From D3 Meetup, To Jam Session, To Writing A JSON to GeoJSON Convertor

Last Tuesday, I went to my first D3 jam session! I haven’t worked with D3 for a while – since early last year actually. So jam session it is – and Micah Stubb, whom I follow on Twitter, is one of the host! Very exciting!

The Meetup

During most of the session, it was just about getting back into the groove , viewing all the cool example on blockbuilder, and checking out all the suggested links provided in the talk and the meetup page itself. There are:

  • Bl.ock Builder, as mentioned earlier. The typical interactive editor of choice for D3. The site have a lot of examples that can be forked into an account on Bl.ock or gist.
  • Observable, another fun interactive editor. I just skimmed the content for this site though and didn’t really got to try it out.
  • D3 Tutorial Page, which have a lot of tutorials.
  • Primer.ai, the event sponsor. Primer.ai is a MI company that automates the analysis of large volumes of data. The company service itself is not exactly D3 related, but their website looked very cool and they uses D3 in their company.

The Jam Session

I was having problem with forking an example I like on Bl.ock when Micah stopped by. He taught me how to fork on gist instead and suggested an npm package that creates a live re-load localhost server. Here are the steps:

  • The block example I viewed was the Proportional Symbol Scaling on Zoom. The url is https://blockbuilder.org/rgdonohue/51c43bb749689e696b8a. It had 3 components: blockbuilder’s domain, username (rgdonohue), and block id (51c43bb749689e696b8a).
  • The blockbuilder domain in the url should be replace with gist’s domain, which resulted in this: https://gist.github.com/rgdonohue/51c43bb749689e696b8a/
  • The rest is the same as forking a github repo – select the fork option on top right, then clone your own repo.
  • The npm package suggested is a live-server, so this step is just a simple npm install -g live-server.

After that, I wanted to apply a different dataset. I was interested in the new transportation on the raise – scootershare! But I couldn’t found SF-specific data. Ok… how about bikeshare? Well, there was Ford Bikeshare, and they even had it in JSON.

Problem: The sample block used GeoJSON. The dataset was a regular, extremely large JSON file. Manually changing it to GeoJSON would be a pain. I started looking into existing solutions. None of which did exactly what I want.

Well, I am a developer. I can write a script!

The JSON-To-GeoJSON Convertor

The mission is quite simple – I want to input a JSON file, and get a GeoJSON output. I want the convertor to be a command line tool, where I can input a file path, input the require filters, and output a local GeoJSON file.

The format of a GeoJSON file according to the RFC 7946  spec is such:

{
       "type": "FeatureCollection",
       "features": [{
           "type": "Feature",
           "geometry": {
               "type": "Point",
               "coordinates": [102.0, 0.5]
           },
           "properties": {
               "prop0": "value0"
           }
       }, {
           "type": "Feature",
           "geometry": {
               "type": "LineString",
               "coordinates": [
                   [102.0, 0.0],
                   [103.0, 1.0],
                   [104.0, 0.0],
                   [105.0, 1.0]
               ]
           },
           "properties": {
               "prop0": "value0",
               "prop1": 0.0
           }
       }, {
           "type": "Feature",
           "geometry": {
               "type": "Polygon",
               "coordinates": [
                   [
                       [100.0, 0.0],
                       [101.0, 0.0],
                       [101.0, 1.0],
                       [100.0, 1.0],
                       [100.0, 0.0]
                   ]
               ]
           },
           "properties": {
               "prop0": "value0",
               "prop1": {
                   "this": "that"
               }
           }
       }]
   }

I easily noticed that therewere 2 key data to extract from a JSON file: the data used for the properties, and the latitude-longitude used for coordinates.

For the language choice, I immediately decided on Python. For data processing, Python was known to be a powerful language for it and had a good collection of related packages if I needed them. I also recalled that it had good support for JSON.

Afterward, it’s time for logic building – what questions should the program ask? How would user format their input and how do the program inform them about it? How should incorrect answers, missing input, and file processing errors be handled?

The part about the property keys was particularly problematic. The program needed to reach inside a multi-layer JSON object, so the input needs to inform the program of the object keys to traverse. Say, the object is this:

{ “data”: { “stations”:[{“station_id”: “S01”, “name”: “Johnson Street”}, {“station_id”: “S02”, “name”: “Gary Street”}]}}

The JSON path to transverse and get S01 would be data.stations.0.station_id in JavaScript. In Python, it would be object[“data”][“stations”][0][“station_id”]. The property keys would have to be dynamically added base on input! My initial thought is that I would had to create the object call from input as string first, then use exec() to invoke it – yes, the infamous exec() that is say to never be use.

The initial draft was this if I remember correctly:

cmd = “object”
for key in keys:
    object.append(‘[“‘ + key + ‘”]”)
exec(object)

I really don’t want to use exec() due to the security risk, especially since I always open source my code. Thankfully, after some brainstorming, I came up with a better solution! I would call the properties layer by layer: first is data = object[“data”], then station = data[“stations”], and so on:

for key in keys:
    if key.isdigit():
        key = int(key)
    object = object[key]

After that, it was just getting the program to write the final files. Took me a day to work out all the kinks and test out possible problem, but now I have a JSON to GeoJSON convertor at my disposal at any time. Please feel free to view it and give suggestions at: https://github.com/amychan331/json-to-geojson

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.