From D3 Meetup, To Jam Session, To Writing A JSON to GeoJSON Convertor

Last Tuesday, I went to my first D3 jam session! I haven’t worked with D3 for a while – since early last year actually. So jam session it is – and Micah Stubb, whom I follow on Twitter, is one of the host! Very exciting!

The Meetup

During most of the session, it was just about getting back into the groove , viewing all the cool example on blockbuilder, and checking out all the suggested links provided in the talk and the meetup page itself. There are:

  • Bl.ock Builder, as mentioned earlier. The typical interactive editor of choice for D3. The site have a lot of examples that can be forked into an account on Bl.ock or gist.
  • Observable, another fun interactive editor. I just skimmed the content for this site though and didn’t really got to try it out.
  • D3 Tutorial Page, which have a lot of tutorials.
  • Primer.ai, the event sponsor. Primer.ai is a MI company that automates the analysis of large volumes of data. The company service itself is not exactly D3 related, but their website looked very cool and they uses D3 in their company.

The Jam Session

I was having problem with forking an example I like on Bl.ock when Micah stopped by. He taught me how to fork on gist instead and suggested an npm package that creates a live re-load localhost server. Here are the steps:

  • The block example I viewed was the Proportional Symbol Scaling on Zoom. The url is https://blockbuilder.org/rgdonohue/51c43bb749689e696b8a. It had 3 components: blockbuilder’s domain, username (rgdonohue), and block id (51c43bb749689e696b8a).
  • The blockbuilder domain in the url should be replace with gist’s domain, which resulted in this: https://gist.github.com/rgdonohue/51c43bb749689e696b8a/
  • The rest is the same as forking a github repo – select the fork option on top right, then clone your own repo.
  • The npm package suggested is a live-server, so this step is just a simple npm install -g live-server.

After that, I wanted to apply a different dataset. I was interested in the new transportation on the raise – scootershare! But I couldn’t found SF-specific data. Ok… how about bikeshare? Well, there was Ford Bikeshare, and they even had it in JSON.

Problem: The sample block used GeoJSON. The dataset was a regular, extremely large JSON file. Manually changing it to GeoJSON would be a pain. I started looking into existing solutions. None of which did exactly what I want.

Well, I am a developer. I can write a script!

The JSON-To-GeoJSON Convertor

The mission is quite simple – I want to input a JSON file, and get a GeoJSON output. I want the convertor to be a command line tool, where I can input a file path, input the require filters, and output a local GeoJSON file.

The format of a GeoJSON file according to the RFC 7946  spec is such:

{
       "type": "FeatureCollection",
       "features": [{
           "type": "Feature",
           "geometry": {
               "type": "Point",
               "coordinates": [102.0, 0.5]
           },
           "properties": {
               "prop0": "value0"
           }
       }, {
           "type": "Feature",
           "geometry": {
               "type": "LineString",
               "coordinates": [
                   [102.0, 0.0],
                   [103.0, 1.0],
                   [104.0, 0.0],
                   [105.0, 1.0]
               ]
           },
           "properties": {
               "prop0": "value0",
               "prop1": 0.0
           }
       }, {
           "type": "Feature",
           "geometry": {
               "type": "Polygon",
               "coordinates": [
                   [
                       [100.0, 0.0],
                       [101.0, 0.0],
                       [101.0, 1.0],
                       [100.0, 1.0],
                       [100.0, 0.0]
                   ]
               ]
           },
           "properties": {
               "prop0": "value0",
               "prop1": {
                   "this": "that"
               }
           }
       }]
   }

I easily noticed that therewere 2 key data to extract from a JSON file: the data used for the properties, and the latitude-longitude used for coordinates.

For the language choice, I immediately decided on Python. For data processing, Python was known to be a powerful language for it and had a good collection of related packages if I needed them. I also recalled that it had good support for JSON.

Afterward, it’s time for logic building – what questions should the program ask? How would user format their input and how do the program inform them about it? How should incorrect answers, missing input, and file processing errors be handled?

The part about the property keys was particularly problematic. The program needed to reach inside a multi-layer JSON object, so the input needs to inform the program of the object keys to traverse. Say, the object is this:

{ “data”: { “stations”:[{“station_id”: “S01”, “name”: “Johnson Street”}, {“station_id”: “S02”, “name”: “Gary Street”}]}}

The JSON path to transverse and get S01 would be data.stations.0.station_id in JavaScript. In Python, it would be object[“data”][“stations”][0][“station_id”]. The property keys would have to be dynamically added base on input! My initial thought is that I would had to create the object call from input as string first, then use exec() to invoke it – yes, the infamous exec() that is say to never be use.

The initial draft was this if I remember correctly:

cmd = “object”
for key in keys:
    object.append(‘[“‘ + key + ‘”]”)
exec(object)

I really don’t want to use exec() due to the security risk, especially since I always open source my code. Thankfully, after some brainstorming, I came up with a better solution! I would call the properties layer by layer: first is data = object[“data”], then station = data[“stations”], and so on:

for key in keys:
    if key.isdigit():
        key = int(key)
    object = object[key]

After that, it was just getting the program to write the final files. Took me a day to work out all the kinks and test out possible problem, but now I have a JSON to GeoJSON convertor at my disposal at any time. Please feel free to view it and give suggestions at: https://github.com/amychan331/json-to-geojson

D3, Data Visualization, & Bubble Wrap: Winning at the 2017 PDA Hackathon

Another hackathon? Do I really want to do two hackathon in a month? On the other hand, my last hackathon was as fun as it was intense. I really enjoyed it. Plus, this hackathon is hosted by Stanford library with digital archiving as the theme.

Archiving. Library. My inner bookworm is dancing. (Or wiggling?)

The temptation is just too great.

So, on a sunny Friday afternoon, I drove from San Francisco to Stanford, and my adventure at the 2017 Personal Digital Archiving (PDA) Hackathon commenced!

The hackathon was part of the PDA Conference. While I didn’t got to attend ($120 for late non-student registering), listening to the participants, the conference seemed to be very well done and highly-praised. They sounded so enthusiastic, particularly about the keynotes and agenda selections, that even I felt excited!

For me, the excitement was also twofold compared to a normal hackathon. I obviously enjoyed the chance to code a project with practical usage in a team environment. But I had also been a library assistant, a teen library council member, and an art history minor student. My love for the history of fine art and literature started when I was just a kid. To be with so many that shared the same interest was rejuvenating after so fully engaged into the tech industry in the last 2 years of my career change.

Naturally, when one of the attendee, Katie, talked about her intention to do a project that address email archiving issues related to her work as an art archivist, I was hyped up!

Art + Archiving = I Am In!!!

Our team decided to build a web platform that organizes and analyzes email in a shareable way. Having an art archivist that is passionate about her work really helped directed the project. While I had looked into Stanford’s email archiving software ePADD, without hearing Katie’s perspective as a professional, it was difficult to fully comprehend the importance of analyzing email and maintaining content privacy in the field of digital archiving.

Thus, with the project decided, our team “Girl With A PERL Earring” began our platform, Bubble Wrap!

Bubble Wrap Logo
Our Platform, Bubble Wrap

Katie and Natasha, whom were both interested in art digital archiving, created the wire frame using Proto.io. Lakshmi, whom was interested in analyzing email, tackled extraction method for a set of sample emails. She applied the NLTK toolkit with Python to extract the data into the required JSON file for our project.

Inspired by the MIT Immersion project Katie showed me, I researched into the project’s background. I knew I saw a similar graph built using a D-something library and 4j-someting while attending a DevWeek presentation. I was right. Immersion was a force-directed graph built using the data visualization D3 library and graph database Neo4j.

D3 LogoNeo4J LogoInital research indicates that D3 have a high learning curve. It required understand of jQuery chain methods, objects in JavaScript, CSS to style SVG, reading coordination… wait, I know those.

And the sample codes. It looks understandable.

I think I can do it.

Gosh, it would kill a lot of my brain cell. I mean, an entirely new library in less than 12 hours?

…….

But yes, I can do it. Let’s do it!!!

Thus began an intense all-nighter that I haven’t attempted since I left architecture school. I did got 3 hours of sleep, but the impression that I didn’t sleep apparently got stuck in the event host’s mind. My teammates were also impressed with my energy. I have a feeling the most memorable part the library host and my teammates will remember about me is that I don’t need to sleep.

But hey, the result was worth it. The data visualization I created looked beautiful, and I learned so much!

D3 required a good grasp of SVG and not just basic CSS styling. I struggled with labeling the circle (it seemed to be a known issues. Text couldn’t just be drawn on the circle) and DOM query (SVG have their own set of query method).

I didn’t have time to go through tutorials, so I quickly analyzed sample codes that seemed applicable. From there, I selected specific methods to research. Force, a D3 module, have its own rules and requirements. In particular, it takes the array key, “source” and “target”, from the imported JSON file. I had changed the JSON keys to make it more applicable to our case, so I was confused when nothing was printing.

As morning came, the essential features of my graph seemed completed.

Working on Bubble Wrap's Graph
Working on Bubble Wrap’s Graph. Photo by Josh Schneider.

Then another issues occurred – our presentation is on Proto.io and Google Slide, so how do we present a dynamic demo? At one point, the discussion involved taking snapshot, but it would defeat the purpose of a moving demo. So, I quickly whipped up a Heroku page, looked into the right webpack (static), and pushed my code up. There, a dynamic demo site!

Bubble Wrap Force-Directed Graph
The Force-Directed Graph
Bubble Wrap Selected Network
Bubble Wrap Selected Network with Annotation Generated

By the way, it is now also hosted permanently at my own website, and it has a Github repo for the curious.

Proto.io was beautiful – elegant but clean. Sadly, while Lakshmi got the JSON set up, it was too late to link it to my D3 script. I ended up using a modified dummy code from the D3 tutorial. Fortunately, Lakshmi set a flowchart in advance, so she was able to go over how the Python script work.

Bubble Wrap Proto.io Login Page
Front/Login Page. Our Proto.io demo was created by Katherine Martinez and Natasha Culbreth.

To our excitement, we won a prize! The prize was for – and since it is long, I dared everyone to say this in one go really really fast – “Most innovative project relating to email management or archiving”.

Bubble Wrap Team Photo
Bubble Wrap Team Photo: Natasha Culbreth, Lakshmi Rao, Amy Chan, and Katherine Martinez

In total, I learned a whole new library in less than 12 hours, met new friends, got to hang out with archivists, listened to some really cool conversations, and won a prize. All in 12 hours!

Safety on the Street: The 2017 All-Women Hackathon

Have you ever been harassed on the street?

That was the question of the day for my team and I last Saturday, when  I attended the 2017 All-Women Hackathon hosted by Expat Woman at Pivotal Lab. The Hackathon have 7 challenges, as listed in their Eventbrite:

  • Engage more Girls in STEM
  • End the Gender Gap in Tech
  • Women’s Empowerment
  • Women’s Safety
  • End Gender Violence
  • Teach Boys/Men to Respect/Empower Women
  • Help Immigrant/Refugee women Navigate the US

My team’s chosen challenge is Women’s Safety, and the project idea stemmed from the experience of our team members, when some random guy just started yelling at them outside the street Pivotal Lab. The incident motivated Grace to go up stage and pitch her ideas, which brought our team together, which consisted of Grace, Magda, Daminika, Anju, Steph, and I.

safe-route team
The Safe Route team, hard at work

Our project is a web app would let user to search their destination using the Google Map API, but instead of just seeing the route, they will also see safety ratings. The rating would be generated from a combination of crime data and user feedback, showing keywords that indicates incidents that occurred. Users would be able to filter and customized feedback result base on time of travel and gender. Upon research, I saw that SFPD have Socrata OpenData set up. With the combination of the OpenData and Google Map API, the path to map a safe route in real-time is not far!

To coordinate our pace, Grace took the advice from one of the hackathon mentor to set up a hourly timer, and I started to write a to-do list that will let our teammates assign themselves to different tasks. Even though I introduced myself as a front-to-full developer, interestingly I ended up working with Grace to establish the data parameters that the app will query.

Using a mix of skills in JSON, REST, SQL (for OpenData’s SoQL), and ES6 JavaScript, I programmed a script that will extract the needed data from OpenData. Grace got an Express server running to connect my script result to Dominika and Magda’s part of the program that deals with Google Map API.

Doing our initial brainstorming, deciding on the project’s direction and presentation content took a lot of discussion. Thankfully, we had a good group of mentors who would periodically come by to talk with us.

In the end, we were able to get a Google Slides set up, a demo video recorded, and time to rehearse our presentation, all in 11 hours! Somehow, I ended up being the presenter, which was a bit nervous-inducing. Thankfully, it went smoothly.

safe-route snapshot
Final Product of our web app, Safe Route

Overall, it was fun to work on the project with my teammates, and my experience at this hackathon was very satisfying. We didn’t win the physical prizes, but the experience was a prize of its own.