D3, Data Visualization, & Bubble Wrap: Winning at the 2017 PDA Hackathon

Another hackathon? Do I really want to do two hackathon in a month? On the other hand, my last hackathon was as fun as it was intense. I really enjoyed it. Plus, this hackathon is hosted by Stanford library with digital archiving as the theme.

Archiving. Library. My inner bookworm is dancing. (Or wiggling?)

The temptation is just too great.

So, on a sunny Friday afternoon, I drove from San Francisco to Stanford, and my adventure at the 2017 Personal Digital Archiving (PDA) Hackathon commenced!

The hackathon was part of the PDA Conference. While I didn’t got to attend ($120 for late non-student registering), listening to the participants, the conference seemed to be very well done and highly-praised. They sounded so enthusiastic, particularly about the keynotes and agenda selections, that even I felt excited!

For me, the excitement was also twofold compared to a normal hackathon. I obviously enjoyed the chance to code a project with practical usage in a team environment. But I had also been a library assistant, a teen library council member, and an art history minor student. My love for the history of fine art and literature started when I was just a kid. To be with so many that shared the same interest was rejuvenating after so fully engaged into the tech industry in the last 2 years of my career change.

Naturally, when one of the attendee, Katie, talked about her intention to do a project that address email archiving issues related to her work as an art archivist, I was hyped up!

Art + Archiving = I Am In!!!

Our team decided to build a web platform that organizes and analyzes email in a shareable way. Having an art archivist that is passionate about her work really helped directed the project. While I had looked into Stanford’s email archiving software ePADD, without hearing Katie’s perspective as a professional, it was difficult to fully comprehend the importance of analyzing email and maintaining content privacy in the field of digital archiving.

Thus, with the project decided, our team “Girl With A PERL Earring” began our platform, Bubble Wrap!

Bubble Wrap Logo
Our Platform, Bubble Wrap

Katie and Natasha, whom were both interested in art digital archiving, created the wire frame using Proto.io. Lakshmi, whom was interested in analyzing email, tackled extraction method for a set of sample emails. She applied the NLTK toolkit with Python to extract the data into the required JSON file for our project.

Inspired by the MIT Immersion project Katie showed me, I researched into the project’s background. I knew I saw a similar graph built using a D-something library and 4j-someting while attending a DevWeek presentation. I was right. Immersion was a force-directed graph built using the data visualization D3 library and graph database Neo4j.

D3 LogoNeo4J LogoInital research indicates that D3 have a high learning curve. It required understand of jQuery chain methods, objects in JavaScript, CSS to style SVG, reading coordination… wait, I know those.

And the sample codes. It looks understandable.

I think I can do it.

Gosh, it would kill a lot of my brain cell. I mean, an entirely new library in less than 12 hours?

…….

But yes, I can do it. Let’s do it!!!

Thus began an intense all-nighter that I haven’t attempted since I left architecture school. I did got 3 hours of sleep, but the impression that I didn’t sleep apparently got stuck in the event host’s mind. My teammates were also impressed with my energy. I have a feeling the most memorable part the library host and my teammates will remember about me is that I don’t need to sleep.

But hey, the result was worth it. The data visualization I created looked beautiful, and I learned so much!

D3 required a good grasp of SVG and not just basic CSS styling. I struggled with labeling the circle (it seemed to be a known issues. Text couldn’t just be drawn on the circle) and DOM query (SVG have their own set of query method).

I didn’t have time to go through tutorials, so I quickly analyzed sample codes that seemed applicable. From there, I selected specific methods to research. Force, a D3 module, have its own rules and requirements. In particular, it takes the array key, “source” and “target”, from the imported JSON file. I had changed the JSON keys to make it more applicable to our case, so I was confused when nothing was printing.

As morning came, the essential features of my graph seemed completed.

Working on Bubble Wrap's Graph
Working on Bubble Wrap’s Graph. Photo by Josh Schneider.

Then another issues occurred – our presentation is on Proto.io and Google Slide, so how do we present a dynamic demo? At one point, the discussion involved taking snapshot, but it would defeat the purpose of a moving demo. So, I quickly whipped up a Heroku page, looked into the right webpack (static), and pushed my code up. There, a dynamic demo site!

Bubble Wrap Force-Directed Graph
The Force-Directed Graph
Bubble Wrap Selected Network
Bubble Wrap Selected Network with Annotation Generated

By the way, it is now also hosted permanently at my own website, and it has a Github repo for the curious.

Proto.io was beautiful – elegant but clean. Sadly, while Lakshmi got the JSON set up, it was too late to link it to my D3 script. I ended up using a modified dummy code from the D3 tutorial. Fortunately, Lakshmi set a flowchart in advance, so she was able to go over how the Python script work.

Bubble Wrap Proto.io Login Page
Front/Login Page. Our Proto.io demo was created by Katherine Martinez and Natasha Culbreth.

To our excitement, we won a prize! The prize was for – and since it is long, I dared everyone to say this in one go really really fast – “Most innovative project relating to email management or archiving”.

Bubble Wrap Team Photo
Bubble Wrap Team Photo: Natasha Culbreth, Lakshmi Rao, Amy Chan, and Katherine Martinez

In total, I learned a whole new library in less than 12 hours, met new friends, got to hang out with archivists, listened to some really cool conversations, and won a prize. All in 12 hours!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.