My First Mapbox Project: Building a Locator

A while I ago, I went to a Map meetup, where I was introduced to Mapbox – I have always been kind of curious about GIS mapping since I interned at SF Bicycle Coalition, where they had another intern worked with GIS maps.

Recently, I finally got back to the codebase that I started during that meetup. The tutorial I did was a map of parks. I wanted to do a locator, and eventually replace the geojson file with park location data with bike shop data – turns out mapping APIs typically uses json-like file call geojson.

Well, not just bike shop – SFBike members get discounts at specific bike shop and even general stores like groceries and book store. I want to do a map of SFBike-related business. That means I would be writing a scraper for the SFBike member page, and then processing Yelp’s API for store hours so the map will display only opened stores.

Lo and behold, Mapbox have a locator tutorial! And while I do that, let put it on Github so I can keep track of my progress!

However, as I code, I came across 2 major problem: how to get a dataset that I already uploaded on Mapbox (most tutorials seems to use local files?) and how to protect my API key in a simple HTML/CSS/JS site without JS framework?

Here’s what I learned and did to troubleshoot:

Retrieving GeoJson Dataset

I tried reading the tutorials first, but it seems to mostly using local files that is already there or add Sources before extracting the sources – that’s whether convoluted…

At one point, I tried queryRenderedFeatures(), but nope, not working. I think the features gets rendered later, so even though I know there is features created from the mapbox tileset, I can’t grab them.

By the way, the flow of mapbox is

  1. Often time, the data are in geoJSON format. GeoJSON can contain FeatureCollection object, which is an object that contain a geometry object and additional properties called features.
  2. In Mapbox, a Dataset is needed, and it is often uploaded GeoJSON.
  3. Dataset creates what’s called a Tileset.
  4. The Tileset can be styled with Style, which is documents that defines the visual appearance of a map.

But first – back to getting the data.

I started searching in the documentation, when I started wondering why there is so many curl when the tutorial never mentions it – only to realize there is a selector for different programming languages on top, JavaScript included.

I resisted the urge to hit the table. The answer was right there!

Here’s what I did to get the uploaded code from Mapbox Studio:

1
2
3
4
5
6
7
8
9
10
11
12
13
map.on('load', function() {
// Retrieve the geoson dataset uploaded in Mapbox Studio
const mapboxClient = mapboxSdk({ accessToken: mapboxgl.accessToken });
mapboxClient.datasets
.listFeatures({
datasetId: 'some datasetId'
})
.send()
.then(
response => { builLocatioList(response.body) },
error => console.log(error)
)
})

API Key in Working .env File, No JS Framework Needed

Since I wanted to keep it to JavaScript, I thought I would use a .env to store the key – but don’t I usually use framework with that? I don’t want a framework for now, so let’s search for ways to use .env without framework!

… Damn, all the express and node solutions online looks ways too complicated for such simple needs (I just need an .env file working!). There seems to be a simple one. But once I try to use process.env.APIKEY, it says that process is not defined.

Maybe I need to require the dotenv package? I mean, I did it in server.js, but maybe they need it in index.js?

Now it is saying require is not defined.

*Flips table, or at least thinks about it*

After some search, I learned that require is for the server side of the app, not the client side. After reading about multiple options, including an hour tweaking with RequireJS, I decided to go with webpack. Basically, I just need add a simple plugin code in my webpack config file:

1
2
3
4
node: { fs: 'empty' },
plugins: [
new Dotenv()
]

The fs: ’empty’ was added because the error code keep saying Can’t resolve fs.

So here are my final file directory structure:
maptime-mapbox-park
-dist
-bundle.js
-node_modules
-public
-index.html
-index.js
-stylesheet.css
-.env
-.gitignore
-package-lock.json
-package.json
-README.md
-webpack.config.js

The steps to create a simple framework-less node app is as follow:

    1. In your project directory root, enter command: touch .env .gitignore README.MD
    2. Inside .gitignore, make sure to include .env and dist. If not, when you push it up to Github, everyone will know your API key! Also add node_modules and any files or directories that you don’t want to upload to Github.
    3. Create a directory named public (some people like app). Inside, create files you needed. You will want index.html and index.js at least.
    4. Now do an npm init -y, which will create a package.json file with default content. You can always change the content later.
    5. Do an npm install express express path. Then npm install -D dotenv-webpack webpack webpack-cli.
    6. Now, time to create your own webpack.config.js file. The output variable will determine what the bundled file output to. In this case, it would be dist/bundle.js.
    7. Since the output bundle file will be at dist/bundle.js, make sure the index.html will use that file instead of index.js. Replace what would normally be <script src=index.js> to <script src = [new bundle file path]>.
    8. Now in index.js, I can use require(‘dotenv’).config();!
    9. Make sure your .env contains your API keys!
    10. When the code is set up, run npm run build to create the bundle file. Everthing looks good? No error? Then go and do a npm run start.
    11. Everything’s setup. You should see your working page in the index.html page.

D3, Data Visualization, & Bubble Wrap: Winning at the 2017 PDA Hackathon

Another hackathon? Do I really want to do two hackathon in a month? On the other hand, my last hackathon was as fun as it was intense. I really enjoyed it. Plus, this hackathon is hosted by Stanford library with digital archiving as the theme.

Archiving. Library. My inner bookworm is dancing. (Or wiggling?)

The temptation is just too great.

So, on a sunny Friday afternoon, I drove from San Francisco to Stanford, and my adventure at the 2017 Personal Digital Archiving (PDA) Hackathon commenced!

The hackathon was part of the PDA Conference. While I didn’t got to attend ($120 for late non-student registering), listening to the participants, the conference seemed to be very well done and highly-praised. They sounded so enthusiastic, particularly about the keynotes and agenda selections, that even I felt excited!

For me, the excitement was also twofold compared to a normal hackathon. I obviously enjoyed the chance to code a project with practical usage in a team environment. But I had also been a library assistant, a teen library council member, and an art history minor student. My love for the history of fine art and literature started when I was just a kid. To be with so many that shared the same interest was rejuvenating after so fully engaged into the tech industry in the last 2 years of my career change.

Naturally, when one of the attendee, Katie, talked about her intention to do a project that address email archiving issues related to her work as an art archivist, I was hyped up!

Art + Archiving = I Am In!!!

Our team decided to build a web platform that organizes and analyzes email in a shareable way. Having an art archivist that is passionate about her work really helped directed the project. While I had looked into Stanford’s email archiving software ePADD, without hearing Katie’s perspective as a professional, it was difficult to fully comprehend the importance of analyzing email and maintaining content privacy in the field of digital archiving.

Thus, with the project decided, our team “Girl With A PERL Earring” began our platform, Bubble Wrap!

Bubble Wrap Logo
Our Platform, Bubble Wrap

Katie and Natasha, whom were both interested in art digital archiving, created the wire frame using Proto.io. Lakshmi, whom was interested in analyzing email, tackled extraction method for a set of sample emails. She applied the NLTK toolkit with Python to extract the data into the required JSON file for our project.

Inspired by the MIT Immersion project Katie showed me, I researched into the project’s background. I knew I saw a similar graph built using a D-something library and 4j-someting while attending a DevWeek presentation. I was right. Immersion was a force-directed graph built using the data visualization D3 library and graph database Neo4j.

D3 LogoNeo4J LogoInital research indicates that D3 have a high learning curve. It required understand of jQuery chain methods, objects in JavaScript, CSS to style SVG, reading coordination… wait, I know those.

And the sample codes. It looks understandable.

I think I can do it.

Gosh, it would kill a lot of my brain cell. I mean, an entirely new library in less than 12 hours?

…….

But yes, I can do it. Let’s do it!!!

Thus began an intense all-nighter that I haven’t attempted since I left architecture school. I did got 3 hours of sleep, but the impression that I didn’t sleep apparently got stuck in the event host’s mind. My teammates were also impressed with my energy. I have a feeling the most memorable part the library host and my teammates will remember about me is that I don’t need to sleep.

But hey, the result was worth it. The data visualization I created looked beautiful, and I learned so much!

D3 required a good grasp of SVG and not just basic CSS styling. I struggled with labeling the circle (it seemed to be a known issues. Text couldn’t just be drawn on the circle) and DOM query (SVG have their own set of query method).

I didn’t have time to go through tutorials, so I quickly analyzed sample codes that seemed applicable. From there, I selected specific methods to research. Force, a D3 module, have its own rules and requirements. In particular, it takes the array key, “source” and “target”, from the imported JSON file. I had changed the JSON keys to make it more applicable to our case, so I was confused when nothing was printing.

As morning came, the essential features of my graph seemed completed.

Working on Bubble Wrap's Graph
Working on Bubble Wrap’s Graph. Photo by Josh Schneider.

Then another issues occurred – our presentation is on Proto.io and Google Slide, so how do we present a dynamic demo? At one point, the discussion involved taking snapshot, but it would defeat the purpose of a moving demo. So, I quickly whipped up a Heroku page, looked into the right webpack (static), and pushed my code up. There, a dynamic demo site!

Bubble Wrap Force-Directed Graph
The Force-Directed Graph
Bubble Wrap Selected Network
Bubble Wrap Selected Network with Annotation Generated

By the way, it is now also hosted permanently at my own website, and it has a Github repo for the curious.

Proto.io was beautiful – elegant but clean. Sadly, while Lakshmi got the JSON set up, it was too late to link it to my D3 script. I ended up using a modified dummy code from the D3 tutorial. Fortunately, Lakshmi set a flowchart in advance, so she was able to go over how the Python script work.

Bubble Wrap Proto.io Login Page
Front/Login Page. Our Proto.io demo was created by Katherine Martinez and Natasha Culbreth.

To our excitement, we won a prize! The prize was for – and since it is long, I dared everyone to say this in one go really really fast – “Most innovative project relating to email management or archiving”.

Bubble Wrap Team Photo
Bubble Wrap Team Photo: Natasha Culbreth, Lakshmi Rao, Amy Chan, and Katherine Martinez

In total, I learned a whole new library in less than 12 hours, met new friends, got to hang out with archivists, listened to some really cool conversations, and won a prize. All in 12 hours!

Safety on the Street: The 2017 All-Women Hackathon

Have you ever been harassed on the street?

That was the question of the day for my team and I last Saturday, when  I attended the 2017 All-Women Hackathon hosted by Expat Woman at Pivotal Lab. The Hackathon have 7 challenges, as listed in their Eventbrite:

  • Engage more Girls in STEM
  • End the Gender Gap in Tech
  • Women’s Empowerment
  • Women’s Safety
  • End Gender Violence
  • Teach Boys/Men to Respect/Empower Women
  • Help Immigrant/Refugee women Navigate the US

My team’s chosen challenge is Women’s Safety, and the project idea stemmed from the experience of our team members, when some random guy just started yelling at them outside the street Pivotal Lab. The incident motivated Grace to go up stage and pitch her ideas, which brought our team together, which consisted of Grace, Magda, Daminika, Anju, Steph, and I.

safe-route team
The Safe Route team, hard at work

Our project is a web app would let user to search their destination using the Google Map API, but instead of just seeing the route, they will also see safety ratings. The rating would be generated from a combination of crime data and user feedback, showing keywords that indicates incidents that occurred. Users would be able to filter and customized feedback result base on time of travel and gender. Upon research, I saw that SFPD have Socrata OpenData set up. With the combination of the OpenData and Google Map API, the path to map a safe route in real-time is not far!

To coordinate our pace, Grace took the advice from one of the hackathon mentor to set up a hourly timer, and I started to write a to-do list that will let our teammates assign themselves to different tasks. Even though I introduced myself as a front-to-full developer, interestingly I ended up working with Grace to establish the data parameters that the app will query.

Using a mix of skills in JSON, REST, SQL (for OpenData’s SoQL), and ES6 JavaScript, I programmed a script that will extract the needed data from OpenData. Grace got an Express server running to connect my script result to Dominika and Magda’s part of the program that deals with Google Map API.

Doing our initial brainstorming, deciding on the project’s direction and presentation content took a lot of discussion. Thankfully, we had a good group of mentors who would periodically come by to talk with us.

In the end, we were able to get a Google Slides set up, a demo video recorded, and time to rehearse our presentation, all in 11 hours! Somehow, I ended up being the presenter, which was a bit nervous-inducing. Thankfully, it went smoothly.

safe-route snapshot
Final Product of our web app, Safe Route

Overall, it was fun to work on the project with my teammates, and my experience at this hackathon was very satisfying. We didn’t win the physical prizes, but the experience was a prize of its own.

What I Have Been Doing 2016-2017

Before I post my 2017 Resolution, let’s list out and organize out what I am doing now. Cause honestly, I think I am doing too much at once. I need to trim down and focus. Ok, so here goes!

  • MIT’s 6.00.1x Introduction to Computer Science and Programming Using Python course
    • Just started this week. I am regretting it a bit now, cause it is asking my to install Anaconda right away, and the last time I did that it messed up my localhost Django practice page – conda does not get along with pip and virtualenv. This is definitely going to be more focus on data science than just Python. I like Python a lot, so a lot of time I forget most people use it for data science, not web development.
  • Functional Programming, cause I want to learn React.
    • This is growing into a monster. I only know basic vanilla JS and jQuery from like 2 years ago when I first got into web dev. One month of study, and I am now on 3 different tutorial cause everyone talks about a different aspect of modern JS. This is what happened:
      1. At first, someone recommended me to watch funfunfunction’s Youtube channel Functional programming in JavaScript. Oh, he’s funny indeed… Oh my god how much had JavaScript changed in the last 2 years!?!?!? Promises? Currying? Map? Filter? What Da H?
      2. Google. Google. How come the functions are all written differently (<— was not fully aware of the whole ES5-ES6-ES7 tangle of mess)
      3. Maybe my foundation just isn’t good. I had never read the holy grail JS book, Eloquent JavaScript. They were just talking about how good it is in the Women Who Code JS Meetup. Let’s do it. Oh, Wes Bos have a really cool video called JavaScript 30? Even better. I will make myself program once a day in JS! Let’s do both.
      4. Combining Eloquent’s Chapter 5 content with Wes Bos’ Array Cardio part 1 and part 2 was really good in helping me to finally understand higher order functions like map() and reduce().
      5. Decided to do Eloquent exercise – which is in ES5 – with ES6/7 instead to learn the differences between the versions.
      6. Wait, wait, wait. Why is object and prototype completely different between ES5 and ES6!? And the constructor! The class! What is even count as object in JS? (<— comes from a C++/PHP7 background)
      7. Found Secrets of the JavaScript Ninja in Barnes & Nobles. Great, another book. But I need it, cause it will explains what on earth is going on with this function-object JavaScript mess.
    • Correction, I am on 1 Youtube channel, 1 video tutorial, and 2 books all at once.
    • Told ya it’s a monster.
  • Doing a side project website of Drupal cause I wanted to brush up on for my internship. Which resulted in learning Bootstrap on Treehouse, cause the theme I selected is based on bootstrap. At least this one’s almost done.
  • The Drupal site is worked on while reading Beginning Drupal 8 by Todd Tomlinson.
  • Revamp of my website, which is PHP.
  • Yea, I am taking on way too much. I think I am going to put a pause on my Drupal project and reading (currently chapter 10)… Should work on my web page first.

Summer 2015 So F ar

One of my goal this summer is work on my project, which ended up with me taking a long detour in both brushing up and learning new skills in JavaScript, JQuery, Ajax, PHP, MVC, and OOP. There was a lot of reading, online video/school, and coding. I have sadly ddiscovered that I have learned more about Ajax in a 2-days self-study using Treehouse video and books than I did a semester of class, but hey, some classes are better than other. It’s been good, although I ended up pushing back my C and Anduino learning.

IT network self-study has been going well. Since I am taking the same professor as I had in my Introduction to Networking for the next semester course in Network Security, I feel that I should brush up on my network knowledge. When I took my first network class over half year ago, I was new to CS and have no idea what is going on. I am half-way through re-reading my books and notes. Now that I have a lot more experience, it makes much more sense, and I feel more comfortable about jumping back to a networking class this Fall.

In term of Fedora, because of that fact that half of my self-study files are local, I lacked a reason to even turn on Virtualbox. But no more. Most of my learning files and even a good portion of my project files are now on Dropbox. I installed both Dropbox and Sublime Text on Fedora, so I can now do most of my coding and studies in Fedora!

Although, Fedora have problem waking up from its sleep in Virtualbox. It would either remain the black screen of death or refuse to let me type anything. In addition, I am consistently clicking on command-C instead of control-C. In Virtualbox, control-C actually exits the Virtualbox out of its full-screen mode instead of copying text. As someone who loves shortcut key and types pretty quick, it’s driving me a bit crazy. I pretty much do a command-key shortcut every 10 minutes when I started using Virtualbox, then when I do return to my Mac OS, I would do the opposite! (As in using control-key for shortcut every 10 minutes). For the first time in my life, I curse my ability to type so quickly…

jQuerySF 2015, Day One Reflection

A two day conference solely dedicated to a jQuery? I know jQuery is an extremely popular library, but I never expected a whole conference dedicated to it! I am still new to JS, especially since I have been taking more back-end language courses lately, but I really would like to brush up on my front-end skills.

Plus, there are lunch and coffee, those are always good incentive!

I had volunteer work that day, so I have to miss the morning. I don’t regret it though – turns out the Americorp people are leaving this week! I am so glad I didn’t missed my last chance to say goodbye. I really enjoyed working with the group this time! They are always so positive and cheerful!

At 12, I leave for the conference after a lot of hugs. I was just in time for the lunch time food truck. I grabbed a bite, settled down, and waited patiently for the next set of speakers.

In jQuerySF, the panels are divided by topics. First up at 2:30 is:

Why Empathize? The speakers are Amanda Glossom, John Resig, and Kelly King.

I like how Amanda began with an explanation of what empathy is: “Empathy is imagining what is like for them in their shoes”. It’s not imagining what is like for us be in their shoes, but what it is like for that individual to be in their shoes. Her ending of “End user isn’t the only one who deserve our empathy. We are people too. ” was also particularly memorable. Very often we get obsess with the end customer, we end up losing those people that are needed to make the whole thing work.

As a JS newbie, the name John Resig sounded familiar. Upon reading his bio, I realized why – he is the author of Secrets of the JavaScript Ninja, a book I have been eying at Barnes & Nobles. His panel did not disappoint. It drawn from his experience during the early days of jQuery. Having read various code language debates, I found it refreshing that John disagreed with the idea that certain code languages are better, and that things that aren’t code can be what differentiate between the different coding language. One of the things he pointed out is level of difficulty to get help from a language community, especially among people just starting to learn the language. His focus of how important community is in the development of a language made his panel particularly useful for those involved in building a language.

The last empathy panel is Kelly King, whom talked about A/B testing. One of the most memorable part for me was when she posted the top liked comment on Netflix, “The new Netflix interface is completely crap.” It sent the room into laughter. The irony though, Kelly revealed, was that A/B testing shows that the new interface actually created better result. As one of my science teacher once told me, science is not always common sense. With A/B testing, they can avoid situations where the decisions are determined based on HiPPO – High Paid Person (Nice name!).

The Importance of Accessibility: The next group of panel consisted of Karo Caran, Jon Kuperman, and Victor Tsaran.

In discussing about accessibility, Karo mentioned not only those who are born with low vision, but also elder and, most interestingly, the invisible audience who wouldn’t acknowledge that they have low vision. She presented 4 pictures of how a person with visual disability might view a laptop screen. The video about how she navigated a website was eye-opening. In the video, she mentioned that she can only see the colors of an app and must magnify for detail. She also mentioned several times how the stripes of color from the navigation bar makes it easier for her. Those seemed like such a small detail when designing a website, but for some people it really makes a difference in their experience of the site.

I like Jon Kuperman’s definition of what accessibility design is, especially the part about how it should create a way that everyone “can contribute to the Web”. His misunderstanding about text-reader was hilarious to read about but also kind of sad. It really does brought to the point of how little most people understand about accessibility. I was fortunate enough that one of my professor did taught us basic accessibility design. However, some of the tools and design considerations that Jon mentioned was never made aware to me before.

So far, all the panels has been inspiring and informational, but Victor Tsaran’s was most educational because I have never even heard of Aria. Turned out ARIA stands for Advanced Rich Internet Applications. It is created to deal with Ajax issues in term of accessibility. It was a short 20-minute, so Victor only got to brush the basic surface of what Aria do, but it is definitely something I hope to look up later.

The last is the keynote – and was it some keynote!

JavaScript State of the Union: The keynote was by Steve Newcomb.

There are times when video and reading can not make up for being in person. This is one of those moment. The combination of his charismatic speech and the presentation of the demo was just amazing. Steve Newcomb has sought to bring the JS language to the three-dimensional, to make browser language compatible with native. I never imagine that something like what the demo showed can be build by JS and jQuery. Listening to him made me wanted to code more projects right after the panel. He was a powerful speaker. His presentation is inspiring and made one look forward to the future of Javascript.

End of Day One

There was so much just on the first day, despite the fact that I have to miss the morning panel. I will try to post a second post later this week, but there are so many notes to process…

For those who missed, there is a video of the conferences at jquerysf.com!