From D3 Meetup, To Jam Session, To Writing A JSON to GeoJSON Convertor

Last Tuesday, I went to my first D3 jam session! I haven’t worked with D3 for a while – since early last year actually. So jam session it is – and Micah Stubb, whom I follow on Twitter, is one of the host! Very exciting!

The Meetup

During most of the session, it was just about getting back into the groove , viewing all the cool example on blockbuilder, and checking out all the suggested links provided in the talk and the meetup page itself. There are:

  • Bl.ock Builder, as mentioned earlier. The typical interactive editor of choice for D3. The site have a lot of examples that can be forked into an account on Bl.ock or gist.
  • Observable, another fun interactive editor. I just skimmed the content for this site though and didn’t really got to try it out.
  • D3 Tutorial Page, which have a lot of tutorials.
  •, the event sponsor. is a MI company that automates the analysis of large volumes of data. The company service itself is not exactly D3 related, but their website looked very cool and they uses D3 in their company.

The Jam Session

I was having problem with forking an example I like on Bl.ock when Micah stopped by. He taught me how to fork on gist instead and suggested an npm package that creates a live re-load localhost server. Here are the steps:

  • The block example I viewed was the Proportional Symbol Scaling on Zoom. The url is It had 3 components: blockbuilder’s domain, username (rgdonohue), and block id (51c43bb749689e696b8a).
  • The blockbuilder domain in the url should be replace with gist’s domain, which resulted in this:
  • The rest is the same as forking a github repo – select the fork option on top right, then clone your own repo.
  • The npm package suggested is a live-server, so this step is just a simple npm install -g live-server.

After that, I wanted to apply a different dataset. I was interested in the new transportation on the raise – scootershare! But I couldn’t found SF-specific data. Ok… how about bikeshare? Well, there was Ford Bikeshare, and they even had it in JSON.

Problem: The sample block used GeoJSON. The dataset was a regular, extremely large JSON file. Manually changing it to GeoJSON would be a pain. I started looking into existing solutions. None of which did exactly what I want.

Well, I am a developer. I can write a script!

The JSON-To-GeoJSON Convertor

The mission is quite simple – I want to input a JSON file, and get a GeoJSON output. I want the convertor to be a command line tool, where I can input a file path, input the require filters, and output a local GeoJSON file.

The format of a GeoJSON file according to the RFC 7946  spec is such:

       "type": "FeatureCollection",
       "features": [{
           "type": "Feature",
           "geometry": {
               "type": "Point",
               "coordinates": [102.0, 0.5]
           "properties": {
               "prop0": "value0"
       }, {
           "type": "Feature",
           "geometry": {
               "type": "LineString",
               "coordinates": [
                   [102.0, 0.0],
                   [103.0, 1.0],
                   [104.0, 0.0],
                   [105.0, 1.0]
           "properties": {
               "prop0": "value0",
               "prop1": 0.0
       }, {
           "type": "Feature",
           "geometry": {
               "type": "Polygon",
               "coordinates": [
                       [100.0, 0.0],
                       [101.0, 0.0],
                       [101.0, 1.0],
                       [100.0, 1.0],
                       [100.0, 0.0]
           "properties": {
               "prop0": "value0",
               "prop1": {
                   "this": "that"

I easily noticed that therewere 2 key data to extract from a JSON file: the data used for the properties, and the latitude-longitude used for coordinates.

For the language choice, I immediately decided on Python. For data processing, Python was known to be a powerful language for it and had a good collection of related packages if I needed them. I also recalled that it had good support for JSON.

Afterward, it’s time for logic building – what questions should the program ask? How would user format their input and how do the program inform them about it? How should incorrect answers, missing input, and file processing errors be handled?

The part about the property keys was particularly problematic. The program needed to reach inside a multi-layer JSON object, so the input needs to inform the program of the object keys to traverse. Say, the object is this:

{ “data”: { “stations”:[{“station_id”: “S01”, “name”: “Johnson Street”}, {“station_id”: “S02”, “name”: “Gary Street”}]}}

The JSON path to transverse and get S01 would be data.stations.0.station_id in JavaScript. In Python, it would be object[“data”][“stations”][0][“station_id”]. The property keys would have to be dynamically added base on input! My initial thought is that I would had to create the object call from input as string first, then use exec() to invoke it – yes, the infamous exec() that is say to never be use.

The initial draft was this if I remember correctly:

cmd = “object”
for key in keys:
    object.append(‘[“‘ + key + ‘”]”)

I really don’t want to use exec() due to the security risk, especially since I always open source my code. Thankfully, after some brainstorming, I came up with a better solution! I would call the properties layer by layer: first is data = object[“data”], then station = data[“stations”], and so on:

for key in keys:
    if key.isdigit():
        key = int(key)
    object = object[key]

After that, it was just getting the program to write the final files. Took me a day to work out all the kinks and test out possible problem, but now I have a JSON to GeoJSON convertor at my disposal at any time. Please feel free to view it and give suggestions at:

Orange Wheel: Backend with Flask. Trying out pytest.

Set up Directories & Environments:

To separate out an environment for my backend as well, the first thing I did was:

  • At the root directory of the project, used command mkdir backend to create a directory for backend.
  • Did cd backend to enter to the backend directory.
  • Did pip -m venv venv, which was what Python 3 use to create virtual environment. Python2 should be virtualenv venv. This project would be Python 3.
  • Invoked the virtual environment with . venv/bin/activate. It should work without problem. To deactivate the environment, just use the command deactivate.

Now for the framework.

I chosen Flask for this project, which I installed with pip install Flask. The reason was that backend shouldn’t be too complicated for this project, so a micro-framework known was ideal. There would be fetch call for the external membership website, a scraper to process the data, and maybe some storage to save user preferences.

Trying out Testing in Python:

Since I set up frontend testing, of course I would do backend. I hadn’t done testing in Python framework before though. Since pytest was what the Flask team had in their documentation on test, let’s go with that!

To get test coverage, I also installed pytest-cov. Together, the command was pip install pytest pytest-cov. From now on, if I want to run a test while also view code coverage, I just have to run pytest –cov=src.

I created a very basic scraper/, with an index() that returned a string:

from flask import Flask

def create_app():
  app = Flask(__name__)

  def index():
    return 'flask index page'

  return app

if __name__=='__main__':
  app = create_app()

I then created a test directory with a to test for the string, but that’s not the only file I have to create. I also had to create what was named a in the test directory. The pytest automatically discovers, which is a file used to share fixture function. I am bit confuse on what fixture function is, but it seems to be a pytest feature that setup and teardown resources in a modular manner. The returned fixture objects are automatically passed into test functions as input arguments? I could be wrong though. If anyone can explain it well, please comment below!

Anyway, the test/

import pytest
from import create_app

def client(app):
    #create a test client for app
    return app.test_client()

def app():
    app = create_app()
    app.debug = True
    return app

This creates the client fixture, which is pass as an argument in the test/

import pytest
from flask import url_for

class TestLanding:
  def test_landing_page(self, client):
    response = client.get('/')
    assert response.status_code == 200
    assert'UTF-8') == 'flask index page'

The data in the returned response is actually in byte, so the test failed because the we are asserting against a string. I had seen tutorials where the solution was to do a assert text in, but I prefer a more precise and less hack-y solution. I ended up doing assert‘UTF-8’) == text.

So now I have a very basic test setup for both frontend and backend. Next up, setting up basic look of the app with Vue!

Errors Encountered Setting Up Django Project

I recently got a project, based in Django. Like all new project, I had to set up the development environment before working on it. The post is about issues I troubleshoot while setting up the development environment:

  • Python Virtual Environment: When a tool like virtualenv is setting up a virtual environment, it creates it base on the computer’s Python version unless we specify a different version in the command. The project was Python 2.7, so the setup encountered errors because my computer use Python 3 natively. If you are not sure what Python your computer is running on, a simple python –version will show it. To ensure there are no error, I use the command virtualenv -p python2.7 venv.
  • OS & Package Dependencies: Apparently Linux require manually installing several dependencies for matplotlib & scipy. At least according to the project README, which was written for Linux. As MacOS user, I didn’t have to worry about that.
  • Compiler in OS: I do, however, have to worry about command invocation not finding my compiler when I doing npm install. To resolve, use command CXX=clang++ npm install.
  • dyld: Library not loaded: executable_path/../.Python” => Homebrew & Python Path: The errors occurs after ./ runserver, and it is a result of broken Python symlink after a Homebrew update I did for another project. Here are the steps to resolve:
    1. Deactivate the virtual environment.
    2. Upgrade the virtual environment tools. In my case, I did pip install –upgrade virtualenv and pip install –upgrade virtualenvwrapper.
    3. Delete broken symlink using find venv -type l -delete while inside the project directory.
    4. Now redo the virtual environment set up. In my case, it was virtualenv -p python2.7 venv => source venv/bin/activate => pip install -r requirements.txt.

Solve node-gyp compatibility issues in npm install

I had to reinstall a project built in Keystone.js – a JavaScript CMS – lately, and I encountered some compatibility issues that I encountered before. I totally forgot about it, and took me a while to realize it was the same issues I encountered before.

Mainly, the issue was node-gyp. Node-gyp is a command-line tool written in Node.js for compiling addon modules for Node.js. It bundles Chrome’s team’s gyp project, which helps with cross-platform issues. Alas, the tool itself ended up causing its own issues!

Node-gyp error
The error message

For me, there are two problem:

  1. It does not support Python 3.
  2. It needs make – a build automation tool – with its variable CXX set to proper C/C++ compiler toolchain.

In my case, my system’s default Python is 3 and the CXX is not always set to the right toolchain. So, first remove the local directory’s node_modules folder if you did a failed npm install earlier, then input this in the command line:

CXX=c++ npm install –python=python2.7

Most node-gyp rebuild error should disappear!

KeystoneJS success
Dadah!!! Keystone.js now running!

What I Have Been Doing 2016-2017

Before I post my 2017 Resolution, let’s list out and organize out what I am doing now. Cause honestly, I think I am doing too much at once. I need to trim down and focus. Ok, so here goes!

  • MIT’s 6.00.1x Introduction to Computer Science and Programming Using Python course
    • Just started this week. I am regretting it a bit now, cause it is asking my to install Anaconda right away, and the last time I did that it messed up my localhost Django practice page – conda does not get along with pip and virtualenv. This is definitely going to be more focus on data science than just Python. I like Python a lot, so a lot of time I forget most people use it for data science, not web development.
  • Functional Programming, cause I want to learn React.
    • This is growing into a monster. I only know basic vanilla JS and jQuery from like 2 years ago when I first got into web dev. One month of study, and I am now on 3 different tutorial cause everyone talks about a different aspect of modern JS. This is what happened:
      1. At first, someone recommended me to watch funfunfunction’s Youtube channel Functional programming in JavaScript. Oh, he’s funny indeed… Oh my god how much had JavaScript changed in the last 2 years!?!?!? Promises? Currying? Map? Filter? What Da H?
      2. Google. Google. How come the functions are all written differently (<— was not fully aware of the whole ES5-ES6-ES7 tangle of mess)
      3. Maybe my foundation just isn’t good. I had never read the holy grail JS book, Eloquent JavaScript. They were just talking about how good it is in the Women Who Code JS Meetup. Let’s do it. Oh, Wes Bos have a really cool video called JavaScript 30? Even better. I will make myself program once a day in JS! Let’s do both.
      4. Combining Eloquent’s Chapter 5 content with Wes Bos’ Array Cardio part 1 and part 2 was really good in helping me to finally understand higher order functions like map() and reduce().
      5. Decided to do Eloquent exercise – which is in ES5 – with ES6/7 instead to learn the differences between the versions.
      6. Wait, wait, wait. Why is object and prototype completely different between ES5 and ES6!? And the constructor! The class! What is even count as object in JS? (<— comes from a C++/PHP7 background)
      7. Found Secrets of the JavaScript Ninja in Barnes & Nobles. Great, another book. But I need it, cause it will explains what on earth is going on with this function-object JavaScript mess.
    • Correction, I am on 1 Youtube channel, 1 video tutorial, and 2 books all at once.
    • Told ya it’s a monster.
  • Doing a side project website of Drupal cause I wanted to brush up on for my internship. Which resulted in learning Bootstrap on Treehouse, cause the theme I selected is based on bootstrap. At least this one’s almost done.
  • The Drupal site is worked on while reading Beginning Drupal 8 by Todd Tomlinson.
  • Revamp of my website, which is PHP.
  • Yea, I am taking on way too much. I think I am going to put a pause on my Drupal project and reading (currently chapter 10)… Should work on my web page first.

Estimating, Mobile, and Websocket on Django

Recently, I went to my first Django Meetup and it was totally worth the trip, even the part where I accidentally took a wrong turn and ended up in Treasure Island instead.


What? The highway entrance to the bridge is literally next to the building! It actually wraps around it, which I thought was pretty cool, but anyway – tip to those going to Four Point: Don’t go up the ramp!

I like the inside too. The Meetup area where the group ate pizza is beneath an interior overlook. Some of the employees had paper decoration hanged up, and they range from pizza to Powerpuff Girls (They really like their pizza, don’t they?). There were rows of wooden benches to the right for the presentation, but for the first half hour, everyone is just enjoying the food.

Having finally started to learn Django and on my first web developer job hunt, it was pretty fun to hear from working Python developer at the table talk about how a Python project typically progress. For example, I got to hear about the typical requests from ecommerce clients and how there are features that most ecommerce clients eventually decides to get even if the clients didn’t thought of it at first. There were discussion of language version craziness (one had a client that use all version of Django starting from… 1.4 I think? Django is 1.10 right now, by the way). Of course, framework differences were also brought up, though being a Django meetup, everyone had found Django to be fairly likeable.

I wouldn’t go into detail about the presentations, since all notes and videos are available to the Meetup event site, Django + Ionic & REST Websockets API with Django Channels , but here are some thoughts:

The first talk was about a Python Estimator. With the name, at first I thought it have to do with finance or concurrency. Turned out it was more about improving the way data scientist shares data with back-end engineers.

The second talk have more to do with the presenter’s journey through building a well-functioning web app via Python and Ionic that tracks food truck – on his spare time. Talk about discipline!

Sam’s talk was about RESTful Socket. I had not read too much about the event profile until the day of the Meetup, so I was surprised to realize that Sam was an engineer at HouseCanary, the company that hosted the hackathon I participated. He was also one of the guys at the table I was sitting in. Realizing that makes me look even more forward to the presentation, since he sounded very knowledgeable at the table.

It is definitely the most technical talk of the day, leaving me much more educated and slightly awestruck about how much more there is to REST and websocket. I had used a bit of REST APIs before, but while I have read some definitely of both of REST and websocket, the definition in relation to their application plus their differences have never really sunk in until this talk. Though I have yet to learn channel, signal, and websocket, the talks today left me eager to learn more.

Deploying my Python Slackbot onto Heroku

I just build a Slack chatbot in Python within a night.

With all the hypes about building a chatbot, that was way easier than I imagine.

I pretty much just followed the tutorial at Full Stack Python: How to Build Your First Slack Bot with Python, which was pretty quick and easy. I hilariously tried to test out my slackbot for 10 minute before realizing I was in localhost. Duh.

So here is the long story version of my troubleshooting:
Most of the time I took to build the slackbot was actually the deployment to Heroku, since I have only ever used it once. When the warning “No default language could be detected for this app. HINT: This occurs when Heroku cannot detect the buildpack to use for this application automatically.” popped up, I was pretty clueless (“What’s a Buildpack?”).

So, I went to the Heroku site and added the Python Buildpack. Since it was giving out error message about not detecting Python, it reminded me that I was working with a virtual environment. Thus, I should do a pip freeze > requirement.txt so the deployment environment knows what to load. So I did that. Let’s git push again…

Python app detected Warning: Your application is missing a Procfile. This file tells Heroku how to run your application… Installing python-2.7.12

Python app is detected, but missing a Procfile? Isn’t that just for Django (my only encounter with Procfile so far is the Django Girls tutorial)? Also, why is it installing Python 2.7.12?

A quick Google revealed that I can create a runtime.txt file to state which Python version I want. I also did some reading on Procfile in term of Heroku and Python deployment. For apps, the example provided was web: gunicorn gettingstarted.wsgi and web: gunicorn hello:app. The later seemed to be more for frameworks like Django, so I went with the later and added web: gunicorn starterbot.wsgi –log-file –. I then pip install gunicorn and redo pip freeze > requirement.txt.

Along the way, I also saw another Heroku deployment tutorial that reminded me that I should use .gitignore. That’s right, I shouldn’t deploy my virtual environment files for security reason. That’s why the “SLACK_BOT_TOKEN” and “BOT_ID” are exported in the first place! So I went and delete my Heroku app, deleted my .git, create the .gitignore and Procfile file, and deploy my app again.

The slackbot, however, didn’t seemed to be running. In the slackchat, the dot that indicate activity status remained grey instead of green. Doing heroku logs revealed that “ImportError: No module named ‘starterbot.wsgi’; ‘starterbot’ is not a package“. So its the Procfile. After some more reading and I narrowed down to the Heroku statement that Procfile “…explicitly declares what command should be executed to start your app.“. So, I tried web: python, because that’s what get the program running, right?

That does get it running! At first, it seems to work well, but the slackbot keeps shutting itself down after a while, and it never respond to my command. Another heroku logs reveals that “Stopping all processes with SIGTERM… Process exited with status 143 …Error R10 (Boot timeout) -> Web process failed to bind to $PORT within 60 seconds of launch… Stopping process with SIGKILL… Process exited with status 137… State changed from starting to crashed“.

It seemed something was wrong with the PORT variable. So, I went to the Heroku site and manually added the PORT config variable, testing the deployment with different port. None seemed to work.

I did a search of “heroku chatbot port”. Surprising, I found the answer in a Node.js tutorial – the author, Luciano Mammino, used “worker: node bin/bot.js” for his Procfile. What would happen if I use “worker: python“?

That got it running, permanently, without any error message about port binding!

My bot is still not outputting anything when I was typing commands though. I added in error checking statement in the if-else statement and eventually realized I have missed a simple colon sign in my command (“@starterbot:”). But once I started adding that, my slackbot worked!

The short, listed version:

  1. heroku create
  2. Find the corresponding app in the Heroku account. Go to Settings and add “SLACK_BOT_TOKEN” and “BOT_ID” to the Config Vars.
  3. pip freeze > requirements.txt
  4. echo “python-3.5.2” > runtime.txt (or whatever Python version you are working with)
  5. echo “worker: python” > Procfile
  6. Add the virtual environment directory name and *.pyc into .gitignore.
  7. Do git add, commit, and push.The bot should be up and running!