Orange Wheel: Backend with Flask. Trying out pytest.

Set up Directories & Environments:

To separate out an environment for my backend as well, the first thing I did was:

  • At the root directory of the project, used command mkdir backend to create a directory for backend.
  • Did cd backend to enter to the backend directory.
  • Did pip -m venv venv, which was what Python 3 use to create virtual environment. Python2 should be virtualenv venv. This project would be Python 3.
  • Invoked the virtual environment with . venv/bin/activate. It should work without problem. To deactivate the environment, just use the command deactivate.

Now for the framework.

I chosen Flask for this project, which I installed with pip install Flask. The reason was that backend shouldn’t be too complicated for this project, so a micro-framework known was ideal. There would be fetch call for the external membership website, a scraper to process the data, and maybe some storage to save user preferences.

Trying out Testing in Python:

Since I set up frontend testing, of course I would do backend. I hadn’t done testing in Python framework before though. Since pytest was what the Flask team had in their documentation on test, let’s go with that!

To get test coverage, I also installed pytest-cov. Together, the command was pip install pytest pytest-cov. From now on, if I want to run a test while also view code coverage, I just have to run pytest –cov=src.

I created a very basic scraper/app.py, with an index() that returned a string:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
from flask import Flask


def create_app():
  app = Flask(__name__)

  @app.route('/')
  def index():
    return 'flask index page'

  return app


if __name__=='__main__':
  app = create_app()
  app.run(port=5000)

I then created a test directory with a test_basic.py to test for the string, but that’s not the only file I have to create. I also had to create what was named a conftest.py in the test directory. The pytest automatically discovers conftest.py, which is a file used to share fixture function. I am bit confuse on what fixture function is, but it seems to be a pytest feature that setup and teardown resources in a modular manner. The returned fixture objects are automatically passed into test functions as input arguments? I could be wrong though. If anyone can explain it well, please comment below!

Anyway, the test/test_basic.py:

1
2
3
4
5
6
7
8
9
10
11
12
13
import pytest
from scraper.app import create_app

@pytest.fixture
def client(app):
    #create a test client for app
    return app.test_client()

@pytest.fixture
def app():
    app = create_app()
    app.debug = True
    return app

This creates the client fixture, which is pass as an argument in the test/test_basic.py:

1
2
3
4
5
6
7
8
9
import pytest
from flask import url_for

class TestLanding:
  def test_landing_page(self, client):
    response = client.get('/')
    print(response.data)
    assert response.status_code == 200
    assert response.data.decode('UTF-8') == 'flask index page'

The data in the returned response is actually in byte, so the test failed because the we are asserting against a string. I had seen tutorials where the solution was to do a assert text in response.data, but I prefer a more precise and less hack-y solution. I ended up doing assert response.data.decode(‘UTF-8’) == text.

So now I have a very basic test setup for both frontend and backend. Next up, setting up basic look of the app with Vue!

Orange Wheel: Unit Testing with Jest

Next up on my side project, Orange Wheel, frontend testing!

It’s been my goal since reading Test-Driven Development with Python by Harry J.W. Percival to get into the habit of setting up basic testing before I program contents. This time is no exception.
As mentioned before, I wanted to use Jest.
There are couple of things to focus on when looking for resources on frontend testing.

The Research

  • Use the official documentation: That is especially true for JavaScript’s testing frameworks – they all have excellent documentations. Vue, in addition, is famed for its documentation clarity. Official documentation tends to start with more bare bone installation instead of adding every optional packages in the universe that the writer personally likes, and it is always the most updated version of instruction.
  • Date of Publish: If you decide to use a tutorial, always try for one that is written within a year. The landscape of JavaScript is very fast changing, and the npm requirements changes equally as quick. If an outdated tutorial is used, error could occur and the debugging can get confusing, especially since there tends to be more packages involve with older technologies.
  • Read what the package or program you installed do: Just because a tutorial or documentation tells you to download a package, doesn’t mean you should blindly do it. If you come back to the project, you may have no idea what you did. That is not good if you are trying to explain your project, debug it, or replicate the process for another project. It’s also a good habit as a developer to have an idea of what your third-party package do for the sake of security.

In fact, during this week when I was suppose to work on my unit testing, I was asked to help out with the curriculum material on Enzyme testing for Techtonica. The course intended to use a Medium posting for an in-class exercise. I remember reading the post while looking at the documentation on Jest and Enzyme. There were references to previously required npm package installation and optional packages. As a result, in my slides, I made sure to address it. Following the Medium exercise steps wouldn’t harm the project, but new programming students should know that Enzyme is not that complicated and don’t require those packages or configurations. It would also lessen their confusion when they look at other tutorials and notice that those packages and configurations are not used.

The Installation & Config

To get Jest set up in Vue, there is this simple command:

npm install –save-dev jest @vue/test-utils vue-jest babel-jest

There are also some package.json configuration. In particular, to enable the use of @ as a reference for src folder, I have to add new property, “moduleNameMapper”, to package.json’s “jest” property.

The Reason

  • @vue/test-utils is needed for obvious reason – it’s the utility to enable testing in Vue! Needing to install jest goes without saying, of course.
  • This should be followed by adding “test”: “jest” to package.json file’s “scripts” property.
  • vue-jest is the preprocessor to enable import of single-file component (SFC) built by the Vue.js team.
  • babel-jest adds support for ES modules syntax. Without it, the import statement didn’t work in my case.

The Source

The detail of the installation and configuration are very nicely written in the docs of Vue Test Utils by the Vue.js team. Well-written office documentation rules!!! ✊

Planning & Setup for Frontend of Vue-Based Side Project, Orange Wheel

Time to continue on building my new side project, Orange Wheel! As mentioned in my last post, I have decided to go with plain Vue – no NativeScript or other mobile framework – and Python-based web scraper. Once I start setting up, there are several decision planning that I must do. Here are some of them:

Templates: Selecting a good Vue templates sets a good foundation and makes life easier for beginners. After some searches, I decided to go with one that use the PWA template. Their features can be read on their Github repos, but most are related to the fact that it was designed for a PWA site – read the Google developer docs on PWA here if you are not familiar with the term!

Unit Testing: The PWA template do actually have a possible built-in testing upon installation, which uses PhantomJS with Karma, Mocha, and karma-webpack. However, I have made a decision to use Jest prior to template selection, so I will give that a try and see what happens first.

Directory Structure: At first I just started with all the node modules and npm stuff in root directory like I usually do, but then while setting up, I realize that since I have Python as backend, I really should separate frontend and backend. Right now, in the root directory, there is are files (README and .gitignore) and 2 directories (frontend and backend). Hmm, I somehow lost .git while restructuring, so I will have to link the directory back to my Github repo later.

Icons: Dear god is there many icon images in the PWA template! I actually deleted it at first, not realizing that it was needed for a better PWA experience. I have to re-download them later (new lesson of the day: one can download a directory in Github with command svn export [Github link with “tree/master ” replaced by “trunk”] – well, as long svn is installed in the system in the first place anyway). However, I have to replace the default Vue icons with actual project-relevant icons eventually. So… I have to create an icon, export it in various file format and size? Work this inconvenient yet repetitive have to have a web tool. Let’s google… yep, a simple google search on PWA icons displays a bunch of web tool for this exact purpose. I am cover.

Well, that’s it for now. I still have to do the actual set up of Jest and icon creation. I also have to update my changes to my Github repo. After that, I will set up the backend.

Let’s Try To Build My Locator in NativeScript-Vue!

Building a locator with Mapbox without JS framework had worked out pretty well in my last post. This time, I decided to try it with a JS framework. I have been working with React for a while, so I decided to use a new framework this time -Vue, to be precise.

Since I wanted the app to be used while a bicyclist may be on the road, I wanted the app to work well with mobile. React have React Native. How about Vue? Lo and behold, NativeScript! They even have a tutorial: Include Feature-Rich Maps in a NativeScript-Vue App with Mapbox.

Things I noticed on my first run of the tutorial:

  • NativeScript installation required pod setup, which took forever but didn’t displayed progress bar or related message. I thought the installation got stuck and was quite confused. In the end,  command pod setup –verbose showed me that it was indeed setting up and not getting stuck.
  • The command vue init nativescript-vue/vue-cli-template set a MIT open-source license by default. Nice feature – I usually don’t really think about license when I am building a side project, so this was a good reminder.
  • The vue/devtools package was behind in its electron dependency update, so I got a npm audit alert. When I checked their Github issues, it seemed they were aware of it but haven’t gotten around to fixing it yet.
  • Like pod setup,the npm install also took a while. I downed an entire cup of coffee, and it was still loading! I don’t work with mobile app development framework a lot. Was that normal?
  • Although the tutorial was made only this July, there seemed to be quite a bit of changes for the vue template. For one thing, I saw a hook directory – wasn’t hook only out this or last month? Some changes related to the tutorial content:
    • Directory “src” is now “app“.
    • HelloWorld.vue is now App.vue
    • npm run watch:ios is not working…
    • … the vue template “vue-cli-template” went through some major changes. The one used in the tutorial is now legacy. I should had changed the command for initiating the template to “vue init nativescript-vue/vue-cli-template#legacy <directory-name>” instead. In the new template, instead of npm run watch:ios, the command tns run ios  –bundle should be use instead.

Sadly, after tns run ios –bundle, it tried to run the iPhone emulator in my Macbook, and I was reminded of the main reason why I don’t do much mobile development. My emulator has always been ridiculously slow and would even freeze my Macbook sometimes.

There was tns build ios –bundle, which build and run the code in NativeScript’s iPhone app, but that one also took forever. In fact, it’s still bundling right now as I am writing this post – and I would have to build it every time I want to see changes in this case.

I really just want to learn Vue, but I had spent over a day trying to set up and troubleshoot NativeScript issues. The tns run, tns build, and the various package download and setup took a lot of time. NativeScript isn’t my goal – Vue and Mapbox is. So for my next step, I will take a step back and re-focus on building a web app with Vue.

My First Mapbox Project: Building a Locator

A while I ago, I went to a Map meetup, where I was introduced to Mapbox – I have always been kind of curious about GIS mapping since I interned at SF Bicycle Coalition, where they had another intern worked with GIS maps.

Recently, I finally got back to the codebase that I started during that meetup. The tutorial I did was a map of parks. I wanted to do a locator, and eventually replace the geojson file with park location data with bike shop data – turns out mapping APIs typically uses json-like file call geojson.

Well, not just bike shop – SFBike members get discounts at specific bike shop and even general stores like groceries and book store. I want to do a map of SFBike-related business. That means I would be writing a scraper for the SFBike member page, and then processing Yelp’s API for store hours so the map will display only opened stores.

Lo and behold, Mapbox have a locator tutorial! And while I do that, let put it on Github so I can keep track of my progress!

However, as I code, I came across 2 major problem: how to get a dataset that I already uploaded on Mapbox (most tutorials seems to use local files?) and how to protect my API key in a simple HTML/CSS/JS site without JS framework?

Here’s what I learned and did to troubleshoot:

Retrieving GeoJson Dataset

I tried reading the tutorials first, but it seems to mostly using local files that is already there or add Sources before extracting the sources – that’s whether convoluted…

At one point, I tried queryRenderedFeatures(), but nope, not working. I think the features gets rendered later, so even though I know there is features created from the mapbox tileset, I can’t grab them.

By the way, the flow of mapbox is

  1. Often time, the data are in geoJSON format. GeoJSON can contain FeatureCollection object, which is an object that contain a geometry object and additional properties called features.
  2. In Mapbox, a Dataset is needed, and it is often uploaded GeoJSON.
  3. Dataset creates what’s called a Tileset.
  4. The Tileset can be styled with Style, which is documents that defines the visual appearance of a map.

But first – back to getting the data.

I started searching in the documentation, when I started wondering why there is so many curl when the tutorial never mentions it – only to realize there is a selector for different programming languages on top, JavaScript included.

I resisted the urge to hit the table. The answer was right there!

Here’s what I did to get the uploaded code from Mapbox Studio:

1
2
3
4
5
6
7
8
9
10
11
12
13
map.on('load', function() {
// Retrieve the geoson dataset uploaded in Mapbox Studio
const mapboxClient = mapboxSdk({ accessToken: mapboxgl.accessToken });
mapboxClient.datasets
.listFeatures({
datasetId: 'some datasetId'
})
.send()
.then(
response =&gt; { builLocatioList(response.body) },
error =&gt; console.log(error)
)
})

API Key in Working .env File, No JS Framework Needed

Since I wanted to keep it to JavaScript, I thought I would use a .env to store the key – but don’t I usually use framework with that? I don’t want a framework for now, so let’s search for ways to use .env without framework!

… Damn, all the express and node solutions online looks ways too complicated for such simple needs (I just need an .env file working!). There seems to be a simple one. But once I try to use process.env.APIKEY, it says that process is not defined.

Maybe I need to require the dotenv package? I mean, I did it in server.js, but maybe they need it in index.js?

Now it is saying require is not defined.

Flips table, or at least thinks about it

After some search, I learned that require is for the server side of the app, not the client side. After reading about multiple options, including an hour tweaking with RequireJS, I decided to go with webpack. Basically, I just need add a simple plugin code in my webpack config file:

1
2
3
4
node: { fs: 'empty' },
plugins: [
new Dotenv()
]

The fs: ’empty’ was added because the error code keep saying Can’t resolve fs.

So here are my final file directory structure:
maptime-mapbox-park
-dist
-bundle.js
-node_modules
-public
-index.html
-index.js
-stylesheet.css
-.env
-.gitignore
-package-lock.json
-package.json
-README.md
-webpack.config.js

The steps to create a simple framework-less node app is as follow:

    1. In your project directory root, enter command: touch .env .gitignore README.MD
    2. Inside .gitignore, make sure to include .env and dist. If not, when you push it up to Github, everyone will know your API key! Also add node_modules and any files or directories that you don’t want to upload to Github.
    3. Create a directory named public (some people like app). Inside, create files you needed. You will want index.html and index.js at least.
    4. Now do an npm init -y, which will create a package.json file with default content. You can always change the content later.
    5. Do an npm install express express path. Then npm install -D dotenv-webpack webpack webpack-cli.
    6. Now, time to create your own webpack.config.js file. The output variable will determine what the bundled file output to. In this case, it would be dist/bundle.js.
    7. Since the output bundle file will be at dist/bundle.js, make sure the index.html will use that file instead of index.js. Replace what would normally be <script src=index.js> to <script src = [new bundle file path]>.
    8. Now in index.js, I can use require(‘dotenv’).config();!
    9. Make sure your .env contains your API keys!
    10. When the code is set up, run npm run build to create the bundle file. Everthing looks good? No error? Then go and do a npm run start.
    11. Everything’s setup. You should see your working page in the index.html page.