HomeBlogBooksProjectsArchiveAboutlogo

A Curious Animal

Born to be curious, born to be animal!

Generate and host your own raster tiles customized with Mapbox Studio

26 July, 2015
- 9 min read

If you have never saw maps hosted in Mapbox platform you would probably agree on the quality of its designs. The business of Mapbox is to host and server geospatial data. For this reason, all the great tools Mapbox facilitates are oriented to help their users to prepare and work with their data. One of the provided tools is Mapbox Studio. Mapbox Studio (MbS) is a desktop application that allows to create CartoCSS themes that are later used to generate raster tiles. Briefly explained, what MbS does is to download OpenStreetMap data in vector format and render it on the fly applying the specified CartoCSS style. The result of working with MbS is not a set of tiles but a style, that is, a set of rules that express which colour must be used to render roads, at which levels must labels appears and with which size, which colour must be used for ground, etc. This style can be later uploaded to Mapbox platform so that raster tiles were generated on the cloud and we can consume the tiles paying for the service. (Hope one day I can contract their services, they deserve by their great job). The question we can make us is: how we can generate the raster tiles locally from a given MbS style? Well, this article is about that. Continue reading. Working with Mapbox Studio and create your custom style Let's start from the beginning so download Mapbox Studio application and install on your system. Once installed execute it and you will be asked to be connected to the Mapbox platform. There are two main reasons why Mapbox requires you to register as a user. First, the power of the platform is on the cloud and the goal is you upload all your data to the servers. That includes the styles you create. Second, MbS retrieves data in vector format from Mapbox servers. When you register as a user you get an API token that identifies your requests. Each time MbS makes a request to extract data it has your token that identifies you as user. This way Mapbox can control if any user is making a bad usage of their platform. Once logged in you will be allowed to create new map styles. The easiest way is to start using one of the starter styles created by the great Mapbox designers: Here we have chose the Mapbox Outdoors style. In the image you can see the style code (CartoCSS which is inspered by CSS) and the resultant tiles obtaining from painting the vector information with the given style rules: CartoCSS is a Mapnik stylesheet pre-processor developed by MapBox and inspired by Cascadenik. It is like a CSS language specially developed to style maps. Store the style with a new name somewhere on your computer, for example, . If you look at your disk you will see a folder has been created containing a bunch of files that defines the style rules (take a look they are not dangerous). Finally, modify some properties, for example or colors and save to see the result: Great !!! You just have created your first custom style. Generating raster tiles from MbS style Looking for a solution I discovered the tessera and tl tools. Tessera is a node based command line application. It is based in some modules from mapbox (concretely tilelive) plus others implemented by the author (Seth Fitzsimmons). The result is we can execute tessera passing a MbS defined style, open a browser pointing to a local address and see a map with the raster tiles generated with our MbS style. Similarly, tl is a node based command line tool we can execute, passing a set of options, to generate a MBTiles file or a pyramid of tiles following the well known format. I know about both tools at the article Converting Mapbox Studio Vector Tiles to Rasters from Azavea Labs. How to install the tools? NOTE: You need to have NodeJS installed in your system, along with the npm package manager command line tools. I don't like to install global node packages (or at least more than the necessary) so I'm going to install the previous tools in a custom folder: {% highlight bash %} mkdir tiletools cd tiletools {% endhighlight %} Inside the directory execute next sentence, which install the and packages among others: {% highlight bash %} npm install tessera tl mbtiles mapnik tilelive tilelive-file tilelive-http tilelive-mapbox tilelive-mapnik tilelive-s3 tilelive-tmsource tilelive-tmstyle tilelive-utfgrid tilelive-vector tilejson {% endhighlight %} You will see a hidden directory named  has been created which contains some subdirectories with the same name as the previous packages. Running tessera Let's try to run tessera for the first time. Because it is installed as a local node module execute: {% highlight bash %} ./node_modules/tessera/bin/tessera.js Usage: node tessera.js uri options uri tilelive URI to serve Options: -C SIZE, --cache-size SIZE Set the cache size (in MB) 10 -c CONFIG, --config CONFIG Provide a configuration file -p PORT, --port PORT Set the HTTP Port 8080 -r MODULE, --require MODULE Require a specific tilelive module -S SIZE, --source-cache-size SIZE Set the source cache size (in # of sources) 10 -v, --version Show version info A tilelive URI or configuration file is required. {% endhighlight %} Tessera requires you pass an URI so it can server its content. It accepts URIs from Mapbox hosted file, Mapnik, Tilemill, Mapbox Studio, ... Repeat again indicating the path to our previously created style indicating the protocol . {% highlight bash %} ./node_modules/tessera/bin/tessera.js tmstyle://./customstyle.tm2 Listening at http://0.0.0.0:8080/ /Users/antonio/Downloads/tiletools/nodemodules/tessera/server.js:43 throw err; ^ Error: A Mapbox access accessToken is required. `export MAPBOXACCESS_TOKEN=...` to set. ... {% endhighlight %} First seems tessera is working at port 8080 but later we get an error about . If you remember from the first section, Mapbox requires all the requests be signed with the user token. So, you need to get the access token from your account and set it as environment variable before execute tessera: {% highlight bash %} export MAPBOXACCESSTOKEN=yourtokenhere ./node_modules/tessera/bin/tessera.js tmstyle://./customstyle.tm2 Listening at http://0.0.0.0:8080/ /Users/antonio/Downloads/tiletools/node_modules/tessera/server.js:43 throw err; ^ Error: Failed to find font face 'Open Sans Bold' in FontSet 'fontset-0' in FontSet {% endhighlight %} We are close to make it work. The problem now is our MbS style is using a font we have not installed in our system. One easy, but brute force, solution is to install all Google Web Fonts on your system. For this purpose you can use the Web Font Load installation script. In my case I have installed them in the user's fonts folder . Once fonts were installed try executing tessera again: {% highlight bash %} ./node_modules/tessera/bin/tessera.js tmstyle://./customstyle.tm2 Listening at http://0.0.0.0:8080/ /Users/antonio/Downloads/tiletools/node_modules/tessera/server.js:43 throw err; ^ Error: Failed to find font face 'Open Sans Bold' in FontSet 'fontset-0' in FontSet {% endhighlight %} That' s a bit strange, we have just installed the fonts but they are not found. What is happening? Well, tessera uses mapnik to create the raster tiles and it looks for fonts in the folders specified by the environment variable , so let define the variable: {% highlight bash %} export MAPNIKFONTPATH=~/Library/Fonts/ {% endhighlight %} and execute the script again: {% highlight bash %} ./node_modules/tessera/bin/tessera.js tmstyle://./customstyle.tm2 Listening at http://0.0.0.0:8080/ /Users/antonio/Downloads/tiletools/node_modules/tessera/server.js:43 throw err; ^ Error: Failed to find font face 'Arial Unicode MS Regular' in FontSet 'fontset-0' in FontSet {% endhighlight %} OMG !!! This seems a never ending story. Now we need to install the Arial Unicode font. Look for it, install in your system and execute tessera again: {% highlight bash %} ./node_modules/tessera/bin/tessera.js tmstyle://./customstyle.tm2 Listening at http://0.0.0.0:8080/ {% endhighlight %} Great !!! It seems tessera is working fine. Let's go to open our browser pointing to and see the result: A map implemented using Leaflet web mapping library is shown, rendering raster tiles that are created in the fly. Look at the console to see the tessera output information: We can see how tiles at current zoom, the zoom level 8, has been generated. At this point we have tessera working but what about generate a local pyramid of tiles for a given zoom levels and a given bounding box? Generating a custom pyramid of tiles with tl command line tool Before continue we need to know which bounding box we want to generate, the whole World? or only a piece. In my case I want three zoom levels (7, 8 and 9) wrapping Catalonia. There are some online tools you can use to get the bbox of a region, but one I like it the Bounding Box Tool from Klokan Technologies. The tl tool can run three main commands but are only interested in the copy one, which copies data between two providers. In our case the MbS style is one provider and the file system is the other. Run the tl command to see the available options: {% highlight bash %} ./node_modules/tl/bin/tl.js copy --help Usage: node tl.js copy options source source URI sink sink URI Options: -v, --version Show version info -b BBOX, --bounds BBOX WGS84 bounding box -180,-85.0511,180,85.0511 -z ZOOM, --min-zoom ZOOM Min zoom (inclusive) 0 -Z ZOOM, --max-zoom ZOOM Max zoom (inclusive) 22 -r MODULE, --require MODULE Require a specific tilelive module -s SCHEME, --scheme SCHEME Copy scheme scanline -i FILE, --info FILE TileJSON copy data between tilelive providers {% endhighlight %} So let's go to execute the command to copy data from our MbS style to the local folder. We want to generate tiles from zoom level 7 to 9 and indicating a bounding box wrapping Catalonia. Remember the options must be indicated as . {% highlight bash %} ./node_modules/tl/bin/tl.js copy -z 7 -Z 9 -b "0.023293972 40.4104003077 3.6146087646 42.9542303723" tmstyle://./customstyle.tm2/ file://./tiles Segmentation fault: 11 {% endhighlight %} Ough !!! That hurts, a segmentation fault. After looking for a while I realised it seems a bug. To solve it go to and remove the folder dependency. It is redundant because there is one installed in parent folder. Execute the command again and see the output: The tl tool has created a local tiles directory and generated all the raster tiles for the given zoom levels and bounding box. The output shows in addition the time required to generate each tile. That's all. Now we only need to host the tiles at our own servers !!!

From SublimeText to Atom text editor

15 July, 2015
- 2 min read

I have used regularly SublimeText (v2) for the past year. I used it mainly to programming in JavaScript, HTML, CSS and to write in Markdown syntax (I'm sorry but to programming in Java NetBeans continues to be my preferred IDE). In that time I used a bunch of SublimeText plugins to help me in my day to day: Markdown Editing. Provides a decent Markdown color scheme (light and dark) with more robust syntax highlighting and useful Markdown editing features. Markdown Preview. Preview and build your markdown files quickly in your web browser. Markdown TOC. Search headings in document and insert/update TOC(Table Of Contents) to it. DocBlockr. Simplifies writing DocBlock comments in many languages, one of them JavaScript. To the previous list we need to attach the spellchecker, autocomplete, minimap, highlight selected text and some more great core features of SublimeText. A week ago I start using Atom text editor (v1.0.2) so I looked for a list of similar plugins that helps me in my day to day in the same way as previous plugins on SublimeText: Minimap, A preview of the full source code. Replicates the minimap core feature of SublimeText. Highlight Selected: Double click on a word to highlight it throughout the open file. Replicated the highlight selected text core feature of SublimeText. Markdown TOC: Generate/updates TOC (table of contents) of headlines from parsed markdown file. DocBlockr: Designed to make writing documentation faster and easier. We need to note by default atom.io has support for markdown syntax, autocompletion and spellchecker too.

The mystery of no flash session variables in Express + Passport auth

30 March, 2015
- 5 min read

Recently I started an application using NodeJS with ExpressJS framework and decided to use passport for authenticate the users. As many other times I wanted to use flash messages so, when user authentication fails, the application shows a message informing about bad credentials. Nothing new on the horizon until.... OMG !!! I can't see the flash messages !!! Disclaimer: This is a really stupid history with me as starring. I like to learn from my errors and because of this I decide to write this post both as a punishment and to ensure I don't forget it again. The crime scene I was working implementing a sign up process, where the user writes its credentials and system creates as a new user or returns an error message like "Sorry, but a username with that email exists" or similar. Before introduce any code, the flow is as follows: User access the page via GET method. Data is sent to resource via POST method, which is responsible to: Check if data is fine, create a new user and redirected to the page. If a user with the same email exists we redirect again to the page (that is, using the GET method) with a flash message related to bad credentials. Note: A flash message is a variable stored within a session that is only available once, for the next request. That is if we put a flash variable and renders a page, the flash variable is available but if we render the same (or other) page again the flash variable is not present (it is destroyed). The approximate code for the previous flow is as follows. First, the next code is responsible to receive the post data and register the user: {% highlight javascript %} // process the signup form router.post('/signup', passport.authenticate('local-signup', { successRedirect : '/profile', // redirect to the secure profile section failureRedirect : '/signup', // redirect back to the signup page if there is an error failureFlash : true // allow flash messages })); The authentication is delegated to passport, which is implemented as:

Reading/writing compressed and not compressed files in Java

03 January, 2015
- 3 min read

Main reason for this post is trying don't repeat yourself (DRY) because, often, I fall in the recursive need to read and write compressed and not compressed files (mainly JSON and CSV). Let's to see first how to read text files. Note I'm working with (relatively small) text files so: The read methods returns an String with the whole content. I'm using  to read line by line. {% highlight java %} private String readFile(String fileName) { StringBuilder sb = new StringBuilder(); try { BufferedReader input = new BufferedReader(new FileReader(new File(fileName))); try { String line = null; while ((line = input.readLine()) != null) { sb.append(line); } } finally { input.close(); } } catch (IOException ex) { // Handle exception return null; } } {% endhighlight %} Note: there are more than one way to do things. In the entry Best way to read a text file, where you can find many different ways to read a text file depending on your JDK version and the size of the file. Similarly to write a String to a file: {% highlight java %} private void writeFile(String fileName, String value) { try { FileWriter fw = new FileWriter(fileName); BufferedWriter bw = new BufferedWriter(fw); bw.write(value); bw.close(); } catch (IOException ex) { // Handle exception } } {% endhighlight %} To read/write compressed files, that is with binary data, we need to work with streams and buffers. So to read a GZIP compressed file and obtain a String: {% highlight java %} private String readCompressedFile(String fileName) { try { GZIPInputStream gis = new GZIPInputStream(new FileInputStream(fileName)); ByteArrayOutputStream fos = new ByteArrayOutputStream(); byte[] buffer = new byte1024; int len; while ((len = gis.read(buffer)) != -1) { fos.write(buffer, 0, len); } fos.close(); gis.close(); return new String(fos.toByteArray()); } catch (IOException ex) { // Handle exception return null; } } {% endhighlight %} and similarly to write a String to a GZip compressed file: {% highlight java %} private void writeCompressedFile(String fileName, String value) { try { InputStream is = new ByteArrayInputStream(value.getBytes()); GZIPOutputStream gzipOS = new GZIPOutputStream(new FileOutputStream(fileName)); byte[] buffer = new byte1024; int len; while ((len = is.read(buffer)) != -1) { gzipOS.write(buffer, 0, len); } gzipOS.close(); is.close(); } catch (IOException ex) { // Handle exception } } {% endhighlight %} References Next you can find a couple of great links with Java code for various JDK versions: Reading and writing text files Reading and writing binary files

Why OpenLayers3 does not render my GeoJSON?

30 December, 2014
- 2 min read

OpenLayers3 offers the class that allows to read data from a GeoJSON source (an URL, a JavaScript object or a text string). Maybe you, like me, has spent some time trying to understand why your GeoJSON data is not rendering properly: projection is fine, your GeoJSON is well formed and validated but OpenLayers3 doesn't return any features and so nothing is rendered in the map. What is the problem? The class is a subclass of that uses an instance to read content: A source class, by definition, acts as a source of features for a vector layers, that is, it is like a container of features. Because of this, the source is limited to read GeoJSON features and not geometries. So next GeoJSON will be ignored by OpenLayers3 (really if you use de debug version you will see an assertion message): {% highlight javascript %} { "type": "Point", "coordinates": -105.01621, 39.57422 } {% endhighlight %} While the next is a valid GeoJSON suitable to be read by the source: {% highlight javascript %} { "type": "Feature", "geometry": { "type": "Point", "coordinates": -105.01621, 39.57422 }, "properties": { "name": "Some feature property" } } {% endhighlight %}  That means we can't read GeoJSON geometries? Absolutely no, simply means source classes follows the source concept, and that means, work with features. The format class allows to read and write features and geometries. To read a GeoJSON file composed of geometries we need to read the geometries and, manually, create a feature for each one. For example: {% highlight javascript %} var point = { "type": "Point", "coordinates": -105.01621, 39.57422 }; // Read the GeoJSON geometries using the format class var format = new ol.format.GeoJSON(); var geometry = format.readGeometry(point); // Create a feature using the geometry var feature = new ol.Feature({ geometry: geometry, propA: 'Some feature property' }); // Add feature to the source of some vector layer vectorLayer.getSource().addFeature(feature); {% endhighlight %}

How the JavaScript heatmap implementation works?

23 December, 2014
- 2 min read

A heatmap is a powerful way to visualise data. Given a matrix of data each value is represented by a color. The implementation of the heatmap algorithm is expensive in computation terms: for each grid's pixel you need to compute its colour from a set of known values. As you can thing, it is not feasible to be implement it on the client side because map rendering would be really slow. But OpenLayers3 comes with a handy class, , which allows to render vector data as a heatmap, so the question is: how it is made? Really, the layer uses a smart approximation to the algorithm which produces great results and is really fast. The steps can be summarised as: A gradient of colors is created as a 1x256 pixel size image. Each known value is rendered in a canvas as a grey blurred point using some radius. This produces a canvas where the blurred points can overlap each other and create more obscure zones. Something similar to this. Finally, an image is obtained from the canvas and for each pixels a color is assigned. The color is obtained from the previous 1x256 pixel image obtained the color specified by the grey value (which goes from 0..255). The coloured image is then rendered in the map canvas, obtaining a nice effect suited to be used for density maps. The offers some properties we can use to play better: , , , and . This last can be configured per feature, allowing to assign a level of importance to each feature determining in more or less measure the final color.