How to build vector tiles from scratch

Summary

As I add more data to the NYC Chaos Dashboard, a website that maps live urban activity, I have been looking for a more efficient way to render the map. Since I collect all of the data in one process and return the Dashboard as one HTML file, I kept wondering how I could optimize the map’s loading time by pre-processing the data as much as possible in the backend. This is where vector tiles come in. The code shown in this post is written in Go. Why generate tiles? Initially, all of the map’s data was passed to the rendering library in GeoJSON format (embedded directly in the HTML file). For those who don’t know, GeoJSON is a JSON based standard to represent geographic information. You can go see the full RFC here, but here’s a quick preview of what it looks like so you can get an idea: { "type": "Feature", "geometry": { "type": "Point", "coordinates": [-74.04452395542852, 40.68987850656795] }, "properties": { "name": "Statue of Liberty", "status": "open" } } Now, I’m definitely not JSON’s greatest fan. It’s all text, meaning that a number is stored in a base 10 ASCII representation, where a number like 42 gets stored as "4" and "2". I could go on, but I think you see the problem: it’s not the most efficient way to store data. Nonetheless, JSON has a lot of merits: it’s human-readable and easy to share between systems, so I find myself using it more than I’d like to - more often than not, simplicity is the way to go, and a simple format like GeoJSON just gets the job done, and that alone makes it a worthy geographic standard. So what happens when I start adding more layers to the map? It gets slow. I’m working on adding flood sensor data (thank you Floodnet for granting me access to the API), LIRR and MetroNorth data, NYISO power data, and many more datasets which will start adding a lot of layers to the map. I can already see that the HTML file, at the time of writing, is 4.5Mb (once decompressed) and takes ~770ms to transfer from Cloudflare’s CDN to my browser. This...