New OSM file format: 30% smaller than PBF, 5x faster to import

Summary

The OSM dataset is huge, and keeps growing every day. Great news, of course, but sometimes the sheer volume can be overwhelming – there are just gobs and gobs of data! Hence, we created GOB (“Geo-Object Bundle”), a new file format that makes tackling OSM data faster and easier. It’s a companion format to our now-familiar Geo-Object Library (essentially, a tightly-compressed GOL with its indexes stripped). To support this new format, GOL Tool 2.1 has two new commands: save GOLs as GOBs and load GOBs into a GOL (Of course, like all of the GeoDesk Toolkit, the GOL Tool is free & open-source). Advantages of GOB GOB files are on average half the size of a GOL, and 30% smaller than PBFs. Importing a GOB is 5 times faster than building a GOL from a PBF. A modern system loads a planet-size GOB into a GOL in 3 minutes. The speed advantage grows more pronounced on memory-constrained machines: gol build starts paging heavily with less than 32 GB of RAM, whereas gol load requires minimal resources (even a decade-old laptop loads the whole planet in under an hour). GOBs are organized into tiles, so it’s easy to extract regional subsets (basically at file-copy speed) and stitch them back together; that makes GOB a convenient format for archiving and distributing geodata. The image above shows some of the tiling structure, which mimics that of tile renderers. On the left, the smallest squares are zoom 6, the right shows the most granular level (zoom 12). A typical planet GOB has about 60,000 tiles. Below are some size statistics for the planet file and popular regional extracts (without metadata): PBF GOL GOB Planet 65.4 GB 93.6 GB +43.1% 46.0 GB -29.7% California 1.18 GB 1.59 GB +35.0% 770 MB -36.5% France 4.54 GB 5.89 GB +29.7% 2.84 GB -36.3% Germany 4.29 GB 5.92 GB +38.0% 2.67 GB -37.5% Italy 1.96 GB 2.63 GB +34.0% 1.34 GB -31.6% Japan 2.13 GB 2.91 GB +36.1% 1.34 GB -37.0% Poland 1.84 GB 2.72 GB +47.6% 1.29 GB -29.7% Switzerland 487 MB 634 MB +30.1% 311 MB -36.2% Dense, well-ma...