On File Formats

Summary

Blog On File Formats I've been meaning to write something about file format design for a while, and might as well do a blog post. This is pretty unorganized, in a moment thing, so take it as it is. So You Need a File Format There's a few questions you have to ask. Different files have different goals, after all. 1. Does a file format exist for this yet? It's a good idea to check what the prior art is, as it's more than likely that there's already something out there that fits the bill. Let's say you need an image format. There's tons of image formats out there, so one of those might already solve your problem, possibly with tool integration and plenty of library code to boot. But it's also possible you're in need of a image format for something really specific, like a 4-bit cpu driving a custom LCD with a strange pixel order pattern. Hitting that with jpeg is probably not the way to go. So you need something custom. Assuming, again, that the community for that specific use case doesn't already have a standard format. That you don't have a problem with. There are cases where you may want to use existing complex systems, such as SQLite, or something that automatically serializes your data; some programming languages even provide functionality for that. There are pros and cons for these approaches, so you may end up having to design something from scratch. 2. Does it need to be human readable? Parsing text is a pain, so again you need to ask whether you can get away with json, xml, ini or some other commonly used format, and just design fields in it. Just picking one of these formats isn't the end, though, since the structure of the data matters. There may be a reason you don't want to use those formats, in which case you need to make decisions based on your specific use case, and the reasons why you can't use a common format. In this case I would concentrate on minimizing potential errors, both technical and human; Make the format as easy and foolproof to parse as pos...