UIT – performant, modular, low-memory file processing at scale, in the Cloud

Summary

UIT - Universal Information Terminal UIT is a library for performant, modular, low-memory file processing at scale, in the Cloud. It works by offering a 4-step process to gather a file hierarchy from any desired modality, apply filters and transformations, and output it in any desired modality. performance : speed is of essence when navigating and searching through large amounts of data : speed is of essence when navigating and searching through large amounts of data low-memory by applying streaming and parallelization we can run this in low-memory environments such as Cloudflare workers by applying streaming and parallelization we can run this in low-memory environments such as Cloudflare workers modular: modularity is beneficial because by making it composable we get a clear high-level overview of all building blocks. also, not all building blocks can be ran in the same runtime or location. Important This is an early pre-release. See TODO for current challenges. Try the demo at https://uuithub.com UIT has come about after many iterations of the platform of uithub, which started as a simple node-based parser of zipfiles. While building more and more features and add-ons, I found myself limited by the memory a lot as I was not streaming enough, and going back to JSON too early (because using the Streams API is tricky!). Thus, as features and complexity grew the need was born to create a more modular extensible architecture with good serverless practices in mind. FormData has a long history [RFC 1867 (1995)] [RFC 2388 (1998)] [RFC 7578 (2015)] and is deeply embedded into the web. It offers an excellent way to serve multiple files, binary and textual, over a single request. Although FormData does not support stream-reading directly from Request and other Web Standards yet, UIT leverages the fact that intermediate results can be read using the Streams API using multipart-formdata-stream-js. UIT cleverly modularizes filters and transformations on file hierarchies by pro...