TreeStore: Endowing Your Data with Hierarchical Structure

Summary

When working with large and complex datasets, having a way to organize your data efficiently is crucial. blosc2.TreeStore is a powerful feature in the blosc2 library that allows you to store and manage your compressed arrays in a hierarchical, tree-like structure, much like a filesystem. This container, typically saved with a .b2z extension, can hold not only blosc2.NDArray or blosc2.SChunk objects but also metadata, making it a versatile tool for data organization. What is a TreeStore? A TreeStore lets you arrange your data into groups (like directories) and datasets (like files). Each dataset is a blosc2.NDArray or blosc2.SChunk instance, benefiting from Blosc2's high-performance compression. This structure is ideal for scenarios where data has a natural hierarchy, such as in scientific experiments, simulations, or any project with multiple related datasets. Basic Usage: Creating and Populating a TreeStore Creating a TreeStore is straightforward. You can use a with statement to ensure the store is properly managed. Inside the with block, you can create groups and datasets using a path-like syntax. import blosc2 import numpy as np # Create a new TreeStore with blosc2.TreeStore("my_experiment.b2z", mode="w") as ts: # You can store numpy arrays, which are converted to blosc2.NDArray ts["/dataset0"] = np.arange(100) # Create a group with a dataset that can be a blosc2 NDArray ts["/group1/dataset1"] = blosc2.zeros((10,)) # You can also store blosc2 arrays directly (vlmeta included) ext = blosc2.linspace(0, 1, 10_000, dtype=np.float32) ext.vlmeta["desc"] = "dataset2 metadata" ts["/group1/dataset2"] = ext In this example, we created a TreeStore in a file named my_experiment.b2z. It contains two groups, root and group1, each holding datasets. Reading from a TreeStore To access the data, you open the TreeStore in read mode ('r') and use the same path-like keys to retrieve your arrays. # Open the TreeStore in read-only mode ('r') with blosc2.TreeStore("my_experiment.b2z", m...