ThalamusDB: Query text, tables, images, and audio

Summary

ThalamusDB: Semantic Queries on Multimodal Data ThalamusDB is an approximate processing engine supporting SQL queries extended with semantic operators on multimodal data. Find the full ThalamusDB documentation here: https://itrummer.github.io/thalamusdb/. Try It on Google Colab To get a first impression of ThalamusDB, try it on Google Colab here. Execute the code cell, enter your OpenAI API key when asked, then enter your queries in the ThalamusDB console. Quick Start Install ThalamusDB using pip: pip install thalamusdb ThalamusDB can use language models from various providers, including OpenAI and Google. Store the access key of the provider you plan to use in an environment variable. For instance, if using OpenAI, set the OPENAI_API_KEY environment variable using the following command on Linux platforms: export OPENAI_API_KEY=[Your OpenAI API Key] Now you can run the ThalamusDB console using the following command: thalamusdb [Path to DuckDB database file] --modelconfigpath=[Path to model configuration file] For instance, try out the example database in this repository: git clone https://github.com/itrummer/thalamusdb cd thalamusdb thalamusdb data/cars.db --modelconfigpath=config/models.json The cars database contains a single table with the following schema: cars(description text , pic text ) The description column contains a text description of images, and the pic column contains the path to the associated image file. Run the following command in the ThalamusDB console to see the picture paths: select pic from cars; You will see relative paths of JPEG images, located in the images sub-folder. Now, you can try semantic queries such as the following: select count ( * ) from cars where nlfilter(pic, ' the car in the picture is red ' ); After less than a minute, ThalamusDB should produce the correct answer (1). You may try more complex queries that require a certain degree of commonsense knowledge to evaluate, e.g.: select count ( * ) from cars where nlfilter(pic, ' ...