Verified project record

lance-format/lance

lance-format/lance is a high-performance lakehouse format built for multimodal AI workloads, providing 100x faster random access, native vector search, and data versioning. It stores images, videos, audio, text, and embeddings in a single unified format while integrating with the existing data ecosystem.

Project overview

It directly addresses a core bottleneck in AI and ML pipelines: traditional lakehouse formats optimized for SQL analytics deliver poor random access and vector search performance. By delivering 100x faster random reads and native vector search alongside multimodal storage, it fills a critical infrastructure gap for teams building retrieval-augmented generation, feature engineering, and custom model training pipelines.

Project type: RAG · Model Development · Data Processing
Use cases: Knowledge Q&A · Data Analysis
Deployment: Refer to project documentation
License: Apache-2.0

Best for

AI engineers and developers who need to store, search, and iterate on multimodal datasets combining media and embeddings with high-performance random access.
Teams building retrieval-augmented generation or custom model training pipelines who want a single format for vector search and analytics without a paid managed service.

Key capabilities

Store images, videos, audio, text, and embeddings in a single unified format with efficient blob encoding and lazy loading.
Efficiently add columns with backfilled values without full table rewrites, designed for ML feature engineering.

Limitations and risks

The 'next' file format alias is unstable and should only be used for experimentation, never for production data.

Getting started

Install easily using standard package managers by running pip install pylance, then import lance in Python to begin working with the format.

Evidence and sources

GitHub project description: Open Lakehouse Format for Multimodal AI. Convert from Parquet in 2 lines of code for 100x faster random access, vector index, and data versioning. Compatible with Pandas, DuckDB,…
README: The key features of Lance include: * **Expressive hybrid search:** Combine vector similarity search, full-text search (BM25), and SQL analytics on the same dataset with accelerate…
README: pip install pylance
README: parquet = pa.dataset.dataset(uri, format='parquet') lance.write_dataset(parquet, "/tmp/test.lance")
README: sift1m.create_index("vector", index_type="IVF_PQ", num_partitions=256, # IVF num_sub_vectors=16) # PQ

Reviewed at: 2026-07-19T18:57:40Z