A feed-forward 3D foundation model for reconstructing 3D scenes from streaming image or video data. It unifies coordinate grounding, dense geometric cues, and long-range drift correction, and supports stable inference at approximately 20 FPS on 518×378 resolution over long sequences exceeding 10,000 frames.
Project overview
The model architecture unifies coordinate grounding, dense geometric cues, and long-range drift correction within a single streaming framework, enabling high-efficiency inference.
Deployment
Refer to project documentation
License
Apache-2.0
Best for
Researchers and developers working on streaming 3D scene reconstruction who need to process image or video data and generate point clouds, visualizations, or flythrough videos.
Key capabilities
Architecturally unifies coordinate grounding, dense geometric cues, and long-range drift correction via anchor context, pose-reference window, and trajectory memory.
Provides a browser-based viser viewer for interactive 3D visualization of reconstructed scenes.
Processes long sequences to produce headless point-cloud flythrough MP4 videos, supporting windowed inference and extensive camera path configurations.
Uses an ONNX sky segmentation model to filter out sky points from the reconstructed point cloud, improving visualization quality for outdoor scenes.
Provides evaluation scripts and preprocessing tools for benchmarks like KITTI and Oxford Spires.
Enables inference on long sequences (>3000 frames) by using a sliding window approach that resets the KV cache.
Allows offloading per-frame predictions to CPU and reducing bidirectional scale frames to avoid out-of-memory issues.
Limitations and risks
Performance degrades when the KV cache stores more than 320 views without a keyframe strategy.
Maximum inference range is bounded by the longest distance seen during training unless state resetting is used.
Users may run into out-of-memory issues on limited GPUs. The system allows offloading per-frame predictions to CPU and reducing bidirectional scale frames to mitigate this.
Getting started
Create a Conda environment and manually install PyTorch with CUDA 12.8 support. Then install the lingbot-map package via pip.
Download the required LingBot-Map model checkpoints and the skyseg.onnx ONNX sky segmentation model from HuggingFace or ModelScope.
Run the demo using the command line interface. An example command processes an image folder with sky masking enabled using the downloaded model checkpoint.