Streaming reconstruction

🎬 Interactive Demo (demo.py)

Run demo.py for interactive 3D visualization via a browser-based viser viewer (default http://localhost:8080).

Try the Example Scenes

The example/ directory contains four scenes that can be run directly:

# courthouse scene
python demo.py --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/courthouse --mask_sky

# University scene
python demo.py --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/university --mask_sky

# Loop scene (loop closure trajectory)
python demo.py --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/loop

# Oxford scene with sky masking (outdoor, large scale scene)
python demo.py --model_path /path/to/lingbot-map-long.pt \
    --image_folder example/oxford --mask_sky

🎯 Featured: indoor walkthrough (~25 000 frames, 13 minutes)

Sequence is too long for the interactive viser viewer — this clip was rendered with the Offline Rendering Pipeline. See that section for the full command.

Streaming with Keyframe Interval

Use --keyframe_interval to reduce KV cache memory by only keeping every N-th frame as a keyframe. Non-keyframe frames still produce predictions but are not stored in the cache. This is useful for sequences longer than 320 frames. The model is trained with video RoPE on 320 views; performance may degrade when the KV cache stores more than 320 views. The keyframe strategy supports inference over longer sequences.

Dataset: Download the demo sequences from robbyant/lingbot-map-demo on Hugging Face.

Example run on the travel sequence from the dataset above (sky masking on, 4 camera optimization iterations, keyframe every 2 frames):

python demo.py \
    --image_folder /path/to/lingbot-map-demo/travel/ \
    --model_path /path/to/lingbot-map-long.pt \
    --mask_sky \
    --camera_num_iterations 4 \
    --keyframe_interval 2

Note on inference range. Inference does not reset state by default, so the maximum range is bounded by the longest distance in the training data. Beyond that distance, state resetting is required. If pose collapse occurs, switch to windowed mode (--mode windowed). In most cases, only --keyframe_interval needs adjustment.

Windowed Inference (for long sequences, >3000 frames)

python demo.py --model_path /path/to/lingbot-map-long.pt \
    --video_path video.mp4 --fps 10 \
    --mode windowed --window_size 128 --overlap_keyframes 16 --keyframe_interval 2 

Sky Masking

Sky masking uses an ONNX sky segmentation model to filter out sky points from the reconstructed point cloud, which improves visualization quality for outdoor scenes.

Setup:

# Install onnxruntime (required)
pip install onnxruntime        # CPU
# or
pip install onnxruntime-gpu    # GPU (faster for large image sets)

The sky segmentation model (skyseg.onnx) will be automatically downloaded from HuggingFace on first use.

Usage:

python demo.py --model_path /path/to/checkpoint.pt \
    --image_folder /path/to/images/ --mask_sky

Sky masks are cached in <image_folder>_sky_masks/ so subsequent runs skip regeneration. You can also specify a custom cache directory with --sky_mask_dir, or save side-by-side mask visualizations with --sky_mask_visualization_dir:

python demo.py --model_path /path/to/checkpoint.pt \
    --image_folder /path/to/images/ --mask_sky \
    --sky_mask_dir /path/to/cached_masks/ \
    --sky_mask_visualization_dir /path/to/mask_viz/

Visualization Options

Argument	Default	Description
--port	8080	Viser viewer port
--conf_threshold	1.5	Visibility threshold for filtering low-confidence points
--point_size	0.00001	Point cloud point size
--downsample_factor	10	Spatial downsampling for point cloud display

Performance & Memory

Without FlashInfer (SDPA fallback)

python demo.py --model_path /path/to/checkpoint.pt \
    --image_folder /path/to/images/ --use_sdpa

Running on Limited GPU Memory

If you run into out-of-memory issues, try one (or both) of the following:

• --offload_to_cpu — offload per-frame predictions to CPU during inference (on by default; use --no-offload_to_cpu only if you have memory to spare).
• --num_scale_frames 2 — reduce the number of bidirectional scale frames from the default 8 down to 2, which shrinks the activation peak of the initial scale phase.

Faster Inference

Lower the number of iterative refinement steps in the camera head to trade a small amount of pose accuracy for wall-clock speed:

python demo.py --model_path /path/to/checkpoint.pt \
    --image_folder /path/to/images/ --camera_num_iterations 1

--camera_num_iterations defaults to 4; setting it to 1 skips three refinement passes in the camera head (and shrinks its KV cache by 4×).