Skip to main content

Installation and Quick Start

Installation​

Requirements​

  • Python ≥ 3.9
  • PyTorch ≥ 2.0.0
  • CUDA-capable GPU (recommended)

From source​

git clone https://github.com/robbyant/lingbot-depth
cd lingbot-depth

# Install the package (use 'python -m pip' to ensure correct environment)
conda create -n lingbot-depth python=3.9
conda activate lingbot-depth
python -m pip install -e .

Quick Start​

Inference:

import torch
import cv2
import numpy as np
from mdm.model.v2 import MDMModel

# Load model
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
model = MDMModel.from_pretrained('robbyant/lingbot-depth-pretrain-vitl-14-v0.5').to(device)

# Load and prepare inputs
image = cv2.cvtColor(cv2.imread('examples/0/rgb.png'), cv2.COLOR_BGR2RGB)
h, w = image.shape[:2]
image = torch.tensor(image / 255, dtype=torch.float32, device=device).permute(2, 0, 1)[None]

depth = cv2.imread('examples/0/raw_depth.png', cv2.IMREAD_UNCHANGED).astype(np.float32) / 1000.0
depth = torch.tensor(depth, dtype=torch.float32, device=device)[None]

intrinsics = np.loadtxt('examples/0/intrinsics.txt')
intrinsics[0] /= w  # Normalize fx and cx by width
intrinsics[1] /= h  # Normalize fy and cy by height
intrinsics = torch.tensor(intrinsics, dtype=torch.float32, device=device)[None]

# Run inference
output = model.infer(
    image,
    depth_in=depth,
    intrinsics=intrinsics)

depth_pred = output['depth']  # Refined depth map
points = output['points']      # 3D point cloud

Run example:

The model is automatically downloaded from Hugging Face on first run (no manual download needed):

# Basic usage - processes example 0
python example.py

# Use a different example (0-7 available)
python example.py --example 1

# Use depth completion optimized model
python example.py --model robbyant/lingbot-depth-postrain-dc-vitl14-v0.5

# Custom output directory
python example.py --output my_results

# See all options
python example.py --help

This processes the example data and saves results to result/ (or custom directory):

result/
├── rgb.png                 # Input RGB image
├── depth_input.npy        # Input depth (float32, meters)
├── depth_refined.npy      # Refined depth (float32, meters)
├── depth_input.png        # Input depth visualization
├── depth_refined.png      # Refined depth visualization
├── depth_comparison.png   # Side-by-side comparison
└── point_cloud.ply       # 3D point cloud

Available examples: 8 example scenes (0-7) included in examples/ directory.