Skip to main content

Data, applications, and model details

Method​

LingBot-Depth uses masked depth modeling to learn RGB-D representations through self-supervised learning. A Vision Transformer encoder with depth-aware attention jointly processes RGB and depth inputs.

Depth-aware attention visualization. Visualizing attention from depth queries (Q1–Q3, marked with ⋆) to RGB tokens in two scenes: (a) aquarium and (b) indoor shelf. Each row shows masked input depth, attention weights on RGB, and refined output. Different queries attend to spatially corresponding regions, demonstrating cross-modal alignment.

Key Innovations:

  • Masked Depth Modeling: Self-supervised pre-training via depth reconstruction
  • Cross-Modal Attention: Joint RGB-Depth alignment in unified latent space
  • Metric-Scale Preservation: Maintains real-world measurements for downstream tasks

Training Data​

The training data combines real-world and simulated RGB-D captures:

Training dataset. 2M real-world and 1M simulated samples spanning diverse indoor environments (top). Representative RGB-D inputs with ground truth depth (bottom).

Dataset Composition:

  • Real Captures: 2M samples from residential, office, and commercial environments
  • Simulated Data: 1M photo-realistic renders with perfect ground truth
  • Modalities: RGB images, raw depth, refined ground truth depth
  • Diversity: Multiple sensor types, lighting conditions, and scene complexities

Applications​

4D Point Tracking​

LingBot-Depth provides metric-accurate 3D geometry essential for tracking dynamic targets:

4D point tracking. Robust tracking in gym environments with dynamic human motion. Top: query point selection. Middle: 3D tracking on deforming geometry. Bottom: refined depth maps. Demonstrated on scooter, rowing machine, gym equipment, and pull-up bar.

Dexterous Manipulation​

High-quality geometric understanding enables reliable robotic grasping across diverse objects and materials:

Dexterous grasping. Robust manipulation enabled by refined depth. Top: point cloud reconstruction. Bottom: successful grasps on steel cup, glass cup, storage box, and toy car.

Hardware Setup​

The RGB-D capture system supports large-scale data collection:

RGB-D capture system. Multi-sensor setup with Intel RealSense, Orbbec Gemini, and Azure Kinect for scalable real-world data collection.

Model Details​

Architecture​

  • Encoder: Vision Transformer (Large) with RGB-D fusion
  • Decoder: Multi-scale feature pyramid with specialized heads
  • Heads: Depth regression
  • Training: Masked depth modeling with reconstruction objective

Input Format​

RGB Image:

  • Shape: [B, 3, H, W] normalized to [0, 1]
  • Format: PyTorch tensor, float32

Depth Map:

  • Shape: [B, H, W]
  • Unit: Meters (configurable via scale parameter)
  • Invalid regions: 0 or NaN

Camera Intrinsics:

  • Shape: [B, 3, 3]
  • Normalized format: fx'=fx/W, fy'=fy/H, cx'=cx/W, cy'=cy/H
  • Example: 
    [[fx/W,   0,   cx/W],
     [  0,  fy/H,  cy/H],
     [  0,    0,    1  ]]

Output Format​

The model returns a dictionary:

{
    'depth': torch.Tensor,   # Refined depth [B, H, W]
    'points': torch.Tensor,  # Point cloud [B, H, W, 3] in camera space
}

Inference Parameters​

model.infer(
    image,                                   # RGB tensor [B, 3, H, W]
    depth_in=None,                           # Input depth [B, H, W]
    use_fp16=True,                           # Mixed precision inference
    intrinsics=None,                         # Camera intrinsics [B, 3, 3]
)