Quick Start

⚙️ Quick Start

This codebase is built upon Wan2.2. Please refer to their documentation for installation instructions.

Installation

Clone the repo:

git clone https://github.com/robbyant/lingbot-world.git
cd lingbot-world

Install dependencies:

# Ensure torch >= 2.4.0
pip install -r requirements.txt

Install flash_attn:

pip install flash-attn --no-build-isolation

Model Download

Model	Control Signals	Resolution	Download Links
LingBot-World-Base (Cam)	Camera Poses	480P & 720P	�HuggingFace �ModelScope
LingBot-World-Base (Act)	Actions	480P & 720P	�HuggingFace
LingBot-World-Fast	Camera Poses	480P & 720P	�HuggingFace

Download models using huggingface-cli:

pip install "huggingface_hub[cli]"
huggingface-cli download robbyant/lingbot-world-base-cam --local-dir ./lingbot-world-base-cam

Download models using modelscope-cli:

pip install modelscope
modelscope download robbyant/lingbot-world-base-cam --local_dir ./lingbot-world-base-cam

Inference

Before running inference, you need to prepare:

• Input image
• Text prompt
• Control signals (optional, can be generated from a video using ViPE)
  • intrinsics.npy: Shape [num_frames, 4], where the 4 values represent [fx, fy, cx, cy]
  • poses.npy: Shape [num_frames, 4, 4], where each [4, 4] represents a transformation matrix in OpenCV coordinates

The following commands cover the released inference modes:

• LingBot-World-Base (Cam):
  • 480P:

  torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 480*832 --ckpt_dir lingbot-world-base-cam --image examples/00/image.jpg --action_path examples/00 --dit_fsdp --t5_fsdp --ulysses_size 8 --frame_num 161 --prompt "The video presents a soaring journey through a fantasy jungle. The wind whips past the rider's blue hands gripping the reins, causing the leather straps to vibrate. The ancient gothic castle approaches steadily, its stone details becoming clearer against the backdrop of floating islands and distant waterfalls."

  • 720P:

  torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 720*1280 --ckpt_dir lingbot-world-base-cam --image examples/00/image.jpg --action_path examples/00 --dit_fsdp --t5_fsdp --ulysses_size 8 --frame_num 161 --prompt "The video presents a soaring journey through a fantasy jungle. The wind whips past the rider's blue hands gripping the reins, causing the leather straps to vibrate. The ancient gothic castle approaches steadily, its stone details becoming clearer against the backdrop of floating islands and distant waterfalls."

  Alternatively, you can run inference without control signals:

  torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 480*832 --ckpt_dir lingbot-world-base-cam --image examples/00/image.jpg --dit_fsdp --t5_fsdp --ulysses_size 8 --frame_num 161 --prompt "The video presents a soaring journey through a fantasy jungle. The wind whips past the rider's blue hands gripping the reins, causing the leather straps to vibrate. The ancient gothic castle approaches steadily, its stone details becoming clearer against the backdrop of floating islands and distant waterfalls."

• LingBot-World-Base (Act):
  • 480P:

  torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 480*832 --ckpt_dir lingbot-world-base-cam --image examples/05/image.jpg --action_path examples/05 --allow_act2cam --sample_steps 20 --dit_fsdp --t5_fsdp --ulysses_size 8 --frame_num 121 --prompt "The video presents a soaring journey through a fantasy jungle. The wind whips past the rider's blue hands gripping the reins, causing the leather straps to vibrate. The ancient gothic castle approaches steadily, its stone details becoming clearer against the backdrop of floating islands and distant waterfalls."

  • 480P with user-friendly action string control:

  torchrun --nproc_per_node=8 generate.py --task i2v-A14B --size 480*832 --ckpt_dir lingbot-world-base-cam --image examples/05/image.jpg --action_path examples/05 --action_string "w-10,a-10,d-10,iw-15,none-10,j-10,l-10,s-15" --allow_act2cam --sample_steps 20 --dit_fsdp --t5_fsdp --ulysses_size 8 --prompt "The video presents a soaring journey through a fantasy jungle. The wind whips past the rider's blue hands gripping the reins, causing the leather straps to vibrate. The ancient gothic castle approaches steadily, its stone details becoming clearer against the backdrop of floating islands and distant waterfalls."

Tips: If you have sufficient CUDA memory, you may increase the frame_num parameter to a value such as 961 to generate a one-minute video at 16 FPS. Otherwise if the CUDA memory is not sufficient, you may use --t5_cpu to decrease the memory usage.