Skip to main content

Inference and evaluation

Deploying LingBot-VA for Inference​

LingBot-VA supports both standalone execution and Server-Client architecture which separates the model environment from simulation. By isolating dependencies, the design avoids package clashes and supports distributed inference on GPUs, clusters, and other devices.

Evaluation on RoboTwin-2.0​

Preparing the Environment

You can follow the official instructions from the original RoboTwin-2.0 repository:
https://robotwin-platform.github.io/doc/usage/robotwin-install.html

In summary:

  1. Install Vulkan dependencies:

    sudo apt install libvulkan1 mesa-vulkan-drivers vulkan-tools

  2. Clone the RoboTwin repository:

    git clone https://github.com/RoboTwin-Platform/RoboTwin.git && cd RoboTwin && git checkout 2eeec322

  3. Modify script/requirements.txt with the following content:

    transforms3d==0.4.2
    sapien==3.0.0b1
    scipy==1.10.1
    mplib==0.2.1
    gymnasium==0.29.1
    trimesh==4.4.3
    open3d==0.18.0
    imageio==2.34.2
    pydantic
    zarr
    openai
    huggingface_hub==0.36.2
    h5py
    # For Description Generation
    azure==4.0.0
    azure-ai-inference
    pyglet<2
    wandb
    moviepy
    imageio
    termcolor
    av
    matplotlib
    ffmpeg

  4. Modify line 8 of script/_install.sh:

    pip install "git+https://github.com/facebookresearch/pytorch3d.git@stable" --no-build-isolation

  5. Install dependencies:

    bash script/_install.sh

  6. Download assets:

    bash script/_download_assets.sh

Deploying the Inference Server

# single GPU
bash evaluation/robotwin/launch_server.sh

# multi-GPU
bash evaluation/robotwin/launch_server_multigpus.sh

Executing the Inference Client

# single GPU
task_name="adjust_bottle";
save_root="results/";
bash evaluation/robotwin/launch_client.sh ${save_root} ${task_name}

# multi-GPU
save_root="results/"
task_group_id=0;
bash evaluation/robotwin/launch_client_multigpus.sh ${save_root} ${task_group_id}

Related experiments results will be save in /path/to/your/RoboTwin/${save_root}. Please note that an eval_result folder is also generated. This is a native output from RoboTwin and is identical to the contents in the results folder; it can be safely ignored. It is important to note that the inference server and client must be deployed on the same machine. For launching multi-GPU client, we padded the original 50 tasks to 56 via duplication and partitioned them into 7 groups to align with the 8-GPU configuration of our inference node. You can specify the task_group_id (0-6) to select a particular group for inference. For detailed grouping configurations, please refer to evaluation/robotwin/launch_client_multigpus.sh.

GPU Memory Requirements: Approximately 24GB VRAM for single-GPU RoboTwin evaluation with offload mode enabled (VAE and text_encoder offloaded to CPU).

Evaluation on LIBERO​

Follow the official instructions to install LIBERO, then launch the server and client:

# server
bash evaluation/libero/launch_server.sh

# client
bash evaluation/libero/launch_client.sh

Run Image to Video-Action Generation​

Use the following script for image-to-video-action generation:

NGPU=1 CONFIG_NAME='robotwin_i2av' bash script/run_launch_va_server_sync.sh

GPU Memory Requirements: Approximately 18GB VRAM for single-GPU i2av inference with offload mode enabled (VAE and text_encoder offloaded to CPU).