This repository contains the implementation for CIVIL, a framework designed to enable real robots to learn from multimodal visual-language instruction data using imitation learning.
We use DEVA (Tracking Anything with DEVA) as a dependency for visual tracking of offline data. Configure DEVA by following the instructions at the official repository: Tracking-Anything-with-DEVA
⚠️ Make sure DEVA is installed and working before proceeding to install the environment.
Use the provided environment.yml file to create the conda environment:
conda env create -f environment.yml
conda activate CIVILTo work with the simulation dataset, refer to the instructions provided by CALVIN.
For generating CALVIN data:
- Use
data_generation/generate_calvin.pyto create marker data. - Use
add_segmentation_calvin.pyto generate segmentation masks.
A sample of the real-world dataset we used with Panda robots is available under:
panda_data_example/
For further documentation on training scripts, experiment setup, and user study evaluations, refer to the relevant scripts in the repository.
Check out robot rollouts with CIVIL on our website.