Paper (arXiv preprint) | Project Website

Models made via Blender or taken from @RussTedrake's repository manipulation.

VR Headset communication from RAIL Berkeley's Oculus Reader.

Other contributors, please check out their work!

• Thomas Cohn (@cohnt): IK Calculations, among many
• Adam Wei (@adamw8): Diffusion controllers and code
• Nicholas Pfaff (@nepfaff): models, data regeneration
• Boyuan Chen (@buoyancy99): repository structure and VR controller

If you have any questions, please email me at lkfoland@mit.edu and I will get back to you ASAP.

As mentioned earlier, this repository is based on Boyuan Chen's robot_interface repository. Here's an overview of the organization:

algorithms: This directory is where you should store any algorithms you'll use in your experiments. For example, there are controller algorithms in the controllers directory as well as IK algorithms and basic functions in the common directory.

configurations: You should put a configuration .yaml file for anything you make (experiments, algorithms, datasets, etc.) here under the appropriate sub-directory. Although every experiment/algorithm must have its own .yaml file, you can avoid redundancy by setting an experiment's config file to the default of a similar experiment. For example, see vr_shelves.yaml and sim_data_regen.yaml.

data: Put data that shouldn't be committed in this folder.

datasets: Store dataset code that should be committed in this folder.

debug: From Boyuan's robot_interface repo; use as he describes.

dependencies: A place to store depending repositories. For any form of policy evaluation, you will need to clone the Diffusion Policy repository into this directory and configure paths as needed: https://github.com/real-stanford/diffusion_policy.git. Make sure you follow the setup instructions for this repository, so that way subfolders of it can be referenced like packages!

experiments: This is where most of your coding should take place. Create experiments and put them in this directory.

models: Store custom .sdf files here, and access them in .yamls under the package name models. See vr_shelves.yaml for an example.

outputs: When you run an experiment, the logs of the experiment will be stored here, noted via the time that the experiment began.

scripts: These are bash scripts for calling certain experiments over and over again + calling eval.

• regenerate_camera_data.sh: Generate camera feed from log files with noise added. Be sure to toggle the noise value and number of demos (look at file contents).
• regenerate_noise_data.sh: Add noise to demonstrations in log files. Be sure to toggle the noise value and number of demos (look at file contents).
• run_policy_eval: As the title suggests, run this to do eval with a policy. Toggle parameters (checkpoint path, number of trials, etc) in the file evaluation/eval_box_task.py.

utils: Various utilities for interacting with the VR headset, calculating metrics, et cetera. Try not to tamper with the existing files, but feel free to add your own new files.

conda create env -f environment_macos.yml
conda create env -f environment_linux.yml

To collect VR teleop data (with just arms, or with shelves):

python -m main +name=example_run experiment=vr_experiment

python -m main +name=example_run experiment=vr_shelves

To set a demonstration number, override config default value of 0:

python -m main +name=example_run experiment=vr_experiment demo_num=1

python -m main +name=example_run experiment=sim_noise_regen

python -m main +name=example_run experiment=sim_camera_regen

To visualize a policy rollout, run:

python -m main +name=example_run experiment=sim_image_diffusion

Note that this will use the policy in the evaluation/policy folder.

To run evaluation trials:

scripts/run_policy_eval.sh

To save policy evaluation metrics to a .txt file for a single policy:

scripts/examine_policy_output.sh

To regenerate data:

scripts/regenerate_camera_data.sh

scripts/regenerate_noise_data.sh

Note that these commands were intended to be run from the main directory, for relative paths purposes.

Please email lkfoland@mit.edu with any questions.

Required materials:

1. Oculus Quest VR Headset
2. USB-C Cable
3. RAIL Oculus Berkeley software (https://github.com/rail-berkeley/oculus_reader.git)

Before plugging anytbing in, make sure you have the prerequisites necessary:

sudo apt update
sudo apt install android-tools-adb

Now, create the udev rules file so that you can communicate with the headset:

sudo nano /etc/udev/rules.d/51-android.rules

Once you're in the file, add this line:

SUBSYSTEM=="usb", ATTR{idVendor}=="2833", MODE="0666", GROUP="plugdev"

Save and exit. Make the file executable:

sudo chmod a+r /etc/udev/rules.d/51-android.rules

Reload the rules and make sure you're in the right group; after the last line executes, you should see a list of groups, one of which should be "plugdev."

sudo udevadm control --reload-rules
sudo udevadm trigger
groups $USER

Now, connect the Quest 3 to the computer via USB-C. If you run:

adb devices

You should see a device show up that says "unauthorized." To authorize, put on the VR headset and click "Allow USB Debugging." You should be good to go!

This repo is forked from Boyuan's template repo. For more information, please see the README.md of his repo.