Conference on Robot Learning (CoRL) 2025

Yuxin Chen^*1, Chen Tang^*2, Jianglan Wei¹, Chenran Li¹, Ran Tian¹, Xiang Zhang¹, Wei Zhan¹, Peter Stone^2,3, Masayoshi Tomizuka¹

^*Equal contribution
¹University of California, Berkeley    ²The University of Texas at Austin    ³Sony AI

Aligning robot behavior with human preferences is crucial for deploying embodied AI agents in human-centered environments. A promising solution is interactive imitation learning from human intervention, where a human expert observes the policy's execution and provides interventions as feedback. However, existing methods often fail to utilize the prior policy efficiently to facilitate learning, thus hindering sample efficiency.

In this work, we introduce Maximum-Entropy Residual-Q Inverse Reinforcement Learning (MEReQ), designed for sample-efficient alignment from human intervention. Instead of inferring the complete human behavior characteristics, MEReQ infers a residual reward function that captures the discrepancy between the human expert's and the prior policy's underlying reward functions. Residual Q-Learning (RQL) is then employed to align the policy with human preferences using the inferred reward function. Extensive evaluations on simulated and real-world tasks show that MEReQ achieves sample-efficient alignment from human intervention compared to baselines.

conda env create -f environment.yml
conda activate mereq

# Install customized Gymnasium
cd Gymnasium && pip install -e . && cd ..

# Install customized HighwayEnv
cd HighwayEnv && pip install -e . && cd ..

# Install customized Stable-Baselines3
cd stable-baselines3 && pip install -e . && cd ..

Train a prior policy using the provided training script:

python ./tests/test_train.py -a <ALGO> -e <ENV> -n <N_TIMESTEPS> -tr

Supported Environments:

Supported Algorithms:

Evaluate a trained model:

python ./tests/test_train.py -a <ALGO> -e <ENV> -te

Optional flags:

• -g: Save rendering as a GIF image
• -r: Sample random actions from the action space
• -m <MODEL_PATH>: Evaluate a custom model

See scripts/pretrain.sh for example commands.

Run MEReQ with simulated expert intervention on Highway environment:

python -u MEReQ/highway_intvn.py -e RQL_intvn_plc_<EXP_NUM> 2>&1 | tee outputs/RQL_intvn_plc_<EXP_NUM>.txt

Optional flags:

• -dm: Use expert demonstration (learning from demonstration)
• -tf: Use total expert feature count (accumulated expert feature counts)
• -rp: Use random policy (learning from random policy with Q-learning)

See scripts/IRL_intvn.sh for example commands.

Run MEReQ with human-in-the-loop intervention:

python -u MEReQ/highway_human.py -e RQL_human_<EXP_NUM> 2>&1 | tee outputs/RQL_human_<EXP_NUM>.txt

Optional flags:

• -tf: Use total human feature count (accumulated human feature counts)

We provide implementations of several baseline methods. Below are examples using the fanuc_gribber task:

If you find this work useful, please consider citing:

@inproceedings{chen2025mereq,
  author    = {Chen, Yuxin and Tang, Chen and Wei, Jianglan and Li, Chenran and Tian, Ran and Zhang, Xiang and Zhan, Wei and Stone, Peter and Tomizuka, Masayoshi},
  title     = {MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention},
  booktitle = {9th Annual Conference on Robot Learning (CoRL)},
  year      = {2025},
}

This codebase builds upon the following open-source projects:

• Gymnasium
• HighwayEnv
• Stable-Baselines3

This project is licensed under the MIT License - see the LICENSE file for details.