This is the official repository of the following paper and a project that study positional perception in LVLMs.

Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization
NeurIPS 2024
Xinyu Lyu†, Beitao Chen†, Lianli Gao, Jingkuan Song, Heng Tao Shen

We conducted the theoretical analysis to promote the effectiveness of contrast decoding. Building on this insight, we introduce a novel optimization strategy named Hallucination-Induced Optimization (HIO). This strategy seeks to amplify the contrast between hallucinatory and targeted tokens relying on a fine-tuned theoretical preference model (i.e., Contrary Bradley-Terry Model), thereby facilitating efficient contrast decoding to alleviate hallucinations in LVLMs. Extensive experimental research demonstrates that our HIO strategy can effectively reduce hallucinations in LVLMs, outperforming state-of-the-art methods across various benchmarks.

Visit our 🏠 project page and 📃 paper to explore more!

• [2024.09.29] 📃 Our HIO is accepted by NeurIPS 2024!
• [2024.5.30] 📃 Our paper is accesible at arxiv now.

1. Clone this repository and navigate to HIO folder

git clone https://github.com/BT-C/HIO.git
cd HIO

2. Install Requirements

conda create -n hio python=3.10 -y
conda activate hio
pip install --upgrade pip
pip install -e .

Usage and License Notices: The data, code, and checkpoint are intended and licensed for research use only. They are also restricted to uses that follow the license agreement of LLaMA, Vicuna, and Chat GPT. The dataset is CC BY NC 4.0 (allowing only non-commercial use) and models trained using the dataset should not be used outside of research purposes.

• LLaVA-1.5. The LLaVA-v1.5 part of HIO is based on the official LLaVA-1.5 implementation, which is a great open-source work on LVLM.
• MiniGPT-4. The MiniGPT-4 part of HIO is based on the official MiniGPT-4 implementation.

You can find more details in our paper.

If you're using HIO in your research or applications, please cite using this BibTeX:

@article{chen2024alleviating,
  title={Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization},
  author={Lyu, Xinyu and Chen, Beitao and  Gao, Lianli and Song, Jingkuan and Shen, Heng Tao},
  journal={arXiv preprint arXiv:2405.15356},
  year={2024}
}