Lei Li, Xiangxu Zhang, Xiao Zhou, Zheng Liu,

Gaoling School of Artificial Intelligence, Renmin University of China

Beijing Academy of Artificial Intelligence

In this work, we propose Self-Learning Hypothetical Document Embeddings for zero-shot medical information retrieval, eliminating the need for relevance-labeled data.

We alse develop a comprehensive Chinese Medical Information Retrieval Benchmark and evaluate the performance of various text embedding models on it.

Note that the code in this repo runs under Linux system. We have not tested whether it works under other OS.

1. Clone this repository:

   git clone https://github.com/ll0ruc/AutoMIR.git
   cd automir

2. Create and activate the conda environment:

   conda create -n automir python=3.10
   conda activate automir
   pip install torch==2.4.0 torchvision==0.19.0 torchaudio==2.4.0 --index-url https://download.pytorch.org/whl/cu121
   pip install beir==2.0.0
   pip install mteb==1.1.1
   pip install deepspeed==0.15.1
   pip install peft==0.12.0
   pip install transformers==4.44.2
   pip install sentence-transformers==3.1.1
   pip install datasets==2.21.0
   pip install vllm==0.5.4

CMIRB (Chinese Medical Information Retrieval Benchmark) is a specialized multi-task dataset designed specifically for medical information retrieval.

It consists of data collected from various medical online websites, encompassing 5 tasks and 10 datasets, and has practical application scenarios.

The data preprocessing process can be seen in data_collection_and_processing.

An overview datasets available in CMIRB is provided in the following table:

• CMIRB:

  Place all zip files under ./AutoMIR/dataset and extract them.

For each dataset, the data is expected in the following structure:

${DATASET_ROOT} # Dataset root directory, e.g., ./dataset/MedExamRetrieval
├── query.jsonl        # Query file
├── corpus.jsonl        # Document file
└── qrels.txt         # Relevant label file

Download the medical corpus from huatuo_encyclopedia_qa

python gen_Data.gen_Query_data.py --corpus_path "./train_data/corpus.jsonl" --llm_name Qwen-32b

You will get the query.jsonl file in the train_data folder, which contains the generated queries for each document in the corpus.

python gen_Data.gen_LLM_data.py --query_path "./train_data/query.jsonl" --llm_name_gen_llm qwen

You will get the llm_train_data.jsonl file in the train_data/qwen folder, which contains the generated training data for LLM.

bash run train_llm.sh

You will get the fine-tuned LLM model in the outputs/qwen folder, which can be used as a generator for generating retwriten queries.

python gen_Data.gen_EMB_data.py --llm_name qwen

You will get the emb_train_data.jsonl file in the train_data/qwen folder, which contains the generated training data for retriever.

bash run train_emb.sh

You will get the fine-tuned retriever model in the outputs/qwen folder, which can be used for retrieving relevant documents based on the generated queries.

We evaluate 10+ representative retrieval models of diverse sizes and architectures. Run the following command to get results:

cd ./src
python evaluate.py --retrieval_name bge-FT --llm_name qwen
* `--retriever_name`: the retrieval model to evaluate.
* `--llm_name`: the generator to evaluate.

If this code or dataset contributes to your research, please kindly consider citing our paper and give this repo ⭐️ :)

@inproceedings{li-etal-2025-automir,
    title = "{A}uto{MIR}: Effective Zero-Shot Medical Information Retrieval without Relevance Labels",
    author = "Li, Lei  and Zhang, Xiangxu  and Zhou, Xiao  and Liu, Zheng",
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2025",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.findings-emnlp.1305/",
    doi = "10.18653/v1/2025.findings-emnlp.1305",
    pages = "24028--24047",
    ISBN = "979-8-89176-335-7"
}