Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning

Abstract: Recent advancements in Multimodal Large Language Models (MLLMs) have generated significant interest in their ability to autonomously interact with and interpret Graphical User Interfaces (GUIs). A major challenge in these systems is grounding—accurately identifying critical GUI components such as text or icons based on a GUI image and a corresponding text query. Traditionally, this task has relied on fine-tuning MLLMs with specialized training data to predict component locations directly. However, in this paper, we propose a novel Tuning-free Attention-driven Grounding (TAG) method that leverages the inherent attention patterns in pretrained MLLMs to accomplish this task without the need for additional fine-tuning. Our method involves identifying and aggregating attention maps from specific tokens within a carefully constructed query prompt. Applied to MiniCPM-Llama3-V 2.5, a state-of-the-art MLLM, our tuning-free approach achieves performance comparable to tuning-based methods, with notable success in text localization. Additionally, we demonstrate that our attention map-based grounding technique significantly outperforms direct localization predictions from MiniCPM-Llama3-V 2.5, highlighting the potential of using attention maps from pretrained MLLMs and paving the way for future innovations in this domain.

In this work, we developed an OCG dataset to evaluate the optical character grounding ability of MLLMs. Based on common screen resolutions, we construct 10 different (width:height) aspect ratios to comprehensively assess model's grounding robustness.

We evaluate methods on the ScreenSpot dataset.

Visual Demonstration:

We evaluate methods on the Element accuracy metric on Mind2Web dataset.

Visual Demonstration:

Please download the following three benchmarks: OCG, ScreenSpot and Mind2Web screenshots and annotations. Please DO NOT redistribute the unzipped data files online to avoid risks such as model overfitting.

1. Clone this repository and enter the project folder

git clone https://github.com/HeimingX/TAG.git
cd TAG

2. Create conda environment

conda create -n TAG python=3.10 -y
conda activate TAG

3. Install dependencies

pip install -r requirements.txt

MLLM_PATH=openbmb/MiniCPM-Llama3-V-2_5
OCG_DATAPATH=PATH/TO/OCG
IMG_ASPECTS='[[1, 4], [9, 21], [9, 19], [1, 2], [9, 16], [4, 3], [16, 9], [2, 1], [21, 9], [4, 1]]'

# Evaluate with MiniCPMV2.5
python eval_mm/ocg/eval_MiniCPMV2_5.py \
    --mllm_path ${MLLM_PATH} \
    --data_path ${OCG_DATAPATH} \
    --image_aspects "${IMG_ASPECTS}" \
    --save-dir output/ocg/mv2_5 \
    --sampling

# Evaluate with TAG
python eval_mm/ocg/eval_TAG.py \
    --mllm_path ${MLLM_PATH} \
    --data_path ${OCG_DATAPATH} \
    --image_aspects "${IMG_ASPECTS}" \
    --save-dir output/ocg/tag \
    --batchsize 4

MLLM_PATH=openbmb/MiniCPM-Llama3-V-2_5
SCREENSPOT_IMGS=PATH/TO/IMGS
SCREENSPOT_TEST=PATH/TO/TESTSET

# Evaluate with MiniCPMV2.5
python eval_mm/screenspot/eval_MiniCPMV2_5.py \
    --mllm_path ${MLLM_PATH} \
    --screenspot_imgs ${SCREENSPOT_IMGS} \
    --screenspot_test ${SCREENSPOT_TEST} \
    --save-dir output/screenspot/mv2_5

# Evaluate with TAG
python eval_mm/screenspot/eval_TAG.py \
    --mllm_path ${MLLM_PATH} \
    --screenspot_imgs ${SCREENSPOT_IMGS} \
    --screenspot_test ${SCREENSPOT_TEST} \
    --save-dir output/screenspot/tag

MLLM_PATH=openbmb/MiniCPM-Llama3-V-2_5
MIND2WEB_DATAPATH=PATH/TO/MIND2WEB

TASKTYPES=(task website domain)
for TASK in "${TASKTYPES[@]}"
do
    # Evaluate with MiniCPMV2.5
    python eval_mm/mind2web/eval_MiniCPMV2_5.py \
        --mllm_path ${MLLM_PATH} \
        --data_dir ${MIND2WEB_DATAPATH} \
        --task ${TASK} \
        --save-dir output/mind2web/mv2_5

    # Evaluate with TAG
    python eval_mm/mind2web/eval_TAG.py \
        --mllm_path ${MLLM_PATH} \
        --data_dir ${MIND2WEB_DATAPATH} \
        --task ${TASK} \
        --save-dir output/mind2web/tag \
done

Note: some evaluation log files are provided for reference.

We thank the following MiniCPM-V, SeeClick and Mind2Web for their impressive work and open-sourced projects.

If you find our code/paper helpful, please consider cite our paper 📝 and star us ⭐️！

@inproceedings{xu2025tag,
    title={Attention-driven GUI Grounding: Leveraging Pretrained Multimodal Large Language Models without Fine-Tuning},
    author={Hai-Ming Xu and Qi Chen and Lei Wang and Lingqiao Liu},
    booktitle={The 39th Annual AAAI Conference on Artificial Intelligence},
    year={2025},
}