[CVPR 2026] 3D-SLIM
Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs
for 3D Scene-Language Understanding
Yerim Jeon, Miso Lee, WonJun Moon, Jae-Pil Heo
TLDR: We identify two major issues in conventional LLM decoders: spurious order-dependent correlations and limited instruction-object token interaction. To address both, we propose 3D Spatial Language Instruction Mask (3D-SLIM), an efficient and easily integrable solution that generalizes across various baselines and 3D scene-language tasks.
- [2026/03/23] Support Vicuna-7B, Llama3-8B, and Qwen2-7B.
- [2026/03/20] Released the paper and code of 3D-SLIM.
First, clone the repository:
git clone https://github.com/Jyerim/3D-SLIM
cd 3D-SLIMThen set up the environment using one of the following options:
Option 1: Docker (Recommended)
docker pull yerim0330/chatscene:torch2.7.1-cuda12.8-scatterOption 2: pip install
Step 1. Install PyTorch (CUDA 12.8) and torch extensions:
pip install torch==2.7.1 torchvision==0.22.1 torchaudio==2.7.1 \
--index-url https://download.pytorch.org/whl/cu128
pip install torch-geometric torch-scatter torch-sparseStep 2. Install the remaining dependencies:
pip install -r requirements.txtStep 3. Install Java (required for METEOR metric):
apt-get install -y default-jreOption 1: Download preprocessed data
We provide all preprocessed data on OneDrive.
Download and place the files in the annotations/ directory — no further steps needed.
Option 2: Prepare from scratch
For full data preparation steps (ScanNet, Mask3D, 3D/2D feature extraction, etc.), follow the Chat-Scene preprocess guide.
Once ready, configure the paths in preprocess/run_prepare.sh and run:
bash preprocess/run_prepare.shWe provide model checkpoints for each LLM backbone:
| Model | ScanRefer | Multi3DRefer | Scan2Cap | ScanQA | SQA3D | |||||
|---|---|---|---|---|---|---|---|---|---|---|
| Acc@0.25 | Acc@0.5 | F1@0.25 | F1@0.5 | C@0.5 | B-4@0.5 | C | B-4 | EM | EM-R | |
| Vicuna-7B + Ours | 59.6 | 54.1 | 63.7 | 58.7 | 84.2 | 38.0 | 94.0 | 15.2 | 55.9 | 58.9 |
| Llama3-8B + Ours | 61.8 | 55.7 | 64.3 | 59.2 | 85.1 | 38.8 | 85.1 | 15.6 | 56.3 | 59.2 |
| Qwen2-7B + Ours | 61.0 | 54.8 | 63.9 | 58.8 | 85.0 | 38.9 | 88.5 | 15.9 | 56.0 | 59.2 |
Place the downloaded checkpoint under outputs/<model_name>/.
Before running, download the LLM weights of your choice and place them under llm/:
| Model | Variant | Link |
|---|---|---|
| Vicuna | vicuna-7b-v1.5 |
Download |
| LLaMA3 | Meta-Llama-3-8B-Instruct |
Download |
| Qwen2 | Qwen2-7B-Instruct |
Download |
Then modify the following variables in scripts/run.sh:
| Variable | Description |
|---|---|
OUTPUT_DIR |
Directory where checkpoints and logs will be saved |
llm_model_path |
Path to LLM weights under llm/ |
bash scripts/run.sh Explanation of train_tag and val_tag
Use # to separate different datasets.
| Tag | Dataset |
|---|---|
scanrefer |
ScanRefer |
scan2cap |
Scan2Cap |
scanqa |
ScanQA |
sqa3d |
SQA3D |
multi3dref |
Multi3dRefer |
nr3d_caption |
Captioning dataset from Nr3D |
obj_align |
Alignment dataset from ScanRefer |
Before running, modify the following variables in scripts/eval.sh:
| Variable | Description |
|---|---|
pretrained_path |
Path to trained checkpoint (e.g. outputs/<model_name>/ckpt_01_6890.pth) |
OUTPUT_DIR |
Directory where evaluation results will be saved |
bash scripts/eval.shIf you find our work helpful, please consider citing:
@article{jeon2025masking,
title={Masking Matters: Unlocking the Spatial Reasoning Capabilities of LLMs for 3D Scene-Language Understanding},
author={Jeon, Yerim and Lee, Miso and Moon, WonJun and Heo, Jae-Pil},
journal={arXiv preprint arXiv:2512.02487},
year={2025}
}