• [2025-11] 🎉 Our paper has been selected as an EMNLP 2025 Outstanding Paper Award (7 of 3,200 accepted papers)!
• [2025-08] Paper accepted to EMNLP 2025 Main Conference
• [2025-06] Initial release of DiscoSG-DS dataset and code

DiscoSG addresses the critical gap in discourse-level text scene graph parsing for Vision-Language Models (VLMs):

• 🎯 Novel Task: First benchmark for discourse-level (multi-sentence) text scene graph parsing
• 📊 Rich Dataset: DiscoSG-DS with 400 expert-annotated + 8,430 synthesized instances
• 🚀 Efficient Method: DiscoSG-Refiner achieves 86× faster inference than GPT-4o with comparable performance
• 🔧 Practical Impact: Significant improvements on downstream VLM tasks including caption evaluation and hallucination detection

Traditional scene graph parsers are designed for single-sentence captions and fail to capture:

• ✅ Cross-sentence coreference (e.g., "woman" → "she")
• ✅ Long-range dependencies between sentences
• ✅ Implicit relationships across discourse
• ✅ Global graph coherence

The DiscoSG-DS dataset is located in the DiscoSG_dataset folder:

Legend:

• Avg Len: Average tokens per instance
• Avg Trp/Obj/Rel: Average triples/objects/relations per graph
• Total Trp: Total triples across dataset

Key Insight: DiscoSG instances contain 3× more triples and 30× longer text than existing datasets, capturing complex discourse-level relationships across an average of 9.3 sentences per caption.

Our dataset creation follows a two-stage process combining human expertise with active learning:

• Two-stage annotation process for quality control
• Creates seed training set for bootstrapping
• Establishes baseline teacher model (M₀)

Four-step iterative process:

1. Batch Selection: Random sampling from unlabeled data
2. Draft Annotation: Use current model (Mᵢ) to generate draft annotations
3. Two-Stage Review: Human correction and validation
4. Model Update: Retrain model (Mᵢ → Mᵢ₊₁) with new annotations

DiscoSG-Refiner is a lightweight iterative framework that refines draft scene graphs through a novel 4-step refinement process.

Caption 
→ [Step 1] Initial Graph
→ [Step 2] Deletion
→ [Step 3] Insertion
→ [Step 4] Refined Graph

Process:

• Split multi-sentence caption into individual sentences
• Parse each sentence independently using sentence-level parser
• Merge sentence-level graphs into initial draft

Process:

• Encode caption and each graph triple
• Binary classifier predicts KEEP/DELETE for each triple
• Removes redundant or incorrect triples

Process:

• Encode caption and current graph state after deletion
• Decoder generates missing triples
• Adds complementary information to graph

Process:

• Execute deletion (Step 2)
• Execute insertion (Step 3)
• (Optional) Repeat refinement cycle for further improvement

# Clone the repository
git clone https://github.com/ShaoqLin/DiscoSG.git
cd DiscoSG

# Install dependencies
pip install -r requirements.txt

Configure dataset paths in the following files:

detailcap_discosg_mr.py (line 64):

# Replace with your path to DiscoSG_datasets directory
dataset_path = "path/to/DiscoSG_datasets"

dataset_utils.py (lines 136, 167):

# Replace with your path to DiscoSG_datasets directory
dataset_path = "path/to/DiscoSG_datasets"

💡 Note: Use the same dataset settings as DetailCaps and CapArena

For quick inference, use pre-computed graphs from the reusable_graph directory:

python detailcap_discosg_mr.py \
  --original_parse_dict reusable_graph/original_parse.json \
  --sub_sentence_parse_dict reusable_graph/sub_sentence_parse.json \
  --combined_parse_dict reusable_graph/combined_parse.json

Available parameters:

parser.add_argument("--original_parse_dict", type=str, default=None, 
                   help="Path to the original parse dict file")
parser.add_argument("--sub_sentence_parse_dict", type=str, default=None, 
                   help="Path to the sub sentence parse dict file")
parser.add_argument("--combined_parse_dict", type=str, default=None, 
                   help="Path to the combined parse dict file")

Replace the default capture.py with our modified version:

# Download CAPTURE metric from official repo
wget https://raw.githubusercontent.com/foundation-multimodal-models/CAPTURE/main/capture.py

# Replace with our modified version
cp capture.py path/to/CAPTURE/capture.py

We provide comprehensive materials for result verification:

• 📋 Complete inference logs from our experiments
• 📊 Intermediate graph structures during inference
• 🔄 Pre-computed reusable graphs for fast inference

These materials enable researchers to verify and reproduce our results.

Performance on DiscoSG-DS test sets:

Key Findings:

• ✅ DiscoSG-DS enables significantly better discourse parsing
• ✅ DiscoSG-Refiner offers the best performance-resource trade-off
• ✅ Outperforms GPT-4o while being 86× faster

Graph-based metrics with DiscoSG-Refiner achieve top performance across multiple benchmarks:

• CapArena: Best correlation with human judgments
• DetailCaps: Superior performance on detailed caption evaluation

DiscoSG-Refiner-based metrics excel at detecting hallucinations in VLM outputs, demonstrating the practical value of accurate discourse-level scene graphs.

If you find our work helpful, please cite our papers:

@inproceedings{lin-etal-2025-discosg,
    title = "{D}isco{SG}: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement",
    author = "Lin, Shaoqing and Teng, Chong and Li, Fei and Ji, Donghong and Qu, Lizhen and Li, Zhuang",
    editor = "Christodoulopoulos, Christos and Chakraborty, Tanmoy and Rose, Carolyn and Peng, Violet",
    booktitle = "Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing",
    month = nov,
    year = "2025",
    address = "Suzhou, China",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2025.emnlp-main.398/",
    doi = "10.18653/v1/2025.emnlp-main.398",
    pages = "7848--7873",
    isbn = "979-8-89176-332-6"
}

@inproceedings{li-etal-2023-factual,
    title = "{FACTUAL}: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing",
    author = "Li, Zhuang and Chai, Yuyang and Zhuo, Terry Yue and Qu, Lizhen and 
              Haffari, Gholamreza and Li, Fei and Ji, Donghong and Tran, Quan Hung",
    booktitle = "Findings of the Association for Computational Linguistics: ACL 2023",
    month = jul,
    year = "2023",
    address = "Toronto, Canada",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.findings-acl.398",
    pages = "6377--6390"
}

We are grateful to:

• Dr. Zhuang Li for his exceptional mentorship and generous support throughout this research
• The authors of FACTUAL, DetailCaps, and CapArena for their foundational work