[ICLR 2026] DARE-Bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science

_{Figure 1: DARE-Bench task structure. Each task provides a natural-language question and structured files. An LLM agent executes code within a sandbox to generate predictions, which are compared against ground truth for automatic evaluation.}

Fan Shu^1,* · Yite Wang^2,*† · Ruofan Wu¹ · Boyi Liu² · Zhewei Yao² · Yuxiong He² · Feng Yan¹

¹University of Houston   ²Snowflake AI Research

DARE-Bench is a large-scale benchmark for evaluating LLM agents on data science tasks, featuring 6,300 tasks across classification, regression, and time series forecasting. Here we make available a subset of the tasks. DARE-Bench provides:

• ✅ Verifiable ground truth for objective and reproducible evaluation
• 🎯 Process-aware instruction following tasks with deterministic outcomes
• 📊 Large-scale training data to support supervised fine-tuning and reinforcement learning

The table above reflects the tasks released in this repository.

Note: The paper reports results/statistics on the full dataset (Train: 5,948, Eval: 352, Total: 6,300). For the full dataset statistics (as reported in the paper), expand the section below.

Task type is inferred from the dataset folder suffix: *_class → Classification, *_reg → Regression, *_ts → Time-series.

See LICENSE/LICENSE_DISTRIBUTION.md for the license distribution of the released subset (counted using suggested_license when present; Train: 2,137 source datasets, Eval: 162 source datasets).

We also provide the license metadata in LICENSE/kaggle_license_train.csv and LICENSE/kaggle_license_test.csv.

Performance of various LLMs on DARE-Bench evaluation tasks (score in %):

The table above reflects the results of the tasks released in this repository. For the full evaluation results (as reported in the paper), expand the section below.

import json

# Load evaluation tasks
with open("data/eval/question_list.json", "r") as f:
    tasks = json.load(f)

# Example: Get a task (classification / regression / time_series_analysis)
task = tasks[0]
question_if = task["question_v1"]  # Instruction Following
question_mm = task["question_v2"]  # ML Modeling
data_path = f"data/eval/databases/{task['file_path']}"

Time series tasks: In question_list.json, time series tasks have task = "time_series_analysis" (with variants XF/CF in v1/v2). The provided evaluation script matches predictions by row_id for all tasks (including time series), so your prediction.csv must include row_id and the target column(s) specified in verify/all_metadata.json.

data/eval/databases/ contains dataset files under their original Kaggle/source licenses. All other files are released under Apache-2.0.

dare-bench/
├── data/
│   └── eval/
│       ├── question_list.json
│       └── databases/
├── LICENSE/
├── LICENSE.txt
├── scripts/
│   ├── evaluation.py
│   └── reference_solution.py
├── figs/
└── README.md

The evaluation script computes metrics by comparing model predictions against ground truth:

Prediction format (all tasks, including time series): scripts/evaluation.py expects prediction.csv to contain a row_id column aligned with verify/ground_truth.csv, plus the target column(s) defined in verify/all_metadata.json.

from scripts.evaluation import evaluate_prediction

result = evaluate_prediction(
    prediction_path="path/to/prediction.csv",
    ground_truth_path="path/to/ground_truth.csv",
    metadata_path="path/to/all_metadata.json",
    question_version="v2"
)
print(f"Score: {result['final_score']:.2%}")

Batch evaluation:

python scripts/evaluation.py \
    --predictions_dir ./predictions \
    --databases_dir ./data/eval/databases \
    --question_list ./data/eval/question_list.json \
    --version v2

Each task includes parameters in all_metadata.json that specify the exact ML pipeline (features, model type, preprocessing) required to generate the ground truth.

The reference solution generator creates executable Python code from the task metadata:

from scripts.reference_solution import generate_reference_solution_code

# Generate the code
code = generate_reference_solution_code("path/to/all_metadata.json")

# Execute to get local ground truth (required for IF tasks)
exec(code)

The generated reference solution writes prediction.csv with row_id plus target column(s). This applies to time series tasks as well.

Generate and execute reference solutions for all tasks:

python scripts/reference_solution.py \
    --databases_dir ./data/eval/databases \
    --question_list ./data/eval/question_list.json \
    --output_dir ./reference_solutions \
    --execute  # Flag to run the code and save ground truth

@inproceedings{shu2026darebench,
  title={DARE-Bench: Evaluating Modeling and Instruction Fidelity of LLMs in Data Science},
  author={Shu, Fan and Wang, Yite and Wu, Ruofan and Liu, Boyi and Yao, Zhewei and He, Yuxiong and Yan, Feng},
  booktitle={International Conference on Learning Representations},
  year={2026}
}

• The databases/ assets are subject to their original source licenses from Kaggle.
• All other files in this repository are released under Apache-2.0.

For reference, we provide the license distribution of the released subset in LICENSE/LICENSE_DISTRIBUTION.md. The underlying license metadata is available in LICENSE/kaggle_license_train.csv and LICENSE/kaggle_license_test.csv.