On lr&bsz, MSPE reaches the low-loss target region using only about 1% of the original fitting budget.

Given a pool of runnable experiments and a limited budget, how should we select experiments to ensure that the fitted scaling law extrapolates accurately in the target region?

This repository contains the core method and benchmark for Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection. We formulate scaling-law fitting as a budget-aware sequential design problem: given a pool of candidate experiments with heterogeneous costs, select experiments sequentially to maximize extrapolation accuracy in a high-cost target region.

This repo contains a benchmark for budget-aware scaling-law fitting with 8 tasks and 65 scaling-law instances. Each instance defines:

• a parametric scaling-law family
• a candidate pool of runnable experiments X_cand
• a held-out high-cost target region X_tar
• a task-specific cost proxy

The corresponding datasets live under benchmark/dataset. The dataset names used by the code are:

• parallel_scaling_law
• vocab_scaling_law
• domain_mixture_scaling_law
• moe_scaling_law
• data_constrained_scaling_law
• lr_bsz_scaling_law
• sparsity_scaling_law
• farseer_scaling_law

The main result is that test $R^2$ consistently outperforms Random, Cheapest, Cost Rand, D-opt, and V-opt in low- and mid-budget regimes, and often approaches or exceeds the All Data fit on several tasks. We reproduce Table 2 from the paper below.

Note: the table is written as 1% / 5% / 10%, while the domain and sparsity columns correspond to 20% / 35% / 50% in the paper.

The project is installed via the root setup.py. Core dependencies are numpy, scipy, pyarrow, and matplotlib.

conda create -n active-sl python=3.11
conda activate active-sl

pip install -e .

This installs the benchmark package and includes benchmark/dataset/**/*.parquet as package data.

The following three baselines use the unified benchmark CLI:

• random
• greedy_cheapest, corresponding to Cheapest in the paper
• inverse_cost, corresponding to Cost Rand in the paper

Example commands are below. Results are written to output/<dataset>/...:

python -m benchmark.main --dataset lr_bsz_scaling_law --method random --repeat 10 --n-restarts 64 --workers 8
python -m benchmark.main --dataset lr_bsz_scaling_law --method greedy_cheapest --repeat 10 --n-restarts 64 --workers 8
python -m benchmark.main --dataset lr_bsz_scaling_law --method inverse_cost --repeat 10 --n-restarts 64 --workers 8

To evaluate on other tasks, replace --dataset. Common values are:

parallel_scaling_law
vocab_scaling_law
domain_mixture_scaling_law
moe_scaling_law
data_constrained_scaling_law
lr_bsz_scaling_law
sparsity_scaling_law
farseer_scaling_law

Our method is currently implemented in mspe.py and is invoked directly:

python mspe.py lr_bsz_scaling_law 10

The first positional argument is the dataset name, and the second is the number of repeats N_REPEAT. For example:

python mspe.py data_constrained_scaling_law 10
python mspe.py domain_mixture_scaling_law 10
python mspe.py farseer_scaling_law 10

mspe.py iterates over all scaling-law instances in the dataset and saves results to output/<dataset>/....

• benchmark/: benchmark loader, task definitions, metrics, runner, and baseline methods
• benchmark/dataset/: ScaleBench data and law definitions
• mspe.py: our method

@article{li2026spend,
  title={Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection},
  author={Li, Sijie and Li, Shanda and Lin, Haowei and Sun, Weiwei and Talwalkar, Ameet and Yang, Yiming},
  journal={arXiv preprint arXiv:2604.22753},
  year={2026}
}