AI agents (AutoGen + GPT-4.1/o3/Llama 3.3-70B) autonomously write and iteratively optimize preprocessing/postprocessing code for scientific image analysis. A function bank accumulates all generated solutions with metrics, feeding the best and worst back into prompts to guide exploration. Simple agent designs consistently outperform human-expert baselines across three production-level biomedical imaging tasks.

# Install uv if needed: curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv && source .venv/bin/activate

# Pick the extra for your task:
uv pip install -e ".[cellpose]"    # Cell segmentation
uv pip install -e ".[polaris]"     # Spot detection
uv pip install -e ".[medsam]"      # Medical segmentation
uv pip install -e ".[dev]"         # Development / tests

python -m venv .venv && source .venv/bin/activate
pip install -e ".[cellpose]"  # or .[polaris], .[medsam]

export OPENAI_API_KEY="sk-..."
# Task-specific:
export DEEPCELL_ACCESS_TOKEN="..."   # Polaris only

python main.py \
    --dataset /path/to/data \
    --experiment_name cellpose_segmentation \
    --gpu_id 0 \
    --random_seed 42 \
    -k 3 \
    --num_optim_iter 20 \
    --history_threshold 5

Results are saved to {experiment_name}/{timestamp}/preprocessing_func_bank.json.

1. Download data — see data utilities in utils/ (cellpose_data.py, spotdetection_data.py, medsam_data.py).

2. Task-specific setup:

   • MedSAM — download medsam_vit_b.pth (instructions) and pass --checkpoint_path.
   • Polaris — set DEEPCELL_ACCESS_TOKEN (instructions).
   • Cellpose — comment out fill_holes_and_remove_small_masks in dynamics.resize_and_compute_masks.

3. Run experiments with appropriate --experiment_name (spot_detection, cellpose_segmentation, or medSAM_segmentation).

4. Analyze trajectories:

   python figs/{task_name}_analyze_trajectories.py --data_path /path/to/data

   Creates analysis_results/ under each result folder with test-set evaluations and plots.

Integrate a custom scientific workflow by implementing four components:

See docs/adding_a_task.md for a full walkthrough.

# Unit tests (no GPU/data required)
python -m pytest tests/ -v

# Integration tests (require data + task packages)
python -m tests.test_cellpose_segmentation --data_path /path/to/data
python -m tests.test_spotdetection
python -m tests.test_medsam_segmentation

@inproceedings{wang2026simple,
  title={Simple Agents Outperform Experts in Biomedical Imaging Workflow Optimization},
  author={Wang, Xuefei and Horstmann, Kai A. and Lin, Ethan and Chen, Jonathan and Farhang, Alexander R. and Stiles, Sophia and Sehgal, Atharva and Light, Jonathan and Van Valen, David and Yue, Yisong and Sun, Jennifer J.},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2026},
  note={to appear}
}

Apache License 2.0