Skip to content

docs: update architecture-and-performance.md to reflect AIMD changes#467

Merged
nabinchha merged 5 commits into
mainfrom
nmulepati/docs/466-architecture-and-performance-aimd
Mar 31, 2026
Merged

docs: update architecture-and-performance.md to reflect AIMD changes#467
nabinchha merged 5 commits into
mainfrom
nmulepati/docs/466-architecture-and-performance-aimd

Conversation

@nabinchha

Copy link
Copy Markdown
Contributor

📋 Summary

Update the Architecture & Performance doc to account for AIMD adaptive concurrency control and the ColumnWiseDatasetBuilderDatasetBuilder rename.

Closes #466

🔄 Changes

🔧 Changed

  • "What Data Designer Does" — added bullet for automatic rate-limit adaptation
  • "What Data Designer Does NOT Do" — reworded "Rate limit" to "Impose rate limits" with clarification
  • Concurrency Formula — replaced static max_parallel_requests with AIMD-managed current_throttle_limit; explained decrease/increase behavior
  • max_parallel_requests section — reframed as a ceiling; updated guidance; added AIMD-aware tuning tip
  • Common Problems table — updated "Low throughput" row; added "Frequent 429 → recovery cycles" row
  • Tuning Workflow — added AIMD context and new step for checking throttle logs
  • Execution Model admonition — updated from "column-wise dataset generator" to DatasetBuilder (renamed in chore: rename ColumnWiseDatasetBuilder to DatasetBuilder #437)

✨ Added

  • New "Adaptive Throttling (RunConfig)" section documenting ThrottleConfig parameters
  • Sync engine caveats noting AIMD is fully active on the async engine path only

🤖 Generated with AI

Made with Cursor

…ncy control (#466)

Account for the AIMD throttle manager throughout the architecture doc:
concurrency formula, max_parallel_requests guidance, new ThrottleConfig
section, common problems table, and tuning workflow. Add sync-engine
caveats noting AIMD is fully active on the async path.

Made-with: Cursor
@nabinchha nabinchha requested a review from a team as a code owner March 25, 2026 19:36
@greptile-apps

greptile-apps Bot commented Mar 25, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR updates docs/concepts/architecture-and-performance.md to document the AIMD adaptive concurrency controller and rename ColumnWiseDatasetBuilderDatasetBuilder. The changes are generally accurate and well-structured, but one tip section contains a factual claim about ceiling behavior that is not supported by the implementation in throttle_manager.py.

Key changes:

  • Added AIMD context throughout (concurrency formula, max_parallel_requests guidance, common problems table, tuning workflow)
  • New Adaptive Throttling (RunConfig) section documenting all ThrottleConfig parameters with correct defaults verified against run_config.py
  • Sync-engine caveats accurately noted in two places
  • DatasetBuilder rename (from chore: rename ColumnWiseDatasetBuilder to DatasetBuilder #437) correctly reflected in the execution-model admonition

Issue found:

  • The !!! tip \"How it works in practice\" section claims "If the probe succeeds, the ceiling ratchets up", but release_success() in throttle_manager.py never modifies rate_limit_ceiling — only release_rate_limited() does, and only via min(), so the ceiling can only decrease. The related sentence "If the server rate-limits again, the controller records that level as a ceiling" is also inaccurate: the ceiling is recorded on the first 429 (the rate_limit_ceiling == 0 branch), not only on subsequent ones.

Confidence Score: 4/5

Safe to merge after fixing the factual inaccuracy in the ceiling-ratchet description.

One P1 documentation inaccuracy: the tip claims the rate-limit ceiling ratchets up on probe success, but the code only ever lowers the ceiling. This will mislead users trying to understand recovery behavior. All ThrottleConfig defaults and parameter descriptions are verified correct against the source.

docs/concepts/architecture-and-performance.md — lines 233–236 (ceiling ratchet claim in the tip block)

Important Files Changed

Filename Overview
docs/concepts/architecture-and-performance.md Documentation updated to cover AIMD adaptive concurrency, ThrottleConfig parameters, and the DatasetBuilder rename. One factual inaccuracy found: the tip section incorrectly states the rate-limit ceiling ratchets up on probe success; the implementation only ever lowers the ceiling via min().
Prompt To Fix All With AI
This is a comment left during a code review.
Path: docs/concepts/architecture-and-performance.md
Line: 233-236

Comment:
**"Ceiling ratchets up" claim is not supported by the implementation**

The tip states: *"If the probe succeeds, the ceiling ratchets up; if it triggers another 429, the ceiling lowers."*

Looking at `throttle_manager.py`, `rate_limit_ceiling` is only ever written in `release_rate_limited()`:

```python
if state.rate_limit_ceiling == 0:
    state.rate_limit_ceiling = prev_limit
else:
    state.rate_limit_ceiling = min(state.rate_limit_ceiling, prev_limit)
```

`release_success()` never touches `rate_limit_ceiling`. This means the stored ceiling can only decrease (or stay the same) — it never ratchets up as a result of a successful probe. When the system successfully operates at `ceiling × (1 + ceiling_overshoot)` requests, `rate_limit_ceiling` is unchanged; the soft ceiling returned by `_compute_soft_ceiling()` stays fixed too. If a new 429 fires at `prev_limit = ceiling × (1 + overshoot)` which is greater than the old ceiling, `min(old_ceiling, prev_limit)` still returns `old_ceiling` — no upward movement.

In practice this means a ceiling that was set too low (e.g. during a transient overload) can never self-heal upward; the system is permanently capped at `old_ceiling × (1 + overshoot)` regardless of how well the server performs afterward.

Additionally, "If the server rate-limits **again**, the controller records that level as a ceiling" is also inaccurate — the ceiling is set on the **first** 429 (via the `if state.rate_limit_ceiling == 0` branch), not only on subsequent ones.

How can I resolve this? If you propose a fix, please make it concise.

Reviews (3): Last reviewed commit: "docs: address PR #467 review feedback on..." | Re-trigger Greptile

Comment thread docs/concepts/architecture-and-performance.md
Comment thread docs/concepts/architecture-and-performance.md Outdated
Comment thread docs/concepts/architecture-and-performance.md
andreatgretel
andreatgretel previously approved these changes Mar 31, 2026

@andreatgretel andreatgretel left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, left two minor nits but nothing blocking. 👍

- Clarify that only the first 429 in a burst reduces the concurrency
  limit; subsequent in-flight cascade 429s hold it steady
- Soften sync engine caveats: AIMD engages as a fallback when transport
  retries exhaust, not "no effect"
- Note salvage queue difference between async and sync paths when
  recommending aggressive max_parallel_requests values

Made-with: Cursor
@nabinchha nabinchha merged commit 7087add into main Mar 31, 2026
47 checks passed
@nabinchha nabinchha deleted the nmulepati/docs/466-architecture-and-performance-aimd branch March 31, 2026 22:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Docs: update architecture-and-performance.md to reflect AIMD changes

2 participants