…mula divergence

Updates the addendum's "+1 tile bucketing-formula divergence" subsection
to reference upstream PR flashinfer-ai#3198, which fixes get_max_num_tiles by
replacing fi's L + ceil((E - L) / T) with TRT-LLM's compact closed-form
(E + (T - 1) * L) // T. PR is awaiting review and is based on flashinfer-ai#3067.

Also adds a parenthetical clarifying that the perf impact of the +1
over-allocation was effectively zero at runtime (since the kernel
work is bounded by num_non_exiting_tiles, not buffer capacity), to
prevent future readers from over-attributing the EP=8 gap to this
formula bug.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

… bucket-cap fixes

Updates the post-close addendum and the flashinfer-ai#9 sub-section to reflect the
2026-04-28 root-cause work. The audit's headline +7-9% EP perf gaps at
N=16384 (EP=8 +8.6% kernel, EP=16 +13.7% kernel) are driven by two
paired CuteDSL MoE autotune bugs:

1. CuteDslMoEWrapper prealloc-bias (correctness): wrapper prealloc'd
   buffers sized for self.tile_size only, biasing autotune profiling
   against the other tile_size. Without the fix, fi is locked to
   tile_size=128 in 14/14 cache entries. Side branch
   cute-dsl-moe-wrapper-prealloc-bias-fix at commit a4137e6.
2. Autotuner bucket-cap mismatch (headline-perf): tuner.py:281 caps
   profile buckets at 8192 while runtime can be N=16384. fi profiles
   at half the per-expert workload, autotune-noise on tight margins
   (0.58% at EP=8 bucket=8192) destabilizes the cached choice. 1-line
   fix: bump cap to 16384.

Validation at --num-iters 100 --warmup 10 (3 runs each):

  no fixes:           EP=8 +5.5%/9.5pp  EP=16 +1.8%/9.7pp
  prealloc-fix only:  EP=8 +4.4%/11pp   EP=16 +2.4%/10.5pp
  both fixes:         EP=8 -0.8%/1.1pp  EP=16 -5.6%/10pp

Prealloc-fix alone does not close the headline gap. The bucket-cap-fix
is what closes it. Both fixes are required.

The EP=16 10pp spread under both-fixes is trt's autotune coin-flip
on its 0.08% tile=128 vs tile=256 profile-time margin -- fi's chosen
tactic is stable across all 3 runs; trt's is the noise source.

Several earlier 2026-04-28 revisions to this addendum had wrong
framings (EP=16 fully closed by prealloc-fix alone at --num-iters 1
was a lucky 2-run sample; necessary-but-not-sufficient at EP=8 was
partly true at --num-iters 1 but the residual was the bucket-cap
mismatch, not an unfixable deeper issue). The final state captured
here supersedes all earlier revisions in this same audit doc.

Neither fix is upstream as of 2026-04-28 -- held pending flashinfer-ai#3067
(fix-cutedsl-moe-gemm2-tactic-enum-tile256) and flashinfer-ai#3198
(get_max_num_tiles off-by-one) merging first. Thin-adapter refactor
remains the recommended medium-term direction; both short-term fixes
patch symptoms in code the refactor would delete.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…, mark superseded sub-sections

Lee asked for a thorough audit-of-the-audit pass: look for inconsistencies,
discrepancies, redundancies, and opportunities for compaction/simplification.
After reading the full 3952-line doc, the following systematic edits were
applied:

Top-of-doc:

- Added a "Final state (2026-04-29) — read this first" section at the very
  top (immediately after the title, before the chronological correction
  log). Captures: static audit summary, both upstream PRs (flashinfer-ai#3067, flashinfer-ai#3198),
  the two paired wrapper bugs and their side branches, the comprehensive
  30-cell validation matrix, the three-regime classification (large-N
  closure / medium-N wrapper-overhead-residual / small-N noise), the EP=1
  + version-skew explanation, and the closure of the empirical thread.
  This section is now authoritative; the rest of the doc is the
  chronological investigation record leading to it.

- Updated the title's "last updated" tag to 2026-04-29.

Stale references fixed:

- The flashinfer-ai#3067 fix branch was referred to via the stale commit hashes
  d291d17e and e2330e73/f0cf8cd0 in three places; updated to the current
  57de5f8 + 38293a0 (with notes that the original commit-hash references
  are preserved for traceability).
- The "Tunable runner" row in the Summary table marked tile_size=256 as
  "gated off"; updated to "fixed via PR flashinfer-ai#3067 (pending review)".
- Item 3 of the corrections list: "Recommended fix (drafted, not yet
  applied)" → "Fix landed and shipped as upstream PR flashinfer-ai#3067".

Duplicate content removed:

- Two "What still stands (unaffected by the autotune-context bug)"
  blocks were nearly identical; the first was removed and replaced
  with a one-line pointer to the consolidated version below.

Executive verdict updated:

- Rewritten to reflect current state: now references both PRs, the
  paired-fix story, the comprehensive 30-cell sweep matrix, and the
  +13.7% kernel-level closure. Previously it described "Leading
  hypothesis (NOT YET EMPIRICALLY VERIFIED)" framings that have since
  been confirmed and shipped.

Superseded sub-sections marked:

- "EP=8 vs EP=16: the fix has different impact at each" — was an
  early "necessary but not sufficient" framing under prealloc-only
  state; replaced with a SUPERSEDED note pointing to the paired-fix
  story below (the bucket-cap-fix discovery is what closed both EPs).
- "EP=16 confirmation (2026-04-28)" header clarified to indicate
  it's a single-iter prealloc-only snapshot; the H_A/H_B
  hypothesis-discrimination logic and trt 0.08% margin observation
  are still load-bearing and preserved.
- "Final cross-N validation (2026-04-28)" renamed to "Initial 6-cell
  cross-N spot-check" with a pointer to the comprehensive 30-cell
  sweep that supersedes it.
- Status sub-section: removed outdated EP=8 partial-closure / EP=16
  full-closure bullets (those were prealloc-only state); replaced
  with concise paired-fix branch summary and pointer to comprehensive
  sweep for per-cell results.
- Implications for the audit's earlier claims: updated the
  "dominant factor is the prealloc bias" bullet to the paired-fix
  framing (prealloc + bucket-cap, both required).

Follow-ups updated:

- Open follow-ups remaining: appended a "Update 2026-04-29 (final):
  flashinfer-ai#9 also closed" note documenting that the EP=8/EP=16 perf gap was
  root-caused to two paired wrapper bugs, with the empirical
  investigation thread now closed.
- Item flashinfer-ai#9 in the chronological "Follow-ups queued for later sessions"
  list: added a RESOLVED 2026-04-29 header with pointer to the
  paired-fix story, marking the 9a / 9b / 9c sub-mechanism candidates
  as pre-resolution candidate hypotheses (preserved as record of the
  search space, not blocking work).

What was preserved as historical record (not removed):

- Full "2026-04-24/25/28 correction" chronological log
- Per-component static audit detail
- Issue flashinfer-ai#3067 narrative including the original "Leading hypothesis"
  framing and the post-fix verification table
- The 22-day candidate-root-cause investigation that ruled out 5
  mechanisms before landing on the paired wrapper bugs
- Part 2 baseline numbers (with retraction annotations preserved
  as-is)
- Probes, MoE kernel coverage, reproduction runbook

Net effect: doc is more navigable, internally consistent, and the
authoritative summary is at the top. Body content preserves the
investigative reasoning. Diff: +241 / -217 lines, net +24 lines
(intentional — the new Final state section adds explicit
authoritative content).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…g→functional-API migration (NOT committed)

Captures the analysis surfaced 2026-04-30 around an alternative
to the medium-term thin-adapter refactor: migrate SGLang from
`CuteDslMoEWrapper.run(...)` to `cute_dsl_fused_moe_nvfp4(...)`
directly, then deprecate the wrapper. Lee maintains both fi and
SGLang sides, so the cross-project coordination cost normally
faced by such a migration is essentially zero — but the question
of whether to pursue is not decided.

Heavily caveated as a future consideration that **may not be
pursued**. Priority is explicitly lower than the existing PR queue
(PR flashinfer-ai#3171, flashinfer-ai#3198, bucket-cap-fix, prealloc-fix, convergence) and
Phase 1 in follow-up flashinfer-ai#11.

Captured for two reasons:

1. The 5-minute pass on SGLang's `flashinfer_cutedsl.py:ensure_cutedsl_wrapper()`
   surfaces specific gaps in the functional API that would need to
   be closed before any migration. Documenting them here means
   future-Lee (or future-someone) doesn't have to re-derive them.

2. The thin-adapter refactor and the deprecate-the-wrapper migration
   are alternatives, not sequential. If migration is pursued, the
   thin-adapter is redundant. If it's not, the thin-adapter is the
   recommended cleanup. The relationship needed to be on the record.

Captured fi-side prerequisites (if pursued):

- Add `moe_sort_buffers`, `gemm1_out`, `gemm1_out_scale` parameters
  to `cute_dsl_fused_moe_nvfp4`. Forward to `_moe_core_impl` (which
  already accepts them). ~15 lines, additive (existing wrapper
  keeps working).
- Verify `CuteDslFusedMoENvfp4Runner` per-call construction doesn't
  bust the autotune cache or add overhead.
- CUDA-graph capture/replay validation of `cute_dsl_fused_moe_nvfp4`
  under SGLang-style preallocation.

Captured benefits / risks / decision criteria. Status: NOT decided;
revisit only after the existing PR queue settles.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…r-ai#3198 rebased onto main

PR flashinfer-ai#3171 (`fix(cute_dsl/moe): correct tile_size=256 gemm2 tactic
enumeration`) merged into `flashinfer-ai/flashinfer:main`
2026-04-30 as squash-merge commit `070fabf0`.

PR flashinfer-ai#3198 (`fix(cute_dsl/moe): correct off-by-one in get_max_num_tiles
to match TRT-LLM`) was originally stacked on PR flashinfer-ai#3171 (3 commits
on top of `ede2225a`). After flashinfer-ai#3171 merged as a squash-merge, PR
flashinfer-ai#3198 had a test-file conflict against main because git couldn't
reconcile the unsquashed pre-merge commits (`e2330e73` +
`f0cf8cd0`) with the squashed `070fabf0` on main. Rebased PR flashinfer-ai#3198
onto upstream main directly via
`git rebase --onto FETCH_HEAD f0cf8cd0` — drops the now-redundant
PR flashinfer-ai#3171 commits and replays just the off-by-one fix. New HEAD:
`7492ff11`. Single-commit clean branch off main, force-pushed to
fork. PR is now mergeable on GitHub (status went from CONFLICTING
to MERGEABLE / BLOCKED-on-checks).

Audit-doc updates:

- Final state (lines 10-11): PR flashinfer-ai#3171 marked MERGED; PR flashinfer-ai#3198
  reframed as rebased onto main, single clean commit.
- Per-component table (line 451): PR flashinfer-ai#3171 status flipped from
  "pending review" to "MERGED 2026-04-30 as `070fabf0`".
- Issue flashinfer-ai#3067 narrative (line 962): status updated to merged.
- Off-by-one PR section (line 1651): describes the rebase.
- Wrapper-fix gating note (line 1306): added 2026-04-30 update
  that the flashinfer-ai#3171 gating condition is removed.

Memory entries updated separately (closed-audit, prealloc-bias,
PR flashinfer-ai#3198 dedicated, MEMORY.md hook for flashinfer-ai#3198).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…erged 2026-05-04

PR flashinfer-ai#3198 merged 2026-05-04 18:58 UTC as squash-merge commit `393e83ea`.
fi's `get_max_num_tiles` formula now matches TRT-LLM's compact closed-
form, eliminating the +1 tile over-allocation when (E - L) % T != 0.

Updates the line-11 final-state entry and the line-4369 PR queue
annotation in follow-up flashinfer-ai#12.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

…me input (#3216)

<!-- .github/pull_request_template.md -->

## 📌 Description

The autotuner's `DynamicTensorSpec` in
`flashinfer/fused_moe/cute_dsl/tuner.py` declared `gen_tuning_buckets`
as the pre-computed tuple `get_hybrid_num_tokens_buckets(8192)` and
`map_to_tuning_buckets` as `lambda x: map_to_hybrid_bucket(x,8192)`. The
hardcoded 8192 cap silently clamped any runtime workload larger than
that to the 8192-bucket's cached tactic — at DeepSeek-V3 prefill
(N=16384) fi profiled at half the per-expert workload and used a tactic
optimized for the wrong shape.

This PR replaces the pre-computed tuple with the bare callable form
(`get_hybrid_num_tokens_buckets`) and switches the mapper to the
uncapped variant `map_to_hybrid_bucket_uncapped` (added alongside the
hybrid-bucket scheme for exactly this case). The autotuner now invokes
them with the actual input dim at autotune time, matching TRT-LLM's
pattern at `cute_dsl_custom_ops.py:2390-2391` and flashinfer's own
pattern at `gemm/gemm_base.py:_FP8_GEMM_SM100_TUNING_CONFIG`.

## 🔍 Related Issues

#3171
#3198
#3115

## 🚀 Pull Request Checklist

Thank you for contributing to FlashInfer! Before we review your pull
request, please make sure the following items are complete.

### ✅ Pre-commit Checks

- [x] I have installed `pre-commit` by running `pip install pre-commit`
(or used your preferred method).
- [x] I have installed the hooks with `pre-commit install`.
- [x] I have run the hooks manually with `pre-commit run --all-files`
and fixed any reported issues.

> If you are unsure about how to set up `pre-commit`, see [the
pre-commit documentation](https://pre-commit.com/).

## 🧪 Tests

- [x] Tests have been added or updated as needed.
- [x] All tests are passing (`unittest`, etc.).

## Reviewer Notes

<!-- Optional: anything you'd like reviewers to focus on, concerns, etc.
-->


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* MoE autotuner now uses uncapped dynamic hybrid bucket mapping instead
of a fixed-bounded set, improving adaptation to varying input token
sizes.

* **Tests**
* Added offline tests validating autotuner bucket configuration: dynamic
bucket generation, responsiveness to input size, monotonic mapping
behavior, large-input scaling, and alignment with expected power-of-2
bucket values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>