rebase to 2561ed0

* add a sgl_kernel.cpu wrapper for CPU OPs in sgl-kernel

* add wrapper for attention OPs

* set default value of is_vnni to True

* add a sgl_kernel.cpu wrapper for CPU OPs in sgl-kernel

* add wrapper for attention OPs

* set default value of is_vnni to True

* add a sgl_kernel.cpu wrapper for CPU OPs in sgl-kernel

* add wrapper for attention OPs

* set default value of is_vnni to True

* add a sgl_kernel.cpu wrapper for CPU OPs in sgl-kernel

* add wrapper for attention OPs

* set default value of is_vnni to True

fix model loading and add eagle3 inference

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

* Revert "port prefill optimization (sgl-project#7)"

This reverts commit ea0d028.

* improve bfloat16 gemm performance for prefilling

before:
```
gemm_bf16(native): 4.772 ms, gemm_fp8(opt): 0.000 ms, gemm_int8(opt): 0.000 ms, gemm_bf16(opt): 15.328 ms
```

after:
```
gemm_bf16(native): 4.847 ms, gemm_fp8(opt): 0.000 ms, gemm_int8(opt): 0.000 ms, gemm_bf16(opt): 3.927 ms
```

* improve fp8 gemm performance with large M

* enable amx-int8 for gemm, fused moe, shared moe and qkv_proj kernels on PyTorch 2.7

* improve int8 gemm performance with large M

* improve bf16 and int8 moe performance with large nbatches

* update naming for nb0 and nb1 in fused gemm and silu_mul kernel

* improve fp8 moe performance with large nbatches

* remove hardcode numbers

---------

Co-authored-by: mingfeima <mingfei.ma@intel.com>

[Feature] add multimodal attention for rocm and fix qwenvl load error…

…hang

PP PD dev

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

[Wave] Add wave extend attention kernel

Signed-off-by: Harsh Menon <harsh@nod-labs.com>

[Wave] Adding logit_cap and layer scaling to API

Also add support for the wave backend to the model
runner. And use Triton decode kernels for now.

[Wave] Run chunked prefill for perf comparison on Wave test

Need to rename the non chunked/regular prefill version because otherwise
rpd will treat it as the same kernel

Signed-off-by: Stanley Winata <stanley.winata@amd.com>

[Wave] Cache the function that loads the wave kernel

Also maintain a global kernel hash to avoid
recomputing the hash on every call.

[Wave] Don't specify block size and enable buffer ops

[Wave] Enable wave runtime and update scheduling API

[Wave] Update API to use wave_compile & WaveCompileOptions

[Wave] Update wave backend and extend attention to latest

[Wave] Add speculative decode kernel

Signed-off-by: nithinsubbiah <nithinsubbiah@gmail.com>

cache kernels using lru_cache

Update WaveBackend to use Wave Decode  (sgl-project#6)

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

Revert "Update WaveBackend to use Wave Decode  (sgl-project#6)" (sgl-project#7)

This reverts commit eac4599.

Wave Backend decode (sgl-project#8)

* align shapes

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

* fix

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

---------

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

Wave backend fixes (sgl-project#10)

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

More fixes to Wave decode (sgl-project#12)

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

is_causal

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

Enable the grok in3 model (sgl-project#14)

Set unique cache dir for each worker (sgl-project#16)

update kernel (sgl-project#18)

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

updated spec decode test as per wave

Signed-off-by: xintin <gaurav.verma@amd.com>

fix extend (sgl-project#23)

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

Refactor paged decode intermediate arrays shapes (sgl-project#24)

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

remove dyn symbols (sgl-project#26)

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

cleanup shapes (sgl-project#27)

Some fields were removed from `paged_decode_attention_shape`.

Signed-off-by: Ivan Butygin <ivan.butygin@gmail.com>

Remove `mha` param from Wave decode attention kernel (sgl-project#28)

Depends on iree-org/iree-turbine#1039

Signed-off-by: Paul Zhang <paul.zhang@amd.com>

nfc: fix problems reported by linting

update references from iree.turbine to wave_lang

1. rtmp_pusher (sgl-project#10): _force_close_container() (main thread) and
   _run() finally block (background thread) both read/write
   self._container without synchronization, risking a double-close
   segfault.  Add threading.Lock to protect all _container access.

2. fMP4 generator (sgl-project#13): when a client disconnects mid-stream,
   GeneratorExit is raised at a yield point and the generator exits
   without reaching the container.close() at the end, leaking the
   PyAV container and its file descriptors.  Wrap the streaming loop
   and flush in try/finally to ensure container.close() is always
   called.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* add OPTFLAG="moe,w8a8_gemm"

* Update CI workflow and evaluation guide & scripts

Signed-off-by: Xiake Sun <xiake.sun@amd.com>

* Bugfix

Signed-off-by: Xiake Sun <xiake.sun@amd.com>

* Pin transformers version

Signed-off-by: Xiake Sun <xiake.sun@amd.com>

* Update workflow for triton and pyhip install

Signed-off-by: Xiake Sun <xiake.sun@amd.com>

* Debug CI error

Signed-off-by: Xiake Sun <xiake.sun@amd.com>

* Update port to avoid conflict

Signed-off-by: Xiake Sun <xiake.sun@amd.com>

* Revert "Update port to avoid conflict"

This reverts commit dd868e5.

Signed-off-by: Li,Tingqian <tingqli.amd.com>

* Update CI workflow and scripts

Signed-off-by: Li,Tingqian <tingqli.amd.com>

* Fix bug

Signed-off-by: Li,Tingqian <tingqli.amd.com>

* Handle benchmark workflow with port conflict

Signed-off-by: Xiake Sun <xiake.sun@amd.com>

* Fix CI error

Signed-off-by: Xiake Sun <xiake.sun@amd.com>

---------

Signed-off-by: Xiake Sun <xiake.sun@amd.com>
Signed-off-by: Li,Tingqian <tingqli.amd.com>
Co-authored-by: Li,Tingqian <tingqli.amd.com>

torch compile and tuning.

sgl-project#10 Sweep 1: 3 seeds × 5 ratios. Std 3-5% of mean across all ratios;
swing 1.71× (4711→8075) reproduces within noise of original 1.91×.
Variance bands now in paper Table 1.

sgl-project#11 Setting 4 fallback rule:
- Implementation: SGLANG_XPOOL_QDEPTH_TRIGGER added to
  cross_pool_planner.py (gated, legacy preserved).
- Unit tests: 5/5 PASS.
- E2E: both arms fired 21 transfers on Phase 1+2+3 (workload doesn't
  dual-saturate; KV stays <1%). Honest finding documented in §6.4.
- Deeper fix (per-pool admission signal everywhere) is follow-up.

SETTINGS.md scoreboard reflects both items DONE.

…s 28 xfers

v9 pool-binding-shift trace produces real differentiation:
- Phase B (KV-bound 8K random): L1+L2 -37% mean TTFT vs stock
- Phase C (mixed 4K random):     L1+L2 -38% median E2E vs stock
- Cross-pool transfers: stock 0, L1-only 0, L2-only 0, L1+L2 28

Two surprising findings documented:
1. Layer 2 alone fires zero transfers — Layer 1 retention is what
   makes Layer 2 cross the firing threshold.
2. Phase A regresses with L1 (-20% TPS) because K_big=8192 hurts on
   prefix-friendly GSP. Consistent with A2's K_big=0-wins finding.
   Adaptive K_big control marked as follow-up.

Settings status: Setting 1 marked **DONE v6 NULL + v9 PASS**.
All 4 user-requested follow-ups (sgl-project#9 Q3.A 4-arm, sgl-project#10 Sweep 1
multi-seed, sgl-project#11 Setting 4 fallback rule, sgl-project#12 Setting 1 v9 trace)
now complete.

Closes architecture-notes.md "Open risks sgl-project#10": V4-Flash has NO Jinja
chat_template field in tokenizer_config.json (uses Python
encoding/encoding_dsv4.py) — sglang's automatic detection finds nothing
and falls back to a default that doesn't match. Two changes:

1. ChatTemplate(name="deepseek-v4", ...) registered in lang/chat_template.py
   right after deepseek-v3. Same role prefix/suffix structure (system
   empty, user="<｜User｜>", assistant="<｜Assistant｜>"/<｜end▁of▁sentence｜>),
   same stop_str. Thinking-mode + tool-call DSML are handled inside the
   model's decoder; this template only covers the role prefix structure
   sglang uses when it constructs prompts.
2. match_deepseek_v4 matching function registered so model paths matching
   /deepseek-v4/i (and not /base/i) auto-route to the deepseek-v4
   template. Mirrors the existing match_deepseek pattern that handles
   V3 and R1.

Reference: V4 encoding format from /tmp/v4-flash-meta/encoding/README.md
   <｜begin▁of▁sentence｜>{system}<｜User｜>{user}<｜Assistant｜><think>{reasoning}</think>{response}<｜end▁of▁sentence｜>

Phase 4 + Phase 4.5 + Phase 5 / 6 / 7 recipes all depend on this. Until
this commit landed, any sglang serve or train call against
deepseek-ai/DeepSeek-V4-Flash would fail at the chat-template lookup.