add rope scaling as a cli arg so openai server can load rope scaled models

Fix key cache block shape.

Deterministic OpenVINO inference

merge code

BA-78554: Jurassic 2.5

* worked on jurasic2.5 configuration file, updated jurassic2_5 modeling file to support alternating experts/attn layers

* finished working the forward pass of jurassic3.py

* finished working the forward pass of jurassic3.py

* finished working the forward pass of jurassic3.py

* jurassic_3 modeling file works, uses dummy weights initialized by "dummy" flag. Tokenizer raises issues, for now copying the mixtral tokenizer

* changed default tokenizer vocab values, loading of custom .pt weight files works.

* removed notebook

* merging master to jurassic-2.5 to reset head

* Merge branch 'master' into jurassic-2.5

* align to master

Approved-by: Tomer Asida
Approved-by: Mor Zusman

Triton compilation fix

Group Gemm Version

fea: support rfork

Fix vllm-project#1: Runtime instruction tensor fill
- Compile-time tensor is now a TEMPLATE (opcodes set, dimensions zero)
- Runtime code fills M/N/K and A/B/C pointers from function arguments
- DAG analysis resolves which params feed each GEMM's A/B/C inputs
- Intermediate buffers allocated/freed for inter-action data flow

Fix vllm-project#2: bar.sync deadlock
- GEMM pipeline emits bar.sync 0 expecting all 640 CTA threads
- Only 128 threads (4 consumer warps) run the GEMM handler
- Post-process replaces bar.sync 0 with bar.sync 1, 128 (scoped barrier)

Fix vllm-project#3: Shared memory collision (verified non-issue)
- Static smem (inst_state, mbarrier arrays) is separate from dynamic_smem
- GEMM cp.async buffers use dynamic_smem offset 0 — no collision
- Added compile-time assertions that GEMM smem fits in dynamic region
- Documented Phase 3 TODO for when pages carry inter-op data

314 tests pass (84 ferrite-macros + 218 ferrite-ptx + 12 ferrite-runtime).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Nick Mitchell <nickm@us.ibm.com>

…t#38472)

Add 17 unit tests covering the KV cache memory-management fixes from the
previous commit.  Tests run without NCCL, ZMQ, or CUDA by constructing a
minimal engine object via object.__new__ and stubbing the two compiled
extensions (vllm._C, vllm._C_stable_libtorch) in sys.modules.

Test classes
------------
TestRecvTensorPopsFromStore (4 tests)
  - entry_absent_after_recv_tensor          (regression guard vllm-project#1)
  - buffer_size_decremented_after_recv_tensor
  - multiple_layers_all_popped
  - recv_store_does_not_grow_under_sustained_load  (high-QPS OOM scenario)

TestRecvTensorPoolBackedEntries (4 tests)
  - pool_free_called_immediately             (regression guard vllm-project#2)
  - pool_entry_removed_from_recv_store
  - pool_free_not_called_for_none_tensor     (edge case: listener OOM path)
  - pool_free_called_once_per_entry_not_twice

TestGetFinishedStragglerCleanup (6 tests)
  - straggler_gpu_tensor_removed_and_buffer_decremented
  - straggler_pool_tensor_freed
  - multiple_straggler_layers_all_cleaned
  - only_finished_request_cleaned_not_others
  - noop_when_tensors_already_consumed
  - send/recv_tracking_dict_cleaned_by_get_finished

TestGetFinishedNoCompileLayersParamUnused (1 test)
  - Verifies straggler cleanup works even with no_compile_layers={}
    (the old iteration approach would silently skip all cleanup here)

TestPoolFreeCalledOnce (1 test)
  - combined_lifecycle_exactly_one_free: ensures pool.free is called
    exactly once across recv_tensor + get_finished, not twice.

Reproduction of original bugs confirmed:
  - Old recv_tensor (read without pop) leaves entry in recv_store: CONFIRMED
  - Old recv_tensor (no pool.free) leaks pinned RAM: CONFIRMED
  All 17 tests pass against the fixed code.

Co-authored-by: GitHub Copilot

…t#38472)

Add 17 unit tests covering the KV cache memory-management fixes from the
previous commit.  Tests run without NCCL, ZMQ, or CUDA by constructing a
minimal engine object via object.__new__ and stubbing the two compiled
extensions (vllm._C, vllm._C_stable_libtorch) in sys.modules.

Test classes
------------
TestRecvTensorPopsFromStore (4 tests)
  - entry_absent_after_recv_tensor          (regression guard vllm-project#1)
  - buffer_size_decremented_after_recv_tensor
  - multiple_layers_all_popped
  - recv_store_does_not_grow_under_sustained_load  (high-QPS OOM scenario)

TestRecvTensorPoolBackedEntries (4 tests)
  - pool_free_called_immediately             (regression guard vllm-project#2)
  - pool_entry_removed_from_recv_store
  - pool_free_not_called_for_none_tensor     (edge case: listener OOM path)
  - pool_free_called_once_per_entry_not_twice

TestGetFinishedStragglerCleanup (6 tests)
  - straggler_gpu_tensor_removed_and_buffer_decremented
  - straggler_pool_tensor_freed
  - multiple_straggler_layers_all_cleaned
  - only_finished_request_cleaned_not_others
  - noop_when_tensors_already_consumed
  - send/recv_tracking_dict_cleaned_by_get_finished

TestGetFinishedNoCompileLayersParamUnused (1 test)
  - Verifies straggler cleanup works even with no_compile_layers={}
    (the old iteration approach would silently skip all cleanup here)

TestPoolFreeCalledOnce (1 test)
  - combined_lifecycle_exactly_one_free: ensures pool.free is called
    exactly once across recv_tensor + get_finished, not twice.

Reproduction of original bugs confirmed:
  - Old recv_tensor (read without pop) leaves entry in recv_store: CONFIRMED
  - Old recv_tensor (no pool.free) leaks pinned RAM: CONFIRMED
  All 17 tests pass against the fixed code.

Co-authored-by: GitHub Copilot
Signed-off-by: saif <contact@saifmb.com>

…n-files-overview

Add Chinese documentation for vLLM framework structure and GEMM call chains

[WIP] Fix Mamba state contamination in KV cache block reuse

Rebase of PR vllm-project#33315 onto current main. Adds max_tokens_per_doc parameter
to rerank requests, matching Cohere and Jina rerank APIs. Documents
longer than this limit are truncated before scoring.

Handles all three cross-encoder code paths:
- Cross-encoder with sep token (tokenizer built-in truncation)
- Chat template / Jinja path (text truncation before template)
- Score template path (text truncation before template)

Also supports offline usage via PoolingParams(extra_kwargs={"max_tokens_per_doc": N}).

Addresses reviewer feedback from original PR:
- Offline support via PoolingParams (noooop vllm-project#1)
- Score template compatibility tests (noooop vllm-project#2)
- Tests across BAAI/bge-reranker-base, BAAI/bge-reranker-v2-gemma,
  and Qwen/Qwen3-Reranker-0.6B

Original PR: vllm-project#33315
Original author: hustxiayang

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rebase of PR vllm-project#33315 onto current main. Adds max_tokens_per_doc parameter
to rerank requests, matching Cohere and Jina rerank APIs. Documents
longer than this limit are truncated before scoring.

Handles all three cross-encoder code paths:
- Cross-encoder with sep token (tokenizer built-in truncation)
- Chat template / Jinja path (text truncation before template)
- Score template path (text truncation before template)

Also supports offline usage via PoolingParams(extra_kwargs={"max_tokens_per_doc": N}).

Addresses reviewer feedback from original PR:
- Offline support via PoolingParams (noooop vllm-project#1)
- Score template compatibility tests (noooop vllm-project#2)
- Tests across BAAI/bge-reranker-base, BAAI/bge-reranker-v2-gemma,
  and Qwen/Qwen3-Reranker-0.6B

Original PR: vllm-project#33315
Original author: hustxiayang

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rebase of PR vllm-project#33315 onto current main. Adds max_tokens_per_doc parameter
to rerank requests, matching Cohere and Jina rerank APIs. Documents
longer than this limit are truncated before scoring.

Handles all three cross-encoder code paths:
- Cross-encoder with sep token (tokenizer built-in truncation)
- Chat template / Jinja path (text truncation before template)
- Score template path (text truncation before template)

Also supports offline usage via PoolingParams(extra_kwargs={"max_tokens_per_doc": N}).

Addresses reviewer feedback from original PR:
- Offline support via PoolingParams (noooop vllm-project#1)
- Score template compatibility tests (noooop vllm-project#2)
- Tests across BAAI/bge-reranker-base, BAAI/bge-reranker-v2-gemma,
  and Qwen/Qwen3-Reranker-0.6B

Original PR: vllm-project#33315
Original author: hustxiayang

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rebase of PR vllm-project#33315 onto current main. Adds max_tokens_per_doc parameter
to rerank requests, matching Cohere and Jina rerank APIs. Documents
longer than this limit are truncated before scoring.

Handles all three cross-encoder code paths:
- Cross-encoder with sep token (tokenizer built-in truncation)
- Chat template / Jinja path (text truncation before template)
- Score template path (text truncation before template)

Also supports offline usage via PoolingParams(extra_kwargs={"max_tokens_per_doc": N}).

Addresses reviewer feedback from original PR:
- Offline support via PoolingParams (noooop vllm-project#1)
- Score template compatibility tests (noooop vllm-project#2)
- Tests across BAAI/bge-reranker-base, BAAI/bge-reranker-v2-gemma,
  and Qwen/Qwen3-Reranker-0.6B

Original PR: vllm-project#33315
Original author: hustxiayang

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Rebase of PR vllm-project#33315 onto current main. Adds max_tokens_per_doc parameter
to rerank requests, matching Cohere and Jina rerank APIs. Documents
longer than this limit are truncated before scoring.

Handles all three cross-encoder code paths:
- Cross-encoder with sep token (tokenizer built-in truncation)
- Chat template / Jinja path (text truncation before template)
- Score template path (text truncation before template)

Also supports offline usage via PoolingParams(extra_kwargs={"max_tokens_per_doc": N}).

Addresses reviewer feedback from original PR:
- Offline support via PoolingParams (noooop vllm-project#1)
- Score template compatibility tests (noooop vllm-project#2)
- Tests across BAAI/bge-reranker-base, BAAI/bge-reranker-v2-gemma,
  and Qwen/Qwen3-Reranker-0.6B

Original PR: vllm-project#33315
Original author: hustxiayang

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Jesus Federico <jefp@amazon.com>