The logic for initializing the different KV pool types for out-of-tree platforms involves significant code duplication, especially for the constructor arguments. This can be refactored to improve readability and maintainability by extracting common arguments into a dictionary.

        if current_platform.is_out_of_tree() and not self.mambaish_config:
            pool_args = {
                "max_total_num_tokens": self.max_total_num_tokens,
                "page_size": self.page_size,
                "dtype": self.kv_cache_dtype,
                "layer_num": self.num_effective_layers,
                "device": self.device,
                "enable_memory_saver": self.server_args.enable_memory_saver,
                "start_layer": self.start_layer,
                "end_layer": self.end_layer,
            }

            if self.use_mla_backend and is_nsa_model:
                PoolCls = current_platform.get_nsa_kv_pool_cls()
                pool_args.update({
                    "kv_lora_rank": self.model_config.kv_lora_rank,
                    "qk_rope_head_dim": self.model_config.qk_rope_head_dim,
                    "kv_cache_dim": self.calculate_mla_kv_cache_dim(),
                    "index_head_dim": get_nsa_index_head_dim(
                        self.model_config.hf_config
                    ),
                })
            elif self.use_mla_backend:
                PoolCls = current_platform.get_mla_kv_pool_cls()
                pool_args.update({
                    "kv_lora_rank": self.model_config.kv_lora_rank,
                    "qk_rope_head_dim": self.model_config.qk_rope_head_dim,
                    "index_head_dim": (
                        self.model_config.index_head_dim if is_nsa_model else None
                    ),
                })
            else:
                PoolCls = current_platform.get_mha_kv_pool_cls()
                pool_args.update({
                    "head_num": self.model_config.get_num_kv_heads(
                        get_attention_tp_size()
                    ),
                    "head_dim": self.model_config.head_dim,
                })

            self.token_to_kv_pool = PoolCls(**pool_args)

We should call apply_hooks right after the process is created, so the hook can override anything including Scheduler and TpWorker.
A good place is run_scheduler_process

Thanks for the suggestion! Done — load_plugins() (which calls HookRegistry.apply_hooks() internally) is now called in run_scheduler_process() before Scheduler() construction. The call in tp_worker.py has been removed since TpModelWorker is always created inside the Scheduler process, so it was redundant.

simplify the API, we should reduce the call to only one function call.

Done — load_plugins() is now the single entry point. It discovers plugins, executes them, and calls HookRegistry.apply_hooks() internally. Callers no longer need a separate apply_hooks() step.

The name sglang.general_plugins is confusing. We do not have a folder with this name sglang.general_plugins. Can you improve the name?

Makes sense, thanks! Renamed to sglang.plugins — shorter and consistent with the actual package structure.

@yeahdongcn multi modal and SRT handle is cuda alike differently wrt MUSA. leave as a note that this needs to be resolved later?

In DeviceMixin, we unify 'is_cuda_alike()' as CUDA+ROM+MUSA. Therefore, once MM inherits DeviceMixin, it will automatically obtain a consistent definition. The SRT prefix in SRTPlatform (as in the example you provided) is intended to distinguish it from future MMPPlatform (DeviceMixin) - both share the same DeviceMixin foundation but carry subsystem specific factory methods. If you have better naming suggestions, I would be happy to make adjustments.

Sorry, I missed that comment. Yes, the behavior differs between multimodal_gen and SRT. In multimodal_gen, current_platform.is_cuda_alike() is only used to determine graph capture behavior and the communication method.

In contrast, in SRT, it is used to decide whether certain kernels can be imported.

For example:

if _is_cuda_alike:
    from sgl_kernel import (
        cutlass_w4a8_moe_mm,
        get_cutlass_w4a8_moe_mm_data,
    )

this is missing hardware types (e.g NPU is one of the top of my head) and if following my other PR comments, this should be a mixin.

Thanks, addressed! PlatformEnum now covers all current hardware: CUDA, ROCM, CPU, XPU, MUSA, NPU, TPU, MPS, OOT, UNSPECIFIED. All identity queries (is_cuda(), is_npu(), is_musa(), etc.) are defined in DeviceMixin and derived automatically from _enum.

this approach has a few things we need to address:

1. We should have the multimodal and srt platforms share the same core functionalities as multimodal already has a platform object (which will at some point inherit from our base abstraction). We want to avoid diamond inheritance though, so we should move to a device mixin approach IMO.
2. Future ModelRunner / SpecDec refactor: With a mixin, ModelRunner can compose with just DeviceMixin for device operations without needing the full SRTPlatform. This is cleaner than an ABC hierarchy for the planned SpecDec refactor.
3. As far as I understand, the @classmethod decorators / implementation don't focus around failing fast. If a hardware platform does not support something it should be NotImplementedError

Example

# Mixin -- device operations only, no ABC chain
class DeviceMixin:
    name: str
    _enum: PlatformEnum
    def get_device(self, local_rank) -> torch.device: ...
    def get_device_name(self, device_id=0) -> str: ...
    def get_distributed_backend(self) -> str: ...
    def get_available_memory(self, device_id=0) -> tuple[int, int]: ...
    # ... ~15 shared device/memory/distributed methods

class SRTPlatform(DeviceMixin):
    # SRT-specific: graph runners, KV pools, quant, compilation
    def get_graph_runner_class(self) -> type: ...
    def get_kv_pool_class(self, use_mla: bool) -> type: ...

class MMPlatform(DeviceMixin):
    # MM-specific methods
    def get_attn_backend_cls_str(self) -> str: ...

# External packages -- no diamond inheritance!
class CudaDeviceMixin(DeviceMixin):
    name = "npu"
    def get_device(self, local_rank): return torch.device("npu", local_rank)
    def get_distributed_backend(self): return "hccl"

class CudaSRTPlatform(SRTPlatform, CudaDeviceMixin):  # clean MRO
    ...
class CudaMMPlatform(MMPlatform, CudaDeviceMixin):    # clean MRO
    ...

Great suggestion, thank you for the detailed design! Implemented this pattern. Created DeviceMixin in platforms/device_mixin.py with identity queries + device operations (all raising NotImplementedError). SRTPlatform extends DeviceMixin for SRT-specific factory methods. OOT plugins compose via MySRTPlatform(SRTPlatform, MyDeviceMixin) — clean MRO, no diamond inheritance. The MMPlatform(DeviceMixin) slot is ready for when the multimodal subsystem migrates to this pattern.

really what this should be is just

from sglang.srt.platforms import current_platform
current_platform.get_piecewise_backend_cls()

we do not care that the platform is OOT, the hardware plugin itself should be able to implement a PiecewiseBackend Class that we can run (and we should know from flags / server args /. etc wherever we already determine if piecewise backend can be used)

Ideally, we have a unified platform dispatch

Completely agree with the direction! This is tracked as part of the future migration described in plugin.md under "Current Scope & Future Direction". Currently, in-tree platforms (CUDA/NPU) still use direct imports rather than the platform interface. Once each in-tree backend is migrated to its own SRTPlatform subclass, the if/elif chain here will collapse into a single current_platform.get_piecewise_backend_cls() call. For this PR, we took the minimal non-intrusive approach of adding the OOT branch alongside existing logic.

why are these in an interface and not part of the cuda implementation? This comments can be seen as a paint brush for quite a few things in here. Let's chat more about this

Cleaned this up — CUDA-specific methods have been removed from the base class. Methods now raise NotImplementedError (fail-fast). support_cublas() has been deleted entirely. support_cuda_graph() is kept — Many non-CUDA platforms (ROCm, MUSA, and potentially OOT devices) support a similar graph capture mechanism. In the codebase it gates init_device_graphs() and disable_piecewise_cuda_graph, which are hardware-agnostic graph capture paths. It defaults to False (conservative), so platforms opt in explicitly. We could consider renaming it to support_device_graph() in a follow-up if the naming feels misleading.

is this not numerically safe call? (will default to 0)

and same comment as https://github.com/sgl-project/sglang/pull/21388/changes#r2999364494.

the bottom of these should be pinned to their platform implementations

Thanks for looking closely. Without the OOT branch, the if/elif chain falls through with no return value (None), which causes a TypeError downstream when used in arithmetic. The OOT branch calls current_platform.get_device_total_memory() which OOT plugins must implement (raises NotImplementedError if not).

I really like this! However.... we can probably (and should) make this a decorator

@sglang_hook("sglang.srt.managers.scheduler.Scheduler.schedule", type="around")
def my_timer(original_fn, *args, **kwargs):
    ...

Makes things easier (especially if we ever expose this as some form of JIT to something being hooked)

Added the sglang_hook decorator as suggested. Both the imperative HookRegistry.register() API and the decorator @sglang_hook(target, type=HookType.AROUND) are now available, so plugin authors can pick whichever style they prefer.

this entire class is also just

@hook("sglang.srt.some.Class", type="replace")

what do you think?

Done! ClassReplacer has been merged into HookRegistry and class_replacer.py is deleted.
Class replacement is now a special case of HookType.REPLACE within the unified hook system. The plugin_hook decorator handles both function hooks and class replacement

+1 to @merrymercy comments. I do not like this name (slack me if u want and I can help workshop)

Renamed to sglang.plugins — shorter and consistent with the actual package structure.

this logic can be simplified via some toml work

def discover_platforms() -> dict[str, type[SRTPlatform]]:
    platforms: dict[str, type[SRTPlatform]] = {}

    # 1. Built-in (always available)
    from sglang.srt.platforms.cuda import CUDASRTPlatform
    from sglang.srt.platforms.cpu import CPUSRTPlatform
    platforms["cuda"] = CUDASRTPlatform
    platforms["cpu"] = CPUSRTPlatform
    # rest of in tree platforms

    # 2. entry_points from pip-installed packages
    for ep in importlib.metadata.entry_points(group="sglang.platforms"):
        try:
            cls = ep.load()
            platforms[ep.name] = cls
        except Exception as e:
            logger.warning(f"Failed to load platform plugin {ep.name}: {e}")

    # 3. SGLANG_PLATFORM_PLUGIN override (dev/testing)
    if plugin_spec := os.environ.get("SGLANG_PLATFORM_PLUGIN"):
        name, qualname = plugin_spec.split(":", 1) if ":" in plugin_spec else (plugin_spec, plugin_spec)
        cls = resolve_obj_by_qualname(qualname)
        platforms[name] = cls

    return platforms


def get_platform(device: str) -> SRTPlatform:
    """Return the platform for a given device string. Caches instances."""
    # Looks up from discovered platforms, instantiates, caches
    ...

packages register via pyproject.toml:

[project.entry-points."sglang.platforms"]
npu = "sglang_npu.platform:NPUSRTPlatform"

Thank you for the thoughtful design! We simplified init.py to only do entry_points discovery for OOT platforms. Built-in platforms (CUDA/ROCm/NPU/XPU) are not registered as platform plugins yet — they continue to use the existing is_cuda() utility functions (432+ call sites across 195 files). This avoids a massive refactor in this PR while achieving the same goal of clean OOT extensibility. Once the interfaces stabilize, built-in platforms can be gradually migrated to the same plugin architecture, which aligns with your suggested end-state.

as a general comment shouldn't quite a few of these be instant methods and not class methods?

Agreed, thanks! All methods in both DeviceMixin and SRTPlatform are now instance methods. current_platform is an instance (lazy singleton), so everything is accessed via current_platform.some_method().

Do you think it's a good idea to have a class member for priority? @alexnails

# OOT
class OOTPlatform(Platform):
    priority = 500

# built-in
class CUDAPlatform(Platform):
    priority = 100

class CPUPlatform(Platform):
    priority = 10

there are many duplications of class CustomPlatform(SRTPlatform).
Can we extract it and make it shared across tests?

Done. Extracted a shared _StubPlatform subclass that provides minimal concrete implementations, replacing the repeated CustomPlatform(SRTPlatform) boilerplate across tests.

Some *_noop and *_raises_not_implemented tests here are actually testing the "python language itself" instead of sglang logic.
The related methods of DeviceMixin are not overrided by a real device mixin class.

Maybe we should consider removing them.

Done. Removed all *_raises_not_implemented and *_noop tests, as well as other tests that only verify Python behavior (default return values, no-op methods, repr formatting, trivial overrides). The remaining tests focus on SGLang-specific logic: classification rules, validation boundaries, error paths, and branching.

We may need some platform discovery and _resolve_platform related tests, which are very important sglang logic.

Done. Added comprehensive tests for platform discovery: _resolve_platform (both SGLANG_PLATFORM env branch and auto-discover branch), _load_platform_class qualname resolution with type validation