Creating the FusedMoEModularKernel on every forward pass can introduce unnecessary overhead. It's better to initialize it once and cache it for subsequent calls. This can be done with lazy initialization within the apply method.

        if not hasattr(self, "_kernel"):
            self._kernel = None

        if self._kernel is None:
            from vllm.model_executor.layers.fused_moe.flashinfer_cutlass_moe import (
                FlashInferExperts,
            )
            from vllm.model_executor.layers.f

The weight tensors w1 and w2 are unconditionally cast to torch.long. However, this cast should only occur when self.quant_dtype == 'mxfp8' (i.e., for the use_mxfp8_act_scaling=True path). For the else branch (use_w4_group_scaling=True), the weights should be passed as-is (as torch.uint8), as was done in the previous implementation. This incorrect casting can lead to runtime errors or incorrect computation in the kernel.

            if self.quant_dtype == "mxfp8":
                fc1_expert_weights = w1.view(torch.long)
                fc2_expert_weights = w2.view(torch.long)
                fake_input_scale = torch.ones(
                    self.moe_config.num_experts, device=hidden_states.device
                )
                quant_scales = [
                    self.w1_scale.view(torch.int32),
                    fake_input_scale,
                    self.w2_scale.view(torch.int32),
                    fake_input_scale,
                ]
                use_mxfp8_act_scaling = True
            else:
                fc1_expert_weights = w1
                fc2_expert_weights = w2
                assert hidden_states.dtype == torch.bfloat16
                quant_scales = [
                    self.w1_scale,
                    self.w2_scale,
                ]
                a1q_scale = None
                use_w4_group_scaling = True

This refactoring seems to have introduced a potential issue. The previous implementation for the SM100_FI_MXFP4_MXFP8_CUTLASS backend called mxfp8_quantize(x, True, 32), with is_sf_swizzled_layout=True. The new modular kernel path, via MoEPrepareAndFinalizeNoEP and moe_kernel_quantize_input, effectively calls mxfp8_quantize with is_sf_swizzled_layout=False. This discrepancy might lead to incorrect behavior or performance degradation for this backend, as the activation scales will not have the swizzled layout expected by the kernel when weight scales are swizzled. Please ensure the is_sf_swizzled_layout flag is correctly propagated or handled for this backend.

switch to h100 due to resource contraints (we have much more h100 in the ci)

ditto, I think H200 is only for 8xH200

Changed

These are just hardcoded values right? (AFAICT they are: https://github.com/zyongye/vllm/blob/d7d68c3127bc27d97b20ceb901068e709c430bd5/vllm/model_executor/layers/quantization/mxfp4.py#L419-L430)

In that case, I think we should avoid passing these via the quant config and instead just having these parameters in the Kernel itself. This will make things clearer and reduce the surface of the API contract

I moved inside the FlashInferExperts into the init phase.

I dont think this is needed. I think this kernel does support EP

NOTE: the noEP thing here is misnamed. It should be NoDPEP

The kernel interface actually dispatch to multiple kernels. It will error out when I run EP.

I think we should preserve the existing args to just avoid future footguns

I changed it back. Earlier I thought we should align this with nxfp4 quantization function signature.