The variables hidden_states_scale, gemm1_weights_scale, and gemm2_weights_scale are only defined within if blocks. If the conditions are false, these variables will not be defined, leading to a NameError when they are used in the moe_op.trtllm_fp4_block_scale_moe call on line 1094. You should initialize them to None before the conditional blocks to ensure they are always defined.

            extra_input_idx = 0
            hidden_states_scale = None
            if trtllm_gen_dtype_has_scale(self.dtype_act):
                hidden_states_scale = extra_inputs[extra_input_idx]
                extra_input_idx += 1
            gemm1_weights_scale = None
            gemm2_weights_scale = None
            if trtllm_gen_dtype_has_scale(self.dtype_weights):
                gemm1_weights_scale = extra_inputs[extra_input_idx]
                gemm2_weights_scale = extra_inputs[extra_input_idx + 1]
                extra_input_idx += 2

This line is unreachable because the default case in the switch statement above it will always throw an exception with TORCH_CHECK(false, ...). The compiler should have issued a warning about this. Please remove this line.

The value DtypeTrtllmGen.MxE4m3 is duplicated in this list. While this doesn't cause a functional issue, it's redundant and should be cleaned up for better code clarity.

    if dtype in [
        DtypeTrtllmGen.MxE4m3,
        DtypeTrtllmGen.E2m1,
        DtypeTrtllmGen.MxE2m1,
    ]:

The assignment self.top_k = top_k on line 986 is a duplicate of the assignment on line 982. Please remove the redundant line.

This print statement appears to be for debugging purposes. It should be removed from the final code to avoid polluting the output during execution.

There is a typo in the variable name tunning_config. It should be tuning_config for consistency with the rest of the autotuner code.

            tuning_config = (
                MoERunner.tuning_config_no_hidden_states_scales
                if hidden_states_scale is None
                else MoERunner.tuning_config_with_hidden_states_scales
            )

this currenly finds the type by Dtype's Uid bits.
would it be more maintainable if we simply pass strings (so in code review, the case will more obviously match the type name) and document at the binding interface that the available options are in DtypeDecl.h Dtype

the current approach has the advantage of being checked early using class DtypeTrtllmGen(IntEnum). so no objections, just raising options

@aleozlx Thanks for the suggestion! What about defining a new enum in trtllm_fused_moe_kernel_launcher.cu and use macros to map it to btg::Dtype. Then expose to python.

sure . that improves readability. or we can use class Dtype(IntEnum) i proposed in the other thread and remove another conversion. and hopefully __new__ serves to hide impl details in a way that user won't rely on the value (compared to a separate function). although the value changes can cause breakage thru the cpp interface, so we just have to align on something early.

or we can borrow the bits formation from TLLM_ENCODE_DTYPE, and allowing the deletion of get_dtype() conversion

e.g.

class Dtype(IntEnum):
    def __new__(cls, block_format_bit, signed_bit, integer_bit, num_bits, uid):
        value = (block_format_bit << 24) | (signed_bit << 20) | (integer_bit << 16) | (num_bits << 8) | uid
        obj = int.__new__(cls, value)
        obj._value_ = value
        return obj
    
    # keep the values in sync with include/flashinfer/trtllm/batched_gemm/trtllmGen_bmm_export/trtllm/gen/DtypeDecl.h
    Bfloat16 = (0, 1, 0, 16, 0)
    Bool     = (0, 0, 1, 1,  1)
    E2m1     = (1, 1, 0, 4,  2)
    E2m3     = (1, 1, 0, 6,  3)
    E3m2     = (1, 1, 0, 6,  4)
    E4m3     = (0, 1, 0, 8,  5)
    E5m2     = (0, 1, 0, 8,  6)
    Fp16     = (0, 1, 0, 16, 7)
    Fp32     = (0, 1, 0, 32, 8)
    Int8     = (0, 1, 1, 8,  9)
    Int32    = (0, 1, 1, 32, 10)
    Int64    = (0, 1, 1, 64, 11)
    MxE2m1   = (1, 1, 0, 4,  12)
    MxE4m3   = (1, 1, 0, 8,  13)
    UE8m0    = (0, 0, 0, 8,  14)
    UInt8    = (0, 0, 1, 8,  15)
    UInt16   = (0, 0, 1, 16, 16)
    UInt32   = (0, 0, 1, 32, 17)
    UInt64   = (0, 0, 1, 64, 18)
    UInt128  = (0, 0, 1, 128, 19)
    Void     = (0, 1, 0, 0,  20)

shall we always append? so if users are using the optional arguments positionally, they won't break.

or i guess we should have put optionals after , /, ... in hind sight to prevent them from being used positionally, which is easier to break

Just a minor note: we should also take care of torch.float4_e2m1x2 for torch 2.8+

Thanks. But I didn't see float4_e2m1fn_x2 is used anywhere in the flashinfer? Will we add it altogether in the future?

Not yet, we need to prepare for it when frameworks are all upgrading to torch 2.8.

It could be done in later PRs.

This assertion make vllm gpt-oss fail. In that case the scale is tensor([1., 1., 1., ..., 1., 1., 1.], device='cuda:0',dtype=torch.float8_e4m3fn)

@weireweire I created this pr to fix this. Currently vllm's flashinfer tag is 0.2.12. I can mark this pr as ready and increment the flashinfer tag to 0.2.13

This line actually breaks DeepSeek v3 routing. I have left suggest changes in #1494