Upstream Bnb4bitQuantize.convert unwraps input_dict[key] twice (first .values()
then [0]). The patched version did only the first unwrap, which produced a list
for most weight-converter dispatch paths and raised TypeError inside Params4bit
constructor — silently masked by the broad except Exception and falling back to
original_convert (leaving experts unquantized).

Empirically: tiny DeepSeek V3 happens to dispatch experts as bare Tensors so the
bug was invisible there. Qwen3.5-35B-A3B dispatches as list[Tensor] and triggers
the TypeError on every expert param.

Reviewed-by: R1 (correctness), R3 (regression)

…-MoE warning (M7+M8)

Package-wide convention (mxfp4.py, qwen3_moe.py, qwen3_5_moe.py, qwen3_vl_moe.py,
qwen3_next_moe.py, glm4_moe.py, deepseek_v3_moe.py) gates every logger.info on
UNSLOTH_ENABLE_LOGGING. The three info logs in moe_bnb_transformers.py and the
unconditional warning when no experts found weren't gated and spammed every 4-bit
load including dense models (Phi3, GLM4 dense, Llama, Mistral, etc.).

Reviewed-by: R3 (regression), R4 (conventions)

…lause (m4+m6+M1)

- Add __all__ to stop wildcard re-export of bnb/torch/nn/Optional/etc. into
  unsloth_zoo.temporary_patches namespace.
- Tighten param_needs_quantization: only return True when Params4bit AND
  bnb_quantized=False, protecting against re-invocation after first quantize.
- Narrow patched_param_needs_quantization except from bare Exception to
  (KeyError, AttributeError) — the expected failures from get_module_from_name.

Reviewed-by: R1 (correctness), R3 (regression), R4 (conventions)

…h in detector (M3+B2)

M3: _get_base_weight calls bnb.functional.dequantize_4bit(param.data, None) when
param.quant_state has not been populated yet (meta-device placeholder, or before
.to(cuda)). Add explicit guard to fall through to other branches in that case.

B2: _is_moe_experts_module was tightened from (nn.Parameter|Tensor, ndim in (2,3))
to (Params4bit, ndim==2) OR (nn.Parameter, ndim==3). This contradicts the function's
own doc-comment ('After PEFT's nn.utils.parametrize wrapping, accessing gate_up_proj
returns torch.Tensor (not nn.Parameter), so we must accept both.'). Restore the
Tensor branch so the post-parametrize path is still detected.

Reviewed-by: R1 (correctness), R3 (regression)

…issing _original_shape (M5+M6+m5)

R2 found two related issues in patch_peft_param_wrapper_4bit_expert_shape:

M5/M6: _patched_get_param hard-codes the peft 0.18 dim-ordering convention
(num_experts, in_features, out_features = shape) and reassigns self.in_features
and self.out_features on every call. PEFT 0.19's ParamWrapper.update_layer swaps
in_features<->out_features for 3D params (gated on _did_swap_in_out_features),
then calls _move_adapter_to_device_of_base_layer which calls get_param() again.
The reassignment UN-does PEFT's swap, breaking second-adapter add_adapter and
any external reader of layer.in_features. Drop the in_features/out_features
assignment; keep num_experts (which both peft versions derive identically).

m5: the 'else: # TODO: Can we raise an error here? pass' branch silently returned
the packed Params4bit. PEFT's _get_in_out_features would then read the (K, 1)
packed shape and create LoRA factors with wrong dims. Raise ValueError explaining
the failure mode instead.

Reviewed-by: R2 (peft)

Reviewed-by: R4 (conventions)

Three changes from REV findings:

REV-narrowed-except-loses-runtime-context: patched_convert still had broad
except Exception that masked B1 for months. Narrow to (KeyError, AttributeError)
for expected get_module_from_name failures, and use logger.exception() for
unexpected failures so the traceback is preserved.

REV-m3-defer-not-fix: when _get_base_weight sees Params4bit with quant_state=None,
fall-through to subsequent branches just returns the raw packed uint8 data which
crashes grouped_mm later with a different error. Raise an actionable error
instead.

REV-typing-tuple-unused: drop unused Tuple import.

Reviewed-by: REV (independent post-fix review)

_try_attach_block_size is now defined locally in moe_utils_fp8.py (commit
87ba813), but three function bodies still tried to re-import it from
.moe_utils. With the narrowed except in forward_moe_backend, those ImportErrors
no longer silently fall through — they propagate and abort the FP8 forward.
Use the module-local helper directly.

…mm path

_forward_scaled_grouped_mm_fp8 was importing _get_grouped_lora,
_apply_grouped_lora, and _expand_grouped_bias from .moe_utils — none of
those symbols exist anywhere in the codebase, making this whole path
unrunnable on Hopper. Didn't surface on B200 because the function is gated
behind _check_torch_scaled_grouped_mm_supported() (Hopper SM 9.x only).

Mirror the inline pattern from moe_utils.forward_native_grouped_mm (the
validated reference): add small local helpers _moe_separated_lora_delta
and _expand_grouped_bias, and read the (first, second, scaling) tuple
directly from _unsloth_lora_{gate_up,down}_proj injected by
_patched_param_wrapper_forward.

When the fused gate_up_proj has 2*I == H (common in models like
DeepseekV3 and Qwen3.5MoE where intermediate_dim is half hidden_dim),
both 'standard' and 'swapped' shape checks in _detect_moe_lora_layout
pass, and the first-match-wins order picks 'swapped' incorrectly.

For DeepseekV3, the only LoRA path is the fused MoE (target_modules
doesn't attach to shared_experts), so the wrong slicing corrupts every
expert and reload accuracy drops to zero. Qwen3.5MoE was masked because
its shared_expert ALSO got Linear LoRA via standard PEFT matching, and
the shared_expert deltas dominated the codename memorization task.

Disambiguate using _did_swap_in_out_features (set by unsloth on the
PEFT wrapper at training time), defaulting to 'standard' when absent.

For FP8 base models with stored weight in float8_e4m3fn plus companion
weight_scale_inv tensors, _merge_moe_experts_file used to do
W.to(float32) which is a bit-level cast that drops the per-block
scale entirely. The LoRA delta was then added in un-scaled space
and the result re-written as fp8 with the original (now wrong)
scale, corrupting the merged checkpoint.

This patch adds two helpers in saving_utils.py:

  - _fp8_dequant_blockwise(W_fp8, scale_inv): block-wise inverse
    quant matching transformers.integrations.finegrained_fp8
    Fp8Dequantize.convert (W_real = decode_fp8(W) * scale_inv).
  - _fp8_requant_blockwise(W_bf16, block_shape, scale_dtype): per
    block max-abs quant with new_scale_inv = max_abs / 448 and
    W_fp8 = clamp(W / scale_inv, +/- 448), matching Fp8Quantize.

A new _fp8_load_for_merge helper detects fp8 + scale_inv companion
in the safetensors header and returns the dequantized bf16 weight
along with scale metadata. The per-expert merge loop in
_merge_moe_experts_file (gate_proj, up_proj, down_proj) calls this
on read, runs the existing _merge_moe_* math in bf16 / float32, and
then re-quantizes + overwrites both the weight tensor and its
weight_scale_inv slot on write.

Non-FP8 paths are unchanged: the helper short-circuits on
W.dtype != float8_e4m3fn and falls through to the existing
in-place fp8 / bf16 write logic with original dtype.

Validation:
- Qwen3-Coder-30B-A3B-Instruct-FP8: LoRA-adapter reload token
  overlap improved from 1/3 to 2/3 prompts vs the live model
  (saved adapter is now correct in bf16-equivalent space).
- Merged-reload for the same cell still diverges from live on
  this overfit synthetic test (loss 3.4 -> 0.0002). Root cause
  is independent: _write_tensor_direct_torch writes in place
  into fixed-size fp8 slots, so the bf16 merged result has to be
  re-quantized to fp8 with ~3.5% per-block precision loss
  (e4m3 mantissa). Proper fix is to emit bf16 shards for fp8
  sources under save_method="merged_16bit", which is a larger
  refactor and out of scope for this commit.

No regression on non-FP8 cells (bnb4bit, bf16) since the helper
is a no-op for those dtypes.

….18 + 0.19)

`_detect_moe_lora_layout` falls back to a default when both the standard and
swapped shape checks pass (which happens whenever the fused gate_up_proj has
`2*I == H`, i.e. the intermediate is exactly half the hidden dim — the common
case for symmetric MoE tinies like Qwen3-MoE, Qwen3.5-MoE, GLM-4-MoE,
DeepSeek-V3 used in our test suite).

Pre-fix that default was hard-coded to "standard", which is correct for
PEFT 0.19+ (it swaps in/out for 3D non-transposed MoE in
`ParamWrapper.update_layer` so the stored
`lora_A.shape = (r*E, in_features)` matches the "standard" branch) but wrong
for PEFT 0.18 (no swap → `lora_A.shape = (r*E, out_features)` matches the
"swapped" branch). Under PEFT 0.18 the merge math therefore picked the wrong
slicing convention and produced garbled merged checkpoints on every symmetric
4bit MoE cell tested.

Fix: at module import time, inspect `peft.tuners.lora.layer.ParamWrapper.update_layer`
for the `_did_swap_in_out_features` substring (introduced in PEFT 0.19). If
present, default the ambiguous case to "standard"; otherwise default to
"swapped". A caller can still override per-call by passing a `lora_module`
whose `_did_swap_in_out_features` attribute is set explicitly.

Empirical verification:

  Cell                         PEFT 0.19            PEFT 0.18 pre-fix   PEFT 0.18 post-fix
  qwen3_moe / 4bit @ 200       [1, 2, 3]            (all garbled)       [0, 0, 2]
  glm4_moe / 4bit  @ 200       [0, 0, 3]            [0, 0, 0]           [1, 1, 1]
  qwen3_5_moe / 4bit @ 200     [3, 2, 2]            [0, 2, 0]           [3, 2, 2]

(numbers are token-set overlap between LIVE first-5 tokens and MERGED-reload
first-5 tokens, per prompt; PEFT 0.18 post-fix now matches PEFT 0.19 baseline).

For non-symmetric shapes (`2*I != H`, e.g. all real-world MoE checkpoints
like Qwen3-Coder-30B-FP8 with I=768, H=2048) the shape check is unambiguous
in either branch and the default is never used, so this fix is a no-op for
those cases.

… 4.58

The patch unconditionally dereferenced `_trainer_utils_mod.validate_quantization_for_training`
to read its sentinel, but the attribute only exists in HF 4.58+. On HF 4.57.6
(one of the CI matrix cells) this raised AttributeError at unsloth import time,
breaking the Core (HF=4.57.6 + TRL<1) job.

Use `getattr(..., None)` and return early if the attribute isn't present —
older HF versions don't have the FP8 guard to relax in the first place.