Description

When using FqnToConfig (or ModuleFqnToConfig) with a configuration that includes both:

• A regex pattern matching to None (intended to skip quantization for certain modules)
• A _default configuration (intended to quantize other modules)

Modules that match the None pattern are incorrectly still quantized by _default instead of being skipped.

Steps to Reproduce

from collections import OrderedDict
import torch
import torch.nn as nn
from torchao.quantization import (
    quantize_,
    Int8DynamicActivationInt8WeightConfig,
    ModuleFqnToConfig,
)

class TestModel(nn.Module):
    def __init__(self):
        super().__init__()
        self.time_embed = nn.Sequential(
            nn.Linear(1024, 1024),
            nn.ReLU(),
            nn.Linear(1024, 1024),
        )
        self.linear1 = nn.Linear(1024, 1024)

    def forward(self, x):
        x = self.time_embed(x)
        x = self.linear1(x)
        return x

model = TestModel()
model.eval()

# Expected: time_embed should NOT be quantized, linear1 should be quantized
cfg = ModuleFqnToConfig(
    module_fqn_to_config=OrderedDict([
        ("re:.*time_embed.*", None),  # Skip time_embed
        ("_default", Int8DynamicActivationInt8WeightConfig()),  # Quantize other modules
    ])
)

quantize_(model, cfg, filter_fn=None)

error logs:
Traceback (most recent call last):
File "/root/workspace/optkit/examples/ttt.py", line 43, in
quantize_(model, cfg, filter_fn=None)
File "/usr/local/lib/python3.10/dist-packages/torchao/quantization/quant_api.py", line 502, in quantize_
_fqn_to_config_handler(module, module_fqn, config)
File "/usr/local/lib/python3.10/dist-packages/torchao/quantization/quant_api.py", line 2690, in _fqn_to_config_handler
return handler(module, c)
File "/usr/local/lib/python3.10/dist-packages/torchao/quantization/quant_api.py", line 1629, in _int8_dynamic_activation_int8_weight_transform
assert hasattr(module, "weight"), (
AssertionError: applying int8 dynamic activation int8 weight quant requires module to have weight attributebut {module} does not have one