Summary
NVIDIA Nemotron-Nano-4B-Instruct achieves only 12.3% pass rate (19/155 tests) during MVP certification. This suggests the Nemotron architecture is largely unsupported in the current APR stack.
Evidence
| Model |
Pass Rate |
Tests |
Source |
| Nemotron-Nano-4B-Instruct |
12.3% |
19/155 |
apr-model-qa-playbook/certifications/nemotron-nano-4b/evidence.json |
Context
Nemotron is NVIDIA's custom architecture based on a modified LLaMA backbone with:
- Custom attention patterns
- Different layer normalization
- NVIDIA-specific optimizations
The very low pass rate (similar to Qwen3 failures) suggests fundamental architecture support is missing rather than edge-case bugs.
Reproduction
cd ../apr-model-qa-playbook
cargo run --release --bin apr-qa -- run playbooks/models/nemotron-nano-4b-mvp.playbook.yaml \
--output certifications/nemotron-nano-4b --no-integrity-check --skip-conversion-tests \
--no-differential --no-trace-payload --timeout 180000Expected Behavior
Should achieve >=90% pass rate once Nemotron architecture is fully supported.
Labels
Architecture support, Nemotron, P3
Summary
NVIDIA Nemotron-Nano-4B-Instruct achieves only 12.3% pass rate (19/155 tests) during MVP certification. This suggests the Nemotron architecture is largely unsupported in the current APR stack.
Evidence
apr-model-qa-playbook/certifications/nemotron-nano-4b/evidence.jsonContext
Nemotron is NVIDIA's custom architecture based on a modified LLaMA backbone with:
The very low pass rate (similar to Qwen3 failures) suggests fundamental architecture support is missing rather than edge-case bugs.
Reproduction
cd ../apr-model-qa-playbook cargo run --release --bin apr-qa -- run playbooks/models/nemotron-nano-4b-mvp.playbook.yaml \ --output certifications/nemotron-nano-4b --no-integrity-check --skip-conversion-tests \ --no-differential --no-trace-payload --timeout 180000Expected Behavior
Should achieve >=90% pass rate once Nemotron architecture is fully supported.
Labels
Architecture support, Nemotron, P3