Summary
Qwen2.5-Coder-0.5B-Instruct qualification failed with 13/24 scenarios falsified. All failures are in format conversion tests (F-CONV-*) with two distinct root causes:
- Conversion fidelity: All format conversions produce outputs differing by 58-90% (tolerance: 1e-6)
- Path resolution bug: Round-trip tests fail with
Invalid model format: No file extension found
Severity: P0 CRITICAL - Blocks model qualification
Environment
- apr-cli: 0.2.12
- Model: Qwen/Qwen2.5-Coder-0.5B-Instruct
- Playbook:
qwen2.5-coder-0.5b-quick.playbook.yaml
- MQS Score: 320/1000 (Grade F, BLOCKED)
Observed Failures
Direct Conversion Gates (P0)
| Gate ID |
Conversion |
CPU Diff |
GPU Diff |
Expected |
| F-CONV-G-A |
GGUF → APR |
7.61e-1 |
7.50e-1 |
< 1e-6 |
| F-CONV-A-G |
APR → GGUF |
7.42e-1 |
7.54e-1 |
< 1e-6 |
| F-CONV-G-S |
GGUF → SafeTensors |
5.86e-1 |
5.86e-1 |
< 1e-6 |
| F-CONV-S-G |
SafeTensors → GGUF |
9.05e-1 |
9.05e-1 |
< 1e-6 |
| F-CONV-A-S |
APR → SafeTensors |
7.55e-1 |
7.38e-1 |
< 1e-6 |
| F-CONV-S-A |
SafeTensors → APR |
8.18e-1 |
8.18e-1 |
< 1e-6 |
Round-Trip/Multi-Hop Gates (P0)
All fail with identical error:
Invalid model format: No file extension found
| Gate ID |
Test |
| F-CONV-RT-001 |
Round-trip GGUF |
| F-CONV-RT-002 |
Multi-hop SafeTensors→Apr→Gguf→SafeTensors |
| F-CONV-RT-003 |
Multi-hop SafeTensors→Apr→Gguf→Apr→SafeTensors |
| F-CONV-IDEM-001 |
Idempotency GGUF→APR twice |
| F-CONV-COM-001 |
Commutativity GGUF→APR vs GGUF→ST→APR |
| F-GOLDEN-RULE-001 |
Golden Rule convert→inference→diff |
Popperian Falsification Checklist
F-LAYOUT-002: Row-Major Layout Violation
| Item |
Status |
| Hypothesis |
All tensors are in row-major layout after conversion |
| Falsified if |
Output contains garbage or diff > ε |
| Result |
⚠️ 58-90% diff suggests LAYOUT-002 violation |
Path Resolution Bug
| Item |
Status |
| Hypothesis |
ConversionTest::execute() receives file path, not directory |
| Falsified if |
Invalid model format: No file extension found |
| Result |
❌ FALSIFIED - directory passed instead of file |
Root Cause Hypotheses (Priority Order)
- H1 (HIGH): Path resolution bug in
apr-qa-runner - ConversionTest passes directory not file
- H2 (MEDIUM): LAYOUT-002 violation - Transpose not applied during GGUF→APR conversion
- H3 (LOW): Quantization mismatch - Q4K block layout differs between formats
Investigation Commands
# 1. Verify layout flag in APR header
apr inspect ~/.cache/apr-models/qwen2-5-coder-0-5b-instruct/apr/model.apr | grep layout
# 2. Check if converter is transposing Q4K tensors
grep -n "transpose_q4k" src/format/converter/*.rs
# 3. Verify GGUF→SafeTensors baseline (no APR involved)
apr rosetta ~/.cache/apr-models/qwen2-5-coder-0-5b-instruct/gguf/model.gguf \
-o /tmp/test.safetensors --verify
Acceptance Criteria
Cross-References
Generated by apr-qa qualification runner with --fail-fast
Summary
Qwen2.5-Coder-0.5B-Instruct qualification failed with 13/24 scenarios falsified. All failures are in format conversion tests (F-CONV-*) with two distinct root causes:
Invalid model format: No file extension foundSeverity: P0 CRITICAL - Blocks model qualification
Environment
qwen2.5-coder-0.5b-quick.playbook.yamlObserved Failures
Direct Conversion Gates (P0)
Round-Trip/Multi-Hop Gates (P0)
All fail with identical error:
Popperian Falsification Checklist
F-LAYOUT-002: Row-Major Layout Violation
Path Resolution Bug
ConversionTest::execute()receives file path, not directoryInvalid model format: No file extension foundRoot Cause Hypotheses (Priority Order)
apr-qa-runner-ConversionTestpasses directory not fileInvestigation Commands
Acceptance Criteria
Cross-References
apr-model-qa-playbook/docs/specifications/apr-playbook-spec.md§ 4.1.1 (LAYOUT-002)apr-model-qa-playbook/certifications/qwen2-5-coder-0-5b-instruct/evidence.jsonGenerated by apr-qa qualification runner with --fail-fast