2026-03-11T12:10:10.6425362Z 0-task-1-0/0 [default0]:
2026-03-11T12:10:10.6426233Z 0-task-1-0/0 [default0]:=================================== FAILURES ===================================
2026-03-11T12:10:10.6426852Z 0-task-1-0/0 [default0]:________ TestVisionTECudaGraphHelper.test_create_and_delete_cudagraphs _________
2026-03-11T12:10:10.6430488Z 0-task-1-0/0 [default0]:
2026-03-11T12:10:10.6431673Z 0-task-1-0/0 [default0]:self = <tests.unit_tests.transformer.test_vision_cuda_graphs.TestVisionTECudaGraphHelper object at 0x7d0f2758a0f0>
2026-03-11T12:10:10.6432841Z 0-task-1-0/0 [default0]:
2026-03-11T12:10:10.6433141Z 0-task-1-0/0 [default0]:    @pytest.mark.skipif(
2026-03-11T12:10:10.6433619Z 0-task-1-0/0 [default0]:        not (HAVE_TE_GRAPHS and is_te_min_version("2.7.0")),
2026-03-11T12:10:10.6434259Z 0-task-1-0/0 [default0]:        reason="TE CUDA graph capture requires TransformerEngine >= 2.7.0",
2026-03-11T12:10:10.6434751Z 0-task-1-0/0 [default0]:    )
2026-03-11T12:10:10.6435644Z 0-task-1-0/0 [default0]:    def test_create_and_delete_cudagraphs(self):
2026-03-11T12:10:10.6436332Z 0-task-1-0/0 [default0]:        """Full lifecycle: create graphs, verify state, delete, verify cleanup."""
2026-03-11T12:10:10.6436907Z 0-task-1-0/0 [default0]:        self.llava_model.cuda()
2026-03-11T12:10:10.6437369Z 0-task-1-0/0 [default0]:        helper = self._make_helper(num_microbatches=1)
2026-03-11T12:10:10.6437792Z 0-task-1-0/0 [default0]:    
2026-03-11T12:10:10.6438128Z 0-task-1-0/0 [default0]:        assert not helper.graphs_created()
2026-03-11T12:10:10.6438498Z 0-task-1-0/0 [default0]:    
2026-03-11T12:10:10.6438804Z 0-task-1-0/0 [default0]:>       helper.create_cudagraphs()
2026-03-11T12:10:10.6439131Z 0-task-1-0/0 [default0]:
2026-03-11T12:10:10.6439506Z 0-task-1-0/0 [default0]:tests/unit_tests/transformer/test_vision_cuda_graphs.py:359: 
2026-03-11T12:10:10.6439993Z 0-task-1-0/0 [default0]:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2026-03-11T12:10:10.6440519Z 0-task-1-0/0 [default0]:megatron/core/transformer/cuda_graphs.py:2751: in create_cudagraphs
2026-03-11T12:10:10.6440954Z 0-task-1-0/0 [default0]:    super().create_cudagraphs()
2026-03-11T12:10:10.6441531Z 0-task-1-0/0 [default0]:megatron/core/transformer/cuda_graphs.py:2253: in create_cudagraphs
2026-03-11T12:10:10.6442145Z 0-task-1-0/0 [default0]:    sample_args, kwargs = self._get_cuda_graph_input_data()
2026-03-11T12:10:10.6442666Z 0-task-1-0/0 [default0]:megatron/core/transformer/cuda_graphs.py:2197: in _get_cuda_graph_input_data
2026-03-11T12:10:10.6443157Z 0-task-1-0/0 [default0]:    kwargs = get_make_graphed_callables_kwargs()
2026-03-11T12:10:10.6443672Z 0-task-1-0/0 [default0]:megatron/core/transformer/cuda_graphs.py:2143: in get_make_graphed_callables_kwargs
2026-03-11T12:10:10.6444253Z 0-task-1-0/0 [default0]:    (10 - self.config.cuda_graph_warmup_steps * get_num_microbatches())
2026-03-11T12:10:10.6444732Z 0-task-1-0/0 [default0]:_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
2026-03-11T12:10:10.6445066Z 0-task-1-0/0 [default0]:
2026-03-11T12:10:10.6445352Z 0-task-1-0/0 [default0]:    def get_num_microbatches() -> int:
2026-03-11T12:10:10.6445874Z 0-task-1-0/0 [default0]:        """Get number of microbatches."""
2026-03-11T12:10:10.6446279Z 0-task-1-0/0 [default0]:>       return _GLOBAL_NUM_MICROBATCHES_CALCULATOR.get()
2026-03-11T12:10:10.6446730Z 0-task-1-0/0 [default0]:E       AttributeError: 'NoneType' object has no attribute 'get'
2026-03-11T12:10:10.6447221Z 0-task-1-0/0 [default0]:
2026-03-11T12:10:10.6447540Z 0-task-1-0/0 [default0]:megatron/core/num_microbatches_calculator.py:19: AttributeError