Skip to content

[Issue] Windows Pytorch nightly (2.12) test failures with Segmentation fault while running TestNN::test_CTCLoss_critical_target_len - gfx110x, gfx1151 #3987

@chiranjeevipattigidi

Description

@chiranjeevipattigidi

Platform: Windows
Effected Archs: gf110x-all, gfx1151
Pytorch Version: nightly
Python Versions: 3.10, 3.11, 3.12, 3.13

Run:

Windows fatal exception: access violation

Thread 0x00001fd8 (most recent call first):
  <no Python frame>

Current thread 0x0000248c (most recent call first):
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\nn\functional.py", line 3111 in ctc_loss
  File "B:\runner\_work\TheRock\TheRock\external-builds\pytorch\pytorch\test\test_nn.py", line 2845 in test_CTCLoss_critical_target_len
  File "B:\runner\_work\_tool\Python\3.12.10\x64\Lib\unittest\case.py", line 589 in _callTestMethod
  File "B:\runner\_work\_tool\Python\3.12.10\x64\Lib\unittest\case.py", line 634 in run
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\testing\_internal\common_utils.py", line 3514 in _run_custom
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\testing\_internal\common_utils.py", line 3545 in run
  File "B:\runner\_work\_tool\Python\3.12.10\x64\Lib\unittest\case.py", line 690 in __call__
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\unittest.py", line 351 in runtest
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\runner.py", line 174 in pytest_runtest_call
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\runner.py", line 242 in <lambda>
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\runner.py", line 341 in from_call
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\runner.py", line 241 in call_and_report
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\runner.py", line 132 in runtestprotocol
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\runner.py", line 113 in pytest_runtest_protocol
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\main.py", line 362 in pytest_runtestloop
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\main.py", line 337 in _main
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\main.py", line 283 in wrap_session
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\main.py", line 330 in pytest_cmdline_main
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\pluggy\_hooks.py", line 512 in __call__
  File "B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\_pytest\config\__init__.py", line 175 in main
  File "B:\runner\_work\TheRock\TheRock\external-builds\pytorch\run_pytorch_tests.py", line 272 in main
  File "B:\runner\_work\TheRock\TheRock\external-builds\pytorch\run_pytorch_tests.py", line 281 in <module>
Exception Code: 0xC0000005
0x00007FFFB0420000, B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\lib\torch_hip.dll(0x00007FFFB0420000) + 0x0 byte(s)
0x00007FFFB2A70819, B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\lib\torch_hip.dll(0x00007FFFB0420000) + 0x2650819 byte(s), ?_use_miopen_ctc_loss@cuda@at@@YA_NAEBVTensor@2@000_J@Z() + 0x89 byte(s)
0x00007FFFB2BF53B3, B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\lib\torch_hip.dll(0x00007FFFB0420000) + 0x27D53B3 byte(s), ?_fused_adagrad_@cuda@at@@YAXV?$ArrayRef@VTensor@at@@@c10@@000AEBVTensor@2@NNN_NAEBV?$optional@VTensor@at@@@std@@3@Z() + 0x62393 byte(s)
0x00007FFFB9C4CADF, B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\lib\torch_cpu.dll(0x00007FFFB85B0000) + 0x169CADF byte(s), ?redispatch@_use_miopen_ctc_loss_Tensor@_ops@at@@SA_NVDispatchKeySet@c10@@AEBVTensor@3@111_J@Z() + 0xAF byte(s)
0x00007FFFBC5EE293, B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\lib\torch_cpu.dll(0x00007FFFB85B0000) + 0x403E293 byte(s), ??0JitDecompRegisterer@impl@autograd@torch@@QEAA@PEAUJitDecompInterface@123@@Z() + 0x64AC93 byte(s)
0x00007FFFB9C4C734, B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\lib\torch_cpu.dll(0x00007FFFB85B0000) + 0x169C734 byte(s), ?call@_use_miopen_ctc_loss_Tensor@_ops@at@@SA_NAEBVTensor@3@000_J@Z() + 0x114 byte(s)
0x00007FFFB8CD8882, B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\lib\torch_cpu.dll(0x00007FFFB85B0000) + 0x728882 byte(s), ?ctc_loss@native@at@@YA?AVTensor@2@AEBV32@000_J1_N@Z() + 0x122 byte(s)
0x00007FFFBA231423, B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\lib\torch_cpu.dll(0x00007FFFB85B0000) + 0x1C81423 byte(s), ?ctc_loss@compositeimplicitautograd@at@@YA?AVTensor@2@AEBV32@0V?$ArrayRef@_J@c10@@1_J2_N@Z() + 0x443 byte(s)
0x00007FFFB9C2B68D, B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\lib\torch_cpu.dll(0x00007FFFB85B0000) + 0x167B68D byte(s), ?call@ctc_loss_Tensor@_ops@at@@SA?AVTensor@3@AEBV43@000_J1_N@Z() + 0x15D byte(s)
0x00007FFF7EC8007E, B:\runner\_work\TheRock\TheRock\.venv\Lib\site-packages\torch\lib\torch_python.dll(0x00007FFF7E940000) + 0x34007E byte(s), ?release@?$THPPointer@U_object@@@@QEAAPEAU_object@@XZ() + 0x1CE5AE byte(s)
0x00007FF80E3E79E4, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x379E4 byte(s), PyThread_acquire_lock_timed() + 0x5F0 byte(s)
0x00007FF80E432018, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x82018 byte(s), PyObject_Vectorcall() + 0x688 byte(s)
0x00007FF80E4319C5, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x819C5 byte(s), PyObject_Vectorcall() + 0x35 byte(s)
_PyObject_Call_Prepend() + 0x7F byte(s)
0x00007FF80E5030C6, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x1530C6 byte(s), PyDictProxy_New() + 0x2D6 byte(s)
0x00007FF80E4627A6, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0xB27A6 byte(s), PyObject_Call() + 0xB6 byte(s)
0x00007FF80E4366EE, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x866EE byte(s), _PyEval_EvalFrameDefault() + 0x40DE byte(s)
0x00007FF80E43103C, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x8103C byte(s), _PyFunction_Vectorcall() + 0x17C byte(s)
0x00007FF80E463F03, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0xB3F03 byte(s), _PyObject_FastCallDictTstate() + 0xEB byte(s)
0x00007FF80E50319B, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x15319B byte(s), _PyObject_Call_Prepend() + 0x7F byte(s)
0x00007FF80E5030C6, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x1530C6 byte(s), PyDictProxy_New() + 0x2D6 byte(s)
0x00007FF80E432018, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x82018 byte(s), PyObject_Vectorcall() + 0x688 byte(s)
0x00007FF80E4319C5, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x819C5 byte(s), PyObject_Vectorcall() + 0x35 byte(s)
0x00007FF80E432EA5, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x82EA5 byte(s), _PyEval_EvalFrameDefault() + 0x895 byte(s)
0x00007FF80E43103C, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x8103C byte(s), _PyFunction_Vectorcall() + 0x17C byte(s)
0x00007FF80E463F03, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0xB3F03 byte(s), _PyObject_FastCallDictTstate() + 0xEB byte(s)
0x00007FF80E50319B, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x15319B byte(s), _PyObject_Call_Prepend() + 0x7F byte(s)
0x00007FF80E5030C6, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x1530C6 byte(s), PyDictProxy_New() + 0x2D6 byte(s)
0x00007FF80E432018, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x82018 byte(s), PyObject_Vectorcall() + 0x688 byte(s)_PyObject_ClearManagedDict() + 0x384 byte(s)
0x00007FF80E3E6B66, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x36B66 byte(s), PyEval_EvalCode() + 0xB6 byte(s)
0x00007FF80E49E838, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0xEE838 byte(s), _Py_SourceAsString() + 0x90C byte(s)
0x00007FF80E49EA70, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0xEEA70 byte(s), _PyModule_ClearDict() + 0x1FC byte(s)
0x00007FF80E47A5D4, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0xCA5D4 byte(s), PyDict_DelItemString() + 0xFC byte(s)
0x00007FF80E47A432, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0xCA432 byte(s), _PyRun_SimpleFileObject() + 0x126 byte(s)
0x00007FF80E47CD19, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0xCCD19 byte(s), _PyRun_AnyFileObject() + 0x4D byte(s)
0x00007FF80E47CEEA, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0xCCEEA byte(s), _PyEval_MakePendingCalls() + 0x11A byte(s)
0x00007FF80E47CBEB, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0xCCBEB byte(s), PyErr_SetExcFromWindowsErrWithFilenameObjects() + 0x263 byte(s)
0x00007FF80E3D8F1B, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x28F1B byte(s), PyMem_GetAllocator() + 0x5FF byte(s)
0x00007FF80E3D8700, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x28700 byte(s), Py_RunMain() + 0x18 byte(s)
0x00007FF80E3D86E3, B:\runner\_work\_tool\Python\3.12.10\x64\python312.dll(0x00007FF80E3B0000) + 0x286E3 byte(s), Py_Main() + 0x2B byte(s)
0x00007FF6F0E11230, B:\runner\_work\_tool\Python\3.12.10\x64\python.exe(0x00007FF6F0E10000) + 0x1230 byte(s)
0x00007FF81A2EE8D7, C:\Windows\System32\KERNEL32.DLL(0x00007FF81A2C0000) + 0x2E8D7 byte(s), BaseThreadInitThunk() + 0x17 byte(s)
0x00007FF81A92C48C, C:\Windows\SYSTEM32\ntdll.dll(0x00007FF81A8A0000) + 0x8C48C byte(s), RtlUserThreadStart() + 0x2C byte(s)
B:\runner\_work\_temp\3cc90b51-24cd-484a-bec2-bb88b15e122b.sh: line 4:   178 Segmentation fault      python ./external-builds/pytorch/run_pytorch_tests.py -- --continue-on-collection-errors --import-mode=importlib -v
external-builds\pytorch\pytorch\test\test_nn.py::TestNN::test_CTCLoss_critical_target_len 
Error: Process completed with exit code 139.


Metadata

Metadata

Assignees

Type

No type

Projects

Status

In Progress

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions