[Dynamo] Imporve-graph-break-skip-logs by parsshar-RH · Pull Request #167067 · pytorch/pytorch

parsshar-RH · 2025-11-05T09:33:49Z

Summary:

Added frame information (function name, file, line number) to all graph break/skip messages
Standardized message format: "torch.compile will skip tracing the frame ( line ) and fall back to eager. Reason: "

Impacts:

module: dynamo

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @chenyang78 @kadeng @chauhang @amjames @Lucaskabela

pytorch-bot · 2025-11-05T09:33:53Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/167067

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit dd201aa with merge base c940b1f ():

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

trunk / linux-jammy-py3-clang12-executorch / test (executorch, 1, 1, lf.linux.2xlarge, unstable) (gh) (#166072)
backends/xnnpack/test/recipes/test_xnnpack_recipes.py::TestXnnpackRecipes::test_int8_static_quant_recipe

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Lucaskabela · 2025-11-05T18:29:31Z

cc @williamwen42 to help take a look at this

williamwen42

Thanks for submitting! I made a few comments.

williamwen42 · 2025-11-05T18:42:57Z

test/dynamo/test_error_messages.py

        )

+    @make_logging_test(graph_breaks=True)
+    def test_skipped_frame_no_verbose_traceback(self, records):


How does this test differ from test_assert_failure_in_generic_ctx_mgr?

williamwen42 · 2025-11-05T18:43:53Z

test/dynamo/test_error_messages.py

+    def test_skipped_frame_no_verbose_traceback(self, records):
+        def fn(x):
+            with GenericCtxMgr():
+                assert x is None


Could we just do a regular graph break here? Since there's some special handling of asserts that causes such a test to possibly fail internally (see comment above test_assert_failure_in_generic_ctx_mgr)

williamwen42 · 2025-11-05T18:44:49Z

test/dynamo/test_error_messages.py

+
+        self.assertEqual(len(records), 1)
+
+        message = records[0].getMessage()


Prefer to use assertExpectedInline[Munged] here in order to more clearly see what the error message looks like (otherwise, can you provide a reason why this would not be a good idea?)

williamwen42 · 2025-11-05T18:49:34Z

torch/_dynamo/convert_frame.py

+                        if config.verbose:
+                            graph_break_log.debug(
+                                user_stack_trace,
+                                exc_info=True,
+                                stack_info=True,
+                            )
+                        else:
+                            graph_break_log.debug(
+                                user_stack_trace,
+                                exc_info=True,
+                                stack_info=False,
+                            )


Suggested change

if config.verbose:

graph_break_log.debug(

user_stack_trace,

exc_info=True,

stack_info=True,

)

else:

graph_break_log.debug(

user_stack_trace,

exc_info=True,

stack_info=False,

)

graph_break_log.debug(

user_stack_trace,

exc_info=True,

stack_info=config.verbose,

)

williamwen42 · 2025-11-05T18:53:41Z

torch/_dynamo/symbolic_convert.py

+                    frame_info = (
+                        f"{getattr(self.f_code, 'co_name', '<unknown>')} "
+                        f"({getattr(self.f_code, 'co_filename', '<unknown>')} "
+                        f"line {getattr(self.f_code, 'co_firstlineno', 0)})"
+                    )
                    msg = (
-                        "Skipping frame because there is a graph break in a for/while loop\n"
+                        f"torch.compile intentionally decided to skip the frame {frame_info} and fall back to eager.\n"
+                        f"Reason: Skipping frame because there is a graph break in a for/while loop.\n"


Can we refactor this code (and the above) into one function?

williamwen42 · 2025-11-05T18:56:24Z

torch/_dynamo/symbolic_convert.py

+                f"{getattr(self.f_code, 'co_name', '<unknown>')} "
+                f"({getattr(self.f_code, 'co_filename', '<unknown>')} "
+                f"line {getattr(self.f_code, 'co_firstlineno', 0)})"


I'm seeing this code appear a lot - can we move it to a single function in exc.py?

williamwen42 · 2025-11-05T18:57:17Z

test/dynamo/test_nested_graph_breaks.py

+
+        result = torch.compile(f2, backend="eager")(torch.randn(3))
+        expected = f2(torch.randn(3))
+        self.assertEqual(result.shape, expected.shape)


Can you check for UX here (i.e. using assertExpectedInline[Munged])? And same for the added cases below.

torch/_logging/_internal.py

parsshar-RH · 2025-11-06T12:03:17Z

Hi @williamwen42

Thank you for the detailed review and valuable suggestions.
I will incorporate the changes and push an updated commit shortly.

Thanks again for your time and feedback.

parsshar-RH · 2025-11-07T14:39:48Z

@williamwen42

I have implemented the suggested changes.
Please feel free to let me know if any additional updates are needed.

Thanks!

torch/_dynamo/symbolic_convert.py

williamwen42 · 2025-11-07T19:39:39Z

torch/_dynamo/utils.py

        functorch_subclass_name = re.sub(r"\(.*", "", repr(val))
        raise SkipFrame(
-            f"torch.compile cannot be run in context: {functorch_subclass_name}"
+            f"torch.compile intentionally decided to skip the frame and fall back to eager.\n"


Use format_skip_frame_message?

williamwen42 · 2025-11-07T19:40:03Z

torch/_dynamo/variables/functions.py

-            raise SkipFrame(
                f"Skip frame due to `torch._dynamo.skip_frame()`. Message: {skip_frame_msg}"
+            frame_info = format_frame_info(tx.f_code)
+            msg = (


Use format_skip_frame_message?

williamwen42 · 2025-11-07T19:42:52Z

torch/_dynamo/exc.py

+def format_skip_frame_message(code: types.CodeType, reason: str) -> str:
+    frame_info = format_frame_info(code)
+    return (
+        f"torch.compile intentionally decided to skip the frame {frame_info} and fall back to eager.\n"


I don't see a test that checks for this string - I think this might not be logged to the graph_breaks logging artifact (I believe it is logged to the torch._dynamo logger?). Can you make sure this is covered by the tests you added?

parsshar-RH · 2025-11-10T13:57:47Z

@williamwen42

I have implemented the suggested changes.
Please aware me if you have more suggestions.

Thanks!

parsshar-RH · 2025-11-10T23:16:00Z

@williamwen42
Looks like some unrelated CI checks are failing.
Please review.

Thanks in advance!

williamwen42

The lint error looks real - can you move the nested graph break tests you added to test_error_messages?

parsshar-RH · 2025-11-11T17:11:25Z

@williamwen42

I have moved the nested graph break tests to test_error_messages.py

parsshar-RH · 2025-11-12T02:52:26Z

@williamwen42

Lint error is resolved.
But this failing CI check looks unrelated.
Please review and suggest.

Thanks

parsshar-RH · 2025-11-13T16:39:49Z

@williamwen42
Please review and merge the PR.

williamwen42 · 2025-11-13T19:20:46Z

@pytorchbot merge

pytorchmergebot · 2025-11-13T19:22:42Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-11-13T21:35:39Z

Merge failed

Reason: 1 jobs have failed, first few of them are: trunk / linux-jammy-cuda12.8-py3.10-gcc11 / test (default, 4, 5, lf.linux.g6.4xlarge.experimental.nvidia.gpu)

Details for Dev Infra team

Raised by workflow job

williamwen42 · 2025-11-13T21:49:43Z

@pytorchbot rebase

pytorchmergebot · 2025-11-13T21:51:18Z

@pytorchbot started a rebase job onto refs/remotes/origin/viable/strict. Check the current status here

pytorchmergebot · 2025-11-13T21:51:22Z

Successfully rebased improve-graph-break-skip-frame onto refs/remotes/origin/viable/strict, please pull locally before adding more changes (for example, via git checkout improve-graph-break-skip-frame && git pull --rebase)

parsshar-RH · 2025-11-13T23:35:16Z

@pytorchbot merge

pytorchmergebot · 2025-11-13T23:38:33Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-11-13T23:54:50Z

Merge failed

Reason: 1 mandatory check(s) failed. The first few are:

Lint / Test tools / linux-job

Dig deeper by viewing the failures on hud

Details for Dev Infra team

Raised by workflow job

Failing merge rule: Core Maintainers

williamwen42 · 2025-11-14T02:58:59Z

@pytorchbot merge

pytorchmergebot · 2025-11-14T03:00:55Z

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

Summary: Fixes #150477 ### Summary: - Added frame information (function name, file, line number) to all graph break/skip messages - Standardized message format: "torch.compile will skip tracing the frame <name> (<file> line <N>) and fall back to eager. Reason: <reason>" ### Impacts: module: dynamo X-link: pytorch/pytorch#167067 Approved by: https://github.com/williamwen42 Reviewed By: jeanschmidt Differential Revision: D87036500 fbshipit-source-id: 62281bad4609b8ea3557f7139695678bed0679cb

Fixes pytorch#150477 ### Summary: - Added frame information (function name, file, line number) to all graph break/skip messages - Standardized message format: "torch.compile will skip tracing the frame <name> (<file> line <N>) and fall back to eager. Reason: <reason>" ### Impacts: module: dynamo Pull Request resolved: pytorch#167067 Approved by: https://github.com/williamwen42

pytorch-bot bot added the module: dynamo label Nov 5, 2025

pytorchbot added the open source label Nov 5, 2025

Lucaskabela requested a review from williamwen42 November 5, 2025 18:29

williamwen42 added the topic: not user facing topic category label Nov 5, 2025

williamwen42 reviewed Nov 5, 2025

View reviewed changes

janeyx99 added the triaged This issue has been looked at a team member, and triaged and prioritized into an appropriate module label Nov 7, 2025

williamwen42 reviewed Nov 7, 2025

View reviewed changes

williamwen42 approved these changes Nov 10, 2025

View reviewed changes

williamwen42 reviewed Nov 11, 2025

View reviewed changes

parsshar-RH mentioned this pull request Nov 13, 2025

[dynamo, nested graph breaks] Fix-nested-graph-break-suppression #167743

Closed

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 13, 2025

pytorchmergebot added the merging label Nov 13, 2025

pytorchmergebot removed the merging label Nov 13, 2025

parsshar-RH added 3 commits November 13, 2025 21:51

imporve-graph-break-skip-logs

30b62f3

fix-test-case-refactor

9833c67

update functions.py

a09fa68

parsshar-RH added 2 commits November 13, 2025 21:51

add test-case

86308b8

move tests

dd201aa

pytorchmergebot force-pushed the improve-graph-break-skip-frame branch from 683d9ec to dd201aa Compare November 13, 2025 21:51

pytorch-bot bot removed the ciflow/trunk Trigger trunk jobs on your pull request label Nov 13, 2025

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Nov 13, 2025

pytorchmergebot added the merging label Nov 13, 2025

pytorchmergebot removed the merging label Nov 13, 2025

pytorchmergebot added the merging label Nov 14, 2025

pytorchmergebot added the Merged label Nov 14, 2025

pytorchmergebot closed this in 5b1e112 Nov 14, 2025

pytorchmergebot removed the merging label Nov 14, 2025


		self.assertEqual(len(records), 1)

		message = records[0].getMessage()

Conversation

parsshar-RH commented Nov 5, 2025 • edited by pytorch-bot bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary:

Impacts:

Uh oh!

pytorch-bot bot commented Nov 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/167067

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

Lucaskabela commented Nov 5, 2025

Uh oh!

williamwen42 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

parsshar-RH commented Nov 6, 2025

Uh oh!

parsshar-RH commented Nov 7, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

parsshar-RH commented Nov 10, 2025

Uh oh!

parsshar-RH commented Nov 10, 2025

Uh oh!

williamwen42 left a comment

Choose a reason for hiding this comment

Uh oh!

parsshar-RH commented Nov 11, 2025

Uh oh!

parsshar-RH commented Nov 12, 2025

Uh oh!

parsshar-RH commented Nov 13, 2025

Uh oh!

williamwen42 commented Nov 13, 2025

Uh oh!

pytorchmergebot commented Nov 13, 2025

Merge started

Uh oh!

pytorchmergebot commented Nov 13, 2025

Merge failed

Uh oh!

williamwen42 commented Nov 13, 2025

Uh oh!

pytorchmergebot commented Nov 13, 2025

Uh oh!

pytorchmergebot commented Nov 13, 2025

Uh oh!

parsshar-RH commented Nov 13, 2025

Uh oh!

pytorchmergebot commented Nov 13, 2025

Merge started

Uh oh!

pytorchmergebot commented Nov 13, 2025

Merge failed

Uh oh!

williamwen42 commented Nov 14, 2025

Uh oh!

parsshar-RH commented Nov 5, 2025 •

edited by pytorch-bot bot

Loading

pytorch-bot bot commented Nov 5, 2025 •

edited

Loading