Separate provenance tracking to different levels by yushangdi · Pull Request #160383 · pytorch/pytorch

yushangdi · 2025-08-12T00:15:38Z

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In this PR, we prepare to turn on part of the provenance tracking that doesn't have too much overhead by default.

Change provenance_tracking config to provenance_tracking_level
turn on the following provenance tracking by default when basic_provenance_tracking=True
- set_kernel_post_grad_provenance_tracing for kernels, this add mapping between triton kernels and post_grad nodes
- dump_inductor_provenance_info if we're dumping tlparse log
- get_graph_provenance_json and dump reate_mapping_pre_post_grad_nodes. This creates mapping between pre_grad and post_grad nodes. Since we're not turning on the provenance tracking in GraphTransformObserver by default, the mapping here maybe incomplete/limited.
- add stack trace from post grad nodes to inductor IR nodes
- add exception swallowing for all functions above

Test Plan:
CI

Rollback Plan:

Differential Revision: D80031559

cc @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @wenzhe-nrv @jiayisunx @ipiszy @chenyang78 @kadeng @muchulee8 @amjames @chauhang @aakhundov @coconutruben @Lucaskabela

pytorch-bot · 2025-08-12T00:15:42Z

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/160383

📄 Preview Python docs built from this PR
📄 Preview C++ docs built from this PR
❓ Need help or want to give feedback on the CI? Visit the bot commands wiki or our office hours

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (1 Unrelated Failure)

As of commit adc0047 with merge base 211c988 ():

BROKEN TRUNK - The following job failed but were present on the merge base:

👉 Rebase onto the `viable/strict` branch to avoid these failures

inductor / cuda12.8-py3.10-gcc9-sm86 / test (inductor_torchbench, 2, 2, linux.g5.4xlarge.nvidia.gpu) (gh) (trunk failure)
vision_maskrcnn

This comment was automatically generated by Dr. CI and updates every 15 minutes.

facebook-github-bot · 2025-08-12T00:15:50Z

This pull request was exported from Phabricator. Differential Revision: D80031559

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In thi PR, we turn on part of the provenance tracking that doesn't have too much overhead by default. Test Plan: CI Rollback Plan: Differential Revision: D80031559

facebook-github-bot · 2025-08-12T00:23:54Z

This pull request was exported from Phabricator. Differential Revision: D80031559

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In thi PR, we turn on part of the provenance tracking that doesn't have too much overhead by default. Test Plan: CI Rollback Plan: Differential Revision: D80031559

kflu · 2025-08-12T00:56:17Z

Thanks @yushangdi ! Is there potential latency regression by turning it on by default? can we study them? If so, I can also help study some production models latency implication.

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In thi PR, we turn on part of the provenance tracking that doesn't have too much overhead by default. Test Plan: CI Rollback Plan: Differential Revision: D80031559

yushangdi · 2025-08-12T17:59:58Z

Thanks @yushangdi ! Is there potential latency regression by turning it on by default? can we study them? If so, I can also help study some production models latency implication.

@kflu I don't expect any big latency regression, but it may increase a little.

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In thi PR, we turn on part of the provenance tracking that doesn't have too much overhead by default. Test Plan: CI Rollback Plan: Differential Revision: D80031559

facebook-github-bot · 2025-08-12T18:38:04Z

@yushangdi has imported this pull request. If you are a Meta employee, you can view this in D80031559.

kflu · 2025-08-13T07:59:32Z

torch/_inductor/config.py

+    # Save mapping info from inductor generated kernel to post_grad fx nodes to pre_grad fx nodes
+    # Will be changed to default to True
+    # TODO: remove this flag once it's running stable
+    basic_provenance_tracking = os.environ.get("INDUCTOR_PROVENANCE_BASIC", "0") == "1"


Instead of having a "basic", would it be more generic to define it as the "level" of provenance tracking? That way, we can re-use config provenance_tracking and the env vars INDUCTOR_PROVENANCE which is already an integer.

0: disabled
1: normal prvenance
2: basic

kflu · 2025-08-13T08:02:10Z

torch/_inductor/ir.py

-        if config.trace.provenance_tracking:
+
+        if config.trace.basic_provenance_tracking or config.trace.provenance_tracking:
            for node in origins:


This code block also need to be exception handled I think

I moved this code to be called lazily, so it's not in IR node initialization anymore. Currently it's only called when we want to print an IR node.

kflu · 2025-08-13T08:03:28Z

torch/_inductor/compile_fx.py

-            )
+    # Dump provenance artifacts for debugging trace
+    if config.trace.basic_provenance_tracking or config.trace.provenance_tracking:
+        trace_structured(


Is trace_structured exception safe?

we use trace_structured everywhere for tlparse already, it's safe as long as the payload_fn function passed to it is safe.

kflu · 2025-08-13T08:04:28Z

torch/_inductor/compile_fx.py

+                    config.trace.basic_provenance_tracking
+                    or config.trace.provenance_tracking
+                ):
                    provenance_tracking_json = (


This code block needs to be exception free

This block should be exception free already. I added exception check in get_graph_provenance_json and create_mapping_pre_post_grad_nodes

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In this PR, we prepare to turn on part of the provenance tracking that doesn't have too much overhead by default. - Change `provenance_tracking` config to `provenance_tracking_level`. This is defaults to 1 (normal) now, but will be defaults to 2 to turn on basic provenance tracking - turn on the following provenance tracking by default when `basic_provenance_tracking`=True - `set_kernel_post_grad_provenance_tracing` for kernels, this add mapping between triton kernels and post_grad nodes - `dump_inductor_provenance_info` if we're dumping tlparse log - `get_graph_provenance_json` and dump `reate_mapping_pre_post_grad_nodes`. This creates mapping between pre_grad and post_grad nodes. Since we're not turning on the provenance tracking in GraphTransformObserver by default, the mapping here maybe incomplete/limited. - add exception swallowing for all functions above Test Plan: CI Rollback Plan: cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov coconutruben Lucaskabela Differential Revision: D80031559 Pulled By: yushangdi

facebook-github-bot · 2025-08-13T17:27:33Z

This pull request was exported from Phabricator. Differential Revision: D80031559

yushangdi · 2025-08-13T20:28:03Z

@kflu Would it be possible to also verify whether we can turn on "normal" mode by default in the lowering stack?

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In this PR, we prepare to turn on part of the provenance tracking that doesn't have too much overhead by default. - add `basic_provenance_tracking` config. This is defaults to False now, but will be defaults to True to turn on basic provenance tracking - turn on the following provenance tracking by default when `basic_provenance_tracking`=True - `set_kernel_post_grad_provenance_tracing` for kernels, this add mapping between triton kernels and post_grad nodes - `dump_inductor_provenance_info` if we're dumping tlparse log - `get_graph_provenance_json` and dump `reate_mapping_pre_post_grad_nodes`. This creates mapping between pre_grad and post_grad nodes. Since we're not turning on the provenance tracking in GraphTransformObserver by default, the mapping here maybe incomplete/limited. - add stack trace from post grad nodes to inductor IR nodes - add exception swallowing for all functions above Pull Request resolved: pytorch#160383 Test Plan: CI Rollback Plan: cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov coconutruben Lucaskabela Differential Revision: D80031559 Pulled By: yushangdi

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In this PR, we prepare to turn on part of the provenance tracking that doesn't have too much overhead by default. - add `basic_provenance_tracking` config. This is defaults to False now, but will be defaults to True to turn on basic provenance tracking - turn on the following provenance tracking by default when `basic_provenance_tracking`=True - `set_kernel_post_grad_provenance_tracing` for kernels, this add mapping between triton kernels and post_grad nodes - `dump_inductor_provenance_info` if we're dumping tlparse log - `get_graph_provenance_json` and dump `reate_mapping_pre_post_grad_nodes`. This creates mapping between pre_grad and post_grad nodes. Since we're not turning on the provenance tracking in GraphTransformObserver by default, the mapping here maybe incomplete/limited. - add stack trace from post grad nodes to inductor IR nodes - add exception swallowing for all functions above Pull Request resolved: pytorch#160383 Test Plan: CI Rollback Plan: cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov coconutruben Lucaskabela Rollback Plan: Differential Revision: D80139160 Pulled By: yushangdi

kflu · 2025-08-14T18:42:54Z

@kflu Would it be possible to also verify whether we can turn on "normal" mode by default in the lowering stack?

sure, once it's landed we can test both modes.

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In this PR, we prepare to turn on part of the provenance tracking that doesn't have too much overhead by default. - add `basic_provenance_tracking` config. This is defaults to False now, but will be defaults to True to turn on basic provenance tracking - turn on the following provenance tracking by default when `basic_provenance_tracking`=True - `set_kernel_post_grad_provenance_tracing` for kernels, this add mapping between triton kernels and post_grad nodes - `dump_inductor_provenance_info` if we're dumping tlparse log - `get_graph_provenance_json` and dump `reate_mapping_pre_post_grad_nodes`. This creates mapping between pre_grad and post_grad nodes. Since we're not turning on the provenance tracking in GraphTransformObserver by default, the mapping here maybe incomplete/limited. - add stack trace from post grad nodes to inductor IR nodes - add exception swallowing for all functions above Pull Request resolved: pytorch#160383 Test Plan: CI Rollback Plan: cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov coconutruben Lucaskabela Differential Revision: D80031559 Pulled By: yushangdi

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In this PR, we prepare to turn on part of the provenance tracking that doesn't have too much overhead by default. - add `basic_provenance_tracking` config. This is defaults to False now, but will be defaults to True to turn on basic provenance tracking - turn on the following provenance tracking by default when `basic_provenance_tracking`=True - `set_kernel_post_grad_provenance_tracing` for kernels, this add mapping between triton kernels and post_grad nodes - `dump_inductor_provenance_info` if we're dumping tlparse log - `get_graph_provenance_json` and dump `reate_mapping_pre_post_grad_nodes`. This creates mapping between pre_grad and post_grad nodes. Since we're not turning on the provenance tracking in GraphTransformObserver by default, the mapping here maybe incomplete/limited. - add stack trace from post grad nodes to inductor IR nodes - add exception swallowing for all functions above Pull Request resolved: pytorch#160383 Test Plan: CI Rollback Plan: cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov coconutruben Lucaskabela Rollback Plan: Differential Revision: D80139160 Pulled By: yushangdi

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In this PR, we prepare to turn on part of the provenance tracking that doesn't have too much overhead by default. - add `basic_provenance_tracking` config. This is defaults to False now, but will be defaults to True to turn on basic provenance tracking - turn on the following provenance tracking by default when `basic_provenance_tracking`=True - `set_kernel_post_grad_provenance_tracing` for kernels, this add mapping between triton kernels and post_grad nodes - `dump_inductor_provenance_info` if we're dumping tlparse log - `get_graph_provenance_json` and dump `reate_mapping_pre_post_grad_nodes`. This creates mapping between pre_grad and post_grad nodes. Since we're not turning on the provenance tracking in GraphTransformObserver by default, the mapping here maybe incomplete/limited. - add stack trace from post grad nodes to inductor IR nodes - add exception swallowing for all functions above Pull Request resolved: pytorch#160383 Test Plan: CI Rollback Plan: cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov coconutruben Lucaskabela Differential Revision: D80031559 Pulled By: yushangdi

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In this PR, we prepare to turn on part of the provenance tracking that doesn't have too much overhead by default. - add `basic_provenance_tracking` config. This is defaults to False now, but will be defaults to True to turn on basic provenance tracking - turn on the following provenance tracking by default when `basic_provenance_tracking`=True - `set_kernel_post_grad_provenance_tracing` for kernels, this add mapping between triton kernels and post_grad nodes - `dump_inductor_provenance_info` if we're dumping tlparse log - `get_graph_provenance_json` and dump `reate_mapping_pre_post_grad_nodes`. This creates mapping between pre_grad and post_grad nodes. Since we're not turning on the provenance tracking in GraphTransformObserver by default, the mapping here maybe incomplete/limited. - add stack trace from post grad nodes to inductor IR nodes - add exception swallowing for all functions above Pull Request resolved: pytorch#160383 Test Plan: CI Rollback Plan: cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov coconutruben Lucaskabela Rollback Plan: Differential Revision: D80139160 Pulled By: yushangdi

angelayi · 2025-08-15T00:44:49Z

torch/_inductor/config.py

+    # Backward compatibility:
+    #   If TORCH_COMPILE_DEBUG=1, level is set to at least 1.
+    #   If INDUCTOR_PROVENANCE is set, use its integer value.
+    provenance_tracking_level: int = int(


I think you can use something like, Literal[0, 1, 2] like this

I'll add this change to my next PR!

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In this PR, we prepare to turn on part of the provenance tracking that doesn't have too much overhead by default. - Change `provenance_tracking` config to `provenance_tracking_level`. This is defaults to 1 (normal) now, but will be defaults to 2 to turn on basic provenance tracking - turn on the following provenance tracking by default when `basic_provenance_tracking`=True - `set_kernel_post_grad_provenance_tracing` for kernels, this add mapping between triton kernels and post_grad nodes - `dump_inductor_provenance_info` if we're dumping tlparse log - `get_graph_provenance_json` and dump `reate_mapping_pre_post_grad_nodes`. This creates mapping between pre_grad and post_grad nodes. Since we're not turning on the provenance tracking in GraphTransformObserver by default, the mapping here maybe incomplete/limited. - add exception swallowing for all functions above Pull Request resolved: pytorch#160383 Test Plan: CI Rollback Plan: cc voznesenskym penguinwu EikanWang jgong5 Guobing-Chen XiaobingSuper zhuhaozhe blzheng wenzhe-nrv jiayisunx ipiszy chenyang78 kadeng muchulee8 amjames chauhang aakhundov coconutruben Lucaskabela Reviewed By: angelayi Differential Revision: D80031559 Pulled By: yushangdi

facebook-github-bot · 2025-08-15T01:01:35Z

This pull request was exported from Phabricator. Differential Revision: D80031559

facebook-github-bot · 2025-08-15T01:48:40Z

@pytorchbot merge -i

(Initiating merge automatically since Phabricator Diff has merged, merging with -i because oss signals were bypassed internally)

pytorchmergebot · 2025-08-15T01:50:25Z

Merge started

Your change will be merged while ignoring the following 1 checks: pull / linux-jammy-py3.9-clang12 / build

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging

Check the merge workflow status
here

pytorchmergebot · 2025-08-15T05:00:39Z

This PR (#160383) was merged in aa99e09 but it is still open, likely due to a Github bug, so mergebot is closing it manually. If you think this is a mistake, please feel free to reopen and contact Dev Infra.

Summary: as title. We've got request from various parties who are interested in turning on the provenance tracking by default. In this PR, we prepare to turn on part of the provenance tracking that doesn't have too much overhead by default. - Change `provenance_tracking` config to `provenance_tracking_level` - turn on the following provenance tracking by default when `basic_provenance_tracking`=True - `set_kernel_post_grad_provenance_tracing` for kernels, this add mapping between triton kernels and post_grad nodes - `dump_inductor_provenance_info` if we're dumping tlparse log - `get_graph_provenance_json` and dump `reate_mapping_pre_post_grad_nodes`. This creates mapping between pre_grad and post_grad nodes. Since we're not turning on the provenance tracking in GraphTransformObserver by default, the mapping here maybe incomplete/limited. - add stack trace from post grad nodes to inductor IR nodes - add exception swallowing for all functions above Test Plan: CI Rollback Plan: Differential Revision: D80031559 Pull Request resolved: pytorch#160383 Approved by: https://github.com/angelayi

pytorch-bot bot added ciflow/inductor module: inductor labels Aug 12, 2025

facebook-github-bot added the fb-exported label Aug 12, 2025

yushangdi force-pushed the export-D80031559 branch from 21e5e13 to ea96543 Compare August 12, 2025 00:23

yushangdi force-pushed the export-D80031559 branch from ea96543 to 499c47a Compare August 12, 2025 00:36

pytorch-bot bot added the module: dynamo label Aug 12, 2025

yushangdi marked this pull request as draft August 12, 2025 00:46

yushangdi force-pushed the export-D80031559 branch from 499c47a to 7afa5be Compare August 12, 2025 16:58

yushangdi force-pushed the export-D80031559 branch from 7afa5be to 55c4a3f Compare August 12, 2025 17:56

yushangdi force-pushed the export-D80031559 branch from 55c4a3f to c0d0305 Compare August 12, 2025 18:12

pytorch-bot bot added the release notes: fx release notes category label Aug 12, 2025

yushangdi marked this pull request as ready for review August 12, 2025 18:37

pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Aug 12, 2025

kflu reviewed Aug 13, 2025

View reviewed changes

yushangdi force-pushed the export-D80031559 branch from c0d0305 to 4d38dd1 Compare August 13, 2025 17:27

yushangdi requested a review from kflu August 13, 2025 20:28

yushangdi changed the title ~~Turn on part of provenance tracking by default~~ Separate provenance tracking to different levels Aug 13, 2025

yushangdi requested a review from angelayi August 14, 2025 16:53

angelayi approved these changes Aug 15, 2025

View reviewed changes

yushangdi force-pushed the export-D80031559 branch from 4d38dd1 to adc0047 Compare August 15, 2025 01:01

pytorchmergebot added the merging label Aug 15, 2025

pytorchmergebot added the Merged label Aug 15, 2025

pytorchmergebot closed this Aug 15, 2025

pytorchmergebot removed the merging label Aug 15, 2025

Conversation

yushangdi commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pytorch-bot bot commented Aug 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/160383

✅ You can merge normally! (1 Unrelated Failure)

Uh oh!

facebook-github-bot commented Aug 12, 2025

Uh oh!

facebook-github-bot commented Aug 12, 2025

Uh oh!

kflu commented Aug 12, 2025

Uh oh!

yushangdi commented Aug 12, 2025

Uh oh!

facebook-github-bot commented Aug 12, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

yushangdi Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Aug 13, 2025

Uh oh!

yushangdi commented Aug 13, 2025

Uh oh!

kflu commented Aug 14, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

facebook-github-bot commented Aug 15, 2025

Uh oh!

facebook-github-bot commented Aug 15, 2025

Uh oh!

pytorchmergebot commented Aug 15, 2025

Merge started

Uh oh!

pytorchmergebot commented Aug 15, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

yushangdi commented Aug 12, 2025 •

edited

Loading

pytorch-bot bot commented Aug 12, 2025 •

edited

Loading

yushangdi Aug 13, 2025 •

edited

Loading