Handle HIP OOM exceptions#7764
Conversation
|
This also means I can use Perhaps WanImageToVideo should have a tiled option somehow too. |
398805d to
a19cb1a
Compare
|
I've update to pytorch 2.7.1+rocm6.3 and this still seems valuable for my gfx1100. It makes sense that all variants of OOM should be handled. Is there some issue with this approach, or other reason not to merge this? |
|
This is still merge conflict free, want it or shall I close? cc @comfyanonymous @Kosinkadink |
|
Could i get a comment on this, perhaps a short description on why it isn't suitable? Perhaps i could rework it adapt it in some way that still allows oom fallbacks to work on rocm. @comfyanonymous @Kosinkadink |
|
Why don't you set the model_management.OOM_EXCEPTION to whatever the hip oom exception is? |
The error this handles is a generic RuntimeError with just the message identifying it as an oom. Matching all RuntimeErrors could include non-oom ones.
|
Test Evidence CheckIf this PR modifies behavior that requires testing, a test explanation is required. PRs lacking applicable test explanations may not be reviewed until added. Please add test explanations to ensure code quality and prevent regressions. If this PR changes user-facing behavior, visual proof (screen recording or screenshot) is required. PRs without applicable visual documentation may not be reviewed until provided. You can add it by:
|
Enhance exception handling to treat RuntimeException "HIP error: out of memory" the same as
OOM_EXCEPTION.Resolves #7761