Skip to content

config torch to avoid graph breaks caused by logger#6999

Merged
tjruwase merged 1 commit intodeepspeedai:masterfrom
ShellyNR:disable_logger_for_PT2.6
Feb 24, 2025
Merged

config torch to avoid graph breaks caused by logger#6999
tjruwase merged 1 commit intodeepspeedai:masterfrom
ShellyNR:disable_logger_for_PT2.6

Conversation

@ShellyNR
Copy link
Copy Markdown
Contributor

@ShellyNR ShellyNR commented Feb 4, 2025

Following changes in Pytorch trace rules , my previous PR to avoid graph breaks caused by logger is no longer relevant. So instead I've added this functionality to torch dynamo - pytorch/pytorch@16ea0dd
This commit allows the user to config torch to ignore logger methods and avoid associated graph breaks.

To enable ignore logger methods - os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
To ignore logger methods except for a specific method / methods (for example, info and isEnabledFor) - os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
and os.environ["LOGGER_METHODS_TO_EXCLUDE_FROM_DISABLE"] = "info, isEnabledFor"

@ShellyNR ShellyNR requested a review from tjruwase as a code owner February 4, 2025 17:43
@tohtana
Copy link
Copy Markdown
Collaborator

tohtana commented Feb 4, 2025

Thank you @ShellyNR, this is a very clean approach.
I would like to approve this, but DCO prevents tests from running because of the migration to deepspeedai. Can you rebase based on the explanation?

@loadams loadams requested a review from tohtana as a code owner February 10, 2025 22:51
@loadams
Copy link
Copy Markdown
Collaborator

loadams commented Feb 13, 2025

Thank you @ShellyNR, this is a very clean approach. I would like to approve this, but DCO prevents tests from running because of the migration to deepspeedai. Can you rebase based on the explanation?

I believe I've fixed this for now.

@SNahir @ShellyNR
disable_logger_for_pt2.6
fa2fb8a 
Signed-off-by: ShellyNR <shelly.nahir@live.biu.ac.il>
@SNahir SNahir force-pushed the disable_logger_for_PT2.6 branch from 4124fa1 to fa2fb8a Compare February 17, 2025 16:41
@ShellyNR
Copy link
Copy Markdown
Contributor Author

ShellyNR commented Feb 19, 2025

Hi @tohtana sorry for the delay, can you please review this? Thanks

@tjruwase tjruwase added this pull request to the merge queue Feb 24, 2025
Merged via the queue into deepspeedai:master with commit e1903f0 Feb 24, 2025
deepcharm pushed a commit to deepcharm/DeepSpeed that referenced this pull request Feb 26, 2025
@ShellyNR @SNahir @deepcharm
config torch to avoid graph breaks caused by logger (deepspeedai#6999)
ec879b3 
Following changes in Pytorch trace rules , my previous PR to avoid graph
breaks caused by logger is no longer relevant. So instead I've added
this functionality to torch dynamo -
pytorch/pytorch@16ea0dd
This commit allows the user to config torch to ignore logger methods and
avoid associated graph breaks.

To enable ignore logger methods -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
To ignore logger methods except for a specific method / methods (for
example, info and isEnabledFor) -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
and os.environ["LOGGER_METHODS_TO_EXCLUDE_FROM_DISABLE"] = "info,
isEnabledFor"

Signed-off-by: ShellyNR <shelly.nahir@live.biu.ac.il>
Co-authored-by: snahir <snahir@habana.ai>
Signed-off-by: Max Kovalenko <mkovalenko@habana.ai>
deepcharm pushed a commit to deepcharm/DeepSpeed that referenced this pull request Feb 27, 2025
@ShellyNR @SNahir @deepcharm
config torch to avoid graph breaks caused by logger (deepspeedai#6999)
495606a 
Following changes in Pytorch trace rules , my previous PR to avoid graph
breaks caused by logger is no longer relevant. So instead I've added
this functionality to torch dynamo -
pytorch/pytorch@16ea0dd
This commit allows the user to config torch to ignore logger methods and
avoid associated graph breaks.

To enable ignore logger methods -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
To ignore logger methods except for a specific method / methods (for
example, info and isEnabledFor) -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
and os.environ["LOGGER_METHODS_TO_EXCLUDE_FROM_DISABLE"] = "info,
isEnabledFor"

Signed-off-by: ShellyNR <shelly.nahir@live.biu.ac.il>
Co-authored-by: snahir <snahir@habana.ai>
Signed-off-by: Max Kovalenko <mkovalenko@habana.ai>
gyou2021 pushed a commit to gyou2021/DeepSpeed that referenced this pull request Feb 28, 2025
@ShellyNR @SNahir @gyou2021
config torch to avoid graph breaks caused by logger (deepspeedai#6999)
4a86d02 
Following changes in Pytorch trace rules , my previous PR to avoid graph
breaks caused by logger is no longer relevant. So instead I've added
this functionality to torch dynamo -
pytorch/pytorch@16ea0dd
This commit allows the user to config torch to ignore logger methods and
avoid associated graph breaks.

To enable ignore logger methods -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
To ignore logger methods except for a specific method / methods (for
example, info and isEnabledFor) -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
and os.environ["LOGGER_METHODS_TO_EXCLUDE_FROM_DISABLE"] = "info,
isEnabledFor"

Signed-off-by: ShellyNR <shelly.nahir@live.biu.ac.il>
Co-authored-by: snahir <snahir@habana.ai>
Signed-off-by: gyou2021 <ganmei.you@intel.com>
tohtana pushed a commit that referenced this pull request Feb 28, 2025
@ShellyNR @SNahir
config torch to avoid graph breaks caused by logger (#6999)
59fe7f6 
Following changes in Pytorch trace rules , my previous PR to avoid graph
breaks caused by logger is no longer relevant. So instead I've added
this functionality to torch dynamo -
pytorch/pytorch@16ea0dd
This commit allows the user to config torch to ignore logger methods and
avoid associated graph breaks.

To enable ignore logger methods -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
To ignore logger methods except for a specific method / methods (for
example, info and isEnabledFor) -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
and os.environ["LOGGER_METHODS_TO_EXCLUDE_FROM_DISABLE"] = "info,
isEnabledFor"

Signed-off-by: ShellyNR <shelly.nahir@live.biu.ac.il>
Co-authored-by: snahir <snahir@habana.ai>
Signed-off-by: Masahiro Tanaka <mtanaka@microsoft.com>
shenzheyu pushed a commit to shenzheyu/DeepSpeed that referenced this pull request Mar 5, 2025
@ShellyNR @SNahir @shenzheyu
config torch to avoid graph breaks caused by logger (deepspeedai#6999)
f0db010 
Following changes in Pytorch trace rules , my previous PR to avoid graph
breaks caused by logger is no longer relevant. So instead I've added
this functionality to torch dynamo -
pytorch/pytorch@16ea0dd
This commit allows the user to config torch to ignore logger methods and
avoid associated graph breaks.

To enable ignore logger methods -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
To ignore logger methods except for a specific method / methods (for
example, info and isEnabledFor) -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
and os.environ["LOGGER_METHODS_TO_EXCLUDE_FROM_DISABLE"] = "info,
isEnabledFor"

Signed-off-by: ShellyNR <shelly.nahir@live.biu.ac.il>
Co-authored-by: snahir <snahir@habana.ai>
Signed-off-by: Zheyu SHEN <zyshen@umd.edu>
ys950902 pushed a commit to ys950902/DeepSpeed that referenced this pull request Mar 6, 2025
@ShellyNR @SNahir @ys950902
config torch to avoid graph breaks caused by logger (deepspeedai#6999)
d1c37e0 
Following changes in Pytorch trace rules , my previous PR to avoid graph
breaks caused by logger is no longer relevant. So instead I've added
this functionality to torch dynamo -
pytorch/pytorch@16ea0dd
This commit allows the user to config torch to ignore logger methods and
avoid associated graph breaks.

To enable ignore logger methods -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
To ignore logger methods except for a specific method / methods (for
example, info and isEnabledFor) -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
and os.environ["LOGGER_METHODS_TO_EXCLUDE_FROM_DISABLE"] = "info,
isEnabledFor"

Signed-off-by: ShellyNR <shelly.nahir@live.biu.ac.il>
Co-authored-by: snahir <snahir@habana.ai>
Signed-off-by: yisheng <yi.sheng@intel.com>
saurabhkoshatwar pushed a commit to saurabhkoshatwar/DeepSpeed that referenced this pull request Mar 8, 2025
@ShellyNR @SNahir @saurabhkoshatwar
config torch to avoid graph breaks caused by logger (deepspeedai#6999)
acc6b59 
Following changes in Pytorch trace rules , my previous PR to avoid graph
breaks caused by logger is no longer relevant. So instead I've added
this functionality to torch dynamo -
pytorch/pytorch@16ea0dd
This commit allows the user to config torch to ignore logger methods and
avoid associated graph breaks.

To enable ignore logger methods -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
To ignore logger methods except for a specific method / methods (for
example, info and isEnabledFor) -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
and os.environ["LOGGER_METHODS_TO_EXCLUDE_FROM_DISABLE"] = "info,
isEnabledFor"

Signed-off-by: ShellyNR <shelly.nahir@live.biu.ac.il>
Co-authored-by: snahir <snahir@habana.ai>
Signed-off-by: Saurabh <saurabhkoshatwar1996@gmail.com>
mauryaavinash95 pushed a commit to DataStates/DeepSpeed that referenced this pull request Mar 20, 2025
@ShellyNR @SNahir
config torch to avoid graph breaks caused by logger (deepspeedai#6999)
31f8da6 
Following changes in Pytorch trace rules , my previous PR to avoid graph
breaks caused by logger is no longer relevant. So instead I've added
this functionality to torch dynamo -
pytorch/pytorch@16ea0dd
This commit allows the user to config torch to ignore logger methods and
avoid associated graph breaks.

To enable ignore logger methods -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
To ignore logger methods except for a specific method / methods (for
example, info and isEnabledFor) -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
and os.environ["LOGGER_METHODS_TO_EXCLUDE_FROM_DISABLE"] = "info,
isEnabledFor"

Signed-off-by: ShellyNR <shelly.nahir@live.biu.ac.il>
Co-authored-by: snahir <snahir@habana.ai>
loadams pushed a commit that referenced this pull request Mar 25, 2025
@ShellyNR @SNahir @loadams
config torch to avoid graph breaks caused by logger (#6999)
fd5273f 
Following changes in Pytorch trace rules , my previous PR to avoid graph
breaks caused by logger is no longer relevant. So instead I've added
this functionality to torch dynamo -
pytorch/pytorch@16ea0dd
This commit allows the user to config torch to ignore logger methods and
avoid associated graph breaks.

To enable ignore logger methods -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
To ignore logger methods except for a specific method / methods (for
example, info and isEnabledFor) -
os.environ["DISABLE_LOGS_WHILE_COMPILING"] = "1"
and os.environ["LOGGER_METHODS_TO_EXCLUDE_FROM_DISABLE"] = "info,
isEnabledFor"

Signed-off-by: ShellyNR <shelly.nahir@live.biu.ac.il>
Co-authored-by: snahir <snahir@habana.ai>
Signed-off-by: Logan Adams <loadams@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tjruwase tjruwase tjruwase approved these changes

@tohtana tohtana Awaiting requested review from tohtana tohtana is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@ShellyNR @tohtana @loadams @tjruwase @SNahir