Skip to content

Fix FSDP offload pin_memory bug#157147

Closed
Edenzzzz wants to merge 2 commits intopytorch:mainfrom
Edenzzzz:fix_pin_mem
Closed

Fix FSDP offload pin_memory bug#157147
Edenzzzz wants to merge 2 commits intopytorch:mainfrom
Edenzzzz:fix_pin_mem

Conversation

@Edenzzzz
Copy link
Contributor

@Edenzzzz Edenzzzz commented Jun 27, 2025

@pytorch-bot
Copy link

pytorch-bot bot commented Jun 27, 2025

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/157147

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (2 Unrelated Failures)

As of commit e60eb70 with merge base 8a88c6e (image):

FLAKY - The following job failed but was likely due to flakiness present on trunk:

UNSTABLE - The following job is marked as unstable, possibly due to flakiness on trunk:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@pytorch-bot pytorch-bot bot added oncall: distributed Add this issue/PR to distributed oncall triage queue release notes: distributed (fsdp) release notes category labels Jun 27, 2025
@Edenzzzz
Copy link
Contributor Author

cc @kwen2501

@weifengpy weifengpy self-requested a review June 27, 2025 21:47
@weifengpy
Copy link
Contributor

weifengpy commented Jun 27, 2025

good catch!

there are 2 usage of pin_memory(device=) in fully_shard. Is it possible to remove another as well?

padded_sharded_param = padded_sharded_param.pin_memory(
device=self.device
)

note to myself:
at aten level, there is a major refactor to deprecate the device argument in pin_memory(device=) #126376

now it throws warnings if using device argument https://github.com/pytorch/pytorch/blame/20e40492b046b9287726d3ec656117e4dc38f0e2/aten/src/ATen/native/Memory.cpp#L45-L49

Copy link
Contributor

@weifengpy weifengpy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commented

@weifengpy
Copy link
Contributor

cc @jeejakp12 as we remove device from pin_memory

@weifengpy
Copy link
Contributor

there is a chance of breaking 3rd party device. but according to #126376, even 3rd party device should avoid using pin_memory(device) as there are other ways in that PR

@weifengpy
Copy link
Contributor

approved. but probably need to wait for CI (linter etc)

@weifengpy
Copy link
Contributor

@pytorchmergebot merge

@pytorch-bot pytorch-bot bot added the ciflow/trunk Trigger trunk jobs on your pull request label Jun 28, 2025
@pytorchmergebot
Copy link
Collaborator

Merge started

Your change will be merged once all checks pass (ETA 0-4 Hours).

Learn more about merging in the wiki.

Questions? Feedback? Please reach out to the PyTorch DevX Team

Advanced Debugging
Check the merge workflow status
here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ciflow/trunk Trigger trunk jobs on your pull request Merged oncall: distributed Add this issue/PR to distributed oncall triage queue open source release notes: distributed (fsdp) release notes category

Projects

None yet

Development

Successfully merging this pull request may close these issues.

FSDP cpu offload hits CUDA error: invalid argument when pin_memory is True

4 participants