Update to allow large models to be checked for mobile support. #18357
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add ability to set flag to keep initializers as external data - Handle issue with ONNX shape inferencing silently failing
Contributor
|
@skottmckay do you think this PR would also fix #14697? |
Contributor
Author
It will get closer in that it would get to the point where it attempted to create the flatbuffer for the ORT format model, however the flatbuffer offsets are unsigned 32-bit int so there is at most 4GB of data that could be written out. Better than the 2GB protobuf limit though. Not clear what the scenario is where you'd need to use an ORT format model. That implies a minimal build to save a few MB to load a model that is multiple GB. Due to that we haven't prioritized supporting these models in ORT format. |
…geModelsInUsabilityChecker
…geModelsInUsabilityChecker
edgchen1
reviewed
Nov 15, 2023
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
…geModelsInUsabilityChecker
edgchen1
approved these changes
Nov 15, 2023
kleiti
pushed a commit
to kleiti/onnxruntime
that referenced
this pull request
Mar 22, 2024
…soft#18357) ### Description <!-- Describe your changes. --> Update usability checker and related infrastructure to support checking models > 2GB. - Add ability to set flag to keep initializers as external data - we optimize the model as part of the checking so need to write out a new copy. - Handle issue with ONNX shape inferencing silently failing - use API that supports large models but requires writing the model to a new file - automate cleanup of that copy of the model ### Motivation and Context <!-- - Why is this change required? What problem does it solve? - If it fixes an open issue, please link to the issue here. --> Allow analysis of LLMs to determine gaps for mobile usage. --------- Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Update usability checker and related infrastructure to support checking models > 2GB.
Motivation and Context
Allow analysis of LLMs to determine gaps for mobile usage.