Skip to content

"ValueError: Value out of range" when uploading large files (~>= 4.5GB) with st.file_uploader #5938

@snehankekre

Description

@snehankekre

Checklist

  • I have searched the existing issues for similar issues.
  • I added a very descriptive title to this issue.
  • I have provided sufficient information below to help reproduce this issue.

Summary

Uploading a large file with st.file_uploader can reliably lead to a:

ValueError: Value out of range <num-uploaded-bytes>

File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 565, in _run_script
    exec(code, module.__dict__)
File "/Users/skekre/Downloads/streamlit-misc/file-uploader-size.py", line 3, in <module>
    f = st.file_uploader("Upload a file")
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/runtime/metrics_util.py", line 311, in wrapped_func
    result = non_optional_func(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/elements/file_uploader.py", line 338, in file_uploader
    return self._file_uploader(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/elements/file_uploader.py", line 418, in _file_uploader
    file_uploader_state = serde.serialize(widget_state.value)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/streamlit/elements/file_uploader.py", line 108, in serialize
    file_info.size = f.size

where "large" in my case was approximately close to or greater than 4.5GB. I'm not sure whether the specific size after which this error is thrown is machine dependant. @serdar-bayramov who first reported this on the Streamlit forum encountered the error for files >= 2GB.

Reproducible Code Example

import streamlit as st

f = st.file_uploader("Upload a file")

if f is not None:
    st.write(f.name)
    st.write(f.size)

Steps To Reproduce

  1. Create a .streamlit/config.toml file to set the upload limit to 4.5 GB:

    [server]
    maxUploadSize = 4500
  2. Create a 4.4 GB file:

    truncate -s 4280m example.file
  3. Run the above app and upload example.file to the file uploader widget

Expected Behavior

As the uploaded 4.4 GB file is under the 4.5 GB limit set in .streamlit/config.toml, one would expect:

  1. to see the upload successfully complete
  2. the file uploader widget to be visible after the upload
  3. see the output of the filename and filesize displayed

Current Behavior

  1. Once the progress bar on the file uploader widget hits the maximum, the app greys out for a few seconds while the Running man icon is displayed in the top right corner
  2. Immediately after that, the file uploader widget disappears, leaving the user with the following error:

image

Is this a regression?

  • Yes, this used to work in a previous version.

Debug info

  • Streamlit version: 1.16.0
  • Python version: 3.9
  • Operating System: macOS
  • Browser: Chrome
  • Virtual environment: NA

Additional Information

First reported by @serdar-bayramov in https://discuss.streamlit.io/t/how-to-upload-large-files-using-st-file-uploader/35654

Are you willing to submit a PR?

  • Yes, I am willing to submit a PR!

Metadata

Metadata

Assignees

Labels

feature:st.file_uploaderRelated to the `st.file_uploader` widgetpriority:P2Medium-high priority - fix within one monthstatus:confirmedBug has been confirmed by the Streamlit teamtype:bugSomething isn't working as expected

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions