-
Notifications
You must be signed in to change notification settings - Fork 775
Unable to download large snapshots on 32bit arm flatforms #2047
Description
Home Assistant release with the issue:
Home Assistant 0.114.4
Operating environment (HassOS/Generic):
HassOS 4.13
Supervisor logs:
20-09-12 17:28:38 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:28:46 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:28:58 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 73945d7b
20-09-12 17:29:09 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:29:13 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:30:00 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:30:04 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:30:12 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:30:32 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:31:00 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81Description of problem:
I'm unable to download larger snapshots from the supervisor on 32-bit arm platforms. A 1 GB snapshots downloads fine, but a 2GB snapshot fails.
Minimal Steps to Reproduce:
- Download and install the latest installation image for the pi2 https://github.com/home-assistant/operating-system/releases/download/4.13/hassos_rpi2-4.13.img.gz
- Install the Samba addon and add a 2GB file to the /config folder. I generated such a file with:
dd if=/dev/urandom of=1GB.bin bs=64M count=32 iflag=fullblock
- Create a snapshot from
Supervisor>Snapshots>CREATE - Attempt to download the new snapshot, and chrome reports "Failed - Network Error"
. Nothing shows up in chrome's network tab.
I noticed this problem because users reported seeing it with my addon, the same issue occurs when using the /snapshot/download addon api method (I'm guessing this and the browser download use the same path internally).
I managed to reproduce this on a 32-bit raspberry pi 2 (its the hardware I have on hand) but I've had users report it happen on a Pi3B and pi4 as well when using the 32 bit image. I've been trying to narrow this problem down for a while, for a long time I thought it was a problem in my addon until I noticed it also happens when trying to download a snapshot through the supervisor's web ui. Here is everything I've learned about it:
- I'm unable to get it to happen using the 64 bit pi4 image, which is why I suspect this only affects the 32 bit images.
- I haven't had any reports of users running into this outside of the 32 bit images.
- Smaller snapshots (1GB and less) download fine, larger ones (2GB) fail.
- The supervisor logs that a snapshot download was attempted, but does not log any errors:
20-09-12 17:30:04 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81 - It doesn't seem to matter if create the snapshot in the supervisor or upload it into /backup from another supervisor, the bigger snapshot fails to download either way.
- For these larger snapshots, the /api/snapshots method returns HTTP 200 and whats looks like the appropriate headers, but then the connection is closed immedietaly before sending any snapshot bytes. Here is the pattern of request start/end one of my users saw (this is based on reporting from aiohttp's Client Tracing):
08-28 09:46:42 TRACE [backup.tracing_session] Request started: GET http://hassio/snapshots/<redacted>/download 08-28 09:46:42 TRACE [backup.tracing_session] sent headers: 08-28 09:46:42 TRACE [backup.tracing_session] Authorization: <redacted> 08-28 09:46:42 TRACE [backup.tracing_session] Request ending: GET http://hassio/snapshots/<redacted>/download 08-28 09:46:42 TRACE [backup.tracing_session] status: 200 08-28 09:46:42 TRACE [backup.tracing_session] received headers: 08-28 09:46:42 TRACE [backup.tracing_session] Content-Type: application/tar 08-28 09:46:42 TRACE [backup.tracing_session] Last-Modified: Sat, 22 Aug 2020 23:34:44 GMT 08-28 09:46:42 TRACE [backup.tracing_session] Content-Length: 2676981760 08-28 09:46:42 TRACE [backup.tracing_session] Accept-Ranges: bytes 08-28 09:46:42 TRACE [backup.tracing_session] Date: Fri, 28 Aug 2020 07:46:42 GMT 08-28 09:46:42 TRACE [backup.tracing_session] Server: Python/3.8 aiohttp/3.6.2