Skip to content

Unable to download large snapshots on 32bit arm flatforms #2047

@sabeechen

Description

@sabeechen

Home Assistant release with the issue:

Home Assistant 0.114.4

Operating environment (HassOS/Generic):

HassOS 4.13

Supervisor logs:

20-09-12 17:28:38 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:28:46 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:28:58 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 73945d7b
20-09-12 17:29:09 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:29:13 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:30:00 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:30:04 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:30:12 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:30:32 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
20-09-12 17:31:00 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81

Description of problem:
I'm unable to download larger snapshots from the supervisor on 32-bit arm platforms. A 1 GB snapshots downloads fine, but a 2GB snapshot fails.

Minimal Steps to Reproduce:

  • Download and install the latest installation image for the pi2 https://github.com/home-assistant/operating-system/releases/download/4.13/hassos_rpi2-4.13.img.gz
  • Install the Samba addon and add a 2GB file to the /config folder. I generated such a file with:
     dd if=/dev/urandom of=1GB.bin bs=64M count=32 iflag=fullblock
  • Create a snapshot from Supervisor > Snapshots > CREATE
  • Attempt to download the new snapshot, and chrome reports "Failed - Network Error"
    image. Nothing shows up in chrome's network tab.

I noticed this problem because users reported seeing it with my addon, the same issue occurs when using the /snapshot/download addon api method (I'm guessing this and the browser download use the same path internally).
I managed to reproduce this on a 32-bit raspberry pi 2 (its the hardware I have on hand) but I've had users report it happen on a Pi3B and pi4 as well when using the 32 bit image. I've been trying to narrow this problem down for a while, for a long time I thought it was a problem in my addon until I noticed it also happens when trying to download a snapshot through the supervisor's web ui. Here is everything I've learned about it:

  • I'm unable to get it to happen using the 64 bit pi4 image, which is why I suspect this only affects the 32 bit images.
  • I haven't had any reports of users running into this outside of the 32 bit images.
  • Smaller snapshots (1GB and less) download fine, larger ones (2GB) fail.
  • The supervisor logs that a snapshot download was attempted, but does not log any errors:
     20-09-12 17:30:04 INFO (MainThread) [supervisor.api.snapshots] Download snapshot 8d0f4a81
    
  • It doesn't seem to matter if create the snapshot in the supervisor or upload it into /backup from another supervisor, the bigger snapshot fails to download either way.
  • For these larger snapshots, the /api/snapshots method returns HTTP 200 and whats looks like the appropriate headers, but then the connection is closed immedietaly before sending any snapshot bytes. Here is the pattern of request start/end one of my users saw (this is based on reporting from aiohttp's Client Tracing):
    08-28 09:46:42 TRACE [backup.tracing_session] Request started: GET http://hassio/snapshots/<redacted>/download
    08-28 09:46:42 TRACE [backup.tracing_session]   sent headers:
    08-28 09:46:42 TRACE [backup.tracing_session]     Authorization: <redacted>
    08-28 09:46:42 TRACE [backup.tracing_session] Request ending: GET http://hassio/snapshots/<redacted>/download
    08-28 09:46:42 TRACE [backup.tracing_session]   status: 200
    08-28 09:46:42 TRACE [backup.tracing_session]   received headers:
    08-28 09:46:42 TRACE [backup.tracing_session]     Content-Type: application/tar
    08-28 09:46:42 TRACE [backup.tracing_session]     Last-Modified: Sat, 22 Aug 2020 23:34:44 GMT
    08-28 09:46:42 TRACE [backup.tracing_session]     Content-Length: 2676981760
    08-28 09:46:42 TRACE [backup.tracing_session]     Accept-Ranges: bytes
    08-28 09:46:42 TRACE [backup.tracing_session]     Date: Fri, 28 Aug 2020 07:46:42 GMT
    08-28 09:46:42 TRACE [backup.tracing_session]     Server: Python/3.8 aiohttp/3.6.2
    

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions