Skip to content

Bug introduced in distributed 2022.5.1 -- unpack_frames #6448

@evamaxfield

Description

@evamaxfield

What happened:

Seems like potentially two bugs related to each other or one bug that is causing two different flakey behaviors.

First bug is that the compute call is returning an array asking for PiB of memory: https://github.com/AllenCellModeling/aicsimageio/runs/6594900687?check_suite_focus=true#step:8:137

Second bug is that the unpack_frames is requiring a buffer of a certain length but isn't receiving it: https://github.com/AllenCellModeling/aicsimageio/runs/6594900687?check_suite_focus=true#step:8:709

Ultimately both bugs use the unpack_frames function in the traceback though.

image

What you expected to happen:

This exact same code ran fine on 2022.5.0: https://github.com/AllenCellModeling/aicsimageio/runs/6563614927?check_suite_focus=true

image

(I also checked the underlying file reading library version tifffile and it stays the same 2022.5.4 for both the PR CI and the post merge CI.)

Minimal Complete Verifiable Example:

Unlike the last bug I caught via CI (#6255) this one is seems deep in the weeds of worker magic but I do think I have a cuplrit from the changelog.

  • Improve ensure_memoryview test coverage & make minor fixes #6333 -- the actual code for host_array in utils was changed in 2022.5.0 but not in 2022.5.1 however there was a change made in that PR ensures a memory view for each frame to unpack. Memory view -> incorrect allocation size / buffer size seems like a likely culprit?

Anything else we need to know?:

Environment:

  • Dask version: 2022.5.1
  • Python version: CI tests on 3.8, 3.9, and 3.10, all are failing
  • Operating System: CI tests on Windows, MacOS, and Ubuntu -- MacOS tests fail fast, the others tests are taking 3 hours so I am assuming something is going wrong with them too.
  • Install method (conda, pip, source): pip
Cluster Dump State:

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions