There are two race conditions in here. I could only produce one of them in a test

If a key is transitioned to "flight", a gather_dep task for this key is scheduled. This race condition requires the task to be scheduled in batch with another set of tasks. If one task of this batch is released before the gather_dep coroutine is first executed but after the transition in ensure_communicate, the gather_dep coroutine will filter the released task and will only try to fetch results of the remote for the tasks still in state "flight".
Once the response is available, all keys are tried to be parsed in the response, not only the actually fetched ones. I marked this situation with a FIXME a while ago but didn't anticipate the impact nor did I have an actual test backing this up.

Two things can now happen:

1. The key is actually released and stays released. In this case we'll take the "already-executing" code path which is wrong. We'll get a log for a released task claiming it was executed. No big deal. We'll survive a bad log message.
2. However, If during the data collection, i.e. during get_data_from_worker the forgotten key is rescheduled, this results in us taking the "missing-data" path since the data for the key is not in the response, the task is not in state memory, the remote was not busy and the task has still dependents since it was rescheduled. In this case the worker issues a missing data signal to the scheduler which deletes the possibly only replica of that task on the remote which in turn might escalate pretty badly.

I tried very hard in reproducing 2.) using multiple mocks and locks 🤯 but it is extremely difficult to produce due to the rather monolithic design of that particular section of code. I settled for for a test which covers only 1.) since 2.) is a special case of 1.) I intend to eliminate such race conditions by implementing a cancelled state as described in #4413 (comment) but this is way out of scope for this PR. The test will still be useful, though.