Our HUB has a bunch of storages that we lock for read or write where needed. Once the entry points start being called from different threads, we can easily deadlock between two entry points. I think we can solve this by enforcing the order of locks: any entry point has to strictly lock the hub resources in the order their storages are declared (in hub.rs).

Explanation

Supposing we make a node of a graph for every storage, and we mark an arrow from A to B to C if an entry point locks storage A followed by B followed by C. A deadlock occurs if at some point these arrows form a cycle, i.e. entry X locks A -> B -> C, while entry Y locks B -> C -> A. Of course, a cycle can also be formed by more than two entry points.

Now, let's assume the code follows a strict order for locking like suggested above. Imagine we hit a deadlock. Let's take a look at the last storage type being locked (pick a subset of all storages that are locked and take the last according to our order), call it A. Since we are in the deadlock, it's locked by some entry X and some other entries try (but fail) to lock it as well. However, there is nothing preventing X from proceeding, since whatever else it needs has to be locked after A, and those are not locked at this point. So X can proceed and free A eventually, so this is not a deadlock.

Plan

1. Audit the code and re-order the locks.
2. Make sure that non-HUB resources are also locked in the order (whatever it is).
3. Document the rules thoroughly in the code.
4. Implement run-time debug checks for the order, asserting accordingly.