We had Cassandra slowness which caused the write queues to fill in MT, but we noticed that on many of our instances, one worker never drained. Over the course of a couple minutes this caused all ingest in MT to stop.

Attaching a snapshot of the dashboard. It's a bit hard to tell, so I added an arrow showing where queue 5 just hangs as the other queues all start to drain. Also attached a stack from one of our hung instances.

metrictank.20170911.stack.txt