We identified the root cause as an errant migration workflow, which exacerbated IO pressure by calling an IO hot path concurrently with other retry queues. The migration workflow had an issue where it would call a redeploy for every replica, not every deployment This meant if a machine happened to call redeploy on something with say, 50 replicas and 3 replicas on the same box, it would trigger 150 redeploys This caused retries across the queues, which pushed queued status requests to the workers, in all regions, which made other regions unschedulable, filling up THEIR retry queues.