During a routine upgrade of our GKE (Google Kubernetes Engine) cluster which powers the dashboard, Google’s Metadata server experienced a large sagging under load which caused encrypt and decrypt requests to become severely delayed. As a result, we experienced a rush of users from 21:45 UTC to 23:24 UTC, during which the degraded metadata service became significantly delayed under load. After several of short-term attempts to achieve resolution, such as disabling logging to reduce the pressure on the affected metadata service, Railway engineering determined that rolling out a newer version of GKE unaffected by the known issue of the previous version of GKE would bring resolution.