Replication was restarted several times, but on the week of the 20th a decision was made to entirely reload mysql2 and examine some alternative replication methods (primarily row-based replication). Doing backups using Percona XtraBackup, allowing for much faster full restores Using mydumper rather than mysql-zrm to improve the speed of our logical backups Work on our documentation and training for our complex systems, including Regularly testing full restores as part of our backup process on a spare server Gather more accurate ETAs for the restoration process Regularly audit the databases we host – Multiple test and ballooning databases (100GB+) seriously delayed the restore process Migrate to a bigger, more powerful mysql cluster (already planned before this outage) Of resource use &…