At Thu Nov 16 17:15 server db1063, master of the s5 shard of databases (dewiki and wikidata) started to have storage crashes (T180714#3767308)
Update Fri Nov 17 07:55:06 UTC 2017:
Pending things:
- Revert schema change on codfw [DONE on master - replication will bring it to the slaves]
- move dbstore1001 under the new master (db1070) [DONE]
- revert schema change on dbstore1001 [DONE]
- move dbstore1002 under the new master (db1070) [DONE]
- revert schema change on dbstore1002 [ONGOING]
- decide what to do with db1063 (definitely not a master for anything else, we might rebuild it as vslow from db1071 for example) as it is broken: https://phabricator.wikimedia.org/P6337
- db1071 moved under db1070
IR document: https://wikitech.wikimedia.org/wiki/Incident_documentation/20171116-s5-dewiki-wikidata