| Boshomi | |
| May 24 2018, 7:32 PM |
| F18514381: Screenshot_20180524-171600-01.jpeg | |
| May 24 2018, 9:59 PM |
| F18514237: 2018-05-24_22-40-12_preview.jpeg | |
| May 24 2018, 9:09 PM |
| F18512793: 33378763_2097967926898471_8812372984074338304_n.jpg | |
| May 24 2018, 7:51 PM |
| F18512789: capture-20180524-214313.png | |
| May 24 2018, 7:51 PM |
just now i got this:
(Auf die Datenbank konnte nicht zugegriffen werden: Cannot access the database: No working replica DB server: Unknown error (10.64.32.198:3318))
Error also:
(Cannot access the database: Cannot access the database: No working replica DB server: Unknown error (10.64.32.113))
Change 435044 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.4] Do not register SpecialItemDisambiguation to stop DoS.
A hot fix has been applied to keep the sites up. Wikidata functions may be returning incorrect data in the meantime.
Also property suggester is disabled and article placeholder won't show much and also Special:ItemDisambiguation
There is a separate task about the broken text: T195525: MWExceptionRenderer.php doesn't always declare the encoding used. It already has a patch pending, so in case another issue like this occurs, at least the error message will be displayed correctly ;)
Change 435055 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.5] Log the query that would hit wb_terms
Change 435057 had a related patch set uploaded (by Daniel Kinzler; owner: Daniel Kinzler):
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.4] Log the query that would hit wb_terms.
Pinging @Lydia_Pintscher. This mostly affected Wikidiata, the site was down for a while (an hour)? As far as I can tell, this is unrelated to the lexeme deployment. It may be related to the wb_terms work @Ladsgroup was doing, and/or something hitting Special:ItemDisambiguation rather hard. Analysis is ongoing.
Status summary:
There are some suggestions of further actions on https://etherpad.wikimedia.org/p/wb_terms_solution. Needs cleaning up and proper task breakdown.
Logging more info now, see https://gerrit.wikimedia.org/r/q/Id9fdc74829e6268ecc3861602adf6666c2eaffc4
Some users on ptwiki related that coudn't access the Meta, rollback editions or see diffs. And when I tried access wikidata appeared a server error warning:
Change 435057 merged by Ladsgroup:
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.4] Log the query that would hit wb_terms.
Change 435055 merged by Ladsgroup:
[mediawiki/extensions/Wikibase@wmf/1.32.0-wmf.5] Log the query that would hit wb_terms.
Mentioned in SAL (#wikimedia-operations) [2018-05-24T22:11:42Z] <ladsgroup@tin> Synchronized php-1.32.0-wmf.4/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Log the query that would hit wb_terms. (T195520) (duration: 01m 21s)
Mentioned in SAL (#wikimedia-operations) [2018-05-24T22:21:12Z] <ladsgroup@tin> Synchronized php-1.32.0-wmf.5/extensions/Wikibase/lib/includes/Store/Sql/TermSqlIndex.php: Log the query that would hit wb_terms. (T195520) (duration: 01m 20s)
Change 435079 had a related patch set uploaded (by Ladsgroup; owner: Amir Sarabadani):
[mediawiki/extensions/Wikibase@master] Use TypeDispatchingEntitySearchHelper for SearchHelper in several places
Incident report (in progress): https://wikitech.wikimedia.org/wiki/Incident_documentation/20180524-wikidata
Suggestions for the future:
The TermSqlIndex::getMatchingTerms method which was patched out during the incident has now been added back.
* 15:11 addshore: Wikibase - Re enable wb_terms things window done
* 14:57 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: [[gerrit:436007{{!}}TermSqlIndex::getMatchingTerms actually execute select]] (duration: 02m 19s)
* 14:49 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: [[gerrit:436006{{!}}TermSqlIndex::getMatchingTerms actually execute select]] (duration: 02m 18s)
* 14:32 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: [[gerrit:436004{{!}}Re add TermSqlIndex::getMatchingTerms select, but dont call]] (duration: 02m 18s)
* 14:29 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: [[gerrit:436003{{!}}Re add TermSqlIndex::getMatchingTerms select, but dont call]] (duration: 02m 13s)
* 14:13 addshore@tin: Synchronized php-1.32.0-wmf.4/extensions/Wikibase: [[gerrit:436001{{!}}track all wb_terms table access via statsd]] (duration: 02m 19s)
* 14:10 addshore@tin: Synchronized php-1.32.0-wmf.5/extensions/Wikibase: [[gerrit:436000{{!}}track all wb_terms table access via statsd]] (duration: 02m 21s)ArticlePlaceholder search exposure, PropertySuggester and Special:ItemDisambiguation are still disabled individually.
There is also a new grafana dashboard monitoring the calls to the wb_terms table now. https://grafana-admin.wikimedia.org/dashboard/db/wikibase-wb_terms
I created T195792: Add support for setting individual query timeout in wikimedia/rdbms as something that would be useful to have right now for this case to avoid it taking down the whole shard.
Another thing that I would personally like to look into is pages to wikidata deployers for s8 issues / lag and or www.wikidata.org downtime / exceptions (not filed a ticket yet)
Mentioned in SAL (#wikimedia-operations) [2018-05-29T15:52:02Z] <addshore@tin> Synchronized wmf-config/Wikibase.php: [[gerrit:435147|Revert - Dont load PropertySuggester]] T195520 (duration: 01m 19s)
Change 435079 merged by jenkins-bot:
[mediawiki/extensions/Wikibase@master] Use TypeDispatchingEntitySearchHelper for SearchHelper in several places
So to wrap this ticket up the incident report can be found at https://wikitech.wikimedia.org/wiki/Incident_documentation/20180524-wikidata
There are still a collection of mid term actionables in progress, but the outage itself stopped quite some time ago.
Change 435042 abandoned by Jforrester:
Do not register SpecialItemDisambiguation to stop DoS.
Reason:
Production wikis are now running wmf.8 or wmf.10.
Change 435044 abandoned by Jforrester:
Do not register SpecialItemDisambiguation to stop DoS.
Reason:
Production wikis are now running wmf.8 or wmf.10.