Page MenuHomePhabricator
Create Task
Maniphest T315510

Start maintenance script to backfill talk page comment database
Closed, ResolvedPublic
Actions

Assigned To
• matmarex
Authored By
ppelberg
Aug 18 2022, 12:12 AM
Tags
Referenced Files
F55289143: done.webm
Jun 13 2024, 3:29 PM
F41465575: matmarex-T315510-enwiki.log.gz
Nov 7 2023, 2:12 PM
F41458218: persistentRevisionThreadItems-s7.log.gz
Nov 6 2023, 2:50 PM
F41458198: persistentRevisionThreadItems-s6.log
Nov 6 2023, 2:50 PM
F37160298: image.png
Aug 2 2023, 12:34 PM
F35498612: T315510maintScriptOutput.txt
Aug 30 2022, 9:11 PM
Subscribers
Aklapper
DatGuy
Esanders
Ladsgroup
• Lucas_Werkmeister_WMDE
• matmarex
ppelberg
View All 11 Subscribers
Tokens
"Stroopwafel" token, awarded by VPuffetMichel."Like" token, awarded by ppelberg.

Description

T315353 will create the database that will store the talk page comments and topics people will generate permanent links to.

This task involves the work of running the persistRevisionThreadItems.php maintenance script for long enough to populate said database with all wikis' existing wikitext comments and topics.

This way, by the time we expose the affordances that will enable people to generate links to specific comments (T275729) and topics (T273341), nearly all of the comments or topics people might want to generate a link to will have permanent links associated with them.

Open questions

  • [@matmarex] What – if any – adjustments do we need to make to the script to ensure that we can run on production without issue?

Progress

Details

SubjectRepoBranchLines +/-
ParserOutputAccess: Limit local cache sizemediawiki/corewmf/1.42.0-wmf.4+10 -0
ParserOutputAccess: Limit local cache sizemediawiki/coremaster+10 -0
Rewrite persistRevisionThreadItemsmediawiki/extensions/DiscussionToolsmaster+99 -31
persistRevisionThreadItems: Allow processing current revisions onlymediawiki/extensions/DiscussionToolswmf/1.39.0-wmf.26+7 -0
persistRevisionThreadItems: Allow processing current revisions onlymediawiki/extensions/DiscussionToolsmaster+7 -0
Customize query in gerrit

Related Objects
Search...

StatusSubtypeAssignedTask
 OpenNoneT314973 Improve Topic Subscription support for discussions that are merged, moved, or deleted
 OpenNoneT315050 Enable people to visit the page the discussion they were subscribed to has been moved to
 OpenNoneT318449 [Tracking] Improve the sharing of MediaWiki content
 OpenNoneT318446 [Tracking] Improve sharing link functionality of pages
 OpenNoneT315507 Introduce support for topic and comment permanent links within the Android and iOS apps
 OpenNoneT349353 Expose comment/topic permanent links on Special:Contributions
 OpenNoneT265269 Assist with linking to other comments within the Reply Tool
 ResolvedppelbergT302011 [Release Ticket] Introduce permalinks on wikitext talk pages
 ResolvedDLynchT365974 Deploy talk page permalinks to en.wiki
 Resolved• matmarexT315353 Create database tables for permalinks in production wikis, and enable the feature
 Resolved• matmarexT315510 Start maintenance script to backfill talk page comment database
 Resolved• matmarexT316548 DiscussionTools' maintenance scripts cannot be executed in Wikimedia production
 Invalid• matmarexT316915 It's 2022 and we still can't use Parsoid in production
 ResolvedPRODUCTION ERROR• matmarexT315383 MWException: Error contacting the Parsoid/RESTBase server (HTTP 404) from DiscussionTools (on closed wikis)
 Resolved• matmarexT315688 MWException: Error contacting the Parsoid/RESTBase server (HTTP 404) from DiscussionTools (on open wikis) – permalinks unavailable for some edits
 Resolved• matmarexT315689 MWException: Error contacting the Parsoid/RESTBase server (HTTP 403): (no message) from DiscussionTools (on private wikis) – permalinks unavailable
 Resolved• matmarexT316914 MWException: Error contacting the Parsoid/RESTBase server (HTTP 403): Rerenders for this article are blacklisted in the config from DiscussionTools – permalinks unavailable for some pages
 Resolved• matmarexT329627 Optimize permalinks maintenance script to avoid listing non-discussion pages
 Resolved• matmarexT351732 Debug memory leak in maintenance script
 ResolvedSecurity• matmarexT356884 Specific fr.wp page is very slow to parse

Event Timeline

There are a very large number of changes, so older changes are hidden. Show Older Changes
Stashbot added a comment.Feb 7 2024, 3:40 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-07T15:40:08Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint2002:~$ mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --start '["67578461"]' | tee ~/T315510-enwiki # in tmux

Stashbot added a comment.Feb 7 2024, 4:03 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-07T16:03:34Z] <Lucas_WMDE> STOP persistRevisionThreadItems on rowiki for T315510 – according to T315510#9328399, it should be done already (it was at --start '["2075226"]' and had processed 31000, updated 0) [relog from 15:45, stashbot was down]

Stashbot added a comment.Feb 7 2024, 4:15 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-07T16:15:19Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint2002:~$ mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki ukwiki --current --all --touched-after=20230613000000 --start '["1685316"]' | tee ~/T315510-ukwiki # in tmux

• Lucas_Werkmeister_WMDE added a comment.Feb 7 2024, 4:26 PM

I started the ukwiki run based on the last output in T315510#9328399 / P53378, but it hasn’t had much to do yet (updated should be > 0, for enwiki it’s around 1-2% of the number of processed rows):

Processed 5000 (updated 0) of 4508541 rows
--start '["1708556"]'

@Urbanecm did you run the script again? (I’m not sure how to interpret the last sentence of the comment.) If yes, how far did it get?

• matmarex added a subtask: T356884: Specific fr.wp page is very slow to parse.Feb 7 2024, 6:40 PM
• Lucas_Werkmeister_WMDE added a comment.Feb 8 2024, 10:15 AM

Apparently ukwiki found something to do, so disregard the above question. We can just let it run to completion.

Processed 40700 (updated 0) of 4508541 rows
--start '["1826788"]'
Processed 40800 (updated 1) of 4508541 rows
--start '["1827152"]'

(By now it’s even “updated 2”, gasp.)

• Lucas_Werkmeister_WMDE added a comment.Feb 8 2024, 10:31 AM

There are also occasional errors in the enwiki run (so far I haven’t seen them in the ukwiki run, though they don’t end up in the tee’d file and so it’s possible they just scrolled out of the tmux buffer):

Processed 529800 (updated 121015) of 42066285 rows
--start '["68647345"]'
Error while processing revid=1042438330, pageid=68647413
[8202ffe7a01340425a6ed25c] [no req]   TypeError: Argument 1 passed to ExternalStoreDB::getDomainId() must be of the type array, bool given, called in /srv/mediawiki/php-1.42.0-wmf.16/includes/externalstore/ExternalStoreDB.php on line 165                                                                                                                                                
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.16/includes/externalstore/ExternalStoreDB.php(192)
#0 /srv/mediawiki/php-1.42.0-wmf.16/includes/externalstore/ExternalStoreDB.php(165): ExternalStoreDB->getDomainId(boolean)
#1 /srv/mediawiki/php-1.42.0-wmf.16/includes/externalstore/ExternalStoreDB.php(299): ExternalStoreDB->getReplica(string)
#2 /srv/mediawiki/php-1.42.0-wmf.16/includes/externalstore/ExternalStoreDB.php(67): ExternalStoreDB->fetchBlob(string, string, boolean)
#3 /srv/mediawiki/php-1.42.0-wmf.16/includes/externalstore/ExternalStoreAccess.php(73): ExternalStoreDB->fetchFromURL(string)
#4 /srv/mediawiki/php-1.42.0-wmf.16/includes/Storage/SqlBlobStore.php(535): ExternalStoreAccess->fetchFromURL(string, array)
#5 /srv/mediawiki/php-1.42.0-wmf.16/includes/libs/objectcache/wancache/WANObjectCache.php(1726): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL, array)                                                                                                                                                                                           
#6 /srv/mediawiki/php-1.42.0-wmf.16/includes/libs/objectcache/wancache/WANObjectCache.php(1556): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#7 /srv/mediawiki/php-1.42.0-wmf.16/includes/Storage/SqlBlobStore.php(539): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#8 /srv/mediawiki/php-1.42.0-wmf.16/includes/Storage/SqlBlobStore.php(432): MediaWiki\Storage\SqlBlobStore->expandBlob(string, array, string)
#9 /srv/mediawiki/php-1.42.0-wmf.16/includes/Storage/SqlBlobStore.php(268): MediaWiki\Storage\SqlBlobStore->fetchBlobs(array, integer)
#10 /srv/mediawiki/php-1.42.0-wmf.16/includes/libs/objectcache/wancache/WANObjectCache.php(1726): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL, array)                                                                                                                                                                                          
#11 /srv/mediawiki/php-1.42.0-wmf.16/includes/libs/objectcache/wancache/WANObjectCache.php(1556): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#12 /srv/mediawiki/php-1.42.0-wmf.16/includes/Storage/SqlBlobStore.php(276): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#13 /srv/mediawiki/php-1.42.0-wmf.16/includes/Revision/RevisionStore.php(1200): MediaWiki\Storage\SqlBlobStore->getBlob(string, integer)
#14 /srv/mediawiki/php-1.42.0-wmf.16/includes/Revision/RevisionStore.php(1518): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, NULL, NULL, NULL, integer)
#15 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#16 /srv/mediawiki/php-1.42.0-wmf.16/includes/Revision/SlotRecord.php(324): call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#17 /srv/mediawiki/php-1.42.0-wmf.16/includes/Revision/RevisionRecord.php(158): MediaWiki\Revision\SlotRecord->getContent()
#18 /srv/mediawiki/php-1.42.0-wmf.16/includes/parser/Parser.php(3677): MediaWiki\Revision\RevisionRecord->getContent(string)
#19 /srv/mediawiki/php-1.42.0-wmf.16/includes/parser/Parser.php(3550): Parser::statelessFetchTemplate(MediaWiki\Title\Title, Parser)
#20 /srv/mediawiki/php-1.42.0-wmf.16/includes/parser/Parser.php(3444): Parser->fetchTemplateAndTitle(MediaWiki\Title\Title)
#21 /srv/mediawiki/php-1.42.0-wmf.16/includes/parser/Parser.php(3185): Parser->getTemplateDom(MediaWiki\Title\Title)
#22 /srv/mediawiki/php-1.42.0-wmf.16/includes/parser/PPFrame_Hash.php(274): Parser->braceSubstitution(array, PPFrame_Hash)
#23 /srv/mediawiki/php-1.42.0-wmf.16/includes/parser/Parser.php(2903): PPFrame_Hash->expand(PPNode_Hash_Tree, integer)
#24 /srv/mediawiki/php-1.42.0-wmf.16/includes/parser/Parsoid/Config/DataAccess.php(384): Parser->replaceVariables(string, PPFrame_Hash)
#25 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wikitext/Wikitext.php(43): MediaWiki\Parser\Parsoid\Config\DataAccess->preprocessWikitext(MediaWiki\Parser\Parsoid\Config\PageConfig, MediaWiki\Parser\ParserOutput, string)                                                                                                                                               
#26 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(1068): Wikimedia\Parsoid\Wikitext\Wikitext::preprocess(Wikimedia\Parsoid\Config\Env, string)
#27 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(1111): Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->expandTemplate(Wikimedia\Parsoid\Wt2Html\TT\TemplateEncapsulator)                                                                                                                                                                      
#28 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wt2Html/TT/TemplateHandler.php(1152): Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->onTemplate(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)                                                                                                                                                                                  
#29 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wt2Html/TT/TokenHandler.php(138): Wikimedia\Parsoid\Wt2Html\TT\TemplateHandler->onTag(Wikimedia\Parsoid\Tokens\SelfclosingTagTk)                                                                                                                                                                                           
#30 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(113): Wikimedia\Parsoid\Wt2Html\TT\TokenHandler->process(array)
#31 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(176): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunk(array)
#32 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wt2Html/TokenTransformManager.php(174): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunkily(string, array)
#33 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wt2Html/TreeBuilder/TreeBuilderStage.php(483): Wikimedia\Parsoid\Wt2Html\TokenTransformManager->processChunkily(string, array)                                                                                                                                                                                             
#34 [internal function]: Wikimedia\Parsoid\Wt2Html\TreeBuilder\TreeBuilderStage->processChunkily(string, array)
#35 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wt2Html/DOMPostProcessor.php(963): Generator->current()
#36 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipeline.php(170): Wikimedia\Parsoid\Wt2Html\DOMPostProcessor->processChunkily(string, array)
#37 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wt2Html/ParserPipelineFactory.php(279): Wikimedia\Parsoid\Wt2Html\ParserPipeline->parseChunkily(string, array)
#38 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Wikitext/ContentModelHandler.php(161): Wikimedia\Parsoid\Wt2Html\ParserPipelineFactory->parse(string)
#39 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Parsoid.php(191): Wikimedia\Parsoid\Wikitext\ContentModelHandler->toDOM(Wikimedia\Parsoid\Ext\ParsoidExtensionAPI)
#40 /srv/mediawiki/php-1.42.0-wmf.16/vendor/wikimedia/parsoid/src/Parsoid.php(232): Wikimedia\Parsoid\Parsoid->parseWikitext(MediaWiki\Parser\Parsoid\Config\PageConfig, MediaWiki\Parser\ParserOutput, array)                                                                                                                                                                               
#41 /srv/mediawiki/php-1.42.0-wmf.16/includes/parser/Parsoid/ParsoidParser.php(152): Wikimedia\Parsoid\Parsoid->wikitext2html(MediaWiki\Parser\Parsoid\Config\PageConfig, array, NULL, MediaWiki\Parser\ParserOutput)                                                                                                                                                                        
#42 /srv/mediawiki/php-1.42.0-wmf.16/includes/parser/Parsoid/ParsoidParser.php(260): MediaWiki\Parser\Parsoid\ParsoidParser->genParserOutput(MediaWiki\Parser\Parsoid\Config\PageConfig, ParserOptions)                                                                                                                                                                                      
#43 /srv/mediawiki/php-1.42.0-wmf.16/includes/content/WikitextContentHandler.php(397): MediaWiki\Parser\Parsoid\ParsoidParser->parse(string, MediaWiki\Title\Title, ParserOptions, boolean, boolean, integer)                                                                                                                                                                                
#44 /srv/mediawiki/php-1.42.0-wmf.16/includes/content/ContentHandler.php(1656): WikitextContentHandler->fillParserOutput(WikitextContent, MediaWiki\Content\Renderer\ContentParseParams, MediaWiki\Parser\ParserOutput)                                                                                                                                                                      
#45 /srv/mediawiki/php-1.42.0-wmf.16/includes/content/Renderer/ContentRenderer.php(47): ContentHandler->getParserOutput(WikitextContent, MediaWiki\Content\Renderer\ContentParseParams)
#46 /srv/mediawiki/php-1.42.0-wmf.16/includes/Revision/RenderedRevision.php(260): MediaWiki\Content\Renderer\ContentRenderer->getParserOutput(WikitextContent, MediaWiki\Title\Title, integer, ParserOptions, boolean)                                                                                                                                                                       
#47 /srv/mediawiki/php-1.42.0-wmf.16/includes/Revision/RenderedRevision.php(232): MediaWiki\Revision\RenderedRevision->getSlotParserOutputUncached(WikitextContent, boolean)
#48 /srv/mediawiki/php-1.42.0-wmf.16/includes/Revision/RevisionRenderer.php(226): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#49 /srv/mediawiki/php-1.42.0-wmf.16/includes/Revision/RevisionRenderer.php(164): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, ParserOptions, array)                                                                                                                                                                                          
#50 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#51 /srv/mediawiki/php-1.42.0-wmf.16/includes/Revision/RenderedRevision.php(199): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#52 /srv/mediawiki/php-1.42.0-wmf.16/includes/poolcounter/PoolWorkArticleView.php(87): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#53 /srv/mediawiki/php-1.42.0-wmf.16/includes/poolcounter/PoolWorkArticleViewCurrent.php(110): MediaWiki\PoolCounter\PoolWorkArticleView->renderRevision()
#54 /srv/mediawiki/php-1.42.0-wmf.16/includes/poolcounter/PoolCounterWork.php(172): MediaWiki\PoolCounter\PoolWorkArticleViewCurrent->doWork()
#55 /srv/mediawiki/php-1.42.0-wmf.16/includes/page/ParserOutputAccess.php(307): MediaWiki\PoolCounter\PoolCounterWork->execute()
#56 /srv/mediawiki/php-1.42.0-wmf.16/extensions/DiscussionTools/includes/Hooks/HookUtils.php(157): MediaWiki\Page\ParserOutputAccess->getParserOutput(MediaWiki\Page\PageStoreRecord, ParserOptions, MediaWiki\Revision\RevisionStoreRecord, integer)                                                                                                                                        
#57 /srv/mediawiki/php-1.42.0-wmf.16/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(207): MediaWiki\Extension\DiscussionTools\Hooks\HookUtils::parseRevisionParsoidHtml(MediaWiki\Revision\RevisionStoreRecord, boolean)                                                                                                                                              
#58 /srv/mediawiki/php-1.42.0-wmf.16/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                            
#59 /srv/mediawiki/php-1.42.0-wmf.16/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                              
#60 /srv/mediawiki/php-1.42.0-wmf.16/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#61 /srv/mediawiki/php-1.42.0-wmf.16/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#62 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#63 {main}
Processed 529900 (updated 121056) of 42066285 rows
--start '["68647561"]'

Hopefully not a big deal, I think.

sbassett closed subtask T356884: Specific fr.wp page is very slow to parse as Resolved.Feb 8 2024, 4:53 PM
• matmarex added a comment.Feb 8 2024, 6:01 PM
In T315510#9521462, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2024-02-07T15:31:07Z] <Lucas_WMDE> STOP persistRevisionThreadItems on frwiki for T315510 – 100% CPU usage, 15G RAM and counting, no progress output: clearly stuck on something

That should be fixed by T356884, so we could retry it now.

• Lucas_Werkmeister_WMDE added a comment.Feb 12 2024, 11:42 AM

ukwiki is done:

--start '["5174445"]'
Processed 1073600 (updated 63470) of 4508541 rows
--start '["5174771"]'
Processed 1073693 (updated 63470) of 4508541 rows
Finished in 4 d 8 h 4 min 41 s
Stashbot added a comment.Feb 12 2024, 11:46 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-12T11:46:03Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint2002:~$ mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki viwiki --current --all --touched-after=20230613000000 2>&1 | tee ~/T315510-viwiki # in tmux

Stashbot added a comment.Feb 12 2024, 11:48 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-12T11:48:04Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint2002:~$ mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki frwiki --current --all --touched-after=20230613000000 --start '["7544396"]' 2>&1 | tee ~/T315510-frwiki # in tmux

• Lucas_Werkmeister_WMDE added a comment.Feb 12 2024, 11:48 AM
In T315510#9526286, @matmarex wrote:
In T315510#9521462, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2024-02-07T15:31:07Z] <Lucas_WMDE> STOP persistRevisionThreadItems on frwiki for T315510 – 100% CPU usage, 15G RAM and counting, no progress output: clearly stuck on something

That should be fixed by T356884, so we could retry it now.

Yup, looks like it’s making progress now:

lucaswerkmeister-wmde@mwmaint2002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki frwiki --current --all --touched-after=20230613000000 --start '["7544396"]' 2>&1 | tee ~/T315510-frwiki; date                                                                                                                                                
Processing... (estimated 14284762 rows)
--start '["7544396"]'
Processed 100 (updated 34) of 14284762 rows
--start '["7544598"]'
• Lucas_Werkmeister_WMDE added a comment.EditedFeb 12 2024, 12:13 PM

enwiki looks like this btw:

Processed 1842600 (updated 748349) of 42066285 rows
--start '["71449088"]'

Which is pretty worrying – 42 million rows, and it’s not even processed two million, after five days :S

The enwiki run also seems to use more CPU than the others (~70-90% rather than ~40-60%) and to be slower to print new lines – it prints a new “processed” / “--start” line roughly every 30 seconds (though it varies quite a bit), while frwiki and viwiki (and my recollection of rowiki and ukwiki) are closer to ten seconds per line. I’m not sure if that’s because there’s more to do on enwiki or whether there’s some kind of inherent slowdown in the script as it runs for longer and longer.

I think we can keep it running for now, but I want to restart it once a week or so, after the train rolls out, to make sure it doesn’t keep running old code forever. And then I can also check if that seems to speed it up at all.

(In terms of memory, enwiki is still dancing around 4607M VIRT according to htop, which is what I remember from Friday as well, so I think it’s not leaking any memory at the moment.)

• matmarex added a comment.Feb 12 2024, 11:58 PM
In T315510#9533562, @Lucas_Werkmeister_WMDE wrote:

enwiki looks like this btw:

Processed 1842600 (updated 748349) of 42066285 rows
--start '["71449088"]'

Which is pretty worrying – 42 million rows, and it’s not even processed two million, after five days :S

That sounds like excellent progress to me! (Not joking – well, maybe a little. :) ) At this rate it'll be done in about 20 weeks, only a little worse than I predicted in T315510#8200863. We opened this task about 80 weeks ago, so it's fair to say it's almost done. ;)

(Also, the estimate isn't very good, the real number is around 33 million, or at least it was a few months ago – T315510#9247752.)

(Also, both estimates count the rows that were processed in the previous cancelled script runs and skipped with the --start option, so we're actually a million or so ahead.)

The enwiki run also seems to use more CPU than the others (~70-90% rather than ~40-60%) and to be slower to print new lines – it prints a new “processed” / “--start” line roughly every 30 seconds (though it varies quite a bit), while frwiki and viwiki (and my recollection of rowiki and ukwiki) are closer to ten seconds per line. I’m not sure if that’s because there’s more to do on enwiki or whether there’s some kind of inherent slowdown in the script as it runs for longer and longer.

Interesting, I think this is because there’s indeed more to do on enwiki. Almost every article talk page there has relatively expensive content assessment and WikiProject templates, and on average takes ~0.2 seconds to parse. Other projects don't have such templates, or have much simpler ones, so an average frwiki talk page takes ~0.1 s, and a viwiki page ~0.03 s. Here's a quick Ruby script you can run locally to see:

%w[en.wikipedia.org fr.wikipedia.org vi.wikipedia.org].each{|dom|
	puts dom
	10.times{
		system 'curl -s -L "https://'+dom+'/wiki/Special:Random/Talk" | grep "Real time usage:"'
	}
}
• matmarex added a comment.Feb 12 2024, 11:59 PM
In T315510#9524160, @Lucas_Werkmeister_WMDE wrote:

There are also occasional errors in the enwiki run (so far I haven’t seen them in the ukwiki run, though they don’t end up in the tee’d file and so it’s possible they just scrolled out of the tmux buffer):

Error while processing revid=1042438330, pageid=68647413
[8202ffe7a01340425a6ed25c] [no req]   TypeError: Argument 1 passed to ExternalStoreDB::getDomainId() must be of the type array, bool given, called in /srv/mediawiki/php-1.42.0-wmf.16/includes/externalstore/ExternalStoreDB.php on line 165

Hopefully not a big deal, I think.

I browser around that code a little and I think this happens when a database replica server is depooled, but the script still tries to use it. This is supposed to be mitigated by the call to autoReconfigure() here in Maintenance.php, but I guess that doesn't work for external storage databases, where we store the article wikitext? Anyway, if you ever see anyone suggest solving a problem with a maintenance script that takes more than 10 minutes, especially if it's me, whack them upside the head.

• Lucas_Werkmeister_WMDE added a comment.Feb 13 2024, 9:53 AM
In T315510#9536149, @matmarex wrote:

That sounds like excellent progress to me! (Not joking – well, maybe a little. :) ) At this rate it'll be done in about 20 weeks, only a little worse than I predicted in T315510#8200863. We opened this task about 80 weeks ago, so it's fair to say it's almost done. ;)

Okay – I thought I remembered you saying on IRC that the expected runtime was about a month, so only 4–5 weeks ^^

In T315510#9536150, @matmarex wrote:

I browser around that code a little and I think this happens when a database replica server is depooled, but the script still tries to use it. This is supposed to be mitigated by the call to autoReconfigure() here in Maintenance.php, but I guess that doesn't work for external storage databases, where we store the article wikitext? Anyway, if you ever see anyone suggest solving a problem with a maintenance script that takes more than 10 minutes, especially if it's me, whack them upside the head.

I’m guessing that the reconfigure still mostly works (otherwise the script would stop working at all?), but maybe doesn’t prevent the first instance of an error? Not sure. Also https://bash.toolforge.org/quip/a2XhoY0BxE1_1c7sx4wW.

• matmarex mentioned this in T357492: Auto-reconfigure external storage database replicas in long-running maintenance scripts.Feb 14 2024, 1:58 AM
• Lucas_Werkmeister_WMDE added a comment.Feb 15 2024, 10:35 AM
In T315510#9533562, @Lucas_Werkmeister_WMDE wrote:

(In terms of memory, enwiki is still dancing around 4607M VIRT according to htop, which is what I remember from Friday as well, so I think it’s not leaking any memory at the moment.)

It was still in the same range yesterday but now it’s suddenly at 10.8G o_O

One more reason to restart the script, I guess (but I’ll wait until the train reaches group2, so restart either this evening or tomorrow).

Stashbot added a comment.Feb 16 2024, 2:32 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-16T14:32:32Z] <Lucas_WMDE> STOP persistRevisionThreadItems on enwiki for T315510 – for restart on wmf.18; last output: --start '["73436010"]'

Stashbot added a comment.Feb 16 2024, 2:32 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-16T14:32:51Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint2002:~$ mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --start '["67578461"]' | tee -a ~/T315510-enwiki

Stashbot added a comment.Feb 16 2024, 2:33 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-16T14:33:38Z] <Lucas_WMDE> STOP persistRevisionThreadItems.php on enwiki for T315510 again, I forgot to adjust the --start >.<

• Lucas_Werkmeister_WMDE added a comment.Feb 16 2024, 2:34 PM
In T315510#9550627, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2024-02-16T14:33:38Z] <Lucas_WMDE> STOP persistRevisionThreadItems.php on enwiki for T315510 again, I forgot to adjust the --start >.<

So why was it still updating some rows anyway? o_O

lucaswerkmeister-wmde@mwmaint2002:~$ mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --start '["67578461"]' | tee -a ~/T315510-enwiki
Processing... (estimated 42066285 rows)
--start '["67578461"]'
Processed 100 (updated 1) of 42066285 rows
--start '["67578730"]'
Processed 200 (updated 3) of 42066285 rows
--start '["67578906"]'
Processed 300 (updated 4) of 42066285 rows
--start '["67579117"]'
Processed 400 (updated 5) of 42066285 rows
--start '["67579317"]'
^C
Stashbot added a comment.Feb 16 2024, 2:34 PM

Mentioned in SAL (#wikimedia-operations) [2024-02-16T14:34:19Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint2002:~$ mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --start '["73436010"]' | tee -a ~/T315510-enwiki

• Lucas_Werkmeister_WMDE added a comment.Feb 16 2024, 2:40 PM
In T315510#9533562, @Lucas_Werkmeister_WMDE wrote:

The enwiki run also seems to use more CPU than the others (~70-90% rather than ~40-60%) and to be slower to print new lines – it prints a new “processed” / “--start” line roughly every 30 seconds (though it varies quite a bit), while frwiki and viwiki (and my recollection of rowiki and ukwiki) are closer to ten seconds per line. I’m not sure if that’s because there’s more to do on enwiki or whether there’s some kind of inherent slowdown in the script as it runs for longer and longer.

After the restart it seems to be printing new lines substantially faster – previously it was somewhere between 30 and 50 seconds per update, now often below 10 seconds. So it’s probably a good idea to keep restarting this every week or so. (frwiki is also approaching 30 seconds per update, viwiki closer to 10 seconds; maybe restart them every two weeks or so, at least until they’re done.)

• matmarex added a comment.Feb 16 2024, 2:54 PM
In T315510#9550633, @Lucas_Werkmeister_WMDE wrote:
In T315510#9550627, @Stashbot wrote:

Mentioned in SAL (#wikimedia-operations) [2024-02-16T14:33:38Z] <Lucas_WMDE> STOP persistRevisionThreadItems.php on enwiki for T315510 again, I forgot to adjust the --start >.<

So why was it still updating some rows anyway? o_O

The pages might have been edited in the meantime, which would have caused the script to process the new comments.

• Lucas_Werkmeister_WMDE added a comment.Feb 21 2024, 11:05 AM

Apparently frwiki finished earlier today, though there were still a number of these ExternalStoreDB::getDomainId() errors in the output:

Processed 4026800 (updated 2305440) of 14284762 rows
--start '["16150776"]'
Error while processing revid=212609949, pageid=16150928
[1051d195689cfb46e9da7f7a] [no req]   TypeError: Argument 1 passed to ExternalStoreDB::getDomainId() must be of the type array, bool given, called in /srv/mediawiki/php-1.42.0-wmf.17/includes/externalstore/ExternalStoreDB.php on line 165                                                                                                                                                
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.17/includes/externalstore/ExternalStoreDB.php(192)
#0 /srv/mediawiki/php-1.42.0-wmf.17/includes/externalstore/ExternalStoreDB.php(165): ExternalStoreDB->getDomainId(boolean)
#1 /srv/mediawiki/php-1.42.0-wmf.17/includes/externalstore/ExternalStoreDB.php(299): ExternalStoreDB->getReplica(string)
#2 /srv/mediawiki/php-1.42.0-wmf.17/includes/externalstore/ExternalStoreDB.php(67): ExternalStoreDB->fetchBlob(string, string, boolean)
#3 /srv/mediawiki/php-1.42.0-wmf.17/includes/externalstore/ExternalStoreAccess.php(73): ExternalStoreDB->fetchFromURL(string)
#4 /srv/mediawiki/php-1.42.0-wmf.17/includes/Storage/SqlBlobStore.php(535): ExternalStoreAccess->fetchFromURL(string, array)
#5 /srv/mediawiki/php-1.42.0-wmf.17/includes/libs/objectcache/wancache/WANObjectCache.php(1726): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL, array)                                                                                                                                                                                           
#6 /srv/mediawiki/php-1.42.0-wmf.17/includes/libs/objectcache/wancache/WANObjectCache.php(1556): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#7 /srv/mediawiki/php-1.42.0-wmf.17/includes/Storage/SqlBlobStore.php(539): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#8 /srv/mediawiki/php-1.42.0-wmf.17/includes/Storage/SqlBlobStore.php(432): MediaWiki\Storage\SqlBlobStore->expandBlob(string, array, string)
#9 /srv/mediawiki/php-1.42.0-wmf.17/includes/Storage/SqlBlobStore.php(268): MediaWiki\Storage\SqlBlobStore->fetchBlobs(array, integer)
#10 /srv/mediawiki/php-1.42.0-wmf.17/includes/libs/objectcache/wancache/WANObjectCache.php(1726): MediaWiki\Storage\SqlBlobStore->MediaWiki\Storage\{closure}(boolean, integer, array, NULL, array)                                                                                                                                                                                          
#11 /srv/mediawiki/php-1.42.0-wmf.17/includes/libs/objectcache/wancache/WANObjectCache.php(1556): WANObjectCache->fetchOrRegenerate(string, integer, Closure, array, array)
#12 /srv/mediawiki/php-1.42.0-wmf.17/includes/Storage/SqlBlobStore.php(276): WANObjectCache->getWithSetCallback(string, integer, Closure, array)
#13 /srv/mediawiki/php-1.42.0-wmf.17/includes/Revision/RevisionStore.php(1189): MediaWiki\Storage\SqlBlobStore->getBlob(string, integer)
#14 /srv/mediawiki/php-1.42.0-wmf.17/includes/Revision/RevisionStore.php(1507): MediaWiki\Revision\RevisionStore->loadSlotContent(MediaWiki\Revision\SlotRecord, NULL, NULL, NULL, integer)
#15 [internal function]: MediaWiki\Revision\RevisionStore->MediaWiki\Revision\{closure}(MediaWiki\Revision\SlotRecord)
#16 /srv/mediawiki/php-1.42.0-wmf.17/includes/Revision/SlotRecord.php(324): call_user_func(Closure, MediaWiki\Revision\SlotRecord)
#17 /srv/mediawiki/php-1.42.0-wmf.17/includes/Revision/RevisionRecord.php(183): MediaWiki\Revision\SlotRecord->getContent()
#18 /srv/mediawiki/php-1.42.0-wmf.17/includes/Revision/RenderedRevision.php(229): MediaWiki\Revision\RevisionRecord->getContentOrThrow(string, integer, NULL)
#19 /srv/mediawiki/php-1.42.0-wmf.17/includes/Revision/RevisionRenderer.php(226): MediaWiki\Revision\RenderedRevision->getSlotParserOutput(string, array)
#20 /srv/mediawiki/php-1.42.0-wmf.17/includes/Revision/RevisionRenderer.php(164): MediaWiki\Revision\RevisionRenderer->combineSlotOutput(MediaWiki\Revision\RenderedRevision, ParserOptions, array)                                                                                                                                                                                          
#21 [internal function]: MediaWiki\Revision\RevisionRenderer->MediaWiki\Revision\{closure}(MediaWiki\Revision\RenderedRevision, array)
#22 /srv/mediawiki/php-1.42.0-wmf.17/includes/Revision/RenderedRevision.php(199): call_user_func(Closure, MediaWiki\Revision\RenderedRevision, array)
#23 /srv/mediawiki/php-1.42.0-wmf.17/includes/poolcounter/PoolWorkArticleView.php(87): MediaWiki\Revision\RenderedRevision->getRevisionParserOutput()
#24 /srv/mediawiki/php-1.42.0-wmf.17/includes/poolcounter/PoolWorkArticleViewCurrent.php(110): MediaWiki\PoolCounter\PoolWorkArticleView->renderRevision()
#25 /srv/mediawiki/php-1.42.0-wmf.17/includes/poolcounter/PoolCounterWork.php(172): MediaWiki\PoolCounter\PoolWorkArticleViewCurrent->doWork()
#26 /srv/mediawiki/php-1.42.0-wmf.17/includes/page/ParserOutputAccess.php(307): MediaWiki\PoolCounter\PoolCounterWork->execute()
#27 /srv/mediawiki/php-1.42.0-wmf.17/extensions/DiscussionTools/includes/Hooks/HookUtils.php(157): MediaWiki\Page\ParserOutputAccess->getParserOutput(MediaWiki\Page\PageStoreRecord, ParserOptions, MediaWiki\Revision\RevisionStoreRecord, integer)                                                                                                                                        
#28 /srv/mediawiki/php-1.42.0-wmf.17/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(207): MediaWiki\Extension\DiscussionTools\Hooks\HookUtils::parseRevisionParsoidHtml(MediaWiki\Revision\RevisionStoreRecord, boolean)                                                                                                                                              
#29 /srv/mediawiki/php-1.42.0-wmf.17/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                            
#30 /srv/mediawiki/php-1.42.0-wmf.17/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                              
#31 /srv/mediawiki/php-1.42.0-wmf.17/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#32 /srv/mediawiki/php-1.42.0-wmf.17/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#33 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#34 {main}
Processed 4026900 (updated 2305440) of 14284762 rows
--start '["16151006"]'
Processed 4027000 (updated 2305440) of 14284762 rows
--start '["16151214"]'
Processed 4027100 (updated 2305440) of 14284762 rows
--start '["16151443"]'
Processed 4027200 (updated 2305440) of 14284762 rows
--start '["16151635"]'
Processed 4027300 (updated 2305440) of 14284762 rows
--start '["16151897"]'
Processed 4027400 (updated 2305440) of 14284762 rows
--start '["16152201"]'
Processed 4027500 (updated 2305440) of 14284762 rows
--start '["16152454"]'
Processed 4027600 (updated 2305440) of 14284762 rows
--start '["16152710"]'
Processed 4027700 (updated 2305440) of 14284762 rows
--start '["16152914"]'
Processed 4027800 (updated 2305440) of 14284762 rows
--start '["16153101"]'
Processed 4027900 (updated 2305443) of 14284762 rows
--start '["16153262"]'
Processed 4028000 (updated 2305446) of 14284762 rows
--start '["16153470"]'
Processed 4028100 (updated 2305446) of 14284762 rows
--start '["16153762"]'
Processed 4028200 (updated 2305446) of 14284762 rows
--start '["16154005"]'
Processed 4028253 (updated 2305446) of 14284762 rows
Finished in 8 d 18 h 19 min 44 s

real    12619m44.628s
user    9475m47.573s
sys     100m42.534s
Wed 21 Feb 2024 06:07:48 AM UTC

viwiki is currently still running but also dumping tons of these ExternalStoreDB errors, very quickly; I have a feeling this means it’s skipping lots of revisions and might need to be rerun even if the errors stop? :S

And enwiki “finished” with tons of instances of a different error:

Error while processing revid=1208557278, pageid=76122856
[b60cc99c72fc9f7437c3b68c] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php on line 91                                                                                                                               
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorStore.php(187): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.18/includes/Revision/RevisionStore.php(1747): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.18/includes/Revision/RevisionStore.php(1610): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)                                                                                                                                                                                         
#4 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(202): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                             
#6 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                               
#7 /srv/mediawiki/php-1.42.0-wmf.18/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#8 /srv/mediawiki/php-1.42.0-wmf.18/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#10 {main}
Error while processing revid=1208557416, pageid=76122861
[b60cc99c72fc9f7437c3b68c] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php on line 91                                                                                                                               
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorStore.php(187): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.18/includes/Revision/RevisionStore.php(1747): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.18/includes/Revision/RevisionStore.php(1610): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)                                                                                                                                                                                         
#4 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(202): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                             
#6 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                               
#7 /srv/mediawiki/php-1.42.0-wmf.18/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#8 /srv/mediawiki/php-1.42.0-wmf.18/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#10 {main}
Error while processing revid=1208557478, pageid=76122863
[b60cc99c72fc9f7437c3b68c] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php on line 91                                                                                                                               
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorStore.php(187): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.18/includes/Revision/RevisionStore.php(1747): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.18/includes/Revision/RevisionStore.php(1610): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)                                                                                                                                                                                         
#4 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(202): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                             
#6 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                               
#7 /srv/mediawiki/php-1.42.0-wmf.18/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#8 /srv/mediawiki/php-1.42.0-wmf.18/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#10 {main}
Error while processing revid=1208557533, pageid=76122864
[b60cc99c72fc9f7437c3b68c] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php on line 91                                                                                                                               
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.18/includes/user/ActorStore.php(187): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.18/includes/Revision/RevisionStore.php(1747): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.18/includes/Revision/RevisionStore.php(1610): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)                                                                                                                                                                                         
#4 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(202): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                             
#6 /srv/mediawiki/php-1.42.0-wmf.18/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                               
#7 /srv/mediawiki/php-1.42.0-wmf.18/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#8 /srv/mediawiki/php-1.42.0-wmf.18/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#10 {main}
Processed 1366154 (updated 443153) of 42066285 rows
Finished in 33 h 51 min 59 s
Stashbot added a comment.Feb 23 2024, 11:49 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-23T11:49:12Z] <Lucas_WMDE> STOP persistRevisionThreadItems on viwiki for T315510, had been throwing tons of errors since at least Wednesday

• Lucas_Werkmeister_WMDE added a comment.Feb 23 2024, 11:51 AM

I’ll restart viwiki from the last point before it started throwing TypeErrors according to ~lucaswerkmeister-wmde/T315510-viwiki:

Processed 5834600 (updated 9) of 19943094 rows
--start '["7939440"]'
Processed 5834700 (updated 9) of 19943094 rows
--start '["7939540"]'
Processed 5834800 (updated 9) of 19943094 rows
--start '["7939640"]'
Processed 5834900 (updated 9) of 19943094 rows
--start '["7939741"]'
Error while processing revid=60685842, pageid=7939743
[53de2b5697d57afc92c3b25f] [no req]   TypeError: Argument 1 passed to ExternalStoreDB::getDomainId() must be of the type array, bool given, called in /srv/mediawiki/php-1.42.0-wmf.17/includes/externalstore/ExternalStoreDB.php on line 165
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.17/includes/externalstore/ExternalStoreDB.php(192)
#0 /srv/mediawiki/php-1.42.0-wmf.17/includes/externalstore/ExternalStoreDB.php(165): ExternalStoreDB->getDomainId(boolean)
Stashbot added a comment.Feb 23 2024, 11:52 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-23T11:52:08Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint2002:~$ mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki viwiki --current --all --touched-after=20230613000000 --start '["7939741"]' 2>&1 | tee ~/T315510-viwiki # in tmux

• Lucas_Werkmeister_WMDE added a comment.Feb 23 2024, 11:57 AM

And I’ll restart enwiki from the point before it started reporting tons and tons of errors according to ~lucaswerkmeister-wmde/T315510-enwiki:

Processed 883300 (updated 442835) of 42066285 rows
--start '["75193213"]'
Processed 883400 (updated 442898) of 42066285 rows
--start '["75193421"]'
Processed 883500 (updated 442959) of 42066285 rows
--start '["75193642"]'
Processed 883600 (updated 443025) of 42066285 rows
--start '["75193850"]'
Processed 883700 (updated 443081) of 42066285 rows
--start '["75194086"]'
Processed 883800 (updated 443128) of 42066285 rows
--start '["75194261"]'
Error while processing revid=1182893250, pageid=75194364
Error while processing revid=1182893258, pageid=75194365
Error while processing revid=1182893269, pageid=75194367
Error while processing revid=1197319250, pageid=75194368

Note that there were a few isolated errors before then, which I’m assuming we can just ignore:

Error while processing revid=1052801590, pageid=68519093
Error while processing revid=1172741659, pageid=68549096
Error while processing revid=1041192124, pageid=68591215
Error while processing revid=1042438330, pageid=68647413
Error while processing revid=1191770466, pageid=68717121
Error while processing revid=1046396613, pageid=68813635
Error while processing revid=1046985690, pageid=68835892
Error while processing revid=1083572362, pageid=69293796
Error while processing revid=1056644124, pageid=69338707
Error while processing revid=1064393460, pageid=69346232
Error while processing revid=1058797135, pageid=69439927
Error while processing revid=1059531263, pageid=69474810
Error while processing revid=1059704104, pageid=69482712
Error while processing revid=1131531277, pageid=69530619
Error while processing revid=1073473289, pageid=69629573
Error while processing revid=1131496768, pageid=69649019
Error while processing revid=1072924151, pageid=70118686
Error while processing revid=1075174385, pageid=70214059
Error while processing revid=1083059223, pageid=70554712
Error while processing revid=1100740664, pageid=70834013
Error while processing revid=1136982567, pageid=70975300
Error while processing revid=1100205403, pageid=71357055
Error while processing revid=1142667710, pageid=71361867
Error while processing revid=1192640327, pageid=71366976
Error while processing revid=1100569858, pageid=71391747
Error while processing revid=1115485448, pageid=71504477
Error while processing revid=1140236307, pageid=72204118

(Each of these is surrounded by long blocks of “Processed” and “--start” messages, which I left out here, whereas in the first snippet above you can see that past a second point it started erroring out “permanently”.)

Stashbot added a comment.Feb 23 2024, 11:58 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-23T11:58:25Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint2002:~$ mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --start '["75194261"]' | tee -a ~/T315510-enwiki-2 # in tmux

• Lucas_Werkmeister_WMDE added a comment.Feb 23 2024, 1:20 PM

And as seems to be usual, enwiki is doing lots of work while viwiki isn’t actually updating anything, by the way.

enwiki
Processed 41000 (updated 22136) of 42066285 rows
--start '["75271275"]'
viwiki
Processed 46800 (updated 0) of 19151306 rows
--start '["7986677"]'
• Lucas_Werkmeister_WMDE added a comment.Feb 26 2024, 11:18 AM

enwiki “finished” with the same errors again.

Error while processing revid=1209993227, pageid=76174598
[f9b527c18370a109951fb187] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php on line 91                                                                                                                               
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorStore.php(187): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.19/includes/Revision/RevisionStore.php(1747): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.19/includes/Revision/RevisionStore.php(1610): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)                                                                                                                                                                                         
#4 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(202): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                             
#6 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                               
#7 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#8 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#10 {main}
Error while processing revid=1209993258, pageid=76174599
[f9b527c18370a109951fb187] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php on line 91                                                                                                                               
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorStore.php(187): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.19/includes/Revision/RevisionStore.php(1747): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.19/includes/Revision/RevisionStore.php(1610): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)                                                                                                                                                                                         
#4 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(202): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                             
#6 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                               
#7 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#8 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#10 {main}
Error while processing revid=1209993335, pageid=76174601
[f9b527c18370a109951fb187] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php on line 91                                                                                                                               
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorStore.php(187): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.19/includes/Revision/RevisionStore.php(1747): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.19/includes/Revision/RevisionStore.php(1610): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)                                                                                                                                                                                         
#4 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(202): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                             
#6 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                               
#7 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#8 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#10 {main}
Processed 507375 (updated 246594) of 42066285 rows
Finished in 25 h 50 min 5 s

real    1550m4.771s
user    1070m12.899s
sys     20m28.165s
Sat 24 Feb 2024 01:48:28 PM UTC

And viwiki is rapidly churning through pages and just spamming errors again:

Error while processing revid=61052602, pageid=10254705
[0b0535de3852dcb0b1df256c] [no req]   InvalidArgumentException: No server with index '0'
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/ServerInfo.php(80)
#0 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(985): Wikimedia\Rdbms\ServerInfo->getServerInfoStrict(integer)
#1 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(908): Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(integer, Wikimedia\Rdbms\DatabaseDomain, array)
#2 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(775): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(integer, Wikimedia\Rdbms\DatabaseDomain, integer)                                                                                                                                                                                   
#3 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(767): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#4 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(103): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#5 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(117): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#6 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(338): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(741): Wikimedia\Rdbms\DBConnRef->selectField(array, string, array, string, array, array)
#8 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(552): Wikimedia\Rdbms\SelectQueryBuilder->fetchField()
#9 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(490): MediaWiki\Extension\DiscussionTools\ThreadItemStore::MediaWiki\Extension\DiscussionTools\{closure}(Wikimedia\Rdbms\DBConnRef)                                                                                                                                                              
#10 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(564): MediaWiki\Extension\DiscussionTools\ThreadItemStore->findOrInsertIdButTryHarder(Closure, Closure, boolean, MediaWiki\Revision\RevisionStoreRecord)                                                                                                                                        
#11 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(210): MediaWiki\Extension\DiscussionTools\ThreadItemStore->insertThreadItems(MediaWiki\Revision\RevisionStoreRecord, MediaWiki\Extension\DiscussionTools\ContentThreadItemSet)                                                                                                    
#12 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                            
#13 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                              
#14 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#15 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#16 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#17 {main}
Error while processing revid=60985418, pageid=10254706
[0b0535de3852dcb0b1df256c] [no req]   InvalidArgumentException: No server with index '0'
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/ServerInfo.php(80)
#0 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(985): Wikimedia\Rdbms\ServerInfo->getServerInfoStrict(integer)
#1 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(908): Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(integer, Wikimedia\Rdbms\DatabaseDomain, array)
#2 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(775): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(integer, Wikimedia\Rdbms\DatabaseDomain, integer)                                                                                                                                                                                   
#3 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(767): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#4 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(103): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#5 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(117): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#6 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(338): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(741): Wikimedia\Rdbms\DBConnRef->selectField(array, string, array, string, array, array)
#8 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(552): Wikimedia\Rdbms\SelectQueryBuilder->fetchField()
#9 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(490): MediaWiki\Extension\DiscussionTools\ThreadItemStore::MediaWiki\Extension\DiscussionTools\{closure}(Wikimedia\Rdbms\DBConnRef)                                                                                                                                                              
#10 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(564): MediaWiki\Extension\DiscussionTools\ThreadItemStore->findOrInsertIdButTryHarder(Closure, Closure, boolean, MediaWiki\Revision\RevisionStoreRecord)                                                                                                                                        
#11 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(210): MediaWiki\Extension\DiscussionTools\ThreadItemStore->insertThreadItems(MediaWiki\Revision\RevisionStoreRecord, MediaWiki\Extension\DiscussionTools\ContentThreadItemSet)                                                                                                    
#12 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                            
#13 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                              
#14 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#15 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#16 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#17 {main}
Error while processing revid=62345006, pageid=10254707
[0b0535de3852dcb0b1df256c] [no req]   InvalidArgumentException: No server with index '0'
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/ServerInfo.php(80)
#0 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(985): Wikimedia\Rdbms\ServerInfo->getServerInfoStrict(integer)
#1 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(908): Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(integer, Wikimedia\Rdbms\DatabaseDomain, array)
#2 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(775): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(integer, Wikimedia\Rdbms\DatabaseDomain, integer)                                                                                                                                                                                   
#3 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(767): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#4 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(103): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#5 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(117): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#6 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(338): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#7 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(741): Wikimedia\Rdbms\DBConnRef->selectField(array, string, array, string, array, array)
#8 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(552): Wikimedia\Rdbms\SelectQueryBuilder->fetchField()
#9 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(490): MediaWiki\Extension\DiscussionTools\ThreadItemStore::MediaWiki\Extension\DiscussionTools\{closure}(Wikimedia\Rdbms\DBConnRef)                                                                                                                                                              
#10 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(564): MediaWiki\Extension\DiscussionTools\ThreadItemStore->findOrInsertIdButTryHarder(Closure, Closure, boolean, MediaWiki\Revision\RevisionStoreRecord)                                                                                                                                        
#11 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(210): MediaWiki\Extension\DiscussionTools\ThreadItemStore->insertThreadItems(MediaWiki\Revision\RevisionStoreRecord, MediaWiki\Extension\DiscussionTools\ContentThreadItemSet)                                                                                                    
#12 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)                                                                                                                                                                            
#13 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)                                                                                                                                              
#14 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#15 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#16 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#17 {main}
Stashbot added a comment.Feb 26 2024, 11:20 AM

Mentioned in SAL (#wikimedia-operations) [2024-02-26T11:20:14Z] <Lucas_WMDE> STOP persistRevisionThreadItems on viwiki for T315510 again, tons of errors (didn’t even respond to Ctrl+C so I sudo -u www-data kill’ed it)

• Lucas_Werkmeister_WMDE added a comment.Feb 26 2024, 11:26 AM

@matmarex IMHO both of these errors need to be fixed before we continue running the maintenance scripts on enwiki and viwiki.

For the record, enwiki should probably resume from --start '["76082583"]':

…
Processed 462300 (updated 246456) of 42066285 rows
--start '["76082136"]'
Processed 462400 (updated 246517) of 42066285 rows
--start '["76082364"]'
Processed 462500 (updated 246583) of 42066285 rows
--start '["76082583"]'
Error while processing revid=1206745421, pageid=76082621
Error while processing revid=1209038017, pageid=76082622
Error while processing revid=1208107075, pageid=76082625
Error while processing revid=1206753593, pageid=76082626
…

And viwiki should probably resume from --start '["8661638"]':

…
Processed 720200 (updated 0) of 19151306 rows
--start '["8661438"]'
Processed 720300 (updated 0) of 19151306 rows
--start '["8661538"]'
Processed 720400 (updated 0) of 19151306 rows
--start '["8661638"]'
Error while processing revid=55726296, pageid=8661679
[0b0535de3852dcb0b1df256c] [no req]   Wikimedia\Rdbms\DBConnectionError: Cannot access the database: Connection refused (db2118)
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1123)
#0 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(800): Wikimedia\Rdbms\LoadBalancer->reportConnectionError()
#1 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(767): Wikimedia\Rdbms\LoadBalancer->getServerConnection(integer, string, integer)
#2 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(103): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(integer, array, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(117): Wikimedia\Rdbms\DBConnRef->ensureConnection()
#4 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/database/DBConnRef.php(338): Wikimedia\Rdbms\DBConnRef->__call(string, array)
#5 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/querybuilder/SelectQueryBuilder.php(741): Wikimedia\Rdbms\DBConnRef->selectField(array, string, array, string, array, array)
#6 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(552): Wikimedia\Rdbms\SelectQueryBuilder->fetchField()
#7 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(490): MediaWiki\Extension\DiscussionTools\ThreadItemStore::MediaWiki\Extension\DiscussionTools\{closure}(Wikimedia\Rdbms\DBConnRef)
#8 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/includes/ThreadItemStore.php(564): MediaWiki\Extension\DiscussionTools\ThreadItemStore->findOrInsertIdButTryHarder(Closure, Closure, boolean, MediaWiki\Revision\RevisionStoreRecord)
#9 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(210): MediaWiki\Extension\DiscussionTools\ThreadItemStore->insertThreadItems(MediaWiki\Revision\RevisionStoreRecord, MediaWiki\Extension\DiscussionTools\ContentThreadItemSet)
#10 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(170): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)
#11 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)
#12 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#13 /srv/mediawiki/php-1.42.0-wmf.19/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#14 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#15 {main}
Error while processing revid=60718565, pageid=8661680
[0b0535de3852dcb0b1df256c] [no req]   Wikimedia\Rdbms\DBConnectionError: Cannot access the database: Connection refused (db2118)
Backtrace:
• Lucas_Werkmeister_WMDE added a comment.Feb 26 2024, 11:27 AM

(And note to self, if I continue this by digging up my old commands and running them again: remember to add 2>&1 to the enwiki command and update the output file there, and also update the output file for viwiki, i.e. T315510-enwiki-3 and T315510-viwiki-2.)

• Lucas_Werkmeister_WMDE mentioned this in T341553: Allow running one-off scripts manually.Mar 1 2024, 1:43 PM
• Lucas_Werkmeister_WMDE added a comment.Mar 6 2024, 2:45 PM
In T315510#9575955, @Lucas_Werkmeister_WMDE wrote:
Error while processing revid=1209993227, pageid=76174598
[f9b527c18370a109951fb187] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php on line 91                                                                                                                               
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.19/includes/user/ActorStore.php(187): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.19/includes/Revision/RevisionStore.php(1747): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.19/includes/Revision/RevisionStore.php(1610): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)                                                                                                                                                                                         
#4 /srv/mediawiki/php-1.42.0-wmf.19/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(202): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)

Looking at the code, I think this might be enough to fix that error:

diff --git a/maintenance/persistRevisionThreadItems.php b/maintenance/persistRevisionThreadItems.php
index 6187371539..337c3f073e 100644
--- a/maintenance/persistRevisionThreadItems.php
+++ b/maintenance/persistRevisionThreadItems.php
@@ -54,7 +54,7 @@ public function execute() {
 
 		$qb = $dbr->newSelectQueryBuilder();
 
-		$qb->queryInfo( $this->revStore->getQueryInfo( [ 'page' ] ) );
+		$qb->queryInfo( $this->revStore->getQueryInfo( [ 'page', 'user' ] ) );
 
 		if ( $this->getOption( 'all' ) ) {
 			// Do nothing

Though I have no idea why this would only have caused errors sometimes and not all the time. @matmarex what do you think?

Error while processing revid=61052602, pageid=10254705
[0b0535de3852dcb0b1df256c] [no req]   InvalidArgumentException: No server with index '0'
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/ServerInfo.php(80)
#0 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(985): Wikimedia\Rdbms\ServerInfo->getServerInfoStrict(integer)
#1 /srv/mediawiki/php-1.42.0-wmf.19/includes/libs/rdbms/loadbalancer/LoadBalancer.php(908): Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(integer, Wikimedia\Rdbms\DatabaseDomain, array)

And this feels like it might be the same error as T357492, but maybe not – I’m not sure.

Stashbot added a comment.Mar 6 2024, 3:22 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-06T15:22:58Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint2002:~$ mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki viwiki --current --all --touched-after=20230613000000 --start '["8661638"]' 2>&1 | tee ~/T315510-viwiki-2 # in tmux

• Lucas_Werkmeister_WMDE added a comment.Mar 6 2024, 3:24 PM

Trying viwiki again, to see if the “no server with index” errors went away; so far it looks okay (though it hasn’t actually updated anything yet, and the previous stack traces ended in insertThreadItems(), so maybe the problematic code hasn’t been hit yet):

lucaswerkmeister-wmde@mwmaint2002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki viwiki --current --all --touched-after=20230613000000 --start '["8661638"]' 2>&1 | tee ~/T315510-viwiki-2; date                                                                                                                                              
Processing... (estimated 19417998 rows)
--start '["8661638"]'
Processed 100 (updated 0) of 19417998 rows
--start '["8661739"]'
Processed 200 (updated 0) of 19417998 rows
--start '["8661839"]'
Processed 300 (updated 0) of 19417998 rows
--start '["8661939"]'
Processed 400 (updated 0) of 19417998 rows
--start '["8662039"]'
Processed 500 (updated 0) of 19417998 rows
--start '["8662139"]'
• Lucas_Werkmeister_WMDE added a comment.Mar 12 2024, 10:57 AM

viwiki is still running and successfully updated one page, apparently:

…
Processed 4276000 (updated 1) of 19417998 rows
--start '["12964404"]'
Processed 4276100 (updated 1) of 19417998 rows
--start '["12964504"]'
Processed 4276200 (updated 1) of 19417998 rows
--start '["12964605"]'
…

(enwiki is still blocked on the “missing actor” problem.)

Stashbot added a comment.Mar 14 2024, 3:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-14T15:28:21Z] <Lucas_WMDE> STOP persistRevisionThreadItems on viwiki for T315510, restarting to pick up wmf.22

Stashbot added a comment.Mar 14 2024, 3:29 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-14T15:29:56Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint2002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki viwiki --current --all --touched-after=20230613000000 --start '["14615874"]' 2>&1 | tee ~/T315510-viwiki-3; date

• Lucas_Werkmeister_WMDE added a comment.Mar 14 2024, 3:29 PM
Processed 5810400 (updated 11) of 19417998 rows
--start '["14615766"]'
Processed 5810500 (updated 11) of 19417998 rows
--start '["14615874"]'
^C

real    11525m21.820s
user    6512m18.717s
sys     119m36.170s
lucaswerkmeister-wmde@mwmaint2002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki viwiki --current --all --touched-after=20230613000000 --start '["14615874"]' 2>&1 | tee ~/T315510-viwiki-3; date                                                                                                                                             
Processing... (estimated 19443691 rows)
--start '["14615874"]'
Processed 100 (updated 0) of 19443691 rows
--start '["14615996"]'
Processed 200 (updated 0) of 19443691 rows
--start '["14616123"]'
Stashbot added a comment.Mar 20 2024, 10:03 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T10:03:04Z] <Lucas_WMDE> STOP persistRevisionThreadItems on viwiki for T315510, will restart after DC switch is done (resume at: --start '["17099868"]')

• Lucas_Werkmeister_WMDE added a comment.Mar 20 2024, 10:03 AM
Processed 2362900 (updated 1) of 19443691 rows
--start '["17099668"]'
Processed 2363000 (updated 1) of 19443691 rows
--start '["17099768"]'
Processed 2363100 (updated 1) of 19443691 rows
--start '["17099868"]'
^C

real    8313m40.131s
user    6258m55.370s
sys     56m37.000s
Stashbot added a comment.Mar 20 2024, 2:59 PM

Mentioned in SAL (#wikimedia-operations) [2024-03-20T14:59:54Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki viwiki --current --all --touched-after=20230613000000 --start '["17099868"]' 2>&1 | tee ~/T315510-viwiki-4; date # in tmux; note the changed mwmaint host :)

• Lucas_Werkmeister_WMDE added a comment.Mar 28 2024, 11:28 AM

viwiki finished on Wed 27 Mar 2024 07:41:29 AM UTC:

Processed 2245600 (updated 48) of 19042077 rows
--start '["19860795"]'
Processed 2245700 (updated 51) of 19042077 rows
--start '["19861218"]'
Processed 2245701 (updated 51) of 19042077 rows
Finished in 6 d 16 h 41 min 36 s

That leaves only enwiki; I’ve started it again and so far it seems to work and do stuff, but who knows whether the actor problem was really fixed or not.

lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --start '["76082583"]' 2>&1 | tee -a ~/T315510-enwiki-4; date
Processing... (estimated 42067094 rows)
--start '["76082583"]'
Processed 100 (updated 54) of 42067094 rows
--start '["76082783"]'
Processed 200 (updated 119) of 42067094 rows
--start '["76082974"]'
Processed 300 (updated 202) of 42067094 rows
--start '["76083136"]'
Processed 400 (updated 282) of 42067094 rows
--start '["76083306"]'
Processed 500 (updated 290) of 42067094 rows
--start '["76083415"]'
Stashbot added a comment.Mar 28 2024, 11:31 AM

Mentioned in SAL (#wikimedia-operations) [2024-03-28T11:31:00Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --start '["76082583"]' 2>&1 | tee -a ~/T315510-enwiki-4; date

• Lucas_Werkmeister_WMDE changed the task status from Open to Stalled.Apr 2 2024, 8:56 AM

Nope, the enwiki run crashed again, starting at --start '["76318767"]'.

Processed 128800 (updated 57577) of 42067094 rows
--start '["76318424"]'
Processed 128900 (updated 57594) of 42067094 rows
--start '["76318570"]'
Processed 129000 (updated 57653) of 42067094 rows
--start '["76318767"]'
Error while processing revid=1215257050, pageid=76318825
[ed6f868f7e414a24f63adf3d] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php on line 91
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorStore.php(131): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorStore.php(395): MediaWiki\User\ActorStore->newActorFromRow(stdClass)
#3 /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorStore.php(354): MediaWiki\User\ActorStore->findActorIdInternal(string, Wikimedia\Rdbms\DBConnRef)
#4 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/includes/ThreadItemStore.php(575): MediaWiki\User\ActorStore->findActorIdByName(string, Wikimedia\Rdbms\DBConnRef)
#5 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/includes/ThreadItemStore.php(474): MediaWiki\Extension\DiscussionTools\ThreadItemStore->MediaWiki\Extension\DiscussionTools\{closure}(Wikimedia\Rdbms\DBConnRef)
#6 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/includes/ThreadItemStore.php(584): MediaWiki\Extension\DiscussionTools\ThreadItemStore->findOrInsertId(Closure, Closure, boolean, MediaWiki\Revision\RevisionStoreRecord)
#7 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(206): MediaWiki\Extension\DiscussionTools\ThreadItemStore->insertThreadItems(MediaWiki\Revision\RevisionStoreRecord, MediaWiki\Extension\DiscussionTools\ContentThreadItemSet)
#8 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(166): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)
#9 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)
#10 /srv/mediawiki/php-1.42.0-wmf.23/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#11 /srv/mediawiki/php-1.42.0-wmf.23/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#12 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#13 {main}
Error while processing revid=1212878488, pageid=76318828
[ed6f868f7e414a24f63adf3d] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php on line 91
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorStore.php(187): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.23/includes/Revision/RevisionStore.php(1750): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.23/includes/Revision/RevisionStore.php(1613): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)
#4 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(198): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(166): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)
#6 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)
#7 /srv/mediawiki/php-1.42.0-wmf.23/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#8 /srv/mediawiki/php-1.42.0-wmf.23/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#10 {main}
Error while processing revid=1212878586, pageid=76318832
[ed6f868f7e414a24f63adf3d] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php on line 91
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorStore.php(187): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.23/includes/Revision/RevisionStore.php(1750): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.23/includes/Revision/RevisionStore.php(1613): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)
#4 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(198): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(166): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)
#6 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)
#7 /srv/mediawiki/php-1.42.0-wmf.23/maintenance/includes/MaintenanceRunner.php(698): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#8 /srv/mediawiki/php-1.42.0-wmf.23/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#10 {main}
Error while processing revid=1212878623, pageid=76318834
[ed6f868f7e414a24f63adf3d] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php on line 91
Backtrace:
from /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.42.0-wmf.23/includes/user/ActorStore.php(187): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.42.0-wmf.23/includes/Revision/RevisionStore.php(1750): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.42.0-wmf.23/includes/Revision/RevisionStore.php(1613): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)
#4 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(198): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(166): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)
#6 /srv/mediawiki/php-1.42.0-wmf.23/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)
…

@matmarex can you please confirm whether you still need the script to be run on enwiki, and are going to look into those errors? I haven’t heard anything from you since T315510#9550751.

Esanders added a comment.Apr 2 2024, 10:14 PM

We do need the script to be run on en.wiki.

• matmarex added a comment.Apr 4 2024, 2:43 AM

Yes, and I'm going to look into them soon. I'm busy with some other projects, so I only spend maybe a few hours every few weeks checking on this one, but I still want to finish this one day :) Thank you so much for re-running all these scripts.

• matmarex updated the task description. (Show Details)Apr 4 2024, 2:43 AM
• Lucas_Werkmeister_WMDE added a comment.Apr 4 2024, 3:27 PM

Alright, thanks! I’ll just bump this older comment since there’s been a lot of backscroll since then:

In T315510#9607252, @Lucas_Werkmeister_WMDE wrote:

Looking at the code, I think this might be enough to fix that error:

diff --git a/maintenance/persistRevisionThreadItems.php b/maintenance/persistRevisionThreadItems.php
index 6187371539..337c3f073e 100644
--- a/maintenance/persistRevisionThreadItems.php
+++ b/maintenance/persistRevisionThreadItems.php
@@ -54,7 +54,7 @@ public function execute() {
 
 		$qb = $dbr->newSelectQueryBuilder();
 
-		$qb->queryInfo( $this->revStore->getQueryInfo( [ 'page' ] ) );
+		$qb->queryInfo( $this->revStore->getQueryInfo( [ 'page', 'user' ] ) );
 
 		if ( $this->getOption( 'all' ) ) {
 			// Do nothing

I wanted to try this out on mwdebug now, but the last --start from the last run (T315510#9679051, --start '["76318767"]') no longer reproduces the error, unfortunately. Probably the error would still happen at some later point, but that’s not as helpful for debugging. Hopefully you can reproduce the error locally 😅 otherwise I can run the script again and see at what point it starts crashing.

• Lucas_Werkmeister_WMDE added a comment.May 16 2024, 2:08 PM

I’ll try another run; if it passes, great, if it crashes, I can try to apply the above patch on mwdebug again.

Stashbot added a comment.May 16 2024, 2:09 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-16T14:09:19Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --start '["76318767"]' 2>&1 | tee -a ~/T315510-enwiki-5; date

• Lucas_Werkmeister_WMDE added a comment.May 16 2024, 3:39 PM

Hasn’t crashed yet, I’ll look again tomorrow.

Also, I don’t think I realized this last time – --start '["76318767"]' means it’s almost done? The maximum page ID is 76918527, just ~half a million more.

Stashbot added a comment.May 22 2024, 8:39 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-22T20:39:36Z] <Lucas_WMDE> STOPPED lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --start '["76318767"]' 2>&1 | tee -a ~/T315510-enwiki-5; date # ca. 1 hour and 20 minutes ago, after running for a bit over 6 days; some errors

• Lucas_Werkmeister_WMDE added a comment.May 22 2024, 8:42 PM

Well, it made it quite far before encountering the errors again:

Processed 309000 (updated 123619) of 62237982 rows
--start '["76882243"]'
Processed 309100 (updated 123657) of 62237982 rows
--start '["76882470"]'
Processed 309200 (updated 123698) of 62237982 rows
--start '["76882704"]'
Error while processing revid=1225112011, pageid=76882891
[e3b8484f2d4933b27ebac269] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.43.0-wmf.5/includes/user/ActorCache.php on line 91
Backtrace:
from /srv/mediawiki/php-1.43.0-wmf.5/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.43.0-wmf.5/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.43.0-wmf.5/includes/user/ActorStore.php(186): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.43.0-wmf.5/includes/Revision/RevisionStore.php(1756): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.43.0-wmf.5/includes/Revision/RevisionStore.php(1619): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)
#4 /srv/mediawiki/php-1.43.0-wmf.5/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(198): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.43.0-wmf.5/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(166): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)
#6 /srv/mediawiki/php-1.43.0-wmf.5/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)
#7 /srv/mediawiki/php-1.43.0-wmf.5/maintenance/includes/MaintenanceRunner.php(696): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#8 /srv/mediawiki/php-1.43.0-wmf.5/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#10 {main}
Error while processing revid=1223319828, pageid=76882892
[e3b8484f2d4933b27ebac269] [no req]   TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given, called in /srv/mediawiki/php-1.43.0-wmf.5/includes/user/ActorCache.php on line 91
Backtrace:
from /srv/mediawiki/php-1.43.0-wmf.5/includes/user/ActorCache.php(106)
#0 /srv/mediawiki/php-1.43.0-wmf.5/includes/user/ActorCache.php(91): MediaWiki\User\ActorCache->remove(NULL)
#1 /srv/mediawiki/php-1.43.0-wmf.5/includes/user/ActorStore.php(186): MediaWiki\User\ActorCache->add(integer, MediaWiki\User\UserIdentityValue)
#2 /srv/mediawiki/php-1.43.0-wmf.5/includes/Revision/RevisionStore.php(1756): MediaWiki\User\ActorStore->newActorFromRowFields(integer, string, integer)
#3 /srv/mediawiki/php-1.43.0-wmf.5/includes/Revision/RevisionStore.php(1619): MediaWiki\Revision\RevisionStore->newRevisionFromRowAndSlots(stdClass, NULL, integer, MediaWiki\Title\Title, boolean)
#4 /srv/mediawiki/php-1.43.0-wmf.5/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(198): MediaWiki\Revision\RevisionStore->newRevisionFromRow(stdClass)
#5 /srv/mediawiki/php-1.43.0-wmf.5/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(166): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->processRow(stdClass)
#6 /srv/mediawiki/php-1.43.0-wmf.5/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Wikimedia\Rdbms\SelectQueryBuilder, array)
#7 /srv/mediawiki/php-1.43.0-wmf.5/maintenance/includes/MaintenanceRunner.php(696): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
#8 /srv/mediawiki/php-1.43.0-wmf.5/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
#9 /srv/mediawiki/multiversion/MWScript.php(158): require_once(string)
#10 {main}

The MAX(page_id) on enwiki is currently 76991852, so --start '["76882704"]' is just over a hundred thousand away from the end.

Stashbot added a comment.May 24 2024, 1:57 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-24T13:57:04Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --start '["76882704"]' 2>&1 | tee -a ~/T315510-enwiki-6; date

• Lucas_Werkmeister_WMDE added a comment.May 24 2024, 1:57 PM

Trying another run, and if that crashes again, I’ll try the patch on mwdebug again. (But so far it’s working – processed 400, updated 210.)

• Lucas_Werkmeister_WMDE added a comment.May 24 2024, 2:24 PM

Also: at this point the maintenance script is working through pages that were only created less than two weeks ago (e.g. curid=76882704); yet it still has something to do (currently processed 14800, updated 8954). Is this because one of the relevant config settings is not turned on yet on enwiki (perhaps wgDiscussionToolsEnablePermalinksBackend)? Are we going to need one more “cleanup” maintenance script run for the last few enwiki pages after the relevant setting has been turned on?

• Lucas_Werkmeister_WMDE changed the task status from Stalled to Open.May 24 2024, 2:28 PM
Stashbot added a comment.May 24 2024, 3:18 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-24T15:18:11Z] <Lucas_WMDE> FINISHED lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --start '["76882704"]' 2>&1 | tee -a ~/T315510-enwiki-6; date # a few minutes ago

• Lucas_Werkmeister_WMDE added a comment.May 24 2024, 3:19 PM

enwiki should be done:

Processed 42000 (updated 24353) of 60772932 rows
--start '["77003057"]'
Processed 42100 (updated 24401) of 60772932 rows
--start '["77003232"]'
Processed 42200 (updated 24462) of 60772932 rows
--start '["77003429"]'
Processed 42300 (updated 24508) of 60772932 rows
--start '["77003634"]'
Processed 42400 (updated 24536) of 60772932 rows
--start '["77003825"]'
Processed 42462 (updated 24554) of 60772932 rows
Finished in 1 h 18 min 39 s

real    78m39.353s
user    51m17.684s
sys     1m15.778s
Fri 24 May 2024 03:15:41 PM UTC

(Except, potentially, for new pages created now until the config setting is turned on? So we might need one more run with --start '["77003825"]'? No idea.)

• matmarex mentioned this in T316024: PHP Notice: Undefined offset: ... (from ActorCache.php).May 26 2024, 4:09 AM

@Lucas_Werkmeister_WMDE Thank you for persevering with this after I lost patience. I'm really happy to hear that the scripts actually finished!

Yes, we'll need a config change similar to https://gerrit.wikimedia.org/r/c/operations/mediawiki-config/+/965562 (I can do that on Monday) and one last script run afterwards.

I also had a look at all of the error messages you posted:

  1. T315510#9562881 TypeError: Argument 1 passed to ExternalStoreDB::getDomainId() must be of the type array, bool given
  2. T315510#9562881 TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given
  3. T315510#9571716 TypeError: Argument 1 passed to ExternalStoreDB::getDomainId() must be of the type array, bool given
  4. T315510#9575955 TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given
  5. T315510#9575955 InvalidArgumentException: No server with index '0'
  6. T315510#9576002 Wikimedia\Rdbms\DBConnectionError: Cannot access the database: Connection refused (db2118)
  7. T315510#9679051 TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given
  8. T315510#9823357 TypeError: Argument 1 passed to MediaWiki\User\ActorCache::remove() must implement interface MediaWiki\User\UserIdentity, null given

(1, 3) ExternalStoreDB::getDomainId() are the same issue as already filed T357492: Auto-reconfigure external storage database replicas in long-running maintenance scripts (we ran into it earlier in this task).

(2, 4, 7, 8) MediaWiki\User\ActorCache::remove() look similar to T316024: PHP Notice: Undefined offset: ... (from ActorCache.php) – I found entries in Logstash for PHP Notice: Undefined offset: ... preceding these errors. This one seems interesting. I posted more details in that task.

(5) No server with index '0' seems like maybe some database configuration has changed while the script was running. I found T322156: "No server with index" and "Warning: Undefined index" from LoadBalancer::reconfigure, marked as resolved a year ago, with the same error message and similar backtrace. Maybe there's some edge case that the changes there didn't fix. It doesn't seem important, it's just a reason not to do long-running maintenance scripts in the future.

(6) Cannot access the database: Connection refused seems like a temporary database server problem,

Trizek-WMF subscribed.May 27 2024, 9:55 AM

Thank you so much, @Lucas_Werkmeister_WMDE !

Trizek-WMF mentioned this in T365974: Deploy talk page permalinks to en.wiki.May 27 2024, 9:56 AM
Stashbot added a comment.May 27 2024, 2:14 PM

Mentioned in SAL (#wikimedia-operations) [2024-05-27T14:14:11Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --touched-after=20240524120000 2>&1 | tee -a ~/T315510-enwiki-7; date # cc T365974

• Lucas_Werkmeister_WMDE added a comment.May 27 2024, 2:31 PM

The latest run (modeled after T315353#9078672, i.e. with --touched-after rather than --start) is still doing something:

lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --touched-after=20240524120000 2>&1 | tee -a ~/T315510-enwiki-7; date # cc T365974                                                                                                                                                    
Processing... (estimated 61401202 rows)
Processed 100 (updated 94) of 61401202 rows
--start '["3061"]'
Processed 200 (updated 192) of 61401202 rows
--start '["5279"]'
Processed 300 (updated 287) of 61401202 rows
--start '["7290"]'
Processed 400 (updated 381) of 61401202 rows
--start '["9641"]'
Processed 500 (updated 479) of 61401202 rows
--start '["11677"]'
Processed 600 (updated 578) of 61401202 rows
--start '["13759"]'
Processed 700 (updated 676) of 61401202 rows
--start '["16501"]'
Processed 800 (updated 772) of 61401202 rows
--start '["19706"]'
Processed 900 (updated 870) of 61401202 rows
--start '["22311"]'
Processed 1000 (updated 964) of 61401202 rows
--start '["26251"]'

If I understand it correctly, the “correct” total number of rows to process is ca. 4 million (rather than 61 million):

lucaswerkmeister-wmde@stat1011:~$ analytics-mysql enwiki <<< "SELECT COUNT(*) FROM page WHERE page_touched > '20240524120000'\G"
*************************** 1. row ***************************
COUNT(*): 3957142

So by a rough estimate, it will take a bit more than a week to finish. Unfortunately, I’m on holiday from Thursday (E1752), so I won’t be able to report when the maintenance script stops or errors out (unless I hand it over to someone else to run) :/

• Lucas_Werkmeister_WMDE added a comment.EditedMay 27 2024, 3:18 PM

After one hour: Processed 20200 (updated 16783) of 61401202 rows and --start '["920309"]'.

Assuming that the ~four million pages are correct, and 20200 pages were processed in one hour, yields an estimate of just over 8 days:

$ units '3957142 / (20200 / 1 hour)' time
        8 day + 3 hr + 53 min + 53.227723 sec

Whereas assuming that the pages are equally distributed, and --start '["920309"]' represents page ID ~900k out of 77025179 (analytics-mysql enwiki <<< 'SELECT MAX(page_id) FROM page;'), yields an estimate of 3½ days:

$ units '1 hour * (77025179 / 920309)' time
        3 day + 11 hr + 41 min + 41.676285 sec

Remains to be seen which one is more accurate, I guess… though even in the latter case it’s not expected to finish before I’m off.

• Lucas_Werkmeister_WMDE added a comment.May 29 2024, 2:35 PM

FWIW, while I’m on holiday, any deployer should be able to check whether the script is still running with something like ps -ef | grep [p]ersistRevisionThreadItems, and look at the output with less -n +G ~lucaswerkmeister-wmde/T315510-enwiki-7, on mwmaint1002. (Based on the latest output, it might not even finish before I’m back: given “Processed 611000 (updated 153655) of 61401202 rows” and “--start '["38392856"]'” after 2 days + 16 minutes, the above calculations now project 13 days or 4 days, respectively, and I’m inclined to think the former projection is more likely to be accurate: the latter assumes that pages are equally distributed, but in truth I suppose newer pages with higher page IDs are more likely to have been touched recently.)

ppelberg moved this task from Doing to Ready for Sign Off on the Editing-team (Kanban Board) board.Jun 3 2024, 5:28 PM

@Lucas_Werkmeister_WMDE and @matmarex could one of you all please do the honors of marking this task as Resolved?

..."great work" from the entire Editing Team ☀️🎉🔗

• matmarex mentioned this in T366857: InvalidArgumentException from line 80 of ServerInfo.php: No server with index '0' (in a maintenance script).Jun 6 2024, 10:55 PM
In T315510#9842466, @Lucas_Werkmeister_WMDE wrote:

FWIW, while I’m on holiday, any deployer should be able to check whether the script is still running with something like ps -ef | grep [p]ersistRevisionThreadItems, and look at the output with less -n +G ~lucaswerkmeister-wmde/T315510-enwiki-7, on mwmaint1002.

It crashed:

P64214 (An Untitled Masterwork)
1--start '["55019880"]'
2Processed 1464300 (updated 177433) of 61401202 rows
3InvalidArgumentException from line 80 of /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/ServerInfo.php: No server with index '0'
4#0 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(1001): Wikimedia\Rdbms\ServerInfo->getServerInfoStrict(0)
5#1 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(924): Wikimedia\Rdbms\LoadBalancer->reallyOpenConnection(0, Object(Wikimedia\Rdbms\DatabaseDomain), Array)
6#2 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(801): Wikimedia\Rdbms\LoadBalancer->reuseOrOpenConnectionForNewRef(0, Object(Wikimedia\Rdbms\DatabaseDomain), 0)
7#3 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/loadbalancer/LoadBalancer.php(793): Wikimedia\Rdbms\LoadBalancer->getServerConnection(0, 'enwiki', 0)
8#4 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/database/DBConnRef.php(110): Wikimedia\Rdbms\LoadBalancer->getConnectionInternal(-1, Array, 'enwiki', 0)
9#5 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/database/DBConnRef.php(124): Wikimedia\Rdbms\DBConnRef->ensureConnection()
10#6 /srv/mediawiki/php-1.43.0-wmf.6/includes/libs/rdbms/database/DBConnRef.php(420): Wikimedia\Rdbms\DBConnRef->__call('buildComparison', Array)
11#7 /srv/mediawiki/php-1.43.0-wmf.6/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(157): Wikimedia\Rdbms\DBConnRef->buildComparison('>', Array)
12#8 /srv/mediawiki/php-1.43.0-wmf.6/extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php(120): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->process(Object(Wikimedia\Rdbms\SelectQueryBuilder), Array)
13#9 /srv/mediawiki/php-1.43.0-wmf.6/maintenance/includes/MaintenanceRunner.php(696): MediaWiki\Extension\DiscussionTools\Maintenance\PersistRevisionThreadItems->execute()
14#10 /srv/mediawiki/php-1.43.0-wmf.6/maintenance/run.php(51): MediaWiki\Maintenance\MaintenanceRunner->run()
15#11 /srv/mediawiki/multiversion/MWScript.php(158): require_once('/srv/mediawiki/...')
16#12 {main}

Since this is the second time with this error, I filed a task for it: T366857.

There's no timestamp, but judging by logstash entries from the script, it stopped on 2024-06-04: https://logstash.wikimedia.org/goto/634c56e259cef1187f08788a75454c7f

--start '["55019880"]' means that it was mostly done (assuming that 77025179 is the max). We could restart it when you're back, or maybe just leave it that way and let those pages' data get updated when they are touched again. Either way, this doesn't block the enwiki deployment.

• Lucas_Werkmeister_WMDE added a comment.Jun 11 2024, 8:38 AM
In T315510#9869549, @matmarex wrote:

There's no timestamp, but judging by logstash entries from the script, it stopped on 2024-06-04: https://logstash.wikimedia.org/goto/634c56e259cef1187f08788a75454c7f

stat ~/T315510-enwiki-7 agrees (Modify: 2024-06-04 06:21:00.884299307 +0000).

--start '["55019880"]' means that it was mostly done (assuming that 77025179 is the max). We could restart it when you're back, or maybe just leave it that way and let those pages' data get updated when they are touched again. Either way, this doesn't block the enwiki deployment.

Rerunning it again now (though I agree it doesn’t block deployment anyway).

Stashbot added a comment.Jun 11 2024, 8:38 AM

Mentioned in SAL (#wikimedia-operations) [2024-06-11T08:38:53Z] <Lucas_WMDE> lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --touched-after=20240524120000 --start '["55019880"]' 2>&1 | tee -a ~/T315510-enwiki-8; date

• Lucas_Werkmeister_WMDE added a comment.Jun 11 2024, 11:02 AM

or maybe just leave it that way and let those pages' data get updated when they are touched again.

For the record, the script is still doing a little bit but most pages apparently got touched already:

Processed 20500 (updated 324) of 42067094 rows
--start '["55070603"]'
Stashbot added a comment.Jun 12 2024, 9:04 AM

Mentioned in SAL (#wikimedia-operations) [2024-06-12T09:04:14Z] <Lucas_WMDE> STOPPED lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --touched-after=20240524120000 --start '["55019880"]' 2>&1 | tee -a ~/T315510-enwiki-8; date # Ctrl+C, had become very slow, trying restart

Stashbot added a comment.Jun 12 2024, 9:04 AM

Mentioned in SAL (#wikimedia-operations) [2024-06-12T09:04:51Z] <Lucas_WMDE> START lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --touched-after=20240524120000 --start '["55386869"]' 2>&1 | tee -a ~/T315510-enwiki-9; date

• Lucas_Werkmeister_WMDE added a comment.Jun 12 2024, 9:07 AM

Hm, only ~300k pages processed in ~24 hours seems quite slow indeed… not sure the restart was justified, then. (I thought the script had progressed from 500… to 550… over the course of yesterday and then got stuck from 550… to 553… overnight, but clearly I misremembered – according to the messages above, we started at 550… yesterday already.)

• Lucas_Werkmeister_WMDE added a comment.Jun 12 2024, 9:08 AM

(But the 100-page batches do seem to be running much faster now than before the restart, so maybe it helped after all.)

• Lucas_Werkmeister_WMDE added a comment.Jun 13 2024, 2:35 PM

The slowdown still seems to be happening – earlier this morning, each 100-page batch seemed to take about four minutes.

or maybe just leave it that way and let those pages' data get updated when they are touched again.

I’m tempted to come back to this suggestion after all – the slowdown probably isn’t worth investigating. What do you think?

• matmarex added a comment.Jun 13 2024, 3:13 PM

I concur, it seems like unnecessary work at this point.

Stashbot added a comment.Jun 13 2024, 3:28 PM

Mentioned in SAL (#wikimedia-operations) [2024-06-13T15:28:13Z] <Lucas_WMDE> STOPPED lucaswerkmeister-wmde@mwmaint1002:~$ time mwscript extensions/DiscussionTools/maintenance/persistRevisionThreadItems.php --wiki enwiki --current --all --touched-after=20240524120000 --start '["55386869"]' 2>&1 | tee -a ~/T315510-enwiki-9; date # Ctrl+C – had slowed down, unnecessary work by this point; was at --start '["55914913"]'

• Lucas_Werkmeister_WMDE closed this task as Resolved.Jun 13 2024, 3:29 PM

Alright, then I’m going to declare this… DONE.

VPuffetMichel awarded a token.Jun 13 2024, 8:14 PM
Content licensed under Creative Commons Attribution-ShareAlike (CC BY-SA) 4.0 unless otherwise noted; code licensed under GNU General Public License (GPL) 2.0 or later and other open source licenses. By using this site, you agree to the Terms of Use, Privacy Policy, and Code of Conduct. · Wikimedia Foundation · Privacy Policy · Code of Conduct · Terms of Use · Disclaimer · CC-BY-SA · GPL · Credits