For completeness, another option is the varnish "x-key" system, which involves two research projects. One is that implementation of x-key in varnish appears to be incomplete, and the second is that the assignment of appropriate x-keys to URLs is non-trivial as well. There are too many templates used on a page like [[Barack Obama]] to naively assign one x-key to every recursively-included template, so we still need to come up with a mechanism to determine which of the templates deserve an x-key assigned, likely based on purge statistics.

Another option is to subdivide pages into two categories, "high traffic pages" and "long tail low traffic pages". The latter would be put effectively into a no-cache state: the cache lifetime would be very short, and we would never emit purges for them, relying on the natural expiration to deal with vandalism. We'd only emit purges for the high traffic pages.

I like the <figure> inside <figure> option that @lzno proposed makes sense. Switching from inline to block styles makes the semantics clearer for accessibility, in terms of associating the caption with the image.

We're going to work on T369454. The other issue related to whether/how linter recurses into extension-generated content I'm going to treat as a dup/subcase of T331655: Support `outputHasCoreMwDomSpecMarkup='mixed'` in Parsoid extension registration. Right now, it is desired behavior because it is catching a legitimate issue, so Not A Bug.

This is a won't fix, I believe, although I'm going to punt this to @ssastry for comment when he returns from sabbatical. We shouldn't really be emitting fallback text at all, and to the extent that we do it is desirable that it is consistent for machine-readability of citation markup. No humans should see that.

StatsFactory in particular is the thing we want to use, and it ends up depending on all the metric classes implicitly through its method signatures.

Could we change both Parsoid and the legacy parser to agree that === === is not a valid heading? That would avoid the whole "blank ID" issue.

For what it's worth, the new OutputTransform pipeline will (hopefully) maintain the document in DOM form throughout the transformation pipeline, eliminating the repeated HTML parsing steps. That's T347062: Create HtmlHolder interface.

We have a few issues with regression testing:

• Out of disk space. I ran the script in https://wikitech.wikimedia.org/wiki/Parsoid/Common_Tasks#Freeing_disk_space but it failed halfway through (just after "Dumping results table") with a permissions error trying to write results.sql.gz. I was in my homedir and i'm pretty sure the script was running as my user so I don't know what that was about. At any rate, it had freed up enough space by that time to successfully start rt testing, so I didn't investigate further.
• Between 7:30am-8am EST on Jun 13 we had a server issue, resulting in a large number of the following in the logs:

Zmqsmfn7_ueX-TT4hLc5ZwAAAII] /w/rest.php/fr.wikipedia.org/v3/page/wikitext/Badis_badis   Wikimedia\Rdbms\DBUnexpectedError: Database servers in cluster26 are overloaded. In order to protect application servers, the circuit breaking to databases of this section have been activated. Please try again a few seconds.

This resulted in a large number of 500 failures. With @Arlolra's advice I used the instructions in https://www.mediawiki.org/wiki/Parsing/Visual_Diff_Testing#Retesting_a_subset_of_titles to purge the failing files:

$ mysql -u testreduce -p testreduce
>update pages set claim_hash="",claim_num_tries=0, claim_timestamp=null,latest_stat=null,latest_result=null,latest_score=0,num_fetch_errors=0 where latest_score=1000000;

which (a) affected 166 rows, and (b) resulted in a huge spike in load average on the machine, as apparently all of those pages got requeued at once:

Top - 17:42:05 up 134 days,  7:20,  1 user,  load average: 102.97, 76.62, 43.64
Tasks: 148 total,  38 running, 110 sleeping,   0 stopped,   0 zombie

But eventually things settled down, although some of those pages failed with timeouts again and I had to repeat the sql command to requeue a handful of them for a third time.

This seems related to the idea of "page data contexts" which has been floating around (eg T122934#9196348), but as a sorted inverted case -- instead of "an article page have a data context attached", this proposal is closer to "a data page with a presentation context attached". I think both are interesting concepts that I'd like to explore, but we don't have anything similar on our current roadmap. @daniel's proposal is workable and is very similar to the "article page with a data context attached", in that if you could associate a scribunto module with a Title, you could execute that scribunto module to generate the 'data context'. But here we want to execute the scribunto module to generate the article page... generating wikitext from lua is something i'm not a huge fan off. See [[Extension:ArrayFunctions]], which would be a way to generate the wikitext from tabular data.

Would be ideal if legacy could match Parsoid...

With the resolution of T365483: Links from mobile go back to desktop host when using Parsoid Read Views this bug has been resolved for Parsoid read views, which means it will eventually get resolved on all wikis as we phase out legacy read views.

We deprecated the 'object' parameter type used by this in https://gerrit.wikimedia.org/r/c/mediawiki/core/+/1036758 as it was unused in practice.

Linking bugs up: the original issue was T360865: Slow query in Special:LintErrors. Let's just make sure we're not regressing that bug.

T359002: Lead paragraph is not hoisted in new Parsoid HTML is currently stalling further visualdiff testing on the CTT side.

Addressing this issue is high-ish priority for CTT because we find that inconsistent treatment of the lead paragraph between legacy and Parsoid tends to have knock-on effects with positioning that then throw off visualdiff results all the way down the page. So basically we find it hard to discover any /other/ issues with the MFE+Parsoid rendering because the noise from the lead section drowns out everything else.

Seems reasonable. Be careful not to put any HTML block-level content (<p> or <div> tags, eg, which could get hidden inside a caption or other metadata) inside your <span>, but otherwise using CSS to get the block-level display seems to work fine from our perspective.

Note also that LanguageConverter might be an appropriate solution to allow egl and rgn to co-exist on the same wiki.

Created a "midweek release" to break a Kartographer dependency based on a78e700def47f6733e0d81ea3dca3f566cfb5373, which Isabelle has started rt testing on.

No one owns MassMessage, apparently. :(

Tagging @daniel on this. In various places we use ParserOutput::getRawText() is null to indicate the "no request html" case. I could use some more context, though -- what are some of the confusion & bugs related to this? We really can't generate metadata output without generating HTML, so I've always been a bit hazy on what sort of optimization we're aiming for here. If we're trying to respond to a HTTP HEAD request (eg) the "metadata" we're talking about is pretty distinct from the "metadata" usually associated with the ParserOutput (page properties, categories,m etc) -- they overlap only in cache-related properties, and again (unfortunately) we have no way of computing the cache metadata without doing a full parse of the page.

@ihurbain This is probably a task which your i18n fragments can now fix?

In T365433#9846460, @ihurbain wrote:

Strong hypothesis: this happens when CommonsMetadata is involved in the rendering of the page.
Specifically, the DataCollector::verifyAttributionMetadata runs getText, which on VE content is considered Parsoid content, which converts back and forth between content and PageBundle, which loses the version number during that operation.

This is probably also a main path for T365036.

{"reason":"The given page ([0:TransformSource_0ZbZaRyLAW]) does not belong to page ID 383 but actually belongs to 425",

Is this legit? Is the root cause here that we're reusing a page ID and/or a title?

In T365806#9829800, @Soda wrote:

Noting that implementing infinite scrolling will effectively result in a decline of T325607 since infinite scrolling is effectively unsupported by Google bot. :(

Yes, but the solution is specific to DT. This is an attempt to generalize that solution. I plan on chatting with the editing team next week to figure out to what extent their existing code is generalizable. My understanding is that it is pretty tightly tied to the way they look for "new comments" and do notifications; a generalized feature might need a way for an editor to manually make/fix a particular association between page and section rather than try to magically infer the correct link the way that DT does.

These are proposals/help-wanted for now. The WMF annual planning process also deals better with discrete features/interventions, so if we can collaboratively hammer out the mechanics here it makes it easier to insert into planning to get resources allocated.

Please comment on the specific tasks, your concerns are addressed there. But also feel free to suggest other solutions! The point is that @Fuzzy's list of desiderata is an excellent one, and we should continue to work on finding solutions to those issues that don't necessarily involve increasing article size ad infinitum. There are plenty of large topics that could benefit from UX improvements to better tie together a collection of separate pages.

I've created the following five feature requests as strawdog proposals to address each of @Fuzzy's five concerns. Feel free to poke holes in my proposed solutions, suggest improvements or alternatives, etc. But I want to keep the focus on making split documents work better, since we're going to continue to butt up against article size limits.

• T365806: Infinite scroll for articles (split documents on wikisource)
• T365808: "Browser search" across related/split articles
• T365810: Export a collection of pages as a single document (PDF, HTML, printable) *client-side*
• T365812: Generalized stable link mechanism to both *page* and *section*
• T365819: Shared citations for multiple pages

The interwiki map is (in theory) generated by Extension:Interwiki, which could probably be convinced to export its table as a JSON file as well. It seems like the https://noc.wikimedia.org/conf/interwiki.php.txt is actually the target output, I don't see why that can't be generated as an optional export by Extension:Interwiki as well.

For completeness, @subbu has proposed another alternative to #3 and #4 above, which I'll call "#3.5" make class="reference-text mw-reference-text" to both legacy and parsoid; aka have both parsers emit both classes.

So, having written the above two patches to replace byte-size limits with character-size limits, let me set out why I'm afraid this is a Bad Idea and I've done a Bad Thing by even bringing up the possibility of a technical fix here. Sorry:

• Solving the five tasks outlined by @Fuzzy in T275319#9818815 above would be a much better solution to the root cause problems here, and provide features generally useful to a number of projects outside this one particular case:
  • Contextual Understanding: solutions to this might involve the sort of 'infinite scroll' UX common on the web, where we could chain separate pages together to provide a seamless reading experience. This could work on any number of pages where there is a both a hierarchical or other structure as well as a suggested reading order (API docs, historical events, etc) T365806
  • Internal and External References: the Cite extension could be improved to better handle this case. Having citations stored in a separate page or subpage and then referenced in a way that would allow a final "references" section to combine them would be generally useful. I know the "citations in wikidata" folks are also thinking in this general direction. Other wikisources have also solved this problem with templates which manually insert links to a final end-notes section. Improving support for this sort of pattern would be great. T365819 is one proposal.
  • Link Integrity: Again a super general problem which would benefit from work. We have permalinks and a link shortener, and there are templates like {{anchor}} as well. DiscussionTools as well as done some work on maintaining links when sections are renamed and the page title changes (in DT's case, existing topics moved to archive pages). We just need to put the pieces together to allow a standard solution to allow you to write an anchor to a specific section of a page which is robust against both section renaming and the section being moved to a different title. T365812
  • Unsearchable Split Texts: Just bind Ctrl-F to a search query that includes all pages in a certain category or that share a page prefix. @Tgr had some ideas here, apparently the implementation is not hard at all. T365808
  • Exporting Issues: This was the Collection extension's entire raison d'ê·tre. It has suffered from neglect and a lack of maintainers -- as well as from DoS issues which touch on common problems we've discussed here. Creating a PDF for a book containing hundreds of pages of content is computationally expensive. Some way of providing that feature while also protecting it from abuse is needed, but this is something which could 100% be built in Wikimedia Labs. T365810
• Increasing size limits is a one-way ratchet. Once articles of increased size are allowed through and stored in the database, it is really hard to get them back out. For better or worse, MediaWiki's article size limits were built around preventing overly-large articles from being stored in the first place and the code to deal with articles exceeding the limits which are already coming from the database is comparatively immature. I'd like to say "we'll try out larger limits on wiki X for a while, and if this leads to problems (with resource consumption, DoS attacks, etc) then we'll just bump them back to what they were" but that is, unfortunately, not straightforward from a SRE perspective. We'd need a rollback strategy Just In Case before we actually deployed something like this.
• @Krinkle mentioned that there are various situations where we *do* actually want/need to know specific byte size limits on different things. The particular approach in the patch above limits byte size to 4x the character limit, due to the way that UTF-8 works, but that's not a particularly tight bound, and there are extensions of UTF-8 which permit 5- or 6-byte characters as well. You could consider counts based on PHP's grapheme_strlen but that could lead to even larger bytes-per-grapheme counts. We would probably want a combination of byte- and character-based limits just to ensure some amount of predictability.
• As elaborated at length above, these patches are only a band-aid, and it is a near certainty that new source texts will be found which violate the any newly-raised limits. Solving @Fuzzy's five tasks would be a permanent solution.

This is a near-dup of T275319 and further discussion should take place on that ticket. It's not *entirely* the same thing, so I'm not going to formally close this as a dup, but the arguments being made for and against are almost identical so it really doesn't serve much of a point to have that discussion in two different places.