Help talk:Template limits

New template limits, Special:ExpandTemplates

The new template expansion limits, announced on wikitech-l and wikipedia-l, are now in effect on a trial basis. Keep an eye out for any broken articles. Note that there is some information to help track down problems in comments in the HTML source of the parser output. To help editors to substitute templates for literal text in problem articles, I've introduced a new special page: Special:ExpandTemplates. It works like adding subst: to all the templates, except you don't have to repeatedly save the page. -- Tim Starling 07:41, 14 August 2006 (UTC)

Example [1]: Shows a test with inclusions of template:cite web, where the limit is reached at 267 example calls of cite web. The critical thing is the Pre-expand include size, which is now limited to 1MB (Look at the html source of the page and find the string "Pre-expand include size", which is included in a html comment). The html contains {{cite web}}<!-- WARNING: template omitted, pre-expand include size too large --> for calls that exceed the Pre-expand include size limit. --Ligulem 09:13, 14 August 2006 (UTC)

I'm not sure where to discuss this, but I have noticed several pages that use "Template:Familytree" include that template a number of times, and when it gets to about 26 transclusions they aren't expanded any more. Here's an example: User:Timc/Sandbox. --  timc  talk   02:54, 15 August 2006 (UTC)

Template:Familytree is large and ugly. You can either reduce its size, to allow you to fit more of them in the 1MB limit, or you can subst it into pages instead of including. -- Tim Starling 03:33, 15 August 2006 (UTC)

Tim, how does the limit work with things bounded by <noinclude> tags (or everything not bounded by <onlyinclude> tags) on the template page? Do they add towards the limit or does only the info which is actually transcluded for evaluation counted? I'd assume they aren't included in the computation, but worth checking. Also, {{listadmins}} fails now, but lists the Pre-expand include size as 'only' 837703. I assume it stops counting at and doesn't include the item that puts it over the limit? --CBD 10:55, 15 August 2006 (UTC)

I'm not sure, but the relative size of the <noinclude> section is small compared to the rest of the template. --  timc  talk   13:04, 15 August 2006 (UTC)

<noinclude> sections are included in the pre-expand size. If the documentation is large or frequently changed, I'd recommend that you move it to a subpage. Then you can transclude it into both the <noinclude> section and the talk page. It does indeed stop counting when an item puts it over the limit, this behaviour is intended to allow the bulk of a page to render correctly when there is a single large template present plus a number of small templates further down. The large template will be left out, but the small templates will still be included. -- Tim Starling 21:18, 15 August 2006 (UTC)

Indeed. If I remove the <noinclude> part from template:cite web entierly and repeat the test I described above here in this section, I can do 380 inclusions of the cite web template code instead of 267 for that testcase. What a difference! I would strongly recommend that an admin edits template:cite web accordingly. (Cite web is currently included on 18,500 pages) --Ligulem 22:49, 15 August 2006 (UTC)

I am glad I found this section, I was wondering what is happening to Wikipedia:Copyright problems. In this page, we transclude sub-pages for daily reports of copyright problems; everything was going well, but we currently have quite a large backlog, meaning that we seem to hit the limit. Any idea about what we should do ? Remove the backlog would be a good thing to do (but I can't help much with that), but it will come back at some point. Schutz 22:54, 15 August 2006 (UTC)

I would say you need to completely redesign that page. Transcluding that much of stuff into a page is no longer working. --Ligulem 22:58, 15 August 2006 (UTC)

Hum... Ok, we could simply link to the pages for the individual days instead of transcluding them (the way it is done on WP:AFD); that would solve the problem in the short term. However, it could become a problem for pages such as WP:AFD too; looking at one random but recent example, Wikipedia:Articles for deletion/Log/2006 August 11, this page is getting close to the limit. In a "good" (or rather bad) day, there may be many articles listed for deletion, with a few of them attracting a fair number of comments, and it may be enough to get past the limit. Is the 1 Mb limit negotiable if we see that some of these articles get bitten by the limit ? Schutz 23:23, 15 August 2006 (UTC)

The limit is negotiable, maybe we could raise it to 1.5 or 2MB if there was a solid case for that. But note that splitting these pages up does have a significant beneficial impact on server load. It might be better to analyse them and look at why they get so big -- maybe it's a big hassle to manage per-day pages like on AfD, and even more so to split them up further. Maybe new features in MediaWiki or bot-driven automation could help with that. I'm willing to hear your ideas. Increasing the limit might be good in the short term, but it's not going to solve the usability and performance problems that huge pages cause. -- Tim Starling 00:53, 16 August 2006 (UTC)

Wikipedia:Articles_for_deletion/Log/2006_July_28 hits the limit with 13 debates un-transcluded. Pre-expand include size: 1047585 bytes Post-expand include size: 956640 bytes Template argument size: 4759 bytes Maximum: 1048576 bytes Wikipedia:Articles_for_deletion/Log/2006_July_26 is close, don't know about anything beyond 20060810224009 though.

The number of pages on AFD is only going to keep increasing, and I can see how the pages may not be managable for people as is... so I see how just increasing the limit doesn't really solve anything. However, can you have the template parameters listed and/or the transclusion linked? Would make AFD pages being cut off at least bearable still (but WP:AFD/Old's statistics and other bot run things are probably broken) until things are sorted out. Kotepho 01:44, 16 August 2006 (UTC)

Having omitteded transclusions linked would be nice to have, but not critical, as those pages need to be fixed anyway by editors. I don't know how ugly it would be to implement in MediaWiki. If it is ugly to implemenet, I would say leave it as it is. --Ligulem 08:03, 16 August 2006 (UTC)

For WP:CP, as mentioned above, I am going to make subpages instead of transcluding so many pages; this should solve the problem (there aren't that many copyright violations). For WP:AFD, I don't know, but I am happy to help with any bot work (especially given that I found the problem on WP:CP while investigating my bot's work). Schutz 06:46, 16 August 2006 (UTC)

I'm not terribly familiar with what y'all are talking about in this thread, but is this what's going on in List of Hebrew names? —ptk✰fgs 03:51, 16 August 2006 (UTC)

Yes. If you look at the source, you will see errors like "<!-- WARNING: template omitted, pre-expand include size too large -->". --  timc  talk   04:16, 16 August 2006 (UTC)

I've tried to subst template:unicode but that produced a section that looked like this (excerpt):

{{{1}}} {{{1}}}
    * {{{1}}} {{{1}}}, {{{1}}}, Tola. "Worm; grub." Masculine.
    * {{{1}}} {{{1}}}, {{{1}}}, Thomas. "Twin." Masculine.

Substing just some templates on a off-limits page doesn't seem to work. It looks like we can't use MediaWiki server for substing in this case (yes we could subst everything by Special:ExpandTemplates, but I wanted to try substing gradually). --Ligulem 08:44, 16 August 2006 (UTC)

I fixed List of Hebrew names by substing template:unicode in several steps (per sections) in my sandbox. --Ligulem 09:05, 16 August 2006 (UTC)

Tim, above you suggested transcluding documentation into a 'noinclude' section of the template page rather than having it reside there directly. That would seem to imply that the 'pre-expand' size of a template is only the size of the contents of that page NOT including the size of any templates called by it... otherwise there would be no benefit to the procedure you suggest. This almost seems to then become an argument for 'meta-templating'. For instance, wouldn't that mean that if an article contained {{example|param1=Fred|param2=Fish}} that 'template:example' could be set up to be just {{handoff|{{{param1}}}|{{{param2}}}}} with all of the complicated logic/text actually in the 'handoff' template? Thus minimizing the size of the template called while still having exactly the same logic/display - just once removed? Obviously, that's... silly... but I'm trying to understand the rationale behind the methodology here. We have been consolidating alot of things with sub-templates into single templates to avoid 'nested transclusions', but this change now seems likely to lead to the reverse.

Would it somehow be possible TO include 'sub-template' size in the 'pre-expand' size and NOT include 'noinclude' type materials? Since the goal (as I understand it) is to limit the amount of text being 'transcluded in for evaluation' I would think we would want to take these issues into account. For example, if a 900kb page is transcluded, but everything except the word 'frog' is bounded by 'noinclude' tags you are really only transcluding the word 'frog' and the evaluation is quite fast (I just checked)... so why count the full 900kb towards the limit? In contrast, if you call a template which is only six characters long, but those six characters are calling another page which is 900kb long it takes several seconds to evaluate but still only counts as six characters. I can imagine that there are technical difficulties in ignoring 'noinclude' info and including 'sub-template' info, but not doing so seems to give some results wildly inconsistent with the intent - and open an easy loophole for getting around it while page loads are still just as slow. --CBD 00:47, 17 August 2006 (UTC)

Sub-templates are included in the size, just not sub-templates included from within <noinclude> sections. To put it another way, the <noinclude> sections are counted and removed, but they're not expanded. There's no loophole. Documentation subpages have been an agenda of mine for months, because at the moment, editing the documentation for a template adds an item to the job queue for every page that uses the template, and clears the cache of all those pages too. Documentation subpages are not registered in the links for pages using the template, so editing a documentation subpage only clears the cache of the template itself, not the pages that use it. There is a technical solution for this: some people have suggested stripping <noinclude> sections from the template before and after a change, comparing the two, and then only clearing caches and adding items to the job queue if the resulting stripped version has changed. It'll probably be implemented at some stage, but it's not here yet.

As for removing <noinclude> sections from the limit -- I did consider it when I was writing the feature. Although <noinclude> sections are among the cheapest things the parser has to process, they still do have a finite cost. One could easily imagine a DoS attack based on including thousands of templates consisting of only a 1MB <noinclude> section -- or worse, tens of thousands of small <noinclude> sections per template inclusion. Whether or not it has <noinclude> around it, the text still has to be loaded from the database. There are costs such as disk I/O, DB cache size, network bandwidth, and of course the CPU time spent removing the tags. It seems to be a good candidate for inclusion in the limit. -- Tim Starling 05:57, 17 August 2006 (UTC)

I've been thinking about doc subpages for templates too, because people do like documentation on template pages and editing of documentation should be separated from editing template code (at least for heavy use templates). I image we could even have a doc tab for templates (we currently have "template" and "talk"), but I have no idea how ugly this would be to implement. Yet another namespace? In any case, it is very interesting that you are thinking of documentation subpages. --Ligulem 09:02, 17 August 2006 (UTC)

Ok, that (only sub-templates in 'noinclude' sections are 'not counted') makes alot more sense. My primary objection to counting 'noinclude' sections is that I had just been discussing a change to featured lists to mark various sections of a list page 'onlyinclude' so that when the list page were transcluded it would produce a nice little blurb similar to an 'article of the day' page (see User:CBDunkerson/Sandbox2 for an example)... which could then be made into a 'List of the day' and/or 'random list' on the featured content page. Obviously, that is now untenable and separate pages would have to be set up for the blurb on each list and manually updated when the list info changes. While the 'resource cost' of unincluded sections is not zero it is clearly significantly less than the cost of included sections. If there are not significant technical difficulties in doing so it might be nice to somehow reflect those different impacts in the 'expansion limits' logic. --CBD 10:42, 17 August 2006 (UTC)

If it aint broke, don't fix it

In relation to this template expansion limits issue I'd like to suggest that people hold off on 'fixing' things for the time being. If a page isn't displaying right then obviously we need to do something to get it back to working, but I'm seeing people mass moving documentation, substing templates, discussing redesigns, et cetera. We may end up adopting standards and wanting to do these things... but there is no rush. If it isn't causing a page to display improperly right now then changing things before this new limit is fully evaluated/understood and worked into our standards may end up needing to be changed back. Until yesterday it would have been inconceivable to have a 'documentation' sub-page transcluded in for every template (or many templates). Before we do that to hundreds of pages I think we should evaluate it a bit more. It might be easier to just make documentation on template talk pages standard (though I've always preferred it on the template page). Or there may be changes (per my questions above) which make this documentation problem moot. And there are other issues to consider. For instance, note that categories and interwiki links shouldn't be relocated to such a sub-page... which means a popular template that is interwiki'd all over actually becomes less usable because of the increased size from the interwikis - so maybe we want to consider a different way of doing template interwikis. Et cetera. --CBD 01:11, 17 August 2006 (UTC)

In case you are referring to my edits (example): The benefits of the doc subpage for in template included noinclude parts is obvious. Templates which are by their nature intended as meta-templates (templates to be included in other templates), like those from Category:Date computing template or stuff like template:cite news which is intended to be included in pages multiple times (10, 20 times) clearly benefit from the extracted doc pattern. The MediaWiki parser has to parse the whole content of the noinclude, but it can skip templates included in noinclude sections. And per the change on template cite news: if we leave the doc inside the template page itself, it "eats up" from the pre-transclude-size of each page this template is included. Wikipedians will stop using templates like template:cite news if they encounter that the limit is crossed at the very moment they look at a broken preview of their edit, and they will skip that edit. Why should we risk that, if we know better? So what do you want to wait on? What do you need to "evaluate" further? What isn't understood? --Ligulem 07:38, 17 August 2006 (UTC)

What does discussion / evaluation help? Well, until I asked about unincluded sections people were talking mostly about recoding templates to save space... since they have been talking mostly about removing documentation. Until I asked about sub-templates it wasn't clear (to me at least) that while unincluded sections are counted sub-templates in unincluded sections are counted by the length of the template call rather than the template page... though the reverse is true for included sections (i.e. page size not call size).

What is there still to discuss / evaluate? Well, if we are going to do this 'documentation sub-page' methodology we should think about how best to work it. For instance, in your 'MONTHNAME' example above you left the categories and interwikis behind on the template page when you created the documentation sub-page. However, in my experience those are the two unincluded items which get updated most frequently. After thinking about it I wonder if it would make sense to also move these to the 'documentation sub-page' and place them inside 'includeonly' tags there so that they show up on the template page, but not the sub-page? Any drawbacks to that? Would it be a good idea to establish a standard that the 'documentation sub-pages' of protected templates should be unprotected so that anyone can add interwikis and the like? 'Look before you leap' is seldom a bad idea. You reference how removal of unincluded sections (arising BTW out of the discussion here) has increased the 'cite-news limit' from 267 to three-hundred something. Ok, but.... were there really any pages with more than 267 news citations on them? Or running over the limit because of this template in conjunction with others? It seems implausible to me... so what harm in fully evaluating and discussing the best way to handle this new limitation?

Sorry, I'm a 'ducks in a row' kind of guy. I like to get all the answers and general agreement on standards before implementation. Ohhh... another idea, could we standardize on using something short like <noinclude>{{/doc}}</noinclude> for all 'documentation sub-pages' so they can always be found at the same name? Also, if we made a ==Documentation== or somesuch standard at the top of all sub-pages then people who have 'section editing' turned on would see an edit link for the documentation on the main template page, but actually be editing the sub-page when they clicked it... rather than having to know how to find the sub-page at 'Name/doc'. --CBD 11:09, 17 August 2006 (UTC)

Ok. I'm more the quick shot one :) (If I see there is no danger in doing so). BTW, I was fixing [2] by applying the doc subpage pattern. But I will hold off from further actions for now (but I do think I already catched some of the worst cases anyway). --Ligulem 11:28, 17 August 2006 (UTC)

Found a casualty on meta

http://meta.wikimedia.org/wiki/Template:List_of_language_names_ordered_by_code --NE2 16:33, 17 August 2006 (UTC)

Fixed by replacing the template calls with the expanded wiki-source by using m:Special:ExpandTemplates (see project page). --Ligulem 18:36, 17 August 2006 (UTC)

Not very useful

Forgive me, but this isn't a useful feature, particularly without giving any kind of warning on Wiki (the mailing list is all well and good but basically invisible). It breaks WP:CP for a start, so the WMF can just live with noone being able to list, review or delete illegal text. Or, the devs can write us a bot that daily subst's the page. Or just get rid of this 'feature'. -Splash - tk 17:24, 17 August 2006 (UTC)

"It breaks WP:CP for a start", well nothing new (see VPT), that's why I've added it to the new Category:Pages exceeding template inclusion limits. And I must say we shouldn't optimise MediaWiki for this kind of page. It is very atypical. I'm sure we will find another solution for this page. And the "feature" is not a feature per se, it helps running the servers smoothly and disables certain template DOS attacks. These limits make sense to me and I trust Tim that he installed it for good reason. And I also see no problem the way he introduced it. I have read the announcement on the mailing list and I was aware what he intended to do. And you can't get your feet wet by endless theorising upfront. These new limits do work astonishingly well. --Ligulem 18:21, 17 August 2006 (UTC)

Of course it's new. CP wasn't broken until this was introduced. Adding it to a category does what, exactly? Does it allow AfD to function? CP? Does it break articles, processes, policies? And I suppose we're supposed to accept that, say, everyone reads VPT? I didn't ask for theorising, I asked for some advance warning that large numbers of things were going to be wilfully broken. In what sense do they "work astonishingly well"? They are invisible unless they break things, at which point things aren't working at all. I read the mailing list posts now and see that they are protection against a problem that never occurred, which is all very well, but it's a solution that is really a long way suboptimal. -Splash - tk 19:06, 17 August 2006 (UTC)

Yes it's new. What I wanted to say is, it had already been reported on VPT. As such, it's no news. Which "large things are willfully broken"? BTW the problem *did* occur here, caused by such things (which demonstrated the attack vector). Or do you think server response times in the order of 10 or 20 seconds to update the cached html of a page are acceptable? Well, ok, I'm not a WikiMedia dev anyway and do have nothing to say about how this wiki is configured. Tim does. He's also co-responsible for keeping the site running. So probably I should better shut up on this anyway. But Tim has my support on this. --Ligulem 23:13, 17 August 2006 (UTC)

Limit increased, links added

I increased the limit from 1024KB to 2000KB and changed the code to add links to missing templates. WP:CP now works, although it takes 8.5 seconds to render and generates 1.5 MB of HTML, so I wouldn't like to call it "fixed". -- Tim Starling 23:31, 17 August 2006 (UTC)