Just by means of an enwiki update, I wanted to note this discussion on the int-admin's board: https://en.wikipedia.org/wiki/Wikipedia:Interface_administrators%27_noticeboard#Removing_legacy_javascript_globals_from_skin_pages

Basically, the imported user scripts have either been fixed or, if their "owners" are active, they've been notified; if there's no response or action in a few days or a week or so, I'll fix the remainders. The majority of cases, however, are presumably coming from user's skin pages; I've got that list, sorted by active users and skin. After some discussion, I've posted a note at the technical noticeboard for editors to see and be aware of the process; barring major opposition, I'll go ahead in a few days. Presumably that should be a chunk of these, and I'll report back here when done.

legacy variables are slowly decreasing:

My suggestion now. Given that I have cleaned as much as possible for now. I suggest sending a message to tech news and disabling it for group1 one week afterwards. Does that sound good to you @Krinkle and @Jdlrobson ?

Group1 is fine, assuming that any spikes in errors on logstash will be dealt with in follow-ups (or the change rolled back after making that data available). This is important to me given the error rates are currently pretty low and we have error alerting setup. Post-change we'll want to make sure

I'd suggest for that reason deploying on a Monday and making sure at the end of the week we are below the current threshold here: https://grafana.wikimedia.org/d/000000566/overview?viewPanel=16&orgId=1.

I'm cleaning up even more. Hopefully it'll be pretty cleaner by Monday.
@Johan Hey, do you think this should go to tech news or should I compose a dedicated announcement instead?

@Ladsgroup The announcement being that you're cleaning up user scripts, so that people might see their user scripts edited? Sure, that can go into Tech News. Feel free to suggest a few brief sentences I can base the item on if you know what you want it to say. (:

That part is mostly done. It's about disabling legacy global variables deprecated six years ago (i.e. puling the plug) which might break user scripts and such. I cleaned as much as possible but there will be cases regardless.

Just group 1 or should we announce it's about to come to group 2 soon too?

In T72470#6908747, @Johan wrote:

Just group 1 or should we announce it's about to come to group 2 soon too?

Group 2 will have soon as well.

How have I to change something like this in my user script to prevent breaking it soon:

mw.libs.DocTaxon.nsn = mw.config.get( "wgNamespaceNumber" );
mw.libs.DocTaxon.furnish = function () {
   var nsn    = mw.libs.DocTaxon.nsn,
       pgName = mw.config.get( "wgPageName" ),
       server = mw.config.get( "wgServer" ),
       title  = mw.config.get( "wgTitle" ).replace( /\/.*$/, "" );
  ...
}

In T72470#6935908, @doctaxon wrote:

How have I to change something like this in my user script to prevent breaking it soon:

If you're using mw.config.get you're fine.

Please check / update / clarify the introduction at https://www.mediawiki.org/wiki/Manual:Interface/JavaScript#mw.config

Change 674861 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):
[operations/mediawiki-config@master] Disable Legacy javascript in fawikiquote

https://gerrit.wikimedia.org/r/674861

Change 674861 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable Legacy javascript in fawikiquote

https://gerrit.wikimedia.org/r/674861

Mentioned in SAL (#wikimedia-operations) [2021-03-25T11:52:54Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:674861|Disable Legacy javascript in fawikiquote]] (T72470) (duration: 01m 07s)

Change 675751 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):
[operations/mediawiki-config@master] Disable legacy javascript in group1

https://gerrit.wikimedia.org/r/675751

Change 675751 merged by jenkins-bot:
[operations/mediawiki-config@master] Disable legacy javascript in group1

https://gerrit.wikimedia.org/r/675751

Mentioned in SAL (#wikimedia-operations) [2021-03-30T11:21:56Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:675751|Disable legacy javascript global variables in group1]], Some increase in client errors is expected (T72470) (duration: 01m 11s)

@Ladsgroup there was an uptick of errors after the deploy. In last 12hrs there were 3211 errors, which is a percentage increase on the 12hrs the day before (2041 errors)

I've fixed the main offenders which between them caused 712 errors these are listed below:

https://commons.wikimedia.org/wiki/User:BMacZero/AjaxCfdClose.js
https://commons.wikimedia.org/wiki/MediaWiki:Commander.js
https://ca.wikipedia.org/wiki/Usuari:Townie/scripts/infotaules.js
https://commons.wikimedia.org/wiki/User:Strainu/ExCommons.js
https://he.wikipedia.org/w/index.php?title=%D7%9E%D7%A9%D7%AA%D7%9E%D7%A9:Yonidebest/monobook.js/move.js
https://meta.wikimedia.org/wiki/User:DerHexer/testlocalrenamelinks.js
https://he.wikipedia.org/wiki/%D7%9E%D7%A9%D7%AA%D7%9E%D7%A9:%D7%A0%D7%A8%D7%95_%D7%99%D7%90%D7%99%D7%A8/common.js

Please allow about a week to make sure we get these graphs/numbers stable, I can do outreach on https://meta.wikimedia.org/wiki/User:Jdlrobson/User_scripts_with_client_errors to get the rest of the lest prominent errors fixed. Stable = no ReferenceError listed in the first page of "Top 10 errors (group 0+1 - beta cluster)"

When deploying to group 2 can I please request you roll out in stages and leave the top 10 wikis until last?

Note, that we have alerts setup, and given the spike here, I'd expect to see a lot larger spike on wikipedias triggering those.

Yesterday was a federal holiday, compared to the past couple of days, it's not a huge increase.

https://grafana.wikimedia.org/d/000000566/overview?viewPanel=16&orgId=1&from=1616676962670&to=1617117218995

Hey @Ladsgroup I think the majority of these issues on wiki are addressed now. Thanks for pushing this, it's really nice to see us paying off technical debt here.

Your script

The remaining issues are the search results here:
https://commons.wikimedia.org/w/index.php?title=Special:Search&limit=500&offset=0&ns0=1&ns2=1&ns6=1&ns12=1&ns14=1&ns100=1&ns106=1&search=insource%3A%2Fif%5C%28wgNamespaceNumber+%5C%3D%5C%3D+6%5C%29%2F+intitle%3A%2F.%2A%28common.js%7Cvector.js%29%2F&advancedSearch-current={}

I don't know which of these users are active, but the issues I'm encountering now seem to relate to this outdated code fragment. Could your script be run on these?

Moving forward

I think it's okay to push forward with some group 2 wikis (keyword some :))

Remember here that group 1 is a tiny fraction of all our errors, and most of our errors come from external domains so when you cut those out, and look at the filtered dashboard a small difference in the graph can make a difference in the usefulness of the dashboard. I am not concerned about the spike here for group 1, but it does point to potential problems during a rollout in group 2, so a gradual rollout there is preferred. I would like us to maintain the current downward trend of errors and to avoid triggering any sort of alerting.

If I'm reading the graphs correctly for a 12hr period we can expect up to 15k errors from mw.config errors by enabling all wikis and just 2K more will trigger an alert so we need to dice this up some how :).

In particular let's leave en.wikipedia.org, fr.wikipedia.org, ru.wikipedia.org, de.wikipedia.org, zh.wikipedia.org, ja.wikipedia.org for a separate deploy (perhaps making use of dblists/large.dblist or dblists/small.dblist).

I am on vacation next week from Wednesday 7th, so week of 12th would be preferred (or Monday 5th deploy) if you want me to be responsible for fixing the gadget errors that spill out of the deploy. If you are okay to monitor the dashboard and debug and fix the errors with a rate greater than 100 every 12hrs feel free to do this next week without me.

I ran the script and it's mostly clean now. Next week, I'll deploy it on all wikis except the ones you mentioned. I'll try to clean up and fix most of these issues but I'll let it stay like that for a day first so the users themselves fix the most important ones.

Change 677120 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Disable legacy javascript globals on all wikis except some big ones

https://gerrit.wikimedia.org/r/677120

Change 677120 merged by jenkins-bot:

[operations/mediawiki-config@master] Disable legacy javascript globals on all wikis except some big ones

https://gerrit.wikimedia.org/r/677120

Mentioned in SAL (#wikimedia-operations) [2021-04-06T11:07:09Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: [[gerrit:677120|Disable legacy javascript globals on all wikis except some big ones]] (T72470) (duration: 01m 01s)

This broke uzwiki because one of their main scripts was using global variables but using window.wgFoo and my script couldn't catch those. Fixed now: https://uz.wikipedia.org/w/index.php?title=MediaWiki%3AWikibugs.js&type=revision&diff=2187564&oldid=2058805

The errors should slowly recover, I let the community handle most of cases in eswiki. There are some cases in arwiki that's minified and I'm not going to fix that.

I noticed https://sk.wikipedia.org/w/index.php?title=MediaWiki%3ACommon.js&type=revision&diff=7187446&oldid=7156186 was causing about 2000+errors an hour, and https://simple.wikipedia.org/w/index.php?title=MediaWiki%3AGadget-Twinkle.js&type=revision&diff=7462942&oldid=7333050 a similar rate so with those fixed we are now out of the red danger zone and the alert has been resolved.

In thwiki, one of our main scripts https://th.wikipedia.org/wiki/WP:iScript, used by many Wikipedians is broken, but the original creator of this script has left Wikipedia. So there might be a spike in code error over there.

I have since unofficially fixed those errors but only a small group of people have known about me fixing the whole script.

Edit: the older versions is also minified, meaning that the central maintenance couldn't catch those errors. If it could catch those errors, I'm sure couldn't catch all because it couldn't on my version too.

In T72470#6977422, @Jdlrobson wrote:

I noticed https://sk.wikipedia.org/w/index.php?title=MediaWiki%3ACommon.js&type=revision&diff=7187446&oldid=7156186 was causing about 2000+errors an hour, and https://simple.wikipedia.org/w/index.php?title=MediaWiki%3AGadget-Twinkle.js&type=revision&diff=7462942&oldid=7333050 a similar rate so with those fixed we are now out of the red danger zone and the alert has been resolved.

Thanks. I fixed some cases here and there as well now, hopefully it'll stay in the green zone.

Let's revisit this in a week's time. We'll leave community to fix up remaining scripts. If numbers are still high on certain wikis, we might want to address those ourselves our patch Common.js.

We're still hovering dangerously around the threshold level. Before the change 5k was the high for errors in a single day. I'd like to get back to that level for an entire 24hr period before we proceed further (current rate is 5.8k but unclear if that's the new high.. time will tell).

Note: At time of writing I am seeing just under 10k errors in the last 12hrs on https://grafana.wikimedia.org/d/000000037/mw-js-deprecate?orgId=1&viewPanel=7&refresh=1m&from=now-12h&to=now, which is promising.

Hi. We have quite a lot of broken scripts on pl.wiki. I've fixed some gadgets yesterday.

I've made a search for pl.wiki. You might want to pass it along to other communities:
https://pl.wikipedia.org/w/index.php?sort=create_timestamp_desc&search=intitle%3A%2F.*%5C.js%2F+insource%3A%2F%5B%5E%27%22a-z%5C._%5D%28wg%5BA-UW%5D%7CwgVariant%7CwgVector%7CwgVersion%7CwgVisual%7CsearchTerm%7Cskin%5B%5Ea-z%5D%7Cstylepath%5B%5Ea-z%5D%7CtableSorterCollation%5B%5Ea-z%5D%7CadvancedSearch%5B%5Ea-z%5D%29%2F&title=Specjalna%3ASzukaj&profile=advanced&fulltext=1&advancedSearch-current=%7B%7D&ns8=1

This doesn't include window.wg case you mentioned. Would have to be a separate search. This obviously also searches in comments so some false positivies there. I guess you might have something better if you search on DB level (and e.g. remove comments before checking). And there is also a problem that people use a fallback var wgUserName = mw.config.get( 'wgUserName' ). But I guess you might already know that 🙂...

Anyway, just to be sure, example of a script missed in initial sweap:
https://pl.wikipedia.org/w/index.php?title=Wikipedysta%3AMatma_Rex%2FczyWieszSprawdzone.js&type=revision&diff=62544680&oldid=56845244

Missed parts:

if (wgAction=='view') {
...
title: wgPageName,

PS: Above script is just an example. I've already notified the author of the script 🙂

Hi, Thanks for pointing out but let me be clear here. We are not going to fix all scripts users have been writing and using in all wikis. We try our best and more importantly make sure Common.js (and preferably all mediawiki namespace gadgets) are fixed but we can't possibly sweep millions of lines of code to find all of these and fix them. I personally did more 10K fixes just in the past months (More than 50K for all deprecation fixes) and this has been emitting warnings for more five years. We have communicated this in tech news but that's all.

We make sure not to roll out a huge change without proper deprecation warnings and communication and giving enough time to fix (which we did here) but fixing user scripts is responsibility of the maintainer not us.

Change 678345 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[mediawiki/core@master] Deprecate $wgLegacyJavaScriptGlobals in 1.36

https://gerrit.wikimedia.org/r/678345

Change 678345 merged by jenkins-bot:

[mediawiki/core@master] Document $wgLegacyJavaScriptGlobals as already deprecated in 1.36

https://gerrit.wikimedia.org/r/678345

Change 678351 had a related patch set uploaded (by Jforrester; author: Ladsgroup):

[mediawiki/core@REL1_36] Document $wgLegacyJavaScriptGlobals as already deprecated in 1.36

https://gerrit.wikimedia.org/r/678351

Change 678351 merged by jenkins-bot:

[mediawiki/core@REL1_36] Document $wgLegacyJavaScriptGlobals as already deprecated in 1.36

https://gerrit.wikimedia.org/r/678351

Change 678569 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Disable legacy javascript in jawiki

https://gerrit.wikimedia.org/r/678569

Change 678569 merged by jenkins-bot:

[operations/mediawiki-config@master] Disable legacy javascript in jawiki

https://gerrit.wikimedia.org/r/678569

Mentioned in SAL (#wikimedia-operations) [2021-04-12T11:29:50Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:678569|Disable legacy javascript in jawiki (T72470)]] (duration: 00m 56s)

Change 678714 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Disable legacy javascript variables in zhwiki

https://gerrit.wikimedia.org/r/678714

Change 678716 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[mediawiki/core@master] Drop wgLegacyJavaScriptGlobals

https://gerrit.wikimedia.org/r/678716

Change 678714 merged by jenkins-bot:

[operations/mediawiki-config@master] Disable legacy javascript variables in zhwiki

https://gerrit.wikimedia.org/r/678714

Mentioned in SAL (#wikimedia-operations) [2021-04-13T11:14:00Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:678714|Disable legacy javascript variables in zhwiki (T72470)]] (duration: 00m 57s)

Change 679350 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Disable legacy javascript global variables in ruwiki

https://gerrit.wikimedia.org/r/679350

Change 679350 merged by jenkins-bot:

[operations/mediawiki-config@master] Disable legacy javascript global variables in ruwiki

https://gerrit.wikimedia.org/r/679350

Mentioned in SAL (#wikimedia-operations) [2021-04-14T23:27:11Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:679350|Disable legacy javascript global variables in ruwiki (T72470)]] (duration: 01m 16s)

According to https://grafana.wikimedia.org/d/000000037/mw-js-deprecate?orgId=1&viewPanel=7&refresh=1m&from=now-12h&to=now&var-Step=5min at peak we were seeing < 400 events in 5 mins for the mw.config variables (that's < 40,000 unsampled)

A rate of 2000 a minute (not 5) will trigger an alert, so we can probably still want to continue 1 or 2 wikis at a time.

Logged-out users and thus most site-wide traffic is entirely unrecorded in the mw-js-deprecate stats.

Good to know.

Updated @ 12:32 to include the correct sampling rate via @Krinkle, previously I was assuming a 1:10 not 1:100

In T72470#7003748, @Jdlrobson wrote:

According to https://grafana.wikimedia.org/d/000000037/mw-js-deprecate?orgId=1&viewPanel=7&refresh=1m&from=now-12h&to=now&var-Step=5min at peak we were seeing < 400 events in 5 mins for the mw.config variables (that's < 4000 unsampled)

Note that the sampling is 1:100, not 1:10. The sampling is also limited to logged-in users. Logged-out users and thus most site-wide traffic is entirely unrecorded in the mw-js-deprecate stats.

lots of mw-deprecate cases are actually edge case scripts iterating over window object and thus emitting a deprecation warning for every variable. It adds up really quickly.

I'll just disable the rest of wikis (enwiki, frwiki, dewiki) on Monday.

Change 680871 had a related patch set uploaded (by Ladsgroup; author: Ladsgroup):

[operations/mediawiki-config@master] Disable legacy javascript variable for the rest of wikis

https://gerrit.wikimedia.org/r/680871

Change 680871 merged by jenkins-bot:

[operations/mediawiki-config@master] Disable legacy javascript variable for the rest of wikis

https://gerrit.wikimedia.org/r/680871

Mentioned in SAL (#wikimedia-operations) [2021-04-19T11:05:31Z] <ladsgroup@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:680871|Disable legacy javascript variable for the rest of wikis (T72470)]] (duration: 00m 57s)

Change 681119 had a related patch set uploaded (by Jforrester; author: Jforrester):

[mediawiki/vagrant@master] Stop setting wgLegacyJavaScriptGlobals, ignored for 1.37+

https://gerrit.wikimedia.org/r/681119

Neat! Zero errors in the last 3 hrs (😲)

Thanks for pushing this to completion @Ladsgroup

I assume next step is to remove the code once 1.36 is finally released?

At least for case of enwiki. Its interface admins thoroughly cleaned the wiki too. Some of it might be due to caches not being expired yet but regardless I think the numbers are in a good shape.

In T72470#7014916, @Jdlrobson wrote:

I assume next step is to remove the code once 1.36 is finally released?

Sorry. Missed this one. I already made a patch https://gerrit.wikimedia.org/r/678716, given that 1.36 is already cut, the only thing we need to do is to just review that patch and merge it :D

Change 681119 merged by jenkins-bot:

[mediawiki/vagrant@master] Stop setting wgLegacyJavaScriptGlobals, ignored for 1.37+

https://gerrit.wikimedia.org/r/681119

Been a long road, thx to those who worked on this.

Change 678716 merged by jenkins-bot:

[mediawiki/core@master] resourceloader: Remove support for $wgLegacyJavaScriptGlobals

https://gerrit.wikimedia.org/r/678716

Change 681226 had a related patch set uploaded (by Krinkle; author: Krinkle):

[mediawiki/core@master] resourceloader: Move most of mw.log from startup to mediawiki.base

https://gerrit.wikimedia.org/r/681226

Change 820402 had a related patch set uploaded (by Lucas Werkmeister (WMDE); author: Lucas Werkmeister (WMDE)):

[operations/mediawiki-config@master] Remove unused $wgLegacyJavaScriptGlobals

https://gerrit.wikimedia.org/r/820402

Change 820402 merged by jenkins-bot:

[operations/mediawiki-config@master] Remove unused $wgLegacyJavaScriptGlobals

https://gerrit.wikimedia.org/r/820402

Mentioned in SAL (#wikimedia-operations) [2022-08-04T13:34:34Z] <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings.php: Config: [[gerrit:820402|Remove unused $wgLegacyJavaScriptGlobals (T72470)]] (1/2) (duration: 02m 58s)

Mentioned in SAL (#wikimedia-operations) [2022-08-04T13:37:54Z] <lucaswerkmeister-wmde@deploy1002> Synchronized wmf-config/InitialiseSettings-labs.php: Config: [[gerrit:820402|Remove unused $wgLegacyJavaScriptGlobals (T72470)]] (2/2) (duration: 02m 59s)