Server Admin Log: Difference between revisions

Browse history interactively

← Older edit Newer edit →

Content deleted Content added

VisualWikitext

Inline

Revision as of 01:24, 13 November 2016

2016-11-13

01:24 Krenair: Spoken to User:Nirzardp for T150554, set a new password

2016-11-12

19:53 tgr: deployed patch for T150554
18:42 Krenair: done with my shell-granted sysop flag on foundationwiki, have removed it
14:59 reedy@tin: Synchronized wmf-config/CommonSettings.php: Enable OATHAuth for all sysop, crat, oversight and checkuser (duration: 00m 47s)
14:33 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OATHAuth on fishbowl wikis, bump password requirements (duration: 00m 50s)
14:26 Reedy: Created OATHAuth tables on all fishbowl wikis
13:37 Krenair: `mwscript createAndPromote.php foundationwiki --sysop "Alex Monk (WMF)" --force` temporarily
02:21 l10nupdate@tin: ResourceLoader cache refresh completed at Sat Nov 12 02:21:40 UTC 2016 (duration 4m 29s)
02:17 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.2) (duration: 05m 30s)

2016-11-11

21:15 hashar: Restarted Jenkins. This time ZMQ managed to bind to port 8888
21:11 hashar: jenkins: disabled/reenabled the ZMQ Event Publisher. Apparently it refused to start
21:06 hashar: Restarted Jenkins
14:03 mobrovac: restarting RESTBase to pick up https://gerrit.wikimedia.org/r/#/c/320529/
14:02 moritzm: restarting hhvm on canary app servers to pick up libcurl update
13:11 moritzm: installing curl security updates
11:10 ema: cp3043 repooled with gethdr_extrachance=100 (T150503)
10:59 ema: cp3043 depooled, testing https://phabricator.wikimedia.org/P4406 (T150503)
10:51 elukey: restored mw1284 to its normal settings
10:14 marostegui: Deploy alter table dbstore1002 s4 commonswiki.revision - T147305
10:05 elukey: increasing apache log level on mw1284 (depooling, applying config manually, re-pooling with lower weight) for a 503 investigation
09:39 marostegui: Deploy schema change s4 commonswiki.revision db1069 - T147305
07:33 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Repool db1068 - T149079 (duration: 00m 48s)
02:28 l10nupdate@tin: ResourceLoader cache refresh completed at Fri Nov 11 02:28:24 UTC 2016 (duration 5m 14s)
02:23 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.2) (duration: 04m 56s)
01:45 mutante: gerrit now has higher "packedGitLimit" of 2g, goal is to reduce Gerrit slowdowns
01:39 mutante: gerrit restarting for config change 317322 (T148478)
01:04 godog: revert swift ring change for ms-be1027
00:23 godog: swift eqiad-prod: ms-be1027 to weight 1000 - T136631
00:18 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Remove registered trademark symbol from officewiki footer (T95007) (duration: 00m 48s)
00:15 catrope@tin: Synchronized wmf-config/InitialiseSettings.php: Enable as a noindex template on enwiki (2/2) (T149538) (duration: 00m 47s)
00:13 catrope@tin: Synchronized wmf-config/CommonSettings.php: Enable as a noindex template on enwiki (1/2) (T149538) (duration: 00m 49s)

2016-11-10

23:10 mutante: mw1185 - service hhvm restart
23:00 maxsem@tin: Finished scap: https://gerrit.wikimedia.org/r/#/c/320864/ (duration: 22m 42s)
22:38 maxsem@tin: Started scap: https://gerrit.wikimedia.org/r/#/c/320864/
22:35 maxsem@tin: scap sync-l10n completed (1.29.0-wmf.2) (duration: 01m 01s)
21:15 papaul: kafka2003 - signing puppet certs, salt-key, initial run
21:09 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.2
21:08 twentyafterfour@tin: Synchronized php-1.29.0-wmf.2/extensions/ORES/includes/ApiHooks.php: deploy I86e97b05b56b90d956616ef16e8aa86d96403b8c (duration: 00m 47s)
20:54 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: Revert "all wikis to 1.29.0-wmf.2" (Fatal errors spike)
20:49 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.2
20:33 mutante: neon - deactivated puppet node, scheduled icinga downtime, shutdown server permanently (T125023)
20:32 mutante: neon - shutdown -h now (scheduled 3 days downtime, nothing that looked worth saving in homes)
20:28 mutante: neon - deactivate puppet node
20:25 twentyafterfour@tin: Synchronized wmf-config/InitialiseSettings-labs.php: sync LABS: enable mapframe everywhere (I35709ed2903b28a2d4d6e8528ac1fcf361483e76) (duration: 00m 50s)
19:37 dcausse: elastic@eqiad: reindexing commonswiki (logs in terbium.eqiad.wmnet:~dcausse/commons_reindex/cirrus_log) - T150232
19:33 bblack: cache_*: restarting nginx for libssl update (seamless)
19:31 bblack: upgrading libssl1.1 to 1.1.0c on other misc hosts...
19:30 Pchelolo: update RESTBase to 6bfa0f75f
19:24 Pchelolo: update RESTBase to 6bfa0f75f - canary on restbase1007
19:22 bblack: upgrading openssl to 1.1.0c on cache_*
19:19 Pchelolo: update RESTBase to 6bfa0f75f - staging
19:16 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add PageViewInfo log channel (T129602) (duration: 00m 49s)
19:09 thcipriani@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Make beta PageViewInfo use the production pageview API (T129602) (labs-only-change) (duration: 00m 48s)
19:02 cwd: updated payments from c1fa73c649986be89a6a038b9e19f6b3ea19e537 to 3b3c8ce6c7bc24beb520a3e2f507df4514de4679
18:57 moritzm: uploaded openssl 1.1.0c for jessie-wikimedia to carbon
18:42 paravoid: upgrading (and restarting) nginx on sodium
17:46 urandom: T133395: Convert final 25 RESTBase tables to TWCS
17:34 cwd: deployed EVERYTHING... changed config tree to c6a7b17432188a0a4f990061b488208e64fe39dd and civicrm-authonly from 56eadabf705b3d035e11bb9fd3478457c159e40d to df50d2d7b6a8494fb3e049df6b1e101611270
17:33 cwd: restarted slander
17:32 cwd: restarted dash
16:58 urandom: T133395: Performing next 25 RESTBase table conversions to TWCS
15:09 marostegui: Deploy schema change s4 commonswiki.template links (db1068) - https://phabricator.wikimedia.org/T149079
15:01 elukey: restored mw1284 to its settings
14:53 jynus: applying schema change on s3 (page) T69223
14:47 elukey: de-pooling mw1284 to raise mod_proxy_fcgi log level manually (temporary for an ongoing investigation)
14:31 bblack: cache_text: upgrade nginx to 1.11.4-1+wmf14
14:20 bblack: cache_upload: upgrade nginx to 1.11.4-1+wmf14
14:04 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1068 - T149079 (duration: 00m 48s)
13:54 bblack: restarting varnishes on cache_maps + cache_misc
13:51 marostegui: Enabled gtid+ssl on db2010,db2012,db2030
13:45 bblack: cache_misc: upgrade nginx to 1.11.4-1+wmf14
13:29 bblack: cache_maps: upgrade nginx to 1.11.4-1+wmf14
13:25 marostegui: Restarting mysql in misc shard slaves (only codfw - db2010,db2012,db2030) to apply a MySQL config - T149418
13:20 gehel: restart wdqs1* for jvm update
13:07 gehel: restart wdqs2* for jvm update
12:01 ema: upgrading cp1068 (text-eqiad) to varnish 4 -- T131503
11:45 ema: upgrading cp1067 (text-eqiad) to varnish 4 -- T131503
11:25 ema: upgrading cp1066 (text-eqiad) to varnish 4 -- T131503
11:08 ema: upgrading cp1065 (text-eqiad) to varnish 4 -- T131503
10:43 ema: upgrading cp1055 (text-eqiad) to varnish 4 -- T131503
10:29 ema: upgrading cp1054 (text-eqiad) to varnish 4 -- T131503
10:26 _joe_: powercycling mw1280, unresponsive to ping, blank unresponsive console
10:25 moritzm: rolling restart of zookeeper in eqiad to pick up java security update
10:10 moritzm: rolling restart of zookeeper in codfw to pick up java security update
09:53 ema: upgrading cp1053 (text-eqiad) to varnish 4 -- T131503
09:44 marostegui: Deploy gtid_domain_id mysql flag for misc shards - https://phabricator.wikimedia.org/T149418
09:43 elukey: restarting druid daemons on druid100[123] for openjdk updates
09:38 marostegui@tin: Synchronized wmf-config/db-eqiad.php: wmf-config/db-codfw.php Depool db1059 - T149079. Repool db2048 T150334 (duration: 00m 50s)
09:22 dcausse: elastic@codfw: reindexing commonswiki (logs in wasat.codfw.wmnet:~dcausse/commons_reindex/cirrus_log)
09:06 moritzm: rolling restart of elasticsearch on logstash100[4-6] for picking up a Java security update
08:55 moritzm: rebooting ruthenium for kernel update
08:30 moritzm: rebooting copper for kernel update
08:06 moritzm: rebooting bast3001 for kernel update
08:01 marostegui: Deploy schema change s4 commonswiki.revision (dbstore1002) - T147305
04:28 mattflaschen@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Beta Cluster only (duration: 00m 51s)
04:26 mattflaschen@tin: Synchronized wmf-config/CommonSettings-labs.php: Beta Cluster only (duration: 00m 59s)
02:38 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Nov 10 02:38:24 UTC 2016 (duration 4m 36s)
02:33 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.2) (duration: 05m 52s)
02:19 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.1) (duration: 07m 23s)
00:46 bblack: nginx-1.11.4-1+wmf14 uploaded to carbon jessie-wikimedia (only deployed to cp1008 for now) - T93927 - T148917 - T144523
00:37 godog: remove files on iridium:/tmp older than 5d - T150396
00:14 maxsem@tin: Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/313211/ (duration: 00m 52s)
00:09 maxsem@tin: Synchronized wmf-config/CirrusSearch-labs.php: [labs only] https://gerrit.wikimedia.org/r/#/c/313037/ (duration: 00m 47s)
00:06 maxsem@tin: Synchronized wmf-config/CirrusSearch-labs.php: [labs only] https://gerrit.wikimedia.org/r/#/c/313037/ (duration: 00m 48s)

2016-11-09

23:38 godog: update cassandra aggregation scheme for 'count' metrics - T121789
23:25 godog: silence lutetium flapping check_mysql for two days
21:53 eileen1: jobs stopped, dedupe (*2) donations queue
21:21 eileen1: update CiviCRM from 7ee2ce47973c691e32d97ca6fed55df037cf585b to df50d2d7b6a8494fb3e049df6b1e101611270528
21:18 bearND: deployed mobileapps 106f4cd
21:14 bearND: starting mobileapps deploy
20:59 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.2
20:32 urandom: T133395: Converting next 25 RESTBase tables to time-window compaction
20:13 Krenair: created missing wikilove_log tables on azwiki and labtestwiki - T150321
20:05 Krinkle: Killed statsv.py process on hafnium. Seems to have fixed it.
20:05 Krinkle: statsv->graphite has been down for 9 hours since roughly 10AM UTC
19:51 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable <mapframe> on ruwiki (T138057) (duration: 00m 48s)
19:12 godog: upload cassandra-tools-wmf 1.0.0-1 to jessie-wikimedia on carbon - T150304
18:47 marostegui: Stopping MySQL dbstore2001 - taking a snapshot - T149457
18:37 jynus: partitioning db2042- it will have temporarily lag for 10-20 hours
18:36 mutante: neon stopping nsca and apache
18:33 mutante: neon (formerly icinga) remove from puppet, revoke cert, delete salt key, stop icinga service ...
18:33 godog: deploy python-thumbor-wikimedia 0.1.29 to thumbor100[12]
17:41 robh: the puppet failures on the frack hosts are known and have been reported to jeff
17:34 jynus: rebooting db2048 for kernel upgrade
16:18 ema: upgrading cp3043 (text-esams) to varnish 4 -- T131503
15:58 ema: upgrading cp3042 (text-esams) to varnish 4 -- T131503
15:47 urandom: T133395: Converting the next 25 RESTBase keyspaces to TWCS
15:34 ema: upgrading cp3041 (text-esams) to varnish 4 -- T131503
15:30 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2048 (duration: 00m 50s)
15:23 ema: upgrading cp3040 (text-esams) to varnish 4 -- T131503
15:11 ema: upgrading cp3033 (text-esams) to varnish 4 -- T131503
15:07 hashar: restarting Jenkins (java update)
14:59 gehel: clear zero sized log files on logstash* (leftover from disk space issues)
14:58 ema: upgrading cp3032 (text-esams) to varnish 4 -- T131503
14:56 zeljkof: EU SWAT finished
14:53 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Cleanup unused config variables (T148853) (duration: 00m 48s)
14:48 zfilipin@tin: Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Cleanup unused config variables (T148853) (duration: 00m 47s)
14:41 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: [cirrus] Activate BM25 on top 10 wikis: Step 3 (T147508) (duration: 00m 48s)
14:32 ema: upgrading cp3031 (text-esams) to varnish 4 -- T131503
14:29 zfilipin@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: [cirrus] Increase the number of shards to 15 for commonswiki_file (T148736) (duration: 00m 49s)
14:28 zfilipin@tin: Synchronized tests/cirrusTest.php: SWAT: [cirrus] Increase the number of shards to 15 for commonswiki_file (T148736) (duration: 00m 58s)
14:13 elukey: rebooting kafka1014.eqiad.wmnet for kernel and openjdk upgrades
14:11 ema: upgrading cp3030 (text-esams) to varnish 4 -- T131503
13:58 elukey: stopping kafka* daemons on kafka1014 to upgrade its fstab with UUID (T147879)
13:46 elukey: rebooting kafka1012 for kernel and openjdk updates
13:35 elukey: stopping kafka* daemons on kafka1012 to upgrade its fstab with UUID (T147879)
13:02 kart__: Update cxserver to 17f9deb
12:57 elukey: rebooting kafka1022 for kernel + openjdk updates
12:14 hashar: CI gate for MediaWiki is back. Reverted an oojs-ui version bump that triggered tests failure but was not caught properly by CI. T150323
11:17 hashar: CI gate for MediaWiki fails tests. On it. See https://phabricator.wikimedia.org/T150323
11:10 moritzm: rebooting mw1162 for kernel update
11:09 ema: finished upgrading cache_text ulsfo to varnish 4.1.3-1wm3 T150247
11:08 hashar: contint1001 apt-get upgrade packages and purging unneeded ones (left over from a puppet manifest that is no more applied)
10:52 elukey: restarting kafka* on kafka1013 for openjkd upgrades
10:38 mobrovac: change-prop deploying e0040ac
10:33 elukey: rebooting kafka1020 for kernel and openjdk upgrades
10:21 moritzm: rebooting nescio for kernel update
10:11 jynus: stopping and reimaging db2042 for upgrade
10:04 ema: upgrading cache_text ulsfo to varnish 4.1.3-1wm3 T150247
10:03 jynus@tin: Synchronized wmf-config/db-codfw.php: Depool db2042 (duration: 00m 48s)
09:55 moritzm: rebooting maerlant for kernel update
09:35 moritzm: rebooting hydrogen for kernel update
09:35 elukey: rebooting kafka1018 for kernel + openjdk upgrade
08:43 moritzm: rebooting bast1001 for kernel update
08:25 moritzm: rolling reboot of logstash1002/1003 for kernel update
08:16 moritzm: restarted ntp on mw2128 (was stuck in XFAC state)
04:12 moritzm: rebooting notebook1001/1002 for kernel update
03:47 moritzm: installing java security updates on meitnerium/archiva
03:44 apergos: rolling reboots of mw2097-2134 for new kernel
03:44 mutante: gallium.wikimedia.org removed from DNS (T95757)
02:56 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Nov 9 02:56:22 UTC 2016 (duration 5m 24s)
02:51 moritzm: uploaded libuv 1.9.0 for jessie-wikimedia to carbon
02:50 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.2) (duration: 10m 39s)
02:24 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.1) (duration: 05m 42s)

2016-11-08

23:25 eileen1: update CiviCRM from 63d35fc5f5c065168efd2250c2741b3603b2eb92 to 7ee2ce47973c691e32d97ca6fed55df037cf585b
22:40 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.29.0-wmf.2
22:34 twentyafterfour@tin: Finished scap: testwikis to 1.29.0-wmf.2 (duration: 49m 32s)
22:22 moritzm: rebooting ms1001 for kernel update
22:01 moritzm: rolling reboot of mc2* for kernel update
21:56 Pchelolo: RESTBase update to 1d72b8abc
21:44 twentyafterfour@tin: Started scap: testwikis to 1.29.0-wmf.2
21:40 mutante: gallium, ex-CI server, shutdown -h now (the contents of your home dir have been copied to contint1001 in /home/gallium-home/) (T95757)
21:39 mutante: gallium, ex-CI server, shutdown -h now (the contents of your home dir have been copied to contint1001 in /home/gallium-home/)
21:30 Pchelolo: RESTBase update to 1d72b8abc - canary on restbase1007
21:21 Pchelolo: RESTBase update to 1d72b8abc - staging
20:55 godog: upload prometheus-memcached-exporter 0.3.0+ds1-1 to carbon - T147326
20:31 apergos: rolling restarts of mw1218-1222 for new kernel
20:09 mobrovac: change-prop deploying 0c29003
19:39 ejegg: updated fundraising tools from 7ff719a466bb9ecbdb5f444f67d67903456f6fdb to d14d47a83bd822d28da0f2d03afbd74008e215a1
19:29 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1089 (duration: 00m 48s)
19:22 urandom: T133395: Converting 25 additional RESTBase tables to TWCS
19:19 thcipriani@tin: Synchronized wmf-config/CommonSettings-labs.php: SWAT: LABS: fixed incorrect $wgGraphAllowedDomains (housekeeping sync) (duration: 02m 42s)
19:12 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Set timezone for bdwikimedia to "Asia/Dhaka" (T150252) (duration: 00m 47s)
18:59 apergos: rolling reboots of mw1180-1188 for new kernel
18:12 jynus: performing schema change templatelinks on db1089 T139090
18:09 apergos: rolling reboots of mw1170-1179 for new kernel
18:06 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1089 to safely apply pending schema change (duration: 01m 02s)
17:54 jynus@tin: Synchronized wmf-config/db-eqiad.php: Repool db1080 (duration: 02m 45s)
17:41 Krinkle: mwscript deleteEqualMessages.php --wiki ptwikinews (T45917)
17:39 reedy@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Fix variable typo (duration: 00m 59s)
17:27 apergos: rolling restarts of mw1209 - mw1216 for new kernel
17:20 reedy@tin: Synchronized wmf-config/CommonSettings-labs.php: Graphs config (duration: 00m 47s)
17:19 reedy@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Enable Revision Slider (duration: 00m 47s)
17:19 ema: upgrade finished -> cache_text codfw to varnish 4.1.3-1wm3 T150247
17:15 reedy@tin: Synchronized wmf-config/CommonSettings-labs.php: Add PageViewInfo, Remove dupe OATHAuth config (duration: 00m 47s)
17:13 reedy@tin: Synchronized wmf-config/InitialiseSettings-labs.php: Add PageViewInfo (duration: 00m 46s)
17:12 reedy@tin: Synchronized wmf-config/extension-list-labs: Add PageViewInfo (duration: 00m 46s)
16:31 apergos: rolling restart of mw1204-1208 for new kernel
16:24 jynus: performing schema change templatelinks on db1080 T139090
16:21 Krinkle: mwscript deleteEqualMessages.php --wiki kkwiki (T45917)
16:18 ema: upgrading cache_text codfw to varnish 4.1.3-1wm3 T150247
16:11 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1052; depool db1080; reorganize trafic weight for s1 -second try (duration: 00m 46s)
16:04 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool db1052; depool db1080; reorganize trafic weight for s1 (duration: 00m 46s)
15:23 hashar@tin: Synchronized php-1.29.0-wmf.1/extensions/Kartographer/modules/maplink/maplink.js: Search .mw-body instead of #content to support all the skins - T150148 (duration: 00m 47s)
14:09 moritzm: rebooting chromium for kernel update
13:58 hashar: European SWAT on hold while some memcached/elasticsearch issues are being figured out
13:42 apergos: deferring reboots of mw1204-1216 and mw1170-1188 for a while
13:06 moritzm: rolling reboot of restbase-test for kernel update
12:52 ema: upgrading pinkunicorn to varnish 4.1.3-1wm3 T150247
12:41 moritzm: restarted ntp on mw1194, stuck in XFAC state
12:39 moritzm: rebooting iron for kernel update
12:28 moritzm: restarted ntp on mw1166, stuck in XFAC state
12:24 moritzm: depooling/rebooting/repooling scb1002 for kernel update
12:23 moritzm: restarted ntp on hafnium, stuck in XFAC state
12:12 moritzm: depooling/rebooting/repooling scb1001 for kernel update
12:11 jynus@tin: Synchronized wmf-config/db-eqiad.php: Pool back db1051 and api servers to high load after hw issues (duration: 02m 45s)
11:59 apergos: rolling restart of mw1170-1216 for new kernel
11:54 apergos: restart of mw1240, 1253 for new kernel
11:44 moritzm: rebooting logstash1001 for kernel update
11:42 moritzm: rearmed keyholder on mira
11:38 apergos: rolling reboot of mw1161, mw1163-1169 for new kernel
11:37 jynus: running schema change on db1045 (pagelinks) T139090
11:30 moritzm: rebooting mira for kernel update
11:19 apergos: rolling restart of mw2080-2085 for new kernel
11:15 jynus: running schema change on db2070 (pagelinks) T139090
11:02 mark: Activated cr2-eqiad bgp group IX4
10:53 apergos: rolling reboot of mw2090 - mw2096 for new kernel
10:41 moritzm: rearmed keyholder on tin
10:39 apergos: rebooting mw2086 - mw2089 for new kernel
10:36 jynus: rebooting and upgrading db2012
10:33 moritzm: rebooting tin for kernel update
10:23 moritzm: restarted ntp on mw2075, stuck in XFAC state
10:22 moritzm: rebooting hafnium for kernel update
10:08 marostegui@tin: Synchronized wmf-config/db-eqiad.php: Depool db1059 - T149079 T147305 (duration: 00m 57s)
10:06 moritzm: rebooting rhenium for kernel update
09:59 moritzm: rebooting oxygen for kernel update
09:57 apergos: rebooting mw2075 - mw2079 for new kernel
09:49 moritzm: rebooting install2001 for kernel update
09:42 moritzm: rebooting bast2001 for kernel update
09:24 moritzm: rebooting graphite1002 for kernel update
08:44 marostegui: Deploy schema change s4 commonswiki.revision table - T147305
08:21 moritzm: rolling reboot of swift backend servers in esams for kernel update
08:09 moritzm: rolling reboot of parsoid in eqiad for kernel update
08:04 elukey: rebooting stat1001 for kernel upgrades (will cause a brief unavail for analytics websites)
07:30 marostegui: Deploy schema change s5 dewiki.revision on codfw master (db2023) - T148967
07:10 _joe_: stopped logstash, removed large logfiles that were erroneously non-rotated, started logstash across the logstash cluster
02:31 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Nov 8 02:31:30 UTC 2016 (duration 4m 16s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.1) (duration: 09m 52s)
01:48 eileen1: update CiviCRM from 45a1b9a0e1665b1fc71165e1bc4bcfdca2a9adf7 to 63d35fc5f5c065168efd2250c2741b3603b2eb92
01:27 eileen1: enabled GlobalCollect Recurring Donations Dedupe CiviCRM Contacts GlobalCollect audit file download Fundraiser Public Data Export
01:23 eileen1: enabled Thank you mail send
01:17 eileen1: disabled Public Data Export
01:06 eileen1: disable GlobalCollect audit file
01:05 eileen1: disable jobs http://localhost:9000/job/Dedupe%20CiviCRM%20contacts/ http://localhost:9000/job/Dedupe%20CiviCRM%20contacts%20(name-match)/ Thank you mail send
01:04 eileen1: disable jobs GlobalCollect Recurring Donations
00:51 godog: swift eqiad-prod: set weight for ms-be1021 sd[h-n] to 3000 - T139767
00:34 dereckson@tin: Synchronized wmf-config/throttle.php: Nashville Science edit-a-thon (Vanderbilt library) (T150207) (duration: 00m 47s)
00:12 dereckson@tin: Synchronized php-1.29.0-wmf.1/extensions/CentralNotice/: Handle banner loader errors on client (T149107) (duration: 00m 49s)
00:03 Dereckson: projectcom.wikimedia.org wiki creation done

2016-11-07

23:55 Dereckson: Created 'Mjohnson (WMF)' user account on projectcom.wikimedia.org as bureaucrat
23:55 godog: delete parsoid from releases.wikimedia.org and varnish-ban on cache_misc
23:50 Krinkle: mwscript deleteEqualMessages.php --wiki jawikinews (T45917)
23:44 Dereckson: Created storage container for projectcomwiki (private)
23:43 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Initial configuration for projectcom.wikimedia.org (duration: 00m 53s)
23:42 dereckson@tin: rebuilt wikiversions.php and synchronized wikiversions files: Added projectcomwiki
23:41 dereckson@tin: Synchronized dblists/: Added projectcomwiki (duration: 00m 48s)
23:35 Dereckson: projectcomwiki database created
23:32 Krinkle: mwscript deleteEqualMessages.php --wiki jawikibooks (T45917)
23:22 Dereckson: Starting projectcom.wikimedia.org wiki creation
23:22 Dereckson: ec.wikimedia.org wiki creation done
23:19 Dereckson: Created storage container for ec.wikimedia (private)
23:16 dereckson@tin: Synchronized wmf-config/interwiki.php: Update interwiki map for vote. and ec.wikimedia (Gerrit:320308) (duration: 00m 47s)
23:15 Krinkle: mwscript deleteEqualMessages.php --wiki gawiktionary (T45917)
22:51 reedy@tin: Synchronized php-1.29.0-wmf.1/includes/specials/: Deploy security fix T150044 (duration: 00m 54s)
22:44 reedy@tin: Synchronized wmf-config/CommonSettings.php: Set wgOATHAuthAccountPrefix and Don't override message key in badpass log entries (duration: 00m 47s)
22:39 mutante: Un nuevo wiki ha nacido. Bienvenido grupo de usuarios Ecuador Wikimedia. https://ec.wikimedia.org (T135521)
22:35 dereckson@tin: Synchronized multiversion/MWMultiVersion.php: Add ec.wikimedia to MWMultiVersion (T135521) (duration: 00m 49s)
22:25 Dereckson: Created tables for OATHAuth on ec.wikimedia
22:22 dereckson@tin: Synchronized static/images/project-logos/: Logos for ec.wikimedia (T135521) (duration: 00m 48s)
22:20 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: ec.wikimedia initial configuration (T135521) (duration: 00m 47s)
22:18 mutante: gallium - stopped apache, stopped salt, removed zuul cronjob
22:18 dereckson@tin: rebuilt wikiversions.php and synchronized wikiversions files: (no message)
22:17 dereckson@tin: Synchronized dblists: (no message) (duration: 00m 53s)
22:15 mutante: gallium - delete salt key, minion is stopped
22:10 mutante: gallium - revoke puppet cert, deactivate node
22:03 Dereckson: Starting ec.wikimedia.org wiki creation
21:50 gehel: deploying latest wdqs gui and blazegraph
21:38 bearND: deployed mobileapps 4202cbb
21:37 Amir1: ores deployment c61b9c1 is done
21:30 bearND: starting mobileapps deploy
21:29 arlolra: updated Parsoid to version 2c2fe425
21:18 arlolra: starting Parsoid deploy
21:12 Amir1: deploying c61b9c1 from ORES to all nodes (T149730)
21:09 Amir1: deploying c61b9c1 from ORES into canary nodes (T149730)
20:44 urandom: T133395: restbase2001-b.codfw.wmnet: Performing user-defined compaction of la-169239-big-Data.db and la-172629-big-Data.db
20:38 jynus: upgrading new labsdbs to mariadb 10.1.19
20:32 bblack: repooling cp4018 (done experimenting)
20:21 mutante: projectcom.wikimedia.org created in DNS (T143138)
20:14 godog: cmjohnson1 is performing work on LVS in row D, there might be flaps
19:58 ejegg: updated payments-wiki from ed98772ead6365d58356294ddd46bc4312204b1d to c1fa73c649986be89a6a038b9e19f6b3ea19e537
19:42 ejegg: updated civicrm from bdc2786ddaf09e9f412f97406e3cffb13fcc96ab to 45a1b9a0e1665b1fc71165e1bc4bcfdca2a9adf7
18:00 urandom: T133395: Convert local_group_*_title__revisions.{data,idx_by_rev_ever} tables to time-window compaction
17:56 bd808@tin: Synchronized php-1.29.0-wmf.1/includes/exception/MWExceptionHandler.php: MWExceptionHandler: Do not use 'exception' for custom log data (T150106) (duration: 00m 47s)
17:40 jynus: performing schema change on s7 (imagelinks) T139090
16:01 mark: Reactivated cr2-eqiad IX6 BGP group (ipv6 sessions)
16:00 mark: Chris moved cr2-eqiad:xe-5/3/3 to xe-3/3/3
15:54 mark: Disabling cr2-eqiad BGP groups IX4/IX6 (all Equinix Ashburn BGP sessions)
15:44 elukey: started kafka-mirror-main-eqiad_to_analytics.service on kafka1012
15:39 moritzm: rebooting radium for kernel update
15:38 mark: Reenabling OSPF/OSPF3 on cr2-codfw:xe-5/0/1 after eqiad side port move to xe-3/2/3
15:31 mark: Disabling OSPF/OSPF3 on cr2-codfw:xe-5/0/1 for eqiad side port move
15:26 elukey: rebooting kafka1013 for kernel upgrades
15:22 marostegui@tin: Synchronized wmf-config/db-codfw.php: Repool db2042 - T149553 (duration: 00m 49s)
15:19 hashar: Restarting Jenkins (deadlock in beta cluster Jenkins jobs)
15:14 mark: Reactivate cr2-eqiad BGP peering with pfw1-eqiad
15:13 mark: Chris moved cr2-eqiad:xe-5/0/3 to xe-3/3/2
15:10 hashar: European SWAT completed
15:09 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: shortUrl for bdwikimedia and tcywiki T146014 and T150166 (duration: 01m 51s)
15:08 mark: Deactivate cr2-eqiad BGP peering with pfw1-eqiad
15:07 marostegui: Enabling gtid_domain_id db1020 (m2 master) - T149418
15:07 mark: Reactivate cr1-eqiad BGP peering with pfw1-eqiad
15:05 mark: Chris moved cr1-eqiad:xe-5/0/3 to xe-3/3/2
15:03 hashar: T146014 mwscript extensions/ShortUrl/populateShortUrlTable.php --wiki=bdwikimedia (714 titles done)
15:02 hashar: T150166 mwscript extensions/ShortUrl/populateShortUrlTable.php --wiki=tcywiki (1569 titles done)
15:02 moritzm: rebooting mw1261-mw1265 (canary app servers) for kernel update
15:01 hashar: T146014 mwscript sql.php --wiki=bdwikimedia /srv/mediawiki/php-1.29.0-wmf.1/extensions/ShortUrl/schemas/shorturls.sql
15:01 hashar: T150166 mwscript sql.php --wiki=tcywiki /srv/mediawiki/php-1.29.0-wmf.1/extensions/ShortUrl/schemas/shorturls.sql
15:00 mark: Deactivate cr1-eqiad BGP peering with pfw1-eqiad
14:49 hashar: terbium: scap pull to add shortUrl tables to bdwikimedia and tcywiki
14:42 hashar: fawiki: renaming user group 'autopatrol' to 'autopatrolled' for T139246 and T144699 with: mwscript migrateUserGroup.php --wiki=fawiki 'autopatrol' 'autopatrolled'
14:42 hashar: fawiki Done! 417 users in group 'autopatrol' are now in 'autopatrolled' instead.
14:40 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Rename 'autopatrol' to 'autopatrolled' on fawiki - T144699 T139246 (duration: 00m 47s)
14:33 gehel: reboot maps-test* for kernel upgrade
14:30 hashar@tin: Synchronized wmf-config: (no message) (duration: 00m 53s)
14:10 hashar@tin: Synchronized php-1.29.0-wmf.1/extensions/Kartographer/extension.json: Fix monobook <maplink> (missing debounce dep) T145521 (duration: 00m 47s)
13:56 gehel: reboot wdqs1* for kernel upgrade
13:52 bblack: depooling cp4018 nginx+varnish-fe services for debugging
13:36 gehel: reboot wdqs2* for kernel upgrade
13:34 hashar: Flushed nodepool instances. It is bringing up fresh one now.
13:26 moritzm: rebooting labnodepool1001 for kernel update
13:19 hashar: shutting down Nodepool (labnodepool1001.eqiad.wmnet reboot)
13:06 moritzm: rebooting scandium for kernel update
12:09 jynus: performing schema change on s6 (imagelinks) T139090
12:00 moritzm: rebooting wtp1001 for kernel update
11:40 ema: cp3043: repool varnish-be and varnish-be-rand (T149881)
11:33 moritzm: rebooting cassandra test hosts (cerium, praseodymium, xenon) for kernel update
10:49 moritzm: rebooting mw1017/mw1099 for kernel update
10:26 moritzm: rebooting cp1008 for kernel update
10:19 moritzm: rebooting bast4001 for kernel update
10:07 jynus: performing schema change on s5 (imagelinks) T139090
08:46 moritzm: uploaded linux-meta 1.11 to carbon (pointing to the new Linux ABI package)
08:44 marostegui: stopping mysql on db2042 - maintenance- T149553
08:39 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2042 for maintenance - T149553 (duration: 00m 50s)
08:30 marostegui: Deploy schema change on s4 master (db2019) commonswiki.revision - T147305
07:02 _joe_: removing old logfiles on logstash hosts
02:21 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Nov 7 02:21:02 UTC 2016 (duration 4m 18s)
02:16 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.1) (duration: 05m 39s)

2016-11-06

22:13 Dereckson: Run namespacesDupe maintenance script on gl.wikisource (T150143)
10:13 elukey: removing logstash.log.1 from logstash100[123] to free some space
02:22 l10nupdate@tin: ResourceLoader cache refresh completed at Sun Nov 6 02:22:41 UTC 2016 (duration 4m 30s)
02:18 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.1) (duration: 06m 03s)

2016-11-05

21:40 bd808: Deleted huge logstash1003:/var/log/logstash/logstash.log.1 log file; disk full
21:39 bd808: Deleted huge logstash1002:/var/log/logstash/logstash.log.1 log file; disk full
21:36 bd808@tin: Synchronized wmf-config/InitialiseSettings.php: logstash: Temporarily disable EventBus channel (T150106) (duration: 00m 50s)
19:54 bd808: ELK stack problems are related to Elasticsearch index mapping. Some events are being rejected for not matching the expected mappings and that is filling up the disk on the logstash injestion hosts
19:45 bd808: Forced several puppet runs on logstash1001 until things stopped changing; out of disk seemed to have messed up apt upgrades
19:38 bd808: Elasticsearch on logstash1001 won't restart due to missing /etc/elasticsearch/scripts directory
19:23 bd808: Restarted logstash on logstash1001
19:14 bd808: Deleted huge logstash1001:/var/log/logstash/logstash.log.1 log file; disk full and difficult to debug with no free space on /
02:21 l10nupdate@tin: ResourceLoader cache refresh completed at Sat Nov 5 02:21:37 UTC 2016 (duration 4m 36s)
02:17 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.1) (duration: 05m 25s)

2016-11-04

22:43 godog: stop puppet on einsteinium and tegment to avoid log spam - T150061
21:14 urandom: T133395: Starting user-defined compaction of local_group_wikipedia_T_parsoid_html.data, files la-169018-big-Data.db and la-171488-big-Data.db
21:06 godog: compress huge daemon.log on einsteinium into /srv/
18:11 moritzm: uploaded new jessie linux package based on 4.4.30 to carbon
18:01 paravoid: moving mc1033-mc1036 from asw-d-eqiad to asw2-d-eqiad
17:54 paravoid: reactivating cr1-eqiad:ae4 and its subinterfaces (VRRP bug seems to have been worked around)
17:44 paravoid: moved cr1-eqiad:ae4 links from asw-d-eqiad:ae1 to to asw2-d-eqiad:ae1
16:38 ema: upgrading cp4018 (text-ulsfo) to varnish 4 -- T131503
16:22 ema: upgrading cp4017 (text-ulsfo) to varnish 4 -- T131503
16:01 ema: upgrading cp4016 (text-ulsfo) to varnish 4 -- T131503
15:37 ema: upgrading cp4010 (text-ulsfo) to varnish 4 -- T131503
15:36 paravoid: set up 4x10G (ae0) links between asw-d-eqiad<->asw2-d-eqiad
15:35 marostegui: reimage dbstore2002 - T150017
15:20 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Remove wikitech bot group (duration: 00m 47s)
15:17 reedy@tin: Synchronized wmf-config/CommonSettings.php: Simplify some wikitech config (duration: 00m 47s)
15:16 reedy@tin: Synchronized wmf-config/wikitech.php: Stop double loading OATHAuth now, remove commented config (duration: 00m 47s)
15:15 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Normalise wikitech OATHAuth loading config (duration: 00m 48s)
15:06 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OATHAuth on all private wikis (duration: 00m 49s)
15:04 reedy@tin: Synchronized wmf-config/CommonSettings.php: Raise password requirements for private wikis, Abuse filter editors on enwiki, and make minimum bot password length to 8 (duration: 00m 47s)
15:02 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: stage wmgElevateDefaultPasswordPolicy (duration: 00m 48s)
14:49 ema: upgrading cp4009 (text-ulsfo) to varnish 4 -- T131503
14:14 ema: upgrading cp4008 (text-ulsfo) to varnish 4 -- T131503
11:33 mobrovac: restarting zotero
10:58 moritzm: installing tar security updates
10:21 moritzm: upgrading memcached on swift frontend servers in esams
10:00 jynus: stopping db2011 for backup and reimage
09:59 moritzm: upgrading memcached on swift frontend servers in codfw
09:54 moritzm: upgrading memcached on jessie graphite systems
09:26 _joe_: rebooting copper to allow enabling the memory cgroup
09:10 marostegui: Reimage db2034 - T149553
07:20 jynus: disabling alerting for slave lag fleet-wide for 1 hour to deploy new alerting script
06:52 _joe_: restarted manually varnish text-backend on cp3041 - failing automatic restarts with "no space left on device"
02:31 l10nupdate@tin: ResourceLoader cache refresh completed at Fri Nov 4 02:31:26 UTC 2016 (duration 4m 39s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.1) (duration: 09m 08s)
01:54 madhuvishy: Manually reimaging labstore2003 (T149870)
01:35 catrope@tin: Synchronized php-1.29.0-wmf.1/extensions/Thanks: Avoid breakage after Flow uninstallation (duration: 00m 47s)
01:18 catrope@terbium: scap failed: IOError [Errno 13] Permission denied: u'/srv/mediawiki-staging/wmf-config/ExtensionMessages-1.29.0-wmf.1.php' (duration: 00m 20s)
01:17 catrope@terbium: Started scap: (no message)
00:48 catrope@tin: Synchronized dblists/: Disable Flow on enwiki (T148611) (duration: 01m 04s)

2016-11-03

23:23 thcipriani@tin: Synchronized php-1.29.0-wmf.1/extensions/EventBus/EventBus.php: SWAT: Add logging and check for empty JSON encoded body (T148251) (duration: 00m 47s)
23:11 thcipriani@tin: Synchronized portals: SWAT: Bumping portals to master (T146807) (duration: 00m 52s)
23:10 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T146807) (duration: 00m 51s)
22:16 ejegg: updated CiviCRM from 4ff64afd1ac10643f1f2c91c4aa7a5535512c33e to bdc2786ddaf09e9f412f97406e3cffb13fcc96a
20:33 urandom: T133395: Enabling unchecked_tombstone_compaction and setting tombstone_threshold = .6 on "local_group_wikipedia_T_parsoid_html".data
20:26 bblack: codfw cache_text - all nodes v4 and pooled - T131503
20:10 bblack: codfw cache_text - all pooled nodes are v4 (2x still depooled-but-upgraded) - T131503
20:04 ejegg: enabled donations queue consumer
20:02 ejegg: disabled donations queue consumer
20:01 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.29.0-wmf.1
19:32 ejegg: updated CiviCRM from 954bab421fa3435c5dd1a80f59f657407764b620 to 4ff64afd1ac10643f1f2c91c4aa7a5535512c33e
19:31 mobrovac: change-prop deploying f107669
19:28 twentyafterfour@tin: Synchronized php-1.29.0-wmf.1/extensions/PageImages/includes/ApiQueryPageImages.php: T149849 (duration: 00m 47s)
19:23 mobrovac: restbase restarting to re-include wikidata domains for T149114
19:17 Krenair: <twentyafterfour> !log In order to unblock the train for group2: deploying https://gerrit.wikimedia.org/r/#/c/319643/ refs T149059, T149849
18:46 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix comment refs (T148327) (duration: 00m 47s)
18:39 thcipriani@tin: Synchronized wmf-config: SWAT: LABS: Enable Map (GeoJSON) data on Commons (T149548) (housekeeping only sync) (duration: 00m 50s)
18:33 thcipriani@tin: Synchronized php-1.29.0-wmf.1/extensions/Kartographer/includes/ApiQueryMapData.php: SWAT: Fix warning (T149923) (duration: 00m 47s)
18:26 ema: repooling cp2016 (T131503)
18:21 thcipriani@tin: Synchronized php-1.28.0-wmf.23/extensions/EventBus/EventBus.php: SWAT: Log more EventBus HTTP request/response context for HTTP errors (T148251) (duration: 00m 52s)
18:19 thcipriani@tin: Synchronized php-1.29.0-wmf.1/extensions/EventBus/EventBus.php: SWAT: Log more EventBus HTTP request/response context for HTTP errors (T148251) (duration: 00m 49s)
18:02 hashar: Added security rule for "puppet3-diffs" labs project to allow ssh connection from contint1001 instead of gallium
16:20 jynus: stopping and upgrading labsdb1009,10,11 (also disabling temporarily puppet)
16:16 papaul: OS install on labstore2001
16:13 mutante: mw1205 - service hhvm restart
15:59 jynus@tin: Synchronized wmf-config/db-eqiad.php: mariadb: Reduce db1051 load, it has hardware issues (duration: 00m 47s)
15:41 moritzm: installing memcached security updates on graphite hosts
15:39 mobrovac: scb in eqiad enabled puppet back
15:26 mobrovac: scb in eqiad disabling puppet
14:57 akosiaris: failover icinga from tegmen to einsteinium
14:17 moritzm: exim reenabled on fermium after mailman update
14:14 moritzm: temporarily stop exim on fermium for mailman update
14:08 bblack: cache_text: nginx lossless restarts for libssl update - T144626 - T148917
14:00 mobrovac: restbase rolling restart for T149114
13:56 bblack: cache_upload: nginx lossless restarts for libssl update - T144626 - T148917
13:49 bblack: cache_maps + cache_misc: nginx lossless restarts for libssl update - T144626 - T148917
13:42 bblack: cp*: upgrade libssl1.1 to 1.1.0b-1+wmf2 (but no nginx restart yet) - T144626 - T148917
13:36 bblack: cp1065: upgrade libssl1.1 to 1.1.0b-1+wmf2 - T144626 - T148917
13:31 mobrovac: change-prop deploying a1bd739
13:06 hashar@tin: Synchronized wmf-config/CommonSettings.php: Add wmgRevisionSliderBetaFeature (default true) (duration: 00m 46s)
13:05 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: Add wmgRevisionSliderBetaFeature (default true) (duration: 00m 47s)
13:04 paravoid: re-enabling cr2-esams:xe-0/1/3 + cr2-eqiad:xe-4/1/3 (esams-eqiad link)
12:09 marostegui: Deploying schema change s4 commonswiki.revision only codfw - https://phabricator.wikimedia.org/T147305
11:54 mobrovac: change-prop deploying 15eae87
11:52 mobrovac: restbase deploy end of 1ec3b129
11:48 akosiaris: reenable puppet across the fleet on hosts that I had disabled it. https://gerrit.wikimedia.org/r/#/c/316032/1 merged successfully
11:32 mobrovac: restbase deploy start of 1ec3b129
11:16 ema: depooling cp2016, cp2007, cp2019, cp2023: not caching properly (T131503)
10:55 akosiaris: disable puppet throughout the fleet. merging https://gerrit.wikimedia.org/r/#/c/316032/1
09:37 moritzm: uploaded openssl 1.1.0b-1+wmf2 for jessie-wikimedia to apt.wikimedia.org (adding the read_ahead bugfix and dropping the chapoly_pref patch)
09:36 marostegui: Deploy schema change s5 dewiki.revision - only codfw https://phabricator.wikimedia.org/T148967
09:24 ema: upgrading cp2016 (text-codfw) to varnish 4 -- T131503
09:17 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: scb2003.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=apertium'])
09:17 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: scb2004.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=apertium'])
09:16 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=apertium'])
09:16 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=apertium'])
09:00 hashar: contint1001: preliminary transfer of jenkins history from gallium using rsync
08:51 hashar: gallium: unmounted /var/lib/jenkins/tmpfs freeing 512MBytes. Artifact from the past freeing up 512MBytes of memory
08:47 ema: upgrading cp2007 (text-codfw) to varnish 4 -- T131503
08:01 ema: upgrading cp2019 (text-codfw) to varnish 4 -- T131503
07:56 jynus: restarting replication codfw -> eqiad on s1
07:54 ema: repool cp2019 varnish-be, currently depooled for no valid reason
07:51 jynus: stopping mysql on db1042
07:42 ema: upgrading cp2023 (text-codfw) to varnish 4 -- T131503
07:38 _joe_: rolling restart of pybal in esams
07:30 _joe_: restarting pybal on lvs2005
07:18 jynus: stopping and debugging db1073
06:39 yuvipanda: attempting manual re-image of labstore2004
06:28 jynus@tin: Synchronized wmf-config/db-codfw.php: Remove references to db1042 (duration: 00m 46s)
06:22 jynus@tin: Synchronized wmf-config/db-eqiad.php: Depool db1052; Depool db1073; Remove references to db1042 (duration: 00m 47s)
03:44 Krenair: wikitech-static package updates
03:05 l10nupdate@tin: ResourceLoader cache refresh completed at Thu Nov 3 03:05:05 UTC 2016 (duration 5m 36s)
02:59 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.1) (duration: 11m 28s)
02:30 l10nupdate@tin: scap sync-l10n completed (1.28.0-wmf.23) (duration: 12m 08s)
00:28 dereckson@tin: Synchronized php-1.29.0-wmf.1/resources/src/mediawiki.widgets/mw.widgets.TitleWidget.js: Follow-up Id0021594: Remove extra code for redlink suggestions (T149130) (duration: 00m 46s)
00:23 dereckson@tin: Synchronized php-1.29.0-wmf.1/extensions/Kartographer/styles/: Set font size to 14px for both static and interactive maps (T149860) (duration: 00m 47s)

2016-11-02

23:44 bblack: disabling port xe-4/1/3 on cr2-eqiad (wave to esams, level3, other side of earlier disable)
23:37 bblack: disabling port xe-0/1/3 on cr2-esams (wave to eqiad, level3)
22:45 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/319447/ (duration: 00m 47s)
22:10 maxsem@tin: Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/319415/ (duration: 00m 47s)
21:52 ejegg: updated fundraising tools from f83e39291adc55677fc4b49307dc4807eba18019 to 7ff719a466bb9ecbdb5f444f67d67903456f6fdb
21:46 hashar: Restarting Jenkins due to deadlock with the beta cluster jobs
21:41 maxsem@tin: Synchronized php-1.29.0-wmf.1/extensions/Kartographer/: https://gerrit.wikimedia.org/r/#/c/319411/1 (duration: 00m 49s)
21:35 bblack: pooling cp4016 - T149843
21:09 bblack: pooling cp1055 - T149843
21:09 twentyafterfour@tin: Synchronized php-1.29.0-wmf.1/includes/parser/CoreParserFunctions.php: Deploy https://gerrit.wikimedia.org/r/#/c/319400/ refs T149840, T149059 (duration: 00m 51s)
20:49 mdholloway: deployed mobileapps 0ced96c
20:46 mdholloway: starting mobileapps deployment
20:35 bblack: depool cp1055 - T149843
20:34 bblack: depool cp4016 - T149843
20:27 arlolra: updated Parsoid to version 173d7e32 (T149241, T119228, T141723, T141905, T147742, T48580, T133320)
20:09 arlolra: starting Parsoid deploy
20:04 chasemp: maintain-views --databases tcywiki --debug on labsdb1001 and 1003
20:03 chasemp: maintain-views --databases wikimania2017wiki --debug on labsdb1001 and 1003
19:51 twentyafterfour@tin: rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.29.0-wmf.1
19:47 chasemp: maintain-views --databases olowiki on labsdb1001 and 1003 to create view
19:16 XenoRyet: updated DonationInterface from e86f23a371e75a1684de5c102c06d993ead660e0 to ed98772ead6365d58356294ddd46bc4312204b1d
19:08 paravoid: rebooting asw2-d-eqiad
18:56 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: MF Beta: Do not move first paragraph before infobox (T145216) (duration: 00m 49s)
18:31 thcipriani@tin: Synchronized php-1.28.0-wmf.23/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turn off Cirrus AB test on zh and ja (T147499) (duration: 00m 46s)
18:29 thcipriani@tin: Synchronized php-1.29.0-wmf.1/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: SWAT: Turn off Cirrus AB test on zh and ja (T147499) (duration: 00m 47s)
18:22 andrewbogott: rebooting labvirt1013
18:17 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Add maiwiki HD logos (T149790) PART II (duration: 00m 47s)
18:15 thcipriani@tin: Synchronized static/images/project-logos: SWAT: Add maiwiki HD logos (T149790) PART I (duration: 00m 47s)
18:11 bd808: Sending final Tool Labs survey reminder emails from silver (T147336)
18:09 thcipriani@tin: Synchronized portals: SWAT: Updating wikipedia.org portal (T128546) (T135441) (duration: 00m 47s)
18:09 thcipriani@tin: Synchronized portals/prod/wikipedia.org/assets: SWAT: Updating wikipedia.org portal (T128546) (T135441) (duration: 00m 47s)
18:02 andrewbogott: rebooting labvirt1012
17:52 gehel: deploying new GUI on wdqs
17:32 godog: clean syslog/daemon.log on lithium, spam from mtail
16:51 chasemp: copy labstore2001 tools backup to 2003 and others backup to 2004 for emergency maint
16:08 _joe_: banning dbtree.wikimedia.org on cache_misc, T149357
15:48 jynus: restarting db1069 to apply latest wiki list configuration
15:48 volans: re-armed keyholder after it's upgrade on tin, mira and their deployment-prep equivalents
15:15 moritzm: rebooting sodium for kernel update
15:01 marostegui: Deploy schema change s5 dewiki.revision - only codfw T148967
15:01 moritzm: rebooting wasat for kernel update
14:36 moritzm: installing django security updates on Ubuntu servers
14:13 hashar@tin: Synchronized wmf-config/InitialiseSettings.php: MF Beta: Enable moving first paragraph before infobox - T145216 (duration: 00m 47s)
14:12 moritzm: rebooting labvirt1014 for kernel update
13:58 hashar: European SWAT complete
13:56 hashar@tin: Finished scap: ZeroBanner / ZeroPortal extensions.json fix (duration: 22m 01s)
13:34 hashar@tin: Started scap: ZeroBanner / ZeroPortal extensions.json fix
13:27 hashar@tin: Synchronized php-1.29.0-wmf.1/resources/src/mediawiki/page/gallery.css:
12:32 reedy@tin: Synchronized php-1.29.0-wmf.1/includes/EditPage.php: Fix regression from 1.28.0-wmf.23 T149473 (duration: 00m 47s)
12:11 hoo: Updated Wikidata's property suggester with data from Monday's json dump and applied the T132839 workarounds
10:51 moritzm: installing mailman security update
10:35 mobrovac: scb starting back CP and re-enabling puppet
10:33 moritzm: rolling restart of cassandra on restbase in eqiad completed
08:43 marostegui: Stopping mysql dbstore2002 for maintenance - T149457
08:37 moritzm: rolling restart of cassandra on restbase in eqiad to pick up new Java security updates
08:34 mobrovac: scb10ox stopping puppet and CP for Cassandra restarts
08:32 elukey: restarted cassandra-metrics-collector on aqs100[456] for jvm upgrades
08:10 mobrovac: change-prop deploying a28f9ba
07:19 marostegui: Stopping MySQL db2011 for maintenance - T149099
06:00 mutante: re-enable puppet on bromine after gerrit 319268
02:50 l10nupdate@tin: ResourceLoader cache refresh completed at Wed Nov 2 02:50:27 UTC 2016 (duration 5m 26s)
02:45 l10nupdate@tin: scap sync-l10n completed (1.29.0-wmf.1) (duration: 10m 43s)
02:18 l10nupdate@tin: scap sync-l10n completed (1.28.0-wmf.23) (duration: 05m 43s)
01:46 ejegg: enabled CiviCRM queue consumers and dedupe jobs
01:06 ejegg: disabled ingenico audit parser
00:48 ejegg: updated CiviCRM from 38a6c26e65e1b91864ccfc730b32d1e0253df5ff to 954bab421fa3435c5dd1a80f59f657407764b620
00:20 eileen1: enable recurring global collect job - query now dead, 26 contributions run but not in civi
00:13 eileen1: kill recurring global collect job for now (while query dies on server)

2016-11-01

23:25 ejegg: disabled donations queue consumer and dedupe for drush updb
23:22 ejegg: updated CiviCRM from 56eadabf705b3d035e11bb9fd3478457c159e40d to 38a6c26e65e1b91864ccfc730b32d1e0253df5ff
21:55 andrewbogott: rebooting labvirt1013
21:34 andrewbogott: rebooting labvirt1012
21:16 andrewbogott: rebooting labvirt1011
20:59 mutante: stop/start eventlogging on eventlog1001 (after adding IPv6 address appeared to make it stop and removing it again)
20:58 andrewbogott: rebooting labvirt1010
20:33 andrewbogott: rebooting labvirt1005 and labvirt1009
20:09 andrewbogott: rebooting labvirt1008
20:09 andrewbogott: rebooting labvirt1006
19:44 thcipriani@tin: rebuilt wikiversions.php and synchronized wikiversions files: group0 to php-1.29.0-wmf.1
19:37 thcipriani@tin: Finished scap: testwiki to 1.29.0-wmf.1 and rebuild l10n cache (duration: 27m 56s)
19:22 andrewbogott: rebooting labvirt1004, labvirt1007
19:09 thcipriani@tin: Started scap: testwiki to 1.29.0-wmf.1 and rebuild l10n cache
18:48 jynus@tin: Synchronized wmf-config/db-eqiad.php: pool new enwiki api servers to 100% after initial warm-up (duration: 00m 49s)
18:37 andrewbogott: rebooting labvirt1003
18:27 jynus@tin: Synchronized wmf-config/db-eqiad.php: increase api resources for enwiki -high api load (duration: 00m 48s)
18:16 dereckson@tin: Synchronized wmf-config/InitialiseSettings.php: Enable wikilove on bn.wikisource (duration: 01m 44s)
18:00 andrewbogott: rebooting labvirt1002
17:59 godog: ban releases.wikimedia.org/debian from cache_misc to fetch Packages/Release again
17:34 madhuvishy: Rebooting host labstore2001
17:03 thcipriani: starting branch cut for 1.29.0-wmf.1
15:59 godog: graphite-carbon restart after merging https://gerrit.wikimedia.org/r/#/c/316810/
15:58 cmjohnson1: checking all serial cables to row D in eqiad.
15:49 Reedy: Created wikilove tables on bnwikisource for T149683
15:41 mark: Installed nmap on iron
15:01 moritzm: upgrading/rolling restart of remaining wtp nodes in eqiad to nodejs 4.6
14:50 ori: Local-hacking some JavaScript changes on mw1099 to debug T146510
14:35 akosiaris@puppetmaster1001: conftool action : set/pooled=yes; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=cxserver'])
14:08 Reedy: created OATHAuth tables on all private wikis
13:53 Dereckson: scap pull @ tin to sync /srv/mediawiki locally
13:32 Dereckson: sync-portal: Synchronized portals/, purged URLs (Gerrit:319054)
13:27 Dereckson: Synchronized wmf-config/CommonSettings-labs.php: Labs: fix $wgJsonConfigInterwikiPrefix and set isLocal=false for tabular data (Gerrit:319024 + Gerrit:319036, no-op in prod) (duration: 00m 57s)
12:47 gehel: rolling restart of cassandra on maps1* for jvm upgrade
12:32 chasemp: mgmt powercycle of labstore1004
12:05 moritzm: upgrading wtp1001 to nodejs 4.6
12:01 bblack: upgrading cache_text nginx => 1.11.4-1+wmf13
11:13 gehel: rolling restart of cassandra on maps2* for jvm upgrade
11:04 gehel: rolling restart of cassandra on maps-test* for jvm upgrade
10:54 akosiaris: rebooting einsteinium
10:06 moritzm: installing openjdk security fixes on restbase2, rolling restart of cassandra
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Tue Nov 1 02:30:34 UTC 2016 (duration 4m 16s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.28.0-wmf.23) (duration: 09m 08s)

2016-10-31

23:46 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: MF Beta: Do not move first paragraph before infobox (T145216) (T149389) (duration: 00m 46s)
23:40 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn off revision number in graph img srv (duration: 00m 46s)
23:35 thcipriani@tin: Synchronized wmf-config/CommonSettings-labs.php: SWAT: LABS: Enable tabular data lua support (T148745) (housekeeping sync) (duration: 00m 46s)
23:32 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Removed unused wmgUseGraphWithNamespace support PART II (duration: 00m 45s)
23:31 thcipriani@tin: Synchronized wmf-config/CommonSettings.php: SWAT: Removed unused wmgUseGraphWithNamespace support PART I (duration: 00m 47s)
23:26 ejegg: increased payments-wiki session timeout
23:26 thcipriani@tin: Synchronized wmf-config/InitialiseSettings.php: SWAT: Create patroller usergroup for enwiki (T149019) (duration: 00m 46s)
23:21 mutante: disabled puppet on bromine temp. issue with reprepo config for releases
21:08 reedy@tin: Synchronized wmf-config/CommonSettings.php: Enable OATHAuth on officewiki (duration: 00m 48s)
21:07 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Enable OATHAuth on officewiki (duration: 00m 47s)
20:34 arlolra: updated Parsoid to version e503e801 (T149504)
20:16 bblack: upgrading cache_maps to nginx-1.11.4-1+wmf13
20:15 arlolra: starting Parsoid deploy
19:17 elukey: restarted varnishkafka-webrequest on cp2018 and cp3045 (CRITICALs in icinga, librdkafka errors logged for kafka1018.eqiad.wmnet)
19:12 yurik@tin: Synchronized wmf-config/InitialiseSettings.php: touch and sync - logs are flooded (duration: 00m 46s)
18:55 yurik@tin: Synchronized wmf-config: labs syncup https://gerrit.wikimedia.org/r/#/c/318883 (duration: 00m 49s)
17:59 ottomata: kafka preferred-prelica-election for analytics-eqiad to promote kafka1018 as leader
17:27 mark: Chris moved cr2-eqiad:xe-5/0/[0-2] and xe-5/1/2 to xe-3/1/[0-3]
17:05 chasemp: reboot labstore1004
17:00 ottomata: kafka preferred replica election on main-eqiad kafka cluster to promote kafka1003 as leader for its preferred partitions
15:44 mark: Chris moved cr2-eqiad:xe-5/1/[0-3] to xe-3/1/[0-3]
15:35 mark: Disabled ports cr2-eqiad:xe-5/1/[0-3] (row A-D uplinks)
15:28 marostegui@tin: Synchronized wmf-config/db-codfw.php: Depool db2034 for maintenance - T149553 (duration: 00m 46s)
15:21 Reedy: created oathauth_users table on officewiki T135889
15:17 reedy@tin: Synchronized php-1.28.0-wmf.23/extensions/WikimediaMaintenance/createExtensionTables.php: Add OATHAuth (duration: 00m 46s)
14:49 ottomata: adding kafka1003 in as replicas for active main-eqiad topics
14:32 moritzm: rebooting labstore2004 for kernel update
14:12 moritzm: rebooting labstore2003 for kernel update
14:12 ottomata: adding kafka1003 as kafka broker in main-eqiad cluster
14:00 Reedy: that deploy was was "Show changes from last 14 days in watchlist in cswiki T148327 "
14:00 reedy@tin: Synchronized wmf-config/: (no message) (duration: 00m 50s)
13:59 moritzm: powercycling labnet1002
13:58 reedy@tin: Synchronized docroot/noc/: nocnocnoc (duration: 00m 45s)
13:57 reedy@tin: Synchronized wmf-config/: Remove old ContactPage files (duration: 00m 47s)
13:54 reedy@tin: Synchronized wmf-config/CommonSettings.php: Use MetaContactPages (duration: 00m 48s)
13:52 moritzm: powercycling labcontrol1002
13:52 reedy@tin: Synchronized wmf-config/MetaContactPages.php: Stage new file (duration: 00m 46s)
13:46 reedy@tin: Synchronized wmf-config/throttle.php: Throttle rule for T149443 (duration: 00m 46s)
13:39 reedy@tin: Synchronized wmf-config/InitialiseSettings.php: Enable newusermessage on kkwiki T149563 (duration: 00m 55s)
13:35 moritzm: rebooting labstore2001 for kernel update
13:31 moritzm: rebooting labstore1002 for kernel update
13:06 moritzm: rebooting ganeti1001 for kernel update
12:32 bblack: upgrading nginx to 1.11.4-1+wmf13 on cache_misc - T148917
12:31 moritzm: migrating nodes from ganeti1001 for kernel reboot
12:27 moritzm: failover ganeti1002 as new master in eqiad
12:08 bblack: upgrading nginx to 1.11.4-1+wmf13 on cache_upload - T148917
12:07 bblack: upgrading nginx to 1.11.4-1+wmf13 on cache_upload
11:49 bblack: uploaded nginx-1.11.4-1+wmf13 to carbon jessie-wikimedia (logfile spam fixup)
11:32 moritzm: updating parsoid in codfw to nodejs 4.6.0
11:03 jmm@tin: Synchronized wmf-config/ProductionServices.php: Reenabled poolcounter1001 after maintenance (duration: 00m 45s)
11:00 elukey: restarting cassandra on aqs100[456] for OpenJDK upgrades
10:48 moritzm: rebooting poolcounter1001 for kernel update
10:40 moritzm: temporarily disabled poolcounter1001 for maintenance
10:40 jmm@tin: Synchronized wmf-config/ProductionServices.php: disabled poolcounter1001 for maintenance (duration: 00m 47s)
10:08 _joe_: uploaded mcrouter 0.24.0-1 to jessie-wikimedia T132317
08:17 moritzm: rebooting rdb2* for kernel update
07:56 jynus: stopping replication on db1057 (s1-master) from codfw for codfw maintenance
07:43 elukey: powercycled cp2010 (not reachable via ssh, com2 console showed a frozen screen)
07:10 marostegui: Deploying schema change s1 enwiki codfw (db2016 - master) - T147166
05:04 madhuvishy: Upgraded systemd on notebook1002 to 230-7~bpo8+2 from backports
04:48 madhuvishy: Upgraded systemd notebook1001 to 230-7~bpo8+2 from backports
02:59 yuvipanda: start reimaging notebook1001 for T149543
02:20 l10nupdate@tin: ResourceLoader cache refresh completed at Mon Oct 31 02:20:21 UTC 2016 (duration 4m 16s)
02:16 l10nupdate@tin: scap sync-l10n completed (1.28.0-wmf.23) (duration: 05m 12s)

2016-10-30

16:35 jynus: powercycle es2019 after crash T149526
13:54 gehel: disabling completion suggester crons to leave place for terbium reboot
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Sun Oct 30 02:32:14 UTC 2016 (duration 4m 38s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.28.0-wmf.23) (duration: 09m 01s)

2016-10-29

22:26 reedy@tin: Synchronized php-1.28.0-wmf.23/includes/EditPage.php: Fix for T149473 (duration: 00m 49s)
14:25 bblack: nginx-1.11.4-1+wmf12 uploaded to carbon for jessie-wikimedia
11:10 jynus: performing schema change on s4 (imagelinks) T139090
08:06 apergos: reboot dataset1001 for new kernel
02:30 l10nupdate@tin: ResourceLoader cache refresh completed at Sat Oct 29 02:30:52 UTC 2016 (duration 4m 16s)
02:26 l10nupdate@tin: scap sync-l10n completed (1.28.0-wmf.23) (duration: 09m 06s)

2016-10-28

23:19 mutante: re-enabled puppet on phab2001 temp, ran puppet. removed 10.64.31.186/21 from eth0, stopped puppet again
20:42 bd808: Sending Tool Labs survey reminder emails from silver (T147336)
19:24 yurik: deployed kartotherian https://gerrit.wikimedia.org/r/#/c/318575/ - caching is still broken
17:15 mutante: contint1001 - removed php5-* packages (https://puppet-compiler.wmflabs.org/4502/contint1001.wikimedia.org/)
16:43 hasharAway: gallium contint1001: apt-get remove --purge doxygen graphviz
15:44 chasemp: toolschecker seems to have come up wonky, restarting service
15:23 hashar: Restarted nodepool
15:09 andrewbogott: rebooting labservices1001
15:02 andrewbogott: rebooting labnet1001
14:56 moritzm: upgrading openjdk-8/cassandra restart on restbase staging hosts
14:38 moritzm: various reboots of multatuli for systemd tests
13:47 moritzm: uploaded firejail 0.9.44 to carbon
13:42 hoo@tin: Synchronized wmf-config/Wikibase-labs.php: For consistency (duration: 00m 46s)
12:42 jynus: restarting and upgrading labsdb1004
10:46 moritzm: migrating nodes from ganeti1002 for kernel reboot (earlier entry was a typo)
10:46 moritzm: migrating nodes from ganeti1003 for kernel reboot
10:28 moritzm: migrating nodes from ganeti1003 for kernel reboot
10:25 ema: upgrading python-varnishapi to v50.18 on all v4 cache hosts
10:00 moritzm: migrating nodes from ganeti1004 for kernel reboot
09:39 jynus: stopping slave on mariadb labsdb1005 for labsdb1004 reimporting
09:24 hashar@tin: Synchronized /srv/mediawiki-staging/php-1.28.0-wmf.23/extensions/CirrusSearch: https://gerrit.wikimedia.org/r/#/c/318505/ for T149254 (fix log spam/fatal/warnings) (duration: 00m 56s)
09:20 hashar: Pulling CirrusSearch patch https://gerrit.wikimedia.org/r/#/c/318505/ on mw1099 for T149254 (fix log spam/fatal/warnings)
09:10 moritzm: rebooting pool counters in codfw for kernel update
09:06 marostegui: Deploying schema change s1.enwiki - only codfw - T147166
08:01 jynus: applying schema change (imagelinks) to s3 wikis T139090
07:17 moritzm: installing PHP security updates on jessie
02:32 l10nupdate@tin: ResourceLoader cache refresh completed at Fri Oct 28 02:32:31 UTC 2016 (duration 5m 12s)
02:27 l10nupdate@tin: scap sync-l10n completed (1.28.0-wmf.23) (duration: 09m 07s)
01:32 eileen1: update civicrm from 31bea5b36fb22792472990672a49e8dea637546c to 56eadabf705b3d035e11bb9fd3478457c159e40d
00:20 bd808: Testing logging to SAL via stashbot

2016-10-27

23:40 logmsgbot: yurik@tin Synchronized php-1.28.0-wmf.23/extensions/Kartographer/modules/dialog/dialog.js: https://gerrit.wikimedia.org/r/#/c/318457/ (duration: 00m 47s)
22:08 godog: "cassandra" graphite machines LV at 90% used, add 300G via lvresize
21:55 yurik: deployed kartotherian
21:42 yurik: about to deploy kartotherian update
21:10 urandom: T133395: Altering mobileapps keyspaces to use time-windowed compaction
19:27 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.23
19:25 twentyafterfour: Bumping all wikis to 1.28.0-wmf.23 refs T147517
19:18 ejegg: updated payments-wiki from df4c72dbd328307019a82449527dd13c84bf41ab to e86f23a371e75a1684de5c102c06d993ead660e0
19:07 logmsgbot: maxsem@tin Finished scap: https://gerrit.wikimedia.org/r/#/c/318343/ (duration: 37m 38s)
19:06 ejegg: updated SmashPig from f5f49d79fac0887df78ddcc68a2f5ced0670c387 to 142e60bfe09982f9f86b7839cbf35d9e0e96b8d7
18:30 logmsgbot: maxsem@tin Started scap: https://gerrit.wikimedia.org/r/#/c/318343/
18:26 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.23/includes/parser/Parser.php: Remove tracking category stuff that accidentally slipped into 61adc1e14 - T149310 (duration: 00m 46s)
18:14 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Get rid of ip/IP tolerance for throttle rules (T131469) (duration: 00m 46s)
18:09 gehel: wdqs deployment of latest GUI
17:54 godog: upgrading prometheus to 1.2.1 in codfw/eqiad
16:56 moritzm: uploaded gerrit 2.12.5 to apt.wikimedia.org
16:43 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=cxserver'])
16:43 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: mw1241.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=appserver', 'service=apache2'])
16:33 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: mw1241.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=appserver', 'service=apache2'])
16:25 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=cxserver'])
16:18 hashar: Restarted Jenkins Gearman client due to a deadlock with the beta cluster jobs
15:26 akosiaris: start confd on puppetmaster1001 and puppetmaster2001
15:02 akosiaris: disable puppet across the fleet for puppetmaster kernel upgrades
14:26 moritzm: migrating nodes from ganeti2001 for kernel reboot
14:24 moritzm: restarted ntp on ganeti2006 (stuck in XFAC state)
14:17 moritzm: failover of ganeti2002 to new master node in codfw
13:59 moritzm: migrating nodes from ganeti2002 for kernel reboot
13:42 hashar: European SWAT completed
13:39 moritzm: migrating nodes from ganeti2003 for kernel reboot
13:35 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.23/extensions/CirrusSearch/includes/HTMLCompletionProfileSettings.php: Fix comp suggest pref page (duration: 00m 48s)
13:34 gehel: maps / postgres replication checks in error after deployment of https://gerrit.wikimedia.org/r/#/c/315271/ (T147194) - replication is working, only check is failing - icinga is silenced
13:33 gehel: postgres replication checks in error after deployment of https://gerrit.wikimedia.org/r/#/c/315271/ (T147194) - replication is working, only check is failing - icinga is silenced
13:32 moritzm: migrating nodes from ganeti2004 for kernel reboot
13:17 moritzm: migrating nodes from ganeti2005 for kernel reboot
13:10 logmsgbot: hashar@tin Synchronized wmf-config/throttle.php: T146600 T149200 (duration: 00m 53s)
12:53 moritzm: uploaded openjdk-8 8u111 for jessie-wikimedia to apt.wikimedia.org
12:36 moritzm: migrating nodes from ganeti2006 for kernel reboot
12:35 gehel: disabling puppet on maps servers for deployment of https://gerrit.wikimedia.org/r/#/c/315271/
12:20 moritzm: restarted ntp on conf1001 (stuck in XFAC state)
12:04 gehel: restart elasticsearch on relforge to activate GC logs - T134853
11:33 _joe_: stopping all cache-related services on esams spares cp3012-22
10:30 moritzm: rebooting conf1003 for kernel update
10:26 moritzm: rebooting conf1002 for kernel update
10:20 moritzm: rebooting conf1001 for kernel update
10:02 gehel: reboot maps eqiad cluster for kernel update
09:54 moritzm: rolling reboot of zookeeper cluster in codfw for kernel update
09:45 gehel: reboot maps codfw cluster for kernel update
09:42 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: wtp2019.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:42 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: wtp2019.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:41 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: wtp2019.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:15 jynus: applying schema change (imagelinks) to s2 wikis T139090
08:24 moritzm: rolling reboot of logstash cluster for kernel update
07:47 marostegui: Deploying ALTER table s4 commonswiki.templatelinks - https://phabricator.wikimedia.org/T149079 (db2065 only)
07:26 _joe_: creating darmstadtium on ganeti, T148961
07:24 marostegui: Deploying schema change db2034- enwiki.change_tag/tag_summary - T147166
07:15 marostegui: Removed /srv/s5.sql.gz (54G - may 2015) from db1045 to free up some space
06:57 marostegui: Deploy schema change s5 dewiki.revision only codfw - T148967
03:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 27 03:00:35 UTC 2016 (duration 5m 20s)
02:55 logmsgbot: l10nupdate@tin scap sync-l10n completed (1.28.0-wmf.23) (duration: 10m 36s)
02:32 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.23/extensions/OAuth/: deploy fix for T149194 (duration: 00m 47s)
02:31 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.22/extensions/OAuth/: deploy fix for T149194 (duration: 00m 51s)
02:28 logmsgbot: l10nupdate@tin scap sync-l10n completed (1.28.0-wmf.22) (duration: 10m 28s)
01:46 eileen1: disable GlobalCollect Recurring Donations
01:21 mutante: mw1208 - service hhvm restart
00:42 ejegg: updated civicrm from 586433b81decb90b80c40acdbfde4548ca67af43 to 31bea5b36fb22792472990672a49e8dea637546c
00:32 mutante: palladium - removed from DNS
00:30 mutante: palladium - re-shutdown

2016-10-26

23:57 logmsgbot: dereckson@tin Synchronized docroot/noc/conf/: Update noc.wikimedia.org dblist and config files (duration: 00m 45s)
23:52 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.23/extensions/UploadWizard/resources/details/uw.DateDetailsWidget.js: Unbreak Flickr uploads (T149259) (duration: 00m 48s)
23:46 madhuvishy: tools reenabled puppet across proxy hosts. /.well-known/healthz now live on tools-proxy T143638
23:41 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: For $wmgGalleryOptions, use isset() (Gerrit:318223) (duration: 00m 45s)
23:22 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Fix for current Undefined variable: wmgGalleryOptions issue (duration: 00m 48s)
23:19 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Test setting gallery config differently on Beta Cluster enwiki (T141349, 2/2) (duration: 00m 45s)
23:18 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Test setting gallery config differently on Beta Cluster enwiki (T141349, 1/2, no-op in prod) (duration: 00m 49s)
21:08 hashar: gallium: stopped rsync server
21:02 jynus: restarting mysql and rebooting db1035
20:55 jynus: restarting mariadb on db2011 to test configuration change
20:50 hashar: syncing /var/lib/jenkins from gallium to contint1001 . rsync server spawned on gallium in a term, contint1001 using rsync --bwlimit=5m --delete --info=progress2 -az rsync://gallium.wikimedia.org/jenkins /var/lib/jenkins
20:49 Pchelolo: RESTBase deploy e835f9b8
20:46 Pchelolo: RESTBase deploy e835f9b8 - canary on restbase1007
20:43 arlolra: reverted Parsoid to version 63f1e151
20:41 Pchelolo: RESTBase deploy e835f9b8 - staging
20:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.23
20:31 arlolra: reverting Parsoid to version 63f1e151
20:31 arlolra: updated Parsoid to version ede4353
20:28 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.23/extensions/GlobalBlocking/: Deploy fix for T149232 to unblock the train refs T147517 (duration: 00m 51s)
20:14 arlolra: starting Parsoid deploy
19:14 logmsgbot: hoo@tin Synchronized wmf-config/Wikibase-labs.php: For consistency (duration: 00m 45s)
18:47 logmsgbot: hoo@tin Synchronized wmf-config/Wikibase-labs.php: For consistency (duration: 00m 47s)
18:39 mark: Reactivated BGP to AS6461 on cr1-eqiad
18:38 mark: Chris moved cr1-eqiad:xe-5/3/1 to xe-3/3/1
18:31 mark: Disabling BGP session to AS6461 on cr1-eqiad, preparing for port migration
18:27 mark: Chris is moving cr1-eqiad and cr2-eqiad xe-5/3/0 to xe-3/3/0 (both sides)
18:20 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.23/extensions/GeoData/: https://gerrit.wikimedia.org/r/#/c/318138/ (duration: 00m 47s)
18:19 mark: Chris is moving cr1-eqiad and cr2-eqiad xe-5/2/0 to xe-3/2/0 (both sides)
18:12 mark: Disabling cr1-eqiad:xe-5/2/0
18:10 logmsgbot: bd808@tin Synchronized wmf-config/CommonSettings.php: wikitech: Re-enable OAuth management interfaces T149150 (duration: 00m 46s)
18:04 ema: cache_text - finished rolling downtimed reboots for kernel update
17:23 cwd: updated smashpig from daba8c07108246261ca7ec557e3f1e5da430826e to f5f49d79fac0887df78ddcc68a2f5ced0670c387
16:58 mark: Shutting down cr1-eqiad:xe-5/0/[0-2] (part of aggregated links to rows A-C switches)
16:58 mark: Chris moved cr1-eqiad:xe-5/2/1 to xe-3/0/3
16:54 mark: Chris moved cr1-eqiad:xe-5/1/[0-3] to xe-3/1/[0-3]
16:49 mark: Shutting down cr1-eqiad:xe-5/1/[0-3] (part of aggregated links to rows A-D switches)
16:39 moritzm: restarted ntp on mc2008 (stuck in XFAC state)
16:36 jynus: stopping db2011 to replace disks T149099
15:59 mark: Disabling cr1-eqiad:ae4; VRRP conflict
15:58 mark: Reenabling cr1-eqiad:ae4
15:57 mark: Restored cr1-eqiad:ae4
15:37 ema: cache_text - start rolling downtimed reboots for kernel update (~3 hours to completion)
15:27 ema: cp1054 reboot for kernel update
15:16 bblack: restarting grrrit-wm
14:34 moritzm: rearmed keyholder on mira after reboot
14:24 bblack: cache_upload - start rolling downtimed reboots for kernel update (~4 hours to completion)
14:19 moritzm: rolling reboot of ocg cluster for kernel update
14:10 logmsgbot: demon@tin Synchronized w: replacing wiki.phtml with a symlink (duration: 00m 47s)
13:39 hashar: European SWAT deploy completed
13:37 hashar: mw2098 is all set now after I ran "scap pull". It is properly in tin:/etc/dsh/group/mediawiki-installation
13:30 hashar: mw2098: scap pull . It failed yesterday on reboot and is back in pull
13:28 hashar: mw2098 spurts bunch of Notice: Undefined variable: wmgWatchlistDefault in /srv/mediawiki/wmf-config/CommonSettings.php on line 1871
13:25 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: Remove obsolete config values (duration: 00m 46s)
13:24 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings-labs.php: LABS: Enable Tabular data on Commons - T148745 (duration: 00m 45s)
13:23 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.23/extensions/Kartographer: T149145: Fix empty groups params T149154: Fix external links (duration: 00m 57s)
13:11 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: wikitech: Set wgMWOAuthCentralWiki = false - T149150 (duration: 00m 47s)
13:07 moritzm: rebooting labtest hosts for kernel update
12:36 marostegui: Deploy schema change s5 dewiki.revision only codfw - T148967
12:31 moritzm: rebooting mira for kernel update
12:20 _joe_: turned off mw1152, removed salt/puppet data, T149185
12:19 bblack: moving git-ssh LVS from low-traffic -> high-traffic2 - T143915
12:05 bblack: moving ocg LVS from high-traffic2 -> low-traffic - T143915
10:40 bblack: rebooting eqiad lvs primaries (lvs100[1-3])
10:19 bblack: rebooting esams lvs primaries (lvs300[12])
10:13 bblack: rebooting ulsfo lvs primaries (lvs400[12])
10:02 bblack: rebooting codfw lvs primaries (lvs200[1-3])
09:52 jynus: starting schema change (imagelinks) on s1 T139090
08:43 elukey: increasing the AQS cassandra system_auth keyspace replication from 1 to 6 (and running nodetool-{a,b} repair system_auth on all nodes)
08:29 elukey: downgraded memcached on mc2009 to the Debian Jessie version (was part of a performance experiment)
08:25 dcausse: elastic@eqiad reindexing enwiki (take 3) with BM25 from wasat.codfw.wmnet T147508 (logs in ~dcausse/bm25_reindex/cirrus_log)
08:06 marostegui: Stoppping replication on db2058 - using it to clone another host - T146261
07:47 moritzm: bounced ntp on oxygen (stuck in XFAC state)
07:45 moritzm: rebooting mc* servers in codfw for kernel update
07:20 moritzm: rebooting oxygen for kernel update
07:14 marostegui: Deploying ALTER table s4 commonswiki.templatelinks - db2051 - T149079
07:10 moritzm: rebooting nescio for kernel update
06:34 moritzm: repooled mw2098 (was previously down for hardware check)
03:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 26 03:00:28 UTC 2016 (duration 5m 28s)
02:55 logmsgbot: l10nupdate@tin scap sync-l10n completed (1.28.0-wmf.23) (duration: 10m 13s)
02:28 logmsgbot: l10nupdate@tin scap sync-l10n completed (1.28.0-wmf.22) (duration: 09m 38s)
01:36 mutante: lead - (formerly gerrit) - shutdown -h now (T147905)
01:13 mutante: palladium - shutdown -h now
00:10 mutante: restarted icinga-wm, now there is /var/log/icinga/irc.log, it should talk now, but doesnt
00:04 mutante: let icinga own /var/log/icinga on einsteinium, restart icinga

2016-10-25

23:45 logmsgbot: addshore@mira Synchronized wmf-config/InitialiseSettings.php: gerrit:293243 Add a project namespace on tg.wikipedia (duration: 00m 47s)
23:41 logmsgbot: addshore@mira Synchronized wmf-config/CommonSettings.php: gerrit:315121 Stop adding Category:Uploaded_with_UploadWizard (duration: 00m 47s)
23:39 logmsgbot: addshore@mira Synchronized wmf-config/InitialiseSettings.php: gerrit:318013 Enable static maps on testwiki (duration: 00m 48s)
23:26 mutante: Submitted 'deactivate node' for palladium.eqiad.wmnet
23:21 mutante: removed palladium from puppet (T147320). puppet node clean
23:02 logmsgbot: maxsem@mira Synchronized php-1.28.0-wmf.23/extensions/ZeroBanner/: https://gerrit.wikimedia.org/r/#/c/318004/ (duration: 00m 50s)
23:00 logmsgbot: maxsem@mira Synchronized php-1.28.0-wmf.23/extensions/ZeroPortal/: https://gerrit.wikimedia.org/r/#/c/318004/ (duration: 01m 32s)
22:49 ejegg: updated CiviCRM from 844495a64a34f16167a5f5300826b9531ae360dd to 586433b81decb90b80c40acdbfde4548ca67af43
22:34 Pchelolo: revert RESTBase to f9017adc
22:33 ejegg: updated payments-wiki from 1e8f6a22b71710163fc4e4d90ea03a9729ab33d6 to df4c72dbd328307019a82449527dd13c84bf41ab
22:02 Pchelolo: RESTBase update to 3e53f00e
21:55 Pchelolo: RESTBase update to 3e53f00e - staging
21:54 ejegg: updated SmashPig from 31c0757ca17499deac6a8f49eb7ed17924630419 to daba8c07108246261ca7ec557e3f1e5da430826e
21:51 logmsgbot: maxsem@mira Synchronized php-1.28.0-wmf.23/extensions/Graph/: https://gerrit.wikimedia.org/r/#/c/317989/ (duration: 01m 23s)
20:49 ejegg: updated civicrm from 9e288691de2e918d00928c9164e0f2389563888d to 844495a64a34f16167a5f5300826b9531ae360dd
20:42 logmsgbot: maxsem@mira Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/#/c/317879/ (duration: 00m 47s)
20:32 ostriches: gerrit doing a quick reboot, config pick up
20:16 logmsgbot: filippo@mira Synchronized wmf-config/ProductionServices.php: Put back potassium as poolcounter1002 (duration: 00m 51s)
20:14 logmsgbot: demon@mira Synchronized php-1.28.0-wmf.23/extensions/CiteThisPage/SpecialCiteThisPage.php: T149112 (duration: 01m 39s)
19:41 dcausse: elastic@eqiad reindexing enwiki with BM25 from terbium T147508 (logs in ~dcausse/bm25_reindex/cirrus_log)
19:23 hasharAway: Python PyPi mirror has some issue. Impacts all CI jobs relying on tox https://status.python.org/
19:22 twentyafterfour: phabricator is back from reboot and it appears that all is well
19:19 twentyafterfour: twentyafterfour@iridium: The system is going down for reboot NOW!
19:11 twentyafterfour: rebooting iridium (phabricator) in ~ 3 minutes
19:11 logmsgbot: demon@mira rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.23
19:03 logmsgbot: ebernhardson@mira Synchronized php-1.28.0-wmf.22/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: (no message) (duration: 00m 47s)
18:46 logmsgbot: demon@mira Finished scap: Moving testwiki to wmf.23 for l10n bootstrap (duration: 42m 45s)
18:44 godog: restbase eqiad rolling reboot for kernel update
18:43 Alex: �18�<�godog�18�>�� !log restbase eqiad rolling reboot for kernel update
18:03 logmsgbot: demon@mira Started scap: Moving testwiki to wmf.23 for l10n bootstrap
17:27 moritzm: rebooting phab2001 for kernel update
17:24 moritzm: rebooting notebook1002 for kernel update
17:23 jynus: powercycling db2015, unresponsive
17:22 ostriches: gerrit: quick reboot, picking up logging config changes for jvm
17:19 logmsgbot: filippo@mira Synchronized wmf-config/ProductionServices.php: Put helium back in service during potassium reimage (duration: 01m 34s)
16:08 gehel: restarting ferm on elastic2020
16:04 gehel: delete dangling indices on elasticsearch codfw: jawiki_general_first, jawiki_content_first, zhwiki_general_first and zhwiki_content_first
15:21 ori: Synchronized wmf-config/throttle.php: I049bd463: Use correct IP for Vanderbilt 2016-10-25 edit-a-thon throttle exception (T149063) (duration: 01m 20s)
14:32 gehel: reboot of wdqs cluster eqiad for kernel upgrade
14:24 moritzm: rebooting maerlant for kernel update
14:20 gehel: reboot of wdqs cluster codfw for kernel upgrade
14:14 elukey: removed logstash filter for Apache (https://logstash.wikimedia.org/app/kibana#/dashboard/apache2log) - T144005
14:01 _joe_: refreshing puppet facts
13:34 moritzm: rebooting rcstream servers for kernel update
13:11 moritzm: rebooting etcd1006 for kernel update
13:07 hashar: European SWAT complete
13:06 logmsgbot: hashar@mira Synchronized wmf-config/throttle.php: Nashville Architecture edit-a-thon (Vanderbilt library) throttle rule - T149063 (duration: 02m 07s)
13:03 moritzm: rebooting etcd1005 for kernel update
12:57 moritzm: rebooting etcd1004 for kernel update
12:54 moritzm: repooled maerlant (was depooled for some reason, possibly forgotten to repool after maintenance)
12:50 moritzm: rebooting etcd1003 for kernel update
12:42 moritzm: rebooting etcd1002 for kernel update
12:30 moritzm: rebooting etcd1001 for kernel update
12:24 elukey: rebooting druid100[123] for kernel upgrades
11:56 moritzm: rebooting hydrogen for kernel update
11:53 dcausse: elastic@eqiad reindexing top10 wikis with BM25 from terbium T147508 (logs in ~dcausse/bm25_reindex/cirrus_log)
11:53 moritzm: rolling reboot of mc2* for kernel update
11:31 moritzm: rebooting copper for kernel update
11:16 moritzm: bounced ntp on hassium (stuck in XFAC state)
11:14 logmsgbot: marostegui@mira Synchronized wmf-config/db-eqiad.php: Repool db1059 - T146261 (duration: 01m 22s)
10:37 moritzm: rebooting acamar for kernel update
10:11 elukey: reimaging mc103[1-6] to Jessie
10:09 logmsgbot: marostegui@mira Synchronized wmf-config/db-eqiad.php: Depool db1059 to clone another host from it - T146261 (duration: 01m 36s)
09:58 moritzm: rearmed keyholder on tin
09:50 moritzm: rebooting tin for kernel update
09:45 moritzm: rebooting achernar for kernel update
09:43 marostegui: Deploying ALTER table s4 commonswiki.templatelinks - T149079 (db2058 only)
09:23 moritzm: rebooting hassium for kernel update
09:09 moritzm: rebooting hassaleh for kernel update
08:49 marostegui: Stopping replication db2058 s4 - using it to clone another host - T146261
08:13 akosiaris: reimaging tegmen
08:05 dcausse: elastic@codfw reindexing jawiki, thwiki and zhwiki T147498 (logs in terbium:~dcausse/bm25_reindex/cirrus_log)
08:05 moritzm: rebooting chromium for kernel update
07:40 moritzm: rebooting netmon1001 for kernel update
07:07 moritzm: rebooting tungsten for kernel update
07:05 gehel: rebooting elasticsearch relforge cluster for kernel update
06:56 moritzm: rebooting wezen for kernel update
06:52 gehel: rebooting elastic1035 for kernel update
06:52 moritzm: rebooting osmium for kernel update
02:39 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 25 02:39:53 UTC 2016 (duration 5m 12s)
02:34 logmsgbot: l10nupdate@tin scap sync-l10n completed (1.28.0-wmf.22) (duration: 10m 46s)
02:33 logmsgbot: ori@mira Synchronized php-1.28.0-wmf.22/resources/src/mediawiki/mediawiki.js: I1d61f4dcf: mw.loader: Fix off-by-one error in splitModuleKey() (duration: 02m 15s)

2016-10-24

23:24 logmsgbot: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Set collation to uca-no-u-kn on no.wikipedia (146675, T148488) (duration: 00m 50s)
23:14 logmsgbot: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Remove duplicated wmgCirrusSearchClusterOverrides entry (duration: 00m 50s)
23:07 dapatrick: Deployed patch for T148600 to wmf22
22:10 ejegg: updated civicrm from 3b76de564aef4dad7807be2d3e570dec38696227 to 9e288691de2e918d00928c9164e0f2389563888d
21:43 godog: rolling-restart of ms-fe in codfw/eqiad for kernel update
21:01 Amir1: rollbacking ores to 8bbd3ab
20:49 Amir1: deploying 0caa589 to all ores nodes
20:44 Amir1: deploying 0caa589 on ores canary node
20:42 arlolra: updated Parsoid to version 63f1e151 (T139032, T146612, T141905)
20:29 bearND: deployed mobileapps f872894
20:26 bearND: starting mobileapps deploy
20:25 gehel: stopping elasticsearch eqiad cluster restart for the night.
20:18 arlolra: starting Parsoid deploy
20:06 gehel: powering on elastic2020 (no idea why it is powered off)
19:34 ejegg: updated SmashPig from f9e185b7b14749ae24155c69bc1927dc6222e5f7 to 31c0757ca17499deac6a8f49eb7ed17924630419
19:30 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: [cirrus] Activate BM25 on top 10 wikis: Step 2 (take 2) (T147508) (duration: 00m 50s)
19:22 logmsgbot: thcipriani@mira Synchronized php-1.28.0-wmf.22/extensions/CirrusSearch/includes/SearchConfig.php: SWAT: Add wgContentNamespaces to the list of vars loaded by SearchConfig (T148840) (duration: 00m 58s)
19:16 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix capitalization for change 317387 (T148328) (duration: 00m 51s)
19:11 logmsgbot: thcipriani@mira Synchronized wmf-config/CommonSettings.php: Fix capitalization for change 317387 (T148328) PART II (duration: 00m 50s)
19:10 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: Fix capitalization for change 317387 (T148328) PART I (duration: 00m 50s)
19:03 logmsgbot: thcipriani@mira Synchronized php-1.28.0-wmf.22/resources/src/mediawiki/mediawiki.js: SWAT: resourceloader: Make cache-eval in mw.loader.work asynchronous (T142129) (duration: 00m 52s)
18:32 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Switch bs, hr and uk wikis to numeric collation (T148682) (duration: 00m 50s)
18:19 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Set $wgCategoryCollation to uca-hr for Croatian wikipedia (T148749) (duration: 00m 57s)
18:06 ejegg: updated payments-wiki settings to f4b79f02c2897349a96b023aaef4cbf24e7708f0
18:05 bblack: downgrading nginx(+linked openssl implicitly) on cp*
17:02 gehel: deplyoing latest GUI for WDQS
16:22 ejegg: updated SmashPig from e28b2cd9f0c1429acdd2a08c68f95884dbffb594 to f9e185b7b14749ae24155c69bc1927dc6222e5f7
15:04 bblack: enabling/running puppet on caches for 8x varnish ports changes - T107749
14:57 paravoid: restarting ferm on es2015
14:54 bblack: starting ferm server on eeden, radon
14:41 logmsgbot: gehel@puppetmaster1001 conftool action : set/pooled=yes; selector: dc=eqiad,cluster=maps,service=kartotherian,name=maps1002.eqiad.wmnet
14:38 logmsgbot: dereckson@mira Synchronized wmf-config/CommonSettings.php: Toggle wgDefaultUserOptions['watchdefault'] on for cs.wikipedia, off elsewhere (T148328, 2/2) (duration: 00m 50s)
14:36 logmsgbot: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Toggle wgDefaultUserOptions['watchdefault'] on for cs.wikipedia, off elsewhere (T148328, 1/2) (duration: 00m 54s)
14:36 bblack: disabling puppet on all caches ahead of port# work, to test - T107749 / https://gerrit.wikimedia.org/r/#/c/317405
14:29 yurik: re-deployed current kartotherian to all servers (maps1002 & maps-test* were stale)
14:11 marostegui: Deploy schema change s5 dewiki.revision - only codfw T148967
14:03 logmsgbot: l10nupdate@mira ResourceLoader cache refresh completed at Mon Oct 24 14:03:07 UTC 2016 (duration 6m 17s)
13:56 logmsgbot: dereckson@mira scap sync-l10n completed (1.28.0-wmf.22) (duration: 10m 46s)
13:42 bblack: restarting all varnish frontends (serially per-cluster with proper depooling, etc)
13:20 elukey: reimaging mc120[89] and mc1030
13:18 Dereckson: Started manually l10nupdate, as it didn't run for 6 days, and more especially to fix T148921 user-facing issue.
13:13 logmsgbot: dereckson@mira Synchronized wmf-config/throttle.php: Edit-a-thon BDA (Poitiers) throttle rule (T148852) (duration: 01m 13s)
10:47 elukey: reimaged mc102[56], currently doing mc1027
10:21 _joe_: rebooting kubernetes1002
09:20 mobrovac: change-prop deploying c7feda2
09:09 mobrovac: restbase deploy end of f9017ad
08:55 akosiaris: rebooting cobalt (gerrit) for kernel upgrades
08:53 elukey: reimaging mc1024
08:46 mobrovac: restbase deploy start of f9017ad
08:38 gehel: continue rolling restart of elasticsearch eqiad cluster
08:38 hashar: Restarting gallium (Jenkins/Zuul) for kernel upgrades
08:36 akosiaris: rebooting labnodepool1001 for kernel upgrades
08:36 akosiaris: rebooting scandium for kernel upgrades
08:33 hashar: rebooting contint1001
08:20 elukey: reimaging mc1023.eqiad.wmnet
07:46 elukey: reimaging mc1022.eqiad.wmnet (T137345)
07:09 marosteg1i: Deploying alter table s1.enwiki on codfw - T147166

2016-10-22

15:37 logmsgbot: oblivian@puppetmaster1001 conftool action : set/pooled=no; selector: name=cp1052.eqiad.wmnet
15:02 logmsgbot: bblack@puppetmaster1001 conftool action : set/pooled=yes; selector: name=cp1052.eqiad.wmnet
15:02 bblack: repool cp1052 - T148891
14:52 bblack: rebooted cp1052 - T148891
14:27 bblack: depooled cp1052 (cache_text@eqiad, ethernet linkdown for unknown reasons)
12:34 marostegui: Stopping replication in db2055 to use it to clone another host - T146261

2016-10-21

23:45 mutante: depooling maps1002 (by running "depool" on the server itself)
23:35 yurik: maps1002.eqiad is running older/incorrect/misbehaving software for some reason, restart didn't help. Need to depool
22:17 mutante: cp4006,cp4014 gzipped some logs in home for disk space
22:08 mutante: cp4006, cp4014 were running out of disk, apt-get clean
21:40 mutante: phab2001 that IP was also on iridium/phab1001, it should not be hardcoded in puppet, causing issues in T143363
21:37 mutante: phab2001 - ip addr del 10.64.32.186/21 dev eth0
21:06 bblack: restarting varnish backends (depooled, etc) for eqiad cache_upload: cp1049, cp1072, cp1074
19:50 cmjohnson1: dataset1001 array 1 swap failed disk slot 4
19:40 cmjohnson1: labvirt1005 swapping disk 0
19:40 gehel: routing traffic for cache-maps in codfw -> eqiad
19:29 gehel: running puppet on eqiad cache nodes to activate maps traffic redirection
19:06 gehel: shutting down cassandra on maps2004, seems to have lost data
18:22 ejegg: updated SmashPig from d1ca0632d00dfb608f70ca4b70251a5ba49f4411 to e28b2cd9f0c1429acdd2a08c68f95884dbffb594
16:45 ejegg: updated fundraising tools from 09ae6e24d8ca8350dc099d63a6ca0d9ec9fdef2b to f83e39291adc55677fc4b49307dc4807eba18019
16:33 mutante: rebooting planet1001 - *.planet.wm.org will be right back
16:30 mutante: rebooting planet2001
16:05 elukey: reimaging mc1021 with wmf-auto-reimage (T137345)
15:28 elukey: reimaging mc1019 with wmf-auto-reimage (T137345)
14:50 elukey: reimaging mc1020 with wmf-auto-reimage (T137345)
14:31 _joe_: rebooting all kubernetes worker nodes in production
14:31 moritzm: rolling reboot of thumbor* for kernel update
14:30 marostegui: Stopping replication on db2055 to use it to clone another host - T146261
13:55 bblack: restart isc-dhcp-server on carbon
13:55 moritzm: rolling reboot of thumbor* for kernel update
13:40 moritzm: completed rolling reboot of restbase in codfw
13:14 marostegui: Deploying schema change S6 ruwiki for table ores_model - T147734
12:24 moritzm: rebooting ruthenium for kernel update
12:02 moritzm: rebooting bromine for kernel update
11:28 gehel: starting rolling restart of elasticsearch eqiad cluster
11:05 moritzm: rebooting hafnium for kernel update
10:49 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: mariadb: pool db1053 as the new rc special slave after maintenance (duration: 01m 00s)
10:36 marostegui: Deploying schema change S2 several wikis for table ores_model - T147734
10:28 bblack: rebooting radon (ns0)
10:22 moritzm: rolling reboot of restbase in codfw for kernel update
10:09 marostegui: Deploying schema change S7 fawiki.ores_model - T147734
10:04 moritzm: rebooting seaborgium (labs LDAP server) for kernel update
09:51 marostegui: Deploying schema change S5 wikidatawiki.ores_model - T147734
09:48 moritzm: rebooting neon (icinga host) for kernel update
09:35 marostegui: Deploying schema change S1 enwiki.ores_model in eqiad - T147734
09:32 elukey: rebooting kafka100[12] for kernel upgrades (EventBus hosts)
09:26 moritzm: rebooting krypton for kernel update
09:18 godog: start rolling reboot of ms-be machines in eqiad for kernel update
09:15 moritzm: rebooting meitnerium (archiva.wikimedia.org) for kernel update
09:13 jynus: reviewing and applying new watchdog events to all core dbs T148790
09:06 moritzm: rebooting serpens (labs LDAP server) for kernel update
08:49 moritzm: rebooting ununpentium (RT) for kernel update
08:40 marostegui: Deploying schema change S1 enwiki.ores_model in codfw - T147734
08:38 moritzm: rebooting radium (tor relay) for kernel update
08:35 moritzm: rebooting aluminium (url_downloader for eqiad) for kernel update
08:25 moritzm: rebooting alsafi (url_downloader for codfw) for kernel update
08:23 jynus: applying events_coredb_slave.sql to db1070
08:12 moritzm: rebooting bast1001 for kernel update
08:05 moritzm: rolling reboot of swift backend servers in codfw
07:52 moritzm: rebooting bohrium (hosting piwik) for kernel update
07:20 elukey: rebooting stat100[234] for kernel upgrades
06:26 elukey: restarting stat1001 for kernel upgrades (will cause a brief outage for some analytics websites like analytics.w.o and pivot.w.o)

2016-10-20

23:51 bblack: rebooting eeden (ns2) for kernel
23:48 logmsgbot: dereckson@mira Synchronized php-1.28.0-wmf.22/extensions/CentralNotice: Bump CentralNotice version to fix T145738 and T145447 (Gerrit:317077) (duration: 00m 54s)
23:45 logmsgbot: dereckson@mira Synchronized php-1.28.0-wmf.22/includes/cache/MessageCache.php: Use checkKeys for large messages (T144952) (duration: 00m 50s)
23:37 bblack: rolling restarts of citoid on scb* (for recdns update)
23:30 logmsgbot: dereckson@mira Synchronized php-1.28.0-wmf.22/extensions/UploadWizard/resources/ui/steps/uw.ui.Upload.js: Fix a weird ghost "or" for non-Flickr users (Gerrit:317013) (duration: 01m 31s)
22:52 bd808: Finished sending Tool Labs survey emails from silver (T147336)
21:52 ejegg: updated fundraising tools from f6d200dc76520298171196c5419b382fe86d9dcd to 09ae6e24d8ca8350dc099d63a6ca0d9ec9fdef2b
21:18 ejegg: updated SmashPig from 961fc4c14f94181a0e364615a0a05dd8f4646912 to d1ca0632d00dfb608f70ca4b70251a5ba49f4411
21:01 bd808: Started sending Tool Labs survey emails from silver
18:39 logmsgbot: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Revert "[cirrus] Activate BM25 on top 10 wikis: Step 2" (duration: 00m 54s)
18:30 logmsgbot: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Activate Cirrus BM25 algo on top 10 wikis (step 2, T147508) (duration: 00m 50s)
18:12 XenoRyet: updated payments wiki from 27b464fd4383647fc2e7f0a613f290d6edccd22f to 1e8f6a22b71710163fc4e4d90ea03a9729ab33d6
18:12 logmsgbot: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Enable Visual Editor for all users remaining phase 6 Wikipedias (T142589) (duration: 00m 50s)
18:11 mutante: mailing list server back, normal operation
18:08 mutante: rebooting fermium (lists.wm.org)
18:08 logmsgbot: dereckson@mira Synchronized wmf-config/CommonSettings.php: wikitech: Fix Undefined variable: wgMWOAuthCentralWiki (Gerrit:316981) (duration: 01m 26s)
17:50 dcausse: warming up elastic@codfw from wasat.codfw.wmnet (take 3)
17:42 urandom: T133395, T113805: Starting a primary-range, incremental repair of local_group_wiktionary_T_parsoid_html.data on restbase2001.codfw.wmnet
17:38 mutante: rebooting kraz - short downtime of irc.wikimedia.org please prepare to reconnect your clients if they dont automatically do it
17:35 apergos: reboot of last few stragglers for mw* hosts in codfw/eqiad: mw2152 mw2079 mw1239
17:29 mutante: rebooting install2001
17:00 apergos: rolling reboot of video scalers in codfw/eqiad: mw1259 mw1260 mw2152 mw2246
16:48 apergos: rolling reboot of testservers in codfw/eqiad: mw1017 mw1099 mw2017 mw2099
16:45 mutante: rebooting install1001
16:44 logmsgbot: gehel@puppetmaster1001 conftool action : set/pooled=yes; selector: dc=eqiad,cluster=logstash,service=kibana
16:35 godog: reboot graphite1001 for kernel upgrade
16:30 apergos: rolling reboots for jobrunners in eqiad: mw1161-1169, mw1299-1306
16:26 gehel: deploying new LVS service for kibana - T132458
16:25 godog: reboot graphite1003 for kernel upgrade
16:08 moritzm: bounced ntp on mw2089/mw2241 (XFAC state)
15:59 mutante: short downtime of ganglia web ui
15:59 mutante: rebooting uranium
15:36 apergos: rolling reboots for jobrunners in codfw: mw2080-2085, mw2153-mw2162, mw2247-2250
15:14 apergos: rolling reboot of image scalers for codfw, eqiad: mw2086-2089, mw2148-2151, mw1293-1298
15:10 ottomata: restarted statsv on hafnium
14:55 moritzm: bounced ntp on mw2196/mw2197 (XFAC state)
14:34 moritzm: rebooting rutherfordium for kernel update
14:27 logmsgbot: filippo@puppetmaster1001 conftool action : set/pooled=no; selector: name=prometheus1001.eqiad.wmnet
14:26 logmsgbot: filippo@puppetmaster1001 conftool action : set/pooled=yes; selector: name=prometheus1002.eqiad.wmnet
14:24 akosiaris: bounce ntpd on bast4001
14:20 moritzm: rebooting auth* servers
14:20 ottomata: starting rolling restart of analytics-eqiad kafka brokers to apply kernel update
14:18 logmsgbot: filippo@puppetmaster1001 conftool action : set/pooled=no; selector: name=prometheus2001.codfw.wmnet
14:18 logmsgbot: filippo@puppetmaster1001 conftool action : set/pooled=yes; selector: name=prometheus2002.codfw.wmnet
14:17 apergos: rolling reboot of remaining app servers in codfw: mw2221-2245, and in eqiad: mw1261-1275
14:11 logmsgbot: jmm@puppetmaster1001 conftool action : set/pooled=inactive; selector: mw2098.codfw.wmnet
14:09 logmsgbot: jynus@mira Synchronized wmf-config/db-eqiad.php: mariadb: move db1053 from s1 to s4 (duration: 02m 06s)
13:38 moritzm: restarting mx1001 for kernel update
13:22 moritzm: restarting francium for kernel update
13:15 godog: rolling reboot of prometheus machines for kernel update
13:14 moritzm: restarting ms1001 for kernel update
13:10 elukey: force failover from temporary Hadoop Master node (an1002) to its stanby (an1001) to restore the standard configuration
13:05 elukey: correction: force failover for Hadoop Master node (an1001) to its stanby (an1002) and rebooting an1001 for kernel upgrades
12:59 elukey: force failover for Hadoop Master node (an1002) to its stanby (an1002) and rebooting an1001 for kernel upgrades
12:59 moritzm: ferm on baham (failed to start due to failing DNS resolution in early boot)
12:52 moritzm: restarting mx2001 for kernel update
12:48 moritzm: bounced ntp on mw2116 (XFAC state)
12:39 elukey: restarting an1003 for kernel upgrades (oozie/hive master)
12:35 moritzm: bounced ntp on baham (was stick in INIT phase)
12:31 apergos: more app server rolling restarts for codfw: mw2163-2199
12:29 apergos: more API server rolling restarts for eqiad: mw1221-1235, 1276-1290
12:27 apergos: more APP server rolling restarts for eqiad: mw1209-1216, 128-1220, 1236-38, 1240-1258
12:12 moritzm: restarting bast2001 for kernel update
12:11 apergos: retaction. those are app servers, not starting them yet
12:10 apergos: more api server rolling restarts for eqiad: mw1209-1216, 128-1220, 1236-38, 1240-1258
12:08 moritzm: bounced ntp on mw2206 (XFAC state)
12:05 bblack: correction: rebooting baham / ns1.wikimedia.org for kernel
12:04 bblack: rebooting baham / ns2.wikimedia.org for kernel
11:53 elukey: rebooting an1027 (camus job launcher) for kernel upgrades
11:48 moritzm: bounced ntp on mw2101 and mw2147 (XFAC state)
11:48 bblack: depool cp1047 (cache_maps eqiad)
11:23 apergos: rolling restarts of more api servers in codfw: mw2200 - 2220
11:17 elukey: rebooting all the Analytics Hadoop nodes for kernel upgrades
11:07 mobrovac: change-prop restarting in codfw after kafka kernel upgrade
10:58 apergos: rolling reboots for first batch of app servers in eqiad: mw1170-1188
10:50 elukey: rebooting kafka200[12] for kernel upgrades (Kafka main-codfw non live cluster)
10:38 apergos: rolling restarts on first batch of api servers in eqiad: mw1189-1208
10:21 apergos: while the first batch of codfw api servers trundle along, starting rolling reboots for appservers in codfw starting with mw2090-2098, 2100-2119
10:20 moritzm: removing a few older kernels on analytics1036, was short of disk space in /boot partition
10:05 elukey: rebooting the Analytics Hadoop cluster for kernel upgrades
09:50 jynus: stop sql thread replication for db1053 and applying partitioning as a "special slave"
09:32 godog: rolling restart of graphite machines for kernel upgrade
09:16 apergos: restarts of mw2075,6,7 done, starting rolling restarts shortly of 8,9, 2120-2147
08:57 akosiaris: rebooting wtp10{02,06,12,13,17,22} for kernel upgrade
08:57 elukey: rebooting eventlog2001 for kernel upgrades (EL spare host)
08:54 elukey: rebooting eventlog1001 for kernel upgrades (Eventlogging host)
08:53 moritzm: rebooting bast4001 for kernel update
08:49 moritzm: rebooting restbase-test* for kernel upgrade
08:43 akosiaris: rebooting wtp10{01,03,04,05,18,23} for kernel upgrade
08:34 akosiaris: rebooting wtp10{07,08,09,10,19,24} for kernel upgrade
08:32 elukey: rebooting aqs100[456] for kernel upgrades (one at the time, de-pool/reboot/pool)
08:31 elukey: rebooting aqs100[123] for kernel upgrades (one at the time, de-pool/reboot/pool)
08:25 akosiaris: rebooting wtp10{10,14,15,16,20,21} for kernel upgrade
08:19 akosiaris: reboot the rest of the wtp20XX hosts for kernel upgrade
08:15 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: wtp2019.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
08:10 akosiaris: reboot wtp20{03,05,08,09,12,15,17,18,20} for kernel upgrade
08:09 mobrovac: change-prop deploying 3a11886
07:52 moritzm: rebooting bast3001 for kernel update
07:51 gehel: start of elasticsearch codfw rolling restart
07:32 moritzm: rebooting snapshot1001 for kernel update
07:27 moritzm: rebooting snapshot1005-1007 for kernel update
01:17 logmsgbot: legoktm@mira Synchronized wmf-config/InitialiseSettings.php: Revert Enable AbuseFilterCachingParser by default - T148673 (duration: 00m 51s)
00:23 bblack: restarting pybal on lvs1002 for new recdns IP

2016-10-19

23:58 logmsgbot: krenair@mira Synchronized php-1.28.0-wmf.22/extensions/OpenStackManager: https://gerrit.wikimedia.org/r/#/c/316909/ (duration: 01m 00s)
23:41 mutante: Host mw1239 is not in mediawiki-installation dsh group
23:35 logmsgbot: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Reverting votewiki back to English (T148352) (duration: 00m 50s)
23:20 logmsgbot: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Switching 10 more wikis to numeric category collation (T146675) (duration: 00m 59s)
21:02 bearND: deployed mobileapps 2551db4
20:59 bearND: starting mobileapps deploy
20:11 logmsgbot: demon@mira Synchronized docroot/mediawiki/keys/: adding tylers key (duration: 01m 09s)
20:04 ejegg: disabled CiviCRM generic dedupe job
20:00 ejegg: enabled CiviCRM major gifts dedupe
19:56 bblack: installing new kernel packages on lvs:primary
19:30 bblack: upgrading nginx+openssl on remaining cache nodes (eqiad+esams/text+upload) - T144523
19:26 bblack: installing new kernel packages on lvs:secondary
19:26 bblack: installing new kernel packages on authdns
19:18 bblack: installing new kernel packages on cp*
18:33 bblack: restarting stuck Jenkins
18:02 urandom: T133395: RESTBase: Altering keyspace local_group_wikipedia_T_parsoid_html.data to enable time-window compaction
17:41 logmsgbot: ori@mira Synchronized wmf-config/InitialiseSettings.php: I6d28e534: Disable AbuseFilterCachingParser on bgwiki (T148660) (duration: 00m 50s)
17:34 moritzm: rebooting xenon/cerium/praseodymium to new kernels
17:33 bblack: cp1008 / pinkunicorn reboot
17:30 logmsgbot: ori@mira Synchronized wmf-config/InitialiseSettings.php: Ieb8cdab9: Enable AbuseFilterCachingParser by default (duration: 01m 01s)
17:15 elukey: depooled mw1239.eqiad.wmnet to allow hw investigation (T148421) (was done today but didn't logged properly)
16:36 bblack: depooling cp3009 (esams cache_misc), possible HW issues
15:58 Mark Issuing secure erase on cp3021 sdb
14:17 akosiaris@puppetmaster1001:conftool action : set/pooled=yes; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=graphoid'])
14:14 mobrovac: [done] scb expanding and deploying services to scb[12]00[1234] change-prop citoid cxserver graphoid mathoid mobileapps
14:02 chasemp: bdsync tools from labstore1001 to labstore2001
14:00 mobrovac: scb expanding and deploying services to scb[12]00[1234] change-prop citoid cxserver graphoid mathoid mobileapps
13:36 mafk: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Create a 'templateeditor' user group at en.wiktionary plus adittional configuration (T148007) (duration: 02m 33s)
13:31 bblack: upgrading nginx on ulsfo text+upload caches - T144523
13:21 mobrovac: change-prop stopping instances on scb100[12] so that scb1003 picks up more load
13:21 logmsgbot: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Create a new namespace "Príloha" for skwikt (T148563) (duration: 00m 50s)
13:14 mobrovac: change-prop stopping instance on scb1004 so that scb1004 picks up more load
13:14 logmsgbot: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Create a 'templateeditor' user group at en.wiktionary plus adittional configuration (T148007) (duration: 02m 33s)
13:11 ema: stopping varnishlog service on v4 cp hosts and removing log file
12:52 bblack: upgrading nginx on codfw text+upload caches - T144523
12:24 marostegui: Stopping db2055 to clone another host - T146261
12:17 akosiaris: update cr{1,2}-eqiad configuration to add tegmen+einsteinium
12:01 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb2004.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=ores'])
12:01 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb2003.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=ores'])
12:00 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
12:00 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
11:55 marostegui: Deploying schema change db2055 - S1 enwiki.change_tag - T147166
11:35 bblack: upgrading nginx on cp2002 (codfw upload canary) - T144523
11:30 bblack: upgrading nginx on cp2001 (codfw text canary) - T144523
10:27 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: scb1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=mobileapps'])
10:27 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: scb1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=mobileapps'])
10:05 _joe_: converting owner of files for l10nupdate usermod on tin
10:02 _joe_: ran usermod -u 10002 l10nupdate on tin
09:50 logmsgbot: marostegui@mira Synchronized wmf-config/db-eqiad.php: Repool db1064 after finishing the ALTER table - T147305 (duration: 01m 08s)
09:22 moritzm: installing rsyslog bugfix updates
09:20 marostegui: Deploying schema change on db1069 S4 instance commonswiki revision table - T147305
08:15 marostegui: Stopping db2062.codfw.wmnet to use it to clone another server - T146261
08:14 moritzm: installing tor security update on radium
08:12 moritzm: installing quagga security updates
07:31 _joe_: disabled profiling on mw1189, hhvm keeps crashing
06:50 _joe_: installing jemalloc with memory profiling enabled on mw1189

2016-10-18

23:04 Dereckson: This full scap pulled three changes of the EU SWAT: gerrit:316069 TimedMediaHandler, gerrit:316585 MobileFrontend, gerrit:315901 ULS
23:03 logmsgbot: demon@mira Finished scap: bringing full cluster back into sync (duration: 25m 13s)
22:38 logmsgbot: demon@mira Started scap: bringing full cluster back into sync
22:28 logmsgbot: demon@mira Synchronized README: Bringing co-masters back in sync (duration: 13m 10s)
21:37 mutante: added Dpatrick to WMF LDAP group
18:32 logmsgbot: dereckson@mira Synchronized wmf-config/LabsServices.php: Elastic@deployment-prep: Remove deployment-elastic08 from the cluster (no-op in prod, labs only) (duration: 00m 47s)
18:30 logmsgbot: dereckson@mira Synchronized wmf-config/CirrusSearch-labs.php: Elastic@deployment-prep: force the number of replicas to 1 max (no-op in prod, labs only) (duration: 01m 18s)
17:55 dcausse: warming up elastic@codfw from wasat.codfw.wmnet
17:34 jynus: stopping mysql, cloning db1064->db1053; upgrading
17:01 bblack: upgrading nginx on cache_maps - T144523
16:57 ejegg: updated payments-wiki from b4ad60e739b9dbb97f08a3623db961a74682422a to 27b464fd4383647fc2e7f0a613f290d6edccd22f
15:47 godog: eqiad-prod: ms-be1022 to weight 3000 T136631
15:16 andrewbogott: upgrading puppetmaster on labtestcontrol2001 to trusty/3.8.5
15:06 bblack: upgrading nginx on all remaining cache_misc (eqiad, esams) - T144523
14:54 bblack: upgrading nginx on all cache_misc @ codfw - T144523
14:54 chasemp: rsync tools from labstore1001 to labstore1004
14:43 bblack: upgrading nginx on all cache_misc @ ulsfo - T144523
14:40 marostegui: Shutting down es2015 for hardware maintenance - T147769
14:21 bblack: upgrading nginx on cp4001 (cache_misc ulsfo) as prod canary
14:18 bblack: uploading nginx-1.11.4+wmf3 to carbon jessie-wikimedia - T144523
13:58 jynus: restarting and upgrading db2049 and es2019 to test new config
13:53 jynus: applying new init.d script on all mariadb 10 servers
12:52 elukey: mw1169 back in service after reimage (MW Jobrunner)
11:55 elukey: removed /etc/mysql/conf.d/research-client.cnf from stat1002 (root:root perms, not supposed to be there but only on stat1003)
11:37 elukey: reimaging mw1169 to Debian Jessie (MW Jobrunner)
10:40 elukey: mw1168.eqiad.wmnet back in service after reimage (MW Jobrunner)
09:29 elukey: reimaging mw1168 to Debian Jessie (MW Jobrunner)
09:25 elukey: varnishkafka restarting in upload/misc/maps with new settings (https://gerrit.wikimedia.org/r/316306)
09:18 gehel: upgrade nodejs to 4.6.0 on maps2* servers
08:56 moritzm: reimaging tin to jessie
08:53 marostegui: Deploying ALTER table on S4 commonswiki (db1064 — last host) - T147305
08:42 jynus: clone db1052 -> db1053, will perform maintenance (db restarts, reboots on both) at the same time
07:57 logmsgbot: marostegui@mira Synchronized wmf-config/db-eqiad.php: Depool db1064 as it needs an ALTER table and pool db1068 temporarily to serve vslow and dump service - T147305 (duration: 02m 53s)
03:19 mutante: restarted grrrit-wm
03:18 mutante: gerrit has logs now in /var/log/gerrit/
03:15 mutante: restarting gerrit for logging config change
02:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 18 02:37:01 UTC 2016 (duration 5m 49s)
02:31 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.22) (duration: 10m 20s)
00:48 bblack: restarting API hhvms with >40% mem usage via salt every 10 minutes in a loop from here forward. screen session on neodymium, named api-hhvm-restarts
00:39 mutante: restarted hhvm on mw1281 (was at 47.7% usage)
00:31 bblack: restarting hhvm on API nodes where it's using >30% mem
00:22 bblack: restarting hhvm on *API* nodes where it's using >50% mem
00:22 bblack: restarting hhvm on nodes where it's using >50% mem
00:05 mutante: restarted hhvm on mw1194,mw1197,mw1198

2016-10-17

23:27 Pchelolo: running import deletions script on restbase1007
22:26 mutante: restarted gerrit on cobalt
22:07 Pchelolo: running restriction import script on restbase1007
20:59 mutante: tegmen - stopped duplicate icinga-wm (ircecho)
20:53 mutante: maintenance servers, terbium and wasat, now have IPv6 connectivity
20:33 bearND: deployed mobileapps 13fa4b4
20:32 Krenair: updated status.wm.o apache config on wikitech-static box to correctly serve static assets again (T148438)
20:30 bearND: starting mobileapps deploy
19:31 cwd: disabled all dedupe jobs besides "contacts"
18:38 gehel: deploying latest gui and binaries for wdqs
18:35 Jeff_Green: switch payments-listener back to eqiad
18:17 _joe_: dumping core on mw1194
17:32 Jeff_Green: switch payments-listener to codfw
17:20 _joe_: restarting lvs on lvs1003/1006 for the api change
16:42 ottomata: restarting hadoop nodemanagers 1 at a time
16:18 ori: Restarted HHVM on API cluster EQIAD
15:33 ottomata: rebootting analytics1030
15:13 elukey: ran kafka preferred-replica-election to allow kafka1018 to be back as broker replica leader
14:38 elukey: mw1167 back in service after reimage (MW Jobrunner)
14:30 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.22/extensions/EducationProgram/includes/Events/EditEventCreator.php: Id02366ef: Fix-up for Ia3d767e86 (duration: 00m 52s)
14:06 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I8562f8e1: Enable AbuseFilterCachingParser on metawiki and commonswiki (duration: 00m 56s)
13:06 elukey: reimage mw1167 to Debian (MW Jobrunner)
12:31 marostegui: Stopping MySQL db2055 (S1-codfw) to import S1 to dbstore2001 - T146261
11:39 akosiaris: T148830 poweroff sca1001, sca1002, sca2001, sca2002
11:38 jynus: stopping db1048 for general upgrade & reconfiguration
10:57 godog: deploy thumbor 0.1.28 to thumbor100[12]
10:38 moritzm: uploaded openssl 1.1.0b1+wmf1 for jessie-wikimedia to carbon (patched to be co-installable with our default 1.0.2 packages, build against libssl11-dev to use openssl 1.1)
10:31 mobrovac: citoid deploying df4c92e
10:04 mobrovac: mathoid deploying 52f345b
09:51 akosiaris: T148380 disable puppet on sca1001, sca1002, deactivate them on puppetmasters
09:02 godog: reset power on ms-be1025 - off and no logs to be found on ilo
08:54 jynus: stopping, upgrading and restarting es2014
08:16 _joe_: restarting hhvm on mw1175, stuck in HPHP::FastCGISession::blockingWriteStdOut after OOM
08:15 elukey: upgrading nodejs on aqs100[56]
08:10 jynus: disabling notifications of es2014 before it pages
07:49 marostegui: Stopping MySQL in db2057.codfw.wmnet to use it to clone another server
07:15 marostegui: Dropping memory tables hitcounter, _counters from S7 - T132837
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 17 02:26:33 UTC 2016 (duration 4m 56s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.22) (duration: 07m 34s)

2016-10-16

20:58 Amir1: ladsgroup@scb[12]00[12]: sudo service celery-ores-worker restart
14:05 Amir1: mwscript resetUserEmail.php --wiki=fawiki Ebrambot <email removed>
10:36 _joe_: restarting hhvm on mw120[0-8]
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 16 02:26:45 UTC 2016 (duration 5m 41s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.22) (duration: 06m 50s)

2016-10-15

06:32 logmsgbot: tstarling@mira Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 02m 30s)
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 15 02:31:13 UTC 2016 (duration 5m 37s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.22) (duration: 05m 41s)

2016-10-14

21:49 yurik: deployed & restarted kartotherian, all's good now
21:47 yurik: about to sync kartotherian fix
21:47 ejegg: updated SmashPig from 2306010284ae2452a6a81f61fa44fa35fcc6a42f to 961fc4c14f94181a0e364615a0a05dd8f4646912
21:43 logmsgbot: krenair@mira Synchronized wmf-config/InitialiseSettings-labs.php: https://gerrit.wikimedia.org/r/#/c/316004/ - no-op here, only labs reads this file. just keeping it in sync (duration: 02m 07s)
19:51 Jeff_Green: flip payments-listener back to eqiad
19:43 matt_flaschen: Manual DB update for https://www.wikidata.org/wiki/User_talk:Doror and https://fr.wikipedia.org/wiki/Discussion_utilisateur:Robur15 . T148057
18:08 Jeff_Green: flip payments-listener service from eqiad to codfw
17:46 ejegg: enabled pending queue consumer
17:42 ejegg: disabled pending queue consumer
17:34 ejegg: updated SmashPig from 3c3d115d8388515eaf1f88b4263a56985454f3a1 to 2306010284ae2452a6a81f61fa44fa35fcc6a42f
15:52 Jeff_Green: authdns-update to add payments-listener-codfw A record
12:48 dcausse: reindexing top 10 wikipedias with BM25 on elastic@codfw from terbium (logs in ~dcausse/bm25_reindex/cirrus_log/) (T147508)
12:36 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: sca1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=sca', 'service=zotero'])
12:36 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: sca1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=sca', 'service=zotero'])
12:36 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: sca2002.codfw.wmnet (tags: ['dc=codfw', 'cluster=sca', 'service=zotero'])
12:36 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: sca2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=sca', 'service=zotero'])
11:20 mobrovac: change-prop deploying 6dbdaa1
11:18 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: sca2003.codfw.wmnet (tags: ['dc=codfw', 'cluster=sca', 'service=zotero'])
11:17 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: sca2004.codfw.wmnet (tags: ['dc=codfw', 'cluster=sca', 'service=zotero'])
11:17 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: sca1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=sca', 'service=zotero'])
11:17 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: sca1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=sca', 'service=zotero'])
11:02 marostegui: Stopping MySQL db2055 (S1-codfw) to import S1 to dbstore2001 - T146261
11:00 elukey: mw1166 back in service after reimage (MW Jobrunner)
10:28 jynus: stopping and restarting mysql at dbstore2001 for misc tests T146261
09:23 elukey: reimaging mw1166 to Debian Jessie (MW Jobrunner)
08:59 elukey: mw1161 back in service after reimage (MW Jobrunner, scap proxdy)
07:47 elukey: reimaging mw1161 to Debian Jessie (MW Jobrunner, scap proxy)
07:17 marostegui: Dropping hitcounter, _counter memory tables in S7 on db1041 (master) - T132837
07:13 moritzm: upgrading hhvm in codfw to latest 3.12.x bugfix release
02:39 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 14 02:39:27 UTC 2016 (duration 6m 20s)
02:36 Ezarate: mdesploy@eswikinews
02:33 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.22) (duration: 12m 03s)
00:17 logmsgbot: reedy@mira Synchronized php-1.28.0-wmf.22/extensions/PageAssessments: [extensions/PageAssessments] Only update assessment data when talk pages are saved (duration: 00m 51s)

2016-10-13

23:59 logmsgbot: reedy@mira rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.22 take 3
23:50 logmsgbot: reedy@mira Synchronized php-1.28.0-wmf.22/extensions/ZeroPortal: Revert extenson registration (duration: 00m 49s)
23:49 logmsgbot: reedy@mira Synchronized php-1.28.0-wmf.22/extensions/ZeroBanner: Revert extenson registration (duration: 00m 50s)
23:47 logmsgbot: reedy@mira Synchronized wmf-config/mobile.php: Back to pre deploy state (duration: 00m 49s)
23:19 logmsgbot: reedy@mira rebuilt wikiversions.php and synchronized wikiversions files: all wikis back to .21
23:08 logmsgbot: reedy@mira Synchronized wmf-config/mobile.php: Only set remote config if not zerowiki (duration: 01m 15s)
22:53 logmsgbot: reedy@mira Synchronized wmf-config/mobile.php: Revert my hack (duration: 00m 49s)
22:47 logmsgbot: reedy@mira Synchronized php-1.28.0-wmf.22/extensions/JsonConfig/: array_merge_recursive (duration: 00m 50s)
22:41 ejegg: updated SmashPig from e89f1b5316955b5d6576d4c71f8ab840db9b4de2 to 3c3d115d8388515eaf1f88b4263a56985454f3a1
22:30 logmsgbot: reedy@mira Synchronized php-1.28.0-wmf.22/extensions/JsonConfig/: less array + array more array_merge (duration: 00m 57s)
22:01 logmsgbot: reedy@mira rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.22 take 2
21:54 logmsgbot: reedy@mira Synchronized wmf-config/mobile.php: Load wgJsonConfigs in callback (duration: 00m 56s)
21:42 mutante: gerrit is back
21:40 mutante: gerrit is restarting for config change 315571
21:20 bblack: updating nodejs on ocg1003
21:15 bblack: updating nodejs on ocg1002
21:05 bblack: attempting nodejs upgrade on ocg1001
20:26 matt_flaschen: Ran manual DB updates for T148057.
20:17 logmsgbot: thcipriani@mira rebuilt wikiversions.php and synchronized wikiversions files: Rollback 1.28.0-wmf.22 from group2
20:12 logmsgbot: thcipriani@mira rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.22
19:57 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I8f6eb9f6af: Enable AbuseFilterCachingParser on testwiki and mediawikiwiki (duration: 00m 51s)
19:31 yurik: deployed and restarted tilerator[ui] https://gerrit.wikimedia.org/r/#/c/315707/
19:04 jynus: updating dns for labsdb1002 and m3-slave
18:49 jynus: setting phabricator db in read only mode for master failover
18:49 mobrovac: restbase deploy end of d510090
18:44 mutante: contint1001 - systemctl mask jenkins.service
18:43 jynus: setting up circular replication db1043 <-> db1048
18:39 mutante: contint1001 - stop jenkins service
18:32 mobrovac: restbase deploy start of d510090
18:10 logmsgbot: thcipriani@mira Synchronized php-1.28.0-wmf.22/extensions/EventBus/EventBus.hooks.php: SWAT: Do not set the performer property if the user is not available. (T147977) (duration: 01m 38s)
18:09 twentyafterfour: deployed https://phabricator.wikimedia.org/D413 on iridium and restarted apache
17:46 bblack: forced ocsp stapling update on all caches, just in case
17:33 bblack: pushing new intermediate to caches - T148045
17:21 cmjohnson1: powering off mw1001-1148 to be decommissioned (except mw1017 and mw1099) per T141522
17:20 yurik: deployed kartotherian https://gerrit.wikimedia.org/r/#/c/315701/
17:17 bblack: disabling puppet on all cache nodes
15:37 logmsgbot: marostegui@mira Synchronized wmf-config/db-eqiad.php: wmf-config/db-codfw.php db1053 got moved to another rack so updating its IP - T147774 (duration: 00m 50s)
15:31 moritzm: installing libdbd-mysql-perl security updates
15:26 urandom: T133395: RESTBase: Altering keyspace local_group_wikimedia_T_parsoid_html.data to enable time-window compaction
15:18 logmsgbot: marostegui@mira Synchronized wmf-config/db-eqiad.php: Repool db1068 after it was out for an ALTER table - T147305 (duration: 00m 58s)
14:45 moritzm: installing nspr security updates
14:32 marostegui: Shutting down MySQL in db1053, it is going to be moved to another rack - T147774
14:18 cwd|afk: updated smashpig from 28ba033be2f50fc7837c788dc452753ff82284e1 to e89f1b5316955b5d6576d4c71f8ab840db9b4de2
13:22 hashar: European SWAT completed
13:20 moritzm: rolling restart of restbase in eqiad to pick up new nodejs
13:19 logmsgbot: hashar@mira Synchronized php-1.28.0-wmf.22/includes/api/ApiPurge.php: ApiPurge: Set the triggering user for the LinksUpdate T147516 T147977 (duration: 00m 52s)
13:09 logmsgbot: hashar@mira Synchronized wmf-config/InitialiseSettings.php: Adding language name configuration for Wikidata T146707 (duration: 00m 53s)
12:53 marostegui: Dropping hitcounter, _counter memory tables in S6 (frwiki jawiki ruwiki) - T132837
11:14 elukey: mw1165 (MW Jobrunner) back in service after reimage
10:31 hoo: Ran (updated) T132839-Workarounds.sh from my home in terbium
09:54 marostegui: Deploying schema change on commonswiki.revision - db1068 - T147305
09:49 elukey: reimaging mw1165 to Debian Jessie (MW Jobrunner)
09:45 logmsgbot: marostegui@mira Synchronized wmf-config/db-eqiad.php: Depool db1068 for an ALTER table - T147305 (duration: 04m 58s)
09:15 marostegui: Stopping MySQL in db2057.codfw.wmnet to use it to clone another server
09:09 moritzm: reimaging wasat to jessie
08:35 elukey: restarting aqs on aqs1004 to pick up the new nodejs package
08:31 moritzm: updating app server canaries to new hhvm package
07:25 marostegui: Dropping hitcounter, _counter memory tables in S6 (frwiki jawiki ruwiki) on db1050 (master) - T132837
07:14 marostegui: Dropping hitcounter, _counter memory tables in S5 (dewiki, wikidatawiki) - T132837
06:52 moritzm: installing ghostscript security updates
03:01 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 13 03:01:31 UTC 2016 (duration 6m 42s)
02:54 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.22) (duration: 12m 47s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.21) (duration: 08m 24s)

2016-10-12

23:41 ejegg: updated civicrm settings to revision a7c7919cf6ff7a4722e0f98aca90c43edfd3ac9c
23:38 logmsgbot: maxsem@mira Synchronized php-1.28.0-wmf.22/extensions/Kartographer/: https://gerrit.wikimedia.org/r/#/c/315603/ (duration: 01m 03s)
23:08 logmsgbot: dereckson@mira Synchronized wmf-config/InitialiseSettings.php: Raise abuse filter emergency threshold for es.wikibooks (T145765) (duration: 01m 19s)
21:47 ejegg: updated SmashPig from ac6a0f04aad1b5519e8ee3a112dacf2a53efff30 to 28ba033be2f50fc7837c788dc452753ff82284e1
21:29 ejegg: updated SmashPig from 00772cd5c821ad11267bd9355cc24160756361bf to ac6a0f04aad1b5519e8ee3a112dacf2a53efff30
21:20 ejegg: updated SmashPig from 94c7f0d45cb42b8b7dced341e4dea920473991b0 to 00772cd5c821ad11267bd9355cc24160756361bf
21:03 ejegg: disabled pending queue consumer
20:57 ejegg: updated SmashPig on all hosts from fa0267b6f23505d835fd1557a82c2ea99a6985d8 to 94c7f0d45cb42b8b7dced341e4dea920473991b0
20:48 ejegg: updated SmashPig on listener host from fa0267b6f23505d835fd1557a82c2ea99a6985d8 to 94c7f0d45cb42b8b7dced341e4dea920473991b0
19:27 logmsgbot: thcipriani@mira rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.22
19:14 cmjohnson1: disconnecting production cable from old pay-lvs1002 (replaced with new) T147932
18:39 logmsgbot: ebernhardson@mira Synchronized wmf-config/CirrusSearch-common.php: SWAT T66829 Prefer articles in a users language on multilingual wikis (duration: 00m 51s)
18:37 logmsgbot: ebernhardson@mira Synchronized wmf-config/InitialiseSettings.php: SWAT T66829 Prefer articles in a users language on multilingual wikis (duration: 00m 50s)
18:28 logmsgbot: ebernhardson@mira Synchronized wmf-config/CirrusSearch-common.php: SWAT wgCirrusSimilarityProfile -> wgCirrusSearchSimilarityProfile (duration: 00m 53s)
18:27 logmsgbot: ebernhardson@mira Synchronized wmf-config/InitialiseSettings.php: SWAT wgCirrusSimilarityProfile -> wgCirrusSearchSimilarityProfile (duration: 00m 49s)
18:19 ejegg: updated fundraising tools from cbf4dcd734956f95a7485ce1f420d05222c09e5a to f6d200dc76520298171196c5419b382fe86d9dcd
18:17 logmsgbot: ebernhardson@mira Synchronized wmf-config/InitialiseSettings.php: SWAT T138310 Re-enable Flow beta feature on frwikiquote (duration: 00m 50s)
18:13 logmsgbot: ebernhardson@mira Synchronized php-1.28.0-wmf.22/extensions/EventBus/EventBus.hooks.php: SWAT Dont set added/removed properties if they are empty (duration: 00m 52s)
18:10 logmsgbot: ebernhardson@mira Synchronized wmf-config/CirrusSearch-common.php: SWAT T147508 Activate BM25 on top 10 wikis: Step 1 (duration: 00m 50s)
18:08 logmsgbot: ebernhardson@mira Synchronized wmf-config/InitialiseSettings.php: SWAT cirrussearch config updates (duration: 01m 10s)
17:36 urandom: T133395: RESTBase: Altering keyspace local_group_wiktionary_T_parsoid_html.data to enable time-window compaction
17:10 robh: gerrit: system rebooted (cobalt) to enable HT, system back online as of a few minutes ago
17:00 ostriches: gerrit: stopping momentarily for system reboot
15:51 urandom: T133395: Restarting Cassandra instances in RESTBase, eqiad, rack 'd'
15:37 gehel: upgrading nodejs to 4.6.0 on maps1* servers
15:10 urandom: T133395: Restarting Cassandra instances in RESTBase, eqiad, rack 'b'
14:51 urandom: T133395: Restarting Cassandra instances on restbase1011.eqiad.wmnet
14:47 bblack: traffic cache nginxes: seamless upgrade-restart for new openssl lib
14:45 elukey: uploaded zuul 2.5.0-8-gcbc7f62-wmf4precise1 to precise-wikimedia/third-party (T145057)
14:38 elukey: uploaded zuul 2.5.0-8-gcbc7f62-wmf4jessie1 to jessie-wikimedia/third-party (T145057)
14:35 hashar: zuul-merger on scandium restarted. CI is resumed.
14:34 urandom: T133395: Restarting Cassandra instances on restbase1010.eqiad.wmnet
14:31 akosiaris: disable puppet on neon. Merging https://gerrit.wikimedia.org/r/315510
14:28 elukey: install zuul_2.5.0-8-gcbc7f62-wmf4jessie1_amd64.deb on scandium - T145057
14:27 hashar: stopped zuul-merger on scandium pausing CI as a result. Snipe upgrade going on
14:22 urandom: T133395: Restarting Cassandra instances on restbase1007.eqiad.wmnet
14:04 urandom: T133395: Restarting Cassandra instances on restbase2009.codfw.wmnet
14:02 moritzm: installing imagemagick security updates
14:01 bblack: upgrading openssl + confctl on cp*
13:58 hashar: Upgrading Zuul zuul_2.5.0-8-gcbc7f62 wmf3..wmf4
13:52 moritzm: restart restbase on restbase1007 to pick up new nodejs
13:27 moritzm: rolling restart of restbase in codfw to pick up new nodejs
13:23 hashar: European SWAT completed.
13:21 logmsgbot: hashar@mira Synchronized wmf-config/abusefilter.php: Send abusefilter hit notifications from es.wikibooks to UDP T147744 (duration: 00m 52s)
13:14 logmsgbot: hashar@mira Synchronized wmf-config/InitialiseSettings.php: Create 'massmessage-sender' group for tr.wikipedia T147740 (duration: 03m 42s)
12:23 elukey: mw1164 back in service (MW Jobrunner)
11:15 elukey: reimaing mw1164 to Debian Jessie (MW Jobrunner)
10:10 godog: upgrade grafana on krypton to 3.1.1-1470047149 T146354
09:20 kart_: Update cxserver to da7d4f6 (T146731)
09:09 moritzm: upgrading nodejs on etherpad1001
08:56 elukey: mw1163 (MW Jobrunner) back in service after the reimage
08:53 moritzm: upgrading nodejs on ruthenium
08:34 moritzm: installing c-ares security updates
07:38 elukey: reimaging mw1163 to Debian (MW Jobrunner)
06:42 moritzm: reimaging mw1099 (test application server) to jessie
03:14 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 12 03:13:59 UTC 2016 (duration 7m 12s)
03:06 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.22) (duration: 12m 50s)
02:36 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.21) (duration: 12m 50s)
00:02 logmsgbot: ebernhardson@mira Synchronized php-1.28.0-wmf.22/extensions/CirrusSearch/: SWAT CirrusSearch Add completion support to ClusterOverride, Remove position_increment_gap on source_text.trigram (duration: 00m 58s)
00:00 ebernhardson: pulled cirrus changes (315440, 315441) to mw1099

2016-10-11

23:54 logmsgbot: ebernhardson@mira Synchronized php-1.28.0-wmf.22/includes/ForkController.php: SWAT T147881 Call destroy method that actually exists instead of one that doesnt anymore. (duration: 00m 52s)
23:52 logmsgbot: ebernhardson@mira Synchronized php-1.28.0-wmf.22/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.init.js: SWAT T147890 Only enable VE tabs if VE is available (duration: 00m 50s)
23:48 ebernhardson: pulled ve update (315424) to mw1099
23:41 logmsgbot: ebernhardson@mira Synchronized php-1.28.0-wmf.21/extensions/CentralAuth/: SWAT T147029 Add ignorestatus option for fixing stuck renames (duration: 00m 53s)
23:34 logmsgbot: ebernhardson@mira Synchronized w/robots.php: SWAT robots.php: Use WikiPage instead of Article class (duration: 00m 50s)
23:31 ebernhardson: pulled config change (314790) to mw1099
23:30 logmsgbot: ebernhardson@mira Synchronized wmf-config/InitialiseSettings.php: SWAT T143829 Disable bottom language button in Minerva (duration: 00m 50s)
23:29 ebernhardson: pulled config change (315450) to mw1099
23:28 ebernhardson: pulled config change (315314) to mw1099
23:24 ebernhardson: pulled config change (315314) to mw1099
23:23 ejegg|afk: updated fundraising tools from 206799da4d006e6e1e2c747a28039950f47bf2e6 to cbf4dcd734956f95a7485ce1f420d05222c09e5a
23:23 logmsgbot: ebernhardson@mira Synchronized php-1.28.0-wmf.21/extensions/MobileFrontend/includes/skins/SkinMinerva.php: SWAT: Fix logic of MinervaBottomLanguageButton T143829 (duration: 00m 50s)
23:22 logmsgbot: ebernhardson@mira Synchronized php-1.28.0-wmf.22/extensions/MobileFrontend/includes/skins/SkinMinerva.php: SWAT: Fix logic of MinervaBottomLanguageButton T143829 (duration: 00m 50s)
23:18 ebernhardson: pulled MobileFront update to mw1099
23:17 logmsgbot: ebernhardson@mira Synchronized wmf-config/CirrusSearch-common.php: Set defaults for wgCirrusSearchClusterOverrides (duration: 00m 56s)
23:15 logmsgbot: ebernhardson@mira Synchronized wmf-config/InitialiseSettings.php: Set defaults for wgCirrusSearchClusterOverrides (duration: 00m 53s)
23:12 ebernhardson: pulled config change to m21099
23:11 mutante: lead - revoke puppet cert, node clean
22:45 ejegg: disabled donation queue consumer
22:44 ejegg: enabled donation queue consumer
22:43 ejegg: updated CiviCRM from 17fab4ded647bad51b30ce65157c88a87e1f7e40 to 8682821cb591b2861aee1fd167157a1b7aa27abd
22:07 ejegg: disabled donations queue consumer
22:04 ejegg: updated fundraising tools from 112b3fa8cf097c8f702d3824ff169d7a1059f723 to 206799da4d006e6e1e2c747a28039950f47bf2e6
21:53 ejegg: updated fundraising tool from cc7628355e25422bda2b479a009757acbb702072 to 112b3fa8cf097c8f702d3824ff169d7a1059f723
21:44 ejegg: disabled paypal audit parser
21:16 urandom: T133395: Restarting Cassandra instances on restbase2006.codfw.wmnet
21:14 moritzm: repooling all services on scb1001 after earlier revert to nodejs 4.4.6
21:10 urandom: T133395: Restart Cassandra on restbase2005-c.codfw.wmnet
21:06 urandom: T133395: Restart Cassandra on restbase2005-b.codfw.wmnet
21:01 Jeff_Green: replaced pay-lvs1001
20:58 urandom: T133395: Restart Cassandra on restbase2005-a.codfw.wmnet
20:57 ejegg: enabled PayPal audit parsing job
20:55 ejegg: updated fundraising tools from 6e36fd547b97c426bfb6810d5bc2c9fd4b66efa5 to cc7628355e25422bda2b479a009757acbb702072
20:33 ottomata: repooled scb1001 for mobileapps
20:06 Pchelolo: repooling mobileapps on scb1001
20:03 urandom: T133395: Restarting Cassandra: restbase2008-c
20:01 logmsgbot: thcipriani@mira rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.22
19:55 urandom: restarting restbase in codfw
19:51 logmsgbot: thcipriani@mira Finished scap: testwiki to 1.28.0-wmf.22 and rebuild l10n cache (duration: 45m 28s)
19:06 logmsgbot: thcipriani@mira Started scap: testwiki to 1.28.0-wmf.22 and rebuild l10n cache
18:59 urandom: restarting restbase: restbase2004.codfw.wmnet
18:55 elukey: kafka1018 back in service after maintenance
18:40 logmsgbot: thcipriani@mira Synchronized wmf-config/CommonSettings.php: SWAT: Enable $wgPageTriageNoIndexUnreviewedNewArticles on all wikis that have PageTriage (T147544) PART II (duration: 00m 52s)
18:33 urandom: T133395: Restarting Cassandra in RESTBase (codfw) to apply https://gerrit.wikimedia.org/r/314603
18:26 urandom: T133395: Starting dumps (3) in RESTBase Staging
18:20 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable $wgPageTriageNoIndexUnreviewedNewArticles on all wikis that have PageTriage (T147544) PART I (duration: 00m 50s)
18:18 mutante: cobalt (new gerrit) run reviewer-count cron, works now
18:17 mutante: lead (old gerrit) manually remove reviewer-count cron, puppet is disabled
18:13 urandom: T133395: Restarting Cassandra in RESTBase Staging to apply https://gerrit.wikimedia.org/r/314603
18:13 logmsgbot: thcipriani@mira Synchronized dblists/visualeditor-nondefault.dblist: SWAT: Enable the visual editor for logged-in users on remaining phase 6 Wikipedias (T142589) PART II (duration: 00m 59s)
18:11 logmsgbot: thcipriani@mira Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the visual editor for logged-in users on remaining phase 6 Wikipedias (T142589) PART I (duration: 01m 56s)
18:10 twentyafterfour: updated php on iridium
18:09 urandom: T133395: Restarting xenon.eqiad.wmnet to apply https://gerrit.wikimedia.org/r/314603
17:51 bearND: deployed mobileapps fc900fc
17:29 bearND: starting mobileapps deploy
17:25 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/CleanDuplicateScores.php on eight wikis (T145356)
17:20 mutante: mw1272 kernel: [10254957.470558] BUG: Bad page map in process hhvm
17:16 elukey: rebooting kafka1018
17:14 mutante: mw1272 reboot
17:05 thcipriani: starting branch cut for 1.28.0-wmf.22
14:58 hashar: Upgrading Zuul on gallium 2.5.0-8-gcbc7f62-wmf2precise1 2.5.0-8-gcbc7f62-wmf3precise1 (merely a noop for zuul scheduler) T147070
14:57 hashar: Upgrading Zuul on gallium 2.5.0-8-gcbc7f62-wmf2precise1 2.5.0-8-gcbc7f62-wmf3precise1 (merely a noop for zuul scheduler)
14:28 elukey: upgraded zuul on scandium (T147073)
14:20 urandom: T133395: Restarting xenon.eqiad.wmnet to apply https://gerrit.wikimedia.org/r/314603
13:56 hashar: European SWAT is done.
13:33 hashar: mira: purging portals URLs for jan_drewniak_ : cat /srv/mediawiki-staging/portals/urls-to-purge.txt | mwscript purgeList.php
13:14 logmsgbot: hashar@mira Synchronized portals: (no message) (duration: 01m 01s)
13:13 logmsgbot: hashar@mira Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 01m 46s)
12:47 Amir1: on terbium ^
12:47 Amir1: mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=ptwiki --logwiki=metawiki "Zhyar Merlin" "Zhiar Merlin"
12:44 elukey: restarted keyholder-proxy on mira
12:39 moritzm: nodejs reverted to 4.4.6 on scb1001, depooling for service restarts
12:30 elukey: rearming the keyholder on mira
12:27 mobrovac: change-prop scb1001: disabled puppet to try and debug why change-prop master is failing on node v4.6.0
12:14 moritzm: upgrading nodejs on scb2001 to 4.6.0
11:38 elukey: decomissioning the old AQS cluster - aqs100[123] for good https://gerrit.wikimedia.org/r/#/c/314542/
11:37 moritzm: repooling scb1001
11:20 jynus: stopping and starting mysql on labsdb1008 (not active) for new package/config testing
11:18 elukey: reimaging mw1162.eqiad.wmnet to Debian (MW Jobrunner)
11:14 moritzm: depooling scb1001 for service restarts
11:11 moritzm: upgrading nodejs on scb1001 to 4.6.0
11:00 logmsgbot: hashar@mira Synchronized README: testing deploy from mira (duration: 02m 38s)
10:01 moritzm: switching deployment server to mira
08:31 marostegui: Removing Not needed file from dbstore1001 to free up space (/srv/tmp/db1064.tar.gz.enc)
07:56 moritzm: reimaging mw1017 to jessie (test application server in eqiad)
06:50 moritzm: installing django security updates
06:17 marostegui: Deploying schema change S4 commonswiki.revision - T147305
02:38 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 11 02:38:48 UTC 2016 (duration 4m 45s)
02:34 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.21) (duration: 15m 28s)

2016-10-10

21:29 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Add upload_by_url right to Commons bots (duration: 00m 50s)
21:26 logmsgbot: reedy@tin Synchronized wmf-config/InitialiseSettings.php: Fix typo in group name. Add message-format logging group (duration: 00m 50s)
21:20 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: More wfLoadExtension, no config changes (duration: 00m 49s)
21:12 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Remove some legacy cruft that is unused (duration: 00m 50s)
21:08 logmsgbot: reedy@tin Synchronized wmf-config/: Re-enable OAuth on Wikitech T147804 (duration: 00m 52s)
20:39 Reedy: Created up to date oauth tables on wikitech
20:38 Reedy: Dropped oauth tables from wikitech
20:36 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Allow Commons 'crats to manage accountcreator group (T144689) (duration: 00m 50s)
20:16 Amir1: deploying 8bbd3ab to all ores nodes (T146680)
20:09 Amir1: deploying 8bbd3ab to ores canary nodes (T146680)
18:29 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.21/extensions/UploadWizard/resources/controller/uw.controller.Details.js: Don't show warning confirmation dialog when there are no warnings (T147659) (duration: 00m 48s)
18:25 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.21/extensions/ORES/maintenance/CleanDuplicateScores.php: Fixup maintenance/CleanDuplicateScores.php (duration: 00m 54s)
18:23 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.21/includes/Linker.php: Do not normalise external links to special pages (T147685) (duration: 01m 07s)
18:22 logmsgbot: dereckson@tin scap aborted: file php-1.28.0-wmf.21/includes/Linker.php Do not normalise external links to special pages (T147685) (duration: 00m 03s)
18:22 logmsgbot: dereckson@tin Started scap: file php-1.28.0-wmf.21/includes/Linker.php Do not normalise external links to special pages (T147685)
17:20 gehel: upgraded maps1* to postgis 2.3.0 - T144763
15:50 mobrovac: mathoid deploying adb8e548
14:13 moritzm: upgraded PHP on bohrium/piwik.wikimedia.or
14:03 gehel: reimage maps-test200[34] - T147194
13:38 marostegui: Dropping hitcounter, _counter memory tables in S4 - db1040 (master) - T132837
13:20 zeljkof: ending EU SWAT!
13:15 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Activate subphrases autocomplete on wikisources, mw.org and wikitech (T146208) (duration: 00m 50s)
13:12 gehel: reimage maps-test2002 - T147194
12:47 moritzm: uploaded nodejs 4.6.0 for jessie-wikimedia to carbon
12:44 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Restore original weight for db1082 after its RAID controller firmware - T145533 (duration: 00m 55s)
12:06 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.21/extensions/Wikidata: Update Wikibase, add EntityHandler::supportsCategories (T147748) (duration: 02m 25s)
11:52 godog: swift eqiad-prod: ms-be1022 to weight 2000 T136631
11:33 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Increase weight for db1082 after its RAID controller firmware - T145533 (duration: 00m 49s)
10:52 marostegui: Dropping hitcounter, _counter memory tables in S2 - db1069 - T132837
10:42 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1082 with some small weight after its RAID controller firmware - T145533 (duration: 00m 50s)
10:04 jynus: running ALTER TABLE search_documentfield ENGINE=InnoDB, FORCE; on phabricato db replica (db1043)
09:39 marostegui: Dropping hitcounter, _counter memory tables in S2 - db1063- T132837
09:32 moritzm: rolling reboot of swift frontend servers in eqiad for kernel security update
09:30 moritzm: pruning older, unused kernel images on labstore1003
08:57 marostegui: Dropping hitcounter, _counter memory tables in S2 - dbstore1002 - T132837
08:56 jynus: reboot db1043 to test new mysql configuration and general upgrade- proxy will complain
08:53 marostegui: Dropping hitcounter, _counter memory tables in S2 - dbstore1001 - T132837
08:53 godog: reboot graphite2001 and graphite1001 for trusty kernel upgrade
08:40 hoo: Populated the sites/ site_identifiers tables on olowiki (T146614)
08:38 marostegui: Dropping hitcounter, _counter memory tables in S2 - dbstore2002 - T132837
08:12 akosiaris: clear mx1001's queues from backscatter spam T147173
07:34 marostegui: Deploying schema change on S4 codfw only commonswiki.revision - T147305
07:29 moritzm: installing php security updates on jessie systems
06:15 marostegui: db1082: Upgrading RAID controller firmware
06:13 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Depool db1082 to upgrade its RAID controller firmware - T145533 (duration: 00m 50s)
05:42 jynus: reseting slave on es2 eqiad master (es1015)
05:11 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: mariadb: Depool es2015 (master, crashed); replaced by es2016 (duration: 00m 49s)
04:50 jynus: changing topology of es2 @ codfw
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Oct 10 02:30:47 UTC 2016 (duration 4m 51s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.21) (duration: 09m 40s)

2016-10-09

09:07 elukey: chmod o+r /var/lib/varnish/frontend/_.vsm and /var/lib/varnish/cp2008/_.vsm on cp2008 to avoid gmond errors
09:01 jynus: dropping unneded files on db1026 to mitigate disk issues for the next week
08:45 elukey: powercycling cp2008, no ssh and mgmt console frozen
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Oct 9 02:27:12 UTC 2016 (duration 4m 36s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.21) (duration: 08m 59s)

2016-10-08

19:48 bd808: cp2008 Strongswan failures for both ipv4 and ipv6 across a larg number (all?) hosts
09:40 elukey: masked the kafka systemd unit on kafka1018 and re-enabling puppet
09:10 apergos: puppet disabled on kafka1018, leave broker down, bad disk /dev/sdi (see dmesg for sample errors)
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Oct 8 02:31:15 UTC 2016 (duration 6m 20s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.21) (duration: 09m 09s)

2016-10-07

23:16 Krenair: test morebots
23:15 logmsgbot: thcipriani@tin Synchronized w/robots.php: Revert change to robots.php (duration: 00m 49s)
22:22 mutante: etcd servers have puppet issue with Etcd_user[root]
22:04 ejegg: updated payments-wiki from 1fbe171bca30d1fb7a4b7d937a740d87f7c9c8e3 to b4ad60e739b9dbb97f08a3623db961a74682422a
20:30 bblack: lead.wikimedia.org: replaced by cobalt functionally, please leave it untouched for now with puppet disabled!
19:46 mutante: deleted old /var/lib/gerrit2/ data on cobalt, syncing from lead
19:45 mutante: rsyncing /var/lib/gerrit2 from lead to cobalt
19:30 mutante: removed gerrit IPs from cobalt interfaces
19:29 mutante: disabled puppet on lead and cobalt
19:21 mutante: re-enabling puppet on cobalt
19:21 mutante: removed gerrit IP from lead's interface, v4 and v6
19:09 mutante: rsyncing gerrit data one more time from lead to cobalt
19:08 cmjohnson1: db1065 swapping failed disk slot 9 T147396
19:07 ostriches: stopped puppet on lead
19:07 mutante: stopping gerrit on lead
19:02 mutante: cobalt, disabled puppet, removed service IP from interface
17:28 mutante: rsyncing gerrit data from lead to cobalt
16:53 jynus: testing img_metadata nuking for T145953 and T147015 (backups on neodymium)
16:25 gehel: reimage maps-test2001 - T147194
16:14 akosiaris: build python-irclib for jessie and upload it to apt.wikimedia.org jessie-wikimedia/main
16:06 moritzm: updated hhvm package for jessie to 3.12.9
14:41 moritzm: uploaded openssl 1.0.2j for jessie-wikimedia to carbon
14:41 kart_: Update cxserver to fa2f715 (T147552)
13:56 elukey: reimaing mw123[45] to Debian Jessie (last two api appservers)
12:39 elukey: reimaging mw123[23] to Debian Jessie
12:09 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1082 with its original weight - T145533 (duration: 00m 52s)
11:17 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1082 with a bit less weight than usual to start with - T145533 (duration: 00m 55s)
10:52 moritzm: reimaging mw1238, mw1239 to jessie
10:28 elukey: reimaging mw123[01] to Debian Jessie
10:27 elukey: mw122[89] back in live api server pool
10:00 _joe_: updated conftool to 0.3.1 on all the cluster except caches, T147480
09:48 _joe_: creating etcd100[1-6].eqiad.wmnet on ganeti, T147620
09:32 moritzm: reimaging mw1220, mw1236, mw1237 to jessie
09:28 moritzm: installing pillow/python-imaging security updates on Ubuntu systems
09:20 gehel: reimaging maps-test2004 - T147194
09:18 moritzm: installing php security updates on precise hosts
08:53 gehel: reimaging maps-test2003 - T147194
08:33 logmsgbot: oblivian@puppetmaster1001 conftool action : set/weight=20; selector: cluster=api_appserver,dc=eqiad,name=mw123.*
08:31 logmsgbot: oblivian@puppetmaster1001 conftool action : set/weight=0; selector: cluster=api_appserver,dc=eqiad,name=mw123.*
08:20 _joe_: restarting hhvm on a few api appservers, due to memory leaks (T146451)
07:17 elukey: reimaging mw1228 and mw1229 (api appservers) to Debian Jessie
06:32 moritzm: reimaging mw1216, mw1218, mw1219 to jessie
06:31 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Depool db1082 to get its raid controller firmware upgraded - T145533 (duration: 00m 49s)
06:10 _joe_: restarted hhvm, jobrunner on mw1161
05:55 marostegui: Deploying schema change on S4 master commonswiki.revision table - T147113
04:49 kart_: Update cxserver to 84fb704 (T147368)
02:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Oct 7 02:41:54 UTC 2016 (duration 5m 43s)
02:36 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.21) (duration: 16m 03s)
00:54 Dereckson: https://olo.wikipedia.org has been successfully created (T146612).
00:44 Dereckson: mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php olowiki --backend=local-multiwrite
00:39 logmsgbot: dereckson@tin Synchronized wmf-config/interwiki.php: Interwiki cache update for pmid, HTTPS links and olo.wikipedia.org (duration: 00m 50s)
00:25 logmsgbot: dereckson@tin Synchronized langlist: +olo (duration: 00m 49s)
00:22 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Initial configuration for olo.wikipedia.org (T146612) (duration: 00m 50s)
00:18 logmsgbot: dereckson@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
00:17 logmsgbot: dereckson@tin Synchronized dblists: Create olo.wikipedia.org (T146612) (duration: 00m 50s)

2016-10-06

23:46 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.21/extensions/WikimediaMessages/i18n/wikimediainterwikisearchresults/: olo.wikipedia.org project name (duration: 00m 49s)
23:44 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.21/extensions/WikimediaMessages/i18n/wikimediaprojectnames: olo.wikipedia.org project name (duration: 00m 51s)
23:37 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.21/includes/Revision.php: Revision->insertOn: Set READ_LATEST flag (T138310) (duration: 00m 49s)
23:34 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Logo for pt.wikimedia (T126832, 2/2, no-op for the moment) (duration: 00m 50s)
23:31 logmsgbot: dereckson@tin Synchronized static/images/project-logos/: Logo for pt.wikimedia (T126832, 1/2) (duration: 00m 50s)
23:23 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable RelatedArticles on Minerva skin for all but top 6 wikis (T144812) (duration: 00m 50s)
23:11 logmsgbot: dereckson@tin Synchronized wmf-config/throttle.php: Clean expired throttle rules (Gerrit:313166) (duration: 00m 50s)
23:11 ejegg: enabled donations, recurring and refund queue consumers
23:02 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable footer v2 on Minerva for all wikis (T145442) (duration: 00m 50s)
22:38 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.21
22:31 ejegg: enabled donations queue consumer
22:15 ejegg: disabled donations, refund, and recurring queue consumers
22:14 ejegg: disabled paypal audit processor
22:08 twentyafterfour: phd enabled on iridium
22:07 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.21
22:04 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php: Ignore reuseConnection() errors after LoadBalancer/LBFactory destruction (T147520) (duration: 00m 50s)
22:01 ejegg: enabled paypal nightly audit parser
22:01 logmsgbot: ori@tin Synchronized wmf-config/abusefilter.php: If794eb2a: AbuseFilter: Use new parser from I4aea5f00 on Labs (duration: 00m 49s)
21:53 ejegg: updated tools from bde60d5ebbdb89c8ecdf0653e92d29f6a5939ec7 to 6e36fd547b97c426bfb6810d5bc2c9fd4b66efa5
21:53 logmsgbot: thcipriani@tin Synchronized wmf-config/Wikibase-production.php: SWAT: Add config for units on Wikidata (T117032) PART II (duration: 00m 50s)
21:51 logmsgbot: thcipriani@tin Synchronized wmf-config/unitConversionConfig.json: SWAT: Add config for units on Wikidata (T117032) PART I (duration: 00m 48s)
21:38 ejegg: updated payments-wiki from 27ffd8cb01a5820a0a4e143503b1976e858984bd to 1fbe171bca30d1fb7a4b7d937a740d87f7c9c8e3
21:22 mutante: restarting zuul on gallium
21:12 ostriches: lead: enabling & running puppet again, should bring things back up
21:02 akosiaris: rebooting lead one more time
20:17 akosiaris: rebooting lead once more
19:47 ostriches: lead: rebooting, because what have we got to lose
19:23 twentyafterfour: disabled phd and puppet on iridium
18:31 twentyafterfour: stopped phd on iridium to relieve some load on gerrit
18:24 ostriches: lead: restarting apache to force error page to show for now
18:21 ostriches: lead: disabled puppet for now, gerrit's sick
18:04 ostriches: gerrit: kicking gerrit and apache, something is unhappy...
17:11 ema: power-cycling cp2017
16:54 akosiaris: uploaded to apt.wikimedia.org precise-wikimedia/main: php5_5.3.10-1ubuntu3.25+wmf1
16:31 ema: power-cycling cp2022
15:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 6 15:48:38 UTC 2016 (duration 7m 12s)
15:41 logmsgbot: dereckson@tin scap sync-l10n completed (1.28.0-wmf.21) (duration: 15m 46s)
15:41 _joe_: upgrading conftool to 0.3.1 on all mw*, wtp* servers, T147480 T145518
15:25 ema: powercycle cp3045
15:05 logmsgbot: dereckson@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 15m 59s)
15:03 ema: cp3034 hanging during boot, power-cycled
14:41 jynus: restarting db1069:3133 mysql instance
14:40 _joe_: uploaded conftool 0.3.1 to apt.w.o, T147480
14:33 ema: cache_upload: rolling reboots for kernel upgrades
14:26 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.21/extensions/WikimediaMessages/i18n/wikimedia: Wikimedia messages for new 'engineer' group for ruwiki (T144599) (duration: 00m 49s)
14:25 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.20/extensions/WikimediaMessages/i18n/wikimedia: Wikimedia messages for new 'engineer' group for ruwiki (T144599) (duration: 00m 49s)
14:23 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: New 'engineer' group for ruwiki (T144599) (duration: 00m 52s)
13:59 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Configure Visual Editor namespaces on sv.wikipedia (gerrit:309808 and gerrit:314558, T144688) (duration: 00m 50s)
13:54 mobrovac: citoid deploying 4d97774
13:50 elukey: added mw122[67] back to the api appservers live pool
13:50 ema: cache_text: rolling reboots for kernel upgrades
13:38 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Disable Upload Wizard blacklist issues on Commons (T146417) (duration: 00m 49s)
13:23 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.21/extensions/EventBus: Send a resource_change event on page_image property change (T145569) (duration: 00m 48s)
13:16 logmsgbot: dereckson@tin Synchronized wmf-config/CirrusSearch-common.php: Initialize subphrases autocomplete on wikisources, mw.org and wikitech (T146208, 3/3) (duration: 00m 49s)
13:14 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Initialize subphrases autocomplete on wikisources, mw.org and wikitech (T146208, 2/3) (duration: 00m 49s)
13:11 logmsgbot: dereckson@tin Synchronized tests/cirrusTest.php: Initialize subphrases autocomplete on wikisources, mw.org and wikitech (T146208, 1/3, no-op in prod part) (duration: 00m 50s)
12:50 ema: cache_misc: rolling reboots for kernel upgrades
12:36 elukey: reimaging mw122[67] to Debian Jessie
12:33 elukey: adding mw122[45] back to the live api appservers pool (note: mw1224 was pooled => no before the reimage, but I don't see any blocker in adding it back to serve live traffic)
11:57 mobrovac: restbase deploy end of fa4dc79
11:50 moritzm: rebooting video scalers for kernel security update
11:36 mobrovac: restbase deploy start of fa4dc79
11:14 elukey: reimaging mw122[34] to Debian Jessie
11:12 elukey: added mw122[23] back to the api appservers live pool
10:50 mobrovac: change-prop deploying 403eec8
10:27 ema: cache_maps: rolling reboots for kernel upgrades
10:16 moritzm: reimaging mw1209, mw1210, mw1215 to jessie
10:09 ema: power cycling cp2015, reboot failed
10:02 elukey: reimaging mw122[23] to Debian jessie (api appservers)
09:57 ema: cp1046 cp2015 depooled reboot for kernel upgrades
09:55 moritzm: installing jackrabbit security updates on Ubuntu and Debian systems
09:36 elukey: adding mw1208 and mw1221 back to the api appservers live pool
09:22 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Restoring db1082 original weight: 500 (duration: 00m 52s)
08:54 ema: jessie dist-upgrade on cp* cache hosts
08:53 moritzm: reimaging mw1212-mw1214 to jessie
08:27 twentyafterfour: Restarted apache on iridium to apply hotfix to phab calendar form. refs T147525
08:25 moritzm: restarted hhvm on mw1213
08:02 marostegui: Dropping tables in S3.testwiki - T57676
07:43 moritzm: upgrading labtestvirt2001 to Linux 4.4
07:40 marostegui: Dropping tables in S1.enwiki - T57676
06:38 elukey: reimaging mw1208 and mw1221 to Debian Jessie (API appservers)
06:36 moritzm: reimaging mw1187, mw1188, mw1211 to jessie (the latter is a scap proxy)
03:14 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Oct 6 03:14:25 UTC 2016 (duration 7m 1s)
03:07 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.21) (duration: 15m 38s)
02:32 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 11m 53s)
00:15 twentyafterfour: phabricator update complete and service is restored
00:15 bblack: cache_upload: rolling depooled frontend restarts for libvmod-netmapper upgrade
00:11 twentyafterfour: scheduled phabricator update starting momentarily. service will be offline for (hopefully) less than 5 minutes
00:08 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.21/extensions/Flow/: Make more types of exceptions loggable (Gerrit:314452, T135545, T138310) (duration: 01m 12s)
00:03 Dereckson: Created Flow tables on labswiki (wikitech.wikimedia.org)
00:02 bblack: cache_maps: rolling depooled frontend restarts for libvmod-netmapper upgrade

2016-10-05

23:57 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set Flow database for wikitech (T127792) (duration: 00m 50s)
23:54 bblack: cache_misc: rolling depooled frontend restarts for libvmod-netmapper upgrade
23:38 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Always set wgFlowDefaultWikiDb (Gerrit:314194 and Gerrit:314453) (duration: 00m 50s)
23:22 bblack: rebooting radon for kernel update (ns0.wikimedia.org)
23:13 logmsgbot: dereckson@tin Synchronized dblists/commonsuploads.dblist: Disable local upload on bat-smg.wikipedia (T142632) (duration: 00m 49s)
23:09 logmsgbot: dereckson@tin Synchronized wmf-config/CirrusSearch-common.php: Cirrus: Support document versioning (T144039) (duration: 00m 50s)
22:49 bblack: rebooting primary LVS hosts for kernel updates
22:43 ejegg: updated civicrm from 8bc490854c397cca1a0fa75f313e2ce5f068d3a1 to 17fab4ded647bad51b30ce65157c88a87e1f7e40
22:35 bblack: jessie dist-upgrade on primary LVS servers
22:28 awight: update fundraising-tools from 5427a601ed64ecfe01ef84eda13f28c0dd08a3af to bde60d5ebbdb89c8ecdf0653e92d29f6a5939ec7
22:25 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.21/includes/libs/rdbms/loadbalancer/LoadBalancer.php: Add more information to reuseConnection() exceptions (duration: 00m 51s)
22:15 bblack: rebooting secondary (inactive) LVS hosts for kernel updates
22:00 ejegg: updated civicrm from d52c04db4de38d3961661d719b462d27f44c831c to 8bc490854c397cca1a0fa75f313e2ce5f068d3a1
21:59 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.21/extensions/TimedMediaHandler: Revert "Rewrite discovery of TimedText tracks" (duration: 00m 54s)
21:30 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 to 1.28.0-wmf.20
21:14 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.21
21:09 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.21/includes/libs/rdbms: Make LoadMonitor use $serverIndexes in the cache key (T147359) PART II (duration: 00m 55s)
21:08 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.21/maintenance/lag.php: Make LoadMonitor use $serverIndexes in the cache key (T147359) PART I (duration: 00m 50s)
20:34 Krenair: ran package updates on wikitech-static vm
19:56 Pchelolo: deploy RESTBase 810b6aa563
19:52 bblack: jessie dist-upgrade on secondary LVS servers
19:37 Pchelolo: deploy RESTBase 810b6aa563 canary on restbase1007
19:15 gehel: rebooting maps1* for kernel upgrade
19:05 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: remove dumb commented setting, dumb me (duration: 00m 49s)
18:51 awight: update fundraising CRM from 5f53ef867cf3c2bb2f919c2feb9323eededfb7e7 to d52c04db4de38d3961661d719b462d27f44c831c
18:46 urandom: T146211: Performing rolling restart of RESTBase eqiad rack 'd' Cassandra instances, and marking SSTables unrepaired.
18:39 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: disable gzip internally, T125938 (duration: 00m 50s)
18:25 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.21/extensions/TimedMediaHandler/: fix fatal (duration: 00m 54s)
18:18 XenoRyet: updated civicrm from 412d999b671e13982d718437ab3be56efaa25baf to 5f53ef867cf3c2bb2f919c2feb9323eededfb7e7
18:17 urandom: T146211: Performing rolling restart of RESTBase rack 'b' Cassandra instances, and marking SSTables unrepaired.
17:58 urandom: T146211: Performing rolling restart of restbase1011.eqiad.wmnet Cassandra instances, and marking SSTables unrepaired.
17:35 moritzm: installing chromium security updates on osmium
17:32 urandom: T146211: Performing rolling restart of restbase1010.eqiad.wmnet Cassandra instances, and marking SSTables unrepaired.
17:23 moritzm: installing libav security updates
16:35 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Don't grant editcontentmodel to all users yet (duration: 01m 01s)
16:28 godog: upgrade mysqld_exporter to 0.9.0 on db2030 T147476
15:54 urandom: T146211: Restarting Cassandra on restbase1007-c.eqiad.wmnet to mark parsoid.data-parsoid tables unrepaired
15:48 urandom: T146211: Restarting Cassandra on restbase1007-b.eqiad.wmnet to mark parsoid.data-parsoid tables unrepaired
15:44 paravoid: upgrading JunOS on cr2-knams
15:39 urandom: T146211: Restarting Cassandra on restbase1007-a.eqiad.wmnet to mark parsoid.data-parsoid tables unrepaired
15:38 moritzm: restarted hhvm on mw1274, was stuck
15:35 godog: reimage lithium with bigger disks T143307
15:18 paravoid: upgrading JunOS on cr1-esams
15:04 godog: add lpxelinux.0 to volatile/tftpboot on puppet.eqiad.wmnet
14:33 logmsgbot: gehel@puppetmaster1001 conftool action : set/pooled=yes; selector: dc=codfw,cluster=wdqs,service=wdqs
14:32 logmsgbot: gehel@puppetmaster1001 conftool action : set/pooled=yes; selector: dc=eqiad,cluster=wdqs,service=wdqs
14:27 gehel: restarting pybal on lvs1003 - T132457
14:25 cmjohnson1: db1055 replacing disk slot 0
14:22 gehel: restarting pybal on lvs1006 - T132457
14:18 gehel: restarting pybal on lvs1003 - T132457
14:17 bblack: rebooting baham (ns1.wikimedia.org)
14:17 gehel: restarting pybal on lvs1012 - T132457
14:15 gehel: restarting pybal on lvs1009 - T132457
14:11 gehel: restarting pybal on lvs1006 - T132457
14:02 bblack: rebooting eeden (ns2.wikimedia.org)
13:58 paravoid: upgrading JunOS on cr1-ulsfo
13:47 elukey: adding mw120[67] back to the api appservers live pool after reimage
13:46 gehel: deploying new LVS configuration for WDQS service - T132457
13:42 moritzm: upgrading neodymium to Linux 4.4
13:32 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: wmf-config/db-codfw.php Remove db1019 entries as it is going to be decommissioned - T146265 (duration: 00m 49s)
13:32 paravoid: upgrading JunOS on cr2-ulsfo (attempt 2)
13:17 bblack: upgrading kernel packages on cp* cache hosts (no reboots yet)
13:07 paravoid: upgrading JunOS on cr2-ulsfo
12:50 marostegui: dropping views jamwiki_p.abuse_filter_history drop view adywiki_p.abuse_filter_history - T147413
12:14 logmsgbot: reedy@tin Synchronized php-1.28.0-wmf.21/extensions/UserMerge: Fix fatal when using special page (duration: 00m 50s)
12:10 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=apertium'])
11:44 elukey: reimaging mw120[67] to Debian Jessie
11:23 moritzm: reimaging mw1184-mw1186 to jessie
10:41 moritzm: reimaging mw1181-mw1183 to jessie
10:29 elukey: adding mw120[01] back to the mw api live pool after reimage
10:18 kart_: Update cxserver to 0b2c3fa (T144588)
10:12 godog: reimage bast3001 with /srv partition scheme
10:05 akosiaris: restart pybal on lvs1003, lvs2003 T147288
09:43 akosiaris: restart pybal on lvs1006, lvs1009, lvs1012, lvs2006 T147288
09:40 akosiaris: pool all scb hosts for apertium service
09:39 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=apertium'])
09:39 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb2002.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=apertium'])
09:39 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=apertium'])
09:39 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=apertium'])
09:29 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=cxserver'])
09:22 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=cxserver'])
09:22 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=yes; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=cxserver'])
09:21 akosiaris: enable puppet on scb1002. T147288
09:11 ema: repooling varnish-be-rand on cp2014 and cp1073 T147209
08:42 moritzm: installing PHP security updates on Ubuntu systems
08:28 moritzm: reimaging mw1172,mw1179, mw1180 to jessie
08:20 logmsgbot: akosiaris@puppetmaster1001 conftool action : set/pooled=no; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=cxserver'])
08:12 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Increase weight for db1082 from 100 to 300 (duration: 00m 52s)
08:11 akosiaris: T147288 disable puppet on scb1001, scb1002, scb2001, scb2002
08:10 akosiaris: disable puppet on scb1001, scb1002, scb2001, scb2002
07:57 elukey: reimaging mw120[01] to Debian Jessie (mw1201 is a scap proxy)
07:08 moritzm: reimaging mw1176-mw1178 to jessie
03:20 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Oct 5 03:20:16 UTC 2016 (duration 7m 7s)
03:13 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.21) (duration: 19m 19s)
02:37 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 13m 47s)
00:05 logmsgbot: maxsem@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/#/c/314206/1 (duration: 00m 49s)
00:01 logmsgbot: maxsem@tin Synchronized wmf-config/: (no message) (duration: 00m 52s)
00:00 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/313158/2 (duration: 01m 57s)

2016-10-04

23:39 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/313157/2 (duration: 01m 38s)
23:31 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/313156/2 (duration: 00m 50s)
23:29 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/313156/2 (duration: 00m 57s)
23:24 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/313155/2 (duration: 00m 49s)
23:20 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/313155/2 (duration: 00m 49s)
23:13 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/313154/2 (duration: 00m 50s)
22:41 XenoRyet: roll back civicrm from 5f53ef867cf3c2bb2f919c2feb9323eededfb7e7 to 412d999b671e13982d718437ab3be56efaa25baf
22:29 XenoRyet: updated civicrm from 412d999b671e13982d718437ab3be56efaa25baf to 5f53ef867cf3c2bb2f919c2feb9323eededfb7e7
22:08 ejegg: updated payments-wiki from e6027d57b021509d4dd3668aa3b67c10b3a4e246 to 27ffd8cb01a5820a0a4e143503b1976e858984bd
21:44 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.21
21:37 bblack: cache_upload: rolling frontend restarts for https://gerrit.wikimedia.org/r/#/c/313847/ (sequential depooled, ~30s per host)
21:36 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: testwiki to 1.28.0-wmf.21
21:34 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.21/includes/libs/rdbms/loadmonitor/LoadMonitor.php: Add version to LoadMonitor::getCacheKey() (T147359) (duration: 00m 53s)
21:05 ejegg: enabled recurring donation consumer
20:59 ejegg: updated civicrm from 7502c0bfec772a9d2a9bccee05ee42a685818217 to 412d999b671e13982d718437ab3be56efaa25baf
20:47 ejegg: updated civicrm from 51b790b6e866f9be2d55a85873b1273923681127 to 7502c0bfec772a9d2a9bccee05ee42a685818217
20:46 ejegg: disabled recurring donation queue consumer
20:38 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: testwiki back to 1.28.0-wmf.20
20:22 logmsgbot: thcipriani@tin Finished scap: testwiki to 1.28.0-wmf.21 and rebuild l10n cache (duration: 55m 51s)
20:03 Pchelolo: RESTBase deploy 810b6aa563 to staging
19:26 logmsgbot: thcipriani@tin Started scap: testwiki to 1.28.0-wmf.21 and rebuild l10n cache
19:09 XenoRyet: update civicrm from b45b155befaf0f2f9ff663df156c1882e34b429c to 51b790b6e866f9be2d55a85873b1273923681127
18:40 RoanKattouw: Running extension/Echo/removeInvalidNotification.php on testwiki, test2wiki and mediawikiwiki (T147138)
18:33 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Flow opt in: Temporarily disable all, MW.org is redundant (Gerrit:314042) (duration: 00m 50s)
18:25 thcipriani: cutting branch 1.28.0-wmf.21 of mediawiki and extensions
18:12 XenoRyet: update civicrm from e2b5bbfbdaaad29925fc60586ce7a2da8297cc2d to b45b155befaf0f2f9ff663df156c1882e34b429c
18:10 awight: fundraising campaigns reenabled
17:57 yurik: deployed tilerator (disabled on maps-test*) https://gerrit.wikimedia.org/r/#/c/314030/
17:51 awight: disabled fundraising campaigns
17:44 logmsgbot: krinkle@tin Synchronized docroot/noc/db.php: (no message) (duration: 00m 48s)
17:38 yurik: deployed kartotherian https://gerrit.wikimedia.org/r/#/c/314018/ -- Possible issue https://phabricator.wikimedia.org/T147334
17:19 yurik: deploying kartotherian & tilerator updates
16:55 logmsgbot: krinkle@tin Synchronized docroot/noc: (no message) (duration: 01m 01s)
16:32 mutante: new wiki language Livvi-Karelian -> olo.wikipedia.org has been added to DNS (T146612)
16:30 moritzm: upgrading labvirt1014 to Linux 4.4
16:30 mutante: authdns commands from T97051#1994679 to add olo.wp for T146612
16:29 mutante: authdns-gen-zones -f /srv/authdns/git/templates /etc/gdnsd/zones && gdnsd checkconf && gdnsd reload-zones
15:18 elukey: adding mw120[45] back to the api live pool after reimage
15:00 volans: created marostegui account into Racktables
14:50 godog: eqiad-prod: ms-be1022 to weight 1000 T136631
14:11 cmjohnson1: ms-be1002 replacing failed disk slot 11
14:07 cmjohnson1: db1055 swapped disk 0
13:40 hoo: Updated Wikidata's property suggester with data from Monday's json dump and applied the T132839 workarounds
13:33 marostegui: Remove db1019 from prometheus also adding it to spare as it is going to be decommissioned
13:33 godog: upgrade grafana to 3.1.1 on labmon1001 - T146354
13:16 logmsgbot: hashar@tin Synchronized rpc/RunJobs.php: trick mw into generating a raw exception report (duration: 00m 47s)
13:08 elukey: reimage mw120[45] to Jessie
13:04 hashar: Purged namespace 0 pages for arbcom_nlwiki (T147186) via: mwscript purgeList.php --wiki=arbcom_nlwiki --namespace=0 --verbose
13:04 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Enable subpages for main namespace in arbcom_nlwiki T147186 (duration: 00m 49s)
12:56 logmsgbot: hashar@tin Synchronized wmf-config/throttle.php: [throttle] Increase account creation limits for an event in Perpignan on 201 T147293 (duration: 00m 50s)
11:14 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Removing db1019 entry as it is going to be decommissioned - T146265 (duration: 00m 51s)
11:14 elukey: adding mw120[23] back to the live api servers pool
10:23 elukey: installed memcached 1.4.28-1.1+wmf1 on mc2009 as part of a performance test - T129963
10:07 elukey: reimaging mw120[23] to Jessie
09:56 elukey: adding mw119[89] to the live api server pool (volans provides magic)
09:03 hashar: Regenerating configuration of all Jenkins job due to https://gerrit.wikimedia.org/r/#/c/313306/
08:44 elukey: reimaging mw119[89] to jessie
07:09 elukey: rebooting eventlog1001 for kernel upgrades
07:04 elukey: executed salt -C 'G@cluster:jobrunner and G@site:eqiad' cmd.run 'find /var/log/hhvm/ -type f -user root -exec chown www-data:www-data {} \;' (also in codfw) to reduce cronspam
02:41 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Oct 4 02:41:38 UTC 2016 (duration 4m 55s)
02:36 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 18m 19s)
01:09 Dereckson: echo 'https://noc.wikimedia.org/' | mwscript purgeList.php
00:26 logmsgbot: dereckson@tin Synchronized docroot/noc/index.html: Remove dead pybal link on noc. (Gerrit:313162) (duration: 00m 48s)
00:08 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.20/extensions/Flow/includes/BoardMover.php: SWAT: BoardMover: do not try to save a null edit (T138310) (duration: 00m 49s)
00:04 gwicke: Started run of exportRestrictions script on terbium (T135278); this is running in screen as user gwicke. It is not expected to generate noticeable load.

2016-10-03

23:45 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Wikidata descriptions on Japanese and Spanish Wikipedias (T145786) (duration: 00m 49s)
23:36 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.20/resources/lib/oojs-ui: SWAT: Update OOjs UI to v0.17.10 (duration: 00m 48s)
23:34 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.20/vendor: SWAT: Update OOjs UI to v0.17.10 (duration: 01m 33s)
23:24 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Disable wmgEchoFooterNotice (duration: 00m 49s)
23:15 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Reduce number of replicas for titlesuggest indices (T147192) (duration: 00m 51s)
22:26 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1091 with regular weight (duration: 00m 51s)
22:08 jynus: running schema change (innodb conversion) on phabricator db hosts T146673
21:54 cscott: OCG deploy temporarily disabled PDF render on en.wiktionary.org to combat DoS.
21:50 cscott: updated OCG to version 0bf27e3452dfdc770317f15793e93e6e89c7865a (T147211, T144120)
21:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1091 with low weight after maintenance (duration: 00m 50s)
21:47 cscott: starting OCG deploy
21:01 jynus: disabling puppet on labsdb1002 and shutting it down for decommission
20:57 bearND: deployed mobileapps 17bc059
20:53 bearND: starting mobileapps deploy
20:32 yurik: deployed graphoid update - https://gerrit.wikimedia.org/r/#/c/313887/
20:28 yurik: about to deploy graphoid update - https://gerrit.wikimedia.org/r/#/c/313887/
20:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1091 (duration: 00m 48s)
19:48 cscott: cleared OCG queue again, while I work on a blacklist patch for the OCG frontend
19:33 ejegg: updated payments-wiki settings
19:28 XenoRyet: updated payments-wiki from cc27f83f31ecc609d4400050e73905b7364f1d42 to e6027d57b021509d4dd3668aa3b67c10b3a4e246
18:56 cscott: cleared rapidly-growing OCG queue w/ mw-ocg-service/scripts/clear-queue.js to cope with someone trying to render all of enwiktionary to PDF.
18:47 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.20/includes/exception/MWExceptionHandler.php: Restore prior render() logic (T147122) (duration: 00m 48s)
18:39 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Set $wgDefaultExternalStore for wikitech before Flow settings (T127792) (duration: 01m 04s)
18:36 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.20/includes/exception/MWExceptionHandler.php: Restore delegation to MWException::report (T147098) (duration: 00m 48s)
18:34 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Use === for $wgDBname comparisons (duration: 01m 53s)
18:24 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow beta feature on elwiki (T144384) (duration: 00m 49s)
18:15 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Enable PageAssessments on enwiki (T146679) (duration: 00m 49s)
17:17 ejegg: updated SmashPig from efe8720e454b03740e4fb4e846657cc92d5ecf7c to fa0267b6f23505d835fd1557a82c2ea99a6985d8
17:16 gehel: deploying latest wdqs updater and gui
16:19 ejegg: updated payments-wiki settings to stop sending completed donations to ActiveMQ
15:15 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 after its maintenance to its original value: 500 - T147113 (duration: 00m 48s)
14:58 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Increase weight for db1084 after its maintenance - T147113 (duration: 00m 48s)
14:53 chasemp: adding volans (RCoccioli) to phab security, confirmed staff account association and membership in ops acl already, confirmed w/ riccardo he is missing, and there is a long standing agreement all members of ops should be in #security
14:37 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1084 after its maintenance - T147113 (duration: 00m 48s)
14:32 zeljkof: ending EU SWAT
14:28 logmsgbot: zfilipin@tin Synchronized robots.txt: SWAT: Fix an invalid empty line in the global robots.txt (T146908) (duration: 00m 47s)
14:24 logmsgbot: zfilipin@tin Synchronized static/images/project-logos/olowiki-2x.png: SWAT: Add 1.5 and 2x logos for olowiki (T146745) (duration: 00m 48s)
14:23 logmsgbot: zfilipin@tin Synchronized static/images/project-logos/olowiki-1.5x.png: SWAT: Add 1.5 and 2x logos for olowiki (T146745) (duration: 00m 48s)
14:20 hashar: T146271 mwscript purgeList.php --wiki=testwikidatawiki --namespace=121 --verbose
14:19 hashar: Purged wikidata wiki property talk page, they now allow subpages (T146271). Ran: mwscript purgeList.php --wiki=wikidatawiki --namespace=121 --verbose
14:15 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable subpages in 121 namespace in wikidata (T146271) (duration: 00m 49s)
14:02 zeljkof: extending EU SWAT
13:55 logmsgbot: zfilipin@tin Synchronized static/images/project-logos/olowiki.png: SWAT: Upload 1x logo for olowiki (T146745) (duration: 00m 48s)
13:51 logmsgbot: zfilipin@tin Synchronized static/images/project-logos/hewiki.png: SWAT: Fix hewiki logos (T145017) (duration: 00m 47s)
13:38 godog: reenable puppet on scb1* ores/celery spamming is over
13:31 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Change protection level autoreview in arwiki (T146575) (duration: 00m 48s)
13:23 dcausse: elasticsearch@eqiad: reducing replica count from 5 to 2 for jawiki_titlesuggest and eswiki_titlesuggest
13:21 logmsgbot: zfilipin@tin Synchronized wmf-config/throttle.php: SWAT: [throttle] Rule for Winona State University (T146600) [throttle] Ada Lovelace Day Edit-a-thon (T146654) (duration: 00m 49s)
13:19 dcausse: elasticsearch@eqiad: reducing replica count from 5 to 2 for ruwiki_titlesuggest
13:15 dcausse: elasticsearch@eqiad: reducing replica count from 5 to 2 for zhwiki_titlesuggest
13:13 gehel: reimage of maps-test2001 - T147194
13:11 dcausse: elasticsearch@eqiad: reducing replica count from 5 to 2 for frwiki_titlesuggest
13:09 gehel: shutting down services on maps-test* servers prior to reimage -T147194
13:02 zeljkof: starting EU SWAT
13:00 dcausse: elasticsearch@eqiad: reducing replica count from 5 to 3 for enwiki_titlesuggest
12:52 dcausse: elasticsearch@eqiad: reducing replica count from 5 to 3 for dewiki_titlesuggest
12:23 marostegui: Deploying alter table in S4 - T147113
12:14 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Depool db1084 for maintenance - T147113 (duration: 00m 48s)
11:51 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Increase db1081 weight to its original value after finishing maintenance - T147113 (duration: 00m 48s)
11:21 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Increase db1081 weight after finishing its maintenance - T147113 (duration: 00m 48s)
10:33 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Increase db1081 weight after finishing its maintenance - T147113 (duration: 00m 48s)
10:21 akosiaris: restarting slapd on dubnium.wikimedia.org T143302
10:16 akosiaris: restarting slapd on seaborgium.wikimedia.org T143302
10:13 akosiaris: restarting slapd on serpens.wikimedia.org T143302
10:11 akosiaris: restarting slapd on pollux.wikimedia.org T143302
09:54 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1081 after finishing its maintenance - T147113 (duration: 00m 49s)
09:48 elukey: lowered down builds log retention from 90 to 60 days for the puppet compiler (https://integration.wikimedia.org/ci/job/operations-puppet-catalog-compiler/)
09:32 akosiaris: T147173 clean exim queues on mx1001 from backscatter spam. Seems to be originating from mx.{east,west}.cox.net, blocked them for now
09:28 marostegui: dbstore2001 going to be reimaged as jessie
09:27 gehel: rolling restart of elasticsearch codfw cluster for kernel upgrade - T146123
09:14 akosiaris: T147173 clean exim queues on mx1001 from backscatter spam
09:08 akosiaris: clean exim queues on mx1001 from backscatter spam
08:46 elukey: rebooted compiler02.puppet3-diffs.eqiad.wmflabs (not reachable by Jenkins, pingable from bastions but no ssh available)
08:04 _joe_: powercycling mw1207
07:56 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Depool db1081 for maintenance - T147113 (duration: 00m 50s)
07:24 volans: emptying /var/log/debug on dubnium because of disk full (the same data is on syslog) T147173
06:30 marostegui: altering S3,S4,S5,S6,S7 user_groups tables in sanitarium to avoid tokudb bug - T146121
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 13m 16s)

2016-10-02

07:29 gehel: silencing wdqs response time alerts, it is flapping, related to traffic - T147130
04:58 cwd|afk: updated smash pig from 4b36376f4b206406b5b88661cfcecf1b588d5bcf to efe8720e454b03740e4fb4e846657cc92d5ecf7c
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 13m 07s)

2016-10-01

11:03 Amir1: ladsgroup@terbium:~$ mwscript extensions/CentralAuth/maintenance/fixStuckGlobalRename.php --wiki=commonswiki --logwiki=metawiki Gautehuus Neuraxıs
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 12m 51s)

2016-09-30

22:06 Krinkle: Re-run mwscript deleteEqualMessages.php on all wikis it was previously run on (T45917)
18:49 ejegg: updated SmashPig from 8ff1950ccd87c649f1748f25e1a0a708c3337206 to 4b36376f4b206406b5b88661cfcecf1b588d5bcf
17:57 ejegg: updated civicrm from 18e59abac57ba85ee9d9dbd50f9f25df64522974 to e2b5bbfbdaaad29925fc60586ce7a2da8297cc2d
13:33 yuvipanda: restart grafana-server on labmon1001
02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 30 02:33:12 UTC 2016 (duration 4m 49s)
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 13m 31s)
00:29 matt_flaschen: Manually updated the DB to fix already-broken cases caused by since-fixed T138310
00:25 ejegg: updated civicrm from 637659ee8257562492405385d3fadaee53db998b to 18e59abac57ba85ee9d9dbd50f9f25df64522974
00:12 ejegg: updated SmashPig from 3811f0f1c4bed1bd0b02264b5865ae36021cb275 to 8ff1950ccd87c649f1748f25e1a0a708c3337206
00:12 ejegg: updated SmashPig // FIXME: var map can't put one thing in two places
00:07 ejegg: re-enabled donations queue consumer

2016-09-29

23:38 ejegg: updated CiviCRM from 6768613907598c52b0af1ffae35faa9078f15f63 to 637659ee8257562492405385d3fadaee53db998b
23:25 ejegg: disabled donations queue consumer
23:06 ejegg: enabled mirroring completed donations queue from payments-wiki
23:03 ejegg: updated SmashPig from 2169b71016e2deb1114655d0256d3286ff057943 to 3811f0f1c4bed1bd0b02264b5865ae36021cb275
21:14 ejegg: updated SmashPig from 077ffcc3c4485f62a2f9c80eb4843ef8c72d0c4f to 2169b71016e2deb1114655d0256d3286ff057943
18:25 ejegg: disabled CiviCRM dedupe jobs
18:24 bd808: https://tools.wmflabs.org/sal/ missing some entries for 2016-09-29; consider https://wikitech.wikimedia.org/wiki/Server_Admin_Log canonical
18:21 cwd: rolled forward PaymentListeners again
17:54 cwd: updated smashpig from 0d88feaf8ecab0286e36a91303bb234c68fd6384 to 077ffcc3c4485f62a2f9c80eb4843ef8c72d0c4f
17:25 cwd: rolled back PaymentListeners
17:05 cwd: updated PaymentListeners from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c
16:59 cwd: updated smashpig from 40c4a7c664dc53f16943aa0b83f30ab1ce435c15 to 0d88feaf8ecab0286e36a91303bb234c68fd6384
16:51 ejegg: enabled adyen job runner
16:48 cwd: updated smashpig from 3458f93599084815da46a2540e9ed762c8b120ce to 40c4a7c664dc53f16943aa0b83f30ab1ce435c15
16:34 elukey: executed 'sudo salt -C 'G@cluster:imagescaler and G@site:eqiad' cmd.run 'find /var/log/hhvm/ -type f -user root -exec chown www-data:www-data {} \;' to reduce cronspam
16:32 elukey: executed 'sudo salt -C 'G@cluster:imagescaler and G@site:codfw' cmd.run 'find /var/log/hhvm/ -type f -user root -exec chown www-data:www-data {} \;' to reduce cronspam
16:32 urandom: T133395: restbase staging: starting bootstrap of restbase-test2001-b.codfw.wmnet (test of decomm/bootstrap under time-windowed compaction)
15:18 urandom: T133395: restbase staging: decommissioning restbase-test2001-b.codfw.wmnet (test of decomm/bootstrap under time-windowed compaction)
13:29 cwd|afk: disabled Adyen job runner
10:28 hashar: Upgrading Jenkins plugins with zeljkof :]
09:32 robh: received notification of ulsfo.1.23.pdu flapping power status via united layer icinga, yet checking router shows no power interruption for cr1-ulsfo. seems to be a monitoring false alarm (from united layers end, not ours)
08:38 logmsgbot: reedy@tin Synchronized wmf-config/mobile-labs.php: Remove transfers of non existent $wmg variables (duration: 00m 48s)
02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 29 02:32:47 UTC 2016 (duration 4m 46s)
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 12m 42s)
02:14 eileen: civicrm upgraded from 86371233a9218a526f85cd76d7008f966042fde7 to 6768613907598c52b0af1ffae35faa9078f15f63
01:04 cwd: rolled smashpig back because JSON_UNESCAPED_UNICODE is unavailable in php5.3 and don't want consumers to explode
00:56 cwd: updated SmashPig from 3458f93599084815da46a2540e9ed762c8b120ce to 3d0a76b5918f280602e5dabeeb0373c1a118590d

2016-09-28

23:51 ejegg: updated SmashPig from d2589271d43d8895fbb928f54f8feb43fcbc43c9 to 3458f93599084815da46a2540e9ed762c8b120ce
22:06 cwd: PaymentListeners rolled back
21:54 cwd: PaymentListeners updated from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7
21:33 Krenair: Fixed labs 205.21.68.10.in-addr.arpa. entry to remove another broken contintcloud name, unbreaking beta scap
21:25 XenoRyet: update SmashPig from 372cd4008fee3fd02ad2eae9163cf7b28d2ef7c8 to d2589271d43d8895fbb928f54f8feb43fcbc43c9
20:51 logmsgbot: apergos is awesome and made the bot work again by restarting it
20:50 apergos: restarted logmsgbot on neon
20:47 bd808: logmsgbot seems to be down: "error: [Errno 111] Connection refused" from scap sync-file
20:46 bd808: scap sync-file wmf-config/throttle.php "IP cap lift for eswiki on 2016-09-30 (T146788)"
19:48 awight: update fundraising CRM from 6b2bd98fce5006030423bccc4f4b7fd9b5d14821 to 86371233a9218a526f85cd76d7008f966042fde7
19:22 XenoRyet: reverted SmashPig from 4b930ada57d99070713be18e235238078e1c1e48 to 372cd4008fee3fd02ad2eae9163cf7b28d2ef7c8
17:22 jynus: restarting db1069.s3 (stagnant replication)
16:42 awight: update CRM from 88000acc826592e535491bde6a47f8976deaf07a to 6b2bd98fce5006030423bccc4f4b7fd9b5d14821
08:15 twentyafterfour: twentyafterfour@iridium:/srv/phab/phabricator$ sudo bin/search index --type PhabricatorProject --force
03:34 eileen: tools upgrade from b0be0f9ca04191c4bab869bb81191c5c77c432ca to 5427a601ed64ecfe01ef84eda13f28c0dd08a3af
01:20 eileen: updating CiviCRM from 26e5214e66f2e4b5de89c27dd5121dfde43269e2 to 88000acc826592e535491bde6a47f8976deaf07a
01:05 eileen: civicrm upgrade from d30a5e454e6614214faef8354a3a23d71d573c7f to 26e5214e66f2e4b5de89c27dd5121dfde43269e2

2016-09-27

05:32 _joe_: rebooting ms-be1002, stuck in a failed disk

2016-09-26

19:37 awight: enabling awight_test5 banner at 1% of nlwiki
18:09 ejegg: rolled back paypal IPN listener to b4d77a991e100f97d98fcd72eaf03940a4e1845d
17:59 ejegg: updated standalone paypal IPN listener from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c
17:47 ejegg: rolled back paypal IPN listener to b4d77a991e100f97d98fcd72eaf03940a4e1845d
17:39 ejegg: updated standalone paypal IPN listener from b4d77a991e100f97d98fcd72eaf03940a4e1845d to 21647c8f4b781b74ae2dc4377334410b4eed7e3c
13:52 marostegui: phabricator is back in write mode - search is degraded. we are regenerating the indexes
13:52 chasemp: iridium phab ./bin/search index --all
03:39 cwdent_: disabled civicrm dedupe contacts job

2016-09-25

16:26 Amir1: deploying 9cc2009 on ORES nodes (T146581)
15:24 elukey: executed https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_.2Fmnt.2Fhdfs on stat1002 (fusermount didn't succeed to umount though)
14:39 gehel: restarting blazegraph on wdqs1002 - T146576
14:30 gehel: putting wdqs1002 in maintenance mode, server looks unstable, investigating... - T146576
14:28 godog: temporarily redirect ores-celery-worker logs to /srv/log/celery/syslog.log and remove old daemon.log.1 to avoid scb100* filling up the / filesystem

2016-09-24

19:30 ema: hhvm 1283-1290 rolling restart
12:21 godog: apply temporary cleanup of old (+20m) thumbor temporary files - T146262
10:47 _joe_: systemctl restart thumbor-instances.service on thumbor1001 freed 3 GB of space
02:45 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 24 02:44:59 UTC 2016 (duration 5m 57s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 16m 49s)

2016-09-23

22:05 matt_flaschen: Deployed patch for T146425
21:42 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: Additional logging to track down autocomplete timing regression (duration: 00m 50s)
20:52 gehel: cleaning up leftover system unit files on wdqs1*
18:41 gehel: killing stuck tilerator notification processes on maps1001 - T145534
17:57 mutante: mira restarted cron
17:53 ejegg: updated SmashPig from 8ac116037440746eaf64b9e99e1ee962d5d33475 to 372cd4008fee3fd02ad2eae9163cf7b28d2ef7c8
17:46 logmsgbot: thcipriani@tin Synchronized README: Test sync for new mira (duration: 01m 27s)
17:43 mutante: mira - changing UID of l10nupdate to 10002, chown'ing files (1001 -> 10002)
17:35 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/ElasticsearchIntermediary.php: Add timing marks to narrow down autocomplete timing regression (duration: 00m 50s)
17:31 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.20/extensions/CirrusSearch/includes/CompletionSuggester.php: Add timing marks to narrow down autocomplete timing regression (duration: 18m 43s)
17:04 mutante: stat1002 - before it was hanging and then fixed due to https://wikitech.wikimedia.org/wiki/Analytics/Cluster/Hadoop/Administration#Fixing_HDFS_mount_at_.2Fmnt.2Fhdfs
17:03 mutante: stat1002 - starting nagios-nrpe-server
14:55 jynus: deployed dns update (removing db1010) T129395
12:20 moritzm: rearmed keyholder on mira
12:03 _joe_: rolling restart of mw1280-90, high cpu usage due to memory leaks.
10:16 moritzm: reimaging mira to jessie (again, previously installer config still pointed to trusty)
10:05 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=wikidatawiki (T146461) and for 'trwiki', 'plwiki', 'fawiki', 'nlwiki', 'ruwiki', 'ptwiki'
10:00 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=enwiki
09:58 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.20/extensions/ORES/includes/Cache.php: No int typehinting (causes jobs to crash) T146461 (duration: 00m 42s)
09:58 moritzm: rearmed keyholder on mira
09:48 jynus: disabling alerts and shutting down db1010 in preparation for decommissioning T129395
09:08 moritzm: reimaging mira to jessie
09:06 elukey: reboot eventlog2001.codfw.wmnet for kernel upgrades
08:52 elukey: upgrading varnishkafka to 1.0.12-1 in cache:misc
08:44 ema: depooled nginx restart on cp4003 and cp1045 for libssl upgrade
08:30 elukey: upgrading varnishkafka to 1.0.12-1 in cache:maps
07:34 elukey: executed 'find /var/log/hhvm/ -type f -user root -exec chown www-data:www-data {} \;' for all the api and appservers to remove/prevent cronspam (root:adm files also related to new reimaged hosts, Rsyslog needs to be configured before hhvm) - T132324
07:02 moritzm: rebooting francium for kernel security update
04:03 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.20/includes/deferred: 5af1b93db1bb3d14844c55e4e3ed17fe963de551 (duration: 00m 48s)
04:02 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.20/includes/libs/rdbms: 5af1b93db1bb3d14844c55e4e3ed17fe963de551 (duration: 00m 51s)
02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 23 02:46:04 UTC 2016 (duration 6m 10s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 17m 04s)
02:13 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.20/extensions/SecurePoll/: https://gerrit.wikimedia.org/r/#/c/312450/1 (duration: 00m 51s)
02:10 mutante: mw1206, mw1224 - restarted hhvm and apache
01:49 bblack: depooled mw1224 service apache2
00:38 Krenair: mw1224 apache stuck, not restarting for now in case someone wants to investigate later. possibly T89912?
00:17 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/312339 (duration: 00m 48s)
00:16 logmsgbot: krenair@tin Synchronized wmf-config/mobile.php: https://gerrit.wikimedia.org/r/312339 (duration: 00m 47s)
00:08 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.20/extensions/FlaggedRevs/business/RevisionReviewForm.php: https://gerrit.wikimedia.org/r/#/c/312423/ (duration: 00m 48s)

2016-09-22

23:46 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/310483 (duration: 00m 48s)
23:19 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.20/resources/src/mediawiki.less/mediawiki.ui/mixins.less: https://gerrit.wikimedia.org/r/#/c/312340/ (duration: 00m 48s)
22:49 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.20/includes/libs/rdbms/loadbalancer/LoadBalancer.php: a73a7ef9286275f797411646f9c5af60d4894c73 (duration: 01m 04s)
22:18 mutante: added slaporte and zhousquared to wmf LDAP group (T146227)
21:24 hasharAway: Nodepool is all back and operational. Reduced amount of queries to the OpenStack API by more than 10%
21:14 yurik: deployed tilerator https://gerrit.wikimedia.org/r/#/c/312329/
21:05 hasharAway: stopped nodepooled and restarted it with 0.1.1-wmf5
21:04 mutante: upgraded nodepool to 0.1.1-wmf5 on labnodepool1001
21:04 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.20/resources/src/mediawiki/mediawiki.js: T146099 (duration: 00m 48s)
21:02 mutante: imported nodepool_0.1.1-wmf5_amd64 into jessie-wikimedia (T145142)
20:52 urandom: T133395: RESTBase Staging: starting dumps (3, eqiad)
20:47 urandom: T133395: RESTBase Staging: altering table to set TWCS on wikipedia parsoid.html table
20:20 urandom: T133395: RESTBase Staging: Restarting Cassandra to pick up TWCS jar in classpath
20:08 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.20
20:00 thcipriani: rolling out wmf.20 to all wikis
19:23 SMalyshev: Deploying new version of WDQS GUI
19:09 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.20
19:01 thcipriani: wmf.20 to group1 will watch until 20 UTC and move forward to all wikis
18:56 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/CentralNotice: SWAT: Update extensions/CentralNotice submodule (T144952) (duration: 00m 50s)
18:50 yuvipanda: enabling puppet on labcontrol1001, run on labtestcontrol2001 seems ok
18:47 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.20/extensions/CentralNotice: SWAT: Update extensions/CentralNotice submodule (T144952) (duration: 00m 52s)
18:46 yuvipanda: disable puppet on labcontrol1001 for https://gerrit.wikimedia.org/r/#/c/312301/
18:38 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.20/includes/libs/rdbms/database/Database.php: 844cfd568a7c7953faa6ac69acebff1cee943b7f & 014a420b4525798b1202cc488b337acdaf09c49a (duration: 00m 49s)
18:31 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.19/extensions/TimedMediaHandler/MwEmbedModules: SWAT: Update ogv.js to 1.2.0 (T145983) (duration: 00m 48s)
18:28 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/TimedMediaHandler/MwEmbedModules: SWAT: Update ogv.js to 1.2.0 (T145983) (duration: 00m 51s)
17:24 moritzm: rebooting ms-be1016, high load caused by XFS bug
17:09 moritzm: rolling reboot of trusty swift backend servers in eqiad completed
16:40 elukey: forced logrotation for /etc/logrotate.d/upstart on labvirt1014 to investigate cronspam
16:17 godog: offline sdd on ms-be1004 via megacli T144499
15:29 mobrovac: restbase deploy end of d96fbc1
15:10 mobrovac: restbase deploy start of d96fbc1
15:02 bblack: upgrading openssl on cp*
13:22 moritzm: resume rolling reboot of trusty swift backend servers in eqiad for kernel security update
13:02 moritzm: uploaded openssl 1.0.2i for jessie-wikimedia to carbon
12:31 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.20/extensions/Popups: Merge mw.popups.experiment into mw.popups.core T146035 (duration: 00m 49s)
12:27 mobrovac: restbase deploy end of d5538ad
12:25 elukey: installing varnishkafka 1.0.12 on cache:upload ulsfo and eqiad
12:24 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-ro_0.7.3~r57551-2+wmf1
12:13 hashar: Early SWAT for mobile team ( https://gerrit.wikimedia.org/r/#/c/311977/ )
12:11 mobrovac: restbase deploy start of d5538ad
11:34 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1082 with some light weight (duration: 00m 52s)
10:24 moritzm: rolling reboot of trusty swift backend servers in eqiad for kernel security update
09:59 moritzm: rebooting subra/suhail for kernel security update
09:50 hashar: updated jobrunner code to a0e82166 (tweak errors reporting in logs) | Does not include 51014242 "Batch stats to statsd" (poke addshore )
09:19 gehel: upgrade / restart of elasticsearch eqiad cluster done T145404 / T146123
09:02 elukey: installing varnishkafka 1.0.12 on cache:upload codfw
08:49 marostegui: Deploying schema change on S7 master - T141951
08:43 elukey: installing varnishkafka 1.0.12 on cache:upload esams
08:40 elukey: installed varnishkafka 1.0.12 on cp1099
08:35 elukey: restarted varnishkafka on cp1099 (log abandoned )
08:19 hashar: Cleanup jobrunner list of minions in redis ( "deploy:jobrunner/jobrunner:minions" )
08:09 hashar: Resyncing all jobrunner deployment installations since only 41/68 minions have completed fetch/checkout
08:01 elukey: rolling restart of the whole Analytics Hadoop cluster for kernel upgrades (analytics* hosts)
07:58 elukey: uploaded varnishkafka 1.0.12-1 to reprepro
07:52 elukey: rebooted stat100[23] for kernel upgrades
07:40 moritzm: rolling restart of trusty swift frontend servers in codfw for kernel security update
07:33 elukey: rebooting stat1004 for kernel upgrades
06:45 elukey: Puppet disabled on analytics1027 to stop periodic Java daemons (prep step for Hadoop cluster reboots)
03:18 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 22 03:18:48 UTC 2016 (duration 6m 48s)
03:12 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.20) (duration: 17m 08s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 16m 39s)
02:03 eileen: all back on
01:50 eileen: turned off jenkins jobs to run dbupdate: dedupe(s) thank-you & donate import
01:40 eileen: update civicrm from 5393a13727e8dbad05ffef9ddd44e965eb9282d1 to d30a5e454e6614214faef8354a3a23d71d573c7f
01:35 twentyafterfour: reboot successful, iridium is back online
01:28 twentyafterfour: Rebooting iridium to apply kernel update
00:39 awight: update paymentswiki from d572ee9e3ef0a97c044f02e9866469a8f3fa5858 to cc27f83f31ecc609d4400050e73905b7364f1d42; mirror unsubscribe queue

2016-09-21

23:32 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Blacklist minerva from showing Related Articles in the footer (T144912, currently no-op) (duration: 00m 47s)
23:31 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Blacklist minerva from showing Related Articles in the footer (T144912, currently no-op) (duration: 00m 49s)
22:57 mobrovac: change-prop deploying ea8cdf8
22:35 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings.php: Add CentralNotice debug log bucket for T144952 (duration: 00m 48s)
22:33 logmsgbot: awight@tin Synchronized php-1.28.0-wmf.20/extensions/CentralNotice: Correct CentralNotice logging for T144952 (duration: 00m 51s)
22:31 logmsgbot: awight@tin Synchronized php-1.28.0-wmf.18/extensions/CentralNotice: Correct CentralNotice logging for T144952 (duration: 00m 51s)
22:08 cwd: updated SmashPig from f308ba490635c75b6fe735e7dc1c558250202365 to 8ac116037440746eaf64b9e99e1ee962d5d33475
20:39 bearND: deployed mobileapps bf6943b
20:36 yurik: deployed kartotherian geoshape lines support - https://gerrit.wikimedia.org/r/#/c/312097/
20:35 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.20
20:31 bearND: starting mobileapps deploy
20:24 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.20 and rebuild l10n cache (duration: 52m 01s)
20:17 arlolra: updated Parsoid to version a802de0
20:05 arlolra: starting Parsoid deploy
19:32 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.20 and rebuild l10n cache
19:24 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.js: T146099 (duration: 01m 41s)
17:41 bblack: bits.wikimedia.org hostname removed from DNS (if related real complaints/problems occur, revert https://gerrit.wikimedia.org/r/305533 )
16:46 Krenair: running P3833 script against designate to clean up existing T120797 mess
16:46 mobrovac: restbase deploy end of a75510d
16:30 mobrovac: restbase deploy start of a75510d
15:03 moritzm: installing wireshark security updates
13:59 akosiaris: disabled puppet on neon, puppet migration in progress
13:48 hashar: European SWAT completed
13:45 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: (no message) (duration: 00m 46s)
13:44 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 46s)
13:43 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings-labs.php: (no message) (duration: 00m 48s)
13:39 logmsgbot: hashar@tin Synchronized wmf-config/mobile.php: For phuedx or is that for yurik? (duration: 00m 47s)
13:37 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/extensions/Kartographer: For yurik or phuedx? :D (duration: 00m 48s)
13:37 gehel: adding planet_osm_lines and roads indexes on maps*
13:34 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/extensions/Wikidata: (no message) (duration: 02m 22s)
13:31 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/extensions/Kartographer/: (no message) (duration: 00m 50s)
13:17 logmsgbot: hashar@tin Synchronized wmf-config: (no message) (duration: 00m 49s)
13:13 logmsgbot: hashar@tin Synchronized wmf-config: New wikitext editor: Enable the Beta Feature in Beta Cluster (duration: 00m 50s)
13:10 logmsgbot: hashar@tin Synchronized wmf-config: New wikitext editor: Enable the Beta Feature in Beta Cluster (duration: 00m 51s)
12:30 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1094 after the ALTER table - T141951 (duration: 00m 47s)
12:09 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Depool db1094 for an ALTER table - T141951 (duration: 00m 47s)
11:45 moritzm: rolling restart of trusty swift backend servers in codfw for kernel security update
11:25 mobrovac: restbase deploy end of ca55669
11:16 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Repool db1086 after the ALTER table - T141951 (duration: 00m 47s)
11:07 mobrovac: restbase deploy start of ca55669
11:04 marostegui: Rebuilding tables in db1082 (non pooled) - T137191
11:03 elukey: adding mw1197 back to serving live traffic after the reimage
10:51 elukey: restarted varnishkafka on cp1048 (VSLQ_Dispatch: Varnish Log abandoned or overrun.)
10:45 elukey: adding mw1196 back to serving live traffic after the reimage
10:06 moritzm: rebooting lithium for kernel security update
09:39 moritzm: reimaging mw1173-mw1175 to jessie
09:26 marostegui: Stopping mysql at db1019 for a few days as it will be decommissioned - T146265
09:19 gehel: powercycling elastic1027 - T145404
09:09 godog: reimage bast3001.wikimedia.org with separate /srv
08:31 marostegui: schema change on S7 - T141951
08:30 elukey: reimagining mw1196-7 to jessie
08:29 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Depooling db1086 for an alter table - T141951 (duration: 00m 49s)
07:19 elukey: Moved some hhvm logs (/var/log/hhvm) from root:adm to www-data:www-data on mw127[678] to remove cronspam (T132324)
07:18 moritzm: reimaging mw1170-mw1172 to jessie
06:59 marostegui: dropping tables in S1,S3,S4 - T54924
06:21 elukey: removing aqs100[123] from live traffic - aqs.svc.eqiad.wmnet - T144497
06:03 awight|afk: update SmashPig from 6651835c816ce62112cbfea96863e4244f133c17 to f308ba490635c75b6fe735e7dc1c558250202365
03:58 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.js: I221cd6c2b (duration: 00m 46s)
03:56 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.requestIdleCallback.js: I221cd6c2b (duration: 00m 48s)
03:54 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.19/resources/src/mediawiki/mediawiki.requestIdleCallback.js: I221cd6c2b (duration: 00m 47s)
03:31 awight: update SmashPig from 4530dc9113a83bc5492a7da6f8e9ee3a6c789cda to 6651835c816ce62112cbfea96863e4244f133c17
03:13 awight: update SmashPig from f1f55096ac20899fa9700aa0f80dece5b47a7fd3 to 4530dc9113a83bc5492a7da6f8e9ee3a6c789cda
02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 21 02:46:22 UTC 2016 (duration 7m 11s)
02:45 mutante: thumbor1002 moved nginx access logs to /srv for more space on /
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 16m 28s)
02:11 mutante: thumbor1001/1002 - moved logs from /var/log/thumbor to /srv/thumborlogs to free some space, the actual issue is in /tmp though. lots of systemd-private-* dirs with large sizes. like https://bugzilla.redhat.com/show_bug.cgi?id=1183684 ?
01:58 awight: update SmashPig from 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c to f1f55096ac20899fa9700aa0f80dece5b47a7fd3
01:51 mutante: thumbor servers ran out of disk space
01:15 awight: updated SmashPig to 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c
00:45 eileen: jobs enabled again - update failed to run due to trigger - which didn't affect staging maybe not on - will edit & try again

2016-09-20

23:49 eileen: disabled Dedupe CiviCRM contacts
23:47 eileen: disabled Project Dedupe Major gifts contacts
23:47 eileen: disabled CiviCRM contacts (high numbers)
23:46 eileen: disabled Thank you mail send
23:46 eileen: disabled Donations queue consume
23:45 eileen: CiviCRM update from de1df9ec4fe58487eeb61cc69160f228e009f2cf to 5393a13727e8dbad05ffef9ddd44e965eb9282d1
23:35 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/mediawiki.js: Always use requestIdleCallback polyfill for batchEval (T146099) (duration: 00m 46s)
23:22 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.19/extensions/Echo/: SWAT (duration: 00m 55s)
23:20 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.19/extensions/TimedMediaHandler: SWAT (duration: 00m 50s)
23:17 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.18/extensions/TimedMediaHandler: SWAT (duration: 00m 50s)
22:26 mobrovac: change-prop deploying 4417255
21:41 mutante: powercycled mw1294 (down, frozen console)
21:26 thcipriani: starting branch cut for wmf.20
21:17 chasemp: rsync initial transfer of others on labstore1001 to labstore1004
21:08 awight: update SmashPig from db68be988194c960aebca691d0fd8e6a6d24246a to 285d8ce25f22d8538c693c1b3c3e7ce9e688f22c
20:26 Pchelolo: restbase deploy ca41acd3f
20:20 Pchelolo: restbase deploy ca41acd3f canary on restbase1007
20:13 Pchelolo: restbase deploy ca41acd3f to staging
20:07 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.18/extensions/ProofreadPage/modules/page/ext.proofreadpage.page.edit.js: Initializes the zoom widget after page loading (duration: 00m 47s)
19:51 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.18/extensions/ProofreadPage/modules/page/ext.proofreadpage.page.edit.js: Makes sure that the zoom widget is initialized before zooming in/out - https://gerrit.wikimedia.org/r/#/c/311765/ (duration: 00m 48s)
17:47 Pchelolo: update RESTBase to 4829630f
17:27 Pchelolo: update RESTBase to 4829630f canary on restbase1007
17:01 elukey: adding aqs1006 to live traffic - aqs.svc.eqiad.wmnet - T144497
16:58 elukey: adding aqs1005 to live traffic - aqs.svc.eqiad.wmnet - T144497
16:48 gehel: increase recovery bandwidth on elasticsearch eqiad to match codfw - T145404
16:32 elukey: restarting cassandra on aqs100[56] (started the work earlier on today, stopped due to T146130)
14:31 dcausse: restarting relforge100[12].eqiad.wmnet servers for kernel upgrade and java settings change
14:23 moritzm: installing tomcat security updates on Ubuntu servers
13:41 jynus: disabling puppet on labtestweb2001
13:40 ottomata: merged --until flag change in check_graphite script (this could affect all graphite based alerts)
13:37 zeljkof: executed script: mwscript maintenance/updateArticleCount.php --wiki=wikidatawiki --update
13:34 zeljkof: EU SWAT finished
13:28 gehel: restarting for elasticsearch and kernel upgrade - eqiad cluster - T145404 / T146123
13:05 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Change $wgArticleCountMethod in Wikidata from default (link) to any (T144687) (duration: 00m 47s)
11:41 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 46s)
11:38 moritzm: upgrading ganeti2001 to Linux 4.4 (ganeti2006 has been promoted to new master node)
11:05 moritzm: upgrading ganeti2006 to Linux 4.4
10:49 moritzm: upgrading ganeti2005 to Linux 4.4
10:44 moritzm: reimaging app servers mw1240-mw1242 and API servers mw1194/mw1195 to jessie
10:43 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 48s)
10:36 moritzm: upgrading ganeti2004 to Linux 4.4
10:28 akosiaris: force mw2232 to use palladium for report handler testing
10:25 jynus: deploying schema change on s1 hosts T139090
10:15 moritzm: upgrading ganeti2003 to Linux 4.4
10:12 mobrovac: change-prop deploying e1ef51e
09:57 moritzm: upgrading ganeti2002 to Linux 4.4
09:13 moritzm: reimaging API servers mw1192/mw1193 to jessie
08:56 moritzm: reimaging mw1243-mw1245 to jessie
07:36 elukey: restart cassandra on aqs100[456] for T130861 - only aqs1004 is taking live traffic
02:47 eileen: CiviCRM update from c9157ba6aac3ad3c2fd61c7bb0475851fb8ec421 to de1df9ec4fe58487eeb61cc69160f228e009f2cf
02:40 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 20 02:40:17 UTC 2016 (duration 6m 52s)
02:33 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 10m 35s)
00:43 mutante: wtp2019 - down again, powercycled, probably damaged RAM

2016-09-19

23:22 Pchelolo: restart restbase in staging
23:01 awight: update orphan rectifier config for T145848
22:58 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: Restore testwiki to 1.28.0-wmf.18
22:53 dapatrick: Deployed patch for T144573 to wmf18 and wmf19
22:48 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.19 and rebuild l10n cache (duration: 52m 03s)
22:31 bblack: cache_upload: pooling cp1099 (storage experiment - T145661)
22:02 awight: update paymentswiki from 4cd18776ca67fe0db04b4173cb47f39dda59f43b to d572ee9e3ef0a97c044f02e9866469a8f3fa5858
22:00 bblack: cp1099: depooling varnish backends for storage size experimentation
21:56 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.19 and rebuild l10n cache
21:41 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: Revert group0 wikis to 1.28.0-wmf.19
21:30 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.28.0-wmf.19
20:40 hoo: Removed today's Wikidata json dumps: All shards succeeded, but final dump composition apparently failed.
20:14 chasemp: reboot labstore1004
20:08 Pchelolo: restbase deploy 4829630f staging
18:49 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Throttle for RCL (T145838) (duration: 00m 47s)
18:43 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: ORES default threshold to high for wikidatawiki (T144784) (duration: 00m 47s)
18:32 jynus: emergency/unscheduled restart of mariadb @ labsdb1003 - close to OOM, unusable
18:14 thcipriani: ran on terbium: mwscript extensions/ShortUrl/populateShortUrlTable.php --wiki=bdwikimedia
17:47 awight: reenable banner history queue consumer
17:46 awight: update civicrm from 1df25962834885121bd7a9ba856c0c69c2b9cfda to c9157ba6aac3ad3c2fd61c7bb0475851fb8ec421
17:39 ejegg: updated payments-wiki from 392d67520d14998d61823755b22df50ab45afb35 to 4cd18776ca67fe0db04b4173cb47f39dda59f43b
17:36 Krenair: Reset wikitech/horizon 2fa for Greg per request
15:50 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-nno-nob_1.1.0~r66076-1+wmf1
15:50 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-br-fr_0.5.0~r61325-1+wmf1
14:40 chasemp: testing nfs export performance on labstore1004/1005 cluster
14:35 yurik: depl graphoid https://gerrit.wikimedia.org/r/#/c/311374/
14:28 moritzm: uploaded apache 2.4.10-10+deb8u7+wmf1 for jessie-wikimedia to carbon
14:21 elukey: adding aqs1004 to live traffic - aqs.svc.eqiad.wmnet - T144497
14:10 moritzm: reimaging mw1246-mw1248 to jessie
14:10 hashar: European SWAT is complete.
14:05 zeljkof: EU SWAT finished
14:05 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add WT namespace alias to NS_PROJECT in mywiktionary (T140998) (duration: 00m 47s)
14:01 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/extensions/Graph: Fixed wikiraw: protocol bug T146010 (duration: 00m 47s)
14:00 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/extensions/Graph: Fixed wikiraw: protocol bug T146010 (duration: 00m 48s)
13:58 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable WikidataPageBanner on itwikiwoyage (T145328) (duration: 00m 48s)
13:55 hashar: Europe SWAT extended as we still have some patches to process
13:47 logmsgbot: zfilipin@tin Synchronized wmf-config/throttle.php: SWAT: Throttling rule for RCL (T145838) [throttle] Allow the same number accounts Throttle for RCL (duration: 00m 47s)
13:18 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/includes/api/ApiQueryBacklinksprop.php: API: Force straight join for prop=linkshere|transcludedin|fileusage T145079 (duration: 00m 47s)
13:18 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/includes/api/ApiQueryBacklinksprop.php: API: Force straight join for prop=linkshere|transcludedin|fileusage T145079 (duration: 00m 50s)
12:59 moritzm: installing wget updates from jessie 8.6 point update
12:51 elukey: adding mw1191 back to serving traffic after reimage
11:43 mobrovac: restbase cassandra truncating local_group_wikipedia_T_feed_aggregated.data
11:32 jynus: rebooting again db1061 for upgrade
11:13 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Temporarily depool db1062 and repool db1034, in order to be able to ALTER a large table. T141951 (duration: 00m 48s)
10:49 moritzm: reimaging mw1249, mw1250, mw1258 to jessie
10:23 jynus: powercycle db1061, unresponsive since ~1am
09:20 moritzm: invalidated squid cache on carbon
08:49 akosiaris: increase /var/lib/puppet to 50GB on puppetmaster1002, puppetmaster2001, puppetmaster2002
08:48 logmsgbot: addshore@tin Synchronized wmf-config/CommonSettings.php: 311118 NOOP Some inline comments added (duration: 00m 58s)
08:04 marostegui: renaming tables in S1, S4 and S4 in eqiad before dropping them T54924
07:50 elukey: reimaging mw1191.eqiad.wmnet to jessie
07:49 moritzm: installing updates for file/libmagic from jessie 8.6 point update
07:42 moritzm: reimaging mw1255-mw1257 to jessie
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 19 02:28:46 UTC 2016 (duration 5m 4s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 10m 39s)

2016-09-18

20:37 bblack: restart upload backend: cp3036
20:22 bblack: restart upload backend: cp3039
19:58 bblack: restart upload backend: cp1074 (stats indicate LRU_Fail imminent)
19:28 bblack: restart upload backend: cp1064 (already in LRU_Fail, caught early)
19:28 bblack: restart up
18:06 bblack: restart upload varnish backend: cp1050 (already in LRU_Fail)
17:58 bblack: restart upload varnish backend: cp2008
17:42 bblack: restart upload varnish backend: cp1071
17:32 bblack: restart upload varnish backend: cp2020
17:06 bblack: restart upload varnish backend: cp2026
16:35 bblack: restarting upload varnish backend: cp2011
15:43 bblack: restarting upload varnish backend: cp2017
15:13 bblack: restarting upload varnish backend: cp2005
14:52 bblack: restarting upload varnish backend: cp1049
14:42 bblack: restarting upload varnish backend: cp2022
14:17 bblack: restarting varnish backend on cp1073 (503 LRU_Fail pattern, has been up a few days...)
13:29 bblack: disabling puppet on cp1074, to experiment with vhtcpd regex filter
11:17 ema: repooling varnish-be in codfw
11:00 ema: varnish-backend restart on cp3037
10:58 ema: varnish-backend restart on cp3044
10:54 ema: repooling varnish on cp1050
10:53 ema: repooling varnish on cp1062
10:52 ema: repooling varnish on cp1064
10:50 ema: repooling varnish on cp1071
10:50 ema: repooling varnish on cp1072
10:49 ema: repooling varnish on cp1073
10:49 ema: repooling varnish on cp1074
09:47 _joe_: varnish-backend-restart on cp1063
09:30 _joe_: varnish-backend-restart on cp1048
06:09 ejegg|afk: updated civicrm from 5ba6976f2552564b51085abff0afd5f76195229b to 1df25962834885121bd7a9ba856c0c69c2b9cfda
02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 18 02:32:49 UTC 2016 (duration 5m 58s)
02:26 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 10m 18s)

2016-09-17

16:40 _joe_: rolling restart of HHVM on part fo the API cluster in eqiad, T133674
08:15 _joe_: enlarged puppet partition on puppetmaster1001, rendered full by reports
07:06 p858snake: set +z on -operations, allows messages sent by +b or +q users (normally blocked) to be seen by users that currently op'ed
06:55 p858snake: see T145924 or email to ops list for more info
06:54 p858snake: silenced (+q) icinga-wm in operations channel, due to channel spam from low disk space on puppetm1001
02:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 17 02:34:33 UTC 2016 (duration 7m 3s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 11m 40s)

2016-09-16

23:09 mutante: titanium - shutdown -h now
21:23 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: Set some database logging groups to log (duration: 00m 47s)
20:34 logmsgbot: reedy@tin Synchronized wmf-config/: Load CN via extension registration. Only load jsonconfig once (duration: 00m 56s)
20:28 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: Couple more to extension.json (duration: 00m 47s)
19:32 mutante: fermium disabled puppet again
19:31 mutante: fermium starting mailman qrunner (T144933)
19:26 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.19/includes/jobqueue/JobQueueGroup.php: 01254b0a72a8619117ecad103427e2431e89cc52 (duration: 00m 47s)
18:11 mutante: fermium - re-enabled puppet (after merging gerrit 310746(
17:55 mutante: gallium rm /etc/sudoers.d/jenkins-slave (to go with gerrit 311161)
17:03 Pchelolo: deploy changeprop to apply gerrit 311153 config change
16:16 yuvipanda: puppet developers are you reading this? just checking...
15:18 akosiaris: enable puppet on puppetmaster1001 again
15:03 akosiaris: disabling puppet on puppetmaster1001
15:00 hashar: gallium: dpkg --purge php5-mysql (mysql got removed)
14:46 gehel: disabling shard allocation check on relforge to test shard allocation issues
13:56 elukey: mw1189 back serving traffic after reimage
13:41 logmsgbot: akosiaris@tin Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 46s)
13:35 hashar: gallium: removing MySQL which is no more defined in puppet and running puppet. Did: apt-get remove mysql-common mysql-server mysql-server-core-5.5
13:19 logmsgbot: akosiaris@tin Synchronized wmf-config/db-eqiad.php: (no message) (duration: 00m 48s)
12:50 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: All wikis back to 1.28.0-wmf.18 :( T145819
12:40 hashar: Going to rollback all Wikis back to 1.28.0-wmf.18 . Despite much investigation, a bunch of jobs are broken due to T145819 which includes Special:CreateAccount :(
12:37 elukey: mw1190 back serving traffic after the reimage
12:24 gehel: rolling restart of codfw elasticsearch cluster completed - T145404
12:20 moritzm: installing security updates for mysql 5.5 (one off systems running mysql as packaged by Ubuntu/Debian and not running wmf-mariadb10)
12:15 moritzm: installing python-imaging security updates on precise
11:24 akosiaris: silence icinga-wm for a while
11:22 akosiaris: restarted puppetmaster on all puppetmasters
11:22 akosiaris: stop puppetmaster on all puppetmasters, resizing /var/lib/puppet
10:21 marostegui: renaming tables before dropping them in codfw S1,S3,S4 - T54924
09:34 elukey: reimage mw1189-90 to Jessie (trying Riccardo's script!)
09:02 moritzm: reimaging mw1252-mw1254 to jessie
08:59 moritzm: installing tomcat7 security updates
08:52 moritzm: installing tomcat8 security updates
08:43 moritzm: installing libidn security updates in eqiad
07:36 elukey: forced logrotation with debug of /etc/logrotate.d/graphite-web on graphite1001 to find cronspam source
03:21 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 16 03:21:54 UTC 2016 (duration 5m 18s)
03:16 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.19) (duration: 18m 44s)
02:41 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 18m 25s)
01:28 mutante: mw1294 - down and frozen, powercycled
00:53 awight: remove "pending" from AMQ old message consumer
00:50 awight: update paymentswiki config to disable legacy orphan rectifier

2016-09-15

23:38 legoktm: legoktm@terbium:~$ foreachwiki extensions/WikimediaMaintenance/createExtensionTables.php babel # T145366
23:24 awight: rolled back paymentswiki to 392d67520d14998d61823755b22df50ab45afb35
23:22 awight: updating paymentswiki to possible broken 609c1e5f03ddcd9d72f562d7911cf77cc459371b
23:00 awight: update paymentswiki to from 996ca30076946c6148d9688a905c82e2e346e165 to 392d67520d14998d61823755b22df50ab45afb35 (reverted DI submodule update)
22:55 awight: Reenabled donations and fredge consumers
22:50 awight: update fundraising CRM from f381bd1c15e30e4d47fe372e128143baee6a7c7a to 5ba6976f2552564b51085abff0afd5f76195229b
22:30 awight: rollback paymentswiki from 609c1e5f03ddcd9d72f562d7911cf77cc459371b to 996ca30076946c6148d9688a905c82e2e346e165
22:28 awight: update paymentswiki from 996ca30076946c6148d9688a905c82e2e346e165 to 609c1e5f03ddcd9d72f562d7911cf77cc459371b
22:19 ejegg: updated SmashPig from af19422065c08669269179019fea1e7c208d8a7e to db68be988194c960aebca691d0fd8e6a6d24246a
22:10 ejegg: updated SmashPig from 12a7b78cf280f96ff3d2314abb2a9290506ae454 to af19422065c08669269179019fea1e7c208d8a7e
21:41 ejegg: updated SmashPig from e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574 to 12a7b78cf280f96ff3d2314abb2a9290506ae454
20:55 hashar: All wikis are on 1.28.0-wmf.19 wikidatawiki / testwikidatawiki stick to .18 for now.
20:35 gehel: increasing number of shards per node for dewiki_content index to 2 on elasticsearch codfw
20:28 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: All wiki to .19. Keep testwikidata and wikidata at .18 (commits: 38603f0 770d336)
20:14 yuvipanda: remove self from github wikimedia org, was getting spammed for each new repo creation
20:13 ema: varnish-be esams cache_upload: rolling depool and restart
20:11 ejegg: updated SmashPig from e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574 to 12a7b78cf280f96ff3d2314abb2a9290506ae454
20:08 gehel: increasing number of shards per node for enwiki_content index to 2 on elasticsearch codfw
20:01 yuvipanda: restart puppetmaster on labcontrol1001 to pick up hiera changes
19:59 ema: depool and restart varnish-be on cp1048
19:56 ema: depool and restart varnish-be on cp1064
19:52 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
19:50 ema: depool and restart varnish-be on cp1062
19:45 ema: depool and restart varnish-be on cp1073
19:33 ema: depool and restart varnish-be on cp1050
19:20 ema: depool and restart varnish-be on cp1063
19:09 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.19
19:08 ema: depool and restart varnish-be on cp1072
19:08 awight: disabled fredge, donations, and banner-history consumers
19:07 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php: To unblock renames stuck on mediawiki.org T145596 (duration: 00m 47s)
19:06 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/extensions/CentralAuth/maintenance/fixStuckGlobalRename.php: To unblock renames stuck on mediawiki.org T145596 (duration: 00m 47s)
19:05 logmsgbot: thcipriani@tin Finished scap: SWAT: Add missing close button title message (T145774) and Revert "Remove jquery.arrowSteps module" (T144974) (duration: 28m 08s)
18:59 ema: depool and restart varnish-be on cp1099
18:43 ejegg: updated smashpig consume pending job with new queue name
18:37 logmsgbot: thcipriani@tin Started scap: SWAT: Add missing close button title message (T145774) and Revert "Remove jquery.arrowSteps module" (T144974)
18:33 mutante: titanium - stop salt, stop puppet, revoke puppet cert, delete salt key
18:23 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.19/extensions/Kartographer: SWAT: Fix map popup CSS (T145716) (duration: 00m 56s)
18:23 ema: depool and restart varnish-be on cp1074
17:56 awight: Purging GC messages from pending with timestamp < '2016-09-15 13:44:55'
17:44 ema: depool and restart varnish-be on cp1049
17:18 Pchelolo: change-prop deploy 310877
16:51 Pchelolo: change-prop deploy gerrit 310873
16:49 akosiaris: uploaded to apt.wikimedia.org precise-wikimedia: zuul_2.5.0-8-gcbc7f62-wmf3precise1
16:49 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: zuul_2.5.0-8-gcbc7f62-wmf3jessie1
15:55 moritzm: uploaded trebuchet-trigger 0.5.6-1~jessie1 to carbon (no change rebuild for jessie)
15:45 kart_: Update cxserver to a1949e9
14:59 _joe_: starting a noop run on all nodes to puppetmaster2001 to test puppetdb
14:57 elukey: deployed new-aqs-cluster branch (--rev new-aqs-cluster) to aqs100[456] (new AQS cluster not serving live traffic)
14:30 _joe_: removing old reports from the puppet directory
14:03 godog: empty big log file on thumbor1001 /var/log/thumbor/thumbor.log
13:31 zeljkof: EU SWAT done!
13:28 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Update outdated comment for Wikibase (duration: 00m 48s)
13:25 logmsgbot: zfilipin@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove $wgTranslateEC (duration: 00m 48s)
12:43 logmsgbot: addshore@tin Finished scap: Update RevisionSlider i18n (duration: 30m 26s)
12:13 logmsgbot: addshore@tin Started scap: Update RevisionSlider i18n
11:11 hashar: CI is catching up. It is starved processing a long serie of dependent changes in Gerrit
11:08 hashar: CI / Jenkins is starved. Investigating
10:09 moritzm: reimaging mw1251 to jessie
09:35 moritzm: reimaging mw1250 to jessie
08:11 marostegui: altering tables in S7 - eqiad hosts - T141951
07:01 moritzm: installing libidn security updates
06:54 marostegui: renaming tables before dropping them - T145487
06:29 moritzm: installing chromium security updates on osmium
06:24 _joe_: turning off nitrogen for memory reduction, reimage
03:23 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 15 03:23:49 UTC 2016 (duration 6m 44s)
03:17 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.19) (duration: 18m 25s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 49s)
02:04 mutante: ms-be1022 - down per icinga, but also mgmt is not reachable

2016-09-14

23:44 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.19/extensions/Kartographer/tests/phpunit/KartographerTest.php: Always serve all the data on preview (T145615, 2/2, no-op part) (duration: 00m 50s)
23:43 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.19/extensions/Kartographer/includes/Tag/TagHandler.php: Always serve all the data on preview (T145615, 1/2) (duration: 00m 47s)
23:34 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add logging channel for NewUserMessage (T131957) (duration: 00m 47s)
22:36 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.19/includes/resourceloader/ResourceLoaderWikiModule.php: T145673 (duration: 00m 47s)
21:18 Pchelolo: RESTBase update to fd43f3a58
21:13 Pchelolo: RESTBase update to fd43f3a58 canary on restbase1007
20:57 Pchelolo: RESTBase update to fd43f3a58 staging
20:28 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.19/extensions/CentralAuth/includes/LocalRenameJob/LocalRenameJob.php: Fix LocalRenameJob transaction owner to match JobRunner T143328 T145596 (duration: 00m 48s)
20:23 yuvipanda: manually raise max_connections on labtestcontrol2001, see T145679 for ticket
20:13 Pchelolo: revert RESTBase is staging to d10d759d42
20:11 arlolra: updated Parsoid to version aed15dda
20:05 Pchelolo: update RESTBase to 5ae9a506 - staging
20:03 arlolra: starting Parsoid deploy
19:25 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: Revert group1. Hebrew wiki has templates on the wrong side / CSS is off
19:14 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.19
19:09 Pchelolo: revert RESTBase to d10d759
19:08 awight: update paymentswiki config to 9919bad6897f2eb7199dd076f6fc04a60c713cf8
19:07 mutante: titanium - puppet node clean
18:59 Pchelolo: RESTBase deploy d39580f14
18:57 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/Kartographer/modules/box/Map.js: SWAT: Map should take viewport width/height instead of body width/height (T145521) (duration: 00m 47s)
18:53 Pchelolo: RESTBase deploy d39580f14 canary on restbase1007
18:50 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/ZeroBanner/modules: SWAT: Display edit icon and page actions (duration: 00m 47s)
18:45 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.19/extensions/Kartographer/modules/box/Map.js: SWAT: Map should take viewport width/height instead of body width/height (T145521) (duration: 00m 47s)
18:23 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.19/includes/pager/ReverseChronologicalPager.php: SWAT: Partially reverting I8e684f06 to restore some legacy behavior (T145597) (duration: 00m 48s)
18:17 urandom: T133805: Renabling Pupppet, forcing run, and restarting Cassandra to restore 8M region size on restbase1013-a.eqiad.wmnet
17:57 AaronSchulz: Deleted big pages per https://meta.wikimedia.org/w/index.php?title=Steward_requests/Miscellaneous&oldid=15908701#Deleting_a_pages_with_a_.3E5000_revisions_in_ruwiki
17:53 mutante: meitnerium - oops, an unrelated rsyncd is supposed to be running on this, puppet re-created files
17:50 mutante: meitnerium - stop rsyncd, remove config fragments
17:20 mobrovac: change-prop deploying 19e2d51
17:19 volans: reimage mw2198 as it failed before
16:32 jynus: stopping mysql and shutting down db1082
16:05 elukey: restarting cassandra on aqs100[23] T130861
16:01 jynus: starting mysql on db1082
15:57 elukey: restarting cassandra on aqs1001 T130861
15:54 akosiaris: updated cr1-eqiad,cr2-eqiad puppet rules
15:50 hoo: Ran T132839-Workarounds.sh from my home in terbium (see T132839)
15:49 urandom: T130861: Performing rolling Cassandra restart, restbase staging
15:45 urandom: T130861: Restarting Cassandra, xenon.eqiad.wmnet
15:41 urandom: T130861: Forcing puppet run in restbase staging
14:38 mobrovac: change-prop deploying ddc091e
14:30 gehel: increasing delayed allocation to 10m on elasticsearch codfw to speed up cluster restart - T145404
14:26 gehel: upgrading elasticsearch codfw to elasticsearch 2.3.5 - T145404
13:33 gehel: upgrading logstash to elasticsearch 2.3.5 - T145404
13:20 marostegui: renaming tables in s3 codfw - T132837
13:11 logmsgbot: hashar@tin Synchronized portals: Bumping portals to master T128546 (duration: 00m 47s)
13:10 logmsgbot: hashar@tin Synchronized portals/prod/wikipedia.org/assets: Bumping portals to master T128546 (duration: 00m 48s)
13:07 akosiaris: stop ircecho (icinga-wm) temporarily on neon
12:12 akosiaris: stop ircecho (icinga-wm) temporarily on neon
12:05 _joe_: restarting apache on puppetmaster1001
10:31 akosiaris: stopped temporarily ircecho (icinga-wm) on neon
10:25 ema: varnish-be restarted on cp4005
10:03 marostegui: alter localuser table in db2054 - T141951
09:47 marostegui: Renaming tables before dropping them T54924
08:43 marostegui: alter localuser table in db2047 - T141951
07:35 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.19/includes/MediaWiki.php: Use cpPosTime cookie for same-domain redirects on DB change - https://gerrit.wikimedia.org/r/#/c/310494/ (duration: 00m 45s)
07:34 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.19/includes/db/ChronologyProtector.php: Use cpPosTime cookie for same-domain redirects on DB change - https://gerrit.wikimedia.org/r/#/c/310494/ (duration: 00m 47s)
07:32 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.19/includes/db/loadbalancer/LBFactory.php: Use cpPosTime cookie for same-domain redirects on DB change - https://gerrit.wikimedia.org/r/#/c/310494/ (duration: 00m 46s)
03:22 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Sep 14 03:22:13 UTC 2016 (duration 7m 6s)
03:15 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.19) (duration: 18m 08s)
02:46 mutante: restarted grrrrit-wm
02:44 mutante: gerrit back to normal
02:42 mutante: gerrit restarting to apply config changes 256663 and 308885
02:40 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 56s)
01:56 logmsgbot: aaron@tin Synchronized wmf-config: Set $wgAPIMaxLagThreshold => 3 and "max lag" => 6 (duration: 00m 51s)

2016-09-13

23:32 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.18/resources/lib/moment/locale: T145382 (duration: 00m 47s)
23:19 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.19/resources/lib/moment/locale: T145382 (duration: 00m 49s)
22:44 awight: update paymentswiki config to da01ae9045af7b8b5f3d34a5f81419fde3bde1d5
22:13 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.19/includes/MediaWiki.php: Avoid stupid warnings on url parsing (duration: 00m 47s)
21:45 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.19/includes/DefaultSettings.php: for lego <3 (duration: 00m 47s)
20:46 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.19/extensions/Echo: For Roan <3 (duration: 00m 54s)
20:23 Krenair: restarted designate-api on labtestservices2001, now designate in labtest is working again
20:21 Krenair: restart rabbitmq-server on labtestcontrol2001
19:01 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to wmf.19
18:35 gehel: starting data import on wdqs200?
18:24 logmsgbot: demon@tin Finished scap: testwiki to wmf.19 + l10n bootstrap (try 2) (duration: 47m 21s)
18:08 gehel: moving to scap deployed configuration for wdqs - T144380
17:58 mutante: wtp2019 - powercycled, back up without the error, services started
17:55 mutante: wtp2019 Uncorrectable Memory Error
17:55 mutante: wtp2019 - down sinc a couple days. console says "Alert! System fatal error during previous boot"
17:53 volans: reimaging mw2198 that failed early today
17:36 logmsgbot: demon@tin Started scap: testwiki to wmf.19 + l10n bootstrap (try 2)
17:24 logmsgbot: demon@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="testwiki" --outdir="/tmp/scap_l10n_4282891950" --threads=4 --lang en --quiet' returned non-zero exit status 1 (duration: 09m 23s)
17:15 logmsgbot: demon@tin Started scap: testwiki to wmf.19 + l10n bootstrap
16:59 jynus: beta dbs back in rw mode
16:54 mobrovac: restbase deploy end of d10d759
16:53 logmsgbot: demon@tin Synchronized multiversion/updateWikiversions: unbreak myself (duration: 00m 48s)
16:41 mobrovac: restbase deploy start of d10d759
16:25 mobrovac: change-prop deploying d701a69
16:24 godog: dump hhvm backtrace on mw1162 and restart hhvm, apache gets connection refused
16:17 urandom: T144826: Removing compaction rate limit, increasing compactor threads (from 10 to 20), and beginning scrub of local_group_wikipedia_T_parsoid_html.data (restbase2004-b.codfw.wmnet)
16:06 jynus: power resetting db1082
16:02 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1082 (duration: 01m 00s)
15:46 gehel: test_* indices remvoed on relforge cluster, cluster restarted
15:44 jynus: setting deployment-db1 and deployment-db1 mysqls in read only mode
15:40 gehel: shutting down relforge cluster for indices cleanup
15:18 marxarelli: starting 2-hour read-only maintenance window for beta cluster migration
14:50 godog: drain and reboot restbase2004
14:26 gehel: deleting test_* indices on relforge cluster
14:24 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-kaz-tat_0.2.1~r57554-1+wmf1
14:03 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-kaz_0.1.0~r61338-1+wmf1
13:33 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: Stop logging xff from 127.0.0.1 T129982 (duration: 00m 47s)
13:28 hashar: Pulling "Stop logging xff from 127.0.0.1" patch on mw1299 and mw1161-mw1169 T129982
13:21 hashar: Pulling "Stop logging xff from 127.0.0.1" patch on mw1300-1303 T129982
13:19 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.18/extensions/UploadWizard/resources/uw.EventFlowLogger.js: SWAT: [[gerrit:310180|uw.EventFlowLogger: Fix NS_ERROR_NOT_AVAILABLE debug logging (duration: 00m 49s)
13:15 logmsgbot: hashar@tin Synchronized portals: Bumping portals to master (duration: 00m 50s)
13:10 logmsgbot: addshore@tin Synchronized dblists/clldefault.dblist: SWAT: Deploy Compact Language Links out of beta for Tulu Wikipedia (duration: 00m 46s)
13:08 logmsgbot: addshore@tin Synchronized wmf-config/flaggedrevs.php: SWAT: Fix illegal wgFlaggedRevsWhitelist for arwiki (duration: 00m 47s)
13:04 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RevisionSlider BetaFeature on all wikis (duration: 00m 49s)
12:47 marostegui: alter localuser table in db2040 - T141951
12:24 jynus: putting db1024 under maintenance (potential lag, etc.) to test solutions for T145079
12:06 mobrovac: zotero translators deploying cad95af
11:49 gehel: restarting elasticseaarch on relforge1001 - OOM heap space
11:18 volans: reimaging mw2198 and mw2199 to test the automation script T143536
10:58 mobrovac: citoid deployed e79430f for T144597
10:58 godog: finished rolling restart swift-proxy for thumbor change T139606
10:48 elukey: zuul upgraded to zuul_2.5.0-8-gcbc7f62-wmf2jessie1 on scandium (T145057)
10:45 elukey: uploaded 2.5.0-8-gcbc7f62-wmf2jessie1 to jessie-wikimedia/thirdparty (T145057)
10:10 godog: enable shadow requests to thumbor for small wikis T139606
09:39 marostegui: alter localuser table in dbstore2002 - T141951
08:52 gehel: relforge is taking more time than expected to recover after upgrade, most probably related to ~3k indices that were created for test purpose
08:46 gehel: relforge is taking more time than expected to recover after upgrade, most probably related to >10k indices that were created for test purpose
08:33 marostegui: renaming tables in db1015 - T145487
08:22 marostegui: alter localuser table in https://tendril.wikimedia.org/host/view/dbstore2001.codfw.wmnet/3306 - T141951
08:04 gehel: upgrading elasticsearch & plugins to 2.3.5 on relforge - T145404
07:52 elukey: wrong package name for my prev entry - remove kafkatee from stat1002 - not in puppet and causing cronspam (T132324)
07:50 elukey: remove kafkacat from stat1002 - not in puppet and causing cronspam (T132324)
07:07 moritzm: installing openjdk6 security updates
02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 13 02:46:54 UTC 2016 (duration 7m 12s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 41s)
01:58 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Lower $wgMaxUserDBWriteDuration to 3 (duration: 00m 47s)
01:27 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/extensions/GlobalUsage: 1843a856c7c648a310a1983d53139e5b79ea6585 (duration: 00m 48s)
01:24 bblack: cache_upload: reverting codfw to file storage
01:15 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/extensions/GeoData: 2dedca3789266b53cbde5bd88913c01175926974 (duration: 00m 49s)
01:14 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/includes/jobqueue/utils/PurgeJobUtils.php: (no message) (duration: 00m 52s)
00:50 bblack: cache_upload: reverting eqiad to file storage
00:50 bblack: reverting eqiad to file storage
00:17 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/extensions/GeoData: 2dedca3789266b53cbde5bd88913c01175926974 (duration: 00m 48s)
00:16 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/includes/jobqueue/utils/PurgeJobUtils.php: 0419831bbae66bda1e4dca8d690b607b493b2f5f (duration: 00m 47s)

2016-09-12

23:40 bblack: switch cache_upload codfw to -sdeprecated_persistent...
23:40 logmsgbot: dereckson@tin Synchronized wmf-config/throttle.php: Women in Science throttle rules (T145115 and T145253) (duration: 00m 47s)
23:33 urandom: T144826: Restarting Cassandra on restbase2004-b.codfw.wmnet (scrub complete, re-joining cluster)
23:14 bblack: switch cache_upload eqiad to -sdeprecated_persistent...
21:54 eileen: from 3f01d93237c2b8af4fbda5e629a4a59c77dab3c1 to f381bd1c15e30e4d47fe372e128143baee6a7c7a
21:17 arlolra: For completeness, "back" in my last log is a mistake. I scap deployed the wrong --rev, but that was ultimately the version we wanted deployed anyways, so no harm no foul. (T145460)
20:40 arlolra: Parsoid back on f7c43009c
20:32 arlolra: Parsoid deploy failed, rolling back
20:28 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-b.eqiad.wmnet
20:28 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1014-b.eqiad.wmnet
20:28 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1009-b.eqiad.wmnet
20:21 mobrovac: change-prop deploying 86a60b3
20:13 arlolra: starting Parsoid deploy
19:52 logmsgbot: demon@tin Synchronized multiversion/getMWVersion: for dumps <3 (duration: 00m 46s)
19:44 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-c.eqiad.wmnet
19:44 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-c.eqiad.wmnet
19:44 urandom: !log T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-c.eqiad.wmnet
19:42 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-c.eqiad.wmnet
19:42 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-c.eqiad.wmnet
19:42 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-c.eqiad.wmnet
19:13 urandom: T133805: Restarting Cassandra to apply G1 region size of 32M on restbase1013-a.eqiad.wmnet
19:12 urandom: T133805: Disabling Puppet for GC experiment on restbase1013.eqiad.wmnet
19:10 logmsgbot: thcipriani@tin Synchronized static/images/project-logos: SWAT: Fix HD logos for hewiki (T145017) (duration: 00m 48s)
18:43 ejegg: updated civicrm from 93091637a0a68950efb8955a2cfef031d6ba7883 to 9cbee66e067f788b2634000a33c010dd3f4606b6
18:24 ori: Changing wikiversion for group2 wikis on mw1017 to debug regression (T145359)
18:23 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/resources/Resources.php: Add missing dependency to 'mediawiki.Upload.BookletLayout' module (T145315) (duration: 00m 47s)
18:13 gehel: rolled back wdqs to HEAD^1
17:50 gehel: wdqs1001 put in maitnenance, some issue with config file deployment
17:46 gehel: deploying latest wikidata query service
17:29 godog: roll-restart cassandra in eqiad with new CA and certs T143044
16:30 ema: wiping/repooling cp4015
15:41 godog: roll-restart cassandra in codfw with new CA and certs T143044
15:15 godog: drain and restart cassandra instances on restbase2001 with new CA - T143044
14:51 ema: depool cp4015, restart and repool cp4006's backend
14:38 moritzm: powering down mw2017 for hardware maintenance
14:38 mobrovac: change-prop deploying 5d5d39e
14:36 urandom: T144826: Removing compaction rate limit, increasing compactor threads (from 10 to 20), and beginning scrub of local_group_wikipedia_T_parsoid_html.data (restbase2004-b.codfw.wmnet)
14:10 mobrovac: change-prop deploying 404b07c to enable scap config deploys
13:48 logmsgbot: hashar@tin Synchronized wmf-config: Remove upload7 references T129586 (duration: 00m 50s)
13:32 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/maintenance/cleanupUploadStash.php: Revert "Clean up user handling in UploadStash" T145228 (duration: 00m 46s)
13:31 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/includes/upload/UploadStash.php: Revert "Clean up user handling in UploadStash" T145228 (duration: 00m 46s)
13:27 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.18/extensions/Kartographer: Fix mw.Uri crushing bug T145178 (duration: 00m 49s)
13:21 ema: upgrade cp1099 to varnish 4 T131502
13:17 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 48s)
13:16 logmsgbot: hashar@tin Synchronized wmf-config/throttle.php: Add throttling rule for University of Canterbury T145327 (duration: 00m 46s)
13:15 logmsgbot: hashar@tin Synchronized static/images/project-logos/: Add HD logos for hewiki T145017 (duration: 00m 50s)
13:04 ema: upgrade cp1074 to varnish 4 T131502
12:47 ema: upgrade cp1073 to varnish 4 T131502
12:30 ema: upgrade cp1072 to varnish 4 T131502
12:11 ema: upgrade cp1071 to varnish 4 T131502
12:00 ema: upgrade cp1064 to varnish 4 T131502
11:47 ema: upgrade cp1063 to varnish 4 T131502
11:40 mobrovac: change-prop deploying 79b172a
11:32 ema: upgrade cp1062 to varnish 4 T131502
11:17 ema: upgrade cp1050 to varnish 4 T131502
11:03 ema: upgrade cp1049 to varnish 4 T131502
10:48 ema: upgrade cp1048 to varnish 4 T131502
10:28 marostegui: renaming tables in db1015 - T132837
10:19 moritzm: decomissioning mw2061-mw2074 (Bug: T144745)
10:07 volans: reimage mw2198, mw2199 to Jessie (again) T143536
10:04 marostegui: Testing schema change on db1039 - T141951
10:02 jynus: deploying schema change on s4 hosts T139090
09:47 ema: depool cp4006 (503 Could not get storage)
07:25 moritzm: reimaging mw2077-mw2079, mw2017 to jessie
07:16 moritzm: installing openjpeg security updates
04:34 bblack: upgrade cp3049 to varnish 4 T131502
04:20 bblack: upgrade cp3048 to varnish 4 T131502
04:06 bblack: upgrade cp3047 to varnish 4 T131502
03:51 bblack: upgrade cp3046 to varnish 4 T131502
03:36 bblack: upgrade cp3045 to varnish 4 T131502
03:21 bblack: upgrade cp3044 to varnish 4 T131502
03:05 bblack: upgrade cp3039 to varnish 4 T131502
02:49 bblack: upgrade cp3038 to varnish 4 T131502
02:34 bblack: upgrade cp3037 to varnish 4 T131502
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 12 02:29:27 UTC 2016 (duration 5m 53s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 10m 37s)
02:06 ema: upgrade cp3036 to varnish 4 T131502
01:28 ema: upgrade cp3035 to varnish 4 T131502
00:51 ema: upgrade cp3034 to varnish 4 T131502

2016-09-11

22:33 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Lower wgMaxUserDBWriteDuration to 4 (duration: 00m 47s)
08:44 Amir1: ladsgroup@tin:~$ mwscript resetUserEmail.php --wiki=fawiki Sinasalek <email removed>
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 11 02:30:18 UTC 2016 (duration 5m 55s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 10m 03s)

2016-09-10

22:30 urandom: T144826: Restarting Cassandra on restbase2004-b.codfw.wmnet (scrub complete, re-joining cluster)
12:36 urandom: T144826: Removing compaction rate limit, increasing compactor threads (from 10 to 20), and beginning scrub of local_group_wikipedia_T_parsoid_html.data (restbase2004-b.codfw.wmnet)
02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 10 02:46:34 UTC 2016 (duration 6m 11s)
02:40 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 53s)

2016-09-09

19:29 logmsgbot: demon@tin Synchronized wmf-config/wikitech.php: bizarro config loading (duration: 00m 46s)
18:23 logmsgbot: demon@tin Synchronized wmf-config/: prune old ext messages files (duration: 00m 52s)
16:54 logmsgbot: demon@tin Synchronized multiversion/: rm one more ugly file (duration: 01m 05s)
16:53 logmsgbot: demon@tin Synchronized docroot/noc/conf/: Updating activeMWVersions data (duration: 00m 47s)
16:29 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.18/extensions/JsonConfig/: Unbreak Zero namespace, Check globals in addition to attributes https://gerrit.wikimedia.org/r/309598 (duration: 00m 51s)
16:10 legoktm: live hacking on mw1017
15:15 urandom: T133805: Renabling Pupppet, forcing run, and restarting Cassandra to restore 8M region size on restbase1013-a.eqiad.wmnet
14:52 Jeff_Green: authdns-update for pay-lvs1001 & pay-lvs1002
14:51 mobrovac: change-prop deployed 34b23e7
13:25 elukey: analytics1032 back in service after disk swap
12:45 elukey: running authdns-update on ns0.w.o to pick up the new domain pivot.wikimedia.org (T138262)
12:27 elukey: reimaging mw213[789] and mw2075 to Jessie
12:05 moritzm: reimaging mw2133-mw2136 to jessie
10:19 moritzm: reimaging mw2080, mw2083-mw2085 to jessie
10:04 volans: reimage mw2132 to Jessie
10:03 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=maps,service=kartotherian
09:25 gehel: restarting pybal on lvs1003
09:05 gehel: deploying new LVS configuration for kartotherian.svc.eqiad.wmnet
09:03 elukey: reimage mw2128->mw2131 to Jessie
09:02 godog: reimage ms-be1022 - T140597
08:55 godog: reset power on ms-be2019, cpu "soft lockup"
08:31 moritzm: reimaging mw2124-mw2127 to jessie
07:17 elukey: puppet disabled on analytics1032, Hadoop services stopped - T145170
06:48 moritzm: reimaging mw2120-mw2123 to jessie
05:22 jynus: deploying schema change on s5 hosts T139090
03:32 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: Avoid $wmfMasterDatacenter notices from noc files (duration: 00m 46s)
03:31 logmsgbot: aaron@tin Synchronized docroot/noc/db.php: Avoid $wmfMasterDatacenter notices from noc files (duration: 00m 48s)
02:45 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 9 02:45:47 UTC 2016 (duration 6m 11s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 37s)
01:38 logmsgbot: aaron@tin Synchronized wmf-config/filebackend-production.php: Bump description text expiry for files (duration: 00m 46s)
01:07 logmsgbot: aaron@tin Synchronized tests/Defines.php: (no message) (duration: 00m 46s)
01:02 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/extensions/SpamBlacklist: 56effa952c48725a2665dec72782bc8f7c7915a2 (duration: 00m 49s)
00:06 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.18/extensions/Wikidata: Don't use multiple return values (T145138) (duration: 02m 24s)

2016-09-08

23:53 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/CentralNotice: Bump production version to 4dbd3f9 (duration: 00m 51s)
23:47 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/Popups/extension.json: ext.popups.core depends on mediawiki.storage (Gerrit:309469) (duration: 00m 46s)
23:41 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/Kartographer/modules/box/Map.js: Switch to geojson for geoshapes srv (T144777) (duration: 00m 48s)
23:36 awight: update fundraising crm from cf19366e7f651785276d0071ce2c944d393c0ad5 to 93091637a0a68950efb8955a2cfef031d6ba7883
23:27 awight: rolling back fundraising crm from 946a3f1338bbaf3b65070b03a5a4c4aff5313a90 to cf19366e7f651785276d0071ce2c944d393c0ad5
23:25 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/resources/src/mediawiki/page/rollback.js: RollbackAction: Allow 'from' to be an empty string (T141985, 2/2) (duration: 00m 46s)
23:23 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/includes/actions/RollbackAction.php: RollbackAction: Allow 'from' to be an empty string (T141985, 1/2) (duration: 00m 46s)
23:18 awight: update fundraising crm from cf19366e7f651785276d0071ce2c944d393c0ad5 to 946a3f1338bbaf3b65070b03a5a4c4aff5313a90
23:02 yurik_: scaped kartotherian https://gerrit.wikimedia.org/r/#/c/309473/
22:52 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.18/includes/jobqueue/jobs:
21:56 awight: update SmashPig from 7f9eb7475d194c67ff070b5d8bbc9fc6837b462f to e11af5793df9a0e2dbcf9ac138a8f2a3fa1bf574
20:47 gehel: redeploy wdqs on wdqs2001.codfw.wmnet
20:03 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.18/extensions/Echo/includes/SeenTime.php: Trying to stop some duplicate redis fetches (duration: 00m 52s)
19:57 bblack: repooling normal traffic to cache_upload in ulsfo
19:02 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group2 to wmf.18
18:23 logmsgbot: demon@tin Synchronized multiversion/: So much junk to remove (duration: 01m 06s)
17:58 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase200[1-9]-b.codfw.wmnet
17:53 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-b.eqiad.wmnet
17:53 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-b.eqiad.wmnet
17:53 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-b.eqiad.wmnet
17:46 chasemp: reboot labstore1004 & labstore1005
17:45 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-b.eqiad.wmnet
17:45 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-b.eqiad.wmnet
17:45 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-b.eqiad.wmnet
17:41 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-a.eqiad.wmnet
17:41 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1014-a.eqiad.wmnet
17:41 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1009-a.eqiad.wmnet
17:11 gehel: reverting deploying new LVS configuration for kartotherian.svc.eqiad.wmnet - puppet error, let's analyse slowly...
17:06 gehel: deploying new LVS configuration for kartotherian.svc.eqiad.wmnet
17:04 bd808: Updated Striker to 7d7c8ee
16:55 logmsgbot: demon@tin Synchronized multiversion/: removing more junk - getMWVersion (duration: 01m 07s)
16:47 logmsgbot: demon@tin Finished scap: removing obsolete p symlink (duration: 04m 25s)
16:43 logmsgbot: demon@tin Started scap: removing obsolete p symlink
16:13 godog: roll-restart cassandra instances on restbase-test cluster T143044
16:07 moritzm: uploaded linux-meta 1.10 to carbon (pointing to the new 4.4.19 kernel image)
14:48 logmsgbot: addshore@tin Finished scap: SWAT: Update jquery.uls from upstream (duration: 50m 36s)
14:41 gehel: deploying new DNS entries for kartotherian.svc.eqiad.wmnet
14:36 godog: bounce restbase-test2001 cassandra-a instance T143044
13:58 logmsgbot: addshore@tin Started scap: SWAT: Update jquery.uls from upstream
13:43 moritzm: powering down mw2075-mw2079 for hardware maintenance (T142726)
13:28 ema: upgrading cache_upload ulsfo to varnish 4, dns depooled T131502
13:15 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable mention status notifications everywhere (duration: 00m 47s)
13:12 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add massmessage-sender group to urwiki (duration: 00m 47s)
13:08 logmsgbot: addshore@tin Synchronized wmf-config/extension-list: SWAT: RESTBaseUpdateJobs: Un-deploy the extension 3/3 (duration: 00m 47s)
13:07 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: RESTBaseUpdateJobs: Un-deploy the extension 2/3 (duration: 00m 46s)
13:06 logmsgbot: addshore@tin Synchronized wmf-config/CommonSettings.php: SWAT: RESTBaseUpdateJobs: Un-deploy the extension 1/3 (duration: 00m 49s)
12:52 gehel: redeploying wdqs on wdqs2002.codfw.wmnet - T144380
12:44 moritzm: uploaded new linux package for jessie (based on 4.4.19 with bumped kernel ABI=2)
11:35 moritzm: reimaging mw2161, mw2162, mw2081, mw2082 to jessie
10:14 mobrovac: change-prop deploying a991e25
09:46 godog: roll-reboot thumbor machines to apply memory cgroup enablement T144938
08:52 gehel: initial data mimport on wdqs codfw cluster - T144380
08:23 marostegui: Drop tables: ImageMetricsLoadingTime_10078363 and ImageMetricsCorsSupport_11686678 - T141407
06:51 moritzm: reimaging mw2212-mw2214 to jessie
06:44 elukey: reimaging mw2208->mw2211 to jessie
03:33 ottomata: merging dns change to point archiva.wikimedia.org at new archiva node meitnerium
03:24 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 8 03:24:15 UTC 2016 (duration 7m 23s)
03:16 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 18m 14s)
02:40 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 17m 56s)
00:46 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.18/extensions/VisualEditor/extension.json: https://gerrit.wikimedia.org/r/#/c/309213/ (duration: 00m 46s)
00:42 yurik: kartotherian synced T145042
00:15 twentyafterfour: upgrade complete. Service restored and everything seems normal.
00:14 twentyafterfour: phabricator upgrade is running database migrations now, taking longer than expected
00:03 twentyafterfour: Phabricator upgrade starting momentarily. Service will be offline for a short time, most likely less than 5 minutes.

2016-09-07

23:55 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Translate on fr.wiktionary (T138972) (duration: 00m 47s)
23:45 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Revert "Turn on CirrusSearch bm25 A/B test" (T143588) (duration: 00m 46s)
23:43 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Revert "Turn on CirrusSearch bm25 A/B test" (T143588) (duration: 00m 46s)
23:41 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/VisualEditor/modules/ve-mw/ui/pages/ve.ui.MWParameterPage.js: Fix parent constructor call (Gerrit:309180) (duration: 00m 46s)
23:39 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/VisualEditor/lib/ve: Fix bad serialization of DOM elements in cloneElement (through Gerrit:309156) (duration: 00m 47s)
23:38 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/VisualEditor/lib/ve: Fix bad serialization of DOM elements in cloneElement (through Gerrit:309155) (duration: 00m 47s)
23:36 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.18/extensions/ORES/includes/Hooks.php: Get results when the score is not stored too (T144999) (duration: 00m 46s)
23:24 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Correct dblist definition (T143345, 2/2) (duration: 00m 47s)
23:23 logmsgbot: dereckson@tin Synchronized dblists/: Correct dblist definition (T143345, 1/2) (duration: 00m 49s)
22:02 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/PurgeScoreCache.php --wiki=wikidatawiki --model damaging
21:39 awight: update payments wiki from fafb6b476da3239039780b86d4b8f8d91bb54faa to 996ca30076946c6148d9688a905c82e2e346e165
21:36 Dereckson: Created tables for Translate extension on fr.wiktionary (T138972)
21:35 awight: reprocessing 20160906 PayPal audit files, take 2
21:21 awight: update fundraising-tools from b3ed7ab3deac94c4e465d3768109bca05b6f0a0c to b0be0f9ca04191c4bab869bb81191c5c77c432ca
21:04 urandom: T139961: Stopping RESTBase htmldumper in codfw
21:03 awight: rollback fundraising-tools from b71c504835454572ff70e48297c38b6ca3cbaece to b3ed7ab3deac94c4e465d3768109bca05b6f0a0c
20:50 awight: Reprocessing 20160906 PayPal audit files
20:11 mobrovac: restbase deploy end of 38d8c41
20:07 mobrovac: restbase cassandra truncating local_group_wikipedia_T_feed_aggregated.data for T144990
19:59 mobrovac: mobileapps deploying 2cd4f6a
19:52 mobrovac: restbase deploy start of 38d8c41
19:51 awight: update fundraising-tools from b3ed7ab3deac94c4e465d3768109bca05b6f0a0c to b71c504835454572ff70e48297c38b6ca3cbaece
19:40 urandom: T139961: Actually starting RESTBase htmldumper processes in codfw (read testing)
19:12 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.18
19:01 urandom: T139961: Starting RESTBase htmldumper processes in codfw (read testing)
18:47 thcipriani: Morning SWAT complete
18:46 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/MobileFrontend/resources/mobile.notifications.overlay/NotificationsOverlay.js: SWAT: Count local unread notifications when mark-all-read is clicked (T141404) (duration: 00m 44s)
18:44 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/Echo/modules/model/mw.echo.dm.ModelManager.js: SWAT: Add method to get local unread notifications in the manager (T141404) (duration: 00m 45s)
18:37 gehel: deploying wdqs, fix for T144913
18:35 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Extension:SandboxLink for tcywiki (T144925) (duration: 00m 47s)
18:29 logmsgbot: thcipriani@tin Synchronized dblists/nowikidatadescriptiontaglines.dblist: SWAT: Remove wikidata descriptions from additional projects (duration: 00m 45s)
18:24 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.18/extensions/UniversalLanguageSelector: SWAT: Revert "Update jquery.uls to a9dc11b" (T144871) (duration: 00m 47s)
18:21 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.18/includes/db/loadbalancer/LoadBalancer.php: ddd35a6ccedb68bad41d17c67e4408afe5ca4ae6 (duration: 00m 45s)
18:13 logmsgbot: thcipriani@tin Synchronized dblists/nowikidatadescriptiontaglines.dblist: SWAT: Revert "Enable Wikidata descriptions on all wikipedias" (duration: 00m 47s)
17:23 logmsgbot: aaron@tin Synchronized wmf-config/redis.php: Avoid pointless ChronologyProtector duplicate key notices (duration: 00m 47s)
16:42 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.18/includes/libs/objectcache/WANObjectCache.php: for aaron <3 (duration: 02m 50s)
16:09 volans: restarted ircecho on neon after rotating the irc.log file
16:03 cmjohnson1: db1020 swapping failed disk slot 5
15:59 gehel: deploying wdqs, fix for T144913
15:53 cmjohnson1: graphite1002 swapping failed disk slot10
15:43 madhuvishy: Rebooting host labstore1004
15:36 logmsgbot: krenair@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/309015/ (duration: 02m 50s)
15:08 andrewbogott: re-imaging labnet1002 for T136718
15:02 moritzm: shutting down mw2080-mw2085 for hardware maintenance (T142726)
14:55 mobrovac: mobileapps deploying 8b929cfe
14:49 urandom: T144826: Restarting Cassandra on restbase2004-c.codfw.wmnet (scrub complete, re-joining cluster)
14:45 urandom: T144826: Removing compaction rate limit, increasing compactor threads from 10 to 20, and beginning scrub of local_group_globaldomain_T_mathoid_png.data (restbase2004-c.codfw.wmnet)
14:45 urandom: T144826: Removing compaction rate limit, increasing compactor threads from 10 to 20, and beginning scrub of local_group_globaldomain_T_mathoid_png.data
14:29 hoo: Deployed d4ad9ddbdd21a6460bb3f3a6fa6b74998e33d020 of wikidata/query/deploy: UI improvements
14:16 moritzm: shutting down mw2120-mw2139 for hardware maintenance (T142726)
14:13 moritzm: reimaging mw2157-2160 to jessie
13:58 elukey: reimaging mw2204->mw2207 to jessie
13:57 moritzm: upgrading labvirt1014 to Linux 4.4
13:55 mobrovac: restbase start end of 3852f72
13:31 mobrovac: restbase start deploy of 3852f72
13:15 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.18/extensions/RevisionSlider/modules/ext.RevisionSlider.DiffPage.js: SWAT: [[gerrit:308943|Revert "Do not nest mw-content-text element when reloading a diff" (duration: 00m 47s)
13:10 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable mention status notifications on mediawikiwiki and metawiki (duration: 00m 47s)
13:06 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable RC patrol for fiwiki and some related changes (duration: 00m 47s)
12:58 gehel: enabling row aware allocation on elasticsearch eqiad - T143571
12:51 ema: depool upload in ulsfo
12:38 moritzm: restarted mailman on fermium
12:35 logmsgbot: filippo@palladium conftool action : set/pooled=yes; selector: ms-fe1001.eqiad.wmnet
12:20 mobrovac: mobileapps deploying fc09d0d
11:48 moritzm: reimaging mw2153-mw2156 to jessie
11:04 logmsgbot: filippo@palladium conftool action : set/pooled=no; selector: ms-fe1001.eqiad.wmnet
10:32 hashar: https://yarn.wikimedia.org/ for the lazies
10:30 elukey: yarn.w.o is now available to all the users in the wmf ldap group (Basic Auth)
10:02 godog: add mw:thumbor to read/write ACLs for thumbnail containers of a subset of wikis T139606
09:05 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: pooled db1064 and removed db1019 which was replacing it - T144723 (duration: 00m 52s)
08:06 moritzm: reimaging mw2200-mw2203 to jessie
07:58 elukey: executed apt-get purge tmpreaper on gallium (T132324)
07:27 elukey: reimaging mw2144->mw2147 to jessie
07:00 gehel: increase cluster_concurrent_rebalance on elasticsearch codfw - T143571
06:40 moritzm: reimaging mw2140-mw2143 to jessie
05:38 gehel: enabling row aware allocation on elasticsearch codfw - T143571
03:16 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.18) (duration: 17m 58s)
02:40 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 17m 37s)
02:06 urandom: T144826: Restarting Cassandra restbase2004-b.codfw.wmnet (putting back into service)
01:07 hoo: Updated Wikidata's property suggester with data from Monday's json dump and applied the T132839 workarounds
01:04 Krenair: labtest ldap: created dc=codfw,ou=hosts,dc=wikimedia,dc=org
00:13 Dereckson: Ran namespaceDupes maintenance script on frwiki

2016-09-06

23:54 yurik: deployed kartotherian - adjusting geoshapes arg, and bumping deps - https://gerrit.wikimedia.org/r/#/c/308899/
23:52 Dereckson: Ran namespaceDupes maintenance script on skwiki (0 pages to fix, 1 links to fix, fixed) (T143472)
23:47 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Fix User namespace localisation for sk.wikipedia (T143472) (duration: 00m 47s)
23:23 logmsgbot: dereckson@tin Synchronized dblists/nowikidatadescriptiontaglines.dblist: Update wikis where Wikidata descriptions is shown or not (Gerrit:307968 and Gerrit:307969, T143345) (duration: 00m 46s)
23:14 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Do not use $wgExtensionFunctions to set globals (T143055) (duration: 00m 47s)
22:58 arlolra: Parsoid restarted to pick up new wiki config after <maplink> deploy (T144062)
22:52 arlolra: restarting Parsoid
22:26 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/308890/3 (duration: 00m 47s)
22:09 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.17/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/308887/ (duration: 00m 48s)
21:48 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.17/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/308300/ (duration: 00m 48s)
21:47 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.18/extensions/VisualEditor: https://gerrit.wikimedia.org/r/#/c/308300/ (duration: 00m 49s)
20:59 urandom: T133805: Restarting Cassandra to apply G1 region size of 16M on restbase1013-a.eqiad.wmnet
20:56 urandom: T133805: Disabling Puppet for GC experiment on restbase1013.eqiad.wmnet
20:53 logmsgbot: demon@tin Finished scap: scap 4 wikidata <3 (duration: 25m 53s)
20:27 logmsgbot: demon@tin Started scap: scap 4 wikidata <3
19:48 logmsgbot: demon@tin Synchronized multiversion/: rm useless file (duration: 01m 05s)
19:43 logmsgbot: demon@tin Finished scap: group0 to wmf.18 (duration: 27m 33s)
19:15 logmsgbot: demon@tin Started scap: group0 to wmf.18
19:05 robh: cache misc updates for wmfusercontent complete
18:58 Pchelolo: restbase deploy 4c239f2fa
18:54 Pchelolo: restbase deploy 4c239f2fa canary on restbase1007
18:49 Pchelolo: restbase deploy 4c239f2fa to staging
18:24 logmsgbot: demon@tin Synchronized multiversion/checkoutMediaWiki: rm branch pointer junk (duration: 00m 45s)
18:23 robh: running updates on cache_misc systems to update wmfusercontent certificate
18:05 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Enable VisualEditor by default for logged-out users on Indic-script wikipædias (duration: 00m 50s)
17:47 logmsgbot: demon@tin Finished scap: 1.28.0-wmf.18 initial scap for l10n build (testwiki) (duration: 51m 52s)
17:39 ejegg: enabled banner history queue consumer
17:38 arlolra: updated Parsoid to version 7863e6ad (T142617)
17:36 ejegg: updated CiviCRM from 7484c9085209b83850d0802da8beea74dc594749 to cf19366e7f651785276d0071ce2c944d393c0ad5
17:34 ejegg: disabled banner history queue consumer
17:26 arlolra: starting Parsoid deploy
17:20 urandom: T144826: Ephemerally increasing compactor thread count from 10 to 20
17:13 urandom: T144826: Lifting compaction throttle
17:10 urandom: T144826: Starting online scrub
17:02 urandom: T144826: Restaring Cassandra on restbase2004-a.codfw.wmnet
16:55 logmsgbot: demon@tin Started scap: 1.28.0-wmf.18 initial scap for l10n build (testwiki)
16:11 mobrovac: change-prop restarting to pick up https://gerrit.wikimedia.org/r/308230
11:32 mobrovac: change-prop deploying e14892b
11:19 marostegui: disabled puppet on db1064 - going to be reimaged - T144723
10:50 jynus: shutting down db2001-2009
09:45 marostegui: rysnc running from db1064 to dbstore1001 (T144723)
09:23 marostegui: Stopping mysql on db1064 for maintenance - T144723
09:12 moritzm: shutting down mw2153-mw2162 for hardware maintenance (T142726)
08:54 akosiaris: T144174 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-srd-ita_0.9.0~r72554-1+wmf1
08:20 akosiaris: T144174 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-srd_0.9.0~r72792-1+wmf1
08:20 akosiaris: T144174 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-ita_0.9.0~r72553-1+wmf1
07:41 moritzm: correction: shutting down mw2140-mw2147 and mw2200-mw2214 for hardware maintenance (T142726)
07:40 moritzm: shutting down mw2140-mw2214 for hardware maintenance (T142726)
07:22 elukey: Increasing MaxRequestWorkers on Eqiad Imagescalers - mw129[3-8] - from 30 to 100 (one at the time checking metrics)
07:21 moritzm: reimaging mw2170 to jessie
06:54 moritzm: installing chromium security update on osmium
02:45 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Sep 6 02:45:17 UTC 2016 (duration 6m 28s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 16m 46s)

2016-09-05

23:27 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set wgMathFileBackend to false for wikitech wikis (T126628) (duration: 00m 48s)
23:23 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2002.codfw.wmnet
23:17 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2002.codfw.wmnet
23:17 logmsgbot: dereckson@tin Synchronized dblists/closed.dblist: Close wikimania2015 (T139032) dblist update (duration: 00m 47s)
23:17 ema: upgrading cp2002 to varnish 4 T131502
23:14 logmsgbot: dereckson@tin Synchronized wmf-config/: Close wikimania2015 (T139032). So long and thanks for all the fish. (duration: 00m 51s)
22:55 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2005.codfw.wmnet
22:50 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2005.codfw.wmnet
22:49 ema: upgrading cp2005 to varnish 4 T131502
22:28 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2008.codfw.wmnet
22:24 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2008.codfw.wmnet
22:24 volans: restarting ircecho on neon to get back icinga-wm
22:24 ema: upgrading cp2008 to varnish 4 T131502
22:08 volans: stopped ircecho on neon to avoid the spam of recovery, monitoring icinga, I'll re-enable it in a bit
22:04 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2011.codfw.wmnet
22:00 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2011.codfw.wmnet
21:59 ema: upgrading cp2011 to varnish 4 T131502
21:58 volans: restarting ircecho on neon to get back icinga-wm
21:36 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2014.codfw.wmnet
21:28 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2014.codfw.wmnet
21:28 ema: upgrading cp2014 to varnish 4 T131502
20:56 bd808: Updated striker to b5fdbf9 (T144040, T144296)
20:45 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2017.codfw.wmnet
20:41 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2017.codfw.wmnet
20:41 ema: upgrading cp2017 to varnish 4 T131502
19:49 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2020.codfw.wmnet
19:45 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2020.codfw.wmnet
19:45 ema: upgrading cp2020 to varnish 4 T131502
19:26 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp2024.codfw.wmnet
19:22 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp2024.codfw.wmnet
19:22 ema: upgrading cp2024 to varnish 4 T131502
18:53 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.17/extensions/UploadWizard/resources/mw.UploadWizardUpload.js: SWAT: mw.UploadWizardDetails, mw.UploadWizardUpload: Use amenableparser to handle templates in error messages Part 2/2 (duration: 00m 46s)
18:52 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.17/extensions/UploadWizard/resources/mw.UploadWizardDetails.js: SWAT: mw.UploadWizardDetails, mw.UploadWizardUpload: Use amenableparser to handle templates in error messages Part 1/2 (duration: 00m 48s)
18:51 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.17/resources/src/mediawiki/mediawiki.Upload.BookletLayout.js: SWAT: mw.Upload.BookletLayout: Use amenableparser to handle templates in error messages (duration: 00m 47s)
18:50 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.17/resources/src/mediawiki/api/messages.js: SWAT: mw.api.messages: Allow passing extra parameters for the API call (duration: 00m 53s)
18:31 ema: upgrading cp2026 to varnish 4 T131502
17:43 ema: restarting pybal on lvs2002 T134893
17:35 ema: upgrading cp2022 to varnish 4 T131502
17:34 mobrovac: change-prop deploying 222fcf8
14:42 elukey: upgrading apache httpd to the latest version on mw129[3-8] (eqiad image scalers)
14:05 logmsgbot: marostegui@tin Synchronized wmf-config/db-eqiad.php: Changing db-eqiad config to depool db1064 - T144723 (duration: 00m 48s)
14:00 elukey: upgrading mw1306/mw1299 to the latest version of Apache httpd
13:47 mobrovac: change-prop restarting for https://gerrit.wikimedia.org/r/306308
13:40 elukey: upgrading mw130[012345] to the latest version of Apache httpd (eqiad jobrunners, one at the time)
13:23 moritzm: reimaging mw2087 to jessie
13:12 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable the RevisionSlider on test.wikidata.org (duration: 00m 48s)
13:07 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add source wikis in import page in sawiki (duration: 00m 53s)
12:49 ema: repool cp4005 with varnish 3
12:47 moritzm: depooling/rebooting/repooling sca1002 for upgrade to Linux 4.4 (T144492)
12:41 ema: downgrading cp4005 to varnish 3 T131502
11:40 elukey: Reimaging mw217[89] and mw219[6789] to Debian jessie
10:37 moritzm: depooling/rebooting/repooling sca1001 for upgrade to Linux 4.4 (T144492)
10:18 moritzm: reimaging mw2192-mw2195 to jessie
09:24 elukey: reimaging mw21(8[89]|9[01]) to Debian Jessie
09:14 elukey: reimaging mw218[4567] to Debian Jessie
09:01 moritzm: reimaging mw2174-mw2177 to jessie
07:12 moritzm: reimaging mw2169-mw2172 to jessie
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Sep 5 02:29:51 UTC 2016 (duration 5m 42s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 10m 50s)

2016-09-04

20:11 elukey: re-enabled ircecho on neon
20:03 elukey: stopped ircecho on neon temporarily
19:32 elukey: restarting apache2 on rhodium (attempt to fix it)
19:14 hashar: Puppet is falling since ~ 18:05 UTC. At least a couple european ops are looking at it
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Sep 4 02:31:20 UTC 2016 (duration 6m 7s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 09m 57s)

2016-09-03

14:24 ema: depool cp4005
10:21 jynus: deploying schema change on s7 hosts T139090
02:46 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Sep 3 02:46:12 UTC 2016 (duration 6m 22s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 17m 35s)

2016-09-02

22:31 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.17/includes/page/WikiPage.php: No-op sync - 880c7f9526610bfd396dddb14f7772f9247850cb (duration: 00m 49s)
16:50 thcipriani: jenkins restarted
16:50 mark: rebooting bast3001
16:49 ema: downgrading cp4015 to varnish 3 T131502
16:40 thcipriani: restarting jenkins shortly for plugin upgrade
16:10 ema: downgrading cp4014 to varnish 3 T131502
15:33 ema: downgrading cp4013 to varnish 3 T131502
14:27 ema: downgrading cp4007 to varnish 3 T131502
14:14 mutante: gallium deleting jenkins/config-history files older than an hour
13:48 moritzm: rebooting labnet1002 for kernel update
12:09 ema: pooling ulsfo
11:37 elukey: upgrading mw1283.eqiad.wmnet to the latest httpd version
10:17 moritzm: reimaging mw2180-mw2183 to jessie
10:10 elukey: reimaging mw216[5-8] to jessie (IPMI fixed)
09:48 mark: Raised OSPF metrics on cr2-ulsfo<-->cr1-codfw link from 388 to 1000 in both directions
09:13 elukey: upgrading httpd on mw127[6789] to 2.4.10-10+deb8u6+wmf2 (eqiad api canaries)
09:03 ema: reboot cp4006 for kernel upgrade
08:38 moritzm: reimaging mw2099, mw2117, mw2163, mw2164 to jessie
08:34 elukey: upgrading httpd on mw1289 to 2.4.10-10+deb8u6+wmf2
08:19 elukey: upgrading httpd on mw128[0124] to 2.4.10-10+deb8u6+wmf2
08:04 elukey: upgrading httpd on mw1290 to 2.4.10-10+deb8u6+wmf2
06:40 moritzm: reimage mw2101 to jessie
06:36 moritzm: reimage mw2149-mw2151 to jessie
04:35 mutante: tin - re-enabled puppet
02:57 awight: Began bulk refund for T144489
02:43 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Sep 2 02:43:50 UTC 2016 (duration 5m 15s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 16m 12s)
01:06 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Customize wgMathDirectory for wikitech (T126628, 2/2) (duration: 00m 46s)
01:05 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Customize wgMathDirectory for wikitech (T126628, 1/2) (duration: 00m 47s)
00:22 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Math on wikitech (T126338) (duration: 00m 47s)
00:13 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: End lazy loading reference experiments (T144240) (duration: 00m 47s)
00:10 logmsgbot: dereckson@tin Synchronized wmf-config/CirrusSearch-common.php: Disable phrase suggester for wikidata (T143260, 2/2) (duration: 00m 46s)
00:09 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Disable phrase suggester for wikidata (T143260, 1/2) (duration: 00m 46s)
00:03 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/ZeroBanner: Update router code (T143425) (duration: 00m 47s)
00:02 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/MobileFrontend/resources/mobile.startup/Skin.js: Ensure lazy image placeholders without height can be loaded (T143768) (duration: 00m 46s)
00:00 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/Flow/modules/flow/ui/widgets/editor/editors/mw.flow.ui.VisualEditorWidget.js: Flow Fixes related to Visual Editor (T138356 and T139972) (duration: 00m 45s)

2016-09-01

23:57 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/VisualEditor/lib/ve: Update lib/ve submodule for Ib9bbaccfff9 (duration: 00m 47s)
23:51 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.17/extensions/CirrusSearch/includes/Query/FullTextQueryStringQueryBuilder.php: Do not use the suggest reverse field if it's a non local search (Gerrit:307955) (duration: 00m 48s)
23:39 mutante: tin removed mw2187 from /etc/dsh/group/scap-proxies
23:38 mutante: tin stopping puppet
23:32 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Enable Wikidata descriptions taglines on labs (T143345, no-op in prod) (duration: 02m 52s)
23:15 XenoRyet: updated payments-wiki from ef2f2f85461d25a3938c173d0116fa0d2dee58d1 to fafb6b476da3239039780b86d4b8f8d91bb54faa
23:06 MaxSem: That for https://gerrit.wikimedia.org/r/#/c/308084/
23:06 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 02m 53s)
23:01 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.17/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/308085/ (duration: 02m 54s)
22:45 ejegg: enabled fredge queue consumer
22:38 ejegg: updated civicrm from 1678e1f52468dae596882a323cfa5365b966d5f0 to 7484c9085209b83850d0802da8beea74dc594749
22:36 ejegg: disabled fredge queue consumer
22:31 ejegg: updated paymentswiki settings from 393944f179b6823f27dc1a9430b4a3661388ccff to 18451935c65f711e5c50794211a05309fd903a40
22:11 awight: reenabling recurring Ingenico job and kicking one-off run
22:03 awight: updating wmf_civicrm schema to 7240
21:59 awight: update fundraising CRM from 0c6bf3813ee0f2e58d5948fc7c000cf20c114841 to 1678e1f52468dae596882a323cfa5365b966d5f0
21:25 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-a
21:25 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-a
21:24 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-a
21:22 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-a
21:22 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-a
21:21 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-a
20:58 mutante: mw2187 - shut down
20:35 urandom: T143226: Clearing repair status: eqiad, rack 'd' nodes
20:35 urandom: T143226: Clearing repair status: eqiad, rack 'dd' nodes
19:58 hashar: 1.28.0-wmf.17 rolled on group2 and apparently all fine
19:54 gehel: reloading ferm rules on elasticsearch eqiad cluster
19:44 urandom: T143226: Clearing repair status: eqiad, rack 'b' nodes
19:39 urandom: T143226: Clearing repair status restbase1011-c.eqiad.wmnet
19:19 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.17
19:14 mutante: tin temp. disabled puppet
19:11 mutante: tin removing mw2167 thru mw2199 from dsh file manually, re-running puppet
18:33 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Test PageAssessments on English Wikivoyage (T142056) (duration: 02m 48s)
18:26 mutante: mw2187 - powercycled
18:19 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: name=elastic1029.eqiad.wmnet
18:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: REVERT because proxy down SWAT: Test PageAssessments on English Wikivoyage (T142056) (duration: 03m 15s)
18:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Test PageAssessments on English Wikivoyage (T142056) (duration: 04m 54s)
17:55 Jeff_Green: switching fundraising database reader from db1008 to frdb1001
17:12 bd808: Updated striker to ac555bd; fixes T144064
16:49 ema: cp4006 repooled after downgrade
16:44 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: name=elastic1028.eqiad.wmnet
16:24 ema: restarting pybal on lvs4002 T134893
16:10 ema: downgrading cp4006 to varnish 3 T131502
15:36 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: name=elastic102[12].eqiad.wmnet
15:34 chasemp: reboot nova-compute on labvirt1013 as stuck (no logs, not applying any changes or taking any instruction)
14:58 moritzm: powered down several hosts for hardware maintenance (T142726): mw2099, mw2102, mw2117, mw2163-mw2199
14:42 mobrovac: restbase deploy end of 9cca320
14:40 moritzm: powered down several hosts for hardware maintenance (T142726): mw2087, mw2149-mw2151
14:39 elukey: upgrading httpd/apache to 2.4.10-10+deb8u6+wmf2 on mw128[56]
14:27 mobrovac: restbase deploy start of 9cca320
14:11 godog: wipe and reinitialize corrupted xfs on /dev/sdn1 on ms-be1016
13:53 eileen: turned off queue for http://localhost:9000/job/GlobalCollect%20Recurring%20Donations/ in jenkins
13:42 elukey: upgrading httpd/apache to 2.4.10-10+deb8u6+wmf2 on mw128[78]
13:41 bblack: uploaded openssl-1.1.0-1+wmf1 to jessie-wikimedia/experimental
13:39 elukey: not upgrading mw130[01] since I'd need more info before proceeding
13:34 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.17/includes/page/WikiPage.php: T144484 (duration: 00m 49s)
13:30 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.17/includes/page/WikiPage.php: T144484 (duration: 00m 35s)
13:16 elukey: upgrading httpd/apache to 2.4.10-10+deb8u6+wmf2 on mw130[01].eqiad.wmnet
13:01 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: name=elastic1047.eqiad.wmnet
12:53 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-be'])
12:12 gehel: rolling restart of ferm on elasticsearch eqiad cluster to account for moved servers - T143685
10:50 moritzm: installing libidn security updates
10:39 godog: reboot ms-be1016, stuck again
10:24 logmsgbot: hashar@tin Synchronized docroot/noc/index.html: link to conftool and wikitech pages on https://noc.wikimedia.org/ (duration: 00m 47s)
10:17 gehel: repooled elastic104[456] - T144450
09:58 jynus: adding marostegui to wmf and ops on wikitech LDAP
09:57 logmsgbot: gehel@palladium conftool action : set/pooled=inactive; selector: elastic1046.eqiad.wmnet
09:57 logmsgbot: gehel@palladium conftool action : set/pooled=inactive; selector: elastic1045.eqiad.wmnet
09:56 logmsgbot: gehel@palladium conftool action : set/pooled=inactive; selector: elastic1044.eqiad.wmnet
09:51 moritzm: reimaging mw2200-2203 to jessie
09:37 moritzm: reimaging mw2061-2064 to jessie
09:35 elukey: reimaging mw2167 -> mw2170 to Jessie
09:24 moritzm: reimaging mw2163-2166 to jessie
09:06 moritzm: installing postgres security updates on labsdb1004/1006/1007
09:04 godog: reboot ms-be1016, stuck and nothing on console
07:45 moritzm: reimaging mw2116-2119 to jessie
03:12 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Sep 1 03:12:34 UTC 2016 (duration 7m 14s)
03:05 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 17m 59s)
02:30 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 12m 21s)
02:22 bblack: ulsfo depool
01:00 Jeff_Green: reboot db1025 for kernel update
01:00 Jeff_Green: reboot db1025 for kernel update
00:29 eileen: CiviCRM upgrade from from d067c476da074afa70297c29e3d471743c110e3a to 0c6bf3813ee0f2e58d5948fc7c000cf20c114841
00:13 eileen: fr campaigns disabled
00:09 eileen: stop Donations q consumer job on jenkins

2016-08-31

23:53 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/307886/ (duration: 00m 48s)
23:42 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/268627/ (duration: 00m 47s)
23:40 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/268627/ (duration: 00m 47s)
23:34 urandom: T143226: Clearing repair status: eqiad, rack 'a' nodes
23:34 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: Updating logos, step 2 (duration: 00m 48s)
23:31 logmsgbot: maxsem@tin Synchronized static: Update logos, step 1 (duration: 00m 47s)
23:20 urandom: T143226: Clearing repair status: codfw, rack 'd'
23:15 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.17/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/307883/ (duration: 00m 49s)
22:33 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: name=elastic1047.eqiad.wmnet
22:32 gehel: depooling elastic1047 from LVS
22:17 gehel: restarting elasticsearch on elastic1028
22:12 urandom: T143226: Clearing repair status: restbase2008.codfw.wmnet
22:02 hashar: surge of since 31th 7:00 UTC : Retrying connection to search.svc.eqiad.wmnet
22:01 urandom: T143226: Clearing repair status: restbase2004.codfw.wmnet
21:52 urandom: T143226: Clearing repair status: restbase2003.codfw.wmnet
21:46 urandom: T143226: Clearing repair status: restbase2007.codfw.wmnet
21:44 urandom: T143226: Clearing repair status: restbase2002.codfw.wmnet
21:39 urandom: T143226: Starting restbase2001-{b,c}.codfw.wmnet
21:38 urandom: T143226: Stopping restbase2001-{a,b} to clear tables marked repaired
21:27 urandom: T143226: Stopping restbase2001-a.codfw.wmnet to clear tables marked repaired
19:59 hashar: group1 to 1.28.0-wmf.17 done. There is a couple explicit commit of implicit transactions for Wikidata T144433 T144434 not much of a worry
19:50 apergos: restarted udp2log-mw on fluorine again after deployment of https://gerrit.wikimedia.org/r/#/c/307812/
19:46 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.17
19:15 apergos: stopped/started udp2log-mw on fluorine
18:54 gehel: shutting down elasticsearch on elastic1028 to prepare moving server - T143685
18:52 Pchelolo: RESTBase deploy fa1a5aab
18:47 Pchelolo: RESTBase deploy fa1a5aab canary on restbase1007
18:34 Pchelolo: RESTBase deploy fa1a5aab to staging
18:22 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.16/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Enable CirrusSearch BM25 AB test (duration: 00m 47s)
18:17 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.17/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: Enable CirrusSearch BM25 AB test (duration: 00m 50s)
17:41 mobrovac: change-prop deploying bf221599c
14:49 mobrovac: parsoid first scap3 deploy to wtp20xx
14:40 godog: reenable puppet on wtf* post-merge of scap3 conversion
14:39 elukey: the previous entry was related to apache2
14:38 elukey: upgrading the remaining codfw mw* to 2.4.10-10+deb8u6+wmf2
14:38 mobrovac: parsoid first scap3 deploy to wtp2001 only
14:25 godog: stopping puppet on wtp* prior to merging https://gerrit.wikimedia.org/r/#/c/304470/
14:19 bblack: packages updates for cp* (new kernel, new nginx, misc updates)
13:59 bblack: uploaded nginx 1.11.3-1+wmf2 for jessie-wikimedia to carbon (ssl curve variable support)
13:58 jynus: running alter table on db1047 enwiki.categorylinks to mitigate lag issues
13:52 elukey: upgrading the codfw jessie appservers to 2.4.10-10+deb8u6+wmf2
13:40 chasemp: restart rabbitmq on labcontrol1001
13:36 bblack: repooling ulsfo traffic
13:25 moritzm: installing libarchive security updates on jessie
13:13 moritzm: reimaging mw2212-mw2215 to jessie
11:39 godog: swift eqiad-prod: set weight for ms-be1021 sd[h-n] to 0 - T139767
11:10 gehel: restarting elasticsearch104[456] to take new rack configuration into account - T143685
10:33 akosiaris: starting a gradual deployment of wikidiff2 across the mw* fleet. T140443
09:53 mobrovac: change-prop deploying e999f517
09:51 elukey: restarting cassandra on aqs100[123] to verify performances after daemon restart
09:42 moritzm: reimaging mw2108-mw2111 to jessie
08:24 akosiaris: drained ulsfo of traffic, it seems to be unstable currently
07:10 moritzm: removed cached files on stat1001 (/var/www/limn-public-data/caching) after checking with joal, disk space on root partition was depleted
07:07 jynus: removed db1027 from our dns
07:05 moritzm: reimaging mw2104-mw2108 to jessie
04:28 ejegg: updated civicrm from e1feb34ff688e230cced92c46e5c2a78e2b3cffa to d067c476da074afa70297c29e3d471743c110e3a
03:48 ejegg: rolled back civicrm to e1feb34ff688e230cced92c46e5c2a78e2b3cffa
03:43 ejegg: updated civicrm from e1feb34ff688e230cced92c46e5c2a78e2b3cffa to 8895f0e8ac9ad03cdf11b81542c17c5a39ca5cd8
03:17 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 31 03:17:37 UTC 2016 (duration 7m 13s)
03:11 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.17/resources/src/mediawiki/mediawiki.js: Ie4e7464aa811 and I90eea4bfe (duration: 00m 47s)
03:10 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.17) (duration: 19m 02s)
02:57 ejegg: rolled back civicrm to e1feb34ff688e230cced92c46e5c2a78e2b3cffa
02:53 ejegg: updated civicrm from e1feb34ff688e230cced92c46e5c2a78e2b3cffa to c0a7f45bd913ebd28e16c0d78b009a7af95209e9
02:51 ejegg: disabled fredge queue consumer
02:34 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 12m 45s)
00:06 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.17/extensions/Flow: SWAT (duration: 01m 09s)
00:05 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.17/extensions/Echo: SWAT (duration: 00m 53s)

2016-08-30

23:59 logmsgbot: catrope@tin Synchronized wmf-config/CirrusSearch-common.php: Fix CirrusSearch BM25 A/B test similarity config (duration: 00m 48s)
23:59 RoanKattouw: Whoops, I meant scap sync-file, not scap sync
23:59 logmsgbot: catrope@tin scap aborted: wmf-config/CirrusSearch-common.php Fix CirrusSearch BM25 A/B test similarity config (duration: 00m 01s)
23:59 logmsgbot: catrope@tin Started scap: wmf-config/CirrusSearch-common.php Fix CirrusSearch BM25 A/B test similarity config
23:58 logmsgbot: catrope@tin Synchronized wmf-config/logging.php: Require ack of kafka logging (T135159) (duration: 00m 47s)
23:40 ejegg: updated SmashPig from 4ba14a10518f80fe7e20fadc493d78c796c15921 to 7f9eb7475d194c67ff070b5d8bbc9fc6837b462f
23:35 ejegg: rolled back SmashPig to 4ba14a10518f80fe7e20fadc493d78c796c15921
23:34 logmsgbot: catrope@tin Synchronized wmf-config: Enable Wikidata description taglines on all projects except top 6 wikis (T143344) (duration: 00m 54s)
23:33 logmsgbot: catrope@tin Synchronized dblists/nowikidatadescriptiontaglines.dblist: (no message) (duration: 00m 53s)
23:30 ejegg: updated SmashPig from 4ba14a10518f80fe7e20fadc493d78c796c15921 to b94325bea6158e80f194689e247b16239053baa1
22:19 ejegg: rolled back SmashPig to 4ba14a10518f80fe7e20fadc493d78c796c15921
22:18 mobrovac: change-prop deploying a87a61d
21:57 ejegg: updated SmashPig from 4ba14a10518f80fe7e20fadc493d78c796c15921 to b94325bea6158e80f194689e247b16239053baa1
20:16 gehel: restarting ferm on elasticsearch eqiad cluster after reinstall of elastic104[4567] - T143685
20:09 yuvipanda: cleanup /var/cache/iegreview for bd808
19:52 legoktm: deployed https://gerrit.wikimedia.org/r/306710, moving 4 parsoid CI jobs from nodepool trusty to nodepool jessie
19:47 mutante: rsyncing archiva data from titanium to meitnerium, runs in a screen
19:38 hashar: 1.28.0-wmf.17 successfully pushed to group0
19:34 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.17 (bis) T144307
19:30 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.16/extensions/CentralAuth/includes/CentralAuthUser.php: Remove verbose cache miss log that was making notices T144307 (duration: 00m 48s)
19:05 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: Rollback 1.28.0-wmf.17 T142117
19:03 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.17 T142117
18:28 gehel: deploying latest version of wikidata query service
18:17 chasemp: restart nodepool
18:12 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-nondefault.dblist: SWAT: Enable VisualEditor by default for logged-in users on Indic-script Wikipedias (T142586) PART II (duration: 00m 49s)
18:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor by default for logged-in users on Indic-script Wikipedias (T142586) PART I (duration: 00m 48s)
18:08 mobrovac: change-prop deploying c793e4a2
17:48 gehel: shutting down elasticsearch on elastic1046 to prepare moving server - T143685
17:08 gehel: shutting down elasticsearch on elastic1045 to prepare moving server - T143685
17:05 gehel: shutting down elasticsearch on elastic1044 to prepare moving server - T143685
16:58 gehel: shutting down elasticsearch on elastic1047 to prepare moving server - T143685
16:13 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.17/includes/EditPage.php: (no message) (duration: 00m 48s)
16:04 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable allowDataAccessInUserLanguage on Wikidata (T122670) (duration: 00m 51s)
15:44 logmsgbot: hashar@tin Finished scap: testwiki to php-1.27.0-wmf.17 T142117 (duration: 53m 28s)
15:35 godog: restart pybal on lvs primaries in codfw/eqiad
14:51 logmsgbot: hashar@tin Started scap: testwiki to php-1.27.0-wmf.17 T142117
14:46 ema: banning objects with status code 200 and content-length 0 from upload frontends in ulsfo T144257
14:44 godog: bounce pybal to pick up prometheus.svc on low-traffic in eqiad/codfw
14:41 godog: bounce pybal to pick up prometheus.svc
14:40 ema: banning objects with status code 200 and content-length 0 from upload backends in ulsfo T144257
14:39 hashar: Applying security patches for 1.28.0-wmf.17 T142117
14:35 hashar: mw2101 running scap pull , it missed bunch of files
14:35 zeljkof: European SWAT is done!
14:34 logmsgbot: filippo@palladium conftool action : set/pooled=yes; selector: prometheus1001.eqiad.wmnet
14:34 logmsgbot: filippo@palladium conftool action : set/pooled=yes; selector: prometheus2001.codfw.wmnet
14:32 logmsgbot: zfilipin@tin Synchronized php-1.28.0-wmf.16/extensions/CirrusSearch/includes/CirrusSearch.php: SWAT: Initialize the UserTesting framework before creating a Connection (duration: 00m 49s)
14:20 logmsgbot: zfilipin@tin Synchronized dblists/s3.dblist: SWAT: Sort s3.dblist in lexicographical order (duration: 02m 44s)
14:14 hashar: Moved mediawiki-core-phpcs job back to Nodepool T143938
14:01 hashar: European SWAT extended
13:53 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-be'])
13:48 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow bureaucrats to manage account creators group on ar.wikipedia (T143844) (duration: 00m 50s)
13:11 gehel: cleanup openjdk 7 on maps2002 - T142977
13:06 gehel: removing unused openjdk 7 on maps1001
11:47 gehel: banning elastic10(44|45|46|47) from elasticsearch eqiad cluster - T143685
11:42 moritzm: upgrading remaining jessie-based mw systems to hhvm 3.12.7 (now that the systemd unit override is in place)
11:01 godog: roll-restart xenon/cerium/praseodymium cassandra instances to pick up new certs
10:53 hashar: Cutting MediaWiki branch 1.28.0-wmf.17 T142117
09:57 ema: restarted morebots
02:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 30 02:36:06 UTC 2016 (duration 6m 36s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 57s)

2016-08-29

23:47 chasemp: stop nfs server on labstore1001 to try to fix scratch mount
23:34 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.16/extensions/ORES/includes/Cache.php: Improvements to purging cache (T144216) (duration: 00m 47s)
23:27 Dereckson: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add Əlavə namespace to az.wiktionary (T143851) (duration: 00m 47s)
23:26 chasemp: restart nfs-kernel-server on labstore1003 and labstore1001
23:25 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: EditSubmitButtonLabelPublish: Temporarily don't do this (currently no-op) (duration: 00m 47s)
23:25 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: ores: Update thresholds (T144101) (duration: 00m 48s)
22:45 ejegg: enabled donations queue consumer
22:40 ejegg: updated CiviCRM from d040549105151ccb66a464effbbd56a3e4bfbb8b to e1feb34ff688e230cced92c46e5c2a78e2b3cffa
22:39 ejegg: disable donations queue consumer
22:39 dapatrick: Deployed patch for T125177 to 1.28.0-wmf.16
21:56 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: T131132 (duration: 00m 48s)
21:32 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-a.eqiad.wmnet
21:31 urandom: T143226: Starting restbase1015-a.eqiad.wmnet
21:29 urandom: T143226: Stopping restbase1015-a.eqiad.wmnet to remove repairedAt attribute on a table
21:11 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1007-b.eqiad.wmnet
20:33 akosiaris: restarted ircecho on neon.wikimedia.org (icinga-wm)
20:12 bd808: Updated striker to ffe13c1; see https://wikitech.wikimedia.org/wiki/Toolsadmin.wikimedia.org/Deployments#2016-08-29
20:11 bd808: starting striker deploy
20:10 subbu: finished deploying parsoid sha 48cf803e
20:04 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:02 subbu: starting parsoid deploy
19:41 bd808: Taking stashbot offline (hopefully briefly)
19:26 Niharika: Updated iegreview to 29e98bb (Show all users in 'Manage Users' list (blocked and unblocked both))
18:42 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Define variables for AB test for all wikis and Revert "Revert "CirrusSearch BM25 A/B test config"" (duration: 00m 47s)
18:32 chasemp: make greg a phab admin to fight surge of spam bots
18:29 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: REVERT SWAT: CirrusSearch BM25 A/B test config (T143586) (duration: 00m 48s)
18:23 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: CirrusSearch BM25 A/B test config (T143586) (duration: 00m 46s)
18:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Switch enwiki to uca-default collation (T136150) (duration: 00m 47s)
17:31 logmsgbot: demon@tin Synchronized multiversion/: delete deleteMediawiki (duration: 01m 09s)
16:12 Amir1: mwscript extensions/ORES/maintenance/PopulateDatabase.php --wiki=fawiki (and seven other wikis) T144101
16:07 Amir1: mwscript extensions/ORES/maintenance/PurgeScoreCache.php --wiki=enwiki (and seven other wikis) (T144101)
15:34 halfak: restarted ores-celery-worker service on scb1002
15:23 Amir1: ladsgroup@terbium:~$ mwscript extensions/ORES/maintenance/CheckModelVersions.php on (fa|tr|pl|ru|pt|nl|wikidata|en)wiki
15:12 halfak: deploying ores b8598dd (see T144101)
14:28 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4015.ulsfo.wmnet
14:20 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4015.ulsfo.wmnet
14:20 ema: upgrading cp4015 to Varnish 4 (T131502)
14:06 gehel: restart elasticsearch on logstash1004 to validate fix for T142357
14:03 hashar: European SWAT deploy is complete
13:59 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.16/includes/api/ApiUpload.php: ApiUpload: Better handle unreasonably large metadata in 'imageinfo' T143993 (duration: 00m 46s)
13:44 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4014.ulsfo.wmnet
13:37 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4014.ulsfo.wmnet
13:36 ema: upgrading cp4014 to Varnish 4 (T131502)
13:36 logmsgbot: hashar@tin Synchronized wmf-config/missing.php: Do not prepend protocol in missing.php T141208 (duration: 00m 47s)
13:26 logmsgbot: hashar@tin Synchronized wmf-config/throttle.php: Add throttling exception for UBC T143951 (duration: 00m 46s)
13:19 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Disable ORES for reverted, goodfaith and wp10 models T143988 (duration: 02m 43s)
13:15 hashar: Pulled https://gerrit.wikimedia.org/r/#/c/306951/ on mw1099
13:14 hashar: Pulled https://gerrit.wikimedia.org/r/#/c/306904/ on mw1099
13:11 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.16/extensions/CirrusSearch/includes/Query/FullTextSimpleMatchQueryBuilder.php: Fallback to QueryString if we detect acronyms T143541 (duration: 00m 50s)
13:08 hashar: Pulled https://gerrit.wikimedia.org/r/#/c/307261/ on mw1099
12:51 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4013.ulsfo.wmnet
12:42 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4013.ulsfo.wmnet
12:42 ema: upgrading cp4013 to Varnish 4 (T131502)
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-urd-hin_0.1.0~r64379-1+wmf1
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-sme-nob_0.6.0~r61921-1+wmf1
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-isl-eng_0.1.0~r66083-1+wmf1
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hbs-slv_0.1.0~r59294-1+wmf1
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hbs-eng_0.1.0~r57598-1+wmf1
11:52 elukey: reimaging mw209[01] to jessie
11:31 moritzm: reimaging mw2088 to jessie
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: giella-sme_0.0.20150917~r121176-1+wmf1
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-isl_0.1.0~r65494-1+wmf1
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hbs_0.5.0~r68212-1+wmf1
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eus_0.1.0-1+wmf1
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-en-es_0.8.0+svn~57502-2+wmf1
11:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-en-ca_0.9.3~r61328-2+wmf1
10:52 hashar: Upgrading Zuul server on gallium zuul_2.5.0-8-gcbc7f62-wmf1precise1 zuul_2.5.0-8-gcbc7f62-wmf2precise1 T144088
10:09 moritzm: disabling puppet on jessie mediawiki hosts in eqiad for staged merged of https://gerrit.wikimedia.org/r/#/c/306225/
09:58 elukey: Executed 'kafka preferred-replica-election' on kafka1012 to rebalance Kafka broker leaders (T144158)
09:55 moritzm: depooled mw1266, running some tests with systemd unit override and hhvm update
09:44 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4007.ulsfo.wmnet
09:28 ema: upgrading cp4007 to Varnish 4 (T131502)
08:18 hashar: Upgrading Zuul server on gallium zuul 2.1.0-391-gbc58ea3-wmf2precise1 => zuul_2.5.0-8-gcbc7f62-wmf1precise1 T144088
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 29 02:31:06 UTC 2016 (duration 5m 53s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 25s)

2016-08-28

16:51 hoo: Ran T132839-Workarounds.sh from my home in terbium (see T132839)
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 28 02:30:02 UTC 2016 (duration 5m 58s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 19s)

2016-08-27

02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 27 02:30:35 UTC 2016 (duration 5m 50s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 12s)

2016-08-26

22:51 apergos: from a few hours ago: (08:42:47 μμ) bd808: !log Updated striker to fix T143956
22:49 apergos: running html dmp for en wikipedia manually out of screen session (ariel) on francium

2016-08-25

15:13 ema: pooling cp4005 backend (varnish 4 cache_upload) T131502
14:37 elukey: upgrading httpd to 2.4.10-10+deb8u6+wmf2 on mw1269/mw127[01]
14:17 moritzm: re-enabled puppet on kafka*, mc* and rdb* hosts
13:45 hashar: European SWAT is done
13:44 logmsgbot: zfilipin@tin Synchronized php-1.28.0-wmf.16/extensions/UniversalLanguageSelector: SWAT: ext.uls.compactlinks: consistently normalize language codes (T143867) (duration: 00m 47s)
13:42 logmsgbot: zfilipin@tin Synchronized php-1.28.0-wmf.15/extensions/UniversalLanguageSelector: SWAT: ext.uls.compactlinks: consistently normalize language codes (T143867) (duration: 00m 49s)
13:40 moritzm: temporarily disabling puppet on kafka*, mc* and rdb* hosts (to enable ferm ipsec change in controlled stages)
13:09 moritzm: puppet re-enabled on all kafka* hosts
12:00 moritzm: temporarily disabling puppet on kafka* hosts (to enable ferm changes in controlled stages)
10:00 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe'])
10:00 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx'])
09:52 moritzm: temporarily stop irc bot, until puppet has self-healed
08:17 moritzm: restarted hhvm on mw1191 and mw1216, got stuck
07:07 moritzm: installing harfbuzz security updates
03:28 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.16/includes/specialpage/AuthManagerSpecialPage.php: UBN fix for T143840 (duration: 00m 49s)
02:55 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 25 02:55:50 UTC 2016 (duration 6m 30s)
02:49 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 11m 09s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 41s)
01:49 mutante: iridium - temp. stopping phd service for home dir change
01:08 mutante: radium apt-get autoremove; apt-get upgrade (openssh, openssl, passwd, sudo, libpam, libc6 :)
00:51 urandom: T137474: Stopping dumps in RESTBase staging, and reverting xenon.eqiad.wmnet to Cassandra 2.2.6-wmf1
00:32 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.16/extensions/CirrusSearch/includes/Job/CheckerJob.php: Fix CirrusSearch CheckerJob stuck in a loop (duration: 00m 47s)
00:08 mutante: chromium back in service - both eqiad DNS recursors now on jessie

2016-08-24

23:43 logmsgbot: dzahn@palladium conftool action : set/pooled=yes; selector: name=chromium.wikimedia.org
23:39 mutante: chromium - install ntpdate, stop ntp, sync time with hydrogen, start ntp, remove ntpdate
23:36 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.15/extensions/UploadWizard/resources: More debug logging for Firefox's 'NS_ERROR_NOT_AVAILABLE' exceptions (T136831) (duration: 00m 49s)
23:35 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.16/extensions/UploadWizard/resources/: More debug logging for Firefox's 'NS_ERROR_NOT_AVAILABLE' exceptions (T136831) (duration: 00m 47s)
23:33 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: Fix ProofreadPage::updatePrIndex signature (T143817) (duration: 00m 50s)
22:59 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings-labs.php: labs-only change: https://gerrit.wikimedia.org/r/298919 (duration: 00m 46s)
22:58 logmsgbot: krenair@tin Synchronized wmf-config/LabsServices.php: labs-only change: https://gerrit.wikimedia.org/r/305837 (duration: 00m 48s)
22:49 mutante: chromium - revoking and re-signing puppet certs, salt keys, initial puppet run..
22:43 yuvipanda: force a puppet run on californium
22:27 bd808: Deployed striker (2fdf103) to californium
22:20 mutante: rebooting chromium into PXE
22:15 papaul: installing puppetmaster200[1-2]
22:05 mutante: stopping puppet and pdns-recursor on chromium
21:57 logmsgbot: krinkle@tin Synchronized multiversion/MWMultiVersion.php: Ie9c568a87ae - No-op clean-up (duration: 00m 49s)
21:54 yuvipanda: forcing puppet run on californium
21:50 mutante: running puppet on lvs servers, removing chromium from resolv.conf for reinstall
21:45 yuvipanda: re-enabled puppet, ran puppet and verified ORES is ok on scb1001 / 1002
21:40 yuvipanda: re-enabled puppet, ran puppet and verified ORES is ok on scb2001 / 2002
21:27 logmsgbot: dzahn@palladium conftool action : set/pooled=no; selector: name=chromium.wikimedia.org
21:27 mutante: depooling chromium for reinstall. scheduled downtime for host and service IPs
21:25 yuvipanda: disable puppet on scb2001 and 2002 as well
21:24 yuvipanda: disable puppet on scb1001 and 1002
21:20 yuvipanda: forcing puppet run on tin
21:20 yuvipanda: forcing puppet run on scb1001
21:14 bd808: Forced puppet run on logstash1003 but cron had beaten me to it
21:12 bd808: Forcing puppet run on logstash1002
21:08 hoo: Ran DELETE FROM wbs_propertypairs WHERE pid1 = '641' on Wikidata for T132839
21:07 bd808: Forcing puppet run on logstash1001
20:59 urandom: T137474: Upgrading xenon.eqiad.wmnet to cassandra_2.2.6-wmf2
20:57 bd808: Created initial tables for striker in striker db on m5-master
20:53 yuvipanda: gave bd808 password for striker db on terbium
20:39 yurik: deployed kartotherian https://gerrit.wikimedia.org/r/#/c/306518/
20:31 yurik: deployed tilerator https://gerrit.wikimedia.org/r/#/c/306514/
20:03 urandom: T137474 Starting htmldumper in RESTBase Staging
20:02 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.16/extensions/ProofreadPage/ProofreadPage.body.php: Fix hooks signatures T143817 (duration: 00m 47s)
19:47 mobrovac: change-prop deploying 28a0057
19:00 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.16
18:44 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-production.php: SWAT: Cirrus: Send more like this queries to default cluster (duration: 00m 46s)
18:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Record content security policy events in log stash (duration: 00m 52s)
18:23 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Change ORES threshold to soft as default (T143738) (duration: 01m 01s)
18:17 paravoid: reenabling cr2-codfw:xe-5/0/1 (link to cr2-eqiad), recovered since 17:02 UTC
18:17 ebernhardson: restarted kibana on logstash1001
17:44 jynus: overwritting /etc/default/prometheus-mysqld-exporter on all trusty mysql nodes on codfw
17:27 gehel: deleting logstash indices from before august 1st (T142357)
17:04 logmsgbot: demon@tin Synchronized wmf-config/: Remove old obsolete ExtensionMessages files (duration: 00m 49s)
16:50 godog: bounce uwsgi on labmon1001 T143556
16:15 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe'])
16:15 logmsgbot: ema@palladium conftool action : set/pooled=no; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx'])
15:58 papaul: !log wdqs200[1-2] - signing puppet certs, salt-key, initial run
15:41 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=varnish-fe'])
15:41 logmsgbot: ema@palladium conftool action : set/pooled=yes; selector: cp4005.ulsfo.wmnet (tags: ['dc=ulsfo', 'cluster=cache_upload', 'service=nginx'])
15:26 _joe_: attempting to restart apache on rhodium, swapping, load exploding
15:22 papaul: installing wdqs200[1-2]
14:41 logmsgbot: demon@tin Synchronized multiversion/getMWVersion.php: fixup error message (duration: 00m 47s)
13:53 gehel: rolling restart of logstash nodes to validate fix to T142357
13:50 hashar: European SWAT done
13:49 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: Fully restrict uploads on ms.wikipedia T141227 (duration: 00m 46s)
13:48 logmsgbot: zfilipin@tin Synchronized dblists/commonsuploads.dblist: Fully restrict uploads on ms.wikipedia T141227 (duration: 00m 46s)
13:39 ema: upgrading cp4005 to Varnish 4 (T131502)
13:27 logmsgbot: zfilipin@tin Synchronized wmf-config/InitialiseSettings.php: Make default ORES threshold soft (higher threshold) T143738 (duration: 00m 59s)
12:56 mobrovac: changeprop deploying c5fd932
12:29 gehel: remove old logstash indices that were not deleted after 31 days
11:10 godog: enable collection of mysqld metrics from prometheus2002 too
10:49 moritzm: start installing hhvm updates in eqiad
10:13 gehel: increase recovery throttling on elasticsearch codfw to reduce rolling restart time
09:53 moritzm: disabled creation of unprivileged user namespaces on trusty systems via sysctl kernel.unprivileged_userns_clone
09:30 gehel: starting rolling restart of elasticearch codfw for JVM and elasticsearch upgrade
09:03 hashar: gallium contint::firewall: Limited to production networks https://gerrit.wikimedia.org/r/301627 . For monitoring do: grep iptables-dropped /var/log/syslog
08:39 ema: dns: re-enabling codfw
08:29 paravoid: disabling cr2-codfw:xe-5/0/1 (link to cr2-eqiad), flapping since 07:31 UTC
08:09 ema: disabling codfw in dns
03:07 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 24 03:07:38 UTC 2016 (duration 6m 55s)
03:00 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.16) (duration: 18m 24s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 16s)
01:57 mutante: temp stop dhcp service on install2001 - debug

2016-08-23

23:45 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Related Articles on fr.wikinews (T143480) (duration: 00m 53s)
23:28 mutante: restarted grrrit-wm, apache and gerrit on lead
23:23 mutante: gerrit restarting for config changes 303355, 304977
22:04 bd808: Updated tin:/srv/deployment/logstash/plugins to d18b1c6 (Add output plugin for Sentry)
21:27 SMalyshev: redeploying WDQS GUI to fix examples breakage
21:27 logmsgbot: dzahn@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=dns,name=hydrogen.wikimedia.org
21:05 mutante: hydrogen - reinstall finished, re-added to salt, restarted ntpd
20:42 mutante: hydrogen - signing new puppet cert
20:22 mutante: hydrogen - reinstalling one more time, wrong partitioning
19:55 mutante: re-signing new puppet certs for hydrogen, initial run, new salt key
19:34 hashar: 1.28.0-wmf.16 to group0 looks successful.
19:19 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: fix compile & sync of wikiversions
19:17 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 50s)
19:13 logmsgbot: hashar@tin Synchronized wikiversions.json: Group0 to 1.28.0-wmf.16 T141551 (duration: 00m 48s)
19:04 mutante: rebooting hydrogen
19:04 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix UI l10n for Help page link on commons.wikimedia.org (T143564) (duration: 00m 47s)
18:56 logmsgbot: bblack@palladium conftool action : set/pooled=no; selector: dc=eqiad,cluster=dns,name=hydrogen.wikimedia.org
18:26 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.15/resources/src/mediawiki.widgets/mw.widgets.CategoryCapsuleItemWidget.js: SWAT: Debug logging for "queue[title] undefined" (T139130) (duration: 00m 50s)
18:24 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.16/resources/src/mediawiki.widgets/mw.widgets.CategoryCapsuleItemWidget.js: SWAT: Debug logging for "queue[title] undefined" (T139130) (duration: 00m 50s)
18:22 mutante: all mw appservers are installing all the font packages now, not just imagescalers. this should fix some issues with EasyTimeline on zh projects and more
18:13 kaldari: mwscript maintenance/updateCollation.php --wiki=svwiki --force
18:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Switching Swedish Wikipedia to uca-sv-u-kn collation (T142113) (duration: 00m 58s)
17:35 mobrovac: changeprop deploying 519ad9d
16:27 jynus: rebooting es2004 for hardware maintenance T143220
14:25 moritzm: installing libgd security updates
14:13 moritzm: rebooting conf1002 for kernel update
14:10 gehel: cleanup list of banned node from elasticsearch eqiad cluster
14:05 logmsgbot: hashar@tin Finished scap: testwiki to php-1.28.0-wmf.16 T141551 (duration: 28m 45s)
13:48 moritzm: rebooting conf1001 for kernel update
13:37 Dereckson: Run namespaceDupes maintenance script on azwiki and azwiktionary (T143580)
13:37 godog: remove some old compilations from puppet compiler filling up compiler02.eqiad.wmflabs disks T143671
13:36 logmsgbot: hashar@tin Started scap: testwiki to php-1.28.0-wmf.16 T141551
13:34 hashar: Restarting Jenkins
13:28 hashar: European SWAT deploy is complete
13:27 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.15/maintenance/namespaceDupes.php: Run LinksDeletionUpdate after commit() in namespaceDupes.php T143631 (duration: 00m 52s)
13:25 moritzm: rebooting conf1003 for kernel update
13:23 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: Remove no longer relevant $wgTranslateTasks overrides (duration: 00m 48s)
13:23 _joe_: restarted pybal on lvs1011 for testing with etcd reboots
13:21 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: Remove English for all groups from $wgTranslateBlacklist T124013 (duration: 00m 54s)
13:16 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Enable T143073 debug log channel (duration: 02m 19s)
12:45 mobrovac: citoid deploying f711219
12:43 hashar: on tin dropping stall versions /srv/mediawiki-staging/php-1.28.0-wmf.{8,9,10} T141551
11:22 hashar: Cutting MediaWiki branch 1.28.0-wmf.16 | T141551
10:07 moritzm: upgrading hhvm on codfw mediawiki servers
09:00 Krenair: updated wikitech-static to MW 1.27.1
08:52 hashar: Jenkins had some deadlocks preventing builds from processing. Resolved by disabling/reenabling the Gearman client
07:29 moritzm: installing botan security updates on trusty systems
05:52 mutante: install2001 - "MD RAID" and "MegaRAID" icinga checks and both fail? new/test? install1001 doesn't have these checks - disabled notifications
03:07 ejegg: updated payments-wiki from 2b027e313ccecc2b93f214a94738b4f94899f347 to a472ff0a8c62f697221d71647f4013e5c2dfcd45
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 23 02:28:07 UTC 2016 (duration 5m 44s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 16s)

2016-08-22

23:58 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.15/extensions/AbuseFilter/Views/: Let abusefilter-modify users see history of hidden filters (gerrit:305596+gerrit:306077, T143365) (duration: 00m 50s)
23:22 Dereckson: NamespaceDupes maintenance script run on sk.wikipedia
23:19 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Fix user namespaces on Slovak Wikipedia (T143472) (duration: 00m 58s)
22:43 Dereckson: Run maintenance ssript namespaceDupes.php on azwiktionary (T143580)
21:19 urandom: Restarting restbase staging in codfw
20:29 adamw_: update CRM from a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e to 43cca607ed0d6d2f91e3f18d9df1473021b40f88
20:13 subbu: finished deploying parsoid sha df53a991
20:08 papaul: wezen - signing puppet certs, salt-key, initial run
20:08 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:05 subbu: starting parsoid deploy
18:47 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1002.eqiad.wmnet
18:45 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1002.eqiad.wmnet
18:44 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet
18:39 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet
18:39 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka2002.codfw.wmnet
18:36 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka2002.codfw.wmnet
18:35 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka2001.codfw.wmnet
18:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka2001.codfw.wmnet
18:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka2001.eqiad.wmnet
18:30 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2009-a.codfw.wmnet
18:29 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2008-a.codfw.wmnet
18:29 ottomata: deploying eventlogging eventbus and a topic config patch, will depool each node as i do and check that all is well
18:26 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1015-a.eqiad.wmnet
18:26 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1014-a.eqiad.wmnet
18:25 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1009-a.eqiad.wmnet
18:24 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1012-a.eqiad.wmnet
18:24 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1011-a.eqiad.wmnet
18:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES review tool for English Wikipedia (T140003) (duration: 01m 04s)
18:23 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1010-a.eqiad.wmnet
18:14 urandom: Restarting Cassandra instances to apply new TLS cert restbase-test2003.codfw.wmnet
18:13 urandom: Restarting Cassandra instances to apply new TLS cert restbase-test2002.codfw.wmnet
18:04 urandom: Restarting Cassandra instances to apply new TLS cert restbase-test2001.codfw.wmnet (for reals this time)
17:48 godog: cassandra: replace certs for restbase-test200[123]-[ab] - T120662
17:35 urandom: Restart Cassandra instances to apply updated certificates, restbase-test2001.codfw.wmnet
17:09 gehel: deplyoing latest GUI + updater version on wdqs100? servers
16:57 urandom_: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1013-a.eqiad.wmnet
16:21 papaul: installing wezen new syslog server
14:46 hashar: European swat completed 8/8 100%
14:46 bblack: text caches: geoip testing looks good, re-enabling+running puppet for the rest
14:46 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Set Flow as default for User talk on kabwiki - T140588 (duration: 00m 59s)
14:36 bblack: text cache puppets disabled, cp1065 testing merged https://gerrit.wikimedia.org/r/253619
13:44 jynus: stopping db1073 to clone compressed Innodb data to db2034
13:43 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.15/extensions/ProofreadPage/ProofreadPage.body.php: Fix unknown constant AS_HOOK_ERROR issue in ProofreadPage (T143471) (duration: 00m 48s)
13:42 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.15/extensions/UniversalLanguageSelector/resources/js/ext.uls.compactlinks.js: Apply toLowerCase when reading featured articles (T143527) (duration: 00m 50s)
13:41 logmsgbot: dereckson@tin scap aborted: php-1.28.0-wmf.15/extensions/UniversalLanguageSelector / resources/js/ext.uls.compactlinks.js Apply toLowerCase when reading featured articles (T143527) (duration: 00m 04s)
13:41 logmsgbot: dereckson@tin Started scap: php-1.28.0-wmf.15/extensions/UniversalLanguageSelector / resources/js/ext.uls.compactlinks.js Apply toLowerCase when reading featured articles (T143527)
13:40 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: Add Collection render note for articles rdf2latex -t - T135613 (duration: 00m 48s)
13:39 logmsgbot: hashar@tin Synchronized wmf-config/CommonSettings.php: Add Collection render note for articles & rdf2latex -t - T135613 (duration: 00m 49s)
13:35 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Enable WikidataPageBanner on ro.wikivoyage - T142963 (duration: 00m 49s)
13:25 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Restrict local upload on ar.wikipedia - T142450 (duration: 00m 49s)
13:16 logmsgbot: hashar@tin Synchronized wmf-config/throttle.php: [cleanup] Remove old throttle rules (duration: 00m 48s)
13:09 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Enable transwiki upload for tcywiki T143397 (duration: 00m 58s)
12:10 mobrovac: restbase deploy end of e9e5ff1
11:57 mobrovac: restbase deploy start of e9e5ff1
10:17 gehel: starting rolling restart of elasticearch eqiad for JVM and elasticsearch upgrade
09:12 jynus: stopping db2034 for cloning and reimage
08:43 _joe_: restarting hhvm on mw1278, deadlock in HPHP::Treadmill::getAgeOldestRequest
08:41 moritzm: restarted hhvm on mw1162, was deadlocked
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 22 02:29:09 UTC 2016 (duration 5m 45s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 26s)

2016-08-21

15:01 Dereckson: Run deleteEqualMessages maintenance script on urwiki (T45917)
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 21 02:27:39 UTC 2016 (duration 5m 41s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 10m 05s)

2016-08-20

13:34 bblack--: rolling, depooled restarts for varnish-frontends: all text, upload in esams, eqiad, codfw.
02:35 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 20 02:35:10 UTC 2016 (duration 5m 58s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 09m 46s)

2016-08-19

22:58 mutante: works fine, no more "bastiononly" group - all users get automatically added on bastions
22:55 mutante: temp. disabled puppet on bastion hosts, confirming change 301149 works as expected
20:03 mutante: removed 2fa for user Florianschmidtwelzow on labswiki, confirmed via file on tools-login and IRC identity
18:00 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2006-a.codfw.wmnet
17:52 logmsgbot: dereckson@tin Finished scap: (no message) (duration: 26m 44s)
17:26 Dereckson: Current scap is for Gerrit:305662 (T143402)
17:26 logmsgbot: dereckson@tin Started scap: (no message)
16:20 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2004-a.codfw.wmnet
16:19 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2007-a.codfw.wmnet
16:11 bblack: cache_maps: rolling frontend cache restarts, ~5 minute window
13:45 bblack: cache_misc: varnish-frontend global rolling restart (~3 mins to completion)
08:56 jynus: deploying schema change on s2 hosts T139090
07:51 moritzm: depooling mw2215 for some tests with the hhvm systemd unit
07:30 moritzm: installing gnupg security updates
06:32 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.15/includes/OutputPage.php: T143357 (duration: 00m 55s)
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 19 02:26:42 UTC 2016 (duration 6m 6s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 08m 59s)
02:08 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change (duration: 00m 54s)
01:31 logmsgbot: mattflaschen@tin Synchronized wmf-config/InitialiseSettings-labs.php: Beta-only change (duration: 00m 51s)
00:31 awight: rollback civicrm to a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e

2016-08-18

23:58 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:305558 Set timezone to Europe/Ljubljana on sl. projects (duration: 00m 49s)
23:52 logmsgbot: tgr@tin Synchronized wmf-config/abusefilter.php: SWAT gerrit:293109 Remove AbuseFilter B/C config (duration: 00m 49s)
23:46 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:303939 Remove $wgDisableAuthManager (duration: 00m 49s)
23:45 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: SWAT gerrit:303939 Remove $wgDisableAuthManager (duration: 00m 48s)
23:43 logmsgbot: tgr@tin Synchronized wmf-config/wikitech.php: SWAT gerrit:303939 Remove $wgDisableAuthManager (duration: 00m 49s)
23:42 kaldari: mwscript sql.php --wiki=enwikivoyage /srv/mediawiki/php/extensions/PageAssessments/db/addReviewsTable.sql
23:41 kaldari: mwscript sql.php --wiki=enwikivoyage /srv/mediawiki/php/extensions/PageAssessments/db/addProjectsTable.sql
23:36 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:275299 Disable wgUseFilePatrol in huwiki (duration: 01m 00s)
22:44 mutante: phab2001 - puppet re-enabled (but phd service stopped, after gerrit 305591)
22:16 logmsgbot: krenair@tin Synchronized wmf-config/LabsServices.php: for labs, no-op in prod: https://gerrit.wikimedia.org/r/#/c/305589/ (duration: 00m 56s)
21:23 Dereckson: Run deleteEqualMessages maintenance script on eswikibooks, eswikiquote, eswikisource, eswiktionary and eswikiversity (T45917)
21:19 awight: update civicrm from a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e to da572ab911763612a4b7005056821918ac630cbe
20:22 mutante: phab2001 disabled phd service again
20:04 Dereckson: Run initSiteStats maintenance script on mhrwiki and newwiki (T143352)
19:45 mutante: phab2001 - reenabled puppet temp
19:10 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.15
18:49 Guest64769: phab2001 manually removed phabricator crons since puppet is disabled there
18:48 awight: rollback civicrm from 41844071e3018b21f2c36ab692c4b43b92cab8d0 to a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e
18:45 awight: update civicrm from a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e to 41844071e3018b21f2c36ab692c4b43b92cab8d0
17:24 logmsgbot: gehel@tin Synchronized wmf-config/InitialiseSettings.php: switching search back to eqiad (duration: 00m 49s)
17:19 gehel: elasticsearch eqiad recovery throttling back to standard 20mb recovery
17:15 godog: reinstall ms-be1027 after ssd replaced T140374
17:12 chasemp: nodepool restart with new settings for rate/env/ready for jessie from puppet
16:56 mobrovac: mathoid deploying 75606c71
16:16 chasemp: restart nodepool with STATSD_HOST env variable for test
15:52 gehel: reset postgresql replication on maps1003 (correction, not maps1001)
15:51 gehel: reset postgresql replication on maps1001
15:21 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Bump cache epoch for Wikidata (duration: 00m 49s)
15:12 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable lazy loaded images on mobile web (duration: 01m 00s)
14:54 cmjohnson1: ms-be1005 replacing disk Slot Number: 3
14:41 paravoid: cr1-eqiad and cr2-eqiad: replace obsolete PIM BFD statements with new family inet/inet6 ones
14:38 paravoid: cr1-eqiad: JunOS, SCB and linecard upgrade is over
14:34 paravoid: cr1-eqiad: activating PyBal/LVS BGP sessions
14:32 paravoid: cr1-eqiad: activating Transit4/6 BGP sessions
14:31 paravoid: cr1-eqiad: activating Private-Peer4/6 BGP sessions
14:31 paravoid: cr1-eqiad: reenabling Private-Peer/Transit interfaces
14:29 mark: cr2-eqiad: setting ae3.1019 inet vrrp priority to default (from 50)
14:27 paravoid: cr1-eqiad: removing deprioritization of all VRRP groups (priority=50)
14:24 paravoid: cr1-eqiad: reenabling xe-5/0/3 (link to pfw) and the Fundraising BGP group
14:24 paravoid: cr1-eqiad: reenabling xe-4/2/0 (link to cr1-codfw) and xe-4/2/2 (link to cr2-knams)
14:20 paravoid: cr1-eqiad: reenabling all asw row D interfaces
14:19 paravoid: cr1-eqiad: reenabling all asw row B/C interfaces
14:18 gehel: increasing elasticsearch recovery throttling to 40mb to speed up eqiad recovery
14:18 paravoid: cr1-eqiad: reenabling all asw row A interfaces
14:12 paravoid: cr1-eqiad: activate chassis redundancy graceful-switchover
13:57 paravoid: cr1-eqiad: cmjohnson1 replacing both SCBs with SCBE2s and adding a new linecard
13:54 paravoid: cr1-eqiad: shutting down both routing-engines and powering off
13:38 paravoid: cr1-eqiad: setting "chassis network-services enhanced-ip" and rebooting both REs
13:20 paravoid: cr1-eqiad: upgrading re1 and rebooting
13:18 paravoid: cr1-eqiad: toggling mastership between routing-engines (re1->re0)
13:00 jynus: reloaded dbproxy1010's haproxy service to point to the original master
12:58 paravoid: cr1-eqiad: upgrading re0 and rebooting
12:56 paravoid: cr2-eqiad: remove VRRPv3 backwards compatibility (delete protocols vrrp checksum-without-pseudoheader)
12:53 paravoid: cr1-eqiad: deactivate chassis redundancy graceful-switchover
12:52 paravoid: cr1-eqiad: disabling all asw row A-C interfaces
12:44 paravoid: cr1-eqiad: disabling all asw row D interfaces
12:41 logmsgbot: gehel@tin Synchronized wmf-config/InitialiseSettings.php: switching search to codfw (duration: 00m 56s)
12:40 gehel: switching search traffic to codfw
12:13 paravoid: cr1-eqiad: reenabling all asw row A/B/C interfaces
12:05 paravoid: cr1-eqiad: disabling all asw row B/C interfaces
12:04 paravoid: cr1-eqiad: disabling all asw row A interfaces
12:01 paravoid: cr1-eqiad: disabling Transit/Fundraising interfaces
11:56 paravoid: cr1-eqiad: deactivating Fundraising BGP session
11:54 paravoid: cr1-eqiad: deactivating PyBal/LVS backup static routes
11:47 paravoid: cr1-eqiad: deactivating Transit4/6 and Private-Peer4/6 BGP sessions
11:31 moritzm: uploaded new Linux package for jessie-wikimedia to carbon (now based on 4.4.18)
11:06 paravoid: cr1-eqiad: deactivating BGP sessions with PyBal/LVS
11:04 paravoid: cr1-eqiad: disabling xe-4/2/0 (link to cr1-codfw) and xe-4/2/2 (link to cr2-knams)
10:51 paravoid: cr1-eqiad: deprioritizing all groups (priority=50)
10:20 moritzm: upgrading remaining canary application servers to hhvm 3.12.7
08:59 moritzm: enabled scaling of huge SVGs on image scalers (T111815)
08:58 logmsgbot: jmm@tin Synchronized wmf-config/CommonSettings.php: enable SVG scaling of huge files (duration: 00m 48s)
08:10 jynus: stoping mysql on db1042, db2009 for testing (both are depooled and alerts disabled)
03:06 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 18 03:06:44 UTC 2016 (duration 7m 5s)
02:59 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 08m 44s)
02:34 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 58s)
00:27 logmsgbot: maxsem@tin Finished scap: https://gerrit.wikimedia.org/r/#/c/305424/ (duration: 52m 22s)

2016-08-17

23:53 XenoRyet: updated payments-wiki from b449f65dba5697905a87261592e934d7f4898a54 to 2b027e313ccecc2b93f214a94738b4f94899f347
23:35 logmsgbot: maxsem@tin Started scap: https://gerrit.wikimedia.org/r/#/c/305424/
23:27 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/304328/ (duration: 00m 55s)
23:12 logmsgbot: maxsem@tin Synchronized wmf-config/CirrusSearch-production.php: https://gerrit.wikimedia.org/r/#/c/304204/3 (duration: 00m 53s)
22:16 chasemp: restart nodepool
22:15 chasemp: openstack quota set --instances 15 contintcloud
22:03 chasemp: restart nodepool with 10s cycle rate
21:59 chasemp: openstack quota set --instances 14 contintcloud
21:59 chasemp: disable puppet on labnodepool for testing of instance threshold
21:50 chasemp: set contintcloud project instance quota to 12 for testing
21:47 cscott: updated OCG to version e3e0fd015ad8fdbf9da1838c830fe4b075c59a29 (T133001, T142226)
21:46 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.15/extensions/CentralAuth: ef4b5d45f9bb59c978e23f21ed09649aa628c4d1 (duration: 00m 59s)
21:43 cscott: starting OCG deploy
20:43 bearND: deployed mobileapps 81bd74f
20:26 urandom: Restarting Cassandra, aqs1004-a.eqiad.wmnet
20:25 bearND: starting mobileapps deploy
20:09 subbu: finished deploying parsoid sha 3cf877bb
20:04 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:01 subbu: starting parsoid deploy
19:23 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.15 refs T140971
18:41 ejegg: disabled donations queue consumer, adyen job runner, pending queue consumer for db rename
18:40 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2005-a.codfw.wmnet
18:37 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2003-a.codfw.wmnet
18:31 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase2002-a.codfw.wmnet
18:17 urandom: T143226: Perform major compaction on local_group_wikipedia_T_parsoid_html.data, restbase1008-a.eqiad.wmnet
17:31 ottomata: deploying Kafka main-eqiad -> main-codfw 'eqiad.*' topic mirroing
16:02 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.14/extensions/ContentTranslation: SWAT: Avoid deadlock patterns in cx_corpora updates (T134245) (duration: 00m 50s)
15:59 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.15/extensions/ContentTranslation: SWAT: Avoid deadlock patterns in cx_corpora updates (T134245) (duration: 00m 52s)
15:40 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: wmgEchoMentionStatusNotifications true for test/test2wiki (T141995) (duration: 00m 50s)
15:29 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.15/includes/api/ApiUpload.php: SWAT: Do not call the "UploadStashFile" hook for partially uploaded files (T143161) PART II (duration: 00m 50s)
15:27 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.15/includes/upload/UploadBase.php: SWAT: Do not call the "UploadStashFile" hook for partially uploaded files (T143161) PART I (duration: 00m 53s)
15:15 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor by default for logged-out users on Arabic-script Wikipedias (T142587) (duration: 00m 50s)
14:50 jynus: deploying schema change on s6 hosts T139090
14:47 gehel: rolling restart of elasticsearch logstash cluster for elasticsearch upgrade to 2.3.4
14:45 godog: bounce carbon on graphite machines
14:42 gehel: rolling restart of elasticsearch relforge cluster for elasticsearch upgrade to 2.3.4
14:29 gehel: upgrading elasticsearch plugins to 2.3.4 on elasticsearch, relforge and logstash clusters. Rolling restart coming next.
14:26 gehel: upgrading elasticsearch to 2.3.4 on relforge cluster
13:28 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable allowDataAccessInUserLanguage on meta (T122672) (duration: 00m 56s)
13:10 gehel: rolling restart of relforge100* for JVM upgrade. Short downtime expected.
12:57 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-swe-nor_0.2.0~r69544-1+wmf1
12:57 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-swe-dan_0.7.0~r66063-1+wmf1
12:57 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-dan-nor_1.3.0~r67099-2+wmf1
12:18 godog: restart pybal on low-traffic for thumbor - T139606
12:10 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hin_0.1.0~r59158-1+wmf1
12:03 moritzm: repooling mw1298 with config change to allow scaling of huge SVGs (to testdrive further before enabling this in general)
11:56 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-nob_0.9.0~r69513-1+wmf1
11:55 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-dan_0.5.0~r67099-2+wmf1
10:07 Amir1: ladsgroup@scb[12]00[12]:~$ sudo service celery-ores-worker restart (T143105)
09:45 godog: upload scap3 3.2.3-1 to carbon T127762
09:37 moritzm: upgraded mw1017 to HHVM 3.12.7 (plus patches)
09:31 moritzm: uploaded hhvm 3.12.7+dfsg+wmf1~trusty1 for trusty-wikimedia to carbon (also includes a fix for T137642)
08:00 moritzm: installing openjdk security updates on the elastic* clusters
07:36 gehel: cleanup and shutdown of nobelium before reclaim
06:34 moritzm: installing openjdk security updates on the stat* hosts
03:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 17 03:15:05 UTC 2016 (duration 7m 22s)
03:07 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.15) (duration: 17m 59s)
02:33 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 31s)

2016-08-16

23:57 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.15/includes/OutputPage.php: 653232f90605 (duration: 00m 52s)
23:56 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.15/includes/resourceloader/ResourceLoaderClientHtml.php: 653232f90605 (duration: 00m 48s)
23:55 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.15/includes/skins/SkinTemplate.php: 05c82731c9831c465 (duration: 00m 49s)
23:49 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/304632/2 (duration: 00m 54s)
23:43 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.14/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/305080/ (duration: 00m 51s)
23:39 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.15/extensions/BetaFeatures: deploy https://gerrit.wikimedia.org/r/#/c/305148/ (duration: 00m 49s)
23:32 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: touch (duration: 00m 52s)
23:25 logmsgbot: maxsem@tin Synchronized wmf-config: https://gerrit.wikimedia.org/r/#/c/304631/2 (duration: 00m 53s)
23:13 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.15/extensions/Kartographer: https://gerrit.wikimedia.org/r/#/c/305080/ (duration: 00m 51s)
23:11 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.15/extensions/Graph: https://gerrit.wikimedia.org/r/#/c/305145/ (duration: 00m 59s)
22:34 ejegg: restarted donation queue consumer
22:26 ejegg: paused queue consumer to set up pending db comparison testing
22:02 ejegg: updated Civicrm from 1d24161dd3f0f3ad64c0bf77f06022f30a2b3f2f to a30fc0aa2c2dc3a7a9a0b09bef19d112bdf5f98e
21:37 mutante: restarted grrrit-wm for config change 304746
20:48 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 wikis to 1.28.0-wmf.15
20:40 logmsgbot: twentyafterfour@tin Finished scap: sync testwiki to 1.28.0-wmf.15 refs T140971 (duration: 52m 45s)
19:47 logmsgbot: twentyafterfour@tin Started scap: sync testwiki to 1.28.0-wmf.15 refs T140971
18:02 gehel: starting tile generation for zoom levels 0-10 on maps eqiad
17:46 moritzm: uploaded hhvm 3.12.7+dfsg+wmf1 for jessie-wikimedia to carbon (also includes a fix for T137642)
16:14 bblack: rolling depooled restarts of varnish-frontend on ulsfo upload caches
16:09 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.14/extensions/Flow/maintenance/FlowRestoreLQT.php: SWAT: Query wiki DB for logging table, not Flow DB (T119509) (duration: 00m 57s)
16:09 logmsgbot: bblack@palladium conftool action : set/pooled=yes; selector: name=cp4006.ulsfo.wmnet
16:09 bblack: repooling cp4006
15:58 bblack: rebooting cp4006
15:48 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bumping portals to master (T140153) (duration: 00m 47s)
15:47 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T140153) (duration: 00m 51s)
15:34 ema: depooling cp4006
15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES review tool in plwiki (T140005) (duration: 00m 55s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Change login cookies (for "Keep me logged in") to a one year expiry. (T68699) (duration: 01m 08s)
12:53 hoo: Put a better workaround for T132839 in place: Only remove property pairs with context = "item". This keeps ref and qualifier pairs for ext ids intact.
12:13 ema: re-enabling puppet on cache hosts (T138546)
10:52 ema: rolling restart of v4 varnishes (T142810)
06:35 moritzm: restarted hhvm on jobrunner mw1162 (deadlocked)
04:02 mutante: restarted grrrit-wm
03:58 mutante: gerrit restarting to apply config change 302229
03:42 mutante: restarted grrrit-wm
03:21 mutante: gerrit restarting to apply config change 304838
03:21 mutante: lead - restarted apache
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 16 02:31:10 UTC 2016 (duration 5m 42s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 31s)
00:21 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: remove utc timezone overrides (duration: 00m 48s)
00:17 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set timezone to Europe/Ljubljana on sl. projects (T142701) (duration: 00m 49s)
00:06 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add autopatrolled and rollbacker user groups to it.wikinews (T142571) (duration: 00m 52s)

2016-08-15

23:51 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Fix VE namespaces in he, fa and ko Wikipedias (T118060). Remove redundant svwiktionary wmgVisualEditorAvailableNamespaces entry (Gerrit:304936). (duration: 00m 52s)
23:15 logmsgbot: dereckson@tin Synchronized wmf-config/interwiki.php: Update interwiki map (Gerrit:304622) (duration: 00m 49s)
23:12 eileen: upgrading CiviCRM from cad2c59081e53b877f51be4c853f307f3c4d4949 to 1d24161dd3f0f3ad64c0bf77f06022f30a2b3f2f
23:12 kaldari: mwscript maintenance/updateCollation.php --wiki=mkwiki --force
23:08 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set collation to uca-mk-u-kn on mk.wikipedia (T26953) (duration: 01m 00s)
21:35 awight: update Thank-you letter to point to the new unsubscribe server
21:17 awight: update paymentswiki from 6b1f3c278308f831a149c986d2269342f94e2a16 to b449f65dba5697905a87261592e934d7f4898a54; update config for FundraisingEmailUnsubscribe
20:17 Krenair: labtestcontrol2001: Raise max_connections mysql global to 500 to match real-labs' on m5-master (db1009). old value was 151, see my comment at T132422
20:10 subbu: finished deploying parsoid sha f039dcf6
20:05 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:03 subbu: starting parsoid deploy
19:39 urandom: T140008: Starting major compaction (WP parsoid html, split output) on restbase1007-a.eqiad.wmnet
19:27 urandom: T140008: Staring user-defined compaction (10 tables, highest droppable tombstones), restbase2001-b.codfw.wmnet
18:56 awight: roll back paymentswiki from 75e4d2509a53910e5630ca3dca5950474b587a17 to 6b1f3c278308f831a149c986d2269342f94e2a16
18:52 awight: update paymentswiki from 6b1f3c278308f831a149c986d2269342f94e2a16 to 75e4d2509a53910e5630ca3dca5950474b587a17
18:13 ottomata: starting main-eqiad Kafka upgrade to confluent 0.9, will be stopping and starting brokers on kafka1001 and kafka1002
17:03 gehel: deploying latest wdqs on wdqs100[12].eqiad.wmnet
16:16 mobrovac: restbase restarting in eqiad after cassandra restarts for the openjdk upgrade
16:07 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1015.eqiad.wmnet
16:00 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1014.eqiad.wmnet
15:52 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1013.eqiad.wmnet
15:45 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1012.eqiad.wmnet
15:38 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1011.eqiad.wmnet
15:30 urandom: Restarting Cassandra to apply OpenJDK 8u102 upgrade, restbase1010.eqiad.wmnet
15:03 ostriches: gerrit: upgrading 2.12.2 -> 2.12.3. Quick restart will happen.
14:24 moritzm: upgrading firejail on mw* to 0.9.40
13:59 moritzm: upgrading mw1293 to firejail 0.9.40
11:08 moritzm: rolling restart of cassandra on restbase1* to pick up openjdk security update
10:54 moritzm: uploaded gerrit 2.12.3 to apt.wikimedia.org
10:49 mobrovac: restbase restring RB in codfw following the jdk upgrade and cassandra restarts
10:06 moritzm: rolling restart of cassandra on restbase2* to pick up openjdk security update
07:29 moritzm: upgrading openjdk on maps clusters (along with cassandra restart)
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 15 02:30:30 UTC 2016 (duration 5m 51s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 21s)

2016-08-14

17:12 paravoid: bumping cr2-knams<->cr1-eqiad OSPF/OSPF3 metric to 1820 (thus activating the new cr2-esams<->cr2-eqiad link which has a metric of 840)
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 14 02:30:34 UTC 2016 (duration 5m 53s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 15s)

2016-08-13

20:59 Krenair: labtestcontrol2001: restarted mysql to unbreak labtesthorizon login again. we really need to figure out why this becomes necessary
11:38 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: wfLoadExtension all around (duration: 00m 49s)
10:57 logmsgbot: reedy@tin Synchronized wmf-config/: Simplify WikimediaMessages loading, Remove Orphaned OpenStreetMapSlippyMap config (duration: 00m 50s)
10:45 logmsgbot: reedy@tin Synchronized wmf-config/: Simplify WikimediaMessages loading, Remove Orphaned OpenStreetMapSlippyMap config (duration: 00m 52s)
10:33 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: Fix erroneous plural (duration: 00m 50s)
09:52 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: Fix path to FancyCaptcha extension.json in extension-list (duration: 00m 47s)
09:46 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: Many more to extension.json in extension-list (duration: 00m 50s)
09:13 elukey: extended the maximum kafka topic partition size (350GiB) to upload as well (it was only text before)
08:57 elukey: added temporary override to Kafka topic settings to free disk space: retention.bytes=375809638400
08:42 elukey: restarting cassandra on aqs100[456] (non live cluster) for jvm upgrades and new settings
08:22 elukey: added temporary override to Kafka topic settings to free disk space: retention.bytes=429496729600
08:14 elukey: added temporary override to Kafka topic settings to free disk space: retention.bytes=483183820800
07:31 awight: Removed Ingenico audit processing post-build action to immediately start a thank-you mailing job.
02:33 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 13 02:33:32 UTC 2016 (duration 6m 7s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 11m 28s)

2016-08-12

20:13 halfak: running FLUSHALL on oresrdb1001 T142857
20:07 Amir1: deploying ores 2ef24f2 to all nodes T142857
20:03 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: T68699: Enable one-year login on Beta Cluster. Scheduled for prod on Tuesday. (duration: 00m 52s)
20:03 Amir1: deploying ores 2ef24f2 to scb2001.codfw.wmnet (canary node) T142857
16:34 chasemp: reboot labstore1005 to test failure mode
15:36 gehel: reset postgresql maps slave for maps1003
15:18 yurik: deployed tilerator to fix krtotherian restarts on maps100*
15:13 moritzm: upgrading openjdk on restbase-test/xenon/praseodymium/cerium) (along with cassandra restart)
14:33 mobrovac: zotero restarted, mem usage was at 11%
14:16 chasemp: reboot labstore1004/1005
13:54 ema: cache_maps varnish backends rolling restart (T142810)
13:21 gehel: initial configuration of maps1002
11:00 elukey: upgrading httpd to 2.4.10-10+deb8u6+wmf2 on mw126[5678]
10:47 jynus: dropping aft tables from all other hosts after archiving its contents T59185
10:06 jynus: dropping aft tables from all enwiki hosts after archiving its contents T59185
09:38 jynus: dropping aft tables from db1052 T59185
09:32 elukey: restarting Druid java daemons on druid100[123] for openjdk upgrades
08:36 akosiaris: roll restart parsoid to apply https://phabricator.wikimedia.org/rOPUPb32dd25950f1499a79e74fc811c050291b9ec6b8
08:25 moritzm: install postgres security updates on maps clusters
02:45 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 12 02:45:40 UTC 2016 (duration 5m 57s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 18m 02s)
02:04 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.14/extensions/Echo: Revert self-mentions pending further investigation and discussion, due to accidental self-mentions. (duration: 01m 04s)
00:51 yurik: deployed kartotherian & tilerator. maps100[134].eqiad are still down (non production)
00:21 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.14/extensions/CentralNotice: CentralNotice deployment gerrit:304420 (duration: 00m 49s)
00:15 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.14/: VE: Fix TextState#getChangeTransaction bug (T141573) ; Echo: Revert "Hack around browser bug in IE breaking badge alignment in Monobook" (gerrit:304415) ; Core: Revert CSS fix (gerrit:304412, T142750) (duration: 08m 58s)

2016-08-11

23:47 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Remove 'gather-hidelist' user right gerrit:303803 (duration: 00m 48s)
23:42 ostriches: gerrit: running puppet to pick up config change, gerrit will do a quick restart
23:37 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: UrlShortener: Whitelist *.wikidata.org (T142055) (duration: 00m 47s)
23:24 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Re-enable thank-you-edit notifications (T128249) (duration: 00m 58s)
23:22 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Re-enable Echo footer notice (T141414) (duration: 00m 54s)
20:25 ejegg: updated SmashPig from 5c180de6e424be10f9d61052c2c8dca7e0e825af to 4ba14a10518f80fe7e20fadc493d78c796c15921
20:08 yuvipanda: restart nova-compute on labvirt1010
20:06 yuvipanda: tools restart rabbitmq-server on labcontrol1001
20:04 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group2 wikis to 1.28.0-wmf.14
19:44 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.14/extensions/CentralNotice: deploy https://gerrit.wikimedia.org/r/#/c/304130/ (duration: 00m 58s)
18:18 MaxSem: restarted hhvm on mw1017 to catch up with hhvm-wikidiff2 upgrade
17:49 ema: switched LVS schedulers for text, upload, maps and misc port 80 to source hash scheduling T108827
17:37 ema: applying https://gerrit.wikimedia.org/r/#/c/297418/ to eqiad load balancers
17:16 ema: applying https://gerrit.wikimedia.org/r/#/c/297418/ to codfw load balancers
17:02 Amir1: restarting celery-ores-worker service in scb[12]00[12] T142361
17:00 ema: applying https://gerrit.wikimedia.org/r/#/c/297418/ to ulsfo load balancers
16:36 ema: applying https://gerrit.wikimedia.org/r/#/c/297418/ to esams load balancers
15:51 ema: disabling puppet on lvs hosts to test https://gerrit.wikimedia.org/r/#/c/297418/
14:20 jynus: disabling puppet on db2001-2009; preparing for decomission
13:53 chasemp: restarting nodepool to test stats collection
13:02 elukey: restarting cassandra on aqs100[123] for jvm upgrades
10:26 elukey: restarting statsv on hafnium (process stuck after kafka brokers restart)
10:25 elukey: kafka restarted on kafka200[12] for jvm upgrades
09:34 jynus: removing leftovers of unmaintained skrillex tool from mira and tin
09:17 elukey: restarting statsv on hafnium (blocked due to analytics kafka brokers restarts)
09:01 elukey: upgrading httpd to 2.4.10-10+deb8u6+wmf2 on mw126[234]
08:50 elukey: uploaded apache2 2.4.10-10+deb8u6+wmf2 to reprepro
08:18 elukey: restarting kafka on the kafka analytics cluster for jvm upgrades
07:38 elukey: increaseing traffic weight to 30 for mw1261 (current: 5)
02:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 11 02:56:27 UTC 2016 (duration 6m 54s)
02:49 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 10m 24s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 11m 12s)
00:41 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.14/maintenance/purgeChangedFiles.php: 91baa668219d8a77b1e300c257e969b04b4b1e3d (duration: 00m 47s)
00:39 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.13/maintenance/purgeChangedFiles.php: 4fe4f803541c4b21c5a118eab426d78d6c0c607b (duration: 00m 50s)
00:10 logmsgbot: reedy@tin Synchronized wikiversions.json: noop for mira (duration: 00m 49s)

2016-08-10

23:59 AaronSchulz: Running purgeChangedFiles.php on all wikis on a terbium screen (T142638)
23:46 logmsgbot: reedy@tin rebuilt wikiversions.php and synchronized wikiversions files: Reinstate .14 as T142638 is fixed
23:41 chasemp: stop nodepool & cleaning out nodepool instances for clean start on project ACL src group removal
23:44 mutante: mw2086 - removing node from cluster failed - backend error, request requires authentication
23:40 logmsgbot: reedy@tin Synchronized php-1.28.0-wmf.14/extensions/VipsScaler: Remove old broken config causing T142638 (duration: 00m 50s)
23:39 yuvipanda: restarted rabbitmq on labcontrol1001
23:22 mutante: connected to mw2086.mgmt (which icinga said was down since a couple hours). i saw it booting up..it came back but i did not powercyle or reboot, just view console
23:17 logmsgbot: reedy@tin rebuilt wikiversions.php and synchronized wikiversions files: Revert to .13 to attempt to fix T142638
22:36 andrewbogott: restarting rabbitmq-server on labvirt1001
21:42 mutante: added tbayer(HaeB) to wmf LDAP group
21:30 eileen: updated civicrm from 024a7ef2badb02743659ffed9856571d96604fd2 to cad2c59081e53b877f51be4c853f307f3c4d4949
20:21 Amir1: deploygin 7aad8e9 for ores in all nodes
20:18 Amir1: deploygin 7aad8e9 for ores in canary (scb2001.codfw.wmnet)
20:17 logmsgbot: mattflaschen@tin Synchronized wmf-config/CommonSettings-labs.php: Re-enable thank-you-edit (milestone notifications for 1st, 10th, 100th, etc. edits) on Beta Cluster (duration: 02m 45s)
20:11 subbu: finished deploying parsoid sha 4de49e26
20:07 subbu: finished syncing code; restarted parsoid on wtp1001 as canary
20:05 subbu: starting parsoid deploy
19:34 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
19:31 twentyafterfour: sync-wikiversions failed for mw2086.codfw.wmnet
19:30 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
17:15 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1002.eqiad.wmnet
17:14 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1002.eqiad.wmnet
17:14 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet
17:12 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet
17:12 ottomata: restarting eventbus in eqiad to apply error_output change
16:13 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on knwiki (T142468) (duration: 02m 55s)
16:08 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on cywiki (T140725) (duration: 02m 49s)
15:50 _joe_: restarting parsoid for T137878
15:41 andrewbogott: rebooting labvirt1014 to 3.16.0-77-generic for testing secgroup issues
15:30 elukey: correct version installed on mw1261 is 2.4.10-10+deb8u6+wmf2
15:22 thcipriani: mw2086 ssh from tin as mwdeploy user failing. Will need to run 'scap pull' when it comes back online
15:20 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Promote new language switcher to all wikis (T129505) (duration: 01m 12s)
15:14 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-nondefault.dblist: SWAT: Enable VisualEditor by default for logged-in users on Arabic-script Wikipedias (T93387) PART II]] (duration: 01m 40s)
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor by default for logged-in users on Arabic-script Wikipedias (T93387) PART I]] (duration: 02m 49s)
14:40 elukey: depooling mw1261 to install/test apache2_2.4.10-10+deb8u6+wmf1_amd64.deb (T73487). After basic checks the host will get back into service with weight 5.
13:54 moritzm: depooling image scaler mw1298 for some local tests with huge SVGs
13:48 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1002.eqiad.wmnet
13:43 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1002.eqiad.wmnet
13:42 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet
13:38 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet
13:38 ottomata: deploying eventbus service to kafka100[12], depooling, deploying, and repooling each one at at time
12:57 moritzm: reboot rdb2001-rdb2004 for updates to Linux 4.4
12:19 akosiaris: T140443 uploaded to apt.wikimedia.org trusty-wikimedia: php-wikidiff2_1.4.1
11:29 akosiaris: T140443 uploaded to apt.wikimedia.org trusty-wikimedia: php-wikidiff2_1.4.0
09:55 moritzm: rebooting dubnium for kernel update to 4.4
09:44 moritzm: rebooting pollux for kernel update to 4.4
09:41 _joe_: uploaded puppetdb deb packages for jessie (T142363)
09:09 moritzm: rebooting hafnium for kernel update to 4.4
03:22 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 10 03:22:29 UTC 2016 (duration 7m 26s)
03:15 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.14) (duration: 18m 02s)
02:40 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 17m 50s)
00:42 logmsgbot: reedy@tin Synchronized php-1.28.0-wmf.14/extensions/timeline/Timeline.body.php: (no message) (duration: 00m 48s)
00:41 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/timeline: (no message) (duration: 00m 47s)
00:33 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/303303/ (duration: 00m 47s)
00:28 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/timeline: https://gerrit.wikimedia.org/r/#/c/303952/ (duration: 00m 50s)
00:25 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.14/extensions/timeline: https://gerrit.wikimedia.org/r/#/c/303950/ (duration: 00m 47s)
00:17 eileen: updated Civicrm from af0048c007ddf9e74c6bd15e50c73c3bd0942492 to 024a7ef2badb02743659ffed9856571d96604fd2
00:11 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/timeline: https://gerrit.wikimedia.org/r/#/c/303947/ (duration: 00m 48s)

2016-08-09

23:44 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/303940/ (duration: 00m 49s)
23:22 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/303936/ (duration: 00m 49s)
23:15 Amir1: ladsgroup@scb2002:~$ sudo service celery-ores-worker restart
23:13 Amir1: ladsgroup@scb2001:~$ sudo service celery-ores-worker restart
23:11 Amir1: ladsgroup@scb1002:~$ sudo service celery-ores-worker restart
23:11 Amir1: sudo service celery-ores-worker restart in scb1001
22:49 paravoid: re-enabling cr2-eqiad:xe-5/2/3 (link to cr2-codfw); vendor reported it as fixed
22:45 ejegg: updated civicrm from d9a765903ac682c0fbd329ced59f5ba3953970c9 to af0048c007ddf9e74c6bd15e50c73c3bd0942492
22:02 ejegg: updated SmashPig from 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 to 5c180de6e424be10f9d61052c2c8dca7e0e825af
21:09 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: Configure 'sourceUrl' for ExtensionDistributor (duration: 00m 50s)
21:02 twentyafterfour: cleaned up stale branch php-1.28.0-wmf.7
20:40 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Fix wgAddGroups/wgRemoveGroups for mediawikiwiki - T142492 (duration: 00m 50s)
20:27 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.13/extensions/FlaggedRevs: 3acc2bd85544072b7fd55abd2829f2bdde7aeef8 (duration: 00m 53s)
20:25 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.14
20:19 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.14/extensions/FlaggedRevs: 52f54661a84f2066bc4c3a13e603ff8c6e5db357 (duration: 00m 55s)
20:17 mutante: root@tin:/srv/mediawiki-staging# find . -uid 0 -exec chown mwdeploy:wikidev {} \;
20:08 bblack: cr[12]-eqiad: added term relforge for instances->relforge100[12] in labs-support
20:05 logmsgbot: twentyafterfour@tin Finished scap: testwiki to 1.28.0-wmf.14 refs T139217 (duration: 49m 45s)
20:02 ejegg: rolled back SmashPig to 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3
20:01 awight: update paymentswiki from 33de2ce5d7b040d733141fc12a74f70a81c09925 to 6b1f3c278308f831a149c986d2269342f94e2a16
19:58 ejegg: enabled fundraising pending queue consumer
19:57 ejegg: updated SmashPig from 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 to 081493d846ad3c1b063f794ca2303576157b27f2
19:40 andrewbogott: restarting rabbitmq-server on labcontrol1001
19:38 awight|eat: update paymentswiki from 4e6a68f9d0b98d9037f81b0c4d312831d7fe66bb to 33de2ce5d7b040d733141fc12a74f70a81c09925
19:15 logmsgbot: twentyafterfour@tin Started scap: testwiki to 1.28.0-wmf.14 refs T139217
19:05 yurik: deployed/restarted kartotherian https://gerrit.wikimedia.org/r/#/c/303841/ and tilerator https://gerrit.wikimedia.org/r/#/c/303840/
18:56 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.deprecate.js: Log RL splitRequest (duration: 02m 08s)
18:54 ejegg: disable pending-new queue consumer
18:21 awight: update paymentswiki from b737b60c87da82543ab812ece4611c68af01307f to 4e6a68f9d0b98d9037f81b0c4d312831d7fe66bb
18:10 ejegg: rolled back SmashPig to 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3
18:05 ejegg: updated SmashPig from 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 to 081493d846ad3c1b063f794ca2303576157b27f2
17:51 ottomata: restarting kafka broker on kafka1013 to test eventlogging leader changes
17:13 subbu: finished deploying parsoid sha a577d80e
17:03 subbu: synced new code; restarted parsoid on wtp1001 as a canary
17:01 subbu: starting parsoid deploy
16:41 mobrovac: restbase deploy end of b800d343
16:21 mobrovac: restbase deploy start of b800d343
15:59 elukey: switching restbase/cassandra user on aqs100[123] to aqs (T142073) - https://gerrit.wikimedia.org/r/303798 will be applied to one node at the time with depool/pool
15:18 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.13/extensions/CirrusSearch/includes/Search/Result.php: SWAT: Use createFragmentTarget instead of setFragment (T142297) (duration: 00m 54s)
14:37 akosiaris: T135176 pool wtp2002
14:36 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2002.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
14:00 paravoid: setting cr2-eqiad:xe-5/2/3 (link to cr2-codfw) to disable; flapping
13:34 akosiaris: all of wtp2001-wtp2020 except wtp2002 have been pooled. T135176
13:33 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
13:07 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-fra-cat_1.1.0~r64309-1+wmf1
12:26 moritzm: depooling image scalers mw2086-mw2089 for reimaging with jessie
12:10 moritzm: rolling restart of hhvm on remaining eqiad mw servers to pick up curl security updates
12:05 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-fra_1.0.0~r65786-1+wmf1
11:56 moritzm: installing fontconfig security updates on jessie systems
09:20 moritzm: rolling restart of hhvm in on eqiad canary trusty app servers to pick up curl security updates
08:05 moritzm: rolling restart of hhvm in codfw to pick up curl security updates
07:42 moritzm: installing php5 security updates on remaining four precise systems
07:25 moritzm: installing curl security updates on Ubuntu systems
06:24 moritzm: installing chromium security updates on osmium
06:13 kart_: Updated cxserver to d3c7d64 (T142340)
05:32 _joe_: removing stale nodes from puppet
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 9 02:26:53 UTC 2016 (duration 5m 51s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 09m 24s)
01:01 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: coding style (duration: 00m 56s)
00:23 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Give autopatrol to translationadmin too (duration: 00m 47s)
00:17 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Rename autoreview group to autopatrolled on mw.org (duration: 00m 48s)
00:12 logmsgbot: catrope@tin Synchronized wmf-config: Various config changes for SWAT (duration: 00m 53s)

2016-08-08

23:53 mutante: git pull on dbtree/db1152, there were previous changes that did not get deployed
23:53 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Fix typo in Russian language switcher config (T129505) (duration: 00m 48s)
23:50 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.13/includes/specials/SpecialNewimages.php: Restore the newimagestext message (T142191) (duration: 00m 47s)
23:43 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Set import sources on enwikibooks (T142333) (duration: 00m 49s)
23:42 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.13/extensions/Echo/modules/nojs/mw.echo.badge.monobook.less: Hack around IE bug breaking badge alignment in Monobook (T142053) (duration: 00m 50s)
23:34 logmsgbot: catrope@tin Synchronized wmf-config/CommonSettings.php: Enable OAuth notifications in config (no-op until Wednesday) (T61172) (T62528) (duration: 00m 50s)
23:19 logmsgbot: catrope@tin Synchronized wmf-config/InitialiseSettings.php: Promote language switcher to top of page on ruwiki (T138961) (duration: 00m 59s)
23:11 bblack: openssl-1.0.2h-1~wmf4 -> caches
23:05 bblack: uploaded to carbon jessie-wikimedia: openssl-1.0.2h-1~wmf4 ( https://gerrit.wikimedia.org/r/#/c/303700/ )
21:57 dapatrick: Deployed patch for T130384 to wmf.13
21:55 hoo: Updated Wikidata's property suggester with data from today's json dump and removed the external identifiers as a workaround for T132839
21:53 ejegg: enabled globalcollect recurring job
21:30 ejegg: updated civicrm from 2d68638471aded73d05a796b05cab11809e31c56 to d9a765903ac682c0fbd329ced59f5ba3953970c9
20:21 subbu: aborting parsoid deploy for today (needs akosiaris to take a look at the deployment setup)
20:16 logmsgbot: aaron@tin Synchronized wmf-config/db-eqiad.php: Use pt-heartbeat and GTIDs on remaning sections (duration: 01m 00s)
20:14 logmsgbot: aaron@tin Synchronized wmf-config/db-codfw.php: Use pt-heartbeat and GTIDs on remaning sections (duration: 00m 53s)
20:02 subbu: starting parsoid deploy
19:50 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.13/includes/db/DatabaseMysqlBase.php: 267c62a530530e (duration: 00m 48s)
19:20 logmsgbot: aaron@tin Synchronized wmf-config/db-eqiad.php: Enable MASTER_GTID_WAIT() on s6 (duration: 00m 48s)
19:19 logmsgbot: aaron@tin Synchronized wmf-config/db-codfw.php: Enable MASTER_GTID_WAIT() on s6 (duration: 00m 48s)
19:18 ottomata: restarting kafka broker on kafka1022 to test more eventlogging rebalances
19:15 logmsgbot: aaron@tin Synchronized tests: (no message) (duration: 00m 50s)
19:13 logmsgbot: aaron@tin Synchronized wmf-config/db-eqiad.php: Switched to pt-heartbeat lag detection on s6 (duration: 00m 53s)
19:12 logmsgbot: aaron@tin Synchronized wmf-config/db-codfw.php: Switched to pt-heartbeat lag detection on s6 (duration: 00m 51s)
19:11 ottomata: restarting kafka broker on 1013 to test more eventlogging rebalances
19:00 ottomata: restarting kafka broker on 1013 to test more eventlogging rebalances
18:48 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/extensions/WikimediaEvents/modules/ext.wikimediaEvents.rlfeature.js: T141344 Track JSON support (duration: 00m 47s)
18:30 ottomata: restarting kafka broker on kafka1013 to test eventlogging leader rebalances
17:09 gehel: updating WDQS to latest and service restart
17:03 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2020.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2019.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2018.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2017.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2016.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:56 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2015.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2014.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2013.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2012.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2011.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2010.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2009.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2008.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2007.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2006.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2005.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2004.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:55 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp2003.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
16:53 urandom: T140008: Starting major compaction, restbase2001-a.codfw.wmnet
15:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add extendedconfirmed user group for fawiki (T140839) (duration: 00m 51s)
15:13 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Simplify the VE RB URL config some more, now that we no longer use wgServerName (duration: 00m 48s)
15:13 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-pt-ca_0.8.2+svn~57507-1+wmf1
15:12 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-fr-es_0.9.2~r61322-1+wmf1
15:08 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: MoodBar: Disable on all wikis except nlwiki (T131340) (duration: 01m 08s)
12:52 moritzm: upgrading firejail on scb1002 (along with service restarts except changeprop)
12:19 moritzm: upgrading firejail on scb1001 (along with service restarts except changeprop)
11:45 moritzm: upgrading firejail on scb* in codfw
11:17 _joe_: creating nihal.codfw.wmnet as a VM for puppetdb, T142365
11:08 _joe_: creating nitrogen.eqiad.wmnet as a VM for puppetdb, T142365
10:10 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.13/extensions/Cite/Cite_body.php: Cite::referencesFormatEntry: Avoid Undefined index: key - T132583 (duration: 00m 49s)
10:05 akosiaris: T135176 depool wtp2001-wtp2020
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2020.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2019.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2018.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2017.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2016.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2015.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2014.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2013.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2012.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2011.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2010.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2009.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2008.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:58 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2007.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:57 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2006.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:57 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2005.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:57 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2004.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:57 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2003.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:57 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp2002.codfw.wmnet (tags: ['dc=codfw', 'cluster=parsoid', 'service=parsoid'])
09:52 akosiaris: uploaded to apt.wikimedia.org precise-wikimedia: php5_5.3.10-1ubuntu3.24+wmf1
09:51 akosiaris: uploaded to apt.wikimedia.org jessie-wikimedia: php5_5.3.10-1ubuntu3.24+wmf1
09:48 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: Fix missed $wmg -> $wg for $wgRestbaseServer (duration: 00m 49s)
09:41 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.13/includes/revisiondelete/: Fix inconsistent RevDelFileItem visibilities - T142228 (duration: 00m 50s)
09:34 elukey: re-imaging aqs1005 to migrate Cassandra partitions to RAID10 (T142075)
09:13 moritzm: restarting elasticsearch on logstash100[56] to pick up java security updates
07:16 moritzm: installing php5 security updates
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 8 02:26:21 UTC 2016 (duration 5m 46s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 09m 21s)

2016-08-07

23:39 logmsgbot: reedy@tin Synchronized wmf-config: last 2 to wfLoadExtension (duration: 00m 59s)
23:12 logmsgbot: reedy@tin Synchronized wmf-config: Handful more extensions to wfLoadExtension (duration: 00m 49s)
22:57 logmsgbot: reedy@tin Synchronized wmf-config: RestbaseUpdateJobs to wfLoadExtension (duration: 00m 51s)
22:55 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Image Area to 100MP (duration: 00m 48s)
22:44 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: 2 more to extension.json (duration: 00m 48s)
22:35 logmsgbot: reedy@tin Synchronized docroot/noc/conf: Cleanup! (duration: 00m 50s)
22:30 logmsgbot: reedy@tin Synchronized docroot: trusted-xff symlink updates (duration: 00m 50s)
22:29 logmsgbot: reedy@tin Synchronized wmf-config/: Swap trusted-xff from cdb to php (duration: 00m 51s)
20:49 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.13/extensions/GlobalBlocking/extension.json: Adding globalblock-exempt grant for OAuth - T142306 (duration: 00m 57s)
16:22 cwd|afk: disabled globalcollect recurring donations
16:13 akosiaris: restarted apache2 on palladium for full depool to take place
12:47 hashar: root cause of CI outage is T126552
12:41 hashar: CI fully back. Root cause was Jenkins that could not properly create slaves config due to : Could not create rootDir /var/lib/jenkins/config-history/xxxx . Deleting via find /var/lib/jenkins/config-history/nodes/ -path '*_deleted_*' -delete
12:12 hashar: CI stuck spawning instances via Nodepool apparently due to : Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) --- Though there is only 8 instances ...
12:10 hashar: CI stuck spawning instances via Nodepool apparently due to : Quota exceeded for instances: Requested 1, but already used 10 of 10 instances (HTTP 403) --- Though there is only 8 instances ...
02:24 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Aug 7 02:24:55 UTC 2016 (duration 5m 51s)
02:19 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 08m 55s)

2016-08-06

23:09 yuvipanda: cleaned and re-accepted salt-key for labvirt1014, minion back up now
22:49 yuvipanda: run 'service mariadb start' on labsdb1003, puppet run didn't do anything
19:43 andrewbogott: rebooting labvirt1012 for a kernel downgrade
19:12 andrewbogott: rebooting labvirt1013 for kernel downgrade
09:36 akosiaris: revert back to old backed up bayes database on mendelevium.eqiad.wmnet (OTRS) to get bayes training working again
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Aug 6 02:26:00 UTC 2016 (duration 5m 48s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 08m 46s)
01:02 andrewbogott: re-imaging labvirt1014

2016-08-05

23:39 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.13/includes/api/ApiLogin.php: temporarily re-add dropped API feature to unbreak Pywikibot T142155 (duration: 00m 48s)
22:37 andrewbogott: rebooting labvirt1014 as part of a protracted iptables/nova-compute investigation
21:03 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Add transitionary timeline config primarily for beta (duration: 00m 57s)
18:26 andrewbogott: restarting rabbitmq-server on labcontrol1001
17:27 ejegg: rolled back SmashPig to 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3
17:25 ejegg: updated SmashPig from 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3 to 2e8a2f4c92840bd999a8742211e0a65d484fde00
16:03 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-spa-arg_0.4.0~r64399-1+wmf1
16:03 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-arg-cat_0.1.0~r64925-1+wmf1
15:16 akosiaris: T135176 pool wtp1019-wtp1024
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1024.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1023.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1022.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1021.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1020.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:13 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1019.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
13:56 akosiaris: strontium has issues, see https://phabricator.wikimedia.org/T142187
13:53 moritzm: uploaded gerrit 2.12.2-wmf2 for jessie-wikipedia to apt.wikimedia.org
13:49 ostriches: gerrit: quick restart to pick up apache and java updates
13:22 paravoid: started spamassassin/exim4 on mendelevium
12:19 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1079 as an api server; reduce main load (duration: 00m 49s)
12:14 akosiaris: just encountered https://wikitech.wikimedia.org/wiki/OTRS#SpamAssassin_stops_reporting_Bayes_results. Recovered the db with sa-learn --sync and then force spam/ham runs via the web interface
11:06 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Document T142135 and apply workaround (duration: 00m 52s)
11:03 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Document T142135 (duration: 00m 56s)
10:54 gehel: killing elasticsearch on logstash1004 (stuck during shutdown)
10:24 akosiaris: T135176 depool wtp1019-wtp1024
10:24 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1024.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:24 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1023.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:24 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1022.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:24 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1021.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1020.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1019.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 akosiaris: T135176 pool wtp1013-wtp1018
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1018.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1017.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1016.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1015.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1014.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:23 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1013.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
09:26 dcausse: creating tcywiki indices on elastic@codfw
09:23 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=30; selector: mw1261.eqiad.wmnet
09:17 moritzm: installing openjdk security updates on logstash cluster, rolling restart to pick up new JRE
09:12 elukey: upgrading httpd on mw126[78] to 2.4.10-10+deb8u4+wmf3 (T73487)
07:23 moritzm: restarting hhvm on mediawiki canaries to pick up curl security update
07:18 akosiaris: T135176 depool wtp1013-wtp1018
07:15 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1018.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:15 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1017.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:15 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1016.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:15 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1015.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:15 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1014.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:14 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1013.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
04:30 moritzm: installing curl security updates on jessie systems
04:07 yuvipanda: restarted nova-network on labnet1002 for T142165
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Aug 5 02:27:30 UTC 2016 (duration 6m 14s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 08m 43s)

2016-08-04

23:37 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/Echo: https://gerrit.wikimedia.org/r/#/c/303095/ https://gerrit.wikimedia.org/r/#/c/303099/ (duration: 00m 54s)
23:37 Pchelolo: restart restbase to apply gerrit:300214 config change
23:34 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/MobileFrontend: https://gerrit.wikimedia.org/r/#/c/303095/ https://gerrit.wikimedia.org/r/#/c/303099/ (duration: 00m 52s)
23:33 awight: update civicrm from 9a971ff6d74ae8e14c1c9f854155d9829e6a0278 to 2d68638471aded73d05a796b05cab11809e31c56
23:29 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.13/extensions/MobileFrontend: https://gerrit.wikimedia.org/r/#/c/302992/ (duration: 00m 56s)
23:17 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/301746/2 (duration: 00m 53s)
23:15 Dereckson: mwscript extensions/WikimediaMaintenance/filebackend/setZoneAccess.php tcywiki --backend=local-multiwrite
23:12 logmsgbot: dereckson@tin Synchronized wmf-config/interwiki.php: Interwiki map update for tcy.wikipedia.org (Gerrit:303098, T140898) (duration: 01m 05s)
22:43 Dereckson: Run mwscript extensions/CirrusSearch/maintenance/updateSearchIndexConfig.php --wiki=tcywiki --baseName=tcywiki
22:31 ejegg: updated SmashPig from b7f5e449aa62cc5518dea580de96b8ed7a2489d0 to 26a475bf5ae03d88ebc4c2fe9707d562d8e3afe3
22:16 Dereckson: Synchronized static/images/project-logos: Logos for tcy.wikipedia.org (T140898) (duration: 00m 48s)
22:12 Dereckson: Synced dblists, wikiversions, langlist, wmf-config/InitialiseSettings.php for tcy.wikipedia
22:07 Dereckson: mwscript extensions/WikimediaMaintenance/addWiki.php --wiki=aawiki tcy wikipedia tcywiki
21:50 urandom: T140825,T140869: Rolling restart of Cassandra instances, eqiad Rack `d'
21:48 urandom: T140825,T140869: Rolling restart of Cassandra instances, eqiad Rack `b', complete
21:26 urandom: T140825,T140869: Rolling restart of Cassandra instances, eqiad Rack `b'
21:17 ejegg: updated SmashPig from e6aa6fe6fdcaab8e961a8b0668cc742d4c443c46 to b7f5e449aa62cc5518dea580de96b8ed7a2489d0
21:14 urandom: T140825,T140869: Rolling restart of codfw Cassandra instances complete
20:40 akosiaris: restart apache on palladium. Managed to get it into a deadlock. OFC puppet spam will flood the channel
20:13 urandom: T140825,T140869: Performing rolling restart of codfw Cassandra instances
20:10 urandom: T140825,T140869: Cassandra instance restarts complete: restbase1011.eqiad.wmnet
20:03 urandom: T140825,T140869: Performing Cassandra instance rolling restart of restbase1011.eqiad.wmnet
19:59 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: More to extension.json in extension-list (duration: 00m 52s)
19:58 ottomata: testing some eventbus kafka failure scenarios in codfw with test.event. (short icinga downtime has been scheduled)
19:51 awight: paymentswiki config fix
19:47 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka2002.codfw.wmnet
19:46 awight: enabling paymentswiki queue mirroring
19:46 akosiaris: restart puppetmaster on palladium to activate loadfactor change. puppet related icinga spam will ensue*
19:46 akosiaris: restart puppetmaster on palladium to activate loadfactor change. puppet related icinga spam will ensure
19:45 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka2002.codfw.wmnet
19:43 logmsgbot: otto@palladium conftool action : set/pooled=yes; selector: kafka2001.codfw.wmnet
19:42 urandom: T140825,T140869: Restarting Cassandra, restbase1010-c.eqiad.wmnet
19:40 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: kafka2001.codfw.wmnet
19:40 urandom: T140825,T140869: Restarting Cassandra, restbase1010-b.eqiad.wmnet
19:39 ottomata: deploying eventlogging-service-eventbus to codfw hosts, depooling and pooling
19:37 urandom: T140825,T140869: Restarting Cassandra, restbase1010-a.eqiad.wmnet
19:06 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.13
18:49 logmsgbot: thcipriani@tin Synchronized README: test logstash host (duration: 00m 51s)
18:36 cwd: updated payments from 3a724bfb1a3e20e17b5886dae0ba7572020abd6b to b737b60c87da82543ab812ece4611c68af01307f
18:12 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1074 as backup rc node in case db1036 lags (duration: 00m 25s)
18:12 awight: update paymentswiki config to 71cc55194f4465600ce4da0ea9f7dfaefdda5479
17:36 elukey: added the analytics-deploy key to the Keyholder for the Analytics Refinery scap3 migration (also updated https://wikitech.wikimedia.org/wiki/Keyholder)
17:18 ostriches: gerrit: Restarting really quick, trying alternative mysql library.
17:17 urandom: T140825,T140869: Restarting Cassandra, restbase1007-c.eqiad.wmnet
17:14 urandom: T140825,T140869: Restarting Cassandra, restbase1007-b.eqiad.wmnet
17:08 urandom: T140825,T140869: Restarting Cassandra, restbase1007-a.eqiad.wmnet
17:07 elukey: restarting keyholder-proxy on tin to let the new analytics key to be picked up
15:08 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bumping portals to master. (duration: 00m 37s)
15:07 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master. (duration: 00m 36s)
14:11 godog: manually config s1 dbs to scrape on prometheus2001 as a test
13:51 akosiaris: delete+accept ganeti1004 salt minion key
13:43 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1012.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
13:43 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1011.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
13:43 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1010.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
13:43 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
13:43 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
10:59 jynus: applying prometheus grants to pc* hosts
10:33 elukey: upgrading httpd on mw126[56] to 2.4.10-10+deb8u4+wmf3 (T73487)
07:31 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1012.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:31 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1011.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:31 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1010.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:31 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:31 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1009.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1008.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
02:43 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Aug 4 02:43:16 UTC 2016 (duration 6m 23s)
02:36 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 05m 52s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 39s)
00:19 awight: revert paymentswiki config to 70b2ff90d9c8f19716f2e9c07a8dc8cfa17991ca
00:17 mutante: gerrit is restarting for config change 301822 (set default project owners). gerrit apache is restarting for 301829 (redirect /r) 301895 (logs for 10 days) and 301824 (renaming logs)
00:16 awight: update payments-wiki config to 793389ac8fa34cfc6a4ba1df67f2f9fac1ca02fe

2016-08-03

23:41 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.13/extensions/Echo/modules/nojs/: Adjust notification badges for monobook (T141923). Prevent IE from rendering the badge SVGs ridiculously big (T142042). (duration: 00m 29s)
23:38 kaldari: ran "mwscript maintenance/updateCollation.php --wiki=testwiki --force"
23:35 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Test numeric collation on testwiki (T141433) (duration: 00m 26s)
23:24 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.13/extensions/MobileFrontend/includes/skins/SkinMinervaBeta.php: Do not output the 'switch language' action on Main Page in beta (T142016) (duration: 00m 28s)
23:19 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable RevisionSlider on plwiki (T141974) (duration: 00m 26s)
23:06 logmsgbot: aude@tin rebuilt wikiversions.php and synchronized wikiversions files: Wikidata back to wmf.13
23:04 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.13/extensions/Wikidata: Update PropertySuggester (duration: 02m 04s)
22:46 mdholloway: mobileapps deployed e7488f6
22:44 mdholloway: starting mobileapps deployment
21:54 bblack: upgrading nginx on cache_text + cache_upload
21:46 bblack: upgrading nginx on cache_misc + cache_maps
21:41 bblack: nginx-1.11.3-1+wmf1 uploaded to carbon jessie-wikimedia
21:02 bearND: deployed mobileapps e48b6a8
21:00 bearND: starting mobileapps deploy
20:05 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: wikidata to 1.28.0-wmf.12
19:38 mutante: ganeti1004, start salt-minion
19:19 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.13
18:59 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.13/extensions/Wikidata: Update PropertySuggester (duration: 02m 02s)
18:53 akosiaris: T135176 pool wtp100[34567] with weight=15
18:53 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1007.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:53 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1006.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:53 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1005.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:53 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:53 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:08 mutante: restarted morebots-production
17:35 mobrovac: citoid deploying 0b9f59fe0
17:01 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1007.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
17:01 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1005.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
17:01 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
17:01 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
16:22 akosiaris: reboot wtp1006
16:17 bblack: upgrading openssl on cache_text, cache_upload
16:01 akosiaris: T135176 pool wtp100[3457] with weight=15. wtp1006 does not look so good
16:00 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1007.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
16:00 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1005.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1007.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1005.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:59 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-urd_0.1.0~r61311-1+wmf1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-tat_0.1.0~r60887-1+wmf1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-swe_0.7.0~r69513-1+wmf1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-spa_0.1.0~r65494-1+wmf1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-pt-gl_0.9.2~r60358-1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-oc-es_1.0.6~r60161-1+wmf1
15:35 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-oc-ca_1.0.6~r60158-1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-nno_0.9.0~r69513-2+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-mlt-ara_0.2.0~r62623-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-mk-en_0.1.1~r57554-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-mk-bg_0.2.0~r49489-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-is-sv_0.1.0~r56030-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-id-ms_0.1.1+svn~57870-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-hbs-mkd_0.1.0~r57554-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eu-es_0.3.3~r56159-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eu-en_0.3.1~r60155-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-pt_1.1.5+svn~57507-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-it_0.1.0~r51165-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-gl_1.0.8~r57542-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-ca_1.2.1+svn~57448-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-es-ast_1.1.0~r60158-1+wmf1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eo-fr_0.9.0~r57551-1
15:34 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eo-es_0.9.1~r60655-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eo-en_1.0.0~r63833-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-eo-ca_0.9.1~r60655-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-en-gl_0.5.2~r57551-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-cy-en_0.1.1~r57554-3+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-cat_1.0.0~r65787-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-ca-it_0.1.1~r57554-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-arg_0.1.2~r65494-1+wmf1
15:33 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-af-nl_0.2.0~r58256-1+wmf1
15:32 bblack: upgrading openssl on cache_maps + cache_misc
15:23 bblack: openssl-1.0.2h-1~wmf3 uploaded to carbon jessie-wikimedia ( https://gerrit.wikimedia.org/r/#/c/301920/ )
15:13 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.13/includes/Linker.php: SWAT: Debug Logging for Undefined index: width in Linker.php (duration: 00m 30s)
14:36 godog: reboot ms-be1022 following firmware upgrade T141756
14:17 mobrovac: restbase deploy end of ff1ee1e7
14:06 mobrovac: restbase deploy start of ff1ee1e7
13:16 godog: reboot ms-be1022 - T140597
12:31 akosiaris: T135176 depool wtp100[34567]
12:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1007.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1006.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1005.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1004.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:30 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1003.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
09:29 jynus: applying prometheus required grants to all databases T128185
08:43 godog: upload scap 3.2.2-1 to carbon T127762
08:32 jynus: restarting replication on db1018
08:31 elukey: upgrading httpd on mw126[34] to 2.4.10-10+deb8u4+wmf3 (T73487)
08:27 jynus: stopping replication to db1018 (s2-master-eqiad)
08:21 jynus: restarting replication on db1024
08:14 jynus: stopping replication to db1024 (depooled) to test replication alerts
08:12 jynus: restarting slave on db2017
08:07 jynus: stopping replication to s2-master-codfw (db2017) to test replication alerts
07:29 jynus: restarting pt-heartbeat-wikimedia on all database masters
05:38 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/extensions/MobileFrontend/: I195f67d061d (duration: 00m 29s)
05:37 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/includes/OutputPage.php: I195f67d061d (duration: 00m 30s)
05:36 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/resources/Resources.php: I195f67d061d (duration: 00m 30s)
05:36 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/autoload.php: I195f67d061d (duration: 00m 42s)
05:35 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.13/includes/resourceloader/: I195f67d061d (duration: 00m 38s)
03:09 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Aug 3 03:09:30 UTC 2016 (duration 6m 58s)
03:02 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.13) (duration: 14m 56s)
02:30 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 34s)
02:24 mutante: neon - restarted ferm service
02:21 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings-labs.php: labs-only, move from ores.wikimedia.org to ores-beta.wmflabs.org (duration: 00m 33s)
02:19 mutante: restarted log bot
01:17 awight: revert paymentswiki config change, new commit 4d0090f50d515047c5e4dfbe565cd04bdd31801d
01:15 awight: update paymentswiki config to f89b8d372753fb2f37c50c062e8cca12c6351e19
01:00 awight: update paymentswiki config to ac5f7a848450f63b4438de926088acd4feacea83
00:27 awight: update paymentswiki config to 3a724bfb1a3e20e17b5886dae0ba7572020abd6b
00:25 logmsgbot: dereckson@tin Synchronized wmf-config: Add $wmgEchoMentionStatusNotifications and enable it in beta labs (no-op, sync labs files) (duration: 00m 28s)
00:20 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Add $wmgEchoMentionStatusNotifications and enable it in beta labs (no-op in prod, T135717, T139623) (duration: 00m 26s)
00:19 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add $wmgEchoMentionStatusNotifications and enable it in beta labs (no-op in prod, T135717, T139623) (duration: 00m 25s)
00:13 logmsgbot: legoktm@tin Synchronized wmf-config: De-deploy CustomData extension - T140847 (duration: 00m 28s)
00:02 logmsgbot: maxsem@tin Synchronized wmf-config: Labs only (duration: 00m 30s)

2016-08-02

23:59 awight: update paymentswiki config to 9b3ee23d54c481343164c50f172184bb7b99e871
23:51 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Reset ar.wikipedia content namespaces (T141906) (duration: 00m 26s)
23:50 awight: update paymentswiki config to 5429797ec902cd3a6fe098d22e12af7a578e80b8
23:50 awight: update paymentswiki config to
23:39 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Move abusefilter permissions to abusefilter.php for azbwiki (T141860, 2/2) (duration: 00m 27s)
23:38 logmsgbot: dereckson@tin Synchronized wmf-config/abusefilter.php: Move abusefilter permissions to abusefilter.php for azbwiki (T141860, 1/2) (duration: 00m 28s)
23:29 logmsgbot: dereckson@tin Synchronized wmf-config/abusefilter.php: Allow abuse filter editors group to edit tags on en.wikipedia (T141847) (duration: 00m 32s)
21:41 logmsgbot: krinkle@tin Synchronized docroot/noc: Update favicon.ico symlink (duration: 00m 34s)
21:20 mutante: gerrit is restarting to apply config changes: 301898 (warm cache, faster startup) 301894 (double size of conflicts cache) 302129 (avoid breaking full phabricator urls)
21:04 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.13
21:01 ejegg: updated civicrm from 4904c4aae3565b65d5f37ecb827ea26c930b72d6 to 9a971ff6d74ae8e14c1c9f854155d9829e6a0278
21:01 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.13 and rebuild l10n cache with Wikidata (duration: 24m 13s)
20:36 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.13 and rebuild l10n cache with Wikidata
20:17 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.13 and rebuild l10n cache (duration: 45m 40s)
19:31 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.13 and rebuild l10n cache
18:21 akosiaris: T135176 set weight for wtp100[12] to 15
18:20 logmsgbot: akosiaris@palladium conftool action : set/weight=15; selector: wtp1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
18:19 logmsgbot: akosiaris@palladium conftool action : set/weight=15; selector: wtp1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
17:01 thcipriani: starting branch-cut for 1.28.0-wmf.13
16:17 andrewbogott: disabling puppet on all lab* hosts for staged upgrade
15:47 mobrovac: restbase rolling restart for firejail upgrade to 0.9.40
15:45 moritzm: upgraded firejail on restbase* to 0.9.40.3
15:32 jynus: performing schema change on heartbeat.heartbeat on all core databases
15:29 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.12/extensions/RevisionSlider/modules/ext.RevisionSlider.init.js: SWAT: Track the load times of RevisionSlider (duration: 00m 26s)
15:25 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove T107711 debug logging (duration: 00m 30s)
15:22 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.12/extensions/ContentTranslation/modules/tools/ext.cx.tools.link.js: SWAT: Fix: Target links has source link titles (duration: 00m 34s)
14:55 ejegg: updated fundraising tools from 28bc2da677caa795c58f906db76a1f8d612ac899 to b3ed7ab3deac94c4e465d3768109bca05b6f0a0c
14:29 moritzm: upgrading restbase2003 system to firejail 0.9.40.3
14:26 akosiaris: T135176 set weight for wtp100[12] to 8
14:24 ema: setting default_ttl=604800 on cache_upload varnish backends
14:24 moritzm: upgrading restbase staging systems to firejail 0.9.40.3
14:22 logmsgbot: akosiaris@palladium conftool action : set/weight=8; selector: wtp1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
14:22 logmsgbot: akosiaris@palladium conftool action : set/weight=8; selector: wtp1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
14:14 jynus: resizing online labsdb1006 postgres fs to increase available disk space
12:34 akosiaris: T135176 repool wtp100[12] with a weight of 1 instead of 15
12:33 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:33 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: wtp1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:33 logmsgbot: akosiaris@palladium conftool action : set/weight=1; selector: wtp1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
12:33 logmsgbot: akosiaris@palladium conftool action : set/weight=1; selector: wtp1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
11:48 mark: puppet node clean & salt-key -d of cp3022.esams.wmnet
10:44 logmsgbot: krenair@tin Synchronized wmf-config: https://gerrit.wikimedia.org/r/302409 - labs-only changes (duration: 00m 34s)
09:49 godog: temporarily stop puppet and disable check_hpssacli on ms-be1023 T136631
09:42 moritzm: installing chromium security updates on osmium
09:39 godog: upload scap 3.2.1-1 to carbon T127762 (at 9:33)
07:55 akosiaris: T135176 depool wtp100[12]
07:54 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
07:54 logmsgbot: akosiaris@palladium conftool action : set/pooled=no; selector: wtp1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=parsoid', 'service=parsoid'])
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Aug 2 02:26:32 UTC 2016 (duration 5m 22s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 25s)

2016-08-01

23:31 eileen: Upgrading CiviCRM from d657255e1edebeccfc0a03bea70b78eb11375cf8 to 4904c4aae3565b65d5f37ecb827ea26c930b72d6
23:27 logmsgbot: dereckson@tin Synchronized wmf-config/: Revert "Add $wmgEchoMentionStatusNotifications and enable it in beta labs" (T135717, T139623) (duration: 00m 27s)
23:25 logmsgbot: dereckson@tin Synchronized wmf-config/: Add $wmgEchoMentionStatusNotifications and enable it in beta labs (T135717, T139623) (duration: 00m 30s)
23:24 eileen: Updating civicrm from d657255e1edebeccfc0a03bea70b78eb11375cf8 to d657255e1edebeccfc0a03bea70b78eb11375cf8
23:21 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Load Elastica extension via extension.json (Gerrit:301856) (duration: 00m 31s)
23:15 mutante: root@tin:/srv/mediawiki-staging# find . -uid 0 -exec chown mwdeploy:wikidev {} \;
23:05 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Labs: Set CategoryCollation for dewiki to 'uca-de-u-kn' (T128806) (duration: 00m 38s)
22:03 dapatrick: Deployed patch for T139670 to wmf.12
21:59 Amir1: restarting uwsgi-ores in scb1001 and scb1002
21:37 Amir1: for ores
21:37 Amir1: deploying 6790ccb
21:20 Amir1: deploying e8d2475 to scb nodes
21:17 chasemp: iridium sudo -u phd /srv/phab/phabricator/bin/repository update 1912
20:33 Amir1: deploying 624d777 to ores
20:18 subbu: finished deploying parsoid sha abf396eb
20:15 mutante: restarted ganglia aggregators on carbon
20:06 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:04 subbu: starting parsoid deploy
19:56 logmsgbot: mobrovac@tin Synchronized wmf-config/LabsServices.php: (no message) (duration: 00m 38s)
19:37 Pchelolo: deploy restbase 840411a4
19:32 Pchelolo: deploy restbase 840411a4 canary on restbase1007
18:49 mutante: rebooting xenon to toggle HT setting in BIOS
18:10 bblack: upgrading openssl on cache_text and cache_upload
17:36 bblack: upgrading openssl package on cache_maps + cache_misc
17:06 Pchelolo: staging deploy restbase 840411a44
16:59 Pchelolo: labs deploy restbase 840411a44
16:10 bd808: Restarted logstash on logstash1001; missing hhvm and service logs; no output to /var/log/logstash/logstash.log for days
15:59 bblack: cp1065: canary upgrade of openssl to 1.0.2h-1~wmf2 (+ nginx upgrade-restart)
15:58 urandom: T134016: Bootstrapping restbase1015-c.eqiad.wmnet
15:57 bblack: uploaded openssl_1.0.2h-1~wmf2 to carbon (jessie-wikimedia) - https://gerrit.wikimedia.org/r/#/c/301903/
15:38 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.12/includes/Linker.php: Debug Logging for Undefined index: width in Linker.php (duration: 00m 25s)
15:29 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: Remove dewiki_diffstats logging (duration: 00m 26s)
15:27 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: Remove T124356 debug logging (duration: 00m 25s)
15:25 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: Debug logging for T138987 (duration: 00m 26s)
15:23 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings-labs.php: Beta move $wgEchoMentionStatusNotifications to CommonSettings PART 2/2 (duration: 00m 24s)
15:22 logmsgbot: addshore@tin Synchronized wmf-config/CommonSettings-labs.php: Beta move $wgEchoMentionStatusNotifications to CommonSettings PART 1/2 (duration: 00m 24s)
15:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Set favicon for mk.wiktionary (T140566) (duration: 00m 25s)
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysops on he.wiktionary to remove autopatroller and patroller user rights (T140563) (duration: 00m 30s)
15:04 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Expanding throttle limits for enwiki Edit-a-thon (T141421) (duration: 00m 24s)
14:41 andrewbogott: restarting slapd on serpens to reclaim leaked memory
13:48 godog: reboot ms-be1023 after raid controller fw upgrade T141756
12:46 mobrovac: zotero translators deployed cde2f75
10:11 godog: reboot ms-be2027 after raid controller fw upgrade T141756
09:25 jynus: dropping index name_type_patrolled_timestamp on zhwiki on db1060 and db1054 T140108
03:20 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.12/includes/filerepo/file/LocalFile.php: c4f34e7a12baa9 (duration: 00m 44s)
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Aug 1 02:26:44 UTC 2016 (duration 5m 38s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 22s)

2016-07-31

18:07 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.12/includes/page/WikiPage.php: 061737668d729ba1a76 (duration: 00m 24s)
17:49 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.12/includes/page/WikiPage.php: 23330222a2af3fb (duration: 00m 34s)
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 31 02:27:53 UTC 2016 (duration 5m 34s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 07s)

2016-07-30

23:36 Jamesofur: deleted 7 files from server for legal compliance
02:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jul 30 02:25:55 UTC 2016 (duration 5m 37s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 10s)

2016-07-29

23:56 YuviPanda: hardreset xenon
23:44 urandom: Rebooting xenon.eqiad.wmnet
23:13 bblack: installed openssl-1.0.2h-1~wmf2 on pinkunicorn for the weekend (not on carbon yet) - https://gerrit.wikimedia.org/r/301903
21:02 ostriches: gerrit: raised log level on sshd to ERROR from WARN. Irrelevant logspam.
20:56 mutante: restarted grrrit-wm but this time only because it died by itself
20:22 mutante: gerrit restarting to apply config change 301873 - tuning caches
20:11 YuviPanda: restart gerrit-wm bot
19:56 mutante: gerrit restarting to apply config change 300446 - up heap size limit
18:59 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings-labs.php: labs only change to enable mobile language bar (duration: 00m 27s)
18:37 urandom: T134016: Bootstrapping restbase2009-c.codfw.wmnet
16:39 YuviPanda: granted addshore admin on labs grafana
15:17 anomie: starting maintenance script for phab:T140811
13:15 Dereckson: Purged static resources related to mk.wiktionary (T141610)
12:43 elukey: upgrading zuul-merger to zuul_2.1.0-391-gbc58ea3-wmf2jessie1_amd64.deb on scandium
10:23 elukey: restarting cassandra on aqs100[123] to apply the latest config (https://gerrit.wikimedia.org/r/#/c/301780/1 - T140869)
10:14 jynus: applying new grants to all s1 servers
09:38 hashar: Upgrading Zuul to get rid of a forced sleep(300) whenever a patch is merged T93812. zuul_2.1.0-391-gbc58ea3-wmf2precise1
08:51 godog: switch back statsite flush period to 60s T101141
07:30 jynus: schema change continues for s2, s1, s4 and s5 T140108
07:24 jynus: fixing s3 replication lag created by TokuDB insert problem
06:59 jynus: powercycling db2069 T141601
04:04 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.12/extensions/TorBlock/extension.json: Move basic torunblocked line to GrantPermissions, not GroupPermissions, see wikitech-l (duration: 00m 38s)
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jul 29 02:26:09 UTC 2016 (duration 5m 57s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 07m 30s)

2016-07-28

23:29 ejegg: updated payments-wiki from 2d9dd79507a42ced0a99bde87b3c45b804610e40 to 3a724bfb1a3e20e17b5886dae0ba7572020abd6b4
23:23 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/301724/1 (duration: 00m 24s)
23:19 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/301644/ (duration: 00m 29s)
22:42 logmsgbot: maxsem@tin Synchronized wmf-config/: Labs only (duration: 00m 45s)
20:25 urandom: T134016: Bootstrapping restbase2006-c.codfw.wmnet
20:02 Pchelolo: deploy restbase cdd164c4e
19:43 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.12/includes/api/ApiQueryUserContributions.php: Fix Undefined variable issue in ApiQueryUserContributions (duration: 00m 32s)
19:42 Pchelolo: deploy restbase cdd164c4e canary on restbase1007
19:22 urandom: T134016: Bootstrapping restbase1014-c.eqiad.wmnet
19:06 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.12
19:01 mutante: restarted grrrit-wm
17:37 ottomata: powercycling analytics1032
16:50 godog: bounce statsite on graphite1001 T101141
16:38 jynus: stopping dbproxy1001 haproxy service
15:41 logmsgbot: aude@tin Synchronized wmf-config/Wikibase.php: Update entityNamespaces setting (duration: 00m 27s)
15:28 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.12/extensions/Wikidata: Fix exception when undeleting items and fix css bug (duration: 01m 52s)
15:20 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.11/extensions/ContentTranslation: Fix DumpCorpora script (duration: 00m 27s)
15:19 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.12/extensions/ContentTranslation: Fix DumpCorpora script (duration: 00m 31s)
15:15 godog: bounce statsite on graphite1001 - T101141
15:10 logmsgbot: aude@tin Synchronized dblists/clldefault.dblist: Enable content translation on more wikis (duration: 00m 23s)
15:09 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable content translation on more wikis (duration: 00m 25s)
15:07 jynus: adding new index (schema change) to recentchanges T140108
14:59 godog: bounce carbon-cache on graphite1001 - T101141
14:11 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1055 after maintenance (duration: 00m 35s)
12:51 elukey: upgrading zuul-merger to zuul_2.1.0-391-gbc58ea3-wmf1jessie (T140894)
12:14 akosiaris: installing updates on mendelevium
11:18 jynus: deploying schema change to all ores databases T140803
11:17 godog: replace statsdlb with statsd-proxy on graphite1001
11:09 jynus: testing schema change on db2038
09:25 moritzm: installing PHP security updates
08:49 godog: swift eqiad-prod: ms-be102[3456] weight 3000
07:43 elukey: starting decom process for old api servers - mw11(1[4-9]|20|3[0-9]|4[0-8]).eqiad.wmnet (tracked in https://etherpad.wikimedia.org/p/appservers-decom)
07:16 _joe_: regenerated the ssl key for rhodium, 1024 bits
06:54 moritzm: installing java security updates on restbase staging systems
06:44 _joe_: installed puppet 3.8 from backports on rhodium
06:36 moritzm: installing perl security updates on eqiad and codfw jessie systems
06:36 _joe_: refreshed puppet facts for the compiler
05:56 jynus: shutting down db1055 for upgrade
02:47 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jul 28 02:47:28 UTC 2016 (duration 6m 22s)
02:41 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 08m 03s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 08m 49s)
02:13 twentyafterfour: restarted apache2 on iridium to deploy 4305a9bb0300650ea40de433261c7e59cc88e4bc
01:30 twentyafterfour: Deploying #phab-2016.30 (https://phabricator.wikimedia.org/project/profile/2118/) - no downtime is expected.

2016-07-27

23:45 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Test numeric sorting on Beta Cluster (Gerrit:301520, labs only, no-op in prod) (duration: 00m 23s)
23:22 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.12/extensions/ORES/maintenance/PopulateDatabase.php: Add revision_id to log for errors (T141368, 2/2, no-op) (duration: 00m 29s)
23:21 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.12/extensions/ORES/includes/Cache.php: Add revision_id to log for errors (T141368, 1/2) (duration: 00m 31s)
22:50 Pchelolo: restbase deploy cdd164c4e8 to staging
21:34 mutante: planet2001 tmp disable puppet for testing
21:17 ejegg: updated payments from 79cb53998c41f72d0fa49130ed1f66dc112b478c to 2d9dd79507a42ced0a99bde87b3c45b804610e40
21:13 bearND: deployed mobileapps e561edf
21:08 bearND: starting mobileapps deploy
20:47 andrewbogott: restarting rabbitmq-server on labcontrol1001 because sometimes that fixes a thing
19:59 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.12
19:38 Pchelolo: deploy restbase 8f5e2897e to staging
19:16 andrewbogott: rebuilding labnet1001 (it's a spare and shouldn't affect Labs)
19:04 urandom: T134016: Bootstrapping restbase2005-c.codfw.wmnet
18:50 Pchelolo: restart changeprop to apply config changes 300681 and 301305
18:08 Pchelolo: deploy restbase 8efbc9282e to staging
17:58 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2009-b.codfw.wmnet)
17:51 mutante: restarted grrriit-wm
17:45 mutante: gerrit is restarting to deploy config change 301381, a couple seconds downtime
17:23 jynus: running analyze table on enwiki.logging db1055 (depooled)
17:22 urandom: Truncating "local_group_wikipedia_T_parsoid_section_offsets".data, "local_group_wikipedia_T_parsoid_dataW4ULtxs1oMqJ".data, and "local_group_wikipedia_T_parsoid_html".data in RESTBase staging
17:21 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 also from the rc/log role (duration: 00m 28s)
16:33 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2009-a.codfw.wmnet)
16:22 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.12/extensions/MobileFrontend/includes/skins/SkinMinerva.php: 301387 Fix watchstar for logged-out user (duration: 00m 32s)
16:04 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.11/includes/EditPage.php: 301356 Count edit conflicts for each namespace separately (duration: 00m 32s)
15:54 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: IP cap lift for Wikipedia Edit-a-thon on 2016-08-03 (duration: 00m 23s)
15:49 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2006-b.codfw.wmnet)
15:45 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add "upwizcampeditors" to $wgAddGroups, $wgRemoveGroups for commonswiki (duration: 00m 24s)
15:38 logmsgbot: thcipriani@tin Synchronized wmf-config/CirrusSearch-common.php: SWAT: Turn on textcat based language detection for search PART II (duration: 00m 23s)
15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Turn on textcat based language detection for search PART I (duration: 00m 27s)
15:21 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2006-a.codfw.wmnet)
15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: beta wgEchoMentionStatusNotifications default true (duration: 01m 28s)
14:58 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1055 for database maintenance (duration: 00m 29s)
14:50 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2005-b.codfw.wmnet)
14:16 urandom: T134016: Cancelling bootstrap of restbase2005-c.codfw.wmnet
14:12 urandom: T134016: Restarting Cassandra instance to apply disabled streaming socket timeout (restbase2005-a.codfw.wmnet)
13:29 elukey: Restart Cassandra on aqs100[123] to apply the latest configuration (T140869)
12:50 elukey: disabling puppet on restbase*, aqs* and maps* as extra careful step for https://gerrit.wikimedia.org/r/301083 (no-op but better safe than sorry)
12:28 bblack: starting wipe of cache_misc caches
12:23 akosiaris: puppet enabled on netmon1001 (correction of previous log line)
12:22 akosiaris: puppet enabled on net1001
12:05 akosiaris: disable puppet on netmon1001, debugging servermon
11:25 Dereckson: Run initSiteStats to update statistics count on ast.wikipedia (T141432)
10:42 jynus: add extra grants to db1016 and all of m1 for servermon
10:20 moritzm: restarting slapd on serpens
08:43 elukey: Decomissioning mw1018-25 (T139353)
08:30 logmsgbot: hashar@tin Synchronized wmf-config/InitialiseSettings.php: Lower loglevel for resourceloader to info https://gerrit.wikimedia.org/r/#/c/301336/ (duration: 00m 26s)
07:49 jynus: update m1-master to point to dbproxy1006
07:11 moritzm: installing perl security updates in esams and codfw
06:56 jynus: dropping tables from m4 shard T141407
03:06 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jul 27 03:06:43 UTC 2016 (duration 6m 49s)
02:59 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.12) (duration: 15m 29s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 09m 11s)
02:09 mutante: restarted grrrit-wm after removing bugzilla password from gerrit
01:57 mutante: lead removed reviewer_count job from root's crontab
01:41 mutante: ytterbium - shutdown -h now, over and out
01:05 ejegg: rolled back paymentswiki from 79d2b67067fd7e579372b63e0d619eccfa3b9143 to 79cb53998c41f72d0fa49130ed1f66dc112b478c
00:56 mutante: xenon - rsync cassandra-test data to restbase-test2001 /srv/backups/eqiad/
00:53 mutante: restbase-test2001-2003 - test rsyncing, create temp data dir. mkdir -p $(grep path /etc/rsync.d/frag-parsoid-html | cut -d= -f2)

2016-07-26

23:59 logmsgbot: reedy@tin Synchronized php-1.28.0-wmf.12/extensions/MobileFrontend/: Deploy revert for group0 for T141386 (duration: 00m 30s)
23:25 logmsgbot: reedy@tin Synchronized dblists/commonsuploads.dblist: Disabling local uploads on ms.wikipedia.org (duration: 00m 23s)
23:18 logmsgbot: reedy@tin Synchronized wmf-config/event-schemas: Bump event-schemas submodule commit to master (duration: 00m 28s)
23:08 ejegg: updated payments from 79cb53998c41f72d0fa49130ed1f66dc112b478c to 79d2b67067fd7e579372b63e0d619eccfa3b9143
23:07 logmsgbot: reedy@tin Synchronized wmf-config/: Remove rest of ImageMetrics config (duration: 00m 33s)
23:04 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Undeploy ImageMetrics (duration: 00m 27s)
21:05 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: moar extension.json (duration: 00m 33s)
20:43 mdholloway: mobileapps deployed fd3f33b
20:41 mdholloway: starting mobileapps deployment
20:37 urandom: Bootstrapping restbase2005-c.eqiad.wmnet
20:23 urandom: T134016: Bootstrapping restbase1009-c.eqiad.wmnet
19:58 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1015-b.eqiad.wmnet)
19:54 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1015-a.eqiad.wmnet)
19:53 urandom: T140825: Setting vm.dirty_background_bytes=24576 (restbase1015.eqiad.wmnet)
19:49 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1014-b.eqiad.wmnet)
19:43 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1014-a.eqiad.wmnet)
19:42 urandom: T140825: Setting vm.dirty_background_bytes=24576 (restbase1014.eqiad.wmnet)
19:40 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.12
19:37 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1009-b.eqiad.wmnet)
19:34 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.12 and rebuild l10n cache (duration: 25m 29s)
19:33 urandom: T134016, T140825: Restarting Cassandra to disable trickle_fsync and streaming socket timeouts (restbase1009-a.eqiad.wmnet)
19:33 urandom: T140825: Setting vm.dirty_background_bytes=24576 (restbase1009.eqiad.wmnet)
19:09 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.12 and rebuild l10n cache
19:07 logmsgbot: thcipriani@tin Purged l10n cache for 1.28.0-wmf.10
18:36 logmsgbot: addshore@tin Synchronized php-1.28.0-wmf.11/extensions/WikimediaEvents/WikimediaEventsHooks.php: dewiki_diffstats add rev timestamps & feature state 301119 (duration: 00m 28s)
18:28 logmsgbot: addshore@tin Synchronized wmf-config/InitialiseSettings.php: Enable RevisionSlider on mediawikiwiki 301105 (duration: 01m 28s)
18:07 bd808: Restarted elasticsearch on logstash1003, couldn't find master
17:12 subbu: finished deploying parsoid version 285b6983
17:10 thcipriani: starting branch cut for 1.28.0-wmf.12
17:06 subbu: synced new parsoid code; restarted parsoid on wtp1007 as a canary
17:03 subbu: starting parsoid deploy
15:55 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Configuration changes for mk.wiktionary.org PART III (duration: 00m 26s)
15:54 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/mkwiktionary.png: SWAT: Configuration changes for mk.wiktionary.org PART II (duration: 00m 24s)
15:54 logmsgbot: thcipriani@tin Synchronized static/favicon/wiktionary/mk.ico: SWAT: Configuration changes for mk.wiktionary.org PART I (duration: 00m 24s)
15:47 godog: reimage mw1292 as thumbor1002
15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove EchoBundleEmailInterval (T135446) PART II (duration: 00m 26s)
15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove EchoBundleEmailInterval (T135446) PART I (duration: 00m 34s)
15:00 paravoid: installed cr2-eqiad FPC 3
14:36 moritzm: uploading openjdk-8 security update (8u102-b14-1~bpo8+1) to carbon
14:32 godog: reimage mw1291 as thumbor1001
13:55 jynus: compressing 300GB table on dbstore2002 (expect warnings, slowdown, lag -but it is a passive analytics slave)
12:42 moritzm: installing perl security updates
11:48 moritzm: installing exim4 updates related to perl security release
11:39 logmsgbot: filippo@palladium conftool action : set/pooled=inactive; selector: name=mw1291.eqiad.wmnet
11:39 logmsgbot: filippo@palladium conftool action : set/pooled=inactive; selector: name=mw1292.eqiad.wmnet
11:29 logmsgbot: filippo@palladium conftool action : set/pooled=no; selector: name=mw1292.*
11:28 logmsgbot: filippo@palladium conftool action : set/pooled=no; selector: name=mw1291.*
10:43 elukey: restarting cassandra on aqs100[456] instances (not serving live traffic)
09:18 moritzm: updating debhelper, cdbs, devscripts, libintl-perl, libmodule-build-perl and libnet-dns-perl on jessie systems for compatibility with perl security update
07:38 kart_: Update cxserver to 447a6c9 - registry: Remove 'en' as target from Apertium MT - disables machine translation to English in ContentTranslation
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jul 26 02:31:43 UTC 2016 (duration 6m 8s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 09m 00s)
01:51 mutante: cerium testing is over?
01:11 Amir1: deploying from 2d9817b to a291da1 for ores in scb nodes
00:53 mutante: lead - stopped rsyncd
00:49 urandom: T134016: Bootstrapping restbase2008-c.codfw.wmnet
00:06 Pchelolo: restbase deploy ae5fbac to staging

2016-07-25

23:11 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add dewiki_diffstats to wmgMonologChannels (Gerrit:288158, T134861) (duration: 00m 25s)
22:48 legoktm: restarted zuul due to depends-on lockup
22:46 logmsgbot: reedy@tin Synchronized docroot/noc/conf/: Update dblist symlinks (duration: 00m 37s)
21:52 tgr: deployed security patch for T137551
20:37 Pchelolo: restbase deploy 8efbc92
20:10 Pchelolo: restbase deploy 8efbc92 canary deploy to restbase1007
20:05 Pchelolo: restbase deploy 8efbc92 to staging
20:00 Pchelolo: restbase deploy 8efbc92 to deployment-prep
19:21 urandom: T134016: Bootstrapping restbase1013-c.eqiad.wmnet
18:31 ottomata: upgrading kafka to 0.9 in main-codfw, first kafka2001 then 2002
18:15 mutante: ytterbium - revoke puppet cert, delete salt-key, remove from icinga
16:16 urandom: T134016: Restarting Cassandra to apply stream timeout (restbase1013-b.eqiad.wmnet)
16:10 urandom: T134016: Restarting Cassandra to apply stream timeout (restbase1013-a.eqiad.wmnet)
16:06 urandom: T140825, T134016: Restarting Cassandra to apply stream timeout, and disable trickle_fsync (restbase1012-c.eqiad.wmnet)
16:02 urandom: T140825, T134016: Restarting Cassandra to apply stream timeout, and disable trickle_fsync (restbase1012-b.eqiad.wmnet)
15:54 urandom: T140825, T134016: Reststarting Cassandra to apply stream timeout, and disable trickle_fsync (restbase1012-a.eqiad.wmnet)
15:53 urandom: T140825: Setting vm.dirty_background_bytes=24M on restbase1012.eqiad.wmnet
15:43 urandom: T140825, T134016: Reststarting Cassandra to apply stream timeout, and 8MB trickle_fsync (restbase1008-c.eqiad.wmnet)
15:39 urandom: T140825, T134016: Reststarting Cassandra to apply stream timeout, and 8MB trickle_fsync (restbase1008-b.eqiad.wmnet)
15:34 urandom: T140825, T134016: Reststarting Cassandra to apply stream timeout, and 8MB trickle_fsync (restbase1008-a.eqiad.wmnet)
15:28 elukey: Standardized the jmxtrans GC metric names to pick up automatically variations in settings. This introduces metric name changes in Hadoop, Zookeeper, Kafka. (https://gerrit.wikimedia.org/r/#/c/299118/)
12:53 moritzm: installing squid security updates
10:10 _joe_: remove spurious puppet facts
10:10 _joe_: remove spurious puppet facts
10:04 moritzm: installing Django security updates
09:18 godog: swift eqiad-prod: ms-be102[3456] weight 1500
03:26 hashar: scandium: migrating zuul-merger repos from lead to gerrit.wikimedia.org: find /srv/ssd/zuul/git -path '*/.git/config' -print -execdir sed -i -e 's/lead.wikimedia.org/gerrit.wikimedia.org/' config \;
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jul 25 02:28:21 UTC 2016 (duration 5m 52s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 09m 09s)
02:03 ostriches: gerrit: reindexing lucene now that we have new data. searches/dashboards may look a tad weird for a bit
01:53 hashar: starting Zuul
01:51 mutante: restarted grrrit-wm
01:39 ostriches: lead: turning puppet back on, here we go
01:38 jynus: m2 replication on db2011 stopped, master binlog pos: db1020-bin.000968:1013334195
01:37 hashar: scandium: restarted zuul-merger
01:36 ostriches: ytterbium: Stopped puppet, stopped gerrit process.
01:34 mutante: switched gerrit-new to gerrit in DNS
01:30 ostriches: lead: stopped puppet for a few minutes
01:17 hashar: scandium: migrating zuul-merger repos to lead find /srv/ssd/zuul/git -path '*/.git/config' -print -execdir sed -i -e 's/ytterbium.wikimedia.org/lead.wikimedia.org/' config \;
01:10 hashar: stopping CI
01:09 jynus: reviewdb backup finished, available on db1020:/srv/tmp/2016-07-25_00-54-31/
01:02 ostriches: rsyncing latest git data from ytterbium to lead
00:57 mutante: manually deleted reviewer-counts cron from gerrit2 user, runs as root and puppet does not remove crons unless ensure=>absent
00:55 jynus: starting hot backup of db1020's reviewdb

2016-07-24

02:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 24 02:25:08 UTC 2016 (duration 4m 34s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 08m 59s)

2016-07-23

15:38 godog: stop swift in esams test cluster, lots of logging from there
15:37 godog: lithium sudo lvextend --size +10G -r /dev/mapper/lithium--vg-syslog
04:58 ori: Gerrit is back up after service restart; was unavailable between ~ 04:29 - 04:57 UTC
04:56 ori: Restarting Gerrit on ytterbium
04:48 ori: Users report Gerrit is down; on ytterbium java is occupying two cores at 100%
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jul 23 02:26:49 UTC 2016 (duration 5m 41s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 08m 24s)
01:02 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.11/extensions/CentralAuth/includes/CentralAuthPlugin.php: T141160 (duration: 00m 29s)
01:01 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.11/extensions/CentralAuth/includes/CentralAuthHooks.php: T141160 (duration: 00m 27s)
01:00 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.11/extensions/CentralAuth/includes/CentralAuthPrimaryAuthenticationProvider.php: T141160 (duration: 00m 28s)
00:37 tgr: doing an emergency deploy of https://gerrit.wikimedia.org/r/#/c/300679 for T141160, creates dozens of new users per hour to be unattached on loginwiki which probably has weird consequences

2016-07-22

22:19 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: Enable debug logging for DBTransaction (duration: 00m 38s)
21:10 ejegg: updated civicrm from 2f4805fa2d2a7c57881408be2b3a017d26d8f43e to d657255e1edebeccfc0a03bea70b78eb11375cf8
20:58 ejegg: disabled Worldpay audit parser job
18:59 ejegg: rolled back payments from 79d2b67067fd7e579372b63e0d619eccfa3b9143 to 79cb53998c41f72d0fa49130ed1f66dc112b478c
18:54 mutante: restart grrrit-wm
16:05 Jeff_Green: running authdns-update to correct a DKIM public key on wikipedia.org
15:24 anomie: Starting script to populate empty gu_auth_token phab:T140478
15:16 urandom: T140825: Restarting Cassandra to apply 8MB trickle_fsync (restbase1015-a.eqiad.wmnet)
14:21 gehel: rolling restart of logstash100[1-3] - T141063
14:19 urandom: T134016: Boostrapping restbase2004-c.codfw.wmnet
12:42 jynus: applying new m5 db grants
11:12 jynus: reimage dbproxy1009 T140983
11:04 jynus: applying new m2 db grants
10:47 jynus: reimage dbproxy1007 T140983
10:36 jynus: applying new m1 db grants
10:27 hashar: Restarting Jenkins entirely (deadlocked)
10:23 hashar: Jenkins has some random deadlock. Will probably reboot it
09:45 jynus: reimage dbproxy1006
09:36 jynus: applying new m3 db grants
08:19 jynus: reimage dbproxy1008
06:43 jynus: updating dns records: m3-slave to db1043; m2-master to dbproxy1002
04:08 jynus: backing up, shutting down and reimage db1043
03:14 jynus: stopping db1043 db
03:06 twentyafterfour: restarted apache2 and phd on iridium
03:04 jynus: reverting m3-master dns back to the proxy
02:59 jynus: restarted phd on iridium
02:35 jynus: SET GLOBAL read_only=0; on db1048
02:34 jynus: updating m3-master dns
02:33 jynus: setting db1043 as read-only (phabricator/m3)
02:31 jynus: making dbstore1002.eqiad.wmnet:3306 a child of db1048.eqiad.wmnet:3306
02:27 jynus: making db2012.codfw.wmnet:3306 a child of db1048.eqiad.wmnet
02:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jul 22 02:25:53 UTC 2016 (duration 5m 47s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 08m 23s)
00:53 bd808: Restarted elasticsearch on logstash1003; couldn't find master (even though the master thought 1003 was fine)
00:43 mutante: restarted grrrit-wm
00:01 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings-labs.php: Labs-only cleanups (duration: 00m 25s)

2016-07-21

23:53 Amir1: deploying 2d9817b to ores in scb nodes
23:49 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: (no message) (duration: 00m 24s)
23:46 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#q,298344,n,z (duration: 00m 24s)
23:41 MaxSem: on tin: ran mwscript extensions/ShortUrl/populateShortUrlTable.php --wiki=urwiki
23:39 MaxSem: created ShortUrl tables on urwiki
23:37 ori: Restarted statsv on hafnium (cc Krinkle). 'gaierror: [Errno -3] Temporary failure in name resolution'
23:34 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.11/extensions/CirrusSearch/: https://gerrit.wikimedia.org/r/#q,300430,n,z https://gerrit.wikimedia.org/r/#q,300436,n,z (duration: 00m 32s)
23:32 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.11/extensions/EventBus/: https://gerrit.wikimedia.org/r/#q,300332,n,z (duration: 00m 26s)
23:27 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/299619/ (duration: 00m 24s)
23:22 logmsgbot: maxsem@tin Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/299615/ (duration: 00m 29s)
23:21 Amir1: restarting uwsgi and celery for ores in scb1002
23:20 logmsgbot: maxsem@tin Synchronized dblists/wikidatadescriptions.dblist: https://gerrit.wikimedia.org/r/#/c/299615/ (duration: 00m 24s)
23:19 Amir1: restarting uwsgi and celery for ores in scb 1001
23:09 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/298933/ (duration: 00m 29s)
22:46 ebernhardson: restart elasticsearch on logstash1002
22:22 bd808: Restarted kibana4 on logstash1001 for "node[18588]: segfault at 2fcb25f00009 ip 0000000000ad9846 sp 00007ffe526bbb40 error 4 in node[400000+1383000]"
22:01 mutante: stat1002 - puppetized git pull from "refinery_source" fails
21:11 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Moved WMF specific SiteMatrix data to CommonSettings (duration: 00m 26s)
20:28 ejegg: re-enabled fundraising campaigns after schema update
19:27 logmsgbot: demon@tin Synchronized wikiversions.json: because sync-wikiversions doesn't care about co-masters ugh (duration: 00m 29s)
19:09 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: last wikis to wmf.11
19:07 ejegg: disabled fundraising CentralNotice campaigns for paymentswiki schema update
18:34 ejegg: updated payments-wiki from f23f15656eb488f5008b45b940077abbaa779004 to 79d2b67067fd7e579372b63e0d619eccfa3b9143
17:33 mutante: restarted grrrit-wm
17:26 logmsgbot: krinkle@tin Synchronized w/static.php: allow short-lived caching of 400/500 errors (duration: 00m 24s)
17:15 ostriches: gerrit: restarting
17:13 ostriches: gerrit: killed a couple of long-running git-upload-pack's for mediawiki/core
17:07 gehel: cleaning leftover crons on logstash* servers - T140973
16:51 urandom: T134016: Starting bootstrap of restbase2003-c.codfw.wmnet
16:50 urandom: T134016: Restart of codfw rack 'c' instances to apply stream socket timeout complete
16:47 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase2008-b.codfw.wmnet)
16:46 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.11/extensions/CirrusSearch/includes/Searcher.php: T140950: Deploy UBN fix to CirruSearch (duration: 00m 31s)
16:46 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase2008-a.codfw.wmnet)
16:43 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase2004-b.codfw.wmnet)
16:41 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase200r-a.codfw.wmnet)
16:38 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase2003-b.codfw.wmnet)
16:36 urandom: T134016: Restarting Cassandra to apply new stream timeout (restbase2003-a.codfw.wmnet)
16:30 subbu: finished (test) deploy of parsoid sha ed2f8228
16:28 urandom: Cancelling 2003-c bootstrap, and disabling Puppet on restbase2003.codfw.wmnet to keep instance down : T134016
16:27 subbu: synced parsoid code; restarting parsoid on wtp1001 as a canary
16:24 subbu: starting (test) parsoid deployment
16:17 subbu: aborted (test) parsoid deployment
16:13 subbu: starting parsoid deployment
14:58 jynus: stopping dbstore1002 for scheduled maintenace T119488
14:44 paravoid: cr2-eqiad is now upgraded, passing transit and cross-DC traffic and is the VRRP master in eqiad
14:43 paravoid: cr2-eqiad: restoring VRRP priorities
14:40 paravoid: cr2-eqiad: restoring PyBal BGP sessions
14:39 paravoid: cr2-eqiad: reenabling IX interface & BGP
14:37 paravoid: cr2-eqiad: reenabling Transit interfaces & BGP
14:35 paravoid: cr2-eqiad: enabling Fundraising interface & BGP
14:30 paravoid: cr2-eqiad: reenabling xe-4/2/0 (link to cr1-eqord) and xe-5/2/3 (link to cr2-codfw)
14:26 paravoid: cr2-eqiad: reenabling all asw-*-eqiad interfaces
14:14 logmsgbot: demon@tin Synchronized wmf-config/: extension list cleanups (duration: 00m 34s)
14:07 paravoid: cr2-eqiad: halting both routing engines(!)
14:04 paravoid: cr2-eqiad: disabling xe-4/2/0 (link to cr1-eqord)
14:04 paravoid: cr2-eqiad: disabling xe-5/2/3 (link to cr2-codfw)
14:02 paravoid: cr2-eqiad: disabling all asw-*-eqiad interfaces
13:41 paravoid: cr2-eqiad: fabric upgrade bandwidth for FPC 4/5
13:38 paravoid: cr2-eqiad: toggling mastership between routing-engines (re1->re0)
13:31 paravoid: cr2-eqiad: setting scb 0 to offline and replacing it
13:31 paravoid: cr2-eqiad: setting fabric plane 0/1/2/3 to offline
13:30 paravoid: cr2-eqiad: powering off re0 (backup)
13:28 paravoid: cr2-eqiad: toggling mastership between routing-engines (re0->re1)
13:18 mobrovac: mathoid deploying 36be4ea
13:12 paravoid: cr2-eqiad: setting scb 1 to offline and replacing it
13:10 paravoid: cr2-eqiad: setting fabric plane 5/6/7 to offline
13:10 paravoid: cr2-eqiad: setting fabric plane 4 to offline
13:10 paravoid: cr2-eqiad: setting "chassis state cb-upgrade on" and powering off re1 (backup)
13:01 godog: bounce gerrit on ytterbium
12:58 godog: manually flipping m2-master to db1020
12:49 paravoid: cr2-eqiad: re-enabling GRES and toggling mastership between routing-engines (re1->re0)
12:48 paravoid: cr2-eqiad: fixing IPv6 VRRP interoperatbility between the cr1/cr2 ( http://www.juniper.net/documentation/en_US/junos14.2/topics/concept/vrrpv3-junos-support.html )
12:41 mobrovac: citoid deployed 5134e49e
12:36 mobrovac: change-prop deploying b7079fd9c
12:27 paravoid: cr2-eqiad: rebooting backup RE (re0)
12:19 paravoid: cr2-eqiad: toggling mastership between routing-engines (re0->re1)
12:15 paravoid: cr2-eqiad: setting "chassis network-services enhanced-ip" and rebooting re1 (then re0 will follow)
11:49 paravoid: upgrading cr2-eqiad:re1 and rebooting
11:38 paravoid: cr2-eqiad: toggling mastership between routing-engines (re1->re0)
11:24 paravoid: upgrading cr2-eqiad:re0 and rebooting
11:15 paravoid: cr2-eqiad: deactivate chassis redundancy graceful-switchover
10:59 paravoid: cr2-eqiad: disabling IX/Transit/Fundraising interfaces
10:55 paravoid: cr2-eqiad: deactivating Fundraising BGP session
10:52 paravoid: cr2-eqiad: deactivating Transit BGP sessions
10:50 paravoid: cr2-eqiad: deactivating IX BGP sessions
10:35 paravoid: cr2-eqiad: increase cross-datacenter link OSPF metrics
09:47 gehel: reinstalling and configuring relforge1001/1002 - T137256
08:10 _joe_: restarting apache on palladium
07:47 apergos: restarted gerrit on ytterbium, it was refusing to complete git fetches for large repos (mw core, puppet...)
03:03 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jul 21 03:03:21 UTC 2016 (duration 7m 2s)
02:56 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 08m 57s)
02:31 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 09m 33s)

2016-07-20

23:43 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT, no-op (duration: 00m 24s)
23:42 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.10/extensions/EventBus/EventBus.hooks.php: Add rev_by_bot flag to revision_create event (2/2) (duration: 00m 23s)
23:41 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.10/extensions/EventBus/extension.json: Add rev_by_bot flag to revision_create event (1/2) (duration: 00m 26s)
23:40 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.11/extensions/ORES/: Let ORES extension score for some namespaces instead of all (Gerrit:300083]) (duration: 00m 30s)
23:38 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: ORES score edits in main and Property namespaces in wikidatawiki (Gerrit:300086) (duration: 00m 33s)
21:59 mutante: new language "tcy" (Tulu) has been approved and added today - https://meta.wikimedia.org/wiki/Requests_for_new_languages/Wikipedia_Tulu
21:49 mutante: DNS authdns-gen-zones on all servers to add new language tcy (bug T97051)
20:28 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.11/extensions/PagedTiffHandler/: (no message) (duration: 00m 25s)
20:25 ejegg: updated civicrm from a386eb5a76ec97b3b01c46a49309dfa39bbc58b0 to 2f4805fa2d2a7c57881408be2b3a017d26d8f43e
20:20 awight: update paymentswiki from 7c6fb5a3b90fffdf2229cc903fb546e0e1e47998 to f23f15656eb488f5008b45b940077abbaa779004
20:06 logmsgbot: demon@tin Finished scap: group1 to wmf.11 (3rd bestest try) (duration: 46m 03s)
19:38 urandom: Starting Casssandra on restbase1011-b.eqiad.wmnet
19:20 logmsgbot: demon@tin Started scap: group1 to wmf.11 (3rd bestest try)
19:18 logmsgbot: demon@tin scap aborted: group1 to wmf.11 (2nd best try) (duration: 01m 30s)
19:17 logmsgbot: demon@tin Started scap: group1 to wmf.11 (2nd best try)
19:12 logmsgbot: demon@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript mergeMessageFileList.php --wiki="fawiki" --list-file="/srv/mediawiki-staging/wmf-config/extension-list" --output="/tmp/tmp.KjzXNQRvAU" ' returned non-zero exit status 1 (duration: 00m 28s)
19:12 logmsgbot: demon@tin Started scap: group1 to wmf.11
17:23 urandom: Stopping restbase1013-c bootstrap (pending better timeouts) : T134016
16:09 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: More to extension registration for l10n (duration: 00m 27s)
16:09 logmsgbot: thcipriani@tin Finished scap: SWAT: Fix spelling of RevisionSlider (T140875) (duration: 23m 18s)
15:52 urandom: Resuming failed bootstrap on restbase2003.codfw.wmnet : T134016
15:48 _joe_: removed ruthenium from the list of trebuchet minions
15:46 logmsgbot: thcipriani@tin Started scap: SWAT: Fix spelling of RevisionSlider (T140875)
15:41 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.11/extensions/RevisionSlider/modules/ext.RevisionSlider.HelpDialog.js: SWAT: Open links in the "tutorial" in the new window (T140875) (duration: 00m 27s)
15:32 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: Beta: Test OresEnabledNamespaces on enwiki (duration: 00m 25s)
15:28 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Let bureaucrats in fawiki remove sysop user group (T140810) (duration: 00m 25s)
15:26 mafk: SWAT for 299446 done, which fixes T140544
15:20 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Configuration changes for he.wikinews.org (duration: 00m 28s)
15:08 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Compact Language Links: To beta in ruwikivoyage (duration: 00m 33s)
15:04 urandom: Ammending my last to include 'Rack b'
15:03 urandom: RESTBase Cassandra: raising stream throughput to 25Mbit/s; lowering compaction throughput to 10MB/s : T134016
14:40 Jeff_Green: running authdns-update to deploy SPF/DKIM records for wikipedia.org
14:20 urandom: Restarting bootstrap on restbas1013.eqiad.wmnet (duplicate resume?) : T134016
14:05 urandom: Resuming failed bootstrap on restbase1013-c.eqiad.wmnet : T134016
13:30 urandom: Performing rolling RESTBase restart to work-around Cassandra instance restart fallout : T138314 and T138314
13:28 urandom: Restarting restbase1008-a.eqiad.wmnet to apply a (ephemeral) 7200000ms streaming timeout : T138314
13:15 paravoid: cr2-eqiad: setting VRRP priority to 50 for all subnets, effectively switching the VRRP master to cr1-eqiad
13:12 _joe_: transitioning wtp2011-2020
12:35 _joe_: transitioning wtp2002-2010
12:35 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=wtp2001.codfw.wmnet
12:26 _joe_: transition ongoing on wtp2001
12:23 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=wtp2001.codfw.wmnet
12:22 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=
12:17 _joe_: disabling puppet on all parsoid hosts for the transition to service-runner T90668
07:28 elukey: restarting evenbus on kafka100[12] (T140848)
06:14 _joe_: updating parsoid on wtp100[12]
04:55 mutante: osmium package chromium-browser is missing after upgrade, refered to by jsbench
04:45 mutante: osmium - rsyncing /home , /srv (except /srv/mediawiki created by puppet) back from temp backup on hafnium
04:38 mutante: osmium result: boots into 4.4 kernel which would not work before.. lol
04:35 mutante: osmium edit grub config to boot second entry (3.16), update-grub, reboot
03:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jul 20 03:00:43 UTC 2016 (duration 6m 50s)
02:53 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.11) (duration: 07m 55s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 41s)
02:28 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: delist codereview (duration: 00m 27s)
01:39 logmsgbot: demon@tin Synchronized tests/SiteConfiguration.php: for completeness (duration: 00m 24s)
01:35 mutante: hafnium stopping rsyncd, deleting configs
01:33 mutante: labstore1005, accepting salt key (reinstall 2016-06-25)
01:29 mutante: rhodium, new puppetmaster, add to salt
01:19 mutante: osmium - revoke old puppet cert, salt-key .. sign new ones
01:19 mutante: osmium - after reinstall with jessie, did not boot with 4.4 kernel, _does_ boot with 3.16.04.. still jessie just booted manually into the older kernel in grub
01:13 logmsgbot: demon@tin Synchronized wmf-config/abusefilter.php: Disable abusefilter profiling on commonswiki (duration: 00m 26s)
00:44 logmsgbot: dereckson@tin Finished scap: wmf-config/ upgrade: Gerrit changes 296770, 296767, 296929, 296930, 292623, 292624 (duration: 45m 42s)
00:42 urandom: Temporarily reducing compaction throughput to 10MB/s on restbase1013-c.eqiad.wmnet : T134016
00:32 mutante: osmium - reboot into PXE, reinstall

2016-07-19

23:58 logmsgbot: dereckson@tin Started scap: wmf-config/ upgrade: Gerrit changes 296770, 296767, 296929, 296930, 292623, 292624
22:17 awight: update paymentswiki to fundraising/REL1_27 from e8b600c518b28e3f350ced85d7d1006a76b86596 to 7c6fb5a3b90fffdf2229cc903fb546e0e1e47998
22:10 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/299870/ (duration: 00m 29s)
21:54 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.10/extensions/AbuseFilter: Backported fix for logspam (duration: 00m 38s)
21:43 gwicke: temporarily lowering compaction throughput on all eqiad restbase cassandra instances from 60mb/s to 20mb/s via `nodetool setcompactionthroughput 20` (T140825)
21:23 gwicke: temporarily lowered compaction throughput on all 1012 instances from 60mb/s to 20mb/s via `nodetool setcompactionthroughput 20` (T140825)
21:08 urandom: Bootstrapping restbase2003-c.codfw.wmnet : T134016
20:45 urandom: Lowering compaction throughput to 20MB/s on restbase1013-{a,b}.eqiad.wmnet : T134016
20:44 urandom: Lowering compaction throughput from 35MB/s to 20MB/s on restbase1013-c.eqiad.wmnet : T134016
20:28 urandom: Throttling stream throughput to 20MB/s on all rack 'b' instances : T134016
20:21 urandom: Lowering compaction throughput from 45MB/s to 35MB/s on restbase1013-c.eqiad.wmnet : T134016
20:06 urandom: Disabling Puppet on restbase2003.codfw.wmnet : T134016
20:03 logmsgbot: demon@tin Finished scap: group0 to wmf.11 (duration: 24m 52s)
19:57 urandom: Reducing stream throughput on restbase1013-{a,b} to 20MB/s : T134016
19:47 urandom: Lowering compaction throughput from 60MB/s to 45MB/s on restbase1013-c.eqiad.wmnet : T134016
19:38 logmsgbot: demon@tin Started scap: group0 to wmf.11
19:31 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.9
19:31 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.8
19:31 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.7
19:31 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.7
19:31 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.6
19:12 urandom: Starting bootstrap of restbase1013-c.eqiad.wmnet : T134016
18:59 urandom: Disabling puppet on restbase1013.eqiad.wmnet : T134016
17:56 paravoid: cr1-eqiad: restart chassis-control immediately (should not be traffic affecting)
17:49 jynus: applying new grants to m3 dbs in preparation for db1043 failover/proxy implementation
17:43 mdholloway: mobileapps deployed aa9115a
17:41 mdholloway: starting mobileapps deployment
17:15 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Don't set everywhere, breaks internal to us but external to MW requests (eg gerrit, ocg, etc) (duration: 00m 25s)
17:14 cmjohnson1: dbstore1002 swapping disk at slot 6
16:18 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=wtp1002.*,cluster=parsoid,dc=eqiad
16:01 cmjohnson1: swapping pem0-3 on cr1-eqiad
15:49 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.10/extensions/ContentTranslation/includes/AbuseFilterCheck.php: SWAT: Avoid accessing private $filters field (T139657) (duration: 00m 26s)
15:41 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=wtp1001.*,cluster=parsoid,dc=eqiad
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove Echo transition flags PART II (duration: 00m 30s)
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove Echo transition flags PART I (duration: 00m 26s)
15:18 cmjohnson1: replacing PEM0-3 cr2-eqiad
14:23 logmsgbot: oblivian@palladium conftool action : set/pooled=no:weight=15; selector: cluster=parsoid,name=wtp100[12].*
14:22 logmsgbot: oblivian@palladium conftool action : set/weight=0; selector: cluster=parsoid,name=wtp100[12].*
14:10 jynus: reboot and reimage dbproxy1003 to jessie T125027 T138460
13:48 jynus: restarting dbproxy1005 for kernel upgrade
13:44 jynus: reloading dbproxy1001 to repool db1001 as pasive backend
10:37 elukey: sent SIGHUP to eventbus on kafka100[12] to reload schemas
10:01 jynus: testing haproxy start sequence on dbproxy1005 (unused proxy)
10:01 mobrovac: scb disabling puppet
09:10 godog: upgrade slapd to 2.4.41+dfsg-1+wmf1 on serpens - T130593
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jul 19 02:30:08 UTC 2016 (duration 6m 2s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 37s)

2016-07-18

23:55 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/292622/ (duration: 00m 25s)
23:54 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/292622/ (duration: 00m 24s)
23:54 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/292622/ (duration: 00m 25s)
23:47 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/292621/ (duration: 00m 29s)
23:46 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/292621/ (duration: 00m 31s)
23:39 logmsgbot: maxsem@tin Synchronized wmf-config/: https://gerrit.wikimedia.org/r/#/c/292620/ part 2 (duration: 00m 27s)
23:38 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/292620/ part 1 (duration: 00m 26s)
23:31 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/292619/ part 2 (duration: 00m 24s)
23:30 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/292619/ part 1 (duration: 00m 26s)
23:19 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.10/extensions/MultimediaViewer/: https://gerrit.wikimedia.org/r/#/c/299560/ (duration: 00m 26s)
23:17 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.10/extensions/Kartographer/: https://gerrit.wikimedia.org/r/#/c/299560/ (duration: 00m 27s)
23:15 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/296673/ (duration: 00m 30s)
22:58 bawolff: deploy fix for T129738 to php-1.28.0-wmf.10
22:57 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: rm deprecated fundraising config (duration: 00m 26s)
22:51 mutante: restarted grrrit-wm
22:48 ostriches: lead: puppet turned back on
22:44 dapatrick: Deployed patches for T133147 to wmf.10
22:40 ostriches: lead: disabled puppet for a bit to test some CSS tweaks live.
22:35 mutante: gerrit-new restarting for config change 298710
22:34 dapatrick: Deployed patch for T132926 to wmf.10
22:32 mutante: gerrit restarting for config change 298710
22:18 bawolff: Deployed patch for T136402 on php-1.28.0-wmf.10
22:07 gehel: elasticsearch / kibana upgrade done
22:01 dapatrick: Deployed patch for T115333 to wmf.10
21:52 ebernhardson: brought up kibana4 on logstash.wikimedia.org
21:46 logmsgbot: demon@tin Finished scap: security, se-curity (duration: 08m 12s)
21:38 logmsgbot: demon@tin Started scap: security, se-curity
21:34 ebernhardson: changed replica count for logstash-2016.06-(01|02|03|15|16|17) indices back to 2
21:09 ebernhardson: changed replica count for logstash-2016.06-(01|02|03|15|16|17) indices to 0 to make room for recovering todays index
21:07 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: whitelist some rss feeds for mw.org (duration: 00m 43s)
21:01 ejegg: updated payments from 8d3873f8d6b0600331775e9ccfc0cf4c6ed1e181 to e8b600c518b28e3f350ced85d7d1006a76b86596
20:18 ebernhardson: installed elasticsearch-2.3.3 to logstash1001-6
20:02 ebernhardson: re-shutdown elasticsearch on logstash1001-6
19:55 ejegg: updated payments from 0c14940f4930e94a9287acae978cc6e661e54ee1 to 8d3873f8d6b0600331775e9ccfc0cf4c6ed1e181
19:54 ebernhardson: re-stopping logstash on logstash1001-3
19:52 gehel: disabling puppet on logstash.* nodes for elasticsearch upgrade
19:49 mutante: ytterbium - fixing Apache config, graceful
19:47 ebernhardson: shutdown elasticsearch on logstash1004-6
19:38 bd808: Dropped logstash-2016.07.04 through logstash-2016.07.14 indices for backing Elasticsearch upgrade
19:36 ebernhardson: shutting down logstash and elasticsearch on logstash1001-03
19:05 logmsgbot: demon@tin Synchronized private/: remove obsolete wikitech config file (duration: 00m 32s)
19:04 gehel: starting elasticsearch upgrade for logstash (T136001)
18:57 logmsgbot: demon@tin Synchronized wmf-config/: Pruning 1.27.0-wmf.N ExtensionMessages files (duration: 00m 34s)
18:08 ejegg: rolled back payments to 0c14940f4930e94a9287acae978cc6e661e54ee1
18:05 ejegg: updated payments from 0c14940f4930e94a9287acae978cc6e661e54ee1 to 8d3873f8d6b0600331775e9ccfc0cf4c6ed1e181
17:28 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: globally set $wgHTTPProxy (duration: 00m 26s)
17:18 mobrovac: moblieapps deploying debb3f6
17:08 gehel: updated wdqs to latest version, new blazegraph version, restart of wdqs-updater
15:23 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=mw1261.*
15:21 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=mw1261.*
15:10 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable global abuse filters on ptwiki (T140395) (duration: 00m 38s)
15:09 hashar: gallium upgrading Zuul: zuul_2.1.0-151-g30a433b-wmf3precise1 zuul_2.1.0-151-g30a433b-wmf4precise1_amd64.deb . To support layout validation when multiple connections are used
14:49 mobrovac: mobileapps deploying dfe5f11f5
14:11 mobrovac: moblieapps deploying fb65cea
12:03 hashar: Gerrit was slow processing requests such as git pull since 11:17 UTC . Fixed by killing all idling/waiting tasks T140604
11:08 godog: swift codfw-prod: ms-be202[567] weight 3000 - T136630
10:10 jynus: hard reset for db2056 T140598
08:54 godog: swift eqiad-prod: ms-be102[3-6] to weight 500 - T136631
08:31 hashar: gallium: upgrading Zuul 2.1.0-95-g66c8e52-wmf1precise1 .. zuul_2.1.0-151-g30a433b-wmf3precise1 T137525
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jul 18 02:26:24 UTC 2016 (duration 5m 43s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 23s)

2016-07-17

10:31 godog: restart slapd on serpens - T130593
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 17 02:26:21 UTC 2016 (duration 5m 42s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 24s)

2016-07-16

23:16 Krenair: testing that the SAL bot is still working
20:42 logmsgbot: twentyafterfour@tin Synchronized wmf-config/interwiki.php: (no message) (duration: 00m 26s)
20:41 twentyafterfour: deploying interwiki https config change https://gerrit.wikimedia.org/r/#/c/299299 refs T140206
20:32 awight: update paymentswiki from 25c97ba0f27b61859f90fd205c53d587c2838fec to 0c14940f4930e94a9287acae978cc6e661e54ee1
18:31 awight: enable LogCompleted for Ingenico
18:27 awight: update paymentswiki from 8bf6e911eb43a2d369bf656f07d1b51be0a54f6c to 25c97ba0f27b61859f90fd205c53d587c2838fec
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jul 16 02:27:00 UTC 2016 (duration 5m 44s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 08m 22s)

2016-07-15

21:12 awight: reenable donation queue
21:11 awight: update civicrm from cea316cc57c511c645a92a003028c95e19cac877 to a386eb5a76ec97b3b01c46a49309dfa39bbc58b0
20:32 awight: disable donations queue consumer
18:29 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I8bf7c8dd: Lower default $wgSquidMaxage from 31 days to 14 days (duration: 00m 39s)
15:49 mobrovac: restbase deploy end of 731284b
15:37 mobrovac: restbase deploy start of 731284b
15:29 ottomata: restarting hadoop-mapreduce-historyserver to apply yarn log aggreation retention settings
13:40 godog: stress-test spinning disks on ms-be102[3-6]
12:08 bblack: varnish: rolling frontend restarts for text+upload done
11:56 bblack: varnish: starting rolling, depooled restart of text and upload frontend caches
11:03 godog: swift codfw-prod: ms-be202[567] weight 2500
10:30 twentyafterfour: deployed rPHABacb736547c6595fe09e05bafd7a3b563d3cf67c8 and rPHABcf12fdf248df82dc414d96bddd147c058bc3d636 to address maniphest task dependency graphs. Now related tasks will be shown as a plain list when there are too many tasks to graph.
10:15 mobrovac: restbase deploy end of 018864b
10:04 mobrovac: restbase deploy start of 018864b
09:33 jynus: renabling semisync replication throughout s4
09:31 jynus: restarted circular replication from db2019 -> db1040
09:22 jynus: updating dns record for s4-master.eqiad.wmnet
08:13 jynus: reimporting x1 partial db copy on dbstore1002 from x1-master
07:30 moritzm: installing PHP security updates on jessie systems
07:27 _joe_: powercycling mw1280
06:47 moritzm: installing nspr security updates
06:08 moritzm: installing libarchive security updates
02:35 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jul 15 02:35:16 UTC 2016 (duration 6m 14s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 07m 54s)

2016-07-14

23:11 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.10/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T137169: Turn of TextCat A/B test (duration: 00m 34s)
21:08 matt_flaschen: Started backfillReadBundles.php on all group 2 wikis
21:07 matt_flaschen: Started backfillUnreadWikis.php --rebuild on all group 2 wikis
20:40 yurik: restarted graphoid with the new settings, enabling geoshape protocol
20:02 ottomata: restarting hadoop-yarn-resourcemanager on analytics1002 and then analytics1001 to apply yarn log aggregation change
19:56 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.10/extensions/RevisionSlider: touching js and resource files (duration: 00m 28s)
19:22 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings-labs.php: maps geoshapes stuff for yurik (labs file for completeness) (duration: 00m 27s)
19:21 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: maps geoshapes stuff for yurik (duration: 00m 31s)
19:19 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: Move remaining wikis to wmf.10
19:00 urandom: Dropping legacy Cassandra system_auth tables in RESTBase production to complete RBAC conversion : T139639
15:52 logmsgbot: aude@tin Synchronized wmf-config/CommonSettings-labs.php: (no message) (duration: 00m 32s)
15:50 logmsgbot: aude@tin Synchronized dblists/clldefault.dblist: Enable compact language lists on more wikis (duration: 00m 51s)
15:40 jynus: shutting down es2018, pc2004, es2005 for hardware maintenance T139714
15:35 logmsgbot: aude@tin Finished scap: Update i18n for RevisionSlider (duration: 46m 58s)
15:08 jynus: shutting down es2014, es2015, es2016 for hardware maintenance T139714
14:56 bblack: cache_misc: manually raised default_ttl to 3600 (to match https://gerrit.wikimedia.org/r/#/c/298970/ without restarts)
14:48 logmsgbot: aude@tin Started scap: Update i18n for RevisionSlider
14:48 jynus: shutting down es2011, es2012, es2013 for hardware maintenance T139714
14:47 logmsgbot: aude@tin scap aborted: (no message) (duration: 00m 02s)
14:47 logmsgbot: aude@tin Started scap: (no message)
14:45 logmsgbot: aude@tin Synchronized wmf-config/CommonSettings.php: Enable RevisionSlider on test wikis (duration: 00m 28s)
14:44 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Enable RevisionSlider on test wikis (duration: 00m 27s)
14:38 logmsgbot: aude@tin Synchronized wmf-config/extension-list: Add RevisionSlider to extension-list (duration: 00m 42s)
09:58 godog: powercycle ms-be1012, adding back replaced disk
08:40 Amir1: for ores
08:40 Amir1: deploying 0e9555f to scb nodes
08:39 elukey: restarted hhvm on mw1289 mw1280 mw1288 mw1284 mw1287
08:18 jynus: running "megacli -PDOffline -PhysDrv '[32:6]' -aALL" on dbstore1002 to debug issue T140337
08:06 elukey: upgrading cache misc to varnishkafka 1.0.11-1
08:03 _joe_: removing appservers mw1018-25 from service via conftool for decommissioning (T139353)
08:01 elukey: removing api servers mw111[4-9] from service via conftool as first decom step (T139353)
07:55 elukey: removing api servers mw112[0-9] from service via conftool as first decom step (T139353)
06:45 moritzm: restarted hhvm on mw1170
06:07 moritzm: upgrading hhvm in codfw
04:22 twentyafterfour: Phabricator hotfix: applied patch to disable task graph on tasks with > 100 related tasks.
03:15 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jul 14 03:14:57 UTC 2016 (duration 7m 16s)
03:07 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.10) (duration: 15m 50s)
02:37 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 15m 17s)
00:36 matt_flaschen: Ran backfillReadBundles on labtestwiki
00:35 matt_flaschen: Started backfillReadBundles on labswiki
00:24 matt_flaschen: Started backfillUnreadWikis --rebuild and backfillReadBundles for all group 0 and group 1 wikis earlier
00:06 twentyafterfour: Phabricator maintenance completed. Service restored
00:01 twentyafterfour: preparing to take Phabricator offline momentarily for scheduled maintenance / upgrade. Service should be restored within a couple of minutes.

2016-07-13

23:31 eileen: update CiviCRM from 0898bb9360fe4a5ddea1a41d4e3f3e9823afee27 to cea316cc57c511c645a92a003028c95e19cac877
23:27 logmsgbot: krenair@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/298822 (duration: 00m 26s)
23:25 logmsgbot: krenair@tin Synchronized static/images/project-logos: update for https://gerrit.wikimedia.org/r/298819 and https://gerrit.wikimedia.org/r/298822 (duration: 00m 24s)
23:18 logmsgbot: krenair@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/225509 + https://gerrit.wikimedia.org/r/298899 - create https://meta.wikimedia.org/wiki/Special:Contact/stewards (duration: 00m 26s)
23:17 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.10/includes/resourceloader/ResourceLoaderStartUpModule.php: I882bf7075: ResourceLoader: Update expected length of module version hash (duration: 00m 25s)
21:55 mutante: ytterbium - puppet enabled again, fix deployed
21:48 mutante: ytterbium, disabled puppet, started apache, needs fix
20:40 logmsgbot: demon@tin Synchronized README: no-op to bring co-masters in sync (duration: 00m 28s)
20:28 bearND: deployed mobileapps d1eb1da
20:20 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 to wmf.10
20:19 bearND: starting mobileapps deploy
20:03 ebernhard|lunch: Update codfw elasticsearch cluster sttings with cluster.routing.allocation.disk.watermark.low: 70% to match eqiad and reduce free space icinga warnings
19:28 ostriches: gerrit: restarting, puppet back on, issue fixed.
19:23 logmsgbot: anomie@tin Synchronized php-1.28.0-wmf.10/includes/auth/AuthManager.php: Add timing data logging for T119736 (duration: 00m 28s)
19:23 logmsgbot: anomie@tin Synchronized php-1.28.0-wmf.8/includes/auth/AuthManager.php: Add timing data logging for T119736 (duration: 00m 27s)
19:14 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.8/includes/api/ApiQueryRecentChanges.php: API: Remove index forcing in ApiQueryRecentChanges - T140108 (duration: 00m 26s)
19:04 logmsgbot: demon@tin rebuilt wikiversions.php and synchronized wikiversions files: all group0 to wmf.10
18:41 ostriches: gerrit/ytterbium: flapped for a minute because of incompat 2.12/2.8 config. Working, puppet disabled pending real fix.
18:40 mutante: gerrit has a temp problem. maintenance going on
18:34 logmsgbot: krenair@tin Synchronized wmf-config: labs-only change, should be a noop here: https://gerrit.wikimedia.org/r/298812 (duration: 00m 27s)
18:32 mutante: gerrit will restart shortly for a config change. expect a very short downtime
18:22 legoktm: checkLocalUser.php finished, starting run #2 now
17:58 logmsgbot: legoktm@tin Synchronized wmf-config/: PoolCounterClient.php -> extension.json (duration: 00m 32s)
17:44 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.10/extensions/WikimediaEvents/WikimediaEventsHooks.php: Include the namespace for all pages & Include the resolved special page name for special pages - T138500 (duration: 00m 36s)
17:41 logmsgbot: demon@tin Finished scap: wmf.10 code sync + testwiki to wmf.10 for l10n cache gen (once more with feeling) (duration: 47m 06s)
17:35 ejegg: turned on cURL verbose logging for AstroPay requests on payments
17:27 jynus: drop databases fab_migration, percona and test from m3 T138460
17:25 _joe_: restarting hhvm on mw1229 (stuck in HPHP::Treadmill::getAgeOldestRequest)
17:22 urandom: Starting restbase on restbase1013.eqiad.wmnet
16:54 logmsgbot: demon@tin Started scap: wmf.10 code sync + testwiki to wmf.10 for l10n cache gen (once more with feeling)
16:50 ejegg: updated CentralNotice for cookie cleanup
16:48 logmsgbot: ejegg@tin Synchronized php-1.28.0-wmf.8/extensions/CentralNotice/: (no message) (duration: 01m 52s)
16:47 logmsgbot: demon@tin scap failed: OSError [Errno 1] Operation not permitted: '/var/lock/scap' (duration: 00m 00s)
16:41 logmsgbot: demon@tin scap aborted: wmf.10 code sync + testwiki to wmf.10 for l10n cache gen (duration: 00m 37s)
16:40 logmsgbot: demon@tin Started scap: wmf.10 code sync + testwiki to wmf.10 for l10n cache gen
16:39 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.9
16:38 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.7
16:38 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.6
16:37 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.5
16:19 hashar: CI slightly overloaded / backloaded due to a long tail of Wikibase changes sent in Gerrit.
16:03 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings-labs.php: SWAT: [Beta] Change ORES thresholds in beta (duration: 00m 29s)
16:01 logmsgbot: thcipriani@tin Synchronized wmf-config/LabsServices.php: SWAT: [Beta] Parsoid: direct traffic to deployment-parsoid07 (duration: 00m 26s)
15:30 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES review tool for Turkish Wikipedia (T139992) (duration: 00m 28s)
15:11 urandom: Stopping Staging dumps : T139639
14:44 urandom: Starting offset dump runs from {xenon,cerium,praseodymium}.eqiad.wmnet : T139639
14:38 urandom: Restarting Cassandra on xenon.eqiad.wmnet : T139639
14:32 urandom: Restarting RESTBase on xenon.eqiad.wmnet : T139639
14:29 yurik: deployed kartotherian https://gerrit.wikimedia.org/r/#/c/298731/ & tilerator https://gerrit.wikimedia.org/r/#/c/298732/
14:25 moritzm: depooling mw1298 (image scaler) for some tests
14:21 godog: reboot ms-be1012, many mkfs.xfs stuck on broken sdh
14:17 logmsgbot: oblivian@palladium conftool action : delete; selector: cluster=rcstream
14:16 gehel: shutting down elastic1001-1016 (T139758)
14:16 logmsgbot: oblivian@palladium conftool action : delete; selector: cluster=rcstream
14:13 yurik: about to deploy updated kartotherian & tilerator for node 4.4.6
14:09 urandom: Dropping legacy system_auth tables in staging to complete RBAC conversion : T139639
14:05 bblack: rcstream cleanup done, puppet re-enabled on relevant lvs and rcs100x
13:58 gehel: cleanup puppet / salt from old elasticsearch servers elastic1001-1016 (T139758)
13:58 hashar: T137525 reverted Zuul back to zuul_2.1.0-95-g66c8e52-wmf1precise1_amd64.deb . It could not connect to Gerrit reliably
13:45 ottomata: restarting hadoop nodemanagers to apply log aggregation retention check interval change
13:43 bblack: restarting pybal on primary eqiad high-traffic2 (lvs1002)
13:41 moritzm: upgrading hhvm on remaining appservers in eqiad and codfw
13:34 gehel: disabling puppet and stopping elasticsearch on elastic1001-1016 (T139758)
13:30 bblack: disabling puppet on rcs100[12] for rcstream cleanup
13:29 bblack: disabling puppet on eqiad high-traffic2 lvs for rcstream cleanup
13:20 gehel: scheduling icinga downtime on elastic1001-1016 prior to decommissioning (T139758)
13:01 elukey: upgrading cache maps to varnishkafka 1.0.11-1
13:00 elukey: uploaded varnishkafka 1.0.11-1 to jessie-wikimedia experimental
12:53 hashar: CI is processing with Zuul 2.1.0-151-g30a433b. It might stop processing events at anytime though due to T137525
12:36 hashar: T137525 Upgrading Zuul 2.1.0-95-g66c8e52-wmf1precise1 ... zuul_2.1.0-151-g30a433b-wmf1precise1_amd64.deb
11:52 elukey: installing varnishkafka_1.0.11-1 on cp3008.esams to test it before the complete rollout
11:38 paravoid: cr1-ulsfo: "restart snmp" to fix SNMP hiccup after reboot
11:38 paravoid: cr1/2-ulsfo: disabling flow monitoring
04:22 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: revert http logging change (duration: 00m 31s)
04:20 logmsgbot: legoktm@tin Synchronized wmf-config/interwiki.php: Update interwiki map, make them HTTPS (duration: 00m 39s)
03:44 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Log 'http' at warning level to debug transwiki import errors (duration: 00m 29s)
02:34 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jul 13 02:34:52 UTC 2016 (duration 6m 8s)
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 10m 11s)
00:56 ori: Restarted grrrit-wm
00:41 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.8/extensions/VisualEditor/: https://gerrit.wikimedia.org/r/#/c/298677/ (duration: 01m 01s)
00:07 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: getCentralAuthToken visibility back to protected (Gerrit:298661) (duration: 00m 27s)

2016-07-12

23:43 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add flow-create-board for gomwiki sysop (T139226) (duration: 00m 27s)
23:41 mutante: lithium deleted some logs older than 60 days to make space
23:33 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable lazy loaded references and images on Thai wikipedia (T136731) (duration: 00m 38s)
23:29 mutante: stat1003 still on every puppet run a mongodb gets started..over and over again
23:12 ejegg: updated payments from d9f7027340e5311f38c4224c2fddde087467df87 to 8bf6e911eb43a2d369bf656f07d1b51be0a54f6c
23:12 ostriches: lead: puppet disabled for a bit while index building is in progress.
23:12 ostriches: ytterbium: puppet enabled again, all happy again
22:56 ostriches: ytterbium: disabled puppet for a moment so we can do a config change w/o gerrit restarting itself
22:06 eileen: upgrade Civicrm from f7434730ebd87f6d542c34c080c61eb3f21ccc6b to 0898bb9360fe4a5ddea1a41d4e3f3e9823afee27
21:52 eileen: Updating CiviCRM from 415d7e62bc3bcbd7c5e3682da64ee4847ad63f5b to f7434730ebd87f6d542c34c080c61eb3f21ccc6b
21:24 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T140144: Echo/CentralAuth: Bail if not fully initialized (duration: 00m 49s)
20:18 urandom: Start revision culling script for local_group_wikipedia_T_parsoid_html, from restbase1009.eqiad.wmnet : T140008
19:30 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T119736: T140144: Troubleshoot why Echo is still triggering CA failures (duration: 00m 39s)
19:02 mutante: git pulled on strontium to sync with palladium
18:48 matt_flaschen: Started checkLocalUser.php at ~2016-07-12 17:45 UTC, killed ~18:06 since Echo apparently is not fully fixed after all.
18:38 legoktm: foreachwiki ../../../../home/legoktm/checkLocalUser.php --delete=1 --verbose=1 on terbium
18:24 logmsgbot: anomie@tin Synchronized php-1.28.0-wmf.9/includes/auth/AuthManager.php: Commit transaction after auto-creating a user gerrit:298541 (duration: 00m 29s)
18:22 logmsgbot: anomie@tin Synchronized php-1.28.0-wmf.8/includes/auth/AuthManager.php: Commit transaction after auto-creating a user gerrit:298540 (duration: 00m 30s)
18:08 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: turn cx back on (duration: 00m 29s)
17:44 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.8/extensions/ContentTranslation/: ping limiter fixes (duration: 00m 29s)
17:37 jynus: out of band ALTER TABLE recentchanges ADD KEY `name_type_patrolled_timestamp` on db1054 T140108
16:58 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: prep pinglimiter config for content translation (duration: 00m 33s)
16:56 Amir1: deploying ores f472f65 to scb
16:51 Amir1: deploying ores f472f65 to scb2001
16:42 godog: disable puppet on ms-fe* and re-enable gradually to apply https://gerrit.wikimedia.org/r/#/c/298297/
16:25 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.8/extensions/Echo/includes/ForeignWikiRequest.php: T119736: ForeignWikiRequest: Bail early for non-global users (duration: 00m 32s)
16:12 logmsgbot: demon@tin Synchronized wmf-config/InitialiseSettings.php: Disable content translation, outage right now (duration: 00m 29s)
15:44 bblack: cache nodes: salt manual removal of vm compaction cron via sed ( https://gerrit.wikimedia.org/r/298499 )
15:33 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.9/extensions/Echo/includes/ForeignWikiRequest.php: SWAT: ForeignWikiRequest: Bail early for non-global users (T119736) (duration: 00m 31s)
15:29 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bumping portals to master (T128546) (duration: 00m 29s)
15:28 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T128546) (duration: 00m 29s)
15:20 bblack: upgrading nginx to 1.11.2-1+wmf1 on all caches
15:18 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/trwikimedia.png: SWAT: Revert Logo update for trwikimedia (T140015) (duration: 00m 29s)
15:09 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/trwikimedia.png: SWAT: Logo update for trwikimedia (T140015) (duration: 00m 33s)
14:58 bblack: upgrading nginx to 1.11.2-1+wmf1 on cache_maps
14:33 elukey: Rebuild new AQS Cassandra cluster (aqs100[456]) to remove previous testing settings (no prod traffic is served)
14:23 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: even more extension.json (duration: 00m 26s)
14:17 bblack: nginx 1.11.2-1+wmf1 uploaded to carbon
14:17 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: moar extension.json (duration: 00m 26s)
14:04 bblack: lvs nodes: apt-get install linux-meta
13:52 bblack: lvs nodes: apt-get upgrade to latest (various base system packages)
13:49 bblack: cache nodes: apt-get upgrade to latest (just 3.16 kernel)
13:05 ottomata: restarting nodemanagers on analytics 1039 1046 and 1054
10:45 godog: terbium:~# lvextend --size +70G -r /dev/mapper/terbium--vg-root T139786
09:35 gehel: lowering elasticsearch codfw high watermark to rebalance cluster
09:32 godog: reboot ms-be3004 / high load average and xfs unhappy
09:22 godog: progressively delete esams swift containers, unused and not in production
07:49 elukey: removing api servers mw113[0-9] from service via conftool as first decom step (T139353)
06:31 logmsgbot: legoktm@tin Synchronized wmf-config/CommonSettings.php: Don't block logins if CentralAuthUser::queryAttached() fails - T119736 (duration: 00m 27s)
06:00 legoktm: running checkLocalUsers.php on terbium
04:37 ori: Reverted all wikis to wmf8 due to tenfold increase in T119736
04:35 logmsgbot: ori@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jul 12 02:26:41 UTC 2016 (duration 5m 31s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 08m 44s)

2016-07-11

23:49 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/resources/: https://gerrit.wikimedia.org/r/#/c/298402/ (duration: 00m 28s)
23:36 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/extensions/Echo/: https://gerrit.wikimedia.org/r/#/c/298400/ (duration: 00m 33s)
23:32 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/extensions/Citoid/: https://gerrit.wikimedia.org/r/#/c/298327/ (duration: 00m 27s)
23:30 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/extensions/Wikidata: https://gerrit.wikimedia.org/r/#/c/298386/ (duration: 02m 00s)
23:15 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/297964/ (duration: 00m 32s)
23:11 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/297556/ (duration: 00m 28s)
23:07 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/extensions/Kartographer/: https://gerrit.wikimedia.org/r/#/c/297556/ (duration: 00m 29s)
22:46 logmsgbot: krenair@tin Synchronized wmf-config: more labs-only changes: https://gerrit.wikimedia.org/r/298393 (duration: 00m 36s)
21:50 logmsgbot: krenair@tin Synchronized wmf-config: sync labs-only change, should be a noop here: https://gerrit.wikimedia.org/r/#/c/298299/ (duration: 00m 39s)
21:40 awight: resuming fundraising donations queue consumer
21:37 mutante: ytterbium - graceful'ed Apache, warning about duplicate NameVirtual host is gone
21:03 bd808: Updated default mapping for logstash-* index creation using json generated by https://gerrit.wikimedia.org/r/#/c/298295/. Should take effect starting with the logstash-2016.07.12 index.
20:44 logmsgbot: reedy@tin Synchronized wmf-config/extension-list-labs: nooop for prod (duration: 00m 32s)
20:31 ottomata: rolling restart of hadoop-yarn-nodemanager to apply log aggregation retention seconds
20:29 mdholloway: mobileapps deployed df16702
20:25 mdholloway: starting mobileapps deployment
20:20 awight: disabled fundraising donation queue consumer...
20:09 subbu: finished deploying parsoid sha e738c415
20:05 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:03 subbu: starting parsoid deploy
18:37 chasemp: new hd for failed array in labstore2001
18:23 ejegg: updated payments from 2fc573cbb94e833c4144aa9dad79de8ec374bb09 to d9f7027340e5311f38c4224c2fddde087467df87
18:08 mutante: welcome new mediawiki deployer Brian Wolff (T138635)
17:44 ejegg: updated CiviCRM from f477a42014dd1e6759849b347d5f73d710954d0b to bf029eecb9bfb49d267e60d76344b0170bfa0a83
17:09 awight: reenable fundraising campaigns
17:07 gehel: starting deployment of latest WDQS (second time deploying with scap3)
16:52 twentyafterfour: unclog the phabricator task queue (phd) by cherry-picking upstream fix 12c6f87ca to wmf/stable (+restarted phd on iridium)
15:47 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Update logo settings for the Nepali Wikipedia (T139240) PART II (duration: 00m 26s)
15:46 logmsgbot: thcipriani@tin Synchronized static/images/project-logos: SWAT: Update logo settings for the Nepali Wikipedia (T139240) PART I (duration: 00m 27s)
15:43 elukey: restarted hhvm on mw1170 (Apache errors while reading FCGI headers, HHVM dump debug in /tmp/hhvm.14968.bt.)
15:40 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow to import from zh.wikipedia to beta.wikiversity (T139922) (duration: 00m 26s)
15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: State Compact Language Links is not beta anymore (T136677) (duration: 00m 26s)
15:33 logmsgbot: thcipriani@tin Synchronized wmf-config/interwiki.php: SWAT: Update interwiki map (duration: 00m 28s)
15:25 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: HD version for sqwikiquote logos (T139229) PART II (duration: 00m 27s)
15:25 logmsgbot: thcipriani@tin Synchronized static/images/project-logos: SWAT: HD version for sqwikiquote logos (T139229) PART I (duration: 00m 25s)
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add contentdm.lib.byu.edu to wgCopyUploadsDomains (T139095) (duration: 00m 26s)
15:13 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Remove old throttle rules (duration: 00m 30s)
15:06 elukey: removing mw114[0-8] from service via conftool as first decom step (T139353)
15:06 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/sqwikiquote.png: SWAT: Change Albanian Wikiquote logo (T139229) (duration: 00m 34s)
14:46 ema: upgrading cache_misc to varnish 4.1.3-1wm1
14:39 _joe_: shutting down mw1090-1113,mw1149-51 for decommissioning
14:05 moritzm: rebooting achernar for kernel update
13:31 moritzm: rebooting acamar for kernel update
13:07 ema: upgrading esams cache_maps to varnish 4.1.3-1wm1
12:51 gehel: upgrading nodejs to 4.4.6 on maps2.* servers
12:24 ema: upgrading ulsfo cache_maps to varnish 4.1.3-1wm1
12:14 elukey: restarted hhvm on mw1261
12:03 ema: upgrading codfw cache_maps to varnish 4.1.3-1wm1
11:30 ema: upgrading eqiad cache_maps to varnish 4.1.3-1wm1
11:19 moritzm: installing hhvm updates on canary app servers
10:25 hashar: CI: upgraded Chromium from v49 to v51 (v50 caused qunit jobs to fail / timeout randomly) T136188
10:00 godog: swift codfw-prod: ms-be202[567] weight 2000
09:47 moritzm: installing GCC stable updates on trusty systems (also provides some runtime libs in addition to GCC itself)
09:40 ema: upgrading cp1046 to varnish 4.1.3-1wm1
07:29 mobrovac: change-prop deploying 2b699a6
07:26 mobrovac: graphoid deploying 375d31fd
07:24 mobrovac: mathoid deploying 669cfc0
07:19 mobrovac: cxserver deploying fd8eca47e
07:15 mobrovac: citoid deploying 274c0231d
07:13 mobrovac: mobileapps deploying 6e409f46
06:34 _joe_: restarted hhvm on mw1168
06:13 moritzm: restarted saltmaster on neodymium
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jul 11 02:26:55 UTC 2016 (duration 5m 41s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 08m 41s)

2016-07-10

02:25 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 10 02:25:55 UTC 2016 (duration 5m 43s)
02:20 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 08m 34s)

2016-07-09

19:54 bd808: restarted logstash on logstash1003 for de-dot plugin update (T136001)
19:52 bd808: restarted logstash on logstash1002 for de-dot plugin update (T136001)
19:50 bd808: restarted logstash on logstash1001 for de-dot plugin update (T136001)
19:46 bd808: Updated logstash/plugins to 18b3f1f (Fix de_dot to process keys with falsey values)
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jul 9 02:29:53 UTC 2016 (duration 6m 7s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 08m 21s)
00:28 mutante: analytics1027 - icinga said puppet fail, just ran it, recovery, same on neon.. something kafka graphite checks
00:06 mutante: gerrit is back
00:05 ostriches: gerrit: disabled puppet for a minute so I can unbreak gerrit so I can fix gerrit in puppet.

2016-07-08

23:57 mutante: behold, gerrit might restart now for config change
23:52 Amir1: manually restarting uwsgi-ores in scb1001
23:48 logmsgbot: krinkle@tin Synchronized docroot/foundation/: Remove unused docroot/foundation/index.html (duration: 00m 30s)
23:32 Amir1: manually restarting uwsgi-ores on scb1002
22:59 urandom: Restarting restbase1015-b.eqiad.wmnet to cancel running streams : T139362
22:26 eileen: from bdf2afd417b70332c9542fd3ee4f14cb4e6f93cc to f477a42014dd1e6759849b347d5f73d710954d0b
22:13 urandom: Restarting restbase1014-b.eqiad.wmnet to cancel running streams : T139362
22:02 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.9/extensions/CentralAuth/: Fix job serializing (and status display on Special:GlobalRenameProgress) - T137973 (duration: 00m 32s)
21:59 logmsgbot: reedy@tin Synchronized wmf-config/extension-list: Fix RestBaseUpdateJobs in extension-list (duration: 00m 36s)
21:51 logmsgbot: reedy@tin Synchronized wmf-config/CommonSettings.php: Use Canonical entry point for RestBaseUpdateJobs (duration: 00m 33s)
21:50 logmsgbot: reedy@tin Synchronized multiversion/MWWikiversions.php: Remove some else if spaces (duration: 00m 46s)
21:48 urandom: Restarting restbase1009-b.eqiad.wmnet to cancel running streams : T139362
21:31 elukey: mw1146 powercycled (memory pressure, no ssh/root login)
21:17 urandom: Restarting restbase1015-a.eqiad.wmnet to cancel running streams : T139362
21:11 urandom: Restarting restbase1014-a.eqiad.wmnet to cancel running streams : T139362
21:04 urandom: Restarting restbase1009-a.eqiad.wmnet to cancel running streams : T139362
20:59 urandom: Forcing node removal (restbase1014-c.eqiad.wmnet) : T139362
20:51 urandom: Throttle RESTBase Cassandra outgoing streams to 1mbit cluster-wide : T139362 (actually happened at 21:26)
20:17 bd808: Deleted old l10nupdate caches manually on tin (T130317)
19:56 urandom: Throttle RESTBase Cassandra outgoing streams to 3mbit cluster-wide : T139362
18:52 anomie: Attempting to resubmit LocalRenameUserJobs for T137973
18:48 urandom: "This is going to hurt me more than it does you."; `nodetool removenode' of restbase1014-c.eqiad.wmnet : T139362
18:39 urandom: Stopping restbase1014-c.eqiad.wmnet : T139362
18:29 mutante: db1042 - temp stop puppet, edit ferm rules to allow testing from lead
14:43 jynus: rechecking data consistency after m3 table fixes (could cause lag)
12:21 moritzm: installing glib updates from jessie point release
12:03 jynus: stopping replication on db1043 (m3-slave) for maintenance
09:33 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: aqs1002.eqiad.wmnet
09:30 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: aqs1002.eqiad.wmnet
09:30 elukey: upgrading nodejs packages on aqs100[23]
09:10 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=5; selector: mw1261.eqiad.wmnet
09:06 logmsgbot: elukey@palladium conftool action : set/pooled=no:weight=5; selector: mw1261.eqiad.wmnet
09:03 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=5; selector: mw1261.eqiad.wmnet
09:01 moritzm: rebooting ruthenium for update to Linux 4.4
08:40 hashar: gallium: deleting old log files /var/log/zuul/gearman-server-debug.log*
08:35 hashar: gallium: restarting Zuul to apply logging configuration change https://gerrit.wikimedia.org/r/#/c/291913/
08:18 moritzm: rebooting terbium for kernel security update
08:12 moritzm: rearming keyholder on tin
08:08 moritzm: rebooting tin for kernel security update
08:08 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=mw1261.eqiad.wmnet
07:15 _joe_: removing 20 gb logfile from terbium, only useless debug info
02:37 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jul 8 02:37:29 UTC 2016 (duration 6m 22s)
02:31 logmsgbot: dereckson@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 07m 47s)
02:12 Dereckson: scap pull on terbium (was out of disk space during previous full scap)
02:00 logmsgbot: l10nupdate@tin LocalisationUpdate failed: git pull of extensions failed
01:54 mutante: terbium ran out of disk, deleted rotated nutcracker log
01:48 logmsgbot: dereckson@tin Finished scap: Flow 297914, Echo 297919 297934, ORES 297916 (duration: 27m 48s)
01:26 ostriches: gerrit: readded robots.txt to ytterbium for now
01:20 logmsgbot: dereckson@tin Started scap: Flow 297914, Echo 297919 297934, ORES 297916
00:42 awight: roll back paymentswiki further, to 2fc573cbb94e833c4144aa9dad79de8ec374bb09
00:39 awight: roll back paymentswiki from f54ffb4fad0dc18079a813fbe25813dba36c64aa to c33ddfccf945bd075f0abff9e9de8c09f0174f89
00:27 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.9/includes/api/ApiParse.php: API: Generate head items in the context of the given title (T139565) (duration: 00m 30s)
00:01 awight: CentralNotice campaigns reenabled
00:00 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable ORES review tool as a beta feature in ptwiki (T139692) (duration: 00m 28s)

2016-07-07

23:57 Dereckson: Ran extensions/ORES/maintenance/CheckModelVersions.php and extensions/ORES/maintenance/PopulateDatabase.php on ptwiki (T139692)
23:55 awight: Update paymentswiki from c33ddfccf945bd075f0abff9e9de8c09f0174f89 to f54ffb4fad0dc18079a813fbe25813dba36c64aa
23:50 Dereckson: Created table ores_classification on ptwiki from php-1.28.0-wmf.9/extensions/ORES/sql/ores_classification.sql
23:48 Dereckson: Created table ores_model on ptwiki from php-1.28.0-wmf.9/extensions/ORES/sql/ores_model.sql
23:36 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Remove unused deprecated $wgStyleSheetPath (Gerrit:297511) (duration: 00m 27s)
23:23 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: VisualEditor: Move cite out of primary toolbar except on WP/WB/WV (Gerrit:296573) (duration: 00m 30s)
22:33 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/extensions/VisualEditor/: https://gerrit.wikimedia.org/r/#/c/297795/ and https://gerrit.wikimedia.org/r/#/c/297908/ (duration: 00m 45s)
21:32 urandom: Bootstrapping restbase1009-c : T139362
20:59 urandom: Fin : T126629
20:52 urandom: Restarting Cassandra instance restbase2009-b.codfw.wmnet : T126629
20:50 urandom: Restarting Cassandra instance restbase2009-a.codfw.wmnet : T126629
20:49 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2009.codfw.wmnet : T126629
20:48 urandom: Cassandra upgrade of restbase2006.codfw.wmnet instances complete : T126629
20:45 urandom: Restarting RESTBase on restbase2001.codfw.wmnet
20:41 urandom: Restarting Cassandra instance restbase2006-b.codfw.wmnet : T126629
20:38 urandom: Restarting Cassandra instance restbase2006-a.codfw.wmnet : T126629
20:36 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2006.codfw.wmnet : T126629
20:33 urandom: Cassandra upgrade of restbase2005.codfw.wmnet instances complete : T126629
20:30 urandom: Restarting Cassandra instance restbase2005-b.codfw.wmnet : T126629
20:27 urandom: Restarting Cassandra instance restbase2005-a.codfw.wmnet : T126629
20:27 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.9
20:26 twentyafterfour: deploying wmf.9 to all wikis refs T138555
20:26 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2005.codfw.wmnet : T126629
20:15 gehel: logstash upgrade aborted, rescheduled to Monday July 11th
20:10 bd808: Restarted logstash on logstash1001; hoping to clear up missing de-dot errors
20:07 bd808: Restarted logstash on logstash1002; hoping to clear up missing de-dot errors
20:07 urandom: Cassandra upgrade of restbase2008.codfw.wmnet instances complete : T126629
20:04 urandom: Restarting Cassandra instance restbase2008-b.codfw.wmnet : T126629
20:02 urandom: Restarting Cassandra instance restbase2008-a.codfw.wmnet : T126629
20:02 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.9/extensions/Echo: Fixes for notification sorting and message parsing (duration: 00m 38s)
20:01 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2008.codfw.wmnet : T126629
20:01 urandom: Cassandra upgrade of restbase2004.codfw.wmnet instances complete : T126629
19:59 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.9/extensions/Flow/includes/Notifications/PostReplyPresentationModel.php: flow-post-reply: show compact header on one line (duration: 00m 32s)
19:59 urandom: Restarting Cassandra instance restbase2004-b.codfw.wmnet : T126629
19:57 bd808: Restarted logstash on logstash1003; hoping to clear up missing de-dot errors
19:55 urandom: Restarting Cassandra instance restbase2004-a.codfw.wmnet : T126629
19:55 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2004.codfw.wmnet : T126629
19:53 urandom: Cassandra upgrade of restbase2003.codfw.wmnet instances complete : T126629
19:50 urandom: Restarting Cassandra instance restbase2003-b.codfw.wmnet : T126629
19:46 urandom: Restarting Cassandra instance restbase2003-a.codfw.wmnet : T126629
19:45 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2003.codfw.wmnet : T126629
19:32 urandom: Upgrade of restbase2007.codfw.wmnet instances complete : T126629
19:29 urandom: Restarting Cassandra instance restbase2007-c.codfw.wmnet : T126629
19:27 urandom: Restarting Cassandra instance restbase2007-b.codfw.wmnet : T126629
19:24 urandom: Restarting Cassandra instance restbase2007-a.codfw.wmnet : T126629
19:22 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2007.codfw.wmnet : T126629
19:19 urandom: Upgrade of restbase2002.codfw.wmnet instances complete : T126629
19:16 urandom: Restarting Cassandra instance restbase2002-c.codfw.wmnet : T126629
19:13 urandom: Restarting Cassandra instance restbase2002-b.codfw.wmnet : T126629
19:12 ostriches: gerrit: force all users to log out. sorry ❤️
19:11 urandom: Restarting Cassandra instance restbase2002-a.codfw.wmnet : T126629
19:09 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2002.codfw.wmnet : T126629
19:08 urandom: Cassandra upgrade of restbase2001.codfw.wmnet instances complete : T126629
19:05 urandom: Restarting Cassandra instance restbase2001-c.codfw.wmnet : T126629
19:02 urandom: Restarting Cassandra instance restbase2001-b.codfw.wmnet : T126629
18:56 urandom: Restarting Cassandra instance restbase2001-a.codfw.wmnet : T126629
18:53 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase2001.codfw.wmnet : T126629
18:27 urandom: Disabling Puppet on RESTBase codfw nodes : T126629
18:23 awight: Update SmashPig from 917138e159f0341e3dfbb35818c3ce479927875b to e6aa6fe6fdcaab8e961a8b0668cc742d4c443c46
18:15 urandom: Cassandra 2.2.6 upgrade of restbase1015.eqiad.wmnet instances complete : T126629
18:11 urandom: Restarting Cassandra instance restbase1015-b.eqiad.wmnet : T126629
18:08 urandom: Restarting Cassandra instance restbase1015-a.eqiad.wmnet : T126629
18:07 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase1015.eqiad.wmnet : T126629
17:12 ostriches: gerrit: flush all caches to pick up account disable & rename
16:55 urandom: Restarting Cassandra instance restbase1014-c.eqiad.wmnet : T126629
16:54 awight: update paymentwiki from 2fc573cbb94e833c4144aa9dad79de8ec374bb09 to c33ddfccf945bd075f0abff9e9de8c09f0174f89
16:48 urandom: Restarting Cassandra instance restbase1014-b.eqiad.wmnet : T126629
16:47 jynus: stopping pc2006 for hardware maintenance T139283
16:39 urandom: Restarting Cassandra instance restbase1014-a.eqiad.wmnet : T126629
16:38 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase1014.eqiad.wmnet : T126629
16:29 urandom: Disabling Puppet on restbase101[4-5].eqiad.wmnet : T126629
16:24 gehel: starting elasticsearch and kibana upgrade on logstash cluster (T136001)
16:21 awight: Taking Fundraising campaigns down for maintenance
15:51 elukey: add mw1261 back into service
15:44 bd808: Dropped logstash indices older than logstash-2016.07.01 in preparation for elasticsearch upgrade
15:25 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.9/extensions/Echo/modules/controller/mw.echo.Controller.js: SWAT: Correct section (alert/message/all) (duration: 00m 25s)
15:14 _joe_: uploaded new HHVM package for jessie
15:11 logmsgbot: thcipriani@tin Synchronized dblists/clldefault.dblist: SWAT: Deploy Compact Language Links as default (Stage 4) (T136677) PART II (duration: 00m 34s)
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy Compact Language Links as default (Stage 4) (T136677) PART I (duration: 00m 55s)
14:43 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: aqs1001.eqiad.wmnet
14:38 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: aqs1001.eqiad.wmnet
14:37 elukey: depool aqs1001 for nodejs upgrade
12:05 elukey: depooling mw1261 from service (T73487)
11:34 jynus: breaking m3 replication on db1048 (depooled) to check icinga alert changes
10:44 jynus: disabling all mysql lag alerts cross-fleet T122457
10:29 akosiaris: reboot fermium.wikimedia.org hassium.eqiad.wmnet install1001.wikimedia.org krypton.eqiad.wmnet meitnerium.wikimedia.org mendelevium.eqiad.wmnet T134242
10:22 akosiaris: reboot dubnium T134242
10:22 akosiaris: reboot bromine T134242
10:13 akosiaris: reboot bohrium T134242
10:11 elukey: pooling mw1261 back to service with Apache mod-proxy-fcgi set to trace8 (T73487)
10:07 akosiaris: reboot etherpad1001.eqiad.wmnet, kernel upgrade and qemu upgrade, T134242
10:04 gehel: rolling restart of elasticsearch cluster eqiad completed (T138811)
08:51 _joe_: removing all old servers from the appservers pool but the canaries (T139353)
08:47 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover s4 recentchanges to db1056 (duration: 00m 38s)
08:33 hoo: Updated Wikidata's property suggester with data from Monday's json dump and removed the external identifiers as a workaround for T132839
07:56 gehel: rolling restart of elasticsearch cluster codfw completed (T138811)
07:16 legoktm: mysql:wikiadmin@db1041 [centralauth]> delete from localuser where lu_name ="Philippe" and lu_wiki ="scnwiki";
04:37 eileen: Update CiviCRM from dd24368a897fd78752178ee253e7a890dd57db41 to bdf2afd417b70332c9542fd3ee4f14cb4e6f93cc
03:12 eileen: CiviCRM upgrade from 5f8f7c3236e6bf12c52deea07093fbca165ef4a7 to dd24368a897fd78752178ee253e7a890dd57db41
03:08 chasemp: silence labvirt1011 flapping for 24h, we have a task, and we are attempting to move vms we can
02:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jul 7 02:48:55 UTC 2016 (duration 6m 44s)
02:42 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 07m 07s)
02:26 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 09m 58s)
02:14 chasemp: reboot labvirt1011
01:53 chasemp: start nova-compute again as it seems ot have been killed on labvirt1011 which is acting weird
01:24 andrewbogott: rebooting labvirt1011 (because it is acting crazy)
00:50 eileen: updating CiviCRM from 54e168db2fddc6a9a07036323e01a27dd64333cf to 5f8f7c3236e6bf12c52deea07093fbca165ef4a7
00:23 eileen: Updated CiviCRM from bb9bf136dc0fa82d5d07ebeb33d696e54672b2d6 to 54e168db2fddc6a9a07036323e01a27dd64333cf
00:06 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.8/extensions/CentralAuth/: Make LocalRename jobs run sequentially - T137973 (for real this time) (duration: 00m 30s)
00:05 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.9/extensions/CentralAuth/: Make LocalRename jobs run sequentially - T137973 (duration: 00m 30s)
00:03 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.8/extensions/CentralAuth/: Make LocalRename jobs run sequentially - T137973 (duration: 00m 34s)

2016-07-06

23:56 legoktm: created pageassesments tables on testwiki
23:52 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.9/includes/specials/SpecialContributions.php: Add mediawiki.special.changeslist to SpecialContributions - T139522 (duration: 00m 25s)
23:50 legoktm: running extensions/ORES/maintenance/CheckModelVersions.php and extensions/ORES/maintenance/PopulateDatabase.php on ruwiki
23:49 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: Enable ORES review tool as a beta feature in ruwiki - T139541 (duration: 00m 27s)
23:47 legoktm: created ores_* tables on ruwiki
23:41 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.9/extensions/Echo/: T139321, T139323 (duration: 00m 32s)
23:33 logmsgbot: legoktm@tin Synchronized wmf-config: VisualEditor: Move the citation button out of the primary toolbar on Wikivoyes - T133725 (2/2) (duration: 00m 30s)
23:32 logmsgbot: legoktm@tin Synchronized wmf-config/InitialiseSettings.php: VisualEditor: Move the citation button out of the primary toolbar on Wikivoyes - T133725 (1/2) (duration: 00m 26s)
23:30 logmsgbot: legoktm@tin Synchronized wmf-config: touch (duration: 00m 32s)
23:25 logmsgbot: legoktm@tin Synchronized wmf-config: Test PageAssessments extension on test.wikipedia.org - T137918 (duration: 00m 36s)
22:18 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.9/includes/diff: T139526 (duration: 00m 38s)
20:44 urandom: Restarting Cassandra instance restbase1008-b.eqiad.wmnet : T126629
20:41 bearND: deployed mobileapps 7a73789
20:38 bearND: starting mobileapps deploy
20:38 urandom: Restarting Cassandra instance restbase1008-a.eqiad.wmnet : T126629
20:23 urandom: Disable Puppet on restbase1009.eqiad.wmnet : T126629
20:13 urandom: Upgrade of restbase1013.eqiad.wmnet instances complete : T126629
20:06 urandom: Restarting Cassandra instance restbase1013-b.eqiad.wmnet : T126629
20:00 urandom: Restarting Cassandra instance restbase1013-a.eqiad.wmnet : T126629
19:58 twentyafterfour: deployed 1.28.0-wmf.9 to group1 wikis: T138555
19:57 urandom: Upgrading Cassandra package to 2.2.6-wmf1 on restbase1013.eqiad.wmnet : T126629
19:57 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.9
19:30 urandom: Upgrade of restbase1012.eqiad.wmnet instances complete : T126629
19:24 urandom: Restarting Cassandra instance restbase1012-c.eqiad.wmnet : T126629
19:18 urandom: Restarting Cassandra instance restbase1012-b.eqiad.wmnet : T126629
19:09 urandom: Restarting Cassandra instance restbase1012-a.eqiad.wmnet : T126629
19:07 urandom: Upgrading Cassandra package to 2.2.6-wmf1 on restbase1012.eqiad.wmnet : T126629
18:57 mutante: mw1133 puppet fail because out of memory, stop hhvm, run puppet
18:55 urandom: Cassandra upgrade of restbase1008.eqiad.wmnet instances complete : T126629
18:54 mutante: labstore1001 - failed backup
18:54 mutante: mw1261 syntax error in Apache config
18:52 urandom: Restarting Cassandra instance restbase1008-c : T126629
18:51 mutante: labstore2001 - RAID failure in Icinga (is it T102626 ?)
18:47 urandom: Restarting Cassandra instance restbase1008-b : T126629
18:46 mutante: mw1261 restart hhvm service
18:42 urandom: Restarting Cassandra instance restbase1008-a : T126629
18:40 urandom: Upgrading Cassandra package to 2.2.6-wmf1 on restbase1008 : T126629
17:14 urandom: Disabling Puppet on restbase{1008,1012,1013}.eqiad.wmnet in preparation for rack 'b' Cassandra upgrade : T126629
16:54 urandom: Upgrade of restbase1011.eqiad.wmnet instances to Cassandra 2.2.6 complete : T126629
16:50 urandom: Restarting Cassandra for restbase1011-c.eqiad.wmnet : T126629
16:45 urandom: Restarting Cassandra for restbase1011-b.eqiad.wmnet : T126629
16:41 urandom: Restarting Cassandra fro restbase1011-a.eqiad.wmnet : T126629
16:39 urandom: Upgrading Cassandra package to 2.2.6-wmf1 on restbase1011.eqiad.wmnet : T126629
16:26 awight: update orphan rectifier from 2fc573cbb94e833c4144aa9dad79de8ec374bb09 to 70a7baa9f77c2510739bab0ff9d1b51578a59a6e
16:25 urandom: Upgrade of restbase1010.eqiad.wmnet instances complete : T126629
16:25 awight: update orphan rectifier config to add payments 4 to the Redis pool
16:22 urandom: Restarting Cassandra for restbase1010-c.eqiad.wmnet : T126629
15:59 urandom: Restarting Cassandra for restbase1010-b.eqiad.wmnet : T126629
15:47 urandom: Restarting Cassandra for restbase1010-a.eqiad.wmnet : T126629
15:45 urandom: Re-enabling Puppet on restbase1010 : T126629
15:44 urandom: Upgrading Cassandra to 2.2.6-wmf1 on restbase1010 : T126629
15:36 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.9/extensions/Echo/modules/styles: SWAT: Set width to Special:Notifications (T138433) (duration: 00m 30s)
15:29 elukey: restarting the hdfs datanode on each analytics* Hadoop server to force the new -Xmx2048 heap setting to be picked up
15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Echo transition flags everywhere (duration: 00m 26s)
15:22 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable autopatrolled user group at urwiki (T139302) (duration: 00m 30s)
15:18 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.9/includes/diff/DifferenceEngine.php: Show parser output for diffs unless extension aborts (T139433) (duration: 00m 30s)
15:08 urandom: Disabling puppet on restbase101[0-1].eqiad.wmnet in preparation for 2.2.6 upgrade : T126629
15:02 akosiaris: poweroff prometheus2002 from dbrb -> plain conversion
15:00 _joe_: depooling mw1261, installing an apache package with additional fixes (T73487)
14:19 moritzm: rebooting californium for kernel update (hosting horizon.wikimedia.org)
14:02 moritzm: rebooting silver (hosting wikitech)
13:48 gehel: re-enabling puppet on ^(aqs|restbase).* after confirming that Cassandra puppet module change is a noop
13:38 gehel: disabling puppet on ^(aqs|restbase).* before merging changes to Cassandra puppet module
13:35 elukey: depooling mw1261.eqiad to restore previous fcgi logging settings (T73487)
13:17 dcausse: restarting elastic master node (elastic1040)
13:15 dcausse: truncating elastic main logs on elastic1040 and elastic1034
12:19 mobrovac: restbase deploy end of fa4699a
12:09 mobrovac: restbase deploy start of fa4699a
11:54 elukey: depooling mw1261.eqiad.wmnet to raise Apache's mod-fcgi to trace8 for 503 investigation - T73487 (this will probably slow down a bit the host)
11:15 jynus: shutting down db1048 in preparation for upgrade
10:14 moritzm: upgrading restbase cluster in codfw for nodejs 4.4.6
09:39 _joe_: restarting hhvm on mw1236,mw1215 to test for possible TC cache corruption
09:31 moritzm: installing tomcat security updates on Ubuntu systems (jessie already fixed)
09:13 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1285.eqiad.wmnet
09:10 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1284.eqiad.wmnet
09:09 elukey: pooling into service the last batch of new API appservers - mw1284->mw1290
08:46 godog: lithium:~$ sudo lvextend --size +50G -t -r /dev/mapper/lithium--vg-syslog
08:32 godog: powercycle ms-be2021, unreachable and nothing on console
08:11 jynus: stopping replication on db1056 and performing alter table
07:58 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Fail commons db servers back to its original configuration (duration: 00m 45s)
07:57 moritzm: rolling reboot of sca clusters for kernel update
07:51 moritzm: restarted hhvm on mw1148
03:09 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jul 6 03:09:17 UTC 2016 (duration 6m 53s)
03:02 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.9) (duration: 17m 48s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 09m 16s)
00:23 logmsgbot: maxsem@tin Synchronized wmf-config: No-op (duration: 00m 37s)

2016-07-05

23:57 MaxSem: ran ORES's CheckModelVersions.php and PopulateDatabase.php on nlwiki
23:48 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/297462/ (duration: 00m 29s)
23:42 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.8/extensions/MobileFrontend/: https://gerrit.wikimedia.org/r/#/c/297542/ (duration: 00m 33s)
23:35 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/297542/ (duration: 00m 30s)
23:33 MaxSem: created ORES tables on nlwiki
23:26 chasemp: reboot of labvirt1011 which is being flaky I cannot keep a connection to
23:22 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/296852/ (duration: 00m 53s)
22:07 hasharAway: CI had issue booting instances since 19:50UTC it is operational again as of 21:30 UTC and slowly processing the backlog.
21:34 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: 1.28.0-wmf.9 to group0
20:56 logmsgbot: twentyafterfour@tin Finished scap: Rebuild l10n cache and deploy 1.28.0-wmf.9 to testwiki (duration: 32m 16s)
20:24 logmsgbot: twentyafterfour@tin Started scap: Rebuild l10n cache and deploy 1.28.0-wmf.9 to testwiki
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: lttoolbox_3.3.3~r68466-2+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: hfst_3.10.0~r2798-1+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: hfst-ospell_0.4.0~r4643-5+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: foma_0.9.18+r243-1+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: cg3_0.9.9~r11624-1+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium_3.4.2~r68466-2+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-lex-tools_0.1.1~r66150-1+wmf1
20:01 akosiaris: T107306 uploaded to apt.wikimedia.org jessie-wikimedia: apertium-apy_0.9.1~r343-1
18:54 chasemp: add dpatrick to wmf-nda
18:32 Danny_B: killed wb2-phab @ tools.wikibugs to perform some batch edits
18:30 mutante: added siddharth11 to LDAP group "wmf" per T138369#2425011
16:44 jynus: rebooting pc2006 T139283
16:20 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1024.eqiad.wmnet
16:13 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: mw1024.eqiad.wmnet
16:12 elukey: depooling mw1024 to restore regular fcgi logging settings
16:09 urandom: Bootstrapping restbase1009-c.eqiad.wmnet : T139362
15:49 akosiaris: reenable alerts from smokeping on codfw
15:46 andrewbogott: rebooting labservices1002 to see if it survives the reboot better this time
15:32 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: touch initialisesettings to clear logspam, hopefully (duration: 00m 27s)
15:20 elukey: powercycled mw1140, memory saturated and not reachable via ssh/mgmt-console
15:20 paravoid: mr1-codfw: "request system snapshot media internal slice alternate" + "request system reboot"
15:08 yurik: depl & restarted tilerator https://gerrit.wikimedia.org/r/#/c/297410/
14:42 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1283.eqiad.wmnet
14:41 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1282.eqiad.wmnet
14:37 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1281.eqiad.wmnet
14:26 urandom: Issuing `nodetool cleanup' for restbase1014-b.eqiad.wmnet
14:12 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1280.eqiad.wmnet
14:09 logmsgbot: elukey@palladium conftool action : set/pooled=yes:weight=25; selector: mw1279.eqiad.wmnet
14:07 logmsgbot: oblivian@palladium conftool action : set/pooled=inactive; selector: dc=eqiad,cluster=appserver,service=apache2,name=mw1217.*
14:07 elukey: Pooling first batch of new eqiad api-servers - mw1279->mw1283
14:03 _joe_: depooling permanently mw1091-13 from the appservers pool in eqiad
13:58 logmsgbot: elukey@palladium conftool action : set/weight=20; selector: mw2245.codfw.wmnet
13:58 logmsgbot: elukey@palladium conftool action : set/weight=20; selector: mw2244.codfw.wmnet
13:58 logmsgbot: elukey@palladium conftool action : set/weight=20; selector: mw2243.codfw.wmnet
13:58 logmsgbot: elukey@palladium conftool action : set/weight=20; selector: mw2242.codfw.wmnet
13:57 logmsgbot: elukey@palladium conftool action : set/weight=20; selector: mw2241.codfw.wmnet
13:57 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw2245.codfw.wmnet
13:57 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw2244.codfw.wmnet
13:57 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw2243.codfw.wmnet
13:56 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw2242.codfw.wmnet
13:56 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw2241.codfw.wmnet
13:56 elukey: pooling new codfw appservers - mw224[12345]
12:32 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1024.eqiad.wmnet
12:12 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: mw1024.eqiad.wmnet
12:11 elukey: depooling/re-pooling mw1024.eqiad.wmnet to temporarily set up trace8 logging (503 investigation - T73487)
12:08 jynus: running schema change on db1019 T73563
11:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover all commons special roles to db1081 (duration: 00m 24s)
11:00 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover commons recentachanges (duration: 00m 36s)
10:45 jynus: SET GLOBAL read_only=0; on db1040, our new m4-master
10:38 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover commons master to db1040 (duration: 00m 32s)
10:23 jynus: archiving m3-master phlegal* databases before dropping them
10:21 mobrovac: restbase staging started a no-op dump on cerium to test restbase on node 4.4.6
10:05 logmsgbot: elukey@palladium conftool action : set/weight=30; selector: mw1275.eqiad.wmnet
10:05 logmsgbot: elukey@palladium conftool action : set/weight=30; selector: mw1274.eqiad.wmnet
10:05 logmsgbot: elukey@palladium conftool action : set/weight=30; selector: mw1273.eqiad.wmnet
09:59 logmsgbot: elukey@palladium conftool action : set/weight=30; selector: mw1272.eqiad.wmnet
09:31 _joe_: shutting down mw1009-16 for decommissioning
09:06 _joe_: decommissioning mw1009-16
08:38 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1275.eqiad.wmnet
08:36 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1274.eqiad.wmnet
08:32 gehel: deleting enwikisource_titlesuggest on elasticsearch codfw (index creation issue during cluster restart)
08:31 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1273.eqiad.wmnet
08:24 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: mw1272.eqiad.wmnet
08:21 elukey: adding and pooling new appservers - mw127[2345].eqiad
08:07 godog: swift codfw-prod: ms-be202[567] weight 1500
07:55 jynus: dropping etherpad_restore2 database from m1 T138516
07:40 akosiaris: T138516 forcing a puppet run on cache::misc hosts after merging https://gerrit.wikimedia.org/r/297352
07:29 akosiaris: T138516 stop the secondary etherpad instance on etherpad1001. etherpad-restore.wikimedia.org has served its purpose, killing it
02:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jul 5 02:44:09 UTC 2016 (duration 6m 12s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 17m 13s)

2016-07-04

20:28 jynus: removing /tmp/joal/sstables on all analytics10* hosts
20:22 jynus: deleted 21GB worth of temporary files from analytics1050
19:58 logmsgbot: aaron@tin Synchronized wmf-config/filebackend-production.php: Increase redis lockmanager timeout to 2 (duration: 00m 31s)
19:57 logmsgbot: legoktm@tin Synchronized php-1.28.0-wmf.8/extensions/MassMessage/: MassMessage is no longer accepting lists in the MassMessageList content model - T139303 (duration: 00m 39s)
17:37 jynus: testing slave_parallel_threads=5 on db1073
14:27 moritzm: rebooting lithium for kernel update
14:22 moritzm: installing tomcat7/ libservlet3.0-java security update on the kafka brokers
14:06 _joe_: shutting down mw1001-1008 for decommissioning
14:03 gehel: rolling restart of elasticsearch codfw/eqiad for kernel upgrade (T138811)
13:47 _joe_: stopping jobrunner on mw1011-16 as well, befor decommissioning
13:46 moritzm: depooling mw1153-mw1160 (trusty image scalers), replaced by mw1291-mw1298 (jessie image scalers)
13:44 godog: ack all mr1-codfw related alerts in librenms
13:43 akosiaris: restart smokeping on netmon1001, temporarily disabled msw1-codfw
13:38 gehel: resuming writes on Cirrus / elasticsearch, this did not speedup cluster recovery
13:18 godog: bounce redis on rcs1001
13:16 gehel: restarting elastic1021 for kernel upgrade (T138811)
13:07 elukey: Bootstrapping again Cassandra on aqs100[456] (rack awareness + 2.2.6 - testing environment)
13:02 gehel: pausing writes on Cirrus / elasticsearch for faster cluster restart
12:43 hashar: Nodepool back up with 10 instances (instead of 20) to accomodate for labs capacity T139285
12:39 godog: nodetool-b stop -- COMPACTION on restbase1014
12:29 moritzm: rolling reboot of rcs* cluster for kernel security update
12:10 moritzm: rolling reboot of ocg* cluster for kernel security update
11:40 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover db1053 to db1072 (duration: 00m 40s)
10:56 moritzm: rolling reboot of swift frontends in eqiad for kernel security update
10:30 yuvipanda: stop nodepool on labnodepool1001 and disable puppet to keep it down, to allow stabilizing labs first
10:28 yuvipanda: restart rabbitmq-server on labcontrol1001
10:14 moritzm: installing chromium security update on osmium
10:07 moritzm: installing xerces-c security updates on Ubuntu systems (jessie already fixed)
10:01 _joe_: stopping jobchron and jobrunner on mw1001-10 before decommission
09:50 godog: reimage ms-be300[234] with jessie
09:44 hashar: Labs infra cant delete instances anymore (impacts CI as well) T139285
09:41 moritzm: installing p7zip security updates
09:38 hashar: CI is out of Nodepool instances, the pool has drained because instances can no more be deleted over the OpenStack API
09:25 elukey: Added new jobrunners in service - mw130[256].eqiad.wmnet (https://etherpad.wikimedia.org/p/jessie-install)
08:16 moritzm: rolling reboot of swift backends in eqiad for kernel security update
07:49 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Failover db1034 to db1062 (duration: 00m 30s)
02:26 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jul 4 02:26:54 UTC 2016 (duration 5m 42s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 09m 14s)

2016-07-03

19:27 Reedy: Ran namespaceDupes --fix on gomwiki
14:59 yuvipanda: restart nova-compute process on labvirt1010
14:59 yuvipanda: restart nova-compute process on labvirt10101
09:06 jynus: removing old logs from pc2004
07:42 logmsgbot: legoktm@tin Synchronized static/images/project-logos/: Put high-res enwiktionary logos in the right place - T139255 (duration: 00m 38s)
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jul 3 02:27:13 UTC 2016 (duration 5m 38s)
02:21 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 09m 13s)

2016-07-02

19:15 twentyafterfour: Deployed hotfix to phabricator. Restarted apache2 on iridium
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jul 2 02:29:17 UTC 2016 (duration 5m 40s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 08m 52s)

2016-07-01

22:23 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.8/extensions/WikimediaEvents/extension.json: T128115 (duration: 00m 37s)
22:22 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.8/extensions/WikimediaEvents/modules/: T128115 (duration: 00m 30s)
21:04 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I7a95c0f4: Bump $wgResourceLoaderMaxQueryLength to 5,000 (duration: 00m 32s)
20:08 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I6eb0ae67: Bump $wgResourceLoaderMaxQueryLength to 4,000 (duration: 00m 26s)
19:17 ori: restarted coal on graphite1001 stopped receiving messages from EL 0mq publisher
19:16 ori: restarted navtiming on hafnium; stopped receiving messages from EL 0mq publisher
18:34 mutante: mw1259 - powercycling
18:32 logmsgbot: krinkle@tin Synchronized docroot/default/: (no message) (duration: 00m 31s)
18:31 logmsgbot: krinkle@tin Synchronized errorpages/: (no message) (duration: 01m 06s)
17:47 ebernhardson: restart elasticsearch on elastic1017 to attempt to clear up a continuous backlog of relocating shards
15:53 godog: temporarily run 3x statsdlb instances on graphite1001 to minimise drops - T101141
14:57 dcausse: upgraded and restarted elastic on nobelium@eqiad
14:23 godog: enable another statsdlb instance temporarily on graphite1001 to investigate drops
14:15 moritzm: rearmed keyholder on mira after reboot
13:56 moritzm: rebooting codfw poolcounters for kernel update
13:47 moritzm: rebooting osmium for kernel update
13:28 cmjohnson1: mw1145 swapped eth0 cable
13:04 moritzm: rebooting mira for kernel update
12:59 moritzm: rebooting francium for kernel update
11:23 godog: bounce statsdlb on graphite1001, drops are back after yesterday's reboot T101141
11:15 moritzm: removed two obsolete, older kernel packages from wtp1002 (had flagged an icinga warning on diskspace on /boot)
09:38 elukey: rebooted eventlog2001.codfw.wmnet for kernel upgrades
09:35 moritzm: rolling reboot of swift backends in codfw
09:15 moritzm: powercycling ms-fe2003, stuck after reboot
09:03 moritzm: powercycling ms-fe2002, stuck after reboot
08:41 moritzm: powercycling ms-fe2001, stuck after reboot
08:32 moritzm: rolling reboot of swift frontends in codfw
06:28 moritzm: resuming rolling reboots of elastic* clusters in eqiad and codfw
06:18 moritzm: rolling reboot of wtp1* for kernel security update
02:44 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jul 1 02:44:01 UTC 2016 (duration 5m 7s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 17m 02s)
01:59 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: I3a8057a8: Bump $wgResourceLoaderMaxQueryLength to 3,000 (duration: 00m 28s)
01:46 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: Enable LocalFile log (duration: 00m 32s)
01:08 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: Ie8a71af5: HD logo for en.wiktionary (2/2) (duration: 00m 27s)
01:07 logmsgbot: ori@tin Synchronized static/images/project-logos: Ie8a71af5: HD logo for en.wiktionary (1/2) (duration: 00m 28s)
01:04 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.8/extensions/WikimediaEvents/modules/ext.wikimediaEvents.deprecate.js: Ie28d823c: Log ResourceLoader URL-splitting (duration: 00m 32s)
00:20 logmsgbot: maxsem@tin Finished scap: https://gerrit.wikimedia.org/r/#/c/296819/ - noop in prod (duration: 27m 27s)

2016-06-30

23:53 logmsgbot: maxsem@tin Started scap: https://gerrit.wikimedia.org/r/#/c/296819/ - noop in prod
23:47 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/296257/ (duration: 00m 26s)
23:43 logmsgbot: maxsem@tin Synchronized portals: (no message) (duration: 00m 27s)
23:43 logmsgbot: maxsem@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 28s)
23:41 ori: banned /static/images/project-logos/enwiktionary.png and /static/images/project-logos/adywiki.png
23:37 logmsgbot: maxsem@tin Synchronized docroot/mediawiki/xml/sitelist-1.0/index.html: https://gerrit.wikimedia.org/r/#/c/296788/ (duration: 00m 24s)
23:27 logmsgbot: maxsem@tin Synchronized static/images/project-logos/: (no message) (duration: 00m 27s)
23:12 logmsgbot: maxsem@tin Synchronized static/images/project-logos/enwiktionary.png: https://gerrit.wikimedia.org/r/#/c/296757/ (duration: 00m 30s)
22:04 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: test2wiki wgSquidMaxage (duration: 00m 28s)
21:00 ebernhardson: change cluster.routing.allocation.disk.watermark.high on eqiad elasticsearch cluster to 80%
20:52 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.8/includes/filerepo/file/LocalFile.php: 51d7fb48f2af31d69db115a8b3ed790cdaaf0d2e (duration: 00m 35s)
19:47 logmsgbot: aaron@tin Synchronized wmf-config/InitialiseSettings.php: Set the SaveParse log (duration: 00m 26s)
19:43 twentyafterfour: ran scap pull on mw2123
19:42 twentyafterfour: ran scap pull on mw2098
19:42 gehel: activating statement timeout limitations for kartotherian on maps cluster codfw (T138422)
19:38 ostriches: mw2134: running sync-common, seems out of...sync :)
19:20 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
19:19 twentyafterfour: Deploying 1.28.0-wmf.8 to all wikimedia wikis.
19:18 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.8/includes: adc4c90202d6c44aa58756e3c6bc35918afc5f75 (duration: 01m 19s)
18:24 ori: restarted coal on graphite1001 and navtiming on hafnium due to inexplicably stopped metrics; nothing useful in logs.
17:48 yurik: deployed Kartotherian https://gerrit.wikimedia.org/r/#/c/296787/
17:10 yurik: deployed Graphoid https://gerrit.wikimedia.org/r/#/c/296780/
17:08 jynus: stopping slave on db1073 to test InnoDB compression T139055
16:44 dcausse: restarting elastic1036 (master in eqiad)
16:23 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT:Revert "Use extension registration for TitleBlacklist" (duration: 00m 32s)
16:22 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT:Revert "Use extension registration for TitleBlacklist" (duration: 00m 27s)
16:17 dcausse: truncating elastic logs on elastic1036 and elastic1021
16:16 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Use extension registration for TitleBlacklist (T119117) PART II (duration: 00m 36s)
16:15 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Use extension registration for TitleBlacklist (T119117) PART I (duration: 00m 39s)
15:53 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Use extension registration for LabeledSectionTransclusion (T119117) (duration: 00m 27s)
15:48 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Short array syntax (duration: 00m 30s)
15:39 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: SWAT: Put wikidatawiki back on 1.28.0-wmf.8
15:36 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/Wikidata/extensions/Wikibase/repo/includes/WikibaseRepo.php: SWAT: Update Wikidata - Fix broken editing of statements (T138974) (duration: 00m 25s)
15:35 akosiaris: restarted (actually puppet did) gerrit after merging 4 related changes
15:34 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.8/extensions/Wikidata/extensions/Wikibase/repo/includes/WikibaseRepo.php: SWAT: Update Wikidata - Fix broken editing of statements (T138974) (duration: 00m 31s)
15:27 logmsgbot: thcipriani@tin Synchronized dblists/clldefault.dblist: SWAT: Deploy Compact Language Links as default (Stage 3.5) (T136677) (duration: 00m 26s)
15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Delete old throttle rules (duration: 00m 25s)
15:12 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/enwiktionary.png: SWAT: Revert Change project logo for enwikt (T138801) (duration: 00m 25s)
15:07 logmsgbot: thcipriani@tin Synchronized static/images/project-logos/enwiktionary.png: SWAT: Change project logo for enwikt (T138801) (duration: 00m 25s)
15:04 moritzm: rolling reboot of wtp2 for kernel security update
14:49 mdholloway: mobileapps finished deploying 43538aa
14:46 moritzm: pooling four additional jessie-based image scalers (mw1295-mw1298)
14:45 mdholloway: starting mobileapps deployment
13:42 godog: swift codfw-prod: ms-be202[234] weight 3000
13:10 jynus: upgrading and restarting analytics1003 mysql tables
11:59 moritzm: pooling three additional jessie-based image scalers
10:26 elukey: rebooting stat100[234] and analytics1003 for kernel upgrades
10:20 moritzm: powercycling mw1016, stuck after reboot
09:44 dcausse: truncating current logs again on elastic1045 and elastic1036
09:37 moritzm: powercycling mw1011, stuck after reboot
09:21 dcausse: truncating current logs on elastic1045 and elastic1036
09:19 mobrovac: zotero deployed translators cde2f7531a4
09:13 godog: reboot graphite2001 / graphite1001 to apply trusty kernel update
09:13 dcausse: deleting old logs on elastic1045 and elastic1036
06:57 moritzm: powercycling elastic1015, stuck after reboot
06:37 moritzm: powercycling elastic1014, stuck after reboot
06:26 moritzm: resuming rolling restarts of elasticsearch cluster in eqiad and codfw
06:18 moritzm: rolling restart of mw1001-mw1016 for kernel secuity update
03:00 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jun 30 03:00:28 UTC 2016 (duration 7m 10s)
02:53 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 07m 08s)
02:39 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 17m 23s)
01:33 twentyafterfour: starting phd with only 4 taskmasters to help lighten the load
01:30 twentyafterfour: stopped phd on iridium to investigate large spike in sql insert volume
01:18 mutante: iridium back up, on 3.13.0-91
01:15 mutante: rebooting iridium
00:39 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Don't use always true wmgUseXFFBlocks anymore (2/2) (duration: 00m 25s)
00:38 twentyafterfour: Phabricator upgrade complete, service appears to be stable.
00:32 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Don't use always true wmgUseXFFBlocks anymore (1/2) (duration: 00m 25s)
00:30 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Don't use always true wmgUseXFFBlocks anymore (1/2) (duration: 00m 27s)
00:27 twentyafterfour: Taking phabricator offline momentarily for scheduled update. Expect less than 5 minutes of downtime.
00:25 logmsgbot: maxsem@tin Synchronized wmf-config/: Try again? (duration: 00m 29s)
00:17 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Revert "Cleanup: Move never-altered GlobalBlockingBlockXFF into CommonSettings" (no-op) (duration: 00m 26s)
00:15 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Revert "Cleanup: Move never-altered GlobalBlockingBlockXFF into CommonSettings" (duration: 00m 25s)
00:10 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Clean-up for IS/CS (Gerrit:292615 to Gerrit:292618, no op, 2/2) (duration: 00m 29s)
00:09 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Clean-up for IS/CS (Gerrit:292615 to Gerrit:292618, no op, 1/2) (duration: 00m 28s)
00:08 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.8/extensions/TemplateSandbox: https://gerrit.wikimedia.org/r/#/c/296675/ (duration: 00m 30s)
00:08 logmsgbot: dereckson@tin scap aborted: wmf-config/CommonSettings.php Clean-up for IS/CS (Gerrit:292615 to Gerrit:292618, no op, 1/2) (duration: 00m 20s)
00:07 logmsgbot: dereckson@tin Started scap: wmf-config/CommonSettings.php Clean-up for IS/CS (Gerrit:292615 to Gerrit:292618, no op, 1/2)

2016-06-29

23:21 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Revert "Enable Echo transition flags in production for testing" (duration: 00m 25s)
22:50 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: extdist config for 1.27/1.25 (duration: 00m 31s)
21:43 logmsgbot: krenair@tin Synchronized php-1.28.0-wmf.8/extensions/VisualEditor/ApiVisualEditor.php: https://gerrit.wikimedia.org/r/296661 - VE namespaces issue (duration: 00m 26s)
21:41 chasemp: cleared phab 2fa for ebernhardson for lost phone
21:34 jynus: removing /srv/backups/m2-otrs-* (tranferred to es2001) to make space
21:02 yurik: deployed Graphoid https://gerrit.wikimedia.org/r/#/c/296498/
20:55 yurik: deployed Tilerator https://gerrit.wikimedia.org/r/#/c/296647/
20:50 yurik: deployed Kartotherian https://gerrit.wikimedia.org/r/#/c/296646/
20:21 bearND: mobileapps deployed 1da6bf0
20:15 bearND: starting mobileapps deploy
19:54 logmsgbot: maxsem@tin Synchronized wmf-config/throttle.php: https://gerrit.wikimedia.org/r/#/c/296627/ (duration: 00m 31s)
19:39 mutante: antimony - shutdown -h now (since it's gone from Icinga now)
19:38 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: testwikidata back to wmf.8
19:36 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: Roll back wikidata and testwikidata to 1.28.0-wmf.7 per request by @aude
19:22 mutante: antimony puppetstoredconfigclean.rb to remove icinga monitor remnants
19:14 ostriches: ytterbium: running puppet and reloading replication plugin
19:13 mutante: antimony - stopping gitblit service
19:07 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.8T138555
18:39 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.7/maintenance/dumpBackup.php: Deploy I94ca4a06 (duration: 00m 25s)
18:39 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.7/maintenance/backup.inc: Deploy I94ca4a06 (duration: 00m 24s)
18:37 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.7/includes/export/WikiExporter.php: Deploy I94ca4a06 (duration: 00m 25s)
18:30 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.8/maintenance/dumpBackup.php: Deploy I94ca4a06 (duration: 00m 27s)
18:29 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.8/maintenance/backup.inc: Deploy I94ca4a06 (duration: 00m 33s)
18:28 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.8/includes/export/WikiExporter.php: Deploy I94ca4a06 (duration: 00m 34s)
18:06 mutante: we stopped using gitblit. git.wikimedia.org URLs P3318 T137224
18:05 mutante: git.wm.org URLs switched from gitblit to phab redirects
17:48 ostriches: gerrit: flushed all caches to pick up rename, things may be slow for the next 15m or so
15:49 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Throttling exemption for enwiki (T138167) (duration: 00m 25s)
15:38 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/resources/Resources.php: SWAT: mediawiki.action.edit.stash: Restore dependency to "jquery.getAttrs" (T138931) (duration: 00m 26s)
15:34 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/Flow/maintenance/FlowRemoveOldTopics.php: SWAT: Also delete topics that have more recent updates by (only) talk page manager (T119509) (duration: 00m 25s)
15:29 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/Flow: SWAT: Do not reimport existing header (T119509) (duration: 00m 46s)
15:22 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/Flow/maintenance/FlowRestoreLQT.php: SWAT: Script to restore LQT topics to their pre-import state (T119509) (duration: 00m 26s)
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Echo transition flags in production for testing (duration: 00m 27s)
15:09 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Add $wmgEchoTransition setting for Echo transition flags PART II (duration: 00m 26s)
15:08 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add $wmgEchoTransition setting for Echo transition flags PART I (duration: 00m 50s)
14:46 moritzm: powercycling elastic1012, stuck on reboot
13:22 elukey: rebooting analytics1027 for kernel upgrades
12:58 moritzm: rebooting dataset1001 for kernel update
12:41 moritzm: continuing rolling restarts of elastic* in eqiad and codfw for kernel security update
12:32 moritzm: powercycling elastic1010, stuck on reboot
12:11 moritzm: powercycling mw1260, stuck on reboot
11:38 jynus: halfway moving otrs backups from dbstore1001 to es2001
11:27 gehel: powercycling elastic1009 - stuck in reboot
11:11 moritzm: powercycling mw1223, stuck on reboot
11:09 gehel: deleting broken dewiki_titlesuggest index from codfw (T138811)
10:31 elukey: rebooting analytics100[12] (Hadoop Yarn/HDFS master and standby) - One at the time forcing failover manually with daemon restarts
09:55 moritzm: powercycling mw1163, stuck on reboot
09:23 gehel: banning elastic1001 to 1016 from cluster to prepare their decommissioning (T138329)
09:20 ema: upgrading diamond to 3.5-6 (T138758)
09:01 elukey: rebooting analytics1028->1057 for kernel upgrades (Hadoop worker nodes)
08:55 moritzm: powercycling mw1111, stuck on reboot
08:44 elukey: puppet stopped on analytics1027 to prevent Camus job to run (prep step for Hadoop kernel upgrades)
08:40 moritzm: powercycling mw1108, stuck on reboot
08:12 moritzm: powercycling mw1099, stuck on reboot
08:12 moritzm: powercycling mw1097, stuck on reboot
08:05 moritzm: powercycling mw1092, stuck on reboot
07:47 moritzm: rolling reboot of appservers in eqiad for kernel security update
07:16 moritzm: powercycling snapshot1002, reboot stuck
07:11 moritzm: powercycling snapshot1001, reboot stuck
06:58 moritzm: rebooting most snapshot hosts for kernel security update
03:28 logmsgbot: krinkle@tin Synchronized wmf-config/InitialiseSettings.php: test2wiki (duration: 00m 33s)
02:56 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jun 29 02:56:32 UTC 2016 (duration 6m 30s)
02:50 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.8) (duration: 04m 48s)
02:30 chasemp: labstore1004 is replicating NFS/DRBD shares to labstore1005 and they are large and it's taking a long time
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 09m 21s)
02:18 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: sync wikiversions.json - group0 to 1.28.0-wmf.8 refs T137492
02:16 twentyafterfour: promoting group0 to 1.28.0-wmf.8
00:02 logmsgbot: twentyafterfour@tin Finished scap: sync new branch, testwiki to php-1.28.0-wmf.8 refs T137492 (duration: 51m 59s)

2016-06-28

23:10 logmsgbot: twentyafterfour@tin Started scap: sync new branch, testwiki to php-1.28.0-wmf.8 refs T137492
23:10 Krenair: wikitech-static working now, poke me on IRC or file a #wikitech.wikimedia.org ticket if you find any issues
23:10 twentyafterfour: syncing new branch 1.28.0-wmf.8 refs T137492
23:04 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.7/extensions/EventBus/EventBus.php: SWAT: EventBus: Match the expected format of response log key (duration: 00m 31s)
23:01 Krenair: Updating MW version on wikitech-static to 1.27 (LTS) - https://lists.wikimedia.org/pipermail/mediawiki-announce/2016-June/000191.html
21:59 halfak: deploying ores beec291
21:33 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.7
21:31 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.7/extensions/AbuseFilter/: deploy https://gerrit.wikimedia.org/r/#/c/296464/ refs T138550 T136973 (duration: 00m 36s)
21:24 twentyafterfour: deploying wmf.7 yet again, once CI finishes testing https://gerrit.wikimedia.org/r/#/c/296464/ refs T138550 T136973
20:24 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: once again rolling back to wmf.6 refs T136973 T138550
20:11 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.7
20:09 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.7/extensions/AbuseFilter/: deploying https://gerrit.wikimedia.org/r/#/c/296440/ refs T138550, T136973 (duration: 02m 06s)
20:09 twentyafterfour: deploying https://gerrit.wikimedia.org/r/#/c/296440/ to hopefully unblock wmf.7 deployments. refs T138550, T136973
20:08 gehel: disabling puppet on wdqs100[12] to cleanup after failed scap3 deplyoment
19:33 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: Rolling back to wmf.6: save time regression is still present in wmf.7
19:32 twentyafterfour: Rolling back to wmf.6: T138550 is still a problem
19:24 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.7
19:23 twentyafterfour: Deploying 1.28.0-wmf.7 to all wikis
18:23 mutante: zosma - fresh install, sign puppet certs, initial puppet run
16:16 gehel: starting rolling restart of elasticsearch codfw cluster (T138811)
15:25 logmsgbot: thcipriani@tin Synchronized portals: SWAT: Bumping portals to master (T136874) (duration: 00m 29s)
15:24 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: Bumping portals to master (T136874) (duration: 00m 24s)
15:16 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor by default for all users of the French (T136993), English (T136992), and German (T136991) Wikivoyage (duration: 00m 24s)
15:09 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor by default for all users of the Italian Wikivoyage (T136994) (duration: 00m 25s)
14:52 gehel: powercycling elastic1004 (server not coming up during restart - T138811)
13:47 godog: bounce carbon on graphite machines after applying https://gerrit.wikimedia.org/r/266567
13:40 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: aqs1001.eqiad.wmnet
12:50 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1023, 24, 33, 35, 39, 44, 52, 61, 64, 68, 63, 67, 72, 73 (duration: 02m 39s)
12:00 gehel: powercycling elastic1002 (server not coming up during restart - T138811)
11:43 gehel: powercycling elastic1001 (server not coming up during restart - T138811)
11:21 gehel: rolling restart of elasticsearch eqiad
10:44 moritzm: rolling reboot of mediawiki in codfw for kernel security update
09:39 moritzm: powercycling mw1021, didn't come up after reboot
09:32 elukey: restarted hhvm on mw1238, memory pressure ok but hhvm stuck (hhvm-dump-debug in /tmp/hhvm.14788.bt.)
09:28 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: aqs1003.eqiad.wmnet
09:25 moritzm: powercycling mw1019, didn't come up after reboot
09:25 logmsgbot: reedy@tin Synchronized wmf-config/interwiki.php: Updated IW map (duration: 00m 49s)
09:13 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: aqs1003.eqiad.wmnet
08:57 moritzm: powercycling mw1018, didn't come up after reboot
08:47 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Prepare old servers for decom by sending all queries to new servers (duration: 01m 39s)
08:32 moritzm: rolling reboot of mediawiki canaries for kernel security update
08:30 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: aqs1002.eqiad.wmnet
08:17 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: aqs1002.eqiad.wmnet
08:15 elukey: rebooting aqs100[23].eqiad for kernel upgrades
02:54 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jun 28 02:54:56 UTC 2016 (duration 7m 16s)
02:47 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 08m 59s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 51s)
00:26 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.7/includes/api/ApiMain.php: UsageException to try to catch T138585 issue (duration: 00m 27s)
00:21 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Wikibase descriptions on Catalan and Polish wikis (T135429) (duration: 00m 26s)
00:09 logmsgbot: dereckson@tin Synchronized wmf-config/mobile.php: Introduce config variable to control tagline (T138738, 2/2) (duration: 00m 27s)
00:08 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Introduce config variable to control tagline (T138738, 1/2) (duration: 00m 32s)
00:07 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Introduce config variable to control tagline (no-op) (duration: 00m 27s)
00:05 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.6/extensions/MobileFrontend/: Introduce config variable to control tagline (T138738) (duration: 00m 29s)
00:02 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.7/extensions/MobileFrontend/: Introduce config variable to control tagline (T138738) (duration: 00m 39s)

2016-06-27

20:13 mdholloway: mobileapps deployed 30cc12e
20:08 subbu: finished deploying parsoid sha dd8e644d
20:04 subbu: synced new parsoid code; restarted parsoid on wtp1001 as a canary
20:01 subbu: starting parsoid deploy
17:23 gehel: deploying new logstash config for transition to elasticsearch 2.x (T138335)
15:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Increase move rate limit for extendedmovers in enwiki to 16/60 (duration: 00m 28s)
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Delete old throttle rules (duration: 00m 26s)
15:16 gehel: banning elastic1001 to prepare its decommissioning (T138329)
15:13 logmsgbot: thcipriani@tin Synchronized dblists/clldefault.dblist: SWAT: Deploy Compact Language Links as default (Stage 3) PART II (duration: 00m 23s)
15:07 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Deploy Compact Language Links as default (Stage 3) (duration: 00m 40s)
15:00 elukey: mw1136 powercycled - not responsive to ssh and root login
14:49 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic101[0-6].eqiad.wmnet
14:39 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic100[0-9].eqiad.wmnet
14:37 logmsgbot: gehel@palladium conftool action : get/pooled; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic100[0-9]..eqiad.wmnet
14:34 gehel: removing old elasticsearch servers in eqiad from LVS (elastic1001-1016 - T138329)
10:10 moritzm: pooled mw1291 (jessie imagescaler)
09:48 jynus: stopping and reimporting db2010 (m1)
09:47 gehel: removing maps-test*.codfw.wmnet servers from LVS (T138092)
09:19 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch-ssl,name=elastic104..eqiad.wmnet
09:19 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic104..eqiad.wmnet
09:18 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch-ssl,name=elastic103..eqiad.wmnet
09:18 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: dc=eqiad,cluster=elasticsearch,service=elasticsearch,name=elastic103..eqiad.wmnet
09:10 logmsgbot: gehel@palladium conftool action : get/pooled; selector: elastic10??\.eqiad\.wmnet (tags: ['dc=eqiad', 'cluster=elasticsearch', 'service=elasticsearch'])
09:07 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: elastic1032.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=elasticsearch', 'service=elasticsearch-ssl'])
09:06 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: elastic1032.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=elasticsearch', 'service=elasticsearch'])
09:00 gehel: adding new elasticsearch servers in eqiad to LVS
08:54 godog: swift codfw-prod ms-be202[234] weight 2000
07:15 elukey: puppet stopped on analytics1049 to remove it completely from the Hadoop cluster - broken disk
02:51 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jun 27 02:51:41 UTC 2016 (duration 7m 5s)
02:44 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 08m 09s)
02:27 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 54s)

2016-06-26

02:52 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jun 26 02:52:48 UTC 2016 (duration 6m 19s)
02:46 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 08m 15s)
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 48s)

2016-06-25

09:37 mutante: install2001 killing ganglia aggregator processes, running puppet, for debugging
02:51 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jun 25 02:51:43 UTC 2016 (duration 6m 26s)
02:45 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 07m 58s)
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 53s)
01:07 chasemp: sign labstore1005 puppet certs and bootstrap the server
00:53 chasemp: hand hack apache on labmon to make it work temporarily

2016-06-24

18:41 logmsgbot: krenair@tin Synchronized dblists/mobilemainpagelegacy.dblist: https://gerrit.wikimedia.org/r/#/c/295958/4 - fix mobile main page rendering on a bunch of wikis, effectively putting them back to how they were a few days ago (duration: 00m 37s)
17:19 mobrovac: change-prop deploying df88a75b
17:05 _joe_: re-started changeprop after disabling the dependency module
14:18 paravoid: shutting down ms-fe3002 due to on-site work
14:05 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.7/includes/OutputPage.php: T138586 hotfix (duration: 00m 47s)
14:02 mobrovac: scb100x disabled puppet to clear changeprop queues
13:22 gehel: re-enabling puppet on maps1002 (still in pre-configuration state, only default role)
12:34 hashar: Random resource loader entries are apparently faulty causing issues with css and/or javascript T138586
12:04 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: aqs1001.eqiad.wmnet
12:03 elukey: rebooting aqs1001.eqiad.wmnet for kernel upgrades
10:55 jynus: updated m1-slave dns to be db1001
10:20 hashar: gallium: restarted apache2 , potentially stuck proxy
10:18 moritzm: upgrade nodejs on scb systems in codfw and restart node-based services
09:59 ema: nginx rolling restart to enable TFO on all tlsproxies (T108827)
09:52 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1059 with low weight, increase weight of db1061, db1062 (duration: 00m 33s)
09:48 moritzm: upgrade nodejs on restbase test systems (xenon/praseodymium/cerium/restbase-test) and restart restbase on those
09:09 mobrovac: scb100x stopping puppet to stop change-prop and clear the queue
08:29 moritzm: uploaded nodejs 4.4.6 for jessie-wikimedia to carbon
07:10 elukey: memcached on mc1007 restarted with growth factor 1.05 (T129963)
03:54 robh: data copy for labmon1001 verified complete with proper permissions, re-enabling and running puppet to start back up services
03:19 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jun 24 03:19:55 UTC 2016 (duration 7m 4s)
03:12 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 17m 24s)
02:38 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 17m 08s)
01:22 bblack: stream.wikimedia.org (RCStream) DNS moved to cache_misc termination. If anyone reports bugs with rcstream services, revert https://gerrit.wikimedia.org/r/295385

2016-06-23

23:17 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/295600/ (duration: 00m 29s)
23:15 logmsgbot: maxsem@tin Synchronized dblists/mobilemainpagelegacy.dblist: https://gerrit.wikimedia.org/r/#/c/295600/ (duration: 00m 28s)
22:33 chasemp: reimage labstore1005 post io testing
22:12 chasemp: powercycle labstore1005
21:24 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group2 wikis to wmf.6
21:11 chasemp: silence alerts for labstore1004 for setup
20:31 ebernhardson: synced out latest logstash-plugins via trebuchet
20:17 Dereckson: Run initSiteStats.php on cebwiki (T138533)
20:04 logmsgbot: jzerebecki@tin Synchronized wmf-config/CommonSettings.php: Log PHP/HHVM errors in CLI mode to stderr, not stdout T138291 (duration: 00m 28s)
20:03 robh: labmon1001 data restore at 100gb 50minutes in, 298gb total for restoration
19:29 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.7
19:24 greg-g: 19:21 < RoanKatto> !log Synced patches for T137288 and T137593
18:31 elukey: mw130[0134] - new jobrunners installed and pooled (happened automatically after the fist puppet run)
18:09 robh: labmon1001 powering down for reimage
17:45 subbu: finished deploying parsoid sha 18022c96
17:40 subbu: synced new code; restarted parsoid on wtp1001 as a canary
17:37 subbu: starting parsoid deploy
17:29 robh: labmon1001 cpy changed back to local usb, errors on network transfer for ownership. resumed rsync with append flag to local usb disk.
17:03 bblack: cache perf tuning marker: start rollout of tcp_no_metrics_save:0
16:27 chasemp: remove old log files on ytterbium for T114395
16:18 godog: swift: add ms-be202[234] weight 1000 - T136630
15:31 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings-labs.php: SWAT: LABS: Enable geoshapes graph protocol (duration: 00m 29s)
15:26 akosiaris: stop etherpad-lite, etherpad is down
15:16 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy Compact Language Links as default (Stage 2) PART III (duration: 00m 24s)
15:16 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Deploy Compact Language Links as default (Stage 2) PART II (duration: 00m 28s)
15:15 logmsgbot: thcipriani@tin Synchronized dblists/clldefault.dblist: SWAT: Deploy Compact Language Links as default (Stage 2) PART I (duration: 00m 41s)
15:11 robh: puppet disabled on labmon1001 along with all icinga alerting. data migration to usb in progress via root screen session
15:05 robh: starting data backup of labmon1001, halting statsite/graphite/carbon-relay on system
14:47 akosiaris: change the default message in etherpad to indicate problems
14:47 mobrovac: change-prop deploying 05c72ed24ca
14:45 akosiaris: debugging etherpad. Started the service with a blank db, looks like it's working
14:38 akosiaris: stopping etherpad-lite on etherpad1001, disabling puppet
14:32 jynus: restarting etherpad-lite.service
13:53 hashar: Zuul/CI are slowly catching up. I had to drop a few changes that got force merged on the SmashPig repo.
13:37 awight: update SmashPig from a435adeb130217bda8b95d3c5c6331ace8ad1228 to 917138e159f0341e3dfbb35818c3ce479927875b
13:36 hashar: CI is slowed down due to surge of jobs and lack of instances to build them on ( T133911 ). Queue is 50 for Jessie and 25 for Trusty.
13:30 jynus: db1059 backup and reimage
13:28 awight: update SmashPig from c0cc2a1a6062ad8d114473ea1a444786a0d50833 to a435adeb130217bda8b95d3c5c6331ace8ad1228
13:16 jynus: running scap pool on mw1301
13:13 mobrovac: restarting zotero on sca, 6g mem
13:13 jynus: running scap pool on mw1300
13:11 mobrovac: citoid deploying 0129ab0b
13:11 elukey: purged some puppet output logs on compiler02.puppet3-diffs.eqiad.wmflabs to free space (disk full)
13:09 moritzm: depooled jessie image scaler (mw1291) again, works fine, to be permanently pooled on Monday
12:49 moritzm: pooling new jessie image scaler mw1291 for short production smoke testing
12:35 awight: update SmashPig from f7d65c54bed3ff9c478b0dbcaa1b2d27cc665ace to c0cc2a1a6062ad8d114473ea1a444786a0d50833
12:18 awight: update SmashPig from 90757321a3bfa1045202e06e3dd1960a0043493a to f7d65c54bed3ff9c478b0dbcaa1b2d27cc665ace
12:07 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1059; Repool db1061 & db1062; increase weight of db1068 (duration: 00m 39s)
11:33 gehel: rolling restart of elasticsearch10(01|30|08|36|13|40) to activate new masters
10:13 andrewbogott: restarting rabbitmq-server on labcontrol1001 (random debugging attempt for T138106)
09:49 godog: reimage ms-be202[567] with incorrect raid settings
09:11 jynus: syncing etherpadlite.store (m1) on db2010, which had 2 bad chunks
08:39 mobrovac: change-prop restarting on scb to pick up ores rules https://gerrit.wikimedia.org/r/295576
08:06 mobrovac: change-prop deploying 45db4f84827
06:59 moritzm: installing spice security updates
02:48 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jun 23 02:47:59 UTC 2016 (duration 6m 44s)
02:41 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 07m 05s)
02:26 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 11m 19s)

2016-06-22

23:24 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/295560/ (duration: 00m 25s)
23:23 logmsgbot: maxsem@tin Synchronized wmf-config/CommonSettings.php: https://gerrit.wikimedia.org/r/#/c/295560/ (duration: 00m 24s)
23:23 logmsgbot: maxsem@tin Synchronized dblists/mobilemainpagelegacy.dblist: https://gerrit.wikimedia.org/r/#/c/295560/ (duration: 00m 24s)
23:14 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/294247/ (duration: 00m 24s)
23:09 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/295558/ (duration: 00m 40s)
22:25 ori: Ran hacked maintain-replicas.pl on labsdb100[13] for T135029
21:06 bblack: cache perf: start deploy of -autocorking (probably last experiment I can squeeze in today)
21:00 Dereckson: Run namespaceDupes.php on ptwikinews (T138230) and frwikinews (T138442)
20:33 mdholloway: mobileapps: finished deploying 8046ee2
20:26 yurik: deployed & restarted tilerator https://gerrit.wikimedia.org/r/#/c/295447/
20:25 mdholloway: starting mobileapps deployment
20:20 Reedy: created tmplog_begin_devices on tmplog_end_devices on testwiki.cn_template_log
20:18 yurik: deployed & restarted kartotherian https://gerrit.wikimedia.org/r/#/c/295449/
19:32 bblack: start rollout of first batch of cache sysctl stuff (un-mysterious + disable prequeue timestamps)
19:29 jynus: archiving and dropping reviewdb on m1 shard
19:06 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.7
18:46 jynus: shutting down and reimaging db1001
18:20 papaul: ms-be202[3-7] - signing puppet certs, salt-key, initial run
17:23 akosiaris: restart apache on ununpentium for m1 migration. Hosts RT, just did it for good measure
17:21 akosiaris: restarted bacula-director on helium
17:15 jynus: killing puppet, rt, librenms user connections on db1001
17:10 jynus: failovered m1-master from db1001 to db1016
16:20 gehel: new elasticsearch servers elastic1032-1047 are configured and have joined the eqiad cluster
15:26 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.6/extensions/OATHAuth: SWAT: Fixup qrcode-generating js, to stop race condition. (duration: 00m 33s)
15:23 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Improve style (duration: 00m 33s)
15:18 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/OATHAuth: SWAT: Fixup qrcode-generating js, to stop race condition. (duration: 00m 27s)
15:13 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add www.wpc.ncep.noaa.gov to wgCopyUploadsDomains (duration: 00m 54s)
15:01 elukey: rebooting bohrium.eqiad.wmnet (running piwik) for kernel upgrades
14:32 jynus: checksumming m1 databases in preparation for failover
14:29 tgr: running https://phabricator.wikimedia.org/diffusion/ECAU/browse/master/maintenance/checkLocalUser.php for some users T119736
14:04 moritzm: rolling restart of hhvm/apache on app servers in eqiad for expat security update
13:42 godog: add 500G to fluorine /a (almost full)
13:31 gehel: configuring new elasticsearch servers elastic1038-1042 in eqiad
13:03 hashar: Manually moved some missing build records. Restarting Jenkins
12:49 hashar: T80385 Restarting Jenkins with builds dir set to "${JENKINS_HOME}/builds/${ITEM_FULL_NAME}" which is /var/lib/jenkins/builds/XXX
12:35 gehel: starting reimage of mw1292
12:34 _joe_: disabling puppet on mw1017, live-hacking it
12:34 hashar: T80385 stopping Jenkins and migrating all build records to /var/lib/jenkins/builds
12:06 gehel: configuring new elasticsearch servers elastic1033-1037 in eqiad
10:46 godog: upload libphutil/arcanist 0~git20160620-0wmf1 to carbon
10:32 elukey: mw1140 powercycle after freeze issues due to memory pressure (was not able to ssh to it)
10:18 moritzm: rolling restart of restbase in eqiad to pick up firejail change in service::node
09:46 moritzm: rolling restart of restbase in codfw to pick up firejail change in service::node
09:43 legoktm: live-hacking on mw1017 to debug T115119
09:19 jynus: stopping and reconfiguring mysql on dbstore1001
07:59 moritzm: rolling restart of hhvm/apache on canary app servers in eqiad for expat security update
07:30 jynus: stopping, backing up and reimaging db1061 and db1062
07:06 moritzm: restarted hhvm on mw1131
04:29 chasemp: fix salt key on labtestmetal2001
03:12 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jun 22 03:12:33 UTC 2016 (duration 6m 44s)
03:05 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.7) (duration: 17m 49s)
02:31 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 24s)

2016-06-21

23:14 yurik: updated/restarted kartotherian & tilerator - https://gerrit.wikimedia.org/r/#/c/295440/ https://gerrit.wikimedia.org/r/#/c/295441/
23:05 tgr: deleted localuser rows for Mahir256@orwikisource and A879071@enwiki for T119736
22:19 bd808: Backfilled missing 2016-06-20 data to https://tools.wmflabs.org/sal/production?d=2016-06-20
22:08 logmsgbot: ori@tin Synchronized static/images/mobile: I8f09e825: Optimize mobile static images (duration: 00m 34s)
19:27 bd808: Restarted dead logstash process on logstash1001. Looks to have stopped itself due to the the Elasticsearch OOM earlier
19:18 logmsgbot: thcipriani@tin Purged l10n cache for 1.28.0-wmf.5
19:17 bd808: Restarted ElasticSearch on logstash1001; dead from OOM
19:14 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.7
18:50 bblack: enabled tcp_notsent_lowat optimization on all caches (marking this time for investigation of perf graphs later) - https://gerrit.wikimedia.org/r/#/c/295376/
17:16 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.7/extensions/Graph/lib/graph2.compiled.js: pre-train backport: Updated to latest graph2 lib (duration: 00m 31s)
17:10 yurik_: deployed graphoid https://gerrit.wikimedia.org/r/#/c/295367/
17:06 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: Temporary IP Cap Lift on es.wiki and commons (duration: 00m 24s)
16:33 yurik_: deployed and restarted graphoid with scap3
16:32 gehel: starting installation of new elasticsearch server elastic1032.eqiad.wmnet
15:58 gehel: puppet run on tin to enable scap3 deployment for graphoid
15:53 logmsgbot: catrope@tin Synchronized php-1.28.0-wmf.7/extensions/Echo/: (no message) (duration: 00m 33s)
15:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy Compact Language Links as default (Stage 1) (duration: 00m 25s)
15:42 logmsgbot: thcipriani@tin Synchronized wmf-config/db-eqiad.php: Repool db1068 with low weight; depool db1061 and db1062 (duration: 00m 30s)
15:20 logmsgbot: hashar@tin Finished scap: testwiki to group0 (previously was labtestwiki which does not work) (duration: 51m 45s)
14:47 moritzm: rolling restart of aqs service on aqs1001-aqs1006 to pick up new firejail settings
14:28 logmsgbot: hashar@tin Started scap: testwiki to group0 (previously was labtestwiki which does not work)
14:14 moritzm: correction: restbase1007 was already depooled for cassandra maintenance, thus only rebooting to 4.4
14:12 moritzm: depooling restbase1007 for upgrade to Linux 4.4
14:09 logmsgbot: hashar@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="labtestwiki" --outdir="/tmp/scap_l10n_87423667" --threads=4 --lang en --quiet' returned non-zero exit status 255 (duration: 02m 58s)
14:06 logmsgbot: hashar@tin Started scap: (no message)
14:03 gehel: disabling alerting for maps100?\.eqiad\.wmnet during initial installation
14:02 logmsgbot: hashar@tin scap failed: CalledProcessError Command '/usr/local/bin/mwscript rebuildLocalisationCache.php --wiki="labtestwiki" --outdir="/tmp/scap_l10n_2087727834" --threads=4 --lang en --quiet' returned non-zero exit status 255 (duration: 06m 37s)
13:55 logmsgbot: hashar@tin Started scap: testwiki to 1.28.0-wmf.7 (take three) T136973
13:55 logmsgbot: hashar@tin scap aborted: testwiki to 1.28.0-wmf.7 (take two) T136973 (duration: 01m 35s)
13:53 logmsgbot: hashar@tin Started scap: testwiki to 1.28.0-wmf.7 (take two) T136973
13:53 logmsgbot: hashar@tin scap aborted: testwiki to 1.28.0-wmf.7 T136973 (duration: 04m 17s)
13:48 logmsgbot: hashar@tin Started scap: testwiki to 1.28.0-wmf.7 T136973
13:15 hashar: T136973 applied all security patches to 1.28.0-wmf.7
13:11 RoanKattouw: Running extensions/Echo/maintenance/removeOrphanedEvents.php on all Echo-enabled wikis for T136425
12:57 moritzm: rolling restart of hhvm/apache in codfw for expat security update
12:49 RoanKattouw: Running extensions/Echo/maintenance/backfillReadBundles.php on all Echo-enabled wikis for T136368
12:49 RoanKattouw: Running extensions/Echo/maintenance/backfillReadBundles.php on all Echo-enabled wikis
12:36 hoo: Started a new JSON dump creation on snapshot1003 (after the last one was inconsistent, per T138291)
12:35 gehel: lowering throttling limit for index recovery on codfw elasticsearch cluster
12:33 hoo: Removed Wikidata json dumps from 20160620 (inconsistent, per T138291).
12:30 hashar: T136973 started cut of branch wmf/1.28.0-wmf.7
12:25 gehel: lowering throttling limit for index recovery on eqiad elasticsearch cluster
11:06 jynus: reimaging db1068
10:32 godog: reboot ms-be2003 for disk ordering - T137785
10:22 moritzm: installing expat security updates on Ubuntu systems
10:03 moritzm: installing wget security updates on Ubuntu systems
09:43 gehel: lowering disk high watermark to rebalance elasticsearch eqiad cluster disk space
09:25 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1068; repool db1070 and db1071 as api (duration: 00m 27s)
09:22 moritzm: rolling reboot of logstash cluster to Linux 4.4
07:41 elukey: restarted hhvm on mw1141 - hhvm was getting SEGV (dump in /tmp/hhvm.8735.bt.)
07:39 elukey: restarted hhvm on mw1139 (hhvm-dump in /tmp/hhvm.20736.bt.)
06:41 moritzm: restarted hhvm on mw1252
02:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jun 21 02:10:55 UTC 2016 (duration 6m 36s)
02:04 logmsgbot: l10nupdate@tin LocalisationUpdate failed (1.28.0-wmf.6) at 2016-06-21 02:04:19+00:00

2016-06-20

23:22 Dereckson: `mwscript namespaceDupes.php ptwikinews --fix` (T138230). Some links and revisions are still to fix.
23:16 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Fix pt.wikinews namespace issue (T138230) (duration: 00m 24s)
23:13 logmsgbot: dereckson@tin Synchronized wmf-config/mobile.php: Remove old mobile workaround for Wikidata descriptions (T127250, T138085) (duration: 00m 33s)
21:05 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.6/extensions/Wikidata: Fix property suggester (duration: 01m 59s)
19:50 chasemp: cleaning up /scratch NFS share as it ran out of inodes
19:17 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes/api/ApiStashEdit.php: 82e14dc66f478fbdb9ca6eab1eeb4f9c68c99bd1 (duration: 00m 36s)
18:09 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1071 with low weight after maintenance (duration: 00m 26s)
17:32 bd808: https://tools.wmflabs.org/sal missing events between 2016-06-19T12:29 and 2016-06-20T17:26.
17:26 gehel: deploying latest WDQS
17:19 godog: upload libphutil / arcanist 0~git20160616-0wmf1 to jessie-wikimedia T137770
17:18 mark: Rebooting pfw-codfw
17:00 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: revert cll patch (duration: 00m 25s)
15:44 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Allow sysops to add to/remove from confirmed on ca.wikinews (duration: 00m 25s)
15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable NewUserMessage on pl.wikipedia (duration: 00m 25s)
15:31 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.6/extensions/CentralAuth: SWAT: Split CentralAuthUser::queryAttached into cheap and expensive part (duration: 00m 31s)
15:20 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable Flow beta feature on frwikiquote (duration: 00m 28s)
15:13 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy Compact Language Links as default (Stage 1) PART III (duration: 00m 30s)
15:12 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Deploy Compact Language Links as default (Stage 1) PART II (duration: 00m 29s)
15:12 logmsgbot: thcipriani@tin Synchronized dblists/cll-nondefault.dblist: SWAT: Deploy Compact Language Links as default (Stage 1) PART I (duration: 00m 29s)
15:11 logmsgbot: jmm@palladium conftool action : select; selector: name=mw1099.eqiad.wmnet
15:04 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor by default for all users of French Wikinews (duration: 00m 29s)
13:27 elukey: restarted hhvm on mw1145 after temp. freeze due to memory pressure (hhvm debug in /tmp/hhvm.17794.bt.)
13:27 paravoid: reactivating peerings with Telia Carrier/AS1299 (eqiad/codfw/ulsfo)
13:06 Amir1: full deployment for 8e65182 in ores nodes
13:04 Amir1: deploying 8e65182 to scb2001
12:56 gehel: installing maps1001.eqiad.wmnet (secondary cluster, no traffic there yet) - T138092
12:56 paravoid: deactivating peerings with Telia Carrier/AS1299 (eqiad/codfw/ulsfo)
12:41 moritzm: rebooting ms1001 for update to Linux 4.4
12:13 Amir1: started deploying ores in scb2001 bdc1e2bd
11:36 godog: roll-restart swift on ms-be1* to apply https://gerrit.wikimedia.org/r/294691
11:27 Amir1: for ores in scb nodes
11:27 Amir1: rollbacking ae71d842dfc0958e06922062dd09d49243332a6a
11:13 _joe_: restarting uwsgi orse service
10:58 Amir1: deploying bdc1e2b in ores nodes
10:53 godog: roll-restart swift on ms-be2* to apply https://gerrit.wikimedia.org/r/294691
10:44 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1071 completelly (duration: 00m 25s)
10:35 jynus: db1071 stop, backup and reimage
10:31 mobrovac: restbase started mobile-sections dump for eswiki on restbase1009 for T136964
10:05 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1073 at 100% weight; depool db1071 for reimaging (duration: 00m 27s)
09:50 moritzm: rolling reboot of restbase2001/restbase2002 for upgrade to Linux 4.4
08:57 Amir1: deploying 5dfe738 in ores nodes
08:15 moritzm: installing libxlst security updates
07:43 gehel: rebalancing shards on elasticsearch eqiad cluster
06:47 _joe_: activating the jessie jobrunner, mw1299
05:57 logmsgbot: ori@tin Synchronized wmf-config/CommonSettings.php: Id5804a80: Better cache headers for 'Powered by MediaWiki' badge (2/2) (duration: 00m 35s)
05:56 logmsgbot: ori@tin Synchronized static/images: Id5804a80: Better cache headers for 'Powered by MediaWiki' badge (1/2) (duration: 00m 33s)
02:29 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jun 20 02:29:01 UTC 2016 (duration 5m 44s)
02:23 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 09m 54s)

2016-06-19

12:29 elukey: restarted hhvm on mw1138 - trace in /tmp/hhvm.25048.bt, hhvm killed by OOM
12:27 elukey: restarted hhvm on mw1114 - trace in /tmp/hhvm.11092.bt, hhvm killed by OOM
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jun 19 02:31:25 UTC 2016 (duration 5m 47s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 50s)

2016-06-18

02:32 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jun 18 02:32:26 UTC 2016 (duration 6m 18s)
02:26 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 10m 04s)

2016-06-17

21:21 urandom: Reenabling puppet and resetting configuration on xenon.eqiad.wmnet : T137419
20:39 urandom: Restarting Cassandra on xenon.eqiad.wmnet to apply -XX:+PreserveFramePointer : T137419
20:35 urandom: Disabling puppet on xenon.eqiad.wmnet : T137419
20:23 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.6/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/294958/ (duration: 00m 33s)
18:56 urandom: Restarting Cassandra on xenon.eqiad.wmnet with -XX:+PreserveFramePointer : T137419
18:32 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1073 with low weight after reimage (duration: 00m 35s)
16:29 moritzm: installing squid security updates on carbon
15:59 urandom: Starting html dumps from xenon.eqiad.wmnet and cerium.eqiad.wmnet : T137419
15:54 urandom: Restarting Cassandra on xenon.eqiad.wmnet to enable large pages : T137419
14:55 mobrovac: scb disabling puppet for stopping change-prop to clear transclusion queues
14:16 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase db1072 weight after repooling (duration: 00m 36s)
12:57 jynus: stopping, backuping and reimaging db1073
12:49 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1072 with low weight, depool db1073 (duration: 00m 27s)
12:49 moritzm: rolling reboot of mw1157-mw1160 into new kernels
12:27 moritzm: restarted hhvm on mw1133 and mw1135
11:14 moritzm: stopping puppet on hosts using service::node (restbase, sca, scb, aqs) for step-by-step rollout of two puppet patches for firejail/service::node
09:31 _joe_: powercycling mw1140, OOMd
09:30 moritzm: rolling reboot of mw1153,mw1155,mw1156 into new kernels
08:29 hashar: Restarting Jenkins on gallium. Web interface at least is deadlocked somehow
07:23 jynus: backuping and reimaging db1072
07:18 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1072 for maintenance (duration: 00m 31s)
07:11 mobrovac: restbase started mobile-sections dump on restbase1009 for T136964
07:02 mobrovac: change-prop restarting it to apply https://gerrit.wikimedia.org/r/294880
06:40 moritzm: installing apache update on palladium
06:16 akosiaris: _joe_ restarted zotero on sca1001
06:16 akosiaris: restarted zotero on sca1002
06:04 logmsgbot: root@palladium conftool action : set/weight=25; selector: cluster=api_appserver,name=mw127.*
05:58 logmsgbot: root@palladium conftool action : set/pooled=yes:weight=20; selector: cluster=api_appserver,name=mw127.*
02:31 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jun 17 02:31:00 UTC 2016 (duration 6m 26s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 09m 46s)

2016-06-16

23:44 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T137167: TextCat A/B test for Language Identification (duration: 00m 25s)
23:24 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.6/extensions/WikimediaEvents/extension.json: T137167: TextCat A/B test for Language Identification (duration: 00m 24s)
23:19 logmsgbot: ebernhardson@tin Synchronized php-1.28.0-wmf.6/extensions/WikimediaEvents/modules/ext.wikimediaEvents.searchSatisfaction.js: T137167: TextCat A/B test for Language Identification (duration: 00m 24s)
23:16 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: T137167: search: Dependent config for textcat AB test. (duration: 00m 26s)
23:11 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings.php: T137888: Two permission changes at urwiki (duration: 00m 27s)
23:07 logmsgbot: ebernhardson@tin Synchronized wmf-config/InitialiseSettings-labs.php: T127250: Prepare Wikidata descriptions on mobile for production rollout (duration: 00m 27s)
22:33 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.6/extensions/Kartographer: https://gerrit.wikimedia.org/r/294856 https://gerrit.wikimedia.org/r/294855 (duration: 00m 30s)
22:24 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/294854/ (duration: 00m 26s)
21:15 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.6/extensions/VisualEditor/ApiVisualEditor.php: Pass empty summary to parseAndStash() to avoid warnings T137995 (duration: 00m 39s)
19:05 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.6
18:37 tgr: running invalidateUserSessions.php for T137799
18:22 mobrovac: change-prop deploying bc87a1fecfa
16:36 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Set all new slaves to medium weight (300) after warm up (duration: 00m 25s)
15:37 jynus: deleted sqldata.s6 from labsdb1008 - space issues caused by queries creating temporary tables
15:27 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.6/extensions/ORES/includes/Hooks.php: SWAT: Performance boost on hidenondamaging (duration: 00m 35s)
15:23 moritzm: rolling reboot of restbase1008 - restbase1011 for upgrade to Linux 4.4
15:21 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.6/extensions/ORES: SWAT: Skip when an edit is errored in PopulateDatabase.php (duration: 00m 30s)
15:04 logmsgbot: root@palladium conftool action : set/pooled=yes; selector: name=mw1262.eqiad.wmnet
14:31 twentyafterfour: re-enabled and ran puppet agent --test on iridium. Everything appears to be normal.
13:04 mobrovac: scb1001 enabled puppet back
12:57 gehel: rebalancing shards on elasticsearch equiad cluster
12:33 Amir1: manually restarted celery-ores-worker in scb1001
12:32 moritzm: installing apache2 trusty update on graphite1001
12:32 Amir1: manually restarted celery-ores-worker in scb1002
12:10 moritzm: restarted hhvm on mw1137, got stuck
10:44 moritzm: depooling mw1154 for kernel update/reboot
10:14 mobrovac: scb1001 disabling puppet for a while to manually test changeprop with transclusion rules
09:59 mobrovac: restbase deploy end of ebeaa46
09:56 _joe_: powercycling mw1143, unresponsive on ssh, console
09:48 mobrovac: restbase deploy start of ebeaa46
09:18 logmsgbot: hashar@tin Synchronized php-1.28.0-wmf.6/extensions/MobileFrontend: MobileFrontend RL registration issue preventing Special:Nearby from working properly T137919 (duration: 00m 36s)
08:41 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1085, increase weight of all new db servers (duration: 00m 29s)
08:15 jynus: rebooting db1085 before putting it back into production
02:34 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 15m 49s)
00:57 twentyafterfour: puppet disabled on iridium because https://gerrit.wikimedia.org/r/#/c/294653/ needs to merge (hotfix in preamble.php which puppet will undo if it's allowed to run)
00:43 twentyafterfour: phabricator upgrade/maintenance complete. Everything appears to be back up and running normally.
00:41 twentyafterfour: taking phabricator offline momentarily for scheduled maintenance.
00:24 robh: mw1147 rebooted and manually running scap pull
00:21 robh: mw1147 seems to have died during scap, unresponsive from serial console, powercycled
00:16 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.6/extensions/Kartographer: Search for maplinks inside and outside of content. (duration: 01m 08s)

2016-06-15

23:38 logmsgbot: mattflaschen@tin Synchronized php-1.28.0-wmf.6/extensions/Echo: Sync Echo fix for cross-wiki notifications: 62324e3 (duration: 00m 33s)
21:32 logmsgbot: aaron@tin Synchronized wmf-config/filebackend-production.php: Set "sync" filebackend replication to measure latency effect (duration: 00m 25s)
21:27 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes/libs/objectcache/WANObjectCache.php: faff8f1ef1bfefd1804a3f46e58566711faa3224 (duration: 00m 27s)
21:16 dapatrick: Deployed patch for T137264 to wmf.5 and wmf.6
20:17 logmsgbot: hashar@tin Synchronized wmf-config/throttle.php: Temporary IP Cap Lift on es.wiki T137917 (duration: 00m 30s)
20:09 subbu: finished deploying parsoid sha 3445eceb
20:05 bblack: cache frontend restarts complete
20:04 subbu: synced new code; restarted parsoid on wtp1001 as a canary
20:02 subbu: starting parsoid deploy
19:25 bblack: rolling restart of global varnish frontends (salt -b 1: depool -> sleep 15 -> restart -> repool) - estimated ~35 mins to completion - T107236 (...._
19:15 bblack: varnish frontend restart halted - v4 compat issue to address :P
19:11 bblack: rolling restart of global varnish frontends (salt -b 1: depool -> sleep 15 -> restart -> repool) - estimated ~30 mins to completion - T107236
19:05 logmsgbot: hashar@tin rebuilt wikiversions.php and synchronized wikiversions files: (no message)
18:54 ori: Started MySQL on es2019 (T130702)
16:32 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1023; pool db1085 (disabled), db1088, db1092 w/low weight (duration: 00m 25s)
16:07 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Fix autopatrolled group for ko.wikipedia (duration: 00m 31s)
16:00 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.6/resources/src/mediawiki.special/mediawiki.special.search.styles.css: SWAT: Explicitly specify the width of the search input on Special:Search (duration: 00m 25s)
15:53 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add autopatrolled group in kowiki (duration: 00m 24s)
15:33 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Deploy ORES beta feature in wikidatawiki (duration: 00m 24s)
15:23 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka1002.eqiad.wmnet
15:23 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/ORES: SWAT: Skip when an edit is errored in PopulateDatabase.php (duration: 00m 27s)
15:17 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Send authentication events to logstash (duration: 00m 28s)
15:15 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka1002.eqiad.wmnet
15:11 logmsgbot: thcipriani@tin Synchronized wmf-config/logging.php: SWAT: Fix logging config for authmanager metrics channel rename (duration: 00m 24s)
15:10 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet
15:06 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: SWAT: Remove old throttle rules (duration: 00m 30s)
15:00 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet
15:00 mobrovac: scb disabled puppet for stopped change-prop during kafka nodes upgrade
15:00 elukey: rebooting Eqiad Event Bus for kernel upgrades (one node at the time)
14:24 moritzm: installing php security updates on jessie systems
13:55 moritzm: remove unused PHP packages from the recently provisioned jessie app servers (new installation are fixed in puppet to only install php5-cli, but the initial set needs fixed up manually)
13:40 gehel: rolling back update of firejail on maps2001
13:16 _joe_: stopped jobchron, jobrunner on mw1299, masked in systemd
13:15 mobrovac: change-prop deployed 6ad337
13:06 moritzm: installing libav security updates
12:37 _joe_: rebooting mw1299
12:06 gehel: upgrade of firejail on maps server stopped, pending a patch to service::node
11:46 mobrovac: scb enabled puppet back
11:44 gehel: upgrading firejail to 0.9.38 on maps servers
11:32 mobrovac: scb disabled puppet for 5 min to keep change-prop down
11:30 mobrovac: change-prop deploying 353b926
11:29 jynus: stopping db1023 for cloning to new s6 hosts
11:22 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase new enwiki dbs weight, depool db1023 for cloning (duration: 00m 27s)
11:13 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1033, first pool of db1079, db1086, db1094 with low weight (duration: 00m 25s)
11:11 moritzm: enabed firejail wrapper for imagemagick's convert (for image scalers and the Score extension)
10:59 paravoid: rebooting install2001 again
10:48 logmsgbot: jmm@tin Synchronized wmf-config/CommonSettings.php: firejail security hardening for image scalers (duration: 00m 26s)
09:48 godog: bounce ms-be2003, xfs high load
09:13 moritzm: repooled mw1154 (kernel still the same ATM)
08:53 moritzm: depooling mw1154 (image scaler) for kernel update
08:29 jynus: turning down db1033 for cloning to new s7 slaves
08:15 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1033 for cloning (duration: 00m 38s)
06:59 moritzm: installing apache trusty updates on eqiad app servers
03:51 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes/parser/Parser.php: 4e6e1bc1f2de000f0fdd84dcf04f63a21127d24a (duration: 00m 30s)
03:49 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/parser/Parser.php: 23bac8905a9d60cdc0a068ca025644e091b9027f (duration: 00m 32s)
03:10 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jun 15 03:10:57 UTC 2016 (duration 6m 55s)
03:04 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.6) (duration: 16m 29s)
02:30 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 12m 24s)
02:29 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.5/extensions/Scribunto/engines/LuaCommon/TitleLibrary.php: revert: ad-hoc debug of vary-revision in scribunto (duration: 00m 29s)
02:22 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.5/extensions/Scribunto/engines/LuaCommon/TitleLibrary.php: ad-hoc debug of vary-revision in scribunto (duration: 00m 26s)
01:51 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.5/resources/src/mediawiki.action/mediawiki.action.edit.stash.js: Idfad8407: Improve client-side edit stash change detection (duration: 00m 24s)
01:31 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes/parser: 78de24a20c4662ea709e1f8af84bb5fae4aea2fa (duration: 00m 33s)
01:30 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/parser: 48652dfc27d1bbaab41b3a4d8f7d6be23e2da6b6 (duration: 00m 34s)

2016-06-14

23:40 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.6/resources/src/mediawiki.action/mediawiki.action.edit.stash.js: Idfad8407c8e: Improve client-side edit stash change detection (duration: 00m 25s)
23:30 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: Id800a9d35b: Set import sources for he.wikipedia (T137074) and If66f307a2e: Set import sources for pt.wikinews (T137633) (duration: 00m 27s)
23:28 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.6/extensions/AntiSpoof: I2e407a3ac8: Revert "Make sure AntiSpoof mappings are mapping in the correct direction." (duration: 00m 27s)
23:15 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.6/extensions/Echo: If07369cb1: Allow the primary link to set all bundled notifications as read (T136368) (duration: 00m 34s)
23:09 logmsgbot: ori@tin Synchronized wmf-config/abusefilter.php: I4e5e4d227: Set $wgAbuseFilterConditionLimit = 2000 for commonswiki (T132048) (duration: 00m 28s)
22:43 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes/deferred: 0d038de1414c0b4faed1cc9882151e68d86d3b2d (duration: 00m 25s)
22:15 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/deferred: 29863094805baed7a5fa493c99c87745ce041f49 (duration: 00m 27s)
21:50 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/resources: 7898fd2fa969342a5cc30df6a5757f4642cd6118 (duration: 00m 28s)
21:44 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes: 7898fd2fa969342a5cc30df6a5757f4642cd6118 (duration: 01m 12s)
21:33 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: name=maps-test2.*
21:28 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: name=maps-test2*
21:28 gehel: sending traffic back to old maps servers (T137620)
21:10 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: name=maps-test2*
21:09 logmsgbot: gehel@palladium conftool action : set/pooled=no; selector: maps-test2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=maps', 'service=kartotherian'])
21:09 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: maps2004.codfw.wmnet (tags: ['dc=codfw', 'cluster=maps', 'service=kartotherian'])
21:08 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: maps2003.codfw.wmnet (tags: ['dc=codfw', 'cluster=maps', 'service=kartotherian'])
21:08 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: maps2002.codfw.wmnet (tags: ['dc=codfw', 'cluster=maps', 'service=kartotherian'])
20:58 logmsgbot: gehel@palladium conftool action : set/pooled=yes; selector: maps2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=maps', 'service=kartotherian'])
20:55 gehel: pooling maps2001 (new map server) - T137620
20:50 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.6/includes: ca9068daffb49cc0cdfb84385a29aea34df155cd (duration: 01m 51s)
20:46 gehel: adding new maps servers to LVS
20:09 logmsgbot: demon@tin Finished scap: wikidata submodule update for wmf.6 (duration: 25m 51s)
19:43 logmsgbot: demon@tin Started scap: wikidata submodule update for wmf.6
19:30 logmsgbot: demon@tin Finished scap: group0 to 1.28.0-wmf.6 (duration: 26m 43s)
19:03 logmsgbot: demon@tin Started scap: group0 to 1.28.0-wmf.6
18:56 logmsgbot: demon@tin Purged l10n cache for 1.27.0-wmf.23
18:54 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.4
18:54 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.3
18:54 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.2
18:53 logmsgbot: demon@tin Purged l10n cache for 1.28.0-wmf.1
17:22 Dereckson: Run initSiteStats.php for arcwiki and htwiki (T137827)
16:48 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1040, pool for the first time db1081, db1084, db1091 (duration: 00m 34s)
16:41 godog: reimage ms-fe3002 with jessie T117972
15:57 yurik: deployed & restarted kartotherian (fixing spec.config tests)
15:54 urandom: Restarting cassandra-metrics-collector on restbase1007 : T137304
15:53 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Beta: Enable Compact Language Links for new users (duration: 00m 31s)
15:41 logmsgbot: hashar@tin scap aborted: testwiki to php-1.28.0-wmf.6 and rebuild l10n cache (duration: 01m 31s)
15:40 logmsgbot: hashar@tin Started scap: testwiki to php-1.28.0-wmf.6 and rebuild l10n cache
15:35 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/CentralAuth/includes/CentralAuthHooks.php: SWAT: Account for changed login process (duration: 00m 26s)
15:27 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add nonecho.dblist and echo.dblist PART III (duration: 00m 27s)
15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Add nonecho.dblist and echo.dblist PART II (duration: 00m 26s)
15:26 godog: reimage ms-fe3001 with jessie T117972
15:25 logmsgbot: thcipriani@tin Synchronized dblists: SWAT: Add nonecho.dblist and echo.dblist PART I (duration: 00m 28s)
15:19 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add nonecho.dblist and echo.dblist PART III (duration: 00m 28s)
15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Add nonecho.dblist and echo.dblist PART II (duration: 00m 30s)
15:18 logmsgbot: thcipriani@tin Synchronized dblists: SWAT: Add nonecho.dblist and echo.dblist PART I (duration: 00m 30s)
15:09 yurik: deployed & restarted kartotherian
15:07 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor by default on eleven Wikivoyages (duration: 01m 49s)
13:47 hashar: T136971 Cutting MediaWiki branches 1.28.0-wmf.6
13:40 moritzm: installing apache trusty updates on codfw app servers
13:28 paravoid: rebooting install2001, T137647
12:58 moritzm: installing apache trusty updates on canary app servers
12:55 mobrovac: change-prop deployed f34fb06c99
12:27 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1024, pool for the first time db1090 with low weight (duration: 00m 38s)
11:30 mobrovac: scb disabling puppet for 10 mins or so to keep change-prop down
11:15 akosiaris: T134242 rebooting alsafi.wikimedia.org hassaleh.codfw.wmnet kraz.wikimedia.org mx2001.wikimedia.org planet2001.codfw.wmnet pollux.wikimedia.org pybal-test2001.codfw.wmnet pybal-test2002.codfw.wmnet pybal-test2003.codfw.wmnet for qemu-kvm upgrade
11:13 akosiaris: T134242 install qemu-system-common, qemu-system-x86 1:2.5+dfsg-4~bpo8+1 from jessie-backports on ganeti200{1,2,3,4,5,6}
11:04 _joe_: pooling all the new codfw appservers that have been installed - mw2215-mw2240 (T135466)
10:56 _joe_: pooling the new jessie appservers, mw1263-71
10:52 logmsgbot: oblivian@palladium conftool action : set/weight=30; selector: cluster=appserver,dc=eqiad,name=mw12[67].*
09:27 godog: roll-restart swift proxy in codfw and eqiad
09:04 hashar: gallium: manually removing cron entry zuul_repack from user zuul. Causes cron spam due to zuul merger no more being on gallium T137418
08:59 jynus: stopping db1040 for cloning to new s4 hosts
08:28 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1040 for cloning (duration: 00m 32s)
08:23 _joe_: powercycling mw1154, unresponsive
07:19 jynus: powercycling mw1156, could not regain control after OOM
07:18 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1024, increase weight of db1082, db1087 and db1092 (duration: 10m 50s)
07:05 _joe_: rolling reboot of mw2233-40
06:47 _joe_: rebooting mw2228
06:43 _joe_: rebooting mw2228
06:29 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool db1052, db1080, db1083, db1089 (duration: 01m 31s)
02:39 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jun 14 02:39:50 UTC 2016 (duration 5m 59s)
02:33 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 12m 14s)

2016-06-13

23:50 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Add ORES to whitelisted beta features (T130211) (duration: 00m 23s)
23:42 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.5/extensions/ORES/includes/Hooks.php: Update links to beta features (duration: 00m 25s)
23:33 ejegg: updated payments from 44102c59ac897c9acab470bf83369d233f9b736f to 2fc573cbb94e833c4144aa9dad79de8ec374bb09
23:29 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Update cross-wiki upload configuration (Gerrit:293355) (duration: 00m 23s)
23:10 logmsgbot: dereckson@tin Synchronized portals: (no message) (duration: 00m 24s)
23:10 logmsgbot: dereckson@tin Synchronized portals/prod/wikipedia.org/assets: (no message) (duration: 00m 24s)
22:51 logmsgbot: demon@tin Synchronized wmf-config/CommonSettings.php: Update extension distributor settings (duration: 00m 24s)
22:42 yurik: switched to scap3 and deployed tilerator. Deployed kartotherian. Restarted.
22:41 dapatrick: Deployed patches for T129738 to wmf5
22:36 awight: update fundraising CRM revert from e684b7823e751558772a4de4ac23819bc601eb74 to bb9bf136dc0fa82d5d07ebeb33d696e54672b2d6
22:11 awight: Updating fundraising CRM from b7b46740d701942507dca0a98a75f3f87b6b31b1 to e684b7823e751558772a4de4ac23819bc601eb74
19:15 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/resources: ee2da9c2ae6fac93bf65d17b5ea48e5c47c87d47 (duration: 00m 35s)
18:20 bblack: upgrading nginx (etc) on deployment-prep caches
18:11 gehel: deploying latest GUI on WDQS,
17:58 urandom: Upgrade of restbase1007.eqiad.wmnet (https://people.wikimedia.org/~eevans/debian/cassandra_2.2.6-wmf1_all.deb) complete : T137474
17:55 urandom: Restarting restbase1007-c.eqiad.wmnet : T137474
17:52 urandom: Restarting restbase1007-b.eqiad.wmnet : T137474
17:47 awight: Whitelist Special:PaypalExpressGatewayResult
17:43 godog: enable proxy_http apache module on graphite1003 / graphite2002 and restart apache
17:38 urandom: Restarting restbase1007-a.eqiad.wmnet : T137474
17:37 urandom: Upgrading restbase1007.eqiad.wmnet w/ https://people.wikimedia.org/~eevans/debian/cassandra_2.2.6-wmf1_all.deb : T137474
17:35 awight: update paymentswiki from 63fbe39fbc4d671fd2705ce9e42762b7c49564c2 to 44102c59ac897c9acab470bf83369d233f9b736f
16:51 _joe_: powercycling mw1115
16:49 logmsgbot: thcipriani@tin Finished scap: Update l10n cache for ores (duration: 32m 04s)
16:17 logmsgbot: thcipriani@tin Started scap: Update l10n cache for ores
15:59 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Enable ORES on fawiki PART II (duration: 00m 24s)
15:58 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable ORES on fawiki PART I (duration: 00m 25s)
15:58 logmsgbot: thcipriani@tin Synchronized wmf-config/extension-list: SWAT: Add ORES to extension-list (duration: 00m 25s)
15:42 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Add images.nypl.org to $wgCopyUploadsDomains for commons (duration: 00m 24s)
15:37 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VE in NS_PROJECT in cswiki (duration: 00m 25s)
15:32 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/Echo: SWAT: Use localized weekdays on Special:Notifications (duration: 00m 32s)
15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable transwiki import for la.wiktionary (duration: 00m 26s)
15:22 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Permission changes in zhwiki (duration: 00m 26s)
15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor by default for logged-out users on four Wikipedias too (duration: 00m 24s)
15:10 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/MobileFrontend: Do Not strip srcset on API mobileview action PART II (duration: 00m 38s)
15:09 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/MobileFrontend/includes/MobileContext.php: Do Not strip srcset on API mobileview action PART I (duration: 00m 49s)
15:05 godog: reboot ms-be2012 to fix disk ordering T136395
14:51 godog: truncate syslog.1 on ms-be2012
14:26 bblack: upgrading cp* nginx (and other oustanding minor package updates)
14:23 bblack: uploaded nginx-1.11.1-1+wmf2 to carbon
13:55 dcausse: restarting logstash on logstash1001
11:59 mobrovac: change-prop deployed 54f98b7
11:31 _joe_: rolling reboot of the new appservers in codfw + scap pull
09:55 _joe_: powercycling mw1138, oom, console non-responsive
09:53 jynus: stopping db1052 and cloning it to db1080, db1083 and db1089
09:43 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1052 for cloning (duration: 00m 26s)
08:51 moritzm: removed /var/log/logstash/logstash.log.1 on logstash1001, depleted disk space on the root partition, fallout of T137400
08:43 jynus: powercycling mw1155.eqiad.wmnet , unresponsive on ssh, serial console
08:31 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Increase weight of db1082, db1087, db1092 (duration: 02m 36s)
08:25 logmsgbot: oblivian@palladium conftool action : set/weight=30; selector: name=mw1261.eqiad.wmnet
08:17 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=mw1261.eqiad.wmnet
08:01 logmsgbot: oblivian@palladium conftool action : set/pooled=no:weight=20; selector: name=mw1262.eqiad.wmnet
08:00 logmsgbot: oblivian@palladium conftool action : set/pooled=no:weight=20; selector: name=mw1261.eqiad.wmnet
08:00 logmsgbot: oblivian@palladium conftool action : set/pooled=no; selector: name=mw1261.eqiad.wmnet
06:32 logmsgbot: oblivian@palladium conftool action : set/pooled=yes; selector: name=mw126.*
02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 13m 02s)

2016-06-12

02:28 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 12m 24s)

2016-06-11

03:14 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.5/includes/parser/CacheTime.php: remove ad-hoc logging of updateCacheExpiry(0) traces (duration: 00m 23s)
03:11 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.5/includes/parser/CacheTime.php: ad-hoc logging of updateCacheExpiry(0) traces (duration: 00m 25s)
02:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jun 11 02:36:22 UTC 2016 (duration 6m 30s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 11m 42s)
01:01 mutante: rutherfordium ganeti lockup, gnt-instance console .. and it recovered

2016-06-10

23:37 awight: Update PayPal Express Checkout configuration: add API certificate path
21:58 logmsgbot: ori@tin Synchronized wmf-config/mobile.php: I3d8155d7e14: Remove old config hack that disabled $wgResponsiveImages on mobile (duration: 00m 24s)
19:38 mutante: cp1043/cp1044 - revoke puppet cert, salt key
19:30 logmsgbot: thcipriani@tin Synchronized wmf-config/throttle.php: Fix for ip lift cap for eswiki and Temporary IP Cap Lift for eswiki (duration: 00m 23s)
19:19 logmsgbot: ori@tin Synchronized multiversion: Id432e25c: MWMultiVersion: allow wiki to be specified via the environment (duration: 00m 56s)
17:12 elukey: Updated the puppet compiler with new hosts/facts
16:32 mutante: cp1043,cp1044 shutdown -h, confirmed not in pybal/confctl
16:27 mutante: cp1043/cp1044 - decom'ing, were already "Unused spare system" but running, scheduling downtime in icinga, shutting them down and removing from torrus config and puppet (T133614)
14:17 urandom: Testing patched Cassandra (dpkg -i ...; service cassandra-{a,b} restart) on restbase-test200[1-2] : T137474
14:06 urandom: Testing patched Cassandra (dpkg -i ...; service cassandra-a restart) on restbase-test2001 : T137474
13:59 urandom: Testing patched Cassandra (dpkg -i ...; service cassandra-a restart) on praseodymim : T137474
13:58 urandom: Testing patched Cassandra (dpkg -i ...; service cassandra-a restart) on cerium : T137474
13:15 urandom: Starting html dump(s) in RESTBase staging : T137474
13:13 urandom: Testing patched Cassandra (dpkg -i ...; service cassandra-a restart) on xenon : T137474
11:07 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specials/SpecialUserLogin.php: deploy gerrit:293704 to fix AuthManager metrics (duration: 00m 32s)
11:06 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specials/SpecialCreateAccount.php: deploy gerrit:293704 to fix AuthManager metrics (duration: 00m 52s)
10:56 mobrovac: scb100x enabled puppet back
10:05 mobrovac: scb100x disabling puppet and stopping change-prop to look at zookeeper znodes
09:22 elukey: restarted uwsgi-ores on scb200[12] as deployment follow up
08:27 elukey: restarted uwsgi-ores (after a deployment + puppet run) - service was down
08:01 Amir1: deploying 38df031 into scb100[12] for ores service. Expecting some down time
07:59 dcausse: refilling ttmserver index on all ttm enabled wikis
06:42 moritzm: bounced hhvm on mw1264 (backtrace in /tmp/hhvm.2197.bt)
06:28 papaul: mw2215-mw2238 -signing puppet certs, salk-key initial run
05:54 mutante: re-enabling puppet on carbon
04:48 moritzm: installing squid3 security updates on Ubuntu systems
03:17 logmsgbot: aaron@tin Synchronized wmf-config/CommonSettings.php: Lower $wgAPIMaxLagThreshold to 5 (duration: 00m 36s)
02:35 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jun 10 02:35:15 UTC 2016 (duration 6m 2s)
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 11m 32s)
01:44 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specialpage/LoginSignupSpecialPage.php: deploying gerrit:293668: fix AuthManager warning spam (duration: 00m 25s)
01:43 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specials/SpecialUserLogin.php: deploying gerrit:293667: fix AuthManager dashboard (duration: 00m 33s)
01:42 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specials/SpecialCreateAccount.php: deploying gerrit:293667: fix AuthManager dashboard (duration: 00m 25s)
00:56 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawiki --force
00:40 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikibooks --force
00:39 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikinews --force
00:36 mutante: git pull on strontium because i merged a non-change
00:31 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikiquote --force
00:27 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings-labs.php: ores.wikimedia.org instead of ores.wmflabs.org (duration: 00m 25s)
00:21 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata/extensions/ArticlePlaceholder/includes/SearchHookHandler.php: Update Wikidata - Fix uncaught exception in ArticlePlaceholder (3/3) (duration: 00m 25s)
00:20 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata/vendor/composer/installed.json: Update Wikidata - Fix uncaught exception in ArticlePlaceholder (2/3, no-op) (duration: 00m 25s)
00:19 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata/composer.lock: Update Wikidata - Fix uncaught exception in ArticlePlaceholder (1/3, no-op) (duration: 00m 27s)
00:12 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawikisource --force
00:07 kaldari: ran mwscript maintenance/updateCollation.php --wiki=tawiktionary --force

2016-06-09

23:57 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set Tamil projects to use uca-ta collation II (T75453) (duration: 00m 25s)
23:53 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Enable Flow beta feature on frwiki (T136684) (duration: 00m 27s)
23:47 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Remove HiddenPrefs hack for turning off cross-wiki notifications (T135266) (duration: 00m 27s)
23:31 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: enable AuthManager on group2 wikis T135504 (duration: 00m 24s)
23:29 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: enable use of group1, group2 dblists in config (duration: 00m 23s)
23:28 logmsgbot: tgr@tin Synchronized dblists/group2.dblist: add dblist for group2 (duration: 00m 22s)
23:20 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/includes/specialpage/LoginSignupSpecialPage.php: deploying gerrit:293636 for AuthManager T135504 (duration: 00m 25s)
23:19 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/extensions/MobileFrontend/resources/skins.minerva.special.userlogin.styles/userlogin.less: deploying gerrit:293638 for AuthManager T135504 (duration: 00m 25s)
23:18 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/extensions/ConfirmEdit/FancyCaptcha/resources/ext.confirmEdit.fancyCaptcha.js: deploying gerrit:293637 for AuthManager T135504 (duration: 00m 24s)
22:48 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.5
22:35 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/includes/site/DBSiteStore.php: Revert "Map dummy language codes in sites" Part II (duration: 00m 31s)
22:35 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/includes/ServiceWiring.php: Revert "Map dummy language codes in sites" Part I (duration: 00m 23s)
21:41 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes: 904dd4ae088a8f67942c09b2b28178377955d6a6 (duration: 01m 18s)
20:57 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on nnwiki (T130997) (duration: 00m 24s)
20:53 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on lvwiki (T136100) (duration: 00m 26s)
20:48 logmsgbot: hoo@tin Synchronized wmf-config/InitialiseSettings.php: Enable the ArticlePlaceholder on guwiki (T136517) (duration: 00m 24s)
20:40 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.4/extensions/Wikidata: Update ArticlePlaceholder (duration: 01m 54s)
20:36 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata: Update ArticlePlaceholder (without unrelated T136598 fixes this time) (duration: 01m 51s)
20:33 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/user/User.php: c3b1f80a701d61dc57ccac0c8b1dc7daf03fa925 (duration: 00m 29s)
19:59 urandom: Restarting Cassandra on xenon.eqiad.wmnet (removing patched test build; restoring state) : T137474
19:53 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata: revert, possible s5 master overload (duration: 01m 57s)
19:47 logmsgbot: hoo@tin Synchronized php-1.28.0-wmf.5/extensions/Wikidata: Update ArticlePlaceholder (duration: 02m 04s)
19:44 bearND: mobileapps deployed 71ff97c
19:42 bearND: starting mobileapps deploy
19:11 ejegg: updated cancel page settings on payments-wiki
17:43 urandom: Restarting Cassandra on xenon.eqiad.wmnet (use exponentially decaying resevoirs for metrics histograms) : T126629
17:19 mobrovac: change-prop deploying ecfda93f09d
17:10 ejegg: updated payments-wiki from 3dcf58e3b4e1d02ad4f1874a3e87e55b7e169bfe to 053aaa259382c94aa59e4d0da7317fcafab635cd
15:31 elukey: added topic override retention.bytes=536870912000 to Kafka webrequest_text (T136690)
15:22 hashar: Cleaning git-daemon on gallium (was used by zuul-merger) T137418
15:19 logmsgbot: aude@tin Synchronized wmf-config/InitialiseSettings.php: Add *.nara.gov to wgCopyUploadDomains (duration: 00m 40s)
14:47 mobrovac: change-prop stopped on scb1002
14:38 elukey: Tested temp setting retention.bytes=2G for Analytics kafka topic webrequest_misc
14:37 hashar: Removing zuul-merger from gallium
14:33 hashar: stopped / disabled zuul-merger on gallium T137418
14:12 mobrovac: change-prop restarting on scb1001 for update
14:07 urandom: Re-enabling puppet on xenon.eqiad.wmnet, forcing a run, and restarting Cassandra : T137419
13:52 mobrovac: change-prop restarting on scb1002 for update
13:45 mobrovac: change-prop deploying 2161403c
13:26 urandom: Restarting Cassandra on xenon.eqiad.wmnet to apply 2G file cache : T137419
12:51 urandom: Restarting Cassandra on xenon.eqiad.wmnet : T126629
12:33 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/extensions/LdapAuthentication/LdapPrimaryAuthenticationProvider.php: deploy gerrit:293459 to fix wikitech API login / morebots (T137377) (duration: 00m 47s)
12:19 tgr: !log deploying gerrit:293459 to fix morebots (T137377)
12:11 urandom: !log Restarting Cassandra on xenon.eqiad.wmnet : T126629
12:06 urandom: !log Temporarily disabling puppet on xenon.eqiad.wmnet to test settings : T126629
11:33 Amir1: !log manually restarting ores-uwsgi and celery-ores-worker in scb100[12]
10:51 urandom: !log Restarting Cassandra on {cerium,praseodymium}.eqiad.wmnet (RESTBase staging) : T126629
09:16 gehel: !log lowering disk high watermark to rebalance disk usage on elasticsearch eqiad cluster
09:05 Amir1: !log restarting uwsgi-ores celery-ores-worker in scb1001 and scb1002
08:55 moritzm: !log installing libtasn security updates
08:38 moritzm: !log rolling restart of app server canaries for libtasn security update
07:22 moritzm: !log removed /var/log/logstash/logstash.1 on logstash1001, logspam (similar to the what is described in https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/144) depleted the space on the root partition
02:55 logmsgbot: !log l10nupdate@tin ResourceLoader cache refresh completed at Thu Jun 9 02:55:20 UTC 2016 (duration 6m 19s)
02:53 mutante: !log ms-be2012 ran out of disk due to huge syslog, deleted log, restarted rsyslogd
02:49 logmsgbot: !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 11m 02s)
02:26 logmsgbot: !log mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 11m 00s)
00:03 twentyafterfour: !log Preparing to deploy phabricator update. Tagged release/2016-06-08/1

2016-06-08

23:17 logmsgbot: !log maxsem@tin Synchronized php-1.28.0-wmf.5/extensions/WikimediaEvents/: https://gerrit.wikimedia.org/r/#/c/293439/ (duration: 00m 23s)
23:15 logmsgbot: !log maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/293438 (duration: 00m 25s)
23:07 logmsgbot: !log maxsem@tin Synchronized php-1.28.0-wmf.4/extensions/LiquidThreads/: https://gerrit.wikimedia.org/r/#/c/293247/ (duration: 00m 26s)
23:05 logmsgbot: !log maxsem@tin Synchronized php-1.28.0-wmf.5/extensions/LiquidThreads/: https://gerrit.wikimedia.org/r/#/c/293247/ (duration: 00m 26s)
22:51 hoo: !log Re-started dumpwikidatattl on snapshot1003
22:44 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: enable AuthManager on group1 for reals T135504 (duration: 00m 25s)
22:27 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: enable AuthManager on group1 T135504 (duration: 00m 23s)
22:21 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.5/extensions/OpenStackManager/: backport gerrit:293130 for AuthManager deploy T135504 (duration: 00m 28s)
22:05 ottomata: starting kafka broker on kafka1012 after swapping disk and copying data directory
22:01 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: Bump wgResourceLoaderStorageVersion (T134368) (duration: 00m 28s)
21:12 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.5
21:04 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/includes/specials/SpecialSearch.php: Add a visual clear to Special:Search input box and profile-tabs (duration: 00m 23s)
20:57 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/Renameuser/RenameuserSQL.php: Use master DB when touching the user to signal rename end (duration: 00m 22s)
20:50 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/includes/libs/objectcache/WANObjectCache.php: Avoid getWithSetCallback() warnings on unversioned key migration (duration: 00m 24s)
20:21 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.5/extensions/Kartographer/styles/kartographer.less: Fixed <maplink> autostyling (duration: 00m 26s)
20:18 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.4/extensions/Kartographer: late SWAT: Fix color extraction (duration: 00m 36s)
19:30 mobrovac: change-prop deploying 08a1b1d
19:27 hashar: gallium enabling puppet again now that zuul/jenkins are back
19:18 hashar: Bringing back Jenkins and Zuul on gallium T137265
18:59 logmsgbot: ori@palladium conftool action : set/pooled=yes; selector: name=scb1002.eqiad.wmnet
18:57 yurik: switched kartotherian to scap3, deployed, restarted
18:20 gehel: switching maps to scap3 deployment
16:50 jynus: cloning /var/lib/jenkins from db1085 to contint1001
16:46 ottomata: stopping kafka broker and puppet on kafka1012 to replace sdf
16:37 ottomata: powercycling scb1002
16:36 hashar: Disabled puppet on contint1001 to prevent it from bringing back Jenkins
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=mathoid'])
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=mobileapps'])
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=cxserver'])
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=citoid'])
16:32 logmsgbot: otto@palladium conftool action : set/pooled=no; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=graphoid'])
16:24 ottomata: restarting hadoop-yarn-resourcemanager on analytics1002 to make analytics1001 active
16:07 mobrovac: scb1002 enabling back puppet
16:02 elukey: temporary set a 10TB upperbound to the Kafka webrequest_text topic to free space (T136690)
15:43 ottomata: restarting zk in codfw and eqiad 1 by 1 to apply maxClientCnxns=1024
15:12 ottomata: restarting zookeeper 1 by 1 in eqiad
15:03 _joe_: contint1001: systemctl mask zuul,zuul-merger
14:57 elukey: rolling out the new Varnishkafka version in cache misc (didn't do it before since there was an outage ongoing)
14:53 jynus: rebooting gallium with netboot for hardware maintenance
14:44 mobrovac: scb1001 enabling and running puppet on scb1001
13:44 jynus: running fsck.ext3 /dev/sda2 in read-write mode for gallium
13:42 ottomata: powercycling scb2001 and scb2002
13:30 akosiaris: disabling puppet on scb1001 & scb1002
13:30 mobrovac: change-prop stopped on scb1002
13:29 akosiaris: stopping changeprop on scb1001
13:26 ottomata: powercycling scb1002
13:18 ottomata: powercycling scb1001
13:08 elukey: rolling out new varnishkafka package in cache misc
12:09 jynus: mounted temporarily / partition from gallium sda on db1085:/mnt
10:40 moritzm: uploaded jenkins 1.651.2 for jessie-wikimedia to carbon
10:13 elukey: rolling out the new varnishkafka package to cache maps
10:04 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/deferred/LinksDeletionUpdate.php: fd44d649787ede78687b4cd2ef21e44a4c8b843b (duration: 00m 33s)
08:28 hashar: stopping Jenkins / zuul / zuul-merger / puppet on gallium
08:15 elukey: lowering down webrequest_text kafka topic retention time from 7 days to 4 days to free disk space (T136690)
08:14 hashar: Jenkins has bunch of executors dead for what ever reason preventing jobs from running :(
07:53 mobrovac: change-prop deploying 84d56e53a
06:59 moritzm: enabling ferm on palladium (will lead to temporary puppet failures)
02:58 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jun 8 02:58:28 UTC 2016 (duration 6m 31s)
02:51 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.5) (duration: 06m 49s)
02:51 legoktm: / on gallium is currently read-only for some reason
02:29 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 11m 11s)
00:11 awight_: update fundraising-tools from b2425aef2154d6b689900f4848cca02880321230 to 28bc2da677caa795c58f906db76a1f8d612ac899

2016-06-07

23:46 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.5/includes/deferred/LinksUpdate.php: 6d85caaa9bb5918cb2888fc82f2c7c346cf746a2 (duration: 00m 25s)
23:36 SMalyshev: redeploying WDQS to update the Updater for T128947 fix
23:35 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings.php: SWAT gerrit:292518 User rights configuration for meta. wmf-supportsafety group (duration: 00m 26s)
23:20 logmsgbot: tgr@tin Finished scap: (no message) (duration: 24m 51s)
23:02 awight: update paymentswiki from 28e10141454ef53085aed4c6619a34d3a4b43c58 to de11bfe2273d0bcaa0e713389b2d91e8b3567a1d; add PP cert
22:56 tgr: scapping AuthManager backports + feature switch enabled on group0 T135504
22:56 logmsgbot: tgr@tin Started scap: (no message)
22:10 mutante: icinga config broken: Error: Could not find any host matching 'relforge1001'
21:35 twentyafterfour: restarted apache on iridium to deploy D250
20:02 andrewbogott: dist-upgrade on labvirt1010, in hopes of resolving a nova-compute lockup (possibly related to a kvm upgrade earlier today)
20:00 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: group0 to 1.28.0-wmf.5
19:44 jynus: restarting es2017 due to a bunch of ACPI errors (probably memory-caused)
19:35 logmsgbot: thcipriani@tin Finished scap: testwiki to php-1.28.0-wmf.5 and rebuild l10n cache (duration: 26m 40s)
19:08 logmsgbot: thcipriani@tin Started scap: testwiki to php-1.28.0-wmf.5 and rebuild l10n cache
18:30 andrewbogott: rebooting labvirt1011
17:51 ottomata: restarting broker on kafka1020
17:44 Dereckson: `mwscript initSiteStats.php --wiki kshwiki --update` on Terbium (T137234)
17:33 mutante: furud - shutdown, decom, deleteV VM
17:30 ejegg: updated payments-wiki from 3df3329f75fdbc679baf37bfd3955880091b3ae1 to 28e10141454ef53085aed4c6619a34d3a4b43c58
17:06 logmsgbot: krinkle@tin Synchronized wmf-config/CommonSettings.php: clean-up
17:05 ejegg: rolled back payments-wiki to 3df3329f75fdbc679baf37bfd3955880091b3ae1
17:04 thcipriani: starting branch-cut for mediawiki and extensions for version 1.28.0-wmf.5
17:04 ejegg: updated payments-wiki from 3df3329f75fdbc679baf37bfd3955880091b3ae1 to 413bd3ea92ac570c081532c71891c31391194984
16:01 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Update audit hooks for AuthManager (duration: 00m 24s)
15:53 logmsgbot: thcipriani@tin Synchronized wmf-config/wikitech.php: SWAT: Do not set wgAuth to LdapAuth when AuthManager is enabled (duration: 00m 23s)
15:48 logmsgbot: thcipriani@tin Synchronized portals: SWAT: T135902 adding readme and license to wikipedia.org portal (duration: 00m 25s)
15:48 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: SWAT: T135902 adding readme and license to wikipedia.org portal (duration: 00m 25s)
15:41 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: huwiki: Enable Popups A/B test for 50% of users (duration: 00m 24s)
15:32 logmsgbot: thcipriani@tin Synchronized wmf-config: SWAT: Revert "Send wmf.4 search and ttmserver traffic to codfw" (duration: 00m 26s)
15:24 logmsgbot: thcipriani@tin Synchronized wmf-config/PrivateSettings.php: SWAT: Use bot password for TNBot after touch wmf-config/PrivateSettings.php (duration: 00m 25s)
15:16 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Use bot password for TNBot (duration: 00m 34s)
15:15 logmsgbot: thcipriani@tin Synchronized private/PrivateSettings.php: SWAT: password update for Translation Notification Bot (duration: 00m 41s)
14:47 elukey: installing varnishkafka 1.0.10-1 on cp1046 manually to test the new version.
14:23 jynus: stopping mysql and the OS @ es2017 for hardware maintenance
13:53 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Pool new s5 db hosts: db1082, db1087, db1092 with low weight (duration: 00m 23s)
13:52 logmsgbot: jynus@tin Synchronized wmf-config/db-codfw.php: Add new coredb servers to alias configuration (duration: 00m 38s)
13:49 jynus: about to pool new dewiki/wikidata servers T133398
12:27 moritzm: rolling out gdk-pixbuf security updates
12:23 moritzm: rolling restart of sca cluster for libxml2 security update
11:27 moritzm: restarting apache2 on californium (hosting horizon dashboard) for libxml2 update
11:23 moritzm: restarting apache2 on silver (hosting wikitech) for libxml2 update
11:08 hashar: restarted apache2 on gallium for libxml2 update
10:53 moritzm: restarting apache2 on iridium (hosting Phabricator) for libxml2 update
10:18 moritzm: rolling restart of hhvm on eqiad appservers to pick up libxml2 update
09:09 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Repool db1070 after maintenance (duration: 00m 27s)
09:04 hashar: Upgrading Jenkins IRC plugin 2.25..2.27 and instant messaging plugin 1.34..1.35 . The former should fix a deadlock on shutdowning Jenkins
09:00 moritzm: rolling restart of hhvm on codfw appservers to pick up libxml2 update
08:53 moritzm: rolling restart of hhvm on appserver canaries to pick up libxml2 update
08:28 moritzm: deploying libxml2 security updates on Ubuntu systems (Debian systems already upgraded last week)
07:19 jynus: stopping and cloning db1070 to new s5 servers
07:08 logmsgbot: jynus@tin Synchronized wmf-config/db-eqiad.php: Depool db1070 for cloning (duration: 00m 29s)
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Tue Jun 7 02:30:57 UTC 2016 (duration 5m 32s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 09m 36s)
01:10 logmsgbot: aude@tin Synchronized php-1.28.0-wmf.4/extensions/Wikidata: Fix bug (T136093) in display of labels after edit (duration: 02m 03s)
00:39 Krenair: (TXT record for SPF, actually)
00:39 Krenair: Created MX and SPF records directly for wmflabs.org. for https://phabricator.wikimedia.org/T137160#2359786
00:35 ejegg: updated settings on payments-wiki
00:26 logmsgbot: ori@tin Synchronized php-1.28.0-wmf.4/extensions/CentralAuth/includes/CentralAuthHooks.php: I79cbb1dc: Prefetch $wgCentralAuthLoginWiki DNS (T92864) (duration: 00m 29s)

2016-06-06

23:41 logmsgbot: maxsem@tin Synchronized wmf-config/abusefilter.php: https://gerrit.wikimedia.org/r/#/c/292758/ (duration: 00m 24s)
23:32 logmsgbot: maxsem@tin Synchronized php-1.28.0-wmf.4/extensions/GeoData/: (no message) (duration: 00m 25s)
23:29 logmsgbot: maxsem@tin Synchronized private/PrivateSettings.php: Updated Zero password (duration: 00m 25s)
23:21 Amir1: deploying ae71d84 into ores in prod
23:17 logmsgbot: maxsem@tin Synchronized wmf-config/InitialiseSettings.php: https://gerrit.wikimedia.org/r/#/c/293037/ (duration: 00m 24s)
23:14 logmsgbot: maxsem@tin Synchronized portals: https://gerrit.wikimedia.org/r/#/c/292992/ (duration: 00m 31s)
23:13 logmsgbot: maxsem@tin Synchronized portals/prod/wikipedia.org/assets: https://gerrit.wikimedia.org/r/#/c/292992/ (duration: 00m 30s)
23:04 logmsgbot: maxsem@tin Synchronized docroot/wikipedia.org/.well-known/apple-app-site-association: https://gerrit.wikimedia.org/r/#q,287190,n,z (duration: 00m 25s)
22:05 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.4/includes/api/ApiStashEdit.php: 50ce579046e07 (duration: 00m 23s)
20:25 arlolra: updated Parsoid to version e8d6092e
20:09 arlolra: starting Parsoid deploy
19:15 ottomata: restarting kafka broker on kafka1020 to test python consumption client
19:12 bblack: restarted nginx on rcs1002 (was stuck half-shut-down for reload?), started nginx on rcs1001 (wasn't running at all)
19:08 mutante: ran puppet on carbon because icinga said fail, saw it change STS headers, but no fail
19:06 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.4/includes/page/WikiPage.php: 661c22db3a352 (duration: 00m 30s)
18:08 ori: Running rebuildrecentchanges.php for test2wiki for T133225
17:14 gehel: deploying latest GUI for wikidata query service
16:58 logmsgbot: tgr@tin Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 23s)
16:57 logmsgbot: tgr@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 23s)
16:44 logmsgbot: tgr@tin Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 23s)
16:39 tgr: PrivateSettings changes were for T135074
16:39 logmsgbot: tgr@tin Synchronized wmf-config/PrivateSettings.php: (no message) (duration: 00m 27s)
16:37 logmsgbot: tgr@tin Synchronized private/PrivateSettings.php: (no message) (duration: 00m 26s)
16:23 _joe_: rebooting mw1262
16:22 logmsgbot: tgr@tin Synchronized wmf-config/CommonSettings.php: creating zeroscript grant group on zerowiki, gerrit: 292951 (duration: 00m 28s)
16:00 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Math: Set wgMathFullRestbaseURL to point to wikimedia.org in production (duration: 00m 24s)
15:45 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: ULS: Stop using /static/current (duration: 00m 24s)
15:37 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.4/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.MobileArticleTarget.js: SWAT: Fix config of mobile surfaces (duration: 00m 24s)
15:32 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Use wfLoadExtension for LocalisationUpdate (duration: 00m 27s)
15:21 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT Switch Wikivoyages to Single Edit Tab mode for VE Beta Feature (duration: 00m 24s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable VisualEditor by default for logged-in users on four Wikipedias PART II (duration: 00m 30s)
15:14 logmsgbot: thcipriani@tin Synchronized dblists/visualeditor-default.dblist: SWAT: Enable VisualEditor by default for logged-in users on four Wikipedias PART I (duration: 00m 29s)
15:04 jynus: dropping old outreach databases on m1
14:10 jynus: dropping old bugzilla databases from m1
14:00 jynus: dropping database blog from m1
12:34 hashar: restarted Jenkins, deadlock in IRC plugin
10:46 elukey: re-added kafka1001 to eventbus.svc.eqiad.wmflabs without rebooting since some concerns were raised from the Services team. Will have a discussion with them before proceeding.
10:45 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka1001.eqiad.wmnet
10:33 moritzm: installing perl updates (bugfixes and CVE-2015-8853)
10:27 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka1001.eqiad.wmnet
10:25 elukey: rebooting kafka100[12] for kernel upgrades (one at the time with de-pool/re-pool actions)
09:12 moritzm: installing dpkg bugfix updates on jessie systems
08:45 mobrovac: change-prop deployed 9b04e475
08:27 gehel: lowering elasticsearch high watermark on eqiad cluster to rebalance disk space
08:17 _joe_: rebooting mw1262
07:57 jynus: enabling GTID on pending coredb servers on eqiad
06:18 logmsgbot: aaron@tin Synchronized php-1.28.0-wmf.4/includes/cache/LinkBatch.php: c2ba764f38e44e7 (duration: 00m 30s)
05:34 robh: db2034 locked up via serial console. details on T137084, rebooting since its unresponsive to ssh or serial.
02:28 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Mon Jun 6 02:28:50 UTC 2016 (duration 5m 56s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 09m 34s)

2016-06-05

14:55 Dereckson: `mwscript initSiteStats.php --wiki csbwiki --update` (T137060)
02:27 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sun Jun 5 02:27:38 UTC 2016 (duration 5m 35s)
02:22 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 08m 55s)

2016-06-04

20:18 apergos: rebooting mw1135, unresponsive to ssh or console login
09:51 elukey: restarted hhvm on mw1144 after the host was hanging (OOM killer restored basic host functionalities but not hhvm)
09:47 elukey: removed temporary Analytics Kafka upload retention override
09:38 elukey: Lowering down temporarily the Analytics kafka upload retention time to 24h to free space (T136690)
02:30 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Sat Jun 4 02:30:50 UTC 2016 (duration 5m 39s)
02:25 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 09m 08s)

2016-06-03

22:57 Krinkle: Purged https://en.wikipedia.org/static/images/project-logos/bawiki.png
22:53 logmsgbot: krinkle@tin Synchronized static/images/project-logos/bawiki.png: (no message) (duration: 00m 24s)
21:57 YuviPanda: started copying graphite data from usb back
21:27 awight: update paymentswiki from 28b98ec254b2a15c8df61c568b62f221b328222f to 3df3329f75fdbc679baf37bfd3955880091b3ae1
20:47 ejegg: updated payments-wiki de86eadcd98922ee4207a0c46112585f3ba5c48d to 28b98ec254b2a15c8df61c568b62f221b328222f
20:25 ejegg: updated GatewayReady hook on paymentswiki
19:37 logmsgbot: krinkle@tin Synchronized php-1.28.0-wmf.4/extensions/WikimediaEvents/extension.json: T136920 (duration: 00m 28s)
19:04 mutante: releases apt repo on bromine: export fresh jessie-mediawiki indexes
17:41 mutante: uploaded parsoid 0.5.1 to releases
17:14 robh: bast4001 coming down for second hdd installation. (there are currently no active users on system)
16:58 mutante: magnesium - shutdown -h now, bye
15:30 logmsgbot: tgr@tin Finished scap: revert AbuseFilter + config to pre-extension-registration state T136929 (duration: 06m 13s)
15:24 logmsgbot: tgr@tin Started scap: revert AbuseFilter + config to pre-extension-registration state T136929
14:38 gehel: un-freezing writes from CirrusSearch to eqiad cluster during upgrade (T133126)
13:27 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka2001.codfw.wmnet
13:22 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka2001.codfw.wmnet
13:22 logmsgbot: elukey@palladium conftool action : set/pooled=yes; selector: kafka2002.codfw.wmnet
13:16 hasharAway: Reenabling puppet on gallium. Forgot to put it back yesterday
13:14 logmsgbot: elukey@palladium conftool action : set/pooled=no; selector: kafka2002.codfw.wmnet
13:11 elukey: rebooting kafka200[12] (codfw EventBus) for kernel upgrades
11:18 gehel: freezing writes from CirrusSearch to eqiad clsuter during upgrade (T133126)
10:48 gehel: taking elasticsearch eqiad cluster down for upgrade to 2.3 (T133126)
10:39 gehel: Starting upgrade of elasticsearch eqiad cluster to 2.3 (T133126)
10:35 moritzm: restarting apache on bohrium (serving piwik.wikimedia.org) for libxml2 security update
10:23 moritzm: restarting apache on planet1001 (serving planet.wikimedia.org) for libxml2 security update
08:42 moritzm: rolling restart of scb cluster (mathoid, ores-uwsgi) in eqiad to pick up libxml2 security updates
08:38 jynus: archiving again syslog.1 from ms-be2012 on /srv/swift-storage/sdl1/tmp
08:35 jynus: created new LDAP group grafana-admin, gid=1007
08:34 elukey: rebooting kafka1012 for kernel upgrades.
08:08 moritzm: installing libxml2 security updates on jessie systems
07:19 kart_: Update cxserver to 19a71f1
06:29 moritzm: installing nginx security updates on Ubuntu systems (Debian installs updated some days ago)
02:36 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Fri Jun 3 02:36:39 UTC 2016 (duration 5m 58s)
02:30 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 08m 35s)
01:09 mutante: bromine - puppet currently stopped needs some permission fixes for release upload
01:08 mutante: uploaded parsoid 0.5.0 deb to releases.wm.org
00:24 logmsgbot: awight@tin Finished scap: Deploying labtestwiki AuthManager config; Enabling Popups experiment; CentralNotice fixes for T136408, T136387; Special:Notifications fixes (duration: 25m 08s)

2016-06-02

23:59 logmsgbot: awight@tin Started scap: Deploying labtestwiki AuthManager config; Enabling Popups experiment; CentralNotice fixes for T136408, T136387; Special:Notifications fixes
23:32 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings.php: Add namespace translation 'Portal' for diq (duration: 00m 24s)
23:28 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings.php: Enable AuthManager on beta wikitech (duration: 00m 25s)
23:24 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings.php: Enable Hovercards experiment for 1% of users on huwiki (duration: 00m 24s)
23:23 logmsgbot: awight@tin Synchronized php-1.28.0-wmf.4/extensions/Popups: Do not show Hovercards when NavPopups gadget is enabled on huwiki (duration: 00m 24s)
23:21 logmsgbot: awight@tin Synchronized wmf-config/extension-list-labs: Test PageAssessments on Beta Labs (duration: 00m 25s)
23:20 logmsgbot: awight@tin Synchronized wmf-config/InitialiseSettings-labs.php: Test PageAssessments on Beta Labs (duration: 00m 26s)
23:20 logmsgbot: awight@tin Synchronized wmf-config/CommonSettings-labs.php: Test PageAssessments on Beta Labs (duration: 00m 24s)
22:37 logmsgbot: ori@tin Synchronized wmf-config/InitialiseSettings.php: I9dc532b3: Enable "purge" log group (duration: 00m 42s)
22:20 mutante: removed my gerrit admin flag
20:20 mutante: magnesium (formerly RT) remove from puppet and icinga, revoked cert and salt key, just waiting another day or before shutdown
20:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: logging: disable Wikibase\Client\Changes\WikiPageUpdater channel (duration: 00m 26s)
20:12 logmsgbot: thcipriani@tin rebuilt wikiversions.php and synchronized wikiversions files: all wikis to 1.28.0-wmf.4
19:53 ottomata: stopping kafka broker and restarting kafka1014
19:52 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.4/extensions/CheckUser/specials/SpecialCheckUser.php: Fix Special:Checkuser for log entries when cuc_title = "" (duration: 00m 31s)
19:37 ejegg: re-enabled adyen job runner
19:35 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: scb2002.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=ores'])
19:35 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: scb2001.codfw.wmnet (tags: ['dc=codfw', 'cluster=scb', 'service=ores'])
19:35 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: scb1002.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
19:35 logmsgbot: akosiaris@palladium conftool action : set/pooled=yes; selector: scb1001.eqiad.wmnet (tags: ['dc=eqiad', 'cluster=scb', 'service=ores'])
18:57 ejegg: disabled adyen job runner
18:47 jynus: restarting replication on db1016
18:43 YuviPanda: powercycle labmon1001 again, get into bios
18:29 YuviPanda: going to try to intentionally trip the NFS check on tools-checker. This will not page
18:24 YuviPanda: powercycle labmon1001 again
18:19 mutante: db2007, revoke puppet cert, delete salt key, nuke from stored configs / icinga
18:19 bearND: mobileapps deployed b2fee30
18:18 mutante: db2007 shutdown, schedule eternal downtime
18:04 bearND: starting mobileapps deploy
17:40 subbu: finished deploying parsoid version 7188080b
17:34 subbu: synced new code; restarted parsoid on wtp1001 as a canary
17:29 subbu: starting deploy of new parsoid code
17:21 mutante: ran ALTER TABLE character set utf8 .. (https://phabricator.wikimedia.org/T119112#2311402) on RT db
17:16 mutante: running RT database upgrade from 4.0.4 to 4.2.8
17:13 awight: update paymentswiki from d26426c4225080c95f0bd5a6a31c54e4826287b1 to de86eadcd98922ee4207a0c46112585f3ba5c48d
17:05 mutante: stopped exim on magnesium
17:05 jynus: stopping replication from db1001 to db1016 (pasive m1 node) before schema change
16:52 mutante: magnesium (RT), tmp. stopped RT and puppet
16:50 YuviPanda: begin reinstall of labmon1001
15:19 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.4/extensions/Math: SWAT: Use img instead of meta tags for SVGs and Fix iterator in batchGetMathML (duration: 00m 28s)
15:12 logmsgbot: thcipriani@tin Synchronized portals: deploying new localized top-links on wikipedia.org (duration: 00m 31s)
15:11 logmsgbot: thcipriani@tin Synchronized portals/prod/wikipedia.org/assets: deploying new localized top-links on wikipedia.org (duration: 00m 32s)
14:33 jynus: acked ores icinga checks on some scb hosts and pointing to T124201 (it seems the checks arrived before the actual setup)
13:52 moritzm: installing imagemagick security updates on Ubuntu systems (but affected decoders already neutralised by policy changes) (also Debian systems already addressed)
13:34 hashar: Downgrading Zuul back to zuul_2.1.0-95-g66c8e52-wmf1precise1_amd64.deb . Paramiko cant acquire ssh connection with Gerrit for some reason... https://phabricator.wikimedia.org/P3204
12:10 hashar: Upgraded Zuul upstream code being 66c8e52..30a433b package is 2.1.0-151-g30a433b-wmf1precise1
11:39 logmsgbot: jmm@tin Synchronized wmf-config/CommonSettings.php: disable firejail security hardening for image scalers, needs more work for the Score extension (duration: 00m 36s)
10:55 hashar: Restarted Zuul and reenabled puppet on gallium
10:50 hashar: gallium: stopped puppet agent
10:49 hashar: gracefully stopping Zuul, will upgrade / take traces etc over the next half hour or so
10:14 jynus: archiving again syslog.1 from ms-be2012 on /srv/swift-storage/sdl1/tmp
10:08 mobrovac: restbase enabling puppet back in production
08:40 mobrovac: restbase deploy end of 19f25925
08:29 mobrovac: restbase deploy start of 19f25925
08:09 mobrovac: restbase disabling puppet in production for testing https://gerrit.wikimedia.org/r/#/c/292109/ in staging
07:23 moritzm: rebooting etherpad1001 (hosting etherpad.wikimedia.org) for upgrade to Linux 4.4
07:02 jynus: performing schema change for db1057
03:04 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Thu Jun 2 03:04:44 UTC 2016 (duration 6m 40s)
02:58 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 15m 37s)
02:24 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 10m 06s)
01:52 mutante: scb1001/2001 ores - connection refused
01:52 mutante: mw1136 service hhvm restart
01:37 mutante: labsdb1001 /etc/init.d/mysql start
01:32 YuviPanda: service mysql start on labsdb1001
01:25 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Set $wgSpamBlacklistEventLogging to true on testwiki (duration: 00m 22s)
01:25 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings.php: Set $wgSpamBlacklistEventLogging to true on testwiki (duration: 00m 23s)
01:23 YuviPanda: reboot labsdb1001
01:21 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.4/extensions/Flow/handlebars/: HACK: Hide reply form for locked topics (T135848) (duration: 00m 24s)
01:19 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.4/extensions/Echo/includes/special/NotificationPager.php: Fix notification pager (T136759) (duration: 00m 25s)
01:18 YuviPanda: restart mysql on labsdb1001
01:00 bearND: mobileapps reverted to 8d6d648c943074b7d3999baf31d60ad99249cd51
00:55 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Revert "Test PageAssessments extension on Labs" (no-op) (duration: 00m 22s)
00:55 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings-labs.php: Revert "Test PageAssessments extension on Labs" (no-op) (duration: 00m 23s)
00:26 logmsgbot: awight@tin Synchronized php-1.28.0-wmf.4/extensions/CentralNotice: Fix for T136387 (duration: 00m 38s)
00:05 urandom: Deploy of cdff5e3 to RESTBase production complete
00:03 YuviPanda: started nfs-exports on labstore1001

2016-06-01

23:57 urandom: Deploying cdff5e3 to RESTBase production
23:51 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Revert Use extension registration for SpamBlacklist (T119117) (duration: 00m 24s)
23:49 urandom: Deploying cdff5e3 to restbase1008.eqiad.wmnet (canary node)
23:44 urandom: Deploy of RESTBase to staging environment complete
23:40 urandom: Deploying RESTBase to staging environment
23:39 urandom: RESTBase deploy to xenon.eqiad.wmnet (canary node) complete
23:38 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings-labs.php: Test PageAssessments extension on Labs (no-op) (duration: 00m 26s)
23:37 logmsgbot: dereckson@tin Synchronized wmf-config/InitialiseSettings-labs.php: Test PageAssessments extension on Labs (no-op) (duration: 00m 30s)
23:36 urandom: Deploying RESTBase to xenon.eqiad.wmnet (canary node)
23:26 logmsgbot: dereckson@tin Synchronized php-1.28.0-wmf.4/extensions/VisualEditor/modules/ve-mw/init/targets/ve.init.mw.DesktopArticleTarget.js: Simplify teardown of toolbar save button (T136421) (duration: 00m 23s)
23:21 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Use full URL in $wgNoticeHideUrls (T130442) (duration: 00m 23s)
23:17 urandom: Deploying d8fa5c0 to RESTBase production
23:10 logmsgbot: dereckson@tin Synchronized wmf-config/CommonSettings.php: Use HTTPS URL to citoid instead of protocol-relative (T136423) (duration: 00m 32s)
23:06 urandom: Update restbase staging to f05b66f
22:36 cwd: updated paymentswiki from 44bd699d6700ac4faf3c2d772ba713b093ae8cb8 to d26426c4225080c95f0bd5a6a31c54e4826287b1
22:30 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.4/extensions/CentralNotice/: deploy https://gerrit.wikimedia.org/r/#/c/292279/ (duration: 00m 26s)
21:38 twentyafterfour: train has left the station
21:37 logmsgbot: twentyafterfour@tin Synchronized wmf-config/InitialiseSettings.php: deploy /wmf-config/InitialiseSettings.php for eranroz ( T132972 ) (duration: 00m 25s)
21:31 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.4/includes/specials/SpecialPrefixindex.php: sync https://gerrit.wikimedia.org/r/#/c/292228/ ( T136738 ) (duration: 00m 26s)
21:26 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.3/includes/specials/SpecialPrefixindex.php: sync https://gerrit.wikimedia.org/r/#/c/292234/ ( T136738 ) (duration: 00m 30s)
20:58 bearND: mobileapps deployed ed0e2e4
20:56 gehel: restarting postgresql on maps2001
20:55 bearND: starting mobileapps deploy
20:49 ejegg: updated paymentswiki from 7d222320b35ad8a44d8c77a4c3019364a49e53f2 to 44bd699d6700ac4faf3c2d772ba713b093ae8cb8
20:44 logmsgbot: twentyafterfour@tin rebuilt wikiversions.php and synchronized wikiversions files: group1 wikis to 1.28.0-wmf.4
20:39 logmsgbot: twentyafterfour@tin Synchronized php-1.28.0-wmf.4/includes/cache/LinkBatch.php: deploy https://gerrit.wikimedia.org/r/#/c/292217/ (duration: 00m 27s)
20:17 subbu: finished deploying parsoid sha afb0d522
20:17 urandom: Rolling restart of RESTBase (redistribute Cassandra client connections?) : T126629
20:10 subbu: synced new code; restarted parsoid on wtp1001 as a canary
20:07 subbu: starting parsoid deploy
19:43 ema: cp* hosts rebooted (T131928)
19:40 bblack: restarting pybals for healthcheck config changes
18:25 urandom: restarting Cassandra on restbase1007.eqiad.wmnet
18:19 ejegg: updated payments-wiki from 5bb160e9898224e1d7d0a5c57fe408edb998a262 to 7d222320b35ad8a44d8c77a4c3019364a49e53f2
18:16 ottomata: stopping kafka broker on kafka1018 and rebooting node
17:51 urandom: Restarting Cassandra on restbase1007.eqiad.wmnet : T126629
17:48 ema: depooled reboot of cp4* hosts (T131928)
17:47 urandom: Temporarily disabling puppet to test setting on restbase1007.eqiad.wmnet : T126629
17:15 ejegg: rolled back payments-wiki from a335a3a6f8909d1e7e1a79877512a12a0561aa2a to 5bb160e9898224e1d7d0a5c57fe408edb998a262
17:06 ejegg: updated payments-wiki from 5bb160e9898224e1d7d0a5c57fe408edb998a262 to a335a3a6f8909d1e7e1a79877512a12a0561aa2a
17:05 akosiaris: powered on lvs2006. disk change did not happen
17:05 akosiaris: powered off lvs2006 for disk swap
16:54 logmsgbot: tgr@tin Synchronized wmf-config/InitialiseSettings-labs.php: T135504: enable AuthManager in beta (duration: 00m 32s)
16:39 logmsgbot: tgr@tin Synchronized php-1.28.0-wmf.4/extensions/NewUserMessage/: backport gerrit:292168 to update NewUserMessage for AuthManager (duration: 00m 29s)
16:22 urandom: Disabling traces on restbase1008-a.eqiad.wmnet : T126629
16:01 logmsgbot: thcipriani@tin Finished scap: SWAT: Update for AuthManager (duration: 26m 05s)
15:58 urandom: Setting trace probability on restbase1008-a.eqiad.wmnet to 5% : T126629
15:58 jynus: updating dns entry for db1080.eqiad.wment
15:58 urandom: Disabling trace probability on restbase1007-a.eqiad.wmnet : T126629
15:48 urandom: Setting trace probability to 5% on restbase1007-a.eqiad.wmnet : T126629
15:35 logmsgbot: thcipriani@tin Started scap: SWAT: Update for AuthManager
15:33 logmsgbot: thcipriani@tin Synchronized php-1.28.0-wmf.4/resources/src/moment-locale-overrides.js: SWAT: Avoid passing integers to mw.RegExp.escape (duration: 00m 24s)
15:29 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove centralauth-autoaccount right (duration: 00m 25s)
15:26 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Enable bot passwords on zerowiki (duration: 00m 24s)
15:19 paravoid: Re-enabling OSPF on all cr1-codfw row subnets
15:18 paravoid: Re-enabling cr1-codfw et-0/* interfaces
15:18 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Revert "Enable RC patrol on ta.wikiquote" (duration: 00m 25s)
15:15 logmsgbot: thcipriani@tin Synchronized wmf-config/InitialiseSettings.php: SWAT: Remove no longer used Echo configuration PART II (duration: 00m 26s)
15:14 logmsgbot: thcipriani@tin Synchronized wmf-config/CommonSettings.php: SWAT: Remove no longer used Echo configuration PART I (duration: 00m 33s)
15:13 paravoid: Rebooting cr1-codfw FPC 0
15:09 paravoid: Upgrading cr1-codfw FPC 0 all PICs firmware
15:08 logmsgbot: thcipriani@tin Synchronized static/images/sul: SWAT: Make SUL icons square and use global defaults (duration: 00m 41s)
15:07 paravoid: Disabling cr1-codfw et-0/* (all row uplinks)
15:03 akosiaris: restarted grrrit-wm after gerrit restart
15:03 paravoid: Disabling OSPF on all cr1-codfw row subnets to drain FPC0
15:02 akosiaris: restarted gerrit to enforce 100m maxObjectSizeLimit
14:59 paravoid: Restoring VRRP priority on cr2-codfw
14:57 bblack: depooled reboot of cp3048 (T131928)
14:57 paravoid: Re-enabling OSPF on all cr2-codfw row subnets
14:54 paravoid: Re-enabling cr2-codfw et-0/* interfaces
14:49 paravoid: Rebooting cr2-codfw FPC 0
14:48 paravoid: Upgrading cr2-codfw FPC 0 all PICs firmware
14:42 paravoid: Disabling cr2-codfw et-0/2/0, et-0/2/1 (row C/D uplinks)
14:34 paravoid: Disabling cr2-codfw et-0/0/0 (row A uplink)
14:29 paravoid: Disabling cr2-codfw et-0/0/1 (row B uplink)
14:15 paravoid: Disabling OSPF on all cr2-codfw row subnets to drain FPC0
14:08 ema: depooled reboot of cp1* hosts (T131928)
12:49 paravoid: draining cr2-codfw for firmware upgrade
12:26 bblack: upgrade nginx to 1.11.1-1+wmf1 on all clusters
11:50 elukey: rebooting kafka1022 for kernel upgrade (4.4)
11:05 ema: rebooting cp3* spares (T131928)
10:47 Dereckson: Script done for uca-it collation on itwiki: 10 599 758 rows processed
10:47 ema: depooled reboot of cp3046 (T131928)
10:47 ema: depooled reboot of cp3003 (T131928)
10:45 ema: depooled reboot of cp3034 (T131928)
10:39 ema: depooled reboot of cp3005 (T131928)
10:38 ema: depooled reboot of cp3044 (T131928)
10:35 ema: depooled reboot of cp3047 (T131928)
10:31 ema: depooled reboot of cp3004 (T131928)
10:28 ema: depooled reboot of cp3009 (T131928)
10:14 ema: depooled reboot of cp3037 (T131928)
10:11 jynus: moved syslog1 to ms-be2012:/srv/swift-storage/sdl1/tmp to avoid / fillup
10:10 ema: depooled reboot of cp3008 (T131928)
10:09 ema: depooled reboot of cp3035 (T131928)
09:37 moritzm: installing libgd security updates
09:28 ema: depooled reboot of cp3039 (T131928)
09:23 ema: depooled reboot of cp3045 (T131928)
09:21 ema: depooled reboot of cp3010 (T131928)
09:18 ema: depooled reboot of cp3006 (T131928)
09:16 ema: depooled reboot of cp3007 (T131928)
09:10 ema: depooled reboot of cp3036 (T131928)
08:25 mobrovac: mobileapps deploying 8d6d648
08:24 ema: depooled reboot of cp3049 (T131928)
08:22 hashar: Nodepool came back up just fine after labnodepool1001 reboot and is fully operational.
08:15 jynus: deleting mysql logrotate scripts to avoid root spam
08:14 moritzm: reboot labnodepool1001 for update to Linux 4.4
07:56 elukey: event logging restarted on eventlog1001.eqiad.wmnet
07:46 elukey: stopping kafka on kafka1020.eqiad and rebooting the host for Linux 4.4 upgrades
07:43 moritzm: rolling reboot of scb in eqiad for update to Linux 4.4
07:32 moritzm: restarted hhvm on mw1180
07:05 mobrovac: change-prop restarting to apply https://gerrit.wikimedia.org/r/291201
05:41 mobrovac: restbase deploy end of 5c99693
05:26 mobrovac: restbase deploy start of 5c99693
04:31 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.4/includes/: reapplied new version of I03739e94 (duration: 01m 21s)
04:27 logmsgbot: demon@tin Synchronized php-1.28.0-wmf.3/includes/: reapplied new version of I03739e94 (duration: 01m 34s)
03:11 logmsgbot: l10nupdate@tin ResourceLoader cache refresh completed at Wed Jun 1 03:11:11 UTC 2016 (duration 6m 39s)
03:04 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.4) (duration: 15m 42s)
02:30 logmsgbot: mwdeploy@tin scap sync-l10n completed (1.28.0-wmf.3) (duration: 09m 30s)
00:04 Dereckson: Started `mwscript updateCollation.php itwiki --previous-collation=uppercase` on Terbium (T136647)

2000s

Archive 1: 2004 Jun - 2004 Sep
Archive 2: 2004 Oct - 2004 Nov
Archive 3: 2004 Dec - 2005 Mar
Archive 4: 2005 Apr - 2005 Jul
Archive 5: 2005 Aug - 2005 Oct, with revision history 2004-06-23 to 2005-11-25
Archive 6: 2005 Nov - 2006 Feb
Archive 7: 2006 Mar - 2006 Jun
Archive 8: 2006 Jul - 2006 Sep
Archive 9: 2006 Oct - 2007 Jan, with revision history 2005-11-25 to 2007-02-21
Archive 10: 2007 Feb - 2007 Jun
Archive 11: 2007 Jul - 2007 Dec
Archive 12: 2008 Jan - 2008 Jul
Archive 12a: 2008 Aug
Archive 12b: 2008 Sept
Archive 13: 2008 Oct - 2009 Jun
Archive 14: 2009 Jun - 2009 Dec

2010s

2020s

@@ Line 1: / Line 1: @@
+== 2016-11-13 ==
+* 01:24 Krenair: Spoken to User:Nirzardp for T150554, set a new password
 == 2016-11-12 ==
 * 19:53 tgr: deployed patch for T150554