Page MenuHomePhabricator

Liz
User

Projects

User does not belong to any projects.

Today

  • Clear sailing ahead.

Tomorrow

  • Clear sailing ahead.

Saturday

  • Clear sailing ahead.

User Details

User Since
Jan 27 2019, 6:24 AM (300 w, 4 d)
Availability
Available
LDAP User
Unknown
MediaWiki User
Liz [ Global Accounts ]

Recent Activity

Sep 30 2024

Liz edited projects for T375988: Quarry shows error: This web service cannot be reached, added: Quarry; removed Cloud-Services.
Sep 30 2024, 3:08 AM · Quarry
Liz updated the task description for T375988: Quarry shows error: This web service cannot be reached.
Sep 30 2024, 3:07 AM · Quarry
Liz created T375988: Quarry shows error: This web service cannot be reached.

The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

Sep 30 2024, 3:04 AM · Quarry

Aug 12 2024

Liz added a comment to T367856: Cleanup revision table schema.

Yes, looks like the long wait is over. Thank you for helping the servers/system run smoothly.

Aug 12 2024, 4:26 PM · Data-Engineering (Q2 2024 October 1st - December 31th), Schema-change-in-production, DBA, Data Products
Liz added a comment to T367856: Cleanup revision table schema.

But, the million dollar question is, when will it ever be completed? Obviously not in 26 hours since we are currently at 209 hours. I know that whenever the lag ends, the system will catch up, it's just that it's been 8+ days now. I don't remember it ever taking more than 3 or 4 days or replag. But then I've only recently been paying attention.

Aug 12 2024, 3:44 AM · Data-Engineering (Q2 2024 October 1st - December 31th), Schema-change-in-production, DBA, Data Products

Aug 11 2024

Liz added a comment to T367856: Cleanup revision table schema.

I've been looking at that note for 7 days now. Hopefully, there won't be a "next time". Or is that wishful thinking?

Aug 11 2024, 2:41 AM · Data-Engineering (Q2 2024 October 1st - December 31th), Schema-change-in-production, DBA, Data Products

Aug 10 2024

Liz added a comment to T367856: Cleanup revision table schema.

@Liz. I know you use a couple of database reports to do some important daily tasks on enwiki. What are the links to those reports please?

If I understand it correctly, there are basically 2 sanitized replicas (analytics and web), and only 1 of them (analytics) currently has replag. Perhaps someone can generate a one-time report for you using the one without replag (web), so that you can start catching up on your daily tasks.

Aug 10 2024, 10:44 PM · Data-Engineering (Q2 2024 October 1st - December 31th), Schema-change-in-production, DBA, Data Products
Liz added a comment to T367856: Cleanup revision table schema.

This is crazy! Replag at 158 hours now!

Aug 10 2024, 1:01 AM · Data-Engineering (Q2 2024 October 1st - December 31th), Schema-change-in-production, DBA, Data Products

Aug 8 2024

Liz added a comment to T367856: Cleanup revision table schema.

The replag is at 0 but when I do Quarry queries, the information is all from August 3rd, nothing more recent than that and there should be a backlog of errors to fix. And the bots I rely on haven't started reporting again. When will the system "catch up"?

Aug 8 2024, 6:39 PM · Data-Engineering (Q2 2024 October 1st - December 31th), Schema-change-in-production, DBA, Data Products
Liz added a comment to T367856: Cleanup revision table schema.

Well, it didn't finish in less than a day. Currently a 112 hour lag.

Aug 8 2024, 2:56 AM · Data-Engineering (Q2 2024 October 1st - December 31th), Schema-change-in-production, DBA, Data Products

Aug 5 2024

Liz added a comment to T367856: Cleanup revision table schema.

I know, right now the replag is 61 hours.

Aug 5 2024, 11:22 PM · Data-Engineering (Q2 2024 October 1st - December 31th), Schema-change-in-production, DBA, Data Products
Liz added a comment to T367856: Cleanup revision table schema.

Do you have any idea how much longer this job will take? It's been a couple days now.

Aug 5 2024, 6:25 PM · Data-Engineering (Q2 2024 October 1st - December 31th), Schema-change-in-production, DBA, Data Products

Jul 28 2024

Liz added a comment to T371198: [bug] Quarry queries on terminal "queue" status.

It looks like this, https://phabricator.wikimedia.org/T365136, was a similar problem.

Jul 28 2024, 8:02 PM · Quarry
Liz updated the task description for T371198: [bug] Quarry queries on terminal "queue" status.
Jul 28 2024, 8:00 PM · Quarry
Liz created T371198: [bug] Quarry queries on terminal "queue" status.
Jul 28 2024, 7:59 PM · Quarry

Jun 19 2024

Liz added a comment to T367464: [bug] Quarry queries not completing .

Well, the queries were running and finishing up for a few hours last night (at least it was night where I was at) but now they are back to not completing at all. But for a few hours, everything was in sync.

Jun 19 2024, 7:41 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry
Liz added a comment to T367464: [bug] Quarry queries not completing .

But hey, a few of my queries just issued reports! I don't know what happened since I posted this message but something has changed for the better. Surprised me.

Jun 19 2024, 12:21 AM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry
Liz added a comment to T367464: [bug] Quarry queries not completing .

I detected the problem whenever I opened this bug report. I remember filing a different one for Quarry about two weeks ago but this kind of incident, with queries running endlessly and never finishing, only happened starting last week at least for me. You'll notice in my original bug report, I say that about 1/2 of my queries were completing but that soon turned into ZERO queries finishing.

Jun 19 2024, 12:18 AM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry

Jun 18 2024

Liz added a comment to T367464: [bug] Quarry queries not completing .

The status says this case is open and is a high priority but it's not getting any attention from those who might be in a position to resolve this problem.

Jun 18 2024, 2:48 AM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry

Jun 17 2024

Liz added a comment to T367464: [bug] Quarry queries not completing .

Well, maybe it depends on the database, queries aren't running on enwiki_p. But I've noticed that editors have just given up on running Quarry queries since about 2 days ago when it stopped working

Jun 17 2024, 4:53 AM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry

Jun 16 2024

Liz added a comment to T367464: [bug] Quarry queries not completing .

But hitting "Explain" fills up the part of the page with a table where the query results should be plus it is in code that only a developer or coder would understand, not us mere editors.

Jun 16 2024, 8:25 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry
Liz added a comment to T367464: [bug] Quarry queries not completing .

Now, there is a little grey box saying "Explain" at the bottom of the page under Query Status (see this Quarry query ) that just produces a table of code I don't understand and isn't helpful at all, at least to me.

Jun 16 2024, 4:21 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry

Jun 14 2024

Liz reopened T367464: [bug] Quarry queries not completing as "Open".
Jun 14 2024, 2:49 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry
Liz added a comment to T367464: [bug] Quarry queries not completing .

I don't see that there has been any improvement. My queries are still not finishing their run.

Jun 14 2024, 2:47 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry

Jun 13 2024

Liz created T367464: [bug] Quarry queries not completing .
Jun 13 2024, 8:27 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry
Liz added a comment to T365374: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO).

I'm not getting that message any more but some queries are just running endlessly and never finishing. I had one that ran for more than 3 hours before I finally cancelled it. Does this warrant a new case to be opened?

Jun 13 2024, 8:21 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry

Jun 12 2024

Liz added a comment to T365374: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO).

Some of my queries are running okay now, others are just spinning their wheels, like they are running but they never conclude or take an unusual length of time to do so. So, a partial fix. And with one query, I get an error message

Jun 12 2024, 6:12 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry

Jun 11 2024

Liz added a comment to T365374: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO).

Sorry to post a duplicate notice. I couldn't find this one when I wanted to post about this bug.

Jun 11 2024, 10:51 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry
Liz created T367237: [bug] Trouble running Quarry queries.
Jun 11 2024, 9:09 PM · Quarry
Josve05a awarded T365374: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO) a The World Burns token.
Jun 11 2024, 4:01 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry

May 21 2024

Liz added a comment to T365374: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO).

Thanks to whomever fixed this problem.

May 21 2024, 11:09 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry
Liz added a comment to T365374: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO).

If you want to see an example of this, try Category redirects and look at the history.

May 21 2024, 4:39 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry

May 20 2024

Liz added a comment to T365374: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO).

I spoke too soon. It's happening again.

May 20 2024, 8:04 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry
Liz added a comment to T365374: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO).

Sorry, but whatever was the problem is fixed, it's working now. Feel free to close this. Thanks.

May 20 2024, 4:36 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry
Liz updated the task description for T365374: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO).
May 20 2024, 4:33 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry
Liz created T365374: [bug] Access denied for user 'quarry'@'172.16.2.72' (using password: NO).
May 20 2024, 4:31 PM · cloud-services-team (FY2023/2024-Q3-Q4), Quarry

May 8 2024

Liz added a comment to T363757: Add tm: as alias to template: on English Wikipedia.

DreamRimmer, the page you tagged for deletion created dozens of broken redirects. I'd reconsider doing this in the futuree.

May 8 2024, 2:42 AM · Wikimedia-Site-requests

Apr 27 2024

Liz added a comment to T207959: [Pages created] Investigate allowing querying of users with over max edit count.

Is there any discussion here any more about raising the count of edits for productive editors to see their Edit Count information? I could see all of my data until it reached 600,000 edits, then I was given permission to reach 650,000 edits and five months later I reached this ceiling. It is very useful to see the XTools information which I checked several times a day, both my editor counts and admin counts.

Apr 27 2024, 5:40 AM · XTools
Liz added a comment to T318036: Investigate raising max edit count above 500,000.

Is there any discussion here any more about raising the count of edits for productive editors to see their Edit Count information? I could see all of my data until it reached 600,000 edits, then I was given permission to reach 650,000 edits and five months later I reached this ceiling. It is very useful to see the XTools information which I checked several times a day, both my editor counts and admin counts.

Apr 27 2024, 5:36 AM · XTools

Apr 22 2024

Liz added a comment to T363077: High replication lag for enwiki (db1154 s1 replication crashed).

Well, my "complaints" are actually a request for information on when this problem will be fixed. Where else can I ask? I was directed to come here.

Apr 22 2024, 3:49 AM · DBA, Data-Services
Liz added a comment to T363077: High replication lag for enwiki (db1154 s1 replication crashed).

Now a 7 hour system lag.

Apr 22 2024, 12:59 AM · DBA, Data-Services
Liz added a comment to T352010: Gradually drop old pagelinks columns.

Oh, no, this task isn't done yet? System lag again? I hope this one is shorter than the last one.

It's not related to this ticket per se. you should check T363077: High replication lag for enwiki (db1154 s1 replication crashed)

Apr 22 2024, 12:56 AM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data

Apr 21 2024

Liz added a comment to T352010: Gradually drop old pagelinks columns.

Oh, no, this task isn't done yet? System lag again? I hope this one is shorter than the last one.

Apr 21 2024, 7:21 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data

Apr 18 2024

Liz added a comment to T362732: enwiki_p database replica has stopped updating.

Yeah! Life returns to normal.

Apr 18 2024, 2:05 AM · Data-Services
Liz added a comment to T352010: Gradually drop old pagelinks columns.

Looks like everything is back to normal. Thanks for all of the work you do for the project.

Apr 18 2024, 2:03 AM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data

Apr 17 2024

Liz added a comment to T352010: Gradually drop old pagelinks columns.

Looks like the lag is decreasing now.

Apr 17 2024, 8:25 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Liz added a comment to T352010: Gradually drop old pagelinks columns.

Do you have any idea when this task will be done, the system will catch up to the 10 hour time lag and everything goes back to normal?

It'll take a while:

grafik.png (306×1 px, 59 KB)

My rough estimate is around 6-8 more hours

Apr 17 2024, 4:32 PM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Liz added a comment to T362732: enwiki_p database replica has stopped updating.

I updated the description. Any tools that rely on database replicas, including all toolforge tools that rely on data not available through the API, are affected by this.

Apr 17 2024, 4:26 PM · Data-Services
Liz added a comment to T352010: Gradually drop old pagelinks columns.

Do you have any idea when this task will be done, the system will catch up to the 10 hour time lag and everything goes back to normal?

Apr 17 2024, 3:27 AM · Schema-change-in-production, DBA, MediaWiki-Page-derived-data
Liz added a comment to T362732: enwiki_p database replica has stopped updating.

I can't run any of my Quarry queries and all of the bots I use for editing purposes aren't updating as well. When will this system update be complete and the system lag catch up? Tonight? Tomorrow morning? Next week?

Apr 17 2024, 1:21 AM · Data-Services

Jun 4 2023

Liz added a comment to T338006: [jobs-api,jobs-cli,infra] Transient cronjob scheduling failures on Toolforge k8s.

And both reports were skipped again tonight for the 4th day in a row. We work with a 7 day lag between when the G13 soon report is issued and when action is required and we're now down to a three day supply of reports until we are really in trouble.

Jun 4 2023, 12:14 AM · Toolforge, Kubernetes

Jun 1 2023

Liz added a comment to T337446: Rebuild sanitarium hosts.

I'm not sure what's causing it (regarding s1), but I'm finding some bots are not returning up-to-date reports. With s1 down for 5 days, there should be a backlog of lengthy reports but I'm seeing short reports or none at all. Did every new edit since May 25th get restored and integrated? Sorry that I don't know the correct terminology.

Jun 1 2023, 12:55 AM · User-notice-archive, TaxonBot, cloud-services-team, Data-Engineering, Data-Services, DBA

May 30 2023

Liz added a comment to T337446: Rebuild sanitarium hosts.

Looks like it's just s1 to be restored (or whatever the correct term is).

May 30 2023, 5:34 PM · User-notice-archive, TaxonBot, cloud-services-team, Data-Engineering, Data-Services, DBA

May 26 2023

Liz added a comment to T337446: Rebuild sanitarium hosts.

Any status updates on this issue? Replag is at 19 hours now.

May 26 2023, 1:18 AM · User-notice-archive, TaxonBot, cloud-services-team, Data-Engineering, Data-Services, DBA

May 22 2023

Liz added a comment to T169452: Replace Quarry with an installation of Superset.

My only concern is that I'm not a programmer, just a regular editor, and I run several queries evert day that were written for me by more tech-minded editors. I hope if we move from Quarry to Superset, it is at least as user-friendly and easy to use as Quarry is (and hopefully, even more so!).

May 22 2023, 10:57 PM · cloud-services-team (FY2024/2025-Q1-Q2), superset.wmcloud.org, Quarry

Aug 23 2022

Liz added a comment to T307314: Lag in updating Special Pages?.

I have been told, probably repeatedly, that Special Pages are generated automatically every 3 days but the ones I regularly use https://en.wikipedia.org/wiki/Special:UnusedCategories and https://en.wikipedia.org/wiki/Special:WantedCategories, were supposed to update some time on August 22nd and didn't.

Aug 23 2022, 7:38 PM · DBA, serviceops, WMF-General-or-Unknown

Jun 13 2022

Liz added a comment to T310431: Unable to view all Wikimedia projects.

This happened again in the past 15 minutes and lasted about 4 or 5 minutes.

Jun 13 2022, 4:16 AM · Wikimedia-Incident, SRE, Traffic
Liz added a comment to T310325: s1 (enwiki) wiki replicas replication has stopped.

Well, I noticed that some bots, like AnomieBOT III, is back to issuing reports, while others, like SDZeroBot, are still affected. I guess it takes a while for everything to return to normal.

Jun 13 2022, 12:55 AM · cloud-services-team (Kanban), DBA, Data-Services

Jun 12 2022

Liz added a comment to T310325: s1 (enwiki) wiki replicas replication has stopped.

I don't understand the difference between what is "production" and what is not (and what difference that makes to the developers) but, again, I'm with Jonesey95. All information that is communicated is valuable to us volunteer editors, including what you have just shared here today. Most of us have our daily editing routines and handling problems that show up on bot or database reports is part of our daily work. When they go down, we understandably have questions.

Jun 12 2022, 12:36 AM · cloud-services-team (Kanban), DBA, Data-Services

Jun 11 2022

Liz added a comment to T310325: s1 (enwiki) wiki replicas replication has stopped.

Monday is not "soon", at least not to me on this Friday night. But I appreciate any specific estimate.

Jun 11 2022, 5:54 AM · cloud-services-team (Kanban), DBA, Data-Services

Jun 10 2022

Liz added a comment to T310325: s1 (enwiki) wiki replicas replication has stopped.

And now there is a 34 hour lag! How can a system catch up with a lag longer than a day?

Jun 10 2022, 8:20 PM · cloud-services-team (Kanban), DBA, Data-Services
Liz added a comment to T310325: s1 (enwiki) wiki replicas replication has stopped.

And now it's 18 hours behind.

Jun 10 2022, 4:08 AM · cloud-services-team (Kanban), DBA, Data-Services

Jun 1 2022

Liz added a comment to T307314: Lag in updating Special Pages?.

Just a reminder that today is June 1st and the Special Pages, at least the ones we utilize on En.Wiki, typically get generated on the 1st of the month...and then the 4th, 7th, 10th, 13th, 16th, 19th, 22nd, 25th, 28th and sometimes the 31st. If they could get back to being generated on their old, regular schedule, that would be great!

Jun 1 2022, 7:22 PM · DBA, serviceops, WMF-General-or-Unknown

May 31 2022

Liz added a comment to T307314: Lag in updating Special Pages?.

This happened again today. The special pages involving categories eventually updated at around 00:30 UTC on May 29th but so far, no updates today on May 31st and they usually also update on the 1st of every month so the reports should have been issued both today and also tomorrow....so we'll see.

May 31 2022, 11:00 PM · DBA, serviceops, WMF-General-or-Unknown
Liz added a comment to T309570: quarry is unable to access enwiki_p.page table.

Looks like everything is fixed now. Thank you for getting right on this bug report!

May 31 2022, 4:13 AM · Quarry, Wikimedia-production-error
Liz added a comment to T309569: HTTP 500 - View 'enwiki_p.page' references invalid table(s) or column(s) or function(s) or definer/invoker of view lack rights to use them.

I'm also experience problems with Quarry, 2 bots on en.wiki which haven't issued their regularly scheduled reports and receiving "overflow" messages when I try to look at page histories earlier today. Are these connected?

May 31 2022, 12:47 AM · XTools
Liz added a comment to T309570: quarry is unable to access enwiki_p.page table.

I'm also experience problems with 2 bots on en.wiki which haven't issued their regularly scheduled reports and receiving "overflow" messages when I try to look at page histories earlier today. Are these connected?

May 31 2022, 12:43 AM · Quarry, Wikimedia-production-error
Liz added a comment to T157670: Periodically run refreshLinks.php on production sites..

I'm also experience problems with two bots on en.wiki which haven't issued their regularly scheduled reports and receiving "overflow" messages when I try to look at page histories earlier today. Are these connected?

May 31 2022, 12:43 AM · Platform Engineering Roadmap Decision Making, MediaWiki-Parser, MediaWiki-Page-editing, Parsing-Team--ARCHIVED

May 28 2022

Liz reopened T307314: Lag in updating Special Pages? as "Open".

Looks like this special pages posting lag is happening again, on En.Wiki, special pages involving categories typically, for years, updated around 13:00-15:00 UTC time every 3 days but on May 25th, they updated at 19:00 UTC and today, they haven't updated at all. I'd prefer that they "catch up" and post sometime today rather than skipping a date and just updating again on June 1st as they did at the beginning of May when this lag was first reported. A late update is better than omitting an update!

May 28 2022, 8:11 PM · DBA, serviceops, WMF-General-or-Unknown

May 1 2022

Liz created T307314: Lag in updating Special Pages?.
May 1 2022, 9:51 PM · DBA, serviceops, WMF-General-or-Unknown

Apr 27 2021

Liz created T281281: Error message when trying to get edit count/summary.
Apr 27 2021, 3:43 PM · XTools

Feb 23 2021

Liz created T275466: Split wikistats metrics out by namespace.
Feb 23 2021, 3:43 AM · Data-Engineering, Data-Engineering-Wikistats