The Cloud-Services project tag is not intended to have any tasks. Please check the list on https://phabricator.wikimedia.org/project/profile/832/ and replace it with a more specific project tag to this task. Thanks!

Yes, looks like the long wait is over. Thank you for helping the servers/system run smoothly.

But, the million dollar question is, when will it ever be completed? Obviously not in 26 hours since we are currently at 209 hours. I know that whenever the lag ends, the system will catch up, it's just that it's been 8+ days now. I don't remember it ever taking more than 3 or 4 days or replag. But then I've only recently been paying attention.

I've been looking at that note for 7 days now. Hopefully, there won't be a "next time". Or is that wishful thinking?

In T367856#10055126, @Novem_Linguae wrote:

@Liz. I know you use a couple of database reports to do some important daily tasks on enwiki. What are the links to those reports please?

If I understand it correctly, there are basically 2 sanitized replicas (analytics and web), and only 1 of them (analytics) currently has replag. Perhaps someone can generate a one-time report for you using the one without replag (web), so that you can start catching up on your daily tasks.

This is crazy! Replag at 158 hours now!

The replag is at 0 but when I do Quarry queries, the information is all from August 3rd, nothing more recent than that and there should be a backlog of errors to fix. And the bots I rely on haven't started reporting again. When will the system "catch up"?

Well, it didn't finish in less than a day. Currently a 112 hour lag.

I know, right now the replag is 61 hours.

Do you have any idea how much longer this job will take? It's been a couple days now.

It looks like this, https://phabricator.wikimedia.org/T365136, was a similar problem.

Well, the queries were running and finishing up for a few hours last night (at least it was night where I was at) but now they are back to not completing at all. But for a few hours, everything was in sync.

But hey, a few of my queries just issued reports! I don't know what happened since I posted this message but something has changed for the better. Surprised me.

I detected the problem whenever I opened this bug report. I remember filing a different one for Quarry about two weeks ago but this kind of incident, with queries running endlessly and never finishing, only happened starting last week at least for me. You'll notice in my original bug report, I say that about 1/2 of my queries were completing but that soon turned into ZERO queries finishing.

The status says this case is open and is a high priority but it's not getting any attention from those who might be in a position to resolve this problem.

Well, maybe it depends on the database, queries aren't running on enwiki_p. But I've noticed that editors have just given up on running Quarry queries since about 2 days ago when it stopped working

But hitting "Explain" fills up the part of the page with a table where the query results should be plus it is in code that only a developer or coder would understand, not us mere editors.

Now, there is a little grey box saying "Explain" at the bottom of the page under Query Status (see this Quarry query ) that just produces a table of code I don't understand and isn't helpful at all, at least to me.

I don't see that there has been any improvement. My queries are still not finishing their run.

I'm not getting that message any more but some queries are just running endlessly and never finishing. I had one that ran for more than 3 hours before I finally cancelled it. Does this warrant a new case to be opened?

Some of my queries are running okay now, others are just spinning their wheels, like they are running but they never conclude or take an unusual length of time to do so. So, a partial fix. And with one query, I get an error message

Sorry to post a duplicate notice. I couldn't find this one when I wanted to post about this bug.

Thanks to whomever fixed this problem.

If you want to see an example of this, try Category redirects and look at the history.

I spoke too soon. It's happening again.

Sorry, but whatever was the problem is fixed, it's working now. Feel free to close this. Thanks.

DreamRimmer, the page you tagged for deletion created dozens of broken redirects. I'd reconsider doing this in the futuree.

Is there any discussion here any more about raising the count of edits for productive editors to see their Edit Count information? I could see all of my data until it reached 600,000 edits, then I was given permission to reach 650,000 edits and five months later I reached this ceiling. It is very useful to see the XTools information which I checked several times a day, both my editor counts and admin counts.

Is there any discussion here any more about raising the count of edits for productive editors to see their Edit Count information? I could see all of my data until it reached 600,000 edits, then I was given permission to reach 650,000 edits and five months later I reached this ceiling. It is very useful to see the XTools information which I checked several times a day, both my editor counts and admin counts.

Well, my "complaints" are actually a request for information on when this problem will be fixed. Where else can I ask? I was directed to come here.

Now a 7 hour system lag.

In T352010#9732464, @Ladsgroup wrote:

In T352010#9732305, @Liz wrote:

Oh, no, this task isn't done yet? System lag again? I hope this one is shorter than the last one.

It's not related to this ticket per se. you should check T363077: High replication lag for enwiki (db1154 s1 replication crashed)

Oh, no, this task isn't done yet? System lag again? I hope this one is shorter than the last one.

Yeah! Life returns to normal.

Looks like everything is back to normal. Thanks for all of the work you do for the project.

Looks like the lag is decreasing now.

In T352010#9721868, @Ladsgroup wrote:

In T352010#9721159, @Liz wrote:

Do you have any idea when this task will be done, the system will catch up to the 10 hour time lag and everything goes back to normal?

It'll take a while:

My rough estimate is around 6-8 more hours

In T362732#9722693, @Ahecht wrote:

I updated the description. Any tools that rely on database replicas, including all toolforge tools that rely on data not available through the API, are affected by this.

Do you have any idea when this task will be done, the system will catch up to the 10 hour time lag and everything goes back to normal?

I can't run any of my Quarry queries and all of the bots I use for editing purposes aren't updating as well. When will this system update be complete and the system lag catch up? Tonight? Tomorrow morning? Next week?

And both reports were skipped again tonight for the 4th day in a row. We work with a 7 day lag between when the G13 soon report is issued and when action is required and we're now down to a three day supply of reports until we are really in trouble.

I'm not sure what's causing it (regarding s1), but I'm finding some bots are not returning up-to-date reports. With s1 down for 5 days, there should be a backlog of lengthy reports but I'm seeing short reports or none at all. Did every new edit since May 25th get restored and integrated? Sorry that I don't know the correct terminology.

Looks like it's just s1 to be restored (or whatever the correct term is).

Any status updates on this issue? Replag is at 19 hours now.

My only concern is that I'm not a programmer, just a regular editor, and I run several queries evert day that were written for me by more tech-minded editors. I hope if we move from Quarry to Superset, it is at least as user-friendly and easy to use as Quarry is (and hopefully, even more so!).

I have been told, probably repeatedly, that Special Pages are generated automatically every 3 days but the ones I regularly use https://en.wikipedia.org/wiki/Special:UnusedCategories and https://en.wikipedia.org/wiki/Special:WantedCategories, were supposed to update some time on August 22nd and didn't.

This happened again in the past 15 minutes and lasted about 4 or 5 minutes.

Well, I noticed that some bots, like AnomieBOT III, is back to issuing reports, while others, like SDZeroBot, are still affected. I guess it takes a while for everything to return to normal.

I don't understand the difference between what is "production" and what is not (and what difference that makes to the developers) but, again, I'm with Jonesey95. All information that is communicated is valuable to us volunteer editors, including what you have just shared here today. Most of us have our daily editing routines and handling problems that show up on bot or database reports is part of our daily work. When they go down, we understandably have questions.

Monday is not "soon", at least not to me on this Friday night. But I appreciate any specific estimate.

And now there is a 34 hour lag! How can a system catch up with a lag longer than a day?

And now it's 18 hours behind.

Just a reminder that today is June 1st and the Special Pages, at least the ones we utilize on En.Wiki, typically get generated on the 1st of the month...and then the 4th, 7th, 10th, 13th, 16th, 19th, 22nd, 25th, 28th and sometimes the 31st. If they could get back to being generated on their old, regular schedule, that would be great!

This happened again today. The special pages involving categories eventually updated at around 00:30 UTC on May 29th but so far, no updates today on May 31st and they usually also update on the 1st of every month so the reports should have been issued both today and also tomorrow....so we'll see.

Looks like everything is fixed now. Thank you for getting right on this bug report!

I'm also experience problems with Quarry, 2 bots on en.wiki which haven't issued their regularly scheduled reports and receiving "overflow" messages when I try to look at page histories earlier today. Are these connected?

I'm also experience problems with 2 bots on en.wiki which haven't issued their regularly scheduled reports and receiving "overflow" messages when I try to look at page histories earlier today. Are these connected?

I'm also experience problems with two bots on en.wiki which haven't issued their regularly scheduled reports and receiving "overflow" messages when I try to look at page histories earlier today. Are these connected?

Looks like this special pages posting lag is happening again, on En.Wiki, special pages involving categories typically, for years, updated around 13:00-15:00 UTC time every 3 days but on May 25th, they updated at 19:00 UTC and today, they haven't updated at all. I'd prefer that they "catch up" and post sometime today rather than skipping a date and just updating again on June 1st as they did at the beginning of May when this lag was first reported. A late update is better than omitting an update!