Well, as usual the actual logic is pretty easy but I'm fighting with java/scala to build this jar with the correct version of scala. For some reason it's picking up 2.13. But in theory I kind of have this working, the patch is super simple and would be very easy to build and maintain as we upgrade Spark. It only requires a few lines of code to be changed. So except for java being insane, this kind of thing could work well:

I have deployed this so it should be picked up by the job that starts on Sep. 1st. I have not dropped the column out of the table because old data is deleted quickly anyway and it doesn't hurt the insert.

this got lost in the sprint move and almost broke a bunch of stuff, but I had a nightmare about it so it's all good

Seems easy enough to extend, here's the PR that added support for timestamp. but I doubt upstream would ever want to merge. So this would just be our personal little hack.

temporarily grabbing this to look into the "modify the JDBC data source to accept a function" option.

I'm back on dumps after some time, things I'm going to look at, in this order:

So to confirm, this means:

• over 1% of page views (after deducting known bots and spiders) are coming from clients with user agents that are entirely unknown to ua-parser. That is, the "Other" is already there in the raw wmf.webrequest_text dataset, and we've not created or normalized anything else to "Other".
• 0.26% is "Redacted" where we replace/normalize/summarise for privacy reasons browser/OS names in our pipeline.

In T342267#10060142, @Krinkle wrote:

Hm.. it seems the "Other" bucket has grown slightly larger than our predictions of 0.26% prediction at T342267#9998984

the new graphs are up. The pivot transformation failed for all the browser family reports, so I'm still fixing that. But, for example, we can now compare these two:

The backfill job should be done sometime this weekend, and I'll rerun the weekly job then.

UPDATE: the below didn't work, I just ended up deleting the DAG and setting its start date to 2015-06-01

Status update on this: the new job is running, I'm going to keep it here until we vet the data. But new data should start showing up right away, and we can compare dashboards side by side and day by day:

I think I need to speak with Sam and understand this N / M enrollments problem some more because I'm not getting it right now. With that caveat, thoughts:

@VirginiaPoundstone adding this to our sprint because it's basically a no-op and an easy resolution to an old task.

Oh! Thanks for the reminder, this is now available but not included in the sqoop lists. I'll make a patch, easy enough. labswiki seems to be available in both the analytics replicas and cloud replicas (as labswiki_p in the latter)

Some initial thoughts: https://docs.google.com/document/d/1NorKzBiQyz2nXCUUkGqdFP8SfPD0bzBMdfPO183dqNY/edit

Adding this to our Sprint board as it needs to get a look and deployment. But the change looks good, just have to alter the table and make sure the timing of all the jobs works out.

great, moving this to get deployed. Steps will be:

Just for the record, we met and discussed @Joe's proposal (this task's description) and were in general agreement that it's the best way forward. We have follow-up discussions to have and coordination to do, but we're aligned on the idea.

K, as a final update here, the pipeline is:

manually running this in a screen on an-launcher1002:

Ok, this ended up being very involved. I believe the root of all the confusion is that all the dumps jobs assume the PREVIOUS dump finished and work only on the CURRENT dump. So we ran around dumpsdata and snapshot hosts, hardcoding 20240701 where it was looking for "latest" and we're not sure whether we broke anything. At the end of the day, we basically figured that the snapshot1010 version of dumps files seemed all good, and we just rsynced them over to dumpsdata and clouddumps hosts. The rsync service that runs ALSO assumes this "latest" thing, but not for all files, just for the status files. So as far as we could tell everything was already rsynced except the status and html files. The monitor/html generation service ALSO assumes this "latest" thing so we weren't able to run that to generate the html, even after trying to hack it, but the html files were already on the snapshot hosts so we just moved those over with rsync too. The base rsync excludes json and html, so we just hacked it to include them instead.

Quick spike to size up how many revisions we're dealing with on a daily basis:

oof, I just realized this is for the month BEFORE. I see that's still in-process:

Ok, dug into this a bit more. Looks like the job set up to import the dumps XML is running fine but the status file says wikidatawiki is still in progress. Specifically it says this:

The airflow sensor timed out. But I never saw an alert for it (maybe it was before this week). I cleared it and will report back here in a bit after it has a chance to think about running again.

Deployed, started job, waiting to see if it works.

I deployed this and started the job, checking in now to make sure it runs.

• wmf.mediawiki_history: duplicate revision/create records indeed exist, some have 4 copies and some 2 copies but all spot-checked duplicates come in even numbers
• wmf_raw.mediawiki_revision: does not show the same duplication
• analytics mysql replicas: the pages those revisions belong to were moved and had some delete/restore and delete/revision actions in the logging table
• cloud replicas: agrees with analytics replicas

+1 to decom

We do not currently use globalblocks anywhere I know of or searched.

Sorry to ask this very basic question, but I found a bunch of others didn't know: how exactly is Dumps blocking the php 8 upgrade? Like, if we leave everything exactly as-is and just upgrade PHP, would it not run the way it is currently set up? On surface I see no big difference between the current setup and a containerized MW running on the same servers, so I'm curious about the nuance I'm missing here.

Sorry, I just signed it, I'm sure I signed it or some form of it at some point before, I've been an employee for like 12 years almost :P

ok, moving to ready to deploy. I'm going to ping @Krinkle one more time for data review. I executed this as I was testing and the results are available in milimetric.browser_general_test. You can query this like this:

weird... quick steps as I look into this.

Ok, sent updated code, it's fast now due to a CACHE statement, but that doesn't change the query plan which is still absolutely nuts, check this out:

Quick spark-sql query to get link changes where someone tags a new wiki project on the talk page:

OK, so it seems most problems do indeed track back to not applying delete and restore events. It feels like we can mark this task complete. We can find a way to apply delete/restore/merge, and then run these queries again and see what we need to reconcile. The period I looked at above was 10 days of enwiki revisions. If anyone disagrees, do move this task back.

That one mismatch_page that has no other reason listed is apparently part of a merge, so if we're not following up on delete/restore properly then this makes perfect sense because merges are more complicated still. Here are the two pages involved and the logging table records for them:

Ok, I think I got this query to make sense... the results:

I am still trying to find an elegant way to change the queries and show all this, but I just wanted to share results so far:

Apologies for the week delay here, I was out sick, picking it back up soon.

Quick summary of last meeting. Luke started working on a draft of what we were talking about (see the reconciliation flow on https://miro.com/app/board/uXjVNfaohl0=/).

My first hunch, that the revisions were coming from only specific pages, is wrong:

are we literally saying that we should just change the value of statistics-users-active to Editors? Code here: https://gerrit.wikimedia.org/g/mediawiki/core/+/1aa990f1725bf81caaf44527b9e778b5a8fe7e4d/languages/i18n/en.json#1950

Thanks for pinging us, we don't use abuse filter tables anywhere I'm aware of, so this shouldn't affect us.

Thanks for pinging us on this. The sqoop code should run without modification, so we're good downstream. Thank you!

obligatory reference: https://www.mediawiki.org/wiki/Extension:NavigationTiming (is this roughly related?)

This is now done.

Great question, @mforns. This was mostly for performance reasons. I couldn't find a way to get Spark to optimally work on the full day of pageviews without first aggregating it like this to > 250. But the execution plan I ended up with looks pretty wild. Let's talk tomorrow when you have some time. I'm attaching the change here.

I've migrated and shut off the old instances. I will delete them in a couple of days, just in case. But everything's working fine without them. Did not know about the wmflabs -> wmcloud automatic redirect, that made everything very simple.

I grouped a couple of tasks under this so we're less likely to lose them in the fray.

The simpler way to do this, just two phases as opposed to progressive, gets us fairly similar results, with about 200 fewer rows which are all detailing specific browser versions.

We get a ton more detailed results this way, and the total coverage increases to 99.7%. Still not 99.9%, but I think we may have too much detail at some point. I'm fairly happy with these results, and I'm going to prepare the new browser general query as a gerrit change. It'll be good to get some review.

This might affect some data we sqoop into HDFS and some of how we compute commons impact metrics or similar future metrics. We have to wait until a schema change is proposed to know for sure.

From a discussion with @Krinkle about the data, a preliminary idea of how to roll up is: