Change 637895 had a related patch set uploaded (by Hoo man; owner: Hoo man):
[operations/puppet@production] Do weekly dumps of Wikidata Lexeme

https://gerrit.wikimedia.org/r/637895

I still need to test my changes (sadly there are no automated tests for these scripts and it would be excruciatingly hard to write some for the current bash scripts). Once that is done, this is good to go from my side.

Moving this back to todo because we need to get this unstuck.

Change 637895 merged by ArielGlenn:
[operations/puppet@production] Do weekly dumps of Wikidata Lexeme

https://gerrit.wikimedia.org/r/637895

Dump generation is currently scheduled for Wednesday (3:15 UTC, just like the non-lexeme JSON dumps are on Monday 3:15 UTC).

These ran and are available at https://dumps.wikimedia.org/other/wikibase/wikidatawiki/20210122/

How do they look?

Are you sure they ran? That directory only contains RDF dumps as far as I can tell (Turtle and NTriples), we’ve been generating those for a while (compare 20210122 with 20201218). I haven’t found any lexeme JSON dumps yet.

In T264883#6786811, @Lucas_Werkmeister_WMDE wrote:

Are you sure they ran? That directory only contains RDF dumps as far as I can tell (Turtle and NTriples), we’ve been generating those for a while (compare 20210122 with 20201218). I haven’t found any lexeme JSON dumps yet.

Ah crap. Yeah I see that now.

I didn't get any failure emails about it, but when I looked in the log I saw this:
root@snapshot1008:~# more /var/log/wikidatadump/dumpwikidatajson-wikidata-20210127-lexemes-main.log
File size for shard 0 is only 26086402. Aborting.

I guess those values need to be adjusted for lexemes.

Ah, thanks. Looks like the threshold was last adjusted in June 2019.

I guess for the inaugural lexeme JSON dumps it would be best to skip the check completely? And then use the first dumps’ sizes going forward. (Or we could try to estimate the size from the existing RDF dumps?)

…or maybe use the file size that the error message already tells us as a guideline 🤦

Change 659945 had a related patch set uploaded (by Lucas Werkmeister (WMDE); owner: Lucas Werkmeister (WMDE)):
[operations/puppet@production] Update minimum expected file size for lexeme JSON dumps

https://gerrit.wikimedia.org/r/659945

Change 659945 merged by ArielGlenn:
[operations/puppet@production] Update minimum expected file size for lexeme JSON dumps

https://gerrit.wikimedia.org/r/659945

All set. We should check on these again in the middle of next week, as the run starts on Monday at ridiculous-o-clock when we are all sleeping.

This seemed to be ready for verification, as it has already ran a couple of times:
https://dumps.wikimedia.org/wikidatawiki/entities/20210203/ and https://dumps.wikimedia.org/wikidatawiki/entities/20210210/.

I am doing some prep work before I try to test this on buster. Getting close!

These look fine to me from today, and I've done all the buster-side testing so that's ok too. Closing this! Ah, do we want to anounce it anywhere though? Maybe I won't close it pending that answer. Places it could be announced: xmldatadumps-l, wikitech-l, research list, wikidata list.