Seems to be similar to the issue I had in T382715: Error using FileImporter and undelete file on Commons because of "local-multiwrite/local-public...is in an inconsistent state within the internal storage backends"

Awesome, thank you guys.

This should be fixed now that ms-be2075 is taken out of the ring. Thanks to @MatthewVernon for doing all the heavy lifting.

In T382705#10422179, @BCornwall wrote:

(it's been quite a day for swift!)

@TheDJ That was a result of a separate issue that is now resolved (it's been quite a day for swift!)

Mentioned in SAL (#wikimedia-operations) [2024-12-23T20:16:56Z] <Emperor> weighted ms-be2075 to zero T382705 T382707

Mentioned in SAL (#wikimedia-operations) [2024-12-23T20:16:56Z] <Emperor> weighted ms-be2075 to zero T382705 T382707

Ehm. it this a problem ? or a side effect of the depool taking effect after that 20:15 window ?

Great, thanks, I'll close this ticket now :)

Ooops. Sorry. I undeleted the file and it worked fine.

@Sreejithk2000 Could you try undeleting the file again now?

Done, in both clusters.

Mentioned in SAL (#wikimedia-operations) [2024-12-23T17:22:29Z] <Emperor> swift delete wikipedia-commons-local-public.88 8/88/Model_4000-First_of_Odakyu_Electric_Railway_2.JPG T382694

if the image exists in deleted container, I agree, just deleting the file from the public container is the right thing to do. In fact, I think we should actively look for images in the deleted container that are in public too as these images (for legal reasons such as copyright) are not supposed to accessible publicly.

(with the caveat that I'm not super familiar with this)

The depool won't entirely help (writes always go to both clusters), but diverting read traffic to eqiad swift should help mitigate user impact a bit. We should restore it before US staff stop work at the end of today, though.

Mentioned in SAL (#wikimedia-operations) [2024-12-23T14:10:02Z] <Emperor> depool codfw swift T382705

ms-be2075 will be effectively removed from the ring (weights set to 0), but a small snag: Swift rings have an enforced minimum time between changes for data integrity reasons and the next availability for application will be at 20:15 UTC. Unfortunately, we'll need to wait.

Change #1106303 merged by BCornwall:

[operations/puppet@production] Swift: Mark ms-be2075 as failed, remove from prod

https://gerrit.wikimedia.org/r/1106303

ms-be2075 has a data link reset a few times a minute:

oh, wrong link, and wrong screenshot, I copied from the wrong browser tab :D

Change #1106303 had a related patch set uploaded (by BCornwall; author: BCornwall):

[operations/puppet@production] Swift: Remove ms-be2075 from prod hosts

https://gerrit.wikimedia.org/r/1106303

Thank you for reporting @Yann. I created T382705 for this one.

For any sort of maintenance, we either have to reset the mime type of all svgs, or preferably, we need to list files on swift by a header property and only reset the text/plain files. I don't think we have something for that last option I MediaWiki however, and I'm not sure swift itself even allows that ?

This happened again while trying to upload a new version of https://commons.wikimedia.org/wiki/File:The_Lion_of_the_Moguls_-_Le_Lion_des_Mogols_(1924)_dir._Jean_Epstein.webm

04369: finalize/189> Still waiting for server to publish uploaded file
04374: FAILED: stashfailed: An unknown error occurred in storage backend "local-swift-codfw".

Both swift clusters do indeed have an object there - in eqiad last modified 2021-06-30, in codfw last modified 2021-01-05, and both objects are the same image.

My vague, ancient, memory is SVG files without an XML processing instruction used to be tagged as text/plain.

I vaguely remember that this happened for invalid svgs when MediaWiki did not yet supply the content type to swift, and instead we relied on the swift side to determine the content type at upload time..