I'll close the task as the major version upgrade is finished. If any new errors are introduced with the new version, feel free to reopen or link a subtask like the two above.

This was bot traffic: https://superset.wikimedia.org/superset/dashboard/p/l39rmY3r05p/. Alert recovered after 5 minutes. I'll take a deeper look if that happens again.

I'll close the task, the downtime was around 5 minutes. We have a task to improve the downtimes in the upgrade cookbook: T363564

All instances (except the WMCS Bookworm Runner) are now on GitLab version 17. I'll leave the task open over the weekend in case there are any errors to report.

I deleted packager02.packaging.eqiad1.wikimedia.cloud, which was the last buster instance. So I'll resolve the task. All buster instances are replaced or deleted.

I created the bookworm host packager-etherpad01.packaging.eqiad1.wikimedia.cloud to replace packager02.packaging.eqiad1.wikimedia.cloud. I successfully rebuilt Etherpad version 1.9.7 (the current production version) on the new bookworm host.

As discussed in IRC, the release VMs should probably have a separate disk mounted at /srv (similar to the 150GB disk mounted at /srv/docker). However, I'm not sure if /srv/docker needs all that space. The Docker partition is using only 5%, with no significant change over the past month. Therefore, we could use this larger disk at /srv and mount a smaller one for Docker, or just use the 150GB disk for /srv (including /srv/docker).

I'll resolve this task optimistically because this was likely caused by a bigger database incident: T368098.

The custom nginx rule to block non-admin configuration changes using the web UI is in place now. So I'm resolving the task.

gitlab1003 was upgraded successfully to 17.0.2: https://gitlab-replica-b.wikimedia.org. I'll proceed with the other replica on Monday.

I will upgrade gitlab1003 to version 17.0 later today. If everything goes as planned, I will update the other replica on Monday and the production system during the deployment window on Friday, June 28th.

I think all of the options above can be used. I updated the frequency of the scheduled pipeline from every 30 minutes to every 15 minutes. Also I added a [DO NOT EDIT] description to the Trusted Runners.

Hello! We are transitioning from the legacy Miscweb setup to Kubernetes. Most services have already been migrated (see main tracking task T300171). However, the migration for the query service is still pending (T350793).

As discussed in T367544, the VM used for building the Etherpad Debian package will be deleted soon. Therefore, it might be sensible to attempt one last version upgrade. This would give us additional time to evaluate new systems for building and deploying Etherpad (like Kubernetes, wmf-debci or another bookworm build vm).

It might be possible to schedule another Etherpad upgrade (T362432) before the packager02.packaging.eqiad1.wikimedia.cloud host is deleted.

According to the etherpad upgrade docs this host is used to build the etherpad Debian package. I also used the host in the past to build the etherpad package. The dedicated host is used because "etherpad builds fetches npm modules during the build time".

I updated the test instance and test gitlab-runners to GitLab version 17. Everything looks good and the migration was successful. @brennen we can test the account approval bot on the test instance now.

I'm going to upgrade the test instance to 17.0 next Monday June 17th.

This should be fixed with the change above. We ignore user-generated script_failures in the alert now.

Related to ongoing upgrade in T367382. I silenced the alert. This should resolve later today.

This was triggered by failing jobs due to faulty CI jobs/scripts. Jobs came mostly from https://gitlab.wikimedia.org/repos/security/ci-cd-testing-gitlab-ci-security-templates. So this was user generated.

This is resolved after GitLab is back. I'll try to catch the error in a new version of the exporter.

This is resolved after GitLab is back. I'll try to catch the error in a new version of the exporter.

^ another patch was needed to make sure to probe the service IPs and not the default host IPs (similar to the blackbox::http check).

resolved after fixing the IPs which should be probed. https://gerrit.wikimedia.org/r/c/operations/puppet/+/1042191

This seemed to be a puppet dependency issue and not a SSH issue. I checked the sshd on all instances and IPv6 worked on both replicas but not on production. I restarted the git sshd and after that sshd listens on IPv6:

After digging through the docs I did not find any way to restrict the edit page for runners in the projects CI menu, for example here in airflow-dags: https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags/-/runners/1484/edit. All users with elevated permissions on the project can edit the CI settings for every runner which is assigned to the project.

The gitlab-ssh service is listening on IPv4 only. So the above ProbeDown alert fired for IPv6.

This is resolved after switching to monitor IPv4 only. In T367021 I'll discuss more how to properly add IPv6 to the gitlab-ssh service.

One end-user edited the runner within their project to solve a building/dependency issue:

the alert is right, the protected checkbox was missing. I manually protected the runner again in https://gitlab.wikimedia.org/admin/runners/1484#/.

I left a comment in https://gitlab.wikimedia.org/repos/phabricator/phabricator/-/merge_requests/52#note_86778. +1 for removing the custom rate limiting code. I'd try to move any existing extra logic to a single place (requestctl).

In T365675#9855610, @brennen wrote:

We could probably bump https://gitlab.devtools.wmcloud.org/ to v17...

I executed the cleanup command on all instances, I'll resolve the task.

In T365675#9842110, @jnuche wrote:

Can't speak for other tools, but Kokkuri uses the new id tokens instead of the old deprecated JWT variables. So at least I can say that Kokkuri is not blocking the upgrade.

For what it's worth, I've done some searching in GitLab and the only repos that I could find that are still using CI_JOB_JWT* tokens are an old fork of Kokkuri and two playgrounds.

A different issue is the deprecation of the /-/jwks endpoint, I know of at least two repos that are still using that, I've created MRs to update them. As in the case of the tokens, I'm not aware of whether we have other repos using the old endpoint:

1. https://gitlab.wikimedia.org/repos/releng/reggie/-/merge_requests/78
2. https://gitlab.wikimedia.org/repos/releng/jwt-authorizer/-/merge_requests/17

All instance updated to new postgres version.
I'd move the cleanup (mentioned in the scripts output) to somewhere next week:

This is fixed now, the exporter is retrying:

@jnuche just to be sure, you replaced the old versions of JSON web tokens (CI_JOB_JWT*) in T337474 and we should not be affected by the deprecation anymore?

This was caused by the daily backup-restore. In v1.0.10 of the exporter I added better error handling : https://gitlab.wikimedia.org/repos/sre/gitlab-exporter/-/tags/v1.0.10

I can upload a fix for that in the exporter tomorrow