Moved public_html.old to public_html.old.20240927 to unblock this for now, however manually moving the directory is not a sustainable solution.

This may be related to T375279 somehow: it started around the same moment (17 Sept at 15:30 UTC) and also seems to have an odd issue with file ownership. Specifically, the out file for this job (and several others) seems to be owned by root:

Some more thoughts:

I'm more than happy to give access to anyone who wants to pick this up. It's not immediately clear to me whether it's possible to add CCs through the "gerrit" command line (https://gerrit-review.googlesource.com/Documentation/cmd-index.html#user_commands); adding an extra flag to the template isn't too complex.

I have no objections to relicensing code that I have contributed under GPL-3.

D'oh! Yeah, looks like I completely overlooked that fairly essential step 😅

My assumption at the time was that the API would not just provide the current tag but also a list of aliases, in which case the current matching logic could be applied to the set of tags _and_ their aliases.

I think indeed the current system was primarily a "well this seems good enough" solution rather than taking a principled approach.

The original design choice was to start from "now" to prevent the situation where the bot is down for one reason or another, and then starts to catch up on several days of backlog.

In theory we should be able to have the best of both worlds using https://docs.gitlab.com/ee/ci/pipelines/merge_request_pipelines.html#run-pipelines-in-the-parent-project but I haven't really looked at it in detail.

Sounds like a good way to bring wikibugs up to date with what has been learned in the software engineering world over the last 10 years :-)

From what I can see, this role has been taken over by Crystal-bot so I would suggest to just remove the code althogether.

I don't 😅. My guess would be that it has something to do with incorrect IRC formatting being ignored, but I can't find an obvious "except Exception: ..." in the 2015-era source code. So closing, if it had been important it wouldn't have stayed a 1-line bug for 10 years.

In T357729#9552163, @bd808 wrote:

Until @valhallasw gets annoyed by me and kicks me out anyway. ;)

I, for one, welcome our new unicorn overlords.

Is anything visible from other IRC clients? i.e. do they see a connection reset by peer, a forced close due to X, ...?

I also think this may have been broken for 6 years now without too much notice except when folks are staring at the bot's logs for some other reason.

The corollary of that is that maybe linking to a specific comment/action isn't that relevant to begin with - so just removing the functionality might be a sensible option as well :-)

This could now presumably be built on top of https://wikitech.wikimedia.org/wiki/GitLab/Phabricator_integration (either using Redis as intermediary between the two repos, or by moving Wikibugs under the gitlab-webhooks repo).

In T289712#9489602, @dancy wrote:

@valhallasw Can you explain this section of ReviewerFactory._reviewer_generator() in the Gerrit reviewer bot code:
https://github.com/valhallasw/gerrit-reviewer-bot/blob/dcb31716af142494716a35a5fad239add8e368af/add_reviewer.py#L61

elif part.name == "every":
    modulo = self._tryParseInt(part.value, 1)
    if modulo < 2:
        modulo = 1

I wasn't able to find any documentation about an every template parameter. It appears that the intention is to add matching reviewers somewhat probabilistically. I notice in ReviewerFactory._filter_reviewers() that variable i is never advanced past 0 which appears to be a bug.

Tested with https://gerrit.wikimedia.org/r/c/labs/tools/wikibugs2/+/977293, seems to work as expected:

Took an initial stab at adding the scaffolding for changing the behavior in the WIP patchset above. Implementation still to be done (and open to pick up as I might not get to it).

Did an online test with Jon and we were able to reproduce and log the issue. See attached (anonymized)

Having logged a WIP change, I'm actually a bit confused by the behaviour you're describing. From what I can see, there _is_ now a new event, but that event shouldn't be logged as it's not one of the supported events:

The last time I looked at it, the json provided by stream-events looked like this:

Hacked a timeout -k 3700 3600 into forrestbot.sh which (assuming this is an issue in the python script/pop handling/etc and not in the toolforge jobs framework) would kill the bot if not done in an hour.

Merlijn van Deen - alternative — valhallasw-alt
from LDAP

Typical error log entry:

{'id': 'operations%2Fpuppet~production~I5017ee944346ac8bb797b4ef862156418f6a3eb1', 'project': 'operations/puppet', 'branch': 'production', 'attention_set': {'9922': {'account': {'_account_id': 9922, 'name': 'Hokwelum', 'email': '[email protected]', 'username': 'hokwelum'}, 'last_update': '2023-07-17 22:51:15.000000000', 'reason': 'Reviewer was added'}, '10626': {'account': {'_account_id': 10626, 'name': 'Jennifer Ebe', 'email': '[email protected]', 'username': 'jebe'}, 'last_update': '2023-07-17 22:51:15.000000000', 'reason': 'Reviewer was added'}, '9379': {'account': {'_account_id': 9379, 'name': 'Btullis', 'email': '[email protected]', 'username': 'btullis'}, 'last_update': '2023-07-17 17:52:43.000000000', 'reason': '<GERRIT_ACCOUNT_6> replied on the change', 'reason_account': {'_account_id': 6, 'name': 'ArielGlenn', 'email': '[email protected]', 'username': 'ariel'}}}, 'hashtags': [], 'change_id': 'I5017ee944346ac8bb797b4ef862156418f6a3eb1', 'subject': 'make sure certain systemd jobs run only on the primary xml dumps NFS shares', 'status': 'NEW', 'created': '2023-07-17 10:11:43.000000000', 'updated': '2023-07-17 22:51:15.000000000', 'submit_type': 'REBASE_IF_NECESSARY', 'mergeable': True, 'insertions': 35, 'deletions': 3, 'total_comment_count': 3, 'unresolved_comment_count': 1, 'has_review_started': True, 'meta_rev_id': 'd740927c470fc97cec7a9889f2fb9540bbd882c5', '_number': 938816, 'owner': {'_account_id': 6, 'name': 'ArielGlenn', 'email': '[email protected]', 'username': 'ariel'}, 'current_revision': 'a6c30abf227160608c960199bb99b273bc6ff146', 'revisions': {'a6c30abf227160608c960199bb99b273bc6ff146': {'kind': 'REWORK', '_number': 6, 'created': '2023-07-17 18:30:11.000000000', 'uploader': {'_account_id': 6, 'name': 'ArielGlenn', 'email': '[email protected]', 'username': 'ariel'}, 'ref': 'refs/changes/16/938816/6', 'fetch': {'anonymous http': {'url': 'https://gerrit.wikimedia.org/r/operations/puppet', 'ref': 'refs/changes/16/938816/6'}}, 'files': {'modules/role/manifests/dumps/generation/server/xmlfallback.pp': {'lines_inserted': 2, 'size_delta': 129, 'size': 791}, 'modules/profile/manifests/dumps/generation/server/exceptionchecker.pp': {'lines_inserted': 2, 'size_delta': 111, 'size': 434}, 'modules/role/manifests/dumps/generation/server/misccrons.pp': {'lines_inserted': 2, 'size_delta': 129, 'size': 669}, 'hieradata/common/profile/dumps/generation/server.yaml': {'status': 'A', 'lines_inserted': 8, 'size_delta': 314, 'size': 314}, 'hieradata/common/profile/dumps/generation/server/xmldumps.yaml': {'status': 'A', 'lines_inserted': 4, 'size_delta': 220, 'size': 220}, 'hieradata/common/profile/dumps.yaml': {'lines_inserted': 2, 'size_delta': 102, 'size': 1761}, 'hieradata/hosts/dumpsdata1006.yaml': {'lines_deleted': 1, 'size_delta': -65, 'size': 591}, 'modules/dumps/manifests/generation/server/exceptionchecker.pp': {'lines_inserted': 3, 'lines_deleted': 1, 'size_delta': 67, 'size': 1160}, 'modules/role/manifests/dumps/generation/server/spare.pp': {'lines_inserted': 3, 'size_delta': 167, 'size': 684}, 'hieradata/common/profile/dumps/generation/server/alldumps.yaml': {'status': 'A', 'lines_inserted': 4, 'size_delta': 220, 'size': 220}, 'modules/profile/manifests/dumps/generation/server/jobswatcher.pp': {'lines_inserted': 2, 'size_delta': 106, 'size': 534}, 'modules/dumps/manifests/generation/server/jobswatcher.pp': {'lines_inserted': 3, 'lines_deleted': 1, 'size_delta': 67, 'size': 1022}}}}, 'requirements': [], 'submit_records': [{'rule_name': 'gerrit~DefaultSubmitRule', 'status': 'NOT_READY', 'labels': [{'label': 'Verified', 'status': 'OK', 'applied_by': {'_account_id': 75, 'name': 'jenkins-bot', 'username': 'jenkins-bot', 'tags': ['SERVICE_USER']}}, {'label': 'Code-Review', 'status': 'NEED'}]}]}
 caused exception:Traceback (most recent call last):
  File "/data/project/gerrit-reviewer-bot/src/gerrit-reviewer-bot/pop3bot.py", line 134, in <module>
    main()
  File "/data/project/gerrit-reviewer-bot/src/gerrit-reviewer-bot/pop3bot.py", line 122, in main
    add_reviewers(changeset['id'], reviewers)
  File "/data/project/gerrit-reviewer-bot/src/gerrit-reviewer-bot/add_reviewer.py", line 150, in add_reviewers
    raise Exception(command + ' was not executed successfully (code %i)' % retval)
Exception: gerrit set-reviewers --add ' Xcollazo ' --add jebe --add Hokwelum 'operations%2Fpuppet~production~I5017ee944346ac8bb797b4ef862156418f6a3eb1' was not executed successfully (code 1)

Wikibugs uses the task page html for

See T288381 for the related discussion from two years ago on Wikibugs. Long story short: No, not really, unless you pay for GitLab EE (which I think we don't).

@Jdforrester-WMF note that due to the bot hanging for so long, there will be quite a few tasks tagged with a newer release (anything that went to master will be marked as MW-1.41-notes (1.41.0-wmf.9; 2023-05-15) rather than the applicable wmf release). Hopefully not a huge deal to sort out by hand (and maybe it doesn't matter too much to fold them into one set of release notes for wmf5-9?)

tools.forrestbot@tools-sgebastion-10:~$ toolforge-jobs list
Job name:         Job type:            Status:
----------------  -------------------  --------------------
releasetaggerbot  schedule: 0 * * * *  Running for 28d1h44m

so it seems that the job is hanging in some sort of error loop.

For home directories who's capacity goes above 1 gigabyte, the largest files will be removed to bring the total usage below 1 gigabyte.

Hiya, the reviewer bot is in very low maintenance mode a a) Gerrit has much more powerful features than when the bot was originally written, and b) we're moving over to Gitlab at... some point?

Not entirely sure what's going on. The bot is running but there do not seem to be any emails coming in:

Thanks! It seems that it's an issue with some of the k8s hosts, but it happens relatively rarely. Combined with the overall migration to Gitlab (which will likely require a complete rethink of the bot) it's not a high priority to fix for me (but happy to give access to anyone who is interested to dig in!)

Cronjob disabled (one-off project which already finished a long time ago, but had forgotten to turn off the cronjob before...)

I think someone just restarted everything. Looking at the logs, there doesn't seem to be anything wrong with the Phab integration, rather the IRC bot had trouble connecting.

@Marostegui which command(s) did you run, exactly?

And it works if I first kill the webservice. Probably a quota issue then? A limit of 3 continuous jobs seemed pretty limited when I read it the first time, and when the webservice also counts against it...

I get the same exception if I try to run the job by hand:

tools.wikibugs@tools-sgebastion-10:~$ toolforge-jobs run redis2irc --command "libera/wrapper.sh redis2irc" --image tf-python39 --continuous --emails all
ERROR: unable to create job: "HTTP 403: likely an internal bug: 403 Client Error: Forbidden for url: https://k8s.tools.eqiad1.wikimedia.cloud:6443/apis/apps/v1/namespaces/tool-wikibugs/deployments. k8s JSON: {\"kind\": \"Deployment\", \"apiVersion\": \"apps/v1\", \"metadata\": {\"name\": \"redis2irc\", \"namespace\": \"tool-wikibugs\", \"labels\": {\"toolforge\": \"tool\", \"app.kubernetes.io/version\": \"1\", \"app.kubernetes.io/managed-by\": \"toolforge-jobs-framework\", \"app.kubernetes.io/created-by\": \"wikibugs\", \"app.kubernetes.io/component\": \"deployments\", \"app.kubernetes.io/name\": \"redis2irc\", \"jobs.toolforge.org/filelog\": \"yes\", \"jobs.toolforge.org/emails\": \"all\"}}, \"spec\": {\"template\": {\"metadata\": {\"labels\": {\"toolforge\": \"tool\", \"app.kubernetes.io/version\": \"1\", \"app.kubernetes.io/managed-by\": \"toolforge-jobs-framework\", \"app.kubernetes.io/created-by\": \"wikibugs\", \"app.kubernetes.io/component\": \"deployments\", \"app.kubernetes.io/name\": \"redis2irc\", \"jobs.toolforge.org/filelog\": \"yes\", \"jobs.toolforge.org/emails\": \"all\"}}, \"spec\": {\"restartPolicy\": \"Always\", \"containers\": [{\"name\": \"redis2irc\", \"image\": \"docker-registry.tools.wmflabs.org/toolforge-python39-sssd-base:latest\", \"workingDir\": \"/data/project/wikibugs\", \"command\": [\"/bin/sh\", \"-c\", \"--\", \"libera/wrapper.sh redis2irc 1>>redis2irc.out 2>>redis2irc.err\"], \"resources\": {}, \"env\": [{\"name\": \"HOME\", \"value\": \"/data/project/wikibugs\"}], \"volumeMounts\": [{\"mountPath\": \"/data/project\", \"name\": \"home\"}]}], \"volumes\": [{\"name\": \"home\", \"hostPath\": {\"path\": \"/data/project\", \"type\": \"Directory\"}}]}}, \"replicas\": 1, \"selector\": {\"matchLabels\": {\"toolforge\": \"tool\", \"app.kubernetes.io/version\": \"1\", \"app.kubernetes.io/managed-by\": \"toolforge-jobs-framework\", \"app.kubernetes.io/created-by\": \"wikibugs\", \"app.kubernetes.io/component\": \"deployments\", \"app.kubernetes.io/name\": \"redis2irc\", \"jobs.toolforge.org/filelog\": \"yes\", \"jobs.toolforge.org/emails\": \"all\"}}}}"

fdsfadsdfsa

bump²

fdsa

Test

See https://www.mediawiki.org/wiki/Wikibugs#Configuring_channels for how to do this.
An admin has to autovoice Wikibugs in the channel before the channel can be added to the configuration.

From the log file:

The main bit of code seems to be in https://github.com/pywikibot/Pywikibot-nightly-creator

I think I know what's going on -- it's an issue I fixed earlier for ReleaseTaggerBot (which uses the same email based system), but forgot to also apply here: T284587: https://github.com/wikimedia/labs-tools-forrestbot/commit/76a76db66a2fa6a54493b402bd9c68376f0d70e0

https://gerrit-reviewer-bot.toolforge.org/ contains the last 50-or-so log lines; if you have access to tool labs I'm fairly certain you can just read the .out and .err logs in the home dir. The logging is not super extensive but should give a decent idea of what's happening.

In T288381#7315659, @Legoktm wrote:

:S

Do you have a patch or an idea how much changes are needed for adding the integration? That does seem like the best way forward. cc'ing @brennen since I'm not really sure what our capacity for local patches is .

by "building" do you mean patching the GitLab code?

@Legoktm mentioned Debian's gitlab instance (Salsa) has some irc integration as well. Seems to be the built-in Irker plus something based on webhooks & https://salsa.debian.org/kgb-team/kgb/-/blob/master/script/kgb-bot. This is something we could also deploy in principle (with wikibugs just handling phab, and kgb handling merge requests etc).

With a little bit of hacking, I was able to make the Mattermost integration act like a regular webhook:

A few further notes after fiddling with a fresh Gitlab instance.

Some thoughts on architecture when using a webhook

https://gitlab.wikimedia.org/api/v4/events?target_type=merge_request seems to be pretty much what we need, except it can only filter by after={date} and not by date and time... so it's not particularly convenient for polling. We may have to go for the inverse route (either emails or webhooks).

Oooh, yes, that's going to be fun. A few thoughts.

Would be happy to be on the front lines with Wikibugs2, as it needs to be adapted for GitLab (T288381). Eat your own dog food, etc. (cc @Legoktm)

Note that I may have missed changes before 6 jun (which is the oldest log file).

Decided to use a script after realizing the number of clicks required for the manual approach. Should all be sorted now :-)

List seems to be the following:

Deployed and re-enabled bot. Looking into the logs now.

For (b): found a source email with the same issue -- the email has mixed CRLF and LF newlines which explains why things got confused. Created test + fix (see above). Will deploy this first, and will then investigate what I can find on other situations where the \r may have caused issues.

Hmm, that doesn't look good:

forrestbot.log.2021-06-05:2021-06-05 15:00:13,036 - forrestbot - DEBUG - <https://gerrit.wikimedia.org/r/c/mediawiki/core/+/698040>: skipping (SkipMailException('Project mediawiki/core\r is not being watched',))

It's always great when you fix a bug twice, merge once, then come back yet another year later to merge the first one.

Ooof. That was a long time ago :-)

The main issue seems solved when replacing connect_ by connect; however the tests then still fail, most likely because the font settings etc. in Windows 10 have changed significantly over where they were in Windows 7. Given that the code is largely 'battle-tested' and the tests cannot be run automatically indeed removing the tests seems like a reasonable approach.

I'm not entirely convinced this should be solved by Wikibugs -- isn't this better solved at the client side by ignoring bots when it comes to alerts?