Change #1081427 merged by JHathaway:

[operations/alerts@master] replace exim alert with postfix alert

https://gerrit.wikimedia.org/r/1081427

Change #1081427 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/alerts@master] replace exim alert with postfix alert

https://gerrit.wikimedia.org/r/1081427

Change #1081371 merged by Alexandros Kosiaris:

[operations/dns@master] cleanup old mx1001, mx2001 references

https://gerrit.wikimedia.org/r/1081371

Change #1081371 had a related patch set uploaded (by Alexandros Kosiaris; author: Alexandros Kosiaris):

[operations/dns@master] cleanup old mx1001, mx2001 references

https://gerrit.wikimedia.org/r/1081371

cookbooks.sre.hosts.decommission executed by jhathaway@cumin1002 for hosts: mx1001.wikimedia.org

• mx1001.wikimedia.org (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster eqiad to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Started forced sync of VMs in Ganeti cluster eqiad to Netbox

cookbooks.sre.hosts.decommission executed by jhathaway@cumin1002 for hosts: mx2001.wikimedia.org

• mx2001.wikimedia.org (PASS)
  • Downtimed host on Icinga/Alertmanager
  • Found Ganeti VM
  • VM shutdown
  • Started forced sync of VMs in Ganeti cluster codfw to Netbox
  • Removed from DebMonitor
  • Removed from Puppet master and PuppetDB
  • VM removed
  • Started forced sync of VMs in Ganeti cluster codfw to Netbox

Change #1079529 merged by jenkins-bot:

[analytics/refinery/source@master] Update the smtp server settings for email from refine

https://gerrit.wikimedia.org/r/1079529

We have migrated postfix

No, and I apologise for my tone.

@Primefac: Do you think it's better not to reply at all, instead of adding a quick comment explaining how to realistically see progress? People are busy. People need to prioritize things. There are thousands of open tickets. Please just do it if you have the skills and time to help out instead of criticizing people that they provided you a pointer. See https://www.mediawiki.org/wiki/Gerrit/Tutorial and https://developer.wikimedia.org/ . Thanks.

If people knew how to make patches, they would be creating pull requests after they had finished, not asking for patches to be written (ostensibly because they don't know how). Don't be an arse about it if you have no intention of helping.

Feel free to make patches.

@Ladsgroup Can you unbreak this at least for community-managed MediaWiki instances? I.e. provide a setting for the backend administrators to opt-in for sending emails on bot edits

In T358087#9727479, @Qwerfjkl wrote:

For what it's worth I've made https://watchlistemail.toolforge.org/ to send emails for bot edits. (It will only work on enwiki, ping me if anyone wants it working on another wiki).

Change #1079529 had a related patch set uploaded (by Btullis; author: Btullis):

[analytics/refinery/source@master] Update the smtp server settings for email from refine

https://gerrit.wikimedia.org/r/1079529

I would like it to automatically filter out the emails that are not active or are returning blank/bouncing back.

Fixed with change in the config, also no longer relative, as we are now running Postfix

retry logic has been added which should resolve the issue, please reopen if it occurs again.

Change #1070671 merged by JHathaway:

[operations/puppet@production] vrts_aliases: add retry logic

https://gerrit.wikimedia.org/r/1070671

Thanks, everone. I think @MatthewVernon 's suggestion is fair, and something I should have done. I will update the code's headers so bounces get sent to root@. While I know mail is not reliable, I just found weird that the same kind of message (as it is automated) got filtered only that one time.

Change #1070671 had a related patch set uploaded (by JHathaway; author: JHathaway):

[operations/puppet@production] vrts_aliases: add retry logic

https://gerrit.wikimedia.org/r/1070671

So looks like we're back to the original problem only.

The patch was needed, the last error was at Sep 03 17:08:32. After that it run smoothly except for one run at Sep 03 20:10:57 that failed with ERROR:/usr/local/bin/vrts_aliases:Connection unexpectedly closed.

Looks like the last alert was at 2:42 UTC so the patch likely wasn't needed after all.

In T368257#10112246, @Volans wrote:

@LSobanski the current failures are because there are 2 email addresses in the config that are managed by gsuite (I've redacted some part):

`

vrts_aliases[1485203]: ERROR:/usr/local/bin/vrts_aliases:email is handled by gsuite: **[email protected]
vrts_aliases[1485203]: ERROR:/usr/local/bin/vrts_aliases:email is handled by gsuite: **[email protected]

Change #1070222 merged by Dzahn:

[operations/puppet@production] Filter out addresses that cannot be removed from VRTS

https://gerrit.wikimedia.org/r/1070222

Change #1070222 had a related patch set uploaded (by LSobanski; author: LSobanski):

[operations/puppet@production] Filter out addresses that cannot be removed from VRTS

https://gerrit.wikimedia.org/r/1070222