Yapperbot

Down since mid-December?

Yapperbot's last message was posted on 12/18/23; it looks like it needs to be migrated off of GridEngine and has been shut down by the toolforge maintainers. Either komla needs to migrate it or someone else needs to take up maintenance. See T320195 – SJ + 01:54, 29 January 2024 (UTC)

Sj I just noticed that too. I am willing to help with maintenance and debug. I'm not at this point willing to take full responsibility for the code, but more than happy to assist anyone else.

In addition, I would like to address a problem I pointed out to the author (Naypta) of the program long ago that has never been addressed: User_talk:Yapperbot/Archive_2#Concern_about_probability_of_receiving_notifications. It's been a few years since I discussed it with the author. In the end the author didn't feel it worth fixing. I just mentioned the bug to Novem Linguae, who replied "I agree that that sounds like a bug..." here This seems like a good time to consider fixing the bug.

Right now I am reviewing all those old discussions and will see about looking at the code and possibly try to help with documentation before making any recommendation on how to fix it. --David Tornheim (talk) 22:28, 14 February 2024 (UTC)

Naypta hasn't edited in about two years. First things first, someone who is not Naypta needs to 100% fork and take over the bot's source code and get it set up on Toolforge. Are you thinking about doing that David? –Novem Linguae (talk) 22:36, 14 February 2024 (UTC)

Considering it--especially if there are no other takers. --David Tornheim (talk) 22:48, 14 February 2024 (UTC)

Moving Forward

I see there are still no takers. I have been reviewing the code here and here(I believe this is an old version). I am also aware of Wikipedia:Bots/Requests_for_approval/Yapperbot.

Regarding my background and experience: I have a strong background in programming in countless languages popular before 2000, e.g.: C (I read every bit of Kernighan and Richtie's famous book and did all the exercises), Verilog, Pascal, FORTRAN, LISP, BASIC, HPL, Prolog, and Z-80, 8085, 8086, VAX and MIPS assembly. Since then, I have learned some of PERL and Python. Lately, data problems that I might have solved with a program, I just find it faster and easier to maintain an Excel spreadsheet or Access Database. Within the last 5 years, I took a class in Database Management (and learned SQL) and one in HTML. In the old days, I found using a debugger very helpful. I haven't used a recent debugger in quite a while. Those are the pluses.

I do not have experience with GoLang, bots, or any coding on Wikipedia/WikiMedia--other than Wikitext and HTML. I have not used github other than to look at Yapperbot. So if you want me to help get this bot restarted, I have a few questions:

(1) Where is the code that *calls* the bot?

(2) How can a bot be tested without changing Wikipedia pages? I saw the mention of test verification here, but it's not clear to me how that process worked.

(3) Are there any simple bots or simple test GoLang programs you suggest I play with before diving further into the Code? I am also reviewing [1] and read over Pike's post.

--David Tornheim (talk) 22:17, 18 February 2024 (UTC)

You definitely seem to have more than enough background :) I've never run a Wikipedia bot, but here's my interpretation:

1. The bot's entry point seems to be main.go. Bot calls aren't from Wikipedia; the bot operator runs the bot's code on their machine to monitor changes on Wikipedia.
2. See Help:Creating a bot § Testing. Basically do a dry run of your bot.

Aaron Liu (talk) 22:26, 18 February 2024 (UTC)

It was actually run from Toolforge, presumably via a crontab that ran "jsub main.go", although I don't have access to Toolforge so can't be 100% sure. * Pppery * _{it has begun...} 22:53, 18 February 2024 (UTC)

Since the bot inactivity coincided with the grid engine shutdown, that does seem likely. Aaron Liu (talk) 02:16, 19 February 2024 (UTC)

Pinging @Enterprisey: who might have advice. 𝗐♥, – SJ + 01:43, 19 February 2024 (UTC)

If I were you, I would apply for a ToolForge account. ToolForge is where Wikipedia bots are often hosted and set up to automatically run.

Once approved, use an FTP client to take a peek at the Yapperbot files. Toolforge files are usually public to other Toolforge users, unless set to Linux chmod 0600 or similar. The Yapperbot files are probably located at the path "/data/project/yapperbot" or similar. These may be more up-to-date than what is in GitHub repos, and may also include some configuration files that tell you how the cronjob worked (time interval, bash code to start the bot, etc.)

To set up your own bot or your fork of Yapperbot to use Toolforge cron jobs, you'll want to SSH in to your Toolforge account and then set up a "job". This is the new way to do Toolforge cron jobs. The help file is at wikitech:Help:Toolforge/Jobs framework. One of the old ways to do Toolforge cron jobs is the "grid engine", and this also happens to be why Yapperbot was shut down. The Toolforge sysadmins want to force upgrading to the jobs framework, so they are shutting down all bots still running on the grid engine, including Yapperbot.

How can a bot be tested without changing Wikipedia pages? Use https://test.wikipedia.org/ as a big sandbox, if needed. Those edits will go live on test.wikipedia.org, but won't hurt anything. Or see if the bot has a "dry run" option of some sort by reading its program code. –Novem Linguae (talk) 02:41, 19 February 2024 (UTC)

Thank you both for the advice. Will keep you posted on progress, or you can always ask if you are concerned. --David Tornheim (talk) 06:35, 20 February 2024 (UTC)

All the src and job-related files at /data/project/yapperbot/frs are protected, so we can't access them. – DreamRimmer (talk) 16:23, 20 February 2024 (UTC)

Hello @Legoktm. I hope you're doing well! Does Toolforge have a procedure for taking over an inactive account? Maybe the Yapperbot files chmod can be set to public, or David Tornheim and myself can be added as maintainers? Thanks a lot. Looking forward to your feedback. –Novem Linguae (talk) 17:01, 20 February 2024 (UTC)

See wikitech:Help:Toolforge/Abandoned tool policy. * Pppery * _{it has begun...} 17:09, 20 February 2024 (UTC)

Could we just have the responsibility for FRS be taken over by User:Legobot? It may be a big ask, but it would help with consolidation. Maybe a general purpose RfC bot might be in order. Awesome Aasim 20:36, 20 February 2024 (UTC)

Could we just have the responsibility for FRS be taken over by User:Legobot?

Based on my experience with Yapperbot, I would prefer it, because I believe Legobot's function was superior and met the specification better than the new implementation. I am also hoping that Legobot is documented. There was no real documentation for Yapperbot that I found, despite repeated asks.

Maybe a general purpose RfC bot might be in order. What do you mean? I don't know how others feel, but I wonder if it might be better to separate the RfC notification function from the GA notification function. I can't remember if Legobot did that or not. I can see how the functioning of GA and RfC notification might be similar, but it feels like these a significantly different tasks and priorities, and I would feel more comfortable if the code is separated for clarity and to avoid having any change to the code affect both or inadvertently affect the function we were not trying to correct. Is that what you mean?

Of course, getting the bot (either Legobot or Yapperbot) up and running--as it was--will be the first task! --David Tornheim (talk) 22:11, 20 February 2024 (UTC)

Legobot's FRS task when it ran had its own bugs, i.e Wikipedia talk:Feedback request service/Archive 1#Still receiving messages. Legoktm has never been interested in doing anything to this Legobot task other than fixing the most urgent bugs and occasionally deploying urgent code, and has wanted a new maintainer for years, which makes sense since it wasn't his code to begin with - he took it over from Chris G, who took it over from Harej.

I think Legobot's implementation doesn't have any better documentation, and it depended on a SQL database written by the Rfc list maintenance task, so it will be even more of a pain to resurrect whereas Yapperbot at least has the advantage of having no external dependencies. * Pppery * _{it has begun...} 01:15, 21 February 2024 (UTC)

@David Tornheim When I say a general purpose RfC bot may be in order, I mean that having one bot, maybe named User:RfC Bot, handle the tasks of RfC and RfC's FRS. For GAN we can have a different bot for that. I think having one bot for a single task will be a lot less likely to fail, depending on implementation, than one bot for 30 different tasks. Awesome Aasim 04:20, 21 February 2024 (UTC)

Do you have any evidence for that belief? Legobot failed at this because it's original operator abandoned it and Legoktm picked it up in order for the task not to die without a clear plan for long-term maintenance, not because the bot runs 30 other tasks. Counterexample AnomieBOT * Pppery * _{it has begun...} 04:47, 21 February 2024 (UTC)

As long as the code for each task is separable, running them through the same account does not provide much benefits. Aaron Liu (talk) 14:48, 21 February 2024 (UTC)

Back again

It appears Yapperbot has sprung back to life :) ‍—‍a smart kitten_[meow] 07:35, 21 February 2024 (UTC)

Indeed! --David Tornheim (talk) 08:23, 21 February 2024 (UTC)

and the world is saved! Aaron Liu (talk) 14:47, 21 February 2024 (UTC)

I assume the toolforge folks mass re-enabled bots that have been shut down during the Grid deprecation, but I don't see this in the timeline. Wikitech:News/Toolforge Grid Engine deprecation#Timeline. Bit of a mystery, eh? –Novem Linguae (talk) 15:48, 21 February 2024 (UTC)

@Novem Linguae: After looking at the phab link above (phab:T320195), I think Legoktm might have manually migrated the bot as an emergency measure. All the best, ‍—‍a smart kitten_[meow] 15:51, 21 February 2024 (UTC)

Hi! Yes, I got it working last night (well, it started working after I thought it wasn't and went to sleep). This was just a one-time emergency measure, it really needs a permanent maintainer. I'm happy to assist with getting people set up with Toolforge, etc. @David Tornheim do you have a Toolforge account? Would be best to set up a new tool and bot account for this and then we can work on transitioning.

We should set a 1 to 2 month deadline for a new maintainer to get set up (otherwise just turn it off again). Legoktm (talk) 19:05, 21 February 2024 (UTC)

@Legoktm: Thanks for doing that and thanks for offering to help with the transition. Yes, I just set up an account under the name "davidtornheim".

Have you looked at the code for both? Yapperbot has little to no documentation. And according to what I read above, apparently Legobot did not either. I have expressed concerns that Yapperbot does not really meet the desired specification the way I believe Legobot did. One of my goals will be to correct the code to meet the specification. It sounds like one of the reasons for the end of Legobot--besides not having a maintainer--was that it relied on a database not maintained within Wikipedia. That sounds like a more serious problem. Is that your understanding of the main reason Legobot was not revived for RfC (and possibly also GA) notices? Or was it simply because the new maintainer preferred writing their own code? If there is a location that I could read more about the history of the demise of Legobot's RfC notification, I would like to jump into that. --David Tornheim (talk) 20:38, 21 February 2024 (UTC)

I don't think there's any real story to read here. In short, Legobot's FRS task died for unknown (to this day) reasons in December 2019 at a time when Legoktm was almost entirely inactive, people started complaining at Wikipedia talk:Feedback request service/Archive 1#Not getting any invitations?, and Naypta noticed the complaints and was inspired to write their own code. Naypta couldn't have reused Legobot's FRS code because it is tightly coupled with Legobot's RfC list code. They wrote at Wikipedia:Bots/Requests for approval/Yapperbot, in what now seems like wishful thinking, This has the main advantage that there is no database - all the needed information is stored on-wiki - so it would be a whole lot easier for someone else to pick up the bot in the event that I'm unable to continue to manage it and it breaks. I also suspect that, due to the compiled nature of Golang, it'd be a hell of a lot faster - although that's a suspicion, rather than any kind of actual metric. * Pppery * _{it has begun...} 23:47, 21 February 2024 (UTC)

@Pppery: Thanks for the explanation. I do remember that problem now. When you say wishful thinking, do you question the claim that it runs faster and/or that it would be harder to pick up? It may, in fact, be easier to figure out than the old Legobot's code that does the same thing. The fact that no one knows why Legobot stopped working in 2019 is interesting. I don't know if that is worth looking into or not. For now I intend to put my focus on Yapperbot, since it is now working. If there is anything that Yapperbot requires of Legobot, I would want to know that up front. At some point, I will probably try to compare both programs to see if some of the code in Legobot seemed superior. The way Yapperbot chooses who to inform does not seem right to me. I believe in needlessly does too few notifications. --David Tornheim (talk) 00:02, 22 February 2024 (UTC)

The fact that the bot would be easy to pick up again when it breaks was wishful thinking - this discussion is showing that it isn't easy. * Pppery * _{it has begun...} 00:04, 22 February 2024 (UTC)

To be fair, it seems like the difficulty was due to bot code on toolforge not being readable for whatever reason and an external factor of the grid engine shutdown. Aaron Liu (talk) 00:06, 22 February 2024 (UTC)

I agree with Aaron, I suspect that if someone compiled the Yapperbot code on their laptop and just started running it, it would've worked fine. There's actually very little in the Toolforge tool itself, just a few config files and the golang binaries. Legoktm (talk) 00:10, 22 February 2024 (UTC)

@David Tornheim: great, I just approved your Toolforge request, you should have access to the general platform now. Here's what I suggest to move forward:

• File a request to adopt the yapperbot tool as it's abandoned.
• Create a new bot account and file a BRFA to take over the Yapperbot tasks
• Once the adoption request goes through and the BRFA is approved, switch it over to use your new bot account
• This should take no more than a month and it'll meet my deadline to find a new maintainer.
• At this point you're in full control and you're welcome to switch to other code, whether it's Legobot's old stuff (I wouldn't recommend it) or something entirely new.

I would also strongly suggest finding a second person to act as a backup maintainer who also has access to the Toolforge tool. Let me know what you think of this plan. Legoktm (talk) 00:21, 22 February 2024 (UTC)

Thanks. Novem Linguae offered to act as a backup. That makes me more comfortable too. And thanks for further explanations from everyone. And yes, the plan sounds fine. --David Tornheim (talk) 00:27, 22 February 2024 (UTC)

P.S. It looks like Legobot might not be entirely dead yet. And there is discussion here: User_talk:Legobot. I will start reading up on that. Does Legobot still work *with* Yapperbot? I got that feeling from reading User_talk:Legobot#Mysterious_non-RfCs --David Tornheim (talk) 23:36, 21 February 2024 (UTC)

The current status is that Legobot is responsible for maintaining pages like Wikipedia:Requests for comment/Wikipedia policies and guidelines, whereas Yapperbot is responsible for sending messages to people's talk pages. These tasks are completely unrelated technically and share no code with each other. The two bots work together, in a way. * Pppery * _{it has begun...} 23:47, 21 February 2024 (UTC)

Got it. --David Tornheim (talk) 00:02, 22 February 2024 (UTC)

FYI. Update coming soon. I am almost done writing up a post on my progress in reviewing code, creating accounts and bots, etc. and in general preparing for handling Yapperbot functions. I'll post in another section. Would it be better, if I instead post on the talk page for User:Feedback_Request_Service_bot and ping editors there? If one of you wants to create a talk page there with the best opening parameters including archiving, please be my guest. --David Tornheim (talk) 04:58, 13 March 2024 (UTC)

Maybe throw a link to the BRFA on the page User:Feedback_Request_Service_bot, then ping Primefac asking to add a bot flag. –Novem Linguae (talk) 05:38, 13 March 2024 (UTC)

@Novem Linguae:  Done. When I post my update about my work studying Yapperbot, Legobot, creating accounts, etc., and ask for some questions, would it be better to post here--where we have been discussing the possible adoption--or on the new bot page, or on my personal talk page? I can see advantages & disadvantages to all three.

FYI. While working on this, I did try you suggestion of posting a question to Village Pump.[2]. Although, they did mostly answer my question, I found discussion here easier.) --David Tornheim (talk) 01:18, 14 March 2024 (UTC)

This section is getting a bit long. Maybe starting anew at User talk:Feedback Request Service bot and pinging key editors to the new discussion would be a good choice. –Novem Linguae (talk) 01:30, 14 March 2024 (UTC)

I posted the update here: User_talk:Feedback_Request_Service_bot#Update --David Tornheim (talk) 05:40, 15 March 2024 (UTC)

Notifying tool maintainer, Naypta

FYI. I asked about any further notification that is required before I proceed with the adoption request. Please respond there:

• User_talk:Naypta#Yapperbot_is_down,_we're_thinking_about_forking

If you feel that discussion should be here instead, feel free to move it here. --David Tornheim (talk) 09:08, 23 February 2024 (UTC)

Notification to Maintainer of possible adoption of Yapperbot

@Naypta: This is your official notification per Abandoned Tool Policy that because:

(1) Yapperbot was inactive for approximately two months from December 19, 2023 through February 20, 2023.[3]

(2) You have not been active on any Wikimedia project for over 28 days.[4] The last edit we have recorded is 5 May 2023.

That, unless you respond within 14 days, your maintenance for the tool Yapperbot may be put up for adoption by a new maintainer(s). Please let us know if you object to such an adoption and can resume maintenance of the tool. --David Tornheim (talk) 22:39, 23 February 2024 (UTC)

Feedback request service

I was wondering what was happening with RFC notifications and I noted the status of the bot running the Feedback request service has been killed. Does anyone know if there is going to be some replacement for this? TarnishedPath^talk 13:56, 21 August 2024 (UTC)

@TarnishedPath: I will look into it ASAP! --David Tornheim (talk) 02:45, 22 August 2024 (UTC)

You are correct that the Yapperbot's Feedback Request Service is no longer enabled. Last I checked a couple of months ago, it was still active and I assumed it was still running. I will work on getting it restarted and/or a new bot to do the Feedback Request Service notification tasks. Thanks for putting this notice here and drawing my attention to the problem. --David Tornheim (talk) 06:35, 22 August 2024 (UTC)

An anon had made a bogus edit that disabled the bot. This has been fixed and the kill page has now been protected by @Pppery so this should not reoccur. See phab:T361426#10086509 for more details. -- BDavis (WMF) (talk) 23:19, 22 August 2024 (UTC)

@BDavis (WMF): Thanks. I bookmarked that page too, and I put warnings on the two IPs pages for vandalism.[5][6] I will keep an eye to make sure FRS is still enabled and properly running until we finish up adoption or transferring the bot. I have been somewhat concerned that some of the internal data that Yapperbot uses is stored on Wikipedia and is vulnerable to these attacks. I don't think the author Naypta was sufficiently familiar with how Wikipedia works to understand the vulnerability. The new page protection will help, but there are some other internal data places that are still vulnerable to similar attacks by new IPs. I can give you a list of the ones I am aware of as I start spending time reviewing the code again.

Please note that the second IP messed with the kill switch to the "Uncurrenter" Yapperbot process too, but reverted the change.[7] All the kills switches should minimally be protected to either autoconfirmed users and/or more established user or possibly just to admins. --David Tornheim (talk) 10:18, 23 August 2024 (UTC)

I have semi-protected User:Yapperbot/kill/Uncurrenter. Pruner doesn't seem to have a kill switch at all. * Pppery * _{it has begun...} 14:42, 23 August 2024 (UTC)

Strange. Because it has the "Stop" button on the User:Yapperbot page. I did try to navigate to User:Yapperbot/kill/pruner and User:Yapperbot/kill/Pruner, and it does indicate that the pages don't exist. You think this was an oversight? I might look at Naypta's edit history to see when the kill pages were created and see if there is more to this.--David Tornheim (talk) 20:01, 23 August 2024 (UTC)

Special:PrefixIndex/User:Yapperbot/ might help you find some of the kill pages. For example, the FRS one is User:Yapperbot/kill/FRS. –Novem Linguae (talk) 20:18, 23 August 2024 (UTC)

Novem Linguae,BDavis (WMF),Pppery: The FRS function of Yapperbot is still not working. Take a look at: User:Yapperbot/FRS/SentCount.json. When Yapperbot/FRS is running smoothly, it gets updated every day. It hasn't changed since 6/13/24. I tried logging in to the Toolforge shell to look at the files and look for errors, but I can't get in. Can someone help me get in? User_talk:Novem_Linguae#Need_help (permalink). --David Tornheim (talk) 04:14, 27 August 2024 (UTC)

FRS is working again! Sohom_Datta fixed the most recent problem. --David Tornheim (talk) 17:40, 29 August 2024 (UTC)

Also, JSON pages like User:Yapperbot/FRS/SentCount.json and User:Yapperbot/FRS/RFCsdone.json are implicitly protected so only the bot and admins can edit them, so you don't need to worry about any security issues there. * Pppery * _{it has begun...} 14:43, 23 August 2024 (UTC)

Pppery That's good. I believe I saw one one or more of those JSON pages saying, "Do not edit this page". I thought that meant editors were capable of it, and he hoped the warning was sufficient to deter them. Glad that is not the case. I'll still keep an eye out for any other similar vulnerabilities as re-familiarize myself with the code. I haven't looked at the code for "kill" recently, so I might look at that first. If there is anything you would like me to focus on or investigate, please let me know. --David Tornheim (talk) 20:01, 23 August 2024 (UTC)

Bug report when the bot removes a Current template

I don't know anything about this bot nor the seemingly inactive operator, but I just want to let anyone who cares know that when the bot removed a {{Current}} template at Queensland Government, it broke the adjacent table. See Special:Diff/1253732866 and my edit to fix it at Special:Diff/1253849470. Fork99 (talk) 06:27, 28 October 2024 (UTC)