From https://www.mediawiki.org/wiki/Growth/Personalized_first_day/Newcomer_tasks/Experiment_analysis,_November_2020#Detailed_findings

Constructive activation: the effect is larger when looking only at constructive activation. Newcomers with Growth features are 26.7% more likely to make a first unreverted article edit. On our four pilot wikis, the baseline constructive activation rate is 16.1%. The Growth features are estimated to increase this to 20.4%, which is a 26.7% increase over the baseline.

I wonder how much things have changed in the past 4 years!

In T360829#9655634, @mpopov wrote:

From https://www.mediawiki.org/wiki/Growth/Personalized_first_day/Newcomer_tasks/Experiment_analysis,_November_2020#Detailed_findings

Constructive activation: the effect is larger when looking only at constructive activation. Newcomers with Growth features are 26.7% more likely to make a first unreverted article edit. On our four pilot wikis, the baseline constructive activation rate is 16.1%. The Growth features are estimated to increase this to 20.4%, which is a 26.7% increase over the baseline.

I wonder how much things have changed in the past 4 years!

Wow! What a great reference point...thank you for finding and sharing, @mpopov.

One other decision that needs to be made before we decide on the baseline:
How do we define "constructive activation"?

Growth has a definition that we have used across several previous experiments:
https://www.mediawiki.org/wiki/Growth/Personalized_first_day/Structured_tasks/Add_a_link/Experiment_analysis,_December_2021#Glossary

Will we use that constructive activation definition, or do we want to consider different time frames?

@nettrom_WMF will be developing it as an essential metric in the next FY for SDS 2.2 and will likely use that definition.

Morten, can you please confirm if you're probably going to use that definition or if there's a good chance you change the time frame based on your experience over the past several years? (No need to commit to anything here, just a simple vibes check.)

I also want to note that if the exact definition of the constructive activation changes in the course of developing it as an essential metric, we can always re-calibrate the KR based on the finalized definition.

In T360829#9662018, @mpopov wrote:

Morten, can you please confirm if you're probably going to use that definition or if there's a good chance you change the time frame based on your experience over the past several years? (No need to commit to anything here, just a simple vibes check.)

I think it's very likely that the definition stays the same. While our analysis where we defined it was only based on two wikis (Czech and Korean Wikipedia), the results were consistent between them. There are also three more recent analyses I can think of that points to activation (or "newcomer actions") happening quickly:

1. The Mentorship preliminary analysis shows that when newcomers ask their mentor a question the timeframe is "hours".
2. When we looked at newcomers who reach newcomer task milestones (e.g. 5 or 10 edits), we were again look at "hours" for many of them.
3. For one of the Thank You page/banner campaigns, we used the same 24-hour default but were asked to extend it. I tried 7 and 30 days, but saw only minor increases in the activation rates.

The latter analysis correlates well with what we saw in the original analysis, where 85.7% of those who edited within a year on Czech WP would do so on the first day. Extending it to a week only increased it to 90.7%. Adding additional weeks only adds small percentages. If I remember correctly the first-day percentage was slightly higher on Korean WP, but the one-week diff was still about +5pp.

The areas where there's uncertainty is that we used all edits rather than non-reverted article edits, and that there might be a subset of users as defined by the wiki, platform of registration, or some other characteristic, that behaves differently. I'm looking forward to doing some data digging next FY to see if we can discover that, or perhaps find that things are surprisingly stable.

In T360829#9659170, @KStoller-WMF wrote:

One other decision that needs to be made before we decide on the baseline:
How do we define "constructive activation"?

Growth has a definition that we have used across several previous experiments:
https://www.mediawiki.org/wiki/Growth/Personalized_first_day/Structured_tasks/Add_a_link/Experiment_analysis,_December_2021#Glossary

Will we use that constructive activation definition, or do we want to consider different time frames?

I propose we move forward with what I understand @KStoller-WMF to be proposing here...

Defining constructive activation in the same way the Growth Team has historically defined it with a coupleof edits (bolded): "a newcomer making their first edit to an article in the main namespace on a mobile device within 24 hours of registration, and that edit not being reverted within 48 hours of being published."

To put the above into practice, I'm going to:

1. Update the task description to include the definition above
2. Ensure all WE 1.2 hypothesis owners whose work is meant to directly affect constructive activation are onboard with this definition and set up to evaluate it
3. Once "2)" is done, I'll include this definition in the DRAFT APP FY24-25: OKRs sheet and talk with @CBogen about having this definition be incorporated into the on-wiki documentation.

Meta: great spot naming the importance of us all aligning on how we'll define "constructive activation" in this context, @KStoller-WMF. Also, thank you @mpopov and @nettrom_WMF for offering the context needed to do so.

@ppelberg
I've completed an initial analysis of the baseline for constructive activation per the definition documented in DRAFT APP FY24-25: OKRs sheet and defined below.

See summary of findings below and let me know if you have any questions.

Constructive activation definition [1] : A newcomer making at least one edit to an article in the main namespace of a Wikipedia project on a mobile device within 24 hours of registration on a mobile device and that edit not being reverted within 48 hours of being published. This will be measured on a per platform basis.

[1] This aligns with how Growth has defined this metric with the following primary exceptions: (1) we are looking only a main namespace edits (page_namepace = 0) and (2) we are only reviewing edits completed on a mobile device (mobile web or app) with per platform splits.

Constructive Activation Baseline Numbers (Based on data from March and April 2024)

• Overall
  • Mobile web: 17.6%
  • Mobile Apps: 2.1%

• By Registration Month (minimal variation over time. Two most recent months reflected below)
  • • Mobile Web:
      • March 2024: 17.9%; April 2024: 17.3%
  • Mobile Apps:
    • March 2024: 2.2%; April 2024: 2.1%

• By Wiki Trends
  • Mobile web constructive activation rates range from a low of about 11% (observed on arwiki and uzwiki) to a high of 36% on fiwiki (limited to wikis where there were at least 100 newcomers that made a mobile edit over the reviewed time period)
  • For apps, there is a little more variance due to lower number of editors available to review on a per wiki basis. For wikis with at least 25 newcomers that made a mobile app edit, constructive activation rates range from about 1.1% on zhwiki to 6.8% on fawiki.

Some other relevant findings:

• Mobile app editors (iOS and Android) represent about only 3% of all newcomers that made an edit within 24 hours of registering on a mobile device. 4% of all mobile edits completed newcomers within 24 hours of registering.
• Almost half (around 46%) of mobile editors (both mobile web and app) make more than one mobile edit on a main namespace 24 hours after registering. As long as one of these edits is not reverted, we count them as as being constructively activated.

Methodology notes
I gathered mobile registrations from March and April 2024, as the most recent months [1] available prior to the end of this fiscal year. I also collected data on API registrations as those are typically mobile apps. For those registrations, data was gathered on edits to a main namespace completed on a mobile device within 24 hours of registration and the reverts of those edits. I reviewed activation rates by platform (mobile web and mobile apps). I also looked at splits by registration month and wiki to get a sense of how much this number varies by time and wiki.

[1] This analysis relies on mediawiki_history. We need to wait for the June mediawiki_history snapshots to obtain May data, because we require a full 24 hours of data into the next month (for users that registered at the end of May).

I referenced and adapted past constructive activation analyses written for Growth's Structured Task Analyses by @nettrom_WMF.

OPEN QUESTIONS/NOTES
[1] The current definition specifies that the edit be completed on a mobile device. I also limited the analysis to users that registered on a mobile device so we are reviewing users that complete a constructive edit on the same platform where they registered. This would exclude desktop registrations. Any concerns or suggested changes to this approach?

[2] For the constructive activation rate of mobile app users, I used API registrations logged in the ServerSideAccountCreation table to try to identify users that registered on wiki via the app and then made an edit. Will follow-up with @SNowick_WMF to see if this is the correct approach or if there are other specific considerations that should be applied to the calculation of activation for mobile app users.

Code Repo

In T360829#9903872, @MNeisler wrote:

OPEN QUESTIONS/NOTES
[1] The current definition specifies that the edit be completed on a mobile device. I also limited the analysis to users that registered on a mobile device so we are reviewing users that complete a constructive edit on the same platform where they registered. This would exclude desktop registrations. Any concerns or suggested changes to this approach?

Saw this but it got left by the wayside in the flurry that was the end of last quarter, hopefully stopping by is still useful.

I don't have a concern with this approach. I tried to look for numbers on this, because I know that I at some point in the past year investigated platform switching for newcomers and found that it only occurred on a single wiki and for a relatively small subset of users, but couldn't find those numbers again. If it turns out that we'd really like them, I'll take another look.

In the meantime, as mentioned, what I do remember is that newcomers rarely switch platforms. Since you're measuring activation, meaning they're editing within 24 hours of registration, I think it'll be even rarer that they'll edit on a platform different from the one they registered on. In other words, limiting this to both mobile registrations as well as edits is very reasonable.

Meta: the next step on this ticket is describing the process I'm proposing KR 1.2 hypothesis "owners" and I follow to converge on a target improvement.

In T360829#9903872, @MNeisler wrote:

@ppelberg
I've completed an initial analysis of the baseline for constructive activation per the definition documented in DRAFT APP FY24-25: OKRs sheet and defined below.

This looks wonderful, @MNeisler. Specific responses below.

More broadly, we can consider this task resolved.

Note: I've included the findings you brought together here in the July Monthly WE1.2 Monthly Report.

OPEN QUESTIONS/NOTES
[1] The current definition specifies that the edit be completed on a mobile device. I also limited the analysis to users that registered on a mobile device so we are reviewing users that complete a constructive edit on the same platform where they registered. This would exclude desktop registrations. Any concerns or suggested changes to this approach?

For the reasons @nettrom_WMF helpfully identified in T360829#9978677, no concerns and/or changes from me.

[2] For the constructive activation rate of mobile app users, I used API registrations logged in the ServerSideAccountCreation table to try to identify users that registered on wiki via the app and then made an edit. Will follow-up with @SNowick_WMF to see if this is the correct approach or if there are other specific considerations that should be applied to the calculation of activation for mobile app users.

Code Repo

If/when you come to find this approach needs revision, I'll assume you'll make us aware. In the meantime, I'm going to assume the approach you described above is fine.

In T360829#10005075, @ppelberg wrote:

Meta: the next step on this ticket is describing the process I'm proposing KR 1.2 hypothesis "owners" and I follow to converge on a target improvement.

Done in T371726.