Meta: I've updated the task description to clarify the requirements of this task based on what @MNeisler and I talked about offline on 17 July 2024.

In T368988#10009174, @ppelberg wrote:

Meta: I've updated the task description to clarify the requirements of this task based on what @MNeisler and I talked about offline on 17 July 2024.

I've made another round of updates to the task description based on the initial scope @MNeisler and I converged on offline this week.

Here are the initial results exploring Citoid feature use by user experience level and platform. Please let me know if you have any questions or any further breakdowns would be useful. cc @ppelberg

Methdology Notes

Reviewed all edit attempts and saves recorded in July 2024 that were made to a main namespace on a Wikipedia Project. Data on engagement with the Citoid feature are tracked in visualeditorfeatureuse and other edit data was obtained from editattemptstemp and mediawiki-history.

How does Citoid usage vary by experience level and platform?

Of all the edits in the Wikipedia main namespace where Citoid is used, what proportion of these edits were completed by 1) brand new editors, 2) Junior Contributors, 3) Senior Contributors (grouped by platform (mobile / desktop)?

• Overall, across both platforms, most edits (82.9%) where citoid was used were completed on desktop

• The majority (76.2% of all citoid edits on desktop and 70.7% of all citoid edits on mobile) are completed by senior editors; however, if we look at the proportion of distinct users that completed at least one citoid edit, the frequency of usage is slightly higher among junior contributors. In other words, while junior editors complete less overall citoid edits than senior editors, a higher proportion of distinct junior contributors use the feature. See details in the charts below.

Proportion of distinct Citoid edits completed by experience level and platform

Proportion of distinct Citoid users by experience level and platform

How heavily do people depend on Citoid to insert citations

Of all the unreverted edits made with VE in the Wikipedia main namespace that add a new reference, what proportion of those edits did people use Citoid to make? What percentage of those edits did people use other means to add references?

• Citoid was used for more than half (62%) of all edits with a reference that were not reverted within 48 hours.
• There are similar rates of Citoid dependency on both mobile and desktop platforms as shown in table below.

Unreverted edits made with VE that add a reference

• The use of Citoid to complete edits with a reference is slightly higher for brand new editors. 71.4% of edits completed by brand new editors that were published with a new reference used Citoid compared to 60.1% of edits completed by senior editors.

To what extent are people finding success with Citoid?

Of the edits attempts in Wikipedia's main namespace that involve someone using Citoid, what percentage of those edits involve people doing the following? i) Successfully/unsuccessfully inserting a reference (grouped by experience level and platform)

• A reference was successfully inserted for 50% of editing sessions where citoid was opened.
• People are less likely to be able to successfully insert a reference after opening citoid on mobile compared to desktop. Across all experience levels, 29% of users successfully inserted a reference after opening citoid on mobile compared to 57% on desktop.
• Senior editors are more likely to successfully insert a reference with Citoid compared to junior or brand new editors on both mobile and desktop platforms. Senior editors success rate on both platforms is around 60% while brand new editors successfully insert a reference for less than half of editing sessions where Citoid is opened.
• Brand new editors are much more likely to insert a reference successfully on desktop compared to mobile. Only 12.8% of citoid edits attempts on mobile by brand new editors involved a reference being successfully inserted compared to 45.9% on desktop.

and ii) Publishing/not publishing an edit with a new reference (grouped by experience level and platform). Note: This excludes any edits reverted within 48 hours and is limited to edits that were published with a new reference as indicated by the editcheck-newreference tag.

• 44.8% of all desktop citoid edit attempts were successfully saved with a new reference compared to 20.7% on mobile. Note: This includes users who may have opened the citoid editor but did not attempt to generate or insert a reference. The chart below clarifies the proportion of edits that reach each step of a successful citoid edit from open -> click -> insert -> successfully publish.
• The largest percent decrease between stages is between the user opening the ciotid editor (action = open) to clicking the create button to generate a citation (action = create). 63.5% of editing sessions on desktop and only 35.4% of editing sessions on mobile include a click to generate a citation after opening Citioid.
• Users are more likely to successfully publish an edit after clicking the create button. About 70.5% of edits that included a click to generate a citation were successfully saved on desktop and 59% on mobile.
  
  By Platform

• As shown in the chart below, brand new editors are the least likely group to complete the remaining steps of publishing a citoid edit after opening citoid. Only 19.9% of edits by brand new editors that opened citoid were successfully published.

By Editor Experience

Of the times when people open Citoid and do NOT follow through to insert a reference, what reasons explain why this might be the case?

• 81% of edit attempts when people open Citoid and do not insert a reference is because they abandoned the workflow. Only 19% of all citoid edit attempts failed to insert a reference because the automatic citation generation failed.
• Per platform splits are provided below:

• People abandoning the workflow is the most common reason for not inserting a reference across all experience level groups as well. Senior editors that open citoid are the most likely group to not insert a reference due to the automatic citation generation failure (36.6% of citoid edits by senior editors that do not insert a reference)

However, many of the citoid edits attempts were abandoned prior to clicking the create button to generate a citation. If we limit to editing sessions where people click the create button to insert a citation, then the primary reason for not inserting a citation is due to an automatic citation generation fail.

• 63% of editing sessions on desktop and 61% of editing sessions on mobile where people click the create button do not follow through to insert a reference due to an automatic citation generation fail.
• Similar rates were observed for junior editors and senior as well
• For brand new editors that click the create button, there is closer to an even split between reasons for not inserting a reference. 48.3% of these editing sessions are abandoned and 51.7% of these editing sessions included an automatic citation generation fail.

In T368988#10072946, @MNeisler wrote:

Here are the initial results exploring Citoid feature use by user experience level and platform. Please let me know if you have any questions or any further breakdowns would be useful. cc @ppelberg

This looks great, @MNeisler. Comments/questions in-line below...

In parallel, I've pulled out – what I think are – the notable findings into this document for us to share internally and externally (by way of T370682).

Methdology Notes

Reviewed all edit attempts and saves recorded in July 2024 that were made to a main namespace on a Wikipedia Project. Data on engagement with the Citoid feature are tracked in visualeditorfeatureuse and other edit data was obtained from editattemptstemp and mediawiki-history.

Assuming doing so would not require a great deal of effort, I think it would be worthwhile to re-run this analysis for the 30 days that followed August 3, 2024 to see how – if at all – the decrease in Citoid request volume impacts these numbers.

How does Citoid usage vary by experience level and platform?

Of all the edits in the Wikipedia main namespace where Citoid is used, what proportion of these edits were completed by 1) brand new editors, 2) Junior Contributors, 3) Senior Contributors (grouped by platform (mobile / desktop)?

NOTE: "use" in this context means someone successfully tapped Insert to add a citation into the article they're editing. Limited analysis to edits that were published.

To be doubly sure, would it be accurate for me to understand the data in the table below as meaning the following?

Of all published edits where someone tapped the Insert button within Citoid, 82.9% of said edits occurred on desktop and 17.2% occurred on mobile.

Note: I assume 17.2% and 82.9% summing to >100% is simply a result of rounding.

Assuming doing so would not require a great deal of effort, I think it would be worthwhile to re-run this analysis for the 30 days that followed August 3, 2024 to see how – if at all – the decrease in Citoid request volume impacts these numbers.

This makes sense and would not require much effort. I'll make a note to rerun this around September 3rd. This would be 30 days after the citoid volume decrease on August 3rd and the August mediawiki_history snapshot should be available by then as well.

To be doubly sure, would it be accurate for me to understand the data in the table below as meaning the following?
Of all published edits where someone tapped the Insert button within Citoid, 82.9% of said edits occurred on desktop and 17.2% occurred on mobile

Yes that's correct (limited to main namespace edits on a Wikipedia project).

Note: I assume 17.2% and 82.9% summing to >100% is simply a result of rounding.

Yes. The more exact percentages are 82.85% and 17.15%

In T368988#10085223, @MNeisler wrote:

Assuming doing so would not require a great deal of effort, I think it would be worthwhile to re-run this analysis for the 30 days that followed August 3, 2024 to see how – if at all – the decrease in Citoid request volume impacts these numbers.

This makes sense and would not require much effort. I'll make a note to rerun this around September 3rd. This would be 30 days after the citoid volume decrease on August 3rd and the August mediawiki_history snapshot should be available by then as well.

Excellent. Let's do it.

To be doubly sure, would it be accurate for me to understand the data in the table below as meaning the following?
Of all published edits where someone tapped the Insert button within Citoid, 82.9% of said edits occurred on desktop and 17.2% occurred on mobile

Yes that's correct (limited to main namespace edits on a Wikipedia project).

Wonderful, ok.

Note: I assume 17.2% and 82.9% summing to >100% is simply a result of rounding.

Yes. The more exact percentages are 82.85% and 17.15%

Got it. Thank you for clarifying ^ _ ^

@ppelberg On August 3rd, overall citoid requests from MyBib dropped substantially. I re-ran this analysis to include 30 days post August 3rd and did not identify any significant changes to Citoid feature use due to this change in request volume.

See details below:

How does Citoid usage vary by experience level and platform?

Proportion of distinct Citoid edits completed by experience level and platform

Proportion of distinct Citoid users by experience level and platform

How heavily do people depend on Citoid to insert citations

• No significant changes in citoid dependency. Following August 3rd, Citoid has been used for 60% of all edits with a reference that were not reverted within 48 hours across both platforms, compared to 62% prior in July.

Unreverted edits made with VE that add a reference by platform

• The use of Citoid to complete edits with a reference is still higher for brand new editors following August 3rd. 72.1% of all published edits with a reference by brand new editors used Citoid compared to 57.3% of edits completed by senior editors. This is comparable to July numbers (71.4% by brand new editors and 60.1% by senior editors).

To what extent are people finding success with Citoid?

Before and after August 3rd, a reference was successfully inserted for 50% of editing sessions where citoid was opened. There have not been any significant changes in citoid success rate observed by platform or by experience level.

See updated per platform and per editor experience splits in the citoid editing session funnel charts below:

Of the times when people open Citoid and do NOT follow through to insert a reference, what reasons explain why this might be the case?

• Before and after August 3rd, 81% of edit attempts when people open Citoid and do not insert a reference is because they abandoned the workflow. Only 19% of all citoid edit attempts failed to insert a reference because the automatic citation generation failed.
• proportion of citoid edits that did not insert a reference by platform

• Following August 3rd, people abandoning the workflow is still most common reason for not inserting a reference across all experience level groups. Senior editors that open citoid are the most likely group to not insert a reference due to the automatic citation generation failure. This is primarily because most senior editors do not abandon their edit prior to clicking on the citation create button compared to more inexperienced users.

After clicking the create button to generate a citation on desktop, 63% of all failed citoid editing sessions failed due to an automatic generation failure. This is the same rate observed prior to August 3rd.

We observed a slight decrease on mobile. 59% of failed citoid editing sessions on mobile failed due to an automatic generation failure after clicking the create button compared to 61% prior to August 3rd. We are unable to confirm if this slight decrease is the result of the Citoid volume decrease.

Thank you for pulling this all together, @MNeisler.

Per what we talked about offline, we have NOT yet detected any change in the likelihood that people will be successful using Citoid to generate a citation before/after the 3 August 2024 change in third-party request volume.

Now, in order for us to say – with some confidence – the 3 August 2024 change did, in fact, have no impact, we'd first need to be able to distinguish cases where Citoid fails because of a network failure (e.g. blocked request) and cases where Citoid fails because no search results were found for the provided input.

Note this work is happening in T374572.