Future Audiences/Experiment:Citation Needed
| Part of the WMF Annual Plan, 2023 – present |
| Future Audiences |
|---|
 |
| 2024-25 Objectives & Key Results (Summary) |
|
FA1: Test Hypotheses |
| Experiments |
|
| Team |
|
MPinchuk (WMF) – Product lead
|
| Commons category |
Citation Needed is an experimental feature developed in 2024 by the Wikimedia Foundation Future Audiences team. It uses generative AI to test if and how we can engage people to use Wikipedia to verify information as they browse the web.
This feature builds off our previous experiment of building a Wikipedia plugin for ChatGPT.
Timeline
| Date | Milestone | Status |
|---|---|---|
| October-December 2023 | Early ideation & feedback-gathering | Done |
| January-March 2024 | Minimum viable product definition & development | Done |
| Feb 26-March 14 2024 | Internal acceptance testing (small number of WMF and community testers) | Done |
| March 22-27 2024 | Extension submitted to and accepted in the Chrome Store | Done |
| March 28-May 24 2024 | Soft launch
(Extension is available to any community members who want to test it + invited group of US donors) |
Done |
| May 09 2024 | General release
(Extension is available to anyone) |
Done |
| May - June 2024 | User survey released to get more in depth feedback | Done |
| June 2024 | Analysis and planning next steps (e.g., wind down, try a new experiment, etc.) | Done |
Results
Top-level findings from qualitative (survey) and quantitative (usage data):
- Users who responded to our surveys were generally enthusiastic about the idea of verifying information on 3rd party platforms with Wikipedia
- ~1000 users installed the extension, further confirming that there is interest in/demand for this workflow
- However, sustained usage was low/nonexistent, indicating some/all of the following: it is too manual/clunky/hard to remember to use the extension; websites are not generally where suspicious claims show up and this experience might be needed elsewhere – i.e., on social media apps; mis/disinformation is more subtle/systemic and needs more than claim-level verification
More detailed findings can be found in the full report here.
Experiment FAQs
What does Citation Needed do?
- User installs a Chrome browser extension for Citation Needed
- While browsing, the user can select a sentence-sized text (the claim) to verify with Citation Needed
- Once Citation Needed is activated, the extension:
- Uses ChatGPT API to extract search keywords
- Uses the Wikimedia CirrusSearch API to search for relevant articles, and article sections
- Returns text from relevant article sections
- Uses ChatGPT API to parse returned article text for most relevant quote
- Uses ChatGPT API to provide an inference on whether the Wikipedia content supports the claim
- Returns the relevant quote and inference for the user, along with links to article and quality signals
The user will see:
- the original claim
- clickable links to a relevant Wikipedia article (section)
- a relevant quote from the Wikipedia article
- quality signals of the article: last edit date, number of references, number of editors
- an inference suggesting that the claim is un/supported by Wikipedia
What do we hope to learn from this experiment?
- Do people browsing the web want to verify the content they're seeing on other websites?
- Do they see Wikipedia as a trustworthy/credible knowledge source that is useful for such verification?
- Can we reach new audiences or create new opportunities for current audiences to use Wikipedia by meeting users where they are (i.e., on various external sites)?
- Can generative AI assist in searching for, retrieving, and making inferences about the content on Wikipedia?
Why is this experiment only in one language and one browser?
Focusing just on one popular global language and web browser with this experiment allows us to quickly validate the hypothesis that people want to verify third-party web content with Wikipedia, before investing a large number of resources into developing this as a fully-featured product. If we see high demand for supporting this use-case from this initial experiment, we will recommend "graduating" it from an experimental idea to a product (which will mean making it available to more users in more languages).
What happens after the experiment?
Depending on the results of the initial 3-month experimental period, we may:
- Test other features (e.g.,: recommend a new fact to add to Wikipedia, based on a claim that is not present in Wikipedia but present on a reliable source) that help us understand how else a feature like this might support the free knowledge ecosystem.
- Recommend "graduating" this from an experimental idea to a product. This would involve taking some time to make the feature more stable, scalable, and able to support more languages and users.
- Recommend ending the experiment. If we do not find that this feature is useful, we will document what we have learned and move on to other experiments.
How/when can I try Citation Needed?
The Citation Needed extension is currently available in the Chrome Store – you can find and install it via this link and give feedback via the "Feedback" link in the extension.
As of March 27, 2024, we are inviting any interested Wikimedia community members and some US donors to test out this first version of the extension in order to evaluate its quality and usefulness. If this "soft launch" is successful, we may release some updates (based on user feedback), publicize the extension to a larger audience, and continue to gather data from this experiment for a ~3 month period.
Is it open source?
Yes. You can find the code for the API and for the browser extension in the "Future Audiences" repository of the Wikimedia GitLab instance.