A number of websites have blocked Citoid due to the high volume of traffic its activity is placing on their websites. This results in Citoid errors when attempting to cite content by inserting these websites' URLs.
Known blocked websites
Strategy and state
Strategy
At present, there are four strategies we are pursuing to ensure volunteers are able to reliably generate citations using Citoid in ways that meet our partners' needs and expectations.
These four strategies are as follows…
1. Align with partners so that we can:
- Understand their needs and expectations to further improve how Citoid behaves.
2. Improve UX so that we can:
- Offer volunteers clear path(s) forward when Citoid fails
- Simplify the steps to generate a reference when Citoid is unable to do so automatically
3. Increase observability so that we can:
- Swiftly address issues, when they emerge
- Ensure Citoid is behaving in ways that meet volunteers and partner needs
- Evaluate the impact of changes we're making to Citoid
4. Reconsider internal assumptions so that we can:
- Ensure Citoid behaves in ways that accommodate the technical and business constraints that ensure the longevity of partner infrastructure
State
The section contains the actions we are taking, and will consider taking in the future, to deliver the impact described in the Strategy section above.
Strategy | Ticket(s) | Description | Status |
---|---|---|---|
Improve Citoid UX | T364595 | Offer people an alternative path for generating citations from within Citoid's error state | ✅ Done; deployed 12 June 2024 |
T364594 | Revising Citoid's error message to be more specific | ✅ Done; deployed 13 June 2024 | |
Increase observability | T364901 | Log data about which domains are failing most frequently | ✅ Done; data being logged as of ~24 June 2024 |
T365583 | Log data when Citoid fails because the media type is (e.g. PDFs) is no supported | ✅ Done; deployed 12 June 2024 | |
T364903 | [SPIKE] Determine how specific we can be about logging why Citoid is failing | ✅ Investigation complete; results informing work in T365583 and T364901 | |
T368802 | Identify patterns in data now being logged about Citoid performance | Up next | |
Reconsider internal assumptions | T366093 | Change Citoid user agent to use same pattern as Zotero | ✅ Done; deployed 12 June 2024 |
T367194 | Citoid/Zotero: Create rate limiting configurable on a per site basis | Exploring technical feasibility; work not yet prioritized | |
T367452 | Reduce Citoid HTTP request volume by using HTTP HEAD instead of HTTP GET | ✅ Done; deployed week of 17 June 2024 | |
Ticket needed | Cache metadata results to reduce amount of traffic we're sending to domains | Investigation required to assess feasibility; this work has not yet been prioritized | |
Ticket needed | Enable people to do the metadata scraping themselves. | Investigation required to assess feasibility; this work has not yet been prioritized | |
Ticket needed | Write Citoid as a layered set of data adapters | Investigation required to assess feasibility; this work has not yet been prioritized | |
T95388 | Fallback to archive.org when Citoid request fails | 🟢 Investigation is active | |
Align with partners | - | Talk with partners directly to understand what they need from Citoid to fulfill the requests people are making with it | In progress |
Original description
first seen today at an event: https://en.wikipedia.org/wiki/Special:Diff/1218432547
later during same event had a problem with NY times. https://en.wikipedia.org/wiki/Special:Diff/1218452300
I went home, pulled a link off NY times front page and tried a test at [[Wikipedia:Sandbox]]. (didn't save)
link: https://www.nytimes.com/2024/04/11/us/politics/spirit-aerosystems-boeing-737-max.html
error message: We couldn't make a citation for you. You can create one manually using the "Manual" tab above.
NY times was definitely working here, (2024-02-13) this URL also now broken: https://en.wikipedia.org/wiki/Special:Diff/1207056572