Wikidata:WikiProject Source Reliability

(Redirected from Wikidata:CRAP)

WikiProject source reliability, aka the Credibility Ratings + Assessments Project, is an effort to identify and aggregate online sources of assessments of the reliability and credibility of sources. These assessments include estimates of bias, verifiability of claims, level of editorial oversight or peer review, and expertise in specific topics. Assessments may be of individual documents, or aggregate assessments of authors or domains.

Types of assessments include:

  • Compilations of self-assessments. Examples include the TRANSPOSE project for collating self-reported peer-review practices of journals).
  • Evaluations by groups created specifically to evaluate source reliability. Examples include Media Bias Fact Check, other fact checking sites, and sites like Ad Fontes Media that produce visuals referenced by others.
  • Evaluations by communities of practice, as a by-product of their work reviewing or sourcing information to others. Examples include Perennial Sources lists on various language Wikipedias, topical Reliable Sources lists from individual WikiProjects, and newsrooms that publish their internal measures of source reliability.
  • Compilations of secondary assessments, including the above. Examples include Iffy.news.

Goals

edit

Short-term

edit
  • Add source credibility ratings on Wikidata (including topical focus on news outlets and vaccines)
    • the catalog of sources,
    • for each catalog entry, a timestamped snapshots of its reliability assessments,
    • adding assessments to a WD entity for the source [create it if it doesn't exist yet]
  • Curate master Wikipedian-source reliability lists (as versioned tables w/ explicit schema)
    • Federating WD (specialist sets of tabular data)
    • citeable (as archival soure w/ identifier)
    • explicit schemas (+ maps between them)
    • versioned (schemas + datasets)
    • provenanced (chains of derivatives for composite data)

Mid-term

edit
  • Model source credibility as a schema using new or existing properties.
  • Catalog a range of sources for such ratings
  • Set up versioned, archival backups of the snapshots

Long-term

edit
  • Integration into future shared citations / wikicite efforts

Tasks

edit
  • Propose properties (or identify existing properties) for: assessment of factual-reporting, assessment of credibility, assessment of political bias
    assessed source reliability - could this be a single property w/ qualifiers for "type of reliability" and also for "according to whom"?
  • Propose properties for Permanent IDs for these sources
    Compare: MBFC ID, AllSides ID, PolitiFact ID. Get sources to create identifiers for entities if they don't have one.
  • Add these properties to entities in the datasets below (CU, Iffy, &c)
  • Create entities for news associations that recognize other orgs (State + national press associations)
    Relate publications to the press association to which they belong (instance of: member?).
  • Define a shared schema : data model
  • Figure out how to add "hijacked journal" as "sourcing circumstances" option, or otherwise make it an accessible qualifier. (Example)
  • ...

Interested participants

edit
  1. Ocaasi (talk) 18:43, 7 March 2023 (UTC)[reply]
  2. Sj (talk) 18:49, 7 March 2023 (UTC)[reply]
  3. Harej (talk) 18:49, 7 March 2023 (UTC)[reply]
  4. SuperHamster (talk) 18:51, 7 March 2023 (UTC)[reply]
  5. Hearvox (talk) 20:48, 10 April 2023 (UTC)[reply]
  6. Andy Mabbett (Pigsonthewing); Talk to Andy; Andy's edits
  7. So9q (talk) 14:45, 29 April 2023 (UTC)[reply]
  8. Daniel Mietchen (talk)
  9. AmandaSLawrence (talk) 12:06, 6 September 2023 (UTC)[reply]
  10. Nikola Tulechki (talk) 16:45, 13 March 2024 (UTC)[reply]

Data model

edit

Derived from "perennial sources" data model:

Property proposals

edit


see also: Proposed schema for reusable source data

Indexes

edit
  • Media Bias Fact Check [MBFC] (mainly news)
    • Measures: Accuracy, right-left bias (US scale)
  • Iffy.news (incorporates MBFC)
    • Measure: Iffyness
    • Thoughts on incorporating into WD: notes
  • All Sides
  • Ad Fontes rankings by source (mainly news; slow site)
    • Measures: Reliability + news value, Right-Left bias (US scale),
  • NewsQ[1] (mainly news)
    • Measures: identification, journalistic content, journalistic behavior, diversity, external assessment, external affiliation, traffic + engagement
  • Beall's List (journals)
    • Measures: tags: predatory journals, hijacked journals

Source lists

edit

General sources lists

edit

Perennial sources

edit
wide range of source sites, including journals + popular websites

Topical sources lists

edit

Current compilations of reliability indicators

edit

External sources:

See also

edit

Related projects

Related policy

Research into credibility and reliability assessments

In the broader context of citations