Monitoring, Evaluation and

Learning Training for Local

Partners

Learning Project

June 2024

1
Module 4 |
Data Quality Assessment

2
Module 4 | Objective
• Understand data quality and data quality standards

• Understanding on common threats on data quality

standards

• Aware data quality assessment by USAID and IPs

3
 PHOTO: USAID
Data Quality Assessment
• Data informs decisions
across the USAID
Program Cycle
• High quality data is the
cornerstone for evidence-
based decision-making
• USAID’s credibility when
communicating and
reporting requires realistic
understanding of the
limitations of data
 What is Data Quality?
• Data Quality refers to the accuracy or worth of the
information collected.

5
How often should DQAs be
conducted?
USAID DQAs: Internal IP DQAs
• Standard indicators reported to • Recommended
USAID/Washington: annually, especially
for complex
– Within 12 months prior to the new indicators
indicator data being reported and every 3 • Can be conducted
years thereafter by MEL team.

– USAID may choose to conduct DQAs more

frequently if needed
• The Mission’s Custom indicators:
– There is no requirement for data that are
not reported to USAID/Washington
– Should be conducted as needed.
 USAID’s Five Data Quality
Standards
Remember: VIP – Roundtrip.
VALIDITY

TIMELINESS INTEGRITY

DATA
QUALITY

RELIABILITY PRECISION

7
 1) VALIDITY

Definition: Data (and indicator) should clearly

and adequately represent the intended result.

Key Questions: Does the indicator and do the

data collection methods reflect the outputs or
outcomes we seek to measure?
Validity | Common Threats
Data collection process Instrument design
• Inconsistencies in data collection • Poorly structured
process. instruments
• Instruments not always • Reliance on proxy
completed measures
• Transcription errors
• Small or possibly biased
sample/not representative
• Under-skilled or under-
supervised data collectors
Validity | Examples
Possible causes of poor data VALIDITY for the
indicator:
"Number of community workers trained in
• psychosocial
Double counting - People support"
counted who were
trained in topics other psychosocial
• Attendance sheets missing information (name and
date of events, topic of training)
• Sign-in sheet not reviewed and signed by event
supervisor or IP
• Transcription errors
 2) RELIABILITY

Definition: Data should reflect stable and

consistent data collection processes and analysis
methods over time and across sites/partners.
Key Question: Assuming no real change in the
variable, if an external party repeated the same
methods of data collection, would they end up with
the same data?
Reliability vs. Validity
How should we distinguish reliability and
validity?

Valid? Valid? Valid?

Reliable? Reliable? Reliable?

12
 Conti … Reliability
Data is collected and analyzed in a consistent
manner over time and across sites.
Indicator: Number of youth employed
• Organization A counts all youth who are receiving
wages.
• Organization B counts only youth receiving full-
time wages.
• Organization C counts those who work 3 days per
week.
Indicator: Number of people vaccinated for
COVID-19
Reliability | Common
Threats
• Data collection technique is unstructured

• Data collection too costly to repeat

• Changes were made in the instruments

• PIRS lack details

Reliability |
Examples
Possible causes of poor RELIABILITY for the
indicator:

"% of vulnerable community members with

improved level of satisfaction with local
services"
• Time lag between changes in quality of services
and change in perception
• “Local services” is very general: would need to
ask about different types of services
 3) INTEGRITY
Definition: Data collected should have safeguards to
minimize the risk of transcription error or data
manipulation.

Key Question: Are mechanisms in place to reduce the

possibility that data is improperly recorded or
manipulated for political or personal reasons?
Data should be accurate and established
data management systems prevent error
and risk of manipulation.
Integrity | Common
Threats
• No procedures in place to safeguard
• Transcription or data entry errors
• Calculation errors
• Data manipulation
• Uncertainty about data quality from
secondary source
 4) PRECISION

Definition: Data should have sufficient level of

detail to permit management decision-making.
Key Question: Is the magnitude of margin of error
in the data tolerable?
Precision | Common
Threats
• Response categories not sufficiently fine grained
• Rounding at too high a level
• Unacceptable margin of error
 5) TIMELINESS

Definition: Data should be available at a useful

frequency, should be current.
Key Question: Are data up-to-date (internal and external
deadlines) and analyzed often enough to be useful for
program management?

The sooner we know about a

problem, the sooner we can fix it!
Additional Criteria to
Consider
1. Completeness
Definition:
• To the degree to which all the necessary steps
in data collection, data entry, data cleaning,
and data analysis have been carried through.
• In addition, no data are missing and no
responses are incomplete, uncollected, or,
because of other data quality issues, unusable.

22
Additional Criteria to
Consider
2. Confidentiality
Definition:
• Involves the researcher’s
protecting the participant’s
private information.

https://www.usaid.gov/responsibled
ata

23
Additional Criteria to
Consider
3. Ethics
Definition:
• Ethics are a system of moral ideas and rules about our conduct
that reflect international standards and the values of the culture we
work in and of the communities we serve.
• Data ethics are the rules or standards governing the conduct of a
person collecting, collating, reporting on, or utilizing data, and
represent our standard of what’s “right.”
• Informed consent, privacy, confidentiality, and integrity are
basic ethical tenets of any data collection effort. You and your
organization are responsible for ensuring the ethical treatment of
participants.

24
DQA: Implementing Partners
There are no requirements for when partners must conduct
a DQA. But, we still should!
How do you select indicators for DQA?

• For F indicators or those in USAID’s PPR, or for

Project monitoring
• When you identify data quality risks (many staff
or partners reporting into indicator)
• When you have findings or data that do not match
When should you conduct a DQA?
• Earlier is better, to catch and resolve issues (and
note changes in your PIRS!)
• Remember, F indicators are assessed every 3
years (or 1 year if new)

25
A DQA determines if…
• Data quality standards are met.
• Proper data collection methods are used.
• Written procedures are available.
• Qualified personnel are available at all levels.
• Data protections are in place.
• Source documents are maintained.
• Duplicate data are detected.

26
Daily Quality Assessment
Process

Preparation
Desk Review Field Review

Follow-up action Documentation

27
 DQA Process…
 Preparation  Documentation
• Identify DQA team members • Analyze and produce the DQA
• Develop DQA Agenda and timeline report and share as appropriate
• Prepare the presentation on MEL
• system and data flow  Follow-up
• Ensure the implementation of
 Desk Review
the DQA findings and
• Review the activity PIRS improvements as a result
• Review the AMELP

 Field Review
• Conduct interview
• Data verification
• Debriefing for field staff
28
Q&A

29
DQA | Exercise

Please read a short Activity scenario on data

quality and answer the questions (print out to
be distributed).

30