IMPLEMENTATION GUIDANCE FOR

ASSESSMENTS OF FRONTLINE SERVICE

READINESS
Strengthening real-time monitoring of health services in the context
of the COVID-19 pandemic

1 July 2021
WHO continues to monitor the situation closely for any changes that may affect this implementation guidance.
Should any factors change, WHO will issue a further update. Otherwise, this implementation guidance document
will expire 2 years after the date of publication.

© World Health Organization 2021. Some rights reserved. This work is available under the CC BY-NC-SA 3.0 IGO
licence.

WHO reference number: WHO/2019-nCoV/HCF_assessment/Frontline_services/2021.1

Contents

1. Introduction ....................................................................................................................................................... 1
1.1 Context ...................................................................................................................................................... 1
1.2 Tools .......................................................................................................................................................... 1
1.3 Objectives and recommended approach .................................................................................................. 2
2. Core modules..................................................................................................................................................... 3
2.1 COVID-19 case management capacities: diagnostics, therapeutics, vaccine readiness and other health
products – facility assessment tool ....................................................................................................................... 3
2.2 Continuity of essential health services: facility assessment tool .............................................................. 5
2.3 Community needs, perceptions and demand: community assessment tool ............................................ 8
3. Preparation for implementation ..................................................................................................................... 11
3.1 Governance and coordination ................................................................................................................. 12
3.2 Sampling .................................................................................................................................................. 16
3.3 Survey planning and preparation ............................................................................................................ 18
4. Guide for interviewers ..................................................................................................................................... 26
4.1 Interview skills ......................................................................................................................................... 26
4.2 Obtaining consent ................................................................................................................................... 27
4.3 Completing the questionnaire ................................................................................................................. 27
4.4 Using and updating the interviewer call log ............................................................................................ 31
4.5 Troubleshooting ...................................................................................................................................... 31
5. Guide for data managers ................................................................................................................................. 33
5.1 Data download, cleaning, management and analysis ............................................................................. 34
5.2 Field-check tables .................................................................................................................................... 36
5.3 Chartbooks .............................................................................................................................................. 36
5.4 Dashboard ............................................................................................................................................... 40
6. Data use and action plan for system strengthening........................................................................................ 41
6.1 Purpose .................................................................................................................................................... 41
6.2 Action plan for system strengthening ..................................................................................................... 41
7. References ....................................................................................................................................................... 49
8. Annex 1. Suite of health service capacity assessment modules ..................................................................... 50
9. Annex 2. Interpreting results from a stratified sample ................................................................................... 51

iii
Abbreviations

COVID-19 novel coronavirus 2019 disease

CSV comma-separated values
IMST incident management support team
IPC infection prevention and control
MoH Ministry of Health
PPE personal protective equipment
WHO World Health Organization

iv
1. Introduction
1.1 Context
Countries face a multitude of questions and decisions that must be addressed to prepare for and respond to the
COVID-19 (novel coronavirus 2019 disease) pandemic while simultaneously maintaining the delivery of other
health services. Key decisions made and actions taken to mitigate the risk of potential health system collapse
must be informed by accurate and real-time data collected through ongoing monitoring during all phases of the
COVID-19 pandemic. Against this rapidly evolving situation, many countries are facing challenges in ensuring the
availability of accurate and timely data on the capacities of frontline health providers and facilities to deliver
essential COVID-19 tools. At the same time, routine data systems are falling short in their ability to detect and
track the extent of disruptions across essential health services that could inform mitigation strategies and
responses to evolving community needs and barriers to accessing care. In such contexts, countries should
consider implementing regular and rapid assessments in facilities and communities to prioritize needs for
service availability; workforce capacities, training and protection; the availability of essential health products
and supplies; vaccine readiness; infection prevention and control (IPC) capacities; and safety measures.

1.2 Tools
To address the dual-track challenge of responding to COVID-19 while maintaining the delivery of essential health
services, the World Health Organization (WHO) has developed the Suite of health service capacity assessments
in the context of the COVID-19 pandemic, a new collection of tools for health facilities and communities to
support rapid and accurate monitoring of current, surge, and future frontline service capacities throughout the
different phases of the COVID-19 pandemic. The suite consists of modules that can be used to prioritize actions
and decision-making at the health facility, subnational and national levels. The suite includes three core
modules.

• COVID-19 case management capacities: diagnostics, therapeutics, vaccine readiness and other health
products for COVID-19 – facility assessment tool. This tool aims to assess health facilities’ capacities for
COVID-19 case management, including the availability of diagnostics, therapeutics and other essential
health products, such as oxygen and personal protective equipment (PPE), as well as cold chain
capacities.
• Continuity of essential health services: facility assessment tool. This tool aims to assess the capacities
of primary care and hospital facilities to deliver essential health services (including the availability of
health workers and their infection rate, isolation and triage capacities, adherence to IPC standards, and
the availability of essential medicines and supplies) and helps to track changes in service utilization and
modifications to service delivery.
• Community needs, perceptions and demand: community assessment tool. This tool aims to collect
information on unmet health needs, changes in care-seeking behaviours, and barriers to care affecting
service demand using information from interviews with key community informants.

When implemented regularly, the tools can help alert authorities and other stakeholders to modifications
needed in service delivery or investment, or both, to guide the response of health systems. The suite also
includes additional modules that can be used for one-time or recurrent in-depth assessments, which countries
may select and customize according to their context and needs. The modules have been developed based on
WHO’s guidance on COVID-19 preparedness and response, facility readiness for COVID-19 case management
and on ensuring the continuity of essential health services during the COVID-19 outbreak, including Maintaining

1
essential health services: operational guidance for the COVID-19 context (1). The complete set of modules is
listed in Annex 1.

1.3 Objectives and recommended approach

The primary objective of the assessment (or survey) is to rapidly detect and monitor bottlenecks in health
systems and health service capacity as well as gaps in readiness to ensure that essential health services continue
to be provided while health systems respond to the rapidly changing context of the COVID-19 pandemic. As
such, the surveys must respond to urgent needs for regular, real-time data, and they must be implemented
rapidly, efficiently and safely in the emergency setting of the pandemic (e.g. surveys must be implemented in a
manner that respects social distancing and travel restrictions and makes use of limited resources that have a
high return, thus reducing the burden on facilities to the greatest extent possible).

The recommended method of implementation is to use an electronic questionnaire delivered rapidly and
regularly through telephone interviews to a sample of sentinel facilities. (These types of surveys are also
referred to as computer-assisted telephone interviews.) While such a survey will not necessarily be fully
representative of the national context, sentinel facilities can nonetheless provide early evidence of changes in
the provision and utilization of health services, and findings can be used to inform strategies for modifications to
service delivery and to guide investments of resources. A single round of data collection should be completed in
a short time frame (approximately 1 week) to enable the use of real-time data.

Telephone interviews also have the potential to save costs, as travel is not required and a greater number of
people in a greater number of facilities can be interviewed during a given day. Nevertheless, compared with in-
person interviews, the method carries its own unique complexities, and so clear governance structures,
coordination and planning, operational protocols, and trainings to build skills to address the challenges of
telephone interviews (see Section 4) are needed.

2
2. Core modules
2.1 COVID-19 case management capacities: diagnostics, therapeutics, vaccine readiness and other
health products – facility assessment tool
This tool was developed to assess capacities for COVID-19 case management, and it focuses on ensuring the
provision of health products for COVID-19 patients to facilities designated to serve these patients. The tool
allows regional or national governments, or both, and health facilities (if it is used for self-assessment) to
evaluate the availability of and status of stock-outs of critical COVID-19 medicines, equipment and supplies at
each site and to identify areas that need further attention to enable the facility to respond effectively to the
pandemic. It is intended for use in health facilities treating moderate, severe and critical cases of COVID-19,
including hospitals and designated COVID-19 treatment facilities. See Tables 1 and 2 for further details on the
module’s objectives, use, content areas and key performance indicators.

Table 1. COVID-19 case management capacities module: objectives, use and content areas

Assessment Description

Objective Assess current and surge capacities of health facilities for COVID-19
management (i.e. clinical tools and essential supplies)

Use Guide rapid deployment and scale up of essential COVID-19 clinical tools and
supplies

Target audience • Incident management and emergency operations officers

• Facility managers
• Pharmacists
• Biomedical engineers
• Infection prevention and control officers
• Planning officers
• Procurement officers
• Laboratory staff
Respondents Facility managers or facility management team members, or both, in hospitals
and COVID-19 treatment centres

Content Section 1: Health facility identification and description

Section 2: Hospital incident management support team
Section 3: Case management and bed capacities for COVID-19
Section 4: Selected medicines and supplies for COVID-19 case management
Section 5: Personal protective equipment and infection prevention and control
Section 6: COVID-19 laboratory diagnostics
Section 7: Medical equipment for diagnosis, patient monitoring and case
management
Section 8: COVID-19 vaccine readiness (optional)
Section 9: Interview results

When to use From the early stages of the emergency to early recovery

3
Table 2. COVID-19 case management capacities module: questions this module helps to answer and key
performance indicators

Sections Key questions Key performance Indicators

1 Health facility • What are the facility’s • All key performance indicators can be
type and characteristics? disaggregated by facility type, residence area
description (rural/urban) and managing authority
(public/private)

2 Hospital IMST • Have facilities adopted and • Percentage of facilities with IMST protocols
activated IMST protocols? adopted and activated

3 Case • Do facilities have sufficient • Total no. of beds for COVID-19 patients for
management beds and space to manage care for moderate, severe or critical
and bed capacity COVID-19 patients? patients
• No. of beds currently occupied by COVID-
19 patients
• Total no. of beds available for surge
(intensive care unit, respiratory isolation)

4 Medicines and • Do facilities have the necessary • Percentage of facilities with tracer medicines
supplies medicines and medical supplies available
to manage COVID-19 patients? • Percentage of facilities participating in the
Solidarity clinical trial (and availability of trial
medications)

5 PPE and IPC • Do facilities have the necessary • Percentage of facilities with PPE available for
PPE for health workers? staff (masks, gowns, goggles)
• Do facilities have the necessary • Percentage of facilities with IPC supplies
IPC supplies? available (soap, biohazard bags, sanitizer
stations)

6 COVID-19 • Do facilities have the necessary • Percentage of facilities with laboratory

laboratory diagnostic supplies for COVID- diagnostic capacities with tracer items
diagnostics 19 testing? (specimen collection supplies, on-site PCR or
rapid diagnostic tests, system for off-site
testing)
• Percentage of facilities receiving timely test
results

7 Medical • Do facilities have the medical • Percentage of facilities with available and
equipment equipment necessary for functional medical equipment on site for
COVID-19 diagnosis, COVID-19 diagnosis, monitoring and case
monitoring and case management (X-ray capacity, pulse
management? oximeters, ventilators, oxygen)
• Percentage of facilities with malfunctioning
equipment (and reasons for these)

4
 8 COVID-19 • Do facilities have a functioning • Percentage of facilities with functional cold
vaccine cold chain ready to support the chain capacity to deliver COVID-19 vaccines
readiness introduction of COVID-19 (vaccine refrigerator with continuous
vaccines? temperature recorder, vaccine carriers and
cold boxes, ice packs)

IMST: incident management support team; IPC: infection prevention and control; PPE: personal protective
equipment.

2.2 Continuity of essential health services: facility assessment tool

The continuity of essential health services module is a comprehensive tool that can be used to rapidly assess a
health facility’s and workforce’s capacities to continue to provide essential health services during the COVID-19
pandemic. The tool collects information about the health workforce’s capacities, financial management of the
facility, changes in the delivery and utilization of health services and reasons for the changes, IPC capacities, and
capacities to manage mild to moderate COVID-19 cases in primary care settings. It also includes optional
modules that can be used to assess the availability of therapeutics and diagnostics, vaccine readiness and
infrastructure. See Tables 3 and 4 for further details on the module’s objectives, use, content areas and key
performance indicators.

Table 3. Continuity of essential health services module: objectives, use and content areas

Assessment Description

Objective Assess a health facility’s and health workforce’s capacities to continue to safely provide
essential health services throughout the pandemic

Use Detect and track changes in service utilization, modifications to service delivery, staff
capacities and protection for staff to guide strategies and plans to mitigate disruptions
and maintain essential health services
Target • National and subnational health authorities
audience • National and subnational COVID-19 incident management support teams
• Facility managers
• WHO and other partners
Respondents Facility managers or facility management team members, or both, in hospitals and
primary care facilities

5
Content Section 1: Health facility identification and description
Section 2: Staffing
Section 3: Financial management
Section 4: Service delivery and utilization
Section 5: COVID-19 infection prevention and control and personal protective equipment
Section 6: Management of suspected and confirmed COVID-19 cases in primary care
facilities
Section 7: Availability of selected tracer therapeutics (optional)
Section 8: Availability of diagnostics (optional)
Section 9: Vaccine readiness (optional)
Section 10: Facility infrastructure (optional)
Section 11: Interview results
Annex: Routine data record review
When to use From the early stages of an emergency to recovery and continuity after recovery

6
Table 4. Continuity of essential health services module: questions this module helps to answer and key
performance indicators

Sections Key questions Performance Indicators

1 Health facility • What are the facility’s • All key performance indicators can be
type and characteristics? disaggregated by facility type,
description residence area (rural/urban) and
managing authority (public/private)

2 Staffing • How many staff are available? • Percentage of staff diagnosed with
• How many staff have been COVID-19, by occupation
diagnosed with COVID-19? • Percentage of facilities with staff leave
• Are additional training and support or absences and reasons for absences
being provided to staff? or changes in staff management
• Percentage of facilities providing staff
training, supportior supervision

3 Financial • Are facilities continuing to charge • Percentage of facilities that waived or

management user fees during the pandemic? increased user fees
• Are facilities receiving additional • Percentage of facilities receiving
funding for essential health additional funding for essential
services? health services and sources of
• Are staff salaries and overtime funding
being paid on time? • Percentage of facilities maintaining
on-time salary and overtime
payments

4 Service • Has the delivery of services that are • Percentage of facilities with
delivery and unrelated to COVID-19 changed? modifications to service delivery
utilization • Has service utilization increased or • Percentage of facilities with observed
decreased, and, if so, what are the increases or decreases in tracer
reasons for the change? services (outpatient, inpatient,
• Has the facility implemented emergency) and reasons for the
community communication changes
campaigns? • Percentage of facilities with plans for
• Has the facility made catch-up plans restoring services
for missed routine appointments? • Percentage change in service
utilization (record review)

5 IPC and PPE • Are processes and protocols in place • Percentage of facilities with measures
to ensure the safe delivery of health to ensure a safe environment (triage
services? capacity, isolation capacity)
• Do facilities have triage and isolation • Percentage of facilities with IPC
capacities? guidelines in place
• Do staff have sufficient PPE to safely • Percentage of facilities with adequate
deliver essential services? PPE for staff

7
 6 Management • Which COVID-19 primary care • Percentage of primary care facilities
of COVID-19 services are being delivered in the with measures to manage mild cases of
in primary facility? COVID-19
care • What support is being provided to • Percentage of facilities with capacity to
deliver these services? provide COVID-19 services in primary
care (collecting specimens for
diagnosis, on-site testing, referrals)

7 Therapeutics • Do facilities have therapeutics • Percentage of facilities with tracer

available to provide essential health therapeutics, supplies and vaccines
services?

8 Diagnostics • Do facilities have diagnostic tests • Percentage of facilities with tracer

and supplies available to provide diagnostics
essential health services?

9 Vaccine • Do facilities have functioning cold • Percentage of facilities with cold chain
readiness chain capacity? capacity

10 Facility • Have facilities experienced • Percentage of facilities that have

infrastructure unplanned closures? experiences unplanned closures
• Have facilities experienced • Percentage of facilities with
infrastructure-related issues? infrastructure-related issues
IPC: infection prevention and control; PPE: personal protective equipment.

2.3 Community needs, perceptions and demand: community assessment tool

This tool can be used by countries to conduct a pulse survey to assess a community’s health needs, perceptions
around access to essential health services during the COVID-19 outbreak, attitudes towards COVID-19
vaccination, and assets and vulnerabilities. See Tables 5 and 6 for further details on the module’s objectives, use,
content areas and key performance indicators.

Table 5. Community needs, perceptions and demand module: objectives, use and content areas

Assessment Description

Objective Assess community health needs, changes in care-seeking behaviours and barriers to
care affecting the demand for services, and disruptions to community-based care

Use Guide strategies and plans to address unmet health needs and eliminate barriers to care

Target • National and subnational health authorities

audience • National and subnational COVID-19 incident management support teams
• Facility managers
Respondents Community leaders, community health workers, leaders of civil society organizations

Content Section 1: Identification and informed consent

Section 2: Community needs and use of essential health services
Section 3: Barriers to seeking essential health services in the community
Section 4: Attitudes towards COVID-19 vaccination

8
 Section 5: Barriers to the delivery of community-based services
Section 6: Interview results

When to use From the early stages of an emergency to recovery and continuity after recovery

Table 6. Community needs, perceptions and demand module: questions this module helps to answer and key
performance indicators

Section Key questions Key performance indicators

1 Identification and • Who is the key informant • All key performance indicators can be
informed consent providing the responses? disaggregated by type of key informant
• What is the residence setting (community health worker, community
of the community? leader, representative of a civil society
organization) and residence area
(rural/urban)
2 Community needs • How has the COVID-19 • Percentage of key informants who believe
and use of pandemic affected the that community has unmet health needs
essential health utilization of essential health
services services?
• What are the current unmet
needs for health services in
the community?
3 Barriers to seeking • What are the main barriers • Percentage of key informants who
care faced by people seeking to believe that the community faced
use essential health services barriers to seeking care before the
during the COVID-19 COVID-19 pandemic and percentage who
pandemic? believe the barriers have become worse
• Are there marginalized • Percentage of key informants who
groups that have been more believe there are disadvantaged groups
affected during the COVID- in the community
19 pandemic? • Percentage of key informants who report
• Where or what is the first specific type of service delivery points as
point of contact for health the first point contact during the COVID-
care during the COVID-19 19 pandemic in the community
pandemic?
4 Attitudes towards • What is the community’s • Percentage of key informants who believe
COVID-19 attitude towards a COVID-19 there is community demand for the
vaccination vaccine? COVID-19 vaccine (adults, children)
• Reasons for low demand

5 Barriers to the • Have community health • Percentage of community health workers

delivery of workers been able to continue with perceived risks and reasons for these
community-based their work during the COVID- perceptions
services 19 pandemic? • Percentage of key informants lacking
• Have community health support to perform their work
workers experienced stigma

9
while pursuing their • Percentage of communities with changes
responsibilities? in service volume, by type of service
(malaria prevention, social support for
tuberculosis patients)

10
3. Preparation for implementation
Each assessment (or survey)has been designed so that it can be implemented rapidly, frequently and at low cost.
However, it is critical to ensure good governance, coordination and planning to prepare for successful
implementation. Table 7 outlines the generic steps and activities necessary for conducting an effective survey,
from planning and preparation to data analysis, visualization and the use of the collected information. A concept
note and roadmap should document the specific timeline, responsible staff and any required resources for each
activity.

Each assessment prioritizes the rapid collection and use of data in the pandemic context: rapid turnaround is
possible only with sufficient planning and preparation prior to data collection. It is recommended that the
survey manager leads and coordinates detailed activities during the planning and preparation step. In addition,
although the analysis and dissemination of results, and data use is listed separately as Step 5 (see Table 7),
preparation for data analysis starts as soon as the modules are adapted (Table 7, Step 3), and planning for data
use must start when the key stakeholders are identified (Table 7, Step 1). The steps are described in further
detail in the following sections.

Table 7. Outline of survey steps and activities for the workplan for implementing assessments of frontline
service readiness for the COVID-19 pandemic

Step Activities

1 Governance  Ministry of Health submits formal request for support to WHO and partners
and  Establish a dedicated coordination entity led by the Ministry of Health and
coordination comprising key stakeholders in the country and partners
 Determine the scope of the assessment (i.e. the modules to be used) and
frequency of the survey (e.g. monthly or quarterly)
 Identify an in-country team to support the overall management and
implementation of the assessment, including a survey manager, a supervisor, data
collectors or interviewers, a data manager or analyst
 Prepare an overall plan
 Identify resource requirements and prepare a budget
2 Sampling  Obtain a master list of facilities (i.e. a list of all health facility sites, both public and
private) and develop a list of facilities eligible for inclusion
 Determine the sentinel facilities to be sampled based on the sampling strategy
being used
 If conducting a community survey, consider sampling the key informants for
communities within the catchment areas of the sentinel sites
3 Assessment  Review and adapt modules and training materials to meet country-specific needs
planning and  Update online interview tools according to the county-specific adaptations made
preparation to the modules
 Test the tools, evaluate the results and make amendments if necessary
 Review and update the standard analysis plan and automated outputs according to
the county-specific adaptations made to the modules
 Procure the necessary equipment and coordinate logistics
 Plan and conduct training for interviewers
 Develop a contact list based on the facilities being sampled

11
  Do outreach to facilities before the survey to make them aware of it, confirm who
will be the respondent and schedule an appointment
 Develop a schedule for overall data collection
 Develop a daily call schedule for interviewers
 Prepopulate the background questions about facilities in the modules with details
from the master facility list
4 Data collection  Conduct telephone interviews
 Monitor progress and update the data collection schedule, as needed
 Update the daily call schedule for interviewers, as needed
 Validate and check the completion of questionnaires daily
 Download survey data from the server daily
5 Analysis,  Update and produce automated database and outputs daily
results  Develop a roadmap for decision-making that is based on the findings
dissemination  Incorporate feedback on information that needs to be included into the next
and data use round of data collection, as needed
 Integrate findings into or scale up for inclusion in the national preparedness and
performance monitoring dashboard (e.g. long-term focus)

3.1 Governance and coordination

3.1.1 Define roles and responsibilities

Once the steps for the survey have been identified, it is essential to clearly assign roles and responsibilities. The
survey should usually be undertaken under the overall leadership of the Ministry of Health (MoH). Given the
context of the COVID-19 pandemic, it is recommended that the operational organization should be substantially
simplified compared with typical health facility assessments. Summaries of the roles and responsibilities of the
key parties involved are outlined below.

• Ministry of Health: The MoH is responsible for the overall coordination and steering of the implementation
of the assessment. The MoH should designate a focal point (or points) who will have overall responsibility
for coordinating the process, including relevant national institutes and other key stakeholders. The MoH is
responsible for identifying the implementation team, defining the scope and frequency of the surveys,
guiding the adaptation of the questionnaire (see Section 3.3.1), coordinating the survey, ensuring access to
the master list of facilities and permission from the health facilities, and presenting and ensuring that the
data are used to define and update policies and planning to address continuity in delivering essential health
services and improving responses to the COVID-19 pandemic. If relevant, the MoH may also establish a
survey coordinating group with partners.

• Coordinating group: Depending on the presence and interest of stakeholders and partners, a coordinating
group may be established under the leadership of the MoH and involve key partners (e.g. WHO, the United
Nations Children’s Fund, the World Bank, the Global Financing Facility, the Global Fund). The coordinating
group provides oversight of implementation, provides technical input about tools and protocols, helps to
define the scope and frequency of the surveys, and guides the adaptation of the relevant questionnaire.
When there is not a coordinating group, all of the above responsibilities will be coordinated by the MoH.
Even if a formal group is not established, WHO and other partners can provide input into the review
processes and technical or financial support, or both, and support the interpretation and use of results and
the policy dialogue and planning processes.

12
• Implementation team: The implementation team leads all operational and implementation aspects of
the survey and of data collection. The team comprises a survey manager, a supervisor for data collection
(if different from the survey manager), a team of interviewers, and at least one data manager or analyst.
Consultants can be recruited as interviewers or they may be part of the MoH workforce. In cases in
which an existing national public health or statistical agency is available to collaborate in a timely
manner, data collection and analysis responsibilities can be assigned to that agency.

3.1.2 Define the scope and frequency of the surveys and the mode of data collection

Given that the objective is not to conduct a comprehensive assessment of a health system’s performance, the
recommended approach is to conduct regular telephone interviews, facilitated by online data entry, with
respondents in a sample of sentinel facilities to enable rapid analysis and use of data. This approach enables
near real-time data collection, analysis and use, rapid and safe implementation in an emergency context and a
high return with the use of only limited resources.

The modules have been harmonized across the suite to maximize efficiency during the pandemic, and it is
critical that the surveys have a targeted scope. The MoH and survey coordinating group must select appropriate
modules according to the country’s most urgent needs and priorities. The optimal frequency of data collection
will depend on the course of the pandemic, the anticipated magnitude of changes, and the resources and
capacity available to implement the tools. It is recommended to plan for at least quarterly assessments and
consider more frequent implementation if needed and feasible.

The way in which data will be collected should be confirmed early in the planning process. The assessment tools
are available in both a paper and electronic version. For the purpose of quality control and rapid turnaround of
results during the pandemic, it is suggested that the electronic version is used to facilitate data entry and
management. WHO has a standard electronic form that is available for immediate implementation (currently in
LimeSurvey, a web-based survey platform). The standard form must be customized based on any country-
specific adaptations made to the tool (see Section 3.3.1). Programming skills are not required to modify the
LimeSurvey-based forms.

Currently, the LimeSurvey online tool can be used in two different ways: in a web-based format that is accessible
via any browser or as an application (or app) installed on an Android mobile phone. Internet connectivity is an
important consideration when determining the best option for use in a country. Accessing the web-based
format on a computer makes it easy for interviewers to conduct interviews and enter data, with potentially
fewer errors in data entry. All data are saved as the interview proceeds, but an internet connection is required
to conduct the interview. Thus, in places without a stable internet connection, the web-based form can be used
on a mobile phone that has a data plan. Alternatively, the app on Android telephones can be used, since it does
not require an internet connection during an interview. Interviewers can upload the data saved to their phone
when a connection is available. However, data entry on a mobile phone may require more practice to minimize
data entry errors.

Countries may consider using platforms other than LimeSurvey, depending on which systems are available at the
MoH or a national statistical agency, if involved, and their capacity. In such cases, at a minimum the application
should have a built-in check for internal data consistency that pays particular attention to skip patterns and the
entry of numeric data.

13
3.1.3 Identify the implementation team

Even when data output is as fully automated as possible, it is essential to allocate time and human resources to
interpret and use the data, especially during the first round of the survey. The implementation team should
include the following positions:.

• survey manager to plan and coordinate overall implementation at the country level;

• supervisors (if different from the survey manager) to be responsible for overseeing data collection
teams and to troubleshoot during the data collection phase. Note that small implementation teams may
not require supervisors if all responsibilities can be completed by the survey manager;

• data collectors or interviewers to complete questionnaires during telephone interviews with staff at
facilities or key community informants;

• data manager or analyst (at least one) to manage the national database and real-time analyses of the data.

3.1.4 Develop an overall plan for the survey

An overall operational plan or Gantt chart (a bar chart used for project management) should be developed by
the coordinating group; the plan or chart should clearly outline the key steps and activities described in Table 7.

3.1.5 Identify resource requirements and prepare a budget

A budget should be prepared based on the identified requirements for resources. Multiple, context- specific factors
will determine the budget. If possible, it is recommended that existing resources at the MOH be leveraged to avoid
purchasing new equipment for individual interviewers. The key resource requirements are summarized below.

The technical documents needed to plan the assessment include the:

• master facility list;

• standard modules (to be adapted according to the country’s context);
• standard supporting and training materials (to be adapted as necessary);
• standard data analysis code and code for the automated output of figures and tables in Excel (also referred
to as chartbooks, and to be adapted as necessary).

The human resources required include:

• an assessment manager;
• a supervisor;
• a data manager or analyst(s).

The equipment and infrastructure required for the interviewers includes:

• a telephone and headset or earphone;

• a laptop or tablet computer (note that the form can be completed on a smart phone, but data entry is easier
on a tablet or computer);
• a stable internet connection at a central location or in each interviewer’s home;
• mobile phone top-up credits, as necessary.

14
If calls are made from a central location, additional equipment may be needed to strengthen internet and Wi-Fi
connections (e.g. a router, Wi-Fi signal enhancer). If the internet connection used by laptops or tablets is
unstable, then they can be used with a mobile phone connection.

Human resources are an important budget line item. The number of interviewers will depend on the sample size
and the survey plan. The data collection period for each module should not exceed 7 days, and it is expected
that an interviewer can complete 4–5 interviews in a day. Therefore, it is recommended that 1 interviewer is
recruited for about every 20 facilities. For example, for a sample size of 100 facilities, 5 interviewers will be able
to complete data collection in 5 business days. If multiple modules are implemented, countries may use the
same data collection team to implement all modules (lengthening the timeline) or train different data collection
teams for each module (for more rapid deployment).

The budget for interviewers should cover training (2 days for the first round and 1 day for subsequent rounds),
2–3 days to conduct pre-interview outreach (if interviewers are mobilized for this activity) and the data
collection days. Table 8 provides illustrates how to calculate the human resource requirements.

Table 8. Example of how to calculate the number of interviewers and the budget for implementing
assessments of frontline service readiness for the COVID-19 pandemic

Example No. of interviewers Budget calculation

100 health facilities • Assume that 1 Training:
interviewer can 5 interviewers × [daily rate] × [training days per year]
cover 4 facilities per
day Pre-interview outreach:
• 1 interviewer can 5 interviewers × [daily rate] × [outreach days per year]
cover 20 facilities per
week Data collection:
• 5 interviewers can 5 interviewers × [daily rate] × [5 working days] × [frequency
cover 100 facilities per year]
per week
50 community • Assume that 1 Training:
interviews interviewer can 2 interviewers × [daily rate] × [training days per year]
cover 5 communities
per day Pre-interview outreach:
• 1 interviewer can 2 interviewers × [daily rate] × [outreach days per year]
complete 25
community Data collection:
interviews per week 2 interviewers × [daily rate] × [5 working days] × [frequency
• 2 interviewers can per year]
complete 50
community
interviews per week

15
3.2 Sampling
This section describes the recommended sampling approach for the surveys.

3.2.1 Sampling approach

Depending on the resources available and the context, strategies to select the sentinel facilities will vary across
counties. Generally, a stratified sampling approach is recommended for selecting sentinel sites, if feasible and a
master facility list is available. Within each of the select sampling strata, facilities will be selected randomly with
equal selection probability. Since data are collected via telephone, there are no extra cost implications to
surveying sentinel sites that are geographically dispersed, which is a potential result of random sampling within
a stratum. Stratified sampling will provide representative sentinel facilities for each analysis domain as well as
for the country, adjusted for sampling weight. Observed trends from the sentinel sites may be interpreted to be
representative for all facilities in the country, depending on the acceptable level of precision. 1 If a stratified
sample is planned, sampling weights should be provided in the chartbooks (see Section 5.3) in order to calculate
nationally representative estimates. Further information on calculating sampling weights is available elsewhere
(2).

In settings where the master facility list is incomplete or outdated, alternative approaches may be considered,
for example, selecting facilities known to have high-quality data reporting, targeting districts with poor service
provision prior to the pandemic or utilizing existing sentinel facilities that have been selected for other purposes.
If sentinel sites are selected for convenience, observed trends may not be representative of all facilities in the
domain. Therefore, it is important to document the reasons why sites were selected and communicate to data
users any implications for interpreting the results.

3.2.2 Eligible facilities and sampling frame

For the module on COVID-19 case management capacities, eligible facilities include all those that provide COVID-19
case management, and for the module on the continuity of essential health services, eligible facilities are those
that provide essential health services. Eligible facilities must meet any predefined criteria that are specific to the
country. Ideally, an existing master facility list will serve as a sampling frame. For example, if the aim is to include
facilities across all managing authorities and all types in the country, all facilities in the master list are eligible for
the assessment. However, if the assessment focuses only on public facilities in selected regions – which may be a
strategic option in settings in which a majority of essential health services are provided predominantly in the public
sector – then the eligible facilities are all public facilities in the master list that are in the selected regions.

Depending on the country’s context, any special facilities that typically do not provide a common package of
essential health services (e.g. convalescent hospitals) should be excluded from the sampling frame. Further,
since all sentinel facilities must be accessible via telephone, facilities without valid telephone numbers should
also be excluded from the sampling frame.

WHO does not recommend updating or constructing a master facility list for this assessment during the
pandemic (3). Updating or constructing a master facility list is an important step when conducting conventional
health facility assessments. But it can be a labour- and time-intensive undertaking, and it is unlikely to be
feasible during the pandemic, given the many competing priorities. These assessments rely on the existence of a
master list that is fairly up to date in terms of both the number of facilities, by their characteristics, and their
contact information.

1
Unless the magnitude of changes is substantial, the total sample size is most likely too small to make statistical inferences
about the estimates over time, even at the national level (see Annex 2).

16
3.2.3 Sample size

These assessments aim to monitor signals of disruption to the provision of services among sentinel surveillance
facilities during the COVID-19 pandemic. The assessments emphasize monitoring trends among sentinel sites
rather than making statistical inferences about point estimates, as in typical sample surveys. Thus, although it
is unconventional to do so, sample size can be rather simply determined to provide stable estimates among the
sentinel facilities.

The key domains for analysis should be identified. These may include the facility type (e.g. hospital, primary care
facility, long-term care facility), the managing authority (i.e. public or private), administrative units selected by
COVID-19 transmission status (i.e. regions that are more or less affected by COVID-19) and geographical
distribution (i.e. rural or urban). The number of analysis domains and the number of categories within each
domain will need to be determined strategically based on the implementation resources and timeline to ensure
rapid data collection, analysis and use during the pandemic

Since some indicators are calculated among only a subset of facilities (e.g. reasons for staff absenteeism among
facilities that had staff absence during the preceding 3 months), it is recommended that 35 facilities are included
per analysis domain to ensure that at least 25 facilities are in the denominator for most of the key indicators.
The highest number of strata in a domain determines the total sample size. Table 9 illustrates how sample sizes
are calculated based on the type and number of domains to be included.

If a selected facility cannot be reached after three attempts, a replacement facility should be selected to
maintain the sample size (see Section 3.3.5).

Table 9. Illustration of the total number of sentinel sites needed based on different sets of analysis domains.
Note that the total sample size is determined by the domain that has the highest number of strata (in these
scenarios, they are 3, 4 and 2, from top to bottom). The domains that determined the sample sizes are in bold.

Sampling or analysis domain Highest Calculation Sample size

number of
strata
• Facility location (urban versus rural) Three strata 35 × 3 105
• Managing authority (public versus private) (by facility
• Facility type (hospital, health centre and type)
health post)
• Facility location (urban versus rural) Four strata 35 × 4 140
• Managing authority (public versus private) (by
• Facility type (hospital, health centre and administrative
health post) unit)
• Four administrative units purposively
selected
• Facility location (urban versus rural) Two strata (by 35 × 2 70
• Managing authority (public versus private) facility
• Facility type (only primary care facilities) location and
managing
authority)

17
3.2.4 Sentinel surveillance facilities

Target facilities within each domain should be selected randomly from the master facility list. The assessment is
designed to be repeated over time among the sentinel facilities, thus it is critical to avoid targeting interventions
to the sentinel facilities that are included in the assessment. Thus, it is important that only members of the
implementation team and interviewers are aware of the sites that are included in the assessment and during
follow up.

3.3 Survey planning and preparation

This section outlines the key activities necessary for planning and preparation (Table 7, Step 3). Survey managers
should plan and lead all activities described in this section, and they must be included in discussions about the
survey as early as possible, ideally at the outset of implementation (Box 1). Survey managers should receive and
review all technical assistance documents and tools.

Box 1. Roles and responsibilities of the survey manager

Survey managers are critical for successful implementation. They lead and coordinate all activities
during planning, preparation and data collection (Table 7, Steps 3 and 4). They also participates in
analysing the assessment, disseminating the results and ensuring that the data are used appropriately
(Step 5), in close collaboration with the data manager or analyst. The primary responsibilities of the
survey manager are described below; however, some of the responsibilities can be shared with other
team members, depending on the structure of the implementation team.

Assessment managers:

 coordinate the finalization of country-specific adaptations of interview tools and supporting

documents;
 coordinate the finalization of sampling;
 secure appropriate, functioning equipment and manage it and other resources;
 recruit, train and supervise interviewers;
 conduct or manage outreach to sites before data are collected;
 develop and update the daily call schedule for all interviewers;
 manage day-to-day operations during data collection, including troubleshooting;
 with the data manager, monitor the progress of the assessment and quality of the data;
 participate in interpreting the results;
 participate in disseminating the results and in ensuring the data are used appropriately.

Table 10 illustrates a fairly rapid timeline for key preparation activities. Certain activities can be carried out
earlier than shown. For example, the facility contact list can be developed as soon as sampling is completed.
However, certain activities have prerequisites. When the timeline for the prerequisites changes, the timelines
for subsequent activities and data collection must be adjusted accordingly.

18
Table 10. Illustrative timeline of detailed activities necessary for survey planning and preparation before data
collection for assessing frontline service readiness for the COVID-19 pandemic. Note that the number of days
needed may vary depending on the context. Also, a day does not necessarily refer to a full business day.
Shorter timelines may be possible during follow-up rounds

Activity Day and order of activities

1 2 3 4 5 6 7 8 9 10
Develop tools
Review and adapt modules and training materials to meet X X X X X X
country-specific needs
Update and pilot test online toolsa X X X X X
Recruit and train interviewers
Recruitmenta,b X X X X X X
Training and pilot testing in the field X X
Prepare for interviews
Procure necessary equipment and coordinate logisticsa,b X X X X X X
Develop a facility contact listc X
Conduct outreach to facilities to make them aware of the X X
assessment, confirm respondents and schedule
appointmentsd
Develop an interview schedule, including web links to X
online tools specific to each module and facility
Prepopulate the data for facility background questions in X
the modules with details from the master facility list
Prepare for data analysis and of data productse
Review and update the standard analysis plan and X X X X X X X X
automated outputs according to changes made for
county-specific adaptation of modules
Review and update the standard template for the data X X X X
dashboard
a
This step must be completed prior to training interviewers.
b
Recruitment and procurement may take substantially longer depending on the local regulations and context.
c
This step can start as soon as sampling is completed; however, it must be completed prior to pre-interview outreach.
d
This step can start as soon as the facility contact list is completed if staff are available.
e
These activities are completed by data analysts or managers, with inputs from the survey manager. They are described in
detail in Section 5.

3.3.1 Review and adapt modules and materials

Based on the objectives and scope of the assessment in each country, the modules are selected by the MoH and
survey coordinating group (see Section 3.1.2). The modules include standardized questions and response
options that can be used in different settings. But some questions and response options require country-
specific adaptation to reflect the specific needs of national health systems.

It is recommended that the survey manager makes basic changes to the relevant options (e.g. facility type in the
country, name of the administrative units) prior to engaging a technical group. Technical group members should
be requested to review the standard modules and share questions or suggestions prior to a group meeting.
Technical members should be briefed about (i) the goal and specific aims of the assessment and (ii) the sample

19
design. It is recommend to plan for 1–2 hours of discussion and decision-making for each of the facility modules
and less time for the community module. Standard annotated modules can be used during the group meeting to
provide answers to common questions.

There are four types of adaptations: optional sections can be chosen within each module; words or phrases in the
questions and responses can be customized; optional questions and responses can be chosen; and country-
specific questions can be added.

3.3.1.1 Choosing optional sections within each module

Within each selected module, countries should choose to include or exclude the optional sections. The decision about
these sections should be based on whether the information is available through other data sources, the urgency of the
need for the data and potential programmatic responses that may be taken based on the data. Countries may consider
including all or only certain sections in every round of data collection or only during certain rounds. For example, if a
country has data from strong logistics management information systems, the availability of selected tracer
therapeutics may not be prioritized for the module assessing the continuity of essential health services. Or if a
recent health facility assessment covered, for example, diagnostics readiness or infrastructure, the availability of
diagnostics or infrastructure may not need to be included in the assessment.

The module assessing COVID-19 case management capacities includes the following optional section –

• Section 8: COVID-19 vaccine readiness.

The module assessing the continuity of essential health services includes the following optional sections –

• Section 7: Availability of selected tracer therapeutics

• Section 8: Availability of diagnostics
• Section 9: Vaccine readiness
• Section 10: Facility infrastructure.

In addition, in settings for which repeated assessments are planned, certain mandatory (non-optional) sections may
not need to be repeated in every round, based on the expected rate of change during a short period. For example, in
the module assessing COVID-19 case management capacities, Section 7 (Medical equipment for diagnosis, patient
monitoring and case management) may not need to be repeated quarterly.

3.3.1.2 Customizing phrases or words in the questions and in response options

Each module includes options for country-specific questions and responses that will need adaptation, noted as
“county-specific question adaptation” or “country-specific response adaptation” in the paper questionnaires. Examples
of such tailoring include using the country’s names for administrative units and classifications for different types of
facilities.

3.3.1.3 Choosing optional questions and responses

The modules have what are known as country-specific optional questions. These questions are included in the
standard tools because they may be relevant for multiple countries but not necessarily for all countries. Exclude the
optional questions unless (i) the subject is relevant to the country context and (ii) the sample design already allows for
adequate analysis of the particular question. To facilitate decision-making, modules include footnotes that indicate
which country-specific questions are optional.

20
3.3.1.4 Adding country-specific questions
In addition, questions specific to the country can be added. To protect data quality and given the time frame and
intended scope of the survey, it is recommended that new questions be added only when information is
urgently needed and it will have direct and immediate programmatic implications. Also, adding country-specific
questions requires more preparation by data managers who will need to update and test the analysis code for
the country-specific modules, which is a critical part of finalizing the modules.

3.3.2 Test country-specific tools and update training and supporting materials

Adaptation should first be completed using the paper tool, then the online tool should be updated and tested prior to
interviewer training. This preparation is critical to ensure smooth implementation and good data quality, and it is
recommend that at least 1 full day per module should be allotted for updating the online tool prior to training. If the
country-specific adaptation is extensive, more staff time may be needed.

The survey manager coordinates the pilot testing of the online tool. If feasible, this should include online data entry
and interviews with a few facilities that will not be sampled as surveillance sites. If this is not feasible, then mock
interviews can be organized involving supervisors or those who participated in adapting the questionnaires, or both. In
either case, it is recommended that the tool is tested with different types of facilities and respondents (e.g. urban
versus rural facilities, facilities providing high-level care versus those providing low-level care), so that different skip
patterns can be tested and the range of interview times can be estimated. The survey manager should consolidate
all feedback and make amendments as needed to the tools, processes and logistics. Final adjustments must be
reflected in both the paper and online tools.

Once the modules are finalized, other supporting materials must be updated to reflect the country-specific
changes, including:

• the interviewer training slides. Template slides are provided, including example questions that are likely
common across settings. However, details of logistics and processes must be updated according to the
country’s context;
• annotated questionnaires. The questionnaire has been carefully developed with technical experts, and
all questions have been specifically written to provide as much clarity as possible. Still, there may be
questions about the meaning of certain words or phrases during interviewers’ training, field practice and
even the actual interviews, and it is important to have a consistent approach to addressing these
questions and answers. Annotated questionnaires include an additional column describing frequently
asked questions and the answers to them, as well as images of certain items asked about in the tool.
Survey managers should update these annotations so they can be used as a reference during training
and data collection;
• data management and analysis codes, chartbooks and dashboards. Suggestions for updating data
analysis and data products are discussed in Section 5.

3.3.3 Secure equipment and logistics support

All necessary equipment that was identified during the planning and coordination phases must be procured
prior to training for the interviewers (see Section 3.1.5). How the interviewers will be trained and the venue for
training must be determined and prepared for according to the country’s context.

21
3.3.4 Train the implementation team

The number of interviewers needed will be determined by the sample size, timeline and budget for the
assessment (see Section 3.1.5). The survey manager is responsible for recruiting the interviewers. Once the
implementation team is complete, training sessions must be planned and conducted.

3.3.4.1 Interviewer training

The survey manager organizes and leads training. It is recommended that training is as interactive and engaging
as possible, while the experience of the interviewers and dynamics of the group are also considered. Template
training slides are provided. See Sections 4 for further information.

The recommended contents and required materials for training are shown in Table 11. A structured training
schedule should be developed that includes the recommended contents, which can be rearranged as necessary.

Table 11. Recommended contents and required materials for interviewer training for conducting assessments
of frontline service readiness for the COVID-19 pandemic

Category Description

Contents • Introduction of team and survey, including a brief description of the

facilities to be contacted and respondents to be interviewed
• Conducting interviews
­ Skills for successful interviews
­ Tips on completing the questionnaire
­ Module review, using the paper tool
­ Classroom practice, using the online tool
­ Field practice, using the online tool
• Overview of the data collection schedule
­ Daily schedule, including daily debrief and troubleshooting
­ Using and updating the interviewer call log
Materials • Updated country-specific paper tool (one copy per interviewer)
• Updated country-specific annotated modules
• Updated country-specific online tool
• Web link for accessing the online tool
• Training slides
• List of facilities and respondents recruited for field practice and their
telephone numbers

It is important to review the modules using the paper version, but the review can be tedious for the
interviewers. Try to provide as engaging and creative a training as possible. For example, consider reviewing the
paper questionnaire and practicing using the online tool section by section, rather than reviewing all of the
paper questionnaire at one time. Briefly explain the purpose of each section or break down longer sections into
shorter groups of questions. It is recommended that the facilitator reads the first section to the interviewers;
however, for later sections, to keep the interviewers engaged, the facilitator can invite each interviewer to read
a group of questions.

22
For classroom practice, interviewers should be paired up, choose a hypothetical facility and respondent, and
conduct a mock interview by alternating roles (i.e. respondent and interviewer).

For field practice, facilities and respondents who are not included in the sample should be recruited. For this
part of the training, it is recommended that interviewers are grouped into teams with each team interviewing at
least two facilities per module of two different types, if feasible. Interviewers can take turns to cover different
sections while the rest of team listens to the practice interview. The field practice will highlight aspects of
interviewer training that need to be re-emphasized or addressed. It may also suggest minor changes that can be
made in the tool. Any such adjustments must be reflected in both the paper and online tools, as well as in the
data analysis processes.

3.3.4.2 Data manager training

The survey manager should coordinate or conduct training for the data analysts and managers. Template
training slides are provided. For further information, including standard analysis code and information about
chartbooks, see Section 5. Experienced data managers may study the training materials on their own.

3.3.5 Prepare for data collection

There are a number of additional activities that are essential and must be completed prior to data collection.

3.3.5.1 Make adjustments to and confirm availability of equipment and ensure logistics support
Based on feedback from the training for interviewers, make adjustments to equipment and ensure logistics
support for data collection. For example, if data collection is conducted in a central location, potential feedback
might include the need to improve internet connectivity and to provide quiet space for each interviewer.
Confirm any logistics support that will be provided by groups other than the implementation team (e.g.
information technology, the venue, catering for meals)

3.3.5.2 Develop the facility contact list

The survey manager should develop a facility contact list. The list will be used for pre-interview outreach. The
contact list will include the sentinel facilities to be sampled and selected background information (Fig. 1). At a
minimum, it should include three pieces of information: a unique identification number for the facility from the
master list (or sampling frame), the facility’s name and the telephone number. It is recommended that
additional background information is included, such as the first-level administrative unit, facility type and
address. This additional information can be used to verify the sampled facilities during the pre-interview
outreach, and it can be prepopulated into the online tool, saving the interviewer time.

23
Fig. 1. Example of a facility contact list with the results of pre-interview outreach

Facility contact information Pre-interview outreach

Facility Name Region Facility Address Telephone Outreach Main Main Date and Note
identifica type number call date respondent respondent time of
tion telephone appointment
number number

135 AAA XX Health … … 6/12/2020 J Mwangi … Not Facility could

centre applicable not be
reached after
three calls.
Replacement
facility
needed.

890 BB XX Hospital … … 9/12/2020 P Baressi … 17/12/2020

10 AM

159 CC YY Health … … 7/12/2020 C Park … 18/12/2020

centre 12 PM

456 DD YY Health … … 7/12/2020 J Smith … 18/12/2020

centre 2 PM

567 EE ZZ Health … … 7/12/2020 T Gichangi … 18/12/2020

centre 4 PM

3.3.5.3 Pre-interview outreach (only for facility assessments)

Reaching out prior to data collection to facilities that will be sampled is critical for four reasons. First,
operationally the pre-interview call will identify a main respondent to be interviewed, obtain a telephone
number for the respondent (if different from the facility’s telephone number in the master list) and determine a
preferred date and time for the interview (Fig. 1). It is recommended that interview appointments are made
largely during the first two thirds of the data collection period, since some appointments may need to be
rescheduled. Also, sufficient time should be allowed for each interview based on the results of pilot testing and
classroom and field practices. Second, pre-interview outreach is also an opportunity for identifying any facilities
that will be difficult to include in the assessment (Fig. 1). It is recommended that a facility is called three times
(on different days and at different times). If the facility cannot be reached, it should be replaced with another
eligible facility.

Third, if the module on the continuity of essential health services is being used, another important purpose of
the outreach is to alert respondents that data about service volume and trends will be collected during the
interview. This is the only part of the tool that requires respondents to answer based on information in
documents – that is, patient registries in the facility or data provided for health management information
system reporting. Before the interview, respondents should be sent the appropriate form either via mail or
electronically, and they should be encouraged to complete it prior to the interview.

Finally, in large facilities, different sections of the tool may require information from different persons. Consider
sending a copy of the tool to the main respondents so that they can consult the relevant staff as needed.

3.3.5.4 Develop a data collection schedule

Based on the results of outreach, a data collection schedule should be developed and monitored. Then facilities and
appointments should be systematically assigned to interviewers based on the appointment dates and times

24
arranged during outreach and the expected length of interviews. Again, the schedule should allow sufficient time for
each interview, depending on the module and facility type. It is recommended that the data collection period for
each module does not exceed 7 days, and it is expected that an interviewer can complete 4–5 interviews in a day.

The schedule may need to be updated regularly as a result of rescheduled interviews, dropped calls or
interviews that last longer than their allocated time. It is recommended that the same interviewer is assigned to
complete any interviews on their schedule that were missed.

3.3.5.5 Develop a daily call schedule for interviewers (the interviewer call log)
Each day, interviewers will receive a list of facilities to contact and respondents to interview at prearranged
appointment times, and this is referred to as the daily interviewer call log (Fig. 2). The log provides interviewers
with all of the necessary contact information. Interviewers will also complete the final column of the log, where
they record the results of each call (see Section 4.4). If the community module is also used, the call log must
include a list of key community informants and their telephone numbers, although there will be not be a
prearranged interview date or time.

If LimeSurvey is used, the log will also include a web link that is specific to each sentinel facility. The same web
link will be used when an interviewer calls a facility for follow up, if needed, to complete the interview.

Fig. 2. Example of daily interviewer call log for facility assessment

Facility Facility Facility Facility Facility Outreach Main Main Date and Note Interviewer Call results
identific name type address telephone call date respondent respondent time of
ation number telephone appointment
number number

890 BB Hospital … … 9/12/2020 P Baressi … 18/12/2020 Partially A

11 AM completed on
17/12/2020.
Rescheduled.
Start again from
Section 7.

159 CC Health … … 7/12/2020 C Park … 18/12/2020 … A

centre 12 PM

456 DD Health … … 7/12/2020 J Smith … 18/12/2020 … A

centre 2 PM

567 EE Health … … 7/12/2020 T Gichangi … 18/12/2020 … A

centre 4 PM

3.3.5.6 Prepopulate the background information about the facility

If feasible, background information about the facility can be prepopulated into the online tool to save time
during the interview. Information that can be prepopulated includes the facility’s name, identification number
and the administrative unit level, all of which are available from the master facility list. Prepopulating these
responses can improve the efficiency of the interviews if such information is up to date and can be extracted
readily from the master facility list.

25
4. Guide for interviewers
The findings of the assessment are only as good as the data from which they are calculated, and data quality
relies to a large extent on the interviewers. There are core interview skills that are helpful during any kind of
interview, such as interacting with respect and neutrality. Telephone interviews, however, require additional
skills as well as practice to address the specific challenges posed by this type of interview. Below are some basic
instructions for the training and practice.

4.1 Interview skills

Interviewers should be encouraged to develop the following skills. The instructions in the list below may be
helpful.

• Build rapport over the telephone. Greet the respondent in a confident, sincere and friendly manner. A
respondent's first impression of the interviewer will strongly affect their willingness to fully participate in
the interview. All respondents should be treated respectfully and politely. The respondents should know
that their cooperation and the time they are taking to help make the survey successful are appreciated.

• Read the questions verbatim and with the appropriate emphasis. Respondents only hear the questions
and possible responses during the interview (they do not see them), so reading the questionnaire
exactly as written is critical and requires concentration. The wording of each question and the options
for responses has been carefully chosen and, for that reason, it is essential that the interviewer reads
each question to the respondent exactly as it is written and in the same manner to all respondents.

At times, the interviewer may need to repeat a question to be sure that the respondent understands it.
In these cases, the interviewer should not paraphrase the question but repeat it exactly as it is written.
During practice sessions, if interviewers find that they have to repeat certain questions, supervisors and
assessment managers should be made aware so that the wording can be changed, if necessary.

• Speak clearly and slowly. Pay attention to how fast you are speaking, especially when reading an
introduction and throughout the interview. Nervousness may cause interviewers to read too fast or
inflect up at the end of sentences that are not questions, thus making them sound like questions. Also,
pay attention to the position of the mouthpiece. If it is too close to the interviewer’s mouth,
respondents may become distracted by the sound of breathing.

• Control the tempo and tenor of the engagement, subtly but firmly. This is especially important if the
respondent is confused, frustrated or rushed, which may be expressed verbally or nonverbally. For
example, if a respondent starts answering quickly, it can be a sign that they are growing impatient,
bored or annoyed.

• Maintain neutrality. Refrain from making conversational noises that may be interpreted by the
respondent as approval or disapproval of their answers. Such misinterpretation can lead to unintended
biases, as respondents may seek an interviewer’s approval. Also, interviewers should not read the
response options in such a way that may convey their own opinions or thoughts. Interviewers must not
express surprise or astonishment at any answer.

26
 • Listen closely. Listening carefully is as important as asking the questions. Some questions require the
interviewer to listen to what the respondent says and then record the answers by clicking the
corresponding boxes for all applicable categories. Interviewers should not rush into assigning categories
before the respondent has finished replying completely, otherwise interviewers are at risk of attributing
their own biases to the respondent’s replies.

Occasionally, a respondent may answer a question incompletely or seem to have misunderstood the
question. The first step is simply to repeat the question as written. If this does not help, the interviewer
needs to ask for further information without influencing the response – that is, probe for a response. For
example, the interviewer might ask, “Could you explain that a little more?” or “Could you be more
specific about that?” The interviewer must never interpret a respondent's answer and then ask the
respondent if the interpretation is correct.

Finally, the interviewer must listen closely to determine when a respondent is not paying attention,
which may be signalled when respondents rush, say “I don’t know,” give inconsistent or irrelevant
answers, or ask for questions to be repeated. This happens most often when respondents are concerned
about their other responsibilities and want to return to work. In these cases, the interviewer must try to
re-engage the respondent’s interest in the conversation. For example, if the interviewer senses that the
respondent is growing restless, the respondent could be reassured that there are not many more
questions and that their responses are valuable.

• Enunciate clearly. Ensure that all words in the questionnaire, including medical terminology, can be read
clearly and confidently.

• Record information accurately. Never leave a response blank: record responses as they are given
whether negative or positive, including “no”, “not applicable” and “I don’t know.”

Always verify the typed answers on the screen. Sometimes the respondent may appear to understand
the question, but the response may be deemed inconsistent: in this case, the interviewer must record
the response as given by the respondent.

• Thank respondents. Ensure that respondents know that their efforts and time are appreciated. This
should be acknowledged at least at the beginning and end of the survey, and possibly during the survey
as well, if appropriate.

4.2 Obtaining consent

Interviewers are required to obtain consent to participate from each respondent before starting to ask
questions. The consent form, which includes background information for the survey, is located at the start of the
tool.

4.3 Completing the questionnaire

Each module has multiple sections. Each section has instructions as well as questions.
4.3.1 Instructions

The tool has instructions for respondents and for interviewers.

• There is information that introduces a section or a set of questions. These are not questions but
interviewers need to read them to respondents.

27
The following types of instructions are aimed only at interviewers and should not be read to respondents.
• In a small number of open-ended questions, there is a specific instruction for interviewers not to read the
response options aloud.
• In a small number of cases, interviewers are required to record information without asking a question, such
as the date of the interview and the interview results.
• Footnotes that spell out acronyms are provided for the interviewer’s information.
• Skip patterns refers to certain questions that are asked only to selected respondents based on their previous
answers. In the paper form, these are presented in a grey background. In the electronic form, the skip
pattern is built in, and interviewers (or respondents in the case of a self-administered assessment) do not
see the skip instructions. However, it is critical for interviewers to understand the skip pattern to ensure the
correct flow of the survey.

4.3.2 Questions

There are three major types of questions in the tool. This section describes them and provides examples, using
questions and response options in a web-based online form. However, please note that the examples and the
question numbers may not align with those of a specific module used in a particular country.

4.3.2.1 Precoded responses: single response

The most common type of question asks the respondent to select only one of the precoded responses. In the
online interview, the response options will have small circles (radio buttons), and only one can be selected for
each question (Fig. 3).
Fig. 3. Example of a single response question that uses radio buttons (option buttons)

Sometimes responses must be entered in the response grid (or table). When recording a response in one of
these grids, the interviewer must be sure that the answer is entered in the correct row and column (Fig. 4).

28
Fig. 4. Example of a response grid in which the response must be entered in a particular column

In a few cases, a precoded question will include an “Other” category. “Other” should be selected when the
answer provided is not included in the precoded responses. Answers that fall into this category are uncommon
responses and, in most cases, respondents are not asked to explain their answer further, as shown in Fig. 3. For
a small number of questions, however, interviewers are asked to specify to what Other refers (Fig. 5).

Fig. 5. Example of an answer that requires a respondent and interviewer to provide information about a
response coded as Other

Only a small number of questions have a response option for “Do not know”. If “Do not know” is not one of the
options, then interviewers must probe further.

4.3.2.2 Precoded responses: multiple responses

For a small number of questions, interviewers need to listen to the respondent and select all applicable answers
from among the precoded responses. Do not read any of the precoded responses to the respondent. After
respondents complete their answer, ask “Any other reasons?” or “Anything else?”, as written in the
questionnaire. Again, interviewers should not rush into selecting categories before they have finished listening
to the respondent to avoid injecting their own biases into the responses.

29
In the electronic form, these questions have small boxes that allow multiple responses to be selected instead of
radio buttons (Fig. 6).

Fig. 6. Example of a question that allows multiple responses to be selected and has instructions for
interviewers

4.3.2.3 Numeric responses

Several questions require a numeric response. These should be recorded in the appropriate spaces in the right
column of the table (Fig. 7). There are preset ranges of allowed numbers (e.g. a positive integer for the number of
beds); interviewers will get an error message if the numeric entry is outside the range.

Fig. 7. Example of a question requiring a number to be entered into the form

In addition, at the end of each call, the result of the interview must be recorded by selecting one of the
predetermined categories (Fig. 8). This information is critical both for monitoring survey progress and for
analyses.

30
Fig. 8. Example of categories for recording the result of the interview

4.4 Using and updating the interviewer call log

Data collection is conducted during a short period of time, and the survey team prepares a call schedule prior to
the start of the data collection process. Each day, interviewers will receive a list of respondents with their
appointment times, known as the interviewer call log. The log will include basic information about the facilities,
the date and time of interview appointments and any relevant notes about the facility. The notes may include
information about previous attempts to contact the facility or partially completed interviews. See Fig. 2 for an
example.

After each call, interviewers must record the result both in the tool (Fig. 8) and the call log (Fig. 2). This applies
to all kinds of calls, including completed interviews, rescheduled interviews, dropped calls and unsuccessful
attempts to reach respondents. If calls are rescheduled, new dates and times must be provided in the log. In
addition, for rescheduled or dropped calls interviewers need to record the specific section or question from
where the new call should start. This information will be used when the respondent is called again.

At the end of each day, the survey manager will review the logs from all interviews and then create new call logs
for the next day, based on the updated interview schedule.

4.5 Troubleshooting
Occasionally, interviewers may need to skip a section of the survey for logistical reasons. In a large facility,
the main respondent may need to consult other staff for information about a particular section, and it may
be more efficient for interviewers to skip the section and continue the interview with questions that the
main respondent can answer. In such cases, interviewers must ensure that the skipped section is completed
either with other staff at the facility or with the main respondent after consultation with the relevant staff.

Dropped calls are a common and critical challenge in telephone surveys, particularly in low-resource
settings. Interviewers must call back immediately after a call is dropped. If the respondent cannot be
reached on the same day, the interviewer should call back on the next business day. In addition, while
interviewers are strongly encouraged to complete the survey during one call, respondents may request to

31
be called back at another time or day for reasons that are beyond their control. The interviewer should
acknowledge the time they have spent and their efforts, and respectfully accommodate the request.
Interviewers should try to schedule a call back on the same day or the next day.

To resume a rescheduled or dropped call, the interviewer should first open the questionnaire for the specific
facility, go to the section or page where the call will start from (determined by reviewing notes in the call
log) and then call the respondent to resume the interview.

32
5. Guide for data managers
Data managers are an integral part of the implementation team. An important aspect of the assessment is
ensuring rapid data analysis and creation of data products, such as chartbooks and dashboards, to facilitate the
timely use of data during the pandemic. Data managers supervise and adjust the data systems used in the
assessment, ensuring that analyses are accurate and timely and that chartbooks (the automated figures and
tables that are output in Excel) and the assessment dashboard are updated daily, thus providing results in near
real time (Fig. 9). At the end of the data collection period, the assessment results should be available for
immediate review by the survey manager and technical staff. Data managers should support the review and
update the analyses and results, as needed (Box 2).

Figure 9. Data and analysis flow for assessments of frontline service readiness during the COVID-19 pandemic

A data manager must join the implementation team during the planning and preparation processes for the
assessment (Table 7, Step 3). Specifically, during the process of adapting modules to the country’s context, the
data manager will revise the standard analysis code and code for the chartbooks to reflect the adaptations. At
the end of each day of the data collection period, the data manager will update the chartbooks and run the
quality check.

The technical assistance package for data managers includes:

• standard modules;
• data management and analysis codes for two statistical programmes (Stata and RStudio);
• mock data for practice testing;
• standard automated output of figures and tables in Excel (the chartbooks).

The following sections describe the standard tools that will be used by data managers to ensure the timely flow
of data for the assessment (Fig. 9).

33
 Box 2. Roles and responsibilities of the data manager for assessments of frontline service readiness
during the COVID-19 pandemic
 Update and test the analysis code and update and test the chartbooks (automated outputs of
figures and tables in Excel)
 Daily: download and clean data, and produce field-check tables
 Create an analysis data set
 Calculate estimates for key indicators
 Update chartbooks and compile results from the assessment
 With the survey manager, interpret the results
 Assist the survey manager in disseminating the results and using the data appropriately to
achieve necessary changes

5.1 Data download, cleaning, management and analysis

Code for downloading, cleaning and analysing the data is provided for two statistical software packages: Stata
and RStudio. Both produce the same outputs, although there are minor differences in the order of some steps,
and data managers should choose one package, depending on their experience and preferences. Table 12
presents the table of contents for the standard analysis code and notes to facilitate adaptation by data
managers.

In summary, the analysis code will:

• import and clean data sets downloaded from LimeSurvey;

• create field-check tables to monitor survey progress and data quality;
• create indicator variables in the data set; and
• calculate estimates for indicators

In addition, to facilitate further data analysis and transparency, the analysis code exports three different types of
data to Excel chartbooks (Fig. 9; see also Section 5.3):

• a de-identified raw data set at the facility or respondent level, downloaded from LimeSurvey (raw data
in Fig. 9);
• a de-identified, cleaned data set with additional analytical variables at the facility or respondent level
(clean data in Fig. 9);
• estimates of indicators (i.e. results at the overall and analysis domain levels) (summary indicator data in
Fig. 9). This summary data set (also referred to as the purple tab in the chartbook, Section 5.3) is also
used for the dashboard (Section 5.4).

34
Table 12. Contents of the analysis code and possible country-specific adaptations for assessments of frontline
service readiness during the COVID-19 pandemic

Section Adaptation
A. Setting Required actions
• Set working directories in accordance with the
data manager’s local computer settings
• Set local macros for the country’s name, and
survey round, year and month, and the
LimeSurvey identification number
B. Import and drop duplicate cases
B.1. Import raw data from LimeSurvey None
B.2. Export and save the data daily in CSV None
format with the date
B.3. Export the data to the chartbook None
B.4. Assess and drop duplicate cases None
C. Data cleaning
C.1. Change variable names to lowercase None
C.2. Change variable names to make them None
coding friendly
C.3. Assess nonnumeric variables and, as None required unless new country-specific
relevant, change the format from string or questions are added
character to numeric
C.4. Recode yes/no and yes/no/NA answers None required unless new country-specific
questions are added
C.5. Label values None required unless new country-specific
questions are added

D. Create field-check tables None required, but more field-check tables can be
added as needed
E. Create analytical variables
E.1. Country-specific code: local Required actions
• Set local macro for survey implementation and
Section 1 of all modules
E.2. Construct analysis variables Required actions
The data manager must review this section
carefully and revise it based on any country-specific
changes made to the modules, if needed
COVID-19 case management capacities module
• Standard questions: review and update to
reflect any changes made to response options
• Include new country-specific questions, if any
Continuity of essential health services module
• Update the number of staff and COVID
infection variables

35
 • Other standard questions: review and update
to reflect any changes made to response
options
• Include new country-specific questions, if any
Community needs, perceptions and demand
module
• Standard questions: review and update to
reflect any changes made to response options
• Include new country-specific questions, if any
E.3. Merge with sampling weighta Required actions
• Choose one of the two options based on the
sample design used in the country
E.4. Export clean respondent-level data to None
the chartbook
F. Create and export indicator estimate data
F.1. Calculate estimates None
F.2. Export indicator estimate data to the None
chartbook
CSV: comma-separated values.
a
This step is performed only if relevant based on the sample design for facility assessment. It is not relevant for
interviews with key community informants.

5.2 Field-check tables

The analysis code also creates an Excel file with cumulative summary information about the interviews, known
as field-check tables (Table 12). The standard field-check tables include the:

• total number of facilities and respondents, by interview result;

• length of each interview;
• percentage of interviews that have missing responses for selected sets of variables.

The field-check tables allow data managers to monitor the progress of the survey and interviewers’ performance
as a group over time. For example, for the group of questions assessing the availability of PPE, the percentage of
missing responses would ideally be zero. The data manager and survey manager should review the results
together, and the survey manager should use the information to provide feedback to the interviewers. Although
data collection is designed to be completed during a short period of time, it is important to monitor and support
interviewers as much as possible and help them improve their performance if necessary.

5.3 Chartbooks
During each day of data collection, the data should be analysed, with updated results presented in Excel
chartbooks. Chartbooks provide the most comprehensive presentation of data generated from the assessment
(Fig. 10). Their Excel format allows most users to review and explore the results without needing to use other
programmes. Chartbooks also include the de-identified data set for those who want to pursue further analysis
beyond estimating predefined indicators (facility-level cleaned data in Fig. 11).

36
Fig. 10. Example of a chartbook cover page

37
Fig. 11. Different worksheets in a chartbook: required updates and automatically generated figures and tables

Fig. 11 shows that there are three areas that require user modifications: the instructions (red text); weighing, if
applicable (orange tab); and country-specific information pertaining to the cover, acknowledgements and
methodology (dark grey tabs). The indicator estimates (purple tab) are automatically updated by the Stata or
RStudio code based on the comma-separated values (CSV) file that is downloaded daily. Finally, all standard
tables and figures in the light grey tabs are automatically updated based on the indicator estimates (purple tab,
exported from the data analysis code).

Importantly, tables and figures in the light grey tabs can be modified to meet the specific needs of readers in
different contexts. Fig. 12 displays a sample summary results tab, which is automatically updated. Each section
of the survey maps to a separate summary tab, and each summary has five parts:

(1) prepopulated information with details about the module and section, including the key tracer indicators;
(2) figures for each of the key tracer indicators. These figures are generated from the data tables in part 4
(Fig. 12) and are automatically updated daily based on the data that are downloaded;
(3) key results. These need to be updated manually by the data manager or survey manager based on the
results seen in the figures;
(4) tables that summarize the data and populate the figures. These are automatically updated daily based
on the data that is downloaded;
(5) detailed results. These results are automatically updated daily based on the data that is downloaded.
Users can modify or add new figures as needed.

38
Fig. 12. An example of the five parts of a summary results tab in a chartbook

39
5.4 Dashboard
The dashboard presents information about select key indicators from the assessment (Fig. 13). The underlying
database for the dashboard is the summary estimate of indicator data (a CSV file, as described in Fig. 9). Thus,
the results for key indicators are identical between the chartbooks and the dashboard. However, customization
of the dashboard is limited, in contrast to the chartbooks.

When users visit the dashboard, they see the most up to date summary of indicator estimates that have been
generated and uploaded to a designated server and folder by the data manager. Footnotes in the dashboard
indicate when the database was most recently updated. The dashboard is currently hosted on a WHO server,
but it can be hosted on a country’s server, after installation of the bundles required for visualization.

Fig. 13. Example of a dashboard generated for assessments of frontline service readiness during the COVID-19
pandemic

40
6. Data use and action plan for system strengthening
After analysing the data from the frontline services readiness assessment, the findings should be put to use to
address weaknesses in the health system. An action plan for system strengthening should be developed to
prioritize and guide appropriate interventions. This section discusses general guidance on developing an action
plan, 2 and it can be adapted in the context of the pandemic and repeated, rapid telephone surveys.

6.1 Purpose
The purpose of the action plan for system strengthening is to outline the steps and inputs required to address
the causes of gaps in health care found during the readiness assessment. Needs should be identified and
prioritized, and interventions should be developed and costed to address those needs. Mechanisms for
monitoring and coordination should be identified or created to ensure that the interventions are implemented
in good time and within the allotted budget. The goal of the plan is to improve health service delivery and the
performance of the health system. To meet these goals, the plan should provide specific and practical actions
that when implemented will improve the delivery of health services.

6.1.1 Best practices for developing an action plan for system strengthening

The best practices for developing an action plan are summarized below.

• The development and implementation of the action plan for system strengthening should be led by the
MoH or other government ministry responsible for managing the health system.
• The improvement plan should be developed in collaboration with important stakeholders, such as
donors, partners and nongovernmental organizations, to ensure that consensus and stakeholder buy-in
are achieved.
• The activities and interventions in the improvement plan should be relevant to the country’s context
and address the priorities of the country or organization, including its subunits.
• The interventions should build on what already exists and be feasible and appropriate for the context of
the health system and its workforce.
• The activities and interventions should promote and facilitate the sustainability of the health system, so
that the system can satisfy the needs of the present and evolve as those needs change.

6.2 Action plan for system strengthening

6.2.1 Engage stakeholders

To ensure optimum development and implementation of the action plan for system strengthening, important
stakeholders should be encouraged to participate. Being part of the development process will ensure that
stakeholders are invested in the success of the plan and help ensure their continued support and buy-in. The
interests, requirements and priorities of stakeholders should be understood, as well as their capacity to commit
resources to ensure success. There are likely many stakeholders, and not all will need to be involved. Sometimes
having too many stakeholders can inhibit the development of a responsive plan. Know the stakeholders and
choose them strategically – that is, choose those who give the action plan the best chances for success.
Stakeholders can help advocate for necessary changes and mobilize resources to assist with implementation.

A stakeholder engagement matrix can help identify organizations and individuals who have an interest in
improving the information system. The matrix (Table 13) can help identify the organizations, people and groups

2
This section is adapted from reference 4.

41
who are the stakeholders in the process of health system strengthening, as contributors, influencers or
beneficiaries. The matrix is a structured way to define the roles that stakeholders will play in the activity and
assess the resources they can bring to it. It also provides a framework for assessing stakeholders’ interests,
knowledge, positions, alliances, resources, power and importance. For example, who will resist the initiative?
Who will support it? What are their reasons? The matrix helps assess which stakeholders to include in the
process by determining their relative importance. Identifying and engaging with relevant stakeholders
contributes to developing an improvement plan that meets everybody’s expectations and needs.

42
Table 13. Example of a stakeholder engagement matrix that can be used to evaluate which stakeholders should be included in the process of developing a plan for health system
strengthening

Name of Stakeholder Potential role Level of Level of Resources Constraints Engagement Follow-up
stakeholder description in the issue or knowledge of commitment available strategy strategy
organization, activity the issue
group or
individual

(Are they (What is their (What is their (What are their (Will they (Could they (What are their (How will this (What are the
national, primary interest in the specific areas support or provide staff, limitations? stakeholder be plans for
regional or purpose, activity?) of expertise?) oppose the volunteers, Are funds engaged in the feedback or
local?) affiliation, activity, to money, needed to activity?) continued
funding?) what extent, technology, participate? Is involvement?)
and why?) information, there a lack of
influence?) personnel or
political or
other
barriers?)
Government sector

Political sector

Commercial sector

Nongovernmental sector

Other civil society organization

43
Donors and partners

44
6.2.2 Review the results of the assessment of service readiness

Conducting a formal review of the results of the assessment is a good way to understand and prioritize how to
address any problems with data quality that were identified, discuss potential solutions, prioritize
recommendations and prepare a realistic action plan. To encourage and promote ownership of the assessment
results, it is recommended to begin by conducting an internal review with the MoH, followed by a review in a
workshop setting with a broader group of participants. In these phases of reviewing and analysing the
assessment’s results, ensure that the designated participants from the MoH and other stakeholder organizations
have the ability to analyse the the findings of the assessment and are knowledgeable about the country’s
context and health system and, therefore, have the ability to recommend appropriate actions for system
strengthening. To ensure a productive review workshop, the assessment report, charts, graphs and other
reading materials (the information products from the assessment) should be given to the participants in
advance, so they can prepare. To achieve the workshop’s expected outcomes, facilitators who have the skills to
keep participants focused will also need to be identified.

The review workshop can be combined with the action-planning phase. If action planning is part of the
workshop, be sure that the participants have the authority to make decisions. Alternatively, in the first part of
the workshop, health system or health programme experts can review and validate the quality and relevance of
the assessment results and prepare summaries and presentations for the decision-makers. In the second part of
the workshop, the relevant decision-makers can join the health system experts, be briefed on the results and
the recommendations, and contribute to identifying actions and interventions that can be taken to address the
findings. They can also help define the timelines, responsible persons and organizations, and required resources.

For both the review and the formulation of relevant recommendations to be effective, it will be important to
conduct discussions in small groups. The groups should have equal representation from the following categories
of participants:

• health policymakers;
• health programme managers;
• data managers and specialists in monitoring and evaluation;
• health providers.

Each group may be composed of and have tasks assigned according to:

• health programme area (e.g. maternal health, child health, HIV/AIDS, tuberculosis, malaria);
• level of the health system (e.g. national, regional, district, health facility or community health care);
• type of health facility (e.g. hospital, health centre).

A plenary session should follow the group discussions to enable all participants to provide feedback and input on
all groups’ ideas and proposals and to allow participants to learn from one another.

If the results review and the formulation of recommendations are conducted separately from the action-
planning session, the results and recommendations should be disseminated to the relevant decision-makers to
inform them and guide them in identifying the appropriate strategies and actions for strengthening the health
system.

45
6.2.3 Action planning

The process of developing an action plan for system strengthening follows the review and discussion of the
assessment results and recommendations, as well as the identification and prioritization of strategies to achieve
improved service delivery in the health system.

Similar to the earlier part of the process, the planning process requires effective facilitation to ensure the
development of an action plan that describes specific, measurable, achievable, relevant and time-bound (known
as SMART) objectives and activities; the responsibility for the implementation of each activity should be
assigned to a specific person or organization.

6.2.3.1 Prioritizing interventions for system strengthening

When formulating recommendations and developing the action plan, it is important to prioritize those activities
that will lead to the greatest improvement in service delivery with the resources available or for which resources
can be mobilized. The sustainability of the interventions should also be considered when prioritizing activities
for improving service delivery.

Participants in the action planning session can use a prioritization matrix (Table 14) to score the proposed
activities based on their expected impact on health service delivery and on the ability of the organization and
stakeholders to implement the activities. The scores help prioritize the interventions that are the most feasible
and likely to yield the greatest results.

The prioritization exercise is conducted through a consensus process. Participants agree on the level of impact
that each recommended intervention will have and the ability of the stakeholders to implement it while also
considering the available resources (e.g. human, financial, information and communication technology).
Participants can work in small groups to discuss and complete the matrix and then come together in a plenary
session to produce one completed and mutually agreed matrix.

The prioritization matrix is arranged with a scale for impact on the vertical axis and, on the horizontal axis, a
scale for the ability to implement an intervention and the required level of investment (e.g. human and financial
resources, effort and time). Each axis is divided into four scores: 1 represents the lowest score for the attribute
and 4 represents the highest. The interventions with the most impact, that are the easiest to implement and
that require minimal investment are put in cells in the the top right of the matrix, and the interventions with the
least impact and that are least feasible (i.e. require a high level of human or financial resources or efforts) are
put in the cells in the bottom left of the matrix.

46
Table 14. Example of an intervention prioritization matrix that can be used to rate activities proposed for health system strengthening

High Impact 4

1
Low Impact
1 2 3 4

HARD to implement EASY (high ability)

(low ability)

Depending on the context, using this matrix helps distinguish the relevant interventions that are easy or
relatively easy to implement and that produce moderate to high impact from those that are less feasible or yield
only a low impact.

Once the interventions have been identified, they should be broken down into well-defined subactivities so that
a person or organization can be assigned to be responsible for implementation and funding. Table 15 provides
an example of how a main intervention can be split into subactivities that result in improvements in health
service delivery.

6.2.3.2 Scheduling and budgeting activities

The purpose of scheduling and budgeting is to elaborate the overall action plan for system strengthening,
thereby providing a roadmap for the activities supporting each recommended intervention. Understanding the
effort required to implement each recommended intervention allows participants in the action-planning
workshop to split activities into steps and accurately estimate the resources and time required for
implementation. Aligning the activities with the resources they require makes it possible to estimate the costs of
efforts to improve health service delivery, determine how much time is required and the timetable for
implementation.

47
Table 15. Example of a table showing how a primary intervention can be split into subactivities, for example, for scheduling and
budgeting

Priority Objective Activities Short Medium Long Responsible Supporting Budget

actions term term term entity partner
Intervention 1
Action 1 1 Activity 1 X
Activity 2 X
Activity 3 X
Action 2 2 X
X
X

6.2.4 Monitoring and follow up

Monitoring and evaluation helps measure performance and assess the impact of different strategies,
interventions and inputs on the efforts to improve health service delivery. The results of monitoring and
evaluation contribute to the learning experience and help decision-makers improve the interventions.

The action plan itself serves as a monitoring tool that can be used to follow up the implementation of the
interventions and activities it stipulates. Moreover, the use of supervisory checklists can help track the progress
made in improving health service delivery at the facility level. For evaluation purposes, the next scheduled
implementation of the service readiness assessment can be used to measure the success of interventions.
Regularly reviewing the implementation of the action plan and monitoring findings helps stakeholders identify
when mid-course corrections are needed.

48
7. References

1. Maintaining essential health services: operational guidance for the COVID-19 context. Geneva: World
Health Organization; 2020 (interim guidance, 1 June 2020;
https://apps.who.int/iris/handle/10665/332240, accessed 15 March 2021).
2. Service availability and readiness assessment (SARA): an annual monitoring system for service delivery.
Reference manual, version 2.2, revised July 2015. Geneva:World Health Organization; 2015
(https://apps.who.int/iris/handle/10665/149025, accessed 15 March 2021).
3. Master facility list resource package: guidance for countries wanting to strengthen their master facility list.
Geneva: World Health Organization; 2018 (https://apps.who.int/iris/handle/10665/326848, accessed 15
March 2021).
4. Performance of Routine Information System Management (PRISM) user’s kit: moving from assessment to
action. Chapel Hill (NC): MEASURE Evaluation; 2018
(https://www.measureevaluation.org/resources/tools/health-information-systems/prism/performance-of-
routine-information-system-management-prism-users-kit-moving-from-assessment-to-action/view,
accessed 15 March 2021).

49
8. Annex 1. Suite of health service capacity assessment modules
On 30 January 2020, the Director-General of the World Health Organization (WHO), declared the COVID-19
outbreak to be a global public health emergency of international concern under the International Health
Regulations (2005). Following the spread of COVID-19 cases in many countries across continents, COVID-19 was
characterized as a pandemic on 11 March 2020 by the Director-General, upon the advice of the International
Health Regulations (2005) Emergency Committee.

In response to this situation, the Suite of health service capacity assessments in the context of the COVID-19
pandemic has been developed to support rapid and accurate assessments of the current, surge and future
capacities of health facilities throughout the different phases of the pandemic (Table A1.1).

Table A1.1. The WHO suite of tools for assessing health service capacity in the context of the COVID-19
pandemic

Module or tool Use

Health facility and community assessment tools (core modules)

1 COVID-19 case management To assess present and surge capacities for treating
capacities: diagnostics, therapeutics, patients with COVID-19 in health facilities
vaccine readiness and other health
products -facility assessment tool

2 Continuity of essential health services: To assess health facility and health workforce capacities
facility assessment tool to continue to safely provide essential health services

3 Community needs, perceptions and To assess community needs, changes in care-seeking

demand: community assessment tool behaviours and barriers to accessing care

In-depth assessment tools and modules

4 Rapid hospital readiness checklist To assess overall hospital readiness in order to identify
priority actions to prepare for, be ready for and respond
to COVID-19

5 Biomedical equipment for COVID-19 To conduct an in-depth facility inventory of biomedical

case management – inventory tool equipment reallocation and procurement, and planning
measures for COVID-19

6 Ensuring a safe environment for To assess the structural capacities of health facilities to
patients and staff in COVID-19 health allow for safe service delivery and enable planning for
care facilities surge capacity

7 Infection prevention and control To assess infection prevention and control capacities to
health care facility response for respond to COVID-19
COVID-19

50
9. Annex 2. Interpreting results from a stratified sample
Using a stratified sampling approach will ensure that sentinel facilities are representative for each domain as
well as for the country at baseline, after adjustment for sampling weight. Observed trends from the sentinel
sites may be interpreted to be representative of all facilities in the country, depending on the acceptable level of
precision. It should be noted, however, that even at the national level the total sample size can be too small and
the margin of error too large to make statistical inferences about trends unless the change is substantial.

Table A2.1 shows examples of margins of error by estimates for key indicators, desired precision and sample
size.
• With a sample size of 100, if the key indicator is estimated to be 20% and if a 95% confidence interval is
applied, the true value is between 12.4% (20 - 7.6) and 27.6% (20 + 7.6).
• With a sample size of 100, if the key indicator is estimated to be 50% and if a 95% confidence interval is
applied, the true value is between 40.4% (50 - 9.6) and 59.6% (50 + 9.6).

Table A2.1. Margin of error to calculate 95% and 90% confidence intervals, by estimates for key indicators
and sample size, assuming a 95% response rate

Estimate for Margin of error, by confidence interval and sample size

key For 95% confidence interval For 90% confidence interval
indicator (%) Sample size Sample size Sample size Sample size Sample size Sample size
50 100 150 50 100 150
10 8.1 5.7 4.7 6.6 4.7 3.8
20 10.8 7.6 6.2 8.8 6.2 5.1
30 12.4 8.8 7.1 10.1 7.1 5.8
40 13.2 9.4 7.6 10.8 7.6 6.2
50 13.5 9.6 7.8 11.0 7.8 6.4
60 13.2 9.4 7.6 10.8 7.6 6.2
70 12.4 8.8 7.1 10.1 7.1 5.8
80 10.8 7.6 6.2 8.8 6.2 5.1
90 8.1 5.7 4.7 6.6 4.7 3.8

51