TOR for Impact Evaluation Study on RIS to the PNPM Mandiri Project II 1

TERMS OF REFERENCE
Impact Evaluation Study on Rural Infrastructure Support
to the PNPM Mandiri Project II

A. Background
1. The Government of Indonesia has received a loan from the Asian Development Bank
(ADB) towards the cost of the Rural Infrastructure Support to PNPM Mandiri Project II.
The Project will continue the support for RIS-PNPM, financing two cycles of village
planning and block grants in about 1,500 villages in 215 kecamatans in four provinces of
Jambi, Lampung, Riau, and South Sumatra. The Project will follow a community-driven
development approach and will empower communities; strengthen their capacity to
prioritize, design, implement, and monitor community-based projects; and provide block
grants to finance community-identified needs and priorities.
2. The four project provinces of Jambi, Lampung, Riau and South Sumatra have high rural
poverty and unequal access to infrastructure. The provinces contained a total of almost 4
million poor people in 2009. South Sumatra (16%) and Lampung (20%) had poverty
incidence well above the national average. Fewer than 60% of households have access
to safe water supply. South Sumatra (53%) and Lampung (54%) have low school
enrollment rates for upper secondary school and Riau (9.35%) and South Sumatra
(8.45%) have higher than national average unemployment rates.
3. The Project is expected benefiting to about 1.5 million rural poor in about 1,500 poor
and/or isolated villages. It will target poor and isolated villages that lack basic services
and infrastructure. The community investments are expected to reduce time and costs
spent on obtaining water, diminish the incidence and severity of waterborne diseases,
increase the availability and accessibility of education and health services for the rural
poor and near poor, improve access to services and markets, and provide opportunities
to improve incomes and welfare through increased employment and income-earning
opportunities. The capacity development and community facilitation investments under
the Project will support community empowerment, build social capital, and improve local
level governance, resulting in more efficient and equitable provision of services at the
local level. The Project included specific actions to promote gender equity, including
innovative features such as separate women's group meetings to ensure that women's
needs are adequately reflected in village plans, and gender audits for each selected
community investment to ensure that it reflects the needs of men and women.

B. The Project
4. The Project forms part of the Government of Indonesia's ongoing flagship poverty
reduction program-the National Program for Community Empowerment (PNPM Mandiri),
which was launched in 2006 with the commitment to reduce poverty by adopting a
community-driven development approach and providing direct support to poor rural and
urban communities to improve essential social services and basic infrastructure. The
projects implemented in a geographic slice of the PNPM Mandiri Program and include
about 1,500 rural communities in four provinces with the following three outputs: (i)
strengthened capacity for community planning and development, (ii) improved village
services and infrastructure through community development grants, and (iii) improved
capacity for project implementation and monitoring and evaluation.
TOR for Impact Evaluation Study on RIS to the PNPM Mandiri Project II 2

5. The Project’s impact is reduced poverty and improved local governance of rural
communities in the project areas, which is in line with the overall objective of PNPM
Mandiri. The expected outcome is improved access to service delivery and basic rural
infrastructure for the poor, near poor, and women in the project communities.
6. For coordination, implementation and management of the Project, the Government has
established a project organization comprising Project Coordination and Monitoring Unit
(PCMU) and project manager (Satker) at the central level, provincial project
implementation units (PPIUs), and district project implementation units (DPIUs) at the
regional levels. The project organization details are in Appendix 1.
7. In order to determine the impacts of the Project, the Government represented by DGHS
requires Consultant to conduct an external impact evaluation study. The evaluation will
assess the impact of project interventions on poverty, gender, empowerment,
governance and socioeconomic conditions in target villages. Regression discontinuity
and other econometric tools will be used to compare changes in project and control
villages from pre-project baseline data. The impact evaluation will be the first application
of this methodology to assess CDD programs in Indonesia and will complement existing
evidence and contribute to increased knowledge for the institutionalization of the PNPM
Mandiri. The impact evaluation will be conducted in close dialogue with agencies such as
BAPPENAS, Menko Kesra, and the ADB.

C. Roles
8. The Consultants will provide technical assistance and logistical support to undertake the
impact assessment which is consist of: (i) design and implementation of a high-quality
impact evaluation, (ii) assessing the impact of project interventions on reducing poverty,
gender development, communities’ empowerment, improving local governance and
socioeconomic conditions in targeted villages, (iii) determining whether the Project had
impact on governance practices and community empowerment focusing on participation,
transparency and accountability, particularly in the implementation of development
projects, (iv) assessing the impact and utilization of the investments.
9. Impact evaluation is an assessment of how the Project’s intervention being evaluated
affects outcomes, whether these effects are intended or unintended. The proper analysis
of impact requires a counterfactual of what those outcomes would have been in the
absence of the intervention. The Consultants will provide an impact evaluation report
assessing project processes and emerging impacts of the Project. The report will be
reviewed by EA to examine compliance with covenants specified in the Loan Agreement.

D. Objectives
10. Impact evaluation serves both objectives of evaluation: lesson-learning and accountability.
The objective of the study was to make an assessment of economic and social impacts
including local governance impact of the Project in targeted project areas.
11. In line with the objective above, the specific objectives of the impact evaluation study were:
a) To develop a well-designed impact evaluation to answer questions about program
design: which bits work and which bits don’t, and so provide policy-relevant information
for redesign and the design of future programs. A well-designed impact evaluation will
includes reconfirmation of direct (intended) objectives and identification of broader
(indirect) impacts of the Project, and development of appropriate methodology.
TOR for Impact Evaluation Study on RIS to the PNPM Mandiri Project II 3

b) To develop a set of indicators that can meaningfully and reliably define and measure
project inputs, implementation processes, outputs, intended outcomes and impacts.
c) To determine impact of RIS-PNPM II on the various dimensions of human development
that is focusing on standard of living, poverty reduction and gender development.
d) To assess the impact of RIS-PNPM II on local governance practices and community
empowerment that is focusing on participation, transparency and accountability,
particularly in the implementation of CDD rural infrastructure projects.
e) To evaluate the benefit/impact thus far of project interventions and inputs/activities, in
terms of achieving the desired outputs.
f) To assess the achievement of project in terms of the desired outcome to improved
access to service delivery and basic rural infrastructure for the poor, near poor and
women in the project communities as a result of implementing the CDD approach,
including key functions and implementation roles, for delivering services to communities
and its sustainability.
g) To put forward logically sound recommendations based on scientifically rigorous impact
evaluation in line with the above objectives and the National Poverty Reduction Strategy
(SNPK), especially for accelerated development and poverty reduction in a sustainable
way through CDD rural infrastructure projects.
12. The study will help GOI and ADB determine the extent to which the Project has achieved its
intended results, and will help identify factors contributing to successes or failures in
achieving the identified results, adequacy of implementation and monitoring mechanisms. In
addition, the studies will identify lessons learnt and make key recommendations for both
GOI and ADB.
13. Purposive socioeconomic surveys using questionnaires will be conducted to obtain data on
the impact on the ultimate beneficiaries. Relevant documentation to be reviewed for the
impact evaluation study included Reports and Recommendations of the President to the
Board of Directors (RRP), project performance audit reports, and back-to-office reports of
loan administration and review missions. Extensive discussions with PCMU and PPIU/DPIU
officials, EAs, and participating local government’ institutions during the course of various
review and impact evaluation processes will be held.
14. The impact evaluation study uses a post-intervention cross-sectional design in which socio-
economic interviews will be conducted in 150 selected villages out of 1,500 targeted
Project’s villages with a sample of 1,500 households. The sample included “treatment”
households living in the village where the basic infrastructures were developed and “control”
households living in a similar village at a distance of some 5 km with no access to the basic
infrastructures.

E. Working Arrangements and Period of Services

15. The Consultant will establish a fully staffed and well-equipped office at the central level.
The evaluation team will be staffed with specialist consultant, include the following specialists
(i) Evaluation Specialist as Team Leader, (ii) an Infrastructure Engineer, (iii) an Impact
Evaluation Survey Specialist, (iv) a Statistic Specialist, and (v) a Data Management Specialist.
The Consultants will be contracted for a period of 9 (nine) months.
TOR for Impact Evaluation Study on RIS to the PNPM Mandiri Project II 4

F. Scope of Works
16. The evaluation tasks include develop research tools and methodology including village
selection criteria that to be evaluated, collecting data through in-depth, semi-structured
interviews and focus group discussions at the village level, evaluates good governance
practices in terms of participation, transparency and accountability, assess the Project’s
performance against its own goals and principles, with the aim of understanding the factors
that affect to the Project implementation.
17. The Consultants should develop detailed Study Designs as well as Implementation Plans
for carrying out the assignment, which will be subject to approval from PCMU and ADB. The
Detailed Implementation Plans will include key questions to be addressed by the studies,
outline the research methodology and approach, and identify indicators, data sources, and
data collection strategies.
18. Specifically, the evaluation team will examines the Project’ impacts and effects on:
a. Village governance practices as a result of experiences in the Project, focusing on
participation, transparency and accountability, particularly in the implementation of
development projects.
b. Examines whether the community institutions have adopted new behavior as a result of
their experience in the Project process and its implementation.
c. Learn how the Project can foster stronger ties both between villagers and village
governments and among villagers themselves in meeting village needs. This ability to
work together is a fundamental aspect of community empowerment.
d. Examines if and how the women’s ability and their capacity have been changes to meet
their development needs more generally.
e. Examines how the process of poverty targeting works that allowing villagers to target
their own needs and raising the poverty levels of most villagers

G. Reports and Recommendations

19. A draft report format, identifying key areas to be addressed under the benefit /impact
evaluation study will need to be submitted in response to the RFP. Reporting requirements
include submission of a draft report on initial findings, a draft final report and a final report,
the timing of which will be agreed and recorded in the independent firm contract.
20. The impact evaluation study is expected to produce a series of actionable, prioritized
recommendations for the Government of Indonesia (GOI) and ADB for improving the
delivery of basic rural infrastructure services for the poor, near poor and women in the
project communities. The impact evaluation study will need to highlight any similarities
and/or differences with respect to implementing the Project at the community level and at
the district and/or sub-district level.

H. Staffing Requirements
21. The firm will be selected and engaged in accordance with the ADB’s Guidelines on the Use
of Consultants (2010, as amended from time to time). The firm for impact evaluation study
will be contracted through quality and cost-based selection using simplified technical
proposal. The impact evaluation should be completed within 9 months, from April to
December 2012. The firm will provide about 40 person-months of national specialists.
TOR for Impact Evaluation Study on RIS to the PNPM Mandiri Project II 5

22. The evaluation teams requirement experts:

1. Evaluation Specialist as Team Leader (9 PM)
2. Infrastructure Engineer (7 PM)
3. Statistic Specialist (6 PM)
4. Data Management Specialist (9 PM)
5. Evaluation Survey Specialist (9 PM)
The team will also supported by sub-professional staffs, supporting staffs, and field
surveyors. Supporting staff in this assignment required to support office operational.

I. Tasks and Responsibilities of the Specialist

a. Team Leader (9 Person-Months)

23. The Evaluation Specialist (Team Leader) is expected to provide over-all guidance to the
completion of the Impact Evaluation report. He is expected to deliver an output which
conforms to IES’s Guidelines in conducting an impact evaluation.
24. The tasks of the Evaluation Specialist are the following:
a. Provide overall guidance to the team’ members in the conduct of the evaluation,
including refinement of evaluation design, development of field data collection
instruments, data analysis and report writing.
b. Review relevant documents and obtain necessary background information through the
collection of statistics and the conduct of interviews and discussions with the
Government and various stakeholders as identified in the evaluation approach paper;
c. Supervise the conduct of the Evaluation Studies including surveys, focus group
discussions, key informant interviews, site visits, and meetings with the various
stakeholders including the Government and Executing Agencies and other
development partners;
d. Review the outputs of the evaluation specialists and closely coordinate with all
stakeholders involved in the study;
e. In collaboration with PCMU and PPIUs/DPIUs, evaluate/validate the project
implementation report findings on the achievement of project outputs, including their
quality and timing. Examine how the project outputs have contributed to the
achievement of overall project objectives, and consequently determine its impacts on
the beneficiaries;
f. Take the lead in drafting the summary reports, and the Final Report by preparing
relevant sections of the report;
g. Identify key issues and lessons learned from the Project and formulate
recommendations for future ADB assistance in the subsector;
h. Respond to comments and suggestions from peer and inter-departmental reviews,
including those from ADB;
i. Disseminate evaluation findings using appropriate method and forum under the
guidance of EAs and with the support of ADB.
Key Qualification Required
25. The candidate should hold a minimum master in project management or related discipline,
with at least 8 years experience in the project’ benefit monitoring and evaluation field with
significant experience in community driven development (CDD) activities. She/he should:
TOR for Impact Evaluation Study on RIS to the PNPM Mandiri Project II 6

 Possess capability the project’ benefit monitoring and evaluation the impact of
development projects on poverty in rural areas.
 Familiar with donor/governmental project regulations.
 Have at least 5 years relevant experience in project monitoring and evaluation.
 Have capability to lead and manage the team of specialists.
 Be fluent in English and Indonesian, both written and spoken.
 Demonstrated ability to work congenially and productively with consultant team,
counterparts, and other project stakeholders

b. Infrastructure Engineer (7 Person-Months}

26. He/She is responsible to review and assess the overall infrastructure planning and
implementation associated with the project and to coordinate and support the impact
evaluation activities. He/She tasks include but not limited to the following:
a. In consultation with the Team Leader, identify and develop a set of evaluation indicators
reflecting quality of construction, community organization, status of built infrastructures
utilization since handover to the beneficiaries including operations and maintenance
mechanisms, cost recovery arrangements, and other sustainability related issues;
b. Assist Team Leader in liaise with CPMU, provincial and district governments
(PPIUs/DPIUs);
c. Prepare the rural infrastructure survey instrument and pre-test the instrument to gather
relevant data in a systematic way.
d. Inspect a list of pre-identified built infrastructures and conduct survey for selected
communities and surveys through focus group discussions, meetings with CIOs and
other key informants.
e. Identify critical technical, managerial and community factors exerting impact on the
performance of the built infrastructures.
f. Prepare an analytical technical report based on the survey data.
g. Provide a set of key lessons for ADB for future operations, particularly in the area of
technical design, operation and maintenance, communities’ capacity constraints, and
financing mechanism.
Key Qualification Required
27. The candidate should have a minimum bachelor (S-1) degree in civil engineering, with at
least 6 years relevant experience in project’ benefit monitoring and evaluation, and will
advantages with the community based development projects/program. The proposed
engineer with good reputable experiences and having passed the grade of evaluation on
the similar projects will be preferably.
The consultant should;
 Possess capability the technical aspects of basic infrastructures development in rural
areas;
 Familiar with donor/governmental project regulations;
 Possess capability project management: planning, design, implementation, operational
and maintenance services;
 Possess capability to develop systems for monitoring and evaluation of project
technical, the delivery of the planned infrastructure facilities, activities and programs,
the achievement of project objectives and project social and economic benefits;
TOR for Impact Evaluation Study on RIS to the PNPM Mandiri Project II 7

 Good working knowledge in written and spoken English is essential;

 Demonstrated ability to work congenially and productively with consultants,
counterparts, and other project stakeholders in a team context.

c. Impact Evaluation Survey Specialist (9 Person-Months)

28. The Impact Evaluation Survey Specialist would perform following tasks:
a. Conduct an in-depth critical review of theoretical and empirical literature on the impact
of the Project interventions on gender, communities’ income, and other socioeconomic
development areas.
b. Identify modalities and key determinants of impact due to infrastructures development.
c. Based on (b) above, refine the conceptual framework and evaluation matrix for the
impact evaluation study with clear identification of indicators and their measurements at
all levels.
d. Based on (c) above, develop appropriate survey instruments for field data collection
using both qualitative and quantitative methods. These would include knowledge-
attitude-practice surveys, household surveys, focus group discussions and key
informant interviews.
e. Pre-test the survey questionnaires and improve the draft questionnaires.
f. Pre-test data collection instruments in a rural targeted community.
g. Lead, manage and conduct (a) a survey of households selected target areas; (b) a
survey of CIOs assessing their capacity, knowledge, attitude and practice in rural
development needs; (c) key informant interviews with relevant individuals, including
local community leaders.
h. Undertake any other tasks assigned by the Team Leader.
Key Qualification Required
29. He/She should have Bachelor Degree in Social Science and Development with significant, at
least 6 years, experience in the impact evaluation survey field. The specialist should:
 Have knowledge in socialization strategy/development, project’ impact assessment;
 Ability and experience in the utilization of CDD approach on infrastructures
development in rural areas.
 Demonstrated ability to work congenially and productively with team’ members,
counterparts, and other project stakeholders in a team context.

d. Statistic Specialist (6 Person-Months)

30. He/She is responsible to develop an evaluation designs which address selection bias using
statistical methods. Particularly the specialist will:
a. Preparation of sampling framework, training of staff to implement the designed sample,
supervision of the implementation stage to ensure the quality of the sample selected,
and provision of a detailed report outlining all the steps involved in the design and
implementation of the sample.
b. Participation in determining an appropriate strategy for identifying comparison groups
(that is, non-project beneficiaries).
c. Develop a sample selection plan in consultation with the Team Leader and identify
participants for household/individual and CIOs interviews.
d. Follow the advice and guidance of the Team Leader
e. Provide the raw data and data dictionary and generate summary statistical tables as
agreed with the Team Leader and PCMU.
TOR for Impact Evaluation Study on RIS to the PNPM Mandiri Project II 8

f. Prepare a comparative analytical report reflecting with and without project scenario on
key parameters associated with the Project’ components.
g. Regularly update and assist the Team Leader in conducting statistical analysis.
Key Qualification Required
31. He/She should have Bachelor Degree in Statistic Science and Development with significant, at
least 6 years, experience in the evaluation survey field. The specialist should:
 Have knowledge in socialization strategy/development, project’ impact assessment;
 Ability and experience in evaluation designs which address selection bias using
statistical methods, such as propensity score matching or randomization.
 Demonstrated ability to work congenially and productively with team’ members,
counterparts, and other project stakeholders in a team context.
 Documented expertise in economic evaluation

e. Data Management Specialist (9 Person-Months)

32. Data Management Specialist is responsible to development and adaptation of the data
collection instruments and support documentation, including listing materials, questionnaires,
coding guides, manual of operations, data entry manual, and field procedures.
33. He/She tasks include but not limited to the following:
a. Participation in the development of the study protocol
b. Development of data collection instruments including pre-coded data collection
instruments: household survey, knowledge-attitude-practice surveys and built
infrastructures surveys, focus group discussions and key informant interviews.
c. Provide guidance to data collection team in the field to ensure collection of quality data
for the evaluation.
d. Develop a road map for qualitative analysis of field data, including selection of
appropriate analytical techniques.
e. Analyze qualitative and quantitative data and information collected from the field and
prepare draft evaluation report in association with the impact evaluation specialist.
f. Supervision of data collected during field surveys
g. Ensure data completeness, quality, and consistency to the highest standard in all
surveys. Maintain confidentiality of information and data and provide a safe and secure
area to store the completed questionnaires.
h. Manage data entry and process data accurately using user-friendly and most popular
software.
i. Participation in the main analysis to be performed at the end of the study.
Key Qualification Required
34. He/She should have have Bachelor Degree in Management Data/Information System with
significant, at least 6 years, experience in the project’s monitoring and evaluation. He/She
should:
 Familiar with donor/governmental project regulations.
 Possess computer application skills;
 Good working knowledge in written and spoken English is essential;
 Possess a demonstrated ability to work congenially and productively with the team’
members, counterparts, and other project stakeholders in a team context.
TOR for Impact Evaluation Study on RIS to the PNPM Mandiri Project II 9

J. Reporting
35. All written reports will be bi-languages Bahasa and English submitted to the PCMU and
other stakeholders as follows:
1. Inception Report
This report will explain the consultant work plan, methodology, understanding of the
terms of reference, and understanding of the Project, personnel mobilization plan, and
basic strategy and approach in undertaking the assignment, to be submitted not more
than 3 weeks after the issuance of a work order by the project. Ten (10) copies of the
report are to be submitted, and the consultant will present the report to the PCMU
2. Monthly Report
This report contains containing the Consultant administration activities for the current
month, progress of work to date, and work planning for the next month (monthly report
not issued during month of quarterly report).
3. Draft Final and Final Report
This report contains a comprehensive account of the Project impact evaluation,
sustainability and replicability of project impacts; capacity built; institutional and
stakeholder issues, conclusions that insights into the findings; reasons for successes and
failures; innovations, recommendations (based on evidence and insights), lessons learned
with wider relevance and that can be generalized beyond the project, experiences gained,
and conclusions as well as recommendations formed as result of undertaking the
assignment. Draft final report must be presented to the EA as arranged by the PCMU.
The Project Impact Evaluation Report must be submitted not later than twenty (20) days
after the completion of assignments, with ten (10) copies

K. Supports to be provided by Project

36. Providing secondary data: a number of reports were done for specific purposes of the
Project that are going to be useful for end-project evaluation. These include Project’
implementation monitoring reports, database on-line, MoU ADB’ Mission and Supervision,
and auditing reports. Data from PCMU, PPIUs/DPIUs are also among information sources of
the evaluation.
37. The summary of Gender Action Plan (GAP) and the Project’s Design and Monitoring
Frameworks which to be used for the study are in Appendix 2 and Appendix 3
38. Project staff supporting: During the evaluation, an appropriate project staff will be assigned to
support for evaluation team.
39. Facilities support: Office facilities, transportation, or other running costs are included in the
contract.
TOR for Impact Evaluation Study on RIS to the PNPM Mandiri Project II 10

L. Indicative Budget
a. Professional fees (40 person-months@US$2,500) 100,000
b. Sub-professional fees (27 person-months@1,500) 40,500
c. Supporting staffs fees (27 person-months@500) 13,500
d. Airfares and other travel (90 trips@US$300) 27,000
e. Out-off station allowances (360 person-days@US$100) 36,000
f. Per diem for field surveyors (750 person-days@US$50) 37,500
g. Office running costs (9 months@US$2,000) 18,000
h. Training for field surveyors (150 person-days@US$100) 15,000
i. FGDs at community’ level (1,500 person-days@US$10) 15,000
j. Workshops (300 person-days@US$100) 30,000
k. Miscellaneous costs 2,500
Total costs 335,000

Exchange rates: US$ 1,0 = IDR 9,000