Seminar on Performance Budgeting and Fiscal Transparency,

Rabat, Morocco, April 21-23, 2009

Session 3. Challenges
to Performance Budgeting:
Observations and Lessons
from Korea

Nowook Park
Center for Performance Evaluation and Management
Korea Institute of Public Finance
 Contents

1 Description of the current performance budget system

Integration of performance information into the budget

2 and management process

3 Roles of the MOF, line ministries, parliament, and audit office

4 Performance Information : Indicators and Evaluations

5 How to motivate performance? How to engage politicians?

2
1. Description of the current
performance budgeting
system
 Background of Performance-based Budgeting in
Korean Government

Need for enhancing efficiency in public spending

Started 4 major
Expected budget reform programs in
problems public finance
• Increasing public debts • Medium-term expenditure
after 1998’s Asian financial framework: Basis for top
down budgeting
crisis
• Top down budgeting:
• Increasing spending on
autonomy to line ministries
social welfare programs • Performance management
due to aging and economic system: accountability
polarization problems • Digital accounting system:
program budgeting &
accrual accounting

4
 Efforts towards Performance-Based Budgeting

Performance Budgeting (Pilot Project)

2000 ~ 2002 - Developed Strategic Goals, Performance Objectives
and Performance Indicators
- Designed after GPRA

2003 ~ Performance Goal Management System

- Expanded “Performance Budgeting” to all ministries and agencies
- Annual performance plan and report are required

2005 ~ Self-Assessment of Budgetary Program (SABP)

- 1/3 of major budgetary programs are evaluated every year
- Designed after PART

In-Depth Evaluation
2006 ~ - Selected programs are subject to program evaluation
- About 10 programs are subject to program evaluation

5
 Enactment of “National Financial Act”

v “Public Finance Act” was enacted in December, 2006

§ To provide a legal basis for 4 major fiscal reform programs

v Contains articles on performance-based budgeting

§ Annual Performance Plan and Report become legal
requirements for line ministries/agencies
§ SABP and In-depth Evaluation are stipulated.

v It gives stability and continuity which may be a problem

to performance management system.
§ Government has less incentives to maintain and improve
performance management system than to introduce it, because
efforts to improve the system is less visible to the public.
6
 Framework for Performance-Based Budgeting
In Korea

v Performance Monitoring
§ “Performance Goal Management”
- Monitoring based on the performance indicators
- Monitoring unit is program and sub-program
v Program Review
§ “Self-Assessment of Budgetary Program”
- Review based on the checklist
- Review unit is usually sub-program
v Program Evaluation
§ “In-depth Evaluation of Budgetary Program”
- Program evaluation for selected programs
- Evaluation unit is usually sub-program, but cross-cut issues are
sometimes evaluated 7
 Description of “Self-Assessment of
Budgetary Program”

v Ministry of Strategy and Finance (MOSF) reviews self-

assessment of programs done by line ministries/agencies

v Budgetary authority provides a standardized checklist for

reporting self-assessment

v The checklist contains questions on program design,

performance management system, program
implementation, and actual performance

v Entire program will be reviewed in three years.

§ About 1/3 programs are reviewed each year
• About 300 sub-programs are reviewed each year
§ Evaluation unit is usually sub-programs which are unit for budget
allocation by MOSF
8
 Contents of Checklist

Design and • Program purpose

Planning • Rationale for government spending
(30) • Duplication with other programs
• Efficiency of program design
• Relevance of performance objectives and indicators
• Relevance of performance targets
Management • Monitoring efforts
(20) • Obstacles of program implementation
• Implementation as planned
• Efficiency improvement or budget saving
Results and • Independent program evaluation
accountability • Results
(50) • Satisfaction of citizens
• Utilization of evaluation results

9
2. Integration of performance
information into the budget
and management process
 Use of performance information in budget negotiations

Performance
information

MOSF Spending ministry

Enforce budget cut to Allocate budget based

ineffective programs vs on evaluation results

Provide Introduce performance

recommendations for information in their
performance program management
improvement

11
 Performance information in budget statements

Spending
ministries
MOSF Assembly

Submitted with budget : Submitted with budget :

annual performance plan annual performance plan
annual performance report annual performance report

12
 Use of Performance Information from
Performance Budgeting Systems

v Information from monitoring system (performance plan

and report) has not been systematically utilized so far.
§ For internal use, they are useful information
§ From the viewpoint of central budget authority, they do not provide
useful information

v Information from review system are actively used in

budget negotiation process.
§ Programs rated as ineffective are in danger of suffering budget cut
§ Its use is systematically built into the budget process

v Information from program evaluation is usually useful

§ Their use in budget process depends on the quality of evaluation
and the will of central budget authority
§ It is not systematically utilized in the budget process
13
 Observations from Program Review Process

v Evaluation results
§ Quality of performance information has not improved much
§ Programs are showing better results
v Link between evaluation results and budget
§ Evaluation results are utilized at every stage of budget
process
v Moving away from incremental budgeting
§ Evaluated programs are subject to bigger budget change
compared to other programs

14
 Evaluation Results by Total Score
Frequency
300

250

200

150

100

50

0
0 10 20 30 40 50 60 70 80 90 100
Total
2005 2006 2007

15
 Evaluation Results by Ratings

Moderately
Total Effective Adequate Ineffective
Effective

Number 555 28 100 340 87

2005
(%) (100.0) (5.0) (18.0) (61.3) (15.7)
Number 577 30 94 388 65
2006
(%) (100.0) (5.2) (16.3) (67.2) (11.3)
Number 584 66 139 348 31
2007
(%) (100.0) (11.3) (23.8) (59.6) (5.3)

16
 Evaluation Results by Section

Planning(30)
Total
Sub Management Results
Score (20) (50)
Performance
(100) total Design
Planning
(30) (15)
(15)

2005 60.1 23.1 13.8 9.3 15.1 21.9

2006 59.9 22.9 14.3 8.6 14.7 22.3

2007 66.0 23.4 14.2 9.2 15.5 27.1

17
 Utilization of Evaluation Results

v MPB encouraged ministries/agencies to use the

results in reshuffling budget allocation
v MPB announced 10% budget-cut would be done to
“ineffective” programs, in principle
v MPB submitted evaluation results to the National
Assembly upon their request
v Evaluation results are open to public since 2006

18
Use of Performance Information by Agencies (2005)

6
95% CI/Fitted values/agencyuse
0 2-2 4

0 20 40 60 80 100
per f

95% CI Fitted v alues
agenc y us e

19
Use of Performance Information by MOSF (2005)

6
95% CI/Fitted values/mpbuse
0 2
-2 4

0 20 40 60 80 100
per f

95% CI Fitted v alues
mpbuse

20
Use of Performance Information by Legislature (2005)

21
 Link between Evaluation and Budgeting (2005)

( Unit : 100,000won, % )

‘05 ‘06 Budget Difference Ratio

Budget
Rating
Agency MPB Final
(A) B-A C-A D-A Ratio_1 Ratio_2 Ratio_3
(B) (C) (D)

Effective 15,600 24,948 18,955 22,489 9,348 3,355 6,889 0.52 0.38 0.50

Mod.
Effective
92,994 104,335 107,055 105,762 11,342 14,062 12,768 0.32 0.33 0.28

Adequate 208,066 204,473 195,625 201,214 -3,593 -12,441 -6,852 0.10 0.05 0.05

Ineffective 33,081 28,644 29,007 28,505 -4,437 -4,074 -4,576 -0.15 -0.25 -0.19

Total 349,740 362,400 350,642 357,970

22
 Link between Evaluation and Budgeting (2006)

( Unit : 100,000won, % )

‘07 Budget Difference Ratio

‘06
Rating Budget
(A) Agency MPB Final
B-A C-A D-A Ratio_1 Ratio_2 Ratio_3
(B) (C) (D)

Effective 8,891 9,467 9,337 8,872 575 446 -19 0.11 0.10 0.06

Mod.
Effective
33,156 35,701 35,364 35,654 2,545 2,208 2,498 0.12 0.08 0.09

Adequate 297,180 296,769 290,481 289,969 -411 -6,699 -7,211 0.10 0.04 0.03

Ineffective 11,431 6,039 5,400 5,380 -5,392 -6,031 -6,051 -0.15 -0.24 -0.25

Total 350,658 347,975 340,582 339,875

23
 Link between Evaluation and Budgeting (2007)

( Unit : 100,000won, % )

‘08 Budget Difference Ratio

‘07
Rating Budget
(A) Agency MPB
B-A C-A Ratio_1 Ratio_2
(B) (C)

Effective 17,112 18,211 17,503 1,099 391 0.18 0.12

Mod.
Effective
266,051 295,121 291,319 29,070 25,268 0.20 0.13

Adequate 146,034 153,026 142,044 6,992 -3,990 0.28 0.15

Ineffective 3,870 3,457 3,066 -414 -805 -0.001 -0.15

Total 433,067 469,815 453,932

24
 Moving Away from Incremental Budgeting

Programs have been subject to larger budget change

after evaluation

Coefficient of Variation in Funding Change

(Excluding Programs of which funding change is greater than 200%)

CV
(B04-B03)/B03 (B05-B04)/B04 (B06-B05)/B05 (B07-B06)/B06 (B08-B07)/B07
Year

2005 3.1 2.7 9.2

2006 2.7 2.7 -14.3

2007 2.5 3.1 3.9

25
 Cultural Changes among Line Ministries

vBudget allocation based on performance is

spreading among line ministries
§ Utilization of SABP results in budget requests
§ Changing practices of program management
vEvaluation activities become active among line
ministries
§ Need for program evaluations are recognized and
accepted among line ministries

26
 3. Roles of the MOF,
line ministries, parliament, and
audit office
 Ministry of Strategy and Finance (MOSF)

v Evaluate line ministries’ program

§ Issuing guidelines on performance-based budgeting
to line ministries/agencies
§ Evaluating their program’s performance
v Use performance information in budget
formulation
§ Encouraging line ministries/agencies to use
performance evaluation results in preparing their
budget requests
§ Incorporating the performance information into its
decisions during budget formulation
28
 Line ministries/agencies

v Producing performance information in

compliance with central budget authority’s
initiative
§ Submitting strategic plans, annual performance plans
and performance reports
§ Evaluate their programs based on SABP checklist
v Conduct program evaluation
v Use performance information in budget request

29
 The National Assembly

vBefore the enactment of National Finance Act,

there had not been official role of the National
Assembly
§ They had requested evaluation results for
budget deliberation on an ad hoc basis
vThe National Assembly receives annual
performance plan (from 2008) and report (from
2009) with budget documents
§ The National Assembly Budget Office intends to
analyze budget in connection with performance
information
30
 The National Audit Office

v Before the enactment of National Finance Act,

there had not been official role of the National
Audit Office
§ As an independent audit office within the
Administration, it has not played any official role
§ However, they examined the operation of
performance budgeting system and produced report
vThe National Audit Office is assigned with the
role of verifying annual performance report from
2009
31
4. Performance Information :
Indicators and Evaluations
Performance Indicators
Status

v Slight increase in outcome measures but there are

big room for further improvement
§ Outcome performance measures increase by 3.2 percentage
points, while output performance measures decrease by 4.1
percentage points
§ More than 40% of programs do not have relevant performance
indicators in SABP
Type of indicators
Total
Input Process Output Outcome
73 140 837 966
FY 2007 2,016
(3.6) (7.0) (41.5) (47.9)
111 125 761 1,040
FY 2008 2,037
(5.4) (6.1) (37.4) (51.1)

33
Performance Indicators
Examples

Good Bad Ugly

Support the IT business Student Loans Program Percentage of attainment
to go abroad - The amount of loans - rate of planned goal
- Customer satisfaction - The number of loans achievement about
- Increase the amount of employment (x)
export($) -rate of employment (o)

Recommendation : Recommendation :
Using outcome measure Using ‘actual value of
ex) Enrollment rates: The attainment’ instead of
postsecondary enrollment gap ‘percentage of achievement’.
between low- and high-income (FY 2008 manual of KPART)
high school graduates will
decrease each year.
34
Performance Indicators
Problems

v Difficulties to develop outcome measures in some programs

§ Program producing results after long period
• Using milestone indicators may help
• However, using different evaluation cycle needs to be
considered
§ Successful management of program may not change outcome
measures
• Reduce scope of outcome measures
• Particular program may not worth evaluating
§ Data do not exist
• Investment in data is necessary
• Compare cost and benefit of producing new data

35
Performance Indicators
Lessons

vIt is desirable to develop outcome measures,

but it is not possible for every program

vBe aware of some exceptions and apply it

flexibly according to program type

36
Program Evaluations
Status

v Self-Assessment of Budgetary Program

§ Review based on the checklist
§ Among the checklist, there is a question whether program
evaluation is conducted or not
§ It encourage line ministries to conduct program evaluation at
least once in three years

v Budgetary Program Evaluation

§ In-depth evaluation for selected programs by the central budget
authority

37
Program Evaluations
Status - KPART

Q. 3-1. Did you conduct comprehensive program evaluation objectively?

• Has your program been evaluated in depth using

verifiable data ?
• Has your program evaluation been conducted by
independent organization ?
ex) external institution, audit office,
internal evaluation expert, etc.
YES !
• Does evaluation cover important issues of program ?
cf) admit ‘yes’ in case that evaluation is in progress
or conducted within 3 years latest

38
Program Evaluations
Problems

v Little experience with program evaluation among

program managers
v Lack of data to prohibits meaningful program
evaluation
v Lack of fund to conduct program evaluation
v Difficulties in maintaining independence of
evaluators

39
Program Evaluations
Example : Bad

Promotion of the intellectual property (IP)

• Measuring the impact of program performance
Purpose • Realign the way of investment
• Monitoring the performance management system
Does the number of IP center affect the number of industrial
Issue
property ?
Methodology Estimation based on the fixed effect model
Result :
Couldn’t find the evidence for the regional impact of the IP center !

• Insufficient Data
- absence of data which is needed to conduct the evaluation
Obstacle - using customer satisfaction (or number of counseling)
instead of the number of industrial property
• Lack of comprehension about program evaluation
40
Program Evaluations
Example : Good

Development of rural area

• Evaluate the impact of the program
Purpose
• Monitoring the program in progress
Does the amount of fund improve the condition of the
Issue
undeveloped-area ?
Methodology Data analysis and the survey

Result of Effectiveness
Find the evidence for the regional impact !
- There is positive relation between the program and the infrastructure
- In addition, the infrastructure creates value-added business
Result of Management
Find the problems of the management system!
- The management system is not developed and the each manager’s autonomy
is limited
41
Program Evaluations
Lessons

v Plan ahead for data collection

§ Data should go hand in hand with program management
§ Data management is a part of program management

v Be creative in securing fund for evaluation

§ Evaluation should be part of program management
§ In budget negotiation, evaluation plan should be presented

42
 5. How to motivate
performance? How to
engage politicians?
 Institutionalized Incentives for line ministries

v How performance information will be used in budget

preparation is stipulated in guidelines for budget
preparation
§ In principle, 10% budget cut for ineffective program is
encouraged to line ministries
§ It draws attention from program managers but not from high
ranking decision makers
v Results from SABP is used as a part of evaluation
information for the whole department
§ Now high ranking decision makers pay attention
§ But their attention is focused more on getting good scores than
on improving programs’ performance
 Forms of Information does Matter

v Performance information (performance indicator) at

monitoring level has not been able to draw attention from
decision makers
§ They are internally useful information, but not utilized by central
budget authority
§ Performance indicators are good starting point for
communication rather than decision making
v Performance information (program ratings) from program
reviews are actively used by central budget authority
§ It summarizes various information into simple ratings
§ Power of Simplicity!
 Politician’s Engagement

v Although there is no formal mechanism for politician’s

engagement, politicians show interest in performance
information
§ They requests performance information on an ad hoc basis
§ There is not much suspicion on the evaluation results from the
Administration
v It remains to be seen whether politicians will use
performance information more systematically with the
submission of annual performance plan and report to the
National Assembly,
v Publicizing performance information may help to force
politicians to pay attention to PI