WHAT IS AN INDICATOR?

• An indicator is a statement of the area of achievement of a goal or intended

outcome. Evidence is required for the indicator, to show that the goal has
been achieved.
• ‘An indicator can be a proxy measure when the outcomes cannot be directly
measured’. For example, students’ attendance at extra-curricular activities
and meetings could be used as an indicator of student involvement with the
KEY PERFORMANCE INDICATORS University. ‘Sometimes one indicator may be all that is necessary for clear
explanation. Sometimes several indicators may be needed’ (Scottish
Community Development Centre (2007) Learning, Evaluation and Planning).
QUALITY ASSURANCE OFFICE • ‘An indicator is a qualitative or quantitative factor or variable that provides a
simple and reliable mean to express achievement, the attainment of a goal, or
the results stemming from a specific change. It often aggregates or combines
multiple measures.’ (Carebot, 2016: https://thecarebot.github.io/metrics-
measures-and-indicators/).

WHAT ARE KEY PERFORMANCE INDICATORS (KPIs)? WHY DO WE NEED KEY PERFORMANCE INDICATORS?
• KPIs are indicators (often measures) which are used to evaluate the effectiveness • They can indicate how well something is working.
of an institution, its elements and its units in meeting objectives for performance. • They can support evidence-informed evaluation, decision-making,
• A KPI ‘is a metric that embeds performance targets so organizations can chart strategic direction, policy formation and management of the University/
progress toward goals.’1
• ‘A KPI is a composite metric that is tied to targets and that indicates how an unit: navigational and decision-making tools.
organization is performing relative to a specific objective’.2 • They can provide data on overall performance.
• Quantitative indicators are those which are ‘associated with the measurement of • They can contribute to the overall picture of what the University is doing.
quantity or amount, and are expressed as numerical values; something to which
meaning or value is given by assigning it a number’.3 • They identify areas for improvement and development.
• Some people comment that KPIs are quantifiable measure/metrics. Others • They can indicate where to intervene.
suggest that KPIs are not necessarily stated in terms of measurable outcomes and • They contribute to promoting continuous improvement.
that they can be addressed in other terms (e.g. qualitative).
• They contribute to promoting an organizational learning environment.
• They can chart/measure progress.
1. Sahai, P. and Sahni, R. (2016) Strategies for Defining Key Performance Indicators in Research. Clinical Researcher. August 2016,
pp. 16-19. DOI: 10.14524/CR-16-0010), p. 16.
2. Ibid., p. 16.
• They can indicate where to address developments and improvements
3. Chalmers, D. (2008) Teaching and Learning Quality Indicators in Australian Universities. Paper presented at the OECD
Conference ‘Outcomes of higher education: Quality relevance and impact’, p. 4. Paris, 8-10 September. Paris: OECD. with students.
3 4

WHY DO WE NEED KEY PERFORMANCE INDICATORS? BEING REALISTIC ABOUT KEY PERFORMANCE INDICATORS
• KPIs indicate (only); they do not tell the whole story.
• They give focus.
• KPIs inform; humans decide.
• They can contribute to keeping the University/unit(s) on track and goal-
focused: monitoring and direction. • KPIs provide an incomplete, partial and selective coverage of work and activities:
not everything is covered in a KPI or a series of KPIs.
• They can identify and recognize achievement. • KPIs are tools, not ends in themselves.
• They can inform judgements about the consequences of interventions.
• KPIs must be taken within the context of the overall University/unit; using them in
• They can ‘ring an alarm bell’ if matters are not working effectively. isolation risks distorting the overall work, focus and goals of the University/unit.
• They can identify trends over time. • They may focus on what rather than why.
• They can provide (comparative) evidence of performance. • Numbers are useful but not everything is susceptible to/suitable for metrics.
• They can support performance management. • Numbers can contribute to and complement qualitative data, judgements,
• They can acknowledge improvements made  morale. evaluation and policy.
• They contribute to transparency. • KPIs should not be misused: consequential validity is important; there are dangers
• They can be used for accountability. in using them punitively rather than for improvement and development.
5
• KPIs risk focusing on short-term and narrow achievements: only performance. 6

1
 KPIs AT USJ
(Some USJ KPIs might harmonise with those of UCP)
TERMINOLOGY
1. GOALS: Targets; intended outcomes.
LEVEL ACTION WHO HEADS TARGET Example: To raise student attainment through collaborative learning in the University.
UP COMPLETION DATE 2. SUCCESS CRITERIA: How will you know if the goals/intended outcomes have
UNIVERSITY: Overall KPIs to be Overall KPIs to be Vice-rector End of December, been achieved? What standards/criteria will you use to judge whether or not it
developed at University level developed at Prof. Barbosa 2018 has been successful (i.e. what defines the success of the project)
Example: Collaborative learning has raised student attainment in the University.
University level
90% of our students have GPAs of 18/20 or above.
ACADEMIC UNIT: The same KPIs to be discussed Vice-rector End of January 3. KEY PERFORMANCE INDICATORS: Statements of the area of achievement of a
single set of KPIs to be with, developed with Prof. Barbosa; 2019 goal, i.e. operationalizing the goal and success criteria, and the metrics used to
developed for all academic and agreed with Vice-rector for track the success towards a goal (outcome measures of/in the success criteria).
units. Deans. QA; Deans Examples: (a) Rise in student attainment in comparison to the previous year.
(b) Increase in collaborative learning in all modules.
ADMINISTRATIVE, SUPPORT Facilitated discussion Vice-rector End of January 4. EVIDENCE: Data required to inform the indicator.
AND SERVICE UNITS: Each with each unit. Prof. Barbosa; 2019 Examples: (a) Students’ semester grades over two years.
administrative/service/ support Vice-rector for (b) Classroom observation data over two years.
unit to have its own KPIs QA; Unit heads 7

CLARIFYING TERMINOLOGY WHAT IS BEING INDICATED?

QUESTION: What is the difference between success criteria and indicators? (A) The work, operations, achievements, outcomes, achievable measures/steps of
ANSWER: Success criteria are what define the success of the project. Indicators are the University/unit.
statements of areas focused on in order to inform the success criteria: the tools and
(B) The achievement of:
/or metrics to measure or judge the achievement of the success criteria.
Success criterion: Every session in every module uses collaborative learning. 1. the goals of the Action Plan
EXAMPLE 2. the content areas of the Action Plan
Indicator: Collaborative learning by groups of students. 3. the intended outcomes of the Action Plan
Indicators must fairly cover and represent the Action Plan’s goals and contents; they
QUESTION: What is the difference between indicators and evidence? must be sufficiently comprehensive and fair: valid and reliable.
ANSWER: Indicators are statements or measures being used in relation to a field.
Evidence is the data used to inform the indicator. Indicators of:

EXAMPLE
Indicator: Student involvement in the University. • Input
Evidence: Number of students attending a University meeting. • Process
• Outcome

KEY PERFORMANCE INDICATORS (KPIs): KEY PERFORMANCE INDICATORS (KPIs):

TEACHING AND LEARNING TEACHING AND LEARNING
• Input indicators: ‘Input indicators reflect the human, financial and physical
resources involved in supporting institutional programmes, activities and • Outcome indicators: can be quantitative and/or qualitative: ‘Outcome measures
services. Limitations concerning input indicators surround their inability to focus on the quality of educational program, activity and service benefits for all
determine the quality of teaching and learning without extensive stakeholders. Outcome performance indicators typically do not involve
interpretation.’ (Chalmers, 2008, p. 4). generating the quantity of outcomes in the form of numerical data (as do output
• Process indicators: ‘Process indicators are those which include the means used performance indicators), but instead measure complex processes and results in
to deliver educational programmes, activities and services within the terms of their quality and impact. This is the difference between output and
institutional environment . . . . Process indicators allow the collection of outcome measures. While they both measure the effects of higher education,
qualitative information on aspects of teaching and learning quality; such as output performance indicators measure this quantitatively, and outcome
policies and practices related to learning and teaching, performance measures do this qualitatively’ (ibid., p. 5).
management and professional development of staff, quality of curriculum and
the assessment of student learning, and quality of facilities, services and
technology. Process indicators have been identified by empirical research to be
the most practical, useful and appropriate measures of quality teaching and
learning within higher education institutions’ (ibid., p. 5). 11 12

2
 SOME RUDIMENTARY KPIs ARE CATEGORICAL, E.G. YES/NO EXAMPLE OF KPIs FOR MISSION STATEMENT
• USJ has Strategic Plans, Action Plans, Policies at University and unit levels.
• USJ has an appropriate mission to cultivate talent in and for Macau.
• USJ has a Quality Assurance Framework at University level.
• USJ has Guidelines on assessment of students.
• USJ’s mission enables students to reach international standards, both
academically and professionally.
SOME KPIs ARE NUMBERS ONLY • USJ’s mission is understood and supported by all stakeholders.
• Dropout rates. • USJ’s mission underpins and directs the work and development of the
• Number of students enrolled onto a programme. University.
• USJ has QAE for ensuring the achievement of its mission and vision.
SOME KPIs ARE QUALITATIVE
• Clarity and organization of instructor’s presentation of content, concepts, skills, and
techniques.
• Clarity and suitability of instructor’s expectations, requirements, guidance and
directions for student performance.
• Instructor’s helpfulness and availability during and outside class. 13 14

EXAMPLE OF KPIs FOR ADMISSIONS EXAMPLE OF KPIs FOR OPERATIONAL EFFICIENCY

• Number of applicants for each programme in the Faculty
• Low admission criteria to encourage diversity and widen participation. • Student admission number by programme
• High expenditure on library as indicator of commitment to research. • Student grade distribution (by Programme/Study year)
• High expenditure on journals in library (as indicator of commitment to • Student number of ‘normal’ graduation, attrition/dropout rate and graduation delay, by
programme
research). • Number of programmes reviewed/revised
• Low staff-student ratios (as indicator of commitment to the student • Number of new programmes offered
experience). • Teacher-student ratios
• High income levels from international student recruitment (as indicator • Number of research seminars, workshops, visiting speakers and academic presentations
(inside and outside the Faculty)
of commitment to global diversity). • Awards/honours relating to research received by academic staff
• Low staff turnover (as indicator of commitment to the staff experience). • Student achievement for extra-curricular activities
• External academic exchange activities (outbound and inbound visiting teachers/students)
• International academic exchanges (number of visiting and visited HEIs outside Macau
and collaboration agreements signed at Faculty level)
15 • International professional accreditation of programmes 16

PUBLISHED EXAMPLES OF KPIs PUBLISHED EXAMPLES OF KPIs

Standard/Broad Key Performance Indicator Standard/Broad Key Performance Indicator
Area Area
Learning and 4. Ratio of students to teaching staff.
Management of 1. Students overall evaluation on the quality of their learning (Based on full time equivalents)
Quality experiences at the institution. Teaching
5. Students overall rating on the quality of their courses.
Assurance and (Average rating of the overall quality of their program on a five (Average rating of students on a five point scale on overall evaluation of courses.)

Improvement point scale in an annual survey final year students.) 6. Proportion of teaching staff with verified doctoral qualifications.
7. Percentage of students entering programs who successfully complete first year.
2. Proportion of programs in which there was independent 8. Proportion of students entering undergraduate programs who complete those
verification within the institution of standards of student programs in minimum time.
achievement during the year. 9. Proportion of students entering post graduate programs who complete those
programs in specified time.
3. Proportion of programs in which there was independent 10. Proportion of graduates from undergraduate programs who within six months
verification of standards of student achievement by people of graduation are:
(a) employed
external to the institution during the year. (b) enrolled in further study
17 (c) not seeking employment or further study 18

3
 PUBLISHED EXAMPLES OF KPIs PUBLISHED EXAMPLES OF KPIs
Standard/Broad Key Performance Indicator Standard/Broad Key Performance Indicator
Area Area
Student 11. Ratio of students to administrative staff Learning 14. Number of book titles held in the library as a
Administration and Resources proportion of the number of students.
Support Services 15. Number of web site subscriptions as a proportion of
12. Proportion of total operating funds (other than the number of programs offered.
accommodation and student allowances) allocated to 16. Number of periodical subscriptions as a proportion
provision of student services. of the number of programs offered.
13. Student evaluation of academic and career 17. Student evaluation of library services. (Average
counselling. (Average rating on the adequacy of rating on adequacy of library services on a five point
academic and career counselling on a five point scale scale in an annual survey of final year students.)
in an annual survey of final year students.
19 20

PUBLISHED EXAMPLES OF KPIs PUBLISHED EXAMPLES OF KPIs

Standard/Broad Key Performance Indicator Standard/Broad Key Performance Indicator
Area Area
Financial 22. Total operating expenditure (other than
Facilities and 18. Annual expenditure on IT as a proportion of the
Planning and accommodation and student allowances) per student.
Equipment number of students.
Management
19. Number of accessible computer terminals per
Faculty and Staff 23. Proportion of teaching staff leaving the institution in
student.
Employment the past year for reasons other than age retirement.
20. Average overall rating of adequacy of facilities and
Processes
equipment in a survey of teaching staff.
24. Proportion of teaching staff participating in
21. Internet bandwidth per user professional development activities during the past year.
21 22

PUBLISHED EXAMPLES OF KPIs PUBLISHED EXAMPLES OF KPIs

Standard/Broad Key Performance Indicator
Area Standard/Broad Key Performance Indicator
Research 25. Number of refereed publications in the previous year per full time Area
equivalent member of teaching staff.
26. Number of citations in refereed journals in the previous year per Community 31. Proportion of full time teaching and other staff
full time equivalent teaching staff. Service actively engaged in community service activities.
27. Proportion of full time member of teaching staff with at least one
refereed publication during the previous year. 32. Number of community education programs provided
28. Number of papers or reports presented at academic conferences as a proportion of the number of departments.
during the past year per full time equivalent members of teaching staff.
29. Research income from external sources in the past year as a
proportion of the number of full time teaching staff members.
30. Proportion of total operating funds spent on research.
23 24

4
EXAMPLE OF KPIs FOR ACHIEVEMENT OF STRATEGIC PLAN EXAMPLE OF KPIs FOR ACHIEVEMENT OF STRATEGIC PLAN
Teaching and Learning
• Time-to-graduate
• Outcome-based assessment results Institutional Development
• Postgraduate research • Strengthened External Advisory Committee roles
• High enrolment at all levels
• Student awards and internships • Faculty development programs
Research and Scholarship • Joint research labs with companies, research institutes, academic institutes
• Increased international profile of research programmes
• Funded research • Staff development and training
• High quality publications Societal Engagement
• International patents and awards
• Niche research areas to benefit social and economic development in the region • New degree and non-degree programmes
Internationalization • Alumni outreach
• Collaboration with regional and international collaborators
• International students at all levels • Participation in professional societies’ activities
• Hosting international conferences, workshops, symposia • Support of spin-off companies
• Students’ participation in regional and international competitions
• Accreditation of programmes by regional and international accreditation agencies 25 26

EXAMPLE OF KPIs FOR RESEARCH EXAMPLE OF FACULTY KPIs

Overall and in each academic unit: Each Faculty has KPIs which the University uses for setting management objectives,
• The number of ‘research active’ staff’. monitoring Faculty performance, and informing decisions on resource allocation and
• The number of main research areas. budgeting:
• The number of research projects funded by external sources, e.g. Macau Foundation, • Research grants
FDCT, EU, government agencies, NGOs, other external sources.
• Consultancy income
• The amount of external research funding.
• Number of academics producing high quality research outputs (% with average of 3
• The amount of internal research funding.
high impact publications)
• The number of papers in high impact journals.
• Student-staff ratios
• The number of papers in blind peer-reviewed journals.
• Quality of undergraduate entry (entry characteristics, e.g. grades awarded)
• The number of peer-reviewed conference papers. • Undergraduate student achievement (% of high degrees gained by graduates)
• The number of books published. • Employability (% of graduates who seek employment, gaining employment within 3
• The number of chapters in edited volumes. months of graduation)
• The number of citations of published work. • Employability (% of graduates proceeding to graduate level employment or study)
• The number of academic research outputs and rewards/prizes/awards. • Number of postgraduate research students (% of faculty student population)
• The number and types of international patents. • Student satisfaction (% of students in the faculty scoring more than 80% in the Student
• The number of partnership projects with other institutions. 27 Satisfaction Survey). 28

EXAMPLE OF KPIs FOR TEACHING EXAMPLE OF KPIS FOR TEACHING AND LEARNING
• Clarity and organization of instructor’s presentation of content, concepts, skills, and • USJ has policy statements, guidelines and support materials for promoting effective
techniques. teaching and learning.
• Clarity and suitability of instructor’s expectations, requirements, guidance and • USJ provides appropriate funding for learning and teaching activities.
directions for student performance. • USJ ensures adequate and readily accessible teaching and learning resources.
• Instructor’s helpfulness and availability during and outside class. • USJ ensures that student support for learning is provided and continuously enhanced.
• Usefulness, clarity and constructiveness of instructor’s preparation for and • USJ collects and uses feedback from staff and students for the effective management,
formative feedback on student assignments and performance. quality, development and continuous enhancement of teaching and learning.
• Instructor’s promotion of class participation and collaborative, interactive learning • USJ ensures that its programs are delivered in ways that encourages active and
and discussion. interactive learning between students, and that assessment reflects this approach.
• Instructor’s overall teaching effectiveness. • USJ ensures that its teachers are competent to teach in the area required.
• Appropriateness, relevance and suitability of course content. • USJ ensures that its staff-student ratios are sufficient to promote high quality teaching
• Organization of course content. and learning.
• Effectiveness of teaching strategies. • USJ uses outcomes-based approaches (OBA) in its programs, teaching, learning and
• Effectiveness of application and skill development from course content. assessment.
• Links between theory and practice. 29 • USJ provides faculty development in teaching and learning for all its academic staff.
30

5
 WHAT DO YOUR KPIs REALLY INDICATE? WHAT DO YOUR KPIs REALLY INDICATE?
• Number of students • Student dropout/retention rates
• Popularity; quality; program attractiveness; ease of entry; ease of completion; • Low dropout: lax demand/high demand; attractiveness of institution; quality
student support; effectiveness of publicity; resources; business plan; reputation . . . ? of instruction; reputation; opportunities elsewhere; costs; friendships . . . ?
• Gender • Staff turnover
• Equal opportunities; diversity; application; uptake; programmes offered . . .? • Loyalty; job opportunities elsewhere; pay and conditions; career development
• Number of staff and opportunities; workload; institutional culture and ethos . . . ?
• Recruitment; staffing policies; financial situation; pay and conditions; size of Faculty; • Nationalities of staff
workload; reputation; business plan . . . ?
• Internationalism: discrimination/non-discrimination; language issues;
• Number of staff at promoted levels recruitment policy; effective recruitment/poor recruitment; mission
• Management structure; financial situation; attractiveness of Faculty/work; pay and statement; pay and conditions; reputation . . . ?
conditions; opportunities for promotion; promotion policy; opportunities elsewhere;
reputation . . . ? • Nationalities of students
• Internationalism: discrimination/non-discrimination; popularity; language
• Staff-student ratios issues; opportunities elsewhere; attractiveness (e.g. scholarships); family
• Student support; low enrolment; few applications; student dropout; class size policy; matters; types of programme; friendships, reputation . . . ?
staffing problems; staffing benefits; business plan . . . ? 31 32

ADVICE FOR KPIS RESPONSIBLE METRICS

• Make sure that your indicators are valid and reliable – that they indicate Responsible metrics can be understood in terms of number of dimensions:
what they are intended to indicate, and do so reliably and credibly.
• Robustness: basing metrics on the best possible data in terms of accuracy
• Make sure your indicators are realistic. and scope;
• Make sure your indicators are easy to understand. • Humility: recognising that quantitative evaluation should support – but
• Make sure your indicators are connected to goals. not supplant – qualitative, expert assessment;
• Do not have too many indicators: as few as necessary but no fewer. • Transparency: keeping data collection and analytical processes open and
• Make sure that your indicators are not too easy but not so demanding as transparent, so that those being evaluated can test and verify the results;
to be unachievable in terms of success. • Diversity: accounting for variation by field, and using a range of indicators
• Keep the wording of your KPI as brief as possible, but no briefer. to reflect and support a plurality of research . . . across the system;
• Make sure that the KPIs fairly cover and represent the Action Plan’s • Reflexivity: recognising and anticipating the systemic and potential effects
goals and contents, i.e. they must be valid and reliable. of indicators, and updating them in response.
• Be VERY careful what you infer from KPIs; don’t over-interpret! (Wilsdon, J,. (2016) The Metric Tide. London: Sage, p. 138)
33 34

STRATEGIC OBJECTIVE 2: FOLLOWING UP KEY PERFORMANCE INDICATORS AND EVIDENCE

Three questions:
To provide the highest international academic standards
1. What do the KPIs and evidence really tell you about the achievement of
Write two Key Performance Indicators for this Strategic Objective. the goals, contents, intended outcomes and procedures set out in the
Action Plan, i.e. what do you want to do with the KPIs and
evidence/data?
2. Evidence of what, i.e. what does the evidence show and tell you?
3. What will you do now, as a consequence of (1) and (2)?

6
 WHAT DOES IT MEAN FOR ME?
1. You must know the KPIs very well.
2. You must be prepared to create and write KPIs, and to participate
constructively in discussions on the creation and feedback on KPIs and
draft documents.
3. You must be prepared to collect, collate, provide, analyse and report
evidence of using KPIs for improvement and development.
4. You must be prepared to respond to QAE-related requests from the QA
Office.
5. You must be prepared to defend the KPIs and to know their limitations.

37
THANK YOU
38