#7f267f

GUIDE

LEARNING FROM EACH OTHER

SHARING GOOD PRACTICE THROUGH BENCHMARKING

'P O L I F O N I A'
W O R K I N G G R O U P
O N Q U A L I T Y
E N H A N C E M E N T ,
A C C R E D I T A T I O N
AND BENCHMARKING
 GUIDE

LEARNING FROM EACH OTHER

SHARING GOOD PRACTICE THROUGH BENCHMARKING

WWW.POLIFONIA.EU

'P O L I F O N I A'
W O R K I N G G R O U P
O N Q U A L I T Y
E N H A N C E M EN T ,
A C C R E D I T A T I O N
AND BENCHMARKING
October 2014
Design: Daniela Tomaz

ISBN: 978-2-9601378-4-2

DISCLAIMER:
This project is funded with support from the European Commission. This report reflects only
the views of the authors and the Commission cannot be held responsible for any use which
might be made of the information contained herein.
 CONTENTS

ABBREVIATIONS 06

FOREWORD 07
Purpose of this Guide 07
Context of the Project 07
Authors of this Guide 07
Thanks 08

1 WHAT IS BENCHMARKING? 09

2 WHY USE BENCHMARKING? 11

3 EXISTING TYPES OF BENCHMARKING 12

4 HOW TO SET UP A BENCHMARKING PROJECT? 15

4.1 Gaining Commitment for your Project at a Senior Level within your Institution 15
4.2 Selecting Benchmarking Partners and Forming a Benchmarking Group 16
4.3 Defining the Benchmarking Project precisely 17
4.3.1 Topic 17
4.3.2 Purpose/Focus 18
4.3.3 Approach 19
4.3.4 Timeframe 20
4.4 Formulating and Agreeing Benchmarking Questions and/or Indicators 21
4.5 Collecting Data, Cleansing and Reporting 26
4.6 Resourcing and Managing a Benchmarking Project 27
4.7 Producing an Action plan and implementing it 28

5 CHALLENGES 29

6 SOURCES FOR FURTHER INFORMATION 31

ABBREVIATIONS

AEC Association Europenne des Conservatoires, Acadmies de Musique et Musikhochschulen

EQF European Qualification Framework for Lifelong Learning
HME Higher Music Education
HMEI Higher Music Education Institution
PDD Polifonia Dublin Descriptors
WG Working Group

6
ABBREVIATIONS

7
 FOREWORD
PURPOSE OF THIS GUIDE
The purpose of this guide is to explain how the method of benchmarking can be used by music education institutions as a
tool for quality enhancement and, when undertaken in an international context, for internationalisation.This guide aims to
introduce those who are new to the topic with the method of benchmarking, as well as to assist those who are planning
and managing a benchmarking project within a music education setting. It contains an explanation of benchmarking, step-
by-step guidelines and examples of case studies to demonstrate how these steps can be achieved.

CONTEXT OF THE PROJECT

Since its launch in 2004, the ERASMUS Network for Music Polifonia [1] has proactively addressed European higher
education policy issues (such as mobility, research, quality assurance and accreditation, admission and assessment, links
with the profession, etc.) from the perspective of higher music education (HME). Through the consistent output of high
quality products, it has been able to raise the awareness of these issues throughout the sector, which has subsequently
supported the implementation of these outputs at both institutional and national levels. From a general higher education
point of view, Polifonia has often been cited as a good example of what can be achieved through a subject-specific
and European-level approach to the modernisation agenda that was initiated by the Bologna Declaration and is now
embedded in the Europe 2020 strategy.
The Polifonia project, supported by the ERASMUS Networks programme of the European Union [2], is the biggest
European project on professional music training to date. The first project cycle ran from 2004-2007, the second from
2007-2010 and the third, jointly coordinated by the Royal Conservatoire The Hague and the Association Europenne
des Conservatoires, Acadmies de Musique et Musikhochschulen (AEC), from 2011-2014. In this last cycle, experts from
55 different institutions in the field of higher music education and the music profession were involved, coming from 26
European countries and 4 countries outside Europe.
The overall aim of ERASMUS Network for Music Polifonia is to promote innovation in and enhance the quality,
attractiveness and accessibility of European higher music education through cooperation at the European level.

AUTHORS OF THIS GUIDE

Polifonia Working Group (WG) 3 on Quality Enhancement, Accreditation and Benchmarking studied the feasibility of
establishing a European-level accreditation agency in higher music education and an international benchmarking system
for higher music education. The WG improved and reformulated the AEC Criteria for Institution and Programme Review in
order to produce three sets of standards for the external evaluation of higher music education institutions: Standards for
Institutional Review, Standards for Programme Review and Standards for Joint Programme Review. In relation to benchmarking,
the WG studied examples of benchmarking exercises, including some in which higher music education institutions were
engaged, and wrote the present Guide.

[1]
For more information about the Polifonia project, visit its website www.polifonia.eu.
The Erasmus academic networks were supported by the Lifelong Learning Programme (LLP) of the European Commission, the
[2]

European Funding programme in the field of education and training, in place between 2007 and 2014.The Erasmus academic networks
were designed to promote European co-operation and innovation in specific subject areas. For more information on this funding
programme, visit the website http://eacea.ec.europa.eu/llp/erasmus/erasmus_networks_en.php.

8
 FOREWORD

The Working Group on Quality Enhancement, Accreditation and Benchmarking (2011-2014) was composed of:
Stefan Gies (Chair - Hochschule fr Musik Carl Maria von Weber, Dresden)
Dawn Edwards (Royal Northern College of Music, Manchester)
Sam Hope (National Association of Schools of Music - USA, Reston)
Grzegorz Kurzynski (K. Lipinski Academy of Music in Wroclaw, Wroclaw)
Orla McDonagh (The Royal Irish Academy of Music, Dublin)
Claire Michon (Centre dEtudes Suprieures de Musique et de Danse de Poitou-Charentes, Poitiers)
Janneke Ravenhorst (Koninklijk Conservatorium Den Haag, The Hague)
Valentina Sandu Dediu (Universitatea Nationala de Muzica Bucuresti, Bucharest)
Vit Spilka (Janek Academy of Music and Performing Arts, Brno)
Terrell Stone (Conservatorio di Musica di Vicenza Arrigo Pedrollo,Vicenza)
Linda Messas (European Association of Conservatoires AEC Office, Brussels)

THANKS
The WG members would like to express their deep gratitude to representatives of the Royal Northern College of
Music (RNCM) and its partners for sharing with the WG the experience gained within their International Benchmarking
Exercise (IBE), and for enabling the WG to use the IBE project as a case study.
The WG is also very grateful to Mrs. Hilde Sels, Quality manager at Thomas More University College and leader of the
benchmarking project within the KONDOR-project for sharing her experience during an interview, following the article
she published [3], and for her good advice.

[3]
See chapter 6: Sources for further information.

9
1. WHAT IS BENCHMARKING?
Benchmarking is a tool used to analyze and improve performance, which was developed and popularised within the
business industry. It is built on questions regarding the performance of an organisation when compared with other
organisations that offer similar products under comparable conditions. Its goal is to discover which strategies are most
successful in dealing with similar issues.
The European Centre for Strategic Management of Universities (ESMU) defines benchmarking as the voluntary process of
self-evaluation and self-improvement through the systematic and collaborative comparison of practice and performance
with similar organizations. This process allows institutions to identify strengths and weaknesses, and to learn how to
adapt and improve organisational processes in order to face growing competition (ESMU, 2008).
After exploring and discussing several concepts of benchmarking and its use in higher education, the
Polifonia WG decided to adopt the ESMU approach and focus on benchmarking aimed only at internal
quality enhancement (QE). Using benchmarking in terms of internal QE ensures that the results of such a project
can be used to support internally driven QE objectives rather than externally driven factors which may not always be in
an institutions best interests. Indeed, as mentioned during discussions in the working group, benchmarking is like a stick:
you can use a stick to hit something or you can use it as a walking stick. The working group wants to promote the
latter approach (also known as benchlearning) in this guide to demonstrate that the use of benchmarking
as a developmental tool can be very helpful for all music education institutions.
The working group has therefore chosen the following definition:

Benchmarking is a learning tool aimed at improving performance based on the comparison between
institutions / departments / programmes that share common objectives and operate under comparable
conditions. Benchmarking involves choosing appropriate partners at national or international levels,
evaluating and comparing ones home institutions practices and/or performance with those of its partners,
and sharing best practice.

The scope of a benchmarking project can range from very small to large institutional benchmarking and is determined
by the conservatoire itself. Benchmarking can be used to compare two or more institutions or to compare two or more
units within an institution.

EXAMPLES:

Small benchmarking project:

Conservatoire A and conservatoire B compare their processes for examinations. Elements they can look
at:
Regulations: how detailed and how flexible are examination requirements? What are the processes to
prepare and set up examination?
Composition of the jury: are there national/international external examiners? Is there a fixed number
of jury members? Are jury members specialists in the field examined? Are graduated students members
of juries?

10
 WHAT IS BENCHMARKING?

Recital programmes: what is the duration? Are there specific demands for the composition of the
programme?
Public character: is the examination open to the public?
Criteria for evaluation: are there evaluation criteria? Which elements are assessed? How are these
criteria taken into consideration by students/teachers/evaluators?
Feedback to the student: is a feedback given to the student? If so, when, and how?
Resources: are other musicians accompanying or playing with the student provided/paid by the
institution?
Link with the study programme: how are examinations embedded in the study programme?

Large benchmarking project:

Conservatoire A and conservatoire B compare elements such as:
Structure and organigram
Governance policy
Budget - for example, how is the budget spent on different areas in the institution (e.g. staffing, teaching,
research)?
Number of staff in administrative departments (finance, human resources, and estates), cost of staff in
these departments, training provided to staff, number of senior staff, qualifications of senior staff, type
of staff contract, etc.

Benchmarking aims at quality enhancement. It is an internal process which depends on each institution and the issues it
wants to explore, and a process which only works if participants can talk freely and remain open.
However, benchmarking can be used for internal quality assurance purposes as part of an institutions preparation for a
formal accreditation procedure.
Some of the information shared between the partners may be sensitive/confidential to each participating organisation.
Thus, before the start of the project, the level of confidentiality of the data gathered needs to be agreed on by all the
partners: is the data collected for internal use only or also for external use? And, in the latter case, with whom will this
data be shared? An agreement should be reached with the partners to guarantee that the level of confidentiality defined
is respected and, where appropriate, to determine what data can be published.

11
2. WHY USE BENCHMARKING?
Benchmarking is not an end in itself. It is a strategic tool that shapes processes of change within an institution and
supports reliable and informed improvement.
Benchmarking has many benefits!
It identifies and provides an impetus for reflection and change;
It provides a forum for sharing knowledge and experience in a risk-free environment which promotes
the best results;
It enables partners to compare good practice and it highlights possible areas for improvement in
institutional practices/procedures;
It provides data to inform decision-making, and it supports decision-makers in achieving defined goals
and implementing appropriate measures;
It provides the opportunity to build networks and expand contacts;
When undertaken in an international context (i.e. with international partners), it develops both
international relations and provides an international perspective on quality assurance;
It can be a first step in setting up a joint programme or a strategic partnership with other music
institutions;
It brings greater awareness and insight as well as mutual respect.

CASE STUDY 1: the International Benchmarking Exercise (IBE) project

The Royal Northern College of Music (RNCM) in Manchester initiated a project in 2011 with six partners
from Europe, North America, the Far East and Australia. All partners joined the project because they were
interested in finding out what they were good at and not good at, and wanted to focus on what they could
improve. For example, one of the partners explained that the project had so far provided his conservatoire
with very useful management information, for example about a) the composition of the curricula (by
comparing the percentage allocated to theory classes as part of the overall curriculum in each partner
institution) and b) about the total amount of teaching hours allocated to one-to-one instrumental and
vocal tuition, a subject often addressed by teachers.

12
 EXISTING TYPES OF BENCHMARKING

3. EXISTING TYPES OF BENCHMARKING

There are manifold and in some cases quite differing approaches to benchmarking. Not all of them can be suitably
applied in the field of higher music education. Benchmarking as a tool in higher music education is relatively recent, and
traditionally has been restricted to external process and/or performance benchmarking, based on comparisons
between similar institutions.
Indeed, benchmarking can be internal (comparing performance between different units within an institution) or
external (comparing performance with units, institutions or organisations outside ones own institution, in the higher
education sector or across sectors).
Within these broader categories, there are two specific types of benchmarking: process benchmarking and performance
benchmarking.
Although it may be useful to use either type of benchmarking, a benchmarking exercise is most often a combination of
both types. Institutions compare performance to know their strengths and weaknesses, but then learn by comparing
their processes: looking at the processes will usually explain the differences in performance.

Performance benchmarking (quantitative)

This involves comparing the performance levels of institutions and allows the benchmarking partners to assess their
competitive position. This information can then be used for identifying opportunities for improvement and/or setting
performance targets.

Examples of internal performance benchmarking:

Comparing student satisfaction levels on evaluation procedures between departments
Comparing the number of students retained and graduated in various disciplines within the conservatoire
(violin, voice)
Comparing the percentage of graduates working primarily as performers, composers and conductors
between two or more departments within the conservatoire
Comparing the percentage of graduates finding positions as teachers within two years of graduation
Comparing the number of students going in and out on exchange per year between disciplines or
departments within a conservatoire

Examples of external performance benchmarking:

Comparing the proportion of students retained and graduated between institutions
Comparing the proportion of international competition prize-winners between institutions
Comparing the number of partnerships with professional music ensembles and organisations (e.g.
orchestras, music industry providers etc.) between institutions
Comparing the percentage of annual budget spent on staffing (all employees) between institutions

13
It is important to note that performance benchmarking should not be considered as ranking or used towards this
purpose. The results of the comparison exercise reflect very diverse national contexts, legal and financial situations and
are linked to the institutions own mission and vision. It does not make sense to develop a league table based on this data.

Process benchmarking (qualitative)

This is where institutions identify and study institutions that are high performers in particular areas of interest. The
processes themselves of these institutions are observed and studied rather than just the associated performance levels.
Process benchmarking involves the whole process of identifying, capturing, analysing, and implementing best practices.

Examples of internal process benchmarking:

Comparing assessment procedures between two departments of the conservatoire
See the specific questions in the Small benchmarking project example on page 9
Comparing internal communication processes between two or more faculties/departments
How does the programme communicate with its students and staff?
How do students and staff communicate?
How does the programme communicate with its external teachers/examiners/ other external
people who are involved in the programme?
How does the programme ensure the continued effectiveness of its communication systems?

Examples of external process benchmarking:

Comparing admissions procedures between two or more conservatoires
What are the entrance requirements?
In what ways do they assess the abilities (artistic/technical/academic/pedagogical) of the applicants
to successfully complete the study programme?
Do admissions criteria exist?
How have they been developed and how are they implemented?
Comparing practice room management between two or more conservatoires

TIPS AND GOOD PRACTICE!

When designing a benchmarking project, remember that there are many approaches. You could - for
instance - learn a lot from your own institution by starting with an internal benchmarking project,
investigating a topic that affects many departments.
And quite often, thinking out of the box can bring surprising results.

14
 EXISTING TYPES OF BENCHMARKING

THINK OUT OF THE BOX!

A low budget airline company wanted to use benchmarking to identify possibilities to organise processes more efficiently
in order to reduce costs. One of the most costly processes for this company was the length of stay at an airport: an airline
company has to pay for every hour the plane is at the airport, so reducing the length of stay could save considerable costs.
Comparing the processes taking place at the airport (landing and take-off, loading and unloading of baggage, suitcases,
food, garbage etc.) with other airline companies would probably not result in interesting information, since this company
was already low budget and would probably have more efficient processes already.
Therefore, the airline company tried to think differently about finding a benchmarking partner. They did not focus
on their own field anymore, but focused on the exact processes they wanted to benchmark. Eventually they found a
surprising new benchmarking partner, which had nothing to do with the business of airlines but everything to do with
the process that needed to be improved: the formula 1 Pitstop.
It could therefore be interesting and enriching for a conservatoire to study processes implemented in other sectors,
within or outside education, and to use such observations to improve its own processes, for example in relation to
internal communication, governance structures, or other areas.

15
 4. HOW TO SET UP A BENCHMARKING PROJECT?
Benchmarking is only effective as a developmental tool when the benchmarking partners volunteer to participate, when
the goals associated with the process are clearly defined in advance and if an institution decides on its own to adopt a
procedure recommended through the benchmarking project which is adequate and in line with the institutions mission
and vision.
The main steps to set up a benchmarking project are listed below:

4.1 GAINING COMMITMENT FOR YOUR PROJECT AT A SENIOR LEVEL WITHIN YOUR INSTITUTION
If the project is to be sustainable and result in institutional change, active support at a senior level is needed. It is
important that institutional leaders are convinced that benchmarking is a helpful tool, that they are engaged in the
process and do not delegate the responsibility solely to other people in the administration.
Benchmarking is a tool which helps institutions to change, but it can only work effectively if the organisation benchmarking
itself is committed to that change, and is clear that that change is compatible with the institutions culture.[1]

CASE STUDY 1: the IBE project (institutional level project)

As mentioned above, the International Benchmarking Exercise (IBE) project was initiated in 2011 between
seven partners from Europe, North America, the Far East and Australia. The aim of the project is to obtain
from institutions a set of comparative data against which they can benchmark/compare themselves and
which can be used for internal analysis/comparisons (and not to obtain a common set of threshold data
that institutions should meet).
The people that attend to the IBE meetings are the Directors/Vice-Rectors themselves.

CASE STUDY 2: the KONDOR project (programme level project)

Higher education institutes in Flanders that are part of the KU Leuven Association[2] initiated a benchmarking
exercise in September 2010 as part of a project called KONDOR Een Kwaliteitssysteem inzake ONDerwijs
Ontwikkelen en Realiseren, in English Develop and realise a quality system for education).
The first aim of this exercise was building expertise among the quality managers, as the involved institutes
had limited experience and expertise with benchmarking. A call for participants was launched in 2011,
asking each participating programme to appoint at least two persons: the quality manager as well as a
programme coordinator.
In addition, it was agreed from the start that the outcome of the process would be an action plan and
leaders of the involved programmes were asked to commit to using the results of the benchmarking
project and take action accordingly (and possibly adjusting the programmes strategy if needed). Once the
project produced some results, the quality working group of the KU Leuven association, selected common
topics on which to work.

ESMU. (2010). Benchmarking in European Higher Education: A University Benchmarking Handbook. (P. Benneworth, Red.) Brussels:
[1]

ESMU
[2]
The KU Leuven Association is a network linking university colleges across Flanders and Brussels with the University of Leuven (KU Leuven).

16
 HOW TO SET UP A BENCHMARKING PROJECT?

4.2 SELECTING BENCHMARKING PARTNERS AND FORMING A BENCHMARKING GROUP

Your first task is to compose a group of institutions or departments/programmes, and contact persons: the Benchmarking
Group.
The success of the project and quality of the outcome(s) will depend on the composition of the benchmarking group and
the commitment of members of the group to the project.
Benchmarking means relying on having a peer group with a shared strategic interest (). It is necessary to have a
dialogue between committed and self-critical institutions who really want to learn and improve their practice.This means
that all the participants in the collaborative benchmarking exercise need to really believe that the benchmarking exercise
is worth the cost and the effort because it will help them to improve their overall strategic performance. If you are not
committed to improvement, then you owe it to the comparators not to waste their time by getting involved and then
falling out when the going gets tough.[3]
It is important to involve institutions which consider each other as equal partners. Institutions should be aware that
some internal change is needed, but also have strong points/ good practice to offer.

CASE STUDY 1: the IBE project

Partners were selected through an existing network of professional partners that the conservatoire had
long-standing professional/academic links with. Three institutions were added to the group later on in
the process following their expression of interest in joining the benchmarking exercise and with the
agreement of the existing benchmark partners.

CASE STUDY 2: the KONDOR project

The project was carried out by programmes in institutions that are part of the KU Leuven Association,
which unites the University of Leuven with eleven university colleges. The call for participants mentioned
above was launched within the Association.The call required each programme willing to get involved in the
project to appoint at least two persons, and it indicated precisely the amount of work and commitment
expected from these persons: attendance to five day-long meetings and preparatory work in between
meetings.

TIPS AND GOOD PRACTICE!

Choosing partners
We recommend that you choose partners who:
come from an institution with a similar ethos, culture and profile; the mission, size, and target group of
the institution should also be comparable;
are an existing contact, for example, through an existing network;
have access to the relevant data and commit to collecting and submitting this by agreed deadlines;

ESMU. (2010). Benchmarking in European Higher Education: A University Benchmarking Handbook. (P. Benneworth, Red.) Brussels:
[3]

ESMU

17
 are willing to respect confidentiality and follow guidance on the operation of the project and sharing
of data;
have the resources required to support and participate actively in the project, not only financially but
also through the involvement of competent staff members.
Other tips
Although it depends on the scope of the project, it is believed that a benchmarking group should ideally
involve 5 to 10 institutions.With less than 5, you may not have sufficient points of view expressed; with
more than 10, it may be difficult to have efficient discussions and to ensure participants involvement;
It should be agreed from the start if new partners can enter the project once the initial group has been
established;
It is suggested that all participants including the institutions leadership sign a code of conduct addressing
issues mentioned above such as data confidentiality, sharing and publication, commitment to complete
the tasks and attend the meetings, deadlines to be respected, etc.). This document should not only
cover the project period itself but also the post-project period, so that partners agree on how the data
will be used even when the project is finished;
The mode of operation of the project should also be agreed on (virtual, meetings tagged on other
meetings, special meetings?).

4.3 DEFINING THE BENCHMARKING PROJECT PRECISELY

There are two ways to initiate this step: you can choose the topic based on the partners selected, or choose the partners
based on the topic selected. We have chosen to present examples of the first type below, where the partners were
selected first and then agreed on topics for the benchmarking exercise.

4.3.1 TOPIC
When all members of a benchmarking group meet the first time, it is important that each institution has prepared a list
of topics and subtopics they wish to investigate. As it is only realistic to do a benchmarking exercise if at least several
institutions are aiming at comparable objectives and are sharing approximately similar conditions, the first meeting of the
members of a group is primarily used to agree on a common list. The points on the list should be representative of the
institutions objectives, and should reflect in a balanced manner the wishes and interests of the various members.

TIPS AND GOOD PRACTICE!

We advise you to use the MusiQue[4] Standards for Institutional/Programme Review as a starting point for creating a
topic list. This can give you an overview of broad areas the Benchmarking Group (BG) might like to investigate. The
standards are divided into eight areas:
1. Mission,Vision and Context
2. Educational processes

[4]
MusiQuE Music Quality Enhancement is an external evaluation body dedicated to the continuous improvement of the quality of
higher music education across Europe and beyond and, through its accreditation, quality enhancement and advisory services, to assist-
ing higher music education institutions in their own enhancement of quality. MusiQuE takes over the responsibility for the institutional
and programme reviews conducted by the AEC since 2008.

18
 HOW TO SET UP A BENCHMARKING PROJECT?

3. Student profiles
4. Teaching staff
5. Facilities, Resources and Support
6. Communication, Organisation and Decision-making processes
7. Internal Quality Culture
8. Public interaction
In these eight areas, some initial questions/indicators of good practice are suggested. Once the benchmarking group has
chosen an area to focus on, further questions/indicators can be expanded and decided upon by the group.

EXAMPLE:
The Polifonia Working Group (WG) decided to undertake a benchmarking exercise to experience the
process. MusiQuE Standards Area 4: Teaching Staff was chosen as the theme. This area includes a question
on the policies and practices in place within an institution to support teaching staff in their work (i.e.
the professional development of teaching staff): Is there a policy in place for continuing professional
development of teaching staff?
It was agreed that one working group member would formulate a set of questions around this topic, which
would enable the group members to collect information about their institutional situation and to share it
with one another. The questions produced are listed in section 4.4 below.

4.3.2 PURPOSE/FOCUS
It is important to define what you want to find out with the benchmarking project, why you are interested in this and
what you/your partners will do with the results.

CASE STUDY 1: the IBE project

The aims of all partners when committing to this benchmarking exercise were:
To strengthen the links between the institutions of the network
To share best practices
To have a lobbying/political advocacy tool (for example in talks with donors) - this is what our
colleagues in the world are doing
To look at trends provided by the data once the exercise has been running for some years

CASE STUDY 2: the KONDOR project

As mentioned earlier, the first aim of this exercise was building expertise among the quality managers, as
the involved institutes had limited experience and expertise with benchmarking.
Before the exercise started, the quality managers identified issues they were all struggling with. One of
them was selected to be the theme for the benchmarking exercise: How to effectively handle results of

19
 surveys and performance indicators.
This theme was of interest for quality managers and was considered as strategically important in the light
of quality assurance and accreditation. In addition, all participants were looking for answers and were eager
to learn from the others who were dealing with this matter.
In the first stage of the exercise, participants were asked to discuss the above-mentioned theme and to
work on dividing it into smaller subthemes. The group defined three subthemes: effective analysis and
interpretation of results; development of an action plan; and closing the circle, i.e. guaranteeing that also in
the process of handling results there is a systematic evaluation and that there is a link with the strategic
plan.

4.3.3 APPROACH
You need to choose an approach appropriate for the benchmarking project:
Will it be an Internal vs. External project? This affects the choice of benchmarking partners;
Will the project be based on Process/Performance Benchmarking?
Examine the Benchmarking Circle to decide which strategy is best suited to the project: benchmarking
can be considered in terms of four dimensions. It is useful to consider each one of these when designing
a benchmarking project:

TARGET
DIMENSION
What is the aim of the
benchmarking
project

RESULTS DIMENSION The PROCESS DIMENSION

What are the expected Benchmarking How will this be
results at the end of Circle achieved?
the project?

CONDITIONS
DIMENSION
What information/
resources are
necessary to achieve
this?

CASE STUDY 1: the IBE project

The partners first worked on agreeing a list of questions that addressed the themes they had selected to
explore. In a subsequent step, they collected quantitative and qualitative data to answer this questionnaire.

20
 HOW TO SET UP A BENCHMARKING PROJECT?

Since then, meetings are dedicated to analysing and interpreting this data: benchmarking group members
explain the data concerning their institution to the others, interpret it in the light of their national situations
and contexts, compare/contrast the differences in the data between institutions, share their practices, etc.
In addition, the areas addressed by the questionnaire have been progressively extended. The collection of
data on an annual basis will enable the observation of trends after a few years.

CASE STUDY 2: the KONDOR project

The participants first worked on formulating a set of indicators related to the subthemes chosen, which
were realistic and could be measured. The next step was to elaborate upon these indicators by describing
four levels of performance through which these indicators could be met: basic, standard, good and excellent.
The excellent level was the benchmark, which was defined in this project as the best possible performance.
The standard level was what one might consider the normal level of performance. With that in mind the
basic level would be less than satisfactory.
The main part of the exercise was for participants to evaluate their own practice against these indicators
and assess their level of performance, with the aim of reflecting on their own practice and on what needed
to be improved. In addition, sharing the results of the self-assessment with all the partners made it possible
for them to learn about good practices implemented by other partners, and for the whole group to identify
issues to be worked on in future. Finally, the exercise finished with the task for each participant to draft an
action plan based on the results of the benchmarking.

4.3.4 TIMEFRAME
It is important to decide if the benchmarking project will be based on a single data collection or an annual process. Once
the timeframe is set, various dates/deadlines that are possible for all benchmarking partners should be agreed, based on
conditions within their own institutions.

CASE STUDY 1: the IBE project

Phase 1 ran from March 2011 to September 2012. It consisted of an initial data collection and was mostly
focused on quantitative information.
Phase 2 ran from October 2013-February 2014. It aimed at gathering qualitative data to expand on some
of the information gathered in Phase 1, and included the setting up of an online database and data cleansing.
During phases 1 and 2, partners met once a year during the AEC annual Congress to discuss the data
gathered.
The following phase is the collection of data through the online database annually.

CASE STUDY 2: the KONDOR project

The call for participants was launched in 2011 and the project completed in August 2012. Although it was
not possible within this project, the project manager recommended that information be collected one year
later to follow up on the measures undertaken in each institution following the benchmarking process.

21
TIPS AND GOOD PRACTICE!
Things to avoid:
Do not try to address too many questions
Do not be too ambitious in the scope of your project
Do not start without a clear project plan

4.4 FORMULATING AND AGREEING BENCHMARKING QUESTIONS AND/OR INDICATORS

The benchmarking group needs to agree what to measure and for what purpose, to come up with a common definition of
relevant indicators and parameters in order to avoid misunderstandings and to limit the range of possible interpretations
as far as possible. Either all partners work together from the start on building indicators, or the project may have a
coordinator who will draw up a list of questions which should then be discussed and agreed with the partners.
Data may be qualitative and/or quantitative. It is important to combine qualitative indicators (i.e. approaches) with
quantitative indicators (i.e. numbers, data). Indeed, quantitative data alone is not fully significant in determining how
an institution fulfils its duties. Institutions are involved in different contexts and follow different tasks; such differences
are not reflected in quantitative data and neither is the efficiency of the implemented processes. If quantitative data
dominates, there is a danger that the solutions found are neither tailor-made nor appropriate to a specific and unique
situation. Working on qualitative indicators will help to raise questions such as why? or is this really good practice
etc. Also, if quantitative data dominates, there is a danger that the competitiveness of the higher education outputs (the
number and performance of graduates, their competitiveness on the labour market, their flexibility or their contribution
to social development) will underestimate the real potential for benchmarking and quality enhancement.
Three examples are provided below:
A set of questions formulated by the Polifonia Working Group in its work to simulate a benchmarking
exercise. This simulation gives some insight into the process of formulating discussion questions and
illustrates some issues that may be raised in the discussion.
The Case Study of the IBE project, which illustrates the kind of questions that can be formulated in a
music benchmarking exercise, when the exercise is based on the collection of data.
The Case Study of the KONDOR project, which illustrates the kind of indicators and levels of
performance which can be formulated, when the exercise is based on comparing ones institution
against those indicators that are jointly agreed upon.

EXAMPLE OF THE POLIFONIA WG3 BENCHMARKING EXERCISE ON PROFESSIONAL

DEVELOPMENT FOR TEACHING STAFF
The questions suggested by one of the working group (WG) members to simulate an international
benchmarking exercise that focused on professional development for teaching staff were as follows:
1. What is your institutions annual budget for staff development?
2. Are full-time / tenured staff required to participate in staff development as part of their contract?
3. Are part-time / non-tenured staff required to participate in staff development as part of their
contract?

22
 HOW TO SET UP A BENCHMARKING PROJECT?

4. Does your institution have a staff development framework / strategy?

5. Does your institution offer incentives / rewards for staff to participate in staff development?
6. How many hours per year on average does your staff spend in staff development activities?
7. What are the main types of staff development offered by your institution?
8. What percentage of your staff participates in staff development activities?
9. Who has responsibility for the professional development of your teaching staff?
10. Does your institution monitor the engagement of staff in professional development?

These questions were circulated to all WG members who replied with their comments and suggestions
to improve the questions. As a result, the questions became longer and more complex. The adjusted list of
questions was then sent to the WG members, who then tried to answer them and collect the relevant data.
At its following meeting, the Polifonia WG discussed this adjusted list of questions and the preliminary data.
The purpose of this discussion was to simulate a typical benchmarking group meeting aimed at finalizing
the list of questions to be used for the benchmarking exercise, and to find out the types of issues likely to
be raised in such a meeting.

Some of these issues are provided here as example:

How shall the terms professional development be defined? Does it include performances? Presenting
at conferences? Mobility?
Certain questions still need to be clarified.
Should additional data be collected to complete the picture (such as the total budget for staff training)?
Certain questions are irrelevant in certain countries (for example, in countries where there is no
institutional budget for staff development but a national funding decided by the ministry and allocated
by the unions). How can this be handled in such an exercise?

TIPS AND GOOD PRACTICE!

The only way to deal with this diversity is to put together institutions which share common objectives
and operate under comparable conditions and to continue working on the questions until you reach
common questions which everyone understands and is able to answer.
This first result should not be a reason to stop the exercise but rather an encouragement to explore
what means would be necessary to reach the end, i.e. what could be developed by each institution to
enable them to answer the various questions.
It is important to focus on the effectiveness of the processes or measures that are being compared.
For example, when comparing admissions requirements, it may be advisable to additionally compare
student completion rates; or, when comparing the number of practice rooms available in different
conservatoires, it may be advisable to also compare the results on the availability of practice rooms in
student satisfaction surveys.

23
 CASE STUDY 1: the IBE project
Key departments and areas at RNCM were asked what data they felt would be important and useful
in establishing a common database. Reponses included finance, research, the curriculum, governance,
employability, student services and facilities and resources. From this list, 13 main categories to be used
in the benchmarking exercise were established, which were then sent to the partner institutions for
agreement/approval.
Specific questions relating to each category were then drawn up by RNCM. The questions were sent to
partners who were asked if they wanted to collect other types of data. In total the final benchmarking
questionnaire comprises 90 questions. Some simply require a yes/no answer but the majority requires
actual data. Institutions also supplied their mission/vision statements. Questionnaires were then sent to
each participating institution.
Examples of questions developed:
Questions under governance included:
Do you have an independent governing body (please answer 1=yes/0=no)?
Do you have autonomy as regards resource allocation within your institution/faculty/school budget
(please answer 1=yes/0=no)?
Do you have autonomy with regard to curriculum development within your institution/faculty/
school (please answer 1=yes/0=no)?
Questions under Research included:
What is the total number of research/3rd cycle students?
What is the percentage of fail rate?
What is the percentage pass rate at completion?
What is the number of 1:1 tutorial support per student per annum?
Questions under Destinations included:
Percentage of undergraduate/1st cycle students progressing to postgraduate study at their home
institution per annum
Percentage of undergraduate/1st cycle students progressing to postgraduate study in another
country per annum
Percentage of graduates working primarily on other parts of the music profession (e.g. arts
administration, music therapy)
Percentage of graduates pursuing careers outside music
Under Performance opportunities:
Average number of public performances involving students promoted on campus per annum
Total number of public performances involving students promoted off campus per annum
Do you offer an agency service for paid student engagements off campus (please answer 1=yes/0=no)?

24
 HOW TO SET UP A BENCHMARKING PROJECT?

TIPS AND GOOD PRACTICE!

The answers to the questions will show how clear the questions are.
The type and volume of data will depend on the focus of the project.
Only measure what needs to be measured!
If the project is international, as in the above case study, it is important that all partners have a clear
understanding of terminology, definitions, the meaning of each question and what type of data is
required this requires discussion and negotiation. It may be helpful to draw up a Glossary of Terms,
which is a key element in ensuring common understanding.
See as an example the glossary of terms developed by the International Benchmarking Exercise partners for the first
phase of the project (quantitative phase). The terms listed needed clarification as different terminologies were used in
the various countries where partners are based. When partners wanted to share information about the academic staff,
or the student union, they first had to agree on what this term covered.

CASE STUDY 1: the IBE project

Abstract of the glossary of terms developed by IBE partners
N.B.This glossary is only provided here as an example, as all benchmarking groups will have to agree on their own
understanding of terms, and choose the terms according to the area they will want to investigate.
Academic Staff (or Faculty) includes personnel whose primary assignment is instruction, research,
or public service. This includes staff personnel who hold an academic rank with titles such as professor,
associate professor, assistant professor, instructor, lecturer, or the equivalent of any of these academic
ranks. The category includes personnel with other titles, (e.g. dean, director, associate dean, assistant dean,
chair or head of department), if their principal activity is instruction or research[5]. It does not include
student teachers or teacher aides; source: http://stats.oecd.org/glossary
Completion Rate is the percentage of students completing their last year of studies;
Conversion Rate is the percentage of applicants who take up a place as a student/the number of students
who applied for a study place, got accepted and started the programme;
Home Students are the students who either originate from the country where they study or have been
resident in it for 3 years prior to starting studies;
Overseas Students are students who originate from countries outside of EU;
Partnership is a formalised relationship through Memorandum of Understandings, agreements, contracts
with education providers, professional bodies and associations, employers, etc;
Sabbatical Officer is a full- or a part-time officer elected by the members of the student union or
association;
Student Union is a student organisation staffed and run by students.

[5]
Does not include those faculty in administrative-only positions.

25
 CASE STUDY 2: the KONDOR project
The participants worked on producing sets of indicators for the three sub-themes chosen. They first
produced a long list of indicators, which they then had to narrow down. This part of the exercise was
challenging and many questions were raised. Eventually, the following indicators were agreed upon:
For analysis and interpretation the consensus indicators were:
professional level of the analysing team
process of analysing and interpreting
communication of the results to stakeholders
For developing an action plan they were:
availability of decision criteria and targets
formulation of goals
action plan itself
For closing the circle, these indicators were:
evaluation and adjustment of the tools used to measure
effectiveness of actions for improvement
link between actions and strategic goals of the program
As an example, the description of the four levels of performance for the indicator communication of
results to the stakeholders was:
Basic: the program informs stakeholders occasionally about results of surveys;
Standard: in less than half of the surveys the program communicates the results and the actions
taken to the stakeholders;
Good: in more than half of the surveys the program communicates the results and the actions taken
to the stakeholders;
Excellent: the program always communicates the results and the actions taken to the stakeholders.

TIPS AND GOOD PRACTICE!

It is important (but challenging) to provide a definition of the indicator to ensure common understanding
and to find the right level of specification (not too specific but specific enough). An external expert (in
music and benchmarking) could be very useful in assisting in this process.
The starting point for a benchmarking project following the approach of Case Study 2 (the KONDOR
project) should always be the description of good practice in order to compare this with good practice
at other institutions or across departments within the institution. Although these good practice models
often rely on an ideal situation, they must also be realistic.

26
 HOW TO SET UP A BENCHMARKING PROJECT?

4.5 COLLECTING DATA, CLEANSING AND REPORTING

The essence of a good benchmarking process is institutional learning; learning about how what the university does
relates to the wider environment and stakeholder set. Data gathering is a very small element of this activity, which takes
place later on in the benchmarking activity, once the key processes are understood.[6]
In the approach followed by the RNCM International Benchmarking Exercise, a template was designed for the
questionnaire or survey. Data was collected by sending the survey to partners with a completion date and/or through
site visits. Data was gathered in a standardised format and entered into an appropriate database.
Once the data has been collected and entered into the database it was checked for errors by the project coordinator
and any anomalies were referred to the relevant project leader for checking/correction.
Following the initial data collection phase it may be necessary to review the benchmarking questions. It may have been
difficult to provide data for some questions, whereas other questions may have provided data that is not useful. In these
cases the group may agree to remove these questions from the database and subsequent phases of data collection, or
refine them.
It is important in a benchmarking project to give consideration to the reporting of the data collected. The nature of
the report will depend on the purpose of the project. Two types of report can be produced for use internally or to be
shared amongst partners:
1. Detailed results of the data gathered.
2. Summary reports focussing on good practices, comparisons of performance, and if required, target
setting.
They should be written/presented in a format that is clearly understood by all those involved.

CASE STUDY 1: the IBE project

Data was first collected in an excel file: the template was prepared by RNCM with all questions agreed
upon; each partner filled it in and sent its own file back to RNCM; finally, the RNCM coordinator compiled
all the data in a single excel file, which was circulated. In December 2013, an online tool was developed by
RNCM to simplify the collection and sharing of data between the partners which is available to partners
on a secure website. It aimed at reducing the administrative overhead associated with keeping the data
up-to-date, and also at facilitating easy access to the data for authorised users.
Data Reporting will take 2 forms with this tool:
1. A set of standard reports displaying the data in suitable formats including tables, bar charts and
pie charts.
2. Data downloadable in Excel spreadsheet format for offline analysis.
In the approach followed by the KONDOR project, data is collected as evidence of the performance level
met by the institution for each indicator.

CASE STUDY 2: the KONDOR PROJECT

Once the levels of performance for each indicator were drafted, participants were asked to assess their

[6]
Idem.

27
 institutional practices against the various indicators and to score these practices using scoring cards (with
a score of 1-basic to 4-excellent for each indicator). Participants were asked to explain the reasons for the
score selected, which involved collecting data on the institutional practice and presenting it.
Finally, participants had to indicate which level of performance they wanted to reach in the future for each
indicator and how, and to make a list of strengths and points of improvement.

4.6 RESOURCING AND MANAGING A BENCHMARKING PROJECT

The following resources are needed to successfully carry out a benchmarking project:
Staff time - an investment of staff time is needed to establish the benchmarking group, develop the
tool(s) to capture information, organise meetings of the group, analyse and report the data. Partners
will need to devote time to the project.
Specific skills and competences to generate useful results, it is important to develop the right
questions, collect the right data and interpret it properly. Specific skills and competences, for example
in relation to dealing with the statistical material, will need to be developed in the partner institutions;
Administration costs each partner may need to cover the costs of site visits, meetings of the group
and general administrative costs depending on the nature of the project.
Implementation costs if changes are to be made to institutional practices as a result of a benchmarking
project this may incur costs.
The overall investment will depend on the size and scale of the project, especially if the project will continue over a
number of years to monitor trends and changes over time. A budget should be agreed at the outset with partners
agreeing to any costs they are required to meet.
For collaborative benchmarking projects one institution should be appointed as coordinator. Each partner institution
should nominate a project manager to coordinate the necessary input from various people within the institution and to
liaise with senior staff in the institution.

CASE STUDY 1: the IBE project

A contact person has been designated in each institution. This person is the link person and responsible
in their institution for ensuring the annual return of data.
Persons responsible for gathering the data were:
Head of Administration and Public relations, International Office, Students representatives, the Secretary
of the Board of Governors and the Vice-Principal (partner from the Netherlands);
A PhD student hired for this purpose (partner in Australia);
The Administrative staff in charge of accreditation procedures (partner in the US);
The Head of Administration in consultation with various colleagues (partner in Switzerland);
The Head of Studies in consultation with various colleagues (partner in Norway).

28
 HOW TO SET UP A BENCHMARKING PROJECT?

CASE STUDY 2: the KONDOR project

The work was conducted by the quality manager and the programme coordinator of each programme
involved in the project. In addition, the benchmarking project was supported by a working group consisting
of five members (including a project leader and a project researcher), which prepared and coordinated
the exercise.

4.7 PRODUCING AN ACTION PLAN AND IMPLEMENTING IT

The goal of the partners undertaking a benchmarking exercise should primarily be to learn from this exercise (for example,
how partner institutions function and deal with certain issues). The specific results of the project are never known at
the beginning of the exercise and all benchmarking partners will need to ensure that they obtain useful information from
each other to be able to identify areas for improvement and change in their home institution/programme.
The success of a benchmarking exercise ultimately comes down to the capability of managers to use that information
to better understand their institutional situation and produce an agenda for strategic institutional change. Benchmarking
can help managers to learn about their institution, what is possible and what is being done elsewhere, but it is the
responsibility of the managers to join up those various lessons into an action plan that makes sense in the wider
institutional context.[7]

[7]
Idem.

29
 5. CHALLENGES
To be effective, institutions have to be willing to hear criticism and to persevere with a benchmarking exercise that is
demanding and challenging internally, and bringing potentially unwelcome messages to those that commissioned it.[1]
All seven steps to set up a benchmarking project (listed in chapter 4) will be challenging in different ways depending on
the project itself. The following difficulties were encountered by the institutions involved in Case Study 1 and/or Case
Study 2 presented in this guide:

In Step 1 - Gaining commitment for your project at a senior level within your institution:
Changes in the senior management structure of institutions during the timeframe of the benchmarking
project.

In Step 2 - Selecting Benchmarking Partners and Forming a Benchmarking Group:

Changes of partner institutions.

In Step 3 - Defining the benchmarking project precisely:

All partners have to understand that benchmarking is not a goal in itself but only an instrument, and
carefully consider the pros and cons before they participate in the benchmarking project;
The need to consider the best approach to benchmarking, given the theme selected.

In Step 4 - Formulating and Agreeing Benchmarking Questions and/or indicators:

The definitions of terms: partners need to understand what is discussed (even terms such as teaching
staff, student support, etc.) and to come to a common understanding of the meaning of the questions/
indicators.

In Step 5 - Collecting data, cleansing and reporting/ Conducting a self-evaluation process:

The partners need to determine the main contact in their institution for completion of the questionnaire.
The approach followed in Case Study 1 required input from many different departments within the
institution, which created challenges relating to the appointment of contact persons, the collection of
all the data requested and in a timely manner;
It is very important to contextualise the data collected, for example to avoid referring to the student/
teacher ratio alone without relating that figure to the funding/finance situation.

In Step 6 - Resourcing and Managing a Benchmarking Project

The need to use the institutions existing resources in Case Study 1, as there was limited or no financial
resources to undertake the benchmarking project;
The need to develop specific skills and competences in the institution in order to develop the right
questions, collect the right data, interpret it properly, and therefore generate useful results;

[1]
Idem.

30
 CHALLENGES

The lack of dedicated time for project / lack of dedicated staff, as the amount of work needed is
significant.

In Step 7 - Producing an Action plan and implementing it:

The need to contextualise the lessons learned, as it is usually irrelevant and challenging to simply
implement good practices from others into ones own organisation;
The need to consider all the factors that affect a process. Several factors may interact with/depend on
each other and if only one factor is highlighted, there is a risk of losing sight of the actual possibilities
for improvement;
The translation of the data into a meaningful quality enhancement strategy (from a practical point of
view): how to really use the data to build an action plan, who will be involved inside the institution to
draft and implement this plan?

Remember that every step in the process is important and deserves equal attention, although there might be pressure
to skip certain steps.
Despite the challenges listed, the institutions which have been involved in the benchmarking projects studied by the
Polifonia Working Group shared their satisfaction with these projects and strongly recommended this method to
colleagues.
If you feel that benchmarking is the right tool for your institution, or would like to receive advice from experienced
colleagues in this field, please contact the AEC Office. A counselling visit to your institution can be arranged.

31
6. SOURCES FOR FURTHER INFORMATION

Hooge, N. & Sels, H. Benchmarking, an appropriate tool for decision-making and improving or just
another hype? In Papers from the 7th European Quality Assurance Forum (EUA)
Weblink: www.eua.be/Libraries/EQAF_2012/Sels_Hooge.sflb.ashx

ESMU. (2008). Benchmarking in European Higher Education. A Practical Guide. Brussels: ESMU.
Weblink: http://lllp.iugaza.edu.ps/Files_Uploads/634956737013680415.pdf

ESMU. (2010). Benchmarking in European Higher Education: A University Benchmarking Handbook. (P.
Benneworth, Red.) Brussels: ESMU.
Weblink: www.che-consult.de/downloads/Handbook_Benchmarking_EBI_II.pdf

32
SOURCES FOR FURTHER INFORMATION

33
The ERASMUS Network for Music POLIFONIA promotes innovation in European Higher Music
Education (HME) and aims to assist institutions in enhancing the quality and relevance of HME
through cooperation at the European level.

This publication provides step-by-step guidelines with examples of case studies for those who are
new to the topic, with the aim of introducing them to the method of benchmarking, and assisting
those who are planning a benchmarking project within a music education setting.