FRAMEWORK QUALITY PLAN

REFERENCE TEMPLATE TO BE FINALISED IN AGREEMENT WITH AWARDED BT3

CONTRACTOR UNDER SC01

B-TRAIN 3 TAXUD/2015/AO-01
FOR THE PROVISION OF SERVICES COVERING MULTIFACETED EU
TRAINING SUPPORT PROGRAMMES, ELEARNING DEVELOPMENT AND
COMMUNICATION SERVICES IN THE FIELD OF CUSTOMS AND
TAXATION

Commission européenne/Europese Commissie, 1049 Bruxelles/Brussel, BELGIQUE/BELGIË - Tel. +32 22991111

Table of Contents
1 INTRODUCTION ..................................................................................................... 4
1.1 Objective of this Document ............................................................................... 4
1.2 Structure of this Document ................................................................................ 4
2 TERMINOLOGY ....................................................................................................... 6
2.1 Abbreviations and Acronyms ............................................................................ 6
3 PROJECT MANAGEMENT ..................................................................................... 8
3.1 Global Project Planning ..................................................................................... 8
3.2 Responsibilities in the eLearning Courses Project ............................................ 8
3.2.1 Roles and Responsibilities of the B-Train 3 Contractor ...................... 8
3.3 Organisation of the Project Team .................................................................... 10
3.3.1 DG TAXUD Project Team ................................................................ 10
3.3.2 Contractor Project Team.................................................................... 11
4 DOCUMENTATION ............................................................................................... 12
4.1 Contract Management Deliverables ................................................................ 12
4.1.1 Monthly Progress Report ................................................................... 12
4.1.2 Weekly Project Progress Report (Informal) ...................................... 13
5 DEVELOPMENT, REVIEW AND ACCEPTANCE CYCLE ............................. 15
5.1 Monthly Progress Report ................................................................................. 15
5.2 Documentary Review ...................................................................................... 15
5.2.1 Contractor Internal Review ............................................................... 15
5.2.2 Delivery of Draft Versions ................................................................ 15
5.2.3 Contractor Internal QA Process......................................................... 15
5.2.4 DG TAXUD Review and Acceptance Process ................................. 16
5.2.5 T1/T2/T3 Review Cycle .................................................................... 21
5.2.6 T1/T2/T3 Review Timeline ............................................................... 21
5.3 Acceptance Testing ......................................................................................... 22
5.3.1 Factory Acceptance Test ................................................................... 23
5.3.2 Test Event .......................................................................................... 25
6 QUALITY INDICATOR......................................................................................... 27
6.1 The Principles .................................................................................................. 27
6.2 Calculation of the SQI ..................................................................................... 28
6.2.1 The General Quality Indicator ........................................................... 29
6.2.2 Liquidated damages ........................................................................... 30
6.2.3 General Indications on the Use of SQI & GQI.................................. 31
6.3 The Service Quality Indicators (SQIs) ............................................................ 32
6.3.1 Definition of the SQI ......................................................................... 32
6.3.2 SQIs’ weight in the GQI .................................................................... 36
7 AUDITS AND MANAGERIAL REVIEWS......................................................... 37
7.1 Managerial reviews ......................................................................................... 37
2
 7.2 Project Quality Audit ....................................................................................... 37
7.2.1 Audit Objective ................................................................................. 37
7.2.2 Confidentiality ................................................................................... 38
7.2.3 Audit Preparation............................................................................... 38
7.2.4 Audit Execution and Reporting ......................................................... 38
7.2.5 Project Audit Follow-Up ................................................................... 38
8 ESCALATION ......................................................................................................... 39
9 TOOLS, TECHNIQUES, AND METHODOLOGIES ...................................... 39
9.1 Comments Excel Sheet .................................................................................... 39
9.2 MPR Annexes .................................................................................................. 40
10 MEDIA CONTROL ................................................................................................. 42
11 RECORDS COLLECTION, MAINTENANCE, AND RETENTION .............. 42
12 RISK MANAGEMENT ........................................................................................... 42
13 ANNEXES ................................................................................................................ 44
13.1 Annexe I: Example of Comments Sheet ......................................................... 44
13.2 Annex II: Example of Qualitative evaluation Questionnaire .......................... 45
13.3 Annex II: Example of Weekly Progress Report .............................................. 52
Milestone Plan: ................................................................................................ 52
Items from Weekly Progress Meeting (DDMMYYY):................................... 53
Open items from previous Weekly Progress Meeting:.................................... 53

3
B-Train 3 - Framework Quality Plan

1 INTRODUCTION

1.1 OBJECTIVE OF THIS DOCUMENT

This document is the Framework Quality Plan (FQP) for the European Commission's
Taxation and Customs Union Directorate, DG TAXUD, which will cover the execution
of Framework Contract B-Train 3. The FQP also defines the quality expectations for
the scope and duration of the contractual relationship under this Framework Contract
(FWC).
This FQP defines the conditions applying to the working relationships between the
contractor and DG TAXUD in the context and the duration of the FWC. It defines the
standards, rules and procedures applicable to the execution of this FWC. In
particular, it defines the procedures that will govern the delivery of the services
along with their monitoring mechanisms and follow up actions, as well as the rules
for measurement of the quality of services rendered and products delivered by the
contractor.
The purpose of the FQP is to ensure that contractual requirements are met and
TAXUD satisfaction is obtained.
For each Specific Contract drawn under this FWC the contractor will prepare a
Specific Quality Plan (SQP) to demonstrate that he/she:
 Understands the project, its characteristics, peculiarities, risks and
innovations;
 Has selected the proper project organisations, techniques, resources and
tools to cope with the above;
 Has defined the deliverables, Quality Control, approval, responsibilities and
processes involved;
 Has identified the major risks and the corresponding mitigation actions.
More specifically, the SQP defines the following items:
 Roles and responsibilities;
 Content, format, sign-off, review process and responsibilities for the project
deliverables;
 Management of changes and problems;
 Working procedures, i.e. handling of incidents, change requests and requests
for estimate;
 Measurement and monitoring of service quality;
 Content and structure for reporting to DG TAXUD;
 Interaction processes between the different entities involved in DG TAXUD
projects.
It is the responsibility of the Contractor’s Portfolio/Project Manager to ensure that
the key principles and procedures defined in the SQP are communicated and well
understood by all Project Managers and team members.

1.2 STRUCTURE OF THIS DOCUMENT

This document is structured into the following chapters:

Chapter 1 ‘Introduction’ includes the standard sections for the related
documents and the document conventions.
Chapter 2 ‘Terminology’ gives the terminology required for a good
understanding of the document.
Chapter 3 ‘Management’ presents the overall organisation of the project and
the responsibilities of the actors.
Chapter 4 ‘Documentation’ presents the contractual and management
documents covering this FWC.
Chapter 5 ‘Development, Review and Acceptance Cycle’ proposes the
overall review cycle of the document and software deliverables.
Chapter 6 ‘Standards, Practices, Conventions, and Metrics’ presents general
standards to cope with for the documentation, and the metrics that
will be used to measure the performances.
Chapter 7 ‘Audits and Managerial Review’ provides elements of managerial
review and audit processes.
Chapter 8 ‘Escalation’ presents the escalation processes.
Chapter 9 ‘Tools, Techniques, and Methodologies’ provides information
about techniques and tools used in support of the FWC's
implementation.
Chapter 10 ‘Media Control’ gives information on media used in support of
delivery processes.
Chapter 11 ‘Records Collection, Maintenance, and Retention’ presents the
operational processes in support of deliverables management process.
Chapter 12 ‘Risk Management’ presents an overview of the risk management
process.
Chapter 13 ‘Annexes’ refers to the annexes of the current document.

5
2 TERMINOLOGY

2.1 ABBREVIATIONS AND ACRONYMS

Abbreviation Meaning

AP Activity Package
BT3 B-TRAIN 3 Framework Contract
Coms Communications
DD Design Description
DLV Deliverable
DTM Deliverables Tracking Matrix
FAT Factory Acceptance Test
FWC Framework Contract
GQI Global Quality Indicator
ITT Invitation to Tender
Mgt Management
MPR Monthly Progress Report
PC Participating Countries (includes Member states and Candidate
countries)
QA Quality Assurance
FQP Framework Quality Plan
SQP Specific Quality Plan
QP Quality Plan
QP/A Quality Plan / Addendum
RD Requirements Description
RfA Request for Action
RfE Request for Estimates
SC Specific Contract
SfA Submitted for Acceptance
SfR Submitted for Review
SP Service Proposal
GQI Global Quality Inidicator (or Overall Quality Indicator)
SQI Service Quality Indicator
TA Technical Annex
TE Test Event
ToR Terms of Reference
WP Work Package
WPM Project Weekly Progress Meeting

6
WPR Weekly Progress Report

7
3 PROJECT MANAGEMENT

3.1 GLOBAL PROJECT PLANNING

The overall project planning for each SC will be annexed to each specific Quality
Plan, with the major milestones of the deliverables and the dates of the review
cycles, defined according to T1/T2/T3 method as defined in section 5.1 of this
document. Dates of deliverables and review cycle have to be reflected in the DTMs.
The planning and the DTMs will be maintained all along a SC project life and will be
delivered on a monthly basis as annexes of a SC dedicated Monthly Progress Report.

3.2 RESPONSIBILITIES IN THE ELEARNING COURSES PROJECT

3.2.1 ROLES AND RESPONSIBILITIES OF THE B-TRAIN 3 CONTRACTOR

This section is provided as an example and should be adapted in line with selected
contractor's offer. The titles and the different responsibilities hereunder are neither
exhaustive nor complete.
This should not be understood as being TAXUD's preferred organisational solution!

3.2.1.1 B-Train 3 Portfolio Manager

The B-TRAIN 3 Portfolio Manager is the person responsible for the successful
completion of the contracted activities, reporting to DG TAXUD. During the projects,
he/she will be the person who manages contractor staff working under this SC.
This person’s responsibilities include, but are not limited to the following items:
 Collaborate closely with DG TAXUD appointed Project Management team for
all questions regarding the B-TRAIN 3 projects;
 Perform the Portfolio Management activities under WP.1 and take ownership
of the MPR;
 Draw up the Quality Plans (QP) for the projects, in compliance with this
Framework Quality Plan and the SQP;
 Plan, control and report on the production of deliverables;
 Ensure that all work is performed to the agreed standards and quality;
 Manage the budget, work plan and all contractual management procedures
(contract management, scope management, issues management, risk
management, etc);
 Ensure the coordination with all involved actors;
 Provide Leadership on the projects, ensuring a long-term vision.

3.2.1.2 B-Train 3 Project Manager

The Project Manager is responsible for the services delivery of the individual project
APs.
The main responsibilities of the Project Manager cover:
 The detailed planning of the project activities;
 The assignment of his team’s resources on specific tasks;
 The monitoring of the progress on each activities;

8
  The organisation of detailed planning meeting (e.g.: every 2 weeks) with its
team to identify possible slippage in delivery dates;
 The coaching of his team;
 The preparation of the required information to support the B-TRAIN 3
Portfolio Manager in the delivery of the Monthly Progress Report (e.g.:
progress on milestones and budget consumption);
 The escalation to the B-TRAIN 3 Portfolio Manager of issues, risks and
potential delays. All delays will have to be reported on a timely manner to the
B-TRAIN 3 Portfolio Manager and discussed with TAXUD Project team;
 The ownership of weekly Project Progress meetings. The frequency of these
meetings will vary in fu whenever these are requested for close follow up
purpose. The normal frequency would be weekly,
The Project Manager manages day-to-day operations through the support of several
key individual roles:
 Developers;
 Designers.
The Project Manager is also responsible for organizing the important Take Over and
Hand Over work packages.

3.2.1.3 B-Train 3 Technical Architect

The Technical Architect is responsible for all content related Technical activities.
His main responsibilities cover:
 Setting up the work environment for the Design and Development team:
configuration of the PCs and the Collaborative Content Development Server
and the basic LCS features required in the BT3 project;
 Supporting and fixing issues related to content dissemination in the various
countries (due to the diversity of LMS in the countries). Especially all the
AICC and SCORM LMS compliance potential issues;
 Interfacing with the development teams whenever there is a requirement for
new components.

3.2.1.4 B-Train 3 Lead Pedagogue

The Lead Pedagogue is responsible for the global pedagogic vision and consistency of
the whole B-TRAIN 3 project. He/she has a key role in setting up the Standards
Specifications, and then in all the content projects.
His/her main responsibilities cover:
 The development of all the pedagogy-related sections of the Standard
Specifications;
 Supporting the project manager for important demos or workshops, where
pedagogical expertise will be required to make decisions;
 Assessing the pedagogical elements in the Feasibility Studies;
 Defining the high-level pedagogical outline of all the content development
projects, using the content development environment;
 Supervise and manage the other pedagogues on the design team;
 Coordination with the content development team.
Having a role of consolidating all the key learning elements provided during a
specific project, the lead pedagogue has to prepare intensively on the subject, based
on all available material given prior to the start of the project. (Technical Annexes,
available legislation, etc) During Project Group Meetings the Lead Pedagogue has to

9
come up with already digested information and present implementation ideas and
suggestions for a group validation and/or further elaboration.

3.2.1.5 B-Train 3 Quality Assurance Manager

The quality assurance is made of all actions taken to ensure that standards and
procedures are adhered to and that delivered products or services meet performance
requirements.
The quality control is made of all the operational techniques and activities that are
used to fulfil the requirements for quality.
The Quality Assurance Manager is responsible for all Quality Management activities
under WP.0.3 (Internal QA and QC and Risk Management). He/she first confirms the
quality objectives with DG TAXUD and supports the Project Manager in the
implementation of the Framework Quality Plan by defining the quality procedures
and the checklists to be applied.
The main responsibilities cover:
 The Quality Reviews of the deliverables;
 The planning and organisation of the Quality Inspections;
 The organisation of the Internal Quality Audits;
 The support to DG TAXUD during the external Quality Audits;
 The organisation of the Quality Meetings;
 The follow-up of the non-compliances and of the associated corrective
actions;
 The follow-up of the Quality Improvement action plan.
The Quality Assurance Manager supervises the Quality Controllers for the different
specific projects.

3.2.1.6 B-Train 3 Quality Controller

The Quality Controller (QC) works under the direction of the Quality Assurance
Manager and is the key role for day-to-day quality management.
In B-TRAIN 3 the QC has to cope with different complications regarding the quality
process:
 There are several distinct types of common training modules that are being
created under several pedagogical models;
 There is a complex Quality Management lifecycle that will have to be
observed in parallel with the consortium’s own quality requirements.
The Quality Controller has an active role in setting up the QP for the different sub-
projects.
The Quality Controller is the liaison factor as long as quality is concerned between
the management of the project and the project teams.
The Quality Controller plays an active role to support the Commission in conducting
quality and security audits.

3.3 ORGANISATION OF THE PROJECT TEAM

3.3.1 DG TAXUD PROJECT TEAM

For each SC the QP will contain a list of responsible persons from the DG TAXUD
team for a specific AP.

10
3.3.2 CONTRACTOR PROJECT TEAM

In the QP for each SP the contractor will provide DG TAXUD with a list of persons
assigned to a specific project together with their roles and the AP to which they are
assigned in the framework of the specific project.

11
4 DOCUMENTATION

4.1 CONTRACT MANAGEMENT DELIVERABLES

4.1.1 MONTHLY PROGRESS REPORT

The B-TRAIN 3 contractor will report on a monthly basis through a specific

deliverable: the Monthly Progress Report (MPR).
The MPR contains a description of achievements, progress on activities and
deliverables, pending problems, short-term actions, resource usage and activities
issues and risks, status on the RfA and SQI analysis.
The proposed structure of the MPR is presented here below. This structure can be
amended upon request from DG TAXUD at any time during the SC with no change on
the QP:
 Introduction;
 Tasks planned during the month, including actual progress on those tasks:
For each task mentioned, a short description of the contribution to the
progress is given:
o Description of the activities carried out;
o Description of the results achieved;
o Comments on ongoing tasks when appropriate;
o Justification of the schedule deviations.
 Monthly Service Report, including the services regarding the work packages
in the contract:
 Status of the deliverables up to the concerned reporting period (reference to
the DTM annex attached to the MPR);
 List of all deliverables of the reporting period:
o Due, but delayed
o SfR
o SfA
 Identification of the deliverables allocated with an acceptance mechanism of
the type “for acceptance with MPR” as per specific contract:
o Submitted for acceptance during the reported month;
o Delayed from this month and from previous ones with due
justifications.
o Mentioning the initially planned submission date for acceptance.
 Status of the Service Quality Indicators (SQI) and Global Quality Indicator
(GQI) up to the concerned reporting period (reference to the SQI annex
attached to the MPR);
 Status of the issues met during the execution of the tasks (Reference to the
issue log annex attached to the MPR);
 Risk Management Status (Reference to the Risk Management log annex
attached to the MPR).
 Tasks planned for next month: An extract of the project Time Plan, listing all
the tasks that are planned to start or to be continued during next month. This
list is the basis for point (1) of the MPR made at the end of the next reporting
period (‘Tasks planned during the concerned month’);

12
  Status of the Request for Estimate (RfE) and Requests for Action (RfA)
(Reference to the annex attached to the MPR);
 Log of Decisions taken in the course of any B-TRAIN 3 project, that will be
valid for the whole duration of the Framework Contract
 List of Change Requests as identified as deviation from the original Offer and
RfA assumptions. Those Changes should be listed with an explanation, an
impact analysis, a cost estimate and a status.

The MPR is the supportive document of the monthly progress meeting, which will be
held after the end of the reporting period. The minutes of the monthly progress
meeting include a section for review of the MPR, and therefore need to be reviewed
prior to amending and submitting the MPR SfA version.
The submission and review cycle of this management document is the following:
 After the end of the reporting period, the MPR is submitted to DG TAXUD for
review;
 With the MPR as supportive document, the Bilateral monthly meeting (BMM)
is organised. This meeting is also the place where the MPR is discussed and
the comments collected for amendment of the MPR;
 The minutes of the BMM are submitted to DG TAXUD for review;
 DG TAXUD provides the comments on the minutes of the BMM;
 The minutes of the BMM and the MPR are then amended with those
comments then submitted to DG TAXUD as a bundle for acceptance.
The timeline is summarised here below:
 T0: End of the reporting period;
 T0 + 5 working days: SfR version of the corresponding MPR;
 T1 = T0 + 5 working days (first suitable date): Bilateral Monthly Meeting
(BMM);
 T1 + 3 working days: SfR version of the minutes of the Bilateral Monthly
Meeting;
 T1 + 5 working days: Comments on the SfR version of the minutes of the
BMM;
 T1 + 8 working days: SfA version of MPR and Minutes of BMM submitted as a
bundle.
The MPR is accompanied by a list of annexes, as defined in last section of the MPR:
 Annex 1: Project Plan;
 Annex 2: Delivery Tracking Matrix, Status of the SQI and GQI.
 Annex 3: On Demand/Additional Fixed Price Budget Follow-up; RfE/RfA
Status, including RfA for technical meetings, Training, workshops and
missions;
 Annex 4: Actions Log; Issue Log; Risk Management Log; Decision Log;
Change Requests details.
The annexes are further described in section 9.2.

4.1.2 WEEKLY PROJECT PROGRESS REPORT (INFORMAL)

Each week, B-Train 3 Project Managers are reporting on the status of their project.
This weekly Project Progress Report (PPR) summarizes all activities performed within
past week and details the following projects outcomes:

 Project Summary

13
 Project Status

 Project Adapted Milestone Plan

 Project Actions/Issues/risks/Decisions

The PPR is initially submitted prior to the Weekly Progress Meeting, then updated
with any information that is needed for follow up after the meeting. Typically, PPR
are initially distributed to the Portfolio Manager and DG TAXUD at the end of the
week. This rule can be adapted per project.
During the whole course of the development process, the Contractors plan a weekly
checkpoint with DG TAXUD. The PPR is reviewed and project items are discussed
(actions, planning, deliveries, milestones, issues, changes, risks). These meetings
can either be held as phone conferences, webcasts or face to face meetings at DG
TAXUD’s premises.
After the weekly PPM, the contractor sends an updated version of the PPR, whenever
appropriate.
An example of PPR is given in Point 13.3

14
5 DEVELOPMENT, REVIEW AND ACCEPTANCE CYCLE

5.1 MONTHLY PROGRESS REPORT

While the MPR itself follows the Documentary Review cycle for acceptance, the
acceptance of the MPR implies the acceptance of the services which are integrated in
the MPR (see WP 1.3).

5.2 DOCUMENTARY REVIEW

5.2.1 CONTRACTOR INTERNAL REVIEW

Any document will be assigned an author who will keep the ownership of the
document all along the life cycle of this document. In case of shared document, the
author of the document remains the accountable person for this document.
During the development of the deliverable, peer-to-peer reviews will be organised
internally with the purpose to ensure a consistency of the document in itself (self-
consistency) and a conformance with the documents which the document being
developed depends on.

5.2.2 DELIVERY OF DRAFT VERSIONS

The Review and Acceptance of documents by DG TAXUD is based on the SfR and SfA
mechanisms.
However, in order to ensure that the deliverables match DG TAXUD expectations,
additional drafts could be produced for review before the SfR delivery, at the
discretion of the contractor but in coordination with DG TAXUD’s Project Coordinator.
The objective is to avoid extensive rework of the SfR version. Therefore, it is
essential that the decisions on the content and structure of the documents are taken
on the draft documents and not questioned during the formal SfR/SfA review cycle.
The draft versions will be working documents. There will be no formal quality review
by the contractor and no Comments database or Author’s Positions for those
documents. The review of those documents will take the form of embedded revision
marks and comments.
DG TAXUD should deliver all feedback on any draft document within 4 working days
from delivery via one single consolidated e-mail.
When relevant, the informal working/validation meetings will be organised within 2
days after reception of the comments on the draft document. Those meetings will be
considered as technical meetings.

5.2.3 CONTRACTOR INTERNAL QA PROCESS

Once a document is completed from a content perspective, a Quality Analysis is

performed on the document in order to make sure that the document is compliant
with all the rules, procedures, presentation and other standards, correct from a
grammar and spelling perspective, consistent from a numbering perspective, etc.
The quality control concentrates on those aspects, but includes no content aspect in
its scope. The quality reviewer provides its feedback as part of a distinct version to
the author of the document. In order to avoid a loss of control of the deliverable, the
document author reflects the comments of the quality reviewer in the SfR version of
the document. Once the document is amended, it is ready to be submitted for
review.

15
5.2.4 DG TAXUD REVIEW AND ACCEPTANCE PROCESS

The objective of the Quality Assurance process is to ensure that the contractor
follows a thorough analysis and elaboration phase in order to produce a deliverable
which is of very good quality and within the scope defined by DG TAXUD. This
objective should be attained before the deliverable is submitted for review. During
this analysis and elaboration phase the contractor has to make sure that the content
of the deliverable is in line with DG TAXUD's expectation by any appropriate means.
The Quality Control process includes the review of the deliverable by DG TAXUD and
possibly other contractors with the aim to detect and eliminate any possible quality
flaws or scope deviations. The review aims at assuring the quality of the document.
The review and acceptance process of project documents by DG TAXUD is based on
the SfR / SfA mechanisms as illustrated in section 8.5 of the Service Proposal [A5].
Table 1 shows the activities that will be carried on during every step of the review
and acceptance cycle.

T0 Submission for Review

T1 Distribution to DG TAXUD and/or other
contractor reviewers
Individual Review
Collection and consolidation of DG
TAXUD and/or other contractors
comments
Delivery of the comments database to
the author
T2 Preparation and Communication of
Author’s Position
Analysis of the Author’s Position
Review Meeting
Communication of Review Meeting
Decisions
Implementation of Review Meeting
Decisions
Submission for Acceptance
T3 Verification of amendments
Decision to accept the amendments
Table 1: Review Cycle Dedicated Activities
The following sections describe the successive steps of the review cycle.

5.2.4.1 Document Submitted for Review (SfR)

When the document is finalised, reviewed by the Quality Reviewer and authorised by
the Project Manager, it is submitted for review to DG TAXUD and potentially to other
contractors.

5.2.4.2 DG TAXUD Review

DG TAXUD and/or other experts review the document and communicate their
comments.

16
Documents produced in the frame of the project may be reviewed by several people
from different contractors or national administrations. This creates the need for a
synchronisation between all the remarks and recommendations of the different
review teams.
As a general rule, the “Deliverable Comments Excel Sheet” will be used to support
the DG TAXUD review process of the SfR version. This sheet/database must be
consolidated with all comments from DG TAXUD and/or other contractors prior to
being sent to B-Train 3 development team.
However, applicable to both major and common deliverables as well as to PM related
deliverables, DG TAXUD may decide to use the Ms-Word/Ms-Excel comment/revision
marks facility instead of the Comments sheet. This facility can only be used by DG
TAXUD to communicate their review comments. Document SfR and SfA issued by
BT3 may include neither comments nor revision marks.
Whatever the facility used in support of the DG TAXUD review process, the
comments must be consolidated in a single file (Ms-Word Document or Ms-Excel
table). If not, the date of reception of the last comment is used for the calculation of
the following deadlines.
No matter what the format of the file is, the comment document should ALWAYS
contain the minimum following information:

 Item (issue/question/decision) Number

 Reference to Chapter/LU/Section/slide/Page/Paragraph

 Short Description of the issue

 Comments:

 Long and un-ambiguous explanation about the item

 Proposed solution

 Reviewer's name

 Review Date

 Comment type:

 Defect: Error in delivered product, in contradiction with previously defined elements.

Assumptions and decisions are not properly reflected in the delivery

 Change: Assumptions and/or decisions have changed and will require an adaptation of
the delivered item

 Question: Request for explanation or clarification

 Comments: Advice; information that may help the author with further development of
the deliverable

 Severity of the issue:

 Blocking: The identified issue causes the delivered product to malfunction and makes
it impossible to go further with providing comments on parts of this delivery. E.g.:
Technical limitation of a delivered training application that does not allow to proceed
further within the quiz part of the module

17
 Will block: The identified issue has only consequences limited to itself. The problem
has no effect on the rest of the delivered product but still HAS TO be resolved. E.g.: A
Voice-over is de-synchronized with written text.

 Non blocking: The reported comment consists more in a "nice to have" or does not
have to be implemented at all (question/comment/enhancement suggestion…). E.g.:
discussion on the background colour of a text box

The whole review process has to be formulated in English. However, during

localization processes, abstracts of foreign texts may be copied to illustrate the
problem and its solution.
The review step includes the activities as described in following sub-sections.
An example of review excel sheet is given in annex I.
5.2.4.2.1 Distribution of the deliverable for review
As soon as DG TAXUD receives the deliverable, it sends out copies to the identified
reviewers, together with review instructions if needed. This step can also happen at
once (upon approval of DG TAXUD), together with the SfR submission via an upload
on the collaborative platform.
5.2.4.2.2 Individual Review Activity
Comments should be clear to ensure their understanding by the author and the
review meeting participants.
Comments should be formulated as assertions for which the author will have to
make a decision but may be formulated as questions to which the author will have to
give an answer.
The comments will be as specific as possible. If the reviewer is able to propose a
solution, he/she should add it to the comment.
Each record should contain only one comment. In that way, it is easier for the author
to answer to the comment. But, if several comments relate to the same paragraph of
the document, it is better to group these comments in one record for better
understanding. In this case each sub-comment will be numbered.
5.2.4.2.3 Collection and Consolidation
When the technical reviews are completed, the reviewers return their individual
comments to the central point of DG TAXUD.
The consolidation of the comments aims at eliminating duplicated comments and
consolidating all the individual comments into a unique database which is created to
reflect all the comments that will be discussed during the review meeting.
5.2.4.2.4 Delivery of the review comments
After consolidation, DG TAXUD forwards the consolidated Review Report database to
the author of the deliverable under review and to all other reviewers.
A deliverable may be rejected at any point during the review step for any of the
following reasons:
 It contains many elements that are out of scope
 It is of poor quality (including spelling or grammatical errors)

 Its structure is non-conformable to DG TAXUD expectations, and different from the

structure agreed during the corresponding meeting;

 A great number of relevant comments are being produced, showing at evidence that
input from DG TAXUD was not adequately collected although this input was
available. Comments resulting in “no action” could not be invoked for rejection of the

18
 document. That is why the review process must be completed before DG TAXUD
decision for rejection.

When a deliverable is rejected during the review step, DG TAXUD must notify the
issue to the project manager of the contractor, in order to agree on the corrective
actions to perform. The time between the delivery of the SfR version and the
communication of rejection of the deliverable is taken out of the SQI computation.

5.2.4.3 Author’s Position on comments

When receiving the consolidated review report database, the author addresses the
comment by providing, in this database his position for each one. The author
indicates his overall position (whether he agrees or disagrees with the comment) or
request for clarification, if needed. The author proposes a solution or correction to
the text, when relevant. The Author’s position is included in the comments database.
When all the comments are addressed, the author sends the review comments
sheet, completed with his/her position, to DG TAXUD who dispatches it to all
concerned reviewers.

5.2.4.4 Analysis of the Author’s Position and Review Meeting

After reception of the comments database, DG TAXUD walks through and analyses
the author’s positions provided.
A review meeting is organised by DG TAXUD with the reviewers and the author of
the document, with the objective to come to a common agreement on all comments.
A decision is made for each comment; this decision and the solution to be
implemented, are mentioned in the meeting decision field of the database. The
author of the document is responsible for filling in this field of the comment sheet.
The sheet completed with this field is considered as the minutes of the meeting and
is distributed to all reviewers of the document.
The date of the Review Meeting must be planned for and agreed, as early as possible
and be depicted in the contract dedicated DTM.
Any comments raised during the review of the deliverable for which the decision has
been taken to implement, must be consistently implemented throughout the
deliverable with the aim to submit for acceptance a deliverable that is of excellent
quality. That is why meeting decisions should be logged in the Comments Database.

5.2.4.5 Update of the original document

Based on the decisions gotten from the review meeting and logged in the comments
Database, the author amends the original document.
All the agreed changes are implemented in a correct way and consistently
throughout the document. The updated version shall contain the agreed changes
only.
If the author has implemented some changes differently than agreed during the
review meeting, or if specific changes are needed although not decided during the
review meeting, the author must submit those changes to prior agreement from DG
TAXUD. Those changes are documented in the Implementation Information field of
the database.
An internal quality control is applied to the document, with the purpose to check that
all the decisions taken during the review meeting have been reflected adequately in
the documents.
After this process, the document is submitted for acceptance submitted for formal
Acceptance (SfA) to DG TAXUD together with the database.

19
5.2.4.6 Acceptance of the deliverable

Upon reception of the document, DG TAXUD performs a verification of the

implementation of the comments, whether all agreed actions have been properly
carried out, that the amendments have been done in a correct and consistent
manner and that no other changes have been made.
DG TAXUD decides whether the document can be accepted as such or if new
amendments are necessary. New amendments will only be requested based on
remarks concerning the correctness and consistency of the implementation of the
review meeting decision and possible non-agreed changes to the document.
In case of acceptance of the deliverable, DG TAXUD confirms this acceptance by one
of the two following means:
 For the deliverables submitted to individual acceptance, by sending an acceptance
letter containing the version number of the accepted deliverable and signed by the
Head of Unit to the contractor administrative contact.

 For the deliverables submitted to acceptance through DLV-0.5 (MPR), the acceptance
of the MPR implies the acceptance of all the deliverables included in the MPR in
which they have been listed as "Deliverables proposed for acceptance with this MPR".

DG TAXUD has the possibility to reject the document at this stage if:

 At least one of the decisions agreed during the review meeting and reflected in the
comments database has not been implemented in the document;

 At least one of the decisions agreed during the review meeting and reflected in the
comments database has not been implemented in a correct or consistent manner
throughout the document;

 Other changes have been made to the document without prior agreement from DG
TAXUD and without being reported in the implementation notes of a relevant review
comment in the review database.

DG TAXUD reserves the right to decide whether the non-implementation or

inconsistent implementation of review comments or the other changes to the
document are of major (leading to rejection of the deliverable) or minor importance.
When rejected during T3, this new review cycle will include a new SfA date based on
DG TAXUD's judgement.
When a deliverable is rejected during this step, DG TAXUD must notify the issue to
the project manager of the contractor, in order to agree on the corrective actions to
perform. In this case, a new delivery of the SfA version of the deliverable is provided
to DG TAXUD. The time between the delivery of the first SfA version and the
communication of rejection of the deliverable is taken out of the SQI computation.
In any case and for the purposes of SQI calculation, the initially planned SfR and SfA
dates remain the target dates against which SQIs are calculated, besides the time
taken by DG TAXUD to communicate the rejection of the concerned version of the
document.
The following dates are taken into account to define the actual SfR and SfA dates:

 for the calculation of SfR-related SQIs: the date of submission of the last SfR-version
of the deliverable that led eventually to an SfA;

 for the calculation of SfA-related SQIs: the date of submission of the last SfA-version
of the deliverable that was eventually accepted by DG TAXUD.

20
In case of dependency between two deliverables (or two versions of a deliverable),
the SC or the RfE offer must mention the targeted deadline of the second deliverable
(or the second version of a deliverable, i.e. SfA version) in reference to the actual
delivery date of the first deliverable (or the first version of a deliverable, i.e. SfR
version) by a relative number of additional working days.

5.2.5 T1/T2/T3 REVIEW CYCLE

The following review cycle described is proposed for the documents delivered in the
frame of the SC:
T1 T2 T3

T2A T2B T2C

Document
Contractor Writing
AP Preparing Document Update

Acceptance
DG TAXUD Document Review AP Analysis
Process

Author’s Review
Delivery - SfR Comments Delivery - SfA Acceptance
position Meeting

Figure 1: B-Train 3 Document Review Cycle

In this cycle:
 T1 represents the time for DG TAXUD to provide the comments to the author of the
document.

 T2 represents the time needed to the review cycle, from the provision of the Author’s
position up to the document amendment and delivery for acceptance

 T3 is the period during which DG TAXUD will perform is internal acceptance

process.

5.2.6 T1/T2/T3 REVIEW TIMELINE

The timelines (T1/T2/T3) are agreed on between DG TAXUD and the contractor and
are managed through the DTM. The latter will mention the applicable review cycle
for each deliverable and include the derived T0/T1/T2/T3 dates.
Any change of the SfR (T0) or SfA (T2) dates towards an earlier date should be
proposed in an update of the DTM and agreed upon between the contractor and DG
TAXUD in advance. The relevant documents must be referenced to in the DTM.
DG TAXUD reserves the right to keep the initial duration of its T1 and T3 activities
regardless of any delays of the contractor in submitting a deliverable for review or
for acceptance. In the event of a delay by the contractor to submit a deliverable for
review, DG TAXUD reserves the right to arrange a possible review meeting at a
convenient date. In any case the initially planned contractual SfA date remains valid
for the calculation of related SQIs.
For the SQI computation, it is necessary to further split the T2 period into the
following three sub-periods:

 T2A: This period covers the time needed for the contractor to prepare the Author’s
position from the comment received from DG TAXUD;

 T2B: This period covers the time for DG TAXUD to analyse the Author’s position
provided by the contractor, the review meeting and the communication of the related
meeting decisions;

21
 T2C: This period covers the time needed for the contractor to update the document
from the review meeting decisions, to perform an internal QA of the document and to
submit the document for acceptance to DG TAXUD.

In the frame of the SC’s, the following timelines are an example of how the review
cycle can be applied to the documents review cycle (this depends per deliverable):

Timeframe versus SfA according to Review

Cycle
Milestone Actor
10 – 10 – 10 5 – 7 – 10 5–5–5

SfA – 20
Delivery – SfA – 12 SfA – 10
Contractor working
SfR working days working days
days
SfA – 10 SfA – 7 SfA – 5
Comments DG TAXUD
working days working days working days
AP SfA – 6 SfA – 4 SfA – 3
Contractor
Submission working days working days working days
SfA – 4 SfA – 3 SfA – 2
AP Analysis DG TAXUD
working days working days working days
Review SfA – 2
DG TAXUD / SfA – 3 SfA – 2
Meeting and working
Contractor working days working days
Decisions days
Delivery –
Contractor According to Specific Contract or RfA
SfA
SfA + 10 SfA + 10 SfA + 5
Acceptance DG TAXUD
working days working days working days
Table 2: Document Review Cycle Common Timeframes
Other review cycles may be used if defined in a SC contract or Estimates submitted
by the Contractor and accepted by DG TAXUD.

5.3 ACCEPTANCE TESTING

DG TAXUD accepts any major learning software product delivery in the project
following the rules defined in this chapter.
The acceptance test cycle is used to accept the main release of the learning
software. Preceding the Test Event (TE) is the Factory Acceptance Test (FAT).
If the learning module is rejected by DG TAXUD, the contractor will have to modify it
and submit a new version to DG TAXUD who decides the type of acceptance cycle to
follow.

22
 SAT kick-off
Test Plan submitted meeting
for review FAT

SAT
Test Plan End of FAT
Review report
Cycle

TE
End of SAT
FAT meeting
report
Test Plan
review
Accepted
cycle
Decision on Acceptance
by Taxud

FAT QA report

Figure 2: Factory Acceptance Test (FAT)

5.3.1 FACTORY ACCEPTANCE TEST

5.3.1.1 Objectives and Principle

The objective of the FAT is to make sure that the BT-2 developments are ready to
enter the TE with the expectation of passing the acceptance test cases described in
the FAT report.
For eLearning course production, the FAT is considered as integrated
element of the mandatory quality assurance and control to be provided by
the contractor. Only on specific request from DG TAXUD will the TAXUD
project team participate in the FAT and the related TE. The following FAT
process description is therefore to be seen as a reserve for the supply of
other products and services deliverables, requiring a closer involvement of
DG TAXUD in such quality tests.
The acceptance can start when:

 The FAT has been performed successfully. Any issue is recorded and documented in
the FAT Report;

 The results of the FAT are available; the FAT report is sent for review;

 An estimate of the time to fix the issues raised during FAT testing is available in the
FAT report.

DG TAXUD will proceed to the acceptance of the FAT after the planned termination of
the FAT. This is concluded by the acceptance of the FAT report.
The FAT can be made into three steps:
 The contractor elaborates its test scenarios, including:

 The testing strategy;

 The definition of the test scenarios, including the definition of the test cases, with the
corresponding expected results;

 The set-up of the environment for the tests, including the definition of tests data.

 The particular testing of the delivered package, including optical disk completeness,
source files usability, optical disk cover, structure and content…

23
 The contractor performs its tests

 The result of the testing is compiled in a FAT Report;

 Test data, describing all measures used and giving all results must be generated
whenever possible. This data must also identify who did the testing when and how
long it took;

 SCORM compliance/LMS Integration tests are done and documented (test software
used, results).

 A FAT mission is held at the premises of the contractor. The contractor in charge of
the development notifies, by e-mail, DG TAXUD that the FAT mission can start. DG
TAXUD may request the support of the contractor in charge of the Quality Assurance.

5.3.1.2 Acceptance Pre-requisites

Following pre-requisites must be fulfilled for acceptance process:

 The learning module matches the quality criteria and best practice standards;

 All the required users and technical documentation are available for review;

 The tests documentation (test scenarios and test cases) is issued by the contractor and
available for review.

The contractor has to deliver a FAT Report to DG TAXUD in two steps:

 When the contractor has defined the test scenarios and test cases, and has set up the
test data, the FAT Report must be submitted to DG TAXUD as a draft document,
including the expected results. In practice, the Contractor is testing the delivered
modules against

 Any required functional aspect (content structure and content features)

 Any required technical Item (page, animations; media object, play-ability…)

 Any required business rule (if…then…else)

 When the contractor has executed the test scenarios, the FAT Report must be
completed with the actual results of the tests then delivered to DG TAXUD for
review.

The QA Contractor will deliver separate QA FAT reports when he is involved in this
activity.
The results of the FAT activity are discussed and actions on issues are agreed on and
assigned.
A closing meeting will end the validation phase of the FAT. DG TAXUD and the
development contractor attend the kick-off and closing meetings. The QA contractor
attends these meetings if he is involved in the activity in question.
Depending on the case, DG TAXUD may accept a deliverable under the conditions
that any issue or bug, invalid text and any functional/technical malfunction gets
repaired after the Test Event. This may be the case whenever the SfA delivery date
is distant from the Test Event date.

24
5.3.1.3 Responsibilities

The contractor is delivering the FAT Report for DG TAXUD approval. Other
responsibilities include:

 Meeting the prerequisites for FAT;

 Training the testers in the specific technical procedures and techniques required to
execute the tests and manage the test environment;

 Performing the necessary FAT on any corrected release of a software before the
software enters the TE;

 Developing and maintaining the ATS;

 Developing related scenarios, heuristics and metrics measuring freeness from errors,
the instructional quality and the usability;

 Maintaining the traceability and consistency between the ATS and the didactical,
instructional design and usability specifications;

 Providing test documentation to support the testers.

 Specify and identify the testers' profile

 Secure FAT test execution by involving at least a second test profile (external)

The QA contractor is attending the FAT activity and is delivering a QA FAT Report if
he is assigned to the task.

5.3.1.4 Acceptance of the FAT

In case the FAT is not accepted, the identified issues should be resolved and a new
FAT cycle will be scheduled subject to new acceptance.
The FAT acceptance is under the responsibility of DG TAXUD and may be repeated
until DG TAXUD pronounces that the learning module under test is ready to enter in
a TE. The FAT acceptance is made by DG TAXUD after the planned termination of the
FAT. DG TAXUD may request the support of the contractor in charge of the Quality
Assurance.
If there are still major issues remaining after the last run of the FAT, DG TAXUD may
reject the FAT.

5.3.2 TEST EVENT

The TE is performed by a testing team independent from the developer team and
done in a site different from the development site.

5.3.2.1 TE Objectives

The purpose of the TE is to evaluate if the quality of the eLearning modules is

sufficient for distribution.
The TE activities are:

 Identify, raise and log issues regarding the tested learning module;

25
 Identify and raise issues regarding the accompanying technical and user
documentation (ATS, Installation guide, Operation Guide, etc.) and the used testing
material (Metrics, Heuristics, etc.).

5.3.2.2 TE Prerequisites

The TE can start when:

 DG TAXUD has accepted the FAT Report;

 The following elements are available:

 The finalised learning module;

 The updated testing material to carry out an eLearning evaluation

 The team responsible for testing has set up its evaluation group with the necessary
level of experience and competency for running the evaluation;

 The QA contractor has set up its participating team if requested by TAXUD.

5.3.2.3 TE Activities

A starting kick-off meeting will precise the main objectives of the TE. These include:

 Running the tests and recording and maintaining the result in a test log;

 Identifying and raising issues having an impact on the tested e-learning software or
documentation.

The participants in the kick-off meeting are:

 DG TAXUD;

 The team in charge of the testing;

 The contractor in charge of the development;

 QA contractor if requested by DG TAXUD.

A closing meeting is organised, during which the results are discussed and actions on
issues are assigned. This marks the end of the evaluation. The participants in this
meeting should be the same.
The contractor invites all necessary participants to the closing meeting.

5.3.2.4 Output

 The QA contractor’s report recommending the acceptance or non acceptance of the

eLearning module (if requested by TAXUD);

 Quantified results to substantiate TAXUD's decision to accept or reject the learning

module

 An update of all necessary supporting documents.

26
5.3.2.5 Responsibilities

The contractor responsible for development enables the transfer of testing duties to
the team performing the TE. This is done by:

 Providing sufficient evidence of successful FAT;

 Participating in all relevant meetings to take action on issues and change requests.

 The team responsible for carrying out the TE performs the evaluation activities and
produces the Test Report for DG TAXUD acceptance. In addition to planning and
describing the testing activities in detail, it will be:

 Logging a call for each issue raised during the testing;

 Following up the call resolution;

 Qualifying each fixing by running necessary tests;

 Producing intermediate summary reports to inform DG TAXUD on the progress.

If requested, the QA contractor provides the requested quality control and produces
the Testing Quality Control Report for DG TAXUD acceptance.

5.3.2.6 Acceptance after the TE

Based on the TE Report from both the contractor who did the tests and the QA
contractor, DG TAXUD will decide whether the module under test:

 Can be accepted.

 Can be accepted with reserves (which will be implemented in future releases of the
learning module).

 Cannot be accepted.

In the latter case, the changes agreed in the meeting will need to be implemented
and a new evaluation will be planned.

5.3.2.7 Procedure in case of no TE

DG TAXUD reserves the right not to conduct a TE. In this case the procedures
defined in chapter 5.3.2.6 applies right after the completion of the FAT procedures
described in chapter 5.3.1.4.

6 QUALITY INDICATOR

6.1 THE PRINCIPLES

Service quality indicators (SQI) indicate the quality and timeliness of the specific
deliverables. Aggregated per AP, they are computed into a Global Quality Indicator
(GQI) which provides a measurement for the quality of the delivered service/project
as a whole.
The following sections describe a method of computation for the SQI and GQI.
However, this should not be understood as being TAXUD's preferred organisational
solution!

27
The applied method must be designed in mutual agreement and has to comply with
following principles:
 Quantified quality measurements and timeliness are normalised and weighted when
computed into a general quality indicator (GQI) per AP;

 A mechanism to determine the liquidated damages is defined;

 A grace window is foreseen in case the quality of service is below target but within a
limit.

6.2 CALCULATION OF THE SQI

SQI’s are calculated using the following steps in sequence:

Collect Measurement of QoS (M)
The Measurement M (or set of measurements) of QoS has to be collected and
possibly combined according to the definition of the Measurement of the QoS.
If the minimum number of measurements required over the Applicability period to
make the SQI relevant is not attained, then the Measurement (hence SQI) has no
applicable value for that applicability period.
Normalise the Measurement (Mnorm)
For a given Measurement M, the related normalised Measurement MNorm is obtained
by applying the following formula:

M  Target
M Norm 
Target  Limit

Where the M, Target and Limit are values expressed in the same unit and part of the
SQI definition.
SQIprof as a result of the Profiling function

Once the Measurement has been normalised to MNorm, it is profiled (using the f
function) to an SQIprof, which has the following effects:
It limits the SQIprof upwards, versus irrelevant over-performance of QoS above
target;
It defines linear proportionality between the SQIprof and the under-performance of
QoS below Limit;
It sets a grace period (interval defined by the Target and the Limit) which is setting
the SQIprof to a neutral level, immunising the SQI from any positive or negative
factor.
The profiling function (f) is applied on all occurrences of the normalised
Measurements. Those calculations are provided in detail in the SQI report attached
to the Monthly Project Report.
The profiling function f is defined as follows:

Table 3 The definition of the profiling function applied to all normalised

measurements

28
If M Norm  0  SQI prof  f (M Norm )  1 i.e. the QoS leads to a Measurement
above or on Target
If 1  M Norm  0  SQI prof  f (M Norm )  0 i.e. the QoS leads to a Measurement
between Target and Limit – neutral
grace window
If M Norm  1  SQI prof  f (M Norm )  1 i.e. the QoS leads to a Measurement
on Limit
If M norm  1  SQI prof  f (M norm )  M norm i.e. the QoS leads to a Measurement
below the Limit
This profiling function is plotted in the figure below.
Profiling Model "linear penalties w/ a neutral grace window"
2

0
-5 -4 -3 -2 -1 0 1 2 3 4 5
Profiled SQI

-1

-2

-3

At limit At target
-4

-5

Normalised Measurement

Figure 3 Profiled SQI indicators

Averaged profiled SQI
When a single SQIprof is used to measure the QoS of multiple occurences of
services/delivery of the same nature, it is called an "averaged SQI", which is made
of the average of all multiple-SQIi according to the following formula:
n n

 SQI prof i  f (M norm i )

SQI prof  i
 i

n n
Where n is the number of occurrences of the given SQIprof during the applicability
period.

6.2.1 THE GENERAL QUALITY INDICATOR

The GQI can be defined as the weighted average of all SQI1 defined on the
applicability period, allowing a global assessment of the QoS for all services and
deliverables during the whole period (e.g. the duration of a Specific Contract).
To each SQI defined on an applicability period, and participating in the GQI
calculation, a normalised weight factor2 (w) has to be associated.

1
For sake of clarity, as of now, profiled SQI will be simply called "SQI".
2
"Normalised weight" means that the sum of all the weights for all SQI participating in a GQI equals 1.

29
In formula, the General Quality Indicator for the Specific Contract (GQISC) is defined
as:

GQI SC   ( SQI i  wi )
i

6.2.2 LIQUIDATED DAMAGES

The liquidated damages related to deficient QoS during an applicability period are
derived directly from the GQI calculation.
The GQI and the liquidated damages will be calculated at the end of the applicability
period. Nevertheless, over the applicability period, some "intermediary" GQI can be
calculated, in order to assess the QoS at any time.
Liquidated damages may be applied to the Service Provider in the framework of the
Service Level Agreement.

From GQI to liquidated damages calculation

The amount of liquidated damages for an applicability period (in this case the
duration of a Specific Contract) is defined by the following "P" function:

If GQI
SC
 1  Penalty = 20 % * SC/IS (SC/IS is the Fixed
Price budget for
Informatics Services)

If  1  GQI
SC
0  Penalty = 20 % * SC/IS * abs(GQISC);

GQI SC  0  Penalty = 0

Figure 4 Liquidated damage calculation function

abs means absolute-value.
The main idea behind the "P" function is to:
Have no liquidated damage when the GQI is positive, indicating overall positive QoS
for the duration of the SC;
Have liquidated damages linearly proportional to the Fixed Price budget of the SC,
when GQI is negative…
And limit the maximum amount of liquidated damages to 20 % of Fixed Price budget
of the SC when GQI gets below -1, indicating that the global QoS during the SC has
been very negative.

30
 Liquidated damage

20% SC/ISBudget
1

GQI -5 -4 -3 -2 -1
0
0 1 2 3 4 5

-1

Liquidated damages' function

Figure 5 Effect of liquidated damage function

Liquidated damages are calculated at the end of each Specific Contract and applied
on the last payment related to the Specific Contract, when applicable.
The liquidated damage will take the form of an amount to be deducted from the last
invoice for the Specific Contract.

6.2.3 GENERAL INDICATIONS ON THE USE OF SQI & GQI

This section aims at providing a general view on how SQIs will be used throughout
this contract, by:
Identifying the main categories of SQIs;
Mapping of the SQIs to their deliverables or services defined in this Framework
Contract;
Defining their relative importance, i.e. attaching their weights which will intervene in
the GQI calculation.
Note also that non-contractual SQIs could be defined in the CQP or by any mutual
agreement, for the sole purpose of having a convenient normalised instrument to
measure the level of the QoS provided by the contractor. They will not be taken into
account in the calculation of the GQI.
Category of SQIs
The following points are describing the main categories of SQIs, without prejudice to
new categories to be defined at a later stage during the period of the contract:
(1) Delivery date-related SQIs reflect whether those deliverables were delivered in
due time or if there were any delay in their submission (usually for acceptance).
The sensitive factors of the SQIs will be the target set, and moreover the limit,
which for instance, is what will differentiate the SQI on "major deliverables",
from the SQI on "common deliverables".

(2) Discrete SQIs: a way to emphasise the importance attached to a deliverable is to

have a discrete SQI solely dedicated to it.

(3) Service-related SQIs: the range of definition of those SQIs, which are measuring
the quality of the service provided, is very broad, and can be easily extended.
Some sub-categories are presented here, for the sake of illustration:

31
 (4) Training/workshop/demo performance-related SQIs: measure the level of the
quality of those types of services. Note that some of those SQIs are calculated on
the evaluation made by the attendees.

(5) Response time-related SQIs: measure if the response time to a service request has
taken place in the range specified. Typical examples of this are SQIs related to the
time needed to produce SC/RfA offers.

(6) Availability-related SQIs, measuring the availability of the services, or any SQI
related to absence/delay of the contractor at a planned meeting.

This list of categories should not in any way be considered as limiting / complete and
comprehensive.
Mapping of the SQIs to their deliverables or services
The mapping between the SQIs and the deliverables or services they relate to is
provided by the column "SQI" in the deliverables table. The mapping is only
indicative and will be contractually defined at the SC level, unless otherwise stated.
SQIs’ weight in the GQI
The relative importance of an SQI, in comparison to the other SQIs, will be given by
the definition of the weights for the calculation of the GQI.
The precise definition of those weights will be given in the SC. The intention of this
section is to provide a general indication of the importance of the SQI’s weightings,
by defining three categories of weights: the high - medium - and low weights.
Under the high weight category would fall the SQI related to the Take-Over and the
main system specification deliverables.
Under the medium weight category, would fall the SQI related to, in decreasing
order: the other system specifications.
Under the low weight category, would fall the SQI related to all other deliverables
and services for which the Technical Annex specifies SQIs, including the MPRs, the
response time to request of services etc.

6.3 THE SERVICE QUALITY INDICATORS (SQIS)

6.3.1 DEFINITION OF THE SQI

Some or all of the following attributes define a Service Quality Indicator.

SQI Attribute SQI Attribute description

SQI ID Sequential number used to identify the SQI
SQI Name A name, which allows to identify fully the SQI.
SQI Description A complete description of the SQI.
Measurement of the QoS Specifies the measurement of the QoS (or combination of set of
(M) measurements) for the SQI.
Unit of Measurement of Defines the Unit of Measurement of the QoS. For example, a SQI
the QoS aiming to evaluate duration or delays can be expressed in hours
or days.
Application period Specifies the overall period over which the SQI is calculated
(typically duration of a SC);

32
SQI Attribute SQI Attribute description
Target Target, which sets the level of the measurement, that, if reached,
would demonstrate good QoS.
Limit Together with the Target, the Limit defines the "grace window" ",
within which although the QoS is below target, the SQI is still
immunised from negative impact.
Normalised Measurement A normalised Measurement is the result of the transformation of a
(Mnorm) measure (see formula below), which renders a number
independent from the unit of measure of the QoS.
SQI Profiled (SQIprof) A profiled SQI is the result of a profiling function applied to a
normalised SQI (see function f below).
Applicable Defines the set of services and deliverables, to which SQI will
services/deliverables apply.
Minimum number of Minimum number of measurements or set of measurements
Measurements necessary for an SQI to be computable.

The following definitions are provided only as indications of what the final SQI will be
at the moment of signing the Specific Contract.

SQI01 – EXAMPLE TIMELINESS

SQI Attribute SQI Attribute Description

SQI Id SQI01
SQI Name: On time delivery of major deliverables
SQI Results: Measures the on-time delivery of a major deliverable
Calculation Deliverable SfA
formula: Difference between the actual delivery date3 for acceptance
(time stamp of e-mail, post or delivery note) and the planned
delivery date for acceptance.
Deliverable SfR
Difference between the actual delivery date3 for review (time
stamp of e-mail, post or delivery note) and the planned delivery
date for review.
Calculation Once per delivery
period:
Unit: Working Days
Target: 0
Limit: 5% of delivery time (working days, rounded up); if under 200
working days; 10 working days; if delivery time is equal or larger
than 200 working days.
Weight: (high) The weight will be decided by the European Commission
during the handover process.
Applicability SC
Period:

3 For the SQI computation, a working day starts at 08H00 and ends at 20H00 on the same day.

33
 SQI Attribute SQI Attribute Description

Minimum 1
Number of
Events:
Comments All measures will be averaged to provide the SQI measure from
the beginning of the SC to the end of the reporting period.
The liquidated damages will be calculated per GQI and related
budget: per project budget (if GQI per major project) and/or per
horizontal Management & support service budget (if GQI per
Management & service deliveries other than major projects).
The following rules apply as exceptions in SQI calculation:
(1) Any delay in the delivery date of the comments related to the
SfR version4 will postpone the delivery of the SfA version by
the corresponding number of days;

(2) Any delay in the analysis of the AP, the review meeting and
the transmission of meeting decisions versus initial schedule
will postpone the delivery of the SfA version by the
corresponding number of days;

(3) In the case of rejection of a SfR or SfA version of a

deliverable, the number of working days between the delivery
date and the communication of the rejection of the concerned
version of the deliverable is taken out of the delay of the new
SfR or SfA version;

In case of dependency between two deliverables (or two versions

of a deliverable), the SC or the RfE offer must mention the
targeted deadline of the second deliverable (or the second version
of a deliverable, i.e. SfA version) in reference to the actual
delivery date of the first deliverable (or the first version of a
deliverable, i.e. SfR version) by a relative number of additional
working days.

SQI02 – EXAMPLE QUALITY ASSESSEMENT

SQI Attribute SQI Attribute Description

SQI Id SQI02
SQI Name: Didactical adequacy – DG TAXUD’s evaluation
SQI Results: Measures the didactical quality of a learning product (e.g. EN
module) based on the scores obtained during an evaluation
against predefined criteria and quality levels
Calculation Actual average score according to project group evaluation.
formula:
Calculation Once per learning session
period:
Unit: %

4 The date of the last comments received from DG TAXUD is taken as the basis for the date of reception of the comments.

34
 SQI Attribute SQI Attribute Description

Target: 85%
Limit: 50
Weight: The weight will be decided by the European Commission during
the handover process.
Applicability SC
Period:
Minimum 1
Number of
Events:
Comments -

SQI02 would be calculated as follows:

Table 4 Weighting for the didactic adequacy indicator SQI02
Component Description SQI Weight in
Name Profile SQI
KPI001 Quality of language SQI02 15%
(list given as an example)
orthographic aspects
typing errors
clarity of expression
Cross-Course consistency
Compliance with legal texts
KPI002 Quality of interface SQI02 25%
(list given as an example)
intuitive
user-friendly
ease of use
non-labour intensive
balanced content types
Visual aspects quality (video, images,
animations)
Synchronisation of
voice/pics/animations
KPI003 Quality of pedagogical approach SQI02 60%
(list given as an example)
Clear and logical structure
Problem-solving oriented
Adult-learning oriented
Interactivity
Blended in Practice
Accurate translation of learning

35
 objective
Good topic balance
Content consistency

In order to measure the value of each KPIs, it is key to determine the levels of
expectations that are required on the final version of the delivered product.
Prior to the start of the development phase, at the latest, DG TAXUD will formulate
its requirements with respect to dimensions across KPI001, KPI002 and KPI003.
For each KPI, it should be explained in details all tangible elements that will
determine whether a quality criteria belongs to one of the following judgment value:
(1) Significantly below Expectations (Value = 25%): Major improvements are
required.

(2) Needs some Improvement (Value = 50%): Limited Corrections have to be made.

(3) Good and Reliable (Value = 75%): Acceptable Solution that serves its purpose.

(4) Outstanding (Value = 100%): Best Approach, innovative, no objection can be

made.

DG TAXUD may decide to remain very generic on these KPI’s definition or very
detailed in translating these into many quality criteria. The level of granularity is to
be determined upfront.

6.3.2 SQIS’ WEIGHT IN THE GQI

The SQI Weights will be defined by DG TAXUD for each SC.

The relative importance of a deliverable or service to be provided – both called
“event” in the rest of this section – is defined, as the weight per deliverable.
x DLV
wDLV  The following values for x are used:
 xDLV
GQI

 1: Low Category

 2: Medium Category

 3: High Category

 4: Very High Category

Following this definition, the weight of each SQI is evaluated by:

 Counting the number of anticipated events constituting the given SQI for the
concerned Project;

 Multiplying this number by the relative importance of the deliverables of the SQI;

 Translating this value in a percentage vis-à-vis the overall value for all SQIs.

.
However, as the number of events is known only after the execution of a SC or
project, the resulting value presented in the following tables must be considered as
indicative operational values to be used all along the contract, while the contractual

36
values, and the resulting contractual GQI (or project GQI), will be re-calculated at
the end of each SC, according to the actual number of events of the concerned SC
(or project).

<Scope>

SQI Event Nbr of SQI SQI Min Nbr of

relative events importan Relative events
importanc ce weight
e (%)
TBD TBD TBD TBD TBD TBD

Normally, one indicative table per project is needed. The sum of all weights in a
project GQI is 1.
Every Learning Module from the beginning to the issue of the English Optical Disk
version is considered as one project.
The translation and localisation, if done by one stakeholder of the consortium under
an SC is also considered as one project.
Other deliverables fall under the horizontal management (common) GQI.
New projects and related GQIs can be agreed on and defined in RfE/RFA or RFA
procedures.

7 AUDITS AND MANAGERIAL REVIEWS

7.1 MANAGERIAL REVIEWS

All the actions, issues and risks that have been identified during any of the meetings
of any of the projects part of the sub-contract will be logged in a central repository,
which will be delivered on a monthly basis as part of the MPR.
In order to ensure a full follow-up of the projects, all those actions, issues and risks
will be walked through during the BMM.

7.2 PROJECT QUALITY AUDIT

7.2.1 AUDIT OBJECTIVE

A Project Audit is a thorough examination of the project management, the technical

development process and the project deliverables in order to provide the project
management with:

 Evidence that the project is progressing according to plans;

 Evidence that the project is under control: information is accurate and up to date
enabling the managers to take effective decisions;

37
 A clear set of recommendations for immediate and future action.

A Project Audit will definitely be carried out if TAXUD doubts the contractor's
commitment to quality work and realises the contractor's managerial review
mechanisms fail to provide good quality.

7.2.2 CONFIDENTIALITY

The audit is strictly confidential.

7.2.3 AUDIT PREPARATION

The Project Audit shall be carried out by a qualified Auditor at the request of DG
TAXUD.
This Auditor shall design an Audit Plan of intended audit activity and will discuss it
and agree on it with DG TAXUD which will accept it before the audit activity takes
place.

7.2.4 AUDIT EXECUTION AND REPORTING

The Audit execution includes one or several meetings between the Auditor and the
Audited for a detailed discussion of the input material identified during the Project
Audit preparation.
The Auditor prepares the Audit Report. All Project Audit topics identified in the Audit
Plan must be addressed in the Audit Report.
For each topic, the Auditor identifies specific deficiencies as "Negative Results".
Topics that are well implemented are identified as "Positive Results". Each
observation shall be sequentially numbered. These observations will be collected
together and at the conclusion of the audit, after any necessary investigation has
been concluded, the Auditor shall classify each observation as either:

 'Serious' (Category 1) - indicating a total lack of control of a key quality feature (this
will have an impact on the contract);

 'Significant' (Category 2) - indicating a potentially serious lack of control or a number

of minor observations which indicate a worrying trend;

 'Minor' (Category 3) - indicating a single non-compliance with a standard or

requirement which does not in itself signify lack of control.

All Audit observations will be appended to the Audit Report, together with
recommended corrective actions for each of the observations.
The Audit Report is presented to DG TAXUD at a conclusion meeting that is
organised between DG TAXUD and the Auditor.
After having accepted the audit report, DG TAXUD translates some or all
recommendations of the Audit Report into Actions and discusses them with the
Audited and the Auditor during an Action Plan Meeting to take place.
The minutes of this meeting will state the duties and responsibilities of all project
partner(s) as well as the target dates for implementing the actions agreed upon.

7.2.5 PROJECT AUDIT FOLLOW-UP

The implementation of the agreed actions will be monitored by DG TAXUD during

bilateral meetings or by a follow-up Audit.

38
8 ESCALATION

In order to manage possible issues, an escalation mechanism is put in place. When

an issue cannot be solved at a specific level of operation the escalation to the next
level takes place.
When an issue between a contractor and DG TAXUD representative cannot be
solved, DG TAXUD's Project Manager in charge of the project must be alerted. This is
level 2.
When an issue between a contractor and Project Manager of DG TAXUD in charge of
the project cannot be solved, DG TAXUD's Head of unit in charge of the project must
be alerted. This is level 1.
When an issue between contractors cannot be solved, DG TAXUD Head of unit in
charge of the project must be alerted.

 Level 2 involves:

 DG TAXUD's Project Manager;

 the contractor's Contract manager;

 Issues: Contractual issues and complaints on the quality of service provided by the
service provider.

 Level 1 involves:

 Head of Unit of DG TAXUD;

 Director of the contractor;

 Issues: Conflict situation.

9 TOOLS, TECHNIQUES, AND METHODOLOGIES

9.1 COMMENTS EXCEL SHEET

The Comments Excel Sheet will be used to support the review process of the
deliverables covered by the SC (Refer to section 5.2.4.2 for review cycle and
process). This table gathers the following information:

 Reviewers Comments:

 Reviewer Id;

 Comment date;

 Comment Location;

 Comment Type:

 Defect

 Change

39
 Comment

 Question

 Comment

 Severity

 Blocking

 Will Block

 Non-Blocking

 Author Position:

 Author answer: “to be implemented”, “no action”, “to be discussed”;

 Issue Type

 Change Request

 Anomaly

 Comment

 Question

 Author position and comments.

 Implementation Cost

 V-O recording involved

 Review Meeting Decision:

 Meeting Decision;

 Meeting comments.

 Final Implementation Comments;

 Verification Comments.

The Excel sheet is used for the deliverables identified as part of the Activity
Packages. The other deliverables, such as the PM documents (MPR, minutes of
meeting) are reviewed using the mechanism of revision marks in the document.
The Comments sheet is presented as annex I to this document.

9.2 MPR ANNEXES

The following sections describe each of the annexes of the MPR.

Annex 1: Project Plan (PPL)

40
The Project Plan presents the overall planning of the different activities covered by
the SC, including reviewing activities and major milestones (milestones of the
deliverables (SfR and SfA versions).
The Project Plan is maintained all along the project with the percentage complete for
all the activities and the plan resulting from the RfA that are launched during the SC.
The deadlines are not part of the monthly update of this deliverable, as those
deadlines are maintained as part of the DTM. (Refer to next section)
Annex 2: Deliverables Tracking Matrix
The Delivery Tracking Matrix (DTM) provides the detailed status of each deliverable
and of each service provided in the context of the SC.
The DTM contains, on the "per deliverable / service" basis:
 The identification and short description of the deliverable or service;
 The starting date, with the corresponding event;
 The review cycle mechanism and timeline;
 The planned delivery dates for every step of the delivery and of the review
cycle;
 The acceptance mechanism and the status of the deliverable
(SfR/SfA/Accepted);
 Where applicable, The automated computation of the time-based SQI’s up to
the reporting period on a per deliverable basis;
 A free comment used, for instance, in case of change of deadline for a given
deliverable.
The DTM is used to follow-up the deliverable deadlines and review cycles.
If a new RfA is issued, the related deliverables have to be included in the DTM.
Annex 3: On Demand Budget Follow-up
Annex 3 will detail the progression of the on demand budget all along the contract.
This annex will detail the overall on demand budget, the budget that is reserved in
the context of RfE and corresponding offer, and the budget that is consumes as a
result of the approval of the response to the offer.
The travel and subsistence budget is included in this annex.
Annex 4: RfE / RfA Status
This annex will detail the overall status, of the RfE that have been issued all along
the SC, of the offers submitted as response to those RfE, and of the RfA that have
been launched as a result of those offers, or as a result of the activation of the
Additional Fixed Price related activities.
This annex will clearly indicate the RfA that have been ordered for the missions,
technical meetings and workshops. All the budgets will be included as well on a per
RfE / RfA and WP basis.
Annex 5: Actions Log
The actions log annex presents the consolidated list of the actions that have been
raised in the different project meetings and management meetings of the SC,
including the assignee, due date, status and resolution description.
Annex 6: Issue Log
The issue log annex will provide all the issues that have been encountered all along
the SC. The issues will be presented with a description, a qualification of their
priority, and the resolution progress.
Annex 7: Risk Management Log

41
This annex will present all the risks that have been identified all along the SC. The
risks will be presented with a description, a qualification of their priority, the
probability of happening and the potential impact on the deliverables and on the
activities concerned.
Annex 8: Status of the SQI and GQI
This annex presents a complete analysis of the SQI and GQI from the beginning of
the SC. This annex will present the current status on a per SQI basis, including the
number of events and the value of each SQI. An overall value of the GQI per project
will be provided as well.
In a separate sheet, this annex will present the evolution of the SQI and GQI all
along the SC.
Annex 9: Decision Log
This annex presents the different decisions that will be made during the course of
the SC and that will further be valid for further SC’s. An example would be to decide
on the reuse-ability of some developed asset (mascot, template, process…)
Annex 10: Change Request Details
This annex lists each outstanding requested change and summarizes additional
budget cost and planning change.

10 MEDIA CONTROL

All deliverables documents will be submitted to DG TAXUD as attachment to e-mail.

Official AP deliverables will be compressed.
Annexes and Comments Sheet are always exchanged through .zip files attachment.
eLearning modules are delivered to DG TAXUD by FTP. An access to the contractor's
file server must be provided to DG TAXUD.
The contractor must ensure the confidentiality of all deliverables transmitted to DG
TAUXD up to the level of confidentiality ensured by the transmission means.
The contractor must ensure that the confidentiality policy regarding the e-learning
modules and related documentations and specifications is strictly respected by the
people working on the deliverables covered by this contract.

11 RECORDS COLLECTION, MAINTENANCE, AND

RETENTION

All electronic projects’ artefacts (documents, code, applications – SfR and SfA
versions) will be kept on a fileserver and are subject to the contractor's standard
backup strategy. (Incremental Daily backups, full weekly backups)
All physical projects’ artefacts (documents, CD/DVD-ROMs, tapes) documents will be
stored in the contractor’s office.

12 RISK MANAGEMENT

This chapter presents a formal approach for managing risk in the project.

42
Managing risk tries to ensure that adverse events are avoided and/or their negative
impact is minimised. The objective of the project Risk Management is to capture
these possible events and provide a mechanism to control and mitigate them.
Figure 6: Risk Management Process

Plan Risk
Management

Assessment

Identify Identified Analyse Prioritised Plan

Risks Risks
Risks Risk Mitigation

New Risk
Risks Mitigation
Plan
• Mitigation
• Contingency
• Triggers
• Categorisation
• Probability
• Impact & cost
• Owner assignment

Mitigation

Reassess Progress Assess Mitigation Mitigate

Report Actions
Risks Effectiveness Risks

43
13 ANNEXES

13.1 ANNEXE I: EXAMPLE OF COMMENTS SHEET

Deliverable Comments Excel sheet

Deliverable ID DLV-
Deliverable Name
Deliverable Source XXXXXXXXXXXX
Author

Reviewer's list Nam Initials

eBirgit Reiser BRE
Vasco-Sascha Steltenkamp VST
Annette Poro APO

Enter Comments

Value Definition
Error in delivered product, in contradiction with previously defined
elements. Assumptions and décisions are not properly reflected in
Defect the delivery
Assumptions and/or décisions have changed and will require an
Change adaptation of the delivered item
Question Request for explanation or clarification
Advise, information that may help the author with further
Comments development of the deliverable

The identified issue is causing the delivered product to malfunction

and making it impossible to go further with providing with comments
on parts of this delivery. E.g.: Technical limitation of a delivered
training application that does not allow to proceed further within the
Blokking quizz part of the module
The identified issue has only consequences limited to itself. The
problem has no effect on the rest of the delivered product but still
HAS TO be resolved. E.g.: A Voice-over is de-synchronized with
Will Blok written text.
The reported comment consists more in a "nice to have" or does
not have to be implemented at all (question/comment/enhancement
suggestion…). E.g.: discussion on the background colour of a text
Non-Blokking box

Table 5: Comments Cover Sheet

Location Comment Originator Type Severity
General Comment Library content needs to be checked for correctness and completeness DGT 1 - Defect 3 - Non-Blokking
General Comment Add link to Community Customs Code in "Library" (similar to CE module) DGT 1 - Defect 3 - Non-Blokking
General Comment Glossary content needs to be checked for correctness and completeness DGT 1 - Defect 3 - Non-Blokking
Explanation on "when to best use Library and Glossary" needs to be added (similar to CE
General Comment module) DGT 1 - Defect 3 - Non-Blokking
General Comment Video display section (for Commissioner's speech) should be made bigger DGT 1 - Defect 3 - Non-Blokking
General Comment Loading time needs to be shortened in general (between the LU's) DGT 1 - Defect 3 - Non-Blokking
General Comment Speed related to building blocks on various screens need to be checked DGT 1 - Defect 3 - Non-Blokking
General Comment Mock-up does not start via index.html (only menu screen, LUs can not be accessed) DGT 1 - Defect 3 - Non-Blokking
Linguistic revision is needed. Capital letters are where they should not be; small letters are
General Comment where capital letters should be; wrong conjugations… DGT 1 - Defect 3 - Non-Blokking
General Comment Make sure, that always the new, reworked map is used DGT 1 - Defect 3 - Non-Blokking

General Comment Add a teaser animation as introduction to the final evaluation test (compare to NetAGREX) DGT 1 - Defect 3 - Non-Blokking
Improve the transitions between the screens. The module as it is gives too much of the
General Comment impression that it has been cut and pasted from the Customs' Module DGT 1 - Defect 3 - Non-Blokking
General Comment Create additional content / slides according to the structural comments on 'Sheet3' DGT 1 - Defect 3 - Non-Blokking
Old: Hello, i am Max. I am a trader and I need to know more about AEO legislation. I will
follow this course with you. - New: Hello, I am Max. I am a trader and I need to know more
U00-A00-S01 - E11 about AEO legislation. I will follow this course with you. DGT 1 - Defect 3 - Non-Blokking
Old: Hello, I am Stefan. I work for the customs authorities and I need to know more about
AEO legislation. I will follow this course with you. - New: Hello, I am Max. I am a trader and
U00-A00-S01 - G11 I need to know more about AEO legislation. I will follow this course w DGT 1 - Defect 3 - Non-Blokking
Old: The library to access other resources. - New: The library button to access other
U00-A00-S01 - E37 resources. DGT 1 - Defect 3 - Non-Blokking

Table 6: Example of a Comments Sheet (xls)

44
13.2 ANNEX II: EXAMPLE OF QUALITATIVE EVALUATION
QUESTIONNAIRE

The following template is used during a common Test Event in order to evaluate
produced eLearning modules. This questionnaire can serve as a basis to evaluate the
actual quality of an e-Learning Module by computing a KPI a SQI value for each
product.
Container Examination eLearning Course
Test Event, Vienna, Austria, 15th and 16th November, 2007
OVERALL EVALUATION OF THE COURSE

To be Completed by Participants After Studying the Course at the Test Event

Participant Details
1. Name of Test
Person

2. National
Administration

45
 Container Examination e-Learning Course
Test Event, Vienna, Austria, 15th and 16th November, 2007

Course Evaluation Questionnaire

To be Completed by Participants during the Test Event

Overall Evaluation of the Course

When you have finished studying the entire course and fully completed the scoring
sheet , you should only then complete this overall evaluation of the course.

You should TICK ONE OF THE FIVE BOXES FOR EVERY STATEMENT to indicate
your level of agreement or disagreement
with that statement

Ease of Use
Strongly Agree Neither Disagree Strongly
Agree
Agree Disagree
nor
Disagree
The course is easy to use, even
for someone with no e-learning
experience
It is easy for a learner to move
around within the course
It is easy for a learner to find
what he/she needs in the course
A learner on his/her own could
use this course without help

Learner Interest and Motivation

I became quickly interested

when I first opened the course
I felt motivated to continue
studying the course
I was interested in the subject
matter at all times

Learning Content

Strongly Agree Neither Disagree Strongly

Agree
Agree Disagree

46
 nor
Disagree

The learning objectives are clear

for each Unit

The content is correct, accurate

and relevant

The content is well structured

and organised

The amount of content is

required for the learning
objectives

The exercises at the end of Units

are relevant and helpful

The examples and case study

are relevant and useful

There is adequate recap and

summary of key points

The duration of the course is

necessary to meet the objectives

The worksheets help relate the

material to the national
workplace

The glossary is a useful

reference source

The library contains useful

additional material for learners

The Web Links are relevant and

useful for learners

Visual Interface
Strongly Agree Neither Disagree Strongly
Agree

47
 Agree nor Disagree
Disagree
The course is visually attractive
and engaging

I am satisfied with the following

aspects of the visual interface

 The Colour Scheme

 The Screen Layout and

Graphical Design

 The Navigation Options (How

to move around)

 The Customs Sign Mascot

 The Smuggler Mascot

 Use of photographs to support

learning

 Use of drawings to support

learning

 Use of animations to support

learning

 Use of video to support

learning

Interactivity

Strongly Agree Neither Disagree Strongly

Agree
Agree Disagree
nor
Disagree
The course is adequately
interactive

I am satisfied with the following

interactivity elements

 Learner controls pace of

progress through material

 Learner can replay the material

48
 Learner asked to choose correct
answer to text questions

 Learner interacts with photos

during exercises

 Learner asked to input his/her

own answers in text form

 Learner asked to reflect and

discuss questions offline

to apply learning to own

situation

 Learner takes the persona of

the lead officer in the case
study

 Learner is asked to complete

the actual record of the case
study

 The Unit Nine Test of Learning

Sound
Strongly Agree Neither Disagree Strongly
Agree
Agree Disagree
nor
Disagree
The use of sound in this course
supports learning

Sound is adequately used

throughout the course
The quality of the Customs Sign
Mascot voice is adequate and
engaging
The quality of the Smuggler
voice is adequate and engaging
Sound is adequately
synchronised with other course
elements

Use of the Course and Blending the Course With Other Materials and
Methods

Strongly Agree Neither Disagree Strongly

49
 Agree Agree Disagree
nor
Disagree
It will help learners if this
course material is
supplemented with national
training materials relevant to
the national context
It will help learners if this
course is blended with
classroom training
It will help learners if this
course is blended with
practical 'hands- on' training
It will help learners to have
the support of a trainer
Learners will require
technical support to be able
to use the course

Overall Reaction
This is a useful, relevant and
well designed learning
resource that addresses real
learning needs

Potential Use in National Administrations YES NO

I would like to see this course in use in my national
administration
I will recommend or support the use of this course within my
administration
My national Customs administration uses e-learning to train
Customs staff
E-learning is blended with other learning materials and methods
in my administration
Trainers in my administration would be able to use this course
as a training resource
Customs Officers in my administration would be able to use this
course
Customs staff have PCs and internet access in my national
administration
It is possible in my national administration to use an e-learning
course during normal working hours
I agree with the study of e-learning during normal working
hours
My national Customs Administration has at least one dedicated
e-learning room
My national Customs administration has a written training policy

50
 for Customs staff
My national Customs administration has a written training policy
that include support for e-learning

The aspects of this course that work The shortcomings of the course, in my
well, in my opinion are………. opinion are…..

END OF QUESTIONNAIRE

51
13.3 ANNEX II: EXAMPLE OF WEEKLY PROGRESS REPORT

Project: Name
Date: 01/01/20XX

Project Data

Author:
Current stage:
Project planned end date:

Summary

Since last
update
Planned for next
period
Overall Status
AMBE
GREEN RED
R
Scope
Rating
Quality
Planning
Budget
Red: (Risk of) impact on whole Project – Amber: (Risk of) Impact on Key Milestones

MILESTONE PLAN:

Planned
Stage
Stage deliverables /Actual Notes
name
date
Scope Specification
Instructiona Technical Specification
l Design
IDS
Specificatio
n Functional prototype

Storyboard Storyboard

SfR English Module

SfA English Module
Installation/Release Manual

52
 SfA – DEV/FAT
Sign-off

ITEMS FROM WEEKLY PROGRESS MEETING (DDMMYYY):

ID Type Descriptio Due Comment Priority Severit

Action, n (High y (High
Change
Medium Medium
Request
Low) Low)
Decision,
Issue,
Question)

OPEN ITEMS FROM PREVIOUS WEEKLY PROGRESS MEETING:

ID Type Descriptio Due Comment Priority Severit

Action, n (High y (High
Change
Medium Medium
Request
Low) Low)
Decision,
Issue,
Question)

53