Validation Test Plan

Validation Test Plan for Intelligent Transport Systems in Europe

Author(s)

Project Cooperative Systems for Sustainable Mobility and Energy Efficiency

Date Actual: 13 May 2011

Project
Coordinator

Abstract
The main aim of this document is to draw up an overall plan a define a common approach
for the testing and validation in PROJECT to ensure that the Pilot Sites activities are
executed in such a way that valid and rigorous results can be obtained.

Keyword list R&D, Validation, VTP, Impact assessment, Evaluation, Assessment objectives,
Performance indicators, Targeted criteria, Research questions, User acceptance, Driver
behaviour, Assessment methods, Field trials, Test drives, Human factors assessment
methods, Data collection, Data analysis, Measurement plan, Test case form, Pilot sites,
Validation criteria, Applications, Progress indicators, Impact indicators, Validation test
case list, Legal aspects, organizational aspects, CO2 reduction, fuel consumption
reduction, mobility, Baseline, Reference case, V-model methodology, ICT systems

Nature of Report
deliverable

Dissemination Confidential

Project co-funded by the European Commission within the ICT Policy Support Programme

European Commission
CIP PSP

Project number 270952

CIP-ICT-PSP

5/14/2019 Page 1 of 113 Version 2.0

 Validation Test Plan

Control sheet

Version history
Version Date Main author Summary of changes
number

1.0
1.1
1.2
1.3
1.4
1.5
2.0

Approval
Name Date
Prepared

Reviewed

Authorized
Circulation

Recipient Date of submission

European Commission

Statement of originality
This deliverable contains original unpublished work, except where clearly indicated otherwise.
Acknowledgement of previously published material and of the work of others has been made through
appropriate citation, quotation or both.

5/14/2019 Page 2 of 113 Version 1.4

 Validation Test Plan

Table of Contents
EXECUTIVE SUMMARY ................................................................................... 8
1. INTRODUCTION ..................................................................................... 9
1.1. Purpose and scope of this deliverable ............................................................. 9
1.2. Role of validation ...................................................................................... 9
1.3. Structure of the document ......................................................................... 10
1.4. Related documents................................................................................... 10
1.5. Terminology and abbreviations................................................................... 11
2. METHODOLOGY ................................................................................... 13
2.1. Validation fundamentals ........................................................................... 13
2.1.1. Validation objectives............................................................................. 13
2.1.2. Validation categories ............................................................................ 14
2.1.2.1. Environmental ................................................................................... 14
2.1.2.2. Mobility ........................................................................................... 14
2.1.2.3. Safety ............................................................................................. 15
2.1.2.4. Human Factors indicators: user acceptance and driver behaviour .................. 15
2.2. At a glance............................................................................................. 15
2.3. Definition stage ...................................................................................... 17
2.3.1. Research Questions ............................................................................... 19
2.3.2. Hypotheses.......................................................................................... 20
2.3.3. Targeted Criteria ................................................................................. 20
2.3.4. Performance Indicators .......................................................................... 21
2.3.5. Experimental design .............................................................................. 22
The experimental design in such a project as PROJECT, a pilot, begins with the definition
and selection of the suitable indicators according to previous tasks defining RQ, HY and TC
and also considering the resources available. Since PROJECT project is almost 100% real
testing, specific test methods and good quality mesurements are critical. .................... 22
2.3.5.1. Definition and selection of indicators ...................................................... 22
2.3.5.2. Tests methods ................................................................................... 22
2.3.5.2.1. Field trials..................................................................................... 22
2.3.5.2.2. Driving tests .................................................................................. 23
2.3.5.2.3. Subjective assessment methods: interviews/questionnaires ....................... 23
2.3.5.2.3.1. Usability..................................................................................... 24
2.3.5.2.3.2. Acceptance.................................................................................. 24
5/14/2019 Page 3 of 113 Version 1.4
 Validation Test Plan

2.3.5.2.4. Observation ................................................................................... 25

2.3.5.3. Measures and sensors .......................................................................... 25
2.3.5.3.1. Categories ..................................................................................... 25
2.3.5.3.2. Integrity of measures ....................................................................... 26
2.3.6. Test cases definition ............................................................................. 27
2.3.6.1. Header ............................................................................................ 27
2.3.6.2. Test Setup Frame ............................................................................... 28
2.3.6.3. Targeted Criteria ............................................................................... 29
2.3.6.4. Results reporting ............................................................................... 29
2.4. Operation stage ...................................................................................... 30

.......................................... 30
2.4.1. Data acquisition ................................................................................... 30
2.4.2. Data logging ........................................................................................ 31
2.4.3. Data Analysis ....................................................................................... 32
2.5. Impact assessment stage ........................................................................... 33

.......................................... 33
5/14/2019 Page 4 of 113 Version 1.4
 Validation Test Plan

3. PILOT SITES AND APPLICATIONS ............................................................... 37

3.1. Validation environment ............................................................................ 37
3.2. PS#1: Private vehicles in the urban environment ............................................. 38
3.2.1. Pilot Site brief description ...................................................................... 38
3.2.2. Applications to be validated .................................................................... 39
3.2.2.1. Eco-Traffic Management and Control ...................................................... 39
3.2.2.2. Eco-Transit and Dynamic Parking management. ......................................... 40
3.2.2.3. Eco-Dynamic Access Management ........................................................... 40
3.2.2.4. Eco-Driving and Driver Behaviour Change ................................................. 41
3.2.3. Validation criteria ................................................................................ 42
3.2.4. Outline of planned test cases ................................................................... 44
3.2.4.1. Validation Test cases list ..................................................................... 45
3.2.4.2. Timing ............................................................................................. 46
3.2.5. Progress and impact indicators ................................................................ 46
3.3. PS#2: Private vehicles in the Inter-Urban environment ...................................... 50
3.3.1. Pilot Site brief description ...................................................................... 50
3.3.2. Site Legacy Systems............................................................................... 51
3.3.3. Applications to be validated .................................................................... 51
3.3.3.1. Drawing attention to the traffic situation via streetlight ............................. 51
3.3.3.2. Drawing attention to the traffic situation via mobile trailer ......................... 52
3.3.3.3. Drawing attention to the traffic situation via on-board unit ......................... 52
3.3.4. Validation criteria ................................................................................ 52
3.3.5. Outline of planned test cases ................................................................... 53
3.3.5.1. Validation Test cases list ..................................................................... 54
3.3.5.2. Timing ............................................................................................. 55
3.3.6. Progress and impact indicators ................................................................ 55
3.4. PS#3: Public Transportation in the urban environment ..................................... 56
3.4.1. Pilot Site brief description ...................................................................... 56
3.4.2. Site Legacy Systems............................................................................... 57
3.4.3. Applications to be validated .................................................................... 58
3.4.3.1. Enhanced Bus intersection logistic with eco driver support ........................... 58
3.4.3.2. Congestion prevention with eco driver support .......................................... 58
3.4.3.3. ISA/Intelligent Speed Adaptation ............................................................ 59
3.4.4. Validation criteria ................................................................................ 59
5/14/2019 Page 5 of 113 Version 1.4
 Validation Test Plan

3.4.5. Outline of planned test cases ................................................................... 60

3.4.5.1. Validation Test cases list ..................................................................... 61
3.4.5.2. Timing ............................................................................................. 62
3.4.6. Progress and impact indicators ................................................................ 62
4. LEGAL AND ORGANISATIONAL ASPECTS ...................................................... 64
4.1. Briefing of test participants and contractual agreements .................................. 64
4.2. Data privacy .......................................................................................... 64
4.3. Insurance ............................................................................................... 64
4.4. Licensing requirements ............................................................................. 64
4.5. Ethical rules ........................................................................................... 64
5. References ........................................................................................ 65
ANNEXES ................................................................................................ 66

Figures
Figure 1. PROJECT Validation V-model. Validation methodology at a glance ............................ 16
Figure 2. Targeted criteria generation process ............................................................... 19
Figure 3. Key steps for a Study Design ...................................... Error! Bookmark not defined.
Figure 4. Header section of Test Cases template ......................... Error! Bookmark not defined.
Figure 5. Test Setup Frame section ......................................... Error! Bookmark not defined.
Figure 6. Targeted criteria section of Test Cases template ............. Error! Bookmark not defined.
Figure 7. Results reporting of Test Cases template ....................... Error! Bookmark not defined.
Figure 8. Pilot sites environment overview ................................................................... 37
Figure 9. Italian Pilot Site (Luminous Path) ................................................................... 38
Figure 10. Italian Pilot Site (Parking areas) ................................................................... 39
Figure 11. EcoDrive++ application logical architecture...................................................... 42
Figure 12. Italian Pilot Site Validation Test Cases List ...................................................... 45
Figure 13. PROJECT Cooperative Congestion Avoidance scenario on highway ........................... 51
Figure 14. Austrian Pilot Site Validation Test Cases List .................................................... 54
Figure 15. Part of Swedish Pilot Site: intersection at Hjalmar Brantingsgatan .......................... 57
Figure 16. Planning view of the traffic light installation at the intersection of Hjalmar
Brantingsgatan - Wieselgrensgatan – Långängen.............................................................. 58
Figure 17. Austrian Pilot Site Validation Test Cases List .................................................... 61

5/14/2019 Page 6 of 113 Version 1.4

 Validation Test Plan

Tables
Table 1. Example for expected impacts definition........................ Error! Bookmark not defined.
Table 2. System Usability Scale (SUS) ....................................... Error! Bookmark not defined.
Table 3. The nine items in the acceptance assessment questionnaire . Error! Bookmark not defined.
Table 4. Objectives for Impact Assessment ................................................................... 36
Table 5. Hypothesis, targeted criteria and related performance indicators per category ............. 44
Table 6. Italian Pilot Site timing ................................................................................ 46
Table 7. Italian Pilot Site Progress and Impacts indicators ................................................. 50
Table 8. Hypothesis, targeted criteria and related performance indicators per category ............. 53
Table 9. Austrian Pilot Site ...................................................................................... 55
Table 10. Austrian Pilot Site Progress and Impacts indicators .............................................. 56
Table 11. Hypothesis, targeted criteria and related performance indicators per category ............ 60
Table 12. Swedish Pilot Site timing............................................................................. 62
Table 13. Swedish Pilot Site Progress and Impacts indicators .............................................. 63

5/14/2019 Page 7 of 113 Version 1.4

 Validation Test Plan

EXECUTIVE SUMMARY

The objective of work package 5 is to design a validation and assessment procedure for the mobility
services tested in the three project pilot sites which applies a common methodology. It must also
ensure that this is applied in all sites in a coherent way.

The present document provides a detailed description of the Validation & Test Plan. The general
approach adopted by the project for the testing and validation is the often-used V-model
methodology for the development and implementation of ICT systems. The main objective is to
evaluate their impact on the sustainability and energy efficiency of transport, and to verify whether
the targets have been achieved in relation to the reduction in CO2, other emissions and fuel
consumption, energy savings, and the impact on traffic flows and safety. Non-technical
characteristics will also be considered, including legal and organisational aspects and the optimum
installation conditions as well as an assessment of the service level achieved in close-to-real traffic
conditions.

Regarding the tests plan, for each application, the module, the systems, the scenarios, the tools or
test sites and the assessment criteria are defined. The test plans are defined in such a way that the
overlap between tests of different applications is minimized. Furthermore, effort is put in covering
all functionalities. For each application, at least one complete system test is defined. If required,
additional tests will be defined during the test phase. Due to this approach, the test phase
continues after the delivery date of the present document. The project server will provide
information on the current state of testing and reporting.

This will be supported by the provision of suitable assessment methodology, the use of common
templates for the planning, for the installation procedures and for the test development, as well as
the use of predefined objectives from a user point of view to assess user satisfaction and behaviour.

The project systems and applications, will be tested predominantly by field trials in a real life
context which has been stated as an important task for PROJECT tests. The field trials are realised
in two different environments - highway and urban roads -.

The main outcome of D5.1 Validation and Test Plan is the definition of the PROJECT targeted
criteria, in order to validate the impact of PROJECT applications in a real world, as well as provide
the required templates to cover technical and non-technical aspects by the three pilot sites in a
common way.

5/14/2019 Page 8 of 113 Version 1.4

 Validation Test Plan

1. INTRODUCTION
1.1. Purpose and scope of this deliverable

The PROJECT Validation Test Plan identifies the expected impacts of PROJECT applications and
services which will be conducted at the pilot sites (validation sites) of Salerno, Austria and
Gothenburg, together with the individuals or groups who will be affected and the indicators which
could be used to measure those impacts.

This document presents the criteria which will be used to select the impacts for validation, it
provides a brief description of the methods which will be used to measure them and it includes the
validation matrixes to be used for evaluation at the pilot sites.

What should be very obvious after reading this deliverable is the methodology on preparing (firstly)
and analyzing (lastly) the validation, together with the sites and resources involved.

Moreover, this document provides the required templates to cover technical and non-technical
aspects by the three pilot sites in a common way.

1.2. Role of validation

Validation has a major role in PROJECT, since Cooperative Systems are going to be tested which are
innovative, (many elements being used are still prototypes), and there are as yet few quantified
results from their deployment in real-life (or close to real-life) road conditions. Such results are
important as they provide potential future users with concrete information and can hence help to
promote deployment.

Other aspect to be highlighted is that the measurement of the CO2 (and energy-efficiency) impacts
of ITS is a topic of great interest as there is no recognised standard methodology. Among other
initiatives, it is being studied by the project ECOSTAND, which is seeking to define a roadmap to
arrive at a common approach in Europe, Japan and the United States. PROJECT intends to provide
input to this project where appropriate.

Following a reasonable development approach for pilot projects, as PROJECT is, validation is the
last step of the system development process. In order to distinguish and separate clearly validation
in PROJECT from other processes as verification, the following differences are stated:

 Verification or “did we build the system right?” determines if the system is consistent
and performs the selected functions in the correct manner. Using a bottom up
approach verification ensures whether the requirements of functional specifications
at the higher levels are fulfilled after adaptation and integration of applications in
the corresponding pilot sites.

 Validation, or “did we build the right system?” is determining if PROJECT

applications as well as the pilot site system level and the overall system comply with
the high level objectives and performs functions for which it is intended, i.e. meeting
the high level objectives and impacts. Validation checks, using a top down approach,

5/14/2019 Page 9 of 113 Version 1.4

 Validation Test Plan

the performance and effectiveness based on performance indicators and the impact
assessment based on the success criteria.

Figure 1. Validation approach

1.3. Structure of the document

The main body of this validation test plan is divided into 3 further sections as follows:

 Section 2 defines the PROJECT Validation Methodology, presenting the in order to define
the targeted criteria, the expected impacts of the project, the assessment methods and
how define properly the study design

 Section 3 describes the Pilot Sites and the validation criteria for each site as well as the
validation test cases.

 Finally, section 4, describes the legal and organizational aspects that should be taken into
account.

 At the end of the document, a number of annexes can be found. These annexes provide
core information related to PROJECT validation as targeted criteria, indicators, measures,
sensors and test cases. Additional reference information, including templates, lists of
indicators, and questionnaires.

1.4. Related documents

The Validation Test Plan takes into account the following PROJECT documents:
 Part B of PROJECT proposal (DoW)

 D2.1 Italian Pilot Site Design and Planning

5/14/2019 Page 10 of 113 Version 1.4

 Validation Test Plan

 D3.1 Austrian Pilot Site Design and Planning

 D4.1 Swedish Pilot Site Design and Planning

 Excel sheet “Project_Validation_Matrix.xls”_

1.5. Terminology and abbreviations

 Application: An experimental ITS device or system tested in this project in a real-life

environment and which effects are going to be assessed in combination with other
applications, in order to be validated according to the project objectives.

 Assessment: The process of determining the performance and/or impacts of an application

or group of applications, uusually in comparison to a reference case In ITS Pilots as
PROJECT, assessment comprises an experimental process based on real-life traffic
conditions and often involving users.

 DoW: Description of Work

 Evaluation: The process of determining the value of an application in comparison to

alternative applications and/or to a reference case, and deriving recommendations for
decision makers based on identifying requirements on and analysing results of related
experiments.

 Hypothesis (HY): The hypothesis is an answer to a RQ that contains a specific and testable
prediction about the relationship between two variables.

 Impact: Changes or effects brought about by an application as a result of its

implementation in an experimental or ‘real-life’ situation.

 Impact assessment: Measurement or estimation of the impacts (effects) of an application,

for the appraisal groups, i.e. group of people affected by the impacts’
(benefits/inconveniences) application.

 Overall system: It comprises all the PROJECT applications considered as a whole system.
Overall system is a concept used when evaluating the global impact of the project.

 Performance indicator (PI): Quantitative or qualitative measurement agreed on beforehand

expressed as a percentage, index, rate or other value which is monitored at
regular/irregular intervals and can be compared with one or more success criteria.

 PS: Pilot Site

 Reference case: Also called baseline, it´s a type of targeted criteria. Normally the
reference case represents an existing situation, against which the same situation, including
a number of experimental applications, is compared as a method to assess in terms of

5/14/2019 Page 11 of 113 Version 1.4

 Validation Test Plan

validation the effects that these experimental applications cause in a determined test
environment.

 Research question (RQ): Is a statement of what the researcher wants to discover or prove.

 Service: The also called “cooperative mobility services” are ITS applications involving V2V
and V2I communications. In PROJECT project the term “service” is used at higher level
stages (general objectives, impacts, etc.) and “application” at more operative levels. Each
PROJECT service is based, in general, on one PROJECT application

 Targeted criteria (TC): A targeted criteria establishes a quantitative or qualitative

threshold according to which a hypothesis can be evaluated.

 Validation: Did we build the right system? determin PROJECT applications as well as the
overall system compl with the requirements and perform functions for which intended and
meet the high level objectives. Validation us a top down approach the performance and
effectiveness based on performance indicators to be defined in the validation plan.

 VM: Validation Matrix

5/14/2019 Page 12 of 113 Version 1.4

 Validation Test Plan

2. METHODOLOGY
The process adopted in the PROJECT Validation Test Plan is based on a standard framework which is
harmonised within the three pilot sites. It consist of a well-known and frequently used methodology
for the development and realisation of ICT systems referred to as the V-model [1], [2].

One of the most important benefits of using the V-model is that the validation activities are
identified and specified from the start [3]. It also ensures a direct connection between the targeted
criteria, the definition and operation of the tests, and the assessment of the impact.

In this chapter, firstly PROJECT validation objectives and categories are established according to
background information. Secondly, an overview of the application for validation of the V-model and
then a more detailed description of all the comprised stages, is given.

2.1. Validation fundamentals

2.1.1. Validation objectives

The PROJECT validation objectives have been extracted from the PROJECT expected impacts and
general project objectives and also from the Pilot Sites expected results. All of them can be found
as background information in the DoW. A total coherence among the different levels of information
can be found since expected impacts and general objectives establish a framework under which
Pilot sites expected results give specific information on the validation objectives of the
applications, being here grouped and homogenized under main PROJECT validation categories (see
Error! Reference source not found.).

PROJECT Expected Impacts and objectives: they match what requested in the ICT PSP Call4 –
2010, Theme 1 (low carbon economy and smart mobility), Objective 1.3 (Energy efficient co-
operative transport management systems):

1. “Contribute to the uptake of European innovative ICT based mobility services for
sustainable and energy efficient transport systems”: by producing comprehensive specifications
for PROJECT cooperative mobility services specifications with the purpose of offering detailed
and quantified information (technical, legal and organisational aspects) which will stimulate
their deployment through Europe. This information witll be obtained through demonstration of
their funcionalities under realistic conditions.

2. “Improve readiness of the Member States for investments in upgrading their infrastructures
(in particular communications and sensor networks) in support of mobility”: by integrating
cooperative applications for energy efficiency in the key technology areas indicated in the call.

Pilot Sites expected results:

Energy efficiency
 Saving in fuel consumption due to PROJECT services and its quantification.
5/14/2019 Page 13 of 113 Version 1.4
 Validation Test Plan

 Reduction in electric energy consumption due to PROJECT services and its quantification.

Environment
 Reduction in CO2 emissions.

Mobility
 Smoothing of traffic flows.
 Increased, harmonized and smoother traffic flows with less queues and congestions.
 Incresed number of passengers (bus).

User acceptance
 To obtain user acceptability indexes.

Driver behaviour
 Change in driver behaviour avoiding suddenly changes (e.g. hard braking).

Cooperativeness
 Information regarding the deployability of the tested systems.
 Identification of the opportunities derived from the integration of different applications.
 To obtain indications regarding the transferability of the systems to other areas (urban,
interurban).

2.1.2. Validation categories

In the following sections, the categories taken into account during the targeted criteria generation
process, and used to classify the PIs are presented. These categories are: environmental, mobility,
safety, and human-related factors, such as user acceptance and driver behaviour categories.

2.1.2.1. Environmental

Within the environmental category, different aspects should be considered. On one hand, this
category covers the indicators relating to the achievement of a carbon-free society, that is to say,
PIs related to vehicle emissions (including greenhouse gases) and fuel consumption.

On the other hand, it is increasingly recognised in the environmental field that to obtain a realistic
view of the impact of a given instrument or strategy, a global assessment needs to be made. This
means that to assess the environmental impact of an ITS strategy, it is necessary to consider not
only the effect on fuel consumption or CO2 emissions of vehicles but of the energy consumption of
the equipment itself. In two of the Pilot Sites, PROJECT will therefore also consider the energy used
to operate roadside equipment, the product lifecycle and energy harvesting.

2.1.2.2. Mobility

Efficient movement of people and goods - can be said to consider two levels: network and individual.

5/14/2019 Page 14 of 113 Version 1.4

 Validation Test Plan

 The individual level deals with users mobility, i.e. travel time, delay, number of stops or
speed or also a “multimodality index” related to the use in PROJECT of the shuttle service
in the Italian Pilot Site.

 Mobility seen from the network perspective studies the impacts on the network level,
considering indicators such as traffic density (#vehicle/km), traffic volume (#vehicle/h)
and total route/network journey time and delay. These criteria often are different for
diverse parts of the road network. For main roads capacity is an important criterion, while
for small urban roads, capacity and travel time generally are not important.

2.1.2.3. Safety

Regarding the safety category, during technical testing, safety measurements such as the number of
accidents, or crash rates, are not directly measurable. Nevertheless, there are surrogate safety
measures that are measurable or can be estimated. For instance the safe headway distance (and
predicted crashes) or time to collision (TTC). Unfortunately, due to the nature of the tests to be
carried out in the PROJECT Pilot Sites, it is not possible to collect these kind of measurements.
However, the safety impact could be evaluated in other ways.
It is recognised that as well as improving traffic efficiency, cooperative mobility services are able to
increase road safety. Consequently, although safety is not the main focus of the PROJECT project, it
the services will have indirect influence on safety, thanks to other improvements, for instance, as
stated above, in mobility (smoother traffic flows).

2.1.2.4. Human Factors indicators: user acceptance and driver behaviour

The measurement of the impact of PROJECT services on drivers, other road users, and end-
users,needs to be carefully considered in the validation, as human reactions are totally different
from the technical aspects.

When PIs related to human factors are defined, it should be taken into account that many of these
can be easily measured with a structured interview and/or questionnaires. In general, indicators
relating to user acceptance such as satisfaction (this describes the level of ‘comfort’ when using the
system) or usefulness (offering information about the enthusiasm of people using the system for the
support that it provides).

However, depending on the application to be evaluated, it may also be useful to define further
performance indicators that measure behavioural aspects, such as the reaction time of the drivers,
their behaviour in terms of deceleration, acceleration, etc.

2.2. At a glance

The ‘route’ of a V-model starts at the top left-hand side of the ‘V’, passes the bottom and then
continues up the right-hand side. As shown in next figure, in PROJECT the validation is divided into
three stages: definition, operation and impact assessment.

5/14/2019 Page 15 of 113 Version 1.4

 Validation Test Plan

Figure 2. PROJECT Validation V-model. Validation methodology at a glance

Definition stage

 As explained in previous section, validation objectives are based in PROJECT expected

impact and project objectives, being background information.

 The Pilot Sites definition is also background information (DoW, D2.1, D3.1 and D4.1). From
the validation point of view, information is needed from Pilot Sites in order to match
validation objectives (top down approach) with the applications, measures/sensors and
other resources available in each site (bottom-up approach).

 The following step is to define and fill in a Validation Matrix (VM). The VM is a two-
dimensional matrix which contains:
 The identification of the targeted criteria (TC). When the targeted criteria are
known, the performance indicators (PI) related to each TC can be defined.
 Consequently, the second phase is the definition of the PIs and the required measures
(ME) needed to evaluate these PIs.
 The definition of the sensors (SE) required to acquire measures is then needed. While
the sensor table should be completed as well as possible during this phase, it can be
updated at a later stage. Moreover, the SE table should be used in the preparation of,
during, and after the project (since the set of three operational pilot sites is
expected to continue in use after PROJECT has ended, according to final Business
Plans), together with the PI and ME tables.

 The last step during the definition stage is the description of the required validation tests
in each Pilot Site. This step implies the need to:

 Identify the baseline or reference case, i.e. the existing situation.

5/14/2019 Page 16 of 113 Version 1.4

 Validation Test Plan

 Identify the assessment methods (quantitative, qualitative).

 Identify resources and timeline.
 Characterise the validation test sites.

Operational stage

During this phase, the measurements required to evaluate the PIs are acquired, logged and post-
processed. In order to carry out the operational stage correctly, it is important to plan the timing of
measurement of performance indicators for the trial application and the reference application in
such a way that the measurement plan does not itself contribute to bias in the comparison of the
performance indicators of the PROJECT and the baseline. The details of how it is intended to do this
in each Pilot Site are reported in D2.1 Italian Pilot Site Design and planning , D3.1 Austrian Pilot
Site Design and planning and/or D4.1 Swedish Pilot Site Design and planning.

Impact assessment stage

At the end of the validation process, the impact assessment will be done. This stage is divided into
two parts.
 First, the evaluation of the targeted criteria, i.e. check if the criteria established during the
definition stage has been achieved, taking into account the results obtained during the
second phase of the validation, that is to say the operation stage.
 The second step within the validation stage is the impact appraisal. It is a process that
prepares evidence for the policy decision-makers and/or the stakeholders on the advantages
and disadvantages of PROJECT services by assessing their potential impacts, with particular
emphasis on energy-efficiency, safety implications and environmental impact.

Furthermore, the assessment should verify the expected impacts of the PROJECT project. In fact,
the tests cannot be properly designed without a good idea of the nature and scale of likely impacts.
The following section presents the recommendations to define these PROJECT expected impacts.

2.3. Definition stage

5/14/2019 Page 17 of 113 Version 1.4

 Validation Test Plan

Figure 3. Validation methodology: Definition stage

The top-down approach for determining targeted criteria has to match with the bottom-up
approach resulting from the analysis of project resources, technologies and constraints. The
following tasks are necessary in order to bring them together:

 Adjust validation objectives, according to project resources available and timing

 Hypotheses and success criteria definition. Relevant research objectives are the basis for
Research Questions (RQ) formulation. In PROJECT, the main research challenges can be
found in the DoW, under the S&T objectives section (see [DoW]). RQ lead to hypotheses
and success criteria formulation. Success criteria establish the thresholds for performance
indicators.

 Performance indicators definition, which connect validation objectives with available

sensors and measures.

 Scenarios definition, which connect validation objectives, situations and applications for
testing at the pilot sites.

A diagram explaining the process for targeted criteria generation is presented in Figure 4.
Formulation of adequate research questions is essential to focus validation according to the
general objectives of the project.

Targeted criteria generation is a top-down process that moves from high level inputs such as
services/applications and project research challenges, to specific criteria and indicators that allow
validation.

5/14/2019 Page 18 of 113 Version 1.4

 Validation Test Plan

Figure 4. Targeted criteria generation process

2.3.1. Research Questions

Definition:

 A Research Question (RQ) is a statement of what the researcher wants to discover or

prove.

Best practices:
 The RQ should be not too broad or not too narrow. In this case each RQ will be addressed
to a service/application, stated objective and/or a scenario.

 There will be at least one research question per application and scenario.

It is useful to have a reference form for the RQ since this leads more easily to the specific
level of the questions. A proposed syntaxe could be: Will <something> produce <effect> on
<something else>?. Of course, other syntaxes can be used, but the simpler they are, the
more care must be taken to keep them neither too broad, nor too narrow.

 Other forms can be adopted. One could also ask, for instance, what costs are involved or
what stakeholders are involved.

Examples:

 For traffic efficiency category: Do the Enhanced Bus Intersection Logistics and the
Congestion Prevention applications reduce the number of stops made by buses?

 For environmental category: Does the Eco-Dynamic Access Management contribute to

reduction in the CO2 emissions in the defined zone?

5/14/2019 Page 19 of 113 Version 1.4

 Validation Test Plan

2.3.2. Hypotheses

Definition:
 The hypothesis (HY) is an answer to a RQ that contains a specific and testable prediction
about the relationship between two variables.

 Best practices:

 The indicators used should be feasible to obtain (in terms of what to measure, at
reasonable costs, etc...).

 A well thought out and focused RQ leads directly to the HY.

 For proving the hypothesis, a related Performance Indicator is needed. Thus it is

recommended to define the performance indicator(s) directly when defining the
hypothesis.

 The hypothesis is tested by the targeted criteria (TC), which examine whether the
hypothesis is valid or not.

Examples:

 For traffic efficiency category: The Enhanced Bus Intersection Logistics application
together with the Congestion Prevention application will reduce the number of stops.

 For environmental category: The Eco-Dynamic Access Management will contribute to

reduction in the CO2 emissions in the defined zone.

2.3.3. Targeted Criteria

Definition:

 A targeted criteria (TC) establishes a quantitative or qualitative threshold according to

which a hypothesis can be evaluated.

Best practices:

 Within this project three possible types of comparison can be made:

 Comparison to baseline: The performance indicator of systems equipped with the

application/s to be validated is compared to the performance indicator of systems
without these applications.
 Comparison between times: Used for the measurement of long term effects (e.g.,
user learning effects). The performance indicator is measured again with the same
setup of the system after a certain time has elapsed.
 Comparison to value: The performance indicator of the examined applications has to
reach a certain value (e.g. <=, >=, etc.), which is reasonably defined in previous
work.
5/14/2019 Page 20 of 113 Version 1.4
 Validation Test Plan

 The comparison to a previously defined value should be avoided and only be pursued if the
value is defined reasonably, i.e. supported by a PROJECT objective or the literature.

Examples:

 For traffic efficiency category: The rate of stops reduction estimated for the corresponding
previous hypothesis is > 10%.

 For environmental category: The CO2 emissions reduction estimated for the corresponding
previous hypothesis is > 15%.

2.3.4. Performance Indicators

Definition:

 Quantitative or qualitative measurement, agred on beforehand, expressed as a

percentage, index, rate or other value which is monitored at regular or irregular intervals
and can be compared with one or more criteria [4]. The PI can be derived from
measurements that can be direct or derived, from in-vehicle data or from external sensors,
questionnaires or events (e.g. number of stops at traffic lights).

Best practices:

 During the process of developing hypotheses and targeted criteria, it is important to

choose appropriate Performance Indicators (PIs) that will allow the hypotheses to be
tested effectively, but also within the budget and taking into account other limitations of
the project [4]. In addition, PIs are used for measuring how far the objectives will be
achieved by the application.

 Two basic requirements have to be taken into account when defining indicators: they must
be able to reflect clearly the related performance or impact; and they must be capable of
reliable assessment using the experimental tools and measurement methods chosen.

 Furthermore, in the case of a cross-site collaboration, it is necessary to define a set of

common PIs and to determine those indicators in a comparable way at different sites to
enable a generalisation of the results provided by the individual sites.

 Moreover, the calculation of performance indicators has to consider external influences,

which are described in the situational variables.

 Each performance indicator should be derived from the set of measures that is possible to
obtain according to sensors available in the pilot sites.

Example:

 For traffic efficiency category: Number of stops [absolute number or percentage].

5/14/2019 Page 21 of 113 Version 1.4

 Validation Test Plan

 For environmental category: CO2 emissions [g/km]

2.3.5. Experimental design

The experimental design in such a project as PROJECT, a pilot, begins with the definition and
selection of the suitable indicators according to previous tasks defining RQ, HY and TC and also
considering the resources available. Since PROJECT project is almost 100% real testing, specific test
methods and good quality mesurements are critical.

2.3.5.1. Definition and selection of indicators

As stated previously, the performance indicators are used for evaluating the achievements in the
application level and, as well, for estimating the success or impact of the project at a global level.
Accordingly, in order to select suitable indictors, it must be taken into account that they have to
reflect the performance or impact of the application and to deliver reliable results for the
measurement methods and tools chosen. As well, in the case of cross-site or cross-project
collaboration, it is indispensable to define common indicators and measurement methods, in order
to be able to compare results.

2.3.5.2. Tests methods

The assessment of applications can be carried out in several ways. As stated previously, typical
assessment categories are the Technical assessment, Impact assessment, User acceptance
assessment, Socio-economic evaluation, Financial assessment and Market assessment.

This section describes possible techniques for performing the assessment of the applications. These
methods represent a group of quite heterogeneous elements and characteristics. For instance, a
‘questionnaire’ may be regarded as an individual method for evaluation, but a questionnaire may be
also used in combination with test drives. In this case, the test drive creates the situation to be
evaluated and the questionnaire is a subset of the whole test.

In the following, the most common assessment methods are listed and briefly described [4], [5], [6].

2.3.5.2.1. Field trials

The field trial definition implies the testing of applications under real conditions, in order to
identify and evaluate the technical and/or non-technical benefits of these applications prior to
marketing. Within the PROJECT project, the field trials will be performed on Pilot Sites which
include public roads that represent typical driving environments: a highway (Austrian Pilot Site), an
urban road network (Swedish Pilot Site) and an specific geographic area, i.e. the campus of a
university (Italian Pilot Site). The activities carried out in the field trial will be observation (how the
systems and/or drivers react), interviews (does the system work sufficiently well in the real driving
context), measurements (is the signal strength sufficient to broadcast the messages), test drives
(good reception of message in moving vehicles) etc.

5/14/2019 Page 22 of 113 Version 1.4

 Validation Test Plan

2.3.5.2.2. Driving tests

Within PROJECT, test drives may be used for the assessment of the functionality of the applications,
the impact on the end-users or on the traffic situation. Regarding the assessment of the
functionality of applications implemented in the three Pilot Sites, as example can be the on-board
information provided to drivers in order to achieve more ‘ecological driving’.

Furthermore, with regard to the assessment of the impact on the end-users (i.e. drivers in all three
Pilot Sites, but also the highway operator, in the case of the Austrian Pilot Site and bus passengers
in the Swedish Pilot Site), an example could be the behaviour of drivers when ‘eco-information’ is
received, students’ opinions about the improvement in ease of parking in the campus, and so on.

Finally, taking into account the assessment of the impact on the traffic situation, an example from
Austrian Pilot Site can be the reduction of fuel consumption thanks to a smoother traffic flow with
the consequent an increase of safety due to avoidance of accidents caused by sudden braking.

The main advantage of test drives is the direct connection with the real life context. However, at
the same time, this advantage is the main disadvantage. As stated previously, the repeatability of
the measures is a requirement in order to guarantee the integrity of the performance indicators and
impacts. Nevertheless, every real driving situation is unique (the situational variables as traffic
situation, other road users involved, weather, daylight, etc. may change); theoretically only a high
number of samples allows typical situations to be classified or influencing values to be defined as
insignificant. The limited testing undertaken in research projects means that efforts have to be
made in the planning phase to identify the most relevant influencing conditions liely to affect test
drives in order to create typical test environments in order to measure the impacts of the
application.

2.3.5.2.3. Subjective assessment methods: interviews/questionnaires

Drivers, other road users, end-users, etc. should be carefully considered within PROJECT Validation,
as working with human beings is totally different from working with a technical systems (see
chapter 4. “Legal and organisational aspec ts”).

The use of interviews/questionnaires is suitable for collecting systematic information on personal

opinions, knowledge and behaviour. For the performance of tests a wide range of standardised tools
is available. The use of these pre-defined questions, answers or scales simplifies the analysis of
results as well as facilitating their comparability.

Depending upon the research questions and hypotheses underlying the tests, there is often a need
to select a particular group of participants as representative of a determined group of users.
Consequently, the Pilot Leaders are recommended to define (and report) a list of criteria for
selecting the participants for their tests. The following table contains the main parameters for
categories and profiles of participants.

Categories  Experts on driving assistance systems evaluation.

 Experts on human factors evaluation.
 Normal users and potential clients with characteristically little pre-knowledge

5/14/2019 Page 23 of 113 Version 1.4

 Validation Test Plan

about the kind of system tested.

Profile  Demographic variables, such as age, gender, country of origin, socio-economic

variables, and permanent or temporary driver impairments.
 Driving experience (km per year), specific experience with the systems under
evaluation, accident history and the usual time of driving and roads used.
 Experience with human factors evaluation.
 Personality and attitudes.

Table 1. Categories and profiles of participants

This section contains two examples of recommended questionnaires and measuring methods for the
standardized measurement of the main human factor indicators: usability and acceptance.

2.3.5.2.3.1. Usability

Usability is defined as ‘the extent to which a product can be used by specified users to achieve
specified goals with effectiveness, efficiency and satisfaction in a specified context of use’ (ISO
9241-11, 1998) [7], where effectiveness is defined as, accuracy and completeness with which users
achieve specified goals, efficiency as, resources expended in relation to the accuracy and
completeness with which users achieve goals, and finally, satisfaction as, freedom from discomfort
and positive attitudes towards the use of the product.

The proposed method for assessing the usability is the System Usability Scale (SUS) [8], which
provides a reliable, low-cost tool that can be used for global assessments of systems’ usability. The
SUS is applied after a user has used a system, but before any discussion and debriefing. Subjects are
asked to respond immediately, rather than thinking for long. The corresponding table in presents
the System Usability Scale.

2.3.5.2.3.2. Acceptance

Acceptance of driver support systems are of crucial importance. Acceptance will determine whether
such systems will actually be used.

However, the term ‘acceptance’ is a rather wide concept. Acceptance of driving support systems,
can be divided into three categories:

 The utility and usefulness of the system from the driver’s point of view. This category
offers information about the helpfulness of the driver towards the support that the system
is providing.

 The usability of the system and satisfaction of the driver with it. This category describes
the comfort and convenience of using the system.
 The reliability of the system and trust (subjective) in the system by the driver relates to
the amount of trust that the driver has in the system.

Moreover, in addition to the specific operating characteristics of the system and the characteristics
of the drivers, the acceptance of the system also depends on the way it has been evaluated. For
5/14/2019 Page 24 of 113 Version 1.4
 Validation Test Plan

example, a system could be evaluated by describing it and interviewing drivers, evaluating it by

simulator experiments or by field operational tests. In each case the impact is likely to be different.

In order to quantify the total ‘amount’ or level of acceptance, the Van der Laan scale [9] could be
used as a basis and extended with the inclusion of reliability and trust in the system.

The technique used is simple and consists of nine items each rated on a 5-point scale. These items
can be weighted on two scales, one denoting the usefulness of the system and the other designating
satisfaction. The following table shows the nine items in the questionnaire.

Finally, it is recommended that the selected questionnaires and methods are used, and further
questions and measuring methods added where applicable in order to give answers to the testing
hypothesis.

2.3.5.2.4. Observation

This method is based on pure observation, which can be done in an objective or a subjective way,
or both. In general, this kind of assessment is carried out together with another method, mainly
with test drives. An example performed in-vehicle is the 'Wiener Fahrprobe’ [10]. This observation
method involves two observers in the vehicle watching how the driver behaves in real traffic. One
of the observers registers objectives variables such as speed adaptation at junctions or obstacles,
lane changes etc. The other observer makes free observations, e.g. conflicts, communication,
interaction and special events.

2.3.5.3. Measures and sensors

In order to obtain the resulting value of a certain performance indicator, usually a number of
measures are required to derive the PI value from. There is a need to make a distinction at this
point between: measure and sensor.

 A measure can be logged directly from a sensor, read from a simulation or derived from
other measures. A measure is not a rate.

 Sensors indicate how measures will be collected. They can be independent elements or
part of system hardware or also an internal procedure within simulation software.

2.3.5.3.1. Categories

Measures can have different categories and the integrity of the measurement during the operational
process should always be guaranteed. For the first aspect on measure categories, the FESTA
handbook makes the following categorization:

 Direct (raw) measures: a direct measure is logged directly by a sensor, without any
processing before saving the data to the log file. Linear transformations like the conversion
from m/s to Km/h are not considered to be processing. During the definition of the type of
measurement, it must be taken into account that depending on the nature of the
acquisition, a measurement may be direct or not. For example, longitudinal acceleration

5/14/2019 Page 25 of 113 Version 1.4

 Validation Test Plan

is a direct measure if logged directly from an accelerometer, but not if derived from
collected speed and time.

 Derived (pre-processed) measurements: a pre-processed measurement is not directly

logged by a sensor, but a variable that has been filtered or one which is a combination of
two or more direct or derived measurements.

 Events: events can be seen as peculiarities based on direct and/or derived measurements.
They can be short in time, like emergency breaking, or extended over a longer period of
time, such as an overtaking manoeuvre. One or several preconditions must be fulfilled for
an event to be classified as such, that is, one or more ‘trigger’ criteria must be exceeded.

 Self-reported measurements: this kind of measurement is obtained from questionnaires,

interviews, rating scales, etc. The measurements related to self-reported PIs could be the
answer to each single question or the checks on the rating scale, while the ‘sensors’ would
be the questionnaires or rating scale themselves.

 Situational variables: a situational variables are indicators that give information on

current external surrounding and external influences during the testing period that may
affect the results. Situational variables are an aid to understand results and can be logged
like direct measures or computed like derived measurements. They can also be self-
reported measurements or events. In any case, all relevant situational variables have to be
measurable continuously. Example: weather or road friction factor.

2.3.5.3.2. Integrity of measures

To guarantee integrity of the measurement during the operational process, the concepts in the
following table must be taken into consideration:

 Completeness: Disregarding significant indicators can very easy occur when the limited
resources may be concentrated on measuring only the indicators regarded as most
important or easiest to measure. Therefore, totality of the measures should be assured and
choices made regarding this should be reported.

 Repeatability and reproducibility: The measurement process should guarantee that

variability of the measurements obtained by one person or process while measuring the
same item repeatedly (repeatability) and the variability of the measurement system
caused by differences in operator behaviour (reproducibility) will be within an admissible
range. It means that the ability of a test or experiment to be accurately reproduced, or
replicated, must be ensured.

 Disturbance of validation: The conditions surrounding (and influencing) the measurements

during tests should be, as far as possible, controlled and homogeneous and ideally they
should be the same to of the ones during the reference cases data collection. But from a
realistic point of view, situational variables almost always change and then as many
critical situational variables as possible should be measured in order to identify and
understand possible deviations due to these surruounding conditions and external
influences. Conditions to be taken into consideration could be the weather conditions, the
traffic situation, the test driver, the type of vehicle, the daylight conditions, and so on. In
5/14/2019 Page 26 of 113 Version 1.4
 Validation Test Plan

order to guarantee the required integrity of measurements, as stated previously,

repeatability of the test is required. As a consequence, when the measurement conditions
and their variations are known, it is recommended to establish the control factors. These
factors are the parameters that have to be controlled during the testing of the hypothesis
to ensure that they do not disturb the comparability.

 Furthermore, there may be accidental or intended bias introduced into the measurement
process by factors such as respondent fatigue - subjects become mentally and/or
physically tired -, policy response bias - subjects wish to influence the results of the
validation -, or justification bias - subjects consciously or subconsciously give answers
which they think are more acceptable to an interviewer. These potential disturbances
introduced during the validation must be considered and special countermeasures taken in
order to avoid this effect.

2.3.6. Test cases definition

This section presents the template to be used for Validation test cases. The main objective of this
template is to ensure a common approach for the definition of the tests within all Pilot Sites
involved in the PROJECT project.

The Test Case template is a Microsoft Excel file, based on the SAFESPOT project templates [6],
which includes:

 One sheet, where a list of the test cases is given, called ‘Test Cases List’.
 Another sheet with the required explanations in all fields, called ‘Explanations.
 Plus, one sheet for every Test Case within each Pilot Site, labelled ‘Test Case.xx’, where x
is a consecutive number.

A detailed description of the structure and contents of this template is included in Annex #6.
However, the following sections present a brief description of the Test Case template.

The ‘Test Case.xx’ template is divided into four sections:

 A Header Section that collects general information such as the relevant task or application,
authors, hardware and software components used, test type, test purpose, etc.

 Test Setup Frame with the description of the test setup and scenario

 Statement of Targeted Criteria (?) to be met

 Report of Non-Compliance (things that did not work as expected), results and conclusions.

2.3.6.1. Header

The following picture shows the header section which includes the general information:

5/14/2019 Page 27 of 113 Version 1.4

 Validation Test Plan

Figure 5. Header section of Test Cases template

The first row provides the following information:

 Site: Italian, Austrian, Swedish
 Compiled by / Company
 Date
 Sheet number: the reference number.

 The format proposed is as follows: PROJECT_TC_<Site>_<Index>, e.g.

PROJECT_TC_IT_1.
 The following nomenclature shall be applied.

 a) SITE: Test Site abbreviation: IT (Italian), AS (Austrian) or SW (Swedish

 b) INDEX: A continuous number for each test case

 Application(s) tested: applications / sub-applications tested in the test case.

 Vehicles / RSU, HW components, SW components
 Test Type: technical evaluation, human factors test, any other category of evaluation
(mobility, safety and/or efficiency)
 Test purpose

2.3.6.2. Test Setup Frame

The test setup frame section shown below includes the following test information.

5/14/2019 Page 28 of 113 Version 1.4

 Validation Test Plan

Figure 6. Test Setup Frame section

 Scenario Definition: describe in detail the test setup and the scenario you are running.
 Scenario Schema: refer to the vehicles / RSU applied, their position and movements and the
use cases.
 Scenario Objective: human factors, how many subjects planned, the experimental design.
 Test Procedure: technical evaluation, how many repetitions.
 Situational variables: e.g. (give some examples)
 Control Factors: e.g. (give some examples)
 Test variations: describe any variations from the original plan.

2.3.6.3. Targeted Criteria

The targeted criteria section includes the hypothesis or hypotheses, and the related targeted
criteria, which will be evaluated during the test.

Figure 7. Targeted criteria section of Test Cases template

2.3.6.4. Results reporting

5/14/2019 Page 29 of 113 Version 1.4

 Validation Test Plan

The last section of the Test Cases template includes space for the reporting of non-compliances, for
summarising the results and providing conclusions about the evaluated test.

Figure 8. Results reporting of Test Cases template

2.4. Operation stage

Figure 9. Validation methodology: Operation stage

The main tasks during this second phase of the validation methodology are data acquisition, logging
and analysis of the required measures in order to evaluate the performance indicators and impacts.

2.4.1. Data acquisition

The measurement process is carried out during the operational stage of the Validation V-model.
Measurements are defined as planned actions whose aim is to obtain quantitative information on a
performance indicator and, in general, are mainly related to the technical assessment, although
they may also refer to other assessment categories. The preparation of the measurement process
needs to be appropriately planned in the design phase (validation matrix).

Methods for data collection are, basically, direct measurements from the field trials or tests drive,
observation of the test or subjective methods based on questionnaires, interviews and rating scales.

5/14/2019 Page 30 of 113 Version 1.4

 Validation Test Plan

Same as in the test methods, there are different data types that are needed in order to support the
hypotheses validation:

 Driver background data: Data obtained from interviews or questionnaires, concerning the
driver. The selection of participants and test drivers for the sites is dependent on the
selected indicators and the hypothesis underlying the tests. Depending upon the research
question, there is often a need to select a particular group of participants and ensure that
this group is, in some way, representative of those users who will, at the end, interact
with the PROJECT the system (professional drivers, human factors experts or even
potential clients).

 Nomadic devices data: Data provided either by the application under test or the
acquisition system.

 Subjective data: Data coming from interviews and questionnaires on satisfaction or use of
the system tests.

 Real time data: Online data coming from the drivers and systems to be evaluated. This
type of observation must not be intrusive, in order not to affect natural behaviour of
drivers. In-vehicle data – coming from in-vehicle systems - can also be considered under
this type.

 Data from infrastructure: Special requirements licenses are generally necessary to obtain
these types of data. Contacts with the road authority or a service provider are necessary
to obtain these data.

 Data from specific sensors: Data coming from video cameras or driver monitoring systems
are subjected to issues discussed in Error! Reference source not found..

2.4.2. Data logging

Data logging or data registration must take into account the selection of the technologies to be
used, otherwise problems of interfacing may occur. Furthermore, whether data logging equipment
is implemented in a vehicle or not will influence the operational licence of the vehicle.

In order to evaluate the impact of the applications, it is necessary to estimate the expected effects
and representativeness or statistical confidence of the experiment. Unfortunately, the limited
resources of research projects do not always guarantee representativeness. In such cases, it is
necessary to decide if a statistical approach is possible. If so, it is necessary to link the expected
improvement with a level of statistical confidence. The sample used should be as large as possible.
Although the statistical approach may be desirable, it is not always possible, especially when user
acceptance is being evaluated with a small sample (e.g. a limited number of vehicles). This is
common for operators of Traffic Control Centres, and must be taken into account in SP5
applications, for instance.

5/14/2019 Page 31 of 113 Version 1.4

 Validation Test Plan

The measurement plans refers to the timing of measurement (during the operational stage), i.e. the
collection of the required data for the evaluation of the performance indicators or impacts. This
period includes both:

 The required time to know the baseline, i.e., how long is the before measurement in
order to obtain a representative set of data, unless it is based on literature, historical or
statistical data.

 The actual development of the test itself

Reference case or baseline:

The reference case or baseline refers to the existing situation before implementing the PROJECT
project applications in the validation location. The reference case is used when there is the need to
know if the developed applications have the effect of improving or worsening the existing situation,
i.e. the baseline.

The reference case is required in particular for the ”before-after analysis” approach. It would also
be needed for an alternating with-without system test is done. The technical assessment of
standardised components principally represents an uncritical category of assessment, while for user
acceptance the same baseline is usually used over a number of assessment objectives (impact
analysis and socio-economic evaluation).

Finally, when applications are tested only in one validation location, this has the advantage that the
same study environment and infrastructure can be used. However, validating the new application to
a long-term time horizon may introduce differences in relative performance. Nevertheless, when
applications are tested on more than one site it is recommended to define the reference case as
clearly as possible to facilitate the comparison of results. It is essential then to identify and clearly
define the reference cases needed.

This section will be further elaborated in version 2 of the validation plan and will then describe the
guidelines for developing a baseline scenario.

2.4.3. Data Analysis

The data analysis is mostly related to the technical assessment. Information collected and stored
during the ‘Data acquisition’ and the ‘Data logging’ phases within operational stage, is analysed to
conclude this validation stage, e.g. functions or reactions. By definition the data analysis is a very
exact method of evaluation, however, data analysis is not an automatic task limited to some
calculations algorithms. It is the place where logged data is prepared to confront the hypothesis,
data and methods. For instance, typical risks of error regarding time response originate from wrong
synchronisations (e.g. different reference times, latencies of systems) or comparison of ‘equal’ data
that has been processed in a different way because of the lack of a common definition (e.g. to use
the mean or median value). Therefore, the strategy of data analysis need to be planned in order to
provide an overall assessment of the impact of a system from the experimental data.

5/14/2019 Page 32 of 113 Version 1.4

 Validation Test Plan

In order to evaluate the impact of the applications, it is necessary to estimate the expected effects
and representativeness or statistical confidence of the experiment. Unfortunately, the limited
resources of research projects do not always guarantee representativeness. In such cases, it is
necessary to decide if a statistical approach is possible. If so, it is necessary to link the expected
improvement with a level of statistical confidence. The sample used should be as large as possible.
Although the statistical approach may be desirable, it is not always possible, e.g. when user
acceptance is being evaluated with a limited number of people. This is also common for operators
of Traffic Control Centres, and must be taken into account in the Austrian Pilot Site for example.

Before – after analysis:

The before-after analysis is the typical method of evaluation for the PROJECT applications. The
situation before (i.e. without the PROJECT application) will be compared to the situation after
implementing the PROJECT application.

During the first phase of the evaluation stage, a ‘before’ analysis will be made in order to provide
the reference case or baseline results. This will then permit the evaluation of the targeted criteria.
through comparison with the situation when the application/s are operational.

The relevant situations for testing the applications therefore have to be defined as well as the
parameters and indicators for assessing the impacts. It is important to ensure that the surrounding
conditions remain constant during the tests so that to any changes observed result from the
implementation of the application and not other factors.

2.5. Impact assessment stage

Figure 10. Validation methodology: Impact Assessment stage

The main objectives of the impact assessment within the PROJECT project are to:

 Carry out the impact assessment, i.e. provide evidence for the policy decision-makers
and/or the stakeholders, on the advantages and disadvantages of PROJECT services by

5/14/2019 Page 33 of 113 Version 1.4

 Validation Test Plan

assessing their potential impacts, with particular emphasis on energy-efficiency, safety

implications and environmental impact.

 Provide content for the ‘validated specifications’ report, which must include detailed and,
where possible, quantified results of the testing and validation activities in all three Pilot
Sites.

In order to carry out the impact assessment [13] there are six major analytical steps:

 Identifying the problem:

 Describe the nature and extent of the problem.

 Identify the key players/affected populations.
 Establish the drivers and underlying causes.
 Is the problem in the European Union's remit to act? Does it pass the necessity and
value added test?
 Develop a clear baseline scenario, including, where necessary, sensitivity analysis and
risk assessment.

 Define the objectives:

 Set objectives that correspond to the problem and its root causes.
 Establish objectives at a number of levels, going from general to specific/
operational.
 Ensure that the objectives are coherent with existing EU policies and strategies.

 Develop main policy options:

 Identify policy options, where appropriate distinguishing between options for content
and options for delivery mechanisms (regulatory/non-regulatory approaches).
 Check the proportionality principle.
 Begin to narrow the range through screening for technical and other constraints, and
measuring against criteria of effectiveness, efficiency and coherence.
 Draw-up a shortlist of potentially valid options for further analysis.

 Analyse the impacts of the options:

 Identify (direct and indirect) economic, social and environmental impacts and how
they occur (causality).
 Identify who is affected (including those outside the EU) and in what way.
 Assess the impacts against the baseline in qualitative, quantitative and monetary
terms. If quantification is not possible explain why.
 Identify and assess administrative burden/simplification benefits (or provide a
justification if this is not done).
 Consider the risks and uncertainties in the policy choices, including obstacles to
transposition/compliance.

 Compare the options:

5/14/2019 Page 34 of 113 Version 1.4

 Validation Test Plan

 Weigh-up the positive and negative impacts for each option on the basis of criteria
clearly linked to the objectives.
 Where feasible, display aggregated and disaggregated results.
 Present comparisons between options by categories of impacts or affected
stakeholder.
 Identify, where possible and appropriate, a preferred option.

 Outline policy monitoring and evaluation:

 Identify core progress indicators for the key objectives of the possible intervention.
 Provide a broad outline of possible monitoring and evaluation arrangements.

In the latter stage of the project, the results of the testing and validation activities in all three Pilot
Sites (reported in the deliverables D2.2, D3.2 and D4.2) will be brought together in a single public
document D5.3 “Validated specifications”. It will include:

 Impact and benefits of services in terms of: Safety Impacts, Mobility Impacts,
Environmental Benefits, Social impacts, i.e. evaluation from end-users point of view
(see Table 2. Objectives for Impact Assessment).
 The performance and potential capability of the ICT systems tested.
 Information regarding the product maturity for deployment, providing perspectives
on the market introduction,
 Legal and ethical issues.
 Technical Datasheets of services (diagrams, optimum setups, installation, operation,
etc.).
In the table below it would be better to put the environmental benefits at the top of the list as they
are the main focus of PROJECT.

IMPACT AREA OBJECTIVES

• Determine if drivers drive more safely
Assess Safety Impacts • Estimate crash reductions
• Determine if the implemented ICT system affects other (secondary) crashes
• Assess the direct effect on traffic flow
Assess Mobility Impacts
• Assess the indirect effect of reduced crashes on mobility
Assess Environmental • Assess effect of green driving
Benefits • Assess effect of energy efficiency
• Determine usability of a ICT system
Assess User Acceptance
• Determine if drivers perceive increased stress/workload
& Human Factors
• Determine perceptions of product quality, maturity, etc.
Assess a ICT system’s • Characterize performance/functionality of components
Performance and • Assess capability of components
potential capabilities • Determine reliability and maintainability of components

5/14/2019 Page 35 of 113 Version 1.4

 Validation Test Plan

• Determine if costs are reasonable

• Assess infrastructure investment needs
Asses Product Maturity
• Determine availability of manufacturing capabilities
for deployment
• Assess need for modifications to ITS standards
• Determine if a ICT system is suitable for widespread deployment
Address Institutional
• Identify and determine impact of institutional and legal issues
and Legal Issues

Table 2. Objectives for Impact Assessment

5/14/2019 Page 36 of 113 Version 1.4

 Validation Test Plan

3. PILOT SITES AND APPLICATIONS

The principal focus of PROJECT is in the area of environmentally sustainable traffic management.
The project includes experimentation involving applications in the following areas:

 Environmentally sensitive traffic control strategies

 Eco-driving for private vehicle and for public transport
 Multimodal real-time information systems designed to maximise energy efficiency
 Advanced energy efficient technologies integrated in roadside equipment
 Dynamic City Access Management strategy
 Advanced Real-Time Congestion Management System

These applications are installed and will be tested in three Pilot Sites. These are in Southern Europe
(Salerno Italy), Central (Vienna, Austria) and Northern Europe (Gothenburg, Sweden).

In all sites, the aim is to set up the tests which give the opportunity to make all the required
measurements to enable a comprehensive evaluation of the performance of the systems from the
technical point of view, and also an assessment of the practical and organisation requirements.
Although the systems themselves are different in the different sites, as stated previously, all the
sites are adopted the validation methodology explained. Consequently, the results will be
complementary.

In the following sections, the validation environment is shown, as well as the validation criteria per
site, taking into account the defined targeted criteria. Furthermore, a brief description of the three
Pilot Site is given. However, for in-depth information about each site, the reader is referred to the
specific documents: D2.1 Italian Pilot Site Design and planning , D3.1 Austrian Pilot Site Design and
planning and/or D4.1 Swedish Pilot Site Design and planning.

3.1. Validation environment

Three pilot sites are involved in PROJECT tests. These three sites have their own features and
characterization as can be seen in Figure 11.

Figure 11. Pilot sites environment overview

5/14/2019 Page 37 of 113 Version 1.4

 Validation Test Plan

From the validation point of view, two levels will be considered:

 Pilot Site level: Assessment will be performed and conclusions extracted in the ambit of the
characteristics of each pilot site:

 Evaluation on urban traffic for private cars

 Evaluation on urban traffic for public transport
 Evaluation on inter urban traffic for private cars

 PROJECT level: an overall approach for common assessment objectives that can be found in
the three pilot sites is performed at project level,. Energy efficiency, traffic efficiency and
fuel consumption are the main assessment categories involved.

3.2. PS#1: Private vehicles in the urban environment

The information given along this section is related to the Italian Pilot Site. For further information
view D2.1 Italian Pilot Site: Description and Planning document.

3.2.1. Pilot Site brief description

The Italian Pilot Site will implement several applications in the same geographic area; this approach
allows evaluating both the individual and also combined impact of the proposed solutions in terms
of both environmental impact and traveller behaviour. An important feature is the “total quality”
approach which will be adopted in relation to the energy-efficiency. The measurements will take
into account the energy consumed by the roadside equipment as well as the fuel efficiency of the
transport activities. This Site also intends to investigate co-modality aspects relating to private
car/public transport travel choices for passengers.

Figure 12. Italian Pilot Site. Alternative routes between A and B with luminous path

5/14/2019 Page 38 of 113 Version 1.4

 Validation Test Plan

Figure 13. Italian Pilot Site (Parking areas)

3.2.2. Applications to be validated

3.2.2.1. Eco-Traffic Management and Control

A key element of the Eco-traffic Management and Control system is the Traffic-Adapted Luminous
Path application. This involves the installation of a street lighting system with low energy
consumption based on LED technology able to provide variable levels of luminosity and of power
consumption. When traffic flow is low (e.g. non peak times or during the night) or no cars runs
along the monitored path, a lower level of luminosity is sufficient and the streetlights can be
dimmed to save energy.

In the test zone, vehicles can travel from A to B by two alternative routes (see Figure 12). High
efficiency LED-based lights will be installed on both routes and connected to a Traffic Management
Centre (TMC). The adaptive lighting system with built-in intelligence will automatically adjust on
the basis of the level of traffic flows. A wireless sensor network will constantly measure traffic and,
when flows are low, the luminosity of lights on one route will be reduced and drivers recommended
using the other.

For the trial 10 streetlights will be installed (5 for each path). A wireless sensor network monitors
the traffic; the gateways of the sensor network connect the streetlights and the TMC, putting in
communication the three elements. Furthermore two variable message signs will be placed on the
two entrances that will inform the users about the road lighting. An OBU, connected with a short-
range communication device, installed on the equipped vehicles informs the driver.

5/14/2019 Page 39 of 113 Version 1.4

 Validation Test Plan

3.2.2.2. Eco-Transit and Dynamic Parking management.

The Eco-transit and Dynamic Parking management is a pilot application implemented In the Italian
Pilot Site with the aims to improve the sustainability of the mobility in the Campus of the University
of Salerno. The application fills grade of the two existing parking facilities – i.e. the open space car
park in the Campus and the multi-storey park house at the southern entrance, allowing the
implementation of advanced policies of mobility thanks the cooperation of the vehicles approaching
the Campus with the infrastructure. The application cooperates also with the other ones
implemented in the Italian Pilot Site. In fact, the filling grades are determined by the Eco-Dynamic
Access Management application and forwarded to the Traffic Supervisor.

Students who arrive with private cars are offered a parking lot in the southern park without charge.
In order to reduce the walking effort within the campus, a shuttle bus service is offered. Though
most of the students chose the open space parking “in the green” of the campus area as it is closer
to the respective buildings. This leads to a pressure situation in the open space parking, causing loss
of time and unnecessary driving for finding a parking lot. Hence, the “transport policy” of UniSA
may be stated as a desired user’s shift from open space parking to park house parking in order to
reduce traffic, energy, CO2 and pollution within the campus.

In the test phase a group of at least 100 students will be involved: they download a free application
from the web site of PROJECT project running on smart phone that utilizes the filling grade of these
two parking facilities, which comes along with a behavioural recommendation. In particular, this
Eco-transit and Dynamic Parking management function furnishes a graphical indication of the
suggested parking area to use. An algorithm developed in the Eco-transit and Dynamic Parking
management application by Mizar and proposed on PDA by an application developed/personalized
by Geo-Solutions suggests the parking.

3.2.2.3. Eco-Dynamic Access Management

The Eco-dynamic access management simulates an urban area; the access with private cars to this
area is subject to a scheme, which may change according to ecological parameters. Such
parameters may be the weather situation, the pollution forecast (like PM10 level), the congestion
level, etc. Typical schemes are driving bans under certain conditions for certain vehicles or charges
to pay. The “dynamic” in the Eco-dynamic access management refers to the circumstance, that the
scheme may change dynamically and may be set for vehicles individually, e.g. according to the fuel
consumption.

As the test area – the UniSA campus – grants free access to everybody, consequentially neither a
drive ban may be set nor a charge may be raised; instead, incentives for a certain behavioural
change of drivers who follow the local transport policy recommendation shall be granted.

The test scenario covers the following procedure:

 when entering the campus, test users shall be informed about the general situation” in the
campus, giving the reason for the access scheme at present. Under real-life-conditions a

5/14/2019 Page 40 of 113 Version 1.4

 Validation Test Plan

pollutants threshold level exceedance can be expected. In the test case this is measured
through the occupancy rate of the open space parking.

 the test users get information about the incentive in case they behave properly, i.e. use the
multi storey parking. The incentive is given as “credit points” that may be collected and
turned into espresso coffee in the cafeteria. The number of credit points may vary in order
to achieve certain effects, such as increasing point number with increasing filling grade (i.e.
cut demand peak) or decreasing point number with increasing filling grade (i.e. a general
traffic reduction).

The information is fed to the test users via a DSRC link into the vehicle and displayed on the OBU
and on a smart phone respectively. The reason why not a simple VMS is used is the “dynamic” and
the “personalization” of the Eco-dynamic access management; the scheme is adjustable to the
respective vehicle type and, hence, individually set by means of a I2V cooperation/communication.
So consequentially, every driver gets an individual credit point offer. Furthermore, the offer to the
user is documented for the eventual case where the driver claims more credit points than he
obtained.

3.2.2.4. Eco-Driving and Driver Behaviour Change

The ‘driving recommendations’ is one of the tools that can be used by the driver to have a car
energy savings with consequent fuel consumption reduction and less harmful emissions. Within
PROJECT project, the system that will be developed and installed on the vehicles will suggest to the
driver the right manoeuvres to be carried out to obtain saving of fuel.

Based on a study conducted by the Piedmont Region [14], has emerged that about 40% of the total
travel is recurrent and this value increases until 80% as in the morning and evening time slot. This
recurrent trip is easily explained by home to work path and return and the average mileage is about
16 km and the average time is about 30 minutes.

The application that will be developed, will be pick upped the major benefit in this type of journey
trip that it is called ‘routine trip’. The goal of the application is to try to improve the driver's
driving style which reduction of fuel consumption and pollution emission during the recurrent
journey trip as to go and back from home to work.

The application will be based on a new concept: the application advice to driver some behaviours as
for example accelerate gently on slow with advance to have an even and balanced style of driving
especially during the approach of T-junction and roundabout but, with a larger vision, with the
different aspects of road topology recurring during the day home-work trip. The new and unique
aspect that distinguishes it from other similar application is that it is not based on a permanent
digital mapping or downloaded during the trip. The application would be digital map independent.

The EcoDrive++ application is able to learn itself the recurring trip; after a training period, the
application furnishing to the driver some information, in advance, to improve the style of driving
that approach to the changes of the road geometry. The learning period consist on a phase where
the system acquires data vehicle information combined with the positioning information.
Furthermore, the localization of the vehicle is mandatory to reconstruct the recurrent trip and
calculate if the vehicle carries out the memorized trip or a part of it or not. In the figure below

5/14/2019 Page 41 of 113 Version 1.4

 Validation Test Plan

(Figure 14) is shown the logical architecture that highlight the on board module involved in the
EcoDrive++ application.

Figure 14. EcoDrive++ application logical architecture

3.2.3. Validation criteria

In this section, a summary of the targeted criteria and the related performance indicators defined
for the Italian Pilot Site is shown. For further information view annexes #1 to #5.

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
Traffic-Adapted
Luminous Path
The energy
application will
consumption
Reduce energy reduce the energy Energy consumption
reduction is
consumption consumption thanks [kWh]
estimated to be >
to the adjustment of
10%
a street lighting
ENVIRONMENTAL system
The Eco-Driving
application will The fuel consumption
contribute to a fuel reduction thanks to
Reduce fuel Fuel consumption
reduction providing driving
consumption [l/km]
recommendations to recommendations is
the driver during a estimated to be > 5%
journey

5/14/2019 Page 42 of 113 Version 1.4

 Validation Test Plan

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
The Eco-Dynamic
Access Management The CO2 emissions
will contribute to reduction is
CO2 emissions [g/km]
reduce the CO2 estimated to be >
emissions on the 15 %
defined zone
Reduce CO2 The Eco-Driving
emissions application will
contribute to a CO2
The CO2 emissions
emissions reduction
reduction is CO2 emissions [g/km]
providing
estimated to be >5 %
recommendations to
the driver during a
journey
The Eco-Dynamic
Access Management
The travel time
will contribute to Travel time (‘Parking
Traffic efficiency reduction per test
reduce traffic time’) [s]
user is > 5%
congestions and
MOBILITY improve traffic flow
Eco-transit and The rate of drivers
Dynamic Parking who taking the
Increased co- Use of shuttle
management internal shuttle
modality services [percentage]
contribute to services estimated to
increase the mobility be increase > 3%
Mean deceleration;
or mean number of
The Eco-Driving deceleration events
application does not with system < mean
Deceleration [m/s2]
lead to inefficient deceleration; mean
deceleration number of
deceleration events
without system
The Eco-Driving Mean speed with
application does not system is better than
Mean speed [km/h]
DRIVER Driver behaviour lead to inefficient mean speed without
BEHAVIOUR change speed system
The Eco-Transit Mean speed with
application does not system is better than
Mean speed [km/h]
lead to inefficient mean speed without
speed system
Frequency of 1st and
The Eco-Driving 2nd gear < frequency Frequency of 1st,
application does not without system 2nd and 5th gear
lead to inefficient (frequency of 5th usage [absolute,
gear usage gear > without percent]
system)

5/14/2019 Page 43 of 113 Version 1.4

 Validation Test Plan

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
The Eco-Driving
application will
modify the drivers
behaviour (in this
validation activity The numbers of Rate of use = number
only the CRF recommendations instructions followed
vehicles will be followed is estimated [absolute,
involved and, due to to be increased > 10% percentage]
limitation on driving
permission, only staff
will be involved as
test drivers)
The Eco-Transit
application will
The numbers of Rate of use = number
modify the drivers
recommendations instructions followed
behaviour (in this
followed is estimated [absolute,
validation activity
to be increased > 30% percentage]
only 40-50 students
will be involved)
The Eco-Transit The numbers of Navigation
application will recommendations instructions followed
modify the drivers followed is estimated [absolute,
behaviour to be increased > 10% percentage]
The Eco-Driving
Frequency of stops
application does not
with system < Number of stop&go
lead to an increased
frequency of stops [absolute, percent]
number of vehicle
without system
stoppings.
The Eco-Drive Real
System is not switch System switch off
Time system is
off [bool]
accepted
USER
ACCEPTANCE The applications
High user acceptance
implemented at
score in Usefulness
Italian PS are
questionnaire
accepted

Table 3. Hypothesis, targeted criteria and related performance indicators per category

3.2.4. Outline of planned test cases

For further information view Annex #7.

5/14/2019 Page 44 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

3.2.4.1. Validation Test cases list

Figure 15. Italian Pilot Site Validation Test Cases List

5/14/2019 Page 45 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

3.2.4.2. Timing

Table 4. Italian Pilot Site timing

3.2.5. Progress and impact indicators

As stated previously, the main objectives of PROJECT project are:

 demonstrated reduction in fuel consumption through the use of cooperative systems;

 demonstrated reduction in energy consumption through the use of cooperative systems;
 demonstrated positive behaviour change (in relation to eco-driving and route choice) for
achieving environmental objectives;
 a set of detailed specifications as support for the implementation of the cooperative
systems tested.

The project is designed such that measurements of progress against the above mentioned objectives
will be performed at various timed during the project execution. Technical verification will be
carried out to measure the functionality of applications against technical requirements.
Furthermore, detailed quantifiable indicators are included.

The following table summarizes the progress indicators (PRI) and the impact indicators (IMI) of
Italian Pilot Site:

Expected Progress
Relating to which
Method of
Year 1

Year 2

Indicator No. project objective / Indicator Year 3

measurement
expected result?

Integration and Number of UNISA will monitor

adaptation of streetlights the installation
PRI.WP2.1 5 5 -
components in [poles + LED works and report on
Italian Pilot Site head] the progress
Number of
Integration and UNISA will monitor
[Wireless Sensor
adaptation of the installation
PRI.WP2.2 Nodes] + 5+5 5+5 -
components in works and report on
[repeaters]
Italian Pilot Site the progress
installed
Integration and Number of UNISA will monitor
adaptation of [Variable the installation
PRI.WP2.3 - 2 -
components in Message Signs] works and report on
Italian Pilot Site installed the progress
5/14/2019 Page 46 of 113 Version 1.3
 D5.1 COSMO Validation Test Plan

Expected Progress
Relating to which
Method of

Year 1

Year 2

Year 3
Indicator No. project objective / Indicator
measurement
expected result?

Integration and UNISA will monitor

Number of [CCTV
adaptation of the installation
PRI.WP2.4 cameras] 5 5 -
components in works and report on
installed
Italian Pilot Site the progress
Number of [DSRC
Integration and UNISA will monitor
RSUs] for Eco-
adaptation of the installation
PRI.WP2.5 Access - 2 -
components in works and report on
Management
Italian Pilot Site the progress
application
Integration and UNISA will monitor
adaptation of Number of the installation
PRI.WP2.6 - 1 -
components in [Parking VMS] works and report on
Italian Pilot Site the progress
Integration and UNISA will monitor
adaptation of Number of the installation
PRI.WP2.7 1 1 -
components in SAFESPOT [RSUs] works and report on
Italian Pilot Site the progress
Integration and UNISA will monitor
Number of fully
adaptation of the installation
PRI.WP2.8 equipped [CRF 0 2 -
components in works and report on
vehicles]
Italian Pilot Site the progress
Integration of
equipment in
TMC software:
Integration and 1. Streetlights UNISA will monitor
adaptation of 2. VMS the installation
PRI.WP2.9 3/6 6/6 -
components in 3. WSN works and report on
Italian Pilot Site 4. Parking the progress
5. Multimodal
routing
6. RSU/OBU
Navigation
software
1. VMS data
Integration and UNISA will monitor
2. Multimodal
adaptation of the installation
PRI.WP2.10 routing 2/4 4/4 -
components in works and report on
3. Traffic Info
Italian Pilot Site the progress
4. Access
Manageme
nt data
Est.
Number of
Integration and UNISA will monitor 100
[Smartphones]
adaptation of the installation in
PRI.WP2.11 equipped with - 5
components in works and report on the
the navigation
Italian Pilot Site the progress Pilot
software
area
Integration and Multimodal trip UNISA will monitor
PRI.WP2.12 - 1 -
adaptation of adviser the installation

5/14/2019 Page 47 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Expected Progress
Relating to which
Method of

Year 1

Year 2

Year 3
Indicator No. project objective / Indicator
measurement
expected result?

components in works and report on

Italian Pilot Site the progress
UNISA will monitor
Operation of tests Number of [tests the installation
PRI.WP2.13 40% ALL -
in Italian Pilot Site completed] works and report on
the progress
Streetlight ex- UNISA will monitor
Operation of the ante data the installation
PRI.WP2.14 1 5 -
Italian Pilot Site collection works and report on
[months] the progress
Streetlight ex- UNISA will monitor
Operation of the post data the installation
PRI.WP2.15 - 6 5
Italian Pilot Site collection works and report on
[months] the progress
Ex-ante fuel UNISA will monitor
Operation of the consumption the installation
PRI.WP2.16 1 2 -
Italian Pilot Site data collection works and report on
[months] the progress
Ex-post fuel UNISA will monitor
Operation of the consumption the installation
PRI.WP2.17 - 6 5
Italian Pilot Site data collection works and report on
[months] the progress
Ex-ante fuel CRF will monitor
Operation of the consumption the installation
PRI.WP2.18 - 2 -
Italian Pilot Site data collection works and report on
[months] the progress
Ex-post fuel CRF will monitor
Operation of the consumption the installation
PRI.WP2.19 - 2 -
Italian Pilot Site data collection works and report on
[months] the progress
Ex-ante shuttle UNISA will monitor
Operation of the use data the installation
PRI.WP2.20 2 2 -
Italian Pilot Site collection works and report on
[months] the progress
Ex-post shuttle UNISA will monitor
Operation of the use data the installation
PRI.WP2.21 - 6 5
Italian Pilot Site collection works and report on
[months] the progress
Quantified data
on the fuel
1 (in
saving potential
the
of eco-driving CRF will evaluate
cases
Operation of the [l/100km] the potential fuel
IMI.WP2.1 - where
Italian Pilot Site  CAN data saving of the eco-
it is
before/after driving application
applica
 Eco-driving
ble)
application
before/after

5/14/2019 Page 48 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Expected Progress
Relating to which
Method of

Year 1

Year 2

Year 3
Indicator No. project objective / Indicator
measurement
expected result?

Quantified data UNISA will evaluate

Operation of the
IMI.WP2.2 on the fuel the potential fuel - 1 1
Italian Pilot Site
saving potential saving
Quantified data
on the energy
saving potential
of the adaptable
streetlight
UNISA, SWF and
strategy
MIZAR will evaluate
[Watt/kg CO2]
Operation of the the potential
IMI.WP2.3  LED vs - - 1
Italian Pilot Site energy saving of
Traditional
the adaptable
lights
streetlight strategy
 LED vs Adapt.
LED (without
traffic
management
policy)
Quantified data
on the energy
saving potential
of the Eco-
Traffic
Management
strategy
MIZAR will evaluate
[Watt/kg CO2]
the potential
 LED vs Adapt.
energy saving of
Operation of the LED (without
IMI.WP2.3 the environmental - - 1
Italian Pilot Site traffic
oriented Traffic
management
Management
policy)
strategy
 LED vs LED
adapt. With
traffic
management
policies

>10%
Quantified data
on the energy MIZAR, KTC and
saving potential UNISA will evaluate
of the Eco- the potential
Operation of the dynamic routing energy saving of a
IMI.WP2.4 - - 1
Italian Pilot Site and transit and multimodal eco-
eco dynamic routing strategy
access (incl. P&R) in terms
management of users mode shift
strategies

5/14/2019 Page 49 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Expected Progress
Relating to which
Method of

Year 1

Year 2

Year 3
Indicator No. project objective / Indicator
measurement
expected result?

[Watt/kg CO2]
Modal Shift

>3%
MIZAR, KTC and
Quantified data UNISA will evaluate
on number of the potential
new user of bus growth in the use of
Operation of the shuttle public transport
IMI.WP2.5 - - 1
Italian Pilot Site Modal Shift due to the eco-
routing and eco-
dynamic access
>3% management
strategies

Table 5. Italian Pilot Site Progress and Impacts indicators

3.3. PS#2: Private vehicles in the Inter-Urban environment

In this chapter, the information related to the Austrian Pilot Site is presented. For further
information view D3.1 Austrian Pilot Site: Description and Planning document.

3.3.1. Pilot Site brief description

The goal of the pilot site in Austria is to test a multiple interurban use-case located on a highway. It
will involve a section of the Vienna highway with a temporary construction site (road works). The
aim is to measure the increase in traffic efficiency achieved through the smoothing of the traffic
flow and the resulting reduction in fuel consumption as well as the energy savings from the adaptive
lighting application. These positive effects for the environment are possible through the
implementation of cooperative functions permitting infrastructure to vehicle communications
combined with an innovative traffic monitoring system.

Especially on busy highways, road works result in a safety risk due to the deviation of traffic from
one lane to another and also affect the efficiency of traffic flows. The slowing of vehicles leads to
congestion and stop&go conditions particularly in heavy traffic. Cooperative system applications can
minimise this effect by providing advance warning to vehicles and speed indications which depend
on the real-time traffic situation.

5/14/2019 Page 50 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Figure 16. PROJECT Cooperative Congestion Avoidance scenario on highway

3.3.2. Site Legacy Systems

Typically, an Austrian construction site is not equipped with any of the on-site devices used within
the project. In other words, a normal road works scenario does neither use LED based streetlights,
mobile trailers but static signs nor wireless sensors but cameras controlled manually by an operator.

On the opposite, a traffic management centre provided by ASFINAG does already exist. It is the
control centre for installed traffic guidance systems including variable message signs on highways.
Those devices are linked via wired communication lines located next to the highway to the traffic
management centre.

3.3.3. Applications to be validated

3.3.3.1. Drawing attention to the traffic situation via streetlight

Streetlights are placed at the beginning of the construction site where the lanes are shifted to the
other side of the motorway. During the dark the streetlights are switched on. The test is carried out
every day during the dark.

5/14/2019 Page 51 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

The test scenario remains the same over the period of time: Drivers are passing the illuminated
pilot site and data as mentioned above is acquired constantly.

3.3.3.2. Drawing attention to the traffic situation via mobile trailer

Mobile trailers are installed and connected via a gateway to the traffic management centre. In
general, the mobile trailers are in operation all day long. Aspects to be shown are triggered by the
centre. The test is carried out 10-20 times a day 7 days a week. The test scenario remains the same
over the period of time: Drivers are passing the pilot site and attention is drawn to the current
traffic situation via shown aspects on the mobile trailers and data is being acquired.

3.3.3.3. Drawing attention to the traffic situation via on-board unit

Cars are equipped with an on-board unit and a connected HMI. Information to be shown is sent from
the traffic management centre.

The test is carried out 10-20 times a day 5 days a week. The test scenario remains the same over
the period of time: Drivers are passing the pilot site and attention is drawn to the current traffic
situation via danger signs or textual information on the HMI.

3.3.4. Validation criteria

In this section, a summary of the targeted criteria and the related performance indicators defined
for the Austrian Pilot Site is shown. For further information view annexes #1 to #5.

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
The fuel consumption
The Austrian Site reduction thanks to
Reduce fuel applications will driving Fuel consumption
consumption contribute to a fuel recommendations is [l/Km]
reduction estimated to be >
ENVIRONMENTAL 10%
The Austrian Site The CO2 emissions
Reduce CO2 applications will reduction is
CO2 emissions [g/Km]
emissions contribute to CO2 estimated to be >
emissions reduction 10%
The Austrian The traffic flow with
applications will the Austrian
Traffic flow
MOBILITY Traffic efficiency contribute to a applications will be
[vehicles/hour]
smooth traffic flow better than without
the systems

5/14/2019 Page 52 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
The Austrian The road safety with
applications will the Austrian
Improve road
SAFETY contribute to an applications will be Black spots
safety
improvement of road improve than without
safety the systems
The on-board The numbers of Rate of use = number
DRIVER Driver behaviour Austrian application recommendations instructions followed
BEHAVIOUR change will modify the followed is estimated [absolute,
drivers behaviour to be increased >30% percentage]
The on-board
System is not switch System switch off
Austrian application
off [bool]
is accepted
USER
ACCEPTANCE The applications
High user acceptance
implemented at
score in Usefulness
Austrian PS are
questionnaire
accepted

Table 6. Hypothesis, targeted criteria and related performance indicators per category

3.3.5. Outline of planned test cases

For further information view Annex #8.

5/14/2019 Page 53 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

3.3.5.1. Validation Test cases list

Figure 17. Austrian Pilot Site Validation Test Cases List

5/14/2019 Page 54 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

3.3.5.2. Timing

M1 M2 M3 M4 M5

Year 1 Year 2

Scenario 1 Scenario 2 Scenario 3

Table 7. Austrian Pilot Site1

Refer to D3.1 TEST SITE Pilot site Design and planning for further information.

3.3.6. Progress and impact indicators

As the Italian Pilot Site, the following table summarizes the progress indicators (PRI) and the impact
indicators (IMI) of Austrian Pilot Site:

Expected Progress
Relating to which
Method of

Year 1

Year 2

Year 3
Indicator No. project objective / Indicator
measurement
expected result?

Integration and
SWF will monitor the
adaptation of Number of
PRI.WP3.1 activities and report 2 2 0
components in [mobile VMS]
on the progress
Austrian Pilot Site
Integration and
Number of [DSRC SWF will monitor the
adaptation of
PRI.WP3.2 equipped activities and report 10 - -
components in
vehicles] on the progress
Austrian Pilot Site
Integration and
Number of SWF will monitor the
adaptation of
PRI.WP3.3 [streetlight activities and report 14 0 -
components in
heads] on the progress
Austrian Pilot Site
Integration and
Number of SWF will monitor the
adaptation of
PRI.WP3.4 [COOPERS RSU] activities and report 2 2 0
components in
for the trailers on the progress
Austrian Pilot Site
Integration and
Number of [WSN SWF will monitor the
adaptation of
PRI.WP3.5 nodes + activities and report 5+10 5+10 0
components in
repeaters] on the progress
Austrian Pilot Site
Integration and
SWF will monitor the
adaptation of Number of [WSN
PRI.WP3.6 activities and report 1 1 0
components in gateway]
on the progress
Austrian Pilot Site

1 M1: April 2011, M2: July 2011, M3: October 2011, M4: April 2012 and M5: October 2012

5/14/2019 Page 55 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Expected Progress
Relating to which
Method of

Year 1

Year 2

Year 3
Indicator No. project objective / Indicator
measurement
expected result?

Integration and
Navigation SWF will monitor the
adaptation of
PRI.WP3.7 software activities and report 1 - -
components in
adaptation on the progress
Austrian Pilot Site
Integration and
Integration of SWF will monitor the
adaptation of
PRI.WP3.8 data into the activities and report 1 - -
components in
TMC on the progress
Austrian Pilot Site
Ex-ante
SWF will monitor the
Operation of the streetlight
PRI.WP3.9 activities and report 1 - -
Austrian Pilot Site energy
on the progress
consumption
Ex-post
SWF will monitor the
Operation of the streetlight
PRI.WP3.10 activities and report 1 - -
Austrian Pilot Site energy
on the progress
consumption
Ex-ante traffic SWF will monitor the
Operation of the
PRI.WP3.11 flow data activities and report 3-6 - -
Austrian Pilot Site
collection on the progress
Ex-post traffic SWF will monitor the
Operation of the
PRI.WP3.12 flow activities and report 3-6 3-6 -
Austrian Pilot Site
measurement on the progress
Evaluation of
Quantification of speed/acceleration +
the reduction in car engine model
Operation of the CO2 emissions Fuel consumption
IMI.WP3.1 1 1 -
Austrian Pilot Site model vs. collected
>10% data
Extrapolation from
measured flows

Table 8. Austrian Pilot Site Progress and Impacts indicators

3.4. PS#3: Public Transportation in the urban environment

The information given along this section is related to the Swedish Pilot Site. For further information
view D4.1 Swedish Pilot Site: Description and Planning document.

3.4.1. Pilot Site brief description

The goal of the Swedish pilot site is to demonstrate the impact of combined cooperative
applications involving the public transport fleet (buses) and to quantify their potential for improving
energy and fuel efficiency.

5/14/2019 Page 56 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

The pilot site is located in the City of Gothenburg, along one of the main roads which is the route of
several bus lines. There are a number of intersections which are controlled by traffic lights. The
traffic light controllers will be connected to roadside units (RSU).

The Swedish test site will have access to a number of vehicles (in the range between 7-10) provided
by bus operators connected to the public transportation company (Västtrafiken). The application
will be evaluated by at least 20 commercial bus drivers (end users) during the trial period. The
vehicles will be equipped with a runtime environment for the applications, 802.11p as well as 3G
communication capabilities, a driver interface system and data measurement platform. The legacy
traffic light control system will be integrated to the RSU and will, provide the approaching busses
with information about current and future state and timing information, and the Traffic
Management Centre will be adapted to meet the requirements of the application.

21,24,129,141

Wiselgrensplatsen

Gropegårdsgatan

Eketrägatan

Figure 18. Part of Swedish Pilot Site: intersection at Hjalmar Brantingsgatan

3.4.2. Site Legacy Systems

The test site is composed by three traffic signal installation located according to Figure 18 along
Hjalmar Brantingsgatan. The intersection Hjalmar Brantingsgatan - Wieselgrensgatan – Långängen,
consists of three traffic rings where traffic in the main street will be included in the project.

The crossing is part of a centralized control along the main street, Hjalmar Brantingsgatan where
two additional traffic signals are included with similar approaches.

5/14/2019 Page 57 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Figure 19. Planning view of the traffic light installation at the intersection of Hjalmar
Brantingsgatan - Wieselgrensgatan – Långängen

The RSU will be installed in the existing technical bay, and antennas for both 802.11p and 3G
communications will be mounted in such a way that communication with the test vehicles is
established, if necessary repeaters will be used to extend the 802.11p range. Digital signals from
the traffic controller will be transmitted to the RSU via an USB interface.

3.4.3. Applications to be validated

The Swedish test site will be running naturalistic tests for a period of at least 6 months. It is
expected that equipped vehicles will pass through the test site at least 40 times per day. The test
site applications will only use visual means for driver interaction.

3.4.3.1. Enhanced Bus intersection logistic with eco driver support

The enhanced bus intersection application running in the bus will receive information about the
current and future states of the traffic light it is approaching. This information will be sent from a
road-side unit integrated with the traffic light using 802.11p communication. The application will
calculate the most appropriate speed or span of speeds to arrive at the traffic light during a green
state. This information will be presented to the driver through an in-vehicle HMI.

3.4.3.2. Congestion prevention with eco driver support

The congestion prevention application running in the bus will give the driver speed advice in order
to arrive to a non-occupied bus stop. Also the driver of the bus will be advised not to depart from a
bus stop prematurely but wait in order to synchronize with the traffic light state. The bus will
receive information about the traffic light states from a road-side unit (that is integrated with the
traffic light) over 802.11p communication and information related to other non-equipped vehicles
from a fleet management connection.

5/14/2019 Page 58 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

3.4.3.3. ISA/Intelligent Speed Adaptation

The intelligent speed adaptation application is aimed to provide the driver with continuous speed
information relevant to the road travelled, meaning that both dynamic and fixed speed limits can
be displayed to the driver. The dynamic speed limits will be provided to the vehicle via a fleet
management connection. Dynamic speed limits will not be used within the test site area since the
selected area is lacking that type of infrastructure.

3.4.4. Validation criteria

In this section, a summary of the targeted criteria and the related performance indicators defined
for the Swedish Pilot Site is shown. For further information view annexes #1 to #5.

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
The Enhanced Bus
intersection logistics
application together The fuel consumption
Reduce fuel with the congestion reduction within the Fuel consumption
consumption prevention test site is estimated [l/Km]
application will to be > 10%
reduce the fuel
consumption of buses
ENVIRONMENTAL The Enhanced Bus
intersection logistics
application together
The CO2 emissions
with the congestion
Reduce CO2 reduction is
prevention CO2 emissions [g/Km]
emissions estimated to be >
application will
10%
reduce the CO2
emissions caused by
buses
The Enhanced Bus
intersection logistics
The rate of number
application together
of unplanned stops
with the congestion Number of unplanned
MOBILITY Traffic efficiency reduction is
prevention stops [percentage]
estimated to be >
application will
10%
reduce the number
of unplanned stops

5/14/2019 Page 59 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
The Enhanced Bus
intersection logistics
application together
with the congestion The average speed of
prevention a bus passing the
Travel time [s]
application will intersection will be
increase the average increased > 5km/h
speed of the bus
passing the
intersection
Mean deceleration;
or mean number of
The Eco-Driving deceleration events
application does not with system < mean
Deceleration [m/s2]
lead to inefficient deceleration; mean
deceleration number of
deceleration events
without system
The Eco-Driving Mean speed with
application does not system is better than
Mean speed [Km/h]
DRIVER Driver behaviour lead to inefficient mean speed without
BEHAVIOUR change speed system
The Eco-Driving
Frequency of stops
application does not
with system < Number of stop&go
lead to an increased
frequency of stops [absolute, percent]
number of vehicle
without system
stops.
Mean idling time
The Eco-Driving
with system < mean
application does not
idling time without Time of idling [s]
lead to an increased
system within the
of time of idling.
test site
The applications
High user acceptance
USER implemented at the
score in Usefulness
ACCEPTANCE Swedish TS are
questionnaire
accepted

Table 9. Hypothesis, targeted criteria and related performance indicators per category

3.4.5. Outline of planned test cases

For further information view Annex #9.

5/14/2019 Page 60 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

3.4.5.1. Validation Test cases list

Figure 20. Swedish Pilot Site Validation Test Cases List

5/14/2019 Page 61 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

3.4.5.2. Timing

Table 10. Swedish Pilot Site timing

3.4.6. Progress and impact indicators

As the other sites, the following table summarizes the progress indicators (PRI) and the impact
indicators (IMI) of Swedish Pilot Site:

Expected Progress
Relating to which
Method of
Year 1

Year 2

Year 3
Indicator No. project objective / Indicator
measurement
expected result?

Integration and
LSP will monitor the
adaptation of Number of traffic
PRI.WP4.1 activities and report 3 - -
components in lights connected
on the progress
Swedish Pilot Site
Integration and
LSP will monitor the
adaptation of Number of [RSUs]
PRI.WP4.2 activities and report 3 - -
components in Installed
on the progress
Swedish Pilot Site
Integration and
LSP will monitor the
adaptation of Number of
PRI.WP4.3 activities and report 7 10 -
components in Vehicles involved
on the progress
Swedish Pilot Site
Integration and
Back office data LSP will monitor the
adaptation of
PRI.WP4.4 collection activities and report 1 1 -
components in
installed on the progress
Swedish Pilot Site
Number of traffic LSP will monitor the
Operation of tests
lights activities and report
PRI.WP4.5 in Swedish Pilot - 11 0
connected/opera on the progress
Site
tional [months]

5/14/2019 Page 62 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

Expected Progress
Relating to which
Method of

Year 1

Year 2

Year 3
Indicator No. project objective / Indicator
measurement
expected result?

LSP will monitor the

Operation of tests
Number of RSU:s activities and report
PRI.WP4.6 in Swedish Pilot - 11 0
operational on the progress
Site
[months]
LSP and VTEC will
CO2 reduction:
Operation of tests collect and monitor
>10%
IMI.WP4.1 in Swedish Pilot the CO2 and fuel - 11 0
Fuel consumption
Site consumption of
reduction: >10%
vehicles
LSP and VTEC will
Operation of tests Back office data collect and monitor
IMI.WP4.2 in Swedish Pilot collection the CO2 and fuel - 11 0
Site operational consumption of
vehicles
Completion of
detailed reports
Specifications:
for the state-of- All the partners will
1) procurement
the-art provide a detailed
2) test procedures
cooperative description of the
and quantified
SPC.WP5.1 traffic design and 1 2-3 4
results
management implementation
3) installation
systems which procedures (except
4) operation and
have been for Austria)
monitoring
installed and
tested
Completion of
detailed reports
Specifications:
for the state-of-
1) procurement Austrian site will
the-art
2) test procedures provide a detailed
cooperative
and quantified description of the
SPC.WP5.2 traffic 1 2-3 4
results design and
management
3) installation implementation
systems which
4) operation and procedures
have been
monitoring
installed and
tested

Table 11. Swedish Pilot Site Progress and Impacts indicators

5/14/2019 Page 63 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

4. LEGAL AND ORGANISATIONAL ASPECTS

From the legal and ethical point of view several aspects should be considered in order to the
following orientative categories and main aspects based in FESTA Support Action information [12]. A
check-list for the control of these aspects can be found in annex #10

4.1. Briefing of test participants and contractual agreements

Information provided to test participants

 Information on system limitations and possible malfunctions in intervening and cooperative
systems
 Information on data recording
 Information and agreement on cost allocation and liabilities
 Administrative issues
 Regarded to the issues (fines) from use of on-board test systems and applications

4.2. Data privacy

 Basic principles and regulations. EU minimum required

 Data privacy in research activities
 Legally relevant data and general measures to ensure data privacy
 Consent of test participants
 Data acquisition
 Technical and organisational measures
 Video recording of driver and passengers

4.3. Insurance

 Road traffic liabilities

 Insurance for road traffic in EU countries
 Specific insurance for research and testing activities

4.4. Licensing requirements

 Licensing requirements for motor vehicles in EU countries

 Special regulation for vehicle manufacturers
 Licensing of not approved systems being object of the research
 Special licences for special vehicles and systems

4.5. Ethical rules

 Respect for the person

 Informed Consent
 Privacy
 Research on/with animals

5/14/2019 Page 64 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

5. References
[1] ISO 9001- Section 7.3.1 Design and Development Planning, Section 7.3.5 Design and
Development Verification, Section 7.3.6 Design and Development, Section 4.2.4 Control
of Records. (2009).
[2] Systems Engineering - System life cycle processes, ISO/IEC 15288. (2009).
[3] Driessen, B., Hogema, J., Wilmink, I., Ploeg, J., Papp, Z., & Feenstra, P. (2007). The
SUMMITS Tool Suite: supporting the development and evaluation of cooperative vehicle-
infrastructure systems in a Multi-Aspect Assessment approach (TNO memo 073401-N17).
Delft: TNO Verkeer en Vervoer.
[4] FESTA Support Action, FESTA handbook http://www.its.leeds.ac.uk/festa/downloads.php
[5] CONVERGE Project TR 1101, Deliverable D2.3.1
[6] SAFESPOT Project, Deliverable D5.6.1
[7] ISO 9241-11 (1998). Ergonomic requirements for office work with visual display terminals
(VDTs) - Part 11 :Guidance on usability
[8] Brooke, J. (1996) SUS: a "quick and dirty" usability scale. In P. W. Jordan, B. Thomas, B.
A. Weerdmeester & A. L. McClelland (eds.) Usability Evaluation in Industry. London:
Taylor and Francis.
[9] Van Der Laan, J., Heino, A., & De Waard, D., (1997). “A simple procedure for the
assessment of acceptance of advanced transport telematics”, Transportation Research-C,
5, 1-10.
[10] Risser, R. (1985): Behaviour in traffic conflict situations. Accident Analysis and Prevention
Vol. 17, No. 2, pp 179-197
[11] Zhang, X., et al., (1998): Guidebook for Assessment of Transport Telematics Applications:
Updated Version, CONVERGE Project TR 1101, Deliverable D2.3.1
[12] FESTA Support Action, Deliverable D6.3
[13] European Comission, (January 2009). Impact Assessment Guidelines, SEC(2009) 92.
[14] Regione Piemonte, “Accessibilità e Mobilità in Piemonte: La Gestione del processo di
pianificazione”, Gennaio 2010, in Italian, available at URL:
http://www.regione.piemonte.it/trasporti/prt/dwd/acces.pdf

5/14/2019 Page 65 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

ANNEXES2

ANNEX #1 – TARGETED CRITERIA TABLE

ANNEX #2 – PERFORMANCE INDICATORS TABLE
ANNEX #3 – REQUIRED MEASURES TABLE
ANNEX #4 – SENSORS TABLE
ANNEX #5 – QUESTIONNAIRES
ANNEX #6 – TEST CASE TEMPLATE
ANNEX #7 – ITALIAN TEST CASES
ANNEX #8 – AUSTRIAN TEST CASE
ANNEX #9 – SWEDISH TEST CASES
ANNEX #10 - CHECK-LIST FOR CONTROL OF LEGAL AND ETHICAL ASPECTS

2 Annexes from 1 to 5 are part of the Validation Matrix.

5/14/2019 Page 66 of 113 Version 1.4
 D5.1 COSMO Validation Test Plan

ANEX 1. TARGETED CRITERIA TABLE

5/14/2019 Page 67 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

5/14/2019 Page 68 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

5/14/2019 Page 69 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

5/14/2019 Page 70 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

ANEX 2. PERFORMANCE INDICATORS TABLE

5/14/2019 Page 71 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

ANNEX 3. REQUIRED MEASURES TABLE

5/14/2019 Page 72 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

ANNEX 4. SENSORS TABLE

5/14/2019 Page 73 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

ANNEX 5. QUESTIONNAIRES

5/14/2019 Page 74 of 113 Version 1.4

 D5.1 COSMO Validation Test Plan

ANEX 1. TARGETED CRITERIA TABLES

5/14/2019 Page 75 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

ITALIAN PILOT SITE: TARGETED CRITERIA

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
Traffic-Adapted
Luminous Path
The energy
application will
consumption
Reduce energy reduce the energy Energy consumption
reduction is
consumption consumption thanks [Watt/hour]
estimated to be >
to the adjustment of
10%
a street lighting
system
The Eco-Driving
The fuel
application will
consumption
contribute to a fuel
Reduce fuel reduction thanks to Fuel consumption
reduction providing
consumption driving [l/km]
recommendations to
recommendations is
the driver during a
estimated to be > 5%
ENVIRONMENTAL journey
The Eco-Dynamic
Access Management The CO2 emissions
will contribute to reduction is CO2 emissions
reduce the CO2 estimated to be > [g/km]
emissions on the 15 %
defined zone
Reduce CO2 The Eco-Driving
emissions application will
contribute to a CO2
The CO2 emissions
emissions reduction CO2 emissions
reduction is
providing [g/km]
estimated to be >5 %
recommendations to
the driver during a
journey
The Eco-Dynamic
Access Management
The travel time
Traffic will contribute to Travel time
reduction per test
efficiency reduce traffic (‘Parking time’) [s]
user is > 5%
congestions and
improve traffic flow
MOBILITY
Eco-transit and
The rate of drivers
Dynamic Parking
who taking the Use of shuttle
Increased co- management
internal shuttle services
modality contribute to
services estimated [percentage]
increase the
to be increase > 3%
mobility

5/14/2019 Page 76 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
Mean deceleration;
or mean number of
The Eco-Driving deceleration events
application does not with system < mean
Deceleration [m/s2]
lead to inefficient deceleration; mean
deceleration number of
deceleration events
without system
The Eco-Driving Mean speed with
application does not system is better
Mean speed [km/h]
lead to inefficient than mean speed
speed without system
The Eco-Transit Mean speed with
application does not system is better
Mean speed [km/h]
lead to inefficient than mean speed
speed without system
Frequency of 1st and
The Eco-Driving 2nd gear < Frequency of 1st,
application does not frequency without 2nd and 5th gear
lead to inefficient system (frequency of usage [absolute,
gear usage 5th gear > without percent]
system)
DRIVER Driver behaviour
The Eco-Driving
BEHAVIOUR change
application will
modify the drivers
behaviour (in this
The numbers of
validation activity Rate of use =
recommendations
only the CRF number instructions
followed is
vehicles will be followed [absolute,
estimated to be
involved and, due to percentage]
increased > 10%
limitation on driving
permission, only
staff will be involved
as test drivers)
The Eco-Transit
application will The numbers of
Rate of use =
modify the drivers recommendations
number instructions
behaviour (in this followed is
followed [absolute,
validation activity estimated to be
percentage]
only 40-50 students increased > 30%
will be involved)
The numbers of
The Eco-Transit Navigation
recommendations
application will instructions
followed is
modify the drivers followed [absolute,
estimated to be
behaviour percentage]
increased > 10%

5/14/2019 Page 77 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
The Eco-Driving
Frequency of stops
application does not
with system < Number of stop&go
lead to an increased
frequency of stops [absolute, percent]
number of vehicle
without system
stoppings.
The Eco-Drive Real
System is not switch System switch off
Time system is
off [bool]
accepted
USER
ACCEPTANCE The applications
High user
implemented at
acceptance score in Usefulness
Italian PS are
questionnaire
accepted
AUSTRIAN PILOT SITE: TARGETED CRITERIA

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
The fuel consumption
The Austrian Site reduction thanks to
Reduce fuel applications will driving Fuel consumption
consumption contribute to a fuel recommendations is [l/Km]
reduction estimated to be >
ENVIRONMENTAL 10%
The Austrian Site The CO2 emissions
Reduce CO2 applications will reduction is
CO2 emissions [g/Km]
emissions contribute to CO2 estimated to be >
emissions reduction 10%
The Austrian The traffic flow with
applications will the Austrian
Traffic flow
MOBILITY Traffic efficiency contribute to a applications will be
[vehicles/hour]
smooth traffic flow better than without
the systems
The Austrian The road safety with
applications will the Austrian
Improve road
SAFETY contribute to an applications will be Black spots
safety
improvement of road improve than without
safety the systems
The on-board The numbers of Rate of use = number
DRIVER Driver behaviour Austrian application recommendations instructions followed
BEHAVIOUR change will modify the followed is estimated [absolute,
drivers behaviour to be increased >30% percentage]
The on-board
USER System is not switch System switch off
Austrian application
ACCEPTANCE off [bool]
is accepted

5/14/2019 Page 78 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
The applications
High user acceptance
implemented at
score in Usefulness
Austrian PS are
questionnaire
accepted

SWEDISH PILOT SITE: TARGETED CRITERIA

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
The Enhanced Bus
intersection logistics
application together The fuel consumption
Reduce fuel with the congestion reduction within the Fuel consumption
consumption prevention test site is estimated [l/Km]
application will to be > 10%
reduce the fuel
consumption of buses
EVIRONMENTAL The Enhanced Bus
intersection logistics
application together
The CO2 emissions
with the congestion
Reduce CO2 reduction is
prevention CO2 emissions [g/Km]
emissions estimated to be >
application will
10%
reduce the CO2
emissions caused by
buses

5/14/2019 Page 79 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

STATED PERFORMANCE
CATEGORY HYPHOTESIS TARGETED CRITERIA
OBJECTIVE INDICATOR
The Enhanced Bus
intersection logistics
The rate of number
application together
of unplanned stops
with the congestion Number of unplanned
reduction is
prevention stops [percentage]
estimated to be >
application will
10%
reduce the number
of unplanned stops
MOBILITY Traffic efficiency The Enhanced Bus
intersection logistics
application together
with the congestion The average speed of
prevention a bus passing the
Travel time [s]
application will intersection will be
increase the average increased > 5km/h
speed of the bus
passing the
intersection
Mean deceleration;
or mean number of
The Eco-Driving deceleration events
application does not with system < mean
Deceleration [m/s2]
lead to inefficient deceleration; mean
deceleration number of
deceleration events
without system
The Eco-Driving Mean speed with
application does not system is better than
Mean speed [Km/h]
DRIVER Driver behaviour lead to inefficient mean speed without
BEHAVIOUR change speed system
The Eco-Driving
Frequency of stops
application does not
with system < Number of stop&go
lead to an increased
frequency of stops [absolute, percent]
number of vehicle
without system
stops.
Mean idling time
The Eco-Driving
with system < mean
application does not
idling time without Time of idling [s]
lead to an increased
system within the
of time of idling.
test site
The applications
High user acceptance
USER implemented at the
score in Usefulness
ACCEPTANCE Swedish TS are
questionnaire
accepted

5/14/2019 Page 80 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

ANNEX 2. PERFORMANCE INDICATORS TABLES

5/14/2019 Page 81 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

PI_ID PI_01 Name Average fuel consumption, hot engine, calculated

Category Environmental Group Fuel consumption
Description Average fuel consumption, hot engine, calculated
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit l/km Equation ARTEMIS as a function of road type, gradient and average speed
Comments One problem with the choice of unit is if there is need to describe the PI
during stop-time
REQUIRED MEASURES (Speed_CAN OR Speed_GPS OR Speed_WheelUnitDistance) AND
TrafficSituation_Artemis AND Model_Macro OR P98 (Driving Matrix) AND
Model_Micro
Related Sites Italian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References Keller, M. and Kljun, N. ARTEMIS Road Emission Model. Model description. (EU
Commission – DG Tren – Contract 1999-RD.10429). Workpackage 1100 – Model
version 04c. Deliverable 13. INFRAS. Bern. 2007. See also document
'Classification of RoadTypes and TrafficSituations.doc'.

PI_ID PI_02 Name Average fuel consumption without fuel temp correction
Category Environmental Group Fuel consumption
Description Average fuel consumption of test site pass measured without fuel temp correction
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit l/km Equation
Comments The fuel consumptions within the test site will always be measured on a hot
engine, and only measured within the test site.
The fuel economy will be calculated by either integrating the speed values
over time or by using high accuracy trip meter signals to calculate the distance
driven and the amount of fuel used will be calculated by integrating the fuel
rate over time (FUEL_USED_INTEGRATED). Using the distance and amount of
used fuel a fuel consumption will be calculated (l/km). Thus there will be no
problem describing the PI during stop-time.
REQUIRED MEASURES Fuel_Consumption_1 AND (SPEED OR TRIPMETER)
Related Sites Swedish
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
1 per test site pass N/A N/A N/A
ETHICAL ISSUES N/A
References

5/14/2019 Page 82 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

PI_ID PI_03 Name Absolute fuel consumption

Category Environmental Group Fuel consumption
Description Amount of fuel needed for one single trip
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit l Equation
Comments values will have a high variation, primarily due to differences in traffic. Not
only the value will (obviously) depend on the trip, but also other control
factors will have different impact for different trips.
REQUIRED MEASURES FUEL_USED_DIFFERENCE OR FUEL_USED_INTEGRATED OR
FUEL_USED_CALCULATED OR FUEL_USED_EXPECTED
Related Sites Italian / Austrian?
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
1 per trip Effects are expected to Resolution needs to be N/A
be in the order of a few better than required
percent, so a relative absolute accuracy.
accuracy of better than
1% should be achieved.
The resulting absolute
accuracy depends on the
length of the trip; with
a fixed absolute
accuracy available, the
relative accuracy can be
improved by lengthening
the trip.
ETHICAL ISSUES N/A
References

PI_ID PI_04 Name Average CO2 hot calculated

Category Environmental Group CO2 emissions
Description
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit g/km Equation
Comments One problem with the choice of unit is if there is need to describe the PI
during stop-time
REQUIRED MEASURES (Fuel consumption hot calculated) AND Constant_CO2
Related Sites Italian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A

5/14/2019 Page 83 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

ETHICAL ISSUES N/A

References

PI_ID PI_05 Name Average CO2 estimated based on measured fuel consumption
Category Environmental Group CO2 emissions
Description
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit g/km Equation
Comments One problem with the choice of unit is if there is need to describe the PI
during stop-time
REQUIRED MEASURES (FC average measured) AND Constant_CO2
Related Sites Italian, Swedish, Austrian?
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_06 Name Average energy consumption total calculated

Category Environmental Group Energy consumption
Description
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit Watts/h Equation
Comments
REQUIRED MEASURES WATTS
Related Sites Italian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_07 Name Average energy consumption low lighting level

Category Environmental Group Energy consumption
Description
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN

5/14/2019 Page 84 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Unit Watts/h Equation

Comments
REQUIRED MEASURES WATTS
Related Sites Italian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_08 Name Average energy consumption high lighting level

Category Environmental Group Energy consumption
Description
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit Watts/h Equation
Comments
REQUIRED MEASURES WATTS
Related Sites Italian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_09 Name Number of unplanned stops

Category Mobility Group Traffic efficiency
Description A stop has occurred in the event when the vehicles' speed is < 3km/h
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit N/A Equation Summation of all stopping events (when a vehicles speed
<3km/h)
Comments
REQUIRED MEASURES Speed_CAN OR Speed_GPS OR Speed_WheelUnitDistance
Related Sites Swedish
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A

5/14/2019 Page 85 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

References Not known

PI_ID PI_10 Name Travel time [Parking time]

Category Mobility Group Traffic efficiency
Description Time taken from entering to exiting the test site (parking areas) with and without the
applications
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit s Equation Timestamp when entering - Timestamp when exiting
Comments
REQUIRED MEASURES TIME
Related Sites Italian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
1 per test site pass 1s accuracy N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_11 Name Rate of increment of drivers who parking in the suggested
parking area
Category Mobility Group Traffic efficiency
Description
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit % Equation (TOT.PARKING with SUGGESTION - TOT PARKING without
SUGGESTION) / TOT PARKING without SUGGESTION
Comments Need to measure the use of the parking before and after the suggestion system
in use
REQUIRED MEASURES TOT PARKING with SUGGESTION AND TOT PARKING without SUGGESTION
Related Sites Italian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
1/day 1% accuracy 1% N/A
ETHICAL ISSUES N/A
References

PI_ID PI_12 Name Use of shuttle services

Category Mobility Group Increased co-modality
Description

5/14/2019 Page 86 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN

Unit % Equation
Comments
REQUIRED MEASURES
Related Sites Italian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_13 Name Travel distance

Category Mobility Group Traffic efficiency
Description Distance travelled in the campus per ride
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit km Equation
Comments
REQUIRED MEASURES DISTANCE
Related Sites Italian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
1 per trip N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_14 Name Travel time

Category Mobility Group Traffic efficiency
Description Time taking for the ride in the campus (Italian Pilot Site) /
Time taken from entering to exiting the test site (Swedish Pilot Site)
The average speed of a bus passing the intersection
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit Equation
Comments
REQUIRED MEASURES TIME (ODOMETER) AND DISTANCE
Related Sites Italian /Swedish
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization

5/14/2019 Page 87 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

N/A N/A N/A N/A

ETHICAL ISSUES N/A
References

PI_ID PI_15 Name Traffic flow

Category Mobility Group Traffic efficiency
Description Number of vehicles passing the test site in one hour (Austrian Pilot Site)
[level of service - LoS] Estimation of the level of congestion (Italian Pilot Site)
Subjective (Sub) / Objective (Obj) Obj / Obj Quantitative (QN) / Qualitative (QL) QN / QL
Unit Vehicles/hour Equation
LoS Scale
Comments
REQUIRED MEASURES VEHICLES and TIME (traffic flow) TRAFFIC_LoS AND VEHICLES_PARKED
Related Sites Austrian / Italian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
1/5 min Defined level of service
ETHICAL ISSUES N/A
References

PI_ID PI_16 Name Black spots

Category Safety Group N/A
Description
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit Equation
Comments
REQUIRED MEASURES
Related Sites Austrian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_17 Name Deceleration

5/14/2019 Page 88 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Category Driver behaviour Group N/A

Description Average deceleration or number of deceleration events
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
2
Unit m/s Equation Mean (deceleration); Total number of deceleration events/time
driven
Comments
REQUIRED MEASURES PedalGas AND PedalBrake AND (Speed_CAN OR Speed_GPS OR
Speed_WheelUnitDistance)
Related Sites Italian / Swedish
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_18 Name Mean speed

Category Driver behaviour Group N/A
Description Average velocity
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit Km/h Equation Mean (velocity)
Comments
REQUIRED MEASURES Speed_CAN OR Speed_GPS OR Speed_WheelUnitDistance
Related Sites Italian / Swedish
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_19 Name Time of idling

Category Driver behaviour Group N/A
Description
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit s Equation
Comments
REQUIRED MEASURES TIME
Related Sites Swedish

5/14/2019 Page 89 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_20 Name Frequency of 1st, 2nd and 5th gear usage
Category Driver behaviour Group N/A
Description
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit Number, % Equation
Comments
REQUIRED MEASURES GEAR
Related Sites Italian
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_21 Name Rate of use = number instructions followed

Rate of guidance instructions followed
Category Driver behaviour Group N/A
Description Percentage (or frequency) of use of the system
Percentage of intersections where instructions are given where driver takes the road advised
by the instruction.
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit % Equation Time system used
Comments Unlike driving style advice, guidance instructions are digital in nature: Either
the driver takes one road or the driver takes another road. This allows for
objective measurement. Non-compliance can be intentional (affecting user
acceptance) or unintentional (affecting driver behaviour).
REQUIRED MEASURES GUIDANCE_INSTRUCTIONS_FOLLOWED
Related Sites Italian / Austrian?¿?
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A

5/14/2019 Page 90 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

References Saad and al. (2006) AIDE project

PI_ID PI_22 Name Number of stop & go

Category Driver behaviour Group N/A
Description
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit number, % Equation
Comments
REQUIRED MEASURES NUMBER OF STOPS (&GO) SPEED
Related Sites Italian / Swedish
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_23 Name System switch off / System usage

Category User acceptance Group N/A
Description Indication of user acceptance through information about system usage
Subjective (Sub) / Objective (Obj) Obj Quantitative (QN) / Qualitative (QL) QN
Unit Bool Equation min(System_OnOff) >0
/s Total number of system disengagement events/time driven;
Mean (time)
Comments System is switched off by driver while driving  If the driver does not accept
or like the system it will be switched off
System usage: System is switched off ; time spent while system is switched off
 indication of user acceptance through information about system usage
REQUIRED MEASURES SWITCHED_OFF / SYSTEM_ONOFF
Related Sites Italian / Austrian /Swedish
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References

PI_ID PI_24 Name Ex-ante usefulness

Category User acceptance Group N/A

5/14/2019 Page 91 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Description Level of usefulness perceived by the user prior to usage

Subjective (Sub) / Objective (Obj) Sub Quantitative (QN) / Qualitative (QL) QL
Unit N/A Equation Self reported
Comments Strong dependencies on human/social sciences interpretations framework
REQUIRED MEASURES ACCEPT_QUEST (Questionnaire) / ACCEPT_INTERVIEW (Focus group interview)
Related Sites Italian / Austrian / Swedish
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References Nielsen,1993;Grudin, 1992;

PI_ID PI_25 Name Ex-post usefulness

Category User acceptance Group N/A
Description Self reported (questionnaire item)
Observed (rate of use)
Subjective (Sub) / Objective (Obj) Sub Quantitative (QN) / Qualitative (QL) QL / QN
Unit N/A Equation Self reported /
Comments Strong dependencies on human/social sciences interpretations framework
REQUIRED MEASURES ACCEPT_QUEST (Questionnaire) / ACCEPT_INTERVIEW (Focus group interview)
Related Sites Italian / Austrian / Swedish
OTHER INFORMATION
Required frequency Required accuracy Required resolution Need for synchronization
N/A N/A N/A N/A
ETHICAL ISSUES N/A
References Nielsen,1993;Grudin, 1992;

5/14/2019 Page 92 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

5/14/2019 Page 93 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

ANNEX 5. QUESTIONNAIRES
SYSTEM USABILITY QUESTIONNAIRE (SUS)

DRIVER ACCEPTANCE QUESTIONNAIRE

ANNEX 6. TEST CASE TEMPLATE

5/14/2019 Page 94 of 113 Version 1.3
 D5.1 COSMO Validation Test Plan

5/14/2019 Page 95 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

ANNEX 3. REQUIRED MEASURES TABLES

5/14/2019 Page 96 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

ME_ID ME_01 Name Speed_CAN

Group SPEED
Description Vehicle speed obtained from the CAN bus
Comments
Unit Km/h ?¿?
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs
Related Sites
Italian X Austrian Swedish X

ME_ID ME_02 Name Speed_GPS

Group SPEED
Description Vehicle speed obtained from a GPS-based sensor
Comments
Unit m/s
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs Sensor must have sky-visibility, thus not be mounted under metallic roofing.
Related Sites
Italian X Austrian Swedish ?¿?

ME_ID ME_03 Name Speed_WheelUnitDistance

Group SPEED
Description A pulse sensor on a free rolling wheel
Comments In order to get better accuracy in V and dV/dT compared to using ABS
Unit Accumulated number of pulses
MEASURE SOURCE
SENSOR X SE_ID SE_x Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs The tire rolling radius. Number of pulses/rotation
Related Sites
Italian X Austrian Swedish ?¿?

5/14/2019 Page 97 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

AND TrafficSituation_Artemis AND Model_Macro OR P98 (Driving Matrix) AND Model_Micro

ME_ID ME_04 Name Fuel_Consumption_1

Group ENGINE
Description
Comments Adjustment of density to the reference temperature
Unit cm3/s
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs Different alternatives: if data from CAN fuel is not directly measured. If high accurracy
needed there is need for a special fuel flow meter.
Related Sites
Italian X Austrian Swedish X

ME_ID ME_05 Name FUEL_USED_DIFFERENCE

Group ENGINE
Description Difference between amounts of fuel present in the vehicle at the start and at the end
of a trip, with eventually refill amounts added.
Comments
Unit l
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs Requires high-precision, calibrated in-tank fuel sensor.
Related Sites
Italian X Austrian Swedish

ME_ID ME_06 Name FUEL_USED_INTEGRATED

Group ENGINE
Description The instantaneous fuel usage as report from engine management or from sensors,
integrated over the duration of a trip.
Comments
Unit l
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?

5/14/2019 Page 98 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

QUESTIONNAIRE QT-ID Name

Special needs Requires calibration of the amount of fuel injected or a separate fuel consumption
sensor.
Related Sites
Italian X Austrian Swedish

FUEL_USED_CALCULATED OR FUEL_USED_EXPECTED

TRIPMETER

ME_ID ME_07 Name WATTS

Group ENERGY
Description WATT/H consumed by an electronic device (streetlight)
Comments
Unit Watt/hour
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs N/A
Related Sites
Italian X Austrian Swedish

ME_ID ME_08 Name TIME

Group
Description
Comments
Unit s
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs
Related Sites
Italian X Austrian Swedish

TOT PARKING with SUGGESTION AND TOT PARKING without SUGGESTION

(PI) Use of shuttle service  Required measured?¿?

5/14/2019 Page 99 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

ME_ID ME_09 Name ODOMETER

Group DISTANCE
Description The value of the odometer
Comments Used the calculate the distance driven between entering and existing the test site
Unit km
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs Needs to have high enough accuracy. If it does not, integrating SPEED is better.
Related Sites
Italian Austrian Swedish X

(PI) Black spots  Required measured?¿?

ME_ID ME_10 Name DISTANCE

Group DISTANCE
Description Distance travelled in the campus per ride
Comments the distance will be estimated according to the travel time; i.e. the time until (1) a
parking lot is found or (2) until the campus is left. For (1), some trials may be
performed in order to find typical values, alternatively an application may be generated
in which the driver quits a smart phone button ("leaving car").
Unit m/s2
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs
Related Sites
Italian X Austrian Swedish

ME_ID ME_11 Name PEDAL_GAS

Group PEDALS (VEHICLE CONTROLS)
Description Position of the gas pedal (0-100% depressed)
Comments
Unit %
MEASURE SOURCE
SENSOR X SE_ID SE_xx SENSOR X

5/14/2019 Page 100 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

QUESTIONNAIRE QT-ID QUESTIONNAIRE

Special needs
Related Sites
Italian X Austrian Italian X

ME_ID ME_12 Name PEDAL_BRAKE

Group PEDALS (VEHICLE CONTROLS)
Description Position of the brake pedal (0-100% depressed)
Comments
Unit %
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs
Related Sites
Italian X Austrian Swedish X

ME_ID ME_13 Name GEAR

Group GEAR (VEHICLE CONTROLS)
Description Current gear
Comments
Unit Bool
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs
Related Sites
Italian X Austrian Swedish X

ME_ID ME_14 Name GUIDANCE_INSTRUCTIONS_FOLLOWED

Group USEABILITY
Description Percentage of intersections with guidance instructions where driver takes advised road.
Comments Care must be taken to avoid mismeasurements due to uncertain map-matching right
after an intersection.
Unit %
MEASURE SOURCE

5/14/2019 Page 101 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

SENSOR X SE_ID SE_xx Name ?¿?

QUESTIONNAIRE QT-ID Name
Special needs Needs a software module for evaluation, either while driving or while evaluating
recorded data.
Related Sites
Italian X Austrian Swedish X

ME_ID ME_15 (A) Name SWITCHED_OFF

Group USEABILITY (SYSTEM RELATED)
Description At some point, the driver decides continue without using the system.
Comments This is a proxy for the opinion of the driver that the system is more annoying (or even
harmful) than useful.
Unit N/A
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs
Related Sites
Italian X Austrian X Swedish X

ME_ID ME_15 (B) Name SYSTEM_ONOFF

Group USEABILITY (SYSTEM RELATED)
Description Indicates whether the PROJECT system is active (System_OnOff = 1) or inactive
(System_OnOff = 0)
Comments
Unit N/A
MEASURE SOURCE
SENSOR X SE_ID SE_xx Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs
Related Sites
Italian X Austrian X Swedish X

ME_ID ME_16 Name ACCEPT_QUES

Group QUESTIONNAIRES
Description Acceptance questionnaire
Comments

5/14/2019 Page 102 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Unit N/A
MEASURE SOURCE
SENSOR SE_ID Name
QUESTIONNAIRE X QT-ID Name
Special needs
Related Sites
Italian X Austrian X Swedish X

ME_ID ME_17 Name ACCEPT_INTERVIEW

Group QUESTIONNAIRES
Description Acceptance questionnaire
Comments
Unit N/A
MEASURE SOURCE
SENSOR SE_ID Name
QUESTIONNAIRE X QT-ID Name
Special needs
Related Sites
Italian X Austrian X Swedish X

ME_ID ME_18 Name POSITION

Group POSITION
Description Time-stamped GPS-based position of the subject vehicle
Comments Latitude and Longitude of the vehicle
Unit WGS84 °
MEASURE SOURCE
SENSOR X SE_ID SE_x Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs The tire rolling radius. Number of pulses/rotation
Related Sites
Italian X Austrian Swedish X

ME_ID ME_19 Name TRAFFIC_LoS

Group TRAFFIC MEASURE
Description Level of Service observed in the test site area
Comments

5/14/2019 Page 103 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

Unit N/A
MEASURE SOURCE
SENSOR X SE_ID SE_x Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs
Related Sites
Italian X Austrian Swedish ?¿?

ME_ID ME_20 Name VEHICLES_PARKED

Group TRAFFIC MEASURE
Description Number of vehicles parked in an area
Comments
Unit N/A
MEASURE SOURCE
SENSOR X SE_ID SE_x Name ?¿?
QUESTIONNAIRE QT-ID Name
Special needs
Related Sites
Italian X Austrian Swedish ?¿?

5/14/2019 Page 104 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

5/14/2019 Page 105 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

ANNEX 7. ITALIAN TEST CASES

5/14/2019 Page 106 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

ANNEX 8. AUSTRIAN TEST CASES

5/14/2019 Page 107 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

ANNEX 10. CHECK-LIST FOR CONTROL OF LEGAL AND ETHICAL ASPECTS

1. Briefing of test participants/ Contractual agreements Status

1.1 Have all participants been provided with detailed information of the exact test
design and testing procedures? (This is essential to ensure that they have full
knowledge about the tests and as a basis for the Letter of Agreement),

1.2 Have participants been informed that they must always comply with traffic rules
when taking part in a driving test?

1.3 Does basic technical information exist for participants explaining system
limitations, that can lead to wrong use, and possible malfunctions with
instructions on how to deal with them? (This is important from an operative
point of view to ensure the best results in the tests).

1.4 Is there comprehensive information for participants clearly explaining the

implications for them relating to safe use of the on-board technology and legal
responsibilities? Have the consequences of certain behaviours been explained as
clarifying examples of these responsibilities? (This is important from legal and
safety point of view).

1.5 Is manual override foreseen in cases in which system failure or malfunction can
result in a safety risk? (e.g. intervening cooperative systems).

1.6 Have further legal implications for the researcher under tort law been
investigated related to the risks for participants derived from the use of systems
capable of interfering driving functions (as braking or steering)?

1.7 Have they been informed on the type and method of data to be logged?

1.8 Does a contract or “Letter of Agreement” establishing the relationship between

PROJECT project and test participants been signed by all participants? Does LoA
contain all the related relevant aspects, namely costs allocation, on-board
equipment or vehicles borrowing and return, insurances, responsibilities,
previous requirements for being a test participant?

2. Administrative issues Status

5/14/2019 Page 108 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

2.1 Is it clear for participants that they are legally responsible of every movement of
the vehicle when on-board experimental systems are fully overrideable (even in
the case of driving task being carried out by the system)?

2.2 In particular for experimental informing systems, is it clear that in case of

contradictory information between on-board systems and sign posted
information, the latter provide the only legally relevant information in the case
of legal issue is produced?

3. Data privacy Status

3.1 Are the EU Directive 95/46/EG, related to the minimum standard for data
protection, and specifically 2002/58/EG, related to privacy and electronic
telecommunication, known and understood in the context of PROJECT project?

3.2 Are anonymisation (no possible traceability of data) and use of pseudonyms
(modification of person name) considered in PROJECT as measures to assure
data privacy?

3.3 In order to ensure data privacy for third persons, have the participants given
written authorization to the project to record and track them during the tests?

3.4 Is it clear for researchers and participants that acquired data may only be used
only for the purpose(s) it has been collected/saved and that further
authorization is needed for other purposes?

3.5 Will it be necessary to collect sensitive data from people (ethnic, political
opinion, religious, health, personality or sex life)? If so, has the required special
treatment been taken into account?

3.6 Is the data acquisition limited by the principle of economy, that is, no more
personal data shall be collected and saved than really necessary?

3.7 What kind of technical or organisational measures are going to be taken in order
to preserve data privacy in PROJECT (access to data, authorized persons for data
operation, protection of data against read, copy, alter or delete, data input
control, data storage management, data availability control)?

5/14/2019 Page 109 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

3.8 In order to preserve privacy, has the separation of participants’ personal data
from the data obtained from the research been considered?

3.9 If re-identification of personal with research data is thought to be needed, has

the deposit of necessary data with a bearer of secrets (e.g. a lawyer) been
considered?

3.10 Since video recording is specially delicate in terms of data privacy (high quality
technology, easy access to internet, possibility of recording certain behaviours
during the test, etc.), have specific measures been taken for acquisition,
storage and end of life of this type of data to preserve privacy of participants?

3.11 Have participants been informed and explicitly authorized video recording
during the tests?

3.12 Do the systems and data logging have an off switch in order to avoid recording
drivers other than the test driver using the same test vehicle?

3.13 In case of third parties (passengers) in the test vehicle, is the on board camera
well on view, making it obvious that data is being recorded?

3.14 Have test participants been sensitised to inform passengers of the recording?

3.15 Do test participants know that, in the case of an accident, personal information
which has been recorded might be used to clarify responsibilities regarding
criminal prosecution, as it is not possible to bar the confiscation of data for this
purpose?

4. Insurance Status

4.1 Are national road traffic liability laws in countries of the PROJECT test sites well
known and understood by those responsible for the tests, keepers of the vehicles
and test participants?

4.2 Have all test vehicles the compulsory road traffic insurance?

4.3 Is it seen as necessary to require minimum insurance coverage for test vehicles
(vehicle, driver, passengers)?

5/14/2019 Page 110 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

4.4 Is it considered to be covered as a project cost within PROJECT funding?

4.5 Are test participants aware of general limitations of the insurance coverage?

4.6 Has a personal accident insurance been considered for passengers to cover
specific cases of accidents (hit and run, no insurance coverage of third party,
etc.)?

4.7 In case of third party insurance, is driver supplementary insurance being

considered as far as, is the only car occupant who may not be covered in all
accidents?

4.8 Have sums from insurance specially been considered in detail to know if
sufficient coverage in case of severe injuries, disabilities, etc., exists?

4.9 Do trials need a specific insurance according to the expected risks involved in
the testing?

4.10 Has an insurance been foreseen for test equipment (data acquisition devices,
experimental on-board applications , etc.).

4.11 Is it clear that any type of insurance known today assumes full driver’s control
and that special arrangements are needed for coverage in the case of non-
overrideable systems?

4.12 Specific insurance issues can be found when using cooperative systems in which
the control of a vehicle is dependent on other vehicles. Do the insurance
contracts of cooperative vehicles cover all possible damages between them and
towards surrounding traffic?

5. Licensing requirements Status

5.1 Have any modifications to vehicles foreseen in PROJECT been classified

according to their affect on vehicle licensing requirements and taken into
account for testing development?

5.2 Does PROJECT intend to use manufacturers´ test vehicles? Can they develop the
tests without modifying their operating license?

5/14/2019 Page 111 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

5.3 In the case of non approved systems, is some kind of expert report needed in
order for public authorities to accept the use of these systems during testing?

5.4 Has special licensing been considered in case of vehicles with non-overrideable
systems that do not ensure full control of the driver?

6. Ethical rules Status

6.1 Does PROJECT test and procedures reach a commitment to maximise potential
benefit (actions and personal attitudes aiming to obtain the best project results)
minimising possible risks (ensuring safety of persons and reducing possible
damage in equipment and other goods)?

6.3 Does the PROJECT test and procedures ensure the respect for persons and their
personality?

5/14/2019 Page 112 of 113 Version 1.3

 D5.1 COSMO Validation Test Plan

5/14/2019 Page 113 of 113 Version 1.3