THE DISPLAY OF UNCERTAINTY INFORMATION ON THE CONTROLLER

WORKING POSITION
Dirk Schaefer, Adrian Gizdavu, EUROCONTROL Experimental Centre, Brétigny, France
David Nicholls, RM Consultants Ltd, Abingdon, United Kingdom

Abstract today, and the purpose of Air Traffic Management

(ATM) can therefore be seen as one of managing
In air traffic control – and in many other
aircraft in an uncertain context. ATM is a complex
domains – decisions are frequently based on
socio-technical system with a great deal of potential
uncertain information, most obviously where they
for uncertainty. Most prominently, uncertainty is
involve predictions of future system states.
inherent in tools based on aircraft trajectory
Eliminating the uncertainty completely will not be
predictions that aim at supporting the controller
possible and may in some cases not be desirable, for
during conflict detection or resolution.
example where it restricts the flexibility of the
human operator to an unacceptable degree. One The sources of uncertainty are manifold,
might ask whether there are situations in which including:
quantifying and presenting information about the
• Inaccuracy and error in measurements and
inherent uncertainty might provide additional safety
display
and operational benefits and that, in fact, is the
• Incomplete knowledge about the state of
question the research presented here attempted to
the ATM system
address.
• Meteorological conditions
As part of a study under EUROCONTROL’s • Incomplete knowledge about the intentions
innovative research programme [19], existing of pilots and other controllers
implementations of two elements of the likely Variations in aircraft performance
future controller working position (CWP) were • Ambiguous information
complemented with a depiction of uncertainty. The • Subjective uncertainty of users – the degree
elements were the Vertical Assistance Window of belief or trust in the information they
(VAW), a tool that displays aircraft vertical flight receive, and in their own judgments or
paths, and the Medium Term Conflict Detection those of others.
(MTCD) system. The displays showed ‘uncertainty
bubbles’ in order to represent the predictions’ range The ultimate result is that the predicted future
as well as their mean. Experiments were conducted state of an aircraft involves a certain degree of
in order to investigate the usability of the tools and uncertainty. As new ATM concepts, with increased
the impact it may have on controller workload. reliance on predicted information, are introduced, a
The paper discusses the implementation of the substantial degree of uncertainty will, in the
uncertainty display and explains why certain foreseeable future, remain within the system. Yet,
choices were made. It presents the results of the uncertainty is a relatively unfamiliar concept in
experiment and points towards other potential ATM, and one can often hear remarks such as ‘but
implementations where uncertainty displays might we don’t have uncertainty in ATM’ or 'we don't
offer benefits. want to know about uncertainty – we can only work
with information that is certain’. Although
controllers are certainly aware of the limitations of
Introduction the information they use and the predictions based
If all flights ran exactly on time, if schedules upon them, our discussions have indicated that they
and routes could be designed to allow gate-to-gate do not generally think in terms of ‘uncertainty’, so
flights without any conflicts, and if there were no presenting such a parameter might require a careful
unforeseen events, there might be no need for any introduction.
active air traffic control. Clearly, this is not the case

0-7803-8539-X/04/$20.00 © 2004 IEEE

5.B.3-1
 To date, the ATM culture has largely managed track uncertainty is depicted in Figure 1 where the
uncertainty by seeking to eliminate it at its source, dotted lines identify less certain trajectory
developing ever more reliable and accurate tools. segments.
Uncertainty itself is not part of the information
Similar to the horizontal view, the aircraft
commonly presented on the controllers’ working
trajectory can be displayed in a vertical view. These
positions (CWP) and yet it might be argued that it
displays are less common in ATC, and controllers’
could potentially provide benefits to the air traffic
opinions tend to differ as of their usefulness, but the
controller, ultimately leading to increased safety in
uncertainty inherent in the aircraft’s vertical
air travel.
trajectory might be potentially useful, most
Displaying uncertainty information, i.e. an obviously where it involves climb or descent
indication of how certain and reliable the presented maneuvers.
information is, might lead to a better situation
awareness and thus a better basis for decision-
making; a better indication in case the system is
straying outside normal limits of operation; and a
more motivating working situation because system
predictions are more comprehensible.
On the other hand, displaying additional
information – and expecting the air traffic controller
to process it correctly – might result in a number of
unwanted side effects, including display clutter and
increased cognitive load. Further consequences
might be an increased overall uncertainty and
potential for confusion, because there is more scope
for differences in individual judgment.
Figure 1. Candidate Way of Displaying
In consequence, it seems that how and where Trajectory Uncertainty (Horizontal View)
to display uncertainty information is a question that
needs to be addressed with some prudence, In systems that support the controller during
balancing the advantages and disadvantages it the task of conflict detection, the certainty /
might offer. uncertainty of the detected conflict actually
happening can be calculated as a function of the
uncertainty inherent in the trajectory predictions of
Candidate Displays both aircraft. The uncertainty might be displayed in
Candidate display elements include the both the radar view of the CWP and in a dedicated
horizontal and vertical view of the trajectory, the conflict window.
display of conflict information on the radar view
display and / or in dedicated display items and more More futuristic approaches to ATM, such as
futuristic concepts such as multi-sector planning multi-sector planning (MSP), might benefit from
the display of uncertainty of the future traffic
The horizontal aspect of the aircraft’s situation to enable the operator to make decisions of
trajectory is displayed on the CWP radar view. a more strategic nature. Obviously, the timeline of
Whilst the along track error is likely to increase the prediction and thus the nature of the
with time since deviations from predicted along uncertainties inherent in the predictions are
track progress are not generally corrected for by different from those associated with the horizontal
current aircraft Flight Management Systems (FMS), and vertical conflict prediction views described
the cross track errors generally depend less on the above. This leads to an interesting question, namely
time, since they are predominantly due to when best to display uncertainty information.
inaccuracies in navigation (albeit limited by the Simplistically, we may think in terms of two
FMS) – or a pilot’s conscious decision to fly ‘off- opposing considerations:
track’. A candidate way of displaying the along-

5.B.3-2
• In general, as time progresses towards a Displaying Uncertainty on the
potential conflict, the uncertainty in the
Controller Working Position
situation decreases and the options available
for action become more limited. This would Various human-machine interface elements of
imply that uncertainty presentation is most the CWP that might offer the possibility to display
valuable at early times. uncertainty were studied and two were selected for
• On the other hand, we could consider the a candidate implementation:
problem as one in which decisions become • The Potential Problem Display (PPD) of the
less reversible as time progresses – so that it Medium Term Conflict Detection System
becomes more important to make the right (MTCD), which displays predicted conflicts
decision at late times. Then, if it is true that between pairs of aircraft together with the
good decisions require an awareness of the predicted time of loss of separation and the
uncertainty in the information on which they minimum distance of the conflict; and the
are based, uncertainty presentation could • Vertical Assistance Window (VAW), which
become more important at later times. displays the vertical, i.e. climb/descent
profile of a chosen flight.
The influence of time is however complex and
may be counter-intuitive and there are exceptions to The Potential Problem Display (PPD)
these simplistic general rules. For example, as MTCD supports the air traffic controller in the
pointed out in [8], if it is required to keep the task of detecting conflicts between two or more
probability of conflict below a given target, there is aircraft with a timeline of anywhere between 15 and
a limit on the earliest time of intervention, as well zero minutes into the future. Unlike the Short-Term
as on the more obvious limit on the latest time. At Conflict Alert (STCA) which detects conflicts up to
very early times (i.e. long look-ahead times) the two to three minutes into the future based on an
uncertainty is so great that no intervention is extrapolation of the aircrafts’ actual speed vector,
worthwhile. For the design of probabilistic, medium MTCD is based on a trajectory predictor and
term support tools, the former, early limit is the considers more complex information, such as the
more relevant constraint in practice. Observations aircrafts’ flight plans, entry and exit flight levels,
confirm this consideration: controllers are often and so forth.
reluctant to deal with conflicts too early simply
The potential conflicts identified by MTCD are
because the small likelihood of the conflict actually
brought to the controller’s attention via the
happening does not justify the action.
Potential Problem Display (PPD). PPD displays the
The present paper is mainly concerned with conflict along two axes, namely the time to closest
presentations of uncertainty in the vertical view and point of approach (CPA) and the minimum distance
on displays for presenting conflict information. The (distance at CPA). Each conflict is assigned a
presentation of uncertainty in the context of multi unique number – see Figure 2 for a schema.
sector planning was explored as another strand of
the overall research project [13, 19]. This
complemented the work described here by focusing
on the development and trial application of a user-
centered process for designing innovative
presentations [10], rather than the design and
evaluation of specific products – the potential
displays that are the subject of this paper.

Figure 2. Potential Problem Display (PPD) –

Schematic

5.B.3-3
 Intuitively, the conflict’s position in the PPD As can be seen in Figure 3 conflict number 2 is
would indicate urgency (vertically) and severity less uncertain, so the controller may decide to
(horizontally); the closer the conflict to the lower resolve conflict number 2 first. Note that the colors
left corner the more attention it would apparently depicting the conflict uncertainty range have been
require. Conflict number 1 would obviously be the chosen so as to avoid an intuitive misinterpretation:
first conflict the controller in this situation would one would be tempted to interpret the size of the
attempt to resolve. However, the display presents bubble as an indication of the severity of the
no information that would help determine whether conflict so that uncertain conflicts (large bubbles)
it would be beneficial to resolve conflict number 2 would appear more important to resolve. To balance
before conflict number 3 or vice versa. Which this effect, lighter colors were chosen for larger
option to choose might depend on the certainty with bubbles.
which the two conflicts might occur, arguing that
Advanced implementations such as the
the more unlikely a conflict the less rewarding to
EUROCONTROL MTCD [3], which has recently
address it at an early stage.
been subject to shadow mode field evaluations in
Figure 3 shows the PPD enhanced with three European control centers, include a number of
‘uncertainty bubbles’ displaying the distribution of other features, but owing to the experimental nature
the conflict reflecting the 95 percent range of the of the research presented here, the decision was
conflict prediction. Note that this distribution does made to implement the uncertainty displays within
not refer to the geographical area in which the a prototyping platform. Figure 4 depicts the PPD
conflict might happen but purely time to CPA and with uncertainty bubbles as implemented on the
distance at CPA – and thus does not have a very CWP in the context of this study.
intuitive counterpart.

Figure 3. Potential Problem Display (PPD) with Uncertainty Displayed – Schematic

Figure 4. Potential Problem Display (PPD) with Uncertainty ‘Bubbles’

5.B.3-4
The Vertical Assistance Window (VAW) Experimental Evaluation
The Vertical Assistance Window (VAW) The PPD and the VAW windows were
displays the vertical aspect of the aircraft trajectory implemented with uncertainty displays in a CWP
and can be invoked individually for each aircraft. prototyping environment in EUROCONTROL’s
Note that the VAW displays trajectories of Human Factors Lab. To allow for a comparisons,
individual aircraft only; while conflicts are marked the configuration of the tools permitted the
it does not explicitly serve the visualization of the uncertainty display to be either enabled or disabled.
vertical dimensions of conflicts identified by Experiments with a small number of test subjects
MTCD. were conducted in order to evaluate benefits and
Figure 5 displays the VAW as used in this usability of the uncertainty displays. Being aimed at
study. The aircraft’s vertical profile shows a small an evaluation in the concept phase and recognizing
and constant area of uncertainty for the flight the small number of test subjects, the experiments
segments without altitude changes. Substantial were predominantly based on qualitative rather than
uncertainty only applies to the segment with an quantitative measures, i.e. on the structured
altitude change, in this case a descent. The elicitation of subjective feedback from participants.
uncertainty area for the descent is bounded by the
early, steep descent and the late, shallow descent. Objectives
Uncertainties concerning the vertical path originate
The objectives of the experiments were:
from two basic sources: the point at which the
aircraft will initiate the climb / descent and the • To evaluate the potential benefits of
climb / descent rate. Apart from those, the rotation displaying uncertainty information on with
rate and different winds in different altitudes may Uncertainty Areas the controller working
determine the vertical profile. However, rather than position (CWP), more specifically the
adding to the substantial amount of work Vertical Assistance Windows (VAW) and
concerning the development of algorithms for the Potential Problem Display (PPD).
quantification of uncertainty [1, 7, 8, 9, 14], our • To obtain feedback that helps to improve
research addressed the benefits of presenting existing display concepts for the
uncertainty to the controller and to that end used a presentation of uncertainty information on
rather simplified uncertainty algorithm based on the CWP.
Gaussian distributions whose standard deviations
increased non-linearly with time. Method
Eight air traffic controllers, four active and
four retired, participated in the experiment. After a
general introduction about the concept of
uncertainty and the aims of the experiments and two
training exercises, each controller was presented
with four simulation exercises of 60 minutes
duration. The traffic samples involved two high
traffic (67 aircraft/hour) and two medium traffic (55
aircraft/hour) scenarios. The independent variables
were:
• Uncertainty display enabled / disabled
• Traffic high / medium.

The experimental design involved a within-

Figure 5. Vertical Assistance Windows (VAW) subject comparison of both independent variables.
The sequence was Latin-square counterbalanced to
avoid order effects.

5.B.3-5
 The controllers worked a standalone sector, Airspace
adjacent sectors were implemented as automatic The simulated airspace involved two existing
feed sectors. Controllers did not work in teams upper airspace sectors (FL245 - FL490) in northern
(each controller fulfilled both the Planning and France with a certain percentage of traffic arriving /
Executive Controller roles) and the experiment did departing from Charles-de-Gaulle airport (CDG).
not involve pseudo pilots. The two sectors were combined into one for the
Controllers’ workload was elicited using the sake of the exercise.
NASA Task Load Index. Post-exercise The separation minima were 5NM horizontally
questionnaires were administered after each run. and 1000 ft vertically. The airspace organization is
After the last run an end-of-day questionnaire was displayed in Figure 6.
administered. A semi-structured group debriefing
concluded the experiment.

Figure 6. Airspace Organization

Simulation Environment plus the MTCD PPD and the VAW. The VAW was
Figure 7 shows the experimental setup in the displayed on controller request whilst the PPD was
Human Factors Lab. The controllers worked a visible at all times. MTCD conflict information was
standalone sector, adjacent sectors were also depicted on the radar view display by means of
implemented as automatic feed sectors. Four runs red segments in the involved aircrafts’ trajectories.
were conducted in parallel. A Short Term Conflict Alert (STCA) tool was
available and configured so as to provide the
The controller working positions were controller with conflict information with a two
equipped with a 28” LCD monitor, keyboard and minute look-ahead time.
mouse. The CWP involved the radar view display

5.B.3-6
 Figure 8. VAW and PPD Evaluation
Questionnaire Results

Figure 7. Experimental Setup in the Human Workload

Factors Laboratory Five out of eight controllers stated that the
display of uncertainty did not increase their
Results workload. These results were identical for the PPD,
the VAW and the display of uncertainty in general.
Usability Figure 9 shows the results of NASA TLX;
Out of the sixteen items in the end-of-day even though the results do not provide significant
questionnaire, only five provided significant results. evidence other than the effect of traffic load on
Controllers tended to disagree significantly with the workload, the data seem to suggest an interaction
following statements: between uncertainty and traffic load to the end that
displaying uncertainty actually reduces workload
• ‘The display of uncertainty on the VAW under high traffic. This effect should be studied in
increased my situation awareness.’ more detail involving a greater number of
• ‘I would like a display of uncertainty on the participants.
VAW in my working environment.’
• ‘I would like a display of uncertainty on the
PPD in my working environment.’
• ‘Overall, the display of uncertainty helped
me to make better decisions.’
• ‘I would like a display of uncertainty in my
working environment.’

Figure 8 gives an overview of some of the

questions specific to the PPD and the VAW. As can
be seen, the results are slightly more favorable for
the PPD than for the VAW.

Figure 9. NASA TLX Results

5.B.3-7
Debriefing Results Making) elements of an airport. The reason for
Controllers agreed that the display of suggesting this was that aircraft still at the airport
uncertainty on the PPD might enable them to have a considerable effect on the future workload
prioritize conflicts and might increase their situation and complexity in the control sectors and,
awareness – even though this statement seems to accordingly, need to be taken into consideration for
contradict an opinion expressed in the their calculation. This comment referred more to the
questionnaires. They suggested that the changing calculation of traffic complexity but is interesting in
color was a useful feature. They also stated that the as far as the influence of the uncertainty of the
display did not increase their workload. departure time is mentioned as an important
contributor to complexity and workload.
Controllers expressed the opinion that the
VAW was not helpful since it required extra effort A number of controllers mentioned that the
to invoke the VAW window. Rather than display of uncertainty might be useful for planning
contributing to situation awareness, the display tools such as multi-sector planning. Ground control
might even cause additional workload. In addition, was also mentioned as a potential application of
since it referred to individual aircraft only, the uncertainty displays.
VAW was not considered a tool very suitable for
the detection and resolution of conflicts and hence Critique of the Results
the display of uncertainty was not considered a very Some limitations of the experiments need to be
useful feature. taken into consideration when interpreting the
Controllers mentioned that ‘the concept of results presented here:
uncertainty seems rather strange to them’. They
• About half of the controllers were unfamiliar
estimated that, altogether, the tools as presented did
with Medium Term Conflict Detection and
not have a beneficial effect on their work.
the PPD. More often than not, comments
The participants tended to agree that the related to these tools in general, not
display of uncertainty would be potentially more specifically to the uncertainty displays that
useful for the Planning Controller than for the this study attempted to assess.
Executive Controller. • Controllers were equally unfamiliar with the
Vertical Assistance Window and many
Controller Suggestions comments referred to the usability and
usefulness of this tool in general.
Controllers generally mentioned that the • There were few occasions in which a
various support tools could be better integrated. conflict was seen against the VAW
Even though this comment refers to the tools uncertainty area, and hence not many
themselves rather than the uncertainty display, it opportunities to use this display.
holds interesting ideas for improvements. The
• Furthermore one specific technical problem
suggestion was made that the conflict dot in the
with VAW impeded the correct display of
radar label – which is a feature of MTCD – could
the vertical flight profile in some cases, and
be displayed in a color reflecting the uncertainty of
this may have contributed to a general
the conflict, similar to the color of the conflict in
negative opinion about the VAW.
the PPD. In the same context, controllers agreed
• The number of participants and the
that there is no need to have the uncertainty area
prototyping nature of the experiments helps
depicted in great detail and that a small number of
to get some interesting feedback and
categories – such as certain, less certain, uncertain –
indications but, in most cases, did not
might be sufficient to help prioritize conflicts.
provide statistically significant results.
Another comment referred to the integration of
arrival, departure, and enroute management tools; in Discussion
particular the uncertainty of the departure time in
The general opinion of most of the controllers
relation to the CDM (Collaborative Decision
participating in the experiments was that the display

5.B.3-8
of uncertainty on the CWP, at least in the way Acknowledgements
implemented in this study, provides very little
This work was sponsored in the context of the
benefit. Even though the experimental setup might
program Co-operative Actions of R&D in
partly have contributed to certain reservations, it
EUROCONTROL (CARE) Innovative Actions.
needs to be considered whether the presented
concepts might need to be modified. We wish to thank Mick van Gool, Poul-Eric
Jorgensen, and Seppo Kauppinen for their support
Some important lessons have been learned in
and insights into a complex matter. Our thanks are
the course of the study which could have
also due to the controllers who participated in the
applications beyond the case study tools. The key
experimental evaluation in Brétigny.
guidance points can be summarized as follows:
• Controllers expressed a preference for
integrated displays and tools. Even though References
this comment applies to the CWP as a whole [1] Bakker, GJ, HJ Kremer and HAP Blom, 2000,
it has implications for the display of Geometric and Probabilistic Approaches towards
rd
uncertainty, namely that displaying Conflict Prediction, 3 USA/ Europe ATM R&D
uncertainty on the main CWP window may Seminar, Napoli.
have advantages over displaying it on
secondary windows. [2] EUROCONTROL CARE website,
• The advantage of displaying uncertainty as a www.eurocontrol.int/care/innovative/projects2002/c
potentially useful source of additional s/
information stands against the additional [3] EUROCONTROL, Operational Requirements
workload which processing this information for EATCHIP Phase III ATM Added Functions
causes. It is therefore desirable to limit the Volume 5 – Medium Term Conflict Detection -
complexity of both the concept and the OPR.ET1. ST04. DEL01.5 Edition 2,
display of uncertainty. It might be EUROCONTROL, Brussels, Belgium.
advantageous to present a small number of
‘uncertainty levels ’ rather than the more [4] Gizdavu, A, 2003, Uncertainty 2003 Real-Time
explicit graphical representations assessed Simulation Controller’s Information Book,
here. EUROCONTROL Experimental Centre, Brétigny,
• Careful consideration of the time dimension France.
is required; however, the usefulness of
[5] Gizdavu, A, 2003. Uncertainty 2003 Real-Time
uncertainty as source of information was
Simulation Facility Specification,
generally believed to be greater for working
positions with a more strategic orientation, EUROCONTROL Experimental Centre, Brétigny,
such as Planning Controller of Multi-Sector France.
Planner, than for the more ‘immediate’ work
[6] Hansman, RJ & HJ Davison, 2000. The Effect
of the Executive Controller
of Shared Information on Pilot/ Controller and
• There was a clear preference for uncertainty rd
to be presented only on demand – although Controller/ Controller Interactions. 3 USA/ Europe
this has to be balanced against the general ATM R&D Seminar, Napoli.
principle of HMI design that information [7] Irvine R, 2001, A Geometrical Approach to
should be consistently presented to all users, th
and the potential advantages of continuous Conflict Probability Estimation, 4 USA/ Europe
presentation of uncertainty in providing a ATM R&D Seminar, Santa Fe. Also published in
more complete picture and better situation Air Traffic Control Quarterly, Vol. 10, No. 2 pp85-
awareness 113.
[8] Irvine R, 2003, Lateral Conflict Resolution in
the Medium Term - achieving a Required

5.B.3-9
 th
Probability of Conflict Estimation, 5 USA/ Europe to Air Traffic Controllers - A Human Factors
ATM R&D Seminar, Budapest Perspective, In: Harris, Duffy, Smith, Stephanidis
(Eds.) Human Centred Computing, Vol. 3,
[9] Mondolini S, 2002, Common Trajectory Lawrence Erlbaum.
Modeling for Air Traffic Management Decision
Support Tools. ICAS Conference, September 2002, [16] Schaefer D, Meckiff C Magill, A Pirard B &
Toronto. Aligne, F, 2001, Air Traffic Complexity as a Key
Concept for Multi-Sector Planning, Digital Avionics
[10] Marti P & C Moderini, 2003, A Medieval Systems Conference, Daytona Beach, FL, Oct.
Triptych, Cameos and Books, Innovative Concepts 2001.
th
for future ATM scenarios, 5 USA/ Europe ATM [17] Sorrentino RM & Roney CJR, 1999, The
R&D Seminar, Budapest Uncertain Mind, London, Psychology Press, Taylor
[11] Meckiff C, R Chone and J-P Nicolaon, 1998, & Francis.
The Tactical Load Smoother for Multi-Sector [18] Van Doorn B, B Bakker & C Meckiff, 2000,
nd
Planning, 2 USA/ Europe ATM R&D Seminar, Evaluation of Advanced Conflict Modelling in the
rd
Orlando. Highly Interactive Problem Solver, 3 USA/ Europe
[12] Nicholls DB, P Battino, P Marti and S Pozzi, ATM R&D Seminar, Napoli.
2003, Presenting Uncertainty to Controllers and [19] Van Gool M, 2001. CARE Innovative
Pilots, 5th USA/ Europe ATM R&D Seminar, rd

Budapest. Research and External Technologies. 3 ATM R&D

Symposium. Madri, EUROCAE.
[13] Nicholls DB, 2003, Presenting Uncertainty to
Controllers & Pilots, Project Report CARE-
Innovative-RMC-T4-D2-1.0 Available via the Email Addresses
EUROCONTROL CARE website. Dirk Schaefer, [email protected]
[14] Paelli RA & H Erzberger, Conflict Probability David Nicholls, david.nicholls@rm-
Estimation Generalised to Non-Level Flight, Air consultants.co.uk
Traffic Control Quarterly, Vol. 7, Number 3, 1999,
Adrian Gizdavu,
pp.195 - 222.
[email protected]
[15] Schaefer D, M Flynn & G Skraaning, 2003,
The Presentation of Conflict Resolution Advisories

5.B.3-10