Avionics-Display

HMD, Data Fusion, CAOD, IKBS

BY
KAMLESH KUMAR
D E PA R T M E N T O F A E R O N A U T I C A L
MIT MANIPAL
Highlighted Topics of Lecture Slides

Helmet Mounted Display

Data Fusion

Intelligent Displays Management

Computer Aided Optical Design

Control and Data Entry

New developments in display technology to improve the pilot-display
Interaction…..Continuing progress

1. Head Up Display-HUD

2. Helmet Mounted Display-HMD

3. Multi Function Colour Display-MFCD

4. Digitally generated colour moving map display

5. Synthetic pictorial imagery

6. Display management using intelligent knowledge

based system(IKBS) technology

7. Improved understanding of human factors and

involvement of human factors specialist from the initial
cockpit design

Back
Helmet Mounted Display-HMD

 Defining the Helmet-Mounted Display

 Classifying Visual Helmet-Mounted Display Designs

 The History of Helmet-Mounted Displays

 Advantages of Helmet-Mounted Displays

 Limitations and Disadvantages of Helmet-Mounted Displays

 Helmet-Mounted Display Applications

 Current and Future Helmet-Mounted Display Programs

 Helmet-Mounted Display

 Melzer and Moffitt (1997) describe an HMD as minimally consisting of

"an image source and collimating optics in a head mount."

 Helmet-mounted display (HMD) has been thought of as an optical/visual system.

Importance:-
 To understand the optical parameters involved in the design of
HMDs and the impact these parameters have on image quality.

 The characteristics of the sensor(s) that produce the visual

imagery.

 Advanced HMD designs include significant audio information.

 This requires the HMD designer to also consider auditory

factors such as noise attenuation and communication speech
intelligibility.
 The HMD can function as a ‘HUD on the helmet’ and provide the display
for an integrated night/poor visibility viewing system with flight and
weapon aiming symbology overlaying the projected image from the sensor.

 Although monocular HMDs have been built which are capable of displaying
all the information normally displayed on a HUD, there can be problems
with what is known as ‘monocular rivalry’.

 This is because the brain is trying to process different images from each eye
and rivalry can occur between the eye with a display and that without.

 The problems become more acute at night when the eye without the display
sees very little, and the effects have been experienced when night-flying
with a monocular system in a helicopter.

 It has been shown that a binocular (or biocular) system whereby the same
display is presented to both eyes is the only really satisfactory solution.

 Hence, current HMD designs are generally binocular systems.

 Miniature 0.5 inch diameter CRTs are currently used as the display
sources.

 The development of miniature, high resolution liquid crystal

displays (LCDs), however, has changed this situation.

 HMDs exploiting these devices are now being produced.

 Night viewing applications require the widest possible FOV.

 This is particularly important for helicopters operating at night at

very low level (e.g. 30 ft) flying ‘Nap of the Earth’ missions taking
every possible advantage of terrain screening.

 Binocular HMDs for helicopter applications have minimum FOVs of

40◦ and FOVs of over 50◦ have been produced.
 Fighter/strike aircraft HMDs generally have FOVs of 35 to 40◦, trading off FOV
for a lower weight optical system.

 (Helmet weight must be kept to a minimum because of the loads exerted on

the pilot’s head during high g manoeuvres.)

 A lower weight HMD system can be achieved by the use of a visor projection
system in conjunction with a high efficiency optical design.

 This allows a standard spherically curved aircrew visor to be used to carry out
the combiner and collimation function by the addition of a neutral density
reflection coating.

 The visor coating provides high display brightness whilst maintaining high real
world transmission (>70% can be achieved) with no coloration.
 Display accuracy is also insensitive to visor rotation because of the
spherical shape and partial raising of the visor is possible, for
example to carry out the ‘Valsalva procedure’ (pinching the nostrils
and blowing to equalize the pressure in the ears).

 The combiner and collimation function is thus achieved without

adding additional weight to the optical system as the visor is
essential anyway.

 FOV’s of over 40◦ can be achieved with a binocular visor projection

system; the visor, however, is a long way from the eyes and creating
an extreme FOV is increasingly difficult over 60◦.

 The alternative eyepiece systems can give a larger FOV but at very
high weight penalties.
 A binocular HMD is inherently a better solution for night/poor visibility
operations, there is a large market for a low cost, light weight HMD for
fighter aircraft which are primarily involved in daytime operations to
provide a helmet mounted cueing and sighting system.

 The lower weight and cost achievable with a visor projected monocular
HMD can make this a cost effective solution in such applications.

 Although configured for daytime use, such a system is capable of

displaying video from a sensor, providing the pilot with an enhanced
cueing system after dark or in bad weather.

 Use of image intensification devices as these provide a night vision

capability which essentially complements an infrared viewing system;
each system being better in some conditions than the other.
 The image from the Image Intensifier Tube (IIT) is a
phosphor screeen which emits green light in the centre of the
visual band where the eye is most sensitive.

Helicopter HMD with integrated NVGs (by courtesy of BAE Systems).

This HMD forms part of the mission systems avionics for the Tiger 2
attack helicopter for the German army.
Eurofighter Typhoon binocular visor projected HMD (by courtesy of BAE
Systems).

Right side illustration shows optical ray trace. Displays images are
collimated by reflection from the spherical visor. Note use of ‘brow’
mirror to relay display images.
 Night Viewing Goggles-NVG

 NVGs basically comprise two image intensifiers through which the

observer views the scene at night by means of a suitable optical
arrangement.

 Image intensifiers are highly sensitive detectors which operate in

the red end of the visible spectrum amplifying light about 40,000
times from less than starlight up to visible level.

 An integrated helmet design with the NVGs located near the

natural pivot point of the head so that they do not create
significant out of balance moments avoids this problem.

 NVGs can be incorporated into HMDs, and optically combined

with the CRT displays so that the FLIR video display overlays the
image intensifier display on an accurate one to one basis, enabling
the two systems to be combined in a complementary manner.
 The basic optical mixing of IIT and CRT imagery.

 The image intensifier devices can be battery powered from a small

battery carried by the pilot.

 This enables the pilot to see in pitch darkness conditions on the ground
and walk out to the aircraft/helicopter without any lights being required
during covert operations.
 A binocular HMD system with integrated IITs which are optically combined
with the CRT displays .

 In the case of fighter/strike aircraft, the weight and bulk penalty of optically
combining the IITs with the display optics has led to the adoption of image
intensification devices which image on to small CCD cameras.

 The output of the image intensifier CCD camera is a video signal which is fed
back to the remote display drive electronics where it is electronically
combined with the symbology and displayed on the helmet mounted CRTs.
 The concept of electronic combination of IIT and CRT.

 The advantage of this approach is that it maximizes the performance of

the CRT display by removing the need to mix IIT and CRT imagery
optically.
 The intensified imagery is now presented in a raster format, allowing the
potential to enhance the night vision image, improving image contrast and
reducing some of the unsatisfactory characteristics of directly viewed IITs,
such as image blooming.
 The ability to enhance the image contrast electronically allows the user
to see undulations in terrain, such as featureless desert not visible using
conventional NVGs.

 Similarly, a bright point source of light viewed by a conventional NVG,

results in a halo effect that blots out the surrounding scene.

 This effect can be eliminated, allowing the user to view clearly a scene
containing bright sources of light, such as street lights.
 Four major elements:

 Image source (and associated

drive electronics),

 Display optics,

 Helmet, and

 Head/eye tracker.
 The image source is a display device upon which sensor imagery is
reproduced.

 Early on, these sources were miniature cathode-ray-tubes (CRTs) or image

intensification (I2) tubes.

 More recently, miniature flat panel display technologies have provided

alternate choices.

 The display optics is used to couple the display imagery to the eye. The
optics unit generally magnifies and focuses the display image.

 The helmet, while providing the protection for which it was designed
originally, also now serves as a platform for mounting the image source
and display optics.

 The tracking system couples the head orientation or line-of-sight with that
of the pilotage sensor(s) and weapons.
Manning and Rash (2007) provide a more generalized description of visual HMDs
that is applicable to both military and commercial applications,

where the name “head-worn displays” (HWDs) has been gaining acceptance.

 Basic four building blocks are same, but are expanded in scope:

 A mounting platform,

 An image source

 Relay optics,

 A head-tracker,
 A mounting platform, which can be as simple as a headband or as
sophisticated as a full flight helmet.

In addition to serving as an attachment point, it must provide the stability

to maintain the critical alignment between the user’s eyes and the HWD
viewing optics;

 An image source for generating the information imagery that is optically

presented to the user’s eyes.

Advances in miniature displays have produced a wide selection of small,

lightweight and low power choices at moderate cost, while meeting the
demands of perceptual intensity and resolution
 Relay optics, which transfer to the eye(s) the information at the image
source.
 Relay optics typically consists of a sequence of optical elements (mostly
lenses) that terminates with a beam-splitter (combiner).

 Initial designs for visual applications were monocular with a single beam-
splitter in front of one eye, but as miniature display technologies develop,
binocular designs are becoming dominant; and,
 A head-tracker, which is optional if the HWD is used only to
present status information using non-spatially- referenced symbols.

 However, it often is required if external (outside) imagery is

supplied by a sensor or a synthetic database.

 If such imagery is to be presented, the user’s directional line-of

sight must be recalculated continuously (updated) and used to
point the sensor or to select the synthetic imagery data correlated
with the user’s line-of-sight.

 Presentation of head-referenced information (imagery and/or

symbology) via a head tracker requires a preflight calibration
procedure called bore-sighting, which aligns the sensor’s and user’s
lines-of-sight.
Data Fusion

Back
Data Fusion

 Data fusion is the name given to the process of combining the data
from a number of different sources to provide information which is
not present in the individual sources.

For example,
 A synthetic 3D picture can be derived of the terrain in front of the
aircraft from an accurate terrain data base and accurate information
on the aircraft’s position and attitude.

 The aircraft’s position and attitude information are provided by a GPS

and an INS, or a TRN system(Terrain Reference Navigation ) and an
INS.

 The synthetic picture of the terrain can be overlaid one to one with
the outside scene derived from a FLIR sensor and displayed on a
HUD or a head down display.
 Ground features which may be hard to detect on the FLIR such as
ridge lines can be accentuated together with electricity pylons.

 This enables the pilot to continue visual flight in conditions of

marginal visibility where normally it would be necessary to pull up
and fly at a higher altitude.

 Display enhanced by data fusion as below

 An enhanced vision display and

 A synthetic vision display (by courtesy of BAE Systems).
HUD display while carrying out an erroneous descent in a mountainous scenario.

An enhanced vision display

Synthetic vision display

Alternative ways of displaying primary flight information and guidance

‘3D or 4D’ displays or ‘pathways in the sky’.

 These displays provide a pictorial presentation of the aircraft’s spatial

situation and flight path using synthetically generated ground imagery,
and exploit data fusion.

 Presentation of primary flight

information on colour head down displays-
Classical ‘T’ layout

 The objective of these new pictorial displays is to provide the pilot with a
more intuitive and natural appreciation of the situation in terms of the
aircraft’s state, its actual flight path, and the desired flight path with
respect to the outside world.
Intelligent Displays Management

Back
 There is, however, a big difference between demonstrating an
experimental system and the development of a production system
approved by the airworthiness authorities.

 The costs involved in achieving certification by the civil airworthiness

authorities is a significant sum;
the software is safety critical and the flight trials to prove the system
do not come cheaply.

We could, nevertheless, have such displays ‘tomorrow’ if the need was

agreed.
Intelligent Displays Management

 The exploitation of intelligent knowledge based systems (IKBS) technology,

frequently referred to as ‘expert systems’.

 This systems used to assist the pilot in carrying out the mission

 This is the subject of a number of very active research programmes,

particularly in the United States.

 One of the US programmes is the ‘pilot’s associate program’ which aims to

aid the pilot of a single seat fighter/attack aircraft in a similar way to the
way in which the second crew member of a two crew aircraft assists the
pilot.

 The prime aim is to reduce the pilot work load in high work load situations.

 This is a major importance to the next generation of military aircraft as

these will use a single pilot to carry out the tasks which up to now have
required a pilot and a navigator/weapons systems officer.
 A subset of all the proposed expert systems on an aircraft is an intelligent
displays management system to manage the information which is visually
presented to the pilot in high work load situations.

Unexpected or uncontrollable that leads causes of an excessive work

load,

 The ‘bounce’ – interception by a counter attacking aircraft with very little

warning.

 Evasion of ground threat – SAM (surface–air missile).

 Bird strike when flying at low altitude.

 Engine failure.

 Weather abort or weather diversion emergency.

 A block diagram illustrating the concepts of an intelligent displays
management.

 The system comprises an ‘aircraft state expert’ which deduces ‘what is

happening’ from the aircraft data, pilot inputs, threat warning systems
and the mission plan by the application of an appropriate set of rules.

 The aircraft state expert in turn controls the ‘displays management

expert’ which determines the information displayed on the various
display surfaces:
HUD,
Map,
Head down displays or
HMD according to an appropriate set of
goals and priorities.
Computer Aided Optical Design

Back
 The introduction of the computer revolutionized the way lenses
(optical systems) are designed and evaluated.

 Today the process of designing lenses is more direct, much faster,

and infinitely easier.

 The recent advent of the personal computer and affordable optical

design software has extended this revolution.

 These latest advances allow more people to design lenses than ever
before.

 Practical Computer-Aided Lens Design tells how to design

and evaluate lenses using computers. The approach is general and
fundamental, and is not limited to one specific software program.
Need Software?

 ZEMAX® lens design program, - www.focus-software.com.

 CODE V® lens design program - Optical Research Associates

www.opticalres.com

 OSLO LT® www.sinopt.com -limits the user

to 10 surfaces.

 SYNOPSYS® lens design program (SYNthesis of OPtical SYStems) is a

powerful Windows lens design program- www.gwi.net/osd
What is optical design?

o Optical design is the process of describing the refracting or reflecting elements

in an optical system so that it meets a set of performance specifications.

o Typically, the performance specifications concern the imaging characteristics

of the system, such as the resolution, magnification, numerical aperture, and
field of view.

o Other requirements may include tolerances, size, weight, and cost. The result
of an optical design project is typically a prescription or database that lists the
materials and shapes of the optical elements required.

o A few optical systems that have a limited number of degrees of freedom can be
designed using automatic procedures.
o However, once the complexity of the system or the imaging
requirements reach even fairly modest levels, the art of optical design
becomes important, and all contests have shown that the best designs
are produced by the best designers.

o Although there has been much ballyhoo about global optimization of

optical systems, this does not mean that optical systems can be
designed by computers, without human intervention.
How does optical design software work?

 Optical design software is numerical optimization software that helps convert a

starting design for an optical system into a finished design that meets a given
performance specification.

 The performance specification is described by an error function, often called a

merit function.

 The design software computes changes to the system parameters, or variables,

that minimize the error function.

 Various optimization methods are used to perform the changes:

 The most widely used is the Damped-Least-Squares
method that involves computing the derivatives of the
error-function operands with respect to the variables.

 Other non-derivative, stochastic, and chaotic methods

are also used for special problems.
 Optical design process involves computing a function that describes optical
performance, optical design software is also used for simulation.

 The accuracy of the simulation depends on the particular performance

attribute being computed.

 Some characteristics, such as the magnification or field of view, are easily

and accurately computed, while others, such as the quality of a complex
image, are more difficult to model.

 It is important to realize that the requirements for simulation error functions

are different from those for design error functions.

 The former must be accurate, while the latter is only required to make a
relative quality judgment between one system and another.

 Lenses have traditionally been fabricated using methods that involve

grinding two surfaces together, which automatically generates a spherical
surface.
 Spherical surfaces, together with the fundamental laws of optics,
produce images that contain defects called aberrations.

 Optical design usually involves balancing contrary aberrations,

rather than removing them.

 New materials and aspheric fabrication techniques have expanded

the possibilities for optical design, but the general process remains
one of balancing rather than removing aberrations.

 None of the modern HUDs and HMDs, in fact, would have

been feasible without its use.

 Modern optical design software enables very sophisticated

(sounds better than complex) optical designs to be
optimised and the performance determined.
For example,
 ray tracing for an optical train including de-centred and tilted optical
elements, aspheric elements and holographic elements can be
carried out.

 An automatic optimisation program can be carried out iteratively

whereby the curvature of all the individual elements, their spacing,
tilt, de-centring, refractive index can be varied to seek an optimal
solution using ‘hill climbing’ methods.

 This enables the optical designer to evaluate a very wide range of

configurations to achieve the performance goals.

 These programs take several hours to run even on the most modern
main frame computers.
 Computer generated ray trace for a wide FOV holographic HUD as an
example of the optical complexity involved in such designs.
As shown below:-

 It also allows quick feasibility studies of a range of optical

configurations to be run.

 There are, of course, many more optical analysis functions which can
be carried out using modern optical design software.
 Datum – An aircraft datum line is an imaginary vertical plane or line from
which all horizontal measurements of arm are taken, generally for balance
purposes. Once the datum has been selected, all moment arms and the
location of CG range are measured from this point and the weight and
balance may be calculated.

 Terrain Reference Navigation (TRN)-Terrain elevation data can be

exploited to achieve automatic micro-navigation and covert terrain-
following of the necessary high integrity.

 The micro-navigation employs modern radar altimeters incorporating

variable output power and spread spectrum techniques to provide low
probability of detection.
Synthetic Vision System (SVS)

 It is a computer-mediated reality system for aerial vehicles, that uses 3D

to provide pilots with clear and intuitive means of understanding their
flying environment.

 Synthetic Vision provides situational awareness to the operators by using

terrain, obstacle, geo-political, hydrological and other databases.

 A typical SVS application uses a set of databases stored on board the

aircraft, an image generator computer, and a display.

 Navigation solution is obtained through the use of GPS and Intertial

Reference Systems.
Enhanced Vision Display:-

Enhanced Vision is a related technology which incorporates information

from aircraft based sensors (e.g., near-infrared cameras, millimeter wave
radar) to provide vision in limited visibility environments.

Night vision systems have been available to pilots of military aircraft for
many years.

More recently business jets have added similar capabilities to aircraft to

enhance pilot situational awareness in poor visibility due to weather or
haze, and at night.

“It turns night into day,” Churchville said from the backseat of the 182. “It
makes night flying much easier.”

The systems use infrared sensors, signal processing and advanced cockpit
displays to piece together the display that shows up on the screen.
Ridgelines- The term ridgeline is used to describe a visually prominent
and narrow piece of land that includes the highest points of elevation
within a watershed
Read Reference, Think and Explorer More Knowledge

Thanks