14th International Conference on Information Fusion

Chicago, Illinois, USA, July 5-8, 2011

Improving Maritime Situational Awareness by Fusing

Sensor Information and Intelligence
A.C. van den Broek, R.M. Neef, P. Hanckmann, S.P. van Gosliga, D. van Halsema
TNO
The Hague, The Netherlands
[email protected]

requirements. Next, we describe the fusion process, before

Abstract - In present-day military security operations
we elaborate on fusion methods which can be applied.
threats are more difficult to reveal than in conventional
Finally we discuss a system architecture and simulation
warfare theatres, since they take place during the course
environment for testing the proposed methods and give
of normal life. These maritime missions often take place
conclusions.
in littoral environments, where acts of piracy, drug
trafficking and other threatening events become
obscured in the crowd of everyday fisheries, cargo 2 Operational context
traders, ferries and pleasure cruises, hindering situation
Security operations are often characterized by controlling
awareness. We aim to improve situation awareness and
large areas with a limited number of assets. An example is
threat detection capabilities in maritime scenarios by
the anti-piracy operation in the Gulf of Aden where the
combining sensor-based information with context
operational area extends over thousands of sea miles. One
information and intelligence from various sources. The
of the main operational tasks is to direct assets timely to
fusion and analysis in order to reveal suspect from
the right position. For the command and control process a
normal behavior is based on domain ontologies. A test
common operational picture (COP) is the basis on which
bed allows the study of various exploitation and
decisions and actions are taken. A core part of the COP is
assessments techniques applied to these domain
ontologies. Using an appropriate scenario we have a maritime picture that contains information about vessel
simulated suspect and normal behaviour to test the movements in the complex and detailed coastal
applicability of the various techniques. environment and information about hostile intent and
illegal activity of the vessels. Ideally the COP contains up
to date information about the position of own, enemy and
Keywords: Situation awareness, Intelligence, Sensors, third entities combined with their missions, intentions, and
Threat assessment, Decision support. capabilities.
In present-day military security operations threats are
1 Introduction more difficult to reveal than in conventional warfare
theatres, since they take place during the course of normal
Because of global economic and socio-political changes, life. For example, during maritime missions in littoral
an increase of conflicts near the world's coastlines is environments, acts of piracy, drug trafficking and other
anticipated. The littoral zone is characterized by intense threatening events become obscured in the crowd of
regular vessel traffic. The conduct of Maritime Security everyday fisheries, cargo traders, ferries and pleasure
Operations and Peace support Operations means that cruises. The hostile intent of objects is therefore not
navies have to control instead of dominate the sea, thus always easy to determine because of its ability to cloak and
allowing regular vessel traffic in the area of operations, hide among the regular vessel traffic.
and act against irregular adversaries who nevertheless also
can possess military armaments. In this combined To enable threat recognition appropriate situational
military/non-military setting of operations, naval forces awareness is needed which implies recognition of the
have to protect themselves against threats from land, air objects present in the scene, their interaction with the
and sea, while they continuously have to collect various environment and their intention on basis of threat
data to ensure information superiority over their hypotheses in order to foresee the situation in the near
adversaries and thirds. For the purpose to achieve future. In this way hostile intentions and threats should be
information superiority a research program at TNO, the recognized in time so that timely decisions and counter
Netherlands, has started aiming at improving maritime actions can be taken. In other words the COP should
situation awareness. The study for this improvement contain sufficient actionable information to be retrieved on
focuses on the combined use of intelligence sources and demand. To achieve situation awareness in a wide area,
sensor information. In this paper we first discuss the persistent surveillance, background intelligence, and multi-
operational context and tasks, and define the information source data analysis are required.

978-0-9824438-3-5 ©2011 ISIF 1293

3 Picture compilation and situation approaching with high speed in case of a terrorist attack.
In order to recognize the threat scenario, patterns of
awareness indicators, in which the threat is expressed, should be
Picture compilation implies the collection of data about ‘revealed’ by the observables, which are elements of
activities in the littoral followed by sense making of the information suitable for describing the current situation
data. Sense making can either be done by defining normal and which can be directly derived from measurements
activities and searching for abnormal patterns or by performed by sensors. Sensor measurements yield data
recognizing signatures of normal and threatening about the position, the characterization of the target, the
processes. This results in the so-called Recognized track behavior, AIS information and the observation time.
Maritime Picture (RMP) which is defined as a composite
picture of activities over a maritime area of interest. It 4 Procedures for achieving situation
contains tracks of vessels which have been evaluated with
respect to the activity of the vessels. Analyses of the
awareness
recognized processes in the RMP allow forecasting about Situation and threat awareness should be achieved by
future activities (i.e. situation awareness) including combining information describing the current situation
possible threatening activities. This information is input expressed by observables, with signatures of threatening
for the COP that contains actionable information for and normal processes expressed by threat indicators (see
making decisions. In addition to the situational previous section). In our case the vessel is the object of
information from the RMP, the COP therefore contains all interest in these processes and the requested information
other information necessary for the decision making such for the RMP is the vessel mission: trade or fishery etc
as the position of own assets, capabilities and other (normal process) or smuggling, piracy etc (threatening
relevant geospatial information such sea lanes, harbors etc. process).

3.1 Threat indicators, observables and Observables involve statements which describe the
current situation for which persistent surveillance, i.e.
situation awareness continuous tracking and tracing of vessels with
We describe daily commercial or leisure activities in the observations systems, is a necessity. Therefore a broad
observed world, such as fishing, trade and pleasure suite of platforms equipped with sensors like, radar, AIS
cruising, as process patterns. Processes can be recognized receivers and EO/IR systems is needed. Platforms
by sequences of situations which can be revealed by so- comprise ships (both military as well as commercial),
called indicators. Threats are processes that may occur and UAVs, satellites and VTS (vessel tracking services)
which are obviously not wanted. Situation awareness and ground stations. Examples of observables are statements
threat awareness [1] implies here recognition of these about the size of vessels and ships (large, small), speed
processes before the unwanted situation has occurred. For (slow, high), and track behavior (loitering, stopped, and
example by producing a threat alert an operator and his continuously ahead).
supporting systems can become aware of an imminent Intelligence provides us with information describing the
unwanted situation. (i.e. the alarming situation, figure 1). current context with respect to the vessel mission. This
context consists of knowledge and information about the
geophysical and geopolitical world (long term),
information about current practices and trade activities
(middle term) and information about recent activities of
groups and persons and events (short term). Since the
request for information (vessel mission) for the RMP is
focused on the vessel, an object present in the spatio-
temporal world, this current context should also be
expressed in the same spatio-temporal world. For this
purpose we have chosen the concepts: time, position,
harbor and the vessel itself, for which intelligence should
provide us a priori information with respect the possible
vessel missions.
Figure 1. Diagram showing the relations between the scenarios Figure 2 depicts the combination process where
and the sensor observations. observables, indicators and mission a priori information
are to be combined to determine which vessel missions are
Examples of statements for threat indicators are: a cargo applicable for the vessel under surveillance. In het heart of
vesselheading to a harbor other then the destination in the the combination process, fusion methods are to be applied.
AIS message in the case of smuggling or a small vessel

1294
In the next section we discuss a number of possible fusion most likely intent of a vessel. The context is given by
methods. procedural and factual knowledge about the world, about
relevant objects and events, and their interrelations. The
context can be expressed by a context model, an
information model that contains relevant concepts and
their relations, and encapsulates background and
foreground knowledge required for understanding events.
For example, in our maritime application domain, the
context model should make it feasible to explain
observations about vessels by VTS stations, and interpret
messages from intelligence sources to explain their
importance to the current situation. To this end, the context
model needs to include all a priori knowledge necessary
for the proper situation and threat assessment. We create
such a context model via ontologies. We distinguish two
types of ontologies: content ontologies, and situation
ontologies [2], [3].
Content ontologies capture elements of interest in the
application domain, such as known types of vessels and
ports, and other domain-specific concepts. For most
Figure 2. Information flows for sensor data and for intelligence, applications, one can use existing published ontologies to
which have to be fused and correlated with the signatures of the realise parts of the context model, such as ontologies on
vessel missions.
commonsense knowledge (WordNet, OpenCyc),
geographic (GeoNames) and geopolitical knowledge, and
To build the picture and achieve situation awareness many others. For the other part, one will need to resort to
communication is essential through wireless broadband expert interviews and domain exploration to construct the
networks allowing for information exchange between required ontologies. Also, content ontologies serve to
assets and platforms. Data should be stored in databases in acquire data from information sources in a meaningful and
standardized formats so that members of coalition forces consistent manner. Each element in a data source should
are able to retrieve and process the required information. be related to one or more concepts in the overall domain
An example is the Coalition Shared Database (CSD) ontology, so that services can use information from
developed by NC3A in the NATO/MAJIIC project sources in a consistent manner. In this way data sources
(www.nato.int/docu/update/2007/pdf/majic.pdf). containing information about a vessel but in different
formats can be linked (e.g. databases using different ways
5 Information fusion and analysis to characterise the vessel type).

method Situation ontologies capture a situation or series of states

in the application space using concepts from the content
The fusion process requires that observables, indicators ontologies. For instance, one might create an ontology that
and mission a priori information and their interrelations are defines relevant geospatial vessel behaviours, such as the
represented in a meaningful manner and readily accessible position relative to fishing grounds, or temporal patterns,
to the system. We need an information model that captures such as arrival of a vessel at the scheduled time. We use
the context consisting of concepts and relationships that such situation descriptions to characterise interesting
are relevant in our application domain, and that ensures behaviour patterns, and use those to establish our threat
consistency and a common vocabulary across the system assessment process. The situation ontologies form the
components. constituents for our search patterns of interest, e.g. the
specific intents that we want the system to recognize. For
5.1 Using ontologies to capture the context example, a search pattern for smuggle might consist of a
sensibly connected set of elemental patterns, such as
In order to be able to discriminate between normal and rendez-vous with another vessel mid-sea or the vessel type
suspect and to compare various scenarios use of the does fit the current location and time. Colloquially, one
context is inevitable. The context is the set of facts or might say that the situation ontologies define the
circumstances that surround a situation or event. The observables and indicators to be used for intent
context provides the background against which we can recognition. See [3], [4], [5] for similar approaches.
explain observations, and from which we can infer the Figure 3 depicts the elements of the context model.

1295
 Figure 3. The elements of the context model. The content ontologies jointly form the domain ontology. Situation ontologies describe
relevant situational concepts using concepts from the domain ontology. The situation ontologies form the basis for the definition of
patterns of interest that may be used for threat assessment.

continuous process, which provides a basis for analysis of

5.2 Exploiting the context model the current situation as well as for the building-up prior
Situation recognition requires that relevant relationships knowledge for future situation analyses. In figure 4 we
between threat scenarios, threat indicators, observables and present as an example a linkage diagram showing a subset
mission a priori information are observed and assessed. of the semantic network covering both intelligence and
The content ontologies provide semantic relationships; the sensor data. The right side of the diagram expresses the
situation ontologies define patterns that express how these generic semantic relationships between concepts within the
semantic relationships lead to relevant indicators. Note that application domain (a priori defined knowledge). The left
the initial ontologies are merely schemas, not the data side of the diagram illustrates instantiated data (current
itself. During run-time, the content ontologies are information), in which observed vessel behaviour is related
instantiated by available data. Ontology instantiation is a to concepts that are relevant for threat assessment.

Figure 4. Linkage diagram derived from an underlying semantic network, with right the analysis in the intelligence domain and left the
analysis in the sensor domain. Bold indicates concepts for which mission a priori information is available.

1296
5.3 Assessment methods In practice for (military) security operations there may
not be enough data to instantiate the relations in the
The overall goal of the chain of processes is to provide ontologies to produce evidence and prior information with
alerts to vessel behaviours of interest. These behaviours sufficient certainty. In that case explicit results using the
can be expressed in patterns of interest, grounded in the HMF and therefore actionable information cannot be
context model. Basically, threat alerts can be produced by obtained.
mapping threat patterns on the available, well-structured
data. By exploiting the semantic relationships between Therefore it would be worthwhile to get insight which
data, we can conclude whether a threat alert should be indicators and which relations in the ontologies can be
issued. suitably instantiated. By selecting these indicators for use
in the HMF more explicit results and more decisive
Examples for assessment are probabilistic models such information may be obtained. To get insight in the
as Bayesian belief networks where prior information decisiveness of indicators, the nodes and links in the
(based on previous observations or expert knowledge) is ontologies should receive a value of importance. This
combined with the actual situation description (evidence) value is based on factors such as the prior knowledge, the
derived from the observations to produce the posterior information gain and entropy [10], [11].
probability of an hypothesis.
The Bayesian approach described above is appropriate
Typically a Bayesian belief network is based upon cause when the information obtained is uncertain which is often
and effect relationships [6]. However, for application true in complex situations where hostile intent has to be
domains in which the causal nature of events is a regular inferred in the midst of overwhelming normal activities.
subject of debate one may have to resort to an alternative When situations are less complex and threats are more
approach. The Hypothesis Management Framework explicit, the Bayesian techniques may be used in a more
(HMF) was developed to enable decision support in such straightforward way to obtain actionable information. An
domains [7]. example is given by the identification data combining
Effectively an HMF model is a Bayesian belief network process (IDCP) [12].
that complies with a strict design pattern. Rather than Another approach in less complex situations is the use of
pursuing a static model that represents the ‘true’ causality rules and decision trees for obtaining actionable
of the domain, an HMF model enables a flexible model information. In particular decision trees may be used, when
that is easier to keep up to date with changes in the indicators, which carry decisive information, can be
environment. In HMF competing hypotheses (in our case determined. To get an insight in the decisiveness,
hypotheses about suspect versus normal vessel missions) information entropy or information gain can be used.
are compared using sets of indicators. When an indicator is When sorting the data based on the information gain, the
observed the posterior probability of each hypothesis is more decisive indicators become visible. Decision trees
updated. The sets of hypotheses and indicators can be provide a possible representation of one or more
extended in a flexible way. Each indicator is independently hypothesis. Decision trees can also play a role in a triage
related to each of the hypotheses. Newly added indicators of the data to determine the importance of further
or hypotheses will therefore not affect the integrity of investigation. Consider the case many vessels need to be
existing prior knowledge in the model. assessed simultaneously. Based on a decision tree,
The HMF uses a so-called Naïve Bayesian Classifier decisions can be made which vessels need to be fully
topology. Such a topology has proven to be quite effective evaluated and which do not need further attention.
in getting good results [8]. Due to its structure it is also
inherently robust for imprecise prior knowledge [9]: each 6 Case study
indicator has a relatively modest influence on the
posteriors of hypotheses. HMF is therefore suitable to Above-mentioned procedures and techniques are to be
assess relations between the indicators and observables implemented in a situation awareness support system. For
defined by the situation ontologies and their instantiations, this purpose we have developed a dedicated test bed. In
where the latter provide both the current situation figure 5 we show the processing chain and functional flow
description (evidence) as well as prior knowledge for the of the system.
indicators in combination with the hypotheses.

1297
 Figure 5. Processing chain and functional flow. Colours indicate the relation with the information flows in figure 2.

The system consists of several databases and modules : from the CSD, such as messages may not always
1) The source database contains data collected by be structured in standardized ways and
the sensors, intelligence reports and other intelligence processing is in practice not
collected information. In practice the coalition automated. The output information is updated
shared database (CSD) developed in Majiic can when new information is becoming available and
fulfill this role. Within the coalition the CSD is stored in the Intel database. In this database the
comprises several databases in a distributed actual (a priori) probabilities about the vessel
network where metadata about the contents is mission are stored for the position, time, vessel
shared. On request the actual data can be and harbours. The intelligence processing also
retrieved. produces a so-called validated intelligence picture
2) Observable processing transforms the sensor data which can be used as a layer for the RMP and
and tracks to statements about the detected vessel. COP. The information about time and position
It typically produces statements about the (spatio-temporal intelligence) is input for the
behaviour of the vessel such as moves fast in a observable processing
strait way, stays near coast, is in traffic lane, etc. 4) In the analysis module the actual fusion and
Inputs for the observable processing are analysis takes place. The red box indicates the
geographical data about traffic lanes, coastal fusion and assessment module where the
lines, positions of harbours, anchorage areas etc. techniques discussed in the previous sections are
Observables are produced and updated at regular implemented. Input follows the three aspects
time intervals and processing is mostly depicted in figure 2: requests about the vessel
automated. These observables are becoming mission, the sensor information flow specified by
additional attributes to the tracks and are stored in the observables and the intelligence flow
the observable database. specified by the mission a priori information.
3) Intelligence processing aims at producing (a Output data are vessel mission assignments which
priori) probabilities for the vessel missions with are stored in the mission assignment database. In
respect to harbour, time, position and the vessel the update module the obtained mission
itself. Inputs are the requests about the vessel assignments are compared with previous mission
missions to be revealed and long term a priori assignments to resolve conflicts and to obtain
context information as well as knowledge from final results.
the physical world, historical data etc. Input data

1298
 Once an assessment for the normal and suspect vessel destabilizing groups in Troubledland. There also exists the
missions (threats) are obtained the results can be used in danger of terrorist attacks since Badland accommodates
the situation awareness display module and are combined fanatic groups whose goal is to expel the international
with other layers such as the validated intelligence image, peace keeping force.
mapping and meteo data for the RMP. Also, threat alerts
For the daily life scenario we adopted numerous vessels
can be produced, which can be used for the COP and for
present in the strait, each with ‘normal’ missions. These
decision support modules. Important item here is the
missions are defined as follows: between the two countries
traceability to the source database and information that
local trade and smuggling of small items such food,
caused the threat alerts, and access to other relevant
clothes, and consumer goods etc is ongoing. Also regular
information. The information is also made available for the
transport of persons by ferries exists. In the middle of the
intelligence processing module for future evaluations.
strait a shipping lane is running where large oil tankers and
6.1 Scenario & simulation cargo vessels (international trade) are passing. Fishing
occurs adjacent to the coastlines mixed with pleasure
To try out the system described above we have to supply yachts and cruise ships.
the system with a continuous data stream. Since in practice
data are incomplete we use simulated data. We use J- For each mission we have specified the type of vessel,
ROADS [13] for simulation of the entities, and sensor behaviour, availability of AIS, and AIS content. In the
modeling of the VTS stations and sensors on board of scenario we also defined information about groups,
military ships. individuals and their interactions, special locations and
various events in order to be able to analyze intelligence.
To build normal and suspect situations we use a maritime Information about the events is specified in messages, e.g.
scenario in a sea strait with dense trafficking, and suspect from harbors masters and other local agents. For
behaviour can be introduced (see figure 5). We have monitoring purposes several VTS radar/AIS stations are
adopted the following scenario for our case study. Two positioned along the coast. The UN mission comprises
neighbouring countries are separated by a narrow strait, three ships - two patrol vessels and one frigate - capable of
about 80 NM wide. In Troubledland a traditional monitoring the surrounding environment with their radars.
government resides which is supported by a privileged
minority of the population. In Badland a revolutionary Using the system, scenario and simulation we can
government is advocating worldwide revolution. It determine the applicability of the various techniques:
supports terrorism in general and specifically the ontologies, information certainty, HMF, decision trees &
destabilizing groups in Troubledland. Because of tensions rules. At the time of writing this paper we are producing
between the countries and conflicts in the past a UN peace results which have to be evaluated before conclusions can
keeping mission controls the traffic in the strait to prevent be drawn.
large scale smuggling of weapons from Badland to the

Figure 6. Schematic layout for the scenario

1299
 semantics with the simple event model (SEM).
7 Discussion and conclusions Proceedings of the 1st. ACM International Workshop on
Events in Multimedia, Sheridan Publishers.
We have introduced and described a situation awareness
support system focused on maritime security operations
where sensor information is fused with intelligence data. [6] Pearl, J., (2000), Causality: Models, Reasoning, and
Inference, Cambridge University Press.
The fusion and analysis of the data for revealing suspect
from normal behaviour is based on domain ontologies. A [7] Gosliga, S.P. van, Voorde, I. van de (2008),
test bed allows the study of various exploitation and Hypothesis Management Framework: a flexible design
assessments techniques of the domain ontologies. Using an pattern for belief networks in decision support systems".
appropriate scenario and simulation of suspect and normal Proceedings of the 6th Bayesian Modelling Applications
behaviour we are able to test the applicability of the Workshop at UAI 2008, Helsinki, Finland, July 2008.
various techniques.
In practice requests for on demand information and prior [8] Rish, I., (2001) An empirical study of the naive
information will change and the system should be Bayes classifier, IJCAI Workshop on Empirical Methods
adaptable. Attention may shift to different suspect in Artificial Intelligence, pp. 41–46.
behaviours, or new insights into the intent of vessels may
require the search patterns to be updated. This implies [9] Renooij, S., Gaag, L.C. van der, (2008), Evidence
changing sets of patterns of interest and changes in the and scenario sensitivities in naive Bayesian classifiers,
uncertainty propagation. This means that we need a International Journal of Approximate Reasoning, Volume
method to dynamically update search patterns, and 49, Issue 2, pp. 398-416.
subsequently, update the exploitation and assessment
methods to handle the new observables and indicators. [10] Calmet, J. and Daemi, A., (2004) From entropy to
ontology, in: Trappl R.: Proceedings of the Seventeenth
The introduction of additional observables and indicators European Meeting on Cybernetics and Systems Research
may also imply new concepts in the ontologies. For (EMCSR 2004), 13-16 April 2004
different environments and different threats the system
therefore has to be arranged or configured by human [11] Cho, M., Choi C., Kim, W., Park, J., Kim P., (2007)
operators. A way forward is to identify those parts of the Comparing Ontologies using Entropy, Proceedings of the
system which are suited for implementation of automatic 10th International Conference on Convergence
adaption techniques. Guideline here will be operational Information Technology (ICCIT), 27-29 December 2007
practice.
[12] Kruger, M., Ziegler, J., (2008), User-oriented
References Bayesian identification and its configuration, Proceedings
of the 11th International Conference on Information
[1] Endsley, M.R., (1995), Toward a Theory of Situation Fusion, Cologne, June 30- July 03, 2008
Awareness in Dynamic Systems, Human Factors 37(1), 32-
64 (1995). [13] Wiel, W., van der, (2006), J-ROADS Air Defence
Simulation Support during the 2006 JPOW IX Missile
[2] Llinas, J., Bowman, C., Rogova, G., Steinberg, A., Defence Exercise, NATO RTO, MSG-045-20
Waltz, E., & White, F. (2004). Revisiting the JDL Data
Fusion Model II. Paper presented at the 7th International
Conference on Information Fusion, Stockholm, Sweden.

[3] Kokar, M.M., Matheus, C. J., Baclawski, K. (2009).

Ontology-based situation awareness. Information Fusion,
volume 10, pp. 83-98.

[4] Baumgartner, N., Gottesheim, W., Mitsch, S.,

Retschitzegger, W., Schwinger, W. (2010). BeAware! -
Situation Awareness, the Ontology-Driven Way. Data and
Knowledge Engineering, Special Issue on Contribution of
Ontologies in Designing Advanced Information Systems,
volume 69 (11), Elsevier.

[5] Hage, W.R., Malaisé, V., Vries, G. de, Schreiber, G.,

Someren, M. (2009). Combining ship trajectories and

1300