TECHNISCHE UNIVERSITÄT MÜNCHEN

Lehrstuhl für Integrierte Systeme

A Subjective Logic Based Extensional Approach to

Non-Monotonic Reasoning under Uncertainty and its
Application to Visual Surveillance

Seunghan Han

Vollständiger Abdruck der von der Fakultät für Elektrotechnik und

Informationstechnik der Technischen Universität München zur Erlangung des
akademischen Grades eines

Doktors der Naturwissenschaften

genehmigten Dissertation.

Vorsitzender: Univ-Prof. Dr.-Ing. habil. G. Rigoll

Prüfer der Dissertation:
1. apl. Prof. Dr.-Ing.habil. W. Stechele
2. Univ-Prof. Dr. rer. nat. D. Cremers
3. Prof. Dr. rer. nat. E. Izquierdo
Queen Mary University of London, UK

Die Dissertation wurde am 29.09.2011 bei der Technischen Universität München

eingereicht und durch die Fakultät für Elektrotechnik und Informationstechnik am
27.06.2012 angenommen.
II
Acknowledgements

"He who walks with the wise grows wise." Proverbs 13:20. First and foremost I would like

to thank my sincere advisors, Professor Walter Stechele and Dr. Andreas Hutter for the

guidance, support, and inspiration during my pursuit of the Ph.D. degree at TUM. I owe

the awakening of my interest on multimedia information processing and intelligent visual

surveillance systems. I am also grateful to Professor Gerhard Rigoll, Professor Cremers

and Professor Ebrould Izquierdo for reviewing and offering me advice during their busy

time. I also have to thank Dr. Ramesh Visvanathan and Dr. Vinay Shet for the multiple

times of invitations to SCR (Siemens Corporate Research, Princeton). During my stay at

SCR, I could get better insight on computer vision related researches. I would like to thank

Siemens and German Gorvernment because the work presented here was partially funded

by Siemens AG Corporate Technology and by the German Federal Ministry of economy

and Technology (BMWi) under the THESEUS project.

I wish to acknowledge many of my former and current colleagues at Siemens and

TUM, including Colin Estermann, Andreas Laika, Tobias Schwarze, Jan Neumann, Vasu

Parameswaran, Maneesh Singh, Claus Bahlmann, Subramanyam Vdaygiri, Thomas Riegel,

Gero Bäse and Bonjung Koo for the many times of invaluable discussions on research

directions, technical trends and their valuable feedback on my work.

My sincere friends Daewoo Kim, Sungjin Kwon, Hoochang Shin, Changwook Lee

and Kiwon Park also deserve special thanks. Daewoo Kim was my continuous source of

joy for the discussions and talks not only about technical trends but also about life and

philosophy. Sungjin Kwon especially showed his support offering some of his technical

source codes on image processing. As a Ph. D candidate, Hoochang Shin and Changwook

III
Lee showed their interest on my work and gave me valuable comments. The CCTV video

footages from Kiwon Park was very helpful source of inspiration.

I would like to thank my family and relatives for their love and encouragement.

Especially, this dissertation is dedicated to my parents. My dad were and continue to be a

source of support. There are no words to describe my mom’s generosity, love and kindness

throughout the years.

In particular, I want to thank my dear girl friend Seung-hee. She has made my Ph.D.

career in munich so fruitful and enjoyable not only through her emotional support but also

through her scientific view as a medical researcher. Especially, her attitude towards life

and society, as a christian and as a medical doctor, touched me a lot.

Above all, I wish this dissertation to be my humble praise to God. During my

pursuit of the Ph.D. degree, I realized how God made humans amazingly and miraculously

intelligent. Without his love, I know that I could not even finish this work. Best thanks to

God, the name of Love. "The fear of the LORD is the beginning of wisdom, and knowledge

of the Holy One is understanding." Proverbs 9:10.

IV
Abstract

Most of the automated and intelligent visual surveillance systems, so far, have focused

on real-time observation and understanding abnormal human or object behaviour. The

usual approach is building specific analytic algorithms for well-defined and fixed domains,

(e.g. pedestrian detection and tracking, left baggage detection, intrusion detection, etc.),

Thereby, the semantic space of the approach is bound to the results and capacity of used

algorithms. When this kind of system is used in high-level forensic, such an approach is

inherently too limited. This is due to the fact that modeling all situations at develop-

ment time is very difficult. The difficulty arises not only from the limited vision analytic

power on signal level vision information processing but also from the ambiguity inherent

in interpreting complex situational context. Human cognition usually accomplishes the

interpretation by collecting visual evidences and combining such evidences with contextual

knowledge. Such a human activity includes derivation, modification and retraction of dif-

ferent semantic conclusions upon arrival of new information. Thus, it can be regarded as

non-monotonic reasoning

Bearing such a human ability in mind, the presented work focuses on designing a sys-

tematic approach and methodical support for a more intelligent forensic visual surveillance

system. Especially, we regard traditional vision analytics as perceptional sources of evi-

dences and focus more on the way of manipulating such evidences to derive more complex

semantics of interest. We approach the manipulation from the epistemological stance, that

is a theory aimed at targeting notions such as knowledge, belief, awareness and proposi-

tional attitudes, by means of logical and semantical tools. First, we propose the use of logic

programming for the basis of knowledge representation and reasoning. Second, we propose

V
the use of subjective logic to capture uncertain epistemic belief about knowledge. Third,

upon the proposed subjective logic extended logic programming framework, we present

an inference scheme for so-called default reasoning, that can draw plausible reasoning re-

sults under incomplete and contradictory information. Forth, we present an approach to

model uncertain and ambiguous contextual rules, based on so-called reputation operator

in subjective logic. Fifth, we address a problem that usual logic-based frameworks do

not allow bidirectional interpretation of a rule. Based on so-called deduction and abduc-

tion operators in subjective logic we show that bidirectional reasoning can be done in our

framework. Finally, by the use of abductive logic programming with subjective logic, we

demonstrate a reasoning framework that can answer most feasible hypotheses to the best

of current knowledge-base upon a set of observations that is collected by user. Each of

fore-mentioned aspects are presented with case studies from typical public area scenes with

visual surveillance data.

VI
Contents

1 Introduction 1

1.1 High-level Semantic Reasoning in Visual Surveillance and Epistemology . . 1

1.2 Challenges and Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . 5

1.2.1 Characteristics of Visual Surveillance Systems . . . . . . . . . . . . . 5

1.2.2 Episodic Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

1.2.3 Query Scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

1.2.4 Requirements - Non-monotonic Reasoning under Uncertainty . . . . 8

1.3 Proposed Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

1.4 Key Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1.5 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

2 Prior Art and Related Work 15

2.1 Epistemology - A Philosophical Background . . . . . . . . . . . . . . . . . . 15

2.2 Knowledge Representation in Artificial Intelligence . . . . . . . . . . . . . . 20

2.2.1 Formal Logics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20

2.2.2 Semantic Networks, Frames and Description Logic . . . . . . . . . . 21

2.2.3 The Semantic Web . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

2.3 Reasoning under Uncertainty in Artificial Intelligence . . . . . . . . . . . . . 24

2.3.1 Intensional Approaches . . . . . . . . . . . . . . . . . . . . . . . . . 25

2.3.2 Extensional Approaches . . . . . . . . . . . . . . . . . . . . . . . . . 26

2.3.3 Extensional vs. Intensional : Merits and Deficiencies . . . . . . . . . 27

i
Contents

2.4 Automatic Interpretation of Semantic Occurrence in Video . . . . . . . . . . 30

2.4.1 Pattern Recognition Methods - Data Centric Approaches . . . . . . 30

2.4.2 State Based Models - Intensional Approaches . . . . . . . . . . . . . 31

2.4.3 Semantic Based Models - Extensional Approaches . . . . . . . . . . . 33

3 Preliminaries 37

3.1 Subjective Logic Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

3.1.1 Dempster Shafer Belief Theory . . . . . . . . . . . . . . . . . . . . . 38

3.1.2 Opinion Model in Subjective Logic . . . . . . . . . . . . . . . . . . . 41

3.1.3 The Correlation between Subjective Opinion and Beta Distribution,

The Bayesian Perspective . . . . . . . . . . . . . . . . . . . . . . . . 44

3.1.4 Operators in Subjective Logic . . . . . . . . . . . . . . . . . . . . . . 51

3.2 Logic Formalisms and Logic Programming . . . . . . . . . . . . . . . . . . . 53

3.2.1 Propositional Logic . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

3.2.2 First Order Predicate Logic . . . . . . . . . . . . . . . . . . . . . . . 55

3.2.3 Logic Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . 56

3.2.4 Issues on Supporting Non-monotonicity . . . . . . . . . . . . . . . . 57

4 Proposed Architecture and Case Study on Forensic Queries 59

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

4.2 Proposed Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

4.3 Prototype Implementation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64

4.3.1 Ontological Metadata Model . . . . . . . . . . . . . . . . . . . . . . 64

4.3.2 Logic Programming Extended with Subjective Logic . . . . . . . . . 68

4.4 Case Study I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

4.4.1 Scenario Setting for Case Study I . . . . . . . . . . . . . . . . . . . . 70

4.4.2 Metadata Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

4.4.3 Querying and Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . 76

ii
 Contents

4.5 Case Study II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

4.5.1 Scenario Setting for Case Study II . . . . . . . . . . . . . . . . . . . 81

4.5.2 Metadata Generation . . . . . . . . . . . . . . . . . . . . . . . . . . . 82

4.5.3 Querying and Retrieval . . . . . . . . . . . . . . . . . . . . . . . . . 83

4.6 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87

4.6.1 Scalability of Ontological Instance . . . . . . . . . . . . . . . . . . . 87

4.6.2 Scalability of Rule Engines . . . . . . . . . . . . . . . . . . . . . . . 91

4.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95

5 Contradictory Information Handling 99

5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

5.2 Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

5.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103

5.4 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

5.4.1 Bilattice Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104

5.4.2 L-Fuzzy Set Based Logics (Interval-valued Fuzzy Logic, Intuitionistic

Fuzzy Logic and Fuzzy Belnap Logic) . . . . . . . . . . . . . . . . . 107

5.5 Default Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109

5.5.1 Reiter’s Default Logic . . . . . . . . . . . . . . . . . . . . . . . . . . 109

5.5.2 Bilattice based Multivalued Default Logic . . . . . . . . . . . . . . . 111

5.6 Default Reasoning using Subjective Logic . . . . . . . . . . . . . . . . . . . 113

5.6.1 Fusing Contradicting Dogmatic Opinions in Subjective Logic . . . . 113

5.6.2 On the Interpretation of Contradictory Point in Subjective Logic . . 117

5.6.3 Mapping Multi-Logic-Values into Opinion Space . . . . . . . . . . . 119

5.6.4 Default Inference using Subjective Logic Operators . . . . . . . . . . 122

5.7 Multivalued Default Logic, Square Bilattice and Default Opinion . . . . . . 126

5.8 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

iii
Contents

5.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144

5.9.1 Which way to go? Subjective Logic vs. L-fuzzy Logics . . . . . . . . 144

5.10 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 155

6 Vague Proposition Modeling and Handling 159

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159

6.2 Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 160

6.3 Proposed Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162

6.3.1 Vague Propositional Rules . . . . . . . . . . . . . . . . . . . . . . . . 162

6.4 Case Study I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

6.4.1 Scenario Setting for Case Study I . . . . . . . . . . . . . . . . . . . . 165

6.4.2 Uncertainty Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . 165

6.4.2.1 Distance . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

6.4.2.2 Direction . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167

6.4.3 Rule Encoding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169

6.4.4 Experimental Result . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

6.5 Case Study II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

6.5.1 Scenario Setting for Case Study II . . . . . . . . . . . . . . . . . . . 172

6.5.2 Reasoning Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

6.6 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

7 Hybrid Knowledge Modeling and Reasoning 179

7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

7.2 Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 180

7.3 Conditional Operators in Subjective Logic . . . . . . . . . . . . . . . . . . . 182

7.4 Proposed Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

7.4.1 Extensional Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

7.4.2 Intensional Layer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

iv
 Contents

7.5 Case Study I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189

7.5.1 Scenario Setting for Case Study I . . . . . . . . . . . . . . . . . . . . 189

7.5.2 Rule Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

7.5.3 Experimental Proof of Concept . . . . . . . . . . . . . . . . . . . . . 195

7.6 Case Study II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197

7.6.1 Scenario Setting for Case Study II . . . . . . . . . . . . . . . . . . . 197

7.6.2 Rule Modeling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200

7.6.3 Illustrative Examples as Proof of Concept . . . . . . . . . . . . . . . 204

7.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 206

8 Logical Abduction under Uncertainty 209

8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209

8.2 Background and Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . 210

8.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211

8.4 Abductive Reasoning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212

8.4.1 Abductive Logic Programming . . . . . . . . . . . . . . . . . . . . . 212

8.4.2 Abductive Logic Programming Frameworks . . . . . . . . . . . . . . 213

8.5 Proposed Approach . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

8.6 Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215

8.7 Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 220

9 Conclusion 223

9.1 The Big Picture - Architectural View of the Work . . . . . . . . . . . . . . . 223

9.2 Benefits and Drawbacks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

9.3 Open Issues and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . 228

9.4 Summary and Epilogue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 229

List of Figures 233

List of Tables 237

v
Contents

Bibliography 239

vi
 Chapter 1
1 Introduction

1.1 High-level Semantic Reasoning in Visual Surveillance and

Epistemology

In the security sector of industry, visual surveillance is one of the oldest and most widespread

technologies as cameras are quite informative and relatively inexpensive sensors. Since the

emergence of the first generation of digital recording technology, it is undergoing a digital

revolution, such as compressing video footages, archiving digitized contents, integrating

hundreds of cameras, etc. Consequently, as it is getting digitized, we also get huge amount

of video information that can not be processed only by human security agents. Therefore,

automatic understanding of semantics in surveillance video contents is one of the keys to

success. However, this is an extremely difficult task for visual surveillance systems1 . To

address this problem, there has been active research focus and paradigm shift for the last

decades as shown in Figure 1.1.

Considering that one of the primary objectives of visual surveillance is to prevent

unexpected and potentially dangerous situations for immediate intervention, most of the

automated and intelligent visual surveillance systems, so far, have focused on real-time

observation and understanding of unusual or dangerous activities. The usual approach is

1
This is the same for other visual information systems (VIS), that deal with usual broadcasting or
consumer contents as more and more visual material (i.e., images and videos) is produced [65]. Major
researches in VIS have been driven in terms of efficient access for a massive content database, by means
of (semi-) automated semantic annotation, indexing, content-based retrieval and automated video
abstraction (i.e., shot/scene summarization), etc [52, 47]. Especially, automated semantic annotation
in VIS treats an even wider semantic lexicon of concepts in various content domains, thereby making it
even more challenging [162, 16, 25]. A work introduced by Schreer and Izquierdo et al. [147] is one that
addressed automatic semantic annotation in the RUSHES [5] european project. In this dissertation,
however, we will limit our scope of discussion to visual surveillance contents.

1
Contents

Figure 1.1: Research Spectrum of Intelligent Semantic Analysis in Visual Surveillance and
Paradigm Shift.

building specific analytic algorithms for well-defined and fixed domains (e.g. pedestrian

detection and tracking, left baggage detection, intrusion detection, etc). Such analytics

usually model activities or events as a sequence of a number of ‘states’ that are related

to visual features (such ‘state model’ based approaches are also referred to as ‘intensional

approach’ [130]). Therefore, in this paradigm the focus is more on ‘automated perception’

of specific events as depicted in Figure 1.1 - ‘2nd Paradigm’. Consequently, due to the

difficulty of modeling all situations at development time, the semantic space is inherently

too limited to the results and capacity of used algorithms.

Following the September 2001 attacks on the United States [134] and then those in

London in 2005 [10], it was proven that the primary objective to prevent all potentially

possible threats is not always achievable [148, 74] even in cities where a huge number of

surveillance cameras was deployed (London is the city the most often cited for the number

of cameras deployed in its streets. [74]). Naturally, forensic use of visual surveillance in

an intelligent way is gaining increasing attention, not only for real-time observation but

2
 Chapter 1
Contents

also for post incident investigation. Such investigation usually deals with propositional

semantic assumptions or queries to be investigated after an incident. However, analyzing

such higher level semantics is not a trivial task. The difficulty arises not only from the

ambiguity inherent in visual information and limited vision analytic power but also from

the ‘complex plot’ [148] implied in video scenes in case of planned or intended events.

Humans usually accomplish such investigation by collecting visual evidences and

combining the evidences with contextual knowledge. This can be regarded as ‘cognitive’

human activity that deal with fusion of ‘propositional context knowledge’ and ‘belief ’ on it.

In that sense, it is closely related to ‘epistemology’ 2 that is a theory aimed at targeting

notions such as knowledge, belief, awareness and propositional attitudes, by means of

logical and semantical tools [137, 138]. By this aspect, we advocate that high level semantic

analysis of visual surveillance can be regarded as ‘epistemic reasoning’ upon available

evidences and contextual knowledge as depicted in Figure 1.1 - ‘3rd Paradigm’.

Although it is undergoing the paradigm shift, relatively little work from the ‘auto-

mated epistemic reasoning’ stance has been done. This dissertation is intended to address

this void in semantic analysis research. Thus, the main objectives of this dissertation is as

follows.

• Designing a systematic approach for reuse of metadata acquired from vision analytics.

• Introducing a methodical support for epistemic reasoning under incomplete and uncer-

tain knowledge representation and (vision analytic) metadata.

• Studying the feasibility of the proposed systematic approach and methodical support for

higher level semantic reasoning scenarios in visual surveillance.

In visual surveillance systems, metadata acquired from vision analytic modules can
2
Human ‘cognition’ consists of many aspects such as psychology, neuroscience, logics, linguistics, sociology,
etc., and ‘cognitive science’ is the study on these aspects. While ‘artificial intelligence’ aims to model
some of these aspects for a machine. Among these aspects, ‘epistemology’ addresses the question
‘How do we know what we know?’ and deals with the means of production of knowledge, as well as
skepticism about different knowledge claims. In artificial intelligence, therefore, it is closely related to
logics, automated judgement of belief, etc. For more detailed overview, refer to [116, 138, 137].

3
Contents

be regarded as symbolized visual evidences obtained by artificial vision sensors from the

2nd paradigm view. Then, such evidential metadata should be epistemically fused by con-

textual knowledge. The contextual semantic knowledge can be formal representations in

a set of semantic rules, constraints and relations, i.e., in particular, this is the part that

has a large degree of overlap between how humans describe what constitutes a situational

concept and how it is defined within contextual knowledge modeling formalisms. In other

words, recognizing certain high level contextual semantic concept as it occurs becomes a

problem of ‘explaining’ the observation using the available semantic knowledge. Therefore,

the higher level semantic models should contain modeling approaches that do not just

aim to define the entire ‘state’ space of the event domain as in ‘state model (intensional)’

approaches. Instead, we advocate that it should rather enable human-like ‘epistemic rea-

soning’ that can offer a formal way how we derive and guarantee the target semantics of

interest. In particular, in such circumstance, knowledge should be able to be defined as a

‘modular’ knowledge segments (such knowledge representation guaranteeing modularity of

partial knowledge segments is referred to as ‘extensional approach’ [130]). Regarding these

objectives several fundamental questions can be posed:

• What are the semantic queries that can be answered based on epistemic reasoning under

incomplete and restricted evidences ?

• How can one make the ‘sensing (automated perception)’ result of available vision analytic

modules reusable ?

• Where does the semantic knowledge come from ? How can one represent semantic know-

ledge in a machine understandable way ? What is the proper formalism for representing

such a conceptual knowledge ?

• How humans assess propositional assumptions and contextual concepts along the given

evidences and contextual knowledge ? For example, humans tend to assess a proposition

in a vague manner such as ‘strongly certain’, ‘less certain’, ‘seem to be true but quite not

4
 Chapter 1
Contents

sure’, etc. How such an epistemic status can be modeled ?

• Can one guarantee that a given contextual knowledge is always sound and complete ?

Humans have a capability on reasoning under inconsistent and incomplete knowledge.

What are the epistemic aspects that can arise under such a circumstance and how can

it be modeled in a machine interpretable way ?

• The results from vision analytic modules usually come with uncertainty, and their per-

formance may vary according to its internal analytic mechanisms and training data.

How can one take into account such uncertainty at the time of reuse ?

• What are the practical and pragmatic building blocks given currently available software

components in terms of building a system architecture to enable such a reasoning ?

This dissertation addresses some of these questions by focussing on designing a sys-

tematic approach and methodical support for a more intelligent higher level reasoning

capability.

1.2 Challenges and Requirements

This subsection details characteristics of visual surveillance and derives challenges and

requirements in terms of forensic sense of high level semantic queries (refer to [49, 74] for

a more general view of challenges in visual surveillance related researches.).

1.2.1 Characteristics of Visual Surveillance Systems

In the sense that ‘automated epistemic reasoning’ rely on currently available vision analytic

powers and existing infrastructures, it is important to know the characteristics of visual

surveillance systems. The environmental nature of a surveillance system can be character-

ized as follows.

1. Environment : Distributed

2. Processable Semantics :

5
Contents

- Pre-manual annotation of background objects for fixed camera setting

- Realtime processable objects and some of pre-defined atomic behaviors

3. Types of data :

- Alarms, objects and events representable in symbolic form (e.g. text format, etc)

- Various low-level feature vectors extractable from ROIs

4. Size of data : Large scale

5. Access to data : Very rare

6. Semantic complexity of query : Very high and vary from specific to abstract and vague

7. Degree of automation : User interaction in the loop

8. Performance metrics :

- Acceptable accuracy, robustness and response time

1.2.2 Episodic Case Study

Bearing such an environmental characteristics in mind, a top-down approach is taken to

derive more concrete requirements on the components and the system itself for complex

queries. We analyze real events, e.g. in which the authorites released CCTV footage

after closing investigation on the subway bombing attacks in London on 7th july 2005. A

question roadmap [10] shows the questions dealt by London Police such as ‘Who is the

suspect?’, ‘Was it done by a single person or by a group?’, ‘Why did they do it?’, ‘How

did they do it?’, etc. To be machine supportable, it is important to take into account the

semantic granularity of the query. In the example, the ‘why’ or ‘how’ questions seem much

harder than ‘who’ questions because the more knowledge about the situation is required.

Consequently, queries should have reasonable semantic level to be interpreted by machine.

In terms of machine process-ability, the semantic complexity of query is desired to be

compositional, that can be tied with other atomic semantic meanings with visual cues and

situational context. Figure 1.2 (left) shows steps gathering situational knowledge and how

the acquired knowledge influences the iterative reasoning especially for forensic sense of

search and retrieval. Figure 1.2 (right) shows one sample of the released footage. Although

6
 Chapter 1
Contents

Figure 1.2: Conceptual Workflow of Forensic Reasoning in Case of Video Retrieval (left)
and a Released Retrieved-Footage (right) [10].

this sample seems very ordinary, the people in the scene were regarded as suspects of a

terror when considered in the situational context. Therefore, interpreting the semantic

granularity implied in a question and understanding the situational context knowledge is

important to fulfill the forensic reasoning task.

1.2.3 Query Scenario

For a more concrete discussion, we assume a distributed camera network and an unexpected

incident not directly captured by a camera due to the sparseness of the network and lack

of sufficient vision analytic power. Assuming some clues, let us consider following queries.

• Atomic level of granularity for semantic queries;

Conjunctive queries that can be seen as existence of some of specific atomic metadata

(e.g. shown, detected, etc).

- Q1 ‘scenes in which a person and a red vehicle are shown’

- Q2 ‘scenes in which a person shown on a given photo is in’

• Medium level of granularity for semantic queries;

Compositional queries having relational semantics that may require more contextual in-

formation and that may require iterative and active collection of contextual information.

(e.g. group of, passing by, carrying, talking to, witness of, theft of, etc).

7
Contents

- Q3 ‘group of four people passing by a telephone booth’

- Q4 ‘scenes in which a person is carrying an oil box around the place of the reported

incident’

- Q5 ‘scenes one of selected person, among retrieved after Q4, is talking to somebody’

- Q6 ‘probable witnesses of a selected person after Q4’

- Q7 ‘who is the suspect of the event?’

- Q8 ‘who is the most probable witness of the suspect of the event?’

- Q9 ‘scenes in which a person thieve a bag of other persons’

• Large level of granularity for semantic queries;

Queries that require explanations of an observed or even semantically reasoned event.

- Q10 ‘why did the woman detected in a robby suddenly went to outside?’

- Q11 ‘how did the guy reached at the place of incident?’

- Q12 ‘why did he thieved the bag?’

1.2.4 Requirements - Non-monotonic Reasoning under Uncertainty

In the following, by considering above query scenarios, the most important requirements

towards target high level semantic reasoning capability and systems are described.

• Metadata Representation and Reuse : it is necessary to exploit as many existing

semantic understanding / extraction power and intermediate results as possible to max-

imize the potential utilization for later query and reasoning processing. As shown in

Section 1.2.1 - 2, manual annotation could be also used to denote the concept ‘telephone

booth’ in Q3. Similarly, results from automatic human detection algorithm could be

exploited to serve as atomic ingredient for processing more complex concepts such as

‘group of people’, ‘witness’, ‘suspect’ and ‘theft’ in Q1-Q9. This should include low-level

features as well, as shown in Section 1.2.1 - 3. Thus, flexible machine-readable means

like a suited metadata representation model is required to archive the processing result.

8
 Chapter 1
Contents

• Semantic Knowledge Representation : to cope with the ‘complex plot’ at reasoning

time, the system requires an efficient way to describe contextual knowledge about scenes.

Unlike in the deterministic ‘state-model (intensional)’ based approach [130], here the

reasoning target is not given at development time. Furthermore, a given problem can

not be solved directly by a known set of steps (see Q6-Q8, Q10-Q12) and the input data

is ‘incomplete’ because the available results generated by the analytics can not be fully

pre-determined due to variations in the system configuration or settings of the employed

analytics. However, considering the complexity of possible semantic knowledge ‘model-

free’ approach is also not possible. Thus, we rather need a more flexible ‘extensional’

semantic inference mechanism that allows ‘modularity’ of each knowledge segments [130].

• Uncertainty Representation and Attachment to Metadata : usually, interme-

diate metadata comes with ‘uncertainty’ (e.g. trust worthiness of results and errors in

the generation of metadata such as false alarms). As mentioned in Section 1.2.1 - 8

this will influence the acceptable accuracy and robustness for a semantically complex

queries. Thus, a mechanism is required for dealing with the varying quality to the re-

trieved results, i.e. to cope with the ‘uncertain’ nature of intermediate results captured

as metadata from the analytic side.

• Epistemic Uncertainty Representation and Belief Revision : we can not guar-

antee the ‘soundness’ and ‘completeness’ of given contextual knowledge. There are

‘uncertainty’ in knowledge itself. For example, there can be often situational ‘contra-

dictions’, ‘counter examples’ or ‘stronger knowledge segments’ against given knowledge

segments (see Q7-Q9). Therefore, we need a mean to assess ‘epistemic belief ’ to attach

to given knowledge segments. Furthermore, derived reasoning results should be able to

be revised upon arrival of new evidential data or contextual knowledge. Such capability

is often referred to as ‘belief revision’ [138]. Especially, the handling of contradictory

information is referred to as ‘default reasoning’ [143]. Thus, the reasoning capability

should cover ‘belief revision’ and ‘default reasoning’.

9
Contents

• Epistemic Meta Reasoning and Abduction : in the case of highly abstract and

vague queries such as Q10-Q12, it requires a ‘meta-reasoning’ (i.e. reasoning about

reasoning [45]) power that can even able to reason about the query itself in an iterative

manner. The system should be able to set ‘epistemic hypothesis’ by itself and assess

if the hypothesis satisfies. Especially, such a reasoning explaining a given observation

is known as ‘abduction’ [150] (note that, especially, more complicated handling of this

aspect is closely related to ‘planning’ in the view of artificial intelligent). Therefore, the

desired reasoning power should cover ‘abduction’ capability.

The requirements for ‘belief revision’, ‘default reasoning’, ‘meta-reasoning’ and ‘ab-

duction’ can be categorized under the term of ‘non-monotonic reasoning’ [35]. Considering

‘uncertainty’ both in intermediate metadata and a knowledge segment itself, it can be

denoted as ‘non-monotonic reasoning under uncertainty’. Further, considering that the

knowledge modeling is preferred to be done in an ‘extensional’ way, the requirements can

be summarized as ‘extensional approach to non-monotonic reasoning under uncertainty’.

1.3 Proposed Approach

In this section, we present our approach to the requirements designated in the previous

section.

First, one of the most important aspects throughout above derived requirements is

the representation of uncertainty in terms of both ‘uncertainty in vision analytic metadata’

and ‘epistemic uncertainty in knowledge representation’. We employ subjective logic the-

ory [93] that is a relatively new branch of probabilistic logic which comes with a rich set

of logical operators. Subjective logic also provides an intuitive representation of human

like epistemic belief in a model called ‘subjective opinion’. A subjective opinion not only

allows to describe one’s belief based on degree of truth or falsity but also allows explicit

representation of ignorance about the degree of truth. This aspect is inherited from Demp-

ster Shafer’s belief theory [149]. But it also provides a mathematical mapping between the

10
 Chapter 1
Contents

opinion representation and 2nd order Bayesian especially, via a Beta function representa-

tion [20]. The intuition behind is that the uncertainty representation of subjective logic

covers traditional probability so that we can directly encode traditional bayesian based un-

certainty calculation results coming from vision analytic modules (in the sense that most

of ‘state model’ based approaches rely on Bayesian approach throughout model learning

and recognition phase). This aspect matches to the requirements of uncertainty attach-

ment to metadata. Indeed, including ignorance on trustworthiness, we believe that more

intuitive and human like ‘epistemic status’ can be represented. This aspect matches to the

requirement of ‘epistemic uncertainty representation’ about a given knowledge segments.

Second, for the semantic knowledge modeling and representation, we exploit logic

programing approach. We do not believe that we can derive a ‘whole-model’ that can cover

most of semantic situation. We also do not believe that a ‘model-free’ approach is possible

because introducing a learning mechanism on semantic knowledge out of visual surveillance

data is again extremely difficult task. Therefore, our choice is to bestow more degree of

freedom on modeling semantic knowledge. To achieve this purpose, the most important

aspect is that the system need to guarantee a ‘modularity’ so that any domain engineers can

just concentrate on their partial knowledge segment without considering whole knowledge.

Another aspect is that the knowledge representation formalism should provide a mean

to describe many types of relations. Traditional ‘state-model’ based approaches can be

also seen in a logic sense, in that a pair of nodes connected with edge can be seen as

‘if node1 then node2’ manner. However, this level of conditional description remains in

‘propositional logic’ that only offers one relation ‘influence’. Unlike propositional logic,

logic programming usually offers predicate logic that can build various relations. Although

there are other technologies for knowledge representation such as ‘entity-relation diagram’,

‘semantic network’, ‘ontology and semantic web’ they tend to rely on schematic approach

that more fit to data archiving aspect. Therefore, we adopt semantic web and database

approach for intermediate metadata archival scheme.

Finally and most importantly, to achieve ‘nonmonotonic reasoning’ capability, we

11
Contents

introduce a principled reasoning scheme especially for ‘default reasoning’, ‘vague proposi-

tional rule’ modeling and ‘abductive explanation of observation’. The reasoning mechanism

is realized based on logic programming extended with subjective logic based operations on

epistemic beliefs (subjective opinions). Concerning uncertainty propagation, we also at-

tempts to provide the ability of interpreting conditional knowledges in a bijective manner.

In this way, we give more flexibility and on demand way of using both ‘intensional’ and

‘extensional’ reasoning. To enable ‘default reasoning’, we introduce principled scheme of

handling ‘contradictory information’ so that it can derive plausible reasoning result. For

the vague propositional rule definition, we introduce subjective reputation function for

evaluating trustworthiness of a given proposition. For the semantic explanation scenario,

we take both ‘deductive’ and ‘abductive’ logic programming. Especially, we take both

forward-chaining and backward-chaining logic programming engines for ‘meta-reasoning’.

1.4 Key Contributions

We now summarize the key contributions of this dissertation tackling the challenges of

high level epistemic semantic reasoning for visual surveillance.

• The idea to combine logic programming and subjective logic theory for flexible knowledge

representation, uncertainty handling and epistemic reasoning.

• The study on system architecture for reuse of vision analytic metadata comparing tra-

ditional database, ontology, logic programming based technologies.

• The design of methodical support for contradictory information handling using subjective

logic theory and its comprehensive comparison with L-Fuzzy set based approaches.

• The idea of using reputation concept for modeling vague propositional knowledge.

• The idea of using deduction and abduction operators of subjective logic for enabling

bidirectional interpretation of a conditional proposition.

12
 Chapter 1
Contents

• The idea of layering abductive and deductive logic programming for logical abduction

based diagnostic scene explanation given collected observations.

1.5 Organization of the Thesis

The rest of this dissertation is organized as follows.

In the following Chapter 2, we give a brief historical review of epistemology and logic

both in terms of philosophical view and artificial intelligence. Then we review relevant

literatures to the problems of automatic interpretation of semantic occurrence in video.

In Chapter 3, we present preliminaries fundamental to understand this dissertation.

This will cover subjective logic and logic programming. Except the fundamentals explained

in the preliminaries (Chapter 3), the rest of chapters are organized to be self-contained.

In Chapter 4, we present our framework for representing vision analytic metadata,

semantic knowledge representation and extension of logic programming component with

subjective logic. We discuss the software components, data processing pipeline and system

architecture design. We present some of case studies of forensic query scenarios. We

conclude the chapter with a discussion of experimental evaluation on query performance

on different software component settings.

In Chapter 5, we introduce the problem of handling contradictory information in

extensional system especially in terms of so-called default reasoning. We present our ap-

proach to expand, refine, and translate epistemic status on contradiction by comparing

subjective opinion space with square bilattice. We then models a default reasoning scheme

adopting the idea exploited in bilattice system. There, we also compare our approach to

L-fuzzy set based approaches. We then presents a case study on handling contradictory

information in visual surveillance scenario. Finally, we conclude with discussions especially

on the comparison between proposed approach and L-fuzzy set based approaches such as

square bilattice.

In Chapter 6 we present our research on vague propositional knowledge modeling

13
Contents

and handling. There, we present the specific idea on evaluating a proposition with a

reputation function of subjective logic. We also describe a case study on reasoning under

vague propositional rules. We conclude the chapter with a discussion of the experimental

evaluation of these techniques.

Chapter 7 describes our work on hybrid knowledge modeling and reasoning. First,

we contrast intensional and extensional way of interpretation of given conditional rules.

Then, we present the idea how to enable intensional handling of conditional rules in an

extensional framework using subjective logic. Finally, this chapter presents, in detail, the

case study performed for evaluating a public scene based reasoning.

In Chapter 8, we present our framework for logical abduction based scene explanation

under collected observations. There, we present the idea of layering deduction on query

itself and abduction to get final answer. This chapter also shows how the final answers

can be evaluated in terms of default reasoning. Finally, we conclude the chapter with

discussions on open issues and future work.

In Chapter 9, we present the conclusions of our work with discussions and summary

of contributions and some future research directions.

14
 Chapter 2
2 Prior Art and Related Work

In this chapter, we will review related work ranging from a philosophical background of

epistemology to knowledge representation researches in artificial intelligence. Thereafter, it

follows with reviews of prior art ranging from a semantic multimedia retrieval to automatic

interpretation of semantic occurrences in video.

2.1 Epistemology - A Philosophical Background

Epistemology is the branch of philosophy concerned with the nature and scope of knowledge

[11, 174]. In the view of philosophical history, since Aristotle, Platon to Descartes and Kant,

the majority of philosophy was metaphysics to explain the fundamental nature of being

and the world [105]. Since Aristole, epistemology was dealt as a sub topic of metaphysics

in the focus on structuring existence of entities in the world. Especially, ‘ontology’ received

much interests. Ontology is the theory of objects in terms of the criteria which allow

one to distinguish between different types of objects and the relations, dependencies, and

properties through which they may be described. Since Descartes, epistemology has paid

more attention on how humans think and how humans can sure their belief on knowledge.

Addressing this problem, we face with following fundamental question, "Is the limitation

of one’s cognition bounded via sensory experience ? or is it bounded via innated humans

rationality ?". Through 17 to 18 centuries, we bestowed the name ‘Empiricism’ on the first

category and ‘Rationalism’ on the other category. These two major trends are compromised

by Immanuel Kant in his remarkable work "Die Kritik der reinen Vernunft". Kant thought

humans do not know as knowledge exists but believe a certain knowledge as humans know.

15
Contents

Figure 2.1: Kant’s Epistemological Structure of Thinking [4].

Therefore, to him truth is something that we need to find out of structure of human

thinking [107]. This idea is depicted in Figure 2.1. In terms of rationalism, he introduces

‘a priori innate human rationality’ that serves categories on thinking as posed follows:

• Quantity (Quantität) :

Unity (Allheit), Plurality (Vielheit), Unity (Einheit)

• Quality (Qualität) :

Reality (Realität), Negation (Negation), Limitation (Limitation)

• Relation (Relation) :

Subsistence (Substanz), Causality (Kausalität), Community (Gemeinschaft)

• Modality (Modalität) :

Possibility (Möglichkeit), Existence (Wirklichkeit), Necessity (Notwerdigkeit)

16
 Contents

With the help of such priory categories, he believed that humans can fuse their

empirical sensing ability within these categories to extend their knowledge. This aspect is

Chapter 2
understood as ‘judgement’ and ‘propositional rules’ are the representation of judgement.

Kant also introduces 12 categories of judgement as posed follows :

• Quantity (Quantität) :

- Universal (allgemeine Urteil) ; every S is P

- Particular (besondere Urteil) ; some S is P

- Singular (einzelne Urteil) ; this S is P

• Quality (Qualität) :

- Affirmative (bejahende Urteil) ; S is P

- Negation (verneinde Urteil) ; S is not P

- Infinite (unendliche Urteil) ; S is non P

• Relation (Relation) :

- Categorical (kategorische Urteil) ; S is P

- Hypothetical (hypothetische Urteil) ; if A is B then C is D

- Disjunctive (disjunktive Urteil) ; A is either B or C

• Modality (Modalität) :

- Problematical (problematische Urteil) ; S seems P

- Assertoric (assertorische Urteil) ; S is P

- Apodictic (apodiktische Urteil) ; S must be P

Humans in everyday life face with various situations and events. Then humans

think various ways to solve, analyze the facing situations and events. Of course the ways

would be very varying, however, as long as their thinking is related on ‘judgement’, it

falls into above categories. Categorical judgement (Kategorische Urteil), among others, is

the most basic one and every 12 types can be transformed into the categorical judgement

except ‘Hypothetical’ and ‘Disjunctive’. When the categorical judgment is considered with

17
Contents

‘universal’, ‘particular’, ‘affirmative’ and ‘negation’, following four types of basic judgement

is possible.

• Universal Affirmative : Every S is P

• Universal Negation : Every S is not P

• Particular Affirmative : Some S is P

• Particular Negation : Some S is not P

Based on above mentioned categories, ‘inference’ is the process of deriving ‘new

judgement’ out of known judgements (namely, ‘therefore, new judgement’). In inference,

already known judgements are called ‘premises’. In case there is only one ‘premise’ we

call it ‘direct inference’ and in case of handling ‘multiple premises’, we call it ‘indirect

inference’. Indirect inference can fall into following categories.

• Deductive Inference : Deductive inference attempts to show that a conclusion necessarily

follows from a set of premises or hypotheses. A deductive argument is valid if the

conclusion does follow necessarily from the premises, i.e., if the conclusion must be true

provided that the premises are true. A deductive argument is sound if it is valid and

its premises are true. Deductive arguments are valid or invalid, sound or unsound, but

are never false nor true. Deductive reasoning is a method of gaining knowledge. An

example of a deductive argument: 1) All men are mortal, 2) Socrates is a man, 3)

Therefore, Socrates is mortal. The first premise states that all objects classified as ’men’

have the attribute ’mortal’. The second premise states that ’Socrates’ is classified as a

man - a member of the set ’men’. The conclusion states that ’Socrates’ must be mortal

because he inherits this attribute from his classification as a man.

• Inductive Inference : Inductive inference is a kind of empirical reasoning that constructs

or evaluates inductive arguments. The premises of an inductive logical argument indicate

some degree of support (inductive probability) for the conclusion but do not entail it;

18
 Contents

that is, they suggest truth but do not ensure it. Induction is employed, for example,

in the following argument: 1) Every life form we know of depends on liquid water to

Chapter 2
exist. 2.) All life depends on liquid water to exist. Inductive reasoning allows for the

possibility that the conclusion is false, even where all of the premises are true. For

example: 1) All of the swans we have seen are white, 2) All swans are white. Through

many dictionaries inductive inference are also defined as reasoning that derives general

principles from specific observations.

• Abductive Inference : Abductive inference is a kind of logical reasoning described by

Charles Sanders Peirce as "guessing" [131]. The term refers to the process of arriving

at an explanatory hypothesis. Peirce said that to abduce a hypothetical explanation a

from an observed surprising circumstance b is to surmise that a may be true because

then b would be a matter of course. Thus, to abduce a from b involves determining

that a is sufficient (or nearly sufficient), but not necessary, for b. For example, the

lawn is wet. But if it rained last night, then it would be unsurprising that the lawn

is wet. Therefore, by abductive reasoning, it rained last night. (But note that Peirce

did not remain convinced that a single logical form covers all abduction.) Peirce argues

that good abductive reasoning from P to Q involves not simply a determination that,

e.g., Q is sufficient for P, but also that Q is among the most economical explanations

for P. Simplification and economy are what call for the ’leap’ of abduction. There has

been renewed interest in the subject of abduction in the fields of computer science and

artificial intelligence research [111].

In the context of traditional computer vision research on semantic analysis, the ma-

jority of the work focusses on ‘concept’ level as a result of automated perception. Although

the importance of judgements and inference, little attempts have been paid to ‘automated

judgements and inference’. In this dissertation, we especially pay more attention to the

‘automated inference’ in terms of ‘deduction’ and ‘abduction’. To cope with the quality

property of the judgement, we also pay attention for the type of ‘Particular (besondere

19
Contents

Urteil)’ judgement.

2.2 Knowledge Representation in Artificial Intelligence

In the previous section, we have reviewed that ‘epistemology’ is related to ‘philosophical

ontology’ that tries to encode the relational structure of concepts which one can use to

describe and reason about aspects of the world. Inheriting this tradition, today, formal logic

also focusses on artificially structured languages and its syntax or semantics. Traditional

knowledge representation models developed in the field of artificial intelligence are formal

logics [146], semantic networks [141, 84], frames [118, 84] , Description Logic [24, 2] and

semantic web [7, 48].

2.2.1 Formal Logics

Logics and semantic networks are widely accepted models for effective knowledge represen-

tation. Logics aim at emulating the laws of thought by providing a mechanism to represent

statements about the world - the representation language - and a set of rules to deduce new

statements from previous ones - the proof theory. The representation language is defined

by its syntax and semantics, which specify the structure and the meaning of the statements,

respectively. Different logics make different assumptions about what exists in the world

(e.g. facts) and on the beliefs about the statements. The most basic logic is Propositional

Logic [146]. Propositional logic declaratively deals with pieces of syntax correspond to

facts via partial, disjunctive, negated information. Propositional logic is compositional in

that meaning of A ∧ B is derived from meaning of A and B. Traditional probability theory

can be seen as a propositional logic in the sense that a conditional probability P (B|A) can

be considered as ‘If B then A’. Fuzzy logic [176] also deals facts that comes with degree of

truth. However, propositional logic has very limited expressive power unlike natural lan-

guage. The most widely used and understood logic is First-Order Logic (FOL) [146], also

known as First-Order Predicate Logic (FOPL). Whereas propositional logic assumes the

20
 Contents

Formalism Ontological Commitment Epistemological Commitment

(What exists in the world) (What an agent believes about facts)
Propositional logic facts true / false / (unknown)

Chapter 2
First-order logic facts, objects, relations true / false / (unknown)
Temporal logic facts, objects, relations, times true / false / (unknown)
Probability theory facts degree of belief ∈ [0, 1]
Fuzzy logic facts with degree of truth ∈ [0, 1] known interval value

Table 2.1: Formal Logics and Their Ontological and Epistemological Commitments [146].

world contains facts therefore supports only one abstract relation ‘influences’, first-order

logic (like natural language) assumes the world contains (1) Objects, e.g. people, houses,

numbers, colors, baseball games, wars, etc. (2) Relations, e.g. red, round, prime, brother

of, bigger than, part of, between, etc. (3) Functions, e.g, father of, best friend, one more

than, plus, etc. For example, "Brother (Richard, John) ∧ Brother (John, Richard)" means

that "Richard is the brother of John and John is the brother of Richard"; ∀x King(x) =⇒

Person(x)" means that "All kings are persons". Logics of various kinds and logical reason-

ing and representation languages such as Prolog and KL-ONE have been popular tools for

knowledge modeling, for example in the definition of ‘expert systems’. The ontological and

epistemological commitments of different formal logics are summarized in Figure 2.1.

2.2.2 Semantic Networks, Frames and Description Logic

A semantic network [141, 84] is a directed graph consisting of vertices that represent con-

cepts and edges that encode semantic relations between them. Concepts can be arranged

into taxonomic hierarchies and have associated properties (e.g. the state "Bill is a person"

could be represented by the chain: Bill Clinton Node - Is-A Arc - Person Node). In spite of

their simplicity and support for modular inheritance, semantic networks suffer from limited

expressiveness, as they can not represent negation or disjunction, among others. Frames

[118, 84] are closely related to semantic networks but represent knowledge in terms of hi-

erarchies of frames containing named slots, together with rules such as type constraints,

to define concepts and relationships. It is widely accepted that knowledge in the form of

21
Contents

semantic networks, frames, and scripts can be expressed using logics such as first-order

logic. Much recent work on knowledge modeling has focused on description logics [24, 2],

which have evolved from semantic networks to provide a more rigorous definition of seman-

tics in terms of a (typically restricted) form of first-order logic. Description logics usually

provide a syntax that makes it easy to specify categories and perform inference tasks such

as subsumption and classification.

2.2.3 The Semantic Web

The term ontology has recently undergone a strong revival largely due to the efforts of

the semantic web community [7]. Ontologies are seen in a knowledge management sense

as providing an important tool for the representation of semantic information and the

automated processing of such information for applications such as data mining, retrieval,

and automated discovery and utilization of services by autonomous software agents. The

consensus definition of ontology in this context is as a ‘formal, explicit specification of a

shared conceptualization’ [75], hence the focus is on ‘knowledge sharing’ and ‘sufficiently

formal representation’ to allow manipulation by computer.

As shown in Figure 2.2, the semantic web is based on a layered hierarchy in which

ontologies provide the underpinning that enables metadata to be interpreted automatically

[158, 14]. In addition to the XML and RDF standards for structured document annotation

and resource description, attention is now focussed on the new framework for Web On-

tology languages (OWL) [76]. It is now recognized that ontologies are the natural vehicle

for knowledge representation and interchange at different levels of granularity and across

different domains, hence they are to form a vital cornerstone of future generations of the

internet.

While the proposed formalisms for the semantic web draw on a rich heritage of work

in artificial intelligence and linguistics, they remain limited due to an explicit focus on the

description of deterministic resources especially textual data. Many decades of research

into knowledge engineering have shown the limitations of methodologies such as semantic

22
 Contents

Chapter 2
Figure 2.2: Semantic Web Architecture Layer Cake [14].

nets and description logics on which OWL is based. The World Wide Web Consortium

(W3C) and the knowledge engineering community have yet to address the enormous chal-

lenge of reasoning about non-textual multimedia data and supporting uncertain knowledge

representation. Some efforts have been paid to extend ontology language with uncertainty

representation capability using for instance fuzzy logic [38, 160, 181]. However, there is

no recommended languages for this purpose. So is remained at rule description and logic

framework layer. Even though there is several language candidates for describing rules

such as SWRL and RIF, they are still a working draft in the W3C and not yet a recom-

mendation. Logic framework layer that is assumed to provides more epistemic answer for

the question of why this piece of information is taken or appear to user ? again there is no

technology specification at present for this layer. Proof layer is assumed to answer agents

about the question of why they should believe the results. At present, again there is no

technology recommended by W3C to this layer.

23
Contents

Overall, concerning our purpose of this dissertation, however the semantic web lay-

ered architecture gives us an insight on the component for epistemic reasoning framework.

Therefore, we regards the layered architecture rather as thorough requirements for intel-

ligent systems. At the level of metadata representation, there have been some efforts on

building ontological annotation taxonomies for multimedia description and event descrip-

tion. VERL (Video Event Representation Language) [122, 32] provides taxonomies to

annotate instances of the events and aceMedia project [28] aimed at providing semantic

taxonomies for multimedia annotation. In spite of such efforts, it is not yet widely ac-

cepted for annotation and reasoning due to the lack of handling uncertainties and lack

of technical support on logical reasoning and lack of technical means for proof layer as

explained above. Indeed, the focus of such efforts are to share semantic annotation in an

interoperable manner among systems, rather than focusing on the reasoning power.

2.3 Reasoning under Uncertainty in Artificial Intelligence

Given a knowledge representation, reasoning is the process to draw new knowledge out of

observed patterns of facts and their attributes. Such reasoning methods can be categorized

into two approaches ‘extensional’ and ‘intensional’. These terms can be explained in

terms of defining sets. It is well known that there are two fundamental ways of defining

sets. First, a finite set may be defined by simply listing its members, as in the concept

{Auckland, Hamilton, Wellington}. This is definition by ‘extension’. Alternatively, any set

may be defined by stating the properties of its elements. This is definition by ‘intension’.

Following this convention, a semantic reasoning is ‘extensional’ if it is primarily concerned

with enumerating the final result and not other aspects of computation such as the path

traversed during computation, the amount of space or time required, etc [161]. In formal

logic and its realization such as ‘rule-based systems’ (a.k.a production systems), its primary

aim is to find ‘extensions’ based on their internal ‘search’ mechanism upon input facts.

Conversely, a semantic reasoning is ‘intensional’ precisely when it is concerned with

24
 Contents

such internal details in the given knowledge model [161]. These two classifications are

relative more than absolute, as the definition of ‘final result’ and ‘internal details’ may

Chapter 2
vary depending on the purpose of the semantic models and reasoning aspect.

While efforts in AI have often relied on formalisms such as mathematical formal

logic as a means of representing knowledge and reasoning ‘extensionally’ [130], much of the

recent work in machine learning [62, 130] can be seen as providing ‘intensional’ definitions

of concepts in terms of classifiers. Given a training corpus, these methods by themselves

defines a mechanism of deciding whether a given example is an instance of a particular

concept or not. They do not generally produce an explanation of why it is a member of

that category and do not incorporate a description of the properties of the object. In fact

many statistical learning paradigms such as support vector machines rely on simplifying

and abstracting the classification problem to minimize the amount of information about

the problem that needs to be represented while maximizing the separability of the different

classes. A purely ‘extensional’ representation on the other hand consists of rules or state-

ments whose truth may be assessed independently of the particular extensions involved,

i.e. regardless of which element of the modeled class is being considered.

In the following we will briefly review ‘intensional approaches’ and ‘extensional ap-

proaches’ especially in consideration how these approaches handle uncertainty and seman-

tics.

2.3.1 Intensional Approaches

In intensional approaches, also known as ‘state-based approaches’, uncertainty is attached

to ‘states of affairs’ or subsets of ‘possible worlds’. Such systems treat uncertainty by con-

nectives that combine sets of worlds by set theory operations. For example, the probability

P (A ∧ B) is given by the weight assigned to the intersection of two sets of worlds, those

in which A is true and those in which B is true, but P (A ∧ B) can not be determined

from the individual probabilities P (A) and P (B). In intensional systems, the rules denote
m
elastic constraints about the world. For example, the rule A −→ B does not describe how

25
Contents

an agent reacts to the finding of A, but asserts that the set of worlds in which A and

¬B hold simultaneously has low likelihood and hence should be excluded with probability
m
m. In the Bayesian formalism the rule A −→ B is interpreted as a conditional probability

expression P (B|A) = m, stating that among all worlds satisfying A, those that also satisfy

B constitute an m percent majority.

2.3.2 Extensional Approaches

Extensional approaches also known as rule-based systems or production-systems treat un-

certainty as a generalized truth value; that is, the certainty of a formula is defined to be a

unique function of the certainties of its sub formulas. Thus, the connectives in the formula

serve to select the appropriate weight-combining function. The certaint-factors calculus

used in MYCIN [36] is a well-known example of ‘extensional system’. For example, the

certainty of the conjunction A ∧ B is given by some function of the certainty measures

assigned to A and B individually. The rules in extensional systems provide license for
m
certain symbolic activities. For example, a rule A −→ B may mean “If you see A, then you

are given the license to update the certainty of B by a certain amount which is a function

of the rule strength m”. The rules are interpreted as a summary of past performance of

the problem solver, describing the way an agent normally reacts to problem situations or

to items of evidence.

Consider Figure 2.3 that depicts the combination functions that apply to serial and

parallel rules, from which one can form a ‘rule network’. In extensional approach, it

is required to define a modular procedure for determining the certainty of a conclusion,

given the credibility of each rule and the certainty of the premises (i.e., the roots of the

rule network). To complete the calculus we also need to define combining functions for

conjunction and negation as well.

26
 Contents

Chapter 2
Figure 2.3: An Example of Handling Uncertainty in Extensional Approach - Certainty
Combination Function used in MYCIN [130].

2.3.3 Extensional vs. Intensional : Merits and Deficiencies

Extensional approach tend to be computationally efficient but semantically sloppy while

intensional approach tend to be semantically clear but computationally clumsy .

In extensional systems, setting mathematical detail aside, it is important to notice

that the same combination function applies uniformly to any two rules in such systems,

‘regardless of what other rules might be in the neighborhood’. This is mainly due to the

nature of ‘modularity’ of inference in classical logic. For example, the logical rule “If A

then B” has the following procedural interpretation:

• locality : If you see A anywhere in the knowledge base, then ‘regardless of what other

things’ the knowledge base contains, you are given the license to assert B and add it to

the database.

• detachment : If you see A anywhere in the knowledge base, then regardless of how

A was derived, you are given the license to assert B and add it to the database.

27
Contents

When it is considered with uncertainty, the procedural license provided by the rule
x
A −
→ B reads as follows : “If you see the certainty of A undergoing a change δA , then

regardless of what other things the knowledge base contains and regardless of how δA was

triggered, you are given an unqualified license to modify the current certainty of B by some

amount δB , which may depend on x, on δA , and on the current certainty of B.”

Contrary, in intensional approach such as probability statements, P (B|A) = p, does

not give us license to do anything. Even if we are fortunate enough to find A true in the

database, we still can not assert a thing about B or P (B), because the meaning of the

statement is “If A is true and A is the ‘only’ thing that you know, then you are given license

to attach a probability p to B.”. As soon as other facts K appear in the database, the license

to assert P (B) = p is automatically revoked, and we need to look up P (B|A, K) instead.

The probability statement leaves us totally impotent, unable to initiate any computation,

unless we can verify that everything else in the knowledge base is irrelevant. This is why

verification of irrelevancy is so crucial in intensional systems.

However, there are known semantic penalties imposed in extensional approaches

when relevance considerations are ignored. Such semantic deficiencies are posed :

• The Limits of Modularity in Retracting Conclusions : In extensional systems,

‘detachment’ can create problems. In deductive logic the following holds true: A → B

and B → C =⇒ A → C. In other words, finding evidence for A leads us to conclude C

by chaining. Derived evidence B triggers the rule B → C with the same rigor as would

a directly observed proposition. However consider the case, “ground is wet → it rained”

and ”sprinkler is on → ground is wet”. In this case, if an extensional system is told that

sprinkler is on, it will conclude that it rained. This is incorrect and in fact finding that

the sprinkler was on should only reduce the likelihood that it rained.

• Improper Handling of Bidirectional Inferences : Plausible reasoning requires

that both predictive as well as diagnostic components of reasoning be used. if A → B,

then finding B to be true makes A more credible (abductive reasoning). This requires

28
 Contents

reasoning both ways. Extensional systems do not allow such bidirectional inference i.e.

reasoning from both A to B and B to A. To implement this in extensional systems,

Chapter 2
one has to explicitly specify the reverse rule, possibly risking creation of a cycle that

can cause evidence to be cyclically amplified until both cause and effect are completely

certain with no apparent factual justification. Removing the predictive component pre-

vents system from exhibiting another important pattern of plausible reasoning called

explaining away: if A → B and C → B and B is true, then finding C is true makes A

less credible. To exhibit this kind of reasoning, the system must use bi-directed infer-

ences; from evidence to hypothesis and from hypothesis to evidence. While it might be

possible to get around this problem by exhaustively listing all possible exceptions, to re-

store explaining away (without the danger of circular reasoning), any system that does

that sacrifices on principles of modularity. Inversely, any system that updates beliefs

modularly and treats all rules equally is bound to defy patterns of plausible reasoning.

• Improper Treatment of Correlated Sources of Evidences: Due to ‘locality’,

extensional systems do not store information on how a proposition was derived. As a

result, they risk treating correlated evidence as independent. Consider a situation where

someone hears a piece of news independently from the radio, television as well as the

newspapers. Since from his point of view, the sources are independent, his belief in the

veracity of the piece of news should be very high. However, if that person were to realize

later that all the three sources got their information from the same source, then his belief

in the piece of news should decrease. This can never happen in extensional systems as

they treat each source of information completely independently of the others.

In this dissertation, we propose an ‘extensional reasoning framework’ that comes with

remedies for the problem of ‘bidirectional reasoning’ and ‘limitation of modularity’. The

former was addressed in the sense of interpreting conditional propositions in bi-directional

way and the latter was addressed in the sense of ‘meta-reasoning’ using ‘abductive logic

programming’ [50].

29
Contents

2.4 Automatic Interpretation of Semantic Occurrence in

Video

This chapter examines some relevant research trends and presents an overview of published

work that is related to automatic interpretation of high level semantic occurrence in video

sequences. We have chosen to organize semantic models into three different categories :

‘pattern recognition methods’, ‘state models’ and ‘semantic models’. ‘pattern recognition

methods’ do not generally address the representation aspect of semantic models and rather

trained and specified by feature vectors of training data. ‘state models’ represent domain

knowledge in a graphical model. ‘semantic models’ uses formal languages for knowledge

representation. The ‘state models’ and ‘semantic models’ categorizations again fall into

aforementioned ‘intensional’ and ‘extensional’ approaches. In the following sections, we

will further take a more in-depth look at the three categories of semantic interpretation

methods, and explore examples from the literature.

2.4.1 Pattern Recognition Methods - Data Centric Approaches

This class of techniques in this section is not quite semantic models, in the sense that they

do not consider the problem of semantic representation. Instead, they focus on the event

recognition problem, which is formulated as a traditional pattern-recognition and classi-

fication problem. Accordingly, traditional approaches to these problems such as support

vector machines, neural networks, nearest neighbor classifiers, etc., are applied to the per-

ception scheme. Minimal semantic knowledge is needed in building the semantic classifiers

in this category.

The main advantage of the classifiers in this category is that they are well under-

stood. Usually, they may be fully specified from a set of training data. These approaches

are usually simple and straightforward to implement. This simplicity is afforded by ex-

cluding semantics (i.e., high-level knowledge about the semantic domain) entirely from the

specification of the classifier.

30
 Contents

There are many examples of pattern-recognition methods especially for event recog-

nition in the literature including [29, 180, 27, 71] and [151] (nearest neighbor), [136, 63, 135]

Chapter 2
(Support Vector Machine), [124, 157, 145] (boosting) and [167] (neural networks). A more

comprehensive discussion of these approaches and examples can be found in [12, 102, 165,

86].

2.4.2 State Based Models - Intensional Approaches

“State” based semantic models are a class of formalisms that model the state of the video

semantics in space and time using semantic knowledge. State models improve on pattern-

recognition methods in that they ‘intrinsically’ model the structure of the state space of

the semantic domain. Such approaches fall into the category of ‘intensional’. Modeling

formalisms in this category are also well studied and mathematically well formulated.

This allows for efficient algorithms and sound formulations of problems such as parameter

learning. In most cases, however, the semantic information associated with the model

structure makes this structure difficult to learn from training data. In such approaches,

states are considered as symbolic facts and contextual knowledge is represented as a graph

structure having state nodes that are connected to each other. In the sense of logic,

connected two state nodes can be interpreted as a propositional logic rule that can consider

only one relation, ‘the causality implication’ (that can be often interpreted as ‘influence’,

‘affect’, ‘cause’ in natural language). A piece of propositional knowledge segment should

exist within the whole graph structure, thereby, once uncertainty propagation mechanism is

learnt, adding additional pieces of knowledge will require restructuring causality influence

relation of the whole graph structure. This aspect restricts expressive power and increases

the modeling cost. Due to this complexity and lack of modularity, such approaches have

been focusing on relatively narrow and specific semantics. State modeling formalisms also

include FSMs (Finite-State Machines), Bayesian networks (BNs), hidden Markov models

(HMMs) and dynamic BNs (DBNs).

FSMs assume a ‘fully observable sequence of states’, therefore have been used for

31
Contents

modeling the temporal aspects of video semantics with less concerns on uncertainty. FSMs

appearing in the literature naturally model single thread ‘event’ formed by a sequence of

states. FSM semantic models are utilized include hand gestures [91], single-actor actions

[108] and aerial surveillance [117].

In order to deal with the inherent uncertainty of observations and interpretation that

exists in video, semantic models utilizing probability have been proposed. One such event

modeling formalism is the BNs (also known as probability network) [90]. In BNs, states

are considered as a random variable and they are connected with acyclic edges. Edges

represent joint probability between states using the notion of conditional independence.

BN semantic models are utilized for indoor surveillance [109] such as left luggage detection,

Hongeng et al. [85] considers an activity to be composed of action threads and recognizes

activities by propagating constraints and likelihood of event threads using a BN model,

Remagnino et al. [144] uses BNs for parking lot surveillance, Wang et al. [170] uses a

BN model for primitive activity detection such as ‘jay walking’. One major drawback of

BN based semantic models is that they do not have an inherent capacity for modeling

temporal composition, which is an important aspect of video semantics. Solutions to this

problem include single-frame event classification [37] and choosing abstraction schemes

that encapsulate temporal properties of the input [109].

The benefits of a temporal evolution model (like FSM) and a probabilistic model

(like BN) are combined within the framework of the hidden Markov model (HMM). HMMs

are a class of directed graphical models extended to model the temporal evolution of the

state [142]. Due to this aspect, HMMs have become one of the most popular formalisms for

modeling video semantics. Makris et al. [113] uses HMM to reason about human behaviors

based on trajectory information. Oagale et al. [127] models single-person activities such

as ‘walking’ and ‘kneeling’. Gong et al. [72] applies a HMM model for airport tarmac

surveillance. Oliver et al. [128] use a layered HMM (LHMM) model in the event domain of

office surveillance. Due to the markov assumption that the current state depends only on

the state at a previous time, the semantics recognized in these works are tend to be mostly

32
 Contents

a few seconds in length. One drawback of HMM is that as the model topology becomes

more complex, the efficient exact algorithms associated with the HMM structure are no

Chapter 2
longer applicable and must be replaced with approximation algorithms.

Dynamic BNs (DBNs) generalize BNs with a temporal extent. In fact, HMMs are a

special case of DBNs in which the structure is restricted to provide efficient algorithms for

learning and inference. This, however, often comes at the cost of computational tractability.

Approximation techniques are usually used to perform learning and inference. Thus, DBNs

in their general form appear less often as semantic modeling formalism in the literature.

[120] and [121] apply DBNs for surveillance of people such as ‘entering’, ‘passing’, etc.

Other approaches use a qualitative representation of uncertainty [58], a context rep-

resentation ‘scheme’ for surveillance systems [34], AND/OR tree for the analysis of specific

situations [46], or a GMM based scene representation for reasoning upon activities [114].

These extensions to the formalism have attempted to introduce aspects such as hier-

archy and uncertainty. These methods have largely been applied to specific event domains

and have not been embraced as general solutions.

2.4.3 Semantic Based Models - Extensional Approaches

While semantics such as many types of events can be described as a sequence of a number

of states, an interesting subset of semantics are those defined by the semantic relationships

between their composing sub-events or sub-semantics. The category of “compositional high

level semantic models” groups several primitive unit semantics to allow these kinds of re-

lationships to be represented and recognized. These approaches do not aim to define the

‘entire state space’ of desired semantic domain as in ‘state modeling’ (intensional) ap-

proaches. Instead, semantic knowledge is used to define a set of semantic rules, constraints

and relations.

This type of approach allows the event model to capture high-level semantics such

as long-term temporal dependence (e.g. meeting, eating, etc.), hierarchical semantics (e.g.

boss of, employee of, etc.), semantically complex relations (e.g. friends of, thief of) using

33
Contents

primitive sub-semantics (i.e. mostly basic events or perceptional object/human recognition

results). Thus, approaches in this category are usually applied in domains where the

semantics (e.g. events, human activities or human interactions) of interest are relatively

complex and a particular semantics has large variance. However, because of the high-

level nature of this class of models, they often must be manually specified by a domain

expert (i.e., learning model structure and parameters is generally infeasible). Generally,

the formalisms in this category of event models such as grammars and logic formalisms

comes without uncertainty handling mechanism.

To cope with uncertainty, there have been some work on the use of logic program-

ming languages to achieve better expressive power and on the use of different uncertainty

handling formalisms to reason under uncertainty. Such logic framework based uncertainty

handling approaches can be categorized as ‘extensional’.

Yuri et al. [88] use a stochastic grammar and its parser for parking lot surveillance.

Ogale et al. [126] also use a stochastic grammar for human activity recognition. However,

in logical sense, their grammar rules corresponds to propositional logic, therefore has much

overlap with graphical ‘intensional’ models in the sense that it can not really represent

predicates as first-order logic. In fact, the achievement of better expressive power in ‘exten-

sional’ approaches is mainly due to the first-order predicate logic that logic programming

provides. While propositional logic deals with simple declarative propositions, first-order

logic additionally covers predicates and quantifiers. Akdemir et al. [13] proposed an ontol-

ogy based approach for activity recognition, but without uncertainty handling mechanism

(In ontology community, Description Logics (DLs) are often used as knowledge represen-

tation formalism and DLs are decidable fragments of first-oder-logic [24, 2]). Shet et al.

[152] proposed a system that adopts Prolog based logic programming for high-level reason-

ing. In [155] the same authors extended their system with the bilattice framework [69] to

perform the task of detecting humans under partial occlusion based on the output of parts

based detectors. Jianbing et al. [110] used rule-based reasoning with Dempster Shafer’s

Theory [149] for a bus surveillance scenario. Anderson et al. [17] used Fuzzy Logic [177] to

34
 Contents

Ogale / Yuri Akdemir Jianbing Shet et. al Anderson / Our approach

Approach
et al. [126, 88] et al. [13] et al. [110] [152, 155] Dorado et al. [81, 80, 82]
[17, 54] [83, 79]

Chapter 2
Knowledge Rule Based Ontology Rule Based Rule Based Rule Based Rule Based
Modeling (Prop. L) (DL) (FOL) (FOL) (Prop. L) (FOL)

Uncertainty Stochastic Dempster Fuzzy Subjective

- Bilattice
Formalism Grammar Shafer Logic Logic

Traditional √ √ √
- - -
Logic Operators

Arithmetic √
- - - - -
Operators

Info. Fusion √ √ √
- - -
Operators

Extra Operators √
- - - - -
MT, MP,Rep,etc.
√
Default √
- - - -
Reasoning Chapter 5
√
Vague Rule √
- - - -
Modeling Chapter 6
√
Bidirectional
- - - - -
Inference Chapter 7
√
Diagnostic
- - - - -
Abduction Chapter 8
√
Belief √
- - - -
Revision Chapter 5-8

Table 2.2: A Comparison of Previous Extensional Approaches.

model human activity for video based eldercare. Dorado et al. [54] also applied fuzzy rules

but directly to low-level features for annotation of broadcasting contents. Therefore, their

focus was more on ‘vision based perception’ aspect. They defined propositional logic based

rules in form of ‘<condition> → <action>’, where ‘<condition>’ expresses the instances

of low-level features and ‘<action>’ denotes annotating a word (concept). For example,

using the MPEG-7 edge histogram descriptor (EHD), they mapped an EHD to a fuzzy set

elements such as V (ertical), H(orizontal), D(iagonal) and N (on)D(irectional) then built

a rule such as ‘V is H(igh) ∧ H is M(edium) ∧ D is L(ow) ∧ ND is L(ow) → label as

BU ILDIN G’. Such V , H, D and N D can be considered as state symbols that can be

35
Contents

posed by low-level features. Because their rules are propositional it can be also represented

as a graph having such symbols as nodes that are connected to a word symbol. In this

sense their approach is similar to ‘state base model’.

In this dissertation we proposed the use of logic programming and subjective logic

[93] to encode contextual knowledge with uncertainty handling. Based on our previous

work [78, 81, 80, 82, 83, 79], we extend the system to demonstrate bidirectional condi-

tional inference, belief revision, default reasoning and vague rule modeling. Table 2.2

shows a brief comparison of the previously proposed ‘extensional’ approaches. The ta-

ble shows that the coverage of our subjective logic based approach is most broad. For

example, while some provides information fusion capability for fusing two contradictory

information sources, such as dempster shafer’s fusion operator, bilattice’s operator and

subjective logic’s consensus operator, only some of them support default reasoning that

handles such contradictory information to draw reasonable decision and belief revision.

Indeed, our system also supports modeling vague propositional rules and inference under

such vague rules for high level semantic analysis of visual surveillance data (in the sense

of linguistic interpretation of the rules, the most similar previous approach to the pro-

posed work would be [17]). Finally, bidirectional inference (as is possible in ‘intensional’

approaches) is only supported by subjective logic based approach.

36
3 Preliminaries

Chapter 3
This chapter gives an overview of the fundamental background about subjective logic

theory and logic programming that will be discussed throughout this dissertation.

3.1 Subjective Logic Theory

Audun Jøsang introduced subjective logic as a framework for artificial reasoning [92, 93].

Unlike traditional binary logic or probabilistic logic (the former can only consider true or

false, and the latter can consider degrees of truth or falseness), subjective logic explicitly

represents the amount of ‘lack of information (ignorance) on the degree of truth about a

proposition’ in a model called ‘subjective opinion’ [92]. The idea of explicit representation

of ignorance is inherited from the Dempster Shafer belief theory [149, 92, 149] and the in-

terpretation of an opinion in bayesian perspective is possible by mapping opinions into beta

distributions [93]. Subjective logic also comes with a rich set of operators for the manipu-

lation of opinions. In addition to the standard logical operators, subjective logic provides

some operators specific for Dempster Shafer belief theory such as consensus and recom-

mendation. However, unlike Dempster Shafer’s evidence fusion rule that is inconsistent

with Bayes theorem, it provides an alternative consensus rule with a solid mathematical

basis [92]. It is also different from fuzzy logic: While fuzzy logic maps quantitative mea-

sure to non-crisp premises called fuzzy sets (e.g. ‘fast’, ‘slow’, ‘cold’, ‘hot’, etc.), subjective

logic deals with the uncertain belief itself on a crisp premise (e.g. ‘intrusion happened’,

‘accident happened’, etc.). However, in the sense of interpretation, mapping of an opinion

into the linguistic certainty fuzzy set (i.e., ‘very certainly true’, ‘less certainly true’, etc)

37
Contents

is also possible. In general, subjective logic is suitable for modeling real situations under

partial ignorance on a proposition’s being true or false.

Known application areas are trust network modeling, decision supporting, modeling

and analyzing Bayesian network, etc. However, to the best of our knowledge, the applica-

tion of subjective logic in computer vision related domains has been limited to our previous

work [78, 81, 80, 82, 83, 79]. In this dissertation, based on our previous work, we demon-

strate the capability of default reasoning to handle contradictory information, bidirectional

interpretation of conditional rules using abduction and deduction operators and modeling

vague rules relying on reputation operator in subjective logic. In this section, we will give

a brief introduction to subjective logic theory.

3.1.1 Dempster Shafer Belief Theory

Subjective logic uses theoritical elements from the Dempster-Shafer belief theory [149].

In this section, we give a brief introduction to Dempster Shafer belief theory with an

example. Dempster Shafer belief theory deals with a set of hypotheses called the ‘frame

of discernment’, conventionally denoted as Θ. The elements of Θ are considered to be

mutually exclusive, therefore exactly one element is assumed to be true. To illustrate the

frame of discernment let us consider following example shown by Zadeh [178] (The same

example was also used by Audun Jøsang in [98, 94]).

Example 1. (A Zadeh’s Example on Murder Case) . Suppose that we have a murder

case with three suspects; Peter, Paul and Mary then Figure 3.1 is an example of frame

of discernment shown in [94]. If an element is assumed to be true, its supersets are also

considered to be true as well. An observer can assign belief mass to one or several states

in the powerset of Θ (denoted 2Θ ).

Definition 1. (Belief Mass Assignment) [93] . Let Θ be a frame of discernment. A

belief mass assignment is a mapping mΘ : 2Θ → [0, 1] such that mΘ (x) ≥ 0, mΘ (∅) = 0

∑
and x⊆2Θ mΘ (x) = 1.

38
 Contents

Figure 3.1: An Example of Frame of Discernment Θ containing 3 Elements [94].

For example, if an observer assigned belief mass to a set x4 = {x1 , x3 }, it is considered

Chapter 3
that one of the elements in x4 is true but the observer is uncertain about which of them is

true (i.e., ignorance). In contrast to belief mass, the ‘belief ’ to a set x should be the sum

of belief mass assigned to its subsets. Namely,

Definition 2. (Belief Function) [93] . Let Θ be a frame of discernment, and let mΘ be a

BMA on Θ. Then the belief function corresponding with mΘ is the function b : 2Θ → [0, 1],
∑
such that b(x) = y⊆x mΘ (y), where x, y ∈ 2Θ .

This idea can be expanded to the cases of ‘disbelief ’ and ‘ignorance’ as follows.

Definition 3. (Disbelief Function) [93] . Let Θ be a frame of discernment, and let

mΘ be a BMA on Θ. Then the disbelief function corresponding with mΘ is the function

∑
d : 2Θ → [0, 1], such that d(x) = y∩x=∅ mΘ (y), where x, y ∈ 2Θ .

Definition 4. (Ignorance Function) [93] . Let Θ be a frame of discernment, and let

mΘ be a BMA on Θ. Then the ignorance function corresponding with mΘ is the function

∑
i : 2Θ → [0, 1], such that i(x) = y∩x̸=∅,y⊈x mΘ (y), where x, y ∈ 2Θ .

By the Definition 1, the sum of the ‘belief ’, ‘disbelief ’ and ‘ignorance’ is equal to the

sum of the belief masses which sums up to 1. Therefore, b(x) + d(x) + i(x) = 1. Given the

elements in frame of discernment, the relative number of atomic elements is called ‘relative

atomicity’ which is formally defined as follows.

Definition 5. (Relative Atomicity) [93] . Let Θ be a frame of discernment, and let

x, y ∈ 2Θ . Then for any given y ̸= ∅, the relative atomicity of x to y is the function

|x∩y|
a : 2Θ → [0, 1] such that a(x/y) = |y| , where x, y ∈ 2Θ , y ̸= ∅ (Especially, a(x/Θ) is

simply written as a(x)).

39
Contents

mA
Θ bA (x) dA (x) iA (x) mB
Θ Dempster’s Non-normalised
x∈2 Θ
rule Dempster’s rule
Peter 0.98 0.98 0.01 0.01 0.00 0.490 0.0098
Paul 0.01 0.01 0.98 0.01 0.01 0.015 0.0003
Mary 0.00 0.00 0.99 0.01 0.98 0.490 0.0098
Θ 0.01 1.00 0.00 0.00 0.01 0.005 0.0001
∅ 0.00 0.00 1.00 0.00 0.00 0.000 0.9800

Table 3.1: An Example of Applying Dempster’s Rule shown in [94].

On a frame of discernment, a probability expectation can be calculated by the fol-

lowing definition.

Definition 6. (Probability Expectation) [93] . Let Θ be a frame of discernment with

BMA mΘ , then the probability expectation function corresponding with mΘ is the function
∑
E : 2Θ → [0, 1] such that, E(x) = y mΘ (y)a(x/y), where y ∈ 2Θ .

In Dempster Shafer belief theory, belief mass assignment mΘ plays the most fun-

damental basis for representing belief and calculating probability expectation, etc. In

addition, Dempster Shafer belief theory comes with a rule to combine two different belief

mass assignments which is defined as follows.

Definition 7. (Dempster’s Rule) [94] . Let Θ be a frame of discernment, and let mA

Θ
A ′
and mB Θ be belief mass assignment on Θ of two observers A and B. Then mΘ ⊕ mΘ is a
B

′ B A ′ B
∑
function mAΘ ⊕ mΘ : 2 → [0, 1] such that, 1. mΘ ⊕ mΘ (∅) =
Θ A B
y∩z=∅ mΘ (y)·mΘ (z)−Kand
∑
′
A B
y∩z=x mΘ (y)·mΘ (z)
∑
2. mAΘ ⊕ mΘ (x) = , for all x ̸= ∅, where K = y∩z=∅ mA Θ(y) · mB
B
1−K Θ (z)

and K ̸= 1 in standard Dempster’s rule, and where K = 0 in the non-normalized version

of Dempster’s rule.

Suppose that two observers assigned their belief masses to the frame of discernment

shown in Figure 3.1 that its elements are suspect of a murder case. Table 3.1 shows an

example of using dempster shafer theory and dempster’s rule.

40
 Contents

3.1.2 Opinion Model in Subjective Logic

The ideas of Dempster Shafer belief theory can be simplified in subjective logic theory, by

restricting the frame of discernment to be binary, i.e., it will only contain (focus on) one

particular set and its complement. Such a frame of discernment is called ‘focused frame

of discernment’. Figure 3.2 shows examples of focused frame of discernment that can be

derived from Figure 3.1. The formal definition of ‘focused frame of discernment’ is as

Chapter 3
follows.

Definition 8. (Focused Frame of Discernment) [93] . Let Θ be a frame of discernment

and let x ∈ 2Θ . The frame of discernment denoted by Θ̃x containing only x and ¬x(i.e., x,

where ¬x is the complement of x in Θ is then called a focused frame of discernment with

focus on x.

Remember that the belief, disbelief and ignorance functions are also indexed by a

specific element in a frame of discernment (i.e., b(x), d(x), i(x), respectively). By this

‘focused belief mass assignment’ is defined as follows.

Definition 9. (Focused Belief Mass Assignment) [93] . Let Θ be a frame of discern-

ment with belief mass assignment mΘ where b(x), d(x) and i(x) are the belief, disbelief and

ignorance functions of x in 2Θ , and let a(x) be the real relative atomicity of x in Θ. Let

Θ̃x be the focused frame of discernment with focus on x. The corresponding focused belief

mass assignment mΘ̃x and relative atomicity aΘ̃x (x) on Θ̃x is defined according to:


 m x (x) = b(x) 

 Θ̃  a x (x) = E(x)−b(x) , f ori(x) ̸= 0
Θ̃ i(x)
mΘ̃x (¬x) = d(x) .

  a (x) = a(x), f ori(x) = 0

 Θ̃x
mΘ̃x (Θ̃x ) = i(x)

Opinion model called ‘opinion triangle’ or ‘subjective opinion’ in subjective logic is

represented based on the concept of focused belief mass assignment as shown in Figure 3.3.

The formal definition of subjective opinion model is as follows.

41
Contents

Figure 3.2: Examples of Focused Frame of Discernment Derived from Figure 3.1.

Definition 10. (Opinion) [93] . Let Θ = {x, ¬x} be a state space (a.k.a ‘Frame’)

containing x and its complement x. Let bx , dx , ix represent the belief, disbelief and ig-

norance in the truth of x satisfying the equation: bx + dx + ix = 1 and let ax be the base

rate of x in Θ. Then the opinion of an agent ag about x, denoted by wxag , is the tuple

wxag = (bag ag ag ag
x , dx , ix , ax ).

In the context of the frame of discernment shown in Figure 3.1 and belief mass

assignment shown in Table 3.1, the examples of the focused opinions of agent A are wPAeter =

(0.98, 0.01, 0.01, 1/3), wPAaul = (0.01, 0.98, 0.01, 1/3) and wM
A
ary = (0.00, 0.99, 0.01, 1/3).

The probability expectation can be also defined on the opinion model in subjective logic.

Definition 11. (Probability Expectation) [93] . Let wxag = {bag ag ag ag

x , dx , ix , ax } be an

opinion about the truth of x, then the probability expectation of wxag is defined by: E(wxag ) =

bag ag ag
x + ax ix .

A point inside the triangle represents a (bx , dx , ix ) triple. The corner points marked

with Belief, Disbelief and Ignorance represent the extreme cases, i.e., full belief (1, 0, 0),

full disbelief (0, 1, 0) and no knowledge (0, 0, 1). An opinion lying on base line (the line

connecting Belief and Disbelief) is called ‘dogmatic opinion’ in the sense that they do not

contain any degree of ignorance (i.e., ix = 0). Such dogmatic opinions correspond to

traditional traditional probability. The base rate ax represents the prior knowledge on the

tendency of a given proposition p’s being true and can be indicated along the base line.

42
 Contents

Chapter 3
Figure 3.3: Opinion Triangle.

Figure 3.4: Linguistic Fuzzy Category for Opinion Triangle as a Function of the Base Rate
[139].

43
Contents

For example, in the case we toss a balanced coin with the proposition x as ‘get head’,

then we put ax = 1/2 however, in the case of biased coin we could set different values.

Similarly, in the case we toss a balanced dice we put ax = 1/6 (note that, the original

frame of discernment has six elements). Usually, when we consider balanced binomial

cases, the default value is 1/2. The probability expectation E is then formed by projecting

the opinion onto the base line, parallel to the base rate projector line (see the blue line)

that is built by connecting the ax point with the Ignorance corner (see the red line).

As human language provides various terms to express various types of likelihood

and uncertainty, a linguistic interpretation of an opinion is also possible. Pope et al. [139]

showed an example mapping of fuzzy categories to opinion triangle. As shown in Figure 3.4,

the stronger opinion we have, the lesser changes of mapping areas are made. Similarly, the

weaker opinion we have, we are relying the more on base rate. For example, the area 5A

can be interpreted linguistically as ‘completely certain chances about even’ and the area 2B

can be interpreted as ‘very likely but slightly uncertain’.

3.1.3 The Correlation between Subjective Opinion and Beta Distribution,

The Bayesian Perspective

An interesting property of subjective opinions is their direct mapping to beta distributions.

As shown in the Definition 10, subjective opinion’s frame Θ is binomial, because it deals

with two elements about a proposition x. Namely, x (i.e., x happened, success) and x (i.e.,

x does not happened, failure). Assuming n number of independent observations about the

proposition x with a fixed base rate ax = π, the conditional probabilistic likelihood that x

could happen y times is represented as f (y|π) = n Cy π y (1 − π)n−y , f or y = 0, ..., n. Here,

we are holding ax fixed and are looking at the probability distribution of y over its possible

discrete integer values of 0, ..., n. This is referred to as ‘binomial distribution’ and noted

as binomial(n, ax ). If we look at this same relationship between the probability π and

number of successes y, holding y fixed and let π vary over its possible values 0 ≤ π ≤ 1.

Then we get the likelihood function given by f (y|π) = n Cy π y (1 − π)n−y , f or 0 ≤ π ≤ 1.

44
 Contents

Taking a different view, let us assume that we want to know about the probability density

of the probability π itself given y successes of observations, namely g(π|y). By the Bayes’

theorem, we know that ‘posterior ∝ prior × likelihood’, therefore, g(π|y) ∝ g(π) × f (y|π).

This gives us only the shape of the posterior density. To get the actual posterior, we

need to normalize this by some constant k, to make sure that the area under the posterior

integrates to 1. We find k by integrating g(π) × f (y|π) over the whole range. In general

Chapter 3
we get,
g(π) × f (y|π)
g(π|y) = ∫ 1 , (3.1)
0 g(π) × f (y|π) dπ

In above Equation 3.1, it requires an integration. However, depending on the prior g(π)

chosen, we do not always need to do the actual integration numerically. Assume that we

do not have any idea beforehand what the π is, therefore, we assign,

Using a Uniform Prior : g(π) = 1, f or 0 ≤ π ≤ 1. (3.2)

Clearly, we see that in this case, the posterior density is proportional to the likelihood

therefore, g(π|y) = n Cy π y (1−π)n−y , f or 0 ≤ π ≤ 1. This equation can be slightly changed

by introducing the number of successes α = y + 1 and number of failures β = n − y + 1.

(α+β−2)!
g(π|α, β) = (α+β−2) Cα π
α−1 (1 − π)β−1 = (α−1)!(β−1)! π
α−1 (1 − π)β−1 . Here, each of the

factorial part in the coefficient can be generalized to the cases of real values using the

property of the gamma function1 , Γ(n) = (n − 1)!. Therefore, we get,

Γ(α + β) α−1
g(π|α, β) = π (1 − π)β−1 = Beta(π|α, β), f or 0 ≤ π ≤ 1 (3.3)
Γ(α)Γ(β)

Above Equation 3.3 is the probability density function called beta distribution 2 that its

shape is only dependent on the indexes α and β. Now, let us consider assigning the beta

1
The Gamma function is a generalization of the factorial function over integer ∫to real and complex
∞
numbers, with its argument shifted down by 1. Formally, it is defined as Γ(z) = 0 tz−1 e−t dt.
2 1
Note that, the coefficient part of the beta distribution is B(α,β) , where B(α, β) is a special function
∫1
called Beta function that is formally defined as B(α, β) = 0 tα−1 (1 − t)β−1 dt = Γ(α)Γ(β)
Γ(α+β)
.

45
Contents

prior to g(π) instead, in the step (3.2).

Γ(α + β) α−1
Using a Beta Prior : g(π; α, β) = π (1 − π)β−1 , f or 0 ≤ π ≤ 1. (3.4)
Γ(α)Γ(β)

Then we get, g(π|y) ∝ g(π; α, β) × π y (1 − π)n−y = k × π y+(α−1) (1 − π)n−y+(β−1) again by

the Bayes’ theorem. To make g(π|y) a probability density function that the area under

the function integrates to 1, we need to calculate,

k × π y+(α−1) (1 − π)n−y+(β−1)
g(π|y) = ∫ 1 = k ′ × π y+(α−1) (1 − π)n−y+(β−1) . (3.5)
0k×π
y+(α−1) (1 − π)n−y+(β−1) dπ

In the above Equation 3.5, we can easily recognize that this can be a beta distribution

with parameters α′ = y + α and β ′ = n − y + β. Therefore, we get,

Γ(n + α + β)
g(π|y) = π y+α−1 (1 − π)n−y+b−1 , f or 0 ≤ π ≤ 1. (3.6)
Γ(y + α)Γ(n − y + β)

Again, the posterior density of π has been easily obtained without having to go through

the numerical integration. Furthermore, the posterior is to be the same form as the

Beta(π|α, β) distribution and a production of π to a power times (1 − π) to another

power. When we multiply the beta prior times the binomial likelihood, we just add the

exponents of π and (1 − π), respectively. This makes using Beta(π|α, β) priors when we

have binomial observations particularly easy. Such priors that make their posteriors of the

same form, is called ‘conjugate family of priors’. Such ‘conjugate priors’ play an important

role in bayesian statistics by simplifying numerical computations significantly. Through

our discussion, we explained that beta distribution is the conjugate family for the ‘bino-

mial’ observation distribution. Another advantage of the beta distribution is that it offers

various shapes according to its indexed parameters α and β as shown in Figure 3.6. In fact,

the uniform distribution g(π) = 1 used in (3.2) is a special case of beta distribution that is

indexed with α = 1 and β = 1, namely, Beta(π|1, 1) (see Figure 3.6-(b)). To explain this

advantage further, it is worth to remind the fundamental idea of bayesian statistics.

46
 Contents

Chapter 3
Figure 3.5: Examples of Beta Distributions.

Figure 3.6: Examples of Applying Bayes’ Theorem to Different Priors and Given a Likeli-
hood.

The essence of Bayesian statistics, as explained above, is the use of both the prior

knowledge and the experimental result called likelihood, to get the posterior knowledge.

For example, assume that we have tossed a biased coin 5 times today and got 2 times of

head. Then, what is the chance that we would get head if we toss the coin tomorrow ?

To infer this, ideally, it would be good if we could appropriately fuse a subjective belief or

opinion about the coin and an objective experimental likelihood (2 times of head out of 5

47
Contents

trials). The subjective opinion in bayesian statistics is called ‘subjective probability’ [33]. In

general, the subjective probability is not a single probability value but a distribution over all

possible probability values. Therefore, the characteristics that beta family of distributions

can form a various shapes give us high degree of flexibility on modeling the desired prior

density. Further, it is also known that some of other types of well known priors can be also

approximated to beta distributions [33]. For instance, Alfers et al. introduced a normal

(gaussian) approximation for beta and gamma probabilities [9], Teerapabolarn introduced

a poisson approximation to the beta binomial distribution [164], etc. (refer to [33] for more

details on beta distributions, conjugated priors and bayesian statistics).

In bayesian statistics, however, the effect of the prior we choose will be small when

we have a likelihood from enough data. For example, consider three students Anna, Bart

and Chris are constructing their prior belief about the proportion of residents in their city

who support building a casino (this example is shown in [33]). Based on their subjective

beliefs, let us assume that they have chosen weak priors as shown in Figure 3.6 - (a). Then

assume that they took a random sample of n = 100. Out of the random sample, 26 said

they support and 74 said they do not support building a casino. Figure 3.6 - (b) shows

this likelihood. Despite starting with different priors, when they are considered with the

likelihood, the three posteriors shown in Figure 3.6 - (c) show very similar posteriors. In

other words, in general, putting roughly shaped ‘subjective prior’ is considered to be enough

in case we have reasonable amount of data or reasonably strong likelihood distribution [33].

Audun Jøsang introduced the mapping between beta distribution Beta(Θ|α, β) and

subjective opinion wx = (bx , dx , ix , ax ) [92, 98]. Consider a beta distribution over the frame

(state space) Θ and let π be the probability that x ∈ Θ could happen. Replacing r = α − 1

number of positive observations (success) and s = β − 1 number of negative observations

(failure) in Equation 3.3, regarding the proposition x in the frame Θ, we get,

Γ(r + s + 2)
Beta(π|r, s) = π r (1 − π)s , f or 0 ≤ π ≤ 1, r ≥ 0, s ≥ 0. (3.7)
Γ(r + 1)Γ(s + 1)

48
 Contents

In above case, we have considered a default atomicity ax = 12 , so we had α = r+1 = r+2· 12

and β = s + 1 = s + 2 · (1 − 21 ). If we consider arbitrary atomicity ax , the index parameters

becomes α = r + 2 · ax and β = s + 2 · (1 − ax ). By this, it becomes as follows,

Γ(r + s + 2)
Beta(π|r, s, ax ) = π r+2ax −1 (1 − π)s+2(1−ax )−1 . (3.8)
Γ(r + 2ax )Γ(s + 2(1 − ax ))

Chapter 3
(r+2ax )
Then the probability expectation of Equation 3.8 is known to be as E(π) = (r+s+2) .
Similarly, by the Definition 11, Ewx = bx + ax ix . Then we need to make the two notations
of E to be the same (i.e., E(wx ) = bx + ax ix = E(π) = (r + 2ax )/(r + s + 2) ).

 bx + 
a a
x ix = r/(r + s + 2) + 2x /(r + s + 2)
 b +d +i =1
x x x

∴ by above and ‘affinity’ we get,



 (3.9)

 bx = r/(r + s + 2), increasing function of r



 dx = s/(r + s + 2), increasing function of s



 ix = 2/(r + s + 2), decreasing function of (r, s)



 ax = base rate of x

Now, remember that we have replaced the parameters α and β in Equation 3.7, with
α = r + 2 · ax and β = s + 2 · (1 − ax ), by this and above (3.9) we can get parameters r, s,
α and β as follows,

1. when ix ̸= 0,

 r = 2bx /ix → α = 2bx /ix + 2 · ax s = 2dx /ix → β = 2dx /ix + 2 · (1 − ax )
 1=b +d +i ax = base rate of x
x x x

∴ Beta(π|α, β) = Beta(π|2bx /ix + 2 · ax , 2dx /ix → β = 2dx /ix + 2 · (1 − ax )).

2. when ix = 0,
 (3.10)
 r = 2bx /ix → α = 2bx η s = 2dx /ix → β = 2dx η where, η → ∞
 1=b +d ax = base rate of x
x x

∴ Beta(π|α, β) = Beta(π|bx η, dx η) = δ(t − bx )


 +∞, t = bx
where, δ(t − bx ) is the Dirac Delta function such that, .
 0, t ̸= b
x

49
Contents

Figure 3.7: Examples of Opinion Triangle and Their Mapping to Beta Distribution.

This means, for example, that an opinion with bx = 0, dx = 0, ix = 1 and ax = 0.5 which

maps to Beta(1, 1) is equivalent to a uniform probability density function (see Figure 3.6-

(b)). It also means that a dogmatic opinion with ix = 0 which maps to Beta(bx η, dx η)

where η → ∞ is equivalent to a spike probability density function (i.e., dirac delta) with

infinitesimal width and infinite height at bx (see Figure 3.6-(g) and (h)). Dogmatic opinions

in beta distribution perspective means that infinite amount of evidences converges to the
α 3
ratio of α+β [98].

In Figure 3.7, example 1) shows an opinion about a proposition of an agent, that

can be interpreted as ‘seems likely and slightly uncertain true’, and example 2) shows full

ignorance (a.k.a. ‘vacous’ opinion) at the time of judgement about a proposition. Assuming

base rate to be 0.7 in the example we get expectation value also to be 0.7 and the beta

3
however, we introduce a bound to avoid the actual numerical computation of infinity value and to
visualize the distribution. Currently, we assign a very tiny value such as ix = 1 × 10−5 for numerical
calculation, when ix → 0.

50
 Contents

distribution appears biased towards ‘True’ though the opinion represents full ignorance.

3.1.4 Operators in Subjective Logic

Subjective logic is a generalization of binary logic and probability calculus. This means

that when a corresponding operator exists in binary logic, and the input parameters are

equivalent to binary logic TRUE or FALSE, then the result opinion is equivalent to the

Chapter 3
result that the corresponding binary logic expression would have produced. Table 3.2 pro-

vides a brief overview of the main subjective logic operators. Additional operators exist for

modeling special situations, such as when fusing opinions of multiple observers. Most of the

operators correspond to well-known operators from binary logic and probability calculus,

whereas others are specific to subjective logic. For example addition is simply a generaliza-

tion of addition of probabilities. However, subjective logic also comes with non-traditional

logic operators. For example, deduction[95, 100], abduction[95, 139], discounting [99] op-

erators are generalization of Bayesian interpretation of conditional probability and fusion

of different sources of observations. Cummulative (average) fusion (unfusion) [99, 95] oper-

ators are generalization that is inspired from the dempster shafer’s belief fusion rule [149]

(this particular operators will be further discussed in Chapter 5).

Apart from the computations on the opinion values themselves, subjective logic op-

erators also affect the attributes, i.e. the subjects, the propositions, as well as the frames

containing the propositions. In general, the attributes of the derived opinion are functions

of the argument attributes. Following the principle illustrated in Figure 3.8. the de-

rived proposition is typically obtained using the propositional logic operator correspond-

ing to the subjective logic operator. For example, consider two frames of discernment

X = {healthy, cough, f ever, dizzy} and Y = {dayworker, nightworker}, and a doctor

A examining a patient in terms of X and Y respectively. Then the frame composition

of X and Y will be the cartesian product of X and Y connected through one of above

logical operators, and fpl (x, y) will be one of possible elements in fF C . Similarly, fsc will

simply become A because in this case we are considering only one agent A. Therefore,

51
Contents

Subjective Logic Symbol Propositional / Binary Symbol Subjective Logic

Operator Logic Operator Notation
Addition [115] + XOR ∪ wx∪y = wx + wy
Subtraction [115] − Difference \ wx\y = wx − wy
Multiplication [97] · AND ∧ wx∧y = wx · wy
Division [97] / UN-AND ∧ wx∧y = wx /wy
Comultiplication [97] ⊔ OR ∨ wx∨y = wx ⊔ wy
Codivision [97] ⊔ UN-OR ∨ wx∨y = wx ⊔wy
Complement [93] ¬ NOT x wx = ¬wx
Deduction [95, 100] ⊚ MP ∥ wy∥x = wx ⊚ wy|x
Abduction [95, 139] ⊚ MT ∥ wy∥x = wx ⊚wx|y
Discounting [99] ⊗ Transitivity : wxA:B = wxA ⊗ wxB
Cumulative Fusion [99] ⊕ n/a ⋄ wxA⋄B = wxA ⊕ wxB
Cumulative Unfusion [96] ⊖ n/a ⋄ wxA⋄B = wxA ⊖ wxB
A⋄B
Average Fusion [99] ⊕ n/a ⋄ wx = wxA ⊕wxB
⋄
Average Unfusion [95] ⊖ n/a ⋄ wx∪y = wxA ⊖wxB

Table 3.2: Subjective Logic Operators, Notations, and Corresponding Propositional / Bi-
nary Logic Operators.

Figure 3.8: Composition of Subjective Logic Operators - A Propositional Logic View.

A
for instance, whealthy∧dayworker A
= whealthy · wdayworker
A deals with two focused propositions

‘healthy’ and ‘dayworker’ in X and Y . Then doctor A would have an opinion about the

proposition healthy ∧ dayworker to the amount of opinion that is calculated via subjective

logic’s multiplication operator ‘·’.

The functions for deriving attributes depend on the operator. Some operators, such

as cumulative and averaging fusion, only affect the subject attribute, not the proposition

which then is equal to that of the arguments. Fusion for example assumes that two separate

argument subjects are fused into one. Other operators, such as multiplication, only affect

the proposition and its frame, not the subject which then is equal to that of the arguments.

Multiplication for example assumes that the derived proposition is the conjunction of the

argument propositions, and that the derived frame is composed as the Cartesian product

52
 Contents

of the two argument frames. The transitivity operator is the only operator where both the

subject and the proposition attributes are affected, more specifically by making the derived

subject equal to the subject of the first argument opinion, and the derived proposition and

frame equal to the proposition and frame of the second argument opinion.

It is impractical to explicitly express complex subject combinations and propositional

logic expressions as attributes of derived opinions. Instead, the trust origin subject and a

Chapter 3
compact substitute propositional logic term can be used.

Subject combinations can be expressed in a compact or expanded form. For example,

the transitive trust path from A via B to C can be expressed as A : B : C in compact

form, or as [A, B] : [B, C] in expanded form. The expanded form is the most general, and

corresponds directly with the way subjective logic expressions are formed with operators.

3.2 Logic Formalisms and Logic Programming

In this section we provide a brief introduction to fundamentals of logic formalisms and

logic programming. In this dissertation the ‘extensional’ way of knowledge representation

and uncertainty handling are based on the extension of logic programming with subjective

logic formalism. While subjective logic also is a logic, it’s expressive power is remain within

the expressive power of propositional logic as of many ‘intensional’ approaches. In this

dissertation, to cope with complex semantics, we adopt ‘predicate’ logic that is based on

first-order logic. In the following, we give a preliminary introduction to propositional logic,

first-order predicate logic and logic programming.

3.2.1 Propositional Logic

A propositional logic is the most basic branch of mathematical logic in which the only

objects are propositions, that is, objects which themselves have truth values. Variables

represent propositions, and there are no ‘relations’, ‘functions’, or ‘quantifiers’ except

for the constants T and F (representing true and false respectively). Propositional logic

53
Contents

comes with four basic connectives ¬, ∧, ∨ and → (representing negation, conjunction,

disjunction and implication). A model for propositional logic is just a ‘truth function’ on

a set of variables. Such a truth function can be easily extended to a truth function on all

formulas which contain only the variables are defined on by adding recursive clauses for the

usual definitions of connectives. Propositional logic is decidable. There is an easy way to

determine whether a sentence is a tautology (i.e., a proposition that is true for all possible

interpretations). It can be done using truth tables, since a truth table for a particular

formula can be easily produced, and the formula is a tautology if every assignment of

truth values makes it true.

Formally speaking, propositional formulae (or propositions) are strings of symbols

from a countable alphabet as defined in Definition 12, and formed according to certain

rules stated in Definition 13

Definition 12. (The Alphabet for Propositional Formulae) . this alphabet consists

of:

(1) A countable set P S of propositional symbols : P0 , P1 , P2 , ...

(2) The logical connectives : ∧ (and), ∨ (or), → (implication), ¬ (not) and sometimes ≡

(equivalence).

(3) Auxiliary symbols : ‘(’, ‘)’ (left and right parenthesis respectively).

Definition 13. (Propositional Formulae) . The set P ROP of propositional for-

mulae (or propositions) is the inductive closure of the set P S ∪ F under the functions

C¬ ,C∧ ,C∨ ,C→ and C≡ , defined as follows : For any two strings A, B over the alphabet of

Definition 12,

C¬ (A) = ¬A, C∧ (A, B) = (A ∧ B), C∨ (A, B) = (A ∨ B), C→ (A, B) = (A → B) and

C≡ (A, B) = (A ≡ B).

The truth table in propositional logic can be generalized into continuous space by

replacing the truth values and logical connectives with for instance, probability values

and probabilistic calculation. For instance, given propositional sentences A and B, · for

54
 Contents

∧, + for ∨, complement for ¬ and P (B|A) for A → B. Similarly, subjective logic also

corresponds to propositional logic sense in that it deals with propositions. In this sense,

‘intensional’ approaches that models propositional logic sentences in a graphical model can

also be understood as propositional logic that is generalized to handle continuous truth

values.

Chapter 3
3.2.2 First Order Predicate Logic

In propositional logic, it is not possible to express assertions about elements of a structure.

The weak expressive power of propositional logic accounts for its relative mathematical

simplicity, but it is a very severe limitation, and it is desirable to have more expressive

logics. First-order predicate logic is a considerably richer logic than propositional logic, but

yet enjoys many convinient mathematical properties. For example, in propositional logic

the proposition ‘John is tall’ can not be decomposed into a simple sentence because there is

no logical connectives implied in the proposition. In first-order predicate logic, it is allowed

to decompose the proposition into predicates and individuals as ‘tall (john)’. First-order

predicate logic also allows to handle expressions of generalization using quantificational

expressions. For example, propositions ‘Every cat is sleeping’, ‘Some girls likes David’ or

‘No students are happy’ are possible.

In first-order logic, assertions about elements of structures can be expressed. Tech-

nically, this is achieved by allowing the propositional symbols to have arguments ranging

over elements of structures. For convenience, we also allow symbols denoting functions and

constants.

Following is the formal definition of the syntax of first-order predicate logic.

Definition 14. (The Alphabet for First-Order Predicate Formulae) . this alphabet

consists of the following sets of symbols:

(1) Variables : A countably infinite set V = {x0 , x1 , x2 , ...}

(2) The logical connectives : ∧ (and), ∨ (or), → (implication), ¬ (not), ≡ (equivalence)

and quantifiers ∀ (for all), ∃ (there exists)

55
Contents

(3) Auxiliary symbols : ‘(’, ‘)’ (left and right parenthesis respectively).

A set L of nonlogical symbols consisting of:

(i) Function symbols: A (countable, possibly empty) set F S of symbols f0 , f1 , ..., and

a rank function r assigning a positive integer r(f ) (rank or arity) to every function

symbol f .

(ii) Constants: A (countable, possibly empty) set CS of symbols c0 , c1 , ..., each of rank

zero.

(iii) Predicate symbols: A (countable, possibly empty) set P S of symbols P0 , P1 , ..., and

a rank function r assigning a nonnegative integer r(P ) (called rank or arity) to each

predicate symbol P .

Definition 15. (First Order Predicate Formulae L) . A First Order Formulae L

over the alphabet A of Definition 14 is the collection of all WFFs (Well formed formulas)

that can be constructed from the alphabet A.

Based on the formal definition given above, the aforementioned example propositions

can be defined as follows :

• John is tall : tall (john)

• Every cat is sleeping : ∀.X[cat(X) → sleeping(X)]

• Some girls likes david : ∃.X[girl(X) → love(X, david)]

• No students are happy : ∀.X[student(X) → ¬happy(X)]

3.2.3 Logic Programming

Logic Programming is, in its broadest sense, the use of declarative mathematical logic for

computer programming. However, logic programming, in the narrower sense in which it is

more commonly understood, is the use of ‘predicate’ logic as both a declarative and proce-

dural representation language (in the sense in which it also supports procedural ‘predicate

functions’). Logic Programming considers logic theories of a specific form. The theories,

called logic programs, mainly consists of two types of logical formulae, rules and facts.

56
 Contents

Rules are of the form ‘A ← f0 , f1 , ..., fm ’ where A is rule head and the right hand side is

called body. Each fi is an atom and ‘,’ represents logical conjunction. Each atom is of

the form ‘p(t1 , t2 , ..., tn )’, where ti is a term and p is a predicate symbol of arity n. Terms

could either be variables or constant symbols. In practical implementation of logic pro-

gramming language, it also supports defining procedural ‘predicates functions’ and some

of reserved predicates are used as ‘predicate functions’ to check equality of the terms (e.g.,

Chapter 3
eq(t1 , t2 ) returns true when t1 and t2 are the same in CLIPS [1]) or to compare/calculate

arithmetic values between terms of predicates (e.g., +(t1 , t2 ) the reserved predicate ‘+’

acts as a function returns value of t1 + t2 , similarly, the functional predicate leq, leq(t1 , t2 )

returns true in case that the condition t1 ≤ t2 is satisfied in CLIPS). There is one special

term, called the anonymous term, for which the underscore (_) character is used. When

the character is used, it basically means that it does not care which variable or symbol it

is bound to, as long as it is bound to something. Rules of the form ‘f ←’ (denoted by

just ‘f ’) is called facts and can serve as an atom when used in a rule body. Negation is

represented with the symbol ‘¬’ such that ‘¬¬A=A’. Both positive and negative atoms

are referenced to as literals. Given a rule ‘head ← body’, we interpret the meaning as ‘IF

body THEN head’. Traditionally, a resolved set of facts that matches to a rule is called

‘extension’. In logic programming language based visual surveillance applications as the

ones mentioned in Section 2.4.3, rules have been used to define and reason about various

contextual events or activities.

3.2.4 Issues on Supporting Non-monotonicity

Classical logic is ‘monotonic’ in the following sense: whenever a sentence A is a logical

consequence of a set of sentences T , then A is also a consequence of an arbitrary superset

of T . In other words, adding information never invalidates any conclusions. However, rea-

soning under ‘uncertain knowledge’ (as of our interest for reasoning in visual surveillance)

is different. We often draw plausible conclusions based on the assumption that the world

in which we function and about which we reason is normal and as expected. This is far

57
Contents

from being irrational. To the contrary, it is the best we can do in situations in which we

have only incomplete information. However, as unexpected as it may be, it can happen

that our normality assumptions turn out to be wrong. New information can show that

the situation actually is abnormal in some respect. In this case we may have to revise our

conclusions. Such reasoning, where additional information may invalidate conclusions, is

called ‘nonmonotonic’. It has been a focus of extensive studies by the knowledge represen-

tation community since the early eighties of the last century. This interest was fueled by

several fundamental challenges facing knowledge representation such as modeling and rea-

soning about rules with exceptions or defaults. Another important class of ‘nonmonotonic’

reasoning is ‘abduction’ that is to draw plausible explanation supporting given observation.

Therefore, according to the given set of observations such reasoning need to be able to rate

the plausibility among possible solutions.

To cope with such ‘non-monotonic’ behavior, in this dissertation, we adopt subjective

logic for uncertainty representation formalism. However, subjective logic itself is remain

within the expressive power of propositional logic as of may ‘intensional’ approaches. In

this dissertation to cope with complex semantics, we adopt ‘predicate’ logic that is based

on first-order logic. For the flexible manipulation of facts and predicates on the need of

arithmetic calculation, etc., we also benefit from procedural handling of them. Therefore,

the chosen approach is the use and extension of Logic Programming.

58
4 Proposed Architecture and Case Study on

Forensic Queries

Thus far, we have reviewed fundamental background of this dissertation. The discussions

covered paradigm changes in intelligent visual surveillance, technical challenges and re-

Chapter 4
quirements toward intelligent visual surveillance system, prior art and related work appear

in literature and important preliminaries. Bearing the background of this work in mind,

this chapter starts with the architectural aspect of our approach to high level semantic

reasoning in visual surveillance system.

4.1 Introduction

In this chapter, we present our conceptual system architecture supporting high-level seman-

tic analysis of visual surveillance data. Taking pragmatic view, efficient semantic analysis

of visual surveillance data requires a system oriented approach that optimally combines

the individual legacy vision analytic power and its optimal use. Regarding this objective,

in Section 1.2.4, we have discussed following main requirements : 1) Metadata represen-

tation and reuse. 2) Semantic knowledge representation. 3) Uncertainty representation

and attachment to metadata. 4) Epistemic uncertainty representation and belief revision.

5) Epistemic meta reasoning and abduction. Therefore, reminding the discussion, the

main motivation of the proposed system architecture is to bring components correspond-

ing to each requirement. In terms of functionality, above requirements can be roughly

fall into two parts: ‘1) knowledge representation and reuse’ and ‘2) epistemic reasoning

mechanism’. In this chapter, we give more focus on ‘knowledge representation and reuse’

59
Contents

introducing a ‘data model’ for contextual metadata description. Regarding the ‘epistemic

reasoning mechanism’, we introduce our framework of ‘logic programming with subjective

logic’, and the detailed aspect on particular reasoning mechanisms will be focussed in the

rest of chapters.

We first start with discussion on the software components, data processing pipeline

and system architecture design. Then we introduce our prototype implementation and

present some of case study on forensic query scenarios. We conclude with discussions

on ‘scalability’ issues by conducting performance comparison between different settings of

‘data models’ and ‘rule-engines’.

4.2 Proposed Architecture

Figure 4.1 shows the overall architecture proposed base on the discussed critical require-

ments. The details are as follows.

• Metadata Representation and Reuse: Proper reuse of results generated from

each of individual vision analytic modules introduces ‘interoperability’ problem among

‘heterogeneous vision analytics’. To resolve this issue, individual vision analytics should

share a principled description method. As introduced in Section 2.2.3, there have been

remarkable research efforts on the formal description for the sharable knowledge in the

field of ‘ontological knowledge description’ or ‘semantic web’. Following this recent trend,

we also introduce formal description of metadata. The ontological knowledge descrip-

tion on possible metadata plays an important role in providing a machine process-able

representation: it provides the basis to enable fetching and collecting meaningful subsets

of information from unstructured data sets by giving specific and sharable semantics to

terminologies. Description Logic [24] and its encoding format RDF/OWL-DL [76] is a

good candidate due the widely available supporting tools (such as OWL-DL reasoners,

triple stores and SPARQL query APIs). Along the guideline of metadata ontology, in-

termediate video analysis results will get packaged as ontological metadata instances.

60
 Contents

Chapter 4
Figure 4.1: Conceptual Architecture of the Proposed Intelligent Forensic Reasoning Sys-
tem.

Packaged metadata instances will get fed into the archival layer, where the low-level fea-

ture vectors are indexed in an appropriate index structure [31] and ontological instances

will be archived into triple store [103] backed with a relational data base (RDBMS).

• Semantic Knowledge Representation: To address non-static behavior that causes

difficulties to apply predefined state model based approaches, the logic programming

approach [104] is exploited (see Section 2.3 and Section 2.4 for the detailed comparison

between ‘intensional’ and ‘extensional’ approaches). In declarative logic programming,

the detailed procedures and instructions are omitted but what the machine should do

is described. The steps to solve the given problem are executed by a kind of a runtime

system (called rule engine or logical reasoner) that understands how to use the declarative

information. Therefore, the models for solving the problems are shifted from a set

of procedures to a set of rules. Taking this advantage, contextual knowledge about

a visual surveillance scenario can be represented as sets of rules. Each rule can be

considered independent therefore, adding/modifying/updating rules can be done flexibly

61
Contents

without hitting whole knowledge structure as of state model based approaches (see the

‘modularity’ concept in Section 2.3).

• Uncertainty Representation and Attachment to Metadata : As discussed in

Section 1.2.4, vision analytic metadata usually comes with ‘uncertainty’ and therefore

influence the acceptable accuracy and robustness for a semantically complex queries.

Considering the popularity of ‘probability theory (especially, Bayesian probability)’ in

state-based models (such as FSMs, BNs, HMMs, DBNs, etc. see Section 2.4.2 for de-

tails.), and also considering the popularity of ‘state-based models’ in vision analytic

modules, the resulting values tend to represent probability of being true about target

semantics of vision analytic modules. Therefore, the values normally lay in the interval

[0, 1]. In logic programming, such metadata can be seen as facts (see Section 3.2.3) and

the facts should be fed into rule-engine with the values attached. Our approach is to

encode such values as subjective logic’s opinion [93]. Subjective logic is compatible to

the traditional probability theory but can also handle ‘uncertainty about the probability

itself ’. This way of ‘uncertainty’ representation and attachment will be referred to as

‘opinion assignment to metadata’.

• Epistemic Uncertainty Representation and Belief Revision: The logic pro-

gramming approach, as an inference mechanism, generally works on binary facts which

can be seen as either true or false. As discussed above, however, the facts will come with

‘subjective opinion’ in our design. In addition, semantic knowledge encoded as rules can

not be considered as absolute ones. Rather such rules are also epistemically uncertain.

Thus, we need a means to represent the strength or belief on a particular rule as well.

As ‘opinion assignment to metadata’, we will also attach an epistemic belief to each rule

using subjective logic opinion. This way of ‘epistemic uncertainty representation’ will

be referred to as ‘opinion assignment to rule’. Besides assigning opinions to information

(i.e., metadata in terms of vision analytics, and that can be seen as a fact in terms of

logic programming) and rules, an additional step is needed to deal with the uncertain

62
 Contents

aspect of the gathered information and rules. For example, evaluating the amount of

information sources towards the same conclusion, the amount of information sources

contradicting to others, reputational quality of a given uncertainty value or errors in

the generation of metadata should be dealt with appropriate formalism. Ultimately,

it should be also possible to revise current belief upon arrival of new information and

aforementioned evaluations. We use subjective logic due to its rich set of logical oper-

ators such as consensus, discounting, averaging fusion etc., which are not available in

traditional propositional/probabilistic logic (see Chapter 5, Chapter 6 and Chapter 7 for

more details).

Chapter 4
• Epistemic Meta Reasoning and Abduction: As shown in Section 1.2.3, for large

level of semantic granularity queries, it requires a ‘meta-reasoning’ (i.e., reasoning about

reasoning) power that reason about the possible sub queries from a set of observation.

Thus, such reasoning should be done by setting possible semantic hypotheses upon given

observations and assessing each hypothesis in an iterative manner. This aspect resem-

bles diagnostic reasoning which the so called ‘abductive (constraint) logic programming’

[50] plays an important role. Therefore, we embed the capability of ‘abductive logic pro-

gramming’ inside our rule engine. Both normal logic programming based reasoning and

abductive logic programming lay in the whole reasoning pipeline to interact each other

(see Chapter 8 for the details).

• Query Representation: Although it was not explicitly discussed as a requirement

in Section 1.2.4, one fundamental aspect of such a high level semantic analysis sys-

tem should deal with is the method to deliver user intention into a system. Machine-

processable query representation such as MPQF [53] can be considered as a candidate

format with appropriate query interface. Once a query is delivered, an agent should

decide where to access to fetch proper information in case there are several information

sources. We name this particular conceptual agent as ‘Query Broker’ (and a cooperative

project is undergoing on this issue). However, we will leave this particular component

63
Contents

out of scope in this dissertation but rather focus on above mentioned items.

4.3 Prototype Implementation

As proof of concept, a demonstrator along the afore introduced architectural approach has

been built. In this section, we mainly focussed on metadata representation and inference

mechanism upon acquired metadata instance among all requirements. We first describe

the conceptual ontological metadata description model, Next the subjective logic extension

of logic programming is described.

4.3.1 Ontological Metadata Model

An ideal high-level semantic analysis of visual surveillance system should enable optimal

reuse of currently available assets of vision analytic power. In reality, various vision analyt-

ics could exist and deployed from varying vendors or legacy systems. In this sense, proper

reuse of intermediate results generated from each of individual vision analytic modules in-

troduces ‘interoperability’ problem among ‘heterogeneous vision analytics’. To resolve this

issue, individual vision analytics should share a principled formal metadata description

method. This metadata description should be also considered in the sense of logical rea-

soning. The logical reasoning framework usually gets list of facts and then returns derived

knowledge. Moreover, each fact should be described with specific semantics. However,

considering potentially large scale everyday data, we can not feed all of them into the

rule engine. Rather, we need to be able to prune out most of the unnecessary metadata

except the ones potentially related to the desired reasoning task. Therefore, selecting an

appropriate list of facts from a bunch of surveillance metadata also requires understanding

semantics of each metadata items. Thus, to maximize the potential use of metadata, we

follow an ontological data representation and storing approach that captures dominant

concepts and relations employed in our surveillance system. The proposed data model

supports SPARQL [168] query for data segment selection.

64
 Contents

Six basic ontologies have been created. The first one is formed by data describing the

surrounding conditions and environment, the “Surveillance System Ontology”. The next

“Source Media Ontology” covers fundamental concepts and relations of possible input data

and sensors. The “Color Model” represents fundamental concepts and relations concerning

color model and is required for the similarity-based sub-query based on the MPEG-7 Color

Structure Descriptor [87]. For low-level features, we used slightly modified version of

“MPEG-7 Visual Descriptor Ontology” from AceMediaProject [156, 133]. While the “Video

Analytics” ontology models fundamental concepts and relations concerning the applied

video analysis algorithms, the “Domain Top Ontology” serves basic concepts and relations

for metadata representations.

Chapter 4
Figure 4.2 shows a partial description of these ontologies (For the ontology description

we follow one of well-known knowledge engineering methodology called ORSD (Ontology

Requirement Specification Document) [163]). The whole graphical ontology structure is

represented in Figure 4.3.

65
Contents

Figure 4.2: A Piece of the ‘Video Analytics’ Ontology Description in ORSD (Ontology
Requirement Specification Document) [163] Format.

66
 Contents

Chapter 4

Figure 4.3: Graphical Representation of Ontological Metadata Model: a) Video Analytics,

b) Color Model, c) MPEG-7 Visual Descriptor, d) Surveillance System, e)
Domain Top and f) Source Media Ontologies respectively.

67
Contents

4.3.2 Logic Programming Extended with Subjective Logic

To extend logic programming with subjective logic, the CLIPS [1] rule engine was used as a

basis to provide flexibility for defining complex data structure as well as for providing a rule

resolving mechanism. To extend this system, we defined a data structure having the form of

‘opinion(agent, proposition, b, d, i, a)’. This structure can be interpreted as a fact of arity

6 with the following terms, agent (opinion owner), proposition, belief, disbelief, ignorance,

and atomicity. To represent propositions as first-order predicate logic (see Section 3.2.2 for

details), we extended the structure so that it can take arity n properties as well. Therefore,

given a predicate p the proposition can be described as ‘p(a1 , a2 , ..., an )’. In our system,
agent
therefore, each fact is represented as the form of ‘wp(a 1 ,a2 ,...,an )
’. Namely, rules are defined

with the opinion and predicate logic based proposition structure. Additionally, functions

of subjective logic operators taking opinions as parameters were defined. In this way,

uncertainty in the form of opinion triangle is attached to rules and facts. This aspect is

depicted as follows :

Definition 16. (Opinion Assignment) . Given a knowledge base K in form of declara-

tive language and Subjective Opinion Space O, an opinion assignment over sentences k ∈ K

is a function ϕ : k → O. s.t

1. ϕf act : F act → O, e.g. wp(a

a
1 ,a2 ,..a,n )
= (b, d, i, a)

2. ϕRule : Rule → O, e.g. (wpacc(ac1 ,..,acn ) ← wpa11 (a11 ,..,a1n ) , .., wpain (ai1 ,..,ain ) ) = (b, d, i, a)

3. ϕRuleEval : RuleHead → ( a
⊛ wpaii(aai1 ,..,ain ) = O),
wpii ∈RuleBody
where ⊛ indicates one of subjective logic’s operators.

e.g. for a given rule wpacc(ac1 ,..,acn ) ← wpa11 (a11 ,..,a1n ) , .., wpain (ai1 ,..,ain ) ,

we interpret it as wpacc(ac1 ,..,acn ) = wpa11 (a11 ,..,a1n ) ⊛ ... ⊛ wpain (ai1 ,..,ain ) = (b, d, i, a)

4. ϕinf erence denoted cl(ϕ) : q → O, where K |= q called Closure.

It is important to note that there are different ways of opinion assignment. While

Definition 16 - 2 assigns an opinion to a whole rule sentence itself, Definition 16 - 3

assigns an opinion to the consequence part of the rule (rule head). The assigned opinion is

68
 Contents

functionally calculated out of opinions in the rule body using appropriate subjective logic

operators. Definition 16 - 2 especially plays an important role for prioritizing or weighting

rules for default reasoning (see Chapter 5 and [82]). Given the initial opinion assignment

by Definition 16 - 1 and Definition 16 - 2, the actual inference is performed by Definition 16

- 3 and Definition 16 - 4, where Definition 16 - 4 is further defined as follows :

Definition 17. (Closure) . Given a knowledge base K in form of declarative language

and an opinion assignment ϕ, labeling every sentence k ∈ K into Subjective Opinion Space

O, then the closure over k ∈ K, is the opinion assignment function cl(ϕ)(q) that labels

information q entailed by K (i.e.K |= q).

Chapter 4
For example, if ϕ labels sentences {a, b, c ← a, b} ∈ K as ϕf act (a), ϕf act (b) and

ϕRule (c ← a, b), then cl(ϕ) should also label c as it is information entailed by K. The as-

signment can be principled by the definition of closure. For example, an opinion assignment

to c, in a simple conjunctive sense can be ϕf act (a) · ϕf act (b) · ϕRule (c ← a, b), where · repre-

sent conjunction in Subjective Logic. In our system, to support the rich set subjective logic

operators, we made the specification of Definition 16 - 3 in rule description as follows (note

that, most of rule based systems also support describing actions in the head part of a rule) :

Rule Head (ACTION) :

Assert new Opinion wpacc(ac1 ,..,acn ) ,

where wpacc(ac1 ,..,acn ) = wpa11 (a11 ,..,a1n ) ⊛ .. ⊛ wpain (ai1 ,..,ain )

(4.1)
←

Rule Body :

wpa11 (a11 ,..,a1n ) , .., wpain (ai1 ,..,ain )

Due to the redundancy that arises when describing rules at the opinion structure level, we

will use abbreviated rule formulae as follows :

wpacc(ac1 ,..,acn ) ← wpa11 (a11 ,..,a1n ) ⊛ ... ⊛ wpain (ai1 ,..,ain ) (4.2)

69
Contents

where ⊛ indicates one of subjective logic’s operators. This way of representing rules, we

can build a propositional rules that comprise opinions about a predicate as facts that

can also check logical conjunction based existence of involved opinions and finally define

resulted predicate with opinion attached by calculating opinion values with subjective logic

operators.

4.4 Case Study I

In this section, a case study is presented to apply the described system architecture to

a simple forensic retrieval scenario on assumption of wide area city surveillance. The

case study is designed to show the whole work-through and end-to-end work-pipeline of

the proposed system. In this case study, upon the manually annotated ontological data

instances, query Q5 in Section 1.2.3 is demonstrated.

4.4.1 Scenario Setting for Case Study I

Consider a virtual scenario described below.

“Who did this person talk to?” : Monitoring guards are watching live captured

video sequences in the monitoring room. They were informed that some event happened

around the area of camera 7. According to the report, a suspicious person wearing a

white T-shirt was seen was witnessed. While browsing the video archive of camera 7,

the guards found a person wearing a white T-shirt. They captured the frame, marked

the region of the person, and then narrowed down the location where the reported event

happened. By loading the query ontology, the system enables the “findTalkTo” query.

This query provides an automatic search for additional persons that had been talking

to the selected (suspected) person. By clicking the search button, the system starts to

search to resolve “TalkTo” relation for the selected person and reports a list of key frames

showing instances where the indicated person is talking to other persons. By Clicking

on the key frames the referenced video sequences are displayed on a monitor for a final

70
 Contents

Chapter 4
Figure 4.4: Environmental Assumption : a) Locations of Nine Assumed Cameras, b) Some
of Frames from Camera 7 mounted in front of a Gas Station.

verification by the guards.

As an environmental assumption for the case study, ordinary video sequences are taken

during a day from nine different locations over an area. Figure 4.4 - a) shows the distributed

camera setup. Note that our test data has just normal and ordinary video sequences that

have no prominent unusual behaviors causing any alarm. For example, Figure 4.4 - b)

shows some of frames in our test data captured by camera 7, which is mounted in front of

a gas station.

71
Contents

Figure 4.5: Example Usage of Our Annotation Tool.

4.4.2 Metadata Generation

In this first case study, we assume that all persons and all moving objects have been

detected and annotated. An annotation tool is made to generate metadata instances

along the proposed ontological metadata models. To take into account low-level features,

the extraction capability of MPEG-7 visual features is also implemented. Annotation of

contents using our manual annotation tool is done by selecting bounding boxes from a given

video frames and designating a semantic concept defined in selected domain ontology. At

the same time, to incorporate low-level features and scene structure, we use a variation of

the MPEG-7 ontology (see Section 4.3.1 and [156, 133]). In order to represent the source

of the detection algorithm and related properties (such as, configuration parameters and

equipped camera the algorithm is running and so on) the tool allows additionally linking a

selected region of interest (ROI) (bounding box) to an appropriate algorithm or cameras,

etc. Figure 4.5 shows a simple example of the annotation process with a semantically

labeled bounding box and the associated MPEG-7 CSD feature vector.

The generated annotation instances are encoded in RDF/OWL format. When RDF/

OWL technologies are used, the instances and owl concepts can be represented as DIG (Di-

72
 Contents

Chapter 4
Figure 4.6: Annotation Instance of Figure 4.5 : a) DIG Representation of the Example, b)
Person Instance in N3-Triple Notation, c) ROI Instance in N3-Triple Notation
and d) Color Structure Descriptor in RDF/OWL Format of MPEG-7 Ontology.

rected Graph) consisting of RDF-triples of the form < subjective, predicate, objective >.

There are several encoding formats for RDF triples, among them N3-Triple (a.k.a Nota-

tion3) [169] is one of the simplest representation for human understanding. Figure 4.6

shows an example of a partial DIG of concepts and instances generated by the annota-

tion tool (ideally, this should be generated by the analytic modules automatically) and its

encoding in N3-Triple format. Figure 4.6 can be interpreted as follows:

• the graph shows that there is an instance which is named as Person93 and having the

type “person” coming from the “domain” namespace

• It was classified (evaluatedBy) by an algorithm instance “OmniAnalyticModule” defined

73
Contents

in the system profile.

• It belongs to a ROI instance named as ROI_inst_91. The ROI_inst_91 is an instance

of the MPEG-7 Region concept.

• The ROI exists in a video stream captured from camera id 4 (also defined in the system

profile).

• The ROI also has associated an instance of a visual descriptor named “CSD_inst_91”

which is an instance of MPEG-7 ColorStructureDescriptor defined in the MPEG-7

namespace and extracted by an algorithm defined in the system profile.

Once generated, the metadata should be archived into a persistent repository for later

use. Because RDF/OWL is used as basis for the knowledge representation, it naturally

needs a RDF triple store as data archive. For practical use, the machine should be able

to parse such a graph model. There are several infrastructures available for this purpose

such as Sesame, Jena and Kaon, etc [175]. In the current system integration the Jena

semantic web framework from Hewlet-Packard (HP) is used, which is a Java framework

for building Semantic Web application. It provides a programmatic environment for RDF,

RDFS, OWL, and SPARQL (note. Jena is open source and grown out of work at the HP

labs).

In addition to RDF/OWL archiving, low level feature vectors should be separately

indexed for picking similar vectors. As shown in Figure 4.6 - d), low level features usually

consist of a number of numeric values. In the above case, the shown CSD (Colour Structure

Descriptor) has 256 bins. Indexing such multi-dimensional vectors is however not a trivial

task. In textual data indexing as it has been done in RDBMS side, the most common way is

building a tree structure that balances the average degree of depth of nodes from the root,

by appropriately partitioning the search space. B-tree and its variations (B*-tree, etc) are

well known examples of this approach. However, it turns out that the partitioning is mean-

ingless when we deal with high dimensions, because no matter how we try to partition, the

74
 Contents

Chapter 4
Figure 4.7: Illustrative Example of VA-file : a) VA-file for 2-D Vectors and b) Lower and
Upper Bound between a Given Query Vector q and Feature Vectors in a Cell.

curse of dimensionality causes the sparseness problem meaning that the data points in high

dimensional space will be dispersed so they cannot be well clustered. Multiple approaches

have been proposed so far: dimensionality reduction, vector approximation and locality

sensitive hashing technologies are well known examples [31, 129]. At this stage, considering

the simplicity on implementation, VA file [171] is used. VA file is based on the assumption

that we cannot partition data space that will in turn cause full data search. Therefore,

in the VA file approach, the focus is more on reducing I/O time and distance calculation

time. By quantizing each bin vectors with several bits, one specific feature vector can be

represented by a small number of bits. To cope with K-nearest neighbor search, it also

provides an algorithm to calculate distances at I/O time and thereby sort the requested

K elements. According to the quantization scheme in VA-file, we allocate appropriate bits

(bj ), as follows :

75
Contents


⌊ ⌋  1 if j ≤ (b mod d)
b
bj = +
d 
 0 (4.3)
otherwise

where, b is the total number of bits and d is the number of dimensions.

This concept is shown in Figure 4.7 - a) in the case of b = 2 and d = 2. Once indexed,

the search can be done as shown in Figure 4.7 - b). While reading approximated feature

vectors, for each feature vector Pi we calculate Distmax (Ui ) and Distmin (li ) as shown

above. For K-nearest neighbour search, we first initialize an array with k-index with

appropriately large distance values, then with Pi we calculate the lower bound (Distmin )

if the lower bound is less than values in the array, we set the Pi as a candidate, and repeat

this process till we reach the end of the VA index file. Though the VA file approach is

described here, again, our ultimate aim is not to find another indexing structure, but to

provide knowledge on the index, so that the system can automatically decide what and

how to use such an index, when third party index structures are plugged into our system

and are made available.

4.4.3 Querying and Retrieval

As described, let us assume that someone informed that something happened somewhere.

With this information, we zoom into the reported area and click on one available camera

around the place. We also load a domain knowledge description, which will show us all

possible queries that we can resolve. Clicking a camera will show us a key frame summary

of what happened during the past period, sorted according to the time line. By clicking

on one of the images, we can play the corresponding video in the window on the top left

side (see Figure 4.8 - 1). While browsing the video sequences, we found a guy who is

holding a white box. We were suspicious about the guy and want to know as much as we

can (see Figure 4.8 - 2, 3). After capturing the scene (by clicking the capture button),

we start composing a query. This includes manually selecting the person by drawing a

bounding box, cropping the image and finally extracting a set of low level features for this

76
 Contents

Chapter 4
Figure 4.8: Querying and Retrieval Graphical User Interface and Walkthrough.

region of interest (see Figure 4.8 - 4). From the available set of queries we select the query

‘FindTalkTo’ and trigger it (see Figure 4.8 - 5, 6). This in turn launches a SPARQL [168]

query that is expanded by the DL Reasoner using the Vision Metadata Ontology. The

SPARQL query used at this phase is depicted as follows :

• SELECT DISTINCT ?visionEntity ?am ?type ?ROIx ?ROIy ?ROI_width ?ROI_height ?camID
?timeStamp ?cvstr
#(opinion (agentl ?am) (prop ?type) (args ID ?visionEntity CamID ?camID RectInfo ”?ROIx ?ROIy
?ROI_width ?ROID_height Time ?timeStamp) #OpinionMapperMethod1(?cvstr)#)
FROM NAMED
<http://SicosleOntologyServer/metadata/KB/Sicosle/Instance/DataDocuments/MetaC1:040408:001.owl>

77
Contents

WHERE {
?visionEntity rdf:type sdtop:VisionEntity;
rdf:type ?type;
sdtop:hasRegion ?region;
vao:hasAnalyticModule ?am;
sdTop:hasConfidenceVale ?cvstr;
FILTER (regex (str(?type), ”ˆ http://SicosleOntologyServer/metadata/KB/Sicosle/DomainOnt/
Sicosle_Showcase1Domain.owl”)).
?region mpeg7:hasROITimeStamp ?timeStamp ;
mpeg7:ROI_x ?ROIx;
mpeg7:ROI_y ?ROIy;
mpeg7:ROI_width ?ROI_width;
mpeg7:ROI_height ?ROI_height;
sss:ROIOf ?videoStream .
# FILTER ((?timeStamp > ”2005-01-01 T00:00:00Z”ˆ ˆ xsd:dateTime)).
?videoStream ssm:capturedBy ?cam.
?cam rdf:type ssm:Camera .
?cam ssm:hasSensorID ?camID.

The result set is then transformed into a fact list for the CLIPS rule engine [1]. We

have also enabled describing hints on this translation in the comments (see the lines with

comment symbol ‘#’). Following shows examples of the translated opinion lists in CLIPS

syntax as explained in Section 4.3.2.

• (opinion (agent humanAnnotator) (prop person) (args ID person96 CamID 4 RectInfo "259 75 58 136"
Time 2008-04-04T19:00:20.0) (b 1) (d 0) (i 0) (a 0.5) )
(opinion (agent humanAnnotator) (prop person) (args ID person93 CamID 4 RectInfo "159 55 50 126"
Time 2008-04-04T19:00:21.0) (b 1) (d 0) (i 0) (a 0.5) )
(opinion (agent humanAnnotator) (prop person) (args ID chair4 CamID 4 RectInfo "240 105 31 25"
Time 2008-04-04T19:00:21.5) (b 1) (d 0) (i 0) (a 0.5) )
(opinion (agent humanAnnotator) (prop person) (args ID table3 CamID 4 RectInfo "188 42 100 52"
Time 2008-04-04T19:00:19.0) (b 1) (d 0) (i 0) (a 0.5) )
:

As the metadata is based on manual annotation, the translated opinion values repre-

78
 Contents

sent absolute truth (i.e., (1,0,0,0.5)). Once gathered, rules are executed on gathered data

segments, to find evidential patterns which satisfy given rules. Following shows a rule that

represents ‘talk-to’ semantics in the syntax explained in Section 4.3.2.

talkto_rule
wtalkto(append(V e2,V e3,T s2,T s3),CamID,(duration(T s2,T s3)),(dist(Roi2,Roi3)))
_ _ _
← wf urniture(V e1,CamID,Roi1,T s1) ⊛ wperson(V e2,CamID,Roi2,T s2) ⊛ wperson(V e3,CamID,Roi3,T s3) (4.4)

⊛wdistwithin(centerradius,100,Roi1,Roi2,Roi3)
SLSystem
⊛ wtimeholduntil(min,2,T
SLSystem
s1,T s2,T s3) ⊛ wneq(V e2,V e3)
SLSystem

In our system, actual representation of the above Rule 4.4 is as follows.

• (defrule talkto-rule
?op1<-(opinion(prop ?ty1&:(eq ?ty1 furniture))(args ID ?ve1 CamID ?camID RectInfo ?roi1 Time ?ts1))

Chapter 4
?op2<-(opinion(prop ?ty2&:(eq ?ty2 person))(args ID ?ve2 CamID ?camID RectInfo ?roi2 Time ?ts2))
?op3<-(opinion(prop ?ty3&:(eq ?ty3 person))(args ID ?ve3 CamID ?camID RectInfo ?roi3 Time ?ts3))
(distwithin centerradius 100 ?roi1 ?roi2 ?roi3)
(timeholduntil min 2 ?ts1 ?ts2 ?ts3)
(test (neq ?ve2 ?ve3))
=>
(bind ?new_op (sl-conjunction$ ?op1 ?op2 ?op3))
(modify ?new_op (prop talkto) (args ID (apply str-cat ”talkto” ?ve2 ?ve3 ?ts2 ?ts3)
CamID ?camID Time (duration ?ts2 ?ts3) Dist (dist ?roi2 ?roi3)))
)

Above rule also shows the use of subjective logic’s conjunction operator to derive

a new opinion. However, due to the manual annotation that gives full belief to every

metadata, the rule will also derive an opinion with full belief.

In this step it is e.g. verified in which instance at least two people are appearing

in the same image (because one of the rules defines that the presence of two persons is a

necessary condition for the ‘talk-to’ relation). The result from the rule based filtering will

be a largely reduced candidate set. The final check now is on verifying that one of the

persons is indeed the same as the initially selected (suspicious) person. This is performed

by a similarity match over all the features extracted from the selected ROI. If the distance

79
Contents

between the query feature set and the feature set associated to a person in the candidate

frame is below a certain threshold then it is assumed that both persons are identical. The

remaining candidate instances are presented as a list of key frames. Figure 4.8 - 7) shows

scenes with different people around chairs and table are regarded satisfying the given query.

Clicking ‘view info menu’, it also shows visual similarity matching results using L1 − norm

between features of objects in the retrieved scenes and the extracted query feature vector.

In addition to this, ideally, these retrieved items can be clicked to check the attached

subjective opinion on being the right answer to the query Figure 4.8 - 8). However, at this

stage of case study, we have not considered the uncertainty handling mechanism but rather

used straightforward rule matching. The detailed mechanism on uncertainty handling will

be discussed in the rest of chapters.

4.5 Case Study II

In this section, we present a case study in applying the proposed system to real traffic

surveillance scenes processed with automated vision analytics 1 . Considering that we are

not yet facing the use of vision analytic metadata generated from heterogeneous vision

analytic systems or vendors, we will focus more on logical reasoning aspect rather than the

metadata representation (note that, one of the reasons we use ontological metadata model

was to provide machine-interpretable inter-operability between heterogeneous systems by

giving a sharable formal semantics to metadata. Refer to the Section 2.2.3, Section 4.2 and

Section 4.3.1). Therefore, this case study is to show how logical rules can be used to model

contextual semantics implicitly implied in traffic video scenes. For the convenient imple-

mentation, the generated metadata is directly archived into a DataBase (MySQL) in form

of opinion representation. Total 7 different compositional logical queries are demonstrated.

1
This case study is conducted in cooperation with SCR (Siemens Corporate Research, Princeton, NJ,
US) using SCR’s proprietary vision analytic modules. Especially, special credits must go to Dr. Vinay
Shet, Dr. Gao Xing and Dr. Vasu Parameswaransu

80
 Contents

Figure 4.9: Samples of Traffic Visual Surveillance Data for Case Study 2.

Chapter 4
4.5.1 Scenario Setting for Case Study II

Total 5 hours and 30 minutes ordinary tunnel and highway scenes are collected from 8

different video sources. These video sources are processed using SCR’s (Siemens Corpo-

rate Research) proprietary automated vision analytic modules. The volume of the video

footages is 4.64Gb in compressed form. Figure 4.9 shows some sample frames of the traffic

video data. The applied vision analytic modules generate total 12 primitive semantics (e.g.

trajectory information of vehicles, distances between vehicles, scene with hazard light, oc-

clusion, human appear and disappear, etc.). Upon such semantics, we focus on composing

and answering complex queries by the use of spatio-temporal rules. Following shows the

compositional queries we applied.

• ‘Q1’ Find a Truck following an Emergency Vehicles

• ‘Q2’ Find an Emergency Vehicle following another Emergency Vehicle

• ‘Q3’ Find a Vehicle going past a Pedestrian

• ‘Q4’ Find a Truck going past a Pedestrian

• ‘Q5’ Find a Truck passing by a Stopped Vehicle

• ‘Q6’ Find a Truck following another Truck

• ‘Q7’ Find a Truck passing a vehicle that is Backing up

81
Contents

Vocabulary of Properties of Total # of

Primitive Semantics each Vocabulary Item instances
humanappear (CamID, PedID, Tlx, Tly, Brx, Bry, Time, Ef, Ea) 30
humandisappear (CamID, PedID, Tlx, Tly, Brx, Bry, Time, Ef, Ea) 26
vehicle_trajectory (CamID, VehID, Time, Ef, Ea) 6397
distance (CamID, VehID, LaneID, Distance, AssocVehID, Time, Ef, Ea) 19198
hazardlight (CamID, LaneID, Time, CfgV, Ef, Ea) 899
lostcargo (CamID, LaneID, Y, Time, Ef, Ea) 3
occlusion (CamID, Lambda, Time, Ef, Ea) 649
slowvehicle (CamID, LaneID, Direction, Time, Ef, Ea) 24
speedestimation (CamID, VehID, LaneID, Direction, Speed, Time, Ef, Ea) 6400
stoppedvehicle (CamID, LaneID, Y, Time, Ef, Ea) 52
vehicletype (CamID, VehID, Type, CfgV, Ef, Ea) / Type : 0-small, 1-large 6399
weather (CamID, Type, Time, Ef, Ea) / Type : 1-normal, 2-shiny 17
Total 5h 30m from 8 different sources (≒ 40m for each source) .Total # of instances = 40094

Table 4.1: Metadata Vocabulary and the Property of Each Primitive Semantic Item.

Figure 4.10: Flat-file Style Database Tables in ERD (Entity Relation Diagram) Each Rep-
resenting Metadata Vocabularies of Vision Analytics.

4.5.2 Metadata Generation

Table 4.1 shows the metadata vocabulary used in this case study. Using the vision analytics

available in Siemens (i.e., SCR’s proprietary vision analytics), total 12 primitive semantic

items are generated with corresponding properties. Archiving of the metadata is done using

82
 Contents

a flat-file-like database scheme shown in Figure 4.13, which directly maps each semantic

item into a corresponding database table. In the vocabulary uncertainty is captured in

the [Ef, Ea] pair that satisfies Ef + Ea = 1. The value Ef represents amount of truth

‘Evidence For’ and the other value Ea represents amount of falsity ‘Evidence Against’.

In our system, each metadata item is represented in form of subjective opinion following
SCR_V isionAnalytic
Definition 16 - 1, as shown in Section 4.3.2. For example, whumanappear(CamID,P edID,T ime) =

(b = Ef, d = Ea, i = 0) represents a subjective opinion about ‘humanappear’ metadata

with the amount of opinion (b = Ef, d = Ea, i = 0, a = 0.5 by default).

4.5.3 Querying and Retrieval

Chapter 4
To be able to answer complex queries Q1-Q7 discussed in Section 4.5.1, it should be able

to augment implicit high level semantics, based on the currently available metadata. For

example, the semantics of ‘emergency vehicles’ in ‘Q1’ is not explicitly defined in the meta-

data vocabulary. However, combining ‘hazardlight’ information with ‘vehicle_trajectory’

information, we could compositionally consider an instance of vehicle detection as an ‘emer-

gency vehicle’ if there was also a hazard light at the same time of the vehicle detection.

Using the logical conjunction operator (‘∧’) of subjective logic, this can be represented as

follows :

evrule
wemergencyV ehicle(CamID,V ehID,V ht)
_ _
← whazardlight(CamID,_,Ht) ∧ wvehicletrajectory(CamID,V ehID,V ht) (4.5)

∧wtest(V
SLSystem
ht+25<Ht<V ht+25)

The semantic concept ‘truck’ in ‘Q1’ is also not explicitly defined in the metadata

vocabulary. However, we can also infer the concept ‘truck’ by referring to ‘vehicletype’ that

indicates a large vehicle when the property ‘type’ is set with ‘1’. This can be also depicted

as follows:

truckrule
wtruckV ehicle(CamID,V ehID,V ht)
(4.6)
_ _
← wvehicletype(CamID,V ehID,1) ∧ wvehicletrajectory(CamID,V ehID,V ht)

83
Contents

Based on the inferred concept defined by Rule 4.5 and Rule 4.6, the semantics of

‘Q1’ can be further defined as follows:

q1
wtruckN earAnEmergencyV ehicle(CamID,T vID,EmerT )
_ _
← wemergencyV ehicle(CamID,EvID,EmerT ) ∧ wtruckV ehicle(CamID,T vID,V ht) (4.7)

∧wappearEqual(EmerT,V
SLSystem
ht,70)

Rule 4.7 represents that it will assert a new opinion on the proposition ‘truckNear-

AnEmergencyVehicle’, if there were semantics which can be regarded as ‘emergency vehicle’

and ‘truck vehicle’ at the same time. To manipulate temporal relations, Allen’s temporal

relations were modeled as predicate function [15]. These temporal relation covers ‘{equal,

before, after, meets, meetby, overlaps, overlappedby, starts, startedby, during, contains, fin-

ishes, finishedby }. For example, the predicate function term ‘appearEqual’ corresponds to

‘equal’ in allen’s temporal logic with acceptable tolerance 0.7 sec. Semantics of ‘Q2-Q7’

can be also augmented in the similar way. Table 4.2 shows some of augmented semantics

in form of rules and all the rules directly corresponding to the queries ‘Q1-Q7’.

Figure 4.11 shows the prototype interface of the demonstrator. The user interface

shows retrieved items for the queries ‘Q1-Q7’. Each items comes with basic information and

the item can be played for further examination. The items are ranked along the expectation

value of the calculated subjective opinions attached to items. In this case study, however,

the uncertainty handling is only based on the ‘conjunction’ operator, thereby, represent

rather conjunctive probability sense of subjective opinions. (Note that, the rest of this

dissertation deal with the detailed aspects on handling uncertainty and this section focusses

more on the architectural case study.) As an architectural proof of concept, every queries

seem to retrieve reasonable scenes that matches to the intention of the queries. However,

for some cases as shown in Figure 4.11 - ‘Q4 : Find a truck going past a Pedestrian’, due to

the false alarms of pedestrian metadata, it also retrieved some of wrong items. Therefore,

the robuster analytics we have, the better result would be possible.

84
 Contents

Query Semantics / Corresponding Rules

‘-’ Augmented Semantics :

evrule
wemergencyV ehicle(CamID,V ehID,V ht)
_ _ SLSystem
← whazardlight(CamID,_,Ht) ∧ wvehicletrajectory(CamID,V ehID,V ht) ∧ wtest(V ht+25<Ht<V ht+25)
(4.5)

truckrule
wtruckV
_ ehicle(CamID,V ehID,V ht) _ (4.6)
← wvehicletype(CamID,V ehID,1) ∧ wvehicletrajectory(CamID,V ehID,V ht)

bvrule _
wbackingU pV ehicle(CamID,V hID,V ht) ← wspeedestimation(CamID,V hID,1,V ht) (4.8)

:
‘Q1’ Find a Truck following an Emergency Vehicles
q1
wtruckN earAnEmergencyV ehicle(CamID,T vID,EmerT )
_ _
← wemergencyV ehicle(CamID,EvID,EmerT ) ∧ wtruckV ehicle(CamID,T vID,V ht) (4.7)
SLSystem
∧wappearEqual(EmerT ,V ht,70)

Chapter 4
‘Q2’ Find an Emergency Vehicle following another Emergency Vehicle
q2
wemerV ehicleF ollowingAnotherEmerV ehicle(CamID,EvID2,EmT 2)
_ _
← wemergencyV ehicle(CamID,EvID1,EmT 1) ∧ wemergencyV ehicle(CamID,EvID2,EmT 2) (4.9)
SLSystem
∧wappearBef ore(EmT 1,EmT 2,80)

‘Q3’ Find a Vehicle going past a Pedestrian

q3
wvehicleGoingP astP edestrian(CamID,V hID,Ht)
_ _ SLSystem
← whumanappear(CamID,P edID,Ht) ∧ wvehicletrajectory(CamID,V hID,V t) ∧ wappearBef ore(Ht,V t,80)
(4.10)

‘Q4’ Find a Truck going past a Pedestrian

q4
wtruckGoingP astP edestrian(CamID,V hID,Ht)
_ _ SLSystem
← whumanappear(CamID,P edID,Ht) ∧ wtruckV ehicle(CamID,V hID,V t) ∧ wappearBef ore(Ht,V t,80)
(4.11)

‘Q5’ Find a Truck passing by a Stopped Vehicle

q5
wtruckP assingByStoppedV ehicle(CamID,V hID,St)
_ _ SLSystem
← wstoppedvehicle(CamID,_,_,St) ∧ wtruckV ehicle(CamID,V hID,V t) ∧ wappearBef ore(St,V t,80)
(4.12)

‘Q6’ Find a Truck following another Truck

q6
wtruckF ollowingAnotherT ruck(CamID,T vID1,T 1)
_ _
← wtruckV ehicle(CamID,T vID1,T 1) ∧ wtruckV ehicle(CamID,T vID2,T 2) (4.13)
_ SLSystem SLSystem
∧wdistance(CamID,T vID1,_,Dist,T vID2,_) ∧ wgeq(Dist,25) ∧ wappearBef ore(T 1,T 2,80)

‘Q7’ Find a Truck passing a vehicle that is Backing up

q7
wtruckP assingBackingU pV ehicle(CamID,V hID1,V t1)
_ _
← wtruckV ehicle(CamID,V hID1,V t1) ∧ wbackingU pV ehicle(CamID,V hID2,V t2) (4.14)
SLSystem
∧wappearBef ore(V t1,V t2,30)

Table 4.2: Rules used in Case Study 2.

85
Contents

Figure 4.11: Querying and Retrieval Graphical User Interface and Retrieval Results of ‘Q1-
Q7’ with Sample Items.

86
 Contents

4.6 Discussion

In the previous section, we have demonstrated the proposed architecture approach in two

different case studies. These case studies especially show the feasibility of logical reasoning

for modeling compositional high-level semantics that can be augmented based on available

metadata. While the reasoning aspect is focussed, there are however still open issues such as

‘Scalability’, ‘Proper Uncertainty Handling’, etc. In this section we will further discuss the

‘Scalability’ issue and the issues around ‘Uncertainty Handling’ will be addressed in the rest

of this dissertation. In the proposed architectural pipeline, there are two main components

to discuss in terms of ‘Scalability’. Namely, ‘scalability issues around Ontological instance

Chapter 4
handling’ and the ‘scalability issues around rule-based reasoning’. This section addresses

both of the aspects.

4.6.1 Scalability of Ontological Instance

In Section 4.4, ontological representation of metadata played an important role in the

proper selection of metadata segments for further logical reasoning. The selection is done

by the use of SPARQL [168] query upon OWL/DL reasoner [175]. Although there are

issues such as how to come up with Ontologies that can be agreed by other systems and

venders, etc. (‘in the sense of knowledge sharing and interoperability’), the more essential

and natural question here could be the performance in terms of the scalability due to the

potentially large scale metadata that visual surveillance systems would produce. It is not

only the case of visual surveillance but also the case of many other domains. Therefore, it

is important to know the general performance of OWL/DL reasoners.

There have been performance benchmarks of currently available reasoners [67, 172,

173, 30]. The benchmarks have been conducted in terms of many aspects such as size

of TBox (‘Size of Knowledge Description’), size of ABox (‘Size of Instances of TBox’),

loading time of Ontology, complexity of the queries and currently available large scale

ontologies (i.e., DBLP database for publications, etc.). In a relatively recent comprehensive

87
Contents

Table 4.3: Statistics of Test Ontologies used in the Work of Jürgen Bock et al. [30]. The
Table Indicates # of TBox Parameters (Class, Property, SubClass, etc.) and #
of ABox Parameters (C(a) : Instances of a Class and Instances of a Relation
R(a,b)). The Red Boxed Row Shows the Statistics of Our Manual Annotation
Used in Case Study I (see Section 4.4) Extracted by TopBraid Composer. ‘-’
Represents the Statistics not Available in TopBraid Composer. The Red Ar-
row Indicates that Our Annotation Resembles wine_1 Data Set in terms of
Statistics.

workbench report done by Jürgen Bock et al. [30], they have conducted a performance

benchmark using some of prominent example data sets of RDF/OWL ontology 2 .

Table 4.3 shows the statistics of the test ontologies. We have also extracted the

statistic information of our annotation using an ontology editing tool called ‘TopBraid

Composer’ 3 . Although not every statistics were available as indicated with ‘-’, the red

2
Datasets: VICODI (http://www.vicodi.org), SWRC (http://www.ontoware.org/index.html),
LUBM (http://swat.cse.lehigh.edu/projects/lubm/index.htm) and WINE (http://www.
schemaweb.info/schema/SchemaDetails.aspx?id=62)
3
http://www.topquadrant.com/products/TB_Composer.html

88
 Contents

Chapter 4
Figure 4.12: Benchmark Result in the Work of Jürgen Bock et al. [30]. The Figures show
the Average Load and Query Time with a Particular Reasoner on the Dataset
shown in Table 4.3. The ‘◦’ indicates Time-out (> 5 min).

box and the red arrow in Table 4.3 shows that our annotation is quite similar to the

wine_1 data set. Therefore, it is expected that our ontology model would also show

similar performance aspect as the cases of the wine data set.

The performance evaluation is done with different implementation of reasoners cur-

rently available. The reasoners used in their evaluations fall into three groups: tableau

algorithm based (RacerPro and Pellet), disjunctive datalog based (KAON2) and standard

rule engine based (Sesame and OWLIM) 4 .

4
Reasoners: RacerPro(http://www.RacerPro-systems.com/), Pellet(http://www.mindswap.org/2003/

89
Contents

Figure 4.12 shows the result of performance evaluation. In the benchmark report,

they informed that it was conducted on a Linux 2.6.16.1 system, however, no further

information regarding the hardware setting is available. Unfortunately, it shows that many

of reasoners except KAON2 becomes ‘time-out’ when it is performed with wine data sets

containing more than 5000 instances. In the case of wine_1 data set on Pellet, as expected,

it shows similar performance to the result we experienced with our annotation in the case

study I (Including loading and query processing time, it took approximately 20-25 sec on a

core2duo T7200 1.8 Ghz, 2Gb Ram Windows XP machine). According to the benchmark,

it does not come up with a clear ‘winner’ OWL/DL reasoner that performs well for all

types of ontologies and reasoning tasks. However, Jürgen Bock et al. [30] have concluded

as follows:

• As general conclusions we can summarize our results in that (1) reasoners that employ

a simple rule engine scale very well for large ABoxes, but are in principle very limited

to ‘lightweight language fragments’, (2) classical tableau reasoners scale well for complex

TBox reasoning tasks, but are limited with respect to their support for large ABoxes, and

(3) the reasoning techniques based on reduction to disjunctive datalog as implemented

in KAON2 scale well for large ABoxes, while at the same time they support are rich

language fragment. If nominals are important for a given scenario, Pellet is the only

reasoner in this benchmark, which has adequate support [30] .

The benchmark shows that the complexity of TBox also influence the ABox perfor-

mance. To this extent, the word ‘lightweight language fragments’ in (1) means OWL-Lite

that is a restricted subset of OWL-DL in terms of expressivity to keep computational

tractability. At this stage of the proof of concept, we have used Pellet reasoner not to be

restricted by expressivity on knowledge modeling. However, it seems that a lot more effort

should be paid to derive a matured ontological knowledge model in visual surveillance do-

main, that is not only agreeable by the parties of systems or venders but also balanced in
pellet/index.shtml), KAON2(http://kaon2.semanticweb.org/), Sesame(http://openrdf.org)
and OWLIM(http://www.ontotext.com/owlim)

90
 Contents

terms of TBox and ABox complexity. Therefore, we believe that this should be driven in

the sense of ‘collaborative standardization’. Both the ‘scalability’ and the ‘standardization

of ontology model’ hamper the wide acceptance of the ontology related technology not only

in the visual surveillance but also in other domains. This seems the reason why many of

current data management systems remain in the traditional database technology. However,

considering the current active research effort paid in OWL related technologies, we believe

that it would come up with improved performance in the near future. To this extent, we

believe that the proof of concept (case study I) on the use of ontological metadata shows

a step towards intelligent use of metadata for intelligent visual surveillance.

Chapter 4
4.6.2 Scalability of Rule Engines

In the proposed system architecture, ontological metadata representation is not only to

archive information in a ‘sharable’ and ‘machine interpretable’ way, but also to provide

a mechanism to pick and collect probable amount of partial evidences (called facts in

logic programming) which in turn need to be pumped up to a logical reasoner. Therefore,

another important focus regarding ‘scalability’ is on the rule based logical reasoning engines

that are used after collecting evidential facts. Normally, rule engines implement traditional

backtracking (e.g., SLD resolution [66]) or forward chaining (e.g., Rete algorithm [64])

algorithms. In such algorithms, facts and rules are loaded in memory space called fact

base and rule base respectively. It is known that the size of the facts affects reasoning

performance.

In this section, we represent a simple performance evaluation regarding the ‘scala-

bility’ of representative rule engines such as CLIPS [1] and Prolog 5 . Although it is not

designed for ‘automated’ inference, in logical view, it is important to note that the ‘re-

lational data model’ is also based on the ‘first-order-predicate-logic’ [40, 41] and SQL is

the language to access the relational data model. Namely, they operate across the same
5
For the evaluation, we used SWI-Prolog [8] among many variations due to its GNU-license policy. Com-
parison of Prolog implementations can be found on (http://en.wikipedia.org/wiki/Comparison_of_
Prolog_implementations)

91
Contents

Figure 4.13: The Database Scheme of a Siemens’ Internal Visual Surveillance Prototype
System called VAP (Video Analytic Platform).

conceptual theory, although their focuses are in completely different directions. In rule

engine terms, SQL is primarily a fact and relation(set) engine therefore does not support

automatic pattern tracking that dynamically builds a rule tree at run time. However, giv-

ing up the ‘automatic’ inference aspect, and by taking static view of all possible queries,

SQL can do the other, to a limited extent. Therefore, to contrast the performance to the

case of using traditional database, we manually expanded the chaining of the rules for each

queries ‘Q1-Q7’ in form of SQL. To see the influence of different relational data models,

the SQL expansion is done for two different relational models. The first one is the flat-file

style model as shown in Figure 4.13 explained in Section 4.5. We have also used a more

complex data model that covers many other aspects of surveillance system. The model is

shown in Figure 4.13 and is used in one of Siemens’ internal prototype implementations of

surveillance system called VAP(Video Analytic Platform). Table 4.4 shows some of SQL

query examples expanded upon the two different relational data models.

We have also populated instances of the metadata vocabulary set used for case study

II in Section 4.5. As shown in Table 4.1, originally, 40094 instances were extracted from 8

92
 Contents

Query Semantics / Relational Model (Scheme) / Corresponding SQL Query

‘Q1’ Find a Truck following an Emergency Vehicles
Flat-File DB scheme VAP-DB scheme

select distinct hl.CamID, hl.Time, v.time select distinct ep1.camid, ep1.time, ep2.time
from hazardlight hl, vehicletype vt, vehicletrajectory v from event_property ep1, event_property ep2
where hl.CamID = vt.CamID where ep1.eventtypeid = 2
and vt.CamID = v.CamID and ep1.proptypeid = 7
and vt.VehicleID = v.VehicleID and ep2.eventtypeid = 13
and vt.Type = 1 and ep2.proptypeid = 9
and v.Time > hl.Time and ep2.propvalue = 1
and hl.Time + 80 > v.Time and ep1.camid = ep2.camid
(4.15) and ep2.time > ep1.time
and ep1.time + 80 > ep2.time
(4.16)

‘Q6’ Find a Truck following another Truck

Flat-File DB scheme VAP-DB scheme

select distinct v.CamID, v.Time, d.time select distinct ep2.camid, ep2.time, ep1.time
from distance d, vehicletrajectory v, from event_property ep1, event_property ep2,
vehicletype vt, vehicletype vt2 event_property ep3, .event_property ep4,
where vt.Type = 1 event_property ep5, event_property ep6,
and vt.camID != 0 event_property ep7
and v.CamID = vt.CamID where ep2.eventtypeid = 13
and v.VehicleID = vt.VehicleID (4.17) and ep2.camid != 0

Chapter 4
and d.CamID = v.CamID and ep2.proptypeid = 9
and v.VehicleID = d.VehicleID and ep2.propvalue = 1
and d.Distance < 25 and ep1.eventtypeid = 1
and vt2.Type = 1 and ep1.camid = ep2.camid
and vt2.CamID = vt.CamID and ep3.id = ep1.id
and d.AssocVehicleID = vt2.VehicleID and ep3.eventtypeid = 1
and ep3.proptypeid = 1
and ep4.id = ep2.id
and ep4.eventtypeid = 13
and ep4.proptypeid = 1
and ep3.propvalue = ep4.propvalue
and ep1.proptypeid = 6
and ep1.propvalue < 25
and ep5.eventtypeid = 13
and ep5.proptypeid = 9
and ep5.propvalue = 1
and ep2.camid = ep5.camid
and ep6.id = ep5.id
and ep6.eventtypeid = 13
and ep6.proptypeid = 1
and ep7.id = ep1.id
and ep7.eventtypeid = 1
and ep7.proptypeid = 2
and ep6.propvalue = ep7.propvalue
(4.18)

Table 4.4: Some Examples of SQL Queries for ‘Q1-Q7’.

Data Set # instances time of each video sources total time of 8 video sources
Original Case Study 2 # 40094 ≒ 40 min ≒ 5 hours 30 min
CS2-0.5 27290 ≒ 30 min ≒ 4 hours
CS2-01 42444 ≒ 1 hours ≒ 8 hours
CS2-02 84888 ≒ 2 hours ≒ 16 hours
CS2-03 127332 ≒ 3 hours ≒ 24 hours
CS2-04 169776 ≒ 4 hours ≒ 32 hours
CS2-05 212220 ≒ 5 hours ≒ 40 hours
CS2-06 254664 ≒ 6 hours ≒ 48 hours
CS2-07 297108 ≒ 7 hours ≒ 56 hours
CS2-08 339552 ≒ 8 hours ≒ 64 hours
CS2-24 1018195 ≒ 24 hours ≒192 hours
CS2-48 2036390 ≒ 48 hours ≒ 384 hours

Table 4.5: Statistics of Dataset prepared for the Scalability Evaluation of Rule-engines and
Relational Database.

93
Contents

Figure 4.14: Performance Evaluation Result of Different Rule-engines and Relational

Database Models. The Figures show the Average Response Time on Different
Datasets shown in Table 4.5. The ‘◦’ indicates Time-out (> 1200 sec).

different video sources. The total volume of the data is about 5 hours and 30 minutes, that

is about 40 minutes per each source. Based on the original data set, we have populated

dummy instances to prepared data sets as shown in Table 4.5. The data sets are designed

to contain number of dummy instances correspond to varying hours. For example, the data

set CS2-48 contains, 2036390 instances that corresponds to 384 hours (48hours / source ×

8 source).

94
 Contents

For the evaluation, we set SWI-Prolog 5.6.51, CLIPS 6.3 and MySQL 5.0 on a

core2duo T7200 1.8 Ghz, 2Gb Ram Windows XP machine. Figure 4.14 shows the eval-

uation results. Overall, SQL performs better than rule engines as expected. However,

this was not always the case that SQL performs better than rule engines. For example,

in the case of ‘Q2’, the VAP scheme showed worst result. Indeed, the cases ‘Q1’, ‘Q3’

and ‘Q7’ show similar results between CLIPS and VAP Scheme. This seems because the

relational models (schemes) were not optimized as it is usually done in the reality of appli-

cation development. Therefore, it seems that there would be some optimization issues such

as setting proper indices and separating tables according to correlated data density, etc.

Overall, CLIPS performed better than Prolog. The rules can also be optimized considering

Chapter 4
the data distributions so that it can traverse less amount of instances first in the case of

conjunctive rules. Therefore, speaking overall assessment on the evaluation, it seems that

rule engines could also cover about 150,000 to 200,000 instances of data (in total about 32

to 40 hours 8 camera video sources in our setting) within 5 minutes response time, when

the rule optimization is also considered. Nevertheless, traditional relational database seems

still promising in terms of response time although they can not support automated infer-

ence features such as backtracking/ forwardchaining, unification, lists, or adhoc nested

structures. Therefore, as a hybrid approach, one could consider the automated rules to

single SQL query mapping mechanism, that can be triggered once rule engine binds rule

search tree. To this extent, it appears many literatures on this issue. Some of them are

to simply interface rule engine with relational database [39, 106, 112], to translate and to

compile rules in prolog to SQL [55] and to optimize rules relational query system [89].

4.7 Chapter Summary

In summary of lessons learned in this chapter, while the attempt of using ontological

metadata representation and logic programming showed advantage on flexibility of

representing and resolving epistemic contextual semantics, the scalability is not sufficient

95
Contents

enough and need to be addressed in the future research.

Intelligent analysis of high-level semantics is an important capability in visual surveil-

lance systems with high potential usage. Reminding the requirements discussed in Sec-

tion 1.2.4, such efficient semantic analysis of visual surveillance data requires a system

oriented approach that optimally combines the individual legacy vision analytic power and

its optimal use. In this chapter, we discussed the software components, data processing

pipeline and system architecture design. Among others, the main design philosophy of

our approach was to maximize utilization of available analytic metadata together with

context information. In this view, the proper vision analytic metadata representation and

the contextual knowledge expressive power of the reasoning framework are critical issues

to accomplish the goal. For the former requirement, we introduced ontological metadata

representation and for the latter requirement, we adopted logic programming. The main

advantage of the presented approach is in the ‘flexibility’ of representing and resolving

epistemic contextual semantics by leveraging logic programming and a description logic

based ontological data representation model (this is due to the ‘modularity’ as explained

in Section 2.3.3).

While the reasoning capability can be applied both the real-time analysis and the

forensic post analysis, the two case studies show the potential and the feasibility of the

proposed approach, especially in forensic sense of retrieval. To be fully applicable for

practical real applications, the reasoning power should be ‘scalable’. Section 4.6 shows

performance benchmark of triple stores and rule-engines against different scale of metadata

(i.e., fact base in view of rule-engines). Unfortunately, it seems that logical formalism based

approaches and the triple stores of ontological metadata are by themselves not sufficiently

scalable. The ‘scalability’ problem is common issue in the realm of ontology related research

field and also expert system related researches. One good news, however, is that there have

been undergoing active research focus on the ‘scalability’ issue. We have briefly introduced

some possible remedies shown in literatures. In addition, the ‘scalability’ of low-level feature

96
 Contents

matching is also important and is an active research topic in the realm of high dimensional

vector indexing field. Therefore, we believe and hope that we could benefit from those

researches to resolve the ‘scalability’ issue in the near future.

Another critical and rather essential issue is the proper handling of ‘uncertain’ nature

of vision analytic metadata and knowledge models. In the rest of this dissertation we will

further discuss this aspect in detail.

Chapter 4

97
98
5 Contradictory Information Handling

Thus far, we have discussed architectural aspect of the proposed approach to high-level se-

mantic analysis of visual surveillance data. The main advantage of the proposed approach

is on the high degree of ‘modularity’ that makes more ‘flexible’ contextual knowledge mod-

eling possible. Besides the ability of flexible context modeling, ‘uncertainty’ handling

mechanism is also an important issue. While we have briefly explained the extension of

logic programming with subjective logic (see Section 4.3.2), the detailed aspect to sup-

port ‘epistemic reasoning’ under ‘uncertainty’ is not covered in the previous Section. In

Chapter 5
this chapter, we further deal with the ‘uncertainty’ aspect starting with the discussion

on the ‘default reasoning’ that supports ‘non monotonicity’ explained in the preliminaries

Section 5.5.

5.1 Introduction

In forensic analysis of visual surveillance data, ‘default reasoning’ can play an important

role for deriving plausible semantic conclusions under ‘imprecise’, ‘incomplete’, ‘conflict’

and ‘contradictory’ information about scenes. In default reasoning, not only facts (analytic

metadata) but also rules (contextual knowledge) are the sources of ‘imprecise’ and ‘con-

tradictory’ information. To be epistemically ‘non-monotonic’, in such reasoning, not only

the proper representation of ‘epistemic belief strength’ about given piece of information

but also the proper ‘principled fusion’ of the information is required. A discrete species of

Bilattice for multivalued default logic is one that demonstrated default reasoning in visual

surveillance. In this chapter, we present an approach to default reasoning using subjective

99
Contents

logic that acts in a continuous space. As an uncertainty representation and handling for-

malism, subjective logic bridges Dempster Shafer belief theory and second order Bayesian,

thereby making it attractive tool for artificial reasoning. For the verification of the pro-

posed approach, we further extend the inference scheme on the bilattice for multivalued

default logic to L-fuzzy set based logics that can be modeled with continuous species of bi-

lattice structures. We present some illustrative case studies in visual surveillance scenarios

to contrast the proposed approach with such L-fuzzy set based approaches.

5.2 Background and Motivation

The main objectives of this chapter is as follows: 1) to bestow ‘default reasoning’ capa-

bility upon our subjective logic extended logic programming framework, 2) to compare

the proposed framework with other uncertainty formalisms that could alternatively model

‘default reasoning’ to better position the subjective logic based approach.

The proposed architecture and reasoning framework shown in Chapter 4, can be

regarded as an ‘extensional approach’ in the view of artificial intelligence as explained in

Section 2.3. Extensional approaches treat knowledge as conditional rules that are labeled

with uncertainty [130]. For the labeling uncertainty, in this dissertation, we proposed

the use of subjective logic [93]. However, when it comes to uncertainty representation

formalisms, there are number of other formalisms such as Bilattice [69], fuzzy set based

fuzzy logic [176, 177], Dempster Shafer belief theory [149] and traditional probability based

Bayesian approaches [33], etc. Therefore, for the proper choice of uncertainty formalism,

it is important to know their characteristics and behind philosophy on representing and

handling uncertainty.

As explained in Section 3.1, subjective logic [92, 93] is also one such uncertainty rep-

resentation and handling formalism that can be seen as extended theory derived from both

the Dempster Shafer belief theory and the second order Bayesian. From Dempster Shafer

belief theory, subjective logic inherits the philosophy of explicit representation of ignorance

100
 Contents

about knowledge in a model called subjective opinion triangle that can be also mapped

into beta distribution. The operators of subjective logic are also derived in the sense of

Bayesian. Unlike traditional Dempster Shafer evidence fusion method, that is known to

yield counter intuitive result when it is operated with highly contradictory evidences and

also known to be inconsistent with Bayes’ rule, subjective logic comes with similar opinion

fusion operators that are robust even with such highly contradictory evidences [92]. Com-

pared with bilattice that mainly consists of two lattices, one representing degree of truth

and the other representing degree of information respectively, the degree of information

concept is similar to the degree of ignorance in subjective opinion. The main difference

between bilattice and subjective logic is the operators. While bilattice comes with four

operators that are compositionally defined based on two lattice operators meet and join

from the perspective of set theory, subjective logic comes with 12 operators defined rather

in Bayesian sense. Another formidable uncertainty handling formalism, fuzzy logic is based

on fuzzy set theory that relies on degree of membership concept for a knowledge segment

Chapter 5
and again this is similar to the concept of partial ignorance in subjective logic. Interest-

ingly, it is known that some extensions of fuzzy logics can be modeled with (bi-)lattice

structures. One thing worth to note concerning fuzzy logic is that, even though there are

Zadeh’s original logical operators, there are yet another ways of defining logical operators

as well. However, due to this aspect, there is inconsistent between fuzzy logic operators and

classical probability calculus, thereby often criticized by statisticians who prefer Bayesian

[179]. Thus, we advocate that above aspects make the use of subjective logic attractive as

a means of representing and handling uncertainty for artificial reasoning.

In addition to uncertainty representation aspect, what is also important is the un-

certainty handling in a way supporting non-monotonic property. In reality, the truthness

of a partial knowledge segment is often easy to be fragile, because there can be potentially

possible contradictions or counter examples about the given knowledge segment. Due to

this aspect, the property of retracting and updating existing beliefs upon acquisition of

new information (aka. belief revision) is essential. Default reasoning introduced by Reiter

101
Contents

[143] is one such non-monotonic reasoning method especially under contradictory know-

ledge segments. Default reasoning allows expressing a segment of knowledge as being ‘true

by default’ or ‘generally true’, but could be proven false upon arrival of new information.

A discrete species of bilattice structure that represents multivalued default logic is

one that is used to model default reasoning and demonstrated the usefulness on performing

human identity maintenance and contextual reasoning of event in visual surveillance do-

main [153, 154]. As noted above, the degree of truth and the degree of information concepts

in bilattice are similar to the ones in subjective logic. Focusing on the similarity, we exam-

ine subjective logic operators that have corresponding semantic behavior to the operators

defined on bilattice framework. As mentioned above, what is also interesting is that some

continuous species of bilattice structures are often used to represent two different species

of fuzzy logic. Namely, intuitionistic (or interval-valued) fuzzy logic that can be modeled

with so-called ‘triangle’ bilattice and fuzzy Belnap logic (i.e., fuzzified four-valued logic,

FOUR) that can be also modeled with so-called ‘square’ bilattice [18]. The relationship

between these two fuzzy species of bilattice structures is studied in the work of Cornelis

et al. [42, 43] and Arieli et al. [18, 19]. Interestingly, the uncertainty representation in

intuitionistic fuzzy logic (‘triangle’) is very similar to that of the opinion triangle. There-

fore, to verify the proposed subjective logic based default reasoning approach and to study

its similarity and dissimilarity with fuzzy logics, we further extend the inference scheme

defined on the discrete bilattice structure for the multivalued default logic onto the two

continuous species of bilattice structures. To better verify and contrast the characteristics

of the proposed approach, we present some illustrative case study examples in typical vi-

sual surveillance scenarios. We then compare the default reasoning results yielded from

the proposed subjective logic based approach, bialttice for multivalued default logic, the

intuitionistic fuzzy logic (‘triangle’) and the fuzzy Belnap logic (‘square’ ).

We believe this way of comparison better position the subjective logic as a tool for

artificial reasoning and also give us better insights on the correlations among different

uncertainty formalisms. Namely, by the inherent nature of Subjective logic, it gives a

102
 Contents

bridge between Dempster Shafer belief theory and Bayesian. Then by the comparison in

this work on modeling default reasoning, it shows the bridge among subjective logic, fuzzy

logics and bilattices as well.

5.3 Related Work

As higher-level semantic analysis of visual surveillance data is gaining growing attention,

the flexibility on knowledge representation and proper uncertainty handling mechanism

is becoming more important. To address this aspect, there has been some work on the

use of logic programming languages due to the expressive power and on the use of differ-

ent uncertainty handling formalisms. In general, such approaches can be referred to as

‘extensional approach’. Extensional approaches (also known as rule-based systems) treat

uncertainty as a generalized truth value attached to formulas and compute the uncertainty

of any formula as a function of the uncertainties of its sub formulas [130]. Akdemir et

Chapter 5
al. [13] used an ontology structure for activity analysis, but with no uncertainty handling

mechanism. Shet et al. [152] introduced a system that adopts Prolog based logic program-

ming for higher-level situation reasoning in visual surveillance. The same authors adopted

bilattice based multivalued default reasoning for identity maintenance of human detection

results and context reasoning [153, 154]. Jianbing et al. [110] adopted Dempster Shafer

belief theory with the use of rule-based system for bus surveillance scenario. Anderson et

al. [17] adopted fuzzy logic to model and analyze human activity for video based eldercare

scenario. While different uncertainty handling formalisms are introduced with logic pro-

gramming based knowledge modeling, principled handling of default reasoning has been

only demonstrated by the bilattice based approach [153, 154] (refer to Section 2.4.3 for

more detailed comparision).

When it comes to bilattice framework itself, it is known that some continuous species

of bilattice structures that are called ‘triangle’ and ‘square’ correspond to intuitionistic

fuzzy logic and fuzzy Belnap logic, respectively [18]. Naturally, there has been comparative

103
Contents

study on the characteristics between intuitionistic fuzzy logic and fuzzy Belnap logic [42,

43]. Atanassov [22] introduced a transformation between these two fuzzy logics and proved

that the transformation is bijective. The use of square bilattice is demonstrated to improve

human detection results by the use of rule-based reasoning given high false alarm rate and

partial occlusion based output of different body parts based detectors [155] with the similar

inference scheme shown in their previous work [153, 154]. In this chapter, we show that

the default reasoning behavior on multivalued default and square bilattice can be also

modeled using subjective logic. Relying on the study of Atanassov [22] and Cornelis et al.

[42, 43], we also show the correspondence among subjective logic, intuitionistic fuzzy logic

(‘triangle’) and fuzzy Belnap logic (‘square’).

5.4 Preliminaries

This section gives an overview of the fundamental background about uncertainty represen-

tation and handling formalisms that will be discussed in this chapter in the view of default

reasoning. The preliminaries will cover bilattice theory, and two extensions of fuzzy log-

ics, namely, intuitionistic fuzzy logic and fuzzy Belnap logic. Refer to Chapter 3 for the

theoretical overview of subjective logic theory and logic programming.

5.4.1 Bilattice Theory

Bilattices are algebraic structures which are mainly built on top of the concept poset (i.e.,

partially ordered set, which is a generalization of ordering elements in a set, in terms of

a property of our interest) introduced by Ginsberg [69] and elaborated by Fitting [60].

Ginsberg’s formal definition of bilattice is as follows [69].

Definition 18. (Partial Order) . A partial order is a binary relation ≤ over a set S

which is reflexive, antisymmetric and transitive, i.e., for all a,b and c in S, satisfies : a)

a ≤ a (reflexive) b) if a ≤ b and b ≤ a then a = b (antisymmetric) c) if a ≤ b and b ≤ c

then a ≤ c (transitive).

104
 Contents

Definition 19. (Poset) . A set S with a partial order (S, ≤) is called partially ordered

set (or poset).

Definition 20. (Lattice) . A poset L with a partial order is a lattice (L, ≤, ∧, ∨) if it

satisfies the following two axioms:

a) for any two elements a and b of L, the set {a, b} has a least upper bound ∨ (join).

b) for any two elements a and b of L, the set {a, b} has a greatest lower bound ∧ (meet).

Definition 21. (Pre-bilattice) . A pre-bilattice is a structure

B = (B, ≤t , ≤k ), where B is a nonempty set containing at least two elements, and (B, ≤t ),

(B, ≤k ) are complete lattices (for which all subsets are also lattices).

Definition 22. (Bilattice) . A bilattice is a structure

B = (B, ≤t , ≤k , ¬), such that (B, ≤t , ≤k ) is a pre-bilattice and ¬ is a unary operation on

B that has the following properties: for every x, y in B,

a) if x ≤t y then ¬x ≥t y, b) if x ≤k y then ¬x ≤k ¬y, c) ¬¬x = x.

Chapter 5
The name ‘bi’ - lattice indicates that it is a structure consists of two lattices. Lattices

are any poset that are possible to define meet ∧ (aka. greatest lower bound) and join ∨

(aka. least upper bound) for any two elements in it. A partial order is a generalization

of ordering, i.e., a binary relation ≤ over a set S which is reflexive, antisymmetric and

transitive. Lattices are often expressed as a graph whose edges represent the binary relation

of ‘partial order’ between two elements that can be directly compared. There can be

elements a and b in the lattice L for which an order between them cannot be determined.

However, greatest lower bound meet (a ∧ b) and the lowest upper bound join (a ∨ b)

can always be determined for any of two elements a and b in the lattice L. Namely, by

considering two sets that contain elements that are greater than a and greater than b

respectively, the lowest element which can be found in both of the sets is the meet (and

join can be similarly defined). Figure 5.1 (a) shows a lattice that some elements in it, for

example, a and b are incomparable but still has a ∧ c = f and a ∨ c = t. Conventionally

in lattice theory, ‘1’ represents the greatest elements and ‘0’ is the lowest elements and

105
Contents

Figure 5.1: Examples of Lattice and Bilattices (a) a Lattice (b) Bilattice corresponding to
Traditional Binary Logic (c) Bilattice corresponding to Three-valued Logic (d)
Bilattice corresponding to Belnaps Four-Valued Logic, FOUR. (Refer to [69]
for more detail).

therefore for any a in lattice L, a∧1 = a, a∨1 = 1, a∧0 = 0 and a∨0 = a. Bilattices provide

semantics for reasoning by considering one lattice with partial order in terms of degree of

truth ≤t and the other lattice with partial order in terms of degree of information ≤k (note

that, the semantics of degree of information often can be seen as degree of specificity of

the information as well).

To avoid the confusion that would arise from using the same symbols meet ∧ and

join ∨ for both lattices, following Fitting, we use the symbols meet ⊗ and join ⊕ for

the lattice with partial order ≤k [60]. While the meaning of ∧ and ∨ corresponds to

the standard logical role of conjunction and disjunction, the meaning of ⊗ and ⊕ are

less intuitive. Fitting named ⊗ as consensus operator in the sense that it derives the

most degree of information agreed upon two operands [60, 61]. Likewise ⊕ is named as

gullibility operator in the sense that it accepts any degree of information upon two operands.

In bilattices, therefore, when the gullibility operator ⊕ is used, getting more information

106
 Contents

pushes the overall belief towards true or false with more degree of information except in

case of contradiction. Figure 5.1 (b) (c) (d) shows different bilattice structures that can

model different logics.

5.4.2 L-Fuzzy Set Based Logics (Interval-valued Fuzzy Logic, Intuitionistic

Fuzzy Logic and Fuzzy Belnap Logic)

Since the introduction of Fuzzy set theory and Fuzzy logic by Zadeh [176, 177], it became

popular as a formalism for representing imprecise or vague linguistic concepts (e.g. such

as ‘hot’, ‘cold’, ‘fast’, etc.). The basic idea is to introduce a fuzzy membership function

(conventionally denoted as µ) as a measure of vagueness to elements in a set and it is called

fuzzy set. The membership functions are defined to map an element u to a value within

the interval [0, 1] (i.e. µ(u) → [0, 1] ), thereby assigning exact value makes all elements in

the fuzzy set to be ordered and comparable. Due to this aspect, there has been arguing

Chapter 5
that this makes them inadequate for dealing with incomparable uncertain information [42].

There have been some remedies for this aspect. Noting on the footnote comment of Zadeh

saying ‘in a more general setting, the range of the membership function can be taken to be

a suitable partially ordered set P.’ (p. 359 of [176]), Goguen introduced L-fuzzy set that

uses a membership function that maps an element u to a value in a lattice (i.e. µ(u) → L)

[70]. Interval based representation of degree of membership is also introduced with the

name of interval-valued fuzzy sets (IVFSs, for short) by Gehrke et al. [68]. In IVFSs, an

element u is mapped into a subinterval within (i.e. µ(u) → (vl , vu ) ∈ [0, 1]2 , vl ≤ vu ).

Intuitionistic fuzzy set theory (IFSs, for short) introduced by Atanassov [21] additionally

adopts a non membership function ν, with a weaker constraint ν ≤ 1 − µ (note that, in the

sense of Zadeh’s original fuzzy set the ν is implicitly assumed to be ν = 1 − µ ). Naturally,

the socalled amount of ‘indeterminacy’ or ‘missing information’ can be defined by the

value π = 1 − ν − µ. By dropping the constraint µ + ν ≤ 1 and introducing even weaker

constraint µ + ν ≤ 2, we get fuzzy Belnap set. Unlike IVFSs that requires one to address

values to be vl ≤ vu as lower bound and upper bound of imprecision, IFSs allows one to

107
Contents

Figure 5.2: Lattices (a) L∗ Triangle corresponding to Intuitionistic Fuzzy Logic (and also
can be seen as LI Triangle for Interval-Valued Fuzzy Logic), and (b) L□ Square
corresponding to Fuzzy Belnap Logic.

address the positive and the negative side of an imprecise concept separately. Cornelis et

al. [44], however, showed that IVFSs can also be represented in the form of IFSs, in other

words the two are isomorphic and the truth values in IFSs can be represented as intervals

(e.g. by the mapping f (xl , xu ) = (xl , 1 − xu )). As traditional fuzzy set is used for fuzzy

logic as a measure of uncertainty on a proposition, IVFSs, IFSs and fuzzy Belnap set are

adopted for interval-valued fuzzy logic (IVFL, for short), intuitionistic fuzzy logic (IFL,

for short) and the fuzzy Belnap logic (aka. Fuzzified four-valued logic, FOUR). Following

Goguen [70], IVFL and IFL can be defined on ‘triangle species of lattices denoted LI and

L∗ respectively. Fuzzy Belnap logic can be defined on the ‘square’ lattice denoted L□ .

Therefore, IVFSs, IFSs and Fuzzy Belnap logics are kind of L-fuzzy logic. The formal

definition of ‘square’ and ‘triangle’ are as follows [19, 43, 60].

Definition 23. (L□ Square lattice for fuzzy Belnap logic) . L□ = (L□ , ≤□ ), where

L□ = [0, 1]2 and (x1 , x2 ) ≤□ (y1 , y2 ) iff x1 ≤ y1 and x2 ≥ y2 .

Definition 24. (L∗ Triangle lattice for IFSs) . L∗ = (L∗ , ≤L∗ ), where L∗ = {(x1 , x2 ) ∈

[0, 1]2 |x1 + x2 ≤ 1} and (x1 , x2 ) ≤L∗ (y1 , y2 ) iff x1 ≤ y1 and x2 ≥ y2 .

Definition 25. (LI Triangle lattice for IVFSs) . LI = (LI , ≤LI ), where LI =

{(x1 , x2 ) ∈ [0, 1]2 |x1 ≤ x2 } and [x1 , x2 ] ≤LI [y1 , y2 ] iff x1 ≤ y1 and x2 ≤ y2 .

Figure 5.2 shows the corresponding graphical interpretation of IFL and fuzzy Bel-

108
 Contents

nap logic. As it was shown in Figure 5.1 (c) and (d), they correspond to continuous

extension of three valued logic and Belnap logic (Figure 5.2 (a) and (b), respectively). In

epistemic sense, the values (0, 0) corresponds to ‘unknown’, (1, 0) corresponds to ‘true’,

(0, 1) corresponds to ‘false’, (x, x) corresponds to ‘undecidable’ and (1, 1) corresponds to

‘contradiction’. In Belnap, the ‘contradiction’ point is a special case of ‘undecidable’ points

considered to have even more information than ‘definite truth’ or ‘definite false’. In IFL and

IVFL, however, this way of ‘contradiction’ state is not allowed because IFL and IVFL do

not allow epistemic points that is considered to have even more information than ‘definite

truth’ or ‘definite false’.

5.5 Default Reasoning

Defaults (default assumptions) are statements that can be interpreted as ‘normally, typ-

ically, generally true or false’ as a rule. Contrary to defaults, statements that express

Chapter 5
explicit truth or falsity are called definite rules. In practice, the need to make default

assumptions often occurs in cases where the information at hand is uncertain, incomplete

and potentially contradictory. Default reasoning attempts to draw plausible conclusions

based on known defaults and definite rules. Therefore, in default reasoning, conclusions

can be changed upon acquisition of new information (i.e., ‘nonmonotonicity’). In logic,

Reiter formalized such reasoning aspects as default logic theory using default rules [143].

In the following we give a brief overview on how rules are expressed in logic programming

and a brief introduction to default logic.

5.5.1 Reiter’s Default Logic

This section describes Reiter’s formalization of default logic and an example of default

reasoning in visual surveillance.

Definition 26. (Default Theory) [143] . Let ∆ = (D, W ) be a default theory, where W

is a set of logical formulae (rules and facts) also known as the definite rules and D is a set

109
Contents

Figure 5.3: Illustrative Example of Default Reasoning Scenario in Visual Surveillance.

α:β
of default rules of the form γ , where α is known as the precondition, β is known as the

justification and γ is known as the conclusion.

Any default rule dr ∈ D can be also written as ‘γ ← α, not(¬β)’, where not means

the negation by failure to prove. The interpretation of such rule is that, if the precondition

α is known to be true, and if there were no explicit violations of the justification (facts

and rules that entails ¬β) then it is possible to derive the conclusion γ.

Example 2. (A Default Reasoning Scenario in Visual Surveillance) . Assume a

scene as depicted in Figure 5.3 with two cameras observing the upper and lower parts of

an escalator respectively. The scene also shows stairs next to the escalator. Consider the

following set of rules :

¬escalator_working(T ) ← people_take_stairs(T ),
not(escalator_blocked(T ))
escalator_blocked(T ) ← crowd_at_entrance(T )
where, ¬escalator_working(T ) ∈ D and escalator_blocked(T ) ∈ W .

Assume that Cam1 continuously observes that people appear to be using the stairs

and generates a set of facts as {people_take_stairs(T1 )}Cam1 . Based on the current

set of facts and rules, by default, the rule ¬escalator_working(T1 ) is satisfied because

we can not explicitly prove escalator_blocked(T1 ). However, at a later time we ob-

serve a crowd in front of the escalator and as soon as of Cam2 generates a set of facts

{crowd_at_entrance(T2 )}Cam2 from its observations, then, the proposition

¬escalator_working(T2 ) is no longer supported and is withdrawn.

110
 Contents

5.5.2 Bilattice based Multivalued Default Logic

Ginsberg showed the use of bilattice structure to model default reasoning aspect and ex-

tended the structure to generalized default reasoning framework called multivalued default

logic (aka. prioritized default logic) for artificial reasoning [69]. Ginsberg’s bilattice struc-

tures also inherits the behind philosophy of Belnap logic in the sense that they also adopt

the epistemic states ‘unknown’ and ‘contradictory’. To distinguish definite truth and de-

fault truth value, default truth values assumed to have different amount of truth and

different amount of information are also introduced. Figure 5.4 shows Belnap logic that

has no default truth values (i.e., traditional four-valued logic), default logic and multi-

valued default logic respectively. Based on this truth value setup and the four bilattice

operators, each bilattice operator can be seen as a truth functional binary operator on

those values. Table 5.1 shows (a) the truth table of Belnap logic that has no defaults and

(b) default logic that has default true and default false as epistemic states. Based on the

Chapter 5
truth functional binary operators, inference on the bilattice framework is defined in terms

of truth assignment and closure as follows [69].

Definition 27. (Truth Assignment) . Given a declarative language L and a Bilattice

B, a truth assignment is a function ϕ : L → B.

Definition 28. (Closure) . Given a knowledge base K in form of declarative language

and a truth assignment labeling each sentence k ∈ K with a truth value and a Bilattice B,

then the closure ϕ of a given query sentence q denoted cl(ϕ)(q), is the truth assignment

function such that: cl(ϕ)(q) : {p, p′ |∀S, S ′ ⊆ K, S |= q, S ′ |= ¬q, p ∈ S, p′ ∈ S ′ } → B.

In other words, the implication of cl(ϕ)(q) is a functional mapping from the ‘enu-

meration of all sentences’ that can entail (denoted by the symbol ‘|=’) q and its contra-

dictory information ¬q to a ‘truth value in bilattice B’. For example, if ϕ labels sentences

{p, q ← p} ∈ K as true; i.e., ϕ(p) = T and ϕ(q ← p) = T , then cl(ϕ) should also label q as

true as it is information entailed by K.

111
Contents

Figure 5.4: (a) Belnap Logic, FOUR (b) Default Logic (c) Multivalued (Prioritized) De-
fault Logic.

Table 5.1: Truth Table of Bilattice Operators on (a) Belnap (b) Default Logic (The un-
shaded Part is exactly compatible with Belnap).

Definition 29. (Default Inference) . Given a query sentence q and given S and S ′ that

are sets of sentences such that S |= q and S ′ |= ¬q, then the default inference is the truth

value assignment closure cldi (ϕ)(q) given by :

⊕ ∧ ⊕ ∧
cldi (ϕ)(q) = u∨[ cl(ϕ)(p)] ⊕ ¬ u∨[ cl(ϕ)(p)]. (5.1)
S|=q p∈S S ′ |=¬q p∈S ′

Informally, Equation 5.1 states that for n sets of sentences S that entails q, we first

collect the lowest upper bound (‘∧’) that every sentence in Si can agree on, then take it

112
 Contents

if the result contains more truth than u (unknown) along the partial order ≤t . For each

of these truth values of Si , we evaluate the amount of information and choose the most

informative (certain or specific) one among them using greatest lower bound (‘⊕’) along

the partial order ≤k . We do the same process for all Si′ and by understanding the result

from S ′ as contradictory hypothesis, we again collect the most informative one and apply

the negation operation. Finally, both resulting intermediate values for Si and Si′ are joined

along the partial order ≤k again using greatest lower bound (‘⊕’).

5.6 Default Reasoning using Subjective Logic

This section describes an inference mechanism for default reasoning using subjective logic.

We will discuss how multiple logical values (i.e., default and definite truth values) can

be modeled in subjective logic’s opinion space. Thereafter, we propose an approach to

modeling default reasoning based on subjective logic operators by analyzing the default

Chapter 5
reasoning mechanism on bilattice and identifying corresponding and analogous behaviour.

5.6.1 Fusing Contradicting Dogmatic Opinions in Subjective Logic

The core idea of default theory in Definition 26 is on the ability to discard a weaker

belief by updating current belief based on a more specific or stronger belief. Among other

approaches, the main strategy of the billatice based default inference shown in Definition 29

was to prioritize possible states of beliefs in an ordered set and to update lower ordered

belief with higher ordered belief. Especially, the ‘join operator’ ⊕ played an important

role for combining competing truth values, that for example, draws an ‘undecidible’ point

when it fuses two conflicting beliefs at the same level of information (i.e., dfn and dtn in

Figure 5.4 (c)).

As explained in Section 3.1, subjective logic uses theoretical elements from the

Dempster-Shafer belief theory that the Dempster’s rule (see Definition 7) plays the central

role for fusing different belief mass assignments. Benferhat et al. [26] introduced the use

113
Contents

of Dempster Shafer belief theory for default reasoning. However, Dempster’s rule has been

criticized mainly because highly contradicting beliefs tend to produce counterintuitive re-

sults [98]. Among others, the critique is further formulated and discussed in the form of

examples by Zadeh [178]. Audun Jøsang et al. [98, 94] also introduced such an information

fusion method called ‘consensus’ operator and showed that it is robust even in the highly

contradicting cases.

In this section, therefore, we will briefly review how conflicting or contradictory

information can be fused using subjective logic’s ‘consensus’ operator and contrast to

Dempster’s rule (see Definition 7).

The consensus operator is designed to ‘fairly’ reflect two opinions in a single opinion.

Informally, this means that, each opinion respect the other as much of their ignorance.

The formal definition of ‘consensus’ operator is as follows.

Definition 30. (Consensus ⊕sl ) [94] . Let wxA = (bA A A A B B B B B

x , dx , ix , ax ) and wx = (bx , dx , ix , ax )

be opinions respectively held by agents A and B about the same state x, and let k =

x + ix − ix ix . When ix , ix → 0, the relative dogmatism between wx and wx is defined

iA B A B A B A B

by γ so that γ = iB A A B
x /ix and γ = 1, when both i and i are exactly 0 (i.e., when A and B

forms dogmatic opinions). Let wxA,B = (bA,B A,B A,B A,B

x , dx , ix , ax ) be the opinion such that:
 

 A,B A iB + bB iA )/k 
 bA,B
A +bB
= γbγ+1
 b = (b 
x x


x x x x x 

x

 

 dA,B = (dA B B A  dA,B A +dB
= γdγ+1
x ix + dx ix )/k
x x
x x
k ̸= 0 : k=0:

 


 iA,B
x = (iA B
x ix )/k 
 iA,B
x =0

 

 A,B
 A A B iA −(aA +aB )iA iB 
 aA,B = γaA B
ax = ax ix +aiAx+i x x
B −2iA iB
x x x
x
x +ax
γ+1
x x x x

Then wxA,B is called the consensus opinion between wxA and wxB , representing an imaginary

agent [A, B]’s opinion about x, as if that agent represented both A and B. By using the

symbol ⊕ to designate this operator, we define wxA,B = wxA ⊕ wxB .

The behind philosophy of consensus operator is similar to the one of Bayesian.

Namely, assuming two independent observations (opinions) A and B about a proposi-
tion x, it reflects both observations (opinions) in a ‘fair and equal way’. In Bayesian, the
observation can be represented with the n number of sampling containing α number of

114
 Contents

positives and β number of negatives. Especially, using beta distribution, this can be repre-
sented as Beta(p(x)|α, β). Consider two independent observations observations (αA , β A )
and (αB , β B ). To be ‘fair’, each single observation should be reflected with the same
weight, regardless of the total number of observations. Therefore, the fused observation
should be the observation representing (αA + αB , β A + β B ) that represents again a beta
distribution Beta(p(x)|αA + αB , β A + β B ). According to the opinion to beta distribution
mapping scheme shown in Equation 3.9, this can be formulated as follows.
  

 A A 
 B B 
 Beta(p(x)|αA + αB , β A + β B )

 Beta(p(x)|α , β ) 
 Beta(p(x)|α , β ) 


 
 


 
 


 ⇓ 
 ⇓ 
 ⇓
  
⊕
A B A
α α +αB
bA = αA +β bB = αB +β = bA,B = αA +ααB +β (5.2)


A +2


B +2


A +β B +2

 
 

  
A B A
β β +β B

 dA = αA +β 
 dB = αB +β 
 dA,B = αA +αβB +β
  
A +2 B +2 A +β B +2

 
 

 iA = 2  iB = 2  iA,B = 2
αA +β A +2 αB +β B +2 αA +αB +β A +β B +2

Above (5.2), should be the same as the result derived by the Definition 30. For example,
the consensus calculation of bA,B in opinion space is as follows.

Chapter 5
bA iB +bB iA
bA,B = iA +iB −iA iB
,
by replacing bA , iB , bB , iA and iB with the corresponding elements shown in (5.2).


 bA iB + bB iA = αA +α αA 2 αB 2 2(αA +αB )

 A +2 αB +αB +2 + αB +αB +2 αA +αA +2 = (αA +β A +2)(αB +β B +2)



 2(αA +αB +β A +β B +2)
 iA + iB − iA iB = αA +α 2
A +2 + αB +αB +2 − (αA +β A +2)(αB +β B +2) = (αA +β A +2)(αB +β B +2)
2 4

 +α )
A B
2(α


 (( (((

 → bA,B = bA iB +bB iA ( (A(
(αA +β +2)(α
= ( A B A B
B
+β B +2) αA +αB

 iA +iB −iA iB
= αA +αB +β A +β B +2
.

  +α +β +β (
2(α +2)
(
(( (
((((
(αA +β A +2)(α B
+β B +2)
(5.3)
dA,B and iA,B can be proven similarly as above. Reminding the discussions in Section 3.1.3,
in fact, the ‘consensus’ operator is applying the bayesian theorem itself. The essence
of bayesian theorem is ‘posterior ∝ prior × likelihood’. As the beta distributions are
‘conjugated family’ of distributions, we know that the multiplication can be simplified by
just adding the corresponding index values as follows.

Beta(p(x)|αA , β A ) × Beta(p(x)|αB , β B ) = Beta(p(x)|αA + αB , β A + β B ) (5.4)

115
Contents

mA
Θ mB
Θ Dempster’s Non-normalised Consensus
x ∈ 2Θ rule Dempster’s rule operator
A,B
Peter 0.98 0.00 0.490 0.0098 wpeter = (0.492, 0.503, 0.005, 1/3), E=0.494
Paul 0.01 0.01 0.015 0.0003 wPA,B
aul = (0.010, 0.985, 0.005, 1/3), E=0.012
A,B
Mary 0.00 0.98 0.490 0.0098 wM ary = (0.492, 0.503, 0.005, 1/3), E=0.494
Θ 0.01 0.01 0.005 0.0001 -
∅ 0.00 0.00 0.000 0.9800 -

Table 5.2: An Example of Applying Dempster’s Rule and Consensus Operator for Uncer-
tain Belief Fusion [94].

mA
Θ mB
Θ Dempster’s Non-normalised Consensus
x∈2 Θ
rule Dempster’s rule operator
A,B
Peter 0.99 0.00 0.000 0.0000 wpeter = (0.495, 0.505, 0.000, 1/3), E=0.495
Paul 0.01 0.01 1.000 0.0001 wPA,B
aul = (0.010, 0.990, 0.000, 1/3), E=0.010
A,B
Mary 0.00 0.99 0.000 0.0000 wM ary = (0.495, 0.505, 0.000, 1/3), E=0.495
Θ 0.00 0.00 0.000 0.0000 -
∅ 0.00 0.00 0.000 0.9999 -

Table 5.3: An Example of Applying Dempster’s Rule and Consensus Operator for Dogmatic
Belief Fusion - Dempster’s Rule Deriving Counter Intuitive Result [94].

Even though the ‘numerical multiplication’ ‘×’ was used on beta distribution level of

calculation, it is important to note that, the ‘logical multiplication (conjunction)’ ‘∧’ in

subjective logic is different to the consensus operator. While ‘consensus’ concerns about

different opinions about the ‘same’ proposition, ‘logical multiplication (conjunction)’ deals

with different opinions about ‘dfferent’ propositions, and concerns about the chances that

both may correct. For example, given two different binary frame of concern X = {x, ¬x}

and Y = {y, ¬y}, wxA ∧ wyB concerns about the chances being {xy} among X × Y =

{xy, x¬y, ¬xy, ¬x¬y} (see Definition 31 for detail).

Now, let us again consider the Example 1 in Section 3.1.1, that Zadeh [178] used

to criticize Dempster’s rule. The example deals with a murder case with three suspects;

Peter, Paul and Mary. Assume two conflicting testimonies. In Section 3.1.1, we have

reviewed the result (see Table 3.1) of dempster’s rule when the testimonies have some

amount of uncertainty in the testimony. Similarly, Table 5.2 shows the result of applying

consensus operator to the same example. As shown in Table 5.2, both Dempster’s rule

116
 Contents

and consensus operator derived similar results. However, as shown in Table 5.3, if we fuse

more highly contradicting beliefs having no uncertainty, Dempster’s rule derives counter

intuitive result saying that Paul is the suspect. Unlike Dempster’s rule, consensus operator

derived reasonable result similar to the case of Table 5.2.

5.6.2 On the Interpretation of Contradictory Point in Subjective Logic

As discussed, in subjective logic, fusing two contradictory information derives an opinion

that reflects both opinions fairly. In the case of ‘absolutely conflicting dogmatic opinions’,

namely definite true wA = (1, 0, 0) and definite false wB = (0, 1, 0), the consensus operator

derives wA,B = (0.5, 0.5, 0) which is also a dogmatic opinion. Cognitively speaking, this

opinion can be interpreted in two different views.

In bayesian view, such ‘conflict’ or ‘contradiction’ about a proposition x means that

(no matter how we have observed) we have infinite positive observations and also infinite

negative observations at the same time (i.e., Beta(α → ∞, β → ∞) = w(0.5, 0.5, 0)).

Chapter 5
Namely, in this view, full contradiction would be linguistically said as ‘a situation that we

know it could occur or not occur at the same absolutely definite rate 1/2 with no doubt’.

Given this, assume that a person should finally determine whether the proposition x would

occur or not. The person may say it seems ‘could happen with probability 0.5 but also could

not happen with probability 0.5’ or more simply ‘half and half ’.

While in logical view, the ‘conflict’ or ‘contradiction’ simply indicates the logically

inconsistent state itself, where definite true and definite false can arise for the same propo-

sition. In this sense, it most likely indicates logical error or wrong set up in the considered

frame of discernment. Namely, in this view, the focus is on the fact that the resulting

opinion had been derived through both the definite true and the definite false. In some

multi-valued logic formalisms such as Belnap, this state is labeled with a symbol such as

‘⊥’ called ‘(full) contradiction’. Especially, in bilattice based logic frameworks, the ‘con-

tradictory point’ is defined to have even more information than ‘definite truth’ and ‘definite

false’ as shown in Table 5.4 and Figure 5.5. This is due to the restriction that it is based

117
Contents

on a set theory (i.e., poset), in which every element should be ordered in a way it has

both the ‘meet’ and the ‘join’ operators. (Note that, the ‘contradictory’ point is ‘the only

point’ that is defined to have even more information than definite true and false as shown

in Section 5.4.1).

In summary, while the former focusses on the interpretation of a given opinion itself,

the latter rather focusses on the process how an opinion had been derived. To address

this discussion, Audun Jøsang [94] proposed an extra parameter called ‘degree of conflict’.

Following is the statement of Audun Jøsang in page 13 of [94].

‘An argument that could be used against our consensus operator, is that it does not give any indi-

cation of possible belief conflict. Indeed, by looking at the result only, it does not tell whether the

original beliefs were in harmony or in conflict, and it would have been nice if it did. A possible

way to incorporate the degree of conflict is to add an extra conflict parameter. This

could for example be the belief mass assigned to ∅ in Non-normalised Dempster’s rule, which in

the opinion notation can be defined as cA,B

x = bA B B A A,B
x dx + bx dx where cx ∈ [0, 1]. The consensus

opinion with conflict parameter would then be expressed as wxA,B = (bA,B

x , dA,B
x , iA,B
x , aA,B
x , cA,B
x ).

The conflict parameter would only be relevant for combined belief, and not for original beliefs. A

default value c = −1 could for example indicate original belief, because a default value c = 0 could

be misunderstood as indicating that a belief comes from combined harmonious beliefs, even though

it is an original belief.’

The ‘conflict’ parameter above can be used as an indicator to inform that the opinion

is derived by consensus. Following the setup, fusing definite true wxA = (1, 0, 0, 1/2, −1)

and definite false wxB = (0, 1, 0, 1/2, −1) would introduce the ‘conflict’ parameter cA,B =

1·1+0·0 = 1. Therefore, we get, wxA,B = (0.5, 0.5, 0, 1/2, 1). This gives us the clue how the

opinion was derived. Comparing wxZ = (0.5, 0.5, 0, 1/2, −1) and wxA,B = (0.5, 0.5, 0, 1/2, 1),

while the former indicates one independent observation based opinion and the latter indi-

cates that it was fused by two highly contradicting information, the actual distribution of

them are the same. Namely, once an opinion is derived, one practical interpretation could

118
 Contents

be to indicate to the application that there is a conflict situation. However, any subsequent

numerical calculation upon the resulting opinion will be the same, no matter how it had

been derived. Therefore, in this dissertation, we will not explicitly use the ‘conflict’ pa-

rameter but rather simply accept the Bayesian view for dealing with ‘(full) contradictory’

state. Then, we will further discuss this aspect again in Discussion 1 in Section 5.9.1.

5.6.3 Mapping Multi-Logic-Values into Opinion Space

In this section we discuss the possible mapping of multiple logic values into opinion space.

We start with noting that the implications of ≤t and ≤k in bilattice are similar to the

concept of truth and ignorance in subjective opinion space when visualized in the opinion

triangle. As shown in Figure 5.4 (b) and (c), dtn and df n indicate different levels of

incomplete truth or falsity. The more certain and specific knowledge is obtained, the

higher level of default values result. The degree of certainty or specificity can be considered

Chapter 5
as degree of information, therefore, the levels of default values can be ordered along the

partial order ≤k . Along the information order ≤k , for each pair of dtn and df n there exist

corresponding undecidible states ∗n . As shown in Figure 5.4 (b) and (c), ∗n are assumed

to have more information than their corresponding dtn and df n . Unknown state U is one

of the undecidible states with zero degree of information. Similarly, the full contradictory

state ⊥, that can be reached via definite true (full belief) and definite false (full disbelief),

is also one of the undecidible states with maximum degree of information.

In the sense of degree of information, however, assigning even higher degree of infor-

mation than definite true or false to the full contradiction point ⊥ is an interesting aspect

to discuss. In bilattice, the full contradictory state ⊥ is considered to have even more infor-

mation than definite true or false. While in subjective logic, it is again a dogmatic opinion

having full degree of information (no ignorance). As discussed in the previous Section 5.6.2,

however, this difference can be compromised by adding another dimension of information,

(i.e., the conflict parameter c = 1) to explicitly indicate that a fused opinion had been

derived via the definite true and the definite false. Therefore, strictly speaking, the bilat-

119
Contents

tice way of representing ‘full contradiction’ in subjective logic is (0.5, 0.5, 0, a, c = 1). To

distinguish this slight difference, we will denote the maximum undecidible point (namely,

full contradictory point) in bilattice as ⊥bl and ⊥sl = (0.5, 0.5, 0, a, 1) for subjective logic

(henceforce, the subscripts bl and sl denote bilattice and subjective logic respectively).
Except the full contradictory point, the rest of undecidible states ∗n can be defined
in the opinion triangle as of bilattice. Additionally, such ∗n should be able to reach via an
operation as of join operator ⊕bl in bilattice on the partial order ≤k . This aspect can be
modeled with subjective logic consensus operator ⊕sl (see Definition 30). By definition,
when dtn and df n (having the same level of ignorance) are fused with the consensus operator
⊕sl , it always yields an opinion in the middle of opinion triangle with less ignorance. The
only exception to this is the case of fusing definite true and definite false that yields an
opinion in the middle between definite true and definite false again with no ignorance.
For example, if we consider a tiny amount of ignorance1 ε and take (1 − ε, 0, ε) = t′ and
(0, 1 − ε, ε) = f ′ as any default true or default false, then fusing t′ and f ′ in terms of
degree of information in subjective logic t′ ⊕sl f ′ = ∗′ will always draw the values with less
ignorance ε′ < ε (see definition of i in the case of k ̸= 0 in Definition 30). Following is the
formal proof of this aspect.

Premise :
Given two opinions wxA = (1 − ε, 0, ε) and wxB = (0, 1 − ε, ε) , where ε ∈ (0, 1)
2(1 − ε) 2(1 − ε) ε ε
wxA ⊕ wxB = ( , , ) satisfies ε > .
2−ε 2−ε 2−ε 2−ε (5.5)
Proof :
ε
ε> ⇒
ε(2 − ε) >
ε ⇒ 2 − ε > 1, by the fact that ε ∈ (0, 1) ⇒ (1, 2) > 1
2−ε
ε
∴ε> □
2−ε

This behavior is exactly the same as what bilattice based default structures capture,
thus, in the sense of ordering along ≤k , t′ ≤k ∗′ and f ′ ≤k ∗′ . The only exception to
this is when definite true and false are fused. Namely, t ⊕sl f = ⊥sl (see Definition 30).
This means that only for this single point there is no exact correspondence but rather an
approximate correspondence. Consequently this point in the opinion triangle is denoted
1
Benferhat et al. [26] used similar idea of introducing ε for Dempster Shafer belief theory based default
reasoning.

120
 Contents

Chapter 5
Figure 5.5: (a)The Bilattice and Opinion Triangle Space for Belnap FOUR, (b) The Bi-
lattice and Opinion Triangle Space for Default Logic, (c) The Bilattice and
Opinion Triangle Space for Multivalued Default Logic.

as ⊥sl as depicted in Figure 5.5 (a). Figure 5.5 (a) depicts that the correspondence of the
bilattice used for Belnap logic, FOUR and the opinion triangle for example, by mapping
tbl ≃ tsl = (1, 0, 0, a), fbl ≃ fsl = (0, 1, 0, a), ubl ≃ isl = (0, 0, 1, a) and ⊥bl ≃ ⊥sl =
(0.5, 0.5, 0, a, c = 1). As mentioned in Section 5.6.2, however, considering that the conflict
parameter c does not affect to its subsequent calculation, we will not explicitly use the
conflict parameter in this dissertation. For the default values, following the discussion
above and nontheless values in bilattice are elements of a finite set and the opinion triangle
is of continuous domain, we could pick points along the side edges of the opinion triangle.
However, it is important to note that picking the logical values should be done in the sense
of selecting opinion points to put in a set that satisfies ordering like bilattice. Therefore,

121
Contents

there can be many ways of such mapping as long as we pick values such that:

1. dfn and dtn have the same amount of ignorance.

(5.6)
2. ignorance(dtn−1 ⊕ dfn−1 = ⋆n−1 ) > ignorance(dtn ⊕ dfn = ⋆n ).

Figure 5.5 (b) shows an example of such mapping of default true and default false to the

opinion triangle. In the same manner, we can extend such mapping to the generalized

multivalued default logic in which each of the defaults can be considered with different

priority levels as shown in Figure 5.5 (c).

5.6.4 Default Inference using Subjective Logic Operators

Now, bearing in mind the default inference mechanism defined on bilattice (see Equa-

tion 5.1), we examined subjective logic operators corresponding to ∧bl , ∨bl , ⊗bl and ⊕bl .

Concerning the semantic interpretation of ∧bl and ∨bl representing the standard logical

role of conjunction and disjunction, subjective logic conjunction (·sl ) and disjunction (⊔sl )

operators are examined. For the ⊗bl and ⊕bl operators representing consensus and gulli-

bility , subjective logic consensus ⊕sl and addition +sl operators are examined. Table 5.4

shows the examination results about the correspondence between each of semantically cor-

responding operator pairs. Interestingly, the interpretation of consensus operator ⊗bl does

not match to the consensus ⊕sl in subjective logic. Rather, consensus ⊕sl exhibits char-

acteristics corresponding to the gullibility operator (⊕bl ) in bilattice. The subjective logic

addition operator +sl showed completely different truth table compared to both the ⊕bl and

the ⊗bl . This is because the operator +sl is about adding any beliefs, subtracting any dis-

belief and averaging ignorance from both operands. Thereby, it tends to force rapid change

of belief towards truth direction. The operator ⊗bl seems not to have any corresponding

operator in subjective logic. In Table 5.4, the black points represent that those values are

identical to the ones in bilattice space. Contrary, the red points indicate that the values

are slightly different from the ones in bilattice space. This is due to the difference that the

bilattice operators are defined on a discrete set and the operators in subjective logic are

122
 Contents

Chapter 5
Table 5.4: Comparison of Operators used in cl(ϕ) shown in Equation (5.1) (Black - exact
match and Red - slight different match).

defined on continuous space. However, the semantics of the corresponding operators are

the same as it is explained below.

Definition 31. (Conjunction ·sl ) [97] . Let ΘX and ΘY be two frames and let x and

y be propositions about state in ΘX and ΘY respectively. Let wx = (bx , dx , ix , ax ) and

wy = (by , dy , iy , ay ) be an agent’s opinions about x and y, then conjunctive opinion denoted

as wx · wy is wx∧y = (bx∧y , dx∧y , ix∧y , ax∧y ) such that :



 (1−ax )ay bx iy +ax (1−ay )ix by

 bx∧y = bx by +


1−ax ay

 dx∧y = dx + dy − dx dy

 (1−ay )bx iy +(1−ax )ix by

 ix∧y = ix iy +


1−ax ay

 ax∧y = ax ay .

Definition 32. (Disjunction ⊔sl ) [97] . Let ΘX and ΘY be two frames and let x and

y be propositions about state in ΘX and ΘY respectively. Let wx = (bx , dx , ix , ax ) and

123
Contents

wy = (by , dy , iy , ay ) be an agent’s opinions about x and y, then disjunctive opinion denoted

as wx ⊔ wy is wx∨y = (bx∨y , dx∨y , ix∨y , ax∨y ) such that :



 bx∨y = bx + by − bx by





 dx∨y = dx dy + ax (1−ay )dx iy +(1−ax )ay ix dy
ax +ay −ax ay

 ay dx iy +ax ix dy

 ix∨y = ix iy + ax +ay −ax ay



 ax∨y = ax + ay − ax ay .

The truth functional table of logical conjunction (disjunction) in discrete space should

be a function that is closed to its discrete set of truth values. Therefore, considering the

interpretation of conjunction (disjunction), the binary operator ∧bl (∨bl ) should pick the

greatest upper bound (lowest upper bound) element of given two operands. This forces,

for example, T ∧bl DT (F ∨bl DF ) to be DT (DF ). However, if we were not restricted

by discrete set of values, as is the case in subjective logic, we would expect values in

between T and DT (F and DF ) in a sense that the conjunction (disjunction) operation

are interpreted as intersection (union) of both belief values. This aspect is mainly captured

by the definition of bx∧y (bx∨y ). The amount of ignorance is also captured by ix∧y (ix∨y )

so that it can consider both ignorance values of given two belief values. This aspect is

the main source where the differences of the truth table comes from in Table 5.4. Thus,

considering the semantics of conjunction and disjunction, subjective logic’s conjunction

and disjunction operators model the meaning of the operators under partial ignorance

correctly, but with an additional aspect that is only meaningful in a continuous space.

Similarly to conjunction and disjunction, the operator ⊕bl is also defined to pick a

value among the given discrete set of values. The selection is done in the sense that it

chooses any information that can be accepted upon both operands. In subjective logic, the

consensus operator ⊕sl sees each of operands as one that have observed continuous amount

of positive and negative evidence, thereby, summing up both observations into one opinion.

As discussed in the previous Section 5.6.3, this is similar to interpreting the semantics of

consensus from a bayesian perspective and it will increase the amount of information but

cannot be restricted to be a discrete value. This aspect is captured by the definition of iA,B
x

124
 Contents

in Definition 30, except in the case of dogmatic opinions having no ignorance. Therefore,

the meaning of ⊕bl is modeled also in the sense of partial order ≤k in bilattice, i.e., that

the derived value of given two operands should have more information (less ignorance in

subjective logic). Thus, ⊕sl operator in subjective logic models fusing uncertain beliefs in

a way that it increases the degree of information.

Based on the consideration about the semantics of operators shown above, we now

defined the truth assignment and closure operation for default reasoning using subjective

logic.

Definition 33. (Truth Assignmentsl ) . Given a declarative language L and Subjective

Opinion Space O, a truth assignment is a function ϕsl : L → O.

Definition 34. (Closuresl ) . Given a knowledge base K in form of declarative language

and a truth assignment labeling each sentence k ∈ K with a truth value and Subjective

Opinion Space O, then clsl (ϕ)(q), the closure ϕ of a given query sentence q, is the truth

Chapter 5
assignment function such that :

clsl (ϕ)(q) : {p, p′ |∀S, S ′ ⊆ K, S |= q, S ′ |= ¬q, p ∈ S, p′ ∈ S ′ } → O.

Definition 35. (Default Inferencesl ) . Given a query sentence q and given S and S ′

that are sets of sentences such that S |= q and S ′ |= ¬q, then the default inference is the

truth value assignment closure clsldi (ϕ)(q) given by :

⊕ ∏ ⊕ ∏
clsldi (ϕ)(q) = u⊔[ clsl (ϕ)(p)] ⊕ ¬ u⊔[ clsl (ϕ)(p)] . (5.7)
S|=q p∈S S ′ |=¬q p∈S ′

In fact, reminding the ‘Opinion Assignment’ of Definition 16 described in Sec-

tion 4.3.2, Definition 33 corresponds to the Definition 16 - 1 and 2. Similarly, Definition 34

and Definition 35 corresponds to the Definition 16 - 4. In this dissertation, we will use the

term ‘Opinion Assignment’ to emphasize the use of subjective opinion. In this chapter,

however, we will use the term ‘Truth Assignment’ instead of the term ‘Opinion Assign-

ment’ also to consider the use of other ‘uncertainty representation formalisms’ (such as

bilattice and L-fuzzy sets) and to compare with them.

125
Contents

5.7 Multivalued Default Logic, Square Bilattice and Default

Opinion

In this section, as it was done for subjective logic, we extend the discrete bilattice based

multivalued default logic to L-fuzzy logics in continuous space. We then describe some

properties of L-fuzzy logic representations on bilattice structure and review the possibility

on enabling default reasoning. In the preliminaries section, we have considered IVFL, IFL

and fuzzy Belnap logic in terms of truth value order (≤t ). However, as we have examined in

the previous section, the operators (especially, ⊕bl ) on degree of information (≤k ) play an

important role to model default reasoning aspect as to the case of discrete species bilattice

of multivalued default logic. There has been some work on representing IVFSs, IFSs and

Fuzzy Belnap using bilattice. Following is the definitions of ‘triangle’ and ‘square’ in the

context of bilattice. Arieli et al. introduced the following definitions of ‘square’ bilattice

for fuzzy-belnap logic and ‘triangle’ bilattice for IVFL [19].

Definition 36. (L2 Square bilattice) . Let L = (L, ≤L ) be a complete lattice. A

(bilattice-based) square is a structure L2 = (L × L, ≤t , ≤k , ¬), where ¬(x1 , x2 ) = (x2 , x1 ),

and

a) (x1 , x2 ) ≤t (y1 , y2 ) ⇔ x1 ≤L y1 and x2 ≥L y2 ,

b) (x1 , x2 ) ≤k (y1 , y2 ) ⇔ x1 ≤L y1 andx2 ≤L y2 .

When we set L = [0, 1], this captures the Atanassov’s idea on intuitionistic fuzzy sets

in the sense that it distinguishes between membership function µ and a non-membership ν,

but without imposing the restriction µ + ν ≤ 1. Denoting the join and meet operations of

the complete lattice L by ∧L and ∨L , the following four operators are defined for (x1 , x2 ),

(y1 , y2 ) in L2 ,

(x1 , x2 ) ∧ (y1 , y2 ) = (x1 ∧L y1 , x2 ∨L y2 ) (x1 , x2 ) ∨ (y1 , y2 ) = (x1 ∨L y1 , x2 ∧L y2 )

(5.8)
(x1 , x2 ) ⊗ (y1 , y2 ) = (x1 ∧L y1 , x2 ∧L y2 ) (x1 , x2 ) ⊕ (y1 , y2 ) = (x1 ∨L y1 , x2 ∨L y2 ) .

126
 Contents

Table 5.5: Some of t-norms and t-conorms.

The ∧L and ∨L can be defined by a t-norm and a t-conorm in the sense that they

generalize intersection and union in lattice space that can be seen as a metric space that

Chapter 5
satisfies triangle inequity. (note that, a t-norm is a function T : [0, 1] × [0, 1] → [0, 1] that

satisfies commutative, monotonicity, associative and one act as identity element. And the

same is for a t-conorm S : [0, 1] × [0, 1] → [0, 1] by replacing the last constraint with zero

identity constraint). Table 5.5 shows some of well known t-norms and t-conorms. Choosing

a pair of them to use for ∧L and ∨L on the lattice L , we can define meet ∧ and join ∨ for

partial order ≤t , and meet ⊗ and join ⊕ for partial order ≤k on the bilattice L2 according

to Equation 5.8 shown above. Therefore, considering the semantics of the Equation 5.1

(see Definition 29), we can directly apply the same inference scheme to bilattice based

fuzzy Belnap logic. Similarly to square bilattice for fuzzy Belnap logic, triangle for IVFSs

can be defined as follows.

Definition 37. (I(L) Triangle bilattice for IVFSs) . Let L = (L, ≤L ) be a complete

lattice. and I(L) = {[x1 , x2 ]|(x1 , x2 ) ∈ L2 , x1 ≤L x2 }. A (bilattice-based) triangle is a

structure I(L) = (I(L), ≤t , ≤k ), where

a) [x1 , x2 ] ≤t [y1 , y2 ] ⇔ x1 ≤L y1 and x2 ≤L y2 ,

127
Contents

b) [x1 , x2 ] ≤k [y1 , y2 ] ⇔ x1 ≤L y1 and x2 ≥L y2 .

Though the definitions of triangle bilattice for IFSs are not explicitly defined in

their work, we c-storean easily introduce the triangle structure for IFSs following the

Definition 37.

Definition 38. (I ∗ (L) Triangle bilattice for IFSs) . Let L = (L, ≤L ) be a complete

lattice. and I(L) = {[x1 , x2 ]|(x1 , x2 ) ∈ L2 , x1 + x2 ≤L 1L }. A (bilattice-based) triangle is

a structure I ∗ (L) = (I ∗ (L), ≤t , ≤k , ¬), where ¬(x1 , x2 ) = (x2 , x1 ), and

a) [x1 , x2 ] ≤t [y1 , y2 ] ⇔ x1 ≤L y1 and x2 ≥L y2 ,

b) [x1 , x2 ] ≤k [y1 , y2 ] ⇔ x1 ≤L y1 and x2 ≤L y2 .

As in the case of square bilattice, by setting L = [0, 1], the structure correspond to

LI and L∗ . However, Arieli et al. also showed that I(L) is in fact not a (pre-) bilattice,

since the substructure is not a lattice because the lub (least upper bound, join ∨k ) of any

two elements does not always exist [19]. This corresponds to the interesting aspect that,

the triangles do not allow explicit representation of the epistemic state ‘contradictory’ in

terms of degree of information (note that opinion triangle of subjective logic has the same

aspect). Therefore, for example, the full truth and full falsity in triangle are not comparable

in terms of degree of information. But still (I(L), ≤k ) is a partially ordered set, therefore

the triangle is very much in the same spirit as bilattices. This property is also same in

the case of I ∗ (L). They have also proved that t-norms and t-conorms for the ≤k -order

can’t be properly defined by introducing some theorems such as t-representability theorem,

etc. However, they showed that any t-norms or t-conorms definable in classical fuzzy set

theory have extensions to IVFSs along the partial order ≤t in a compositional manner.

Due to this aspect, it seems that bilattices are not always the key to model the adequate

properties of IVFL and IFL but is quite much adequate for modeling fuzzy Belnap logic.

Therefore, in the context of bilattice, the default inference scheme, Equation 5.8 can not

be set up on IVFL or IFL. While the meet ⊗ and join ⊕ operators can not be defined

on I(L) and I ∗ (L), however, there is an useful mapping between square L□ and triangle

128
 Contents

L∗ . In the work [22], Atanasov, the founder of intuitionistic fuzzy logic, further studied on

the relationship between the ‘triangle’ and the ‘square’, and defined following two bijective

transformations F and G from L□ to L∗ , defined for (x1 , x2 ) ∈ [0, 1]2 such that,

  (
 (0, 0)

 if x1 = x2 = 0  x1 − x2 , x2 ) if x1 ≥ x2

 ( 2 ) G(x1 , x2 ) = 2 2

F (x1 , x2 ) = x1
, xx11+x
x2
if x1 ≥ x2  ( x1 , x − x1 ) if x1 < x2 .
 x +x
 ( 1 2 2
)
2 2 2

 (5.10)
 x1 x2 x22
x1 +x2 , x1 +x2 if x1 < x2 .
(5.9)

Then later, Cornelis et al. showed that the bijective mapping does not preserve the

order, therefore not lattice isomorphism [44]. However, as for the triangle perspective inter-

pretation of values in square, it is still useful. Therefore, rather than directly model default

reasoning scheme for IFSs and IFSs, we do reasoning on square bilattice for fuzzy Belnap

logic, then transform the derived result using above Equations (4) (5). Definition 27, Def-

Chapter 5
inition 28 and Definition 29 hold on square L2 , and we will again distinguish the inference

on square, by denoting subscript L2 to the closure operation i.e. clL2 . For I(L) and
di (ϕ)(q)

I ∗ (L), we define following projection function relying on the above two possible mappings.

Definition 39. (clIF∗ (L) F-Interpretation) . Given a reasoning result clL2 on square
di (ϕ)(q)

L2 , the F-Interpretation is the function such that clIF∗ (L) = F (clL2 (ϕ)(q)), where the func-
di

tion F corresponds to Equation (5.9).

Definition 40. (clIG∗ (L) G-Interpretation) . Given a reasoning result clL2 (ϕ)(q) on
di

square L2 , the G-Interpretation is the function such that clIG∗ (L) = G(clL2 (ϕ)(q)), where
di

the function G corresponds to Equation (5.10).

Reminding that the IVFSs and IFSs are isomorphic [44], in this chapter, we will

show default reasoning on L2 and its F and G interpretations to IFSs. The interpreta-

tions can give us a shedding insight on comparing the reasoning result of the presented

subjective logic based approach with IFL and fuzzy Belnap logic, because the uncertainty

representation using µ and ν in IFL is pretty much similar to the one of subjective logic.

129
Contents

5.8 Case Study

This section deals with illustrative default reasoning examples for visual surveillance sce-

narios. To verify our approach and also to contrast with L-fuzzy logic based approaches,

we will reuse two examples demonstrated by Shet et al. [153, 154] and one scenario in typ-

ical airport scene that is also inspired by Shet et al. [152]. Then we compare the proposed

default inference approach Equation 5.7 to the one of bilattice based default reasoning and

its extension to L2 . The reasoning on L2 will be also interpreted in the view of L∗ (i.e.

I ∗ (L) psuedo-bilattice of L∗ ) with the interpretations clIF∗ (L) and clIG∗ (L) . In this section,

we will not directly concern about IVFL due to the isomorphism between IVFSs and IFSs.

We set truth values as follows :

T ≃ (1, 0, 0)sl ≃ (1, 0)L∗ ≃ (1, 0)L2 F ≃ (0, 1, 0)sl ≃ (0, 1)L∗ ≃ (0, 1)L2
DT ≃ (0.5, 0, 0.5)sl ≃ (0.5, 0)L∗ ≃ (0.5, 0)L2 DF ≃ (0, 0, 5, 0.5)sl ≃ (0, 0.5)L∗ ≃ (0, 0.5)L2
U ≃ (0, 0, 1)sl ≃ (0, 0)L∗ ≃ (0, 0)L2 ∗ ≃ (x, x, 1 − 2x)sl ≃ (x, x)L∗ ≃ (x, x)L2
i.e. undecided or contradiction with some amount of uncertainty (degree of information) on opinion

triangle (on L-fuzzy sets), and ⊥ ≃ (0.5, 0.5, 0)sl ≃ (0.5, 0.5)L∗ ≃ (1, 1)L2 (i.e. the full contradic-

tion).

These mappings are reasonable in the sense that the uncertainty representation of opinion

triangle and IFL I ∗ (L) are similar except the atomicity value of opinion triangle. For the

simplicity we assume that all the propositional knowledge we consider are balanced, there-

fore we set the atomicity of opinion triangle as default a = 0.5, and we will not explicitly

denote a. For the rest of truth values we will use opinion triple representation (b, d, i).

For values of opinion triangle, I ∗ (L) and L2 , that are slightly different to above symbols

but still can be interpreted as one of them, we will denote it with superscript ′ . (e.g.

∗ ≃ ∗′ , DT ≃ DT ′ , etc.).

Example 3. (Identity Inference) [153, 154] . Assume the following truth assignment
and set of rules about determining whether two individuals observed in an image should

130
 Contents

be considered as being one and the same.

ϕ[¬equal(P1 , P2 ) ← distinct(P1 , P2 )] = DT

ϕ[equal(P1 , P2 ) ← appear_similar(P1 , P2 )] = DT

ϕ[appear_similar(a, b)] = T

ϕ[distinct(a, b)] = T

Given two default true rules and facts that can be seen as definite true, the inference for

default logic shown in [153] with bilattice and the default inference with subjective logic

are as follows.

clbldi (ϕ)(equal(a, b))

= [U ∨ (T ∧ DT )] ⊕ ¬[U ∨ (T ∧ DT )] = [U ∨ DT ] ⊕ ¬[U ∨ DT ] = DT ⊕ DF = ∗

clsldi (ϕ)(equal(a, b))

= [U ⊔ (T • DT )] ⊕ ¬[U ⊔ (T • DT )] = [U ⊔ (0.67, 0, 0.33)] ⊕ ¬[U ⊔ (0.67, 0, 0.33)]

Chapter 5
= (0.67, 0, 0.33) ⊕ ¬(0.67, 0, 0.33) = (0.67, 0, 0.33) ⊕ (0, 0.67, 0.33) = (0.4, 0.4, 0.2) = ∗′

And as shown in Table 5.5, choosing one of t-norm and t-conorm pair, and applying

Equation 5.8 in Definition 36, we get following inference result derived on L2 , and its

interpretations clIF∗ (L) and clIG∗ (L) on I ∗ (L). The reasoning results are as follows.

min / max
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (T ∧ DT )] ⊕ ¬[U ∨ (T ∧ DT )] = [(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}] ⊕ ¬[(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}]

= [(0, 0) ∨ (min(1, 0.5), max(0, 0))] ⊕ ¬[(0, 0) ∨ (min(1, 0.5), max(0, 0))]

= [(0, 0) ∨ (0.5, 0)] ⊕ ¬[(0, 0) ∨ (0.5, 0)] = [max(0, 0.5), min(0, 0)] ⊕ ¬[max(0, 0.5), min(0, 0)]

= (0.5, 0) ⊕ ¬(0.5, 0) = (0.5, 0) ⊕ (0, 0.5) = (max(0.5, 0), max(0, 0.5)) = (0.5, 0.5) = ∗′
2
) = (0.25, 0.25) = ∗′
min / max 0.5 0.5·0.5
clIF∗ (L) (clL2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0.5, 0.5)) = ( 0.5+0.5 , 0.5+0.5
di

min / max
clIG∗ (L) (clL2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.5, 0.5)) = (0.5 − 0.5 0.5
2 , 2 ) = (0.25, 0.25) = ∗
di

prod/sum
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (T ∧ DT )] ⊕ ¬[U ∨ (T ∧ DT )] = [(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}] ⊕ ¬[(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}]

131
Contents

= [(0, 0) ∨ (1 · 0.5, 0 + 0 − 0 · 0)] ⊕ ¬[(1 · 0.5, 0 + 0 − 0 · 0)] = [(0, 0) ∨ (0.5, 0)] ⊕ ¬[(0, 0) ∨ (0.5, 0)]

= [0 + 0.5 − 0 · 0.5, 0 · 0] ⊕ ¬[0 + 0.5 − 0 · 0.5, 0 · 0] = (0.5, 0) ⊕ ¬(0.5, 0) = (0.5, 0) ⊕ (0, 0.5)

= (0.5 + 0 − 0.5 · 0, 0.5 + 0 − 0.5 · 0) = (0.5, 0.5) = ∗′

(ϕ)(equal(a, b))) = clIF∗ (L) ((0.5, 0.5)) = (0.25, 0.25) = ∗′

prod/sum
clIF∗ (L) (clL2
di

(ϕ)(equal(a, b))) = clIG∗ (L) ((0.5, 0.5)) = (0.25, 0.25) = ∗′

prod/sum
clIG∗ (L) (clL2
di

Luk
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (T ∧ DT )] ⊕ ¬[U ∨ (T ∧ DT )] = [(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}] ⊕ ¬[(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}]

= [(0, 0) ∨ (max(0, 1 + 0.5 − 1), min(0 + 0, 1))] ⊕ ¬[(0, 0) ∨ (max(0, 1 + 0.5 − 1), min(0 + 0, 1))]

= [(0, 0) ∨ (0.5, 0)] ⊕ ¬[(0, 0) ∨ (0.5, 0)] = [min(0 + 0.5, 1), max(0, 0 + 0 − 1)]

⊕ ¬[min(0 + 0.5, 1), max(0, 0 + 0 − 1)] = (0.5, 0) ⊕ ¬(0.5, 0) = (0.5, 0) ⊕ (0, 0.5)

= (min(0.5 + 0, 1), min(0 + 0.5, 1)) = (0.5, 0.5) = ∗′

′
clIF∗ (L) (clL2 (ϕ)(equal(a, b))) = clI ∗ (L) ((0.5, 0.5)) = (0.25, 0.25) = ∗
Luk F
di

clIG∗ (L) (clL

Luk
2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.5, 0.5)) = (0.25, 0.25) = ∗′
di

drastic
clL2 (ϕ)(equal(a, b))
di

= ([U ∨ (T ∧ DT )] ⊕ ¬[U ∨ (T ∧ DT )] = [(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}] ⊕ ¬[(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}]

= [(0, 0) ∨ (TD (1, 0.5), SD (0, 0))] ⊕ ¬[(0, 0) ∨ (TD (1, 0.5), SD (0, 0))] = [(0, 0) ∨ (0.5, 0)]

⊕ ¬[(0, 0) ∨ (0.5, 0)] = [SD (0, 0.5), TD (0, 0)] ⊕ ¬[SD (0, 0.5), TD (0, 0)]

= (0.5, 0) ⊕ ¬(0.5, 0) = (0.5, 0) ⊕ (0, 0.5) = (SD (0.5, 0), SD (0, 0.5)) = (0.5, 0.5) = ∗′
′
clIF∗ (L) (clL 2 (ϕ)(equal(a, b))) = clI ∗ (L) ((0.5, 0.5)) = (0.25, 0.25) = ∗
Luk F
di

′
clIG∗ (L) (clL 2 (ϕ)(equal(a, b))) = clI ∗ (L) ((0.5, 0.5)) = (0.25, 0.25) = ∗
Luk G
di

N ilpotent
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (T ∧ DT )] ⊕ ¬[U ∨ (T ∧ DT )] = [(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}] ⊕ ¬[(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}]

= [(0, 0) ∨ (TnM (1, 0.5), SnM (0, 0))] ⊕ ¬[(0, 0) ∨ (TnM (1, 0.5), SnM (0, 0))]

= [(0, 0) ∨ (0.5, 0)] ⊕ ¬[(0, 0) ∨ (0.5, 0)] = [SnM (0, 0.5), TnM (0, 0)] ⊕ ¬[SnM (0, 0.5), TnM (0, 0)]

= (0.5, 0) ⊕ ¬(0.5, 0) = (0.5, 0) ⊕ (0, 0.5) = (SnM (0.5, 0), SnM (0, 0.5)) = (0.5, 0.5) = ∗′
N ilpotent
clIF∗ (L) (clL 2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0.5, 0.5)) = (0.25, 0.25) = ∗′
di

N ilpotent
clIG∗ (L) (clL 2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.5, 0.5)) = (0.25, 0.25) = ∗′
di

132
 Contents

Figure 5.6: Reasoning Results of Example 3 in Opinion Space and L2 & I ∗ (L).

Chapter 5
Hamacher
clL 2 (ϕ)(equal(a, b))
di

= [U ∨ (T ∧ DT )] ⊕ ¬[U ∨ (T ∧ DT )] = [(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}] ⊕ ¬[(0, 0) ∨ {(1, 0) ∧ (0.5, 0)}]

= [(0, 0) ∨ (TH0 (1, 0.5), SH2 (0, 0))] ⊕ ¬[(0, 0) ∨ (TH0 (1, 0.5), SH2 (0, 0))]

= [(0, 0) ∨ (0.5, 0)] ⊕ ¬[(0, 0) ∨ (0.5, 0)] = [SH2 (0, 0.5), TH0 (0, 0)] ⊕ ¬[SH2 (0, 0.5), TH0 (0, 0)]

= (0.5, 0) ⊕ ¬(0.5, 0) = (0.5, 0) ⊕ (0, 0.5) = (SH2 (0.5, 0), SH2 (0, 0.5)) = (0.5, 0.5) = ∗′

clIF∗ (L) (clL

Hamacher
2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0.5, 0.5)) = (0.25, 0.25) = ∗′
di

clIG∗ (L) (clL

Hamacher
2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.5, 0.5)) = (0.25, 0.25) = ∗′
di

Figure 5.6 shows the graphical representation of above reasoning results. The resulting

opinion (0.4, 0.4, 0.2) represents same amount of degree of truth and false with uncertainty.

This can also be represented as undecided state ∗′ . All reasoning results on L2 also yielded

similar result (0.5, 0.5) and the same F/G interpretations (0.25, 0.25). Thus, the semantics

of results from the discrete bilattice for multivalued default logic, bilattice based L-fuzzy

logics and subjective logic are the same. While the uncertainty representation semantics

of subjective logic is similar to IFL, when the reasoning result on fuzzy Belnap logic is

133
Contents

interpreted, the distance between subjective opinion and the value of IFL was bigger than

the one of fuzzy Belnap.

Example 4. (Identity Inference with contextual cues) [153, 154] . Assume that
a person enters an office room that we believe to be empty and closed (no other exit).
Suppose that after a while, another person appears from the room who seems dissimilar
from the first person. In this case, inferring equality based on appearance matching is a
weaker default than inferring equality based on the fact that person entered and exited an
empty closed world. This aspect can be represented as following truth assignment and set
of rules.

ϕ[¬equal(P1 , P2 ) ← ¬appear_similar(P1 , P2 )] = DT1

ϕ[equal(P1 , P2 ) ← enter_closed_world(P1 , X, T1 ),

exit_closed_world(P2 , X, T2 ), T2 > T1 ,

empty_bef ore(X, T1 ), empty_af ter(X, T2 ),

not(enter_or_exit_between(P3 , T1 , T2 )).] = DT2

ϕ[¬appear_similar(a, b)] = T

ϕ[enter_closed_world(a, of f ice, 400)] = T

ϕ[exit_closed_world(b, of f ice, 523)] = T

ϕ[empty_bef ore(of f ice, 400)] = T

ϕ[empty_af ter(of f ice, 523)] = T

ϕ[¬(enter_or_exit_between(p3 , 400, 523)] = T

The Inference in this setup is multivalued default reasoning and the bilattice based infer-

ence result shown in Shet et al. [153] and the result of subjective logic based inference are

as follows.

clbldi (ϕ)(equal(a, b))

= [U ∨(T ∧T ∧T ∧T ∧T ∧DT2 )]⊕¬[U ∨(T ∧DT1 )] = [U ∧DT2 ]⊕¬[U ∧DT1 ] = DT2 ⊕DF1 = DT2

clsldi (ϕ)(equal(a, b))

134
 Contents

= [U ⊔ (T • T • T • T • T • DT2 )] ⊕ ¬[U ⊔ (T • DT1 )] = [U ⊔ (0.9, 0, 0.1)] ⊕ ¬[U ⊔ (0.67, 0, 0.33)]

= (0.9, 0, 0.1) ⊕ ¬(0.67, 0, 0.33) = (0.9, 0, 0.1) ⊕ (0, 0.67, 0.33) = (0.75, 0.17, 0.08) = DT2′

Choosing a pair of t-norm and t-conorm, inference result derived on L2 , and its interpre-

tations clIF∗ (L) and clIG∗ (L) on I ∗ (L) are as follows.

min / max
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )] = [U ∨ (T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )]

= [(0, 0) ∨ ((1, 0) ∧ (0.8, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.5, 0))] = [(0, 0) ∨ (min(1, 0.8), max(0, 0))]

⊕ ¬[(0, 0) ∨ (min(1, 0.5), max(0, 0))] = [(0, 0) ∨ (0.8, 0)] ⊕ ¬[(0, 0) ∨ (0.5, 0)]

= (max(0, 0.8), min(0, 0)) ⊕ ¬(max(0, 0.5), min(0, 0)) = (0.8, 0) ⊕ ¬(0.5, 0) = (0.8, 0) ⊕ (0, 0.5)

= (max(0.8, 0), max(0, 0.5)) = (0.8, 0.5) = DT2′

2
) = (0.49, 0.3) = DT2′
min / max 0.8 0.8·0.5
clIF∗ (L) (clL2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0.8, 0.5)) = ( 0.8+0.5 , 0.8+0.5
di

= (0.55, 0.25) = DT2′

min / max
clIG∗ (L) (clL2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.8, 0.5)) = (0.8 − 0.5 0.5
2 , 2 )
di

Chapter 5
prod/sum
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )] = [U ∨ (T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )]

= [(0, 0) ∨ ((1, 0) ∧ (0.8, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.5, 0))] = [(0, 0) ∨ (1 · 0.8, 0 + 0 − 0 · 0)]

⊕ ¬[(0, 0) ∨ (1 · 0.5, 0 + 0 − 0 · 0)] = [(0, 0) ∨ (0.8, 0)] ⊕ ¬[(0, 0) ∨ (0.5, 0)]

= (0 + 0.8 − 0 · 0.8, 0 · 0)) ⊕ ¬(0 + 0.5 − 0 · 0.5, 0 · 0) = (0.8, 0) ⊕ ¬(0.5, 0) = (0.8, 0) ⊕ (0, 0.5)

= (0.8 + 0 − 0.8 · 0, 0 + 0.5 − 0 · 0.5) = (0.8, 0.5) = DT2′

(ϕ)(equal(a, b))) = clIF∗ (L) ((0.8, 0.5)) = (0.49, 0.3) = DT2′

prod/sum
clIF∗ (L) (clL2
di

(ϕ)(equal(a, b))) = clIG∗ (L) ((0.8, 0.5)) = (0.55, 0.25) = DT2′

prod/sum
clIG∗ (L) (clL2
di

Luk
clL 2 (ϕ)(equal(a, b))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )] = [U ∨ (T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )]

= [(0, 0) ∨ ((1, 0) ∧ (0.8, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.5, 0))]

= [(0, 0) ∨ (max(0, 1 + 0.8 − 1), min(0 + 0, 1))] ⊕ ¬[(0, 0) ∨ (max(0, 1 + 0.5 − 1), min(0 + 0, 1))]

= [(0, 0) ∨ (0.8, 0)] ⊕ ¬[(0, 0) ∨ (0.5, 0)] = (min(0 + 0.8, 1), max(0, 0 + 0))

⊕ ¬(min(0 + 0.5, 1), max(0, 0 + 0)) = (0.8, 0) ⊕ ¬(0.5, 0) = (0.8, 0) ⊕ (0, 0.5)

135
Contents

= (min(0.8 + 0, 1), min(0 + 0.5, 1)) = (0.8, 0.5) = DT2′

′
clIF∗ (L) (clL
Luk F
2 (ϕ)(equal(a, b))) = clI ∗ (L) ((0.8, 0.5)) = (0.49, 0.3) = DT2
di

clIG∗ (L) (clL

Luk
2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.8, 0.5)) = (0.55, 0.25) = DT2′
di

drastic
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )] = [U ∨ (T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )]

= [(0, 0) ∨ ((1, 0) ∧ (0.8, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.5, 0))] = [(0, 0) ∨ (TD (1, 0.8), SD (0, 0))]

⊕ ¬[(0, 0) ∨ (TD (1, 0.5), SD (0, 0))] = [(0, 0) ∨ (0.8, 0)] ⊕ ¬[(0, 0) ∨ (0.5, 0)]

= (SD (0, 0.8), TD (0, 0)) ⊕ ¬(SD (0, 0.5), TD (0, 0))(0.8, 0) ⊕ ¬(0.5, 0) = (0.8, 0) ⊕ (0, 0.5)

= (SD (0.8, 0), SD (0, 0.5)) = (0.8, 0.5) = DT2′

clIF∗ (L) (clL

drastic
2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0.8, 0.5)) = (0.49, 0.3) = DT2′
di

clIG∗ (L) (clL

drastic
2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.8, 0.5)) = (0.55, 0.25) = DT2′
di

N ilpotent
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )] = [U ∨ (T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )]

= [(0, 0) ∨ ((1, 0) ∧ (0.8, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.5, 0))] = [(0, 0) ∨ (TnM (1, 0.8), SnM (0, 0))]

⊕ ¬[(0, 0) ∨ (TnM (1, 0.5), SnM (0, 0))] = [(0, 0) ∨ (0.8, 0)] ⊕ ¬[(0, 0) ∨ (0.5, 0)]

= (SnM (0, 0.8), TnM (0, 0)) ⊕ ¬(SnM (0, 0.5), TnM (0, 0)) = (0.8, 0) ⊕ ¬(0.5, 0) = (0.8, 0) ⊕ (0, 0.5)

= (SnM (0.8, 0), SnM (0, 0.5)) = (0.8, 0.5) = DT2′

N ilpotent
clIF∗ (L) (clL 2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0.8, 0.5)) = (0.49, 0.3) = DT2′
di

N ilpotent
clIG∗ (L) (clL 2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.8, 0.5)) = (0.55, 0.25) = DT2′
di

Hamacher
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )] = [U ∨ (T ∧ DT2 )] ⊕ ¬[U ∨ (T ∧ DT1 )]

= [(0, 0) ∨ ((1, 0) ∧ (0.8, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.5, 0))] = [(0, 0) ∨ (TH0 (1, 0.8), SH2 (0, 0))]

⊕ ¬[(0, 0) ∨ (TH0 (1, 0.5), SH2 (0, 0))] = [(0, 0) ∨ (0.8, 0)] ⊕ ¬[(0, 0) ∨ (0.5, 0)]

= (SH2 (0, 0.8), TH0 (0, 0)) ⊕ ¬(SH2 (0, 0.5), TH0 (0, 0)) = (0.8, 0) ⊕ ¬(0.5, 0)

= (0.8, 0) ⊕ (0, 0.5) = (SH2 (0.8, 0), SnM (0, 0.5)) = (0.8, 0.5) = DT2′

clIF∗ (L) (clL

Hamacher
2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0.8, 0.5)) = (0.49, 0.3) = DT2′
di

clIG∗ (L) (clL

Hamacher
2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.8, 0.5)) = (0.55, 0.25) = DT2′
di

136
 Contents

Figure 5.7: Reasoning Results of Example 4 in Opinion Space and L2 & I ∗ (L).

Chapter 5
Figure 5.7 shows above reasoning results. The opinion labeled (a) in opinion triangle is

the most closest one to DT2 compared with L-fuzzy logic based results but a bit biased

to center than original DT2 . In the sense of truth value, this is the same to the cases of

yielded values on L2 . However, in these cases, the amount of falsity is also relatively high

therefore the point is located in the area of overflowed information (µ + ν > 1). The F and

G interpretations are also a bit different but very close to each other. Nonetheless, in the

sense that all reasoning results are pointed on the right-hand side of the line of undecidable

*, semantically, this can be interpreted as the meaning of week truth like DT2 . Thus, the

semantics of results from the discrete bilattice for multivalued default logic, the bilattice

based L-fuzzy logics and subjective logic are the same.

137
Contents

Example 5. (Theft Inference with contextual cues) (The scenario and rules for
‘theft’ have been inspired by Shet et al. [152]) . Assume a typical airport surveillance as
depicted in Figure 5.8 with two cameras. Suppose that a human P1 carrying an object
(Baggage) B is observed and stayed around telephone booth in Cam1. After a while he
disappears from the view of Cam1 without taking his baggage B. Subsequently, P2 enters
the scene, picks up the baggage and leaves. In parallel, according to Cam2 , it seems that
P1 and P2 belong to a same group of people so the two people are considered as friends.
In this scenario, based on the possession relation between an object and person, we could
build a default rule to infer whether a person is a thief or not. Similarly, based on the
friend relation we can also build a bit stronger default rule saying possessing object of
friend is not thief. This aspect is depicted as following truth assignment and set of rules.

ϕ[thef t(P, B, T ) ← human(P ), package(B),

possess(P, B, T ), ¬(belongs(B, P, T ))] = DT1

ϕ[¬thef t(P, B, T ) ← human(P ), package(B),

possess(P, B, T ), belongs(B, P, T )] = DT1

ϕ[¬thef t(P, B, T ) ← human(P ), package(B),

possess(P, B, T ), ¬(belongs(B, P, T )),

f riend(P, P ′ ), belongs(B, P ′ , T )] = DT2

ϕ[human(P )] = T

ϕ[package(B)] = T

ϕ[possess(P, B, T )] = T

ϕ[¬(belongs(B, P, T ))] = T

ϕ[f riend(P, P ′ )]cam2 = T

ϕ[belongs(B, P ′ , T )]cam2 = T

where, DT1 ≃ (0.5, 0, 0.5) and DT2 ≃ (0.8, 0, 0.2)

Given above rules and facts (gathered till P2 is picking up the baggage B), inferring

whether the person is a thief or not is shown as follows.

138
 Contents

Figure 5.8: Illustrative Scenario Setup of Example 5.

Case1. (Inference relying only on Cam1).

clbldi (ϕ)(thef t(P, B, T )) = [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] = [U ∨ DT1 ] = DT1

clsldi (ϕ)(thef t(P, B, T )) = [U ⊔ (T • T • T • T • DT1 )] = [U ⊔ (0.74, 0, 0.26)] = (0.74, 0, 0.26) = DT2′

Choosing a pair of t-norm and t-conorm, inference result derived on L2 , and its interpre-

Chapter 5
tations clIF∗ (L) and clIG∗ (L) on I ∗ (L) are as follows.

min / max
clL2 (ϕ)(thef t(P, B, T ))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] = [U ∨ (T ∧ DT1 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))]

= [(0, 0) ∨ (min(1, 0.5), max(0, 0))] = [(0, 0) ∨ (0.5, 0)] = (max(0, 0.5), min(0, 0))

= (0.5, 0) = DT1
min / max 2
clIF∗ (L) (clL2 (ϕ)(thef t(P, B, T ))) = clIF∗ (L) ((0.5, 0)) = ( 0.5 0.5·0
0.5 , 0.5+0 ) = (0.5, 0) = DT1
di

min / max
clIG∗ (L) (clL2 (ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0)) = (0.5 − 02 , 02 ) = (0.5, 0) = DT1
di

prod/sum
clL2 (ϕ)(thef t(P, B, T ))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] = [U ∨ (T ∧ DT1 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))]

= [(0, 0) ∨ (1 · 0.5, 0 + 0 − 0 · 0)] = [(0, 0) ∨ (0.5, 0)] = (0 + 0.5 − 0 · 0.5), 0 · 0) = (0.5, 0) = DT1
prod/sum
clIF∗ (L) (clL2 (ϕ)(thef t(P, B, T ))) = clIF∗ (L) ((0.5, 0)) = (0.5, 0) = DT1
di

prod/sum
clIG∗ (L) (clL2 (ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0)) = (0.5, 0) = DT1
di

Luk
clL 2 (ϕ)(thef t(P, B, T ))
di

139
Contents

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] = [U ∨ (T ∧ DT1 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))]

= [(0, 0) ∨ (max(0, 1 + 0.5 − 1), min(0 + 0, 1))] = [(0, 0) ∨ (0.5, 0)]

= (min(0 + 0.5, 1), max(0, 0 + 0 − 1)) = (0.5, 0) = DT1

clIF∗ (L) (clL

Luk F
2 (ϕ)(thef t(P, B, T ))) = clI ∗ (L) ((0.5, 0)) = (0.5, 0) = DT1
di

clIG∗ (L) (clL

Luk
2 (ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0)) = (0.5, 0) = DT1
di

drastic
clL2 (ϕ)(thef t(P, B, T ))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] = [U ∨ (T ∧ DT1 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))]

= [(0, 0) ∨ (TD (1, 0.5), SD (0.5, 0))] = [(0, 0) ∨ (0.5, 0)] = (SD (0, 0.5), TD (0, 0)) = (0.5, 0) = DT1

clIF∗ (L) (clL

drastic
2 (ϕ)(thef t(P, B, T ))) = clIF∗ (L) ((0.5, 0)) = (0.5, 0) = DT1
di

clIG∗ (L) (clL

drastic
2 (ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0)) = (0.5, 0) = DT1
di

N ilpotent
clL2 (ϕ)(thef t(P, B, T ))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] = [U ∨ (T ∧ DT1 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))]

= [(0, 0) ∨ (TnM (1, 0.5), SnM (0.5, 0))] = [(0, 0) ∨ (0.5, 0)] = (SnM (0, 0.5), TnM (0, 0))

= (0.5, 0) = DT1
N ilpotent
clIF∗ (L) (clL2 (ϕ)(thef t(P, B, T ))) = clIF∗ (L) ((0.5, 0)) = (0.5, 0) = DT1
di

N ilpotent
clIG∗ (L) (clL2 (ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0)) = (0.5, 0) = DT1
di

Hamacher
clL2 (ϕ)(thef t(P, B, T ))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] = [U ∨ (T ∧ DT1 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))]

= [(0, 0) ∨ (TH0 (1, 0.5), SH2 (0.5, 0))] = [(0, 0) ∨ (0.5, 0)] = (SH2 (0, 0.5), TH0 (0, 0))

= (0.5, 0) = DT1

clIF∗ (L) (clL

Hamacher
2 (ϕ)(thef t(P, B, T ))) = clIF∗ (L) ((0.5, 0)) = (0.5, 0) = DT1
di

clIG∗ (L) (clL

Hamacher
2 (ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0)) = (0.5, 0) = DT1
di

Relying only the facts generated by Cam1 (those are not explicitly subscripted with cam-

era id), all of above inference concluded cl(ϕ)(thef t(P2 , B, T )) = DT1 except subjective
′
logic based approach that derived DT2 . However, the semantic interpretation of the results

are all the same. Namely. theft has taken place with low confidence.

140
 Contents

Case2. (Inference relying on Cam1 and Cam2)

clbldi (ϕ)(thef t(P, B, T ))

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )] = [U ∨ DT1 ] ⊕ ¬[U ∨ DT2 ]

= DT1 ⊕ ¬DT2 = DT1 ⊕ DF2 = DF2

clsldi (ϕ)(thef t(P, B, T ))

= [U ⊔ (T • T • T • T • DT1 )] ⊕ ¬[U ⊔ (T • T • T • T • T • T • DT2 )]

= [U ⊔ (0.74, 0, 0.26)] ⊕ ¬[U ⊔ (0.9, 0, 0.1)] = (0.74, 0, 0.26) ⊕ ¬(0.9, 0, 0.1)

= (0.74, 0, 0.26) ⊕ (0, 0.9, 0.1) = (0.22, 0.7, 0.08) = DF2′

Choosing a pair of t-norm and t-conorm, inference result derived on L2 , and its interpre-

tations clIF∗ (L) and clIG∗ (L) on I ∗ (L) are as follows.

min / max
clL2 (ϕ)(thef t(P, B, T ))
di

= U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )]

= [U ∨ (T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ DT2 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.8, 0))]

Chapter 5
= [(0, 0) ∨ (min(1, 0.5), max(0, 0))] ⊕ ¬[(0, 0) ∨ (min(1, 0.8), max(0, 0))]

= [(0, 0) ∨ (0.5, 0)] ⊕ ¬[(0, 0) ∨ (0.8, 0)] = (max(0, 0.5), min(0, 0)) ⊕ ¬(max(0, 0.8), min(0, 0))

= (0.5, 0) ⊕ ¬(0.8, 0) = (0.5, 0) ⊕ (0, 0.8) = (max(0.5, 0), max(0, 0.8)) = (0.5, 0.8) = DF2′
2
) = (0.3, 0.49) = DF1′
min / max 0.5·0.8 0.8
clIF∗ (L) (clL2 (ϕ)(thef t(P, B, T ))) = clIF∗ (L) ((0.5, 0.8)) = ( 0.5+0.8 , 0.5+0.8
di

= (0.25, 0.55) = DF1′

min / max
2 , 0.8 −
(ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0.8)) = ( 0.5 0.5
clIG∗ (L) (clL2 2 )
di

prod/sum
clL2 (ϕ)(thef t(P, B, T ))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )]

= [U ∨ (T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ DT2 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.8, 0))]

= [(0, 0)∨(1·0.5, 0+0−0·0)]⊕¬[(0, 0)∨(1·0.8, 0+0−0·0)] = [(0, 0)∨(0.5, 0)]⊕¬[(0, 0)∨(0.8, 0)]

= (0 + 0.5 − 0 · 0.5, 0 · 0) ⊕ ¬(0 + 0.8 − 0 · 0.8, 0 · 0) = (0.5, 0) ⊕ ¬(0.8, 0) = (0.5, 0) ⊕ (0, 0.8)

= (0.5 + 0 − 0.5 · 0, 0 + 0.8 − 0 · 0.8) = (0.5, 0.8) = DF2′

(ϕ)(thef t(P, B, T ))) = clIF∗ (L) ((0.5, 0.8)) = (0.3, 0.49) = DT2′
prod/sum
clIF∗ (L) (clL2
di

(ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0.8)) = (0.25, 0.55) = DT2′
prod/sum
clIG∗ (L) (clL2
di

141
Contents

Luk
clL2 (ϕ)(thef t(P, B, T ))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )]

= [U ∨ (T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ DT2 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.8, 0))]

= [(0, 0) ∨ (max(0, 1 + 0.5 − 1), min(0 + 0, 1)] ⊕ ¬[(0, 0) ∨ (max(0, 1 + 0.8 − 1), min(0 + 0, 1)]

= [(0, 0) ∨ (0.5, 0)] ⊕ ¬[(0, 0) ∨ (0.8, 0)]

= (min(0 + 0.5, 1), max(0, 0 + 0 − 1)) ⊕ ¬(min(0 + 0.8, 1), max(0, 0 + 0 − 1))

= (0.5, 0) ⊕ ¬(0.8, 0) = (0.5, 0) ⊕ (0, 0.8) = (min(0 + 0.5, 1), min(0 + 0.8, 1)) = (0.5, 0.8) = DF2′
′
clIF∗ (L) (clL
Luk F
2 (ϕ)(thef t(P, B, T ))) = clI ∗ (L) ((0.5, 0.8)) = (0.3, 0.49) = DT2
di

clIG∗ (L) (clL

Luk
2 (ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0.8)) = (0.25, 0.55) = DT2′
di

drastic
clL2 (ϕ)(thef t(P, B, T ))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )]

= [U ∨ (T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ DT2 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.8, 0))]

= [(0, 0) ∨ (TD (1, 0.5), SD (0, 0)] ⊕ ¬[(0, 0) ∨ (TD (1, 0.8), SD (0, 0)]

= [(0, 0) ∨ (0.5, 0)] ⊕ ¬[(0, 0) ∨ (0.8, 0)] = (SD (0, 0.5), TD (0, 0)) ⊕ ¬(SD (0, 0.8), TD (0, 0))

= (0.5, 0) ⊕ ¬(0.8, 0) = (0.5, 0) ⊕ (0, 0.8) = (SD (0, 0.5), SD (0, 0.8)) = (0.5, 0.8) = DF2′

clIF∗ (L) (clL

drastic
2 (ϕ)(thef t(P, B, T ))) = clIF∗ (L) ((0.5, 0.8)) = (0.3, 0.49) = DT2′
di

clIG∗ (L) (clL

drastic
2 (ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0.8)) = (0.25, 0.55) = DT2′
di

N ilpotent
clL2 (ϕ)(thef t(P, B, T ))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )]

= [U ∨ (T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ DT2 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.8, 0))]

= [(0, 0) ∨ (TnM (1, 0.5), SnM (0, 0)] ⊕ ¬[(0, 0) ∨ (TnM (1, 0.8), SnM (0, 0)]

= [(0, 0) ∨ (0.5, 0)] ⊕ ¬[(0, 0) ∨ (0.8, 0)] = (SnM (0, 0.5), TnM (0, 0)) ⊕ ¬(SnM (0, 0.8), TnM (0, 0))

= (0.5, 0) ⊕ ¬(0.8, 0) = (0.5, 0) ⊕ (0, 0.8) = (SnM (0, 0.5), SnM (0, 0.8)) = (0.5, 0.8) = DF2′
N ilpotent
clIF∗ (L) (clL 2 (ϕ)(thef t(P, B, T ))) = clIF∗ (L) ((0.5, 0.8)) = (0.3, 0.49) = DT2′
di

N ilpotent
clIG∗ (L) (clL 2 (ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0.8)) = (0.25, 0.55) = DT2′
di

Hamacher
clL2 (ϕ)(thef t(P, B, T ))
di

= [U ∨ (T ∧ T ∧ T ∧ T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ T ∧ T ∧ T ∧ T ∧ T ∧ DT2 )]

142
 Contents

Figure 5.9: Reasoning Results of Example 5 in Opinion Space and L2 & I ∗ (L).

Chapter 5
= [U ∨ (T ∧ DT1 )] ⊕ ¬[U ∨ (T ∧ DT2 )] = [(0, 0) ∨ ((1, 0) ∧ (0.5, 0))] ⊕ ¬[(0, 0) ∨ ((1, 0) ∧ (0.8, 0))]

= [(0, 0) ∨ (TH0 (1, 0.5), SH2 (0, 0)] ⊕ ¬[(0, 0) ∨ (TH0 (1, 0.8), SH2 (0, 0)] = [(0, 0) ∨ (0.5, 0)]

⊕ ¬[(0, 0) ∨ (0.8, 0)] = (SH2 (0, 0.5), TH0 (0, 0)) ⊕ ¬(SH2 (0, 0.8), TH0 (0, 0)) = (0.5, 0) ⊕ ¬(0.8, 0)

= (0.5, 0) ⊕ (0, 0.8) = (SH2 (0, 0.5), SH2 (0, 0.8)) = (0.5, 0.8) = DF2′

clIF∗ (L) (clL

Hamacher
2 (ϕ)(thef t(P, B, T ))) = clIF∗ (L) ((0.5, 0.8)) = (0.3, 0.49) = DT2′
di

clIG∗ (L) (clL

Hamacher
2 (ϕ)(thef t(P, B, T ))) = clIG∗ (L) ((0.5, 0.8)) = (0.25, 0.55) = DT2′
di

Figure 5.9 shows above reasoning results. In case of Case1, subjective logic yielded a

rather strong opinion (0.74, 0, 0.26) that is rather close to DT2 than DT1 that other ap-

proaches yielded. Contrary to Case 1, when we take more information also from Cam2,

all of above inference concluded cl(ϕ)(thef t(P2 , B, T )) = DF2 and DF2′ . Namely, no theft

has taken place with rather high confidence. The opinion DF2′ (0.22, 0.7, 0.08) in opinion

triangle is the most closest one to DF2 compared to L-fuzzy logic based results. In the

sense of truth value, this is the same to the cases of yielded values on L2 . However, in

this case, the amount of falsity is also relatively high therefore the point is located in the

143
Contents

area of overflowed information (µ + ν > 1). The F and G interpretations are both close to

each other however the interpretation is rather closer to DF1 . Nonetheless, in the sense

that all reasoning results are pointed on the left-hand side of the line of undecidable *,

semantically, this can be interpreted as the meaning of week false like DF2 . Thus, the

semantics of results from the discrete bilattice for multivalued default logic, the bilattice

based L-fuzzy logics and subjective logic are the same.

As shown with above illustrative visual surveillance inference scenarios, the proposed

default reasoning mechanism Equation 5.7 semantically well models default reasoning and

that is so in the case of L-fuzzy logics.

5.9 Discussion

As shown in the previous section with examples of default reasoning in visual surveillance,

both subjective logic and L-fuzzy logics (especially, IFL and fuzzy Belnap logic) seem rel-

evant for the use of approximate default reasoning. What makes IFSs attractive compared

to other fuzzy set extensions is that it makes geometrical interpretations possible, thereby,

the combination of membership and non-membership functions can be calculated in the

Euclidean plane with a Cartesian coordinate system. [23]. This aspect is also the same

in Subjective logic because subjective logic also makes the geometrical interpretations of

Beta probability distributions possible in the Euclidean plane. However, there are still

some properties worth to discuss to contrast these approaches. In this section, we give

a comparison on the property of both approaches in the view of logical soundness and

feasibility in visual surveillance.

5.9.1 Which way to go? Subjective Logic vs. L-fuzzy Logics

As we noted in the previous default reasoning section, the initial idea of discrete bilattice

assumed an epistemic state called ‘unknown’ and ‘contradiction’ (defined to have even more

information than definite true or false) following Belnaps four-valued logic. In the sense

144
 Contents

of discrete valued logic such as multivalued default logic, the only way of reaching to the

overflowed information state is through the handling of definite true and definite false, and

‘contradiction’ ⊥ is the only epistemic state that is defined in the overflowed information

area. However, as shown in the previous examples, this is not true on L2 , because we can

encounter values in the area of (µ + ν > 1). Similarly to the discussion we have reviewed in

Section 5.6.2, regarding the meaning of the area, and more specifically the epistemic state

‘contradiction’, there has been many discussions on the significance of the epistemic state

in the view of logic. A critical review can be found in a work of Urquhart [166]. There has

been also some report on the possible problems that could be introduced for the formal

specification of software systems such as non-termination error in a work of Hähnle [77].

Dubois, D. [56] formally showed the problems that can arise in Belnap logic, in the sense

of logical soundness. In the following, we briefly review the discussions introduced by [56].

Discussion 1. (Paradoxes in the truth table of multivalued logics) . Table 5.1

Chapter 5
(a) and (b) show the truth table of the Belnap’s four-valued logic and the default logic.

In both logics, the conjunction and disjunction are problematic when applied to the two

extreme epistemic states ‘Unknown’ U and ‘Contradiction’ ⊥. For instance in Belnap logic,

we have U ∧ U = U and U ∨ U = U . Assume that we attached U to a proposition p and

consider p ∧ ¬p and p ∨ ¬p. In Belnap logic, the former and latter both are U because

U ∧ ¬U = U ∧ U = U and U ∨ ¬U = U ∨ U = U . However, in classical logic sense, the

former should be false and the latter should be true. The same anomaly is also introduced

in the case of ⊥. According to the truth table, it claims ⊥ ∧ ⊥ = ⊥ and ⊥ ∨ ⊥ = ⊥. For

p ∧ ¬q and p ∨ ¬q, again, we get ⊥ for both. It breaks the tautology in classical logic sense

[56]. Similar anomaly can be found in the case of U ∧ ⊥ = F and U ∨ ⊥ = T .

From a common sense, the results are counterintuitive and this was even to Belnap

because he stated that this is an unavoidable consequence of his formal setting [56]. This

aspect can be problematic in the following scenario. Consider a proposition p and q, and

agent A1 and A2 saying the p is T and F respectively and this is why p is ⊥. Because A1

and A2 say nothing about q we assume q is U . Now, p ∧ q = ⊥ ∧ U = F that is again

145
Contents

counter intuitive [56]. This aspect also leads to debatable epistemic value assignments.

Suppose two atomic propositions p and q with epistemic state assignment ϕ(p) = ⊥ and

ϕ(q) = U . Then ϕ(p ∧ q) = F as noted above. But since Belnap negation is such that

ϕ(¬p) = ⊥ and ϕ(¬q) = U , we also get ϕ(¬p ∧ q) = ϕ(p ∧ ¬q) = ϕ(¬p ∧ ¬q) = F . Hence,

ϕ((p∧q)∧(¬p∧q)∧(p∧¬q)∧(¬p∧¬q)) = ϕ(p∧q)∧ϕ(¬p∧q)∧ϕ(p∧¬q)∧ϕ(¬p∧¬q) = F ,

however, according to the truth table, (p∧q)∧(¬p∧q)∧(p∧¬q)∧(¬p∧¬q) = F ∧F ∧T ∧T =

⊥. This means ϕ(⊥) = F , therefore hardly acceptable again [56].

This aspect shows that, for any logical connectives ∗, ϕ(p) ∗ ϕ(q) ̸= ϕ(p ∗ q) in Belnap

logic, namely, Belnap logic is ‘non-commutative’. In other words, an epistemic value on

each proposition can not characterize a single epistemic value for the combination of the

propositions. This aspect also hold in the fuzzy Belnap logic as well, because regardless

what t-norms and t-conorms we choose, the truth table values corresponding to definite

true, definite false, unknown and contradictory values will have the same truth functional

values as of discrete Belnap logic.

Unlike fuzzy Belnap logic, in subjective logic we can avoid this problem by the use

of atomicity value a, therefore subjective logic better captures the spirit of classical logic2 .

Consider the same case of U ∧ U and U ∨ U . As shown in Figure 5.10 (a), for subjective

logic opinion w = (0, 0, 1, 0.5) which corresponds to U , subjective logic conjunction also

draws full ignorance but with different atomicity, namely (0, 0, 1, 0.5) ∧ ¬(0, 0, 1, 0.5) =

(0, 0, 1, 0.25). The semantics is clear. Namely, for a proposition that is known to be binary

event that an agent has a full ignorance on the truth of it, the conjunction also draws

full uncertainty but, following the spirit of probabilistic conjunction, it comes with the

atomicity that is the product of both atomicity (i.e. 0.5 · 0.5 in this case). Therefore, even

if we get full ignorance, when it is interpreted in terms of Beta distribution, the overall

expectation should be biased to falsity as traditional logic yields F . This is the same in

the case of U ∧ U that yields a full ignorance opinion but its atomicity is biased to T . The

2
Strictly speaking, subjective logic does not allow a ‘truly full uncertain’ state, in the sense that subjective
logic adds additional dimension of information called atomicity (prior) value. This is the main cause
of the difference shown in Figure 5.10 (a).

146
 Contents

Chapter 5
Figure 5.10: (a) U ∧ U and U ∨ U (b) ⊥ ∧ ⊥ and ⊥ ∨ ⊥ in Subjective Logic.

similar aspect holds in the case of ⊥ ∧ ⊥ = ⊥ and ⊥ ∨ ⊥ = ⊥ 3 . For p ∧ q and p ∨ ¬q,

classical logic should draw T and F . As shown in Figure 5.10 (b), considering the epistemic

state of contradiction as w = (0.5, 0.5, 0, 0.5), we get (0.5, 0.5, 0, 0.5) ∧ ¬(0.5, 0.5, 0, 0.5) =

(0.25, 0.75, 0, 0.25) that is biased to disbelief. Note that, both in Figure 5.10 (a) and (b)

we have the same probability expectation values. However, when represented as Beta

3
This is due to the main difference that subjective logic offers the Bayesian view on interpreting contra-
diction as discussed in Section 5.6.2. Strictly speaking, Belnap like logical view of contradiction can be
also explicitly distinguished using the ‘conflict parameter’ in subjective logic (see Section 5.6.2), even
though the use of the parameter does not affect to subsequent calculation and therefore we did not
explicitly use the parameter in this dissertation. However, for some applications in which it is very
important to know and to distinguish the occurrence of the logical view of conflict, so to regard it
as unhealthy system status and therefore, for example to halt the system, we could explicitly use the
‘conflict’ parameter.

147
Contents

Figure 5.11: (a) Two Examples of U ∧ ⊥ (b) Two Examples of U ∨ ⊥ in Subjective Logic.

distribution, while (b) is almost certain because we have rather pick distribution (b) is

almost uncertain. This aspect is directly captured in the opinion triangle by the value of

ignorance. Now, for the counter intuitive cases of U ∧ ⊥ = F and U ∨ ⊥ = T in Belnap

logic, subjective logic draws a bit different epistemic states. Figure 5.11 (a) depicts two

cases of U ∧ ⊥ one more biased to F and the other more biased to ⊥. The basic idea is

that we take more atomicity in the case of unknown opinion. The same aspect is captured

in the case of U ∨ ⊥ as shown in Figure 5.11 (b). Figure 5.11 (a) more intuitively explains

the above mentioned agent scenario with two propositions p and q. Again, consider the

truth value on proposition p to be ⊥ because agent A1 and A2 saying the p is T and F .

For the proposition q, we assign U because agent A1 and A2 say nothing about q. Now,

148
 Contents

p ∧ q can be determined differently as shown in Figure 5.11 (a).

Finally, the last aspect on whether commutative computation is possible or not

in Belnap logic, has no problem in subjective logic [98, 97, 94]. Suppose two atomic

propositions p and q with epistemic state assignment ϕ(p) = ⊥ = (0.5, 0.5, 0, 0.5) and

ϕ(q) = U = (0, 0, 1, 0.5). Then, ϕ(p ∧ q) = (0.17, 0.5, 0.33, 0.25). The negation in

this case is ϕ(¬p) = ⊥ = (0.5, 0.5, 0, 0.5) and ϕ(¬q) = U = (0, 0, 1, 0.5), we also get

ϕ(¬p ∧ q) = ϕ(p ∧ ¬q) = ϕ(¬ ∧ ¬q) = (0.17, 0.5, 0.33, 0.25). Hence,

ϕ((p ∧ q) ∧ (¬p ∧ q) ∧ (p ∧ ¬q) ∧ (¬p ∧ ¬q))

= ϕ(p ∧ q) ∧ ϕ(¬p ∧ q) ∧ ϕ(p ∧ ¬q) ∧ ϕ(¬p ∧ ¬q)

= (0.17, 0.5, 0.33, 0.25) ∧ (0.17, 0.5, 0.33, 0.25) ∧ (0.17, 0.5, 0.33, 0.25) ∧ (0.17, 0.5, 0.33, 0.25)

= (0.07, 0.75, 0.18, 0.07) ∧ (0.17, 0.5, 0.33, 0.25) ∧ (0.17, 0.5, 0.33, 0.25)

= (0.03, 0.88, 0.1, 0.02) ∧ (0.17, 0.5, 0.33, 0.25) = (0.01, 0.94, 0.05, 0.01)

Chapter 5
In subjective logic, regardless of the order how we calculate opinions, we get the same

result as follows.

ϕ((p ∧ q) ∧ (¬p ∧ q) ∧ (p ∧ ¬q) ∧ (¬p ∧ ¬q))

= ϕ((0.17, 0.5, 0.33, 0.25) ∧ (0.5, 0.5, 0, 0.5) ∧ q ∧ p ∧ ¬q ∧ ¬p ∧ ¬q)

= ϕ((0.11, 0.75, 0.14, 0.13) ∧ (0, 0, 1, 0.5) ∧ p ∧ ¬q ∧ ¬p ∧ ¬q)

= ϕ((0.05, 0.75, 0.2, 0.07) ∧ (0.5, 0.5, 0, 0.5) ∧ ¬q ∧ ¬p ∧ ¬q)

= ϕ((0.03, 0.88, 0.10, 0.04) ∧ (0, 0, 1, 0.5) ∧ ¬p ∧ ¬q)

= ϕ((0.01, 0.88, 0.11, 0.02) ∧ (0.5, 0.5, 0, 0.5) ∧ ¬q)

= ϕ((0.01, 0.94, 0.05, 0.01) ∧ (0, 0, 1, 0.5)) = ϕ(0, 01, 0.94, 0.05, 0.01) = (0, 01, 0.94, 0.05, 0.01)

We believe, above aspect makes subjective logic more solid and sound logic formal-

ism under uncertainty. Especially, compared to fuzzy Belnap logic, the operational order

does not affect on the final result. This is an important aspect, because in fuzzy-Belnap

logic, once we reach at the contradictory point, there is no easy way to escape from the

state unless we use the meet operator ⊗ along the partial order ≤k . Namely, Belnap

149
Contents

logic is non-associative. Therefore, in fuzzy Belnap logic, the sequence of information ar-

rival is important, however, so is not in subjective logic because subjective logic supports

associativity [98, 97, 94]).

Discussion 2. (Feasibility in visual surveillance) . In the previous section for the

examples, we assigned definite true T and definite false F for the logical facts. In practice,
however, such symbolic logical facts are generated from the vision analytics. Because the
vision analytics tend to rely on machine learning and pattern recognition techniques, in
general, the values will be also noisy. Indeed, in practice, it would be more realistic to
attach arbitrary amount of beliefs even to the logical rules rather than values such as
DT, DF, DT1 , DF1 , etc. In the previous examples, the L-fuzzy logic based approaches
generated the same result regardless how we choose t-norms and t-conorms. Therefore, in
this discussion, we will examine how the uncertainty introduced on facts and rules, and
how the choices on t-norms and t-conorms could affect the reasoning result. Consider
Example 3 in the previous section with slightly different settings as follows.

ϕ[¬equal(P1 , P2 ) ← distinct(P1 , P2 )] = (0.5, 0.1)L2 = (0.5, 0.1, 0.4)sl = r1

ϕ[equal(P1 , P2 ) ← appear_similar(P1 , P2 )] = (0.5, 0.1)L2 = (0.5, 0.1, 0.4)sl = r2

ϕ[appear_similar(a, b)] = (0.6, 0.3)L2 = (0.6, 0.3, 0.1)sl = f1

ϕ[distinct(a, b)] = (0.3, 0.4)L2 = (0.3, 0.4, 0.3)sl = f2

In above setting, given two rules that are considered with the same amount of significance,

we attach more strong belief to f1 . Therefore, the expected result is that the two persons

maybe the same one but not quite certainly. Applying the same inference mechanism

Equation 5.1 for L-fuzzy logics and (2) for subjective logic, the inference results are as

follows. (note that, above setting is not applicable to the case of discrete bilattice species

for multivalued default logic.).

clsldi (ϕ)(equal(a, b))

= [U ⊔ (f1 • r2 )] ⊕ ¬[U ⊔ (f2 • r1 )]

= [U ⊔ (0.6, 0.3, 0.1) • (0.5, 0.1, 0.4)] ⊕ ¬[U ⊔ (0.3, 0.4, 0.3) • (0.5, 0.1, 0.4)]

150
 Contents

= [(0, 0, 1) ⊔ (0.4, 0.37, 0.23)] ⊕ ¬[(0, 0, 1) ⊔ (0.24, 0.46, 0.3)]

= (0.4, 0.07, 0.53) ⊕ ¬(0.24, 0.09, 0.67) = (0.4, 0.07, 0.53) ⊕ (0.09, 0.24, 0.67) = (0.37, 0.21, 0.42)

As shown in Table 5.5, choosing one of t-norm and t-conorm pair and applying Equation 5.8

in Definition 36, we get following inference results derived on L2 , and its interpretations

clIF∗ (L) and clIG∗ (L) on I ∗ (L).

min / max
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (f1 ∧ r2 )] ⊕ ¬[U ∨ (f2 ∧ r1 )] = [U ∨ (0.6, 0.3) ∧ (0.5, 0.1)] ⊕ ¬[U ∨ (0.3, 0.4) ∧ (0.5, 0.1)]

= [U ∨ (min(0.6, 0.5), max(0.3, 0.1))] ⊕ ¬[U ∨ (min(0.3, 0.5), max(0.4, 0.1))]

= [(0, 0)∨(0.5, 0.3)]⊕¬[(0, 0)∨(0.3, 0.4)] = (max(0, 0.5), min(0, 0.3))⊕¬(max(0, 0.3), min(0, 0.4))

= (0.5, 0) ⊕ ¬(0.3, 0) = (0.5, 0) ⊕ (0, 0.3) = (max(0.5, 0), max(0, 0.3)) = (0.5, 0.3)
min / max 2
0.5 0.5·0.3
clIF∗ (L) (clL2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0.5, 0.3)) = ( 0.5+0.3 , 0.5+0.3 ) = (0.31, 0.19)
di

min / max
clIG∗ (L) (clL2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.5, 0.3)) = (0.5 − 0.3 0.3
2 , 2 ) = (0.35, 0.15)
di

Chapter 5
prod/sum
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (f1 ∧ r2 )] ⊕ ¬[U ∨ (f2 ∧ r1 )] = [U ∨ (0.6, 0.3) ∧ (0.5, 0.1)] ⊕ ¬[U ∨ (0.3, 0.4) ∧ (0.5, 0.1)]

= [U ∨ (0.6 · 0.5, 0.3 + 0.1 − 0.3 · 0.1)] ⊕ ¬[U ∨ (0.3 · 0.5, 0.4 + 0.1 − 0.4 · 0.1)]

= [(0, 0)∨(0.3, 0.37)]⊕¬[(0, 0)∨(0.15, 0.46)] = (0+0.3−0·0.3, 0·0.37)⊕¬(0+0.15−0·0.15, 0·0.15)

= (0.3, 0) ⊕ ¬(0.15, 0) = (0.3, 0) ⊕ (0, 0.15) = (0.3 + 0 − 0.3 · 0, 0 + 0.15 − 0 · 0.15) = (0.3, 0.15)
prod/sum 2
0.3 0.3·0.15
clIF∗ (L) (clL2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0.3, 0.15)) = ( 0.3+0.15 , 0.3+0.15 ) = (0.2, 0.1)
di

prod/sum
clIG∗ (L) (clL2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.3, 0.15)) = (0.3 − 0.15 0.15
2 , 2 ) = (0.225, 0.075)
di

Luk
clL 2 (ϕ)(equal(a, b))
di

= [U ∨ (f1 ∧ r2 )] ⊕ ¬[U ∨ (f2 ∧ r1 )] = [U ∨ (0.6, 0.3) ∧ (0.5, 0.1)] ⊕ ¬[U ∨ (0.3, 0.4) ∧ (0.5, 0.1)]

= [U ∨ (max(0, 0.6 + 0.5 − 1), min(0.3 + 0.1, 1)] ⊕ ¬[U ∨ (max(0, 0.3 + 0.5 − 1), min(0.4 + 0.1, 1)]

= [(0, 0) ∨ (0.1, 0.4)] ⊕ ¬[(0, 0) ∨ (0, 0.5)] = (min(0 + 0.1, 1), max(0, 0 + 0.4 − 1))

⊕ ¬(min(0 + 0, 1), max(0, 0 + 0.5 − 1)) = (0.1, 0) ⊕ ¬(0, 0) = (0.3, 0) ⊕ (0.0)

= (min(0.3 + 0, 1), min(0 + 0, 1)) = (0.3, 0)

2
0.3 0.3·0
clIF∗ (L) (clL
Luk F
2 (ϕ)(equal(a, b))) = clI ∗ (L) ((0.3, 0)) = ( 0.3+0 , 0.3+0 ) = (0.3, 0)
di

151
Contents

2 (ϕ)(equal(a, b))) = clI ∗ (L) ((0.3, 0)) = (0.3 − 2 , 2 ) = (0.3, 0)

0 0
clIG∗ (L) (clL
Luk G
di

drastic
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (f1 ∧ r2 )] ⊕ ¬[U ∨ (f2 ∧ r1 )] = [U ∨ (0.6, 0.3) ∧ (0.5, 0.1)] ⊕ ¬[U ∨ (0.3, 0.4) ∧ (0.5, 0.1)]

= [U ∨ (TD (0.6, 0.5), SD (0.3, 0.1)] ⊕ ¬[U ∨ (TD (0.3, 0.5), SD (0.4, 0.1)]

= [(0, 0) ∨ (0, 1)] ⊕ ¬[(0, 0) ∨ (0, 1)] = (SD (0, 0), TD (0, 1)) ⊕ ¬(SD (0, 0), TD (0, 1))

= (0, 0) ⊕ ¬(0, 0) = (0, 0) ⊕ (0, 0) = (SD (0, 0), SD (0, 0)) = (0, 0)

clIF∗ (L) (clL

drastic
2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0, 0)) = (0, 0)
di

clIG∗ (L) (clL

drastic
2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.3, 0)) = (0, 0)
di

N ilpotent
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (f1 ∧ r2 )] ⊕ ¬[U ∨ (f2 ∧ r1 )] = [U ∨ (0.6, 0.3) ∧ (0.5, 0.1)] ⊕ ¬[U ∨ (0.3, 0.4) ∧ (0.5, 0.1)]

= [U ∨ (TnM (0.6, 0.5), SnM (0.3, 0.1)] ⊕ ¬[U ∨ (TnM (0.3, 0.5), SnM (0.4, 0.1)]

= [(0, 0)∨(0.5, 0.3)]⊕¬[(0, 0)∨(0, 0.4)] = (SnM (0, 0.5), TnM (0, 0.3))⊕¬(SnM (0, 0), TnM (0, 0.4))

= (0.5, 0) ⊕ ¬(0, 0) = (0.5, 0) ⊕ (0, 0) = (SnM (0.5, 0), SnM (0, 0)) = (0.5, 0)
N ilpotent 2
0.5 0.5·0
clIF∗ (L) (clL2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0.5, 0)) = ( 0.5+0 , 0.5+0 ) = (0.5, 0)
di

N ilpotent
clIG∗ (L) (clL2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.5, 0)) = (0.5 − 02 , 20 ) = (0.5, 0)
di

Hamacher
clL2 (ϕ)(equal(a, b))
di

= [U ∨ (f1 ∧ r2 )] ⊕ ¬[U ∨ (f2 ∧ r1 )] = [U ∨ (0.6, 0.3) ∧ (0.5, 0.1)] ⊕ ¬[U ∨ (0.3, 0.4) ∧ (0.5, 0.1)]

= [U ∨ (TH0 (0.6, 0.5), SH2 (0.3, 0.1)] ⊕ ¬[U ∨ (TH0 (0.3, 0.5), SH2 (0.4, 0.1)]

= [(0, 0) ∨ (0.375, 0.39)] ⊕ ¬[(0, 0) ∨ (0.23, 0.48)] = (SH2 (0, 0.375), TH0 (0, 0.39))

⊕ ¬(SH2 (0, 0.23), TH0 (0, 0.48)) = (0.375, 0) ⊕ ¬(0.23, 0) = (0.375, 0) ⊕ (0, 0.23)

= (SH2 (0.375, 0), SH2 (0, 0.23)) = (0.375, 0.23)

2
0.375 0.375·0.23
clIF∗ (L) (clL
Hamacher
2 (ϕ)(equal(a, b))) = clIF∗ (L) ((0.375, 0.23)) = ( 0.375+0.23 , 0.375+0.23 ) = (0.23, 0.14)
di

clIG∗ (L) (clL

Hamacher
2 (ϕ)(equal(a, b))) = clIG∗ (L) ((0.375, 0.23)) = (0.375 − 0.23 0.23
2 , 2 ) = (0.26, 0.12)
di

Figure 5.12 shows above results in opinion space, I ∗ (L) and L2 . Unlike the case

of using values lying on the boundary of the spaces such as T, F, DT1 , DF1 , when internal

values in spaces are used, the reasoning results are quite dependent on the choice of t-norms

152
 Contents

Figure 5.12: Reasoning Results of Modified Example 3 in Opinion Space and L2 & I ∗ (L).

Chapter 5
and t-conorms. However, what pair of t-norms and t-conorms to use is not easy to answer.

This is a problem common to all fuzzy set based applications. Typically, connectives are

categorized by the properties they satisfy. Lukasewicz connectives are in some sense the

most interesting one because they satisfy the most properties of binary connectives, but

it does not mean that they are best suited for each application. This aspect is some-

times also attacked by statisticians who prefer Bayesian theory. However, Product / Sum

connectives are interesting in Bayesian sense, because Product t-norm and Sum t-conorm

resemble probabilistic conjunction and disjunction. For instance, following Equation 5.8,

fuzzy Belnap connectives on L2 that are compositionally defined upon Product t-norm and

Sum t-conorm pair are as follows :

(bx , dx ) ∧bl (by , dy ) = (bx by , dx + dy − dx dy )

(bx , dx ) ∨bl (by , dy ) = (bx + by − bx by , dx dy )
(bx , dx ) ⊗bl (by , dy ) = (bx by , dx dy )
(bx , dx ) ⊕bl (by , dy ) = (bx + by − bx by , dx + dy − dx dy )

153
Contents

As mentioned throughout this chapter, subjective logic has solid mathematical basis in

Bayesian perspective on dealing binary (crisp) event (see preliminaries and definitions oper-

ators). Therefore, it is worth to compare above Product / Sum connectives with the ones in

subjective logic in the definition level. For example, given two opinions wx = (bx , dx , ix , ax )

and wy = (by , dy , iy , ay ), the conjunction operator of subjective logic generates following

elements (see Definition 31) :

(1−ax )ay bx iy +ax (1−ay )ix by
bx∧sl y = bx by + 1−ax ay

dx∧sl y = dx + dy − dx dy
(1−ay )bx iy +(1−ax )ix by
ix∧sl y = ix iy + 1−ax ay

ax∧sl y = ax ay

and the disjunction of the two opinions are defined as follows (see Definition 32):

bx∨sl y = bx + by − bx by
ax (1−ay )dx iy +(1−ax )ay ix dy
dx∨sl y = dx dy + ax +ay −ax ay
ay dx iy +ax ix dy
ix∨sl y = ix iy + ax +ay −ax ay

ax∨sl y = ax + ay − ax ay

the consensus ⊗sl of the two opinions are defined as follows (see Definition 30):

bA,B
x = (bA B B A
x ix + bx ix )/k

dA,B
x = (dA B B A
x ix + dx ix )/k

iA,B
x = (iA B
x ix )/k
x ix +ax ix −(ax +ax )ix ix
aA A B A A B A B
aA,B
x = iA +i B −2iA iB
x x x x

x + ix − ix ix .
where, k = iA B A B

Although conjunctions (disjunctions) of fuzzy Belnap on Product / Sum and of subjective

logic look similar, they are not exactly the same. The reason is because the definition in

subjective logic is defined so that it can model a beta distribution that approximates the

resulting function by multiplying (comultiplying) the two of corresponding beta distribu-

tions of the given two opinions wx and wy [97] (note that, the result of multiplication and

comultiplication of two beta functions are not always beta function, [97]).

Similarly, while the join operator ⊕bl on Product / Sum just sum both the belief and

the disbelief, subjective logic calculation is designed so that it can model the beta distribu-

154
 Contents

tion derived by merging each pair of parameters of the beta distributions correspond to the

given two opinions wx and wy [94]. Indeed, through (5.2), (5.3) and (5.4) in Section 5.6.3,

we have shown that the ‘consensus’ operator itself is the calculation of ‘Bayes Theorem’

itself. Due to this aspect, even compared with Product / Sum fuzzy Belnap connectives,

subjective logic stays closer to the Bayesian aspect.

When it comes to visual surveillance, number of vision analytics are based on the

pattern recognition and machine learning techniques that are also (in many cases) based

on Bayesian statistics rather than fuzzy theory. Noting this aspect, we advocate subjective

logic could be better suited for visual surveillance applications especially when we want to

stay closer to the way that usual vision analytics generate uncertain symbolic facts.

5.10 Chapter Summary

In summary of lessons learned in this chapter, the attempt to modeling default reasoning

Chapter 5
using subjective logic has given several insights. 1) It shows the feasibility of proposed

reasoning scheme on handling contradictory information. 2) While L-fuzzy logics and

subjective logic have commonality in representing epistemic status, subjective logic is more

close to Bayesian on operation of such status. 3) The logical soundness of the proposed

approach makes it attractive for visual surveillance scenarios compared with L-fuzzy logic

based default reasoning approaches.

In this chapter, we proposed subjective logic based inference framework for default

reasoning, and demonstrated its use for high level semantic analysis of visual surveillance

scenes. Default reasoning is an important aspect of human like non-monotonic reasoning

under incomplete and imprecise knowledge, that can play an important role for deriving

plausible conclusions for many applications. Especially, in the forensic sense of visual

surveillance that needs to reason about a propositional hypothesis to be investigated af-

ter an incident or a report, it is natural to examine all positive and negative contextual

155
Contents

evidences that are related to the given hypothesis and fuse them to derive plausible conclu-

sion based on default reasoning. The keys to enable default reasoning are 1) representing

incompleteness of knowledge and 2) providing appropriate inference mechanism to draw

plausible semantic conclusions by aggregating that knowledge. We adopted subjective logic

due to its property of representing belief with ignorance and its rich set of operators for

handling uncertain beliefs. To contrast the properties and advantage of the proposed ap-

proach, we also applied the inference scheme on L-fuzzy set based logics. The case study

results show that the proposed approach and L-fuzzy set based approaches can be an al-

ternative tool to model default reasoning. Among the L-fuzzy logics, intuitionistic fuzzy

logic is very similar to the uncertainty representation scheme of subjective logic. While

the generalized intuitionistic fuzzy logic, that is fuzzy Belnap logic, can be defined on a

bilattice structure with operators regarding degree of information, intuitionistic fuzzy logic

could not be fully defined on a bilattice structure because the join operator along the axis

of degree of information can not be defined. Contrary to intuitionistic fuzzy logic, even

though it also has triangle structure, subjective logic provides an operator called consensus

that has very similar behaviour as the join operator on degree of information in bilattice.

This is because when two opinions are fused by the consensus operator, it always decreases

ignorance in the derived opinion except in the case of fusing definite true (full belief) and

definite false (full disbelief). Due to this aspect, the comparison of subjective logic based

default reasoning with intuitionistic fuzzy logic was done via a mapping between the fuzzy

Belnap logic and the intuitionistic fuzzy logic. The reasoning result of both the subjective

logic and the fuzzy Belnap logic seem reasonable. However, as noted in the discussion

section, fuzzy Belnap logic has some problems. 1) the truth table has some problematic

aspects, thereby logically not sound. 2) due to 1) the sequence of getting information is

critical. 3) due to 1) once the epistemic state is reached to the contradictory state, it is

not easy to escape that state. 4) the basic four logical operators in L-fuzzy logics can be

determined in many ways, therefore, the semantics of the operators are not sound and clear

in Bayesian sense. Due to these aspects, we advocate subjective logic has advantages as a

156
 Contents

tool for artificial reasoning in visual surveillance. Because, in visual surveillance, due to the

flexibility, and instability of the vision analytics, we can not guarantee the sequence of get-

ting information, therefore, the reasoning system should be robust against the information

acquisition sequence. Indeed, most of the vision analytics are based on probabilistic theory,

therefore, the values from those analytic modules could be well interpreted in subjective

logic. Beside these aspects, there is yet another advantage of the proposed approach, that

is the ability of default reasoning can be fulfilled within a single subjective logic based

reasoning framework that can also offer additional potential usage such as bidirectional

conditional modeling [81], reputation based belief decaying, etc. [80]. Therefore, enabling

default reasoning to subjective logic could offer better expressive power for modeling and

reflecting real world situation.

There are, however, still open issues such as comparing the introduced inference

scheme to more complicated situational reasoning. Therefore, our future research will

cover such comparisons and applying the shown approach to more complicated scenarios

Chapter 5
using automatically generated large scale data.

157
158
6 Vague Proposition Modeling and Handling

In the previous chapter, we have explained that subjective logic can be a good means

to model ‘default reasoning’. The main idea was to label truth of facts and rules with

subjective opinion (see Definition 33) and to fuse both positive and negative sources of

information together (see Definition 35).

In the big picture of our ‘Opinion Assignment’ concept described in Definition 16 of

Section 4.3.2, the former corresponds to 1 - 2 of Definition 16 and the latter corresponds

to Definition 16 - 4. However, there is yet another type of opinion assignment depicted in

Definition 16 - 3. While 1 - 2 of Definition 16 assign a fixed static opinion value to facts

and rules, Definition 16 - 3 assigns an opinion dynamically, according to the subjective

logic calculation scheme written in the rule body. In this chapter, we will further explore

rule modeling in type of Definition 16 - 3.

Chapter 6
6.1 Introduction

This chapter presents an approach to modeling vague contextual rules using subjective logic

for forensic visual surveillance. Unlike traditional real-time visual surveillance, forensic

analysis of visual surveillance data requires mating of high level contextual cues with

observed evidential metadata where both the specification of the context and the metadata

suffer from uncertainties. To address this aspect, in Chapter 4, we proposed the use of

declarative logic programming to represent and reason about contextual knowledge, and

the use of subjective logic for uncertainty handling. Upon this approach, in the previous

Chapter 5, we have demonstrated that subjective logic can be a good means to model

159
Contents

‘default reasoning’. The main idea was to label generalized truth or priority of facts and

rules with subjective opinion and to fuse both positive and negative sources of information

together.

However, there are often cases that the truth value of rule itself is also uncertain

thereby, uncertainty assignment of the rule itself should be rather functional. ‘the more X

then the more Y’ type of knowledge is one of the examples. To enable such type of rule

modeling, in this chapter, we propose a reputational subjective opinion function upon logic

programming, which is similar to fuzzy membership function but can also take uncertainty

of membership value itself into account. Then we further adopt subjective logic’s fusion

operator to accumulate the acquired opinions over time. To verify our approach, we present

a preliminary experimental case studies on reasoning likelihood of being a good witness

that uses metadata extracted by a person tracker and evaluates the relationship between

the tracked persons. The case study is further extended to demonstrate more complex

forensic reasoning by considering additional contextual rules.

6.2 Background and Motivation

In the pipeline of our proposed approach (see Chapter 4), intermediate metadata comes

from vision analytics and additional visual or non visual contextual cues are encoded as

either symbolized facts or rules. Then uncertainty comes with vision analytics are repre-

sented as subjective opinions and attached to their symbolized facts. Similarly, uncertainty

as general trustworthiness or priority among rules is also represented and attached to given

contextual rules. Once such uncertainty attachment is done, principled inference, which

is often nonmonotonic, is conducted. The examples of such principled inferences are de-

fault reasoning [143] to handle inconsistent information (see Chapter 5), abduction [50] to

find most probable hypothesis of given observation and belief revision over time upon the

change of observation (see Chapter 8), etc. Therefore, appropriate uncertainty assignment

plays an important role for proper inference.

160
 Contents

Unlike above mentioned opinion assignment to logical rules, however in reality, there

are linguistically vague logical rules that hinders assigning a single opinion to a rule does

not make sense but opinion assignment itself should be done rather functionally. In this

chapter we will refer to this type of rule as ‘Vague Rule’. An example rule in this type

is ‘the more X then the more Y’. Humans are very skillful in dealing such type of rules.

To enable such type of rule handling in our framework, we first examine a subjective logic

operator called ‘reputation operator’. We then propose a reputational subjective opinion

function that is similar to fuzzy membership function but can also take uncertainty of

membership value itself into consideration. To demonstrate reasoning under such vague

rules, we present a preliminary experimental case study by intentionally restricting the

type of available metadata to the results from human detection and tracking algorithms.

Automatic human detection and tracking is one of the common analytics and becoming

more widely employed in automated visual surveillance systems. The typical types of meta-

information that most human detection analytic modules generate comprise, for instance,

localization information such as coordinate, width, height, time and (optionally) additional

low-level visual feature vectors. We intend to use further such information for evaluating

the relationship between two persons and, more specifically, for estimating whether one

Chapter 6
person could serve as a witness of another person in a public area scene. Examples for

(linguistic) domain knowledge applicable to this scenario include: 1) (At least) two distinct

people are required for building a relationship. 2) The closer the distance between two

people is, the higher is the chance that they may identify each other. 3) If two persons

approach each other directly (face-to-face) then there is a higher chance that they can

identify each other. Such linguistic knowledge can be modeled and encoded as rules by

the proposed approach. The case study is further extended to demonstrate more complex

forensic reasoning by considering additional contextual rules together with the shown vague

rules.

161
Contents

6.3 Proposed Approach

The proposed vague rule modeling approach mainly relies on the logic programming ex-

tended with subjective logic (see Section 4.3.2). Firstly, for a given propositional know-

ledge, we assume a fuzzy-like membership function that grades degree of truth. Then we

focus on that the interpretation of such membership function can be dogmatic, thereby,

when the function is projected on the opinion space, it only lays on the bottom line of the

opinion space. Indeed, in many cases, the exact shape of the function is hard to deter-

mine. To address this aspect, we introduce a reputational function that evaluate the trust

worthiness of the fuzzy-like membership function. Then we introduce accumulation of the

resulted opinions overtime.

6.3.1 Vague Propositional Rules

In logic programming, a conditional proposition ‘y ← x’ is interpreted as ‘IF x THEN y’.

However, there are often cases that we may want to interpret the meaning as ‘the more x

then the more y’ or ‘the more x then the less y’, etc. In this case, the opinion attached to

the consequence of the rule should be rather functional in terms of the elements within the

rule body. Therefore, the opinion assignment suit to this interpretation is Definition 16.

In the sense of intrinsic linguistic uncertainty of the rule, it resembles fuzzy rules shown by

Anderson et al. [17, 177]. In the work, quantitative low level features of human detection

results such as ‘centroid’, ‘eigen-based height’ and ‘ground plane normal similarity’ are lin-

guistically mapped into non-crisp premises (i.e. fuzzy sets) as ‘(H)igh’, ‘(M)edium’, ‘(L)ow’

and ’(V)ery Low’. Then fuzzy rules defines the conjunctive combination of those linguistic

symbols to draw higher semantics such as ‘Upright’, ‘In Between’ and ‘On the ground’

(e.g. ‘U pright(L) ← Centroid(H), EigenHeight(M ), Similarity(H)’ [17]). Therefore,

introducing appropriate fuzzy membership functions for each linguistic terms and proper

handling of the membership functions is of important issue. In this view, Mizumoto et al.

[119] showed comparison of sophisticated mathematical handling of ambiguous concepts

162
 Contents

such as ‘more or less’ having various shapes. One another thing worth to note concern-

ing Fuzzy Logic is that, even if there are Zadeh’s original logical operators, there are yet

another ways of defining logical operators as well. For example, for given two quantita-

tive variables x and y come with corresponding membership functions µa and µb , Zadeh’s

AND operator is defined as ‘x AND y = min(µa (x), µa (y))’. In socalled ‘t-norm fuzzy

logic’, any form of t-norms can be considered as AND operators (see Table 5.5). (note

that, t-norms also played an important role to define lattice operators of L-fuzzy sets as

shown in Definition 5.8 of Chapter 5). For example, in the case of using product t-norm,

the AND operator can be defined as ‘x AND y = µa (x) · µb (x)’ [73]. This aspect still

remains controversial among most statisticians, who prefer Bayesian logic [179]. Contrary,

as explained in the section Section 3.1, subjective logic can be interpreted in the sense of

bayesian and also the final quantitative opinion space can also be interpreted in the sense

of fuzziness (i.e. ‘very certainly true’, ‘less certainly true’, etc). This way, we believe that

subjective logic can better bridges the interpretation of fuzzy intuitive concepts in bayesian

sense. The basic idea of our approach is as follows :

• 1. For a given propositional rule ‘the less (more) y ← the more x’ we could introduce a

membership-like function µi : x → y.

Chapter 6
• 2. It is clear that the function µi should be monotonically decreasing (increasing) but

the shape is not quite clear.

• 3. Considering potentially possible multiple membership like functions µi , however the

values of µi (x) at the two extreme point of (minx ≤ x ≤ maxx ) tend to converge but

the values in between are diverge therefore, the values of later cases are more uncertain.

• 4. Considering the aspect of 3. we introduce socalled reputational opinion function on

the function µi and combine it with raw opinion obtained from µi using subjective logic’s

reputation operator.

163
Contents

Figure 6.1: Vague Rule Modeling using Subjective Logic’s Reputation Operator.

This idea is depicted in Figure 6.1, where the actual reputational operation is defined as

follows :

Definition 41. (Reputation) [99] . Let A and B be two agents where A’s opinion about
A = {bA , dA , uA , aA }, and let x be a proposition
B’s recommendations is expressed as wB B B B B

where B’s opinion about x is recommended to A with the opinion wxB = {bB
x , dx , ux , ax }.
B B B

Let wxA:B = {bA:B

x , dx , ux , ax } be the opinion such that:
A:B A:B A:B


 bA:B = bA bB
x B x dA:B
x = dA B
B dx

 uxA:B = dA + uA + bA uB
B B B x aA:B
x = aB
x

then wxA:B is called the reputation opinion of A. By using the symbole ⊗ to designate this
A ⊗ wB .
operation, we get wxA:B = wB x

For actual evaluation of a given function µi , an opinion assignment function on the

given µi need to be defined. Although there could be also another ways of such function,

in our approach, this is modeled as follows :

164
 Contents



 bx = k + 4(1 − k)(µi (x) − 12 )2


µi (x)

wµreput = 1−bx
dx = Dratio (6.1)
i (x) 



 ux = 1 − bx − dx

where k, represents the minimum boundary of belief about the value from µi (x), and

the Dratio indicates the ratio for assigning the residue of the value µi to disbelief and

uncertainty. This is depicted as Figure 6.1 - D.

6.4 Case Study I

6.4.1 Scenario Setting for Case Study I

At this stage we focused on evaluating the modeling approach itself rather than the reli-

ability of the person detection algorithm. Therefore, we manually annotated a test video

from one of i-LIDS [3] data sample with ground truth metadata for human detection com-

prising bounding boxes and timing information (shown in Figure 6.2). In total, 1 minute

of test video was annotated in which there are 6 people. For our purposes, we intentionally

marked one person as suspect. Then we encoded following linguistic contextual knowledge

according to the proposed approach as explained in Section 4.3.2. 1) (At least) two dis-

Chapter 6
tinct people are required for building a relationship 2) The closer the distance between

two people is, the higher is the chance that they can identify each other. 3) If two persons

approach each other directly (face-to-face) then there is a higher chance that they can

identify each other. Then we calculate subjective opinions between the person marked as

suspect and other human instances over time.

6.4.2 Uncertainty Modeling

6.4.2.1 Distance

The distance between a pair of people would be one of the typical pieces of clue for rea-

soning whether one person could serve as a witness of another person. This relates to the

165
Contents

Figure 6.2: Scenario Setting for Case Study I.

Figure 6.3: Candidate Uncertainty Functions regarding Distance and Direction.

general human knowledge that ‘The closer two people are in distance, the more chances of

perceiving the other are’. Humans are very adapted to operating upon such type of uncer-

tain and ambiguous knowledge. Exactly modeling such a relation is not trivial, but we can

approximate it with a monotonic decreasing function about the possibility of perceiving

each other. This aspect is depicted as three possible curves in the middle of Figure 6.3

- A.), where x represents the distance between the persons as calculated from the person

detection metadata and µi represents the likelihood that two persons at this distance would

perceive each other, maxdist is the maximum possible (i.e diagonal) distance in a frame

and ai is the estimated probability that two humans could’ve recognized each other at the

maxdist distance. However, the value derived from such function is not fully reliable due

to the variety of real world and uncertainty in the correctness of the function and uncer-

tainty in the distance value itself. Considering the aspect of distance, it is clear that both

166
 Contents

extreme cases i.e. very close or very far are much more certain than in the middle of the

range. Thus, to better model the real world situation, the reputational opinion function

need to be applied to any chosen function µi . This is modeled as opinion on the reliability

of µi (x) by applying Equation 6.1. In order to evaluate the impact of choosing different

functions in Figure 6.3 - A.), three different types of µi functions (a concave, convex and

linear) have been applied. The derived reputational opinions showed similar aspects having

peaks of certain belief at each extreme cases as shown in Figure 6.4.

6.4.2.2 Direction

Similarly, we also used direction information between two persons. The linguistic know-

ledge to be modeled is ‘if two persons approach each other directly (face-to-face) then there

is a higher chances of perceiving each other’. The corresponding direction-based relevance

function is shown in Figure 6.3 - B.), where Θ represents the angle between the persons

heading directions as calculated from the person detection metadata and µi represents the

likelihood that two persons at the angle would perceive each other and ai is the expected

minimum probability that two humans could’ve recognized each other at any angle. How-

ever, again the trustworthiness of the values from such functions µi is uncertain, especially

Chapter 6
in the middle range of the Θ. To roughly model such aspect, for a chosen function µi (Θ),

the same reputational function from Equation 6.1 was used again. The impact of choosing

different µi showed similar behavior as of direction based opinions as shown in Figure 6.4.

167
Contents

Figure 6.4: Samples of Reputational Opinion according to Distance and Equation 6.1.

168
 Contents

6.4.3 Rule Encoding

In addition to the uncertainty modeling, logic programming is used to represent the given
contextual rules as explained in Section 4.3.2. Encoded rules in form of Equation 4.2 are
as follows :

Rule1
wwitness(H 1 ,H2 ,T1 )
←
µ( d) (6.2)
HumanD etector HumanD etector µ (d)
(whuman(H1 ,T1 )
∧ whuman(H 2 ,T1 )
) ⊗ (wwitness(H
dist
1 ,H2 ,T1 )
⊗ wµreput
dist (d)
)

Rule2
wwitness(H1 ,H2 ,T1 )
←
µ(d)
(6.3)
HumanD etector HumanD etector µ (d)
(whuman(H1 ,T1 )
∧ whuman(H 2 ,T1 )
) ⊗ (wwitness(H
dir
1 ,H2 ,T1 )
⊗ wµreput
dir (d)
)

Rule3
wwitness(H1 ,H2 ,T1 )
← (wwitness(H
Rule1
1 ,H2 ,T1 )
∧ wwitness(H
Rule2
1 ,H2 ,T1 )
) (6.4)

Rule4
wwitness(H 1 ,H2 ,Tn )
← ⊕n Rule3
i=1 wwitness(H1 ,H2 ,Ti ) (6.5)

The first Rule 6.2 starts considering the necessary condition, meaning that there should

be a distinct pair of two people. Therefore the conjunction operation ∧ (see Definition 31)

on two opinions [97] is used that is very similar to the operation P (A) · P (B) except

that in subjective logic the opinion can additionally represent ignorance. Then, for the

resulting set of frames the reputational opinion about the distance opinions is calculated

as described in Section 6.4.2. Each result is assigned to a new opinion with the predicate

Chapter 6
of the appropriate arity and is assigned the name of agent with the final belief values. In

this case, the final opinion value represents that there is an opinion about two persons

being potential witnesses of each other from an agent named Rule1. The second Rule 6.3

is almost same as Rule 6.2. The only different part of this rule is that the reputational

opinion is about direction. The third Rule 6.4 combines the evidences coming from Rule 6.2

and (6.3). The conjunction operator ∧ is used to reflect that for reliable positive resulting

opinions both evidences should have appeared with a certain amount of belief. The last

Rule 6.5 is about accumulating the belief over time using the consensus operator ⊕ [94]

(see Definition 30). Figure 6.5 shows a graphical representation of the rules in a tree form.

169
Contents

Figure 6.5: Tree Representation of used Rules.

Figure 6.6: Visualization of the Experiment.

6.4.4 Experimental Result

Using the rules described in Section 6.4.3, we calculated subjective opinions between a per-

son marked as suspect and other human instances over time. Figure 6.6 shows a snapshot

of the visualization in the prototype comprising a video player and an opinion visualizer.

While the video is being played the corresponding metadata is transformed into the cor-

responding opinion representation. The translated opinions are fed into the rule-engine

170
 Contents

Figure 6.7: Experimental Result.

which automatically evaluates the rules. The right part of Figure 6.6 shows the opinion

about the proposition ‘human 5 is a witness for the suspects’ marked red and its corre-

sponding mapping to beta distribution. For verification of these results, a questionnaire

was prepared to collect scores about the witnessing chances for each of the ‘pairs’ in the

scene (e.g. human1 and suspect, human2 and suspect , etc). 7 people from our lab took

part in the questionnaire. Then changing the uncertainty functions on vague rules, we

tested the behavior of the proposed approach to check whether it well models human in-

tuition. Although there can be 9 possible combinations of uncertainty functions (i.e. 3

distance functions and 3 direction functions), to better contrast the impact of changing

such uncertainty functions, we have fixed the direction function to the type of µ3 defined

Chapter 6
in Figure 6.3 - B.) and tested with 3 different direction functions shown in Figure 6.3 -

A.). Then the mean and standard deviation, min and max of the ‘human opinions’ were

calculated and compared to the computed results. According to [93], the following criteria

should be applied to the computed results.

1) The opinion with the greatest probability expectation is the greatest opinion.

2) The opinion with the least uncertainty is the greatest opinion.

3) The opinion with the least relative atomicity is the greatest opinion.

In the described experiment, due to the small size of possible pairs, only the first cri-

terion was applied and the final expectation values of each opinion for candidate pairs

were plotted jointly with the questionnaire based result as shown in Figure 6.7. The final

result turns out to be following the tendency of questionnaire based human ‘opinions’. The

171
Contents

change of uncertainty function seems not introducing that critical differences. However,

there were more differences between the expected values, when the final expectation values

were low, for instance, though it was a slight differences, µ3 tend to yield larger expecta-

tion value then µ2 and µ1 . The differences ware smaller when the final expectation values

were getting higher. However, in any cases, the order on the ranking of witnesses show the

same results. Therefore, in the sense of human like reasoning, it seems that the proposed

approach well models human intuition.

6.5 Case Study II

In this section, we further explorer the proposed case study scenario for more complex

contextual forensic reasoning. Especially, we will consider the situation that is needed to

be modeled in the sense of default reasoning [143] explained in Chapter 5.

6.5.1 Scenario Setting for Case Study II

Figure 6.8: Scenario Setting for Case Study 2.

Let us consider a conceptual scenario that a security personnel wants to get sugges-

tions of most probable witnesses of a selected suspect in a scene. Given an assumption

172
 Contents

that automatic vision analytics are running and extracting basic semantics, we will assume

two virtual situations as shown in Figure 6.8, where, witnesses are reasoned according to

the uncertain spatio-temporal rules as demonstrated in Section 6.4. In all situations we

will assume that ‘witness2’ has higher opinion then ‘witness1’. In addition to this, we will

assume optional cases that additional evidential cues are detected. In Figure 6.8 - A.),

‘witness2’ is talking on the phone. In Figure 6.8 - B.), the optional case is the detection of

a license plate of the car seems to belong to the ‘witness1’ and ‘witness2’ comes with face

detection.

6.5.2 Reasoning Examples

Given the scenario with optional cases, we will also assume that 1). people usually don’t

recognize well when they are talking on the phone, 2). identifiable witness is a good wit-

ness. 3) License plate is better identifiable source than face detection because we can even

fetch personal information of the owner easily. Therefore, under optional assumption, for

example, in Figure 6.8 - A.), ‘witness1’ should be better witness, and in Figure 6.8 - B.),

‘witness1’ should be suggested as a better witness. This kind of non monotonic reasoning

under inconsistent information can also be regarded as ‘default reasoning’ [143]. In Chap-

Chapter 6
ter 5, we showed that this aspect can be modeled using subjective logic as well under the

opinion assignment (see Definition 33) and inference mechanism (see Definition 35) shown

in Chapter 5. Here, it is important to note that, unlike the case of vague rule modeling,

the type of opinion assignment to prioritize belong to Definition 16 - 2. and the default

inference scheme belongs to Definition 16 - 4. As shown in Section 5.8, we set T ≃ (1, 0, 0)

(full truth), DT1 ≃ (0.5, 0, 0.5) (weak default true ), DT2 ≃ (0.8, 0, 0.2) (strong default

true), F ≃ (0, 1, 0) (full false), DF1 ≃ (0, 0.5, 0.5) (weak default false), DF2 ≃ (0, 0.8.0.2)

(strong default false), ∗ ≃ (0.33, 0.33, 0, 34) (contradiction), U ≃ (0, 0, 1) (full uncertainty)

and ⊥ ≃ (0.5, 0.5, 0) (full contradiction) [82, 83]. For the rest of truth values we will use

opinion triple representation (b,d,i).

173
Contents

Example 6. (Witness talking on the phone) . Assume the following set of rules

about determining good witness including the uncertain spatio-temporal relation based

witness reasoning rule described in Section 6.4.3. Then also assume the following opinion

assignment that witness2 (denoted as wit_2) has higher opinion being the witness than

witness1 (denoted as wit_1).

Rule4
ϕRule [wwitness(H1)
← ⊕ni=1 wwitness(H
Rule3
1 ,H2 ,Ti )
] = DT1
ϕRule [¬wwitness(H1 ) ← wtalking_on_phone(H1 ) ] = DT2
Rule4
ϕRuleEval [wwitness(wit_1) ] = (0.6, 0.15, 0.25)
Rule4
ϕRuleEval [wwitness(wit_2) ] = (0.7, 0.10, 0.20)
Given two default true and default false rules and facts that can be seen as definite true,

the inference for reasoning better witness using default logic with subjective logic is as

follows.
clsldi (ϕ)(wwitness(wit_1) ) = [U ⊔ ((0.6, 0.15, 0.25) · DT1 )]
= [U ⊔ (0.44, 0.15, 0.41)] = (0.44, 0.15, 0.41) ∼ (Expectation = 0.54)
clsldi (ϕ)(wwitness(wit_2) ) = [U ⊔ ((0.7, 0.10, 0.20) · DT1 )]
= [U ⊔ (0.50, 0.10, 0.40)] = (0.50, 0.10, 0.40) ∼ (Expectation = 0.60)
Above result shows that given the weak rules, ‘witness2’ is more probable witness candidate

than ‘witness1’. Then, let us consider the weak opinion assignment to the additional

contextual cue that witness2 is using the phone. This semantics can be interpreted as ‘the

witness seems to using a phone but not quite sure’.

ϕf act [wtalking_on_phone(wit_2) ] = (0.6, 0.15, 0.25)

Given the additional information, the inference on witness2 is being witness is as follows.
clsldi (ϕ)(wwitness(wit_2) )
= [U ⊔ ((0.7, 0.10, 0.20) · DT1 )] ⊕ ¬[U ⊔ ((0.6, 0.15, 0.25) · DT2 )]
= [U ⊔ (0.50, 0.10, 0.40)] ⊕ ¬[U ⊔ (0.59, 0.15, 0.26)]
= (0.50, 0.10, 0.40) ⊕ ¬(0.59, 0.15, 0.26)
= (0.50, 0.10, 0.40) ⊕ (0.15, 0.59, 0.26)
= (0.34, 0.47, 0.19) ∼ (Expectation = 0.39)
The resulting opinion (0.34,0.47,0.19) on witness2’s being a good witness now weaker than

(0.44,0.15,0.41) which is for the case of witness1’s being a good witness. The expectation

174
 Contents

values also captures this aspect. Thus, this result shows that the inference scheme well

models human intuition.

Example 7. (Witness with Face Detection vs. License Plate Detection) . Con-
sider the following set of rules about determining good witness and the following opinion
assignment to capture the scenario described in Section 6.5.1 as depicted in Figure 6.8.
Rule4
ϕRule [wwitness(H1)
← ⊕ni=1 wwitness(H
Rule3
1 ,H2 ,Ti )
] = DT1
ϕRule [wwitness(H1 ) ← wwitness(H
Rule4
1)
· whasF aceDetectInf o(H1 ) ] = DT1
ϕRule [wwitness(H1 ) ← wwitness(H
Rule4
1)
· whasLicenseDetectInf o(H1 ) ] = DT2
Rule4
ϕRuleEval [wwitness(wit_1) ] = (0.6, 0.15, 0.25)
Rule4
ϕRuleEval [wwitness(wit_2) ] = (0.7, 0.10, 0.20)
ϕf act [whasLicenseDetectInf o(wit_1) ] = (0.6, 0.15, 0.25)
ϕf act [whasF aceDetectInf o(wit_2) ] = (0.6, 0.15, 0.25)
Given two default true and default false rules and facts that can be seen as definite true,
the inference for reasoning better witness using default logic with subjective logic is as
follows.

clsldi (ϕ)(wwitness(wit_1) ) = [U ⊔ ((0.6, 0.15, 0.25) · DT1 · (0.6, 0.15, 0.25) · DT2 )]

= [U ⊔ ((0.44, 0.15, 0.41) · (0.59, 0.15, 0.26))]

= (0.33, 0.28, 0.39) ∼ (Expectation = 0.36)

Chapter 6
clsldi (ϕ)(wwitness(wit_2) ) = [U ⊔ ((0.7, 0.10, 0.20) · DT1 · (0.6, 0.15, 0.25) · DT1 )]

= [U ⊔ ((0.5, 0.1, 0.4) · (0.44, 0.15, 0.41))]

= (0.3, 0.24, 0.47) ∼ (Expectation = 0.33)

Above result shows that given the evidences, ‘witness2’ is slightly more probable witness

candidate than ‘witness1’ because license plate info is more informative thereby strongly

considered than face related information by the opinion assignment. However, due to the

opinion on the fact level is not certain, the values were not strongly forced the belief but

rather increased the uncertainty in the final opinion. The expectation values also captures

this aspect. Thus, this result show that the inference scheme well models human intuition.

175
Contents

6.6 Chapter Summary

In summary of lessons learned in this chapter, the proposed approach on dynamic

assessment of a vague proposition offers more choices to model complex contextual human

knowledge by enriching the expressive power of the framework. The proposed approach

can be used with another principled reasoning scheme such as default reasoning. There

are, however, still open issues on automatic assignment of proper priors and proper

modeling of the reputation function, etc.

Intelligent forensic reasoning upon metadata acquired from automated vision ana-

lytic modules is an important aspect of surveillance systems with high usage potential.

The knowledge expressive power of the reasoning framework and the ability of uncertainty

handling are critical issues in such systems. In this chapter, based on our previous work

on the use of logic programming with subjective logic, we extended the framework so that

it can also handle vague propositional rules. The approach is mainly based on the fuzzy-

like membership function and the reputational operation on it. The main advantage of

the proposed approach is that it offers more choices to model complex contextual human

knowledge by enriching the expressive power of the framework. The other advantage of the

proposed approach is that the modeled vague rules can be used with another principled

reasoning scheme. In this chapter, especially, we have demonstrated how the reasoning re-

sults from uncertain spatio-temporal rules could be used with default reasoning. Another

interesting properties of the system is that, unlike traditional probability based conditional

reasoning, this approach allows for representing lack of information about a proposition.

We could also roughly assign our subjective priors with lack of information, and obser-

vations can also be represented with any degree of ignorance, therefore we believe this

better reflects human intuition and real world situations. Another beneficial property is

the flexibility of assigning opinions to formulae. Especially, rule can embed its own opinion

calculation scheme thereby, allows for sophisticated propagation of opinions through the

inference pipeline. There are, however, still several open issues such as how to better model

176
 Contents

the reputational function, how to automatically assign proper prior opinions to rules, etc.

Although we still need to extend this concept to large scale data. We advocate that this

work showed the potential of the proposed approach.

Chapter 6

177
178
7 Hybrid Knowledge Modeling and Reasoning

Thus far, our discussions have been focused on the ‘extensional’ aspect of knowledge rep-

resentation and uncertainty handling. As reviewed in Section 2.3.3, while ‘extensional’

approach has benefits on ‘flexibility’ and ‘expressive power’ due to the ‘modularity’, it also

has some deficiencies such as ‘improper handling of bidirectional inference’ that can be

better handled in ‘intensional’ approaches. Therefore, it would be worth if we could take

benefits of both approaches in a single framework. In this chapter, we focus on the exten-

sion of our proposed reasoning framework to bestow ‘intensional’ characteristics upon our

proposed ‘extensional’ system, especially in focus on enabling ‘bidirectional inference’.

7.1 Introduction

In forensic analysis of visual surveillance data, conditional knowledge representation and

inference under uncertainty play an important role for deriving new contextual cues by

fusing relevant evidential patterns. To address this aspect, both rule-based (aka. exten-

sional) and state based (aka. intensional) approaches have been adopted for situation or

visual event analysis. The former provides flexible expressive power and computational
Chapter 7
efficiency but typically allows only one directional inference. The latter is computationally

expensive but allows bidirectional interpretation of conditionals by treating antecedent and

consequent of conditionals as mutually relevant states (see Section 2.4 for details). In vi-

sual surveillance, considering the varying semantics and potentially ambiguous causality in

conditionals, it would be useful to combine the expressive power of rule-based system with

the ability of bidirectional interpretation. In this chapter, we propose a hybrid approach

179
Contents

that, while relying mainly on a rule-based architecture, also provides an intensional way of

on-demand conditional modeling using conditional operators in subjective logic. We first

show how conditionals can be assessed via explicit representation of ignorance in subjective

logic. We then describe the proposed hybrid conditional handling framework. Finally, we

present several experimental case studies from a typical public domain visual surveillance

scenes.

7.2 Background and Motivation

In recent years, there has been an increasing research focus on higher level semantic reason-

ing for visual surveillance data by augmenting low level computer vision modules with high

level contextual cues (see Section 1.1 for detailed background on this). Considering the

variety of possible semantics in surveillance scenes, arising from complex spatio-temporal

events, intelligent high level semantic reasoning should provide a means of fusing evidence

from multiple, ‘uncertain’, ‘incomplete’ and potentially ‘contradictory’ information sources.

The key challenges for such high level reasoning approaches are the choice of an appropriate

contextual knowledge representation and the proper handling of uncertainty.

As we have reviewed in Section 2.3 and Section 2.4, depending on how such ap-

proaches handle uncertainty, they can be roughly categorized into ‘extensional’ and ‘inten-

sional’ approaches [130]. Extensional approaches also known as rule-based systems treat

uncertainty as a generalized truth value attached to formulas and compute the uncer-

tainty of any formula as a function of the uncertainties of its sub formulas. In intensional

approaches, also known as state based approaches, antecedents and consequents in condi-

tionals are treated as ‘subsets of possible states’ and handle uncertainty taking into account

relevance between the two states. Extensional approaches have advantages in the ‘flexibil-

ity’ and ‘expressive powe’ due to their ability to derive a new proposition based only on

what is currently known (a) regardless of anything else in the knowledge base (‘locality’)

and (b) regardless of how the current knowledge was derived (‘detachment’). ‘locality and

180
 Contents

detachment’ are together referenced to as ‘modularity’ [130].

Extensional approaches however, are considered semantically less clear compared to

intensional approaches. This deficit of semantic clarity comes from how conditionals are

interpreted. In rule-based frameworks, conditionals are interpreted by ‘Modus Ponens’ 1 .

Therefore in such systems, typically, reverse interpretation (also known as abduction) is

not possible. Although it is possible to explicitly add reverse conditionals, this would in-

troduce cycles that can adversely impact the reasoning mechanism [130]. Another problem

with traditional rule-based frameworks is the quantitative uncertainty assessment function

of conditional formulas. Due to the inference direction and modularity, the uncertainty

propagation is limited to one direction and, thus hinders considering relations between an-

tecedent and consequent in conditional formulas. This is in contrast to traditional Bayesian

reasoning that focusses on the probabilistic relation between antecedent and consequent.

In such intensional approaches, the uncertainty of any formula is computed by combining

possible worlds via set theory operations. Such approaches exhibit neither the property of

locality nor detachment. In other words, they are unable to derive new information unless

they have exhaustively accounted for all other information sources in the knowledge base,

that could possibly influence the final proposition to be reasoned about. While this process

provides better semantic clarity, it comes with an exponential computational cost. Another

property intensional approaches buy us is the ability to perform ‘bi-directional inference’.

Since intensional approaches possess complete knowledge of all information sources influ-

encing all variables of interest, it is possible to ask arbitrary queries of the model without

apriori committing to one direction of inference unlike in extensional appraoches.

Chapter 7
In visual surveillance, there are cases that antecedent and consequent appear to

be related but the causality direction is ambiguous due to the complexity and variety of

semantics. Thus, it would be desirable to combine the two approaches to achieve more

context modeling power and computational efficiency. In this chapter, we present a hybrid

approach to conditional evidence fusion that leverages the avantages of both extensional

1
The rule of deduction (given ‘If P then Q’ infer ‘P therefore Q’).

181
Contents

and intensional approaches using subjective logic. Subjective logic theory provides opera-

tors for both abduction as well as deduction [100, 95, 140] thereby, making it possible to

interpret conditionals bidirectionally, in a Bayesian sense.

7.3 Conditional Operators in Subjective Logic

In this section, we summarize the derivation of conditional operators in subjective logic,

as presented by Jøsang et al, in [100, 95, 140]. Given a conditional of the form ‘y ← x’ we

interpret the meaning as ‘IF x THEN y’. In traditional binary logic, this is seen in a truth-

functional manner following the truth table called ‘material implication’ such that any

antecedent x being true forces y to be evaluated as true as well. However, in practice there

are examples of false conditionals with false antecedent and true consequent. Therefore,

the more natural interpretation is that ‘The truth value of x is relevant to the truth value

of y’. In the sense of relevance connection between x and y, the interpretation should

also consider the case ‘y ← ¬x’ for completeness, so we can properly handle the case that

the antecedent x is false. Following classical probability theory [159], a conditional of the

form ‘y ← x’ can be interpreted in terms of probability calculus and can be expressed via

binomial conditional deduction as:

p(y||x) = p(x)p(y|x) + p(x)p(y|x)

where the terms are defined as follows:

p(y|x) : the conditional probability of y given xis TRUE

p(y|x) : the conditional probability of ygiven xis FALSE (7.1)

p(x) : the probability of the antecedent x

p(x) : the probability of the antecedent’s complement

p(y||x) : the deduced probability of the consequent y

Note that, the notation p(y||x) is meant to indicate that the truth value of y is evaluated

with both the positive and the negative conditionals. This notation is only meaningful in

182
 Contents

a probabilistic sense. In practice, however, p(y|x) or p(y|x) often cannot be determined

directly. In such cases, the required conditionals can be correctly derived by inverting the

available conditionals if we know the base rate p(y) (that is denoted as a(y) in subjective

logic), using Bayes’ theorem as follows:

a(y)p(x|y)
p(y|x) = (7.2)
a(y)p(x|y) + a(y)p(x|y)

In subjective logic, a(y) is interpreted as a subjective opinion wy (0, 0, 1, ay ). Such an

opinion having full ignorance is called ‘vacuous’ opinion. It represents a measure of prior

knowledge. Similarly, p(y|x) can be derived as follows:

a(y)p(x|y) a(y)(1 − p(x|y))

p(y|x) = = (7.3)
a(y)p(x|y) + a(y)p(x|y) a(y)(1 − p(x|y)) + a(y)(1 − p(x|y))

Let the term ‘child frame’ denote a traditional state space of mutually disjoint states and

let the term ‘parent frame’ denote the evidence that was obtained. Bearing the notation

p(consequence|antecedent) in mind, if the ‘parent frame’ is about the antecedent and the

‘child frame’ is the consequent, then such inference is called ‘deduction’. Abductive reason-

ing is the case when the ‘parent frame’ is the consequent and ‘child frame’ is the antecedent

as described in [95]. In subjective logic, conditionals can be generalized to multinomial

frames. However, in this section, we will focus on the binomial case only. Deduction on

binomial frames is as follows.

Definition 42. (Deduction) [100] . Let ΘX = {x, x} and ΘY = {y, y} be two frames
Chapter 7
with arbitrary mutual dependency. Let wx = (bx , dx , ix , ax ), wy|x = (by|x , dy|x , iy|x , ay|x ) and

wy|x = (by|x , dy|x , iy|x , ay|x ) be an agent’s respective opinions about x being true, about y

being true given x is true and about y being true given x is false, then deductive conditional

opinion wy||x = (by||x , dy||x , iy||x , ay||x ) is expressed as :

wy||x = wx ⊚ (wy|x , wy|x ) (7.4)

183
Contents

where ⊚ denotes the general conditional deduction operator for subjective opinions and,

wy||x is defined by


 by||x = bIy − ay K 

 

 
 bI = bx by|x + dx by|x + ix (by|x ax + by|x (1 − ax ))

 d 
 y
y||x = dy − (1 − ay )K
I


where,
 dIy = bx dy|x + dx dy|x + ix (dy|x ax + dy|x (1 − ax ))

 iy||x = iIy + K 


 
 iI = bx i + dx i + ix (i ax + i (1 − ax ))


 a y y|x y|x y|x y|x
y||x = ay
and K can be determined according to 3 different selection criteria detailed in [100].

Abduction on binomial frames is as follows.

Definition 43. (Abduction) [140] . Let ΘX = {x, x} and ΘY = {y, y} be two frames

with arbitrary mutual dependency. Let wy = (by , dy , iy , ay ), wxvac = (0, 0, 1, ax ), wy|x =

(by|x , dy|x , iy|x , ay|x ) and wy|x = (by|x , dy|x , iy|x , ay|x ) be an agent’s respective opinions about

observed consequent y being true, vacous subjective opinion about the base rate of the hy-

pothesis x ,about y being true given x is true, and about y being true given x is false, then

abductive conditional opinion wx||y = (bx||y , dx||y , ix||y , ax||y ) about x being cause of observed

consequent y is expressed as :

wx||y = wy ⊚(wy|x , wy|x , wxvac ) = wy ⊚ (wx|y , wx|y ) (7.5)

where ⊚ denotes the general conditional abduction operator for subjective opinions, then

the inverted conditionals wx|y ,wx|y in the right hand side of the equation can be derived

using the following formula,

wxvac · wy|x
wx|y =
wxvac ⊚ (wy|x , wy|x )
(7.6)
wxvac · ¬wy|x
wx|y = vac
wx ⊚ (¬wy|x , ¬wy|x )

thereby, we can calculate the ⊚ operation with the ⊚ operator and inverted conditionals.

Note that Equation 7.6 involves multiplication and division operators as well as de-

duction operator ⊚ (see Equation 7.4 and Table 3.2 in Definition 43). The multiplication

184
 Contents

in subjective logic is also called ‘conjunction’ and the formal definition is shown in Defini-

tion 31. The inverse operation to multiplication is division. The quotient of opinions about

propositions x and y represents the opinion about a proposition z which is independent of

y such that wx = wy∧z . The formal definition of division operator is as follows.

Definition 44. (Division /) [97] . Let ΘX and ΘY be two frames and let x and y
be propositions about state in ΘX and ΘY respectively. Let wx = (bx , dx , ix , ax ) and wy =
(by , dy , iy , ay ) be an agent’s opinions about x and y, then division opinion denoted as wx /wy
is wx∧y = (bx∧y , dx∧y , ix∧y , ax∧y ) such that :

a (1−a )(1−d )b (1−a )(1−d )i

Defined only when ax ≤ ay , dx ≥ dy , bx ≥ (1−a x y x y
and ix ≥ (1−ayx )(1−dy)
x y


x )ay (1−dy) 
 


 if ax < ay , 

if 0 < ax = ay ,

 


 
 γ(1 − dx )

 ay (bx + ax ix ) ax (1 − dx ) 


 bx∧y = − 
 bx∧y =

 (ay − ax )(by + ay iy ) (ay − ax )(1 − dy ) 
 1 − dy

 


 
 dx − dy
dx − dy dx∧y =
dx∧y = 1 − dy

 1 − dy 


 


 ay (1 − dx ) 
 (1 − γ)(1 − dx )

 −
ay (bx + ax ix ) 
 ix∧y =

 ix∧y = 
 1 − dy

 (ay − ax )(1 − dy ) (ay − ax )(by + ay iy ) 


 


 

 a 
 ax∧y = ax /ay
x∧y = ax /ay
( )
ay (1 − ay ) (1 − dy )bx by
where, γ = − by + .
(ay − ax )(by + ay iy ) 1 − dx by + ay iy

Refer to [97] for the detailed explanation on division operators.

7.4 Proposed Approach

The proposed hybrid conditional handling framework mainly relies on rule-based system

that enables logic programming proposed in Section 4.3.2. The rule-based system is ex-
Chapter 7
tended to allow representation of rules using subjective opinions and operators. The con-

ditional knowledge directly encoded as such rules are handled in extensional manner. To

handle conditionals in intensional manner, special types of predefined rules are introduced.

Such predefined rules drive the abduction and deduction operation in the subjective logic

framework in the presence of required prior opinions for the conditionals. We will first give

185
Contents

a brief overview how rules are expressed in logic programming. Thereafter, comes with

further details of the framework.

7.4.1 Extensional Layer

In Section 4.3.2, we proposed a logic programming framework extended with subjective

logic. In this section, we will briefly explain the framework again. In the proposed frame-

work, the CLIPS [1] rule engine was used as a basis to provide flexibility for defining

complex data structure as well as for providing a rule resolving mechanism. To extend

this system, a data structure ‘opinion(agent, proposition, b, d, i, a)’ was defined that can

be interpreted as a fact of arity 6 with the following terms, agent (opinion owner), propo-

sition, belief, disbelief, ignorance, and atomicity. To represent propositions, we extended

the structure so that it can take arity n properties as well. Therefore, given a predicate p

the proposition can be described as ‘p(a1 , a2 , ..., an )’. In our system, therefore, each fact is
agent
represented as the form of ‘wp(a 1 ,a2 ,...,an )
’. Namely, rules are defined with the opinion and

proposition structure. Additionally, functions of subjective logic operators taking opinions

as parameters were defined. Since in rule-based systems, actions can be executed in the

head part of the rule, the uncertainty assessment ‘ϕ : rules → opinion’ operation, defined

in Definition 16, can be defined in the head part of rules using subjective logic operators

and opinions shown in rule body. This aspect is depicted in Section 4.3.2 as follows :

Rule Head (ACTION) :

Assert new Opinion wpacc(ac1 ,..,acn ) ,

where wpacc(ac1 ,..,acn ) = wpa11 (a11 ,..,a1n ) ⊛ .. ⊛ wpain (ai1 ,..,ain )

(4.1)
←

Rule Body :

wpa11 (a11 ,..,a1n ) , .., wpain (ai1 ,..,ain )

Due to the redundancy that arises when describing rules at the opinion structure level, we

will use abbreviated rule formulae as follows :

186
 Contents

wpacc(ac1 ,..,acn ) ← wpa11 (a11 ,..,a1n ) ⊛ ... ⊛ wpain (ai1 ,..,ain ) (4.2)

where ⊛ indicates one of subjective logic’s operators. More concretely, this implies checking

for the existence of each opinion that matches property constraints among the propositions

of opinions. In general, rules in the form of Equation 4.2 can also be seen as condition-

als. Thus, this way of representing rules and handling uncertainty can be considered as

extensional approach and in our framework such rules represent the extensional layer.

7.4.2 Intensional Layer

In the intensional layer, a conditional proposition ‘y ← x’ is viewed in the sense of relevance

connection between x and y. To generalize this view, we will consider x and y as two facts

with arbitrary mutual dependency without restricting the influence direction. Then we

can set two conditionals ‘y ← x’ and ‘x ← y’. This results in 4 possible conditional

operations as shown in Table 7.1, namely, deduction and abduction for each conditionals.

In the sense of querying for the cases in Table 7.1, Case 1 can be interpreted as ‘given

an observational opinion x give me an opinion that this would make y happen’. Case 2

means ‘given an observational opinion y give me an opinion that it was caused by x’.

Case 3 is ‘given an observational opinion y give me an opinion that this would make

x happen’. Finally, Case 4 says ‘given an observational opinion x give me an opinion

that it was caused by y’. As shown in Section 7.3, the calculation of those operations

requires incidental opinions (i.e. priors) also. According to Equation 7.4 and Equation 7.5

wx||y = wy ⊚(wy|x , wy|x , wxvac ) = wy ⊚ (wx|y , wx|y ) = wx||y by getting inverse opinions using Chapter 7
Equation 7.6. Thus, Case 2 and Case 3 are equal. Similarly, Case 1 and Case 4 are also

equal. This implies that an abduction on a proposition ‘y ← x’ can be interpreted as a

deduction on a proposition ‘x ← y’. Thus, given a single query shown in Table 7.1 we

have two choices of conducting this conditional depending on available priors. Therefore, a

decision making process, examining the availability of required priors in the rule and fact

187
Contents

Observation Priors
Query Operator
wx wy wx|y wx|y wy|x wy|x wxvac wyvac
√ √ √
Case 1 wy||x ⊚
y←x √ √ √ √
Case 2 wx||y ⊚
√ √ √
Case 3 wx||y ⊚
x←y √ √ √ √
Case 4 wy||x ⊚

Table 7.1: Possible Subjective Conditional Operations given x and y with Arbitrary Mutual
Dependency.

base, can be derived in form of a deterministic conjunctive logic program. Following this

scheme, for example, Case 1 corresponds to the rule

Calculate wy||x ← wx , wy|x , wy|x (7.7)

saying if there exist three facts wx , wy|x and wy|x calculate wy||x . Following Equation 4.2,

the rule for Case 1 (i.e., Equation 7.7) will just be denoted as follows.

wy||x ← wx ⊚ (wy|x , wy|x ) (7.8)

Other rules corresponding to the rest of the Cases 2 - 4 can be derived similarly. Note

that such rules are actually not about the context itself but about a mechanism how

the conditionals need to be handled and, therefore, those rules should be considered as

meta-rules. We consider those rules being part of the intensional layer. The CLIPS rule

engine provides prioritization of rules. By adjusting priorities of the meta-level rules and

other extensional rules, we can decide when the meta-level rules should be triggered. For

example, we could first apply the top priority group of extensional rules to infer occurrence

of some evidences that may be used for intensional layer. Then, we could intensionally set

meta-level rules to be the second priority group. Setting lowest priority group of extensional

rules, the inference result from intensional layer could be also used for extensional way of

inference. This way, the proposed approach can provide a combination of ‘extensional’ and

‘intensional’ approaches.

188
 Contents

7.5 Case Study I

7.5.1 Scenario Setting for Case Study I

Figure 7.1: Airport Surveillance Scenes for Case Study.

This section describes a case study for the application of forensic conditional reason-

ing using the proposed hybrid framework on metadata acquired using low level computer

vision analytics. Low level video analytic modules generate a rich set of metadata upon

which the proposed hybrid reasoning can be performed to detect more complex events.

This metadata captures the output of modules that detect ‘humans’ at a particular lo-

cation and the time, modules that detect humans passing, a pre-defined region etc. A

detailed description of these modules is outside the scope of this chapter. Figure 7.1 shows

frames from a video sequence captured from a visual surveillance system observing a typi-

cal airport scene. We also have access to information such as pre-annotated zones of stairs,

escalator, ATM machines, etc. Now assume we need to query this video sequence to detect

Chapter 7
all time instances when the escalator was non-functional. Please note, we do not have

any low-level computer vision module that can directly detect such an event. We wish to

formulate rules to detect this event based on circumstantial evidence of how the behavior

of the humans in the scene changes in response to the escalator not working.

In this case study we will show how we encode rules in the proposed hybrid reason-

ing framework to capture our knowledge of how we expect people behavior to change in

response to this event, and further show how we can use this knowledge to reason about

189
Contents

instances when the escalator is not-functional, given certain observations. Consider the

following rules :

Conditional Knowledge Model - Case1

• Rule 1 - "if escalator is working, people usually use escalator" or

• Rule 2 - "if escalator is not working, people usually use stairs".

Above knowledge can be seen as rules such that

‘people use escalator ← escalator working’ or

‘people use stairs ← ¬ escalator working’.

At the same time we could also think about rather deductive knowledge as follows:

Conditional Knowledge Model - Case2

• Rule 3 - "if people use escalator, usually escalator is working" or

• Rule 4 - "if people use stairs, usually escalator is not working"

Above knowledge can be seen as rules such that

‘escalator working ← people use escalator’ or

‘¬escalator working ← people use stairs’

In this scenario, knowledge about when people use the escalator or the stairs is essential.

Therefore, based on the output of low level vision modules, we need to derive rules to

reason about the tendency of ‘people use escalator’ or ‘people use stairs’. Consider the

following rules:

Observational Knowledge Model

• Rule 5 - "if a person is on escalator, he/she is using escalator "

• Rule 6 - "if a person is on stairs, he/she is using stairs "

Above knowledge can be seen as rules such that

‘he/she uses escalator ← person on escalator zone’ or

‘he/she uses stairs ← person on stairs zone’

190
 Contents

These rules capture our assumption that humans tend to always be in a state of ‘action’

when interacting with certain structures within a building. Therefore, when a low level

vision module detects a human, at a particular time instant, on a structure such as stairs

or escalator, we assume that the human is in the process of ‘using’ it to go somewhere.

Once such an evidence is built, we can assess the plausibility (opinion in Subjective Logic)

about escalator’s working or not using either Case 1 (abductive rules) or Case 2 (deductive

rules) type of rules.

7.5.2 Rule Modeling

This section deals with rule modeling for the scenario, based on the system setting de-

scribed in Section 7.5.1. In both conditional knowledge models shown in Section 7.5.1,

the observation about people use escalator or stairs serves as basis for the assessment.

Therefore, we first start with observational knowledge model described in Section 7.5.1.

Assuming opinions about observational metadata from human detection and annotation,
HumanDetector
we will use opinions such as whuman(H Annotation
, wescalator(E Annotation
i ,LocH ,Tj )i 1 ,LocE 1
,Tj ) , wstairs(S1 ,LocS 1
,Tj ) ,

etc (where for example, the first notation says, there is an opinion from HumanDetector

about a proposition human (i.e., a human exists, more concretely) with id Hi and local-

ization information LocHi at time Tj ). Depending on whether or not a human is ‘within’

a pre-annotated zone Zonek in the scene, low level modules also generate an opinion of
GeometryAgent
the form wwithin(H i ,Zonek ,Tj )
. We combine these opinions with the conjunction operator ∧

[93, 97] of subjective logic as follows.

Chapter 7
wa_human_uses_escalator(Hi ,Tj ) ←
(7.9)
GeometryAgent
HumanDetector
(whuman(Hi ,LocH ,Tj )
∧ wescalator(E
Annotation
1 ,LocE ,Tj ) ∧ wwithin(Hi ,E1 ,Tj ) )
i 1

wa_human_uses_stairs(Hi ,Tj ) ←
(7.10)
GeometryRule
HumanDetector
(whuman(H i ,LocH ,Tj )
∧ wstairs(S
Annotation
1 ,LocS ,Tj ) ∧ wwithin(Hi ,S1 ,Tj ) )
i 1

191
Contents

While above Rule 7.9 and Rule 7.10 will fire for every single human detection, we are

interested in accumulating all the instances of evidences from these rules, to get a single

opinion about ‘people_use_escalator’. An opinion from Rule 7.9 or Rule 7.10 is too weak

evidence to judge the proposition ‘people_use_escalator’. Therefore, we first apply the

‘reputation (discount)’ operator ⊗ [99] (see Definition 41) to every single detections as

follows.

byHi
wpeople_use_escalator(Tj)
←
a_human_uses_escalator
(wa_human_uses_escalator(Hi ,Tj ) ⊗ wpeople_use_escalator )
(7.11)
byHi
wpeople_use_stairs(Tj)
←
a_human_uses_stairs
(wa_human_uses_stairs(Hi ,Tj ) ⊗ wpeople_use_stairs )

Then, we need to fuse each of decisions made from above Rule 7.11. Using consensus

operator ⊕ [93, 94] in subjective logic (see Definition 30 Section 5.6.3), we will combine

both cases of people use escalator or stairs into one observational opinion. The encoded

rule is as follows.

wpeople_use_escalator(Tj ) ←
⊕ n (7.12)
byHi byHi
(⊕ni=1 wpeople_use_escalator(Tj)
) (⊕i=1 ¬wa_people_use_stairs(Tj)
)

Rule 7.12 cumulates every single human detection’s contribution to the opinion about ‘a

human uses escalator’. In the similar way, it also cumulates every single human detec-

tion’s contribution to the opinion on ‘human uses stairs’. Both cumulated opinions are

again fused by consensus operator. In the context that we fuse both positive and negative

evidences, it resembles the subjective logic based ‘default inference’ (see Definition 35).

Furthermore, this is also similar to the binomial observation that deals with number of

positive and negative evidences (see Section 3.1.3), but with uncertainty consideration.

For the simplicity of illustrative explanation, assume that we had perfect human detector

and perfect geometry agent, so every results from Rule 7.9 and Rule 7.10 were definite true.

192
 Contents

Figure 7.2: Examples of Applying Rule 7.12 based on Different # of Observations.

a_human_uses_escalator a_human_uses_stairs
Setting both the discount opinions as wpeople_use_escalator = wpeople_use_stairs =
byHi byHi
(0.5, 0, 0.5), every single opinion of wpeople_use_escalator(Tj)
and wpeople_use_stairs(Tj)
also

becomes (0.5, 0, 0.5). Then, based on this assumption, Figure 7.2 shows examples of cal-

culating opinions about ‘people_use_escalator’ given different number of observations at

a given time Tj .

As explained in Section 7.4.1 these rules belong to ‘extensional layer’. Now, to model

the conditional knowledge models from Section 7.5.1 that work at the ‘intensional layer’,

we need to collect certain prior opinions. In our case, by setting x as escalator_working

and y as people_use_escalator, we could assign opinions for the items shown below.

•wescalator_working|people_use_escalator •wescalator_working|¬people_use_escalator

•wpeople_use_escalator|escalator_working •wpeople_use_escalator|¬escalator_working Chapter 7

•wescalator_working
vac •wpeople_use_escalator
vac

Once Rule 7.9 ∼ Rule 7.12 are triggered, based on available information among above

priors and according to Table 7.1, appropriate conditional reasoning rules are dynamically

triggered. Although, there exist formal ways of extracting priors from data, we will for this

case study, use pre-defined subjective prior opinions for the convenience of our discussion

and considering the scope of this chapter (note that, however, in fact the use of such a

193
Contents

Figure 7.3: Graphical Model of used Conditional Knowledge.

Figure 7.4: Illustration of Abductive Conditional Reasoning for Case Study I

(see http://persons.unik.no/josang/sl/Op.html).

subjective opinion also makes sense in the view of subjectivism in Bayesian, that uses

subjective priors as discussed in Section 3.1.3, refer to chapters on ‘Comparing Bayesian

and Frequentist Inferences’ of [33] for further explanation).

Example 8. (Is the Escalator Working?) . In the case of defining usual tendency

(base rate) of escalator’s working to 90%, opinion about people would use escalator when an

escalator is working to (0.93,0.01,0.06) and opinion about people would still use escalator

194
 Contents

even if an escalator is not working to (0.05,0.9,0.05), following rule will be triggered.

w ←
escalator_working(E1 ,Tj )||people_use_escalator(Tj )

wpeople_use_escalator(Tj ) ⊚(wpeople_use_escalator(Tj )|escalator_working(E1 ,Tj ) , (7.13)

vac
wpeople_use_escalator(Tj )|¬escalator_working(E1 ,Tj ) , wescalator_working(E )
1 ,Tj )

Figure 7.3 depicts graphical model of introduced conditionals for the case study and Fig-

ure 7.4 shows an illustrative example of Rule 7.13 in the case of observational opinions

from Rule 7.9 ∼ Rule 7.12 were as depicted in Figure 7.2. In the case we observed 1 person

using escalator and 1 person using stairs, by the base rate and prior, we get more belief on

escalator working but with high uncertainty. As we observe more people using stairs, we

get stronger disbelief on the proposition ‘escalator_working’.

7.5.3 Experimental Proof of Concept

As proof of concept, a demonstrator has been built. CLIPS rule-engine was used to imple-

ment subjective logic extended logic programming for the proposed reasoning framework.

Eclipse platform was used to integrate other modules such as opinion visualization, video

parser, metadata parser and manipulation UI, etc. At this stage of case study, we used 300

seconds of airport video surveillance data. Then we manually annotated human detection

metadata such as bounding box and time information per seconds as shown in Figure 7.1

with a background annotation (e.g. escalator, stairs, etc). We first set the necessary pri-

ors as shown in Figure 7.4. To consider each of single human detection based judgement

as a reasonably weak and uncertain evidence, we used strong discount opinion factor as Chapter 7
a_human_uses_escalator a_human_uses_stairs
wpeople_use_escalator = wpeople_use_stairs = (0.3, 0.05, 0.65). For more intuitive case

study, we first considered a real-time analysis setup. Meaning that we loaded Rule 7.9 -

7.13 and injected low level metadata along the video playing in synchronized manner. To

see how belief changes over time, we additionally introduced a rule as follows.

195
Contents

Figure 7.5: Screen Captures demonstrating Belief Revision on the Proposition ‘the escala-
tor is not working’.

196
 Contents

Accumulated_Over(T ,T )
1 n
wpeople_use_escalator(Tj)
←
(7.14)
Accumulated_Over(T ,T
1 n−1 )⊕
wpeople_use_escalator(Tn−1 )
(wa_people_use_stairs(Tn ) )

Then we kept tracking the change of the derived opinion about the proposition ‘escala-

tor is not working’. When the system gets opinions about human detection, rules were

automatically triggered. Figure 7.5 shows the change of opinion on ‘escalator is not work-

ing’ along a timely sequential multiple observations. For example, the 4th observation in

Figure 7.5 shows a scene that many people were using stairs while nobody was using the

escalator. In this case, the system correctly computed a strong opinion that the escalator

is not working. While, 5th observation shows an example of retracting current opinion

upon observation of ‘people use escalator’. This demonstrates the ‘non-monotonicity’ in

terms of ‘belief revision’ over time. The demo on this case study shows that it well simulate

human intuition.

7.6 Case Study II

In this section, we further explorer the proposed approach with a virtual scenario.

7.6.1 Scenario Setting for Case Study II

Let us consider a conceptual scenario that a security personnel wants to find a scene of

parking a vehicle, that seems to be done by a ‘novice driver’. Let us also assume a virtual

system setting such that: parking slots are pre-annotated with a polygon, vision analytics

can detect moving and stopped vehicles with motion vector and geometric localization Chapter 7
information (i.e., coordinate, width and height). Given the assumption, Figure 7.6 shows

some of sequential video footages 2 . In Figure 7.6, parking slots are annotated with blue

polygons and vehicles with no motion vectors are labelled with pink ovals. Vehicles with

motion vectors are marked with red boxes and red arrows with its vehicle ‘id’. Humans

2
The video footages are inspired and intensionally synthesized based on the original YouTube video on
http://www.youtube.com/watch?v=_rHY1qKLLws&feature=related.

197
Contents

Figure 7.6: Scenario Setting for Case Study 2.

- Two Cars, C1 and C2. An Empty Parking Slot EP S1.-

can easily infer lots of implied high level semantics of Figure 7.6 as follows.

‘It seems that the car C1 was trying to park to an empty parking slot EP S1 several times.

In the meanwhile the car was trying to park, another car C2 appeared. The newly appeared

car C2 seems to be blocked due to the car C1, that is still trying to park (see t9 ∼ t12 in

Figure 7.6). Because the car C1 was very sluggish at parking, it seems that the driver of the

car C1 was a novice driver. If the driver of C1 was a novice driver, there is a possibility

that the car C1 could have scratched other cars. If such case was really occurred, the car

C2 would be a good witness to get more information on the scene and on the driver of C1,

etc.’

In this case study, we will show how we encode some of above knowledge as rules in the

198
 Contents

proposed hybrid reasoning framework3 . Consider the following rules :

Conditional Knowledge Model - Case1

• Rule 1 - "if a novice driver is driving, he/she is not good at parking" or

• Rule 2 - "if a normal driver is driving, he/she is reasonably good at parking".

Above knowledge can be seen as rules such that

‘¬ good at parking ← novice driver’ or

‘good at parking ← ¬ novice driver’.

At the same time, we could also think about rather deductive knowledge as follows:

Conditional Knowledge Model - Case2

• Rule 3 - "if he/she is not good at parking, usually a novice driver is driving" or

• Rule 4 - "if he/she is reasonably good at parking, usually not a novice driver"

Above knowledge can be formulated as rules such that

‘novice driver ← ¬ good at parking’ or

‘¬ novice driver ← good at parking’

In this scenario, the knowledge about ‘whether a parking was good or not’ is essential.

Therefore, based on the output of low level vision modules, we need to derive rules to

get opinions on ‘good at parking’ or ‘not good at parking’. Consider the following rules:

Observational Knowledge Model

• Rule 5 - "if a car comes in to, and goes out of an empty parking slot multiple times,

it probably is not a good parking"

Chapter 7
• Rule 6 - "if a car comes in to, and goes out of an empty parking slot few times,

it probably is a good parking"

Above knowledge can be seen as rules such that

‘¬ good parking ← areas of car and empty slot are overlapped multiple times’ or

‘good parking ← areas of car and empty slot are overlapped only f ew times’
3
Note that, in this scenario, by assumption, we do not have any low-level computer vision module that
can directly detect the high level semantics such as ‘novice driver’, etc.

199
Contents

The implication of Rule 5 and 6 is similar to the one of binomial observations in the context

that the Rule 5 and 6 also observe multiple times of positive evidences (see Section 3.1.3

for more details on binomial observation). However, in our context, Rule 5 and 6 only

matter positive evidences (i.e., occurrence of ‘geometric overlap between the car and the

empty parking slot of interest’). It can be also stated as ‘the more times the car tries to

park, the higher is the likelihood that it was driven by a novice driver’ which is linguistically

similar to the ‘vague rules’ discussed in the previous Chapter 6. However, while Rule 5

and 6 deal with the number of occurrence of the event (defined in rule head) itself, the

vague rules discussed in Chapter 6 deal with an attribute implied in the condition (rule

body) to occur an event (defined in rule head) 4 (namely, vague rules in Chapter 6 concern

about intrinsic parameters within a ‘predicate’ based proposition5 ). Therefore, the Rule 5

and Rule 6 should be designed as an increasing function of the number of ‘parking trial’.

Namely, the more observational instances of evidence on ‘parking trial’ should entail the

stronger opinion on ‘not good at parking’.

7.6.2 Rule Modeling

Based on the discussion of previous Section 7.6.1, this section deals with actual rule mod-

eling for the given scenario. In both conditional knowledge models shown in Section 7.6.1,

the observation about the goodness of parking serves as basis for the assessment. Therefore,

we first start with observational knowledge model described in Section 7.6.1. Assuming

opinions about observational metadata from vehicle detection and annotation, we will use
V ehicleDetector
opinions such as wvehicle(V Annotation
i ,LocV ,M VV
i 1
,Tj ) , wparking_slot(P Sk ,LocP S k
,Tj ) , etc (where for exam-

ple, the first notation says, there is an opinion from ‘VehicleDetector’ about a proposition

‘vehicle’ (i.e., a vehicle exists, more concretely) with id V1 , localization information LocV1

and motion vector information M VV1 at time Tj ).

4
In Chapter 6, ‘direction’ and ‘distance’ are the concerned attributes.
5
See Section 3.2.2 for detailed explanation about the difference between normal propositional logic and
first-order predicate logic. Also note that, the rule engine used in our framework is based on the
first-order predicate logic (see Section 4.3.2).

200
 Contents

To infer whether a parking slot is occupied or not, we check if a stopped vehicle is

within the parking slot, examining whether the motion vector of the car is Null. Therefore,

we get a rule as follows.

byVi
woccupied_parking_slot(V i ,P Sk ,LocP S ,Tj ) ←
k
GeometryAgent
V ehicleDetector
(wvehicle(Vi ,LocV ,N U LL,Tj )
∧ wparking_slot(P
Annotation
Sk ,LocP S ,T ) ∧ wwithin(Vi ,P Sk ,Tj )
) (7.15)
i k j
byVi byVi
wempty_parking_slot(P Sk ,LocP S ,Tj ) ← ¬woccupied_parking_slot(_,P Sk ,LocP S ,Tj )
k k

As indicated by the agent markup ‘byVi ’, Rule 7.15 derives an opinion about the proposition

‘empty_parking_slot(P Sk , LocP Sk , Tj )’ only when some amount of overlap occurred by

a vehicle ‘Vi ’ and a parking slot ‘P Sk ’. By default, however, every parking slots have

chances of being empty, even if there are no clues about occupancy. Consider following

truth assignment where, DT indicates ‘default true’, where any value reasonably close to

‘true’ but rather ‘uncertain’ can be a candidate of DT as long as we fix the value throughout

reasoning pipeline (see Definition 16 and Definition 35). In this case, following examples

in Chapter 5, we assign DT = (0.5, 0, 0.5) (see Section 5.8).

ϕRule [wempty_parking_slot(SPk ,LocP S ,Tj ) ← wparking_slot(P Sk ,Tj ) ] = DT (7.16)

k

In this setup, following the ‘default reasoning’ scheme introduced in Chapter 5 (see Defi-

nition 35 for detail), we can define a rule as follows.

byDef aultInf erence

wempty_parking_slot(P ←
Chapter 7
Sk ,LocP S ,Tj )
k
(7.17)
byVi
(wparking_slot(P Sk ,Tj ) ∧ DT ) ⊕ (∧ni wempty_parking_slot(P Sk ,LocP S ,Tj ) )
k

Now, we introduce a rule to know whether a car is trying to park. Consider a rule as

follows.

byV
_parking(Vi ,P Sk ,Tj ) ←
i,j
wtrial_of
byDef aultInf erence GeometryAgent
V ehicleDetector
(wvehicle(Vi ,LocV ,N U LL,Tj )
∧ wempty_parking_slot(P Sk ,LocP S ,Tj ) ∧ woverlap(V i ,P Sk ,Tj )
)
i k
(7.18)

201
Contents

While above Rule 7.18 will fire for every single parking trial, we are interested in accu-

mulating all the instances of Rule 7.18, to get a single opinion about ‘good_at_parking’.

Therefore, the accumulation should form a decreasing opinion function6 in terms of number

of observations. To achieve this, we first apply the ‘reputation (discount)’ operator ⊗ [99]

(see Definition 41) to every single parking trial detection as follows, because a single in-

stance of ‘trial_of_parking’ is a weak evidence to judge the proposition ‘good_at_parking’.

byV
i,j
wgood_at_parking(Vi ,P Sk ,Tj )
←
(7.19)
byV trial_of _parking
_parking(Vi ,P Sk ,Tj ) ⊗ wgood_at_parking )
i,j
(wtrial_of

Then, we need to fuse each of decisions made from above Rule 7.19. By default, however,

it would be expected to be a normal parking, and the number of parking trials would form

a negative evidence against good parking. Therefore, using consensus operator ⊕ [93, 94]

in subjective logic (see Definition 30 Section 5.6.3), the encoded rule becomes as follows.

⊕ byV
wgood_at_parking(Vi ,P Sk ,Tj ) ← DT ¬(⊕nj=1 wgood_at_parking(V
i,j
i ,P Sk ,Tj )
) (7.20)

Rule 7.20 cumulates amount of contribution of every single instance of ‘trial_of_parking’,

to the opinion on ‘good at parking’. By the nature of consensus operator ⊕ and ¬, the more

instances from Rule 7.18 and Rule 7.19 (i.e., ‘trial_of_parking’) we detect, the weaker belief

on ‘good_at_parking’ we get. For the illustrative explanation of this aspect, assume that we

had perfect vehicle detector and perfect geometry agent, so every results from Rule 7.19
trial_of _parking
were definite true. Setting the discount opinion as wgood_at_parking = (0.33, 0, 0.67)7 ,
i,j byV
every single opinion of wgood_at_parking(Vi ,P Sk ,Tj )
also becomes (0.33, 0, 0.67). Then, based

on this assumption, Figure 7.7 shows examples of calculating opinions wgood_at_parking by

the by Rule 7.15 ∼ Rule 7.20, given different number of parking trials of Vi . In Figure 7.7,

as more instances of parking trials are considered, Rule 7.20 set stronger disbelief on

6
It becomes increasing opinion function in case we consider ‘NOT good at parking’.
7
The implication of this assignment is to consider at least three times of parking trial to decide possibility
of being a bad parking. Therefore, by intuition, we assigned 1/3 belief of full belief.

202
 Contents

Figure 7.7: Examples of Applying Rule 7.20 based on Different # of Observations on

‘good_at_parking’ proposition by the Vehicle ‘Vi ’.

‘good_at_parking’ proposition.

Thus far, we have reviewed rules belong to ‘extensional layer’. Now to address the

conditional knowledge models from Section 7.6.1 that work at the ‘intensional layer’, we

need to collect certain prior opinions. In our case, by setting x as novice_driver and y as

good_at_parking, we could assign opinions for the items shown below.

•wnovice_driver|good_at_parking •wnovice_driver|¬good_at_parking

•wgood_at_parking|novice_driver •wgood_at_parking|¬novice_driver

•wnovice_driver
vac •wgood_at_parking
vac
Chapter 7
Once Rule 7.15 ∼ Rule 7.20 are triggered, given available priors, one of following meta-level

rules (7.8) in ‘intensional layer’ are triggered.

w ←
novice_driver(Vi )||good_at_parking(Vi )

wgood_at_parking(Vi ) ⊚(wgood_at_parking(Vi )|novice_driver(Vi ) , (7.21)

vac
wgood_at_parking(Vi )|¬novice_driver(Vi ) , wnovice_driver(V )
i)

203
Contents

wgood_at_parking(Vi )||novice_driver(Vi ) ←

wgood_at_parking(Vi ) ⊚(wnovice_driver(Vi )|good_at_parking(Vi ) , (7.22)

wnovice_driver(Vi )|¬good_at_parking(Vi ) )

Note that, Rule 7.21 and Rule 7.22 use different priors, but returns an opinion on the same

proposition that is ‘novice_driver’. Namely, ‘wnovice_driver(Vi ) ’.

7.6.3 Illustrative Examples as Proof of Concept

In this section, we show conceptual reasoning example given the scenario, rules and as-

sumptions described thus far.

Example 9. (Novice Driver) . Assume that we have detected a vehicle with id C1 and a

parking slot EP S1 as depicted in Figure 7.6. Let us also assume that we have observational

opinions on ‘good_at_parking’ of the C1 to the slot EP S1 as shown in Figure 7.7. Assume

that 10% of drivers are novice driver in general. Consider the case we have intuition based

subjective opinions on the proposition ‘good_at_parking|novice_driver’ and also on the

proposition ‘good_at_parking|¬novice_driver’ as follows 8 .

•wgood_at_parking|novice_driver = (0.05, 0.85, 0.1, 0.5)

•wgood_at_parking|¬novice_driver = (0.9, 0.05, 0.05, 0.5)

•wnovice_driver
vac = (0.0, 0.0, 1.0, 0.10) (i.e., 10%)

then, according to Table 7.1, Rule 7.21 will be triggered. Given the assumptions and the

priors above, Figure 7.8 shows an illustrative example of applyging Rule 7.21.

In the case there was only one observational instance on ‘good_at_parking’, the com-

putational result yields more ‘disbelief ’ than ‘belief ’ about the proposition ‘novice_driver’.

However, as we get multiple parking trial based multiple instance of ‘good_at_parking’

opinions, it smoothly changes its opinion toward stronger belief about the proposition

‘novice_driver’. This shows that the reasoning results coincide with human intuition.
8
Note that, in fact, such subjective opinions (based on human intuition) can be regarded as subjective
Bayesian priors as discussed in Section 3.1.3. Refer to [33] for details of Bayesian statistics and its
comparison with traditional frequentist methods.

204
 Contents

Figure 7.8: Illustration of Abductive Conditional Reasoning for Case Study 2

(see http://persons.unik.no/josang/sl/Op.html).

Example 10. (Illegal Parking) . Consider the same situation as the one of Example 9.
Assume that we have detected a new vehicle with id C2. C2 slowly entered in the scene and
suddenly stopped after a while, as depicted in the footages whthin the period (t9 ∼ t12)
in Figure 7.6. Assume a set of rules about determining ‘illegal parking’ and the opinion
assignments to the rules as follows.

ϕRule [willegal_parking(Vi ,Tj ) ← wvehicle(V

V ehicleDetector
i ,LocV i ,N U LL,Tj )
] = (0.2, 0, 0.8)
byVi
ϕRule [¬willegal_parking(Vi ,Tj ) ← woccupied_parking_slot(V ,Tj ) ] = (0.9, 0, 0.1)
Chapter 7
i ,P Sk ,LocP S k

byVi
ϕRule [¬w¬illegal_parking(Vi ,Tj ) ← wblocked_vehicle(V i ,Tj )
] = (0.9, 0, 0.1)

GeometryAgent
wblocked_vehicle(Vi ,Tj ) ← wnovice_driver(Vi ) ∧ wmotion_vector_bef ore_stop_heading(Vi ,V j)
(7.23)

byVi
where, woccupied_parking_slot(Vi ,P Sk ,LocP S ,Tj ) is the opinion determined by the Rule 7.15
k

and wnovice_driver(Vi ) is the opinion after Rule 7.21 and Rule 7.22 are triggered in the

‘intensional layer’.

205
Contents

Consider opinion assignments for the case that C1 has tried to park 4 times. For
this example, we will take the corresponding opinion wnovice_driver from the Figure 7.8 of
Example 9. Also assume that our geometry agent is robust and reliable.

#of inst.=4
ϕRuleEval [wnovice_driver(C1) ] = (0.32, 0.22, 0.46)
GeometryAgent
ϕf act [wmotion_vector_bef ore_stop_heading(C2,C1) ] = T = (1, 0, 0)
V ehicleDetector
ϕf act [wvehicle(C2,Loc C2 ,N U LL,Tj )
] = T = (1, 0, 0)

Given this setting, the inference on ‘C2 parked illegally’ can be conducted as follows.

clsldi (ϕ)(willegal_parking(C2) )
= [U ⊔ ((1, 0, 0) · DT1 )] ⊕ ¬[U ⊔ (((0.32, 0.22, 0.46) · (1, 0, 0)) · DT2 )]
= [U ⊔ (1, 0, 0) · (0.2, 0, 0.8)] ⊕ ¬[U ⊔ ((0.47, 0.22, 0.31) · (0.95, 0, 0.05))]
= [U ⊔ (0.47, 0, 0.53)] ⊕ [U ⊔ ¬(0.55, 0.22, 0.23)]
= (0.47, 0, 0.53) ⊕ (0.22, 0.55, 0.23)
= (0.35, 0.46, 0.19)

The entailed opinion (0.35,0.46,0.19) can be linguistically interpreted as ‘it may not be

an illegal parking, but with some amount of uncertainty’. In the same way, the other

stopped vehicles labelled with pink ovals can be regarded as ‘illegal parking’ when they

are not parked within a predefined parking slot (see Figure 7.6). This example shows that

the inference result from the ‘intensional layer’ can be again used for another semantic

reasoning in the ‘extensional layer’.

7.7 Chapter Summary

In summary of lessons learned in this chapter, we have demonstrated how conditional

premises can be factorized and handled in a Bayesian sense by using subjective logic’s

deduction and abduction operators. The main advantage of the proposed hybrid approach

is that it offers more choices to handle the trade-offs between expressive power, semantic

clarity and computational efficiency. However, there are still several open issues such as

206
 Contents

how to extend this concept to multinomial frame based event, how to automatically get the

prior opinions, etc.

In this chapter, we proposed a hybrid approach to conditional evidence fusion that

mates extensional and intensional approaches of conditional knowledge representation and

inference under uncertainty, for high level semantic analysis of visual surveillance data. For

uncertainty handling, subjective logic was adopted on top of rule-based framework. The

main advantage of the proposed hybrid approach is that it offers more choices to handle

the trade-offs between expressive power, semantic clarity and computational efficiency. For

highly varying and complex compositional events in visual surveillance data, we could ben-

efit from the expressive power and modularity of the extensional approach, while leveraging

the intensional interpretation of conditionals to facilitate bi-directional inference.

There are, however, still several open issues such as how to extend this concept to

multinomial frame based event, how to automatically get the prior opinions, etc. Although

we still need to extend this concept to large scale data. We advocate that this work showed

the potential of the proposed approach. One of interesting properties of the system is

that, unlike traditional probability based conditional reasoning, this approach allows for

representing lack of information about a proposition. We could assign priors with lack

of information, and observations can also be represented with any degree of ignorance,

therefore we believe this better reflects human intuition and real world situations. Another

beneficial property is the flexibility of defining rules by taking a layered approach to leverage

advantages of both extensional and the intensional approaches.

Chapter 7

207
208
8 Logical Abduction under Uncertainty

Thus far, our discussions and reasoning examples have dealt reasoning scenarios that can be

triggered by a ‘specific’ question to be inferred. In reality, however, humans tend to reason

about a situation rather in a diagnostic manner, without setting a specific proposition to

be reasoned about but just by considering set of observations of their interest. Such a

reasoning is called logical abduction and this chapter represents our approach to achieve

the abductive logical reasoning.

8.1 Introduction

This chapter proposes an approach to enable automatic generation of probable semantic

hypotheses for a given set of collected observations for forensic visual surveillance. As video

analytic power exploited in visual surveillance is getting matured, the more automatically

generated intermediate semantic metadata became available. In the sense of forensic reuse

of such data, the majority of approaches have been focused on specific semantic query

based scene analysis. However, in reality, there are often cases in which it is more natural

to reason about the most probable semantic explanation of a scene given a collection

of specific semantic evidences. In general, this type of diagnostic reasoning is known as

abduction1 . To enable such a semantic reasoning, in this chapter, we propose a layered

reasoning pipeline that combines abductive logic programming together with backward
Chapter 8

and forward chaining based deductive logic programming. To rate derived hypotheses, we

1
note that, while the subjective logic ‘abduction’ operator discussed in Chapter 7 deals the ‘bidirectional
interpretation’ of a conditional rule, ‘logical abduction’ deals a diagnostic mechanism to find most
probable hypothesis given set of observations

209
Contents

apply subjective logic. We present a conceptual case study in a distributed camera based

scenario. The case study shows the potential and feasibility of the proposed approach for

forensic analysis of visual surveillance data.

8.2 Background and Motivation

As discussed in Section 1.1, recent advances in computer vision technology have made

it possible to analyze specific patterns of abnormal human or object behavior in visual

surveillance. When it comes to forensic reuse of such analytics, however, it is still beyond

the sufficient intelligence due to the variety of possible semantics and complex plots implied

in surveillance scenes. Such forensic semantic analysis of visual surveillance, therefore,

requires intelligent reuse of low level vision analytic results in a context sensitive manner.

To address this aspect, there has been some work on the use of declarative logic

formalism to represent and reason about high-level contextual semantic knowledge which is

referenced to as ‘extensional’ approach. In Section 2.4.3, we have reviewed such approaches

appear in literature. After the review and comparison, in Chapter 4, we have also proposed

an ‘extensional’ approach that uses logic programming with subjective logic theory.

In such approaches, reasoning is triggered with a specific high level semantic query

to decide the existence of metadata patterns that semantically satisfy the given query.

However, in reality, there are often cases require to ask about the most probable semantic

explanation of a scene based on specific evidential semantic observations. That is, when

the former approach is applied to semantic retrieval, the security personnel is required to

know what semantics to ask. While the later would, ideally, let users collect evidences

of their interest and do simplify queries into a single ‘why?’ or ‘what?’ type of query.

Enabling such type of reasoning would be advantageous especially in the situation that

the semantic ambiguity of a scene is highly implied or multiple semantic interpretations of

a scene are possible. In general, this type of diagnostic reasoning is known as abduction

[131]. In this chapter, we present a layered pipeline of reasoning flow that adopts abductive

210
 Contents

logic programming together with backward and forward chaining based deductive logic

programming and combine subjective logic [92, 93] for the rating of derived diagnostic

hypotheses. We then present a conceptual case study from a distributed camera based

scenario to show the potential and feasibility of the proposed approach in the sense of

diagnostic forensic.

8.3 Related Work

To achieve better expressive power and flexibility on context modelling and reasoning of vi-

sual surveillance data, there has been some work on the use of declarative logic formalisms

and on the use of different uncertainty handling formalisms. As we have already reviewed

in Section 2.4.3, Akdemir et al. [13] used an ontology for human activity recognition, but

without uncertainty handling. Shet et al. [152, 153] proposed a system that adopts Prolog

based logic programming extended with the bilattice framework [69] for default logic [143]

based situation reasoning and human identity maintanance. Jianbing et al. [110] used

rule-based reasoning with Dempster Shafer’s belief theory [149] for a bus surveillance sce-

nario. Anderson et al. [17] used fuzzy logic [177] to model human activity for video based

eldercare. Based on our previous work, [78, 81, 80, 82, 83, 79], in this dissertation, we

proposed the use of logic programming and subjective logic [92, 93] to encode contextual

knowledge with uncertainty handling, then demonstrated bidirectional conditional infer-

ence and default reasoning for visual surveillance scenarios. In the sense of reasoning, such

logic framework based approaches require a specific query to reason about, and yield an

output on the truth of the given query.

However, in the sense of forensic use of visual surveillance data, rather diagnostic

reasoning so called ‘logical abduction’ is also required. In such diagnostic reasoning, the
Chapter 8

initial input to the reasoning system would be desired to be a selected set of interested

event, then the query can be converged into a single ‘why?’ or ‘what?’ type of query to

get explanations or hypotheses to the best of given knowledge base. There has been also

211
Contents

some work on the use of abduction. Espinosa et al. [132] showed description logic [2]

based abduction example getting explanations about an image given ontology based image

annotation. Ferrin et al. [59] also used description logic for visual surveillance to show that

abduction can be used for realtime ‘car theft’ event detection. While both work showed the

potential of abduction in interpreting media contents, the use of description logic makes

it hard to consider uncertainty handling or extending the reasoning framework so that it

can also conduct other types of reasoning such as default reasoning. In contrast to the

previous work, we take more pragmatic approach based on logic programming including

abductive logic programming and rate results with subjective logic with more focus on

situation analysis rather than atomic event analysis.

8.4 Abductive Reasoning

Abduction means a method of logical inference finding plausible explanations (hypotheses)

for a given set of observations. Peirce [131] showed that it can be seen as inverse modus

ponens and formulated it as Σ ∪ ∆ |= Γ, where Σ represents the background knowledge, Γ

represents observations, ∆ represents the explanations to be computed and the symbol ‘|=’

represents logical entailment. During the 90s this aspect has been solidly studied in the field

of AI in the context of logic programming and came up with abductive logic programming

[50]. In this section we provide a brief introduction to abductive logic programming.

8.4.1 Abductive Logic Programming

An abductive logic programming theory is defined as a triple (P, A, IC) consisting of a

logic program P , a set of ground abducible atoms A and a set of classical logic formulas

IC, called the integrity constraints, such that no p ∈ A occurs in the head of a rule of P .

The logic program P also consists of rules R and literals L, namely, P = (R, L). Then

for a given query observation Q (note that, in abductive logic programming, Q is not a set

of observations but rather a single observational query), an abductive explanation for the

212
 Contents

observation is a set ∆ ⊆ A such that : 1) P ∪ ∆ |= Q. 2) P ∪ ∆ |= IC. 3) P ∪ ∆ is

consistent. This mean that the derived explanation ∆ and P should be able to entail Q.

At the same time, should not violate the constraints IC. In the sense of Peirce’s definition

of abduction, (P, A, IC) corresponds to Σ and Q is the Γ and ∆ is the output. Following

shows a simple example of abductive logic programming.

Example 11. (Abduction) . Assume an abductive logic program (P, A, IC), where

P = (R, L) such that :

R ={ grass_is_wet ← it_rained
grass_is_wet ← the_sprinkler_was_on }
L ={ the_sun_was_shining }
A ={ it_rained , the_sprinkler_was_on }
IC ={ f alse ← it_rained, the_sun_was_shining }

then for a given observational query Q = grass_is_wet, the proper explanation

∆ = {the_sprinkler_was_on} should be derived because it is the consistent explanation

given the constraint IC and the fact the_sun_was_shining in P . (note that, in normal

logic programming the output is truth value about the query). There has been some work

on enabling the shown concept of abductive logic programming.

8.4.2 Abductive Logic Programming Frameworks

Most of the implementations of Abductive Logic Programming extend the SLD-resolution

[66] based computational model of logic programming. (note., resolution is a procedure

for showing whether or not a given set of facts are valid according to currently known rule

sets. The SLD-resolution is known to be a computationally efficient resolution algorithm).

SLDNFA [51] is one of well known algorithms to fulfill abduction itself. ACLP [101],

CIFF, SCIFF [6] and Asystem [125] are some of known implementations of abductive logic
Chapter 8

framework. Besides, ABDUAL and ProLogICA., etc., appear in literature but the project

sites are no longer appear. Indeed, it seems that no prominent benchmark or comparison of

those systems appear. Although some are based on Prolog, some of them are also based on

213
Contents

other languages such as ECliPSe language [57]. In this dissertation, considering availability

of executable binary, simplicity on system integration issues, we have used ACLP [101].

8.5 Proposed Approach

The objective of the presented framework is to enable abductive semantic reasoning and

its quantitative evaluation given multiple observational evidence input. For the core part

of the logical abduction, abductive logic programming plays an important role. As shown

in Section 8.4, however, abductive logic programming also requires a specific single query

to be explained and this is far from our intention. Although we could collect a set of

interested observations and put them as part of facts (literals L) in P of (P, A, IC), still

the reasoning should be triggered by a single query. To address this aspect, we take

layered pipeline concept throughout multiple observational input processing, abduction

and evaluation of derived hypotheses. To process the initial multiple observational input,

we adopt a forward chaining based rule engine to derive a possible set of semantics to be

asked. Unlike backward chaining based rule engine such as prolog that only tracks back

patterns that satisfy the given single query, a forward chaining based rule engine derives

all deductive logical conclusions given input literals and rule set at one execution. For

this part, CLIPS [1] rule engine was used. Once we get the list of possible semantics

to be asked and explained, for each of them we conduct the abduction and get a list of

explanations. For the abductive reasoning part, we adopt an implementation of abductive

logic programming framework. For the prototyping purpose, ACLP [101] implementation

of abductive logic programming was used due to its relatively simple structure among

other frameworks such as ASystem [125]. For evaluation of the derived hypotheses we

adopt subjective logic extended deductive rule based framework. Namely, for each of the

potential explanations, we conduct backward chaining based reasoning with subjective logic

based uncertainty evaluation. This layered pipeline is depicted in Figure 8.1. For input

video sources with analytic results, we first select a set of interested evidences to form Γ,

214
 Contents

Figure 8.1: The Abductive Reasoning Framework.

the set of observations which is then sent to the forward chaining based rule engine to

reason about possible semantics to be asked. For each of the semantics to be asked, we

form a single query Q which is sent to the abductive logic programming engine. Once we

get explanation set ∆, this is evaluated with the subjective logic based backward chaining

engine. Finally, the result is presented on explanation viewer.

8.6 Case Study

This section deals with an illustrative case study on the application of forensic abductive

reasoning in a typical distributed camera based scenario. We will assume metadata ac-

quired from low level computer vision analytics such as human detection, vehicle detection

and object detection, etc. As explained in Section 8.4 and Section 8.5, in our setup, such

metadata will be considered as logical facts (literals L, in the sense we care both positive

and negative facts) and will be put into logic program P together with context rules R

(note that, P = (R, L)). Assuming a security personnel examining a video scene for certain

forensic reason, the query for the abductive reasoning will also be assumed to be triggered

by Γ, the set of collected evidential metadata on his interest. Given the Γ, we first use

forward chaining based deductive logic programming to derive possible semantic queries

Qi to be explained and will be sent to the core abductive logic programming part to get
Chapter 8

the set of explanations ∆ij . Then finally, we will evaluate the resulting ∆ij with subjective

logic to get opinion attached set of explanations ∆op

ij .

215
Contents

Figure 8.2: The Setup for Case Study Scenario with 3 Cameras Deployed.

Example 12. Scenario (Why the person suddenly went outside ?) . Assume a

typical distributed visual surveillance with three cameras deployed as shown in Figure 8.2.

Cam1 monitors the lobby of the building, Cam2 and Cam3 monitor front of the gate in

different outside views. Suppose a security personnel was watching a scene of Cam1 and

got interested in why the person who was sitting at the front desk suddenly stood up and

went outside the building. From the scene of Cam1, he collected metadata such as the

person’s standing up and heading to the predefined gate zone as depicted with the box

labelled with Γ in Figure 8.2.

Γ(t1..t2) = { human(pid1), gate(gate1), standing_up(pid1), heading(pid1, gate1) }

Given the Γ, suppose the system had the following deductive logic program segment.

216
 Contents

go_outside(P ) ← human(P ), gate(G), heading(P, G)

The use of the forward chaining rule engine with the given Γ will then assert new semantic

derivations Qi as follows.

Q1 = go_outside(pid1)

Now, each Qi will be considered as the query to be explained by the abductive

reasoning framework. Suppose also that, at the same time of the event in Cam1, there

were a vehicle parked in front of the gate and also a chair beside the vehicle. When the

vehicle leaves, it hit the chair and the chair fell down on the road. These are captured

in Cam3 in Figure 8.2 and the analytic metadata of the scene is stored in form of literals

L. Consider an abductive logic program (P, A, IC) containing context rules R, integrity

constraint IC and abducibles A as follows (note that, P = (R, L)).

L = { vehicle(car1), object(obj1), gate(gate1) }

R={ go_outside(X) ← to_do_smt(X, Y )

go_outside(X) ← saw_smt(X, Y )
to_do_smt(X, Y ) ← take_a_veh(X), vehicle(Y )
to_do_smt(X, Y ) ← buy_smt(X), mart(Y )
saw_smt(X, Y ) ← veh_obj_acc(Y ), mng_veh_obj_acc(X)
saw_smt(X, Y ) ← veh_prs_acc(Y ), mng_veh_prs_acc(X)
veh_obj_acc(Y ) ← vehicle(Y ), object(Z).
veh_prs_acc(Y ) ← vehicle(Y ), person(Z) }

IC = { f alse ← buy_smt(X), take_a_veh(X)

f alse ← buy_smt(X), mng_veh_obj_acc(X)
f alse ← buy_smt(X), mng_veh_prs_acc(X)
f alse ← take_a_veh(X), mng_veh_obj_acc(X)
f alse ← take_a_veh(X), mng_veh_prs_acc(X) }
Chapter 8

A = { buy_smt, take_a_veh, mng_veh_obj_acc, mng_veh_prs_acc }

The rule set R means that, a person may go outside to do something outside or because

the person saw some event. Taking a vehicle or buying something can be seen as doing

217
Contents

something in case that there was a vehicle to take or a mart to go to buy something.

Similarly, if there was an accident in which a vehicle and an object are involved and the

person is trying to manage the accident somehow, it implies that the person noticed the

event. This is also same in the case of an accident in which a vehicle and a person were

involved. In these cases, both a vehicle and an object or both a vehicle and a person

should be existing at the same time. As a constraint, the integrity constraint rule IC

means that some of those event should happen exclusively. Given the knowledge, the

candidate semantic predicates we may want to derive by the execution of abduction are

designated in the set of abducibles A. Now, with the given above abductive logic program

and an input query Q1 = go_outside(pid1), the abductive logic procedure will derive

explanations as follows.

∆11 = { not(mng_veh_prs_acc(pid1)), not(mng_veh_obj_acc(pid1)),

not(buy_smt(pid1)), take_a_veh(pid1) }

∆12 = { not(take_a_veh(pid1)), not(buy_smt(pid1), mng_veh_obj_acc(pid1) }

In above scenario, we get two hypotheses sets for a given query Q1 . In the next step, we

examine each of the elements in ∆11 and ∆12 by the use of the subjective logic extended

logic programming as explained in Section 4.3.2. Consider following logical rule segments.

wtake_a_veh(P1 ,V1 ) ←
Human_Detector V ehicle_Detector
whuman(H1 ,T1 ,X1 ,Y1 ) ∧ wvehicle(V1 ,T1 ,X2 ,Y2 ) ∧ (8.1)
Geometry_M anager
woverwlap(X1 ,Y1 ,X2 ,Y2 )

wmng_veh_obj_acc(P1 ,V1 ) ←
Human_Detector V ehicle_Detector
whuman(H1 ,T1 ,X1 ,Y1 ) ∧ wvehicle(V1 ,T1 ,X2 ,Y2 ) ∧
(8.2)
Object_Detector Geometry_M anager
wobject(O1 ,T1 ,X3 ,Y3 ) ∧ woverwrap(X2 ,Y2 ,X3 ,Y3 ) ∧
Geometry_M anager
wnear(X1 ,Y1 ,X3 ,Y3 )

Rule 8.1 corresponds to the abducible take_a_veh(pid1) in ∆11 and Rule 8.2 corresponds

218
 Contents

to the abducible mng_veh_obj_acc(pid1) in ∆12 respectively. In our scenario, given the

rules and metadata in form of literals L, both rules can be satisfied. Another aspect to

consider is the negative conditions required not to satisfy in ∆. In above cases, ∆11 requires

3 exclusive abducibles not to be satisfied and ∆12 requires 2 exclusive abducibles not to be

satisfied. Indeed, due to the uncertainty implied in metadata, rest of the abducibles with

negation could also come up with certain amount of uncertainty that are also represented

as subjective opinions. In this case, we will take a subjective logic’s complement operation

on the derived opinions. In case we can’t even prove an abducible predicate in ∆ given

logic program P , we will simply assign full ignorance opinion. Considering the positive

abducibles and negative abducibles to be satisfied, it can be seen as default reasoning [143]

and, therefore, we will conduct default reasoning using subjective logic [82, 83] introduced

in Chapter 5. In this work, we slightly modify the reasoning scheme shown in Definition 35

to make it fit in the case of reasoning on ∆. The variant definition is as follows.

Definition 45. (Default Inference on ∆) . Given a set ∆, containing opinion assigned

elements, the default inference is the truth value assignment closure cl(ϕ)(∆) given by :

∧ ⊕ ∧
cl(ϕ)(∆) = [ w(p) ] ¬[ w(p) ] (8.3)
p∈∆ ¬p∈∆

,where ∧, ⊕ and ¬ represent subjective logic’s Conjunction, Consensus and Complement

operators respectively.

Now, consider the case that uncertainty assignment is as follows, that is, having the

same amount of opinion on each positive evidences in ∆11 and ∆12 . Namely,

wtake_a_veh(pid1) = wmng_veh_obj_acc(pid1) = (0.6, 0.3, 0.1).

Assume that, the system failed to prove mng_veh_prs_acc (pid1) and come up with
Chapter 8

buy_smt(pid1) with relatively high confidence that it was not happened, as opinion as-

signment follows:

wmng_veh_prs_acc(pid1) = (0, 0, 1).

219
Contents

wbuy_smt(pid1) = (0.1, 0.8, 0.1).

Given this set up, applying Definition 45, the final evaluation results of each ∆i is as fol-

lows:
∆op
11 = cl(ϕ)(∆11 ) = (0.6, 0.3, 0.1)⊕
¬[(0, 0, 1) ∧ (0.6, 0.3, 0.1) ∧ (0.1, 0.8, 0.1)]

= (0.6, 0.3, 0.1) ⊕ ¬(0.04, 0.86, 0.1)

= (0.6, 0.3, 0.1) ⊕ (0.86, 0.04, 0.1)

= (0.77, 0.18, 0.05), Exp = 0.80

∆op
12 = cl(ϕ)(∆12 ) = (0.6, 0.3, 0.1)⊕
¬[(0.6, 0.3, 0.1) ∧ (0.1, 0.8, 0.1)]

= (0.6, 0.3, 0.1) ⊕ ¬(0.08, 0.86, 0.06)

= (0.6, 0.3, 0.1) ⊕ (0.86, 0.08, 0.06)

= (0.79, 0.17, 0.04), Exp = 0.82

In above example, the final result rated that the hypothesis ∆12 is a bit more plausible

than ∆11 (∵ ∆12 has more belief and less uncertainty than ∆11 , the expectation value Exp

of ∆12 is therefore, greater also). However, the reason the opinion gap between these two

∆1 ’s are not bigger is due to the limited set of rules and metadata we assumed. Therefore,

thorough domain knowledge engineering and rule derivation would be critical on applying

the proposed approach.

8.7 Chapter Summary

In summary of lessons learned in this chapter, we proposed a system enabling diagnostic

abduction based reasoning. It mainly consists of tow layers. 1) a layer that reason about

queries to be asked and 2) a layer that uses abductive logic programming to draw potential

hypotheses given set of observations. Our demonstration showed the feasibility of using

the proposed approach on diagnostic queries. For practical use of this approach, however,

thorough domain engineering and rule derivation should be further tackled.

220
 Contents

In this chapter, we proposed an abductive reasoning framework for forensic visual

surveillance. For logical abduction, abductive logic programming framework was adopted

together with forward chaining based rule engine. The resulting hypotheses are evaluated

by a rule based system that is extended with subjective logic operators. The main advan-

tage of the proposed approach is that it doesn’t require the user to ask a specific query

but let them concentrate on collecting observations of their interest. For a collected set of

observations, the system fulfills an abduction to suggest probable hypotheses and rate the

uncertainty of each hypothesis. We believe this way of using forensic visual surveillance

will offer the user more choices on semantic analysis.

There are, however, still several open issues such as how to accurately define context

rules so that the derived hypotheses make sense, how to scale up the proposed approach

to large scale data, etc. Among them, as shown in the case study, we believe thorough

domain engineering and rule derivation is very critical because the final result would be

affected by this aspect. Nevertheless, we believe the proposed system has benefit and

unique property in the way of composing query especially in the forensic situation that the

semantic ambiguity of a scene is high or multiple semantic interpretations of a scene are

possible.

Chapter 8

221
222
9 Conclusion

In this chapter, we provide an architectural big picture of this dissertation, summarize

main contributions of this dissertation, highlight open issues and discuss extensions to this

work.

9.1 The Big Picture - Architectural View of the Work

This dissertation focussed on issues related to intelligent high-level semantic analysis of

visual surveillance data.

In Chapter 4, we discussed the software components, data processing pipeline and

system architecture design as shown in Figure 9.1. Especially, we focused on the reuse of

intermediate metadata acquired from vision analytics with a discussion of experimental

evaluation on query performance in different software component settings (i.e., ontology,

logic programming and traditional databased techniques). Figure 9.1 - a) shows the key

topics discussed in the Chapter 4.

Based on the proposed architecture, it has been extended to enable advanced reason-

ing aspects such as ‘default reasoning’ (in Chapter 5), ‘vague rule modeling’ (in Chapter 6),

‘bidirectional interpretation of conditionals’ (in Chapter 7) and ‘diagnostic abduction’ (in

Chapter 8). Because these topics were dealt mainly in logical reasoning sense, these chap-

ters correspond to the shaded area shown in Figure 9.1 - b). Especially, for the diagnostic

abduction discussed in Chapter 8, we have introduced an abductive logic programming

part as shown in Figure 9.1 - c).

Throughout Chapter 4 - Chapter 8, rule sets played an important role in visual

Chapter 9

223
Contents

Figure 9.1: The Architectural Bic Picture and Topic Roadmap in terms of Chapters of this
Dissertation.

surveillance scenarios. Such rule sets can be regarded as domain knowledge. Therefore,

the shaded part of Figure 9.1 - d) corresponds to these chapters.

9.2 Benefits and Drawbacks

The main advantage of the presented approach is in the ‘flexibility’ of representing and

resolving epistemic contextual semantics by leveraging logic programming based data rep-

resentation model (this is due to the ‘modularity’ as explained in Section 2.3.3). Based on

the ‘flexible’ knowledge ‘expressive’ power, we have bestowed advanced reasoning features.

The supported reasoning features by approaches appear in literature and covered

by our proposed approach is summarized in Table 9.1 (see Section 2.4.3 for details). The

table shows that the coverage of our subjective logic based approach is most broad. While

224
 Contents

Figure 9.2: Positioning Subjective Logic among Other Uncertainty Representation For-
malisms.

some of previous work support advanced features such as ‘default reasoning’, ‘vague rule

modeling’ and ‘belief revision’, some features such as ‘bidirectional inference’ and ‘diagnos-

tic abduction’ are only supported by our approach. To cope with such ‘non-monotonic’

behavior, in this dissertation, we adopted subjective logic for uncertainty representation

formalism. However, subjective logic itself is remain within the expressive power of propo-

sitional logic as of may ‘intensional’ approaches. In this dissertation to cope with complex

semantics, we adopted ‘predicate’ logic that is based on first-order logic. For the flexible

manipulation of facts and predicates on the need of arithmetic calculation, etc., we also

benefited from procedural handling of them.

The discussion in Section 5.9.1 shows that our approach using subjective logic is ‘log-

ically’ robust. Especially, compared with bilattice based approach that can be considered

as Fuzzy-Belnap, Discussion 1 shows that our approach is free from the ‘paradoxes in the

truth table of multivalued logics’. Discussion 2 shows that the ‘logical soundness’ is impor-
Chapter 9

225
Contents

Ogale / Yuri Akdemir Jianbing Shet et. al Anderson / Our approach

Approach
et al. [126, 88] et al. [13] et al. [110] [152, 155] Dorado et al. [81, 80, 82]
[17, 54] [83, 79]

Knowledge Rule Based Ontology Rule Based Rule Based Rule Based Rule Based
Modeling (Prop. L) (DL) (FOL) (FOL) (Prop. L) (FOL)

Uncertainty Stochastic Dempster Fuzzy Subjective

- Bilattice
Formalism Grammar Shafer Logic Logic
√
Default √
- - - -
Reasoning Chapter 5
√
Vague Rule √
- - - -
Modeling Chapter 6
√
Bidirectional
- - - - -
Inference Chapter 7
√
Diagnostic
- - - - -
Abduction Chapter 8
√
Belief √
- - - -
Revision Chapter 5-8

Table 9.1: The Coverage of the Proposed Approach in terms of Reasoning Power.

tant in visual surveillance system. It makes the use of subjective logic attractive. In the

sense that the Fuzzy-Belnap has similar uncertainty representation tuple, it resembles sub-

jective logic. In addition to this, in Section 3.1.2, we have shown that the subjective logic

is derived from Dempster Shafer belief theory. Similarly, In Section 3.1.3, we have reviewed

that subjective logic is a special case of Bayesian, that uses Beta family of conjugated dis-

tribution among other distributions. Figure 9.2 shows our view of positioning subjective

logic among other uncertainty formalism. The reason for the complementary area between

subjective logic and Bayesian is because subjective logic can not handle other prior distri-

butions that can not be approximated to beta distribution. Naturally, subjective logic can

not handle multi-modal distributions or non-conjugated form of distributions. Moreover,

some of advanced techniques in Bayesian approach such as MCMC (Markov Chain Monte

Carlo) [33] is not captured in subjective logic theory. However, the fact that it bridges

multiple uncertainty formalism gives us a good intuition for interpreting and operating

uncertainty from multiple perspectives.

226
 Contents

For contradictory and imprecise knowledge based reasoning, we could benefit from

our subjective logic based ‘default reasoning’. Especially, in the forensic sense of visual

surveillance that needs to reason about a propositional hypothesis to be investigated after

an incident or a report, it is natural to examine all positive and negative contextual evi-

dences that are related to the given hypothesis and fuse them to derive plausible conclusion

based on default reasoning.

The support of ‘vague rules’ allows for representing lack of information about a

proposition. Therefore, we could roughly assign our subjective priors with lack of informa-

tion, and observations can also be represented with any degree of ignorance, therefore we

believe this better reflects human intuition and real world situations. Especially, rule can

embed its own opinion calculation scheme thereby, allows for sophisticated propagation of

opinions through the inference pipeline. In Chapter 6, especially, we have demonstrated

how the reasoning results from uncertain spatio-temporal rules could be used with default

reasoning.

Another important advantage is the support of ‘bidirectional’ interpretation of con-

ditionals. The advantage is that it offers more choices to handle the trade-offs between

expressive power, semantic clarity and computational efficiency. For highly varying and

complex compositional events in visual surveillance data, we could benefit from the expres-

sive power and modularity of the extensional approach, while leveraging the intensional

interpretation of conditionals to facilitate bi-directional inference.

Finally, the main advantage of the ‘diagnostic logical abduction’ is that it doesn’t

require the user to ask a specific query but let them concentrate on collecting observations

of their interest. For a collected set of observations, the system fulfills an abduction to

suggest probable hypotheses and rate the uncertainty of each hypothesis. We believe

this way of using forensic visual surveillance will offer the user more choices on semantic

analysis.

There are, however, drawbacks as well. One of the biggest drawbacks is the ‘scala-

bility’ against large scale data. To be fully applicable for practical real applications, the
Chapter 9

227
Contents

reasoning power should be ‘scalable’. Section 4.6 shows performance benchmark of triple

stores and rule-engines against different scale of metadata (i.e., fact base in view of rule-

engines). Unfortunately, it seems that logical formalism based approaches and the triple

stores of ontological metadata are by themselves not sufficiently scalable. The ‘scalability’

problem is common issue in the realm of ontology related research field and also expert

system related researches. One good news, however, is that there have been undergoing

active research focus on the ‘scalability’ issue. We have briefly introduced some possible

remedies shown in literatures. In addition, the ‘scalability’ of low-level feature matching

is also important and is an active research topic in the realm of high dimensional vector

indexing field. Therefore, we believe and hope that we could benefit from those researches

to resolve the ‘scalability’ issue in the near future.

9.3 Open Issues and Future Work

Besides the benefits and drawbacks discussed in the previous section, there are still open

issues around the proposed approach.

While the proposed approach offers ‘flexibility’ and ‘expressive power’ for knowledge

modeling, the actual ‘knowledge acquisition’ and ‘expression’ should be done by ‘domain

experts’. Therefore, how to accurately define context rules so that the derived reason-

ing results make sense is an important issue. As shown especially in the case study of

Section 8.7, we believe thorough domain engineering and rule derivation is very critical

because the final result would be affected by this aspect.

This is also related to an important issue on the verification of the proposed ap-

proach applying to more complicated situational reasoning. Therefore, it is important to

generate and share large scale video data set that contain rich ‘epistemic’ semantics. For

the verification and comparison, the video data set should also come with ground truth.

However, unlike data sets for traditional vision analytics such as human or object detection

and tracking, such ‘epistemic’ semantic data set is hard to generate in that : 1) it would

228
 Contents

naturally contain not only multiple objects but also multiple human instances who should

regally agree on the use of the data set. 2) it would also naturally deal distributed multiple

camera that each instance should be labeled properly across the camera sources. 3) the

scenarios should be throughly considered before we build up such data set. Therefore, we

believe proper data set should be generated in a collaborative manner by many researchers

as it was done for TRECVID [123].

There are several open issues also related to subjective logic formalism such as how to

better model the reputational function, how to automatically assign proper prior opinions

to rules, how to extend this concept to multinomial frame based event, etc. Especially,

how to interpret uncertainty values

Therefore, our future research will cover above mentioned issues and extend the

shown approach to more complicated scenarios using automatically generated large scale

data.

9.4 Summary and Epilogue

We believe that there are several important contributions in this dissertation. First, this

dissertation lays out a systematic approach and methodical support for a more intelligent

semantic reasoning system, especially in the sense of ‘epistemic’ forensic (-post) analysis

of visual surveillance data. Traditionally, most of approaches to forensic sense of visual

surveillance applications have remained in signal/low-level feature processing. Content-

based retrieval is another remarkable approach that tries to search a scene that matches

to given image queries. Such methods have been limited to handle only simple and de-

terministic queries due to their heavy dependency on signal level processing. Only a few

researches have been attempted to handle complex semantic aspects such as using tradi-

tional database and ontology technics. However, such work lacks handling uncertainty and

heavily rely on deterministic data scheme thereby, the data processing components are too

much tied on the pre-defined scheme. Even a small change hits the consistency of the sys-
Chapter 9

229
Contents

tem thereby, the change forces entire system to be modified. Unlike legacy approaches, this

work proposes the use of logic programming that is extended with subjective logic theory

for uncertainty handling and epistemic reasoning. To the best of our knowledge, this is the

first attempt that subjective logic theory is applied to visual surveillance application.

It is worth to note that one remarkable research has been attempted in a simi-

lar manner using bilattice framework implemented with prolog. The bilattice framework

demonstrated reasoning power such as belief revision and default reasoning. However, it is

important to note that the bilattice framework based approach lacks soundness in terms of

logic theory. Indeed, the operations in bilattice framework are rather related to fuzzy the-

ory than traditional probability based Bayesian. Most of the vision analytic modules today

tend to based on Bayesian approach throughout its design and training phase. Therefore,

for system engineering to reuse such analytic data, providing a framework that can be also

reasoned in a Bayesian sense is preferred. In contrast, subjective logic not only provides

Bayesian sense of interpreting reasoning data but also bridges many aspects of bilattice

based approach, fuzzy based approach, dempster shaferian approach and Bayesian ap-

proach. Especially, we have demonstrated how conditional premises can be factorized and

handled in a Bayesian sense by using subjective logic’s deduction and abduction operators.

In this dissertation, we also demonstrate dynamic assessment of a vague proposi-

tion by introducing a reputation concept. We believe this way, we better capture human

intuition on modeling and handling vague propositions. We further extend our approach

to model default reasoning behaviour. To the best of our knowledge, this is also the first

time on modeling default reasoning in a schematic way using subjective logic operators.

A comprehensive comparison with L-fuzzy logics and billatice theory is conducted. This

comparison shows many overlap between subjective logic theory and bilattice in the behav-

ioral aspects of handling contradictory information. However, through the comparison we

showed that subjective logic is more robust in handling logical premises. This is again due

to the lack of soundness in bilattice that often derives counter intuitive reasoning result

unlike subjective logic theory.

230
 Contents

Based on the proposed approach, we further extend the system so that it can handle

abductive reasoning. Semantically speaking, such abductive queries are more complex to

handle due to its ambiguity. We tackled this by adding a layer that reason about queries to

be asked and a layer that uses abductive logic programming to draw potential hypotheses

given set of observations. Our demonstration showed the feasibility of using the proposed

approach on diagnostic queries. To the best of our knowledge, this is also the first time

using subjective logic for assessment of abductive reasoning result.

The attempt to semantic reasoning done in this dissertation has given several insights

into the design of intelligent visual surveillance system. Development of better intelligent

visual surveillance systems will require feedback. That is, deriving machine-processable

reasoning categories to develop a semantic reasoning mechanism and the systems are often

made by using the insights during the attempt to building such a system. In fact, this

dissertation especially provides a comprehensive insights on the relationship between dif-

ferent uncertainty formalisms. Preliminary results indicate that this approach is feasible

towards semantic reasoning for visual surveillance applications.

Chapter 9

231
232
List of Figures

1.1 Research Spectrum of Intelligent Semantic Analysis in Visual Surveillance

and Paradigm Shift. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

1.2 Conceptual Workflow of Forensic Reasoning in Case of Video Retrieval (left)

and a Released Retrieved-Footage (right) [10]. . . . . . . . . . . . . . . . . 7

2.1 Kant’s Epistemological Structure of Thinking [4]. . . . . . . . . . . . . . . . 16

2.2 Semantic Web Architecture Layer Cake [14]. . . . . . . . . . . . . . . . . . . 23

2.3 An Example of Handling Uncertainty in Extensional Approach - Certainty

Combination Function used in MYCIN [130]. . . . . . . . . . . . . . . . . . 27

3.1 An Example of Frame of Discernment Θ containing 3 Elements [94]. . . . . 39

3.2 Examples of Focused Frame of Discernment Derived from Figure 3.1. . . . . 42

3.3 Opinion Triangle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

3.4 Linguistic Fuzzy Category for Opinion Triangle as a Function of the Base

Rate [139]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

3.5 Examples of Beta Distributions. . . . . . . . . . . . . . . . . . . . . . . . . . 47

3.6 Examples of Applying Bayes’ Theorem to Different Priors and Given a Like-

lihood. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

3.7 Examples of Opinion Triangle and Their Mapping to Beta Distribution. . . 50

3.8 Composition of Subjective Logic Operators - A Propositional Logic View. . 52

4.1 Conceptual Architecture of the Proposed Intelligent Forensic Reasoning Sys-

tem. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61

233
List of Figures

4.2 A Piece of the ‘Video Analytics’ Ontology Description in ORSD (Ontology

Requirement Specification Document) [163] Format. . . . . . . . . . . . . . 66

4.3 Graphical Representation of Ontological Metadata Model: a) Video Analyt-

ics, b) Color Model, c) MPEG-7 Visual Descriptor, d) Surveillance System,

e) Domain Top and f) Source Media Ontologies respectively. . . . . . . . . . 67

4.4 Environmental Assumption : a) Locations of Nine Assumed Cameras, b)

Some of Frames from Camera 7 mounted in front of a Gas Station. . . . . . 71

4.5 Example Usage of Our Annotation Tool. . . . . . . . . . . . . . . . . . . . . 72

4.6 Annotation Instance of Figure 4.5 : a) DIG Representation of the Example,

b) Person Instance in N3-Triple Notation, c) ROI Instance in N3-Triple No-

tation and d) Color Structure Descriptor in RDF/OWL Format of MPEG-7

Ontology. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

4.7 Illustrative Example of VA-file : a) VA-file for 2-D Vectors and b) Lower

and Upper Bound between a Given Query Vector q and Feature Vectors in

a Cell. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

4.8 Querying and Retrieval Graphical User Interface and Walkthrough. . . . . . 77

4.9 Samples of Traffic Visual Surveillance Data for Case Study 2. . . . . . . . . 81

4.10 Flat-file Style Database Tables in ERD (Entity Relation Diagram) Each

Representing Metadata Vocabularies of Vision Analytics. . . . . . . . . . . . 82

4.11 Querying and Retrieval Graphical User Interface and Retrieval Results of

‘Q1-Q7’ with Sample Items. . . . . . . . . . . . . . . . . . . . . . . . . . . . 86

4.12 Benchmark Result in the Work of Jürgen Bock et al. [30]. The Figures

show the Average Load and Query Time with a Particular Reasoner on the

Dataset shown in Table 4.3. The ‘◦’ indicates Time-out (> 5 min). . . . . . 89

4.13 The Database Scheme of a Siemens’ Internal Visual Surveillance Prototype

System called VAP (Video Analytic Platform). . . . . . . . . . . . . . . . . 92

234
 List of Figures

4.14 Performance Evaluation Result of Different Rule-engines and Relational

Database Models. The Figures show the Average Response Time on Dif-

ferent Datasets shown in Table 4.5. The ‘◦’ indicates Time-out (> 1200

sec). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94

5.1 Examples of Lattice and Bilattices (a) a Lattice (b) Bilattice correspond-

ing to Traditional Binary Logic (c) Bilattice corresponding to Three-valued

Logic (d) Bilattice corresponding to Belnaps Four-Valued Logic, FOUR.

(Refer to [69] for more detail). . . . . . . . . . . . . . . . . . . . . . . . . . . 106

5.2 Lattices (a) L∗ Triangle corresponding to Intuitionistic Fuzzy Logic (and

also can be seen as LI Triangle for Interval-Valued Fuzzy Logic), and (b)

L□ Square corresponding to Fuzzy Belnap Logic. . . . . . . . . . . . . . . . 108

5.3 Illustrative Example of Default Reasoning Scenario in Visual Surveillance. . 110

5.4 (a) Belnap Logic, FOUR (b) Default Logic (c) Multivalued (Prioritized)

Default Logic. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112

5.5 (a)The Bilattice and Opinion Triangle Space for Belnap FOUR, (b) The

Bilattice and Opinion Triangle Space for Default Logic, (c) The Bilattice

and Opinion Triangle Space for Multivalued Default Logic. . . . . . . . . . . 121

5.6 Reasoning Results of Example 3 in Opinion Space and L2 & I ∗ (L). . . . . . 133

5.7 Reasoning Results of Example 4 in Opinion Space and L2 & I ∗ (L). . . . . . 137

5.8 Illustrative Scenario Setup of Example 5. . . . . . . . . . . . . . . . . . . . . 139

5.9 Reasoning Results of Example 5 in Opinion Space and L2 & I ∗ (L). . . . . . 143

5.10 (a) U ∧ U and U ∨ U (b) ⊥ ∧ ⊥ and ⊥ ∨ ⊥ in Subjective Logic. . . . . . . . 147

5.11 (a) Two Examples of U ∧ ⊥ (b) Two Examples of U ∨ ⊥ in Subjective Logic. 148

5.12 Reasoning Results of Modified Example 3 in Opinion Space and L2 & I ∗ (L). 153

6.1 Vague Rule Modeling using Subjective Logic’s Reputation Operator. . . . . 164

6.2 Scenario Setting for Case Study I. . . . . . . . . . . . . . . . . . . . . . . . . 166

6.3 Candidate Uncertainty Functions regarding Distance and Direction. . . . . . 166

235
List of Figures

6.4 Samples of Reputational Opinion according to Distance and Equation 6.1. . 168

6.5 Tree Representation of used Rules. . . . . . . . . . . . . . . . . . . . . . . . 170

6.6 Visualization of the Experiment. . . . . . . . . . . . . . . . . . . . . . . . . 170

6.7 Experimental Result. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 171

6.8 Scenario Setting for Case Study 2. . . . . . . . . . . . . . . . . . . . . . . . 172

7.1 Airport Surveillance Scenes for Case Study. . . . . . . . . . . . . . . . . . . 189

7.2 Examples of Applying Rule 7.12 based on Different # of Observations. . . . 193

7.3 Graphical Model of used Conditional Knowledge. . . . . . . . . . . . . . . . 194

7.4 Illustration of Abductive Conditional Reasoning for Case Study I (see http:

//persons.unik.no/josang/sl/Op.html). . . . . . . . . . . . . . . . . . . 194

7.5 Screen Captures demonstrating Belief Revision on the Proposition ‘the es-

calator is not working’. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 196

7.6 Scenario Setting for Case Study 2. - Two Cars, C1 and C2. An Empty

Parking Slot EP S1.- . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198

7.7 Examples of Applying Rule 7.20 based on Different # of Observations on

‘good_at_parking’ proposition by the Vehicle ‘Vi ’. . . . . . . . . . . . . . . 203

7.8 Illustration of Abductive Conditional Reasoning for Case Study 2 (see http:

//persons.unik.no/josang/sl/Op.html). . . . . . . . . . . . . . . . . . . 205

8.1 The Abductive Reasoning Framework. . . . . . . . . . . . . . . . . . . . . . 215

8.2 The Setup for Case Study Scenario with 3 Cameras Deployed. . . . . . . . . 216

9.1 The Architectural Bic Picture and Topic Roadmap in terms of Chapters of

this Dissertation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224

9.2 Positioning Subjective Logic among Other Uncertainty Representation For-

malisms. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

236
List of Tables

2.1 Formal Logics and Their Ontological and Epistemological Commitments [146]. 21

2.2 A Comparison of Previous Extensional Approaches. . . . . . . . . . . . . . . 35

3.1 An Example of Applying Dempster’s Rule shown in [94]. . . . . . . . . . . . 40

3.2 Subjective Logic Operators, Notations, and Corresponding Propositional /

Binary Logic Operators. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

4.1 Metadata Vocabulary and the Property of Each Primitive Semantic Item. . 82

4.2 Rules used in Case Study 2. . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

4.3 Statistics of Test Ontologies used in the Work of Jürgen Bock et al. [30].

The Table Indicates # of TBox Parameters (Class, Property, SubClass, etc.)

and # of ABox Parameters (C(a) : Instances of a Class and Instances of a

Relation R(a,b)). The Red Boxed Row Shows the Statistics of Our Manual

Annotation Used in Case Study I (see Section 4.4) Extracted by TopBraid

Composer. ‘-’ Represents the Statistics not Available in TopBraid Com-

poser. The Red Arrow Indicates that Our Annotation Resembles wine_1

Data Set in terms of Statistics. . . . . . . . . . . . . . . . . . . . . . . . . . 88

4.4 Some Examples of SQL Queries for ‘Q1-Q7’. . . . . . . . . . . . . . . . . . 93

4.5 Statistics of Dataset prepared for the Scalability Evaluation of Rule-engines

and Relational Database. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93

5.1 Truth Table of Bilattice Operators on (a) Belnap (b) Default Logic (The

unshaded Part is exactly compatible with Belnap). . . . . . . . . . . . . . . 112

237
List of Tables

5.2 An Example of Applying Dempster’s Rule and Consensus Operator for Un-

certain Belief Fusion [94]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116

5.3 An Example of Applying Dempster’s Rule and Consensus Operator for Dog-

matic Belief Fusion - Dempster’s Rule Deriving Counter Intuitive Result [94].116

5.4 Comparison of Operators used in cl(ϕ) shown in Equation (5.1) (Black -

exact match and Red - slight different match). . . . . . . . . . . . . . . . . . 123

5.5 Some of t-norms and t-conorms. . . . . . . . . . . . . . . . . . . . . . . . . . 127

7.1 Possible Subjective Conditional Operations given x and y with Arbitrary

Mutual Dependency. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188

9.1 The Coverage of the Proposed Approach in terms of Reasoning Power. . . . 226

238
Bibliography

[1] CLIPS - Basic Programming Guide - NASA JSC-25012. NASA. 57, 68, 78, 91, 186,

214

[2] Description logics. http://dl.kr.org/. 20, 22, 34, 212

[3] i-lids dataset. http://www.ilids.co.uk. 165

[4] Kant’s epistemological structure of thinking. http://de.wikipedia.org/wiki/Datei:

Kant_Erkenntnis.jpg. 16, 233

[5] Rushes - european research project on multimedia search and retrieval of rushes data.

http://www.rushes-project.eu/. 1

[6] The sciff abductive proof procedure. http://lia.deis.unibo.it/sciff/. 213

[7] Semantic web official website. http://semanticweb.org/wiki/Main_Page. 20, 22

[8] Swi-prolog. http://www.swi-prolog.org/. 91

[9] A normal approximation for beta and gamma tail probabilities. Probability Theory and

Related Fields, 65(3):399–420, 1984. 48

[10] Report of the Official Account of the Bombings in London on 7th July 2005. House of

Commons, May 2006. 2, 6, 7, 233

[11] F. ADams. Epistemology. In McHenry and Yagisawa, editors, Reflections on Philos-

ophy, 2nd Edition, pages 81–101. Longman Press, 2003. 15

[12] J. K. Aggarwal and Q. Cai. Human motion analysis: A review. pages 90–102, June

1997. 31

239
Bibliography

[13] U. Akdemir, P. Turaga, and R. Chellappa. An ontology based approach for activity

recognition from video. In Proceedings of the 16th ACM international conference on

Multimedia, ACM MM ’2008, pages 709–712. ACM, 2008. 34, 35, 103, 211, 226

[14] H. T. Al-Feel, M. Koutb, and H. Suoror. Semantic web on scope : A new architectural

model for the semantic web. Journal of Computer Science, 4(7):613–624, 2008. 22, 23,

233

[15] J. F. Allen. Maintaining knowledge about temporal intervals. Communications of the

ACM, 26(11):832–843, November 1983. 84

[16] A. Altadmri and A. Ahmed. Automatic semantic video annotation in wide domain

videos based on similarity and commonsense knowledgebases. In Proceeding of the 2009

IEEE International Conference on Signal and Image Processing Applications, ICSIPA’

2009, pages 74–79, 2009. 1

[17] D. Anderson, R. H. Luke, J. M. Keller, M. Skubic, M. J. Rantz, and M. A. Aud.

Modeling human activity from voxel person using fuzzy logic. IEEE Transactions on

Fuzzy Systems, 17(1):39–49, February 2009. 34, 35, 36, 103, 162, 211, 226

[18] O. Arieli, C. Cornelis, G. Deschrijver, and E. E. Kerre. Relating intuitionistic fuzzy

sets and interval-valued fuzzy sets through bilattices. In D. Ruan, P. D’hondt, M. D.

Cock, M. Nachtegael, and E. E. Kerre, editors, Applied Computational Intelligence,

pages 57–64. World Scientific, Blankenberge, Belgium, 2004. 102, 103

[19] O. Arieli, C. Cornelis, G. Deschrijver, and E. E. Kerre. Bilattice-based squares and

triangles. In Proceeding of the 8th European Conference on Symbolic and Quantitative

Approaches to Reasoning with Uncertainty, ECSQARU’ 2005, LNAI 3571, pages 563–

575, 2005. 102, 108, 126, 128

[20] R. A. Askey. Beta function. Cambridge Univ. press., 2010. 11

[21] K. T. Atanassov. Intuitionistic fuzzy sets. Fuzzy Sets and Systems, 20(1):87–96,

August 1986. 107

240
 Bibliography

[22] K. T. Atanassov. Remark on a property of the intuitionistic fuzzy interpretation

triangles. Notes on Intuitionistic Fuzzy Sets, 8(1):34–36, 2002. 104, 129

[23] K. T. Atanassov. Intuitionistic fuzzy sets: past, present and future. In Proceedings

of the 3rd Conference of the European Society for Fuzzy Logic and Technology, pages

12–19, September 2003. 144

[24] F. Baader, D. Calvanese, D. McGuinness, D. Nardi, and P. Patel-Schneider. The

Description Logic Handbook. Cambridge Univ. press., 2005. 20, 22, 34, 60

[25] L. Ballan, M. Bertini, B. D. Alberto, L. Seidenari, and G. Serra. Event detection

and recognition for semantic annotation of video. Multimedia Tools and Applications,

51:279–302, 2011. 1

[26] S. Benferhat, A. Saffiotti, and P. Smets. Belief functions and default reasoning. Arti-

ficial Intelligence, 122:1–69, September 2000. 113, 120

[27] M. Blank, L. Gorelick, E. Shechtman, M. Irani, and R. Basri. Actions as space-time

shapes. Number 9, pages 1395–1402, 2005. 31

[28] S. Bloehdorn, K. Petridis, C. Saathoff, N. Simou, V. Tzouvaras, Y. Avrithis, S. Hand-

schuh, I. Kompatsiaris, S. Staab, and M. G. Strintzis. Semantic annotation of images

and videos for multimedia analysis. In Proceedings of the 2005, 2nd European Semantic

Web Conference, May 2005. 24

[29] A. Bobick and J. Davis. The recognition of human movement using temporal tem-

plates. IEEE Transaction on Pattern Analysis and Machine Intelligence, 23(3):257–267,

2001. 31

[30] J. Bock, P. Haase, Q. Ji, and R. Volz. Benchmarking owl reasoners. In Proceedings

of the Workshop on Advancing Reasoning on the Web: Scalability and Commonsense,

ARea ’2008, June 2008. 87, 88, 89, 90, 234, 237

[31] C. Böhm, S. Berchtold, and D. A. Keim. Searching in high-dimensional spaces: Index

structures for improving the performance of multimedia databases. ACM Computing

Surveys, 33(3):322–373, 2001. 61, 75

241
Bibliography

[32] B. Bolles and R. Nevatia. A hierarchical video event ontology in owl. In ARDA

Challenge Workshop Report, 2004. 24

[33] W. M. Bolstad. Introduction to Bayesian Statistics. Wiley-Interscience, 2007. 48, 100,

194, 204, 226

[34] F. Bremond and M. Thonnat. A context representation for surveillance systems. In

Proceedings of the 1996 European Conference on Computer Vision Worshop on Con-

ceptual Descriptions from Images, ECCV’ 1996, April 1996. 33

[35] G. Brewka, I. Niemela, and M. Truszczynski. Nonmonotonic reasoning. In V. Lifschitz,

B. Porter, and F. van Harmelen, editors, Handbook of Knowledge Representation, pages

239–284. Elsevier, 2007. [pdf]. 10

[36] B. G. Buchanan and E. H. Shortliffe. Rule Based Expert Systems: The MYCIN

Experiments of the Stanford Heuristic Programming Project. Addison-Wesley, 1984. 26

[37] H. Buxton and S. Gong. Visual surveillance in a dynamic and uncertain world. Journal

of Artificial Intelligence, 78(1/2):431–459, 1995. 32

[38] S. Calegari and D. Ciucci. Integrating fuzzy logic in ontologies. In Proceedings of

the 2006, 8th International Conference On Enterprise Informational Systems Artificial

Intelligence And Decision Support Systems, pages 66–73, 2006. 23

[39] C. L. Chang and A. Walker. Prosql: a prolog programming interface with sql/ds.

In Proceedings from the first international workshop on Expert database systems, pages

233–246, Redwood City, CA, USA, 1986. Benjamin-Cummings Publishing Co., Inc. 95

[40] E. F. Codd. Redundancy, and consistency of relations stored in large data banks.

IBM Research Report, 1969. 91

[41] E. F. Codd. A relational model of data for large shared data banks. Communications

of the ACM, 13(6):377–387, 1970. 91

[42] C. Cornelis, K. T. Atanassov, and E. E. Kerre. Intuitionistic fuzzy sets and interval-

valued fuzzy sets: a critical comparison. In Proceedings of the 3rd Conference of the

242
 Bibliography

European Society for Fuzzy Logic and Technology, pages 159–163, September 2003. 102,

104, 107

[43] C. Cornelis, G. Deschrijver, and E. E. Kerre. Square and triangle: Reflections on

two prominent mathematical structures for the representation of imprecision. Notes on

Intuitionistic Fuzzy Sets, 9(3):11–21, 2003. 102, 104, 108

[44] C. Cornelis, G. Deschrijver, and E. E. Kerre. Implication in intuitionistic and interval

valued fuzzy set theroy. International Journal of Approximate Reasoning, 35(1):55–95,

2004. 108, 129

[45] S. Costantini. Meta-reasoning: a survey. In Computational Logic: Logic Programming

and Beyond Essays in Honour of Robert A. Kowalski (LNAI Volumes 2408), pages

253–288. Springer-Verlag, 2002. 10

[46] F. Cupillard, F. Bremond, and M. Thonnat. Behavior recognition for individuals,

groups of people, and crowd. IEEE Seminar Intelligent Distributed Surveillance Sys-

tems, March 2003. 33

[47] R. Datta, D. Joshi, J. Li, James, and Z. Wang. Image retrieval: Ideas, influences, and

trends of the new age. ACM Computing Surveys, 39:2007, 2006. 1

[48] J. Davies. Semantic Web Technologies: Trends and Research in Ontology-based Sys-

tems. Wiley, 2006. 20

[49] H. M. Dee and S. A. Velastin. How close are we to solving the problem of automated

visual surveillance?: A review of real-world surveillance, scientific progress and eval-

uative mechanisms. Machine Vision and Applications, 19:329–343, September 2008.

[50] M. Denecker and A. C. Kakas. Abduction in logic programming, in computational

logic: Logic programming and beyond. Lecture Notes in Computer Science, 2407:402–

436, 2002. 29, 63, 160, 212

[51] M. Denecker and D. D. Schreye. Sldnfa: an abductive procedure for normal abductive

243
Bibliography

programs. In Proceedings of the 1992 International Joint Conference and Symposium

on Logic Programming, pages 686–700, 1992. 213

[52] N. Dimitrova, H.-J. Zhang, B. Shahraray, I. Sezan, T. Huang, and A. Zakhor. Appli-

cations of video-content analysis and retrieval. IEEE MultiMedia, 9:42–55, July 2002.

[53] M. Döller, K. Renner, and H. Kosch. Introduction of an mpeg-7 query language. In

Proceedings of the 2007 2nd International Conference on Digital Information Manage-

ment, ICDM’ 2007, Lyon, France, 2007. 63

[54] A. Dorado, J. Calic, and E. Izquierdo. A rule-based video annotation system. IEEE

Transactions on Circuits and Systems for Video Technology, pages 622–633, 2004. 35,

226

[55] C. Draxler. A powerful prolog to sql compiler. Technical report, Center for Information

and Language Processing, Ludwig-Maximillians-Universität München, September 1992.

95

[56] D. Dubois. On ignorance and contradiction considered as truth-values. Logic Journal

of the IGPL, 16(2):195–216, 2008. 145, 146

[57] ECLIPSE. The eclipse constraint programming system. http://eclipseclp.org/.

214

[58] J. Fernyhough, A. G. Cohn, and D. C. Hogg. Building qualitative event models

automatically from visual input. In Proceedings of the 6th International Conference

on Computer Vision, ICCV ’1998, pages 350–355, Washington, DC, USA, 1998. IEEE

Computer Society. 33

[59] G. Ferrin, L. Snidaro, I. Visentini, and G. L. Foresti. Abduction for human behav-

ior analysis in video surveillance scenarios. In Proceedings of the 2008 1st Interna-

tional Workshop on Tracking Humans for the Evaluation of their Motion in Image Se-

quences (THEMIS’2008) in conjunction with British Machine Vision Conference 2008,

THEMIS’ 2008, BMVC’ 2008, 2008. 212

244
 Bibliography

[60] M. Fitting. Bilattices in logic programming. In Proceedings of 20th International

Symposium on Multiple-Valued Logic, pages 238–246, 1990. 104, 106, 108

[61] M. Fitting. Kleenes three-valued logics and their children. Fundamental Informaticae,

20:113–131, 1994. 106

[62] P. Flach. On the state of the art in machine learning : A personal review. Artificial

Intelligence, 13(1-2):199–222, 2001. 25

[63] M. Fleischman, P. Decamp, and D. Roy. Mining temporal patterns of movement for

video content classification. In Proceedings of the 8th ACM International Workshop on

Multimedia Information Retrieval, pages 183–192, 2006. 31

[64] C. Forgy. Rete: A fast algorithm for the many pattern/many object pattern match

problem. Artificial Intelligence, 19:17–37, 1982. 91

[65] S. fu Chang. Compressed-domain content-based image and video retrieval. In Pro-

ceedings of the Symposium on Multimedia Communications and Video Coding, October

1995. 1

[66] J. Gallier. Chapter 9 sld resolution and logic programming. In Logic for Computer

Science: Foundations of Automatic Theorem Proving. Originally published by Wiley,

1986, 2003. 91, 213

[67] T. Gardiner, I. Horrocks, and D. Tsarkov. Automated benchmarking of description

logic reasoners. In Proceedings of the Description Logics Workshop, 2006. 87

[68] M. Gehrke, C. Walker, and E. Walker. Some comments on interval-valued fuzzy sets.

International Journal of Intelligent Systems, 11(10):751–759, 1996. 107

[69] M. L. Ginsberg. Multivalued logics: A uniform approach to inference in artificial

intelligence. Computational Intelligence, 4(3):256–316, 1988. 34, 100, 104, 106, 111,

211, 235

[70] J. A. Goguen. l-fuzzy sets. Journal of Mathematical Analysis and Applications,

8(1):145–174, 1967. 107, 108

245
Bibliography

[71] S. Gong and J. Ng. Learning pixel-wise signal energy for understanding semantics. In

Proceedings of the 2001, British Machine Vision Conference, pages 1183–1189, 2001.

31

[72] S. Gong and T. Xiang. Recognition of group activities using dy- namic probabilistic

networks. In Proceedings of the 2003, 9th IEEE International Conference on Computer

Vision, ICCV’ 2003, pages 742–749, 2003. 32

[73] S. Gottwald and P. Hajek. Triangular norm based mathematical fuzzy logic. In E. P.

Klement and R. Mesiar, editors, Logical, Algebraic, Analytic and Probabilistic Aspects

of Triangular Norms, pages 71–116. Dordrecht Reidel, 2005. 163

[74] V. Gouaillier. Intelligent video surveillance: Promises and challenges. Technical re-

port, 2009. 2, 5

[75] T. Gruber. Towards principles for the design of ontologies used for knowledge sharing.

International Journal Human-Computer Studies, 43:907–928, 1993. 22

[76] M. Guinness and V. Harmelen. OWL Web Ontology Language Overview. W3C Rec-

ommendation, Feb 2004. 22, 60

[77] R. Hähnle. Many-valued logic, partiality, and abstraction in formal specification lan-

guages. Logic Journal of the IGPL, 13(4):415–433, 2005. 145

[78] S. Han, A. Hutter, and W. Stechele. Toward contextual forensic retrieval for visual

surveillance: Challenges and an architectural approach. In Proceedings of the 2009

10th International Workshop on Image Analysis for Multimedia Interactive Services,

WIAMIS’ 2009, London, UK, 2009. 36, 38, 211

[79] S. Han, A. Hutter, and W. Stechele. A reasoning approach to enable abductive

semantic explanation upon collected observations for forensic visual surveillance. In

Proceedings of the 2011 IEEE International Conference on Multimedia and Expo, ICME

’2011, July 11–15 2011. 35, 36, 38, 211, 226

[80] S. Han, B. Koo, A. Hutter, V. Shet, and W. Stechele. Subjective logic based hybrid

approach to conditional evidence fusion for forensic visual surveillance. In Proceedings

246
 Bibliography

of the 2010 7th IEEE International Conference on Advanced Video and Signal Based

Surveillance, AVSS ’2010, pages 337–344, 2010. 35, 36, 38, 157, 211, 226

[81] S. Han, B. Koo, A. Hutter, and W. Stechele. Forensic reasoning upon pre-obtained

surveillance metadata using uncertain spatio-temporal rules and subjective logic. In

Proceedings of the 2010 11th International Workshop on Image Analysis for Multimedia

Interactive Services, WIAMIS’ 2010, 2010. 35, 36, 38, 157, 211, 226

[82] S. Han, B. Koo, and W. Stechele. Subjective logic based approach to modeling default

reasoning for visual surveillance. In Proceedings of the 2010 IEEE 4th International

Conference on Semantic Computing, ICSC ’2010, pages 112–119. IEEE Computer So-

ciety, 2010. 35, 36, 38, 69, 173, 211, 219, 226

[83] S. Han and W. Stechele. Default reasoning for forensic visual surveillance based on sub-

jective logic and its comparison with l-fuzzy set based approaches. International Journal

of Multimedia Data Engineering and Management (IJMDEM), 2(1):38–86, 2011. 35,

36, 38, 173, 211, 219, 226

[84] P. Hayes. On semantic nets, frames and associations. In Proceedings of the 1977

International Joint Conference on Artificial Intelligence, pages 99–107, 1977. 20, 21

[85] S. Hongeng, R. Nevatia, and F. Bremond. Running head: Video-based event recogni-

tion video-based event recognition: Activity representation and probabilistic recogni-

tion methods. Journal of Computer Vision and Image Understanding - Special number

on event detection in video, 96(2):129–162, November 2004. 32

[86] W. Hu, T. Tan, L. Wang, and S. Maybank. A survey on visual surveil- lance of

object motion and behaviors. IEEE Transactions on System, Man, and Cybernetics,

34(3):334–352, 2004. 31

[87] ISO/IEC/JTC1/SC29/WG11/N3966. Multimedia content description interface - part

5 multimedia description schemes. Technical report, Text of 15938-5 FCD Information

Technology, March 2001. 65

[88] Y. A. Ivanov and A. F. Bobick. Recognition of visual activities and interactions by

247
Bibliography

stochastic parsing. IEEE Transactions on Pattern Analysis and Machine Intelligence,

22(8):852–872, 2000. 34, 35, 226

[89] M. Jarke, J. Clifford, and Y. Vassiliou. An optimizing prolog front-end to a relational

query system. In International Conference on Management of Data, volume 14, pages

296–306, 1984. 95

[90] F. V. Jensen. Bayesian Networks and Decision Graphs. Information Science and

Statistics, 2007. 32

[91] K. H. Jo, Y. Kuno, and Y. Shirai. Manipulative hand gesture recognition using task

knowledge for human computer interaction. In Proceedings of the 1998, 3rd Interna-

tional Conference on Face Gesture Recognition, pages 468–473, 1998. 32

[92] A. Jøsang. Artificial reasoning with subjective logic. In Proceedings of the 1997 2nd

Australian Workshop on Commonsense Reasoning, 1997. 37, 48, 100, 101, 211

[93] A. Jøsang. A logic for uncertain probabilities. International Journal of Uncertainty,

Fuzziness and Knowledge-Based Systems, 9(3):271–311, 2001. 10, 36, 37, 38, 39, 40, 41,

42, 52, 62, 100, 171, 191, 192, 202, 211

[94] A. Jøsang. The consensus operator for combining beliefs. Artificial Intelligence Jour-

nal, 38(1):157–170, 2006. 38, 39, 40, 114, 116, 118, 149, 150, 155, 169, 192, 202, 233,

237, 238

[95] A. Jøsang. Conditional reasoning with subjective logic. Journal of Multiple-Valued

Logic and Soft Computing, 15(1):5–38, 2008. 51, 52, 182, 183

[96] A. Jøsang. Cumulative and averaging unfusion of beliefs. In Proceedings of the Inter-

national Conference on Information Processing and Management of Uncertainty, IPMU

’2008, June 2008. 52

[97] A. Jøsang and M. Daniel. Multiplication and comultiplication of beliefs. International

Journal of Approximate Reasoning, 142(1-2):19–51, 2004. 52, 123, 149, 150, 154, 169,

185, 191

248
 Bibliography

[98] A. Jøsang, M. Daniel, and P. Vannoorenberghe. Strategies for combining conflicting

dogmatic beliefs. In Proceedings of the 2003 6th International Conference on Informa-

tion Fusion, pages 1133–1140, 2003. 38, 48, 50, 114, 149, 150

[99] A. Jøsang, S. Marsh, and S. Pope. Exploring different types of trust propagation.

In Proceedings of the 4th International Conference on Trust Management, Pisa, Italy,

2006. 51, 52, 164, 192, 202

[100] A. Jøsang, S. Pope, and M. Daniel. Conditional deduction under uncertainty. In

Proceedings of the 2005 European Conference on Symbolic and Quantitative Approaches

to Reasoning with Uncertainty, ECSQARU’ 2005, Barcelona, Spain, July 2005. 51, 52,

182, 183, 184

[101] A. C. Kakas, A. Michael, and C. Mourlas. Aclp: Abductive constraint logic pro-

gramming. The Journal of Logic Programming, 44(1-3):129–177, 2000. 213, 214

[102] G. Lavee, M. Rudzsky, E. Rivlin, and A. Borzin. Understanding video events: A sur-

vey of methods for automatic interpretation of semantic occurrences in video. Technical

report, Technion - Israel Institute of Technology, Haifa, Israel, Tech. Rep. CIS-2009-06,

2009. 31

[103] R. Lee. Scalability report on triple store applications. Technical report, MIT, July

2004. 61

[104] J. Lloyd and W. John. Foundations of logic programming. Springer-Verlag New York,

Inc., 1987. 61

[105] E. J. Lowe. A Survey of Metaphysics. Oxford University Press, 2002. 15

[106] R. J. Lucas and G. A. Le Vine. A Prolog-relational database interface, pages 67–80.

Halsted Press, New York, NY, USA, 1988. 95

[107] R. Ludwig. Kant für Anfänger - Die Kritik der reinen Vernunft : Eine Lese -

Einführung. Deutscher Taschenbuch Verlag, 1995. 16

[108] F. Lv and R. Nevatia. Single view human action recognition using key pose matching

249
Bibliography

and viterbi path searching. In Proceedings of the 2007, IEEE International Conference

on Computer Vision and Pattern Recognition, CVPR ’2007, pages 1–8, 2007. 32

[109] F. Lv, X. Song, B. Wu, V. K. Singh, , and R. Nevatia. Left-luggage detection using

bayesian inference. In Proceedings of the 2006, 9th IEEE International Workshop on

Performance Evaluation in Tracking and Surveillance, PETS ’2006, pages 83–90, 2006.

32

[110] J. Ma, W. Liu, P. Miller, and W. Yan. Event composition with imperfect information

for bus surveillance. In Proceedings of the 2009 6th IEEE International Conference

on Advanced Video and Signal Based Surveillance, AVSS ’2009, pages 382–387. IEEE

Computer Society, 2009. 34, 35, 103, 211, 226

[111] L. Magnani. Abduction, Reason, and Science. Processes of Discovery and Explana-

tion. Kluwer Academic/ Plenum Publishers, New York, 2002. 19

[112] F. Maier, D. Nute, W. D. Potter, J. Wang, M. Twery, H. M. Rauscher, M. Dass,

and H. Uchiyama. Efficient integration of prolog and relational databases in the ned

intelligent information system. In Proceedings of the 2003 International Conference on

Information and Knowledge Engineering, IKE 2003, pages 364–369, 2003. 95

[113] D. Makris and T. Ellis. Spatial and probabilistic modelling of pedestrian behaviour.

In Proceedings of the 2002 British Machine Vision Conference, volume 2 of BMVC’

2002, pages 557–566, 2002. 32

[114] D. Makris and T. Ellis. Learning semantic scene models from observing activity

in visual surveillance. IEEE Transactions on Systems Man and Cybernetics Part B,

35(3):397–408, 2005. 33

[115] D. McAnally and A. Jøsang. Addition and subtraction of beliefs. In Proceedings of

the 2004, Conference on Information Processing and Management of Uncertainty in

Knowledge Based Systems, IPMU’ 2004, July 2004. 52

[116] R. N. McCauley. Epistemology in an age of cognitive science. Philosophical Psychol-

ogy, 1:143–152, 1988. 3

250
 Bibliography

[117] G. G. Medioni, I. Cohen, F. Bremond, S. Hongeng, and R. Nevatia. Event detection

and analysis from video streams. IEEE Transactions on Pattern Analysis and Machine

Intelligence, 23(8):873–889, August 2001. 32

[118] M. Minsky. A framework for representing knowledge. In P. Winston, editor, The

Psychology of Computer Vision, pages 211–277. McGraw Hill, San Mateo, CA, 1975.

20, 21

[119] M. Mizumoto and H. J. Zimmermann. Comparison of fuzzy reasoning methods.

IEEE Transactions on Fuzzy Sets and Systems, 8:253–283, 1982. 162

[120] N. Moenne-Loccoz, F. Bremond, and M. Thonnat. Recurrent bayesian network for

the recognition of human behaviors from video. In Proceedings of the 2003, Interna-

tional Conference on Computer Vision Systems, ICVS’ 2003, pages 68–77, 2003. 33

[121] J. Muncaster and Y. Ma. Activity recognition using dynamic bayesian networks with

automatic state selection. In Proceedings of the 2007, IEEE Workshop on Motion and

Video Computing, pages 30–37, 2007. 33

[122] R. Nevatia, J. Hobbs, and B. Bolles. An ontology for video event representation.

In Proceedings of the 2004 IEEE Workshop on Event Detection and Recognition, June

2004. 24

[123] NIST. Trec video retrieval evaluation - digital video retrieval at nist. http://

trecvid.nist.gov/. 229

[124] S. Nowozin, G. Bakir, and K. Tsuda. Discriminative subsequence mining for action

classification. In Proceedings of the 2007, 11th International Conference on Computer

Vision, ICCV’ 2007, pages 1919–1923, 2007. 31

[125] B. V. Nuffelen and A. C. Kakas. A-system : Programming with abduction. In

Proceedings of the 2001 6th International Conference on Logic Programming and Non-

monotonic Reasoning, LPNMR’ 2001, pages 393–396. Springer Verlag, 2001. 213, 214

[126] A. Ogale, A. Karapurkar, and Y. Aloimonos. View-invariant modeling and recog-

nition of human actions using grammars. In Proceedings of the 2005 Workshop on

251
Bibliography

Dynamical Vision in conjunction with IEEE International Conference on Computer

Vision, pages 115–126, 2005. 34, 35, 226

[127] A. Ogale, A. Karapurkar, G. G. Filho, and Y. Aloimonos. View-invariant identifica-

tion of pose sequences for action recognition. In Proceedings of the 2004 Workshop on

Video Analysis and Content Extraction, VACE’ 2004, 2004. 32

[128] N. Oliver, E. Horvitz, and A. Garg. Layered representations for human activity recog-

nition. In Proceedings of the 2002, 4th IEEE International Conference on Multimodal

Interfaces, pages 3–8, 2002. 32

[129] L. Paulevé, H. Jégou, and L. Amsaleg. Locality sensitive hashing: A comparison of

hash function types and querying mechanisms. Pattern Recogn. Letter, 31(11):1348–

1358, August 2010. 75

[130] J. Pearl. Probabilistic reasoning in intelligent systems: Networks of plausible infer-

ence. Morgan kaufmann, San Mateo, CA, 1988. 2, 4, 9, 25, 27, 100, 103, 180, 181,

233

[131] C. S. Peirce. Philosophical Writings of Peirce. Dover Publications, New York, 1955.

19, 210, 212

[132] S. E. Peraldi, A. Kaya, S. Melzer, R. Möller, and M. Wessel. Multimedia interpreta-

tion as abduction. In Proceedings of the International Workshop on Description Logics,

2007. 212

[133] K. Petridis, D. Anastasopoulos, C. Saathoff, N. Timmermann, Y. Kompatsiaris, and

S. Staab. M-ontomat-annotizer: Image annotation linking ontologies and multime-

dia low-level features. Lecture Note in Computer Science, Knowledge based intelligent

information and engineering systems, 2006. 65, 72

[134] K. Phillips-Fein. The 9/11 Commission Report - Final Report of the National Com-

mission on Terrorist Attacks Upon the United States. Official Government Edition,U.S.

Government Printing Office, 2007. 2

252
 Bibliography

[135] C. Piciarelli, G. Foresti, and L. Snidaro. Trajectory clustering and its applications

for video surveillance. In Proceedings of the 2005, IEEE International Conference on

Advanced Video and Signal Based Surveillance, pages 40–45, 2005. 31

[136] M. Pittore, C. Basso, and A. Verri. Representing and recognizing visual dynamic

events with support vector machines. In Proceedings of the International Conference

on Image Analysis and Processing, pages 18–23, 1999. 31

[137] J. L. Pollock. Epistemology, rationality, and cognition. In M. Steup, editor, Com-

panion to Epistemology, pages 1–12. Blackwell Publishers, 1994. 3

[138] J. L. Pollock and A. S. Gillies. Belief revision and epistemology. Synthese - Interna-

tional Journal for Epistemology, Methodology and Philosophy of Science, 122(1):69–92,

2000. 3, 9

[139] S. Pope and A. Jøsang. Analsysis of competing hypotheses using subjective logic.

In Proceedings of the 2005, 10th International Symposium on Command and Control

Research and Technology, 2005. 43, 44, 51, 52, 233

[140] S. Pope and A. Jøsang. Abductive reasoning with uncertainty. In Proceedings of the

International Conference on Information Processing and Management of Uncertainty,

IPMU’ 2008, Malaga, Spain, June 2008. 182, 184

[141] M. R. Quillian. Semantic memory. In M. Minsky, editor, Semantic Information

Processing. Cambridge, MA, 1968. 20, 21

[142] L. R. Rabiner. A tutorial on hidden markov models and selected applications in

speech recognition. Readings in speech recognition, 1990. 32

[143] R. Reiter. A logic for default reasoning. Artificial Intelligence, 13:81–132, 1980. 9,

102, 109, 160, 172, 173, 211, 219

[144] P. Remagnino, T. Tan, and K. Baker. Multiagent visual surveillance of dynamic

scenes. Journal of Image and Vision Computing, 16(8):529–532, 1998. 32

[145] P. Ribeiro, P. Moreno, and J. S. Victor. Boosting with temporal consistent learners:

253
Bibliography

An application to human activity recognition. In Proceedings of the 2007, International

Symposium on Visual Computing, pages 464–475, 2007. 31

[146] S. Russell and P. Norvig. Artificial Intelligence: A Modern Approach. Prantice Hall,

second edition, 2002. 20, 21, 237

[147] O. Schreer, I. Feldmann, I. A. Mediavilla, P. Concejero, A. H. Sadka, M. R. Swash,

S. Benini, R. Leonardi, T. Janjusevic, and E. Izquierdo. Rushes - an annotation and

retrieval engine for multimedia semantic units. Multimedia Tools and Applications,

pages 23–49, 2010. 1

[148] N. Shachtman. London bomb plot: Surveillance web slips again? http://www.

wired.com/dangerroom/2007/06/london-bomb-p-1/, 29 June 2007. 2, 3

[149] G. Shafer. A mathematical theory of evidence. Princeton University Press, Princeton,

NJ, 1976. 10, 34, 37, 38, 51, 100, 211

[150] M. Shanahan. Prediction is deduction but explanation is abduction. In Proceedings

of the 1989 International Joint Conference on Artificial Intelligence, pages 1055–1060.

Morgan Kaufmann, 1989. 10

[151] E. Shechtman and M. Irani. Space-time behavior based correlation. In Proceedings

of the 2005 IEEE Conference on Computer Vision and Pattern Recognition, CVPR’

2005, pages 405–401, June 2005. 31

[152] V. D. Shet, D. Harwood, and L. S. Davis. Vidmap: video monitoring of activity with

prolog. In Proceedings of the 2005 2nd IEEE International Conference on Advanced

Video and Signal Based Surveillance, AVSS’ 2005, pages 224–229, 2005. 34, 35, 103,

130, 138, 211, 226

[153] V. D. Shet, D. Harwood, and L. S. Davis. Multivalued default logic for identity

maintenance in visual surveillance. In Proceedings of the 2006 European Conference on

Computer Vision, ECCV’ 2006, pages 119–132, 2006. 102, 103, 104, 130, 131, 134, 211

[154] V. D. Shet, D. Harwood, and L. S. Davis. Top-down, bottom-up multivalued default

254
 Bibliography

reasoning for identity maintenance. In Proceedings of the ACM International Workshop

on Video Surveillance & Sensor Networks, pages 79–86, 2006. 102, 103, 104, 130, 134

[155] V. D. Shet, J. Neumann, V. Ramesh, and L. S. Davis. Bilattice-based logical reason-

ing for human detection. In Proceedings of the 2007 IEEE International Conference on

Computer Vision and Pattern Recognition, CVPR’ 2007, pages 1–8, 2007. 34, 35, 104,

226

[156] N. Simou, V. Tzouvaras, Y. Avrithis, G. Stamou, and S. Kollias. A visual descriptor

ontology for multimedia reasoning. In Proceedings of the 2005 6th International Work-

shop on Image Analysis for Multimedia Interactive Services, WIAMIS’ 2005, Montreux,

Switzerland, 2005. 65, 72

[157] P. Smith, N. da Vitoria Lobo, and M. Shah. Temporal boost for event recognition.

In Proceedings of the 2005, 10th International Conference on Computer Vision, ICCV’

2005, pages 733–740, 2005. 31

[158] S. Staab and R. Studer. Handbook on Ontologies - International Handbooks on In-

formation Systems. Springer, 2004. 22

[159] R. Stalnaker. Probability and conditionals. The University of Western Ontario Series

in Philosophy of Science, D.Riedel Publishing Company, Dordrecht, Holland., 1981.

182

[160] G. Stoilos, G. B. Stamou, and J. Pan. Fuzzy extensions of owl: Logical properties and

reduction to fuzzy description logics. International Journal of Approximated Reasoning,

51(6), 2010. 23

[161] K. V. Stone, F. Pfenning, D. Scott, and C. Gunter. A denotational approach to

measuring complexity in functional programs. Technical report, School of Computer

Science, Carnegie Mellon University, 2003. 24, 25

[162] T. Sumathi, C. Devasena, and M.Hemalatha. An overview of automated image

annotation approaches. International Journal of Research and Reviews in Information

Sciences, 1(1), March 2011. 1

255
Bibliography

[163] Y. Sure, S. Staab, and R.Studer. Methodology for development and employement of

ontology based knowledge management applications. SIGMOD Record, 31(4):18 – 23,

2002. 65, 66, 234

[164] K. Teerapabolarn. Poisson approximation to the beta binomial distribution. Inter-

national Mathematical Forum, 4(25):1253–1256, 2009. 48

[165] P. Turaga, R. Chellappa, V. S. Subrahmanian, and O. Udrea. Machine recognition

of human activities: A survey. IEEE Transactions on Circuits and Systems for Video

Technology, 18(11):1473–1488, 2008. 31

[166] A. Urquhart. Many-valued logics. In D. M. Gabbay and F. Guenthner, editors,

Handbook of Philosophical Logic: Volume III, Alternatives to Classical Logic, pages

71–116. Dordrecht Reidel, 1986. 145

[167] H. Vassilakis, A. J. Howell, and H. Buxton. Comparison of feedforward (tdrbf) and

generative (tdrgbn) network for gesture based control. In Proceeding of the International

Gesture Workshop on Gesture and Sign Languages in HumanComputer Interaction,

pages 317–321. Soringer-Verlag, 2002. 31

[168] W3C. SPARQL Query Language for RDF: W3C Working Draft. http://www.w3.

org/TR/rdf-sparql-query/. 64, 77, 87

[169] W3C. Notation3 - a readable language for data on the web. http://www.w3.org/

Designnumbers/Notation3.html, 2000. 73

[170] X. Wang, X. Ma, and W. Grimson. Unsupervised activity perception by hierarchical

bayesian models. In Proceedings of the IEEE International Conference on Computer

Vision and Pattern Recognition, 2007. 32

[171] R. Weber, H. Schek, and J. Blott. A quantitative analysis and performance study

for similarity-search methods in high-dimensional spaces. In Proceedings of the Inter-

national Conference on Very Large Databases, pages 194–205, 1998. 75

[172] T. Weithöner, T. Liebig, M. Luther, and S. Böhm. Whats wrong with owl bench-

256
 Bibliography

marks. In Proceedings of the Second International Workshop on Scalable Semantic Web

Knowledge Base Systems, SSWS ’2006, pages 101–114, 2006. 87

[173] T. Weithöner, T. Liebig, M. Luther, S. Böhm, F. Henke, and O. Noppens. Real-world

reasoning with owl. In Proceedings of the 4th European Conference on The Semantic

Web: Research and Applications, ESWC ’2007, pages 296–310, Berlin, Heidelberg, 2007.

Springer-Verlag. 87

[174] Wikipedia. Epistemology. http://en.wikipedia.org/wiki/Epistemology. 15

[175] Wikipedia. Semantic reasoners. http://en.wikipedia.org/wiki/Semantic_

Reasoner. 74, 87

[176] L. A. Zadeh. Fuzzy sets. Information and Control, 8:338–353, 1965. 20, 100, 107

[177] L. A. Zadeh. Outline of a new approach to the analysis of complex systems and

decision processes. IEEE Transactions on System, Man, and Cybernetics, 3(1):28–44,

1973. 34, 100, 107, 162, 211

[178] L. A. Zadeh. Review of shafers a mathematical theory of evidence. AI Magazine,

5(3):81–93, 1984. 38, 114, 116

[179] L. A. Zadeh. Is there a need for fuzzy logic? Information Sciences, 178(13):2751–

2779, 2008. 101, 163

[180] L. Zelink-Manor and M. Irani. Statistical analysis of dynamic actions. IEEE Trans-

action on Pattern Analysis and Machine Intelligence, 28(9):1530–1535, 2006. 31

[181] F. Zhang, Z. M. Ma, J. Cheng, and X. Meng. Fuzzy semantic web ontology learning

from fuzzy uml model. In Proceeding of the 18th ACM conference on Information and

knowledge management, CIKM ’2009, pages 1007–1016, New York, NY, USA, 2009.

ACM. 23

257
258