Raciocínio e Percepção Espacial: Uma abordagem

lógica

Paulo Santos

FEI - São Paulo

September 10, 2010

Paulo Santos ( FEI - São Paulo ) September 10, 2010 1 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 2 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 3 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 4 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 5 / 136

Where is FEI?

S. Paulo, SP
FEI Campus
FEI is the largest engineering school in Brazil, with over 8,000
students
it is already a regional centre of scientific development for the
automotive industry and started investing intensively to become
also a regional centre for intelligent robotics.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 6 / 136
Where is FEI?

S. Paulo, SP
FEI Campus
FEI is the largest engineering school in Brazil, with over 8,000
students
it is already a regional centre of scientific development for the
automotive industry and started investing intensively to become
also a regional centre for intelligent robotics.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 6 / 136
Where is FEI?

S. Paulo, SP
FEI Campus
FEI is the largest engineering school in Brazil, with over 8,000
students
it is already a regional centre of scientific development for the
automotive industry and started investing intensively to become
also a regional centre for intelligent robotics.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 6 / 136
Where is FEI?

S. Paulo, SP
FEI Campus
FEI is the largest engineering school in Brazil, with over 8,000
students
it is already a regional centre of scientific development for the
automotive industry and started investing intensively to become
also a regional centre for intelligent robotics.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 6 / 136
Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 7 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 8 / 136

Introduction and motivation

reasoning about space is ubiquitous: it is necessary in situations

from tying a shoe-lace to urban traffic navigation;
automation of spatial reasoning has led to the development of a
number of application domains:
I geographical information systems (GIS)
I robotics
I commonsense reasoning
I natural language processing
I virtual world modelling and animation
I medical analysis and diagnosis systems
I computer vision

Paulo Santos ( FEI - São Paulo ) September 10, 2010 9 / 136

Introduction and motivation

reasoning about space is ubiquitous: it is necessary in situations

from tying a shoe-lace to urban traffic navigation;
automation of spatial reasoning has led to the development of a
number of application domains:
I geographical information systems (GIS)
I robotics
I commonsense reasoning
I natural language processing
I virtual world modelling and animation
I medical analysis and diagnosis systems
I computer vision

Paulo Santos ( FEI - São Paulo ) September 10, 2010 9 / 136

Introduction and motivation

We acquire knowledge about spatial relationships mainly in two ways:

sensory processing: intensively studied in Computer Vision and
Robotics;
being told, or reading, about spatial arrangements (high-level
reasoning). This is the kind of information processing we’re
concerned about here.
In other words, we’ll be talking about qualitative reasoning, in contrast
to numerical processing.
We will also present computer vision systems whose aim is to bridge
the gap between sensory processing and high-level reasoning:
Cognitive Vision systems.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 10 / 136

Introduction and motivation

We acquire knowledge about spatial relationships mainly in two ways:

sensory processing: intensively studied in Computer Vision and
Robotics;
being told, or reading, about spatial arrangements (high-level
reasoning). This is the kind of information processing we’re
concerned about here.
In other words, we’ll be talking about qualitative reasoning, in contrast
to numerical processing.
We will also present computer vision systems whose aim is to bridge
the gap between sensory processing and high-level reasoning:
Cognitive Vision systems.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 10 / 136

Introduction and motivation

We acquire knowledge about spatial relationships mainly in two ways:

sensory processing: intensively studied in Computer Vision and
Robotics;
being told, or reading, about spatial arrangements (high-level
reasoning). This is the kind of information processing we’re
concerned about here.
In other words, we’ll be talking about qualitative reasoning, in contrast
to numerical processing.
We will also present computer vision systems whose aim is to bridge
the gap between sensory processing and high-level reasoning:
Cognitive Vision systems.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 10 / 136

Aim and inspiration for this tutorial

We follow the believe expressed in Takeo Kanade Keynote

Lecture (given at IJCAI 2003): it is now the time to combine
Computer Vision with Relational models and Reasoning.
This tutorial presents:
I tools and methodology of QSR;
I an overview of major QSR calculi;
I an overview of Cognitive Vision Systems;
I examples of QSR systems for Cognitive Vision.
Aim: make a brief overview of these areas, presenting the context
and foundations to kick start new projects.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 11 / 136

Aim and inspiration for this tutorial

We follow the believe expressed in Takeo Kanade Keynote

Lecture (given at IJCAI 2003): it is now the time to combine
Computer Vision with Relational models and Reasoning.
This tutorial presents:
I tools and methodology of QSR;
I an overview of major QSR calculi;
I an overview of Cognitive Vision Systems;
I examples of QSR systems for Cognitive Vision.
Aim: make a brief overview of these areas, presenting the context
and foundations to kick start new projects.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 11 / 136

Aim and inspiration for this tutorial

We follow the believe expressed in Takeo Kanade Keynote

Lecture (given at IJCAI 2003): it is now the time to combine
Computer Vision with Relational models and Reasoning.
This tutorial presents:
I tools and methodology of QSR;
I an overview of major QSR calculi;
I an overview of Cognitive Vision Systems;
I examples of QSR systems for Cognitive Vision.
Aim: make a brief overview of these areas, presenting the context
and foundations to kick start new projects.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 11 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 12 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 13 / 136

Why a “logic-based approach”?

has a semantics
large variety of distinct logics for different kinds of reasoning
variety of inference mechanisms
provides a tool kit from which it is possible to characterise the
calculi
developing relational calculi is well understood

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Why a “logic-based approach”?

has a semantics
large variety of distinct logics for different kinds of reasoning
variety of inference mechanisms
provides a tool kit from which it is possible to characterise the
calculi
developing relational calculi is well understood

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Why a “logic-based approach”?

has a semantics
large variety of distinct logics for different kinds of reasoning
variety of inference mechanisms
provides a tool kit from which it is possible to characterise the
calculi
developing relational calculi is well understood

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Why a “logic-based approach”?

has a semantics
large variety of distinct logics for different kinds of reasoning
variety of inference mechanisms
provides a tool kit from which it is possible to characterise the
calculi
developing relational calculi is well understood

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Why a “logic-based approach”?

has a semantics
large variety of distinct logics for different kinds of reasoning
variety of inference mechanisms
provides a tool kit from which it is possible to characterise the
calculi
developing relational calculi is well understood

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Why a “logic-based approach”?

has a semantics
large variety of distinct logics for different kinds of reasoning
variety of inference mechanisms
provides a tool kit from which it is possible to characterise the
calculi
developing relational calculi is well understood

Paulo Santos ( FEI - São Paulo ) September 10, 2010 14 / 136

Logical Reasoning

Every man is mortal.

Socrates is a man.
Ergo: Socrates is mortal.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136

Logical Reasoning

Every man is mortal.

Socrates is a man.
Ergo: Socrates is mortal.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136

Logical Reasoning

Every man is mortal.

Socrates is a man.
Ergo: Socrates is mortal.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136

Logical Reasoning

Every man is mortal.

Socrates is a man.
Ergo: Socrates is mortal.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 15 / 136

Logical Reasoning

∀x m(x) → mo(x).
m(Socrates).
Ergo: mo(Socrates).

A→B, A
MODUS PONENS: B

Paulo Santos ( FEI - São Paulo ) September 10, 2010 16 / 136

Logical Reasoning

∀x m(x) → mo(x).
m(Socrates).
Ergo: mo(Socrates).

A→B, A
MODUS PONENS: B

Paulo Santos ( FEI - São Paulo ) September 10, 2010 16 / 136

Logic programming

m(X ) :− mo(X ).
m(s).
?- m(X )

?- X = s

Computational processes: resolution, model checking, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136

Logic programming

m(X ) :− mo(X ).
m(s).
?- m(X )

?- X = s

Computational processes: resolution, model checking, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136

Logic programming

m(X ) :− mo(X ).
m(s).
?- m(X )

?- X = s

Computational processes: resolution, model checking, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136

Logic programming

m(X ) :− mo(X ).
m(s).
?- m(X )

?- X = s

Computational processes: resolution, model checking, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 17 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Logical Reasoning

Deduction: Inferring logical truths

I All the beans from this bag are white.
I These beans are from this bag.
I Ergo, these beans are white. (result)
I Prolog, Otter, Spass, and much logic programming systems.
Abduction: jumping to conclusions
I I took one bean from this bag and it is white
I Ergo, all the beans from this bag are white.
I ACLIP, CIFF, ProLogICA, ...
Induction: generalising from examples
I one bean from this bag is white;
I another bean from this bag is white;
I another bean from this bag is white;
I ...
I Ergo, all the beans from this bag are white.
I Progol, HR, Claudien, FOIL, ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 18 / 136

Knowledge Representation and Reasoning

the logic formalisation of reasoning processes, capable of inferring

knowledge from representations of the world;
the construction of a medium for efficient computation, in which
the formal representation provides the means to organise
domain knowledge allowing for efficient (and consistent)
queries, updates and revisions of the knowledge base;
the rigorous treatment of ontological commitments, which provide
the base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actions
I nature of knowledge about temporal entities
I belief change
I vagueness
I spatial entities
I Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993

Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136

Knowledge Representation and Reasoning

the logic formalisation of reasoning processes, capable of inferring

knowledge from representations of the world;
the construction of a medium for efficient computation, in which
the formal representation provides the means to organise
domain knowledge allowing for efficient (and consistent)
queries, updates and revisions of the knowledge base;
the rigorous treatment of ontological commitments, which provide
the base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actions
I nature of knowledge about temporal entities
I belief change
I vagueness
I spatial entities
I Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993

Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136

Knowledge Representation and Reasoning

the logic formalisation of reasoning processes, capable of inferring

knowledge from representations of the world;
the construction of a medium for efficient computation, in which
the formal representation provides the means to organise
domain knowledge allowing for efficient (and consistent)
queries, updates and revisions of the knowledge base;
the rigorous treatment of ontological commitments, which provide
the base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actions
I nature of knowledge about temporal entities
I belief change
I vagueness
I spatial entities
I Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993

Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136

Knowledge Representation and Reasoning

the logic formalisation of reasoning processes, capable of inferring

knowledge from representations of the world;
the construction of a medium for efficient computation, in which
the formal representation provides the means to organise
domain knowledge allowing for efficient (and consistent)
queries, updates and revisions of the knowledge base;
the rigorous treatment of ontological commitments, which provide
the base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actions
I nature of knowledge about temporal entities
I belief change
I vagueness
I spatial entities
I Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993

Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136

Knowledge Representation and Reasoning

the logic formalisation of reasoning processes, capable of inferring

knowledge from representations of the world;
the construction of a medium for efficient computation, in which
the formal representation provides the means to organise
domain knowledge allowing for efficient (and consistent)
queries, updates and revisions of the knowledge base;
the rigorous treatment of ontological commitments, which provide
the base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actions
I nature of knowledge about temporal entities
I belief change
I vagueness
I spatial entities
I Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993

Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136

Knowledge Representation and Reasoning

the logic formalisation of reasoning processes, capable of inferring

knowledge from representations of the world;
the construction of a medium for efficient computation, in which
the formal representation provides the means to organise
domain knowledge allowing for efficient (and consistent)
queries, updates and revisions of the knowledge base;
the rigorous treatment of ontological commitments, which provide
the base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actions
I nature of knowledge about temporal entities
I belief change
I vagueness
I spatial entities
I Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993

Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136

Knowledge Representation and Reasoning

the logic formalisation of reasoning processes, capable of inferring

knowledge from representations of the world;
the construction of a medium for efficient computation, in which
the formal representation provides the means to organise
domain knowledge allowing for efficient (and consistent)
queries, updates and revisions of the knowledge base;
the rigorous treatment of ontological commitments, which provide
the base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actions
I nature of knowledge about temporal entities
I belief change
I vagueness
I spatial entities
I Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993

Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136

Knowledge Representation and Reasoning

the logic formalisation of reasoning processes, capable of inferring

knowledge from representations of the world;
the construction of a medium for efficient computation, in which
the formal representation provides the means to organise
domain knowledge allowing for efficient (and consistent)
queries, updates and revisions of the knowledge base;
the rigorous treatment of ontological commitments, which provide
the base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actions
I nature of knowledge about temporal entities
I belief change
I vagueness
I spatial entities
I Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993

Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136

Knowledge Representation and Reasoning

the logic formalisation of reasoning processes, capable of inferring

knowledge from representations of the world;
the construction of a medium for efficient computation, in which
the formal representation provides the means to organise
domain knowledge allowing for efficient (and consistent)
queries, updates and revisions of the knowledge base;
the rigorous treatment of ontological commitments, which provide
the base rules that guide reasoning about the world. For instance,
I what should or should not be considered as the effects of actions
I nature of knowledge about temporal entities
I belief change
I vagueness
I spatial entities
I Davis et al. “what is Knowledge Representation?”, AI Magazine vol
14, 1993

Paulo Santos ( FEI - São Paulo ) September 10, 2010 19 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 20 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 21 / 136

What is QSR?

The formal representation of (qualitative) spatial knowledge in terms of

some basic entities and primitive relations in order to allow meaningful
and, sometimes, efficient inference methods about space.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 22 / 136

Spatial Reasoning

“The basic stories we know best are small stories of events in space:
The wind blows clouds through the sky, a child throws a rock, a mother
pours milk into a glass, a whale swims through the water. These
stories constitute our world.” (M.Turner, The Literary Mind)

Paulo Santos ( FEI - São Paulo ) September 10, 2010 23 / 136

Philosophical origins of QSR

the fundamental principles of Geometry were first investigated in

ancient Greece by Thales circa 600 B.C.)
the laws of valid argument in terms of logical modes of inference
were studied separately by early Greek philosophers:
analytic geometry (Descartes 1637)
19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolic
geometry (1829)
I Cantor’s point-set topology (1845-1918):
I Poincaré’s algebraic topology

Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136

Philosophical origins of QSR

the fundamental principles of Geometry were first investigated in

ancient Greece by Thales circa 600 B.C.)
the laws of valid argument in terms of logical modes of inference
were studied separately by early Greek philosophers:
analytic geometry (Descartes 1637)
19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolic
geometry (1829)
I Cantor’s point-set topology (1845-1918):
I Poincaré’s algebraic topology

Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136

Philosophical origins of QSR

the fundamental principles of Geometry were first investigated in

ancient Greece by Thales circa 600 B.C.)
the laws of valid argument in terms of logical modes of inference
were studied separately by early Greek philosophers:
analytic geometry (Descartes 1637)
19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolic
geometry (1829)
I Cantor’s point-set topology (1845-1918):
I Poincaré’s algebraic topology

Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136

Philosophical origins of QSR

the fundamental principles of Geometry were first investigated in

ancient Greece by Thales circa 600 B.C.)
the laws of valid argument in terms of logical modes of inference
were studied separately by early Greek philosophers:
analytic geometry (Descartes 1637)
19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolic
geometry (1829)
I Cantor’s point-set topology (1845-1918):
I Poincaré’s algebraic topology

Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136

Philosophical origins of QSR

the fundamental principles of Geometry were first investigated in

ancient Greece by Thales circa 600 B.C.)
the laws of valid argument in terms of logical modes of inference
were studied separately by early Greek philosophers:
analytic geometry (Descartes 1637)
19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolic
geometry (1829)
I Cantor’s point-set topology (1845-1918):
I Poincaré’s algebraic topology

Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136

Philosophical origins of QSR

the fundamental principles of Geometry were first investigated in

ancient Greece by Thales circa 600 B.C.)
the laws of valid argument in terms of logical modes of inference
were studied separately by early Greek philosophers:
analytic geometry (Descartes 1637)
19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolic
geometry (1829)
I Cantor’s point-set topology (1845-1918):
I Poincaré’s algebraic topology

Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136

Philosophical origins of QSR

the fundamental principles of Geometry were first investigated in

ancient Greece by Thales circa 600 B.C.)
the laws of valid argument in terms of logical modes of inference
were studied separately by early Greek philosophers:
analytic geometry (Descartes 1637)
19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolic
geometry (1829)
I Cantor’s point-set topology (1845-1918):
I Poincaré’s algebraic topology

Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136

Philosophical origins of QSR

the fundamental principles of Geometry were first investigated in

ancient Greece by Thales circa 600 B.C.)
the laws of valid argument in terms of logical modes of inference
were studied separately by early Greek philosophers:
analytic geometry (Descartes 1637)
19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolic
geometry (1829)
I Cantor’s point-set topology (1845-1918):
I Poincaré’s algebraic topology

Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136

Philosophical origins of QSR

the fundamental principles of Geometry were first investigated in

ancient Greece by Thales circa 600 B.C.)
the laws of valid argument in terms of logical modes of inference
were studied separately by early Greek philosophers:
analytic geometry (Descartes 1637)
19th century revolution on spatial reasoning:
I Non-euclidean geometries (e.g. Lobachevsky’s hyperbolic
geometry (1829)
I Cantor’s point-set topology (1845-1918):
I Poincaré’s algebraic topology

Paulo Santos ( FEI - São Paulo ) September 10, 2010 24 / 136

Philosophical origins of QSR: 20th century

Bertrand Russell and Alfred Whitehead: principles of logic to

phenomenological theories, describing the world as it is perceived
through sense data;
Whitehead (1920): a theory of the perceived world should have as
basic entities the very ’phenomena’: integral objects or events
I geometry becomes concerned with relationships between regions
occupied by bodies and dynamical laws with qualitative rules about
world events

Paulo Santos ( FEI - São Paulo ) September 10, 2010 25 / 136

Philosophical origins of QSR: 20th century

Bertrand Russell and Alfred Whitehead: principles of logic to

phenomenological theories, describing the world as it is perceived
through sense data;
Whitehead (1920): a theory of the perceived world should have as
basic entities the very ’phenomena’: integral objects or events
I geometry becomes concerned with relationships between regions
occupied by bodies and dynamical laws with qualitative rules about
world events

Paulo Santos ( FEI - São Paulo ) September 10, 2010 25 / 136

Philosophical origins of QSR: 20th century

Bertrand Russell and Alfred Whitehead: principles of logic to

phenomenological theories, describing the world as it is perceived
through sense data;
Whitehead (1920): a theory of the perceived world should have as
basic entities the very ’phenomena’: integral objects or events
I geometry becomes concerned with relationships between regions
occupied by bodies and dynamical laws with qualitative rules about
world events

Paulo Santos ( FEI - São Paulo ) September 10, 2010 25 / 136

Origins of QSR in Artificial Intelligence

phenomenological theories: closer to human reasoning

would it be possible to automate them ?
at least for some particular domains?
the construction of formal theories about the qualitative
relationships between basic spatial (“phenomenological”) theories
is the main goal of qualitative spatial reasoning in AI.

but perhaps we should talk about time before talking about

space...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 26 / 136

Origins of QSR in Artificial Intelligence

phenomenological theories: closer to human reasoning

would it be possible to automate them ?
at least for some particular domains?
the construction of formal theories about the qualitative
relationships between basic spatial (“phenomenological”) theories
is the main goal of qualitative spatial reasoning in AI.

but perhaps we should talk about time before talking about

space...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 26 / 136

Origins of QSR in Artificial Intelligence

phenomenological theories: closer to human reasoning

would it be possible to automate them ?
at least for some particular domains?
the construction of formal theories about the qualitative
relationships between basic spatial (“phenomenological”) theories
is the main goal of qualitative spatial reasoning in AI.

but perhaps we should talk about time before talking about

space...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 26 / 136

Origins of QSR in Artificial Intelligence

phenomenological theories: closer to human reasoning

would it be possible to automate them ?
at least for some particular domains?
the construction of formal theories about the qualitative
relationships between basic spatial (“phenomenological”) theories
is the main goal of qualitative spatial reasoning in AI.

but perhaps we should talk about time before talking about

space...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 26 / 136

Origins of QSR in Artificial Intelligence

phenomenological theories: closer to human reasoning

would it be possible to automate them ?
at least for some particular domains?
the construction of formal theories about the qualitative
relationships between basic spatial (“phenomenological”) theories
is the main goal of qualitative spatial reasoning in AI.

but perhaps we should talk about time before talking about

space...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 26 / 136

Allen’s interval calculus

Figure: adapted from M. Ragni, Reasoning in Dynamic Environments, KI2006

13 relations
any two intervals expressed by one and only one relation (JEPD)
temporal reasoning: derive facts about temporal intervals

Paulo Santos ( FEI - São Paulo ) September 10, 2010 27 / 136

Allen’s interval calculus: applications

represent activities and temporal knowledge

planning and scheduling
temporal databases
natural language understanding

Paulo Santos ( FEI - São Paulo ) September 10, 2010 28 / 136

Tools of temporal reasoning

Conceptual Neighbourhood Diagrams: graphs representing in

their vertices relations on some specific objects; whereas their
edges represent continuous transitions between these relations.
I Continuous transitions: in between adjacent vertices of the graph
there is no other relations that the entities in the relation’s
arguments can assume.
Composition (or transitivity) tables: given two relations on any
objects a, b, ad c (e.g., R1 (a, b) and R2 (b, c)), the composition
table entry for R1 (a, b) and R2 (b, c) gives the minimal set of
disjunctions R3 (b, c) of the possible relations between a and c.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 29 / 136

Tools of temporal reasoning

Conceptual Neighbourhood Diagrams: graphs representing in

their vertices relations on some specific objects; whereas their
edges represent continuous transitions between these relations.
I Continuous transitions: in between adjacent vertices of the graph
there is no other relations that the entities in the relation’s
arguments can assume.
Composition (or transitivity) tables: given two relations on any
objects a, b, ad c (e.g., R1 (a, b) and R2 (b, c)), the composition
table entry for R1 (a, b) and R2 (b, c) gives the minimal set of
disjunctions R3 (b, c) of the possible relations between a and c.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 29 / 136

Tools of temporal reasoning

Conceptual Neighbourhood Diagrams: graphs representing in

their vertices relations on some specific objects; whereas their
edges represent continuous transitions between these relations.
I Continuous transitions: in between adjacent vertices of the graph
there is no other relations that the entities in the relation’s
arguments can assume.
Composition (or transitivity) tables: given two relations on any
objects a, b, ad c (e.g., R1 (a, b) and R2 (b, c)), the composition
table entry for R1 (a, b) and R2 (b, c) gives the minimal set of
disjunctions R3 (b, c) of the possible relations between a and c.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 29 / 136

Conceptual Neighbourhood Diagram: illustration

Figure: adapted from M. Ragni, Reasoning in Dynamic Environments, KI2006

Paulo Santos ( FEI - São Paulo ) September 10, 2010 30 / 136

Transitivity table

< > d di ...

before (<) < no info <; o; m; d; < ...
s
after (>) no info > >; oi; mi; > ...
d; f
during (d) < > no info. d ...
contains <; o; m; di; >; oi; di; o; oi; dur; di ...
(di) fi mi; si con; =
overlaps < >; oi; di; o; d; s <; o; m ...
(o) mi; si
overlapped <; o; m; di; > oi; d; f >; oi; mi; ...
by (oi) fi di;si
meets (m) < >; oi; mi; o; d; s < ...
di; si
met-by <; o; m; di; > d <; o; m; di; ...
(mi) fi fi

Paulo Santos ( FEI - São Paulo ) September 10, 2010 31 / 136

Basic elements in a spatial theory

space: absolute or relative; global or local;

basic entities: points, regions, directions, bodies, shapes, things,
sense-data, ...
primitive relations: meet, between, connect, part-of, ...
I the set of relations is usually Jointly Exhaustive and Pairwise
Disjoints (JEPD)
formal tools: axiomatic (deriving axioms and proving theorems),
algebraic (encode knowledge with operators and equations),
purely logical (design of a spatial logic)

Paulo Santos ( FEI - São Paulo ) September 10, 2010 32 / 136

Tools of QSR

Conceptual Neighbourhood Diagrams

Composition (or transitivity) tables

Paulo Santos ( FEI - São Paulo ) September 10, 2010 33 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 34 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 35 / 136

Region Connection Calculus (RCC)

many sorted, first order logic axiomatisation of spatial regions

based on a primitive binary relation about the connection between
two regions (C/2).
C(x, y ) (“x is connected to y ”), i.e., the topological closures of x
and y share at least one point
∀x C(x, x);
∀xy C(x, y ) → C(y , x);
∀xyz (C(z, x) ↔ C(z, y )) → x = y .

Paulo Santos ( FEI - São Paulo ) September 10, 2010 36 / 136

Region Connection Calculus

P(x, y ), “x is part of y”;

O(x, y ), “x overlaps y”;
PP(x, y ), “x is proper part of y”;
Pi/2 and PPi/2 are inverse relations of P/2 and PP/2, resp;
DC(x, y ), “x is disconnected from y”;
EQ(x, y ), “x is equal to y”;
PO(x, y ), “x partially overlaps y”;
EC(x, y ), “x is externally connected to y”;
TPP(x, y ), “x is tangencial proper part of y”;
NTPP(x, y ), “x is non-tangential proper part of y”;
TPPi/2 NTPPi/2 are inverses of TPP/2 and NTPP/2
{DC(x, y ), EQ(x, y ), PO(x, y ), EC(x, y ), TPP(x, y ), NTPP(x, y ),
TPPi/2, NTPPi/2 } is a JEPD set known as RCC8.
Paulo Santos ( FEI - São Paulo ) September 10, 2010 37 / 136
RCC: conceptual neighbourhood diagram

x TPP
y y
x

x x x NTPP
x y EQ
y y
y NTPPi

y y x
DC EC PO x TPPi

Paulo Santos ( FEI - São Paulo ) September 10, 2010 38 / 136

Region Connection Calculus

((DC(x, y ) ↔ ¬C(x, y ))). (1)

((P(x, y ) ↔ (∀z(C(z, x) → C(z, y ))))). (2)
((PP(x, y ) ↔ (P(x, y ) ∧ ¬P(y , x)))). (3)
((EQ(x, y ) ↔ (P(x, y ) ∧ P(y , x)))). (4)
((O(x, y ) ↔ (∃z(P(z, x) ∧ P(z, y ))))). (5)
((PO(x, y ) ↔ (O(x, y ) ∧ ¬P(x, y ) ∧ ¬P(y , x)))). (6)
((DR(x, y ) ↔ −O(x, y ))). (7)
((EC(x, y ) ↔ (C(x, y ) ∧ ¬O(x, y )))). (8)
((TPP(x, y ) ↔ (PP(x, y ) ∧ (∃z(EC(z, x) ∧ EC(z, y )))))). (9)
((NTPP(x, y ) ↔ (PP(x, y ) ∧ ¬(∃z(EC(z, x) ∧ EC(z, y )))))). (10)
((Pi(x, y ) ↔ P(y , x))). (11)
((PPi(x, y ) ↔ PP(y , x))). (12)
((TPPi(x, y ) ↔ TPP(y , x))). (13)
Paulo Santos ( FEI - São Paulo ) September 10, 2010 39 / 136
RCC: Transitivity table

DC EC PO TPP ...
DC no info. DR,PO, DR,PO, DR,PO,
PP PP PP ...
EC DR,PO, DR,PO, DR,PO, EC,PO, ...
PPi TPP PP PP
PO DR,PO, DR,PO, no info. PO,PP ...
PPi PPi
TPP DC DR DR,PO, PP ...
PP
NTPP DC DC DR,PO, NTPP ...
PP
TPPi DR,PO, EC,PO, PO,PPi PO,TPP ...
PPi PPi
NTPPi DR,PO, PO,PPi PO,PPi PO,PPi ...
PPi
EQ DC EC PO TPP ...

Paulo Santos ( FEI - São Paulo ) September 10, 2010 40 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 41 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 42 / 136

Lines of Sight Calculus

Represent the relative positions between pairs of (non-overlapping)

convex bodies

Figure: Randell et a. From Images to Bodies: Modelling and Exploiting Spatial

Occlusion and Motion Parallax. IJCAI 2001

Paulo Santos ( FEI - São Paulo ) September 10, 2010 43 / 136

Lines of Sight Calculus

B A C

B A JC

JH PH JF JHI PHI JFI

B B A A B B
A A B B A A

B A B A A B A B
A B B A
H EH F HI EHI FI

Paulo Santos ( FEI - São Paulo ) September 10, 2010 44 / 136

Lines of Sight Calculus

14 relations:
I C(x, y ), is clear from;
I JC(x, y ), is just clear from;
I PH(x, y ), partially hides;
I PHI(x, y ), is partially hidden by;
I JH(x, y ), just hides;
I JHI(x, y ), is just hidden;
I H(x, y ), hides;
I HI(x, y ), is hidden by;
I EH(x, y ), exactly hides;
I EHI(x, y ), is exactly hidden;
I F (x, y ), is in front of;
I FI(x, y ), has y in front of it;
I JF (x, y ), is just in front of;
I JFI(x, y ), has y just in front of it;

Paulo Santos ( FEI - São Paulo ) September 10, 2010 45 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 46 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 47 / 136

Region Occlusion Calculus (ROC)

Region Occlusion Calculus (ROC) is an extension of RCC to

represent the various possibilities of interposition (occlusion)
between arbitrary shaped bodies.
Two functions: region and image.
I region: maps a physical body to is occupancy region.
I image maps a physical body to its bi-dimensional projection from a
particular viewpoint.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 48 / 136

Region Occlusion Calculus (ROC)

ROC primitive relations: C/2 and TotallyOccludes(x, y , ν) (“body x

totally ocludes body y from the viewpoint ν”).

∀x∀ν¬TotallyOccludes(x, x, ν)
∀x∀y ∀z∀νTotallyOccludes(x, y , ν) ∧ TotallyOccludes(y , z, ν)) →
TotallyOccludes(x, z, ν)

Paulo Santos ( FEI - São Paulo ) September 10, 2010 49 / 136

Region Occlusion Calculus

The following axioms introduce RCC8 in ROC:.

((TotallyOccludes(x, y , ν) ∧ P(region(z), region(y ))) →

TotallyOccludes(x, z, ν))
(TotallyOccludes(x, y , ν) → ∀z(P(region(z), region(y ))) →
¬TotallyOccludes(z, x, ν))
(TotallyOccludes(x, y , ν) → ∀z∀u(P(region(z), region(x))∧
P(region(u), region(y ))) → ¬TotallyOccludes(u, z, ν))
∃y ∃z(P(region(y ), region(x)) ∧ P(region(z), region(x))∧
TotallyOccludes(y , z, ν))
(TotallyOccludes(x, y , ν) → P(image(y , ν), image(x, ν)))

Paulo Santos ( FEI - São Paulo ) September 10, 2010 50 / 136

Region Occlusion Calculus

We can introduce a weaker notion of occlusion Occludes/3:

Occludes(x, y , ν) ↔ ∃z∃u(P(region(z), region(x))

∧ P(region(u), region(y ))∧
TotallyOccludes(z, u, ν))

Paulo Santos ( FEI - São Paulo ) September 10, 2010 51 / 136

Region Occlusion Calculus

Non occlusion, partial occlusion, mutual occlusion:

PartiallyOccludes(x, y , ν) ↔ Occludes(x, y , ν)∧

¬TotallyOccludes(x, y , ν) ∧ ¬Occludes(y , x, ν)
MutuallyOccludes(x, y , ν) ↔ Occludes(x, y , ν) ∧ Occludes(y , x, ν)
NonOccludes(x, y , ν) ↔ ¬Occludes(x, y , ν) ∧ ¬Occludes(y , x, ν)
NonOccludes(x, y , ν) → DR(image(x, ν), image(y , ν))
PartiallyOccludes(x, y , ν) →
(PO(image(x, ν), image(y , ν)) ∨ PP(image(x, ν), image(y , ν)))
MutuallyOccludes(x, y , ν) →
(PO(image(x, ν), image(y , ν)) ∨ P(image(x, ν), image(y , ν))∨
PI(image(x, ν), image(y , ν)))

Paulo Santos ( FEI - São Paulo ) September 10, 2010 52 / 136

Region Occlusion Calculus
NonOccludesDC MutuallyOccludesNTPP

NonOccludesEC MutuallyOccludesTPP

PartiallyOccludesPO MutuallyOccludesEQ

MutuallyOccludesPO PartiallyOccludesNTPP

PartiallyOccludesTPP TotallyOccludesEQ

PartiallyOccludesPO !1 MutuallyOccludesTPP !1

TotallyOccludesTPPI PartiallyOccludesTPP !1

!1
TotallyOccludesNTPPI PartiallyOccludesNTPP

!1
TotallyOccludesTPPI !1 MutuallyOccludesNTPP

TotallyOccludesEQ !1
TotallyOccludesNTPPI !1

Paulo Santos ( FEI - São Paulo ) September 10, 2010 53 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 54 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 55 / 136

Cardinal Direction Calculus

Cardinal Direction Calculus (CDC) is a formalism for reasoning

about the directions between spatial objects
9 base relations: north, south, east, west, northeast, northwest,
southeast, southwest e EQ (EQ(x, y ) means “x is at the same
direction as y”).
Main goal of CDC is to infer facts about the relative direction of
two objects A and B from the known directions between A and C
(A 6= C and B 6= C).
I E.g., from north(A, B) and northeast(B, C), the task is to calculate
the possible directions between A e C.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 56 / 136

Cardinal Direction Calculus

Cardinal Direction Calculus (CDC) is a formalism for reasoning

about the directions between spatial objects
9 base relations: north, south, east, west, northeast, northwest,
southeast, southwest e EQ (EQ(x, y ) means “x is at the same
direction as y”).
Main goal of CDC is to infer facts about the relative direction of
two objects A and B from the known directions between A and C
(A 6= C and B 6= C).
I E.g., from north(A, B) and northeast(B, C), the task is to calculate
the possible directions between A e C.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 56 / 136

Cardinal Direction Calculus

Figure: Adapted from SparQ User Manual v0.7

Paulo Santos ( FEI - São Paulo ) September 10, 2010 57 / 136
Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 58 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 59 / 136

Double Cross Calculus

is a calculus that defines the direction of a point with respect to a

directed line segment
15 ternary relations on points
represents every distinct relation between the directions left-right
and front-back (e.g.left-front, left-back, left-line, left -perpendicular,
straight-front, ...)
motivation: qualitative description of paths

Paulo Santos ( FEI - São Paulo ) September 10, 2010 60 / 136

Double Cross Calculus

is a calculus that defines the direction of a point with respect to a

directed line segment
15 ternary relations on points
represents every distinct relation between the directions left-right
and front-back (e.g.left-front, left-back, left-line, left -perpendicular,
straight-front, ...)
motivation: qualitative description of paths

Paulo Santos ( FEI - São Paulo ) September 10, 2010 60 / 136

Double Cross Calculus

is a calculus that defines the direction of a point with respect to a

directed line segment
15 ternary relations on points
represents every distinct relation between the directions left-right
and front-back (e.g.left-front, left-back, left-line, left -perpendicular,
straight-front, ...)
motivation: qualitative description of paths

Paulo Santos ( FEI - São Paulo ) September 10, 2010 60 / 136

Double Cross Calculus

is a calculus that defines the direction of a point with respect to a

directed line segment
15 ternary relations on points
represents every distinct relation between the directions left-right
and front-back (e.g.left-front, left-back, left-line, left -perpendicular,
straight-front, ...)
motivation: qualitative description of paths

Paulo Santos ( FEI - São Paulo ) September 10, 2010 60 / 136

Double Cross Calculus

Figure: Adapted from SparQ User Manual v0.7

Paulo Santos ( FEI - São Paulo ) September 10, 2010 61 / 136

Double Cross Calculus

Paulo Santos ( FEI - São Paulo ) September 10, 2010 62 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 63 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 64 / 136

Other Calculi

size and distance: are defined either in absolute scale (whereas

the relations < or >) are introduced in the usual way), or in
relative terms, where the relative connection between three
objects is used to define the relations of proximity (near than, far
than) and equidistance . Size and distance calculi are usually
coupled with other calculi to extend their expressivity,
shapes: one of the least understood areas of QSR. In general,
shape is defined by means of a number of primitives such as
interior or boundary.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 65 / 136

Other Calculi

default: not much has been done wrt default theories about
space; Shanahan formalises a pre-condition about spatial
occupancy of objects assuming that space is empty by default.
spatial change: base to the development of spatial change is the
work of Galton, where both time instants and intervals are
included. Two predicates are used to this end: HoldsT represents
a spatial state that is true at a time instant, whereas HoldsI
represents true stated during a time interval. From this, 8 distinct
kinds of transitions between pairs of states are defined in order to
represent the relation of two states in time.
Qualitative Trajectory Calculi, Line Segments, Dipole Calculi, and
many others!

Paulo Santos ( FEI - São Paulo ) September 10, 2010 66 / 136

SparQ

toolbox for QSR in applications

I reference implementations of QSR
I typical QSR procedures
I uniform interface
http://www.sfbtr8.uni-bremen.de/project/r3/sparq/

Paulo Santos ( FEI - São Paulo ) September 10, 2010 67 / 136

Applications of QSR

qualitative simulation of physical systems

syntax and semantics of visual programming languages
databases integration
GIS
real time event recognition
robotics

Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136

Applications of QSR

qualitative simulation of physical systems

syntax and semantics of visual programming languages
databases integration
GIS
real time event recognition
robotics

Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136

Applications of QSR

qualitative simulation of physical systems

syntax and semantics of visual programming languages
databases integration
GIS
real time event recognition
robotics

Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136

Applications of QSR

qualitative simulation of physical systems

syntax and semantics of visual programming languages
databases integration
GIS
real time event recognition
robotics

Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136

Applications of QSR

qualitative simulation of physical systems

syntax and semantics of visual programming languages
databases integration
GIS
real time event recognition
robotics

Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136

Applications of QSR

qualitative simulation of physical systems

syntax and semantics of visual programming languages
databases integration
GIS
real time event recognition
robotics

Paulo Santos ( FEI - São Paulo ) September 10, 2010 68 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 69 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 70 / 136

Tractability and computability

efficiency is inversely proportional to expressivity

first order formulation of mereotopology is not decidable
I find a decidable subset (e.g. RCC8)
I look for tractable subsets

Paulo Santos ( FEI - São Paulo ) September 10, 2010 71 / 136

Tractability and computability

efficiency is inversely proportional to expressivity

first order formulation of mereotopology is not decidable
I find a decidable subset (e.g. RCC8)
I look for tractable subsets

Paulo Santos ( FEI - São Paulo ) September 10, 2010 71 / 136

Tractability and computability

efficiency is inversely proportional to expressivity

first order formulation of mereotopology is not decidable
I find a decidable subset (e.g. RCC8)
I look for tractable subsets

Paulo Santos ( FEI - São Paulo ) September 10, 2010 71 / 136

Tractability and computability

efficiency is inversely proportional to expressivity

first order formulation of mereotopology is not decidable
I find a decidable subset (e.g. RCC8)
I look for tractable subsets

Paulo Santos ( FEI - São Paulo ) September 10, 2010 71 / 136

Tractability and computability

efficiency is inversely proportional to expressivity

first order formulation of mereotopology is not decidable
I find a decidable subset (e.g. RCC8)
I look for tractable subsets

Paulo Santos ( FEI - São Paulo ) September 10, 2010 71 / 136

Tractability and computability

Tractable subsets of QSR formalism is the subject of a number of

papers. In particular (Renz e Cohn 2008) presents the following
ingredients for finding tractable subsets of such formalism:
a method to prove that a given subset if tractable
a method to suggest possible tractable subsets
in order to prove that a given set of relations is tractable, it is
sufficient to prove that the inclusion of any new relation makes the
set intractable
J. Renz, Qualitative Spatial and Temporal Reasoning: Efficient
Algorithms for Everyone, in: Proc (IJCAI-07)

Paulo Santos ( FEI - São Paulo ) September 10, 2010 72 / 136

Tractability and computability

Tractable subsets of QSR formalism is the subject of a number of

papers. In particular (Renz e Cohn 2008) presents the following
ingredients for finding tractable subsets of such formalism:
a method to prove that a given subset if tractable
a method to suggest possible tractable subsets
in order to prove that a given set of relations is tractable, it is
sufficient to prove that the inclusion of any new relation makes the
set intractable
J. Renz, Qualitative Spatial and Temporal Reasoning: Efficient
Algorithms for Everyone, in: Proc (IJCAI-07)

Paulo Santos ( FEI - São Paulo ) September 10, 2010 72 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 73 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 74 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 75 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 76 / 136

Our knowledge of the external world

(...) what is actually given in sense is much less than most

people would naturally suppose, and (...) much of what at first
sight seems to be given is really inferred. This applies
especially in regard to our space-perceptions. For instance,
we unconsciously infer the “real” size and shape of a visible
object from its apparent size and shape, according to its
distance and our point of view. (...). Thus, the first step in the
analyses of data, namely, the discovery of what is really given
in sense, is full of difficulty.
[B. Russell (1914), Our Knowledge of
the External World, pp.75-76]

Although these ideas are at the foundations of QSR, their application

in scene analysis is still work in progress.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 77 / 136

Our knowledge of the external world

(...) what is actually given in sense is much less than most

people would naturally suppose, and (...) much of what at first
sight seems to be given is really inferred. This applies
especially in regard to our space-perceptions. For instance,
we unconsciously infer the “real” size and shape of a visible
object from its apparent size and shape, according to its
distance and our point of view. (...). Thus, the first step in the
analyses of data, namely, the discovery of what is really given
in sense, is full of difficulty.
[B. Russell (1914), Our Knowledge of
the External World, pp.75-76]

Although these ideas are at the foundations of QSR, their application

in scene analysis is still work in progress.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 77 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 78 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 79 / 136

What is CogVis?
from ECvision network

A Cognitive Vision System can achieve the four levels of generic

visual functionality: Detection, Localisation, Recognition,
Understanding (role, context, purpose)
and exhibits purposive goal-directed behaviour, is adaptive to
unforeseen changes, and can anticipate the occurrence of
objects and events. This is achieved through:
I Learning semantic knowledge (form, function and behaviours)
I Retention of knowledge (about the cognitive system, its
environment, and the relationship with the environment)
I Deliberation about objects and events, including the cognitive
system itself.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 80 / 136

Some approaches for CogVis

VITRA system- Visual Translator: Integration of computer vision

and natural language processing
ALVEN system - textual description of all heart dynamics via
X-Ray image sequences.
Brand’s visual understanding through causal analysis
Siskind’s systems - Event classification from camera input using
force dynamics
Leeds traffic interaction - Modelling traffic interaction using learnt
qualitative spatio-temporal relations and variable length Markov
models
Leeds Protocol learning - Combining continuous and symbolic
models to learn games from observation
FEI contributions - Spatial reasoning image analysis

Paulo Santos ( FEI - São Paulo ) September 10, 2010 81 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 82 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 83 / 136

Nagel’s Image Sequence Evaluation

Requirements and purpose:

exhaustive internal representation for all tasks and experimental
conditions it is expected to handle
situation: suitable intermediate representation during the
evaluation of image sequences
derive scene-specific conceptual descriptions from image
sequences, based on general assumptions about motion as the
cause of observable changes
Image sequence evaluation is based on the idea that the flow of image
sequences reflects (to some extent) the coherency of the conceptual
world.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 84 / 136

Nagel’s Image Sequence Evaluation

Requirements and purpose:

exhaustive internal representation for all tasks and experimental
conditions it is expected to handle
situation: suitable intermediate representation during the
evaluation of image sequences
derive scene-specific conceptual descriptions from image
sequences, based on general assumptions about motion as the
cause of observable changes
Image sequence evaluation is based on the idea that the flow of image
sequences reflects (to some extent) the coherency of the conceptual
world.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 84 / 136

Nagel’s Image Sequence Evaluation

Requirements and purpose:

exhaustive internal representation for all tasks and experimental
conditions it is expected to handle
situation: suitable intermediate representation during the
evaluation of image sequences
derive scene-specific conceptual descriptions from image
sequences, based on general assumptions about motion as the
cause of observable changes
Image sequence evaluation is based on the idea that the flow of image
sequences reflects (to some extent) the coherency of the conceptual
world.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 84 / 136

Nagel’s Image Sequence Evaluation

Requirements and purpose:

exhaustive internal representation for all tasks and experimental
conditions it is expected to handle
situation: suitable intermediate representation during the
evaluation of image sequences
derive scene-specific conceptual descriptions from image
sequences, based on general assumptions about motion as the
cause of observable changes
Image sequence evaluation is based on the idea that the flow of image
sequences reflects (to some extent) the coherency of the conceptual
world.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 84 / 136

Nagel’s Image Sequence Evaluation

Intermediate levels of description:

change: any deviation of sensor signal which significantly differs
from noise
event: any change which has been defined a priori as a primitive
for the construction of more complex descriptions
verb: describe activities
history: extended sequence of activities

Paulo Santos ( FEI - São Paulo ) September 10, 2010 85 / 136

Some early systems

Morio (Nagel 1983), Naos (Neumann 1986), Epex (Nagel 1987),

CityTour (Herzog 1986)
usually describing traffic scenes/situations from an static viewpoint
(using motion verbs such as overtake, approach, move, beside,
recede
but also to describe a walking person (Hogg 1983), heart motion
(Tsotos 1980) and so on

Paulo Santos ( FEI - São Paulo ) September 10, 2010 86 / 136

Some early systems

Morio (Nagel 1983), Naos (Neumann 1986), Epex (Nagel 1987),

CityTour (Herzog 1986)
usually describing traffic scenes/situations from an static viewpoint
(using motion verbs such as overtake, approach, move, beside,
recede
but also to describe a walking person (Hogg 1983), heart motion
(Tsotos 1980) and so on

Paulo Santos ( FEI - São Paulo ) September 10, 2010 86 / 136

Some early systems

Morio (Nagel 1983), Naos (Neumann 1986), Epex (Nagel 1987),

CityTour (Herzog 1986)
usually describing traffic scenes/situations from an static viewpoint
(using motion verbs such as overtake, approach, move, beside,
recede
but also to describe a walking person (Hogg 1983), heart motion
(Tsotos 1980) and so on

Paulo Santos ( FEI - São Paulo ) September 10, 2010 86 / 136

Project Vitra: Visual Translator

long term project: 80’s and 90’s

design and construction of integrated knowledge-based systems
capable of translating visual information into natural language
descriptions
general goals:
I extend the scope of scene analysis beyond the level of object
recognition
I explicit description of spatial configurations by means of spatial
relations
I interpretation of object movement
I automatic recognition of goals and plans of observed agents
specific goals:
I answering queries about traffic situations
I generating reports of football games
I communicating with a mobile robot

Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136

Project Vitra: Visual Translator

long term project: 80’s and 90’s

design and construction of integrated knowledge-based systems
capable of translating visual information into natural language
descriptions
general goals:
I extend the scope of scene analysis beyond the level of object
recognition
I explicit description of spatial configurations by means of spatial
relations
I interpretation of object movement
I automatic recognition of goals and plans of observed agents
specific goals:
I answering queries about traffic situations
I generating reports of football games
I communicating with a mobile robot

Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136

Project Vitra: Visual Translator

long term project: 80’s and 90’s

design and construction of integrated knowledge-based systems
capable of translating visual information into natural language
descriptions
general goals:
I extend the scope of scene analysis beyond the level of object
recognition
I explicit description of spatial configurations by means of spatial
relations
I interpretation of object movement
I automatic recognition of goals and plans of observed agents
specific goals:
I answering queries about traffic situations
I generating reports of football games
I communicating with a mobile robot

Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136

Project Vitra: Visual Translator

long term project: 80’s and 90’s

design and construction of integrated knowledge-based systems
capable of translating visual information into natural language
descriptions
general goals:
I extend the scope of scene analysis beyond the level of object
recognition
I explicit description of spatial configurations by means of spatial
relations
I interpretation of object movement
I automatic recognition of goals and plans of observed agents
specific goals:
I answering queries about traffic situations
I generating reports of football games
I communicating with a mobile robot

Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136

Project Vitra: Visual Translator

long term project: 80’s and 90’s

design and construction of integrated knowledge-based systems
capable of translating visual information into natural language
descriptions
general goals:
I extend the scope of scene analysis beyond the level of object
recognition
I explicit description of spatial configurations by means of spatial
relations
I interpretation of object movement
I automatic recognition of goals and plans of observed agents
specific goals:
I answering queries about traffic situations
I generating reports of football games
I communicating with a mobile robot

Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136

Project Vitra: Visual Translator

long term project: 80’s and 90’s

design and construction of integrated knowledge-based systems
capable of translating visual information into natural language
descriptions
general goals:
I extend the scope of scene analysis beyond the level of object
recognition
I explicit description of spatial configurations by means of spatial
relations
I interpretation of object movement
I automatic recognition of goals and plans of observed agents
specific goals:
I answering queries about traffic situations
I generating reports of football games
I communicating with a mobile robot

Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136

Project Vitra: Visual Translator

long term project: 80’s and 90’s

design and construction of integrated knowledge-based systems
capable of translating visual information into natural language
descriptions
general goals:
I extend the scope of scene analysis beyond the level of object
recognition
I explicit description of spatial configurations by means of spatial
relations
I interpretation of object movement
I automatic recognition of goals and plans of observed agents
specific goals:
I answering queries about traffic situations
I generating reports of football games
I communicating with a mobile robot

Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136

Project Vitra: Visual Translator

long term project: 80’s and 90’s

design and construction of integrated knowledge-based systems
capable of translating visual information into natural language
descriptions
general goals:
I extend the scope of scene analysis beyond the level of object
recognition
I explicit description of spatial configurations by means of spatial
relations
I interpretation of object movement
I automatic recognition of goals and plans of observed agents
specific goals:
I answering queries about traffic situations
I generating reports of football games
I communicating with a mobile robot

Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136

Project Vitra: Visual Translator

long term project: 80’s and 90’s

design and construction of integrated knowledge-based systems
capable of translating visual information into natural language
descriptions
general goals:
I extend the scope of scene analysis beyond the level of object
recognition
I explicit description of spatial configurations by means of spatial
relations
I interpretation of object movement
I automatic recognition of goals and plans of observed agents
specific goals:
I answering queries about traffic situations
I generating reports of football games
I communicating with a mobile robot

Paulo Santos ( FEI - São Paulo ) September 10, 2010 87 / 136

Project Vitra: Visual Translator

Overview of the system:

from raw data, image analysis generates a geometrical
representation of the scene, with object’s locations through time
the geometric description is interpreted by the cognitive level (or
high-level scene analysis)
this high-level analysis extracts:
I spatial relations
I interesting motion events
I presumed intentions, plans and plan interactions between
agents
the final, linguistic, level transform the conceptual descriptions into
utterances

Paulo Santos ( FEI - São Paulo ) September 10, 2010 88 / 136

Project Vitra: Visual Translator

Overview of the system:

from raw data, image analysis generates a geometrical
representation of the scene, with object’s locations through time
the geometric description is interpreted by the cognitive level (or
high-level scene analysis)
this high-level analysis extracts:
I spatial relations
I interesting motion events
I presumed intentions, plans and plan interactions between
agents
the final, linguistic, level transform the conceptual descriptions into
utterances

Paulo Santos ( FEI - São Paulo ) September 10, 2010 88 / 136

Project Vitra: Visual Translator

Overview of the system:

from raw data, image analysis generates a geometrical
representation of the scene, with object’s locations through time
the geometric description is interpreted by the cognitive level (or
high-level scene analysis)
this high-level analysis extracts:
I spatial relations
I interesting motion events
I presumed intentions, plans and plan interactions between
agents
the final, linguistic, level transform the conceptual descriptions into
utterances

Paulo Santos ( FEI - São Paulo ) September 10, 2010 88 / 136

Project Vitra: Visual Translator

Overview of the system:

from raw data, image analysis generates a geometrical
representation of the scene, with object’s locations through time
the geometric description is interpreted by the cognitive level (or
high-level scene analysis)
this high-level analysis extracts:
I spatial relations
I interesting motion events
I presumed intentions, plans and plan interactions between
agents
the final, linguistic, level transform the conceptual descriptions into
utterances

Paulo Santos ( FEI - São Paulo ) September 10, 2010 88 / 136

Project Vitra: Visual Translator

Overview of the system:

from raw data, image analysis generates a geometrical
representation of the scene, with object’s locations through time
the geometric description is interpreted by the cognitive level (or
high-level scene analysis)
this high-level analysis extracts:
I spatial relations
I interesting motion events
I presumed intentions, plans and plan interactions between
agents
the final, linguistic, level transform the conceptual descriptions into
utterances

Paulo Santos ( FEI - São Paulo ) September 10, 2010 88 / 136

Alven (1984)

natural language description of ventricular wall motion

data extracted from X-ray image sequences
knowledge organisation: ontology
Analyse pre-operative and post-operative marker films to evaluate
the efficacy of surgery
analysis using both quantitative and qualitative representations
major issues:
I understanding visual motion from image tokens over time
I reasoning about spatio-temporal relationships

Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136

Alven (1984)

natural language description of ventricular wall motion

data extracted from X-ray image sequences
knowledge organisation: ontology
Analyse pre-operative and post-operative marker films to evaluate
the efficacy of surgery
analysis using both quantitative and qualitative representations
major issues:
I understanding visual motion from image tokens over time
I reasoning about spatio-temporal relationships

Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136

Alven (1984)

natural language description of ventricular wall motion

data extracted from X-ray image sequences
knowledge organisation: ontology
Analyse pre-operative and post-operative marker films to evaluate
the efficacy of surgery
analysis using both quantitative and qualitative representations
major issues:
I understanding visual motion from image tokens over time
I reasoning about spatio-temporal relationships

Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136

Alven (1984)

natural language description of ventricular wall motion

data extracted from X-ray image sequences
knowledge organisation: ontology
Analyse pre-operative and post-operative marker films to evaluate
the efficacy of surgery
analysis using both quantitative and qualitative representations
major issues:
I understanding visual motion from image tokens over time
I reasoning about spatio-temporal relationships

Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136

Alven (1984)

natural language description of ventricular wall motion

data extracted from X-ray image sequences
knowledge organisation: ontology
Analyse pre-operative and post-operative marker films to evaluate
the efficacy of surgery
analysis using both quantitative and qualitative representations
major issues:
I understanding visual motion from image tokens over time
I reasoning about spatio-temporal relationships

Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136

Alven (1984)

natural language description of ventricular wall motion

data extracted from X-ray image sequences
knowledge organisation: ontology
Analyse pre-operative and post-operative marker films to evaluate
the efficacy of surgery
analysis using both quantitative and qualitative representations
major issues:
I understanding visual motion from image tokens over time
I reasoning about spatio-temporal relationships

Paulo Santos ( FEI - São Paulo ) September 10, 2010 89 / 136

Alven

Paulo Santos ( FEI - São Paulo ) September 10, 2010 90 / 136

Alven’s knowledge base

Set of classes organised using relations about

IS_A: generalisation/specialisations (taxonomy)
PART _OF : part/whole (partonomy)
temporal precedence
Should satisfy:
1 motion classes should be sufficient to express the domain
2 image tokens are connected to general knowledge in the leaves of
a PART _OF hierarchy of motion concepts

Paulo Santos ( FEI - São Paulo ) September 10, 2010 91 / 136

Alven’s knowledge base

Set of classes organised using relations about

IS_A: generalisation/specialisations (taxonomy)
PART _OF : part/whole (partonomy)
temporal precedence
Should satisfy:
1 motion classes should be sufficient to express the domain
2 image tokens are connected to general knowledge in the leaves of
a PART _OF hierarchy of motion concepts

Paulo Santos ( FEI - São Paulo ) September 10, 2010 91 / 136

Alven’s control structure

extract tokens from the input signal, instantiating then in the

leaves of the PART _OF hierarchy
follows this hierarchy to activated hypotheses that are aggregates
of the input tokens
this set of hypotheses is specialised by going down one level in
the IS_A hierarchy
each hypothesis is matched with other data instances: matching
leads to further specialisations, failure leads to the selection of
other hypotheses
the best hypotheses generate a set of predictions
predictions are mapped back to the image level and are used as a
guidance to the token extraction procedure

Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136

Alven’s control structure

extract tokens from the input signal, instantiating then in the

leaves of the PART _OF hierarchy
follows this hierarchy to activated hypotheses that are aggregates
of the input tokens
this set of hypotheses is specialised by going down one level in
the IS_A hierarchy
each hypothesis is matched with other data instances: matching
leads to further specialisations, failure leads to the selection of
other hypotheses
the best hypotheses generate a set of predictions
predictions are mapped back to the image level and are used as a
guidance to the token extraction procedure

Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136

Alven’s control structure

extract tokens from the input signal, instantiating then in the

leaves of the PART _OF hierarchy
follows this hierarchy to activated hypotheses that are aggregates
of the input tokens
this set of hypotheses is specialised by going down one level in
the IS_A hierarchy
each hypothesis is matched with other data instances: matching
leads to further specialisations, failure leads to the selection of
other hypotheses
the best hypotheses generate a set of predictions
predictions are mapped back to the image level and are used as a
guidance to the token extraction procedure

Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136

Alven’s control structure

extract tokens from the input signal, instantiating then in the

leaves of the PART _OF hierarchy
follows this hierarchy to activated hypotheses that are aggregates
of the input tokens
this set of hypotheses is specialised by going down one level in
the IS_A hierarchy
each hypothesis is matched with other data instances: matching
leads to further specialisations, failure leads to the selection of
other hypotheses
the best hypotheses generate a set of predictions
predictions are mapped back to the image level and are used as a
guidance to the token extraction procedure

Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136

Alven’s control structure

extract tokens from the input signal, instantiating then in the

leaves of the PART _OF hierarchy
follows this hierarchy to activated hypotheses that are aggregates
of the input tokens
this set of hypotheses is specialised by going down one level in
the IS_A hierarchy
each hypothesis is matched with other data instances: matching
leads to further specialisations, failure leads to the selection of
other hypotheses
the best hypotheses generate a set of predictions
predictions are mapped back to the image level and are used as a
guidance to the token extraction procedure

Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136

Alven’s control structure

extract tokens from the input signal, instantiating then in the

leaves of the PART _OF hierarchy
follows this hierarchy to activated hypotheses that are aggregates
of the input tokens
this set of hypotheses is specialised by going down one level in
the IS_A hierarchy
each hypothesis is matched with other data instances: matching
leads to further specialisations, failure leads to the selection of
other hypotheses
the best hypotheses generate a set of predictions
predictions are mapped back to the image level and are used as a
guidance to the token extraction procedure

Paulo Santos ( FEI - São Paulo ) September 10, 2010 92 / 136

Brand’s visual understanding through causal analysis

describe the causal structure from images

knowledge about physical causality is used in the interpretation
scenes
hypothesis: a small core set of qualitative rules accounts for most
of what humans ordinarily see
it is possible to make useful inferences with qualitative knowledge
about connectivity and free space
qualitative rules of connectivity, friction, attachment and
penetration
interpretation of simple mechanical machines
purpose
I Analyse
I diagnosis
I prediction
I inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis

describe the causal structure from images

knowledge about physical causality is used in the interpretation
scenes
hypothesis: a small core set of qualitative rules accounts for most
of what humans ordinarily see
it is possible to make useful inferences with qualitative knowledge
about connectivity and free space
qualitative rules of connectivity, friction, attachment and
penetration
interpretation of simple mechanical machines
purpose
I Analyse
I diagnosis
I prediction
I inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis

describe the causal structure from images

knowledge about physical causality is used in the interpretation
scenes
hypothesis: a small core set of qualitative rules accounts for most
of what humans ordinarily see
it is possible to make useful inferences with qualitative knowledge
about connectivity and free space
qualitative rules of connectivity, friction, attachment and
penetration
interpretation of simple mechanical machines
purpose
I Analyse
I diagnosis
I prediction
I inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis

describe the causal structure from images

knowledge about physical causality is used in the interpretation
scenes
hypothesis: a small core set of qualitative rules accounts for most
of what humans ordinarily see
it is possible to make useful inferences with qualitative knowledge
about connectivity and free space
qualitative rules of connectivity, friction, attachment and
penetration
interpretation of simple mechanical machines
purpose
I Analyse
I diagnosis
I prediction
I inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis

describe the causal structure from images

knowledge about physical causality is used in the interpretation
scenes
hypothesis: a small core set of qualitative rules accounts for most
of what humans ordinarily see
it is possible to make useful inferences with qualitative knowledge
about connectivity and free space
qualitative rules of connectivity, friction, attachment and
penetration
interpretation of simple mechanical machines
purpose
I Analyse
I diagnosis
I prediction
I inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis

describe the causal structure from images

knowledge about physical causality is used in the interpretation
scenes
hypothesis: a small core set of qualitative rules accounts for most
of what humans ordinarily see
it is possible to make useful inferences with qualitative knowledge
about connectivity and free space
qualitative rules of connectivity, friction, attachment and
penetration
interpretation of simple mechanical machines
purpose
I Analyse
I diagnosis
I prediction
I inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s visual understanding through causal analysis

describe the causal structure from images

knowledge about physical causality is used in the interpretation
scenes
hypothesis: a small core set of qualitative rules accounts for most
of what humans ordinarily see
it is possible to make useful inferences with qualitative knowledge
about connectivity and free space
qualitative rules of connectivity, friction, attachment and
penetration
interpretation of simple mechanical machines
purpose
I Analyse
I diagnosis
I prediction
I inspection
Paulo Santos ( FEI - São Paulo ) September 10, 2010 93 / 136
Brand’s SPROCKET (1997)

Paulo Santos ( FEI - São Paulo ) September 10, 2010 94 / 136

Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 95 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 96 / 136

Siskind’s Event classification from camera input using
force dynamics

similar to Brand’s but on image sequences

force dynamics: SUPPORTED, ATTACHED, PICKUP, PUTDOWN,
CONTACT
more recently: supervised learning of visual event definitions from
video

Paulo Santos ( FEI - São Paulo ) September 10, 2010 97 / 136

Siskind’s Event classification from camera input using
force dynamics

similar to Brand’s but on image sequences

force dynamics: SUPPORTED, ATTACHED, PICKUP, PUTDOWN,
CONTACT
more recently: supervised learning of visual event definitions from
video

Paulo Santos ( FEI - São Paulo ) September 10, 2010 97 / 136

Siskind’s Event classification from camera input using
force dynamics

similar to Brand’s but on image sequences

force dynamics: SUPPORTED, ATTACHED, PICKUP, PUTDOWN,
CONTACT
more recently: supervised learning of visual event definitions from
video

Paulo Santos ( FEI - São Paulo ) September 10, 2010 97 / 136

Siskind’s Leonard System

Paulo Santos ( FEI - São Paulo ) September 10, 2010 98 / 136

Leeds Traffic Interaction

apply QSR to dynamic scene analysis

learning events from the observation of traffic scenes
recognising these events and generate predictions
a scene is interpreted from the comparison of its temporal
development with a (previously learned) transition diagram
representing change in the domain.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 99 / 136

CND of relative positions

Paulo Santos ( FEI - São Paulo ) September 10, 2010 100 / 136
Leeds Traffic Interaction

Paulo Santos ( FEI - São Paulo ) September 10, 2010 101 / 136
Protocol learning

A system for learning protocol behaviour from computer vision

data using ILP
Unsupervised continuous learning of perceptual categories and
unsupervised symbolic learning of protocols (as sets of Horn
clauses);

Paulo Santos ( FEI - São Paulo ) September 10, 2010 102 / 136
Protocol Learning

Off-the-shelf vision system

I Attention: identifies key frames with no motion preceded by a
number of frames with notion;
I Statistical classifier: assigns different classes to clusters of features
Off-the-shelf ILP system: Progol
I Generalises a set of positive only examples according to user
defined mode declarations
I Mode declarations restrict the possible form of the proposed
generalisations

Paulo Santos ( FEI - São Paulo ) September 10, 2010 103 / 136
Protocol Learning

Paulo Santos ( FEI - São Paulo ) September 10, 2010 104 / 136
Protocol Learning

Inducing axioms of ordering and equivalence, without knowing

about numbers
Building equivalence classes to cope with over clustering
sound and completeness of the agent actually playing the game
[AIJ 05]

Paulo Santos ( FEI - São Paulo ) September 10, 2010 105 / 136
Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 106 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 107 / 136
Example of QSR from Vision: Depth Profile Calculus

How much knowledge about a robot’s environment can be from

vision alone?
How can we construct the knowledge about objects in the world
from sensor data of the robot?
Construction of a qualitative spatial reasoning (QSR) system
based on sensor data
Use abduction for sensor data assimilation and deduction for
predictions

Paulo Santos ( FEI - São Paulo ) September 10, 2010 108 / 136
Example of QSR from Vision: Depth Profile Calculus

How much knowledge about a robot’s environment can be from

vision alone?
How can we construct the knowledge about objects in the world
from sensor data of the robot?
Construction of a qualitative spatial reasoning (QSR) system
based on sensor data
Use abduction for sensor data assimilation and deduction for
predictions

Paulo Santos ( FEI - São Paulo ) September 10, 2010 108 / 136
Example of QSR from Vision: Depth Profile Calculus

How much knowledge about a robot’s environment can be from

vision alone?
How can we construct the knowledge about objects in the world
from sensor data of the robot?
Construction of a qualitative spatial reasoning (QSR) system
based on sensor data
Use abduction for sensor data assimilation and deduction for
predictions

Paulo Santos ( FEI - São Paulo ) September 10, 2010 108 / 136
Example of QSR from Vision: Depth Profile Calculus

How much knowledge about a robot’s environment can be from

vision alone?
How can we construct the knowledge about objects in the world
from sensor data of the robot?
Construction of a qualitative spatial reasoning (QSR) system
based on sensor data
Use abduction for sensor data assimilation and deduction for
predictions

Paulo Santos ( FEI - São Paulo ) September 10, 2010 108 / 136
Simplified environment

Paulo Santos ( FEI - São Paulo ) September 10, 2010 109 / 136
Spatial Reasoning about Robot Sensor Data

Attributes:
I Distance, disparity, size;
I Changes in the sensor data;
Representation:
I Depth profiles and time points;
I Displacement between regions;
I Mapping function between images and objects

Paulo Santos ( FEI - São Paulo ) September 10, 2010 110 / 136
Assimilating changes

Axioms of the system:

< Dynamic spatial rel > ← < desc. sensor transition >
< Dynamicspatialrel. > ← < obj − obs relation >

Paulo Santos ( FEI - São Paulo ) September 10, 2010 111 / 136
Depth Profiles

Paulo Santos ( FEI - São Paulo ) September 10, 2010 112 / 136
The model

Extract one horizontal depth profile of each scene from the visual
data;
Objects in the scenes are represented as peaks;
Axiomatise relations on the depth and size of these profiles as
well as displacements;

Paulo Santos ( FEI - São Paulo ) September 10, 2010 113 / 136
Depth Profile Calculus

A theory about displacement, size and depth;

27 base relations;
Large and complex conceptual neighbourhood diagrams and
composition tables;

Paulo Santos ( FEI - São Paulo ) September 10, 2010 114 / 136
Depth Profile Calculus

Paulo Santos ( FEI - São Paulo ) September 10, 2010 115 / 136
DPC example

Paulo Santos ( FEI - São Paulo ) September 10, 2010 116 / 136
Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 117 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 118 / 136
Reasoning about Shadows in Robotics

making explicit the knowledge contained in cast shadows

use it to reason about the robot environment
Computer vision however has largely been filtering out cast
shadows as noise

Paulo Santos ( FEI - São Paulo ) September 10, 2010 119 / 136
Reasoning about Shadows in Robotics

making explicit the knowledge contained in cast shadows

use it to reason about the robot environment
Computer vision however has largely been filtering out cast
shadows as noise

Paulo Santos ( FEI - São Paulo ) September 10, 2010 119 / 136
Reasoning about Shadows in Robotics

making explicit the knowledge contained in cast shadows

use it to reason about the robot environment
Computer vision however has largely been filtering out cast
shadows as noise

Paulo Santos ( FEI - São Paulo ) September 10, 2010 119 / 136
Illusory Motion from Shadows

Paulo Santos ( FEI - São Paulo ) September 10, 2010 120 / 136
Perception of shadows

“no luminous body ever sees the shadows that it generates” [da
Vinci, Notebooks of Leonardo Da Vinci. Project Gutenberg (1888)]
from the light source viewpoint shadows are occluded by their
casters
We model observer-caster-shadow within qualitative spatial
reasoning: ROC + an axiom about shadow :

Shadow(s, o, Scr , L) ↔ PO(r (s), r (Scr ))∧

TotallyOccludes(o, s, L)∧
¬∃o TotallyOccludes(o0 , o, L).
0

“a shadow is totally occluded by its caster from the lightsource

viewpoint”

Paulo Santos ( FEI - São Paulo ) September 10, 2010 121 / 136
Perception of shadows

“no luminous body ever sees the shadows that it generates” [da
Vinci, Notebooks of Leonardo Da Vinci. Project Gutenberg (1888)]
from the light source viewpoint shadows are occluded by their
casters
We model observer-caster-shadow within qualitative spatial
reasoning: ROC + an axiom about shadow :

Shadow(s, o, Scr , L) ↔ PO(r (s), r (Scr ))∧

TotallyOccludes(o, s, L)∧
¬∃o TotallyOccludes(o0 , o, L).
0

“a shadow is totally occluded by its caster from the lightsource

viewpoint”

Paulo Santos ( FEI - São Paulo ) September 10, 2010 121 / 136
Perception of shadows

“no luminous body ever sees the shadows that it generates” [da
Vinci, Notebooks of Leonardo Da Vinci. Project Gutenberg (1888)]
from the light source viewpoint shadows are occluded by their
casters
We model observer-caster-shadow within qualitative spatial
reasoning: ROC + an axiom about shadow :

Shadow(s, o, Scr , L) ↔ PO(r (s), r (Scr ))∧

TotallyOccludes(o, s, L)∧
¬∃o TotallyOccludes(o0 , o, L).
0

“a shadow is totally occluded by its caster from the lightsource

viewpoint”

Paulo Santos ( FEI - São Paulo ) September 10, 2010 121 / 136
Perception of shadows

“no luminous body ever sees the shadows that it generates” [da
Vinci, Notebooks of Leonardo Da Vinci. Project Gutenberg (1888)]
from the light source viewpoint shadows are occluded by their
casters
We model observer-caster-shadow within qualitative spatial
reasoning: ROC + an axiom about shadow :

Shadow(s, o, Scr , L) ↔ PO(r (s), r (Scr ))∧

TotallyOccludes(o, s, L)∧
¬∃o TotallyOccludes(o0 , o, L).
0

“a shadow is totally occluded by its caster from the lightsource

viewpoint”

Paulo Santos ( FEI - São Paulo ) September 10, 2010 121 / 136
Qualitative regions for self-localisation

L
*
4 4
3 3

2 2
o

S
1 1

Paulo Santos ( FEI - São Paulo ) September 10, 2010 122 / 136
In practice

Qualitative robot self-localisation

relative depth from the observation of shadows
threshold finding from qualitative regions

Paulo Santos ( FEI - São Paulo ) September 10, 2010 123 / 136
In practice
L
*
4 4
3 3

2 2
o

S
1 1

located(Region 1, ν, o, s) ← Is_a_Shadow(s, o) ∧
NonOccludesDC(o, s, v ) ∧ v 6= o;
located(Region 2, ν, o, s) ← Is_a_Shadow(s, o) ∧
NonOccludesEC(o, s, v ) ∧ v 6= o;
located(Region 3, ν, o, s) ← Is_a_Shadow(s, o) ∧
PartiallyOccludesPO(o, s, v ) ∧ v 6= o;
located(Region 4, ν, o, s) ← Is_a_Shadow(s, o) ∧
TotallyOccludesTPPI(o, s, v ) ∧ v 6= o;
Paulo Santos ( FEI - São Paulo ) September 10, 2010 124 / 136
Outline I
1 Preface
Where/what is FEI?
2 PART I: The Big Picture
Introduction and motivation
Automated Reasoning 101
Qualitative Spatial Reasoning at a glance
3 Part II: Qualitative Spatial Reasoning
Region Connection Calculus
Lines of Sight Calculus
Region Occlusion Calculus
Cardinal Direction Calculus
Double Cross Calculus
Other calculi
Tractability and computability
4 Coffee Break
20 min ?
Paulo Santos ( FEI - São Paulo ) September 10, 2010 125 / 136
Outline II
5 Part III: Cognitive Vision
Foundations
Cognitive Vision at a glance
Early systems
Modern systems (from 2000)

6 Part IV: QSR in CogVis

Depth Profile Calculus
Reasoning about Shadows in Robotics
The future: Probabilistic Logic Encoding of Spatial Domains

7 Conclusion

Paulo Santos ( FEI - São Paulo ) September 10, 2010 126 / 136
Probabilistic Logic Encoding of Spatial Domains

incorporate incomplete sensor data and domain knowledge in a

probabilistic logic setting
explore inferences about space

Paulo Santos ( FEI - São Paulo ) September 10, 2010 127 / 136
Traffic Scenario

Paulo Santos ( FEI - São Paulo ) September 10, 2010 128 / 136
Sensor model

P(OnTwoWayRoad|SensedTwoWayRoad) = 0.99
P(OnOneWayRoad|SensedOneWayRoad) = 0.99
P(DashedDivider |SensedDashedDivider ) = 0.93
P(SolidDivider |SensedDashedDivider ) = 0.07
P(DashedDivider |SensedSolidDivider ) = 0.20
P(SolidDivider |SensedSolidDivider ) = 0.80

Paulo Santos ( FEI - São Paulo ) September 10, 2010 129 / 136
Taxonomy of concepts

Lane = GoingUp ∪ GoingDown ,

Divider = DashedDivider SolidDivider ,
Vehicle = OnOneWayRoad ∪ OnTwoWayRoad ,
disjoint(Vehicle, Divider, Lane) ,
disjoint(GoingUp, GoingDown) ,
disjoint(DashedDivider, SolidDivider) ,
disjoint(OnOneWayRoad, OnTwoWayRoad) ,

Paulo Santos ( FEI - São Paulo ) September 10, 2010 130 / 136
Hard Constraints

GoingUp ⊂ ∀ri.(GoingUp ∪ ¬Lane),

GoingDown ⊂ ∀le.(GoingDown ∪ ¬Lane),
DashedDivider ⊂ ∃ri.Lane u ∃le.Lane,
SolidDivider ⊂
(¬∃ri.Lane ∪ ¬∃le.Lane) ∪ (∃cdc.GoingUp u ∃cdc.GoingDown)
OnTwoWayRoad ⊂ ∃cdc.OneWayNorth u ∃cdc.OneWaySouth

Paulo Santos ( FEI - São Paulo ) September 10, 2010 131 / 136
Generate a Bayesian Net out of it

Paulo Santos ( FEI - São Paulo ) September 10, 2010 132 / 136
Answer queries such as

On which lane are we? Argmaxli P(v : Onli) : li is the lane with
maximum probability of being the vehicle.
Which driving directions does each lane permit?
∀i : P(li : GoingDown): for each lane li, the probability of being a
GoingUp lane

Paulo Santos ( FEI - São Paulo ) September 10, 2010 133 / 136
Conclusion

We’ve presented a brief overview of QSR and CogVis

Discussed some early work on CogVis systems
presented some new work based mainly at UoL and FEI
Much work remains to be done!

Paulo Santos ( FEI - São Paulo ) September 10, 2010 134 / 136
Conclusion

We’ve presented a brief overview of QSR and CogVis

Discussed some early work on CogVis systems
presented some new work based mainly at UoL and FEI
Much work remains to be done!

Paulo Santos ( FEI - São Paulo ) September 10, 2010 134 / 136
Conclusion

We’ve presented a brief overview of QSR and CogVis

Discussed some early work on CogVis systems
presented some new work based mainly at UoL and FEI
Much work remains to be done!

Paulo Santos ( FEI - São Paulo ) September 10, 2010 134 / 136
Conclusion

We’ve presented a brief overview of QSR and CogVis

Discussed some early work on CogVis systems
presented some new work based mainly at UoL and FEI
Much work remains to be done!

Paulo Santos ( FEI - São Paulo ) September 10, 2010 134 / 136
Guide lines

A. Cohn, Slides of the tutorial: Knowledge Representation and

Reasoning for Computer Vision: Qualitative Spatio-temporal
reasoning
A. Cohn and J. Renz, Qualitative Spatial Representation and
Reasoning, Handbook of Knowledge Representation, 2008
B. Bennett, Logical representations for automated reasoning
about spatial relationships, PhD thesis, School of Computing,
University of Leeds, UK.
P. Santos, Raciocínio e Percepção espacial: uma abordagem
lógica. Working notes of CBA 2010 tutorial. Tutoriais do XVIII
Congresso Brasileiro de Automática, 2010.

Paulo Santos ( FEI - São Paulo ) September 10, 2010 135 / 136
Thanks!

This author was partially supported by:

FAPESP LogProb project:2008/03995-5
CNPq bolsa PQ

Paulo Santos ( FEI - São Paulo ) September 10, 2010 136 / 136