Modeling Human Behaviors in Psychology Using Engineering

Methods 1st Edition

Visit the link below to download the full version of this book:

https://medipdf.com/product/modeling-human-behaviors-in-psychology-using-enginee
ring-methods-1st-edition/

Click Download Now

 Contents

I Modeling Human Behaviors: An Engineering Approach 1

1 Behavioral Signal Processing (BSP): Behavioral Informatics 3
1.1 BSP: Introduction . . . . . . . . . . . . . . . . . . . . . . . 3
1.1.1 BSP: Technical Challenges and Complexities . . . . 4
1.2 BSP: Computational Methods for Dyadic Interaction Dynamics 6
1.2.1 BSP: Further Complexities in Modeling Interaction
Dynamics . . . . . . . . . . . . . . . . . . . . . . . 8

2 Applications in Modeling Human Behaviors Computationally 11

2.1 BSP Application Domains . . . . . . . . . . . . . . . . . . 11
2.2 Case Study I: Emotion Recognition from Speech . . . . . . 12
2.3 Case Study II: Quantifying Implicit Vocal Entrainment . . . 14
2.4 Case Study III: Data-driven Perceptual Experiment . . . . . 15

II Affective Computing from Speech 17

3 Individual Utterance Emotion Recognition 19
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 19
3.2 Emotion Databases and Acoustic Feature Extraction . . . . . 19
3.2.1 The AIBO Database . . . . . . . . . . . . . . . . . 19
3.2.2 The USC IEMOCAP Database . . . . . . . . . . . . 20
3.2.3 Acoustic Feature Extraction . . . . . . . . . . . . . 21
3.2.4 Feature Selection and Normalization . . . . . . . . . 22
3.3 Emotion Classification Framework . . . . . . . . . . . . . . 22
3.3.1 Building the Hierarchical Decision Tree . . . . . . . 22
3.3.2 Building the Hierarchical Decision Tree for the
AIBO Database and the USC IEMOCAP Database . 25
3.3.3 Classifier for Binary Classification Tasks . . . . . . 26
3.4 Emotion Recognition Experiment Setup and Results . . . . . 28
3.4.1 The AIBO Database . . . . . . . . . . . . . . . . . 28

v
vi Contents

3.4.2 The USC IEMOCAP Database . . . . . . . . . . . . 32

3.5 Conclusions and Future Work . . . . . . . . . . . . . . . . . 34

4 Dialog-based Emotion Recognition 37

4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 Emotion Database and Annotation . . . . . . . . . . . . . . 38
4.2.1 The USC IEMOCAP Database . . . . . . . . . . . . 38
4.2.2 Emotion Annotation . . . . . . . . . . . . . . . . . 39
4.3 Dynamic Bayesian Network Model . . . . . . . . . . . . . . 40
4.4 Experimental Results and Discussion . . . . . . . . . . . . . 41
4.4.1 Acoustic Feature Extraction . . . . . . . . . . . . . 41
4.4.2 Experiment Setup . . . . . . . . . . . . . . . . . . . 42
4.4.3 Experiment Results and Discussion . . . . . . . . . 44
4.5 Conclusions and Future Work . . . . . . . . . . . . . . . . . 45

III Quantifying Human Behavior in Psychology 47

5 Implicit Vocal Synchrony Quantification 49
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 49
5.2 BSP Database: The Couple Therapy Corpus . . . . . . . . . 52
5.2.1 Pre-processing and Audio Feature Extraction . . . . 53
5.2.2 Behavioral Codes of Interest . . . . . . . . . . . . . 55
5.3 Signal-derived Vocal Entrainment Quantification . . . . . . 56
5.3.1 PCA-based Similarity Measures . . . . . . . . . . . 57
5.3.2 Representative Vocal Features . . . . . . . . . . . . 61
5.3.3 Vocal Entrainment Measures in Dialogs . . . . . . . 62
5.4 Analysis of Vocal Entrainment Measures . . . . . . . . . . . 63
5.4.1 Natural Cohesiveness of Dialogs . . . . . . . . . . . 64
5.4.2 Entrainment in Affective Interactions . . . . . . . . 67
5.5 Affect Classification using Entrainment Measures . . . . . . 69
5.5.1 Classification Framework . . . . . . . . . . . . . . 69
5.5.2 Classification Setup . . . . . . . . . . . . . . . . . . 73
5.5.3 Classification Results and Discussions . . . . . . . . 74
5.6 Conclusions and Future Work . . . . . . . . . . . . . . . . . 76

6 Analysis of Vocal Synchrony in Couples Therapy 79

6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 79
6.2 BSP Database: The Couple Therapy Corpus . . . . . . . . . 80
 Contents vii

6.3 PCA-based Vocal Entrainment Measures . . . . . . . . . . . 81

6.3.1 Symmetric Entrainment Measures . . . . . . . . . . 81
6.3.2 Directional Entrainment Measures . . . . . . . . . . 82
6.3.3 Canonical Correlation Analysis . . . . . . . . . . . 83
6.4 Analysis of Results and Discussions . . . . . . . . . . . . . 83
6.4.1 Correlation Analysis: The Four Behavioral Dimensions 84
6.4.2 Canonical Correlation Analysis: Withdrawal . . . . 85
6.5 Lessons Learnt from Correlation Analysis . . . . . . . . . . 87
6.6 Vocal Entrainment and Demand-and-Withdraw in Couple
Conflict . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
6.6.1 Demand and Withdraw . . . . . . . . . . . . . . . . 88
6.6.2 Behavioral Influence and Polarization of Demand
and Withdraw . . . . . . . . . . . . . . . . . . . . . 89
6.6.3 Data Analysis . . . . . . . . . . . . . . . . . . . . . 90
6.6.4 Results and Discussions . . . . . . . . . . . . . . . 91

IV Data-driven Perceptual Experiment 93

7 Multiple Instance Learning Framework for Perceptual
Experiment 95
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . 95
7.2 BSP Database: The Couple Therapy Corpus . . . . . . . . . 96
7.3 Computational Framework . . . . . . . . . . . . . . . . . . 97
7.3.1 Multiple Instance Learning . . . . . . . . . . . . . . 97
7.3.2 Sequential Probability Ratio Test . . . . . . . . . . . 98
7.4 Analysis Setup . . . . . . . . . . . . . . . . . . . . . . . . 99
7.4.1 Lexical Feature Extraction . . . . . . . . . . . . . . 99
7.4.2 Classification Setup . . . . . . . . . . . . . . . . . . 99
7.5 Detection Results and Discussions . . . . . . . . . . . . . . 100
7.6 Isolated-Saliency vs. Causal-Integration . . . . . . . . . . . 103
7.7 Conclusions and Future Work . . . . . . . . . . . . . . . . . 103

V Outlook of BSP 105

8 Continuously Emerging Importance of Modeling Human
Behavior 107
viii Contents

Bibliography 109

Author Biography 121

 Part I

Modeling Human Behaviors:

An Engineering Approach
 1
Behavioral Signal Processing (BSP): Behavioral
Informatics

1.1 BSP: Introduction

Human Behavioral Signal Processing (BSP) [77] is an emerging interdisci­
plinary research domain, spanning across engineering and behavior science.
It is operationally defined as computational methods that model human
behavior signals. The concept behind modeling human behavior computa­
tionally is from a ‘signals and systems’ perspective, i.e., humans can be
considered as a complex system, with different internal hidden states and
processes governing the underlying mechanisms of expressive behavioral
generations and perceptions. These expressive behaviors are called behav­
ioral signals in signal processing terms, which encodes information with
regard to humans’ characterizing systems’ hidden and time-varying inter­
nal states. Behavioral signals are manifested in overt and covert cues, are
processed and used by humans explicitly or implicitly, and often time, are
fundamental in facilitating human analysis and decision making. The goal of
BSP is to computationally model these behavioral signals, and the outcome
of BSP is called, “behavioral informatics” – i.e., computational methods
aim at enhancing the capabilities of domain experts in facilitating better
decision-making in terms of both scientific discovery in fields of human be­
havior sciences and better design of human-centered system (e.g., most often
referred to as human–machine interaction interfaces).
These computational methods, i.e. behavioral informatics, potentially
could profoundly impact on the fronts of both behavior science and advanced
engineering systems. Because of the interdisciplinary nature in its algorith­
mic development cycle – encompassing research domains across behavioral
sciences, machine learning, and signal processing – BSP provides an oppor­
tunity to integrate human domain-specific knowledge into constructing novel
computational frameworks addressing problems of societal significances and

DOI: 10.1201/9781003338857-2
3
4 1 Behavioral Signal Processing (BSP): Behavioral Informatics

further transferring that knowledge into better designs of future technologies.

This integration of human knowledge into the end-user system design can
push the boundary of possibilities in the development of human-centric sys­
tems, e.g., striving to achieve a more realistic and natural human–machine
interactions. Furthermore, BSP, with its computational ability, can develop
machine learning algorithms to sense and predict human behaviors through
learning from large amount of data – enabling machines to carry out tasks that
humans are naturally capable of in every day life. BSP provides a grounding
domain for engineers to advance the state of the art machine learning and sig­
nal processing techniques that can better handle the complexities involved in
modeling human behaviors. In the forefront of pushing the field of human be­
havioral science, by modeling human behaviors using complex mathematical
algorithms with objective behavioral signals, BSP can provide computational
tools to the domain experts expanding the capability of behavior science re­
searchers to conduct research into finding new scientific insights about human
behaviors that were not possible before.

1.1.1 BSP: Technical Challenges and Complexities

A standard framework and pipeline of carrying out research work in the
domain of BSP is shown in Figure 1.1. It starts with careful experimental
design with appropriate recruitment of human subjects and well-designed
control and experimental variables. The second part involves extensive data
collection, often in the forms of audio-video recordings, and quantitative data
analysis. The second part of the framework is based on an assumption that a
given subject often has a hidden internal state, and his/her expressed behav­
iors are a full or partial realization of that hidden internal state to carry the
intended messages. Then, we are able to perceive and capture these expressed
behaviors through data recording devices, e.g., audio-video recording, physi­
ological sensors, and other advanced technological devices. In order to extract
meaningful and objective behavioral descriptors, signal processing methods
that are applied on these raw recorded signals are employed. Lastly, with the
appropriate behavioral (mathematical) descriptors and modeling references
given by humans, e.g., references could be either explicit or implicit human
judgments and descriptions of a behavior attribute or internal state, behavioral
modeling techniques can then be used to quantitatively understand or predict
behaviors of interest.
There are many technical challenges associated with this pipeline of de­
riving behavioral informatics. Some of the apparent challenges lie in the data
 1.1 BSP: Introduction 5

Figure 1.1 BSP: An interdisciplinary research domain and applications.

collection process and signal processing methodologies. In order to robustly

and automatically model a behavior, an adequate and representative number
of samples are required for training any machine learning technique, i.e.,
giving large amount of examples of distinct behavioral characteristics for
machines to learn from. Also, the instrumentation of recording spaces can
be a challenge in order to make sure that the data we observe is ecologi­
cally valid, i.e., the behavior that each person exhibits shows no differences
between scenarios when there is no recording happening versus when the
person is aware of the ongoing recordings. Another apparent challenge lies
in the difficulties in developing appropriate signal processing methodologies:
“How to devise a signal processing method to extract the most informative
features to best describe the characteristics of a specific human behavior or
internal state from raw recordings coming out these devices?” It means that
the signal processing techniques need to be formulated in a way that best
explain or capture the actual production process of behavior generation, and
these methods have to also be robust to different recording conditions, e.g.,
background noises, room acoustics, and unwanted interferences from other
sources.
6 1 Behavioral Signal Processing (BSP): Behavioral Informatics

Besides these apparent technical challenges, there are other inherent ab­
stract uncertainties in modeling human behaviors and their associated internal
states due to the fact that human are in-the-loop in this cycle of research
development. First of all, the expressed behaviors only offer a partial window
into the insights of a particular subject’s hidden internal state; the challenge,
on the one hand, is to first figure out whether that partial insight is even
meaningful or adequate, and, on the other hand, is to figure out whether
the underlying hidden state of interest is truly associated with the specific
behavior that is “capture-able” using the behavior recording devices in this
current era. Another further issue arises from the subjective nature of human
perception (evaluation). Human evaluation is commonly used as “ground
truth” for training algorithms; for example, in order to automatically rec­
ognize a person’s emotional state, we have to train algorithms to learn the
mapping between behavioral descriptors and humans’ references. However,
due to the fact that human evaluation is subjective, it further adds unwanted
variation to the validity and reliability of the label that the algorithm is re­
lying upon – making the already difficult tasks of modeling human behavior
even more noise-prone. These multi-level technical challenges caused by the
existence of data heterogeneity and variability in human factors, while are
all complex, but also present themselves as many exciting forefront research
programs. In fact, similar emerging interdisciplinary research fields, such as
Social Signal Processing (SSP) [84, 92] and Computational Behavior Science
(CBS)[79], have all been progressing at the forefront of modeling complex
human behaviors mathematically and objectively.

1.2 BSP: Computational Methods for Dyadic Interaction

Dynamics
A core element in quantitatively understanding human behaviors, both typical
and atypical, is through mathematically modeling of interaction dynamics.
Humans naturally engage in interactions during daily life; in fact, dyadic
interactions, i.e., two-person interaction, are one of the most common real-life
interpersonal interaction scenarios. Across various research domains, dyadic
interaction is a core unit of analysis in understanding detailed interaction
dynamics. The types of scientific research questions related to analyzing
dyadic interactions in various domains of human communication studies
can be roughly categorized with attributes of interactions, such as forms
of communication and relationship strengths and types. Forms of commu­
 1.2 BSP: Computational Methods for Dyadic Interaction Dynamics 7

Figure 1.2 Introduction: Schematics of studying dyadic human interactions.

nication, e.g., face-to-face, telephonic, emails, chats, lectures, etc., dictate

how humans exchange information (verbally, non-verbally, lexical contents
only, etc.). The relationship strength between the interacting dyad exists on a
continuum; it can range from two completely strangers to two intimate part­
ners. The relationship type also varies largely; it can be of casual and typical
interactions, e.g., friends talking and siblings interactions, to a more task-
oriented interacting atmospheres, e.g., job interviews and doctor’s visit, or
even involving atypical interactions, e.g., psychologists interacting with chil­
dren with autism spectrum disorder (ASD), or chronically distressed couples’
spoken interactions.
While across research domains, each domain focuses on different aspects
of various types of dyadic interactions, often time, the flow of the analysis
can be generally conceptualized into three major parts (see Figure 1.2). The
first part involves hypothesis formulation into a problem of interest to study
a particular context of dyadic human interaction. The second part involves
behavioral data collection, often in the forms of audio-video recordings, and
quantitative data analysis. The last part involves interpretations based on the
analysis resulted from the second part along with prior knowledge.
If we unpack this second part of the scientific process, we can see
that it fits very well with the core concept of BSP, as exhibited in Fig­
ure 1.1. Different behavioral science domains have established various ways
8 1 Behavioral Signal Processing (BSP): Behavioral Informatics

of “quantifying” human behaviors, e.g., self-report, observation coding based

on coding manuals, surveys, etc. These measures were established to “mea­
sure and quantify” patterns of variations on the behaviors exhibited by the test
subject during the interaction in order to aid the domain experts in quantita­
tively analyzing and judging human behaviors in relation to the formulated
hypothesis. Quantifying and modeling interaction dynamics is at the core
of providing grounding evidences into the study of human interactions. In
fact, during any human interactions, that is what most of us is doing contin­
uously in our mind, i.e., we learned how to decode and encode information
in the expressive behaviors on-the-fly in order to sustain a smooth flow of
interaction.

1.2.1 BSP: Further Complexities in Modeling Interaction Dynamics

Aside from the challenges existing in deriving per-subject behavioral infor­
matics mentioned above in Section 1.1.1, another layer of complexity exists
due to the fact that coupling effect occurs when the subject is undergoing
interactions. As indicated in psychology study [15], as soon as two people
are engaged in a conversation, their internal states are coupled and behav­
iors become mutually dependent. In terms of signals and systems, the study
of human behaviors can now be abstracted by viewing humans as interact­
ing systems (e.g., stochastically coupled dynamical systems with evolving
internal states) and observable and measurable signals (e.g., speech, video,
gestural kinematics and physiological data) as encoding expressive behaviors
carrying meaningful information. The challenge in such interaction modeling
is multi-fold: the coupling dynamics between each interlocutor in an interac­
tion spans multiple levels, along variable time scales, and differs between
interaction contexts. At the same time, each interlocutor’s internal behav­
ioral dynamics produce a coupling that is multimodal across the verbal and
non-verbal communicative channels. Decoding inter and intrapersonal cou­
pling effects displayed and perceived through behavioral cues is the essential
building block leading to better understanding of human behavior. The chal­
lenges of joint models of inter- and intrapersonal interaction at multiple levels
has brought further exciting new research opportunity that is continuously
pushing the state-of-the-art.
The coupling effect is best illustrated in Figure 1.3. Given a speaker, say,
Speaker 2, with an initial internal state, produces expressed behaviors through
a noisy behavioral production. Speaker 2’s interacting partner, say, Speaker
1, perceives Speaker 2’s behaviors through a noisy perception process. This
 1.2 BSP: Computational Methods for Dyadic Interaction Dynamics 9

Figure 1.3 Introduction: Interpersonal interaction dynamics in dyadic interaction.

perception alters the internal state of Speaker 1, which in terms produces

modulated behaviors of Speaker 1. This, then, feeds into Speaker 2 and cre­
ates a loop of dynamics. The concepts of interaction dynamics is a result
of the coupling between internal states and coordination between interacting
dyad’s behaviors. This additional complexity requires further technological
advances in statistical modeling framework to adequately model multiple in­
teracting processes, since interlocutors’ internal states are inherently coupled.
At the same time, in order to quantify various dependencies of expressed
behaviors, a new computational framework needs to be developed due to the
nature of mutual dependencies between interlocutors’ expressed behaviors.