1 Introduction

This survey systematically reviews Deep Emotion Recognition in Conversations (ERC) works. It elaborates on several ERC challenges and opportunities, explaining how these are currently tackled by the reviewed works and discussing how these can be further addressed.

Emotion recognition capabilities are essential not only for successful interpersonal relationships but also for human–machine interaction. Understanding and knowing how to react to emotions significantly improves the interaction and its outcome. It is therefore a crucial component in the development of empathetic machines, which substantially enriches the experiences these can provide.

ERC modules are useful for a wide range of applications, from automatic opinion mining, to emotion-aware conversational agents and as assisting modules for therapeutic practices. ERC is therefore an actively growing research field, with high applicability and potential. Figure 1 depicts the evolution of the number of ERC publications across the years, using the most established benchmark datasets (Busso et al. 2008; Poria et al. 2019a; Zahiri and Choi 2018; Li et al. 2017), supporting the previous observation. Note that it does not account for datasets addressing emotion causes, a recent research direction.

Fig. 1
figure 1

Evolution of the number of ERC publications across the years

Full size image

It should be noted that emotions are only fully recognizable by the person having the emotion. Thus emotion recognition should be interpreted as “inferring an emotional state from observations of emotional expressions and behavior, and through reasoning about an emotion-generating situation” (Picard 2000).

Progress in ERC benefits from the advancements in Sentiment Analysis, particularly in dialogues, whose tasks include determining the polarity and emotion of text. Research in Sentiment Analysis has come a long way and there is a perception that the field has reached its maturity, but there are still aspects and sub-tasks that need to be thoroughly addressed to attain true sentiment understanding (Poria et al. 2023). Such a statement applies to ERC as well. Emotion and sentiment are used interchangeably throughout this survey, but one should bear in mind they have different meanings: emotion is a psychological state while sentiment is the mental attitude created through the existence of the emotion (Damásio 2020).

The conversational setting presents additional challenges and opportunities, besides the ones of Emotion Recognition, such as the presence of intricate dependencies between the emotional states of the subjects who participate in the conversation, and the use of context and common sense to express emotions, amongst others described in this survey.

1.1 Task definition

The task of ERC consists of determining the emotion of each utterance in a textual conversation, for which speaker information is provided for each utterance. Formally, given a sequence of N utterances \([(u_1, p_1), (u_2, p_2), \dots , (u_N, p_N)]\), where each utterance \(u_i\) is spoken by participant \(p_i\), the goal is to predict the emotion label \(e_i\) for each utterance \(u_i\).

1.2 Survey contributions

To the best of our knowledge, the existent surveys on Emotion Recognition in Conversations are by Poria et al. (2019b), focusing on the challenges and advances in text ERC, pointing out conversational context modelling, speaker-specific modelling, the presence of emotion shift, multiparty conversations and the presence of sarcasm, and a more recent survey by Fu et al. (2023), on multimodal ERC, that covers context modeling in conversations, speaker dependency, methods for fusing multimodal information and presents challenges such as real-world ERC and classification on the typical ERC imbalanced datasets.

Related relevant surveys covered sentiment analysis research challenges and directions (Poria et al. 2023) and textual emotion recognition and its challenges (Deng and Ren 2021), the latter including a section on textual emotion recognition in dialogue, elaborating on utterance context modelling and emotion dynamics modelling. However, these are not specific to conversations, not specifying how the models and practices employed in general textual emotion recognition relate to conversations, nor covering several ERC pertaining challenges.

Our survey discusses additional challenges for this task, such as recognizing emotion causes, different taxonomies across datasets, multilingual ERC, and interpretability. It also extensively covers solutions to dealing with imbalanced ERC datasets and learning with subjectivity from annotators.

This is also the first survey presenting an extensive list of Deep Learning works in ERC, stating their methods, modalities, and performance across various datasets, while also describing key works on the topic. We provide comprehensive descriptions of Deep Learning methods from the Multi-Layer Perceptron, Recurrent Neural Networks, Long Short-Term Memory Networks, Gated Recurrent Units, Convolutional Neural Networks, Graph Neural Networks, and Attention Mechanisms to the Transformer, in the light of Emotion Recognition in Conversations. While these methods are transversal to all emotion recognition tasks, and therefore all emotion recognition surveys, some are more appropriate for ERC. Examples are Transformer-based architectures that are better at preserving long-term dependencies between sequences of utterances than recurrent neural networks.

We also set up an online repositoryFootnote 1 with the sources we present, organized into categories such as overviews, datasets, and works on ERC.

1.3 Survey methodology

We report the steps of our systematic review: identification, screening, eligibility, and inclusion.

In the identification step, to collect ERC works for Sects. 4 and 6, we search the accepted paper’s sections of leading NLP conferences (ACL, EMNLP, NAACL, EACL, COLING, and Findings) and additional conferences (AAAI, INTERSPEECH, and ICASSP), using the title keywords “emotion” in combination with “conversational” or “conversations”. Journal papers are also collected from IEEE Access and IEEE Transactions on Affective Computing.

We limit our search to papers published after 2017, given the significant advances in Deep Learning models since that time, particularly the introduction of the Transformer architecture (Vaswani et al. 2017) and pre-trained Transformer models such as BERT (Devlin et al. 2019) and RoBERTa (Liu et al. 2019).

In the screening step, the inclusion criteria are papers that address emotion recognition within conversations, report performance on one or more of the four main benchmark datasets listed in subsection 3.2 (IEMOCAP, MELD, EmoryNLP, and DailyDialog), and report F1-scores, the adequate evaluation metric for emotion recognition in conversations, as elaborated in subsection 5.2. The exclusion criteria are papers not reporting results on the aforementioned benchmark datasets or not reporting F1-scores.

For Sects. 2, 3, and 5, we conduct topic-specific searches using Google Scholar, employing keywords such as “challenges in ERC” or “data taxonomies in emotion recognition”. Search results are filtered by relevance and publication date to prioritize recent and pertinent research.

To complement our searches, we use tools such as Connected Papers to discover papers related to the ones already retrieved.

To assess paper eligibility we evaluate each paper’s full text, focusing on the reported results and alignment with the survey scope, more specifically, the use of Deep Learning methods.

1.4 Outline

This survey is organised as follows: Sect. 1 introduces the topic of Emotion Recognition in Conversations and Sect. 2 presents its challenges and opportunities. Section 3 describes the emotion taxonomies and a variety of ERC benchmark datasets employing such taxonomies. Section 4 provides descriptions of the most prominent works in ERC and explains the Deep Learning architectures employed. Section 5 describes advisable ERC practices towards better frameworks, elaborating on methods to deal with subjectivity in annotations and methods to deal with the typically unbalanced ERC datasets. Section 6 consists of systematic review tables comparing several works regarding the methods used and their performance and provides insights into how the aforementioned works attempted to solve the ERC challenges presented in section 2. Section 7 elaborates on future work directions. Finally, Sect. 8 presents the conclusions, limitations and key findings of the survey.

2 ERC challenges and opportunities

Research in ERC has come a long way and several challenges have been addressed, as described in this section. From context, to speaker and emotion shift modelling to interpreting common sense, informal language and sarcasm, Deep Learning architectures have demonstrated their potential to address these aspects, although there is room for improvement. Application scenarios such as real-time ERC pose additional challenges. In this section, we describe such challenges and point to the sections that are connected with each challenge, and in subsection 6.1 we wrap up the challenges, elaborating on how they are currently being addressed. In Sect. 7, we suggest future work directions concerning partly unaddressed challenges.

2.1 Context, speaker and emotion dynamics modelling

While early works (Lee et al. 2009; Wöllmer et al. 2010) have tackled challenges from conversational context modelling, to speaker-specific modelling and emotion dynamics, resorting to, for example, gated networks, recent ERC approaches have leveraged the novel developments in Deep Learning, such as Transformers and Transformer-based Pre-trained Models to address these aspects.

Context modelling can be interpreted as leveraging information from previous or surrounding words or utterances. Speaker-specific modelling leverages information about each speaker’s utterances, capitalizing on the fact that an utterance belongs to a given speaker, being extremely important in multiparty conversations. Emotion dynamics concerns the variations of the emotions of the participants in the conversation, being more informative but also harder to model than emotion shift. Emotions concern not only who express them but also the receiver. For example, more conscious and reflective people tend to be more sensitive regarding other’s emotions.

Concerning these challenges, gated and graph-based neural networks and transformer-based architectures are successful at modelling long range dependencies between words in utterances and relations between utterances. Key ERC works in Sect. 4 are described with regards to how these Deep Learning architectures were employed to address these challenges.

2.2 Common sense, informal language and sarcasm

Contextual embeddings from language models, such as BERT (Devlin et al. 2019) and RoBERTA (Liu et al. 2019), are pre-trained with large amounts of data, which makes them suitable to deal with informal language, sarcasm, and use of common sense to express emotions, although there is room for improvement. Most of the ERC works featured in this survey make use of these contextual embeddings.

Regarding common sense, commonsense knowledge bases such as COMET (Bosselut et al. 2019) were developed to aid in leveraging the meaning behind phrases employing such knowledge. Three key ERC works described in Sect. 4 make use of knowledge bases. These knowledge bases, however, are not specific to emotions, which could be an extension to consider.

Concerning sarcasm, speaker-specific modelling can aid in identifying the sarcastic participants in the conversation, to more accurately classify their utterances (Poria et al. 2019b).

2.3 Real-time ERC

Systems to be deployed in real-life conversational scenarios may require real-time emotion recognition (Eyben et al. 2010), which is challenging not only for modelling and recognition but also in annotation because the boundaries of each expression of emotion have to be simultaneously recognized as the emotion category (Picard 2000). Datasets such as the ones described in subsubsection 3.2.5 feature annotations in continuous time, being more suitable for real-time ERC.

2.4 Recognizing emotion causes

Recognizing emotion causes, or what triggered the emotions in the conversation, is also an only recently addressed challenge (Poria et al. 2021; Wang et al. 2023; Jeong and Bak 2023; Hu et al. 2024a; Hua et al. 2024; Ma et al. 2024). It involves resorting to datasets enriched with emotion causes, guiding classifiers to identify not only the emotion at stake but also what caused that emotion to arise. Addressing this challenge can greatly contribute towards a more complete emotional understanding.

2.5 Different taxonomies

With the use of different taxonomies and taxonomy types, as described in subsection 3.1 and reflected in the different emotion representations in the datasets in subsection 3.2, it is not straightforward to compare and establish connections between the different datasets, making it difficult, for example, to fine-tune the same ERC model with data from more than a few datasets in which adjustments can be made to their respective taxonomies.

2.6 Multilingual ERC

Most of the benchmark datasets used to evaluate models and develop ERC applications are in English, as it can be observed in subsection 3.2. Translating the English conversations to the target application languages is not ideal since some emotion can be lost in translation. The collection and annotation of multilingual datasets for ERC would benefit research in this direction. Some dialogues involve code-switching, i.e., featuring several languages in the same dialogue or even utterance (Kumar et al. 2023). The cultural aspect of emotion expression/interpretation also poses a challenge, since emotion is not universally expressed or understood in the same way across different cultures.

2.7 Interpretability

A property lacking in most of the works resorting to deep learning architectures is interpretability. It is difficult to interpret the reasoning behind these models’ decisions and identify what are the characteristic features of each class. Examining attention weights is a way of determining to which features the models were attending to while computing classifications, which can aid interpretability, although there are some issues raised in the community (Jain and Wallace 2019; Wiegreffe and Pinter 2019). Other methods can be proposed to provide more interpretability to the ERC pipeline, such as Fuzzy Fingerprinting Transformer Embedding Representations (Pereira et al. 2023b).

3 ERC taxonomies and benchmark datasets

In this section, a description of different taxonomies, i.e., ways of representing emotions, is provided, followed by the presentation of various datasets that resort to such taxonomies. The “dialogues in the datasets reflect our daily communication way” (Li et al. 2017), therefore encompassing several emotions across different taxonomies.

3.1 Emotion taxonomies

One of the most important issues in the study of emotion is whether there are basic types of affect or whether affective states should be modelled as combinations of locations on shared dimensions (Zachar and Ellis 2012). These two dominant views corresponded to the categorical approach, defended by Panksepp (2004), and to the dimensional approach, proposed by Russell (1980), respectively. This is reflected in two main taxonomy approaches, that are described and compared in this section. The Plutchik wheel of emotions (Plutchik 1982) is also described.

3.1.1 Dimensional vs categorical approach

The description of emotions can follow a dimensional approach, a continuous model that allows for an unlimited states’ description. Following this approach, emotion is described by a range of values for valence, arousal, and dominance, the so-called VAD model (Mehrabian 1980). Valence is a measure of pleasure or displeasure of emotion: happiness has a positive valence, fear has a negative valence; arousal ranges from excitement to calmness: anger is high in arousal, sadness is low in arousal; dominance measures how much choice one has over an emotion: fear is low dominance, admiration is high dominance. The distribution of common emotions over the VAD 3-dimensional space can be observed in Fig. 2.

Fig. 2
figure 2

Distribution of common emotions over the VAD 3-dimensional space (Bălan et al. 2019)

Full size image

A motivation for the use of the dimensional description could be the existence of mixed emotions, elaborated in subsubsection 3.1.3.

The description can also follow a categorical approach, in which an emotion is assigned to one or more labels from a set of classes. Compared to the dimensional approach, this categorical discrete framework makes it easier to annotate emotions but is more limited. Several scholars proposed to identify a set of primary emotions. The basic emotions proposed by Ekman (1999) constitute a standard for this emotion description, categorised into joy, anger, sadness, surprise, fear, and disgust.

3.1.2 Plutchik wheel of emotions

The Plutchik wheel of emotions, depicted in Fig. 3, identifies eight core emotions, organised in opposite pairs of a wheel: sadness and joy, anger and fear, expectation and surprise, and trust and disgust. These have associated emotions according to their intensities and also combine to form new emotional states (Plutchik 1982).

Fig. 3
figure 3

The Plutchik wheel of emotions. Image retrieved from Wikipedia

Full size image

3.1.3 Mixed emotions

A person can experience simultaneously several emotions. For example, one can experience joy and surprise at the same time and sometimes even opposite emotions simultaneously (Berrios et al. 2015). While the VAD model allows for an unlimited description of emotions so that a mixture of emotions can be expressed in a single VAD description, the single-label categorical approach does not account for mixed emotions. This motivates the need to classify emotions with multiple labels when using the categorical approach. This need becomes more obvious with the decreasing number of labels in a dataset, but there are also multi-label datasets with a high number of emotion labels, such as the GoEmotions dataset (Demszky et al. 2020) with 28 emotions.

3.1.4 ERC taxonomy considerations

The existent taxonomies are usually transversally applied along all Emotion Recognition tasks. However, for ERC, it can be observed that there are coincident predominant classes in several benchmark datasets, such as Neutral and Happiness/Joy, as can be seen in the class distribution of the datasets on subsection 3.2. Within different ERC domains such as chit-chat vs task-oriented, e.g. mental health (Dheeraj and Ramakrishnudu 2021) and customer support (Herzig et al. 2016), there are specific classes of emotions that will be more represented than others in each domain. With such a rich diversity of possible classes for this task, as it can be observed in the 28 classes of the GoEmotions dataset, the dimensional approach can encompass all classes but might make it harder to distinguish between differentiated emotional displays.

3.2 ERC benchmark datasets

In this subsection, different categorical, single-label datasets are presented, that were used by the works benchmarked in this survey. These works are organised in Table 3. Other relevant datasets are also presented. Overall, their class imbalance is significant suggesting the use of balancing techniques that will be elaborated on in subsection 5.2. All datasets are in English, which is a limitation concerning multilingual ERC, as pointed out in section 2.

3.2.1 IEMOCAP (Busso et al. 2008)

The IEMOCAP dataset consists of 151 videos of two speakers’ dialogues, comprising 7.433 utterances with an average length of 11,6 words. Each utterrance is labelled with one of 8 emotions: happy, sad, neutral, angry, excited, frustrated, fear, and disgust. Most of the works, however, benchmark their performance on the first 6 classes. The maximum class imbalance ratio is 1:3 (happy: frustrated).

Sample: “Is that, is that - is that just foam? I can’t even tell. Although, if you can’t tell, it’s probably isn’t them. It’ll probably be unmistakable, don’t you think?” Label: Excited

3.2.2 MELD (Poria et al. 2019a)

The Multimodal EmotionLines Dataset (MELD) contains 13.708 utterances from 1.433 dialogues from the TV series Friends. Each utterance encompasses audio, visual, and textual modalities, with an average length of 8,0 words. Emotion categories are Ekman’s basic emotions and neutral. The maximum class imbalance ratio is 1:18 (fear: neutral).

Sample: “Are we okay now?” Label: Fear

3.2.3 EmoryNLP (Zahiri and Choi 2018)

EmoryNLP is a text dataset extracted from transcripts of the TV show Friends. It contains 12.606 utterances, labelled with one of the six primary emotions in Willcox’s feeling wheel (Willcox 1982) and a neutral label. The maximum class imbalance ratio is 1:4 (sad: neutral).

Sample: “What a coincidence, I listen in my sleep.” Label: Peaceful

3.2.4 DailyDialog (Li et al. 2017)

DailyDialog is a large text dataset built from websites used to practise English dialogue in daily life, containing 13.118 multi-turn dialogues, comprising 102.879 utterances with an average length of 14,6 words. Each utterance is labelled with one of the six Ekman’s basic emotions or other. The maximum class imbalance ratio is 1:1156 (fear:other).

Sample: “I was scared stiff of giving my first performance” Label: Disgust

Table 1 shows a comparison of the label distributions of the aforementioned datasets and Table 2 shows a comparison of the corresponding percentages.

Table 1 Label distribution comparison of different ERC datasets
Full size table
Table 2 Label percentage distribution comparison of different ERC datasets
Full size table

3.2.5 Other datasets

The SEMAINE dataset (McKeown et al. 2011) consists in audiovisual recordings of interactions between a human and an operator undertaking the role of an avatar with four personalities. Its five core emotion dimensions are valence, activation, power, anticipation/expectation, and intensity.

The RECOLA dataset (Ringeval et al. 2013) is a multimodal corpus of spontaneous human interactions collected from a collaborative task. It includes not only audio and video, but also electrocardiogram and electrodermal activity data. Emotional behaviours are annotated in terms of arousal and valence, in continuous time, and social behaviours are in terms of agreement, dominance, engagement, performance, and rapport, in discrete time.

The SILICONE dataset (Chapuis et al. 2020) consists of preexisting datasets which have been considered challenging and interesting. It has annotations on Dialog Acts and Sentiment/Emotion. Concerning the latter, it comprises DailyDialog, MELD, IEMOCAP and SEMAINE.

4 Deep learning for emotion recognition in conversations

In this section, key ERC works are described, focusing on the challenges and opportunities of the conversational scenario, presented in Sect. 2, and on the diverse Deep Learning architectures that can be employed for the task. Descriptions of key Deep Learning models are provided along with descriptions of works that resorted to those models. The works on Emotion Recognition in Conversations using Deep Learning described in this survey are chosen to be representative of the different techniques that are employed, covering gated Neural Networks with and without bidirectionality, Memory Networks, Graph Neural Networks, Attention Mechanisms, and Transformers. It is remarkable to observe how in four years research evolved from a simple Long Short-Term Memory Network (LSTM), taking into account the context of the conversation, to complex gated and graph neural network architectures and how the invention of transformers was reflected in not only word embeddings, such as BERT, but also in new classifier architectures. Comparisons between these works are performed based on the reported weighted-F1-score without the majority class on the IEMOCAP dataset, the choice of most authors including the prominent works of Poria et al. (2017), Hazarika et al. (2018b), Majumder et al. (2019), Ghosal et al. (2019), Zhong et al. (2019), Shen et al. (2021a), Ghosal et al. (2020), Wang et al. (2020b), Li et al. (2021a), Shen et al. (2021b) and Lei et al. (2023), since the dataset is not too imbalanced.

4.1 Convolutional neural networks for feature extraction

Convolutional Neural Networks (CNNs) are a type of feedforward neural network inspired by the biology of the visual cortex that mimics the local filtering over the input space performed by the receptive fields. In the case of text, a linear layer is applied to a sequence of words, each represented by its embeddings, separately and in the same way for each sequence. The convolutional layer can have many filters, each extracting a different feature, and is usually followed by a pooling operation. In the case of Emotion Recognition in Conversations, convolutional layers act as feature extractors, useful since strong predictors of the label can appear in different places in the input. With the advent of Pre-Trained Language Models for utterance representation, elaborated on subsection 4.10, convolutional neural networks have been less used.

4.2 The multi-layer perceptron

Emotion Recognition is a classification task. The most simple neural network architecture that can be employed for such a task is the Multi-Layer Perceptron.

The Multi-Layer Perceptron (MLP) is a non-linear function approximator that can be used for classification and regression. It is a class of feed-forward artificial neural networks that is composed of at least three layers: an input layer, a hidden layer, and an output layer. Its complexity and representation power increases with the number of layers. A MLP is mathematically described by the following equation:

$$\begin{aligned} \hat{y}=f_{L}(W_{L}(\ldots f_{2}(W_{2}(f_{1}(W_{1}x+b_{1})+b_{2})\ldots )+b_{L}, \end{aligned}$$
(1)

In which L is the number of layers, \(W_l \in \mathbb {R}^{l-1} \times \mathbb {R}^{l}\) and \(b_l \in \mathbb {R}^{l}\) are the weight matrix and bias vector corresponding to layer l, and \(f_{l}\) is a non-linear activation function corresponding to layer l. Except for the input layer, all layers’ units are followed by a non-linear activation function, such as the sigmoid, the hyperbolic tangent, or the rectifier linear unit. When performing classification, the last layer is followed by the sigmoid activation function for binary classification or the softmax activation function for multiclass classification. A multiclass MLP with one hidden layer is depicted in Fig. 4.

Fig. 4
figure 4

A multy layer perceptron with one hidden layer, input layer with feature dimension N and output layer with feature dimension C

Full size image

The MLP is commonly trained with the Backpropagation algorithm (Rumelhart et al. 1986). The idea is to update the model parameters in the opposite direction of the gradient, such that a loss computed from the prediction error is minimised. The algorithm is composed of a forward pass and a backward pass. In the forward pass, the input is passed through all the layers in the network, yielding a prediction. In the backward pass, the gradients of the model’s parameters are computed with respect to the loss function, making use of the chain rule. These are computed one layer at a time, iterating backward from the last layer. The computation of the gradients along with weight update are done following an optimization method, commonly using Stochastic Gradient Descent (SGD) algorithms.

When used for Emotion Recognition, the input to the network is a set of extracted features representing each utterance obtained for example with the Bag of Words approach for text, which does not account for the order of the words, fitting this type of network but not being the ideal method for the task. That is the main reason why state-of-the-art works do not resort to this architecture.

4.3 Elman recurrent neural network

Feed-forward neural networks, such as the above, assume data has no sequential dependencies which is not the case in Emotion Recognition in which an utterance is a sequence of words that can also comprise sequential data from other modalities. Recurrent Neural Networks (RNNs) introduce memory into the network, being able to learn sequences. They process one input, such as a word, at a time, producing a hidden state which is a summary of the sequence of inputs up to the current timestep. The most simple one, the Elman RNN (Elman 1991), updates the hidden state as in the following equation:

$$\begin{aligned} h_t=f(Wx_t+Uh_{t-1}+b), \end{aligned}$$
(2)

where \(h_t \in \mathbb {R}^{h}\) is the hidden state at time t, \(x_t \in \mathbb {R}^{N}\) is the input at time t, \(W \in \mathbb {R}^{h\times N}\), \(U \in \mathbb {R}^{h\times h}\) and \(b \in \mathbb {R}^{h}\) are parameters to be learned and f is a non-linear activation function such as the sigmoid or the hyperbolic tangent. At each timestep the RNN can produce an output:

$$\begin{aligned} \hat{y}_t=g(Vh_t+c), \end{aligned}$$
(3)

where \(y_t \in \mathbb {R}^{y}\) is the output at time t, \(V\in \mathbb {R}^{y\times h}\) and \(c \in \mathbb {R}^{y}\) are parameters to be learned and g is a non linear activation function.

A visualization of the RNN update through time-steps is provided in Fig. 5.

Fig. 5
figure 5

Two visual descriptions of the recurrent neural network. The description on the right is unrolled, highlighting that parameters W, U and b are shared between timesteps

Full size image

RNNs are trained with a modification of the Backpropagation algorithm, the Backpropagation Through Time (BPTT) algorithm (Werbos 1990). All the parameters are shared between all time steps and it can be observed that an unrolled RNN is comparable to a feed-forward neural network with the same parameters shared over layers.

In the case of Emotion Recognition, the input to the network is a sequential utterance, the textcorresponding to one word being fed per timestep, and the output is the predicted emotion.

In some cases, particularly when dealing with long sequences such as when considering several utterances to model context in Emotion Recognition in Conversations, the backpropagation algorithm yields vanishingly small gradients, preventing the parameters from changing their values and eventually stopping training.

4.4 Long short-term memory networks

To overcome this problem, Long Short-Term Memory Networks (LSTMs) (Hochreiter and Schmidhuber 1997) were proposed. LSTMs are composed of a memory cell and three gates. The memory cell stores the information about the input sequence across timesteps while the gates control the flow of information across the memory cell: the input gate controls the proportion of the current input to include in the memory cell; the forget gate the proportion of the previous memory cell to forget; the information gate the information to output from the current memory cell. This can be described by the following equations:

$$\begin{aligned} i_t= & sigm(W_ix_t+U_ih_{t-1}+b_i) \end{aligned}$$
(4)
$$\begin{aligned} f_t= & sigm(W_fx_t+U_fh_{t-1}+b_f) \end{aligned}$$
(5)
$$\begin{aligned} o_t= & sigm(W_ox_t+U_oh_{t-1}+b_o) \end{aligned}$$
(6)
$$\begin{aligned} c_t= & f_t \odot c_{t-1}+i_t\odot tanh(W_cx_t+U_ch_{t-1}+b_c) \end{aligned}$$
(7)
$$\begin{aligned} h_t= & o_t \odot tanh(c_t), \end{aligned}$$
(8)

where \(i_t\) is the input gate, \(f_t\) the forget gate, \(o_t\) the output gate, \(c_t\) the memory cell and sigm and tahn the element-wise sigmoid and hyperbolic tangent activation functions respectively. The several \(\text{W}, \, \text{U} \, \text{b} \) are parameters to be learned. A visual depiction of an LSTM cell is provided in Fig. 6.

Fig. 6
figure 6

Long-short term memory network cell

Full size image

The vanishing gradient problem is overcome by the gates, specially the forget gate.

As in the Elman RNN, in the case of Emotion Recognition the input to the network can be a sequential utterance and the output is the predicted emotion. LSTMs are usually leveraged to model dependencies between several sequential utterances. In this case, the input to the network is a sequence of utterances.

The LSTM described in the equations above does not capture information from future time steps, which is useful for example in text applications when the context of a word is made up of both previous and subsequent words in the phrase. Therefore, Bidirectional Long Short-Term Memory Networks (Bi-LSTM) (Graves et al. 2005) were proposed. Bi-LSTMs consist of two LSTMs: one executing a forward pass and the other a backward pass. A sequence of hidden states is produced for each direction that are usually joined into a unique sequence of hidden states through concatenation.

4.4.1 Context-dependent emotion recognition (Poria et al. 2017)

There is a strong correlation and influence in emotion distribution between sequential utterances in a conversation. Leveraging those contextual dependencies is key to the performance of a classifier. An utterance is a unit of speech which is bound by breathes or pauses (Olson 1977). In textual data, it is usually delimited by punctuation such as full stops, but can also be delimited by question or exclamation marks. Along these lines, the work of Poria et al. (2017) considers interdependences among utterances. The approach consists of an architecture made of LSTMs to extract contextual features from the utterances. The model enables consecutive utterances to share information while preserving their order. The authors propose the contextual LSTM, consisting of unidirectional LSTM cells, the hidden LSTM, in which the dense layer after the LSTM cell is omitted, and the bi-directional contextual LSTM that considers information from before and after the targeted utterance. It achieved an F1-score of 54.95% on the IEMOCAP dataset.

4.5 Gated reccurent units

Another RNN, which structure is somehow similar but with fewer parameters than the LSTM, is the Gated Recurrent Unit (GRU) (Cho et al. 2014) that can be described by the following equations:

$$\begin{aligned} r_t= & sigm(W_rx_t+U_rh_{t-1}+b_r) \end{aligned}$$
(9)
$$\begin{aligned} z_t= & sigm(W_zx_t+U_zh_{t-1}+b_z) \end{aligned}$$
(10)
$$\begin{aligned} \hat{h}_t= & tanh(W_hx_t+U_hh_{t-1}+b_h) \end{aligned}$$
(11)
$$\begin{aligned} h_t= & z_t \odot h_{t-1}+(1-z_t)\odot \hat{h}_t, \end{aligned}$$
(12)

in which the \(r_t\) is the reset gate that leads the candidate hidden state \(\hat{h}_t\), to ignore or not the previous hidden state, and \(z_t\) is the update gate that controls the proportion of the information from the previous hidden state to carry over to the current hidden state.

GRUs can work better than LSTMs when there is less available data since they have fewer parameters, and are less prone to overfitting.

4.5.1 DialogueRNN (Majumder et al. 2019)

DialogueRNN combines GRUs in a sophisticated way, in which each GRU plays a specific role in modelling the conversation.

DialogueRNN predicts the emotion of an utterance based on the speaker, the context of preceding utterances, and the emotions from preceding utterances. The model has three components all modeled by GRUs. The party-state models the parties’ emotion dynamics throughout the conversation. The global state has the encoding of preceding utterances and the party state, and models the context of the utterance. Finally, the emotion representation is based on the party state and the global state and is used to perform the final classification.

Several variants of the method are proposed. DialogueRNNl considers an additional listener state while a speaker utters, BiDialogueRNN uses a Bi-directional RNN, DialogueRNN+Att uses attention over all surrounding emotion representations and BiDialogueRNN+Att combines the latter approaches.

It yielded an F1-score of 62.75% on the IEMOCAP dataset. The authors argue DialogueRNN variants outperform the contextual LSTM due to better context representation.

4.6 Attention mechanisms

The aforementioned work mentions the use of attention over surrounding emotion representations. To motivate Attention Mechanisms (AM) (Bahdanau et al. 2014) we now introduce Sequence to Sequence (Seq2Seq) models. Seq2Seq models are composed of an RNN encoder that takes the input sequence and outputs a context vector, the last encoder hidden state, that is fed into an RNN decoder that generates an output sequence from this vector. When the input sequence is very long the context vector does not have enough capacity to represent the information from the entire sequence. This is where AMs are useful. Taking inspiration from the human visual system that attends to different parts of the space, while building its representation of the scene, the AM allows the decoder to attend to the relevant encoder hidden states. At each time step, a context vector is obtained by a weighted sum of these hidden states, where the weights of the sum are proportional to the similarity between the current decoded hidden state and the encoded hidden states, as described in the following equations:

$$\begin{aligned} a_{ti}= & \frac{exp(score(h_t^d,h_i^e))}{\sum _{j=1}^{T}exp(score(h_t^d,h_j^e))} \end{aligned}$$
(13)
$$\begin{aligned} c_t= & \sum _{j=1}^{T}a_{tj}h_j^e, \end{aligned}$$
(14)

where \(h_t^d\) is the decoder hidden state and \(h_i^e\) and \(h_j^e\) are the encoder hidden states. The most common similarity score function is the dot product. By introducing AM in an RNN the performance of a classifier usually increases.

4.7 Memory networks

4.7.1 Conversational memory network (Hazarika et al. 2018b)

The recurrent networks described before are limited in terms of long-range summarization since they rely on a sequential processing approach with implicit memory. The Conversational Memory Network (CMN) accounts for this factor through the use of memory networks, which can better capture long-term dependencies and have attention models that can summarize specific details using explicit memory structures.

The work is based on the notion that emotional dynamics involve self and inter-speaker emotional influence. Separate histories for each speaker, consisting of the speaker’s previous utterances, are modeled into memory cells of the Conversational Memory Network using GRUs. Memory networks provide a memory component that can be read from and written to and also perform inference. The CMN then employs an attention mechanism over historical utterances from each speaker to filter out relevant content for the current utterance. This mechanism is repeated for multiple hops of the network to subsequently classify the utterance. CMN achieves an F1-score of 56.13% on the IEMOCAP dataset.

4.7.2 Interactive conversational memory network (Hazarika et al. 2018a)

The Interactive Conversational Memory Network (ICON) is an improvement upon CMN that has an additional dynamic global influence module to model inter-personal emotional influence, a module that maintains a global representation of the conversation, a global state that is updated using a GRU operation on the previous state and current speaker’s history. ICON yields an F1-score of 58.54% on the IEMOCAP dataset.

4.8 Graph neural networks

An alternative to gated neural networks that is also useful to capture long term dependencies is using graph neural networks. Graph neural networks are based on graphs. The latter model a set of objects, the nodes, and their relationships, the edges. While standard neural networks such as CNNs and RNNs cannot handle the graph input properly since they need a specific order for the nodes, the output of GNNs is invariant for the input order of nodes. Furthermore, while in standard neural networks, dependencies between nodes are a node feature, in GNNs there are edges to represent these dependencies. GNNs can propagate information by the graph structure instead of using it as part of features (Zhou et al. 2020).

Two common types of GNNs are recurrent GNNs (RecGNNs) and convolutional GNNs (ConvGNNs). RecGNNs learn node representations with recurrent neural architectures, assuming that a node in a graph constantly exchanges information with its neighbors until a stable equilibrium is reached. ConvGNNs generalize the convolution operation from grid structure to graph structure, generating a node representation by aggregating its features with its neighbours’ features. For classification tasks, such as Emotion Recognition, an end-to-end framework can be constructed for example by stacking graph convolutional layers followed by a softmax layer (Wu et al. 2020).

4.8.1 DialogueGCN (Ghosal et al. 2019)

Dialogue Graph Convolutional Network (GCN) takes into account intra and inter-speaker dependency and background information, having a sequential and a speaker-level encoder. The sequential context encoder encodes the utterances using a bidirectional GRU and is speaker-agnostic. The speaker-level context encoder creates a directed edge-labelled graph in which each utterance, enriched with context, is a node in the graph. The information from neighbor nodes is then aggregated and passed to a neural network in each node, resulting in an updated representation of each node that takes into account its neighbors. In DialogueGCN, node features are initialized with sequentially encoded feature vectors from the sequential context encoders, and edge weights are set using a similarity-based attention module. The utterance level emotion classification is turned into a problem of node classification in the graph. DialogueGCN achieves an F1-score of 64.18% on the IEMOCAP dataset.

4.9 Transformer

Despite the success of recurrent and graph neural networks in a wide variety of tasks including Emotion Recognition, a more powerful network model exists, constituting the current state-of-the-art: the Transformer (Vaswani et al. 2017). The Transformer is also better at capturing long-term dependencies than gated RNNs, due to its shorter path of information flow, which is useful when modelling several utterances in Emotion Recognition in Conversations. It does not resort to recurrence or convolutions. It is also more parallelizable and needs less training time. It is composed of an Encoder and Decoder, which can be visualized in Fig. 7.

Fig. 7
figure 7

The transformer. The encoder on the left and decoder on the right display some similar components

Full size image

Each Encoder layer is composed of two sub-layers. The first is a so called self-attention layer for building the context of the sequence that calculates the relevance of the tokens in the sequence for each given token.

The self-attention mechanism takes as input three matrices, the Query, Key, and Value matrices. These are computed from the dot product between the input sequence matrix and weight matrices learned in the training phase. It then computes a vector of attention scores, as described in the following equation:

$$\begin{aligned} Attention(Q, K, V )=softmax\left( \frac{QK^T}{\sqrt{k}}\right) V, \end{aligned}$$
(15)

in which \(Q \in \mathbb {R}^{N\times m}\), \(K \in \mathbb {R}^{N\times m}\) and \(V \in \mathbb {R}^{N\times m}\) are the Query, Key, and Value matrices, respectively, and m is the model output dimension. The dot product of the Query matrix with the Key matrix represents scores that are then multiplied by the Value matrix to keep intact the values of relevant words and drown-out the irrelevant ones. The \(\frac{1}{\sqrt{k}}\) helps stabilizing the gradients.

The self-attention layer uses a multi-head attention mechanism so that the attention score vectors do not exclusively give a high probability score for its corresponding token, defined by the following equations:

$$\begin{aligned} MultiHead(Q, K, V ) = Concat(head_1, ..., head_h)W^{O}, \end{aligned}$$
(16)

where

$$\begin{aligned} head_i = Attention(QW_i^Q , K W_i^K , V W_i^V ), \end{aligned}$$
(17)

in which \(W_i^Q \in \mathbb {R}^{m\times k}\), \(W_i^K \in \mathbb {R}^{m\times k}\) and \(W_i^V \in \mathbb {R}^{m\times v}\) are projection matrices and \(W^O \in \mathbb {R}^{hv\times m}\), being h the number of heads and k=v=m/h.

The output of the self-attention layer is fed to the second layer, a fully connected feed-forward network, with two linear transformations with a ReLU activation in between, applied separately and in the same way to each position, allowing for parallelization.

A residual connection is applied around each of the two sub-layers, followed by layer normalization.

Each Decoder layer is composed of three sublayers. The first is a masked self-attention layer similar to the self-attention layer described mathematically in Eqs. 16 and 17, but in which each position in the decoder attends to all positions in the decoder up to and including that position, which is attained by masking out values in the softmax. The second is an “encoder-decoder attention” layer, described mathematically in the same equations, where \(\text{K}, \, and \, \text{V} \)come from the output of the encoder and Q comes from the output of the decoder’s masked self-attention layer, mimicking the typical encoder-decoder attention mechanisms in Seq2Seq models. The last layer is a feed-forward neural network, similar to the Encoder layer.

Similar to the Encoder, a residual connection is applied around each of the two sub-layers, followed by layer normalization.

Positional encodings are added to the embeddings that are fed to the Encoder and Decoder to introduce information about the position of the tokens in the sequence.

The most common use of the Transformer is as the backbone for a fine-tuned pre-trained transformer-based language model, such as BERT (Devlin et al. 2019), described in subsection 4.10. These language models are trained with large amounts of data to perform many tasks and only need to be fine-tuned for the specific task at hand. In the case of a classification task such as Emotion Recognition, one just needs to add a linear layer or a more complex neural network architecture on top of the language model. The training is performed with supervised learning, resorting to backpropagation algorithms.

4.9.1 Knowledge-enriched transformer (Zhong et al. 2019)

The invention of the Transformer (Vaswani et al. 2017) led to new state-of-the-art results in several Natural Language Processing tasks. The Knowledge-Enriched Transformer (KET) uses self-attention to model context and response using the Transformer, that a has shorter path of information flow than gated RNNs, overcoming the difficulty in capturing long-term dependencies. Conversations are modeled in the entire Transformer as a single input.

First, concepts are retrieved for each word in the conversation using an external knowledge base, a graph of concepts, then the concept representation is computed using a dynamic context-aware graph attention mechanism over the concepts that are then combined with the input sentence embeddings. Self-attention is used to learn utterance representations individually and hierarchical self-attention is applied to the context to learn the context representation. Finally, the encoder and decoder attention mechanism is applied, followed by max pooling and a linear layer. KET achieves an F1-score of 59.56% on the IEMOCAP dataset.

4.9.2 CESTa (Wang et al. 2020b)

The CESTa or Contextualized Emotion Sequence Tagging approaches ERC as a task of sequence tagging, choosing the set of tags with the highest likelihood for the whole utterance sequence at once, by using a Conditional Random Field (CRF) (Lafferty et al. 2001). In order to capture long-range global context, utterance representations are generated by a multi-layer Transformer encoder. These are then fed to a bi-LSTM encoder that captures self and inter-speaker dependencies, resulting in contextualized representations of utterances that are then passed to the CRF layer. It achieves an F1-score of 67.10% in the IEMOCAP dataset.

4.9.3 DialogXL (Shen et al. 2021a)

DialogXL is an adaptation of the pre-trained language model XLNet (Yang et al. 2019), which is based on the Transformer-XL (Dai et al. 2019), for Emotion Recognition in Conversations. It replaces XLNet’s segment recurrence with a memory utterance recurrence to leverage historical utterances. These utterances’ hidden states are stored in a memory bank to reuse them while identifying a query utterance. The approach also replaces XLNet’s vanilla self-attention proposing a dialog-aware self-attention to grasp useful intra- and inter-speaker dependencies. This is composed of four types of self-attention: global and local self-attention for different sizes of receptive fields, and speaker and listener self-attention for intra- and inter-speaker dependencies. DialogXL achieves an F1-score of 65.94% on the IEMOCAP dataset.

4.10 Embeddings from pre-trained transformer language models

Text features must represent words in a way that is useful for the models, in this case, numeric vectors referred to as word embeddings. These embeddings yield similar representations for similar words.

Static embeddings (Mikolov et al. 2013; Pennington et al. 2014; Mikolov et al. 2018) are obtained based on the co-occurrence of adjacent words and have a fixed representation for the words, not taking into account their context.

Contrarily, contextual embeddings include information from the context in the word representation, a representation that differs according to the occurrence of the word. The only disadvantage of this kind of representation is that we cannot use pre-trained embeddings as in static embeddings so there is an inference time for generating the embeddings. The models are pre-trained with large amounts of data, in order to enable generating good embedings. A first approach, Embeddings from Language Models (ELMo) (Peters et al. 2018) is based on a stack of two Bi-LSTMs, leveraging the full context of a word. It provides a context-free representation of the word along with context information of the sense of the word and its syntax. The second approach, Bidirectional Encoder Representations from Transformers (BERT) (Devlin et al. 2019), built from a Transformer encoder stack, leads to state-of-the-art results in multiple Natural Language Processing tasks. The input given to BERT is the sum of three embeddings, the token embedding corresponding to the sequence of words, the segment embedding containing information on which sentence each token belongs to, and the position embedding, defining the position of each token in the sequence and distance to other tokens taking into account the order of the sequence. An improvement upon BERT, RoBERTa (Liu et al. 2019) is trained with more data and for a longer period of time. Given that context is key for text Emotion Recognition in Conversations, contextual word embeddings are more adequate for the task than static embeddings, as it can be seen in the higher performance of works that resort to these embeddings, namely resultant from pre-trained language models, such as BERT-based embeddings. The latter are also more tailored to deal with common sense, informal language, and sarcasm, ERC challenges described in Sect. 2, since they are pre-trained on large amounts of text.

4.10.1 COSMIC (Ghosal et al. 2020)

Besides choosing appropriate utterance representations and classifier architectures, there are other factors that can be leveraged for the task of ERC. One such factor is the commonsense knowledge of the interlocutors that plays a central role in inferring the latent variables of a conversation, such as speaker state and intent.

COSMIC extracts commonsense features resorting to COMET (Bosselut et al. 2019), a commonsense transformer encoder-decoder model trained on the task of generative commonsense knowledge construction that uses GPT (Radford et al. 2019) as its generative model. The approach uses RoBERTa for context-independent feature extraction, passes each utterance through COMET’s encoder, and extracts the activations from the final time step. COSMIC maintains a context state and attention vector which are always shared between the participants of the conversation. Five GRUs are used to model context state, internal state, external state, intent state, and emotion state, being the latter used for emotion classification. This work achieves an F1-score of 65.28% in the IEMOCAP dataset.

4.10.2 Psychological (Li et al. 2021a)

Psychological also leverages COMET as the commonsense knowledge base and resorts to RoBERTa as the utterance-level encoder. It proposes SKAIG as a conversation-level encoder, a locally connected graph, where the targeted utterance receives information from that past and future context and is also self-connected.

Assuming that the influence of an utterance on contextual utterances is locally effective, the targeted node is connected with contextual nodes in a window of a given size of utterances in the past and future. Knowledge from COMET is introduced to enrich the edges with different relations. To propagate information through SKAIG a Graph Transformer (Shi et al. 2021) is used. Finally, a linear unit predicts the emotion distributions. This work achieves an F1-score of 66.96% in the IEMOCAP dataset.

4.10.3 DAG-ERC (Shen et al. 2021b)

Graph-based methods tend to neglect distant utterances and sequential information. On the other hand, recurrence-based methods leverage those but tend to update the query utterance’s state with limited information from the nearest utterances. Combining both graph and recurrence structures explores their advantages and mitigates their drawbacks. This is the idea behind DAG-ERC, which regards each conversation as a directed acyclic graph (DAG), a combination of both structures. DAG-ERC is based on the DAGNN (Thost and Chen 2021) architecture, with two improvements: a relation-aware feature transformation that gathers information based on speaker identity and a contextual information unit that enhances the information of historical context. RoBERTa-Large is used as the feature extractor. It achieves an F1-score of 68.03% on the IEMOCAP dataset, outperforming the previous approaches.

4.11 ERC within a generative framework

With the advent of Generative Large Language Models (LLMs), such as the GPT series (Radford et al. 2018, 2019; Brown et al. 2020) and Llama (Touvron et al. 2023), classification tasks started to be reformulated within generative frameworks. The idea is to prompt the LLM with the utterance to be classified and other relevant additional input and request an emotion label for the utterance. Open-source LLMs can also be fine-tuned and even pre-trained.

4.11.1 InstructERC (Lei et al. 2023)

In InstructERC the prompt consists of the utterance to be classified, the ERC instruction, the conversational context, the set of possible classification labels, and similar utterances together with its emotion labels. The LLM is pre-trained with a speaker identification task and fine-tuned both with the main ERC task and an emotion influence prediction task to improve overall performance. Experiments were made with several LLMs, and the best results were obtained with LLama2. InstructERC achieves 71.39% in the IEMOCAP dataset, outperforming the previous approaches.

5 Advisable ERC practices

This section provides useful methods for better ERC frameworks, from different ways of dealing with subjectivity in emotion annotation and modelling, to several ways of dealing with the typically unbalanced ERC datasets.

5.1 Learning subjectivity

All human annotation tasks involve some degree of uncertainty, that can stem amongst other factors from the subjectivity inherent to the annotator. Since Emotion Recognition is amongst the most subjective tasks (Uma et al. 2021) and that subjectivity is the main source of uncertainty in Emotion Recognition annotation, we address uncertainty as subjectivity in this survey (Rizos and Schuller 2020). A good annotation practice is to resort to several annotators to avoid biasing the results towards a single annotator’s subjectivity. This will naturally yield different labels from each annotator, the so called annotation disagreement. The more appropriately one deals with the disagreement in the annotated labels the more reliable will the data and classifiers be. Several techniques for dealing with disagreement are described in this section. They are divided into techniques that address subjectivity by adapting the labels, towards an estimate of a gold label, and techniques that embrace subjectivity by adapting the classifier architectures to deal with the subjectivity of the labels, leveraging such property.

5.1.1 Addressing subjectivity

For categorical labels, considering that a gold or true label exists for each sample, the simple way to obtain it would be through majority voting, and choosing the label with the most annotations. However, this does not account for the different annotator skill levels or sample inherent difficulty.

Thus, a common process is to calculate an inter-annotator agreement score to find low agreement data. A very simplistic way of calculating inter-annotator agreement for categorical labels would be to use the percent agreement, dividing the number of agreement classifications by the total number of classifications. However, non-expert annotators do not always know which label to use so they sometimes just guess, resulting in a random label. Cohen pointed out that there is some level of agreement in these random labels and developed Cohen’s Kappa to take that factor into account (McHugh 2012). The related Fleiss’ Kappa (Fleiss 1971) is the adaptation of Cohen’s Kappa for 3 or more annotators.

The low agreement data can then be discarded or trained on and evaluated separately from the high agreement data. It can be considered random noise if it does not reveal the existence of annotator bias (Uma et al. 2021).

For interval labels, weighted fusion of data can be performed with approaches such as the Evaluator Weighted Estimator (EWE) (Grimm and Kroschel 2005), which weights each rater’s annotations based on its rater-specific inter-reliability scores, or the Weighted Trustability Evaluator (WTE) (Hantke et al. 2016), which is instead based on each rater’s performance consistency.

5.1.2 Embracing subjectivity (Rizos and Schuller 2020)

The annotator (dis)agreement can be used as privileged information for master-class learning, being viewed as samples’ additional information (Eyben et al. 2012) to facilitate learning in the training phase, that might or might not be required for making predictions during testing, for example, to weigh positively the loss of the corresponding samples. In this setting, it is useful to learn the meaning of high-rater disagreement for the particular dataset in use.

Instead of considering the existence of a hard label, as in subsubsection 5.1.1, one can calculate the distribution of label annotations to define a soft label distribution per sample, which encodes the subjectivity for each sample and allows for the classifier to learn label correlations. Another approach is to model the annotations of each particular rater, through an ensemble of models. It allows to give more importance to expert raters rather than novices or spammers. The addition of an “unsure” label would facilitate the annotator’s workload and also make the model learn the ambiguity ground-truth, although not having to predict “unsure” labels during testing. Furthermore, methods to understand whether a sample is mislabelled by certain raters or inherently ambiguous could inform the model and annotators during active learning processes.

In ERC, the context of the interactions is not always explicit and the dialogues contain a variety of topics that can exacerbate subjectivity even more, making it much more advisable to embrace subjectivity in this domain.

5.2 Dealing with unbalanced data

All the benchmark datasets used for ERC are unbalanced, some of them are severely unbalanced: the unbalance ratio varies from 1:3 up to 1:1156. In the presence of these unbalances, most classifiers tend to favor the majority classes which, in these datasets, corresponds to the neutral emotion class. Hence, unless the unbalance issue is addressed, most models will be trained and evaluated based on the neutral class results, which is far from intended. All classes should be equally relevant in ERC, and even when Accuracy values look acceptable, a more detailed analysis usually reveals a very poor performance in minority classes.

5.2.1 Metrics for unbalanced data

The performance metrics used for Emotion Recognition in Conversations are Accuracy, Precision, Recall, and F-score or F1, the harmonic mean of Precision and Recall. These are commonly used metrics in many classification applications. One can consider the weighted version of these metrics, that take into account the relative frequencies of each class, or micro-averaging, in which all samples equally contribute to the final averaged metric. Since both weighted versions and micro-averaging depend more on the classes with a bigger number of items, those are not good indicators of the performance of the minority classes. Therefore, those should not be used when the classes have the same importance and unequal frequencies. In unweighted macro-averaging, the metric is computed for each class, and the results are averaged over all the classes. By maximizing an unweighted metric, one is maximizing the performance of the model in correctly classifying all classes regardless of the number of items they have.

ERC is clearly a task that deals with real-world unbalanced data. As such, a proper evaluation should always be performed on a test set that maintains the real-world unbalance of the data and using unweighted metrics such as macro F-score. Ideally, performance results on each of the individual classes should also be presented. Accuracy, micro-averaging, and weighted metrics are mostly irrelevant in what concerns the performance of the models since they are mostly dependent on the majority class, usually neutral. Unweighted metrics that are not affected by dataset unbalance, such as Recall, should also be avoided, unless used in conjunction with those that are, such as Precision.

5.2.2 Balancing techniques

Besides the use of appropriate metrics, another way of dealing with dataset unbalance is to apply balancing techniques to the training set. Common such techniques are random under-sampling, removing observations from the majority class, and random over-sampling, adding copies to the minority class. More elaborate techniques, such as Smote - Synthetic Minority Oversampling Technique (Chawla et al. 2002), usually improve the results.

Balancing a training set has its advantages, but it is of utmost importance that the test set maintains the original set balance. Otherwise, metrics such as Precision and F-score are artificially improved and the model is unlikely to be effective in an unbalanced real-world dataset. It should be noted that Recall is not affected when training and testing on an artificially balanced dataset so ignoring Precision and looking only at Recall as an evaluation metric will not be a good indicator for real-world performance of a model when there is a real-world imbalance among classes.

5.2.3 Few-shot learning

Few-Shot Learning (Wang et al. 2020a) is also an efficient method for dealing with unbalanced data. Contrary to supervised learning, it does not require a high number of training examples. Instead of learning to generalize class identities from a training set to a test set, the Few-Shot Learning models learn to discriminate the similarities and differences between classes, by training a function that predicts similarity.

6 Systematic review of ERC works

In Table 3, a plethora of works on ERC, since 2017, are displayed along with the methods they use and their reported performance.

Table 3 Summary of publications on emotion recognition in conversations, reporting the methods used along with their performance across several datasets
Full size table

Several authors resorted to gated neural networks, which can capture dependencies between words and utterances. This effort led to high reported results on the benchmark datasets. Some authors presented works using graph neural networks and the performances obtained are comparable to the ones obtained with gated neural networks. The best performances, however, were yielded when using transformers or transformer-based models, that better capture dependencies in long utterances, combined with the aforementioned state-of-the-art deep-gated and graph-based network architectures for context modelling. This may reflect the promising role that transformers and pre-trained transformer-based language models play in Natural Language Processing and Emotion Recognition in Conversations and the suitability of recurrent gated and graph-based neural network architectures, as elaborated in previous sections.

6.1 ERC challenges revisited

Practically all works leveraged the information of preceding and sometimes also subsequent (Li et al. 2021a) utterances, resorting to gated or graph neural networks for context modelling of the utterances that were represented by embeddings from fine-tuned pre-trained language models. Another viable way to perform context modelling is to feed several appended utterances to the pre-trained language model (Pereira et al. 2023a). For speaker-specific modelling, also gated (Majumder et al. 2019), graph (Ghosal et al. 2019) neural networks and a combination of both (Shen et al. 2021b) architectures were used. Emotion dynamics modelling was less explored. One work considered a GRU to model the emotion dynamics of each party (Majumder et al. 2019). Some works considered a knowledge base (Zhong et al. 2019; Ghosal et al. 2020; Li et al. 2021a; Zha et al. 2024) to aid in interpreting the meaning behind commonsense knowledge. The majority of the works resorted to fine-tuning large pre-trained language models to obtain embeddings, which also aids in capturing the meaning behind commonsense knowledge expressions and informal language.

7 Directions for future research

We now present suggestions for future work directions.

The first set of directions concerns potentiating the applicability of ERC modules for real-life scenarios. It comprises real-time ERC, recognizing emotion causes, and multilingual ERC, all elaborated in Sect. 2. There are plenty of research opportunities regarding annotation and modelling efforts.

Further exploring mixed emotions with multi-label datasets and classifiers is also promising. Moreover, training the same model with different datasets by matching emotions from one dataset to another improves the generalization capabilities of the classifiers.

Some surveyed works resorted to knowledge bases. These are, however, not specific to emotions, which would be a useful extension.

Most datasets are unbalanced, motivating the need to leverage techniques and use appropriate performance metrics to address this unbalance, as elaborated in subsection 5.2.

Concerning subjectivity uncertainty in annotations, we highlight using inter-annotator (dis)agreement measures to weight or filter the annotator’s opinions, incorporating these measures as additional information to the classifiers, and considering soft label distributions per sample.

Finally, we encourage further tackling interpretability, as elaborated in Sect. 2, since most works focus on performance.

8 Conclusion

Research in Emotion Recognition in Conversations is advancing at a high pace and novel application scenarios pose new challenges and opportunities. Although research in Emotion Recognition in Conversations has come a long way, there is still much room for improvement, as it can be seen by the performance of the classifiers on the benchmark datasets and the several unexplored directions put forward in this section. While we described how current work has addressed several challenges, we also pointed out the opportunities that partly unaddressed challenges constitute.

As main contributions of this survey, we presented partly addressed challenges for ERC, such as recognizing emotion causes, dealing with different taxonomies across datasets, multilingual ERC, and interpretability with associated future work directions. We compiled an extensive list of Deep Learning works in ERC, being the first to simultaneously report their methods, modalities, and performance across various datasets. Finally, our descriptions of Deep Learning methods in the context of ERC provided insights into the suitability of each method for this task, which were not given in previous surveys.

The survey highlights the advantage of leveraging techniques to address unbalanced data, the exploration of mixed emotions, and the benefits of incorporating annotation subjectivity in the learning phase.

Our survey relies on established benchmark datasets. While this aids in comparing different works, it also constitutes a limitation since benchmarking on these datasets may not generalize well to domain-specific applications such as different industries or languages. We also just briefly mention some challenges of real-world applications, not elaborating on the scalability or performance of models in such settings. So our findings may be highly relevant to academic benchmarks but less relevant to domain-specific applications.

Finally, important ethical aspects pertaining to Emotion Recognition are now presented. These aspects are, for example, and not limited to, whether an Emotion Recognition module should be developed or used for a certain purpose, which data to collect and the subjects behind the data, diversity and inclusiveness, privacy and control, and possible biases and misuses of the application (Mohammad 2022). Research in these directions will benefit the community with better Emotion Recognition in Conversation modules for current and novel applications.