MQP ANSWERS

Explain how the recurrent neural network (RNN) processes data sequences.

Recurrent Neural Networks (RNNs) are designed to process sequential data by retaining
information about previous inputs in their internal state. This allows RNNs to model dependencies
in sequences, making them ideal for tasks like time series forecasting, natural language
processing, and speech recognition.

Key Steps in Processing Data Sequences with RNNs

1.​ Input Sequence Processing:

○​ RNNs take a sequence of inputs X={x1,x2,…,xT}, where T is the sequence length.
○​ Each Xt represents the input at time step t, such as a word in a sentence or a value in
a time series.
2.​ Hidden State Update:
○​ At each time step t, the RNN maintains a hidden state ht​, which acts as memory and
captures information about the sequence seen so far.
○​ The hidden state is updated based on the current input xt and the previous hidden
state

○​
3.​ Output Generation:
○​ At each time step, the RNN can produce an output yt based on the hidden state:
4. Sequence Dependency:

○​ The hidden state ht serves as a connection between time steps, allowing the model to
capture dependencies across the sequence.

Discuss about Bidirectional RNNs.

In standard "causal" RNNs, the state at time t captures information from past inputs (x1,x2,…,xt−1​)
and the current input (xt). However, many applications require predictions that depend on the entire
input sequence, including future inputs. For example:

●​ Speech recognition: The interpretation of a current phoneme may depend on upcoming

phonemes or words.
●​ Handwriting recognition: Identifying a character might rely on neighboring characters in
the sequence.
To address this, Bidirectional RNNs (Schuster and Paliwal, 1997) were developed. They
process the sequence both forward (from the start to the end) and backward (from the end to the
start). This allows each output Ot​to depend on both past and future context, focusing primarily on
inputs near time t, without the need for fixed-sized windows or lookahead buffers.

Structure of Bidirectional RNNs

●​ Two RNNs: A bidirectional RNN combines:

1.​ A forward RNN (ht​) that processes the sequence from the start to the end.
2.​ A backward RNN (gt) that processes the sequence from the end to the start.
●​ The output at each time step Ot​is computed based on both ht​(forward state) and gt​
(backward state).
●​ Benefit: This structure enables the network to capture long-range dependencies in both
directions.

Applications

●​ Handwriting Recognition: Helps interpret ambiguous strokes by considering the full

sequence of strokes before and after the current one.
●​ Speech Recognition: Resolves ambiguities in phoneme or word interpretation by
considering preceding and succeeding sounds or words.
●​ Bioinformatics: Used for analyzing DNA sequences where dependencies can occur across
long ranges.

Extensions to 2D Data

●​ For 2D inputs like images, the bidirectional approach can be extended by having RNNs
operate in four directions: up, down, left, and right.
●​ At each pixel (i,j) the output Oi,j is influenced by neighboring pixels and potentially
long-range dependencies.
●​ Advantages over Convolutional Networks:
○​ While CNNs focus on local interactions through filters, bidirectional RNNs can capture
long-range dependencies across the image.
○​ Trade-off: RNNs for 2D data are computationally more expensive than CNNs.
Explain LSTM working principle along with equations.
Long Short-Term Memory (LSTM) is a type of recurrent neural network (RNN) designed to capture
long-term dependencies by managing information flow through gating mechanisms. LSTM cells
address the vanishing and exploding gradient problems in standard RNNs, making them effective
for tasks requiring long-term memory, such as speech and handwriting recognition.

\
Write a note on Speech Recognition and NLP.
Speech recognition aims to map spoken language (acoustic signals) into the corresponding
sequence of words. The process involves the following key points:

Evolution of Speech Recognition

Early Approaches:

●​ GMM-HMM Models (1980s–2000s):

○​ Gaussian Mixture Models (GMMs): Mapped acoustic features to phonemes.
○​ Hidden Markov Models (HMMs): Modeled phoneme sequences.
○​ These systems dominated for decades, but neural networks were explored in the
1990s for similar tasks.

Neural Network Adoption:

●​ Initial neural network-based systems matched GMM-HMM performance but lacked

significant adoption due to the complexity of existing systems.
●​ Early benchmarks (e.g., TIMIT dataset) showed neural networks achieving comparable
performance (26% phoneme error rate).

NLP :

1.​ Definition and Applications:

○​ Natural Language Processing (NLP) allows computers to understand and process
human languages like English or French.
○​ Key applications include machine translation, speech recognition, and text
generation.
2.​ Traditional Approaches - n-grams:
 ○​ n-grams model probabilities of sequences (e.g., unigrams for single words, bigrams
for pairs, trigrams for triples).
○​ Limitation: High sparsity and dimensionality in large vocabularies make it
computationally expensive and less generalizable.
○​ Smoothing techniques distribute probabilities across unseen combinations to handle
sparsity.
3.​ Neural Language Models (NLMs):
○​ NLMs use distributed representations (word embeddings) to overcome sparsity by
placing semantically similar words closer in a low-dimensional space.
○​ Example: Words like dog and cat are neighbors in the embedding space, enabling
generalization across similar contexts.
4.​ Efficient Vocabulary Handling:
○​ Hierarchical Softmax reduces computation by breaking the vocabulary into a tree
structure.
○​ Importance Sampling approximates probabilities by focusing on a subset of words,
reducing the cost of large softmax layers.
5.​ Hybrid Approaches:
○​ Combining n-grams (for quick lookups) and NLMs (for richer representations) offers
the benefits of both models.
○​ Ensemble models or maximum entropy combinations enhance capacity while
maintaining computational efficiency.
Teacher Forcing in Sequence-to-Sequence Models

Teacher forcing is a training strategy used in sequence-to-sequence (Seq2Seq) models,

particularly in tasks like machine translation, text generation, and speech recognition. It involves
feeding the actual target output (ground truth) from the training dataset as the next input to the
model during training, rather than using the model's own predicted output.

Why It Is Used:

1.​ Accelerates Training: By providing the correct output from the ground truth at each time
step, the model learns faster as it avoids compounding errors.
2.​ Prevents Error Accumulation: Using the model's own predictions can lead to cascading
errors when predictions deviate from the ground truth. Teacher forcing mitigates this issue
during training.
3.​ Stabilizes Learning: It ensures that the model stays on the correct path by aligning its
predictions with the ground truth sequence.
4.​ Improves Convergence: It often leads to faster convergence of the model compared to
training with predicted inputs.

Challenges with Teacher Forcing:

●​ Exposure Bias: During inference, the model uses its own predictions as inputs, which can
differ from the training process where it always uses ground truth inputs. This mismatch can
lead to poor performance when the model is deployed.
●​ Dependency on Ground Truth: The model might over-rely on the ground truth during
training and fail to generalize when ground truth inputs are unavailable during inference.

Deep Recurrent Networks

Recurrent Neural Networks (RNNs) traditionally involve three main components for computation:

1.​ Input-to-Hidden Transformation: Converts the input to a hidden state.

2.​ Hidden-to-Hidden Transformation: Passes information between hidden states over time.
3.​ Hidden-to-Output Transformation: Maps the hidden state to the output.

In standard RNNs, each of these transformations is shallow, meaning they involve a single layer of
computation (a learned affine transformation followed by a nonlinearity).

Introducing Depth in RNNs

 ●​ Why Add Depth?
○​ Adding depth to these components increases the model's representational power,
allowing it to capture more complex relationships in the data.
○​ Experimental studies (e.g., Graves, 2013; Pascanu, 2014) show that deep RNNs
perform better for tasks requiring complex mappings.

Different Ways to Make RNNs Deep

1.​ Hierarchical Hidden States:

○​ The hidden state is divided into multiple layers, with lower layers processing raw input
and higher layers refining it.
2.​ Deep Transformations:
○​ Use deep multi-layer perceptrons (MLPs) for the input-to-hidden, hidden-to-hidden,
and hidden-to-output computations.
○​ This adds depth to the RNN but increases the difficulty of optimization due to longer
dependency paths.
3.​ Skip Connections:
○​ Skip connections (bypassing some layers) help mitigate the path-lengthening issue
by shortening the flow of gradients during training, making optimization easier.

Trade-offs

●​ Advantages: Adding depth enhances the model's capacity to process complex data and
extract high-level features.
●​ Challenges: Deeper architectures make optimization harder, as gradients must propagate
through longer paths, increasing the risk of vanishing or exploding gradients.

Recursive Neural Networks (RecNNs)

Recursive Neural Networks (RecNNs) extend the concept of Recurrent Neural Networks (RNNs) by
organizing their computational graph as a tree structure instead of a chain. This allows them to
handle hierarchical data more effectively.

Key Features:

1.​ Tree-like Structure:

○​ Unlike RNNs, which process sequences in a linear chain, RecNNs map inputs into a
tree structure.
 ○​ This makes them suitable for data with hierarchical relationships, such as parse trees
in natural language or structured data in computer vision.
2.​ Efficient Depth:
○​ For a sequence of length τ, RecNNs reduce the depth of nonlinear compositions from
τ (in RNNs) to O(log⁡τ) This helps in capturing long-term dependencies more
efficiently.
3.​ Applications:
○​ Natural Language Processing (NLP): RecNNs are applied to parse trees of
sentences, where each node represents a word or phrase.
○​ Computer Vision: Useful for processing hierarchical features in images.
○​ Data Structures: Can model structured data like trees or graphs.
4.​ Tree Structure:
○​ The tree structure can either be:
■​ Fixed (e.g., a balanced binary tree or a parser-generated tree in NLP).
■​ Learned: Ideally, the model can infer the best tree structure based on the data.
5.​ Variants and Advanced Operations:
○​ Instead of traditional neuron computations (linear transformation + nonlinearity),
RecNNs can use:
■​ Tensor Operations or Bilinear Forms: Useful for modeling relationships
between entities represented as vectors.

Advantages:

●​ Handles hierarchical and structured data effectively.

●​ Reduces the depth of computations, making it more efficient for longer sequences.
●​ Flexible to adapt to different tree structures depending on the task.