M. A.

M College of Engineering and Technology

Siruganur, Trichirappalli 621105
CCS364- SOFT COMPUTING
1. Name :Aadhithya.S, Aathikesavan. S, Ashok. R,

2. Course : BE(CSE) ‘A’,

3. Year&Sem : 3rd&5th ,

4. Reg no: 812022104001,812022104002,812022104011,

5. Regulation: 2021,

6. Title: Recurrent Neural Network in sequential data

7. Date of submission: 21/10/2024.

Date of submission: 20/10/2024.

Aadhithya.S, Aathikesavan. S, Ashok. R, Batch:5
Importance of sequential data
• Sequential data is a type of data where the order of the
elements matters. It is essential in various fields because many natural
processes and phenomena occur in sequences, where the context or state of
one element depends on those that came before. Here’s an explanation of its
importance:
1. Captures Temporal and Contextual
• Information:
Sequential data retains the time or order information, which is crucial
in many applications.

2. Enables Modeling of Dynamic Systems:

• Speech recognition: Audio signals change continuously, so the
order of sound waves must be preserved.
3. Reflects Real-World Processes:
• Biological data, like DNA sequences, contains valuable
information based on the order of nucleotides.
4. Facilitates Predictive Modeling:
• Predictive models often rely on sequential data to make accurate
forecasts.
• For instance: Weather forecasting uses past weather patterns to
predict future conditions.
INTRODUCTION:

• Recurrent Neural Network(RNN) is a type of Neural Network where the output from the previous step is fed
as input to the current step. In traditional neural networks, all the inputs and outputs are independent of
each other. Still, in cases when it is required to predict the next word of a sentence, the previous words are
required and hence there is a need to remember the previous words. Thus RNN came into existence, which
solved this issue with the help of a Hidden Layer. The main and most important feature of RNN is its Hidden
state, which remembers some information about a sequence. The state is also referred to as Memory State
since it remembers the previous input to the network. The main and most important feature of RNN is its
Hidden state, which remembers some information about a sequence. The state is also referred to as Memory
State since it remembers the previous input to the network.

Date of submission: 20/10/2024.

Structure of RNN
• The structure of a Recurrent Neural Network (RNN) is designed to handle sequential data and maintain
information across time steps. Here’s a breakdown of the basic structure:
Input Layer:
• The RNN takes a sequence of inputs, x(1),x(2),…,x(T)x^{(1)}, x^{(2)}, \ldots, x^{(T)}x(1),x(2),…,x(T), where each
x(t)x^{(t)}x(t) represents the input at time step ttt.
• The input can be one-dimensional (e.g., word embeddings in natural language processing) or multi-
dimensional (e.g., features from time series data).
Hidden Layer:
• The RNN has a hidden state h(t)h^{(t)}h(t) that captures information about the sequence up to time step ttt.
The hidden state is updated at each time step using the current input and the previous hidden state.
• The update is given by: h(t)=f(Wxh⋅x(t)+Whh⋅h(t−1)+bh)h^{(t)} = f(W_{xh} \cdot x^{(t)} + W_{hh} \cdot h^{(t-
1)} + b_h)h(t)=f(Wxh​x(t)+Whh​h(t−1)+bh​
⋅ ⋅ ) where:
fff is a non-linear activation function (commonly tanh or ReLU).
WxhW_{xh}Wxh​and WhhW_{hh}Whh​are weight matrices for the input and hidden state, respectively.
bhb_hbh​is the bias term.

Date of submission: 20/10/2024.

Output Layer:
• The output o(t)o^{(t)}o(t) at each time step can be computed based on the hidden state: o(t)=g(Who⋅h(t)
+bo)o^{(t)} = g(W_{ho} \cdot h^{(t)} + b_o)o(t)=g(Who​⋅h(t)+bo​) where:
ggg is an activation function (e.g., softmax for classification tasks).
WhoW_{ho}Who​is the weight matrix connecting the hidden state to the output.
bob_obo​is the bias term for the output layer.
Mathematical Representation of RNNs

• The hidden state update equation is given by:

h(t)=f(Wxh⋅x(t)+Whh⋅h(t−1)+bh)h^{(t)} = f(W_{xh} \cdot x^{(t)} + W_{hh} \cdot h^{(t-1)} +
b_h)h(t)=f(Wxh​⋅x(t)+Whh​⋅h(t−1)+bh​)
• fff is an activation function, typically tanh or ReLU, which helps introduce non-linearity.
• WxhW_{xh}Wxh​ represents the weights connecting the input to the hidden state, while WhhW_{hh}Whh​
connects the previous hidden state to the current hidden state.
• bhb_hbh​is the bias term that adjusts the learning process.
• The output at each time step is computed as:
o(t)=g(Who⋅h(t)+bo)o^{(t)} = g(W_{ho} \cdot h^{(t)} + b_o)o(t)=g(Who​⋅h(t)+bo​)
• ggg can be a softmax function (for classification) or a linear function (for regression tasks).
• WhoW_{ho}Who​is the weight matrix connecting the hidden state to the output layer, and bob_obo​is the
output bias term.
Recurrent Connections in RNNs
•Defining characteristic of Recurrent Neural Networkshe (RNNs) is their ability to connect the hidden
states across time steps. This means that the output at each time step depends not only on the current
input but also on the previous hidden state.

•This recurrent nature forms a feedback loop, allowing information to persist and be passed down
through the network. Essentially, the RNN “remembers” the sequence, making it ideal for tasks requiring
context over time.

•By updating the hidden state recursively, RNNs build a form of memory or context that reflects the
information from previous inputs. This is particularly crucial in tasks like language modeling, where
understanding the context of previous words is necessary to predict the next word accurately.

•In natural language processing (NLP), RNNs use recurrent connections to keep track of the context and
generate coherent sentences or predict the next word based on the entire sequence of previous words.
Unfolding Through Time
•RNNs can be visualized by “unfolding” them over time steps. In this representation, each
time step corresponds to a copy of the network that processes an element of the
sequence.

•This visualization helps in understanding how information and gradients flow during
training.

•Backpropagation Through Time (BPTT) is the method used to train RNNs. It extends
traditional backpropagation by computing gradients over the unfolded time steps,
adjusting weights based on errors calculated from outputs across the entire sequence.
Future of RNNs
1. Advancements in RNN Variants
2. Enhanced LSTM and GRU: Future developments may focus on improving existing RNN variants like Long Short-Term
Memory (LSTM) and Gated Recurrent Units (GRU) to enhance their ability to handle long-term dependencies and
reduce training complexities.
• Attention Mechanisms:Incorporating attention mechanisms into RNNs helps the network focus on relevant
parts of input sequences, improving performance in tasks like language translation and text generation.
2. Integration with Hybrid Models
• Combination with Convolutional Neural Networks (CNNs): RNNs combined with CNNs can efficiently handle
video data and spatiotemporal patterns, enhancing their use in tasks like video analysis and activity recognition.
3. Shift Towards Transformer Model
• RNNs vs. Transformers: With the rise of Transformers and architectures like BERT and GPT, RNNs face
competition, as Transformers often outperform RNNs in handling long-range dependencies and processing
sequences in parallel.
4. Applications in Emerging Field
• Healthcare: RNNs have a future in predictive healthcare, analyzing patient data sequences (e.g., heart rate,
medical history) to predict diseases and outcomes.
RNN VS FEED FORWARD
NETWORK RNN FEED FORWARD
Data Handling: Data handling:
• Designed to handle sequential data where the • Works with independent and static input data,
order of input matters (e.g., time series, speech, meaning the input values are not inherently
text). ordered.
• Has an internal state (hidden state) that • Information flows in one direction, from the input
remembers information from previous time steps, layer through hidden layers to the output layer,
allowing it to retain context across the sequence. with no feedback loops or memory.
Architecture: Architechture:
• Contains recurrent connections where the output • Consists of layered nodes without loops. Each node
of one time step is fed back into the network as passes information to the next layer without
input for the next time step. feedback, and the connections are unidirectional.
Memory Capability: Memory Capability :
• Has the ability to store and recall previous inputs • Lacks memory, so it cannot capture temporal
through its hidden state, making it ideal for tasks dependencies.
that require understanding of previous context.
Benefits and Applications of RNNs
• RNNs are widely used in various domains, text. including:
• Language Modeling and Text Generation: Predicting the next word in a sentence or generating
coherent
• Speech Recognition: Mapping audio signals to text sequences.
• Time Series Analysis: Forecasting future values based on historical data patterns.
• Benefits:Capable of processing inputs of arbitrary lengths.
• Ability to learn dependencies and retain information over time, crucial for understanding context and trends.
Challenges with Basic RNNs

• Vanishing and Exploding Gradients: During training, gradients can become very small (vanishing) or very
large (exploding), making it hard for the network to learn long-term dependencies.
• Short-term Memory Limitations: Basic RNNs may struggle to retain information over long sequences,
making them less effective for tasks where context from much earlier in the sequence is important.
• Advanced Architectures: To address these issues, architectures like Long Short-Term Memory (LSTM) and
Gated Recurrent Unit (GRU) were developed. These models include gating mechanisms to manage the flow
of information, enabling better performance on long sequences.
Conclusion
• Recurrent Neural Networks (RNNs) are powerful tools for processing and analyzing sequential data due to their
recurrent connections, which allow them to maintain context and learn temporal dependencies. Their
structure, characterized by weight sharing and feedback loops, makes them versatile for tasks such as language
modeling, speech recognition, and time series forecasting. However, basic RNNs face challenges like vanishing
and exploding gradients, which can hinder their ability to capture long-term dependencies. To overcome these
limitations, advanced architectures like Long Short-Term Memory (LSTM) and Gated Recurrent Unit (GRU) have
been developed, enhancing the model's ability to retain important information over long sequences. Overall,
RNNs remain a fundamental component of deep learning, enabling the modeling of dynamic, time-based data
across various applications.
References
• https://www.google.com/search?client=firefox-b-
d&sca_esv=3145f941300c1a56&sxsrf=ADLYWIL9oxV_uEOTLJwPVaGGfkKdtm65KQ:1728996352243&q=unfoldin
g+through+time+in+rnn&udm=2&fbs=AEQNm0Aa4sjWe7Rqy32pFwRj0UkWd8nbOJfsBGGB5IQQO6L3J603JUkR
9Y5suk8yuy50qOa0K08TrPholP8ECM8ELoq5GeRrUvU44UjKtPgUX-
2DV1UQVKIioKq9YP8hjr2s4XGUs7BYUWgrA1zGzjnSuLz0Rv9SOxJBYa2HuYoyuz0gUJ8I_0DE-
GtDv_SDOIZzgEUF8lIMmGKJCeFzaPcqEnsoKlWNMQ&sa=X&ved=2ahUKEwj3upKrtZCJAxW5TWwGHRehFNUQtK
gLegQIExAB&biw=1280&bih=649&dpr=1.25#vhid=9_-otUsCKAU9TM&vssid=mosaic