Scribd
Upload
2 views

Module 3 Part 2 Encoder

The document provides an overview of the encoder-decoder architecture used in recurrent neural networks for machine translation, detailing the roles of the encoder and decoder in processing input sequences. It includes a forward pass example demonstrating how the encoder converts input into a latent vector and how the decoder generates the output sequence. Applications of this architecture include machine translation, speech recognition, text summarization, chatbots, and image captioning.

Uploaded by

chinmy anand
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
2 views

Module 3 Part 2 Encoder

The document provides an overview of the encoder-decoder architecture used in recurrent neural networks for machine translation, detailing the roles of the encoder and decoder in processing input sequences. It includes a forward pass example demonstrating how the encoder converts input into a latent vector and how the decoder generates the output sequence. Applications of this architecture include machine translation, speech recognition, text summarization, chatbots, and image captioning.

Uploaded by

chinmy anand
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 14

Contents

Introduction
Encoder and decoder architectures
Example Forward pass
Application

1
Reference:
10.4, Ian Goodfellow, Yoshua Bengio, Aaron
Courville, Deep Learning, MIT Press, 2016
Encoder-Decoder Architecture: Overview
https://www.youtube.com/watch?v=671xny8i
ows

2
Introduction
The encoder-decoder architecture for recurrent neural
networks is the standard neural machine translation
method.
There are three main blocks in the encoder-decoder
model,
1. Encoder: The Encoder will convert the input sequence
into a single-dimensional vector (Hidden/Latent
vector).
2. Hidden/Latent Vector
3. Decoder: The decoder will convert the hidden/Latent
vector into the output sequence.
Encoder-Decoder models are jointly trained to maximize
the conditional probabilities of the target sequence given
the input sequence.
3
RNN vs RNN based Encoder-Decoder
Simple RNN:
• Consists of a single RNN layer that maps inputs xt​
directly to outputs yt​ at each time step limiting its
ability to learn global dependencies over long
sequences.
Encoder-Decoder RNN:
• Composed of two RNNs: the encoder and the
decoder.
• Encoder: Processes the entire input sequence into a
fixed-length representation (final hidden/latent state)
captures global context in a latent representation.
• Decoder: Uses the encoder’s latent representation
to generate the output sequence.
4
Example (same from RNN)
Inputs: 𝑥=[0.5,0.6,0.7] ;
Targets: 𝑦=[0.6,0.7,0.8]
Initial hidden state: ℎ0=0
Weights:𝑊𝑥=0.8, 𝑊ℎ=0.6
Biases:𝑏x= 0.1

5
6
1. Encoder Forward Pass
 Time step 𝑡=1
Hidden state

ℎ1=tanh(0.8⋅0.5+0.6⋅0+0.1)=tanh(0.5)≈0.462
 Time step 𝑡=2
ℎ2=tanh(0.8⋅0.6+0.6⋅0.462+0.1)=tanh(0.857)≈0

 Time step 𝑡=3


.695

ℎ3=tanh(0.8⋅0.7+0.6⋅0.695+0.1)=tanh(1.077)≈0
.792

7
2. Decoder Forward Pass
The decoder generates the target sequence
using the hidden state from the encoder. The
initial hidden state of the decoder is

For each timestep 𝑡:

Weights: 𝑊ℎ=0.6, 𝑊𝑦=1.2


Biases: 𝑏h= 0.1, 𝑏𝑦=0.05

8
2. Decoder Forward Pass

Weights: 𝑊ℎ=0.6, 𝑊𝑦=1.2


Biases: 𝑏h= 0.1, 𝑏𝑦=0.05
Time step 𝑡=1:

9
2. Decoder Forward Pass

Weights: 𝑊ℎ=0.6, 𝑊𝑦=1.2


Biases: 𝑏h= 0.1, 𝑏𝑦=0.05
Time step 𝑡=1:

10
2. Decoder Forward Pass
Weights: 𝑊ℎ=0.6, 𝑊𝑦=1.2
Biases: 𝑏h= 0.1, 𝑏𝑦=0.05
Time step 𝑡=1:

Time step 𝑡=2:

Time step 𝑡=3:

11
Encoder-decoder or sequence-to-
sequence RNN

12
Applications
 Machine Translation
Translating "Bonjour" (French) to "Hello" (English).
 Speech Recognition
Converting spoken words in an audio file to text
 Text Summarization
Summarizing a 1,000-word article into a 100-word
abstract.
 Chatbots and Conversational Agents
Responding to "What's the weather today?" with
"It's sunny and 25°C.“
 Image Captioning
Generating the caption "A dog playing with a ball in
13 the park" for a given image.
14

You might also like