Scribd
Upload
7 views39 pages

07 Dlintro

An Intro to Deep Learning for NLP

Uploaded by

maheshsangamreddiias
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
7 views39 pages

07 Dlintro

An Intro to Deep Learning for NLP

Uploaded by

maheshsangamreddiias
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 39

An Intro to Deep Learning for NLP

Mausam
Disclaimer: this is an outsider’s understanding. Some details may be inaccurate

(several slides by Yonatan Belinkov Yoav Goldberg & Graham Neubig)


Back to the
1980s

[Figure:
Francesconi, 2022]
The Localist vs. Distributed Debate
Distributed vs. Localist
Representations
Localist: “..one computing element for each entity”

Distributed:
• “Each entity is represented by a pattern of activity
distributed over many computing elements”
• “each computing element is involved in representing
many different entitites”
Local Representation

[Thorpe 1989]
Distributed Representation

[Thorpe 1989]
Semi-Distributed Representation

[Thorpe 1989]
Distributed Representations: Pros
and Cons
Distributed
representations:
😀 😀
• Efficient 😀 Localistrepresentations
😀
• Continuous 😀 • Easier to work with(?)
😢
• Degrade gracefully • More interpretable
• Less interpretable

[Pate 2002]
So, who won?

I think we all know…


NLP before DL #1
Assumptions
- doc: bag/sequence/tree of words
- model: bag of features (linear)
- feature: symbolic (diff wt for each)

Features
Model
(NB, SVM, CRF)

Supervised Optimize function


Training (LL, sqd error, margin…)
Data
Learn feature weights
NLP before DL #2
Assumptions
z1 z2 … - doc/query/word is a vector of numbers
- dot product can compute similarity
- via distributional hypothesis

Model
(MF, LSA, IR)

Unsupervised Optimize function


Co-occurrence (LL, sqd error, margin…)
Data
Learn vectors
NLP with DL

Features
Model
(NB, SVM, CRF)

Supervised Optimize function


Training (LL, sqd error, margin…)
Data
Learn feature weights
NLP with DL

Neural Model
Features (NB, SVM, CRF)

Supervised Optimize function


Training (LL, sqd error, margin…)
Data
Learn feature weights
NLP with DL

z1 z2 …

Neural Model
Features (NB, SVM, CRF)

Supervised Optimize function


Training (LL, sqd error, margin…)
Data
Learn feature weights+vectors
NLP with DL

z1 z2 …

Neural Model
Features NN= (NB, SVM, CRF, +++
+ feature discovery)
Supervised Optimize function
Training (LL, sqd error, margin…)
Data
Learn feature weights+vectors
NLP with DL
Assumptions
- doc/query/word is a vector of numbers
z1 z2 … - doc: bag/sequence/tree of words
- feature: neural (weights are shared)
- model: bag/seq of features (non-linear)

Neural Model
Features NN= (NB, SVM, CRF, +++
+ feature discovery)
Supervised Optimize function
Training (LL, sqd error, margin…)
Data
Learn feature weights+vectors
Meta-thoughts
Features
• Learned
• in a task specific end2end way
• not limited by human creativity
Everything is a “Point”
• Word embedding
• Phrase embedding
• Sentence embedding
• Word embedding in context of sentence
• Etc

• Also known as dense/distributed representations

Points are good  reduce sparsity by wt sharing


a single (complex) model can handle all pts
Universal Representations
• Non-linearities
– Allow complex functions

• Put anything computable in the loss function


– Any additional insight about data/external knowledge
Make symbolic operations continuous
• Symbolic  continuous
– Yes/No 
• (number between 0 and 1)
– Good/bad 
• (number between -1 and 1)

– Either remember or forget 


• partially remember
– Select from n things 
• weighted avg over n things
Encoder-Decoder

Symbolic Symbolic
Input z1 Model Neural Model Output
(word) Features (class, sentence..)

Encoder Decoder

Different assumptions on data create different architectures


Building Blocks

+ ; .

Matrix-mult gate non-linearity


x;y
x+y

Can also try


Dimension-wise
Max
(later weighted sum)
Concat vs. Sum
• Concatenating feature vectors: the
"roles" of each vector is retained.

prev current next


word word word

• Different features can have vectors of different dim.

• Fixed number of features in each example


(need to feed into a fixed dim layer).
Concat vs. Sum
• Summing feature vectors: "bag of features"

word word word

• Different feature vectors should have same dim.

• Can encode a bag of arbitrary number of features.


x.y
• degree of closeness
• alignment

• Uses
– question aligns with answer //QA
– sentence aligns with sentence //paraphrase
– word aligns with (~important for) sentence //attention
g(Ax+b)
• 1-layer MLP
• Take x
– project it into a different space //relevant to task
– add some scalar bias (only increases/decreases it)
– convert into a required output

• 2-layer MLP
– Common way to convert input to output
Loss Functions

Cross Entropy
Binary Cross Entropy
Max Margin
Encoder-Decoder
LOSS
P(y) y*

Symbolic Symbolic
Input z1 Model Neural Model Output
(word) Features (class, sentence..)

Encoder Decoder
Common Loss Functions
Common Loss Functions
• Max Margin
Loss = max(0, 1-(score(y*)-score(ybest)))

• Ranking loss (max margin: x ranked over x’)


Loss = max(0, 1-(score(x)-score(x’)))
Regularization
• L1
• L2
• Elastic Net
• DropOut
• Batch Normalization
• Layer Normalization
• Problem-specific regularizations
• Early Stopping
• https://towardsdatascience.com/different-
normalization-layers-in-deep-learning-1a7214ff71d6
Some Practical Advice
Optimization
• Stochastic Gradient Descent
• Mini-Batch Gradient Descent
• AdaGrad
• AdaDelta
• RMSProp
• Adam

Learning rate schedules

https://ruder.io/optimizing-gradient-descent/
Glorot/Xavier Initialization (tanh)
• Initializing W matrix of dimensionality dinxdout

He’s Initialization (relu)


Batching
• Padding
Vanishing and Exploding Gradients
• Clipping

You might also like