Lecture 5: Text Representation II

Distributed Representations
Pilsung Kang
School of Industrial Management Engineering
Korea University
AGENDA

01 Word-level: NNLM
02 Word-level: Word2Vec
03 Word-level: GloVe
04 Word-level: Fasttext
05 Sentence/Paragraph/Document-level
06 More Things to Embed?
 GloVe
Pennington et al. (2014)

• Limitations of Word2Vec
✓ The network spends so much time to train some overwhelmingly used words
▪ Ex: to learn a distribution for P(w|the)

Theatre or theater is a collaborative form of fine art that uses live performers to present the
experience of a real or imagined event before a live audience in a specific place. The performers may
communicate this experience to the audience through combinations of gesture, speech, song, music,
and dance. Elements of art and stagecraft are used to enhance the physicality, presence and
immediacy of the experience. The specific place of the performance is also named by the word
"theatre" as derived from the Ancient Greek (thatron, "a place for viewing"), itself from (theomai, "to
see", "to watch", "to observe"). Modern Western theatre comes from large measure from ancient
Greek drama, from which it borrows technical terminology, classification into genres, and many of its
themes, stock characters, and plot elements. Theatre artist Patrice Pavis denes theatricality, theatrical
language, stage writing, and the specificity of theatre as synonymous expressions that differentiate
theatre from the other performing arts, literature, and the arts in general. Theatre today, broadly
denfied, includes performances of plays and musicals, ballets, operas and various other forms.
 GloVe
Pennington et al. (2014)

• GloVe
✓ Based on matrix factorization method
✓ http://nlp.stanford.edu/projects/glove/
✓ Notations
 GloVe
Pennington et al. (2014)

• Motivation

✓ For words k related ice but not steam (solid), the ratio Pik/Pjk is large
✓ For words k related steam but not ice (gas) the ratio Pik/Pjk is small
✓ For words k that are either related to both ice and steam, or to neither, the ratio
should be close to 1
 GloVe
Pennington et al. (2014)

• Formulation
✓ Express the relationship among three words using a function F

✓ Relationship between wi and wj is expressed by subtraction

✓ Inner product is used to link with wi and wj

 GloVe
Pennington et al. (2014)

• Homomorphism

✓ Want to preserve using

 GloVe
Pennington et al. (2014)

• Homomorphism

inverse element of addition

inverse element of multiplication

✓ Homomorphism preserves an operation, which in turn preserves the inverse element

✓ Need a homomorphism from to
 GloVe
Pennington et al. (2014)

• Homomorphism
✓ Function F: homomorphism that maps to

✓ Finally, we can drive that

 GloVe
Pennington et al. (2014)

• Solution
✓ We know that and
 GloVe
Pennington et al. (2014)

• Objective Function
✓ A least squared objective function
 GloVe
Pennington et al. (2014)

• Objective Function
✓ A least squared objective function
 GloVe
Pennington et al. (2014)

• Results

http://nlp.stanford.edu/projects/glove/
AGENDA

01 Word-level: NNLM
02 Word-level: Word2Vec
03 Word-level: GloVe
04 Word-level: Fasttext
05 Sentence/Paragraph/Document-level
06 More Things to Embed?
 FastText
Bojanowski et al. (2016)

• Limitations of NNLM, Word2Vec, and GloVe

✓ Ignores the morphology or words by assigning a distinct vector to each word
✓ Difficult to apply to morphologically rich languages with large vocabularies and many
rare words (Turkish or Finnish)

• Goal
✓ Learn representations for character n-grams
✓ Represent words as the sum of n-gram vectors
 FastText
Bojanowski et al. (2016)

• Revisit Negative Sampling in Word2Vec

✓ Score is just a dot product between the two embeddings

• Subword model
✓ Define the set of n-grams appearing in w:

✓ Represent a word by the sum of the vector representations of its n-grams

 FastText
Bojanowski et al. (2016)

• Subword model
✓ n-gram representation
▪ Include the word w in the set of its n-grams
▪ Keep all the n-grams of size 3, 4, 5, and 6
▪ Different vectors are assigned to a word and a n-gram sharing the same sequence of
characters
Word2Vec FastText
parameter ... ... ... ... ... p ... ... ... ... ...
... ... ... ... ... ...
r ... ... ... ... ...
pa ... ... ... ... ...
... ... ... ... ... ...
er ... ... ... ... ...
par ... ... ... ... ...
... ... ... ... ... ...
ameter ... ... ... ... ...
Avg. ... ... ... ... ...
 Word Embedding Examples
• Word Embedding examples: English
✓ Word lists that are close to a given word after embedding

Collobert et al. (2011)

✓ Relationship pairs in a word embedding

Mikolov et al. (2013)

 Word Embedding Examples
• Word Embedding examples: English

https://www.tensorflow.org/versions/r0.7/tutorials/word2vec/index.html
 Word Embedding Examples
• Word Embedding examples: Korean
 Word Embedding Examples
• Word Embedding examples: Korean
 Word Embedding Examples
• Word Embedding with two different languages

• Word Embedding with Images