Natural Language Processing

AC3110E

1
 Chapter 5: Syntactic and Semantic
Representations

Lecturer: PhD. DO Thi Ngoc Diep

SCHOOL OF ELECTRICAL AND ELECTRONIC ENGINEERING
HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY
Linguistic structure

• Linguistic structures were an intermediate step toward deeper language

processing in the early history of NLP.
• In modern NLP, don’t generally make explicit use inside the neural language models
• Linguistic structure plays a number of new roles
• practical tool for social scientific studies of text
• practical tool to study about language itself
• provide a useful interpretive lens on neural networks
• Tasks in computational linguistics
• Syntactic parsing
• Semantic parsing

Reference:
+ Jurafsky, Daniel, and James H. Martin. Speech and Language Processing: An Introduction to Natural Language Processing,
Computational Linguistics, and Speech Recognition
+ CS224N: Natural Language Processing with Deep Learning, Stanford / Winter 2023

3
Syntactic Representations

4
Syntactic Representations

• Deal with the grammatical structure and relation of words and phrases
within sentences
• Part-of-speech tagging:
• identify the part-of-speech category for each word in
a sentence based on its grammatical role and context
POS tagging

• [Syntactic] parsing:
• Assigns a syntactic structure to a sentence or identifies the syntactic relationship of
words within a sentence, given the grammar rules of a language

Dependency grammar parsing

Constituent grammar parsing
img src: Deep Learning for NLP
5
Syntactic Representations

• Syntactic parsing is to understand deep structure in text

• Analyze whether a given sentence is in accordance with grammar rules or not
• Applications : grammar checking, intermediate stage of representation for the formal
semantic analysis
• Shallow parsing (Chunking)
• Parsing a limited part of the syntactic information from the given text
• Applications in information extraction, text mining, etc.
• Deep parsing
• For more complex NLP applications
• Applications in dialogue systems and summarization, etc.
• Constituency Parsing and Dependency Parsing
• Part-of-speech tags, chunking, dependency parsing serve well as features for
downstream natural language processing tasks.

6
1. Word Classes – POS tag set

• Word classes: define semantic tendencies

• Parts of speech: tags defined based on their grammatical relationship with
neighboring words
• Also called: parts-of-speech, lexical category, word classes, morphological classes,
lexical tags, POS
• Closed class:
• relatively fixed group of words
• generally function words: short common words which play a role in grammar
• Open class: new created or borrowed words can be added

7
1. Word Classes – POS tag set

• Open classes
• Nouns: for people, places, or things
• proper nouns, common nouns,
• can occur with determiners, and may occur in the plural
• count nouns, mass nouns
• Verbs
• refer to actions and processes
• verbs can have inflections (eat – eats – eating – eaten)
• Adjectives
• describe properties or qualities of nouns
• Adverbs
• generally modify something (often verbs, but also other adverbs and entire verb phrases)
• manner adverbs (slowly), locative adverbs (here), temporal adverbs (yesterday), frequency
(usually, rarely), degree (extremely, very)
• Interjections
• exclamation, greeting, yes/no response, etc. (oh, um, yes, hello)

inside (): Just some examples 8

1. Word Classes – POS tag set

• Closed classes
• Preposition/Postposition: marks a noun’s spacial, temporal, or other relation (in, on,
by, under)
• Auxiliary: helping verb marking tense, aspect, mood, etc. (can, may, should, are)
• Coordinating Conjunction: joins two phrases/clauses (and, or, but)
• Determiner: marks noun phrase properties (a, an, the, this)
• Numeral (one, two, first, second)
• Particle: a preposition-like form used together with a verb (up, down, on, off, in, out,
at, by)
• Pronoun: a shorthand for referring to an entity or event (she, who, I, others)
• Possessive pronouns (my, your, his, her)
• Wh-pronouns (what, who, whom)
• Subordinating Conjunction: joins a main clause with a subordinate clause such as a
sentential complement (that, which, but)
• Other
• Punctuation (˙, , () )
• Symbols ($,%) or emoji

inside (): Just some examples 9

1. Word Classes – POS tag set

• Penn Treebank tagset1

• Universal Dependencies
tagset2

1Marcus et al., 1993

2Nivre et al., 2016a 10
2. Grammar

• Is a formal representation of structures accepted in a language

• G = ( T, N, S, R )
• T (terminal)
• N (non terminal)
• S (start)
• R (rule): R = { }; : production rule

S → NP VP Productions:
NP → John, garbage John laughed. John walks.
VP → laughed, walks Garbage laughed. Garbage walks.
etc.

11
2. Grammar

• Context-free grammar (CFG)

• The simplest grammar concepts
• Contains a set of rules and a set of tokens.
• also called phrase-structure grammars
• R={ };
• CFG rules
• Phrase structure rules
• Sentence structure rules
• Declarative structure
• Imperative structure
• Yes-No structure
• Wh-question structure
• etc.

12
2. Grammar

• CFG can be extracted from Tree bank or User-defined

S -> NP VP # S : entire sentence
VP -> V NP # VP : verb phrase
V -> "eats" | "drinks" # V is verb
NP -> Det N # NP : noun phrase
Det -> "a" | "an" | "the" # Det : determiner
N -> "president" |"Obama" |"apple"| "coke" # N some example nouns

• Formal language: set of sequences generated by a CFG

“President eats apple” => need a deeper understanding of
“Obama drinks coke” semantics structure of the sentence
And also
“Apple eats coke”
“President drinks Obama”

• Sentences can be grammatical or ungrammatical to a certain CFG

13
3. Constituency Parsing

• “I shot an elephant in my pajamas” Structural ambiguity!

(S (S
(NP I) (NP I)
(VP (VP
(VP (V shot) (NP (Det an) (N (V shot)
elephant))) (NP (Det an) (N elephant) (PP (P in)
(PP (P in) (NP (Det my) (N (NP (Det my) (N pajamas))))))
pajamas)))))

Graphical representation of the sentence structure tree

https://www.nltk.org/book 14
Constituency Parsing Approaches

• Rule-based approach
• a top-down approach
• based on rules/grammar
• grammatical rules are coded manually in CFG (context-free grammar)
• Regex-based parser, CKY (Cocke-Kasami-Younger) Parser, Span-Based Neural
Constituency Parsing, etc.
• Probabilistic approach
• a bottom-up approach
• learn rules/grammar by using probabilistic models
• uses PCFG (Probabilistic context-free grammar), in which each rule is associated with
a probability

15
3.1. Rule-based approach

• Syntactic parser is a software that accepts an input text and constructs a

parse tree (syntax tree) based on a set of grammatical rules (grammar)
• Searches through a variety of syntax trees and finds an optimal tree for the given
sentence
• Regex parser reg_parser = RegexpParser(
NP: {<DT>? <JJ>* <NN>*} # NP
P: {<IN>} # Preposition
V: {<V.*>} # Verb
PP: {<P> <NP>}
VP: {<V> <NP|PP>*}
)

=> (S
sent="Mr. Obama played a big
Mr./NNP
role in the Health insurance
Obama/NNP
bill"
(VP
(V played/VBD)
(NP a/DT big/JJ role/NN)
(PP (P in/IN) (NP the/DT)))
Health/NNP
(NP insurance/NN bill/NN))

16
3.2 CKY (Cocke-Kasami-Younger) Parser

• Chart parsing method

• Dynamic Programming Approach
• Require grammars to be in Chomsky Normal Form (CNF)
• CKY Recognition

17
3.2 CKY (Cocke-Kasami-Younger) Parser

• CKY Recognition

S VP/X2 S/VP/X2
Nominal S/VP
Verb
S
Det NP NP

Nominal Nominal

Prep PP

NP
Proper-Noun

18
3.2 CKY (Cocke-Kasami-Younger) Parser

• CKY Parsing
“Book the flight through Houston”
S VP/X2 S/VP/X2 Parsing 1:
Nominal S/VP
Verb (S (Verb book) (NP (Det the) (NN (NN
S flight) (PP (Prep through) (NP (NNP
Det NP NP Houston))))))
Parsing 2:
(S (VP (Verb book) (NP (DT the) (NN
Nominal Nominal flight))) (PP (IN through) (NP (NNP
Houston))))
Parsing 3:
Prep PP (S (X2 (Verb book) (NP (DT the) (NN
flight))) (PP (IN through) (NP (NNP
Houston))))

NP
Proper-Noun => ambiguiation problem

19
3.3 Statistical Constituency Parsing

• As the number of rules increases, the possibility of ambiguity increases

• PCFG (Probabilistic context-free grammar) or Stochastic Context-Free Grammar
(SCFG)
• Probabilities are attached to rules present in CFG.
S -> NP VP [1.0]
NP -> Det N [0.5] | NP PP [0.25] | 'John' [0.1] | 'I' [0.15]
Det -> 'the' [0.8] | 'my' [0.2]
N -> 'man' [0.5] | 'telescope' [0.5]
VP -> VP PP [0.1] | V NP [0.7] | V [0.2]
V -> 'ate' [0.35] | 'saw' [0.65]
PP -> P NP [1.0]
P -> 'with' [0.61] | 'under' [0.39]

• Learn rules/grammar by using probabilistic models

• Given a treebank, compute the probability of each expansion of a non-terminal by counting the
number of times that expansion occurs and then normalizing.
• Without treebank: inside-outside algorithm Baker (1979); Lari and Young (1990), Manning and
Schutze (1999), etc.

20
3.3. Statistical Constituency Parsing

• Probability of a Tree:
• Each rule i in the tree is expressed as LHSi → RHSi

P(T)
= .05 .20 .20 .20 .75 .30 .60 .10 .40
= 2.2×10−6

P(T)
= .05 .10 .20 .15 .75 .75 .30 .60 .10 .40
= 6.1×10−7

21
3.4. Evaluating Constituency Parsers

• PARSEVAL metric
• Measures how much the constituents in the hypothesis parse tree look like the
constituents in a reference parse tree (hand-labeled)
• A constituent in a hypothesis Ch is labeled correct if there is a constituent in the
reference parse Cr with the same starting point, ending point, and non-terminal
symbol.

• Cross-bracket metric:
• The number of constituents for which the reference parse has a bracketing such as
((A B) C) but the hypothesis parse has a bracketing such as (A (B C))

22
4. Dependency parsing

• Do not analyze S, NP, VP, etc.

• Instead, analyze the dependency relationship between words in the
sentence
• Each linguistic unit (words) is connected with each other by a directed link:
dependency link.
Relationship between words -
phrases and between phrases Relationship between words

Constituency Parsing Dependency parsing

23
4. Dependency parsing

• Dependency Relations
• head and its dependent
• label: grammatical functions of dependent
• Universal Dependencies (UD) project
• The largest open community project for building dependency trees
• more than 100 languages, ~ 200 dependency treebanks
• 37 dependency relations
https://universaldependencies.org/

24
4. Dependency parsing

• A good proxy for the semantic relationship between predicates

• Ability to deal with languages that have a relatively free word order
• Currently more common than constituency grammars in natural language
processing
• Families of parsing algorithms
• Transition-Based Dependency Parsing
• Graph-based Dependency Parsing
• new approaches
• End to end learning
• Joint learning
• etc.

25
4.1 Transition-Based Dependency Parsing

• Stack: to build the parse

• Buffer: tokens to be parsed
• Oracle: a predictor
• At each time point:
• The parser walks through the sentence left-to-right, successively shifting items from
the buffer onto the stack
• The parser examines the top two elements of the stack
• The oracle makes a decision about what transition to apply to build the parse
• Insert and delete the transition and word from stack and buffer
• Create Oracle: rule-based classifier, feature-based classifier, neural classifier

Nivre algorithm 26
4.2. Graph-Based Dependency Parsing

• Parser searches through the space of possible trees for a given sentence S
for a tree (or trees) t that maximize a score.
arg max =arg max e t
∈𝒯 ∈𝒯
e: edges of the tree
• Parsing via finding the maximum spanning tree:
• Encode the search space as directed graphs and employ graph theory methods to
search the optimal solutions.
• Score calculation:
• Feature-based:
• Neural algorithm

27
4.3 Evaluation

• Exact match metric:

• How many sentences are parsed correctly
• Labeled and unlabeled attachment accuracy:
• Labeled attachment score (LAS): proper assignment of a word to its head along with
the correct dependency relation.
• Unlabeled attachment score (UAS): the correctness of the assigned head, ignoring the
dependency relation.
• Label accuracy score (LS):
• Percentage of tokens with correct labels, ignoring where the relations are coming
from

28
Treebank

• Treebank : A corpus in which every sentence is annotated with a parse tree

• Treebank has already been annotated with part-of-speech tags
• Can be enhanced with semantic or other linguistic information
• Created completely manually or semi-automatically
• Important in parsing and in linguistic investigations of syntactic phenomena
• Penn Treebank project: includes various treebanks in English, Arabic, and
Chinese
• TreeBank Viewer example

29
Common Syntactic Parsers

• CFG (context free grammar):

• Berkeley : http://nlp.cs.berkeley.edu/software.shtml
• Charniak: http://bllip.cs.brown.edu/resources.shtml
• HPSG (Head-driven Phrase Structure Grammar)
• Enju, deepNLP: https://mynlp.github.io/enju/
• Depedency grammar
• Google SyntaxNet: open-source:
https://research.googleblog.com/2016/05/announcing-syntaxnet-worlds-most.html
• Stanford : https://nlp.stanford.edu/software/lex-parser.shtml

30
Stanford parser

• Stanza: Stanford NLP Group's official Python NLP library

('the', 3, 'det')
('big', 3, 'amod')
('dog', 4, 'nsubj')
('chased', 0, 'root')
('the', 6, 'det')
('cat', 4, 'obj')

31
Application of Syntactic parsing

• Machine translation
(Alshawi 1996, Wu 1997, ...)

• Grammar check (Microsoft)

• Information extraction (Hobbs 1996)

• v.v.

32
Semantics Representations

33
Semantic Analysis

• The process of deducing the meaning of words, phrases, and sentences

within a given context.
• to understand the relationships between words and expressions, draw inferences
from textual data based on the available knowledge.
• Vast complexity and subjectivity in human language => complicated task for
machines.
• Help in:
• Identify the most relevant elements in text
• Determine the primary topic being discussed
• Understand the relationships between different concepts in the text.
• etc.

34
Semantic Analysis

• Applications: extract useful information to facilitate the processes of

• Sentiment analysis in blog post, comments in a forum
• Intent classification for documents, group chat applications
• Question – Answering
• Conversational chatbots
• Automated ticketing support
• Search engine results
• Translation
• etc.

35
Tasks of Semantic Analysis

• Lexical Semantic Analysis: understanding the meaning of each word of the

text individually
• Word senses: Different meanings assigned to a word based on the context
• Hyponyms, Polysemy, Synonyms, Antonyms, Homonyms, etc.

• Syntax Parsing and Semantic analysis

• From syntactic structure, analyze the relationships between words and their
hierarchical roles within the sentence.
• Create and assign meaning representations to linguistic inputs
• Shallow level: label semantic roles (case-roles)
• Deeper level: represents predicate logic form or other form for automatic inference

• Computational semantics: process of designing meaning representations

and associated semantic parsers
• Semantic interpretation: mapping a proper meaning of the words in context.

36
Tasks of Semantic Analysis

• Sub-Tasks involved in Semantic Analysis

• Named Entity Recognition
• identify and label words or phrases in text that refer to a person, location, organization, date, time,
or quantity.
• Word Sense Disambiguation
• identifying the meaning of a word based on its usage and context
• Relationship Extraction
• identifying various entities present in the sentence and extracting the relationships between them
• etc.

The phrase illustrates two different relationships.

‘Elon Musk is one of the co-founders of

Tesla, which is based in Austin, Texas.’

37
Meaning representations

• Meaning representations: linguistic expressions captured in formal

structures
• Basic Units
• Entity: refers to a particular unit or individual in specific such as a person or a
location.
• Concept: a generalization of entities, refers to a broad class of individual units.
• Relations: relationships between various entities and concepts.
• Predicate: Predicates represent the verb structures of the sentences.
• Approaches:
• First-order predicate logic (FOPL)
• Frames
• Conceptual Graphs
• Semantic Nets
• ...

38
Meaning representations

Sentence: “I have a car”

• First-Order Logic representation

• Abstract Meaning Representation (AMR)

• Directed graph form • Textual form

• Frame-based (slot-filler) representation

• Modern Description Logics

• Web Ontology Language
• etc.

39
Meaning representations

• Taking in linguistic input and assigning a meaning representation to it

• Use more or less (or no) syntax
• The compositional rule-to-rule approach
• The meaning of a whole is derived from the meanings of the parts (constituents of
the syntactic parse of the input)
• Attach semantic formation rules to syntactic CFG rules

“AyCaramba serves meat”

40
Semantic Analysis Methods

• A range of techniques and methods designed to extract meaning from text

data
• Rule-Based Approaches
• using predefined linguistic rules and patterns to extract meaning from text.
• Examples :
• Named Entity Recognition (NER)
• Part-of-Speech Tagging
• Syntax Trees: Parsing text to create hierarchical structures representing syntactic and
semantic relationships within sentences.
• Statistical Methods
• using large text corpora to identify patterns and relationships between words.
• Word Embedding
• Distributional Semantics: Latent Semantic Analysis (LSA) and Latent Dirichlet
Allocation (LDA), etc.
• Neural Networks and Deep Learning

41
Semantic Analysis Methods

• Semantic Features Analysis (SFA)

• By identifying the shared features across multiple words, SFA helps determine the
significance and weight of individual factors within a text.
• Common features: Part of speech, Semantic category, Morphological features, Sense
• Common feature weighting: TF-IDF, Normalized term frequency, Global term
weighting, etc.
• Similarity Measurement  Relationships between words, phrases, and concepts:
Cosine similarity, Jaccard similarity, Euclidean distance
• Latent Semantic Analysis (LSA)
• Uses statistical methods to identify latent concepts within the text, reducing
dimensionality and enabling semantic similarity comparisons.
• Creating a term-document matrix
• Make Matrix Decomposition and Dimensionality Reduction (Singular value
decomposition (SVD) technique)
• Represent semantic space and measure similarity

42
Semantic Analysis Methods

• Semantic Content Analysis (SCA)

• understanding and representing the overall meaning of a text by identifying
relationships between words and phrases
• Dependency parsing determines the grammatical relationships between words
• Identifying the thematic roles and case roles of words in a sentence helps reveal the
relationships between actions, participants, and objects
• Thematic roles: Agent, Patient, Theme, Goal, Source
• Semantic Frame Identification
• offer structured representations of events or situations, capturing the meaning within a text
• consist of frame elements and fillers that represent events, situations, or ideas
• refines the understanding of the relationships between words and context.
• Neural Networks and Deep Learning
• Using neural networks to learn semantic representations directly from data
• Transformer-based models: such as BERT, GPT, and T5 (Text-to-Text Transfer
Transformer), have achieved state-of-the-art results in various NLP tasks

43
Conducting Semantic Analysis

1. Preprocessing: removing special characters, stop words, reducing words to their

base forms, Tokenization.
2. Lexical Semantics Analysis: Exploring the context of words and phrases, and
determining their part of speech and semantic relationships, including synonyms,
antonyms, and hyponyms.
3. Parsing and Syntax Analysis: Analyze the grammatical structure of the sentences to
identify relationships between words, as well as their hierarchical roles within the
sentence.
4. Semantic Frame Identification: identifying semantic frames, understanding
relationships between concepts and phrases.
5. Vector Space Models and Word Embeddings: comparisons and processing of word
meanings based on vector representations.
6. Define the Analysis Method: choose the most suitable semantic analysis method:
Semantic Feature Analysis, Latent Semantic Analysis, Semantic Content Analysis,
etc.
7. Perform the Analysis: Apply the chosen semantic analysis method to the text data,
extracting meaning and relationships between words and phrases.
8. Evaluate Results: Measure the quality and effectiveness of the semantic analysis by
comparing the output against predefined benchmarks or datasets.
9. Iterate and Improve: refine and fine-tune the analysis techniques and parameters
to further improve the semantic analysis's effectiveness.
44
Challenges in Semantic Analysis

• Ambiguity and Polysemy

• Cultural and Contextual Differences
• Handling Rare or Unseen Words
• Lack of Common Sense Reasoning
• Scalability
• Multimodal and Cross-lingual Semantics
• etc.

45
Corpus

• PropBank (Proposition Bank)

• a corpus of Penn Treebank sentences fully annotated with semantic roles
• roles are specific to individual verbs,
• FrameNet
• corpus of sentences annotated with semantic roles.
• roles are specific to semantic frames.
• provides a rich set of contexts for the roles within the frame.
• contains over 1200 semantic frames, 13,000 lexical units, and 202,000 example
sentences.

46
• end of Chapter 5.

47