NLP Unit 3 Notes

What is Context Free Grammar? What is the significance of

context free grammar in natural language processing?
A Context-Free Grammar (CFG) is an essential concept in Natural Language Processing
and formal language theory, often used to describe the syntax of languages.

1. Definition of Context-Free Grammar

A Context-Free Grammar is a set of rules that describe how strings in a language are
formed. It is defined as a 4-tuple. G = (V, T, P, S):

● V: A finite set of non-terminal symbols (e.g., S, NP, VP in language parsing).

● T: A finite set of terminal symbols (e.g., actual words or tokens).
● P: A set of production rules in the form A → α, where A is a non-terminal and α is a
string of terminals and/or non-terminals.
■ Single Non-Terminal on the LHS: Every rule in CFG has exactly one
non-terminal on the left.
■ Flexibility on RHS: The right side can contain terminals (actual words
or symbols) and non-terminals, allowing for recursive and complex
language structures.
● S: The start symbol (usually a special non-terminal from which derivations begin).

2. Example of a CFG

Consider a CFG for a simple language that constructs basic sentences:

● V = {S, NP, VP, V, N}

● T = {eats, fish, cat}
● P:
○ S → NP VP
○ NP → N
○ VP → V NP
○ N → "cat" | "fish"
○ V → "eats"
● S=S

With this grammar, you can form sentences like "cat eats fish"

3. Properties of CFG

● Derivations: CFGs allow derivations that generate valid strings for the language. The
derivation can be shown through parse trees or derivation sequences.
 ● Parse Trees: These represent the hierarchical structure of sentences, showing how
CFG rules are applied. Parse trees are crucial for syntactic analysis.

4. Applications in NLP

● Syntax Analysis: CFGs are used in parsing algorithms to determine if a sentence

belongs to a language.
● Natural Language Parsing: For tasks like syntactic parsing in language models,
CFGs are foundational.
● Programming Languages: CFGs describe the syntactic structure of many
programming languages.

5. Advantages and Limitations

● Advantages: They provide a formal way to describe languages and are widely used
in compilers and parsers.
● Limitations: CFGs cannot handle all natural language constructs, especially context-
sensitive ones (e.g., agreement in number and gender).

Sample Parse Tree

G = { V, T, P, S }

V = {S, NP, VP, V, N}

T = {eats, fish, cat}

P= S → NP VP

NP → N

VP → V NP

N → "cat" | "fish"

V → "eats"

S=S

To illustrate, for "cat eats fish," the parse tree would show:

S
/ \
NP VP
 | / \
N V NP
| | |
"cat" "eats" N
|
"fish"
What is top down tree Parsing?
Top-down tree parsing is a parsing strategy that starts from the root of the parse tree
(usually the start symbol) and attempts to derive the input sentence by expanding the
grammar rules. The parser applies production rules recursively, matching the non-terminals
on the left-hand side of rules to the input, until it reaches the leaves of the tree (which are
terminal symbols).

Steps:
● The parser starts from the top of the syntactic tree with the start symbol (S, which
stands for "sentence"). In top-down technique parse tree constructs from top and
input will read from left to right.
● It attempts to break down the S symbol into smaller symbols (both terminal and non-
terminal) according to the grammar rules. If the leaf node or parts of speech do not
match the input string, then it does the backtracking and goes back to the most
recent node processed and applies it to another production.
● It validates the sentence by matching the sequence of symbols (both terminals and
non-terminals) with the words in the input sentence.

Example1: Let us try to validate the sentence by parsing the sentence “Book that flight”
using grammar rules.

Grammar Rules:
S ----> NP VP
S ----> VP
NP ----> ART N
NP ----> ART ADJ N
VP -----> V
VP ------> V NP

Top down parsing:

Example2: Let us try to validate the sentence by parsing the sentence “Rahul is eating an
apple” using grammar rules.

Grammar Rules:
S ----> NP VP
NP ----> N
NP --- > DET N
VP ----> AUX VP
VP ---->V NP
N -----> Rahul| apple
Aux ------> is
V ----> eating
DET ----> an

Top down parsing:

What is bottom-up tree parsing?

Bottom-up parsing is a strategy where the parse tree is constructed starting from the
leaves (the input tokens) and working towards the root (the start symbol). The parser
begins with the individual words (terminals) of the input and tries to combine them into larger
syntactic units (non-terminals) until the start symbol of the grammar is reached.
Steps:
● The parser starts from the bottom of the syntactic tree with terminal leaf nodes and
proceeds towards the root.It attempts to reverse manufacture the tree and return the
phrase to the start symbol S.
● The goal of reaching the starting symbol S is accomplished through a series of
reductions; when the right-hand side of some rule matches; The substring is replaced
with the left-hand of the matched production and the process is repeated until the
starting symbol is reached.

Example: Let us try to validate the sentence by parsing the sentence “John is playing
game” using grammar rules.

Grammar Rules:
S ----> NP VP | NP AUX VP
NP ----> DET N |PN |N
VP ---->V NP
PN ------> Noun
NP ----> N
Aux ------> is
V ----> playing
N----> game | John
What is the Depth First Strategy?

The depth-first strategy explores one possible parse tree completely before moving on to
another possibility. This means the parser will try to expand one branch of the tree as far as
it can go before backtracking to explore other branches.

The parser starts from the root and proceeds down one branch, applying grammar rules to
each non-terminal symbol in the symbol list until it reaches terminal symbols (i.e., words).

If the current branch cannot match the input sentence, the parser backtracks to the last
decision point and tries another alternative.

Characteristics of DFS:

1. Space-efficient :Because it doesn't need to store multiple branches at once, Depth

First Strategy is generally more space-efficient compared to Breadth First Strategy.
 2. Prone to backtracking:If the first path chosen is incorrect, the parser must
backtrack, which can lead to inefficiencies if the correct parse is far down another
branch.
3. Not guaranteed to find the shortest parse first : DFS might find a valid parse deep
down a branch but miss a simpler or more optimal parse closer to the root.

DFS is often preferred when memory is limited or when the grammar contains deep
recursive structures where the correct parse is likely to be found after exploring deep into the
tree.

What is Breadth First Strategy?

In the breadth-first strategy, the parser explores all possible expansions at the current level
before moving on to the next level.
The parser generates all possible symbol lists by applying all relevant grammar rules, then
checks each new symbol list for a match with the sentence.

Rather than committing to a single branch and expanding it fully (as in DFS), BFS keeps
track of multiple branches at the same time and expands them level by level.

Characteristics of BFS:

1. Memory-intensive: BFS needs to store multiple branches at each level, which can
lead to high memory usage, especially for longer sentences or complex grammars.
2. Less backtracking:Since BFS explores all possibilities level by level, there is less
backtracking compared to DFS. Each new state is checked as soon as it's created,
making the algorithm less likely to get stuck deep in an incorrect branch.
3. Guaranteed to find the shortest parse first: If multiple valid parses exist, BFS will
find the one with the fewest steps (i.e., the shallowest tree) first.

BFS is more useful when it's important to find the simplest (or shortest) valid parse first. It
ensures that all possible expansions are checked in parallel, so it is often used when we
care about completeness and finding the shallowest solution.

What is the key Difference between Depth First Strategy (DFS)

& Breadth First Strategy (BFS)?
What is Bottom-Up Chart Parsing?
Bottom-up chart parsing is a parsing technique used to efficiently construct parse trees by
incrementally combining smaller constituents (parts of sentences) into larger ones,
starting from the input tokens and working towards the start symbol of the grammar. This
method uses a data structure called a chart, which stores intermediate parsing results
to avoid redundant computations.

In essence, it’s a bottom-up approach where subtrees are built up from the individual words
(or tokens) and combined into larger subtrees, while the chart ensures that each
constituent is only parsed once. This approach is particularly useful when dealing with
ambiguous grammars or longer inputs, as it allows for reuse of intermediate results.

Example: Let us parse a simple sentence “ The happy dog wagged its tail” using bottom
up chart parsing.

Grammar Rules:
S ----> NP VP
NP ----> DET ADJ N | ProN N | DET N
VP ---->V NP
Example2: Let us parse a simple sentence “ The large can can hold the water” using
bottom up chart parsing.

Grammar Rules:
S -> NP VP
NP -> ART ADJ N
NP -> ART N
NP -> ADJ N
VP -> AUX VP
VP -> V NP
What is Top-Down Chart Parsing?
Top-Down Chart Parsing is a parsing strategy where the parser starts with the start
symbol (S) of the grammar and tries to expand it into the input tokens, following the rules
of the grammar. This method uses a chart to store intermediate results and prevent
redundant computations.

In top-down parsing, the goal is to find a way to derive the input sentence from the start
symbol by applying grammar rules, working from higher-level constituents (non-
terminals) down to the input tokens (terminals).

Top-down chart parsing combines these two ideas:

● Predictive like top-down parsing: It predicts the possible structure of the sentence,
based on the grammar, before looking at the actual words.
● Efficient like chart parsing: It uses a chart to store partial results, ensuring that no
constituent is built more than once. This prevents redundant work that can occur in a
pure top-down approach.

Top-Down Chart parsing Steps:

Step 1: Initialization - The parser starts by placing an active arc in the chart for the start
symbol of the grammar. An active arc represents a partially completed rule.

Step 2: Predictive step- From the start symbol, it predicts the possible rules that could
generate the sentence. For instance, if the sentence is expected to be a noun phrase (NP)
followed by a verb phrase (VP), the parser predicts these possibilities and adds them to the
chart.

Step 3: Matching input - As the parser processes the input, it looks for words that match
the predictions. For example, if the parser predicts that the next word should be an article
(ART) in the NP, it checks if the next word in the sentence matches that.

Step 4: Chart update- When a word matches, the parser advances the position of the dot in
the corresponding arc. The chart is updated with the new position to reflect the progress. If a
constituent is completed (i.e., the dot reaches the end of the rule), it becomes a completed
arc and can be used to extend other active arcs.

Example :Let us parse a simple sentence “ The large can can hold the water” using Top-
Down chart parsing.

Grammar Rules:
S -> NP VP
NP -> ART ADJ N
NP -> ART N
NP -> ADJ N
VP -> AUX VP
VP -> V NP
How are chart parsers more efficient than traditional parsers
like tree parsers?
In traditional parsers, like top-down or bottom-up, you might end up doing the
same work over and over. For example, if the parser tries to recognize a part of the
sentence (like a noun phrase) in multiple places, it may reconstruct the same part
again and again. This repetition increases the number of operations needed to parse
a sentence, making it very slow.

For a sentence of length n, the number of operations needed could grow

exponentially, meaning it increases very quickly as the sentence gets longer. If a
constant C represents how much work the parser does, the complexity can be up to
C^n (C raised to the power of n).

Complexity = C^n

In the worst-case scenario, for a sentence with 12 words, a brute force parser
could take 10^12 operations (1,000,000,000,000 operations).

A chart-based parser avoids this repetition. It keeps track of all the parts of the
sentences that have already been analyzed so it doesn't have to re-analyze them.
This makes the process more efficient.

Even though a chart-based parser does more work at each step (represented by K, a
constant), it ensures that no part of the sentence is analyzed more than once. In the
worst case, it looks at every possible pair of words in the sentence, which results in
K * n³ operations, where n is the sentence length and K is the constant.

Complexity = K * n³

In the worst-case scenario, for a sentence with 12 words, a chart-based parser

would take 1000 * 12³ (1,728,000 operations).

In the example, the chart-based parser is up to 500,000 times faster than the brute
force method.

What is a Finite State Machine?

Finite State Machine (FSM) is a model of computation used to recognize or generate strings
based on regular grammars. It is made up of:

● States (nodes) – Points where the system waits for input or makes decisions.
● Transitions (arcs) – Arcs labeled with word categories or symbols that allow
movement between states.
● Start state – Where the parsing begins.
● Accept state (pop arc) – The state that indicates a successful completion.

FSMs cannot handle recursion or complex nested structures (like sentences with multiple
clauses) because they are only as powerful as regular grammars.

Example for FSM on a regular sentence: The sentence “A purple cow.” can be parsed
using finite state machines using following finite state network:
Where,

NP → ART NP1 (An NP starts with an article followed by NP1)

NP1 → ADJ NP1 (NP1 can have multiple adjectives)

NP1 → N (NP1 ends with a noun)

What is a Recursive Transition Network? Explain how they are

used in Parsing.
FSMs cannot handle recursion or complex nested structures (like sentences with multiple
clauses) because they are only as powerful as regular grammars. To parse such
sentences, we need something more powerful than an FSM—one that can reuse
components (like NP) and handle recursion. This is where Recursive Transition Networks
(RTNs) come in.

A Recursive Transition Network (RTN) extends an FSM by allowing:

● Arcs to call other networks, not just word categories.

● Recursion, where a network can call itself (directly or indirectly).

RTNs are equivalent in power to Context-Free Grammars (CFGs) because they can handle
recursion and nested structures—things that FSMs cannot.

Example for Recursive transition network:The sentence “A purple cow ate the grass” can
be parsed using Recursive transition network as follows.

Let us dive into the steps using the above example:

Step 1: Start at the S network (Sentence Parsing). At the start node of the S network, we
need to parse a noun phrase (NP), so we switch to the NP network.

Step 2: Switch to the NP Network to parse the first NP: "The purple cow".

● Start at the NP node. The input word is "The" (an article). Move to NP1.
● At NP1, the next word is "purple" (an adjective). Stay at NP1.
● At NP1, the next word is "cow" (a noun). Follow the pop arc and return to the S
network.

Step 3: Back to the S Network.

● Now we’ve successfully parsed the first NP ("The purple cow").

● Move to S1 and read the next word: "ate".
● "ate" is a verb, so move to S2.

Step 4: At S2, we need to parse another NP: "the grass". Switch to the NP network again.

● Start at the NP node. "the" is an article, so move to NP1.

● In NP1, "grass" is a noun. Follow the pop arc and return to the S network.

Step 5: Back to the S Network. Now that we’ve parsed the second NP ("the grass"), follow
the final pop arc in the S network.
Top-Down Parsing Using Recursive Transition Networks
(RTNs)
Top-down parsing with Recursive Transition Networks (RTNs) is an approach where the
parser starts from the highest-level rule (root node)—typically the start symbol of a grammar
—and tries to expand it downwards by invoking sub-networks that correspond to non-
terminals. This recursive process continues until the parser matches the input sentence or
identifies that no valid parse is possible.

For Example: Using the below RTN, parse the sentence ”The old man cried”.

Below are the steps, involved in parsing the sentence through the recursive networks: