CS50’s Introduction to Artificial Intelligence with Python

OpenCourseWare

Donate  (https://cs50.harvard.edu/donate)

Brian Yu (https://brianyu.me)
[email protected]

David J. Malan (https://cs.harvard.edu/malan/)

[email protected]
 (https://www.facebook.com/dmalan)  (https://github.com/dmalan) 
(https://www.instagram.com/davidjmalan/)  (https://www.linkedin.com/in/malan/)
 (https://www.reddit.com/user/davidjmalan) 
(https://www.threads.net/@davidjmalan)  (https://twitter.com/davidjmalan)

Lecture 6

Language
So far in the course, we needed to shape tasks and data such that an AI will be able to process
them. Today, we will look at how an AI can be constructed to process human language.

Natural Language Processing spans all tasks where the AI gets human language as input. The
following are a few examples of such tasks:

 automatic summarization, where the AI is given text as input and it produces a summary
of the text as output.
 information extraction, where the AI is given a corpus of text and the AI extracts data as
output.
 language identification, where the AI is given text and returns the language of the text as
output.
 machine translation, where the AI is given a text in the origin language and it outputs the
translation in the target language.
 named entity recognition, where the AI is given text and it extracts the names of the
entities in the text (for example, names of companies).
  speech recognition, where the AI is given speech and it produces the same words in text.
 text classification, where the AI is given text and it needs to classify it as some type of
text.
 word sense disambiguation, where the AI needs to choose the right meaning of a word
that has multiple meanings (e.g. bank means both a financial institution and the ground
on the sides of a river).

Syntax and Semantics

Syntax is sentence structure. As native speakers of some human language, we don’t struggle
with producing grammatical sentences and flagging non-grammatical sentences as wrong. For
example, the sentence “Just before nine o’clock Sherlock Holmes stepped briskly into the room”
is grammatical, whereas the sentence “Just before Sherlock Holmes nine o’clock stepped briskly
the room” is non-grammatical. Syntax can be grammatical and ambiguous at the same time, as
in “I saw the man with the telescope.” Did I see (the man with the telescope) or did I see (the
man), doing so by looking through the telescope? To be able to parse human speech and
produce it, the AI needs to command syntax.

Semantics is the meaning of words or sentences. While the sentence “Just before nine o’clock
Sherlock Holmes stepped briskly into the room” is syntactically different from “Sherlock Holmes
stepped briskly into the room just before nine o’clock,” their content is effectively identical.
Similarly, although the sentence “A few minutes before nine, Sherlock Holmes walked quickly
into the room” uses different words from the previous sentences, it still carries a very similar
meaning. Moreover, a sentence can be perfectly grammatical while being completely
nonsensical, as in Chomsky’s example, “Colorless green ideas sleep furiously.” To be able to parse
human speech and produce it, the AI needs to command semantics.

Context-Free Grammar
Formal Grammar is a system of rules for generating sentences in a language. In Context-Free
Grammar, the text is abstracted from its meaning to represent the structure of the sentence
using formal grammar. Let’s consider the following example sentence:

 She saw the city.

This is a simple grammatical sentence, and we would like to generate a syntax tree
representing its structure.

We start by assigning each word its part of speech. She and city are nouns, which we will mark
as N. Saw is a verb, which we will mark as V. The is a determiner, marking the following noun as
definite or indefinite, and we will mark it as D. Now, the above sentence can be rewritten as

 NVDN
So far, we have abstracted each word from its semantic meaning to its part of speech. However,
words in a sentence are connected to each other, and to understand the sentence we must
understand how they connect. A noun phrase (NP) is a group of words that connect to a noun.
For example, the word she is a noun phrase in this sentence. In addition, the words the city also
form a noun phrase, consisting of a determiner and a noun. A verb phrase (VP) is a group of
words that connect to a verb. The word saw is a verb phrase in itself. However, the words saw the
city also make a verb phrase. In this case, it is a verb phrase consisting of a verb and a noun
phrase, which in turn consists of a determiner and a noun. Finally, the whole sentence (S) can be
represented as follows:

Using formal grammar, the AI is able to represent the structure of sentences. In the grammar we
have described, there are enough rules to represent the simple sentence above. To represent
more complex sentences, we will have to add more rules to our formal grammar.

nltk
As is often the case in Python, multiple libraries have been written to implement the idea
above. nltk (Natural Language Toolkit) is one such library. To analyze the sentence from above,
we will provide the algorithm with rules for the grammar:
 import nltk

grammar = nltk.CFG.fromstring("""
S -> NP VP

NP -> D N | N
VP -> V | V NP

D -> "the" | "a"

N -> "she" | "city" | "car"
V -> "saw" | "walked"
""")

parser = nltk.ChartParser(grammar)

Similar to what we did above, we define what possible components could be included in others.
A sentence can include a noun phrase and a verb phrase, while the phrases themselves can
consist of other phrases, nouns, verbs, etc., and, finally, each part of speech spans some words in
the language.

sentence = input("Sentence: ").split()

try:
for tree in parser.parse(sentence):
tree.pretty_print()
tree.draw()
except ValueError:
print("No parse tree possible.")

After giving the algorithm an input sentence split into a list of words, the function prints the
resulting syntactic tree (pretty_print) and also generates a graphic representation (draw).

n-grams

An n-gram is a sequence of n items from a sample of text. In a character n-gram, the items are
characters, and in a word n-gram the items are words. A unigram, bigram, and trigram are
sequences of one, two, and three items. In the following sentence, the first three n-grams are
“how often have,” “often have I,” and “have I said.”

“How often have I said to you that when you have eliminated the impossible whatever remains,
however improbable, must be the truth?”

n-grams are useful for text processing. While the AI hasn’t necessarily seen the whole sentence
before, it sure has seen parts of it, like “have I said.” Since some words occur together more
often then others, it is possible to also predict the next word with some probability. For
example, your smartphone suggests words to you based on a probability distribution derived
from the last few words you typed. Thus, a helpful step in natural language processing is
breaking the sentence into n-grams.

Tokenization
Tokenization is the task of splitting a sequence of characters into pieces (tokens). Tokens can be
words as well as sentences, in which case the task is called word tokenization or sentence
tokenization. We need tokenization to be able to look at n-grams, since those rely on sequences
of tokens. We start by splitting the text into words based on the space character. While this is a
good start, this method is imperfect because we end up with words with punctuation, such as
“remains,”. So, for example, we can remove punctuation. However, then we face additional
challenges, such as words with apostrophes (e.g. “o’clock”) and hyphens (e.g. “pearl-grey).
Additionally, some punctuation is important for sentence structure, like periods. However, we
need to be able to tell apart between a period at the end of the word “Mr.” and a period in the
end of the sentence. Dealing with these questions is the process of tokenization. In the end,
once we have our tokens, we can start looking at n-grams.

Markov Models
As discussed in previous lectures, Markov models consist of nodes, the value of each of which
has a probability distribution based on a finite number of previous nodes. Markov models can be
used to generate text. To do so, we train the model on a text, and then establish probabilities
for every n-th token in an n-gram based on the n words preceding it. For example, using
trigrams, after the Markov model has two words, it can choose a third one from a probability
distribution based on the first two. Then, it can choose a fourth word from a probability
distribution based on the second and third words. To see an implementation of such a model
using nltk, refer to generator.py in the source code, where our model learns to generate
Shakespeare-sounding sentences. Eventually, using Markov models, we are able to generate text
that is often grammatical and sounding superficially similar to human language output.
However, these sentences lack actual meaning and purpose.

Bag-of-Words Model
Bag-of-words is a model that represents text as an unordered collection of words. This model
ignores syntax and considers only the meanings of the words in the sentence. This approach is
helpful in some classification tasks, such as sentiment analysis (another classification task
would be distinguishing regular email from spam email). Sentiment analysis can be used, for
instance, in product reviews, categorizing reviews as positive or negative. Consider the following
sentences:

1. “My grandson loved it! So much fun!”

2. “Product broke after a few days.”
3. “One of the best games I’ve played in a long time.”
4. “Kind of cheap and flimsy, not worth it.”

Based only on the words in each sentence and ignoring the grammar, we can see that sentences
1 and 3 are positive (“loved,” “fun,” “best”) and sentences 2 and 4 are negative (“broke,” “cheap,”
“flimsy”).

Naive Bayes
Naive Bayes is a technique that’s can be used in sentiment analysis with the bag-of-words
model. In sentiment analysis, we are asking “What is the probability that the sentence is
positive/negative given the words in the sentence.” Answering this question requires computing
conditional probability, and it is helpful to recall Bayes’ rule from lecture 2:

Now, we would like to use this formula to find P(sentiment | text), or, for example, P(positive |
“my grandson loved it”). We start by tokenizing the input, such that we end up with P(positive |
“my”, “grandson”, “loved”, “it”). Applying Bayes’ ruled directly, we get the following expression:
P(“my”, “grandson”, “loved”, “it” | positive)*P(positive)/P(“my”, “grandson”, “loved”, “it”). This
complicated expression will give us the precise answer to P(positive | “my”, “grandson”, “loved”,
“it”).

However, we can simplify the expression if we are willing to get an answer that’s not equal, but
proportional to P(positive | “my”, “grandson”, “loved”, “it”). Later on, knowing that the probability
distribution needs to sum up to 1, we can normalize the resulting value into an exact
probability. This means that we can simplify the expression above to the numerator only: P(“my”,
“grandson”, “loved”, “it” | positive)*P(positive). Again, we can simplify this expression based on the
knowledge that a conditional probability of a given b is proportional to the joint probability of a
and b. Thus, we get the following expression for our probability: P(positive, “my”, “grandson”,
“loved”, “it”)*P(positive). Calculating this joint probability, however, is complicated, because the
probability of each word is conditioned on the probabilities of the words preceding it. It
requires us to compute P(positive)*P(“my” | positive)*P(“grandson” | positive, “my”)*P(loved |
positive, “my”, “grandson”)*P(“it” | positive, “my”, “grandson”, “loved”).

Here is where we use Bayes’ rules naively: we assume that the probability of each word is
independent from other words. This is not true, but despite this imprecision, Naive Bayes’
produces a good sentiment estimate. Using this assumption, we end up with the following
probability: P(positive)*P(“my” | positive)*P(“grandson” | positive)*P(“loved” | positive)*P(“it” |
positive), which is not that difficult to calculate. P(positive) = the number of all positive samples
divided by the number of total samples. P(“loved” | positive) is equal to the number of positive
samples with the word “loved” divided by the number of positive samples. Let’s consider the
example below, with smiling and frowning emojies substituting the words “positive” and
“negative”:

On the right we are seeing a table with the conditional probabilities of each word on the left
occurring in a sentence given that the sentence is positive or negative. In the small table on the
left we are seeing the probability of a positive or a negative sentence. On the bottom left we
are seeing the resulting probabilities following the computation. At this point, they are in
proportion to each other, but they don’t tell us much in terms of probabilities. To get the
probabilities, we need to normalize the values, arriving at P(positive) = 0.6837 and P(negative) =
0.3163. The strength of naive Bayes is that it is sensitive to words that occur more often in one
type of sentence than in the other. In our case, the word “loved” occurs much more often in
positive sentences, which makes the whole sentence more likely to be positive than negative. To
see an implementation of sentiment assessment using Naive Bayes with the nltk library, refer to
sentiment.py.

One problem that we can run into is that some words may never appear in a certain type of
sentence. Suppose none of the positive sentences in our sample had the word “grandson.” Then,
P(“grandson” | positive) = 0, and when computing the probability of the sentence being positive
we will get 0. However, this is not the case in reality (not all sentences mentioning grandsons
are negative). One way to go about this problem is with Additive Smoothing, where we add a
value α to each value in our distribution to smooth the data. This way, even if a certain value is
0, by adding α to it we won’t be multiplying the whole probability for a positive or negative
sentence by 0. A specific type of additive smoothing, Laplace Smoothing adds 1 to each value in
our distribution, pretending that all values have been observed at least once.

Information Retrieval
Information retrieval is the task of finding relevant documents in response to a user query. To
achieve this task, we use topic modeling to discover the topics for a set of documents. How can
the AI go about extracting the topics of documents? One way to do so is by looking at term
frequency, which is simply counting how many times a term appears in a document. The idea
behind this is that key terms and important ideas are likely to repeat. To get term frequencies,
we will work with the tf-idf library.

tf-idf
We can use the tf-idf library for information retrieval tasks. In lecture, we hoped to use term
frequency to establish what the topics of each Sherlock Holmes story are in the corpus we had.
However, when we ran the counting algorithm naively, the top-5 most common words were
function words, words that are used for syntactic purposes in the sentence and not as
independent units of meaning, such as “the,” “by,” “and,” “which,” “yet,” etc. By simply counting all
the words by frequency of appearance we didn’t get the unique words that repeat in the corpus
and define its topics, but the words that are the most common in the English language in
general. The words that we are after are content words, words that carry meaning
independently, such as “crime,” “brothers,” “demons,” “gentle,” “meek,” etc.

We can run the algorithm again, excluding words that we define as function words in the
English language. Now the output becomes more indicative of the content of each document.
However, in the case of Sherlock Holmes, the most common word in each document is,
unsurprisingly, Holmes. Since we are searching the corpus of Sherlock Holmes story, having
Holmes as one of the key words tells us nothing new about each story. This brings us to the idea
of Inverse Document Frequency, which is a measure of how common or rare a word is across
documents in a corpus. It is usually computed by the following equation:
Note that when the word appears in all documents, the value inside the logarithm is 1, and
log(1) = 0. This is where the name of the library comes from: tf-idf stand for Term Frequency —
Inverse Document Frequency. What this library is able to do is to multiply the term frequency of
each word by the inverse document frequency, thus getting a value for each word. The more
common the word is in one document and the fewer documents it appears in, the higher its
value will be. Alternatively, the less common the word is in a document and the more common
it is across documents, the lower its value. This lets us get as output a list of words per
document that are likely to span the main topics that define the document.

Information Extraction
Information Extraction is the task of extracting knowledge from documents. So far, treating text
with the bag-of-words approach was helpful when we wanted the AI to perform simple tasks,
such as recognizing sentiment in a sentence as positive or negative, or retrieving the key words
in a document. However, for a task like information extraction, where we need the AI to
understand the connections between different words, the bag-of-words approach won’t get us
very far.

A possible task of information extraction can take the form of giving a document to the AI as
input and getting a list of companies and the years when they were founded as output. One way
we can go about this is providing the AI with a template, such as “When {company} was
founded in {year}.” This will not get us perfect results, because not all information about
companies and their years of foundation are written in precisely this format. Still, if the dataset
is large enough, we will definitely come across sentences of precisely this form, which will allow
the AI to extract this knowledge. A more advanced approach would be to give the AI an
abstracted example like “Facebook, 2004,” and let it develop its own model to extract the data.
By going through enough data, the AI will be able to infer possible templates for extracting
information similar to the example.

Using templates, even if self-generated by the AI, is helpful in tasks like Information Extraction.
However, if we want the AI to be able to produce text like a human, the AI needs to be able to
understand not just templates, but how all words in the language relate to each other in their
meanings.

Word Net
Wordnet is a database similar to a dictionary, where words are given definitions as well as
broader categories. For example, the word “house” will yield multiple definitions, one of which is
“a dwelling that serves as living quarters for one or more families.” This definition is paired with
the categories “building” and “dwelling.” Having access to these broader categories will allow us
to make connections between words that are semantically similar.

Word Representation
We want to represent word meanings in our AI. As we’ve seen before, it is convenient to provide
input to the AI in the form of numbers. One way to go about this is by using One-Hot
Representation, where each word is represented with a vector that consists of as many values
as we have words. Except for a single value in the vector that is equal to 1, all other values are
equal to 0. How we can differentiate words is by which of the values is 1, ending up with a
unique vector per word. For example, the sentence “He wrote a book” can be represented as four
vectors:

 [1, 0, 0, 0] (he)
 [0, 1, 0, 0] (wrote)
 [0, 0, 1, 0] (a)
 [0, 0, 0, 1] (book)

However, while this representation works in a world with four words, if we want to represent
words from a dictionary, when we can have 50,000 words, we will end up with 50,000 vectors of
length 50,000. This is incredibly inefficient. Another problem in this kind of representation is
that we are unable to represent similarity between words like “wrote” and “authored.” Instead,
we turn to the idea of Distributed Representation, where meaning is distributed across multiple
values in a vector. With distributed representation, each vector has a limited number of values
(much less than 50,000), taking the following form:

 [-0.34, -0.08, 0.02, -0.18, …] (he)

 [-0.27, 0.40, 0.00, -0.65, …] (wrote)
 [-0.12, -0.25, 0.29, -0.09, …] (a)
 [-0.23, -0.16, -0.05, -0.57, …] (book)

This allows us to generate unique values for each word while using smaller vectors.
Additionally, now we are able to represent similarity between words by how different the values
in their vectors are.

“You shall know a word by the company it keeps” is an idea by J. R. Firth, an English linguist.
Following this idea, we can come to define words by their adjacent words. For example, there
are limited words that we can use to complete the sentence “for ___ he ate.” These words are
probably words like “breakfast,” “lunch,” and “dinner.” This brings us to the conclusion that by
considering the environment in which a certain word tends to appear, we can infer the meaning
of the word.

word2vec
word2vec is an algorithm for generating distributed representations of words. It does so by
Skip-Gram Architecture, which is a neural network architecture for predicting context given a
target word. In this architecture, the neural network has an input unit for every target word. A
smaller, single hidden layer (e.g. 50 or 100 units, though this number is flexible) will generate
values that represent the distributed representations of words. Every unit in this hidden layer is
connected to every unit in the input layer. The output layer will generate words that are likely to
appear in a similar context as the target words. Similar to what we saw in last lecture, this
network needs to be trained with a training dataset using the backpropagation algorithm.

This neural network turns out to be quite powerful. In the end, of the process, every word ends
up being just a vector, or a sequence of numbers. For example,

book: [-0.226776 -0.155999 -0.048995 -0.569774 0.053220 0.124401 -0.091108 -0.606255

-0.114630 0.473384 0.061061 0.551323 -0.245151 -0.014248 -0.210003 0.316162 0.340426
0.232053 0.386477 -0.025104 -0.024492 0.342590 0.205586 -0.554390 -0.037832 -0.212766
-0.048781 -0.088652 0.042722 0.000270 0.356324 0.212374 -0.188433 0.196112 -0.223294
-0.014591 0.067874 -0.448922 -0.290960 -0.036474 -0.148416 0.448422 0.016454 0.071613
-0.078306 0.035400 0.330418 0.293890 0.202701 0.555509 0.447660 -0.361554 -0.266283
-0.134947 0.105315 0.131263 0.548085 -0.195238 0.062958 -0.011117 -0.226676 0.050336
-0.295650 -0.201271 0.014450 0.026845 0.403077 -0.221277 -0.236224 0.213415 -0.163396
-0.218948 -0.242459 -0.346984 0.282615 0.014165 -0.342011 0.370489 -0.372362 0.102479
0.547047 0.020831 -0.202521 -0.180814 0.035923 -0.296322 -0.062603 0.232734 0.191323
0.251916 0.150993 -0.024009 0.129037 -0.033097 0.029713 0.125488 -0.018356 -0.226277
0.437586 0.004913]
By themselves, these numbers don’t mean much. But by finding which other words in the corpus
have the most similar vectors, we can run a function that will generate the words that are the
most similar to the word book. In the case of this network it will be: book, books, essay, memoir,
essays, novella, anthology, blurb, autobiography, audiobook. This is not bad for a computer!
Through a bunch of numbers that don’t carry any specific meaning themselves, the AI is able to
generate words that really are very similar to book not in letters or sounds, but in meaning! We
can also compute the difference between words based on how different their vectors are. For
example, the difference between king and man is similar to the difference between queen and
woman. That is, if we add the difference between king and man to the vector for woman, the
closest word to the resulting vector is queen! Similarly, if we add the difference between ramen
and japan to america, we get burritos. By using neural networks and distributed representations
for words, we get our AI to understand semantic similarities between words in the language,
bringing us one step closer to AIs that can understand and produce human language.