m e

lco !
W e 47
C S 4
to
CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 1
What is CS447?
This class is a broad introduction to NLP

Target audience: CS students

Advanced undergraduates,
Masters and PhD students

Prerequisites:
A solid foundation in computer science
Basic knowledge of algorithms
Basic knowledge of probability/statistics, linear algebra
Programming skills (Python 3)

CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 2

What will you learn in this class?
— What is NLP?
The core tasks (as well as data sets and evaluation metrics)
that people work on in NLP

— How does NLP work?

The fundamental models, algorithms and representations
that have been developed for these tasks

— Why is NLP hard?

The relevant linguistic concepts and phenomena
that have to be handled to do well at these tasks

CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 3

NLP is necessary to…
… analyze text automatically at scale
(text = news, documents, social media, search queries,…)

… translate automatically between languages

(language = English, Chinese, Arabic, Hindi, etc.)

… communicate naturally with systems/devices

(‘naturally’ = in human language;
systems/devices = robots, computers, costumer support,
digital assistants, smart devices, navigation systems,…)
… generate text automatically at scale
(generate = produce)
CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 4
Why should you take this class?
NLP is an (increasingly) important area
NLP is now good enough for real-world applications.
There is a huge growth in NLP companies
and NLP jobs (in many industries)

NLP is far from solved (despite rapid progress)

There is still a lot that remains to be done!

Doing NLP well requires a broad mix of knowledge:

— What is natural language?
— What about natural language is challenging for computers?
— What kind of data, algorithms, machine learning approaches
can we use (and which ones do we need to develop)?

CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 5

What about Large Language Models?
LLMs are revolutionizing NLP
LLMs are rapidly transforming the state of the art (SOTA) in NLP.
Better models come out faster than we can update our classes.
LLMs are big money
LLMs have made NLP a multi-billion dollar industry.
Only a few industry players can produce “frontier” (SOTA) LLMs.
Frontier LLMs are poorly understood
Many important details (model size/architecture, training regime/
data) are not publicly known.
But LLMs have not “solved” NLP
There is a lot more to NLP than LLMs

CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 6

Teaching NLP in the LLM era
Why should you still take an NLP class?
Anybody can use chatbots/AI tools.
But not everybody understands how or why LLMs work.
And not everybody understands why NLP is challenging.
LLMs are an important part of NLP now
We will of course teach you the basics of LLMs.
An NLP class is not just an LLM class.
We want you to understand a variety of NLP tasks/problems.
We want you to understand why NLP is challenging.
We want you to be able to do NLP.
We also want you to know not to reinvent the wheel

CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 7

Teaching LLMs
The basic principles behind LLMs are known:
We know the class of neural architecture (transformers)
We know how LLMs can be trained

Most “frontier” LLMs are black boxes:

We don’t know their size/precise architecture
We don’t know what speci c data they have been trained on
We don’t know how much data they have been trained on
We don’t know how exactly they have been trained

This class will teach basic principles of LLMs.

We will only teach the NLP side of LLMs, and won’t get into the
system side of training/storing/accessing huge LLMs.
CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 8
fi
The focus of this class
We want to identify the structure and meaning
of words, sentences, texts and conversations
– N.B.: we do not deal with speech/audio (no signal processing)

We mainly deal with language analysis/understanding,

and somewhat less with language generation/production

We focus on fundamental concepts, methods, models,

tasks and algorithms, not so much on current research:
– Data (natural language): Linguistic concepts and phenomena
– Representations: Embeddings/vectors, grammars, automata,…
– Tasks: Analysis, generation, translation, …
– Models: Neural models, statistical models, …
CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 9
What you should learn
You should be able to answer the following questions:
– What makes natural language dif cult for computers?
– What are the core NLP tasks?
– What are the main modeling techniques used in NLP?

We won’t be able to cover all of the latest research…

(this requires more time, and a much stronger background in
machine learning than we can assume for this class)

… but I would still like you to get an understanding of:

– How well does current NLP technology work (or not)?
– What NLP software and datasets are available?
– How to read NLP research papers [4 credits section]
CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 10
fi
Our Syllabus (tentative)
Week 1: Introduction
Week 2: The Structure and Distribution of Words (HW1 out)
Week 3: Classi cation for NLP
Week 4: The Meaning of Words
Week 5: Introduction to Neural Networks for NLP (HW2 out)
Week 6: Sequence Labeling
Week 7: Neural Sequence Models
Week 8: Machine Translation and Sequence-to-Sequence Models (HW3 out)
Week 9: Large Language Models
Week 10: The Structure of Sentences
Week 11: The Meaning of Sentences (HW4 out)
Week 12: The Structure and Meaning of Discourse
Week 13: Dialogue and Conversational AI
Week 14: Multimodal NLP
Week 15: Ethics in NLP

CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 11

fi
Our Textbook
We loosely follow
Jurafsky and Martin,
Speech and Language Processing 3rd ed.
(February 2024 version)
https://web.stanford.edu/~jurafsky/slp3/

This 3rd edition differs substantially from the earlier 1st and
2nd editions.
We try our best to keep references to relevant chapters/
sections up-to-date, but because this 3rd edition is so different
and has been under development for a number of years,
references to speci c chapters in our materials may be
outdated, because material may have been moved or deleted.
CS447 Natural Language Processing (J. Hockenmaier) https://courses.grainger.illinois.edu/cs447/ 12
fi