Scribd
Upload
12 views

Lecture01 Introduction

The document outlines the course structure for COMS W4705 Natural Language Processing, taught by Dr. Daniel Bauer, including important dates, prerequisites, grading criteria, and course resources. It emphasizes the significance of NLP in AI, its applications, and the evolution of machine learning techniques in the field. Students will learn about language theories, algorithms, and various NLP applications, with a focus on ethical considerations in the use of NLP methods and datasets.

Uploaded by

yl5404
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
12 views

Lecture01 Introduction

The document outlines the course structure for COMS W4705 Natural Language Processing, taught by Dr. Daniel Bauer, including important dates, prerequisites, grading criteria, and course resources. It emphasizes the significance of NLP in AI, its applications, and the evolution of machine learning techniques in the field. Students will learn about language theories, algorithms, and various NLP applications, with a focus on ethical considerations in the use of NLP methods and datasets.

Uploaded by

yl5404
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 35

Natural Language

Processing
Lecture 1: Course Overview and Introduction.
11/06/2024

COMS W4705
Daniel Bauer
The 4705 Team
• Instructor: Dr. Daniel Bauer (he/him/his)
O ce hours: Fri 1:15-2:45pm (after class 704 CEPSR and
on Zoom, starting 9/13).

• Course Assistants:

See Courseworks for contact info and o ce hour


schedule.
ffi
ffi
Important Dates
• Lectures: Fri 10:10-12:40pm (incl. 20 min break)

• Location: 417 IAB (or on Zoom for CVN or by


permission). All sessions will be recorded.

• Exam 1: Friday Oct 18

• Exam 2: Friday Dec 6

There is no additional nal exam.


fi
Course Resources
• Courseworks

• All course materials: videos, lecture notes, code,


announcements, assignments, reading materials

• Homework submission, grade book.

• Ed used for Q & A (shared between sections)


Textbook / Reading
• There is NO o cial textbook for this course.

• Recommended textbook 1 (somewhat outdated, we


won’t follow this too closely, but
references will be provided):

Dan Jurafsky & James Martin


Speech and Language Processing
2nd Ed. Prentice Hall (2009).

• Draft of most 3rd edition chapters:


https://web.stanford.edu/~jurafsky/slp3/
ffi
Textbook / Reading
• Recommended textbook 2:

Yoav Goldberg
Neural Network Methods for
Natural Language Processing
Morgan & Claypool. 2017

• Available as an ebook through the CU library


https://clio.columbia.edu/catalog/13676351
Prerequisites
• Data Structures (COMS W3134 or COMS W3137)

• Discrete Math (COMS W3202, recommended)

• Some experience with basic probability/statistics.

• Some previous or concurrent exposure to AI and machine


learning is bene cial, but not required.

• Some experience with Python is helpful.


fi
Grading

• 40%: 5 programming assignments (lowest score dropped,


10% each)

• 50%: exams (25% each)

• 10%: Participation (in class attendance and on Ed)


Homework
• Homework uploaded through Courseworks. Do not email solutions.
Check your submission!

• ~ 2-week turnaround.

• 72h limit on regrade requests.

• Only programming. Theory done in class / online ungraded


exercises.

• Use Python 3!

• There will typically be some sca olding code to start you o .

• Some assignments will require a GPU (we will use Google Colab)
ff
ff
Academic Honesty
• Submit your own answers and code.

• Review academic honesty policy on the syllabus.

• When in doubt, ask.

• When in trouble, ask for help (and early).

• We will talk _about_ ChatGPT et al. but please do not use


large language models for your homework.
NLP in the Movies
I am fluent in over
six million forms Open the pod bay
of communication doors HAL!

I’m sorry Dave, I’m


afraid I can’t do
that!
Natural Language
Processing
• Important and active research area within AI.

• Timely: Most of our activities online are text based


(web-pages, email, social media, blogs, news, product
descriptions and reviews, medical reports, course content, …)

• NLP leverages more and more available training data and


modern Machine Learning techniques, such as neural
networks (RNNs, transformers etc.)

• Communicating with computers is the “holy grail” of AI.

• NLP may be "AI complete".


Turing Test
(Alan Turing, 1950)

• A computer passes the test of intelligence if it can fool


a human interrogator into believing it is human.

• What skills are needed to build such a system?


• Language processing, knowledge representation,
reasoning, learning.
Image source: Russel & Norvig, Arti cial Intelligence - A Modern Approach
fi
Natural Language
Processing

AI NLP Linguistics

“Every time I re a linguist, my performance goes up” (Fred Jelinek)


fi
Natural Language Processing
vs. Computational Linguistics

• NLP: Build systems that can understand and generate


natural language. Focus on applications.

• Computational Linguistics: Study human language


using computational approaches.

• Many overlapping techniques.


Applications: Information
Retrieval

query

indexed document
corpus

ranked results
Applications: Text
Classi cation

• Spam ltering.

• Detecting topics / genre.

• Sentiment analysis, author recognition, forensic


linguistics, …

• Detecting hate speech.


fi
fi
Applications: Sentiment
Analysis
Fantastic... truly a wonderful family movie

I have a mixed feeling about this movie.

Well it is fun for sure but de nitely not appropriate


for kids 10 and below

My kids loved it!!

The movie is very funny and entertaining. Big A+

I got so boooored...

Disappointed. They showed all fun details in the trailer

Cute but not for adults


fi
Application: Question
Answering
“Where was George Washington born?”

Unstructured
Text

QA system

Knowledge
Base

“Westmoreland County, Virginia”


Applications: Playing
Jeopardy! IBM Watson [2011]

William Wilkinson’s “An Account of the Principalities of Wallachia and


Moldavia“ inspired this author’s most famous novel.

Combines information extraction & natural language understanding.


Applications:
Summarization

Credit: Prof. Kathleen Mc Keown


Applications: Machine
Translation
Machine Translation
• One of the main research areas in NLP, and one of the oldest.
Historical motivation: Translate Russian to English.

• MT is really di cult:

• “Out of sight, out of mind” → “Invisible, imbecile”

• “The spirit is willing, but the esh is weak”


English → Russian → English
“The vodka is good, but the meat is rotten”

• Challenges: Word order, multiple translations for a word


(need context), want to preserve meaning.
ffi
fl
Machine Translation

• Until recently phrase-based translation was the


predominant framework.

• Today neural network models are used.

• Google Translate supports > 100 languages. Near-human


translation quality for some language pairs.
Machine Translation
Applications: Virtual
Assistants
• Siri (Apple), Google Now, Cortana (Microsoft), Alexa
(Amazon).

• Subtasks: Speech recognition, language understanding


(in context?), speech generation, …
Applications: Large
Language Models

• Predictive text, content generation "Generative AI"

• See ChatGPT example.


Applications: Multi-modal
NLP
Image Captioning Visual Question Answering

“Man in black t-shirt is playing guitar.”

RoboNLP (instruction giving / following, summarization etc.)


Evolution of ML techniques
in NLP
• Rules and heuristics, patterns matching, formal grammars.

• Statistical NLP, generative probabilistic models.

• Discriminative models, support vector machines, logistic regression.

• Back to large generative models.

• Neural networks, phase 1 (RNNs including LSTMs, CNNs)

• Neural networks, phase 2, transformer models, large language


models, pretraining

• Few / zero-shot learning. Prompting.


NLP History
rules and heuristics, patterns
matching, formal grammars.
1950s Statistical NLP, generative probabilistic models.
Corpus-based NLP.
1980s and 90s
Discriminative models,
(e.g. logistic regression,
support vector machines)
early 2000s

neural nets, neural sequence models


RNN, LSTM
mid/late 2000s
pre-training,
embeddings
2010s
transformers
large pre-trained LMs
late 2010s, 2020s
•
GPT-2 Examples
GPT is a transformer-based language model created by OpenAI.

• GPT-2 example (Feb 2019 1.5b parameters, trained on 8m web pages)

https://openai.com/blog/better-language-models/#sample1
GPT "prompting" examples

• GPT-3 (GPT3, Jun 2020, 175b parameters, trained on 45TB of text)

• Fine-tuned model InstructGPT trained using Reinforcement Learning from


Human Feedback.
November 2022 ChatGPT

• Fine-tuned GPT 3.5 using dialog data, then optimized using Reinforcement
Learning from Human Feedback.

• Output may be indistinguishable from human output in many cases.


What You Will Learn In This
Course
• How can machines understand and generate natural
language?

• Theories about language (linguistics).

• Algorithms.

• Statistical / Machine Learning Methods, incl. neural


networks.

• Applications.
Course Overview
• Core NLP techniques.

• Language modeling, part-of-speech tagging, syntactic parsing, word-


sense disambiguation, semantic parsing, text similarity.

• Applications.

• text classi cation, machine translation, generation, image


captioning,...

• Machine Learning Techniques:


Supervised machine learning, bayesian models, sequence models (n-
gram models, HMMs), deep learning techniques (RNNs, transformers, ...)

• Critical assessment of NLP methods and data sets (ethics in NLP).


fi

You might also like