Part01 Overview
Part01 Overview
Part I: Overview
Henning Wachsmuth
https://ai.uni-hannover.de
https://wikimedia.org
We mostly speak about text only in this course.
• The goal is to decode structured information
from language, or to encode it in language.
• NLP is a subfield of AI, and one part of computational linguistics.
Computational linguistics
• Roughly, the intersection of computer science and linguistics
• Methods for tackling analysis and synthesis tasks from NLP
• Models to explain linguistic phenomena, using knowledge or statistics
Linguistics
• The study of natural language(s) in terms of form, meaning, and context
Introduction to NLP I Overview © Wachsmuth 2023 4
Natural Language Processing (NLP)
Analysis and Synthesis
Task
• Identify entities, their attributes, and their relations in a given text
• Example. Extract company’s founding dates from a news article
Possible approach
1. Lexical and syntactic preprocessing
2. Named and numeric entity recognition
3. Reference resolution
4. Entity relation extraction
Introduction to NLP I Overview © Wachsmuth 2023 6
Natural Language Processing (NLP)
Example: Language modeling
Task
• Extend a given text word by word until a suitable ending is reached.
• Example. Answer a user’s question to a chatbot
Possible approach
1. Train general language model on huge amounts of text examples
2. Fine-tune model on question-answer training pairs
Introduction to NLP I Overview © Wachsmuth 2023 7
Natural Language Processing (NLP)
Terminology
Terms in NLP
• Task. A specific problem with a defined input and desired output
Examples: Constituency parsing, summarization, ...
• Technique. A general way of how to analyze and/or synthesize a text
Examples: Probabilistic parsing, language model, ...
Applications
https://de.wikipedia.org
• Software that employs NLP to solve real-world problems
• This includes tools, systems, web services, and similar.
The term application is also used in others ways in NLP.
Why applications?
• Automate or human tasks and/or improve over human performance
• Use cases: Writing assistance, text analytics, conversational AI, etc.
Examples
• Writing assistance. DeepL, Grammarly, Booking texts, Google Mail, ...
• Text Analytics. IBM Watson, Facebook Ads & Targeting, Apple Mail, ...
• Conversational AI. ChatGPT, Google Assistant, Siri, Alexa, ...
IBM Watson
https://en.wikipedia.org
• A technology for text analytics and decision support
• Originally: A focused question answering system
• First showcase was the “Jeopardy!” task
... ...
...
Selected milestones
• February 2011. Watson wins Jeopardy
https://www.youtube.com/watch?v=P18EdAKuC1U
• October 2011. Siri starts on the iPhone
https://www.youtube.com/watch?v=gUdVie_bRQo
• August 2014. Skype translates conversations in real time
https://www.youtube.com/watch?v=RuAp92wW9bg
• May 2018. Google Duplex makes phone call appointments
https://www.youtube.com/watch?v=pKVppdt_-B4
• February 2019. Project Debater competes in entire debates
https://www.youtube.com/watch?v=nJXcFtY9cWY
• November 2022. ChatGPT leads conversations on any topic
https://chat.openai.com
Observations
• NLP inside: All main analysis and synthesis tasks are tackled on text.
• None of these applications works perfectly.
Introduction to NLP I Overview © Wachsmuth 2023 15
Challenges
Challenges
Levels of language
• Phonetics. Physical aspects of
speech sounds
https://en.wikipedia.org
• Phonology. Linguistic sounds of a
particular language
• Morphology. Senseful components of words
• Syntax. Structural relationships between words, usually in a sentence
• Semantics. Meaning of single words and compositions of words
• Discourse. Composition of linguistic units larger than a sentence
• Pragmatics. Use of language to accomplish certain goals
Notice
• Phonetics and phonology are usually disregarded in NLP.
• Not all language phenomena clearly refer to one level, e.g., sentiment.
Introduction to NLP I Overview © Wachsmuth 2023 17
Challenges
Ambiguity in Natural Language
Ambiguity
• Linguistic utterances allow for multiple interpretations.
• Fundamental challenge of processing natural language
• Pervasive across all language levels
https://flickr.com
• Attachment. “I saw a man with a telescope.”
• Scope. “I didn’t buy a car.”
• Coordination. “If you love money problems show up.”
• Speech act. “Have you emptied the dishwasher?”
Introduction to NLP I Overview © Wachsmuth 2023 18
Challenges
Missing Phonetics and Phonology
I never said she stole my money. Someone else said it, but I didn’t.
I never said she stole my money. I simply didn’t ever say it.
I never said she stole my money. I might have implied it in some way.
But I never explicitly said it.
I never said she stole my money. I just said she probably borrowed it.
I never said she stole my money. I said she stole someone else’s money.
I never said she stole my money . I said she stole something of mine.
But not my money.
Colloquial language
Special phrases
• Tricky entities. “Let it Be was recorded”, “mutation of the for gene”, ...
• Idioms. “get cold feet”, “lose face”, ...
• Neologisms. “unfriend”, “retweet”, “hangry”, ...
Tricky segmentation
• Hyphens. “the New York-New Haven Railroad”
• Punctuation. “She was a Dr. I was not.”
• Whitespaces. “ ”, “Just.Do.It.”
Introduction to NLP I Overview © Wachsmuth 2023 20
Challenges
Need for Context
World knowledge
• “Maja and Gabriella are sisters.” vs. “Maja and Gabriella are mothers.”
Do Maja and Gabriella belong to the same family?
Domain dependency
• “Read the book!”
Positive sentiment in a book review. Positive in a movie review?
Language dependency
• “Bad”
Sentiment in English? In German (means “bath”)? In French (does not exist)?
In Japanese (not even the characters exist)? ...
Effectiveness issues
• Effectiveness. The extent to which the output of a method is correct
• Methods may not be effective enough for use in real-life applications.
Efficiency issues
• Efficiency. The run-time, space, or energy consumption of a method
• Methods may not be efficient enough when applied to big text amounts.
Robustness issues
• Robustness. The effectiveness of a method across domains of text
• Methods may not be robust enough on data different from training data.
Introduction to NLP I Overview © Wachsmuth 2023 22
Approaches
Approaches
Need for data
• NLP methods are meant to tackle specific analysis or synthesis tasks.
• To this end, they operationalize expert rules and/or statistical patterns.
• Rules and patterns are derived from analyses of training data.
Text corpus
https://pixabay.com
• A collection of real-world texts with known properties,
compiled to study a language problem
• NLP methods are developed and tested on corpora.
Annotation
• An annotation marks a text or a span of text that represents an instance
of a particular type of information.
• Annotations represent meta-information about the marked parts.
• The texts in a corpus are often annotated for the problem to be studied.
Dataset
• A sub-corpus used for developing and/or evaluating methods.
• Typical: a training set for development, a validation set for evaluation
during development, and a test set for the final evaluation
Introduction to NLP I Overview © Wachsmuth 2023 25
Approaches
Inference Techniques
Knowledge-based inference
• Inference is based on manually-encoded expert knowledge.
• Knowledge is represented by rules, lexicons, grammars, and similar.
We will see respective NLP techniques in the earlier part of this course.
Statistical inference
• Inference is based on statistical patterns found in training data.
• Patterns capture frequencies and/or manually-defined text features.
We will see first respective NLP techniques in the later part of this course.
Neural inference
• Inference is based on statistical patterns found in training data.
• Patterns are automatically encoded in neural networks.
Respective NLP techniques are treated in our master courses.
Pipeline approach
• The default way to realize an inference process is in form of a pipeline.
• A pipeline sequentially applies a set of NLP algorithms to a given text.
Alternatives
• Joint approach. Tackle multiple analysis/synthesis tasks simultaneously
• Neural approach. Operates on the raw input text (or tokens)
Even with these, some kind of pipeline is used for most inference processes.
Introduction to NLP I Overview © Wachsmuth 2023 27
Approaches
Development and Evaluation
Input
• Task. An NLP task to be tackled
• Text corpus. A corpus, split into development and evaluation datasets
Output
• Approach. An NLP approach that tackles the given task
• Results. Empirical effectiveness measurements of the approach
Challenges
• Natural language is ambiguous in several ways
• Understanding requires context and world knowledge
• NLP aims to be effective, efficient, and robust
This course
• Teaches how to develop basic NLP methods input
text
Sentence
splitting
Tokenization Part-of-spech
tagging
Phrase
chunking