NLTK - Natural Language Processing in Python

Natural Language Processing and Machine LearningUsing PythonShankar AmbadyMicrosoft New England Research and Development Center, December 14, 2010

Example FilesHosted on Githubhttps://github.com/shanbady/NLTK-Boston-Python-Meetup

What is “Natural Language Processing”?Where is this stuff used?The Machine learning paradoxA look at a few key termsQuick start – creating NLP apps in Python

What is Natural Language Processing? Computer aided text analysis of human language.

The goal is to enable machines to understand human language and extract meaning from text.

It is a field of study which falls under the category of machine learning and more specifically computational linguistics.

The “Natural Language Toolkit” is a python module that provides a variety of functionality that will aide us in processing text.Natural language processing is heavily used throughout all web technologies

ContextLittle sister: What’s your name?Me: Uhh….Shankar..?Sister: Can you spell it?Me: yes. S-H-A-N-K-A…..

Sister: WRONG! It’s spelled “I-T”

Ambiguity“I shot the man with ice cream.“A man with ice cream was shot

A man had ice cream shot at himLanguage translation is a complicated matter!Go to: http://babel.mrfeinberg.com/The problem with communication is the illusion that it has occurred

NLTK - Natural Language Processing in Python

The problem with communication is the illusion that it has occurredThe problem with communication is the illusion, which developed itEPIC FAIL

The “Human Test” Turing testA test proposed to demonstrate that truly intelligent machines capable of understanding and comprehending human language should be indistinguishable from humans performing the same task.I am alsohumanI am ahuman

Setting up NLTKSource downloads available for mac and linux as well as installable packages for windows.Currently only available for Python 2.5 – 2.6http://www.nltk.org/download`easy_install nltk`PrerequisitesNumPySciPy

First stepsNLTK comes with packages of corpora that are required for many modules. Open a python interpreter:importnltknltk.download() If you do not want to use the downloader with a gui (requires TKInter module)Run: python -m nltk.downloader <name of package or “all”>

You may individually select packages or download them in bulk.

Part of Speech Taggingfromnltkimportpos_tag,word_tokenizesentence1='this is a demo that will show you how to detects parts of speech with little effort using NLTK!'tokenized_sent=word_tokenize(sentence1)printpos_tag(tokenized_sent)[('this', 'DT'), ('is', 'VBZ'), ('a', 'DT'), ('demo', 'NN'), ('that', 'WDT'), ('will', 'MD'), ('show', 'VB'), ('you', 'PRP'), ('how', 'WRB'), ('to', 'TO'), ('detects', 'NNS'), ('parts', 'NNS'), ('of', 'IN'), ('speech', 'NN'), ('with', 'IN'), ('little', 'JJ'), ('effort', 'NN'), ('using', 'VBG'), ('NLTK', 'NNP'),('!', '.')]

Penn Bank Part-of-Speech TagsSource: http://www.ai.mit.edu/courses/6.863/tagdef.html

NLTK Textnltk.clean_html(rawhtml)from nltk.corpus import brownfrom nltk import Textbrown_words = brown.words(categories='humor')brownText = Text(brown_words)brownText.collocations()brownText.count("car")brownText.concordance("oil")brownText.dispersion_plot(['car', 'document', 'funny', 'oil'])brownText.similar('humor')

Find similar terms (word definitions) using Wordnetimportnltkfromnltk.corpusimportwordnetaswnsynsets=wn.synsets('phone')print[str(syns.definition)forsynsinsynsets] 'electronic equipment that converts sound into electrical signals that can be transmitted over distances and then converts received signals back into sounds‘'(phonetics) an individual sound unit of speech without concern as to whether or not it is a phoneme of some language‘'electro-acoustic transducer for converting electric signals into sounds; it is held over or inserted into the ear‘'get or try to get into communication (with someone) by telephone'

Meronyms and Holonyms are better described in relation to computer science terms as:Meronym terms: “has a” relationshipHolonym terms: “part of” relationshipHyponym terms: “Is a” relationshipMeronyms and holonyms are oppositesHyponyms and hypernyms are opposites

Cheese, beef, tomato, and bread are meronyms of burger

Going back to the previous example …from nltk.corpus import wordnet as wnsynsets = wn.synsets('phone')print [str( syns.definition ) for syns in synsets]“syns.definition” can be modified to output hypernyms , meronyms, holonyms etc:

Source:http://www.sjsu.edu/faculty/hahn.koo/teaching/ling115/lecture_notes/ling115_wordnet.pdf

fromnltk.corpusimportwordnetaswnsynsets=wn.synsets('car')print[str(syns.part_meronyms())forsynsinsynsets][Synset('gasoline_engine.n.01'), Synset('car_mirror.n.01'), Synset('third_gear.n.01'), Synset('hood.n.09'), Synset('automobile_engine.n.01'), Synset('grille.n.02'),

fromnltk.corpusimportwordnetaswnsynsets=wn.synsets('wing')print[str(syns.part_holonyms())forsynsinsynsets][Synset('airplane.n.01')][Synset('division.n.09')][Synset('bird.n.02')][Synset('car.n.01')][Synset('building.n.01')]

importnltkfromnltk.corpusimportwordnetaswnsynsets=wn.synsets('trees')print[str(syns.part_meronyms())forsynsinsynsets]synset('burl.n.02')

synset('limb.n.02')fromnltk.corpusimportwordnetaswnforhypernyminwn.synsets('robot')[0].hypernym_paths()[0]:printhypernym.lemma_names['entity']['physical_entity']['object', 'physical_object']['whole', 'unit'] ['artifact', 'artefact'] ['instrumentality', 'instrumentation']['device'] ['mechanism'] ['automaton', 'robot', 'golem']

Feeling lonely?Eliza is there to talk to you all day! What human could ever do that for you??fromnltk.chatimportelizaeliza.eliza_chat()……starts the chatbotTherapist---------Talk to the program by typing in plain English, using normal upper-and lower-case letters and punctuation. Enter "quit" when done.========================================================================Hello. How are you feeling today?

Englisch to German to Englisch to German……fromnltk.bookimport*babelize_shell()Babel> the internet is a series of tubesBabel> germanBabel> run0> the internet is a series of tubes1> das Internet ist eine Reihe SchlSuche2> the Internet is a number of hoses3> das Internet ist einige SchlSuche4> the Internet is some hosesBabel>

Do you Speak Girbbhsi??Take the following meaningful textA new study in the journal Animal Behavior shows that dogs rely a great deal on face recognition to tell their own person from other people. Researchers describe how dogs in the study had difficulty recognizing their owners when two human subjects had their faces covered.Let’s have NLTK analyze and generate some gibberish!importnltkwords=‘text'tokens=nltk.word_tokenize(words)text=nltk.Text(tokens)printtext.generate()

Results May Vary but these were mineA new study in the study had difficulty recognizing their owners when two human subjects had their faces covered . their owners when two human subjects had their faces covered . on face recognition to tell their own person from other people. Researchers describe how dogs in the journal Animal Behavior shows that dogs rely a great deal on face recognition to tell their own person from other people. Researchers describe how dogs in the study had difficulty recognizingtheir owners when two human subjects had their faces covered . subjects had their faces covered . dogs rely a greatA new study in the journal Animal Behavior shows that dogs rely a great deal on face recognition to tell their own person from other people. Researchers describe how dogs in the study had difficulty recognizing their owners when two human subjects had their faces covered.

How Similar?#!/usr/bin/env pythonfromnltk.corpusimportwordnetaswnsimilars =[]Aword=‘language'Bword=‘barrier‘

“how similar” (continued)# grab synsets of each wordsynsetsA=wn.synsets(Aword)synsetsB=wn.synsets(Bword)groupA=[wn.synset(str(synset.name))forsynsetinsynsetsA]groupB=[wn.synset(str(synset.name))forsynsetinsynsetsB]

path_similarity()“Path Distance Similarity: Return a score denoting how similar two word senses are, based on the shortest path that connects the senses in the is-a (hypernym/hypnoym) taxonomy.”wup_similarity()“Wu-Palmer Similarity: Return a score denoting how similar two word senses are, based on the depth of the two senses in the taxonomy and that of their Least Common Subsumer (most specific ancestor node).”Source: http://nltk.googlecode.com/svn/trunk/doc/api/nltk.corpus.reader.wordnet.WordNetCorpusReader-class.html

“how similar” (continued)forssetaingroupA:forssetbingroupB:path_similarity=sseta.path_similarity(ssetb)wup_similarity=sseta.wup_similarity(ssetb)ifpath_similarityisnotNone:similars.append({'path':path_similarity,'wup':wup_similarity,'wordA':sseta,'wordB':ssetb,'wordA_definition':sseta.definition,'wordB_definition':ssetb.definition})

“how similar” (continued)similars=sorted( similars,key=\lambdaitem: item['path'] , reverse=True)foriteminsimilars:printitem['wordA'], "- “, item['wordA_definition']printitem['wordB'], "-", item['wordB_definition']print'Path similarity - ',item['path'],"\n“

Languagethe cognitive processes involved in producing and understanding linguistic communicationSimilarity: 0.111~Barrierany condition that makes it difficult to make progress or to achieve an objective

“how similar” (continued)It trickles down from thereSynset('linguistic_process.n.02') the cognitive processes involved in producing and understanding linguistic communicationSynset('barrier.n.02') any condition that makes it difficult to make progress or to achieve an objectivePath similarity - 0.111111111111 Synset('language.n.05') the mental faculty or power of vocal communicationSynset('barrier.n.02') any condition that makes it difficult to make progress or to achieve an objectivePath similarity - 0.111111111111 Synset('language.n.01') a systematic means of communicating by the use of sounds or conventional symbolsSynset('barrier.n.02') any condition that makes it difficult to make progress or to achieve an objectivePath similarity - 0.1 Synset('language.n.01') a systematic means of communicating by the use of sounds or conventional symbolsSynset('barrier.n.03') anything serving to maintain separation by obstructing vision or accessPath similarity - 0.1

Poetic ProgrammingWe will create a program to extract “Haikus” from any given English text.A haiku is a poem in which each stanza consists of three lines. The first line has 5 syllables, the second has 7 and the last line has 5.

Inspired by a GitHub project “Haiku Finder” :https://github.com/jdf/haikufinderWe will be re-implementing this program and adding a few other little features.You will need The nltk_contrib package from Google Code: http://code.google.com/p/nltk/downloads/list

A few paragraphs of text that we will use to create haikus from“poetic programming” (continued)fromnltk_contrib.readability.textanalyzerimportsyllables_enfromnltk.corpusimportcmudict,wordnetaswnfromnltkimportword_tokenizeimportretextchunk=''‘ # we will make Ted Stevens sound more poeticThey want to deliver vast amounts of information over the Internet.And again, the Internet is not something that you just dump somethingon.It's not a big truck. It's a series of tubes. And if you don't understand, those tubes can be filled and if they are filled, when you put your message in, it gets in line and it's going to be delayed by anyone that puts into that tube enormous amounts of material, enormous amounts of material '''

“poetic programming” (continued)textchunk +='‘‘# throw in a few “bush-isms”I want to share with you an interesting program for two reasons, one, it's interesting, and two, my wife thought of it or has actually been involved with it; she didn't think of it. But she thought of it for this speech.This is my maiden voyage.My first speech since I was the president of the United States and I couldn't think of a betterplace to give it than Calgary, Canada.''‘

“poetic programming” (continued)poem=''wordmap=[]words=word_tokenize(textchunk)foriter,wordinenumerate(words): # if it is a word, add a append a space to itifword.isalpha():word+=" "syls=syllables_en.count(word) wordmap.append((word,syls))Tokenize the wordsNLTK function to count syllables

“poetic programming” (continued)Define a function to provide a fallback word in case we end up with lines that do not have the syllable count we need.deffindSyllableWord(word,syllableSize):synsets=wn.synsets(word)forsynsinsynsets:name=syns.namelemmas=syns.lemma_namesforwordstringinlemmas:if(syllables_en.count(wordstring)==syllableSizeandwordstring!=word):return{'word':word,'syllable':syllableSize}return{'word':word,'syllable':syllables_en.count(word)}Given a word , this function tries to find similar words from WordNet that match the required syllable size

“poetic programming” (continued)We loop through the each word keeping tally of syllables and breaking each line when it reaches the appropriate thresholdlineNo=1charNo=0tally=0forsyllabicwordinwordmap:s=syllabicword[1]wordtoAdd=syllabicword[0]iflineNo==1:iftally<5:iftally+int(s)>5andwordtoAdd.isalpha():num=5-tallysimilarterm=findSyllableWord(wordtoAdd,num)wordtoAdd=similarterm['word']s=similarterm['syllable']tally+=int(s)poem+=wordtoAddelse:poem+=" ---"+str(tally)+"\n"ifwordtoAdd.isalpha():poem+=wordtoAddtally=slineNo=2iflineNo==2:…. Abridged

“poetic programming” (continued)printpoemIts not perfect but its still pretty funny!I want to share with ---5you an interesting program ---8for two reasons ,one ---5it 's interesting ---5and two ,my wife thought of it ---7or has actually ---5been involved with ---5it ;she didn't think of it. But she thought ---7of it for this speech. ---5This is my maiden ---5voyage. My first speech since I ---7was the president of ---5 …. Abridged

Let’s build something even cooler

Lets write a spam filter! A program that analyzes legitimate emails “Ham” as well as “spam” and learns the features that are associated with each. Once trained, we should be able to run this program on incoming mail and have it reliably label each one with the appropriate category.

What you will needNLTK (of course) as well as the “stopwords” corpusA good dataset of emails; Both spam and hamPatience and a cup of coffee (these programs tend to take a while to complete)

Finding Great Data: The Enron EmailsA dataset of 200,000+ emails made publicly available in 2003 after the Enron scandal.Contains both spam and actual corporate ham mail.For this reason it is one of the most popular datasets used for testing and developing anti-spam software.The dataset we will use is located at the following url:http://labs-repos.iit.demokritos.gr/skel/i-config/downloads/enron-spam/preprocessed/It contains a list of archived files that contain plaintext emails in two folders , Spam and Ham.

Extract one of the archives from the site into your working directory. Create a python script, lets call it “spambot.py”.Your working directory should contain the “spambot” script and the folders “spam” and “ham”.“Spambot.py”fromnltkimportword_tokenize,\ WordNetLemmatizer,NaiveBayesClassifier\,classify,MaxentClassifierfromnltk.corpusimportstopwordsimportrandomimportos,glob,re

“Spambot.py” (continued)wordlemmatizer = WordNetLemmatizer()commonwords = stopwords.words('english') hamtexts=[]spamtexts=[]forinfileinglob.glob(os.path.join('ham/','*.txt')):text_file=open(infile,"r")hamtexts.append(text_file.read())text_file.close()forinfileinglob.glob(os.path.join('spam/','*.txt')):text_file=open(infile,"r")spamtexts.append(text_file.read())text_file.close()load common English words into liststart globbing the files into the appropriate lists

“Spambot.py” (continued)mixedemails=([(email,'spam')foremailinspamtexts]mixedemails+= [(email,'ham')foremailinhamtexts])random.shuffle(mixedemails)label each item with the appropriate label and store them as a list of tuplesFrom this list of random but labeled emails, we will defined a “feature extractor” which outputs a feature set that our program can use to statistically compare spam and ham. lets give them a nice shuffle

“Spambot.py” (continued)defemail_features(sent):features={}wordtokens=[wordlemmatizer.lemmatize(word.lower())forwordinword_tokenize(sent)]forwordinwordtokens:ifwordnotincommonwords:features[word]=Truereturnfeaturesfeaturesets=[(email_features(n),g)for(n,g)inmixedemails]Normalize wordsIf the word is not a stop-word then lets consider it a “feature”Let’s run each email through the feature extractor and collect it in a “featureset” list

The features you select must be binary features such as the existence of words or part of speech tags (True or False).

To use features that are non-binary such as number values, you must convert it to a binary feature. This process is called “binning”.

If the feature is the number 12 the feature is: (“11<x<13”, True)“Spambot.py” (continued)Lets grab a sampling of our featureset. Changing this number will affect the accuracy of the classifier. It will be a different number for every classifier, find the most effective threshold for your dataset through experimentation.size=int(len(featuresets)*0.7)train_set,test_set=featuresets[size:],featuresets[:size]classifier=NaiveBayesClassifier.train(train_set)Using this threshold grab the first n elements of our featureset and the last n elements to populate our “training” and “test” featuresetsHere we start training the classifier using NLTK’s built in Naïve Bayes classifier

“Spambot.py” (continued)print classifier.labels()This will output the labels that our classifier will use to tag new data['ham', 'spam']The purpose of create a “training set” and a “test set” is to test the accuracy of our classifier on a separate sample from the same data source.printclassify.accuracy(classifier,test_set)0.983589566419

classifier.show_most_informative_features(20)SpamHam

“Spambot.py” (continued)WhileTrue:featset=email_features(raw_input("Enter text to classify: "))printclassifier.classify(featset)We can now directly input new email and have it classified as either Spam or Ham

A few notes:The quality of your input data will affect the accuracy of your classifier.

The threshold value that determines the sample size of the feature set will need to be refined until it reaches its maximum accuracy. This will need to be adjusted if training data is added, changed or removed.A few notes:The accuracy of this dataset can be misleading; In fact our spambot has an accuracy of 98% - but this only applies to Enron emails. This is known as “over-fitting” .

Try classifying your own emails using this trained classifier and you will notice a sharp decline in accuracy.Chunking

Complete sentences are composed of two or more “phrases”. Noun phrase:Jack and Jill went up the hill Prepositional phrase:Contains a noun, preposition and in most cases an adjectiveThe NLTK book is on the table but perhaps it is best kept in a bookshelf Gerund Phrase:Phrases that contain “–ing” verbsJack fell down and broke his crown and Jill came tumbling after

Take the following sentence …..Jack and Jillwent up the hillNoun phraseNoun Phrase

Chunkers will get us this far:[ Jack and Jill ] went up [ the hill ]Chunk tokens are non-recursive – meaning, there is no overlap when chunkingThe recursive form for the same sentence is:( Jack and Jill went up (the hill ) )

NLTK - Natural Language Processing in Python

More Related Content

What's hot (20)

Similar to NLTK - Natural Language Processing in Python (20)

Recently uploaded (20)

NLTK - Natural Language Processing in Python

Editor's Notes