Scribd
Upload
154 views

Natural Language Processing With Python & NLTK Cheat Sheet: by Via

This document provides a cheat sheet summary of key Natural Language Processing techniques using Python and the NLTK library. It covers topics such as text handling, corpus access, tokenization, sentence parsing, part-of-speech tagging, named entity recognition, text classification, lemmatization and stemming, and using regular expressions with Pandas for tasks like extraction and replacement. The cheat sheet is intended as a concise reference guide for common NLP pre-processing, analysis and machine learning tasks.

Uploaded by

Ashwani Rathee
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
154 views

Natural Language Processing With Python & NLTK Cheat Sheet: by Via

This document provides a cheat sheet summary of key Natural Language Processing techniques using Python and the NLTK library. It covers topics such as text handling, corpus access, tokenization, sentence parsing, part-of-speech tagging, named entity recognition, text classification, lemmatization and stemming, and using regular expressions with Pandas for tasks like extraction and replacement. The cheat sheet is intended as a concise reference guide for common NLP pre-processing, analysis and machine learning tasks.

Uploaded by

Ashwani Rathee
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 2

Natural Language Processing with Python & nltk Cheat Sheet

by RJ Murray (murenei) via cheatography.com/58736/cs/15485/

Handling Text Sentence Parsing

text=​'Some words' assign string g=nlt​k.d​ata.lo​ad(​'gr​amm​ar.c​fg') Load a grammar from a file

list(​text) Split text into character tokens g=nlt​k.C​FG.f​ro​mst​rin​g("""..."" Manually define grammar
")
set(t​ext) Unique tokens
parse​r=n​ltk.Ch​art​Par​ser(g) Create a parser out of the
len(t​ext) Number of characters
grammar

trees​=pa​rse​r.p​ars​e_a​ll(​text)
Accessing corpora and lexical resources
for tree in trees: ... print tree
from nltk.c​orpus import import Corpus​Reader object
brown from nltk.c​orpus import treebank

brown.wo​rds​(te​xt_id) Returns pretok​enised document as list treeb​ank.pa​rse​d_s​ent​s('​wsj​_00​0 Treebank parsed sentences


of words 1.m​rg')

brown.fi​lei​ds() Lists docs in Brown corpus


Text Classi​fic​ation
brown.ca​teg​ori​es() Lists categories in Brown corpus
from sklear​n.f​eat​ure​_ex​tra​cti​on.text import
Tokeni​zation CountV​ect​orizer, TfidfV​ect​orizer

text.s​pl​it(​" ") Split by space vect=​Cou​ntV​ect​ori​zer​().f​it​(X_​tr Fit bag of words model to


ain) data
nltk.w​or​d_t​oke​niz​er(​text) nltk in-built word tokenizer
vect.g​et​_fe​atu​re_​nam​es() Get features
nltk.s​en​t_t​oke​niz​e(doc) nltk in-built sentence tokenizer
vect.t​ra​nsf​orm​(X_​train) Convert to doc-term matrix

Lemmat​ization & Stemming


Entity Recogn​ition (Chunk​ing​/Ch​inking)
input​="List listed lists listing Different suffixes
listin​gs" g="NP: {<D​T>?​<JJ​>*<​NN>​}" Regex chunk grammar

words​=in​put.lo​wer​().s​plit(' ') Normalize (lower​case) cp=nl​tk.R​eg​exp​Par​ser(g) Parse grammar


words
ch=cp.pa​rse​(po​s_s​ent) Parse tagged sent. using grammar
porte​r=n​ltk.Po​rte​rSt​emmer Initialise Stemmer
print​(ch) Show chunks
[port​er.s​tem(t) for t in words] Create list of stems
ch.dr​aw() Show chunks in IOB tree
WNL=n​ltk.Wo​rdN​etL​emm​ati​zer() Initialise WordNet
cp.ev​alu​ate​(te​st_​sents) Evaluate against test doc
lemmatizer
sents​=nl​tk.c​or​pus.tr​eeb​ank.ta​gge​d_s​ents()
[WNL.l​em​mat​ize(t) for t in words] Use the lemmatizer
print​(nl​tk.n​e_​chu​nk(​sent)) Print chunk tree
Part of Speech (POS) Tagging

nltk.h​el​p.u​pen​n_t​ags​et Lookup definition for a POS tag


(​'MD')

nltk.p​os​_ta​g(w​ords) nltk in-built POS tagger

<use an altern​ative tagger to illustrate


ambigu​ity>

By RJ Murray (murenei) Published 28th May, 2018. Sponsored by CrosswordCheats.com


cheatography.com/murenei/ Last updated 29th May, 2018. Learn to solve cryptic crosswords!
tutify.com.au Page 1 of 2. http://crosswordcheats.com
Natural Language Processing with Python & nltk Cheat Sheet
by RJ Murray (murenei) via cheatography.com/58736/cs/15485/

RegEx with Pandas & Named Groups

df=pd.Da​taF​ram​e(t​ime​_sents, column​s=[​'te​xt'])

df['t​ext​'].s​tr.sp​lit​().s​tr.len()

df['t​ext​'].s​tr.co​nta​ins​('w​ord')

df['t​ext​'].s​tr.co​unt​(r'​\d')

df['t​ext​'].s​tr.fi​nda​ll(​r'\d')

df['t​ext​'].s​tr.re​pla​ce(​r'​\w+d​ay\b', '???')

df['t​ext​'].s​tr.re​pla​ce(​r'(​\w)', lambda x: x.grou​ps(​)


[0​][:3])

df['t​ext​'].s​tr.ex​tra​ct(​r'(​\d?​\d)​:(​\d\d)')

df['t​ext​'].s​tr.ex​tra​cta​ll(​r'(​(\d​?\d​):(​\d\d) ?
([ap]​m))')

df['t​ext​'].s​tr.ex​tra​cta​ll(​r'(​?P<​dig​its​>\d)')

By RJ Murray (murenei) Published 28th May, 2018. Sponsored by CrosswordCheats.com


cheatography.com/murenei/ Last updated 29th May, 2018. Learn to solve cryptic crosswords!
tutify.com.au Page 2 of 2. http://crosswordcheats.com

You might also like