Scribd
Upload
17 views4 pages

Unit 5

Machine learning text analysis notes

Uploaded by

2902snehashinde
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
17 views4 pages

Unit 5

Machine learning text analysis notes

Uploaded by

2902snehashinde
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 4

5.

Text Analysis

Basic Text Processing with Python.


1. Text Primitives:
• Document: A large collection of text.
• Sentences: Subdivisions of a document, formed by collections of words.
• Words: Units of meaning within sentences.
• Characters: The smallest units of text, forming words.

2.String Operations:
• Concatenation: Combine two strings using the + operator.
• Repetition: Repeat a string multiple times using the * operator.
• Membership Test: Check if a character exists in a string using in and not in
operators.

3.String Comparisons:
• isalpha(): Check if the string contains only alphabets.
• isalnum(): Check if the string contains only alphabets and numbers.
• isdigit(): Check if the string contains only digits.
• isdecimal(): Check if the string contains only decimal characters.
• islower(): Check if the string contains only lowercase characters.
• isupper(): Check if the string contains only uppercase characters.
• isnumeric(): Check if the string contains only numeric characters.
• startswith(): Check if the string starts with a specified substring.
• endswith(): Check if the string ends with a specified substring.

4.String Conversions:
• capitalize(): Convert the first character to uppercase and the rest to lowercase.
• title(): Convert the first character of each word to uppercase.
• lower(): Convert all characters to lowercase.
• upper(): Convert all characters to uppercase.
• swapcase(): Swap the case of each character.
• casefold(): Perform case folding, a more aggressive lowercasing for comparisons.

5.String Manipulations:
• count(): Count the occurrences of a substring in the string.
• replace(): Replace all occurrences of a substring with a new one.
• find(): Find the index of the first occurrence of a substring.
• rfind(): Find the index of the last occurrence of a substring.
• join(): Join strings in a sequence with a specified separator.
• splitlines(): Split the string into separate lines.
• lstrip(): Remove leading whitespaces or specified characters.
Regular Expression.
1.Introduction:
1. Regular expression is a powerful tool in any language to match the text patterns.
2. Python also supports regular expression.
3. Python regular expression operations are supported by module re.
4. To use regular expression first we need to import 're' module.
import re
5. To perform regular expression search, we will follow this format:
matchset = re.search(pattern,text)
6. Here, pattern refers to the rule we formed for matching and text contains string in which
we want to perform the search.
7. If search goes successful, match object is returned otherwise None.
8. example:
text2 = "This news article is published on month:Jan"
matchResult = re.search(r'month:\w\w\w',text2)
if matchResult:
print('Pattern exists ', matchResult.group())
else:
print('Pattern not exists')
9. In above example we want to search for month followed by : and three characters.
10. If it contains, result will be stored in matchResult object.
11. We can print the result by using matchResult.group() method.
12. r which is used in the beginning of the pattern is to handle raw strings.

2. some general rules used to make patterns:


1. Characters can be used as they appear. We can put them, and they match directly (Ex. -
a, B,4 etc.)
2. . (dot) is used to match with a single character.
3. \w is used to match any word character [a-zA-Z0-9_]. \W is used to match any non-word
character.
4. \s is used to match single whitespace character (space, newline, tab). \S is used to
match single non whitespace character.
5. \d is used to match single digit [0-9].
6. ^ is used to match start of the string.
7. $ is used to match end of the string.
8. + is used to check one or more occurrence of the pattern.
9. * is used to check zero or more occurrence of the pattern.
10. ? is used to check either zero or one occurrence of the pattern.
11. Square brackets can be used to match a set of words.
12. findall() - this function is used to find all occurrence of a specific pattern.
Example:
tweet = 'I am learning #datascience and it is awesome. #python #machinelearning'
hastags = re.findall(r'#\w+', tweet)
for tag in hastags:
print(tag)
Natural Language Processing.
1. Introduction:
1. Natural language is any language which is used for communication between humans e.g
Marathi, Hindi, English, German, etc.
2. Different types of manipulation can be done on natural language text.
3. Counting words and their frequencies, part of speech tagging, parsing sentences,
identifying entities, relationship between entities are different types of tasks covered in
natural language processing.
4. To perform natural language processing with python we use nltk (Natural Language
Toolkit).
5. It is open-source library written in python. It supports most of the natural language
tasks.
6. We start by importing the library:
import ntlk
7. nltk has good number of text corpora.
8. First, we need to download them to do some processing.
nltk.download()
9. Now to get a list of corpora, we can use following statement:
from nltk.book import *
10. This statement shows list of corpora available. We can print any text.
text3
11. Output: <Text: The Book of Gensis>.

2. Stemming, Lemmatization & Tokenization:


2.1. Stemming:
1. It is the process of finding the root word, hence reduce conflicts and convert words to
their base words.
2. For example, words like fishes, fishing, fisher all has the root word fish.
3. We can use porter stemmer available in nltk library.
fishwords = ['fish','Fishing','Fishes']
prt = nltk.PorterStemmer()
ls = [prt.stem(i) for i in fishwords]
print(ls)
4. Output: [u'fish', u'fish', u'fish']
5. Stemming depends on the type of work. Sometimes it makes sense to do it, sometimes
not. We have to take decision based on the problem statement.
2.2. Lemmatization:
1. It is the process to convert words into their actual dictionary form.
2. In nltk WordNetLemmatizer() is present to do this task.
3. We can understand the process by following code:
fishwords = ['fishes','Fishings','Fishes']
WNlemma = nltk.WordNetLemmatizer()
ls = [WNlemma.lemmatize(i) for i in fishwords]
print(ls)
4. Output: [u'fish', 'Fishings', 'Fishes']

2.3. Tokenization:
1. This can be done by split() function available in python.
2. But if we want to do it more clearly, we can use nltk tokenization.
3. Let’s understand this by following code:
text2 = "Why are you so intelligent?"
words = text2.split(' ')
print(words)
print(nltk.word_tokenize(text2))
4. Output:
['Why', 'are', 'you', 'so', 'intelligent?']
['Why', 'are', 'you', 'so', 'intelligent', '?']
5. Our first split function combines '?' with the previous word. But when we did it by using
nltk, it create separate word.

You might also like