Creating
transcription helper
functions
S P OK EN LAN GUAGE P ROCES S IN G IN P YTH ON
Daniel Bourke
Machine Learning Engineer/YouTube
Creator
Exploring audio les
# Import os module
import os
# Check the folder of audio files
os.listdir("acme_audio_files")
(['call_1.mp3',
'call_2.mp3',
'call_3.mp3',
'call_4.mp3'])
SPOKEN LANGUAGE PROCESSING IN PYTHON
Preparing for the proof of concept
import speech_recognition as sr
from pydub import AudioSegment
# Import call 1 and convert to .wav
call_1 = AudioSegment.from_file("acme_audio_files/call_1.mp3")
call_1.export("acme_audio_files/call_1.wav", format="wav")
# Transcribe call 1
recognizer = sr.Recognizer()
call_1_file = sr.AudioFile("acme_audio_files/call_1.wav")
with call_1_file as source:
call_1_audio = recognizer.record(call_1_file)
recognizer.recognize_google(call_1_audio)
SPOKEN LANGUAGE PROCESSING IN PYTHON
Functions we'll create
convert_to_wav() converts non- .wav les to .wav les.
show_pydub_stats() shows the audio attributes of a .wav le.
transcribe_audio() uses recognize_google() to transcribe a .wav le.
SPOKEN LANGUAGE PROCESSING IN PYTHON
Creating a le format conversion function
# Create function to convert audio file to wav
def convert_to_wav(filename):
"Takes an audio file of non .wav format and converts to .wav"
# Import audio file
audio = AudioSegment.from_file(filename)
# Create new filename
new_filename = filename.split(".")[0] + ".wav"
# Export file as .wav
audio.export(new_filename, format="wav")
print(f"Converting {filename} to {new_filename}...")
SPOKEN LANGUAGE PROCESSING IN PYTHON
Using the le format conversion function
convert_to_wav("acme_studios_audio/call_1.mp3")
Converting acme_audio_files/call_1.mp3 to acme_audio_files/call_1.wav...
SPOKEN LANGUAGE PROCESSING IN PYTHON
Creating an attribute showing function
def show_pydub_stats(filename):
"Returns different audio attributes related to an audio file."
# Create AudioSegment instance
audio_segment = AudioSegment.from_file(filename)
# Print attributes
print(f"Channels: {audio_segment.channels}")
print(f"Sample width: {audio_segment.sample_width}")
print(f"Frame rate (sample rate): {audio_segment.frame_rate}")
print(f"Frame width: {audio_segment.frame_width}")
print(f"Length (ms): {len(audio_segment)}")
print(f"Frame count: {audio_segment.frame_count()}")
SPOKEN LANGUAGE PROCESSING IN PYTHON
Using the attribute showing function
show_pydub_stats("acme_audio_files/call_1.wav")
Channels: 2
Sample width: 2
Frame rate (sample rate): 32000
Frame width: 4
Length (ms): 54888
Frame count: 1756416.0
SPOKEN LANGUAGE PROCESSING IN PYTHON
Creating a transcribe function
# Create a function to transcribe audio
def transcribe_audio(filename):
"Takes a .wav format audio file and transcribes it to text."
# Setup a recognizer instance
recognizer = sr.Recognizer()
# Import the audio file and convert to audio data
audio_file = sr.AudioFile(filename)
with audio_file as source:
audio_data = recognizer.record(audio_file)
# Return the transcribed text
return recognizer.recognize_google(audio_data)
SPOKEN LANGUAGE PROCESSING IN PYTHON
Using the transcribe function
transcribe_audio("acme_audio_files/call_1.wav")
"hello welcome to Acme studio support line my name is Daniel how can I best help
you hey Daniel this is John I've recently bought a smart from you guys and I know
that's not good to hear John let's let's get your cell number and then we
can we can set up a way to fix it for you one number for 1757 varies how long do
you reckon this is going to take about an hour now while John we're going to try
our best hour I will we get the sealing member will start up this support case
I'm just really really really really I've been trying to contact 34 been put on
hold more than an hour and half so I'm not really happy I kind of wanna get this
issue 6 is fossil"
SPOKEN LANGUAGE PROCESSING IN PYTHON
Let's practice!
S P OK EN LAN GUAGE P ROCES S IN G IN P YTH ON
Sentiment analysis
on spoken language
text
S P OK EN LAN GUAGE P ROCES S IN G IN P YTH ON
Daniel Bourke
Machine Learning Engineer/YouTube
Creator
Installing sentiment analysis libraries
$ pip install nltk
# Download required NLTK packages
import nltk
nltk.download("punkt")
nltk.download("vader_lexicon")
SPOKEN LANGUAGE PROCESSING IN PYTHON
Sentiment analysis with VADER
# Import sentiment analysis class
from nltk.sentiment.vader import SentimentIntensityAnalyzer
# Create sentiment analysis instance
sid = SentimentIntensityAnalyzer()
# Test sentiment analysis on negative text
print(sid.polarity_scores("This customer service is terrible."))
{'neg': 0.437, 'neu': 0.563, 'pos': 0.0, 'compound': -0.4767}
SPOKEN LANGUAGE PROCESSING IN PYTHON
Sentiment analysis on transcribed text
# Transcribe customer channel of call_3
call_3_channel_2_text = transcribe_audio("call_3_channel_2.wav")
print(call_3_channel_2_text)
"hey Dave is this any better do I order products are currently on July 1st and I haven't
received the product a three-week step down this parable 6987 5"
# Sentiment analysis on customer channel of call_3
sid.polarity_scores(call_3_channel_2_text)
{'neg': 0.0, 'neu': 0.892, 'pos': 0.108, 'compound': 0.4404}
SPOKEN LANGUAGE PROCESSING IN PYTHON
Sentence by sentence
call_3_paid_api_text = "Okay. Yeah. Hi, Diane. This is paid on this call and obvi..."
# Import sent tokenizer
from nltk.tokenize import sent_tokenize
# Find sentiment on each sentence
for sentence in sent_tokenize(call_3_paid_api_text):
print(sentence)
print(sid.polarity_scores(sentence))
SPOKEN LANGUAGE PROCESSING IN PYTHON
Sentence by sentence
Okay.
{'neg': 0.0, 'neu': 0.0, 'pos': 1.0, 'compound': 0.2263}
Yeah.
{'neg': 0.0, 'neu': 0.0, 'pos': 1.0, 'compound': 0.296}
Hi, Diane.
{'neg': 0.0, 'neu': 1.0, 'pos': 0.0, 'compound': 0.0}
This is paid on this call and obviously the status of my orders at three weeks ago,
and that service is terrible.
{'neg': 0.129, 'neu': 0.871, 'pos': 0.0, 'compound': -0.4767}
Is this any better?
{'neg': 0.0, 'neu': 0.508, 'pos': 0.492, 'compound': 0.4404}
Yes...
SPOKEN LANGUAGE PROCESSING IN PYTHON
Time to code!
S P OK EN LAN GUAGE P ROCES S IN G IN P YTH ON
Named entity
recognition on
transcribed text
S P OK EN LAN GUAGE P ROCES S IN G IN P YTH ON
Daniel Bourke
Machine Learning Engineer/YouTube
Creator
Installing spaCy
# Install spaCy
$ pip install spacy
# Download spaCy language model
$ python -m spacy download en_core_web_sm
SPOKEN LANGUAGE PROCESSING IN PYTHON
Using spaCy
import spacy
# Load spaCy language model
nlp = spacy.load("en_core_web_sm")
# Create a spaCy doc
doc = nlp("I'd like to talk about a smartphone I ordered on July 31st from your
Sydney store, my order number is 40939440. I spoke to Georgia about it last week.")
SPOKEN LANGUAGE PROCESSING IN PYTHON
spaCy tokens
# Show different tokens and positions
for token in doc:
print(token.text, token.idx)
I 0
'd 1
like 4
to 9
talk 12
about 17
a 23
smartphone 25...
SPOKEN LANGUAGE PROCESSING IN PYTHON
spaCy sentences
# Show sentences in doc
for sentences in doc.sents:
print(sentence)
I'd like to talk about a smartphone I ordered on July 31st from your Sydney store,
my order number is 4093829.
I spoke to one of your customer service team, Georgia, yesterday.
SPOKEN LANGUAGE PROCESSING IN PYTHON
spaCy named entities
Some of spaCy's built-in named entities:
PERSON People, including ctional.
ORG Companies, agencies, institutions, etc.
GPE Countries, cities, states.
PRODUCT Objects, vehicles, foods, etc. (Not services.)
DATE Absolute or relative dates or periods.
TIME Times smaller than a day.
MONEY Monetary values, including unit.
CARDINAL Numerals that do not fall under another type.
SPOKEN LANGUAGE PROCESSING IN PYTHON
spaCy named entities
# Find named entities in doc
for entity in doc.ents:
print(entity.text, entity.label_)
July 31st DATE
Sydney GPE
4093829 CARDINAL
one CARDINAL
Georgia GPE
yesterday DATE
SPOKEN LANGUAGE PROCESSING IN PYTHON
Custom named entities
# Import EntityRuler class
from spacy.pipeline import EntityRuler
# Check spaCy pipeline
print(nlp.pipeline)
[('tagger', <spacy.pipeline.pipes.Tagger at 0x1c3aa8a470>),
('parser', <spacy.pipeline.pipes.DependencyParser at 0x1c3bb60588>),
('ner', <spacy.pipeline.pipes.EntityRecognizer at 0x1c3bb605e8>)]
SPOKEN LANGUAGE PROCESSING IN PYTHON
Changing the pipeline
# Create EntityRuler instance
ruler = EntityRuler(nlp)
# Add token pattern to ruler
ruler.add_patterns([{"label":"PRODUCT", "pattern": "smartphone"}])
# Add new rule to pipeline before ner
nlp.add_pipe(ruler, before="ner")
# Check updated pipeline
nlp.pipeline
SPOKEN LANGUAGE PROCESSING IN PYTHON
Changing the pipeline
[('tagger', <spacy.pipeline.pipes.Tagger at 0x1c1f9c9b38>),
('parser', <spacy.pipeline.pipes.DependencyParser at 0x1c3c9cba08>),
('entity_ruler', <spacy.pipeline.entityruler.EntityRuler at 0x1c1d834b70>),
('ner', <spacy.pipeline.pipes.EntityRecognizer at 0x1c3c9cba68>)]
SPOKEN LANGUAGE PROCESSING IN PYTHON
Testing the new pipeline
# Test new entity rule
for entity in doc.ents:
print(entity.text, entity.label_)
smartphone PRODUCT
July 31st DATE
Sydney GPE
4093829 CARDINAL
one CARDINAL
Georgia GPE
yesterday DATE
SPOKEN LANGUAGE PROCESSING IN PYTHON
Let's rocket and
practice spaCy!
S P OK EN LAN GUAGE P ROCES S IN G IN P YTH ON
Classifying
transcribed speech
with Sklearn
S P OK EN LAN GUAGE P ROCES S IN G IN P YTH ON
Daniel Bourke
Machine Learning Engineer/YouTube
creator
Inspecting the data
# Inspect post purchase audio folder
import os
post_purchase_audio = os.listdir("post_purchase")
print(post_purchase_audio[:5])
['post-purchase-audio-0.mp3',
'post-purchase-audio-1.mp3',
'post-purchase-audio-2.mp3',
'post-purchase-audio-3.mp3',
'post-purchase-audio-4.mp3']
SPOKEN LANGUAGE PROCESSING IN PYTHON
Converting to wav
# Loop through mp3 files
for file in post_purchase_audio:
print(f"Converting {file} to .wav...")
# Use previously made function to convert to .wav
convert_to_wav(file)
Converting post-purchase-audio-0.mp3 to .wav...
Converting post-purchase-audio-1.mp3 to .wav...
Converting post-purchase-audio-2.mp3 to .wav...
Converting post-purchase-audio-3.mp3 to .wav...
Converting post-purchase-audio-4.mp3 to .wav...
SPOKEN LANGUAGE PROCESSING IN PYTHON
Transcribing all phone call excerpts
# Transcribe text from wav files
def create_text_list(folder):
text_list = []
# Loop through folder
for file in folder:
# Check for .wav extension
if file.endswith(".wav"):
# Transcribe audio
text = transcribe_audio(file)
# Add transcribed text to list
text_list.append(text)
return text_list
SPOKEN LANGUAGE PROCESSING IN PYTHON
Transcribing all phone call excerpts
# Convert post purchase audio to text
post_purchase_text = create_text_list(post_purchase_audio)
print(post_purchase_text[:5])
['hey man I just water product from you guys and I think is amazing but I leave a little
'these clothes I just bought from you guys too small is there anyway I can change the s
"I recently got these pair of shoes but they're too big can I change the size",
"I bought a pair of pants from you guys but they're way too small",
"I bought a pair of pants and they're the wrong colour is there any chance I can change
SPOKEN LANGUAGE PROCESSING IN PYTHON
Organizing transcribed text
import pandas as pd
# Create post purchase dataframe
post_purchase_df = pd.DataFrame({"label": "post_purchase", "text": post_purchase_text})
# Create pre purchase dataframe
pre_purchase_df = pd.DataFrame({"label": "pre_purchase", "text": pre_purchase_text})
# Combine pre purchase and post purhcase
df = pd.concat([post_purchase_df, pre_purchase_df])
# View the combined dataframe
df.head()
SPOKEN LANGUAGE PROCESSING IN PYTHON
Organizing transcribed text
label text
0 post_purchase yeah hello someone this morning delivered a pa...
1 post_purchase my shipment arrived yesterday but it's not the...
2 post_purchase hey my name is Daniel I received my shipment y...
3 post_purchase hey mate how are you doing I'm just calling in...
4 pre_purchase hey I was wondering if you know where my new p...
SPOKEN LANGUAGE PROCESSING IN PYTHON
Building a text classi er
# Import text classification packages
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer, TfidfTransformer
from sklearn.model_selection import train_test_split
# Split data into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
X=df["text"],
y=df["label"],
test_size=0.3)
SPOKEN LANGUAGE PROCESSING IN PYTHON
Naive Bayes Pipeline
# Create text classifier pipeline
text_classifier = Pipeline([
("vectorizer", CountVectorizer()),
("tfidf", TfidfTransformer()),
("classifier", MultinomialNB())
])
# Fit the classifier pipeline on the training data
text_classifier.fit(X_train, y_train)
SPOKEN LANGUAGE PROCESSING IN PYTHON
Not so Naive
# Make predictions and compare them to test labels
predictions = text_classifier.predict(X_test)
accuracy = 100 * np.mean(predictions == y_test.label)
print(f"The model is {accuracy:.2f}% accurate.")
The model is 97.87% accurate.
SPOKEN LANGUAGE PROCESSING IN PYTHON
Let's practice!
S P OK EN LAN GUAGE P ROCES S IN G IN P YTH ON
Congratulations!
S P OK EN LAN GUAGE P ROCES S IN G IN P YTH ON
Daniel Bourke
Machine Learning Engineer/YouTube
creator
What you've done
1. Converted audio les into soundwaves with Python and NumPy .
2. Transcribed speech with speech_recognition .
3. Prepared and manipulated audio les using PyDub .
4. Built a spoken language processing pipeline with NLTK , spaCy and sklearn .
SPOKEN LANGUAGE PROCESSING IN PYTHON
What next?
Practice your skills with a project of your own.
Check out speech_recognition 's Microphone() class.
SPOKEN LANGUAGE PROCESSING IN PYTHON
One last transcription
one_last_transcription = transcribe_audio("congratulations.wav")
print(one_last_transcription)
Congratlutions on finishing the Spoken Language Processing with Python course!
You should be proud.
Now get out there and recognize some speech!
SPOKEN LANGUAGE PROCESSING IN PYTHON
Keep learning!
S P OK EN LAN GUAGE P ROCES S IN G IN P YTH ON
