Download the Full Ebook and Access More Features - ebooknice.

com

(Ebook) Advanced Data Analytics Using Python :

With Architectural Patterns, Text and Image
Classification, and Optimization Techniques by
Sayan Mukhopadhyay; Pratip Samanta ISBN
9781484280058, 1484280059
https://ebooknice.com/product/advanced-data-analytics-using-
python-with-architectural-patterns-text-and-image-
classification-and-optimization-techniques-49612318

OR CLICK HERE

DOWLOAD EBOOK

Download more ebook instantly today at https://ebooknice.com

 Instant digital products (PDF, ePub, MOBI) ready for you
Download now and discover formats that fit your needs...

Start reading on any device today!

(Ebook) Advanced Data Analytics Using Python: With

Architectural Patterns, Text and Image Classification, and
Optimization Techniques [2nd Edition] by Sayan
Mukhopadhyay, Pratip Samanta ISBN 9781484280041,
https://ebooknice.com/product/advanced-data-analytics-using-python-
1484280040
with-architectural-patterns-text-and-image-classification-and-
optimization-techniques-2nd-edition-47321334
ebooknice.com

(Ebook) Advanced Data Analytics Using Python: With Machine

Learning, Deep Learning and NLP Examples by Mukhopadhyay,
Sayan ISBN 9781484234495, 1484234499
https://ebooknice.com/product/advanced-data-analytics-using-python-
with-machine-learning-deep-learning-and-nlp-examples-55670484

ebooknice.com

(Ebook) Biota Grow 2C gather 2C cook by Loucas, Jason;

Viles, James ISBN 9781459699816, 9781743365571,
9781925268492, 1459699815, 1743365578, 1925268497
https://ebooknice.com/product/biota-grow-2c-gather-2c-cook-6661374

ebooknice.com

(Ebook) SAT II Success MATH 1C and 2C 2002 (Peterson's SAT

II Success) by Peterson's ISBN 9780768906677, 0768906679

https://ebooknice.com/product/sat-ii-success-
math-1c-and-2c-2002-peterson-s-sat-ii-success-1722018

ebooknice.com
(Ebook) Matematik 5000+ Kurs 2c Lärobok by Lena
Alfredsson, Hans Heikne, Sanna Bodemyr ISBN 9789127456600,
9127456609
https://ebooknice.com/product/matematik-5000-kurs-2c-larobok-23848312

ebooknice.com

(Ebook) Master SAT II Math 1c and 2c 4th ed (Arco Master

the SAT Subject Test: Math Levels 1 & 2) by Arco ISBN
9780768923049, 0768923042
https://ebooknice.com/product/master-sat-ii-math-1c-and-2c-4th-ed-
arco-master-the-sat-subject-test-math-levels-1-2-2326094

ebooknice.com

(Ebook) Cambridge IGCSE and O Level History Workbook 2C -

Depth Study: the United States, 1919-41 2nd Edition by
Benjamin Harrison ISBN 9781398375147, 9781398375048,
1398375144, 1398375047
https://ebooknice.com/product/cambridge-igcse-and-o-level-history-
workbook-2c-depth-study-the-united-states-1919-41-2nd-edition-53538044

ebooknice.com

(Ebook) Advanced Analytics with Transact-SQL: Exploring

Hidden Patterns and Rules in Your Data by Dejan Sarka ISBN
9781484271728, 1484271726
https://ebooknice.com/product/advanced-analytics-with-transact-sql-
exploring-hidden-patterns-and-rules-in-your-data-33557194

ebooknice.com

(Ebook) Python Data Analytics: Data Analysis and Science

Using Pandas, Matplotlib and the Python Programming
Language by Nelli Fabio ISBN 9781484209592, 1484209591
https://ebooknice.com/product/python-data-analytics-data-analysis-and-
science-using-pandas-matplotlib-and-the-python-programming-
language-38169124
ebooknice.com
Sayan Mukhopadhyay and Pratip Samanta

Advanced Data Analytics Using Python

With Architectural Patterns, Text and Image
Classification, and Optimization Techniques
2nd ed.
Sayan Mukhopadhyay
Kolkata, West Bengal, India

Pratip Samanta
Kolkata, West Bengal, India

ISBN 978-1-4842-8004-1 e-ISBN 978-1-4842-8005-8

https://doi.org/10.1007/978-1-4842-8005-8

© Sayan Mukhopadhyay, Pratip Samanta 2018, 2023

Apress Standard

The use of general descriptive names, registered names, trademarks,

service marks, etc. in this publication does not imply, even in the
absence of a specific statement, that such names are exempt from the
relevant protective laws and regulations and therefore free for general
use.

The publisher, the authors and the editors are safe to assume that the
advice and information in this book are believed to be true and accurate
at the date of publication. Neither the publisher nor the authors or the
editors give a warranty, express or implied, with respect to the material
contained herein or for any errors or omissions that may have been
made.

This Apress imprint is published by the registered company APress

Media, LLC, part of Springer Nature.
The registered company address is: 1 New York Plaza, New York, NY
10004, U.S.A.
The reason for the success of this book is that it has original research, so I
dedicate it to the person from whom I learned how to do research: Dr.
Debnath Pal, IISc.
—Sayan Mukhopadhyay
Introduction
We are living in the data science/artificial intelligence era. To thrive in
this environment, where data drives decision-making in everything
from business to government to sports and entertainment, you need
the skills to manage and analyze huge amounts of data. Together we
can use this data to make the world better for everyone. In fact, humans
have yet to find everything we can do using this data. So, let us explore!
Our objective for this book is to empower you to become a leader in
this data-transformed era. With this book you will learn the skills to
develop AI applications and make a difference in the world.
This book is intended for advanced user, because we have
incorporated some advanced analytics topics. Important machine
learning models and deep learning models are explained with coding
exercises and real-world examples.
All the source code used in this book is available for download at
https://github.com/apress/advanced-data-analytics-
python-2e.
Happy reading!
Any source code or other supplementary material referenced by the
author in this book is available to readers on GitHub
(github.com/apress). For more detailed information, please visit
www.apress.com/source-code.
Acknowledgments
Thanks to Labonic Chakraborty (Ripa) and Soumili Chakraborty.
Table of Contents
Chapter 1:​A Birds Eye View to AI System
OOP in Python
Calling Other Languages in Python
Exposing the Python Model as a Microservice
High-Performance API and Concurrent Programming
Choosing the Right Database
Summary
Chapter 2:​ETL with Python
MySQL
How to Install MySQLdb?​
Database Connection
INSERT Operation
READ Operation
DELETE Operation
UPDATE Operation
COMMIT Operation
ROLL-BACK Operation
Normal Forms
First Normal Form
Second Normal Form
Third Normal Form
Elasticsearch
Connection Layer API
Neo4j Python Driver
neo4j-rest-client
 In-Memory Database
MongoDB (Python Edition)
Import Data into the Collection
Create a Connection Using pymongo
Access Database Objects
Insert Data
Update Data
Remove Data
Cloud Databases
Pandas
ETL with Python (Unstructured Data)
Email Parsing
Topical Crawling
Summary
Chapter 3:​Feature Engineering and Supervised Learning
Dimensionality Reduction with Python
Correlation Analysis
Principal Component Analysis
Mutual Information
Classifications with Python
Semi-Supervised Learning
Decision Tree
Which Attribute Comes First?​
Random Forest Classifier
Naïve Bayes Classifier
Support Vector Machine
 Nearest Neighbor Classifier
Sentiment Analysis
Image Recognition
Regression with Python
Least Square Estimation
Logistic Regression
Classification and Regression
Intentionally Bias the Model to Over-Fit or Under-Fit
Dealing with Categorical Data
Summary
Chapter 4:​Unsupervised Learning:​Clustering
K-Means Clustering
Choosing K:​The Elbow Method
Silhouette Analysis
Distance or Similarity Measure
Properties
General and Euclidean Distance
Squared Euclidean Distance
Distance Between String-Edit Distance
Similarity in the Context of a Document
Types of Similarity
Example of K-Means in Images
Preparing the Cluster
Thresholding
Time to Cluster
Revealing the Current Cluster
 Hierarchical Clustering
Bottom-Up Approach
Distance Between Clusters
Top-Down Approach
Graph Theoretical Approach
How Do You Know If the Clustering Result Is Good?​
Summary
Chapter 5:​Deep Learning and Neural Networks
Backpropagation
Backpropagation Approach
Other Algorithms
TensorFlow
Network Architecture and Regularization Techniques
Updatable Model and Transfer Learning
Recurrent Neural Network
LSTM
Reinforcement Learning
TD0
TDλ
Example of Dialectic Learning
Convolution Neural Networks
Summary
Chapter 6:​Time Series
Classification of Variation
Analyzing a Series Containing a Trend
Curve Fitting
 Removing Trends from a Time Series
Analyzing a Series Containing Seasonality
Removing Seasonality from a Time Series
By Filtering
By Differencing
Transformation
To Stabilize the Variance
To Make the Seasonal Effect Additive
To Make the Data Distribution Normal
Stationary Time Series
Stationary Process
Autocorrelation and the Correlogram
Estimating Autocovariance and Autocorrelation Functions
Time-Series Analysis with Python
Useful Methods
Autoregressive Processes
Estimating Parameters of an AR Process
Mixed ARMA Models
Integrated ARMA Models
The Fourier Transform
An Exceptional Scenario
Missing Data
Summary
Chapter 7:​Analytics at Scale
Hadoop
MapReduce Programming
 Partitioning Function
Combiner Function
HDFS File System
MapReduce Design Pattern
A Notes on Functional Programming
Spark
PySpark
Updatable Machine Learning and Spark Memory Model
Analytics in the Cloud
Internet of Things
Essential Architectural Patterns for Data Scientists
Scenario 1:​Hot Potato Anti-Pattern
Scenario 2:​Proxy and Layering Patterns
Thank You
Index
About the Authors
Sayan Mukhopadhyay
has more than 13 years of industry
experience and has been associated with
companies such as Credit Suisse, PayPal,
CA Technologies, CSC, and Mphasis. He
has a deep understanding of applications
for data analysis in domains such as
investment banking, online payments,
online advertising, IT infrastructure, and
retail. His area of expertise is in applying
high-performance computing in
distributed and data-driven
environments such as real-time analysis,
high-frequency trading, and so on.
He earned his engineering degree in
electronics and instrumentation from
Jadavpur University and his master’s
degree in research in computational and data science from IISc in
Bangalore.

Pratip Samanta
is a principal AI engineer/researcher
with more than 11 years of experience.
He has worked for several software
companies and research institutions. He
has published conference papers and has
been granted patents in AI and natural
language processing. He is also
passionate about gardening and
teaching.
About the Technical Reviewer
Joos Korstanje
is a data scientist with more than five
years of industry experience in
developing machine learning tools, of
which a large part is forecasting models.
He currently works at Disneyland Paris
where he develops machine learning for
a variety of tools.
© Sayan Mukhopadhyay, Pratip Samanta 2023
S. Mukhopadhyay, P. Samanta, Advanced Data Analytics Using Python
https://doi.org/10.1007/978-1-4842-8005-8_1

1. A Birds Eye View to AI System

Sayan Mukhopadhyay1 and Pratip Samanta1

(1) Kolkata, West Bengal, India

In this book, we assume that you are familiar with Python programming. In this introductory chapter, we
explain why a data scientist should choose Python as a programming language. Then we highlight some
situations where Python may not be the ideal choice. Finally, we describe some best practices for application
development and give some coding examples that a data scientist may need in their day-to-day job.

OOP in Python
In this section, we explain some features of object-oriented programming (OOP) in a Python context.
The most basic element of any modern application is an object. To a programmer or architect, the world
is a collection of objects. Objects consist of two types of members: attributes and methods. Members can be
private, public, or protected. Classes are data types of objects. Every object is an instance of a class. A class
can be inherited in child classes. Two classes can be associated using composition.
Python has no keywords for public, private, or protected, so encapsulation (hiding a member from the
outside world) is not implicit in Python. Like C++, it supports multilevel and multiple inheritance. Like Java,
it has an abstract keyword. Classes and methods both can be abstract.
In the following code, we are describing an object-oriented question-answering system without any
machine learning. The program’s input is a set of dialogs in input.txt, as shown here:

glob is I
prok is V
pish is X
tegj is L
glob glob Silver is 34 Credits
glob prok Gold is 57800 Credits
pish pish Iron is 3910 Credits
how much is pish tegj glob glob ?
how many Credits is glob prok Silver ?
how many Credits is glob prok Gold ?
how many Credits is glob prok Iron ?
how much wood could a woodchuck chuck if a woodchuck could chuck wood?
Program has a knowledge base in config.txt.
I,1,roman
V,5,roman
X,10,roman
L,50,roman
C,100,roman
D,500,roman
M,1000,roman

Based on this input and the configuration program, the answer to the question is given in input.txt in
standard output, as shown here:
pish tegj glob glob is 42
glob prok Silver is 68 Credits
glob prok Gold is 57800 Credits
glob prok Iron is 782 Credits
I have no idea what you are talking about
The parsing logic is in the Observer class.

import operator

#this class verify the validity of input

class Observer(object):
#store frequecy of symbols
length = {}
#most frequent symbol
symbol = ''
#count of most frequent symbol
count = 0
#calling class
compiler = None

def __init__(self,cmpiler):
self.compiler = cmpiler

def initialize(self, arr):

for i in range(len(arr)):
self.length[arr[i]] = 0

#increase count for each occurence of symbol

def increment(self,symbol):
self.length[symbol] = self.length[symbol] + 1

#claculate most frequent symbol and it's count

def calculate(self):
self.symbol,self.count = max(self.length.items(),
key=operator.itemgetter(1))

#verify if wrong symbol is subtracted ie ( V, ..

def verifySubstract(self, current):
while current % 10 != 0:
current = current / 10
if current == 5:
raise Exception("Wrong Substraction")

def evaluate(self):
#check mximum repeatation is crossing the limit
if self.count > 3:
raise Exception("Repeat more than 3")
#symbol is proper or not
if self.symbol not in self.compiler.symbol_map:
raise Exception("Wrong Symbol")
#check if wrong symbol is repeated ie (V, ..
self.symbol,unit = self.compiler.evaluateSymbol(self.symbol)
while self.symbol % 10 != 0:
self.symbol = self.symbol / 10
if self.count > 1 and self.symbol == 5:
raise Exception("Wrong Symbol repeated")
 #checking if input sentence is proper or not
def evaluateSentence(self, line):
if "is" not in line:
return "I have no idea what you are talking about"
The compilation logic is in the compiler class, as shown here:

import sys

from observer import Observer

class compilerTrader(object):
#store mapping of symbols with score and unit
symbol_map = {}
#store the list of valid symbol
valid_values = []

#read the config and initialize the class member

def __init__(self, config_path):
with open(config_path) as f:
for line in f:
if ',' in line :
symbol, value, type = line.strip().split(',')
self.symbol_map[symbol] = float(value), type
self.valid_values.append(float(value))
f.close()

#evaluate the ultimate numerical score with unit for a symbol

def evaluateSymbol(self, symbol):
while symbol not in self.valid_values:
symbol,unit = self.symbol_map[symbol]
return float(symbol), unit

#compiling the info in line

def compile_super(self, line):
obs = Observer(self)
if 'is' in line:
fields = line.split(' is ')
value = fields[-1]
var = fields[0]
#if one symbol and one value
if ' ' not in var:
if value in self.symbol_map:
self.symbol_map[var] = int(self.symbol_map[value][0]) ,'rom
else:
#logic for value with unit
if ' ' in value:
fields = value.split(' ')
user_unit = fields[-1]
if ' ' not in var:
self.symbol_map[var] = int(fields[0]), user_unit
else:
#logic for multiple symbols in input
total = int(fields[0])
factor = 0
arr = var.split(' ')
obs.initialize(arr)
for i in range(len(arr)):
 obs.increment(arr[i])
if arr[i] in self.symbol_map and arr[i+1] in
self.symbol_map and i < len(arr) -1:
current, current_unit =
self.evaluateSymbol([arr[i]][0])
next, next_unit = self.evaluateSymbol([arr
[0])
if current >= next:
factor = factor + current
else:
obs.verifySubstract(current)
factor = factor - current
else:
if arr[i] in self.symbol_map:
current, current_unit =
self.evaluateSymbol([arr[i]][0])
factor = factor + current
else:
self.symbol_map[arr[i]] =
total/factor, user_unit
self.valid_values.append(total
obs.calculate()
obs.evaluate()
The answering logic is in the answer layer, which calls Observer and compiler. The answering
class inherits the compiler class.

import sys

from observer import Observer

from compiler import compilerTrader

class answeringTrader(compilerTrader):

def __init__(self, config_path):

super().__init__(config_path)

#compiling info in line

def compile(self, line):
super().compile_super(line)

#answering query in line

def answer(self, line):
obs = Observer(super())
if 'is' in line:
values = line.split(' is ')[-1]
ans = 0
arr = values.split(' ')
unit = ''
obs.initialize(arr)
for i in range(len(arr)):
if arr[i] in "?.,!;":
continue
obs.increment(arr[i])
if i < len(arr)-2:
if arr[i] in super().symbol_map and arr[i+1]
in super().symbol_map:
 current, current_unit =
super().evaluateSymbol([arr[i]][0])
next, next_unit =
super().evaluateSymbol([arr[i+1]][0])
if current >= next:
ans = ans + current
else:
if next_unit == 'roman':
obs.verifySubstract(current)
ans = ans - current
else:
ans = ans + current
else:
if arr[i] in super().symbol_map:
current,unit =
super().evaluateSymbol([arr[i]][0])
if unit != 'roman':
ans = ans * current
else:
ans = ans + current

obs.calculate()
obs.evaluate()
values = values.replace("?" , "is ")
if unit == 'roman':
unit = ''
return(values + str(ans) + ' ' + unit)
Finally, the main program calls the answering class and the observer, and then it performs the task
and does unit testing on the logic.

import sys
import unittest

sys.path.append('./answerLayer')
sys.path.append('./compilerLayer')
sys.path.append('./utilityLayer')

from answer import answeringTrader

from observer import Observer

#client interface for the framework

class ClientTrader(object):
trader = None
def __init__(self, config_path):
self.trader = answeringTrader(config_path)

#processing an input string

def process(self, input_string):
obs = Observer(self.trader)
valid = obs.evaluateSentence(input_string)
if valid is not None:
return valid
if input_string.strip()[-1] == '?' :
return self.trader.answer(input_string)
else:
return self.trader.compile(input_string)
#unit test cases
class TestTrader(unittest.TestCase):
trader = None

def setUp(self):
pass

#test case for non-roman symbol unit other than roman

def test_answer_unit(self):
ans = self.trader.process("how many Credits is glob prok Silver
?")
self.assertEqual(ans.strip(), "glob prok Silver is 68.0
Credits")

#test case with only roman symbol in unit case

def test_answer_roman(self):
ans = self.trader.process("how much is pish tegj glob glob ?")
self.assertEqual(ans.strip(), "pish tegj glob glob is 42.0")

#test case if repeatation of symbol is exceed max limit (3)

def test_exception_over_repeat(self):
with self.assertRaises(Exception) as context:
ans = self.trader.process("how much is pish tegj glob glob
glob glob ?")
self.assertTrue("Repeat more than 3" in context.exception)

#test case if wrong symbol repeated ie (V ..

def test_exception_unproper_repeat(self):
with self.assertRaises(Exception) as context:
ans = self.trader.process("how much is pish tegj D D ?")
self.assertTrue("Repeat more than 3" in context.exception)

#test case if wrong symbol substracted ie (V ...

def test_wrong_substraction(self):
with self.assertRaises(Exception) as context:
ans = self.trader.process("how much is V X ?")
self.assertTrue("Wrong Substraction" in context.exception)

#test case if query is not properly formatted

def test_wrong_format_query(self):
ans = self.trader.process("how much wood could a woodchuck chuck
if a woodchuck could chuck wood ?")
self.assertEqual(ans.strip(), "I have no idea what you are
talking about")

if __name__ == '__main__':
if len(sys.argv) != 3:
print("Usage is : " + sys.argv[0] + " <intput file
path> <config file path>")
exit(0)
tr = ClientTrader(sys.argv[2])
f = open(sys.argv[1])
for line in f:
response = tr.process(line.strip())
if response is not None:
print(response)
 TestTrader.trader = tr
unittest.main(argv = [sys.argv[0]], exit = False)
You can run this program with the following command:

python client.py input.txt config.txt

Calling Other Languages in Python

Now we will describe how to use other languages in Python. There are two examples here. The first is calling
R code from Python. R code is required for some use cases. For example, if you want a ready-made function
for the Holt-Winters method in a time series, it is difficult to perform in Python, but it is available in R. So,
you can call R code from Python using the rpy2 module, as shown here:

import rpy2.robjects as ro
ro.r('data(input)')
ro.r('x <-HoltWinters(input data frame)')

(You can use example data given in time series chapter.)

Sometimes you need to call Java code from Python. For example, say you are working on a name-entity
recognition problem in the field of natural language processing (NLP); some text is given as input, and you
have to recognize the names in the text. Python’s NLTK package does have a name-entity recognition
function, but its accuracy is not good. Stanford NLP is a better choice here, but it is written in Java. You can
solve this problem in two ways.
You can call Java at the command line using Python code. You need to install Java with the yum/at-get
install java command before calling it.
For Windows, it is recommended that you install the JRE from
https://adoptium.net/temurin/releases/?version=8. You can also install the JRE from
another distribution. The installation will automatically create JAVA_HOME. If it does not, you need to set
JAVA_HOME as the system variable, and the value should be the location of Java installation folder, for
example, JAVA_HOME=C:\Program Files\Eclipse Adoptium\jdk-8.0.345.1-hotspot\.

import subprocess
subprocess.call(['java','-
cp','*','edu.stanford.nlp.sentiment.SentimentPipeline','-file','foo.txt'])

Please place foo.txt in the same folder where you run the Python code.

You can expose Stanford NLP as a web service and call it as a service. (Before running this code, you’ll
need to download the Stanford nlp JAR file available with the book’s source code.)

nlp = StanfordCoreNLP('http://127.0.0.1:9000')
output = nlp.annotate(sentence, properties={
"annotators": "tokenize,ssplit,parse,sentiment",
"outputFormat": "json",
# Only split the sentence at End Of Line. We assume that this method only
takes in one single sentence.
"ssplit.eolonly": "true",
# Setting enforceRequirements to skip some annotators and make the process
faster
"enforceRequirements": "false"
})

You will see a more detailed example of Stanford NLP in Chapter 2.

Exposing the Python Model as a Microservice

You can expose the Python model as a microservice in the same way that your Python model can be used by
others to write their own code. The best way to do this is to expose your model as a web service. As an
example, the following code exposes a deep learning model using Flask:

from flask import Flask, request, g

from flask_cors import CORS
import tensorflow as tf
from sqlalchemy import *
from sqlalchemy.orm import sessionmaker
import pygeoip
from pymongo import MongoClient
import json
import datetime as dt
import ipaddress
import math
app = Flask(__name__)
CORS(app)
@app.before_request
def before():
db = create_engine('sqlite:///score.db')
metadata = MetaData(db)
g.scores = Table('scores', metadata, autoload=True)
Session = sessionmaker(bind=db)
g.session = Session()
client = MongoClient()
g.db = client.frequency
g.gi = pygeoip.GeoIP('GeoIP.dat')
sess = tf.Session()
new_saver = tf.train.import_meta_graph('model.obj.meta')
new_saver.restore(sess, tf.train.latest_checkpoint('./'))
all_vars = tf.get_collection('vars')
g.dropped_features = str(sess.run(all_vars[0]))
g.b = sess.run(all_vars[1])[0]
return
def get_hour(timestamp):
return dt.datetime.utcfromtimestamp(timestamp / 1e3).hour
def get_value(session, scores, feature_name, feature_value):
s = scores.select((scores.c.feature_name == feature_name) &
(scores.c.feature_value == feature_value))
rs = s.execute()
row = rs.fetchone()
if row is not None:
return float(row['score'])
else:
return 0.0
@app.route('/predict', methods=['POST'])
def predict():
input_json = request.get_json(force=True)
features =
['size','domain','client_time','device','ad_position','client_size',
'ip','root']
predicted = 0
feature_value = ''
for f in features:
if f not in g.dropped_features:
if f == 'ip':
 feature_value =
str(ipaddress.IPv4Address(ipaddress.ip_address(unicode(request.remote_addr))))
else:
feature_value = input_json.get(f)
if f == 'ip':
if 'geo' not in g.dropped_features:
geo =
g.gi.country_name_by_addr(feature_value)
predicted = predicted + get_value(g.session,
g.scores, 'geo', geo)

return str(math.exp(predicted + g.b)-1)

app.run(debug = True, host ='0.0.0.0')
This code exposes a deep learning model as a Flask web service. A JavaScript client will send the request
with web user parameters such as the IP address, ad size, ad position, and so on, and it will return the price
of the ad as a response. The features are categorical. You will learn how to convert them into numerical
scores in Chapter 3. These scores are stored in an in-memory database. The service fetches the score from
the database, sums the result, and replies to the client. This score will be updated real time in each iteration
of training of a deep learning model. It is using MongoDB to store the frequency of that IP address in that
site. It is an important parameter because a user coming to a site for the first time is really searching for
something, which is not true for a user where the frequency is greater than 5. The number of IP addresses is
huge, so they are stored in a distributed MongoDB database.

High-Performance API and Concurrent Programming

Flask is a good choice when you are building a general solution that is also a graphical user interface (GUI).
But if high performance is the most critical requirement of your application, then Falcon is the best choice.
The following code is an example of the same model shown previously exposed by the Falcon framework.
Another improvement we made in this code is that we implemented multithreading, so the code will be
executed in parallel. In addition to the Falcon-specific changes, you should note the major changes in
parallelizing the calling get_score function using a thread pool class.

import falcon
from falcon_cors import CORS
import json
from sqlalchemy import *
from sqlalchemy.orm import sessionmaker
import pygeoip
from pymongo import MongoClient
import json
import datetime as dt
import ipaddress
import math
from concurrent.futures import *
from sqlalchemy.engine import Engine
from sqlalchemy import event
import sqlite3
@event.listens_for(Engine, "connect")
def set_sqlite_pragma(dbapi_connection, connection_record):
cursor = dbapi_connection.cursor()
cursor.execute("PRAGMA cache_size=100000")
cursor.close()
class Predictor(object):
def __init__(self,domain):
db1 = create_engine('sqlite:///score_' + domain + '0test.db')
metadata1 = MetaData(db1)
 self.scores = Table('scores', metadata1, autoload=True)
client = MongoClient(connect=False,maxPoolSize=1)
self.db = client.frequency
self.gi = pygeoip.GeoIP('GeoIP.dat')
self.high = 1.2
self.low = .8
def get_hour(self,timestamp):
return dt.datetime.utcfromtimestamp(timestamp / 1e3).hour
def get_score(self, featurename, featurevalue):
pred = 0
s = self.scores.select((self.scores.c.feature_name ==
featurename) & (self.scores.c.feature_value == featurevalue))
rs = s.execute()
row = rs.fetchone()
if row is not None:
pred = pred + float(row['score'])
res = self.db.frequency.find_one({"ip" : ip})
freq = 1
if res is not None:
freq = res['frequency']
pred2, prob2 = self.get_score('frequency', str(freq))
return (pred1 + pred2), (prob1 + prob2)

conn = sqlite3.connect('multiplier.db')
cursor = conn.execute("SELECT high,low from multiplier
where domain='" + value + "'")
row = cursor.fetchone()
if row is not None:
self.high = row[0]
self.low = row[1]
return self.get_score(f, value)
def on_post(self, req, resp):
input_json = json.loads(req.stream.read(),encoding='utf-8')
input_json['ip'] = unicode(req.remote_addr)
pred = 1
prob = 1
with ThreadPoolExecutor(max_workers=8) as pool:
future_array = {
pool.submit(self.get_value,f,input_json[f]) : f for f in input_json}
for future in as_completed(future_array):
pred1, prob1 = future.result()
pred = pred + pred1
prob = prob - prob1
resp.status = falcon.HTTP_200
res = math.exp(pred)-1
if res < 0:
res = 0
prob = math.exp(prob)
if(prob <= .1):
prob = .1
if(prob >= .9):
prob = .9
multiplier = self.low + (self.high -self.low)*prob
pred = multiplier*pred
resp.body = str(pred)
cors =
CORS(allow_all_origins=True,allow_all_methods=True,allow_all_headers=True)
wsgi_app = api = falcon.API(middleware=[cors.middleware])
f = open('publishers1.list')
for domain in f:
domain = domain.strip()
p = Predictor(domain)
url = '/predict/' + domain
api.add_route(url, p)
Having covered design patterns in Python a bit, let’s now take a look at some essential architecture
patterns for data scientists.

Choosing the Right Database

Before we go, we’ll leave a note for manager on which database is best for which case.
A relational database (MySQL, Oracle, SQL Server) is the preferable choice when data is highly structured
and entities have a clear and strict connection. Mongo, on the other hand, is a better choice when data is
unstructured and unorganized.
Elastic Search or Solr is a better choice when data contains a lengthy textual field and you’re executing lots
of searches in a substring of the text field. With Elastic Search, you get a free data visualization tool called
Kibana as well as an ETL tool called Logstash, and full-stack data analytics solutions are fashionable.
Data must sometimes be represented as a graph. In that situation, a graph database is required. Neo4j is a
popular graph database that comes with many utility tools at a low price.
We occasionally require a quick application. In that situation, an in-memory database like SQLite can be
used. However, SQLite does not support updating your database from a remote host.
You’ll learn more about databases in Chapter 2.

Summary
In this chapter, we discussed fundamental engineering principles for data scientists, which are covered in
separate chapters. The question-answering example can help you understand how to organize your code.
The basic rule is to not put everything into one class. Divide your code into many categories and use parent-
child relationships where they exist. Then you learned how to use Python to call other languages’ code. We
provided two instances of R and Java code calls. Then we showed you how to expose your model as a REST
API and make it perform well by using concurrent programming. Following that, we covered significant
architectural patterns from data scientists.
© Sayan Mukhopadhyay, Pratip Samanta 2023
S. Mukhopadhyay, P. Samanta, Advanced Data Analytics Using Python
https://doi.org/10.1007/978-1-4842-8005-8_2

2. ETL with Python

Sayan Mukhopadhyay1 and Pratip Samanta1

(1) Kolkata, West Bengal, India

Every data science professional has to extract, transform, and load (ETL) data from different data
sources. In this chapter, we will discuss how to perform ETL with Python for a selection of
popular databases. For a relational database, we’ll cover MySQL. As an example of a document
database, we will cover Elasticsearch. For a graph database, we’ll cover Neo4j, and for NoSQL,
we’ll cover MongoDB. We will also discuss the Pandas framework, which was inspired by R’s data
frame concept.
ETL is based on a process in which data is extracted from multiple sources, transformed into
specific formats that involve cleaning enrichment, and finally loaded into its target destination.
The following are the details of each process:
1. Extract: During data extraction, source data is pulled from a variety of sources and moved to a
staging area, making the data available to subsequent stages in the ETL process. After that,
the data undergoes the cleaning and enrichment stage, also known as data cleansing.

2. Transform: In this stage, the source data is matched to the format of the target system. This
includes steps such as changing data types, combining fields, splitting fields, etc.

3. Load: This stage is the final ETL stage. Here, data is loaded into the data warehouse in an
automated manner and can be periodically updated. Once completed, the data is ready for
data analysis.

The previous processes are important in any data analytics work. Once the data goes through
the ETL processes, then it becomes possible to analysis the data, find insights, and so on.
We will discuss various types of ETL throughout this chapter. We discussed in Chapter 1 that
data is not an isolated thing. We need to load data from somewhere, which is a database. We need
to fetch the data from some application, which is extraction. In this chapter and the next, we will
discuss various feature engineering that transforms the data from one form to another.

MySQL
MySQLdb is an API in Python developed to work on top of the MySQL C interface.

How to Install MySQLdb?

First you need to install the Python MySQLdb module on your machine. Then run the following
script:

#!/usr/bin/python
import MySQLdb
If you get an import error exception, that means the module was not installed properly.
The following are the instructions to install the MySQL Python module:

$ gunzip MySQL-python-1.2.2.tar.gz
$ tar -xvf MySQL-python-1.2.2.tar
$ cd MySQL-python-1.2.2
$ python setup.py build
$ python setup.py install

You can download the tar.gz file from

https://dev.mysql.com/downloads/connector/python/. You need to download it to
your working folder.
For Windows, please select the MySQL installer file from
https://dev.mysql.com/downloads/installer/. Once it’s downloaded, double-click
the file to install it and select MySQL Connector/Python as one of the products to install. For
details, you can visit https://dev.mysql.com/doc/connector-
python/en/connector-python-installation-binary.xhtml.

Database Connection
Before connecting to a MySQL database, make sure you do the following:
1. You need to access a database called TEST with the sql "use test" command.

2. In TEST you need a table named STUDENT; use the command sql "create table
student(name varchar(20), sur_name varchar(20),roll_no int");.

3. STUDENT needs three fields: NAME, SUR_NAME, and ROLL_NO.

4. There needs to be a user in TEST that has complete access to the database.

If you do not do these steps properly, you will get an exception in the next Python code.

INSERT Operation
The following code carries out the SQL INSERT statement for the purpose of creating a record in
the STUDENT table:

#!/usr/bin/python
import MySQLdb
# Open database connection
db = MySQLdb.connect("localhost","user","passwd","TEST" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
# Prepare SQL query to INSERT a record into the database.
sql = """INSERT INTO STUDENT(NAME,
SUR_NAME, ROLL_NO)
VALUES ('Sayan', 'Mukhopadhyay', 1)"""
try:
# Execute the SQL command
 cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()

READ Operation
The following code fetches data from the STUDENT table and prints it:

#!/usr/bin/python
import MySQLdb
# Prepare SQL query to INSERT a record into the database.
sql = "SELECT * FROM STUDENT "
try:
# Execute the SQL command
cursor.execute(sql)
# Fetch all the rows in a list of lists.
results = cursor.fetchall()
for row in results:
fname = row[0]
lname = row[1]
id = row[2]
# Now print fetched result
Print( "name=%s,surname=%s,id=%d" % \
(fname, lname, id ))
except:
print "Error: unable to fecth data"

DELETE Operation
The following code deletes a row from TEST with id=1:

#!/usr/bin/python
import MySQLdb
# Prepare SQL query to DELETE required records
sql = "DELETE FROM STUDENT WHERE ROLL_NO =1"
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()

UPDATE Operation
The following code changes the lastname variable to Mukherjee, from Mukhopadhyay:
#!/usr/bin/python
import MySQLdb
# Prepare SQL query to UPDATE required records
sql = "UPDATE STUDENT SET SUR_NAME="Mukherjee"
WHERE SUR_NAME="Mukhopadhyay"
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()

COMMIT Operation
The commit operation provides its assent to the database to finalize the modifications, and after
this operation, there is no way that this can be reverted.

ROLL-BACK Operation
If you are not completely convinced about any of the modifications and you want to reverse them,
then you can apply the roll-back() method.
The following is a complete example of accessing MySQL data through Python. It will give the
complete description of the data stored in a CSV file or MySQL database.
This code asks for the data source type, either MySQL or text. For example, if MySQL asks for
the IP address, credentials, and database name and shows all tables in the database, it offers its
fields once the table is selected. Similarly, a text file asks for a path, and in the files it points to, all
the columns are shown to the user.

# importing files and reading config file

import MySQLdb
import sys
out = open('Config1.txt','w')
print ("Enter the Data Source Type:")
print( "1. MySql")
print ("2. Exit")
while(1):
data1 = sys.stdin.readline().strip()
if(int(data1) == 1):
out.write("source begin"+"\n"+"type=mysql\n")

# taking inputs from user

print ("Enter the ip:")

ip = sys.stdin.readline().strip()
out.write("host=" + ip + "\n")
print ("Enter the database name:")
db = sys.stdin.readline().strip()
out.write("database=" + db + "\n")
print ("Enter the user name:")
 usr = sys.stdin.readline().strip()
out.write("user=" + usr + "\n")
print ("Enter the password:")
passwd = sys.stdin.readline().strip()
out.write("password=" + passwd + "\n")

# making connection to and executing query

connection = MySQLdb.connect(ip, usr, passwd, db)

cursor = connection.cursor()
query = ("show tables")
cursor.execute(query)
data = cursor.fetchall()
tables = []

# appending data to the table

for row in data:

for field in row:
tables.append(field.strip())
for i in range(len(tables)):
print( i, tables[i])
tb = tables[int(sys.stdin.readline().strip())]
out.write("table=" + tb + "\n")
query = ("describe " + tb)
cursor.execute(query)
data = cursor.fetchall()
columns = []
for row in data:
columns.append(row[0].strip())
for i in range(len(columns)):
print( columns[i])
print "Not index choose the exact column names seperated
by coma"
cols = sys.stdin.readline().strip()
out.write("columns=" + cols + "\n")
cursor.close()
connection.close()
out.write("source end"+"\n")
print ("Enter the Data Source Type:")
print ("1. MySql") print ("2. Exit")
out.close()
sys.exit()
Before we go on to the topic of relational databases, let’s talk about database normalization.

Normal Forms
Database normal forms are the principles to organize your data in an optimum way.
Every table in a database can be in one of the normal forms that we’ll go over next. For the
primary key (PK) and foreign key (FK), you want to have as little repetition as possible. The rest of
the information should be taken from other tables.
 First normal form (1NF)
Second normal form (2NF)
Third normal form (3NF)

First Normal Form

There are no repeating groups in the first normal form since only single values are allowed at the
intersection of each row and column.
To get to first normal form, remove the repetitive groups, and establish two new relations to
normalize a connection with a repeating group.
For unique identification, the new connection’s PK is a combination of the old relation’s PK
and a feature from the newly formed relation.
To demonstrate the procedure for 1NF, we’ll use the Student_Grade_Report table, which
comes from a School database.

Student_Grade_Report (StudentNo, StudentName, Major, CourseNo,

CourseName, InstructorNo, InstructorName, InstructorLocation, Grade)

1. The recurring group in the Student Grade Report table contains the course information. A
student can enroll in a variety of courses.

2. Get rid of the group that keeps repeating itself. That’s each student’s course information in
this situation.

3. Determine your new table’s PK.

4. The attribute value must be identified uniquely by the PK (StudentNo and CourseNo).

Student (StudentNo, StudentName, Major)

StudentCourse (StudentNo, CourseNo, CourseName, InstructorNo,
InstructorName, InstructorLocation, Grade)

Second Normal Form

The relation must first be in 1NF for the second normal form. If and only if the PK contains a single
feature, the relationship is automatically in 2NF.
If the connection contains a composite PK, then each nonkey property must be completely
reliant on the entire PK, not just a portion of it (i.e., there can’t be any partial augmentation or
dependency).
A table must first be in 1NF before moving to 2NF.
1. As it has a single-column PK, the Student table is already in 2NF.

2. When looking at the Student Course table, you can observe that not all of the characteristics,
especially the course details, are completely dependent on the PK. The grade is the sole
attribute that is entirely reliant on xxx.

3. Locate the new table containing the course details.

4. Determine the new table’s PK.

 The three new tables are as follows:
Student (StudentNo, StudentName, Major)
CourseGrade (StudentNo, CourseNo, Grade)
CourseInstructor (CourseNo, CourseName, InstructorNo,
InstructorName, InstructorLocation)

Third Normal Form

The connection must be in second normal form to be in third normal form. All transitive
dependencies must be eliminated as well; a nonkey attribute cannot be functionally reliant on
another nonkey attribute.
This is the process for achieving 3NF:
1. From each table with a transitive relationship, remove all dependent characteristics in a
transitive relationship.

2. Make a new table with the dependence eliminated.

3. Inspect new and updated tables to ensure that each table has a determinant and that no
tables have improper dependencies.

Take a look at the four new tables:

Student (StudentNo, StudentName, Major)

CourseGrade (StudentNo, CourseNo, Grade)
Course (CourseNo, CourseName, InstructorNo)
Instructor (InstructorNo, InstructorName, InstructorLocation)

There should be no abnormalities in the third normal form at this point. For this example,
consider the dependency diagram in Figure 2-1. As previously said, the first step is to eliminate
repeated groupings.

Student (StudentNo, StudentName, Major)

StudentCourse (StudentNo, CourseNo, CourseName, InstructorNo,
InstructorName, InstructorLocation, Grade)

Figure 2-1 Dependency diagram

Review the dependencies in Figure 2-1, which summarizes the normalization procedure for
the School database.
 The following are the abbreviations used in Figure 2-1:
PD stands for partially dependent.
TD stands for transitive dependence.
FD stands for full dependency. (FD stands for functional dependence in most cases. Figure 2-1
is the only place where FD is used as an abbreviated form for full dependence.)
A relational database is valuable when structured data and a strict relationship between the
fields are maintained. But what if you do not have structured data in which a strict relationship
between fields has been maintained? That’s where Elasticsearch comes in.

Elasticsearch
You’ll find that data is often unstructured. Meaning, you may end up with a mix of image data,
sensor data, and other forms of data. To analyze this data, we first need to store it. MySQL or SQL-
based databases are not good at storing unstructured data. So here we introduce a different kind
of storage, which is mainly used to handle unstructured textual data.
Elasticsearch is a Lucene-based database, which makes it is easy to store and search text data.
Its query interface is a REST API endpoint. The Elasticsearch (ES) low-level client gives a direct
mapping from Python to ES REST endpoints. One of the big advantages of Elasticsearch is that it
provides a full-stack solution for data analysis in one place. Elasticsearch is the database. It has a
configurable front end called Kibana, a data collection tool called Logstash, and an enterprise
security feature called Shield.
This example has features called cat, cluster, indices, ingest, nodes, snapshot, and tasks that
translate to instances of CatClient, ClusterClient, IndicesClient, CatClient,
ClusterClient, IndicesClient, IngestClient, NodesClient, SnapshotClient,
NodesClient, SnapshotClient, and TasksClient, respectively. These instances are the
only supported way to get access to these classes and their methods.
You can specify your own connection class, which can be used by providing the
connection_class parameter.

# create connection to local host using the ThriftConnection

Es1=Elasticsearch(connection_class=ThriftConnection)

Installation commands for Elastic Search are given here:

curl -fsSL https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo

apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main"
| sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
sudo apt update
sudo apt install elasticsearch

You can start Elasticsearch in Ubuntu with these commands:

service elasticsearch start

service elasticsearch stop

You can check the status with these commands:

service elasticsearch status

# create connection that will automatically inspect the cluster to
get
# the list of active nodes. Start with nodes running on 'esnode1' and
# 'esnode2'
Es1=Elasticsearch(
['esnode1', 'esnode2'],
# sniff before doing anything
sniff_on_start=True,
# refresh nodes after a node fails to respond
sniff_on_connection_fail=True,
# and also every 30 seconds
sniffer_timeout=30
)

Different hosts can have different parameters (hostname, port number, SSL option); you can
use one dictionary per node to specify them.

# connect to localhost directly and

another node using SSL on port 443
# and an url_prefix. Note that ``port`` needs to be an int.
Es1=Elasticsearch([
{'host':'localhost'},
{'host':'othernode','port':443,'url_prefix':'es','use_ssl':True},
])

SSL client authentication is also supported (see Urllib3HttpConnection for a detailed

description of the options); an example is given here:

Es1=Elasticsearch(
['localhost:443','other_host:443'],
# turn on SSL
use_ssl=True,
# make sure we verify SSL certificates (off by default)
verify_certs=True,
# provide a path to CA certs on disk
ca_certs='path to CA_certs',
# PEM formatted SSL client certificate
client_cert='path to clientcert.pem',
# PEM formatted SSL client key
client_key='path to clientkey.pem'
)

Connection Layer API

Many classes are responsible for dealing with the Elasticsearch cluster. Here, the default
subclasses being utilized can be disregarded by handing over parameters to the
Elasticsearch class. Every argument belonging to the client will be added onto Transport,
ConnectionPool, and Connection.
As an example, if you want to use your own personal utilization of the
ConnectionSelector class, you just need to pass in the selector_class parameter.
 The entire API wraps the raw REST API with a high level of accuracy, which includes the
differentiation between the required and optional arguments to the calls. This implies that the
code makes a differentiation between positional and keyword arguments; I advise you to use
keyword arguments for all calls to be consistent and safe. An API call becomes successful (and will
return a response) if Elasticsearch returns a 2XX response. Otherwise, an instance of
TransportError (or a more specific subclass) will be raised. You can see other exceptions and
error states in exceptions. If you do not want an exception to be raised, you can always pass in an
ignore parameter with either a single status code that should be ignored or a list of them.

from elasticsearch import Elasticsearch

es=Elasticsearch()
# ignore 400 cause by IndexAlreadyExistsException when creating an
index
es.indices.create(index='test-index',ignore=400)
# ignore 404 and 400
es.indices.delete(index='test-index',ignore=[400,404])

Neo4j Python Driver

There are a variety of systems, such as network topology and social networks. However, when
difficulties are shown as a graph, they are quickly resolved. Neo4j is a database that stores data in
the form of a graph and executes queries through a graphical interface. The Neo4j Python driver is
supported by Neo4j and connects with the database through the binary protocol. It tries to remain
minimalistic but at the same time be idiomatic to Python.

pip install neo4j-driver

from neo4j.v1 import GraphDatabase, basic_auth
driver11 = GraphDatabase.driver("bolt://localhost",
auth=basic_auth("neo4j", "neo4j"))
session11 = driver11.session()
session11.run("CREATE (a:Person {name:'Sayan',
title:'Mukhopadhyay'})")
result 11= session11.run("MATCH (a:Person) WHERE a.name = 'Sayan'
RETURN a.name AS name, a.title AS title")
for recordi n resul11t:
print("%s %s"% (record["title"], record["name"]))
session.close()

neo4j-rest-client
The main objective of neo4j-rest-client is to make sure that the Python programmers already
using Neo4j locally through python-embedded are also able to access the Neo4j REST server. So,
the structure of the neo4j-rest-client API is completely in sync with python-embedded. But, a new
structure is brought in so as to arrive at a more Pythonic style and to augment the API with the
new features being introduced by the Neo4j team.

In-Memory Database
Another important class of databases is an in-memory database. This type stores and processes
the data in RAM. So, operations on the database are fast, and the data is volatile. SQLite is a
popular example of an in-memory database. In Python you need to use the sqlalchemy library to
operate on SQLite. In Chapter 1’s Flask and Falcon example, I showed you how to select data from
SQLite. Here I will show how to store a Pandas data frame in SQLite:

from sqlalchemy import create_engine

import sqlite3
conn = sqlite3.connect('multiplier.db')
conn.execute('''CREATE TABLE if not exists multiplier
(domain CHAR(50),
low REAL,
high REAL);''')
conn.close()
db_name = "your db name ""
disk_engine = create_engine(db_name)
df.to_sql('scores', disk_engine, if_exists='replace')

MongoDB (Python Edition)

MongoDB is an open-source document database designed for superior performance, easy
availability, and automatic scaling. MongoDB makes sure that object-relational mapping (ORM) is
not required to facilitate development. A document that contains a data structure made up of field
and value pairs is referred to as a record in MongoDB. These records are akin to JSON objects. The
values of fields may be comprised of other documents, arrays, and arrays of documents.

{
"_id":ObjectId("01"),
"address": {
"street":"Siraj Mondal Lane",
"pincode":"743145",
"building":"129",
"coord": [ -24.97, 48.68 ]
},
"borough":"Manhattan",

Import Data into the Collection

mongoimport can be used to place the documents into a collection in a database, within the
system shell or a command prompt. If the collection already exists in the database, the operation
will discard the original collection first.

mongoimport --DB test --collection restaurants --drop --file

~/downloads/primer-dataset.json

The mongoimport command is joined to a MongoDB instance running on localhost on port

27017. The --file option provides a way to import the data; here it’s
~/downloads/primer-dataset.json.
To import data into a MongoDB instance running on a different host or port, the hostname or
port needs to be mentioned specifically in the mongoimport command by including the --host
or --port option.
There is a similar load command in MySQL.

Create a Connection Using pymongo

To create a connection, do the following:

import MongoClient from pymongo.

Client11 = MongoClient()

If no argument is mentioned to MongoClient, then it will default to the MongoDB instance

running on the localhost interface on port 27017.
A complete MongoDB URL may be designated to define the connection, which includes the
host and port number. Let’s take a look at an example.
First, install Mongo using this command: yum/apt install mongo.
Then, launch MongoDB using this command: service mongo start.
The following code makes a connection to a MongoDB instance that runs on
mongodb0.example.net and port 27017:

Client11 = MongoClient("mongodb://myhostname:27017")

Access Database Objects

To assign the database named primer to the local variable DB, you can use either of the following
lines:

Db11 = client11.primer
db11 = client11['primer']

Collection objects can be accessed directly by using the dictionary style or the attribute access
from a database object, as shown in the following two examples:

Coll11 = db11.dataset
coll = db11['dataset']

Insert Data
You can place a document into a collection that doesn’t exist, and the following operation will
create the collection:

result=db.addrss.insert_one({<<your json >>)

Update Data
Here is how to update data:

result=db.address.update_one(
{"building": "129",
{"$set": {"address.street": "MG Road"}}
)

Remove Data
To expunge all documents from a collection, use this:

result=db.restaurants.delete_many({})

Cloud Databases
Even though the cloud has its own chapter, we’d like to provide you with an overview of cloud
databases, particularly databases for large data. People prefer cloud databases when they want
their systems to scale automatically. Google Big Query is the greatest tool for searching your data.
Azure Synapsys has a similar feature; however, it is significantly more expensive. You can store
data on S3, but if you want to run a query, you’ll need Athena, which is expensive. So, in modern
practice, data is stored as a blob in S3, and everything is done in a Python application. If there is
an error in data finding, this method takes a long time. Amazon Redish can also handle a
considerable quantity of large data and comes with a built-in BI tool.

Pandas
The goal of this section is to show some examples to enable you to begin using Pandas. These
illustrations have been taken from real-world data, along with any bugs and weirdness that are
inherent. Pandas is a framework inspired by the R data frame concept.
Please find the CSV file at the following link:

https://github.com/Apress/advanced-data-analytics-python-2e

To read data from a CSV file, use this:

import pandas as pd
broken_df=pd.read_csv('fetaure_engineering_data.csv')

To look at the first three rows, use this:

broken_df[:3]

To select a column, use this:

broken_df[' MSSubClass ']

To plot a column, use this:

broken_df[' MSSubClass’ '].plot()

To get a maximum value in the data set, use this:

MaxValue= broken_df[' MSSubClass’].max() where MSSubClass is the

column header

There are many other methods such as sort, groupby, and orderby in Pandas that are
useful when playing with structured data. Also, Pandas has a ready-made adapter for popular
databases such as MongoDB, Google Big Query, and so on.
One complex example with Pandas is shown next. In the X data frame for each distinct column
value, find the average value of the floor grouping by the root column.

for col in X.columns:

if col != 'root':
avgs =
df.groupby([col,'root'],as_index=False)['floor'].aggregate(np.mean)
for i,row in avgs.iterrows():
k = row[col]
 v = row['floor']
r = row['root']
X.loc[(X[col] == k) &
(X['root'] == r), col] = v2.
You can do any experiment in the Pandas framework with the data given for classification and
regression problems.

ETL with Python (Unstructured Data)

Dealing with unstructured data is an important task in modern data analysis. In this section, I will
cover how to parse emails, and I’ll introduce an advanced research topic called topical crawling.

Email Parsing
See Chapter 1 for a complete example of web crawling using Python.
Like Beautiful Soup, Python has a library for email parsing. The following is the example code
to parse email data stored on a mail server. The inputs in the configuration are the username and
number of mails to parse for the user.
In this code, you have to mention the email user, email folder, and index of the mail-in config;
code will write from the address to handle the subject and the date of the email in the CSV file.

from email.parser import Parser

import os
import sys
conf = open(sys.argv[1])
config={}
users={}

# parsing the config file

for line in conf:

if ("," in line):
fields = line.split(",")
key = fields[0].strip().split("=")[1].strip()
val = fields[1].strip().split("=")[1].strip()
users[key] = val
else:
if ("=" in line):
words = line.strip().split('=')
config[words[0].strip()] = words[1].strip()
conf.close()

# extracting information from user email

for usr in users.keys():

path = config["path"]+"/"+usr+"/"+config["folder"]
files = os.listdir(path)
for f in sorted(files):
if(int(f) > int(users[usr])):
users[usr] = f
path1 = path + "/" + f
data = ""
 with open (path1) as myfile:
data=myfile.read()
if data != "" :
parser = Parser()
email = parser.parsestr(data)
out = ""
out = out + str(email.get('From')) + "," +
str(email.get('To')) + "," + str(email.get('Subject')) + "," +
str(email.get('Date')).replace(","," ")
if email.is_multipart():
for part in email.get_payload():
out = out + "," +
str(part.get_payload()).replace("\n"," ").replace(","," ")
else:
out = out + "," +
str(email.get_payload()).replace("\n"," ").replace(","," ")
print out,"\n"

#updating the output file

conf = open(sys.argv[1],'w')
conf.write("path=" + config["path"] + "\n")
conf.write("folder=" + config["folder"] + "\n")
for usr in users.keys():
conf.write("name="+ usr +",value=" + users[usr] + "\n")
conf.close()

Sample config file for above code.

path=/cygdrive/c/share/enron_mail_20110402/enron_mail_20110402/maildir
folder=Inbox
name=storey-g,value=142
name=ybarbo-p,value=775
name=tycholiz-b,value=602

Topical Crawling
Topical crawlers are intelligent crawlers that retrieve information from anywhere on the Web.
They start with a URL and then find links present in the pages under it; then they look at new
URLs, bypassing the scalability limitations of universal search engines. This is done by
distributing the crawling process across users, queries, and even client computers. Crawlers can
use the context available to infinitely loop through the links with a goal of systematically locating
a highly relevant, focused page.
Web searching is a complicated task. A large chunk of machine learning work is being applied
to find the similarity between pages, such as the maximum number of URLs fetched or visited.

Crawling Algorithms
Figure 2-2 describes how the topical crawling algorithm works with its major components.
Figure 2-2 Topical crawling described
The starting URL of a topical crawler is known as the seed URL. There is another set of URLs
known as the target URLs, which are examples of desired output.
Another intriguing application of crawling is for a startup that wants to uncover crucial
keywords for every IP address. In the HTTP packet header, they acquire the user’s browsing
history from the Internet service provider. After crawling the URL visited by that IP, they classify
the words in the text using name-entity recognition (Stanford NLP), which is easily
implementable by the RNN explained in Chapter 5. All name entities and their types, such as
names of people, locations, and organizations, are recommended for the user.

import requests
from bs4 import BeautifulSoup
import nltk
from nltk.tokenize import word_tokenize
from nltk.corpus import stopwords
from nltk.tag import StanfordNERTagger
import re
import json
import os
import socket
import struct

def ip2int(addr):
return struct.unpack("!I", socket.inet_aton(addr))[0]
def int2ip(addr):
return socket.inet_ntoa(struct.pack("!I", addr))
java_path = '/usr/bin/java'
os.environ['JAVAHOME'] = java_path
os.environ['STANFORD_MODELS'] = '/home/ec2-user/stanford-ner.jar'
nltk.internals.config_java(java_path)

f = open("/home/ec2-user/data.csv")

res = []

stop_words = set(stopwords.words('english'))

for line in f:
fields = line.strip().split(",")
url = fields[1]
ip = fields[-1]
print(ip)
print(url)
tags_del = None
if True:
try:
ip = ip2int(ip)
except:
continue
print(ip)
tagged = None
try:
code = requests.get(url)
plain = code.text
s = BeautifulSoup(plain)
tags_del = s.get_text()
if tags_del is None:
continue
no_html = re.sub('<[^>]*>', '', tags_del)
st = StanfordNERTagger('/home/ec2-
user/english.all.3class.distsim.crf.ser.gz',
'/home/ec2-user/stanford-ner.jar')
tokenized = word_tokenize(no_html)
tagged = st.tag(tokenized)
except:
pass
if tagged is None:
continue
for t in tagged:
t = list(t)
t[0] = t[0].replace(' ', '')
t[-1] = t[-1].replace(' ', '')
print(t)
if t[0] in stop_words:
continue
unit = {}
unit["ip"] = ip
unit["word"] = t[0]
unit["name_entity"] = t[-1]
res.append(unit)
res_final = {}
res_final["result"] = res

#except:
# pass
#except:
# pass

with open('result.json', 'w') as fp:

json.dump(res, fp)

Summary
In this chapter, we discussed different kind of databases and their use cases, and we discussed
collecting text data from the Web and extracting information from different types of unstructured
data like email and web pages.
Other documents randomly have
different content
sometimes allowed myself to drift in my canoe on the river, thinking,
thinking, of Haidee—of the narrow oval of her face curtained in dark
hair streams, of the shadowy eyes of her, of her sweet warm smile.
And then one day I made up my mind suddenly to go to her.
At the first glimpse I had of her cabin, standing a crude, warped,
misshapen thing on the slight rise of ground beneath the cedars, all
my former resolves to give to this habitation some slight air of
comfort and refinement rose up and confronted me, and I saw myself
a weak fellow, who had nursed his despair and disappointment and
failed in his duty to the woman he loved, and who in his cowardice
had absented himself from his loved one, when he might have
brought her comfort and neighborly assistance.
On the back of an old envelope with a stub of a pencil I made a
rough sketch of the improvements I had long since planned, and
when Haidee and Wanza came to the door, I greeted them calmly
and showed them the sketch. Haidee stood there, without her
crutches, her hair unbound about her ivory face. Her gown was
white, and a scarf of rose color swung from her shoulders. She
looked at me for a long moment with eyes dull and faded as morning
stars, and then gradually the old familiar light came back into her
face, her eyes warmed and grew human. She stepped outside, and
joined me on the porch.
“You have laid aside your crutches?” I ventured.
“Yes.”
“You are well?” I asked.
“Oh, yes! I work—hard—at various things. Do I not, Wanza? I sleep.
I have a splendid appetite. And you?”
“I work. I sleep well, too. I drop asleep in the Dingle occasionally
after a hard day’s work. The Dingle is Wanza’s retreat—she walks
there. Do you know it, Wanza?”
She came to my side quickly. Her face displayed signs of
perturbation. “I walk there! What do you mean? Have you seen me?”
“You come on tip-toe. It is hardly walking.”
Her eyes questioned me.
“I’ve seen you only a few times. But I suspect you come frequently.”
“I am sure I don’t, Mr. David Dale.”
She came closer, her cheeks like crimson roses, her bright eyes
angry, her lips scornful.
“You come to visit Joey, I think. You came the first night after your
departure from Cedar Dale. And you went into the cedar room.” I
smiled into her troubled face.
“And what did I do there?”
“You took the magpie’s cage from its hook. You carried it away with
you. But you were like a little trade rat—you left the cedar waxwing
for Joey and me.”
But just here Wanza flung me an odd look and ran into the house,
saying over her shoulder: “That was a funny, funny dream.”
Haidee favored me with a rather intent look, and dropped her gaze to
the envelope in her hand. We walked around the cabin, and I
explained how I planned to build a small rustic pergola with a trellis
for wild honeysuckle at the back door to serve as a breakfast room
next summer, and timidly at last, I told her that I wished that I might
cover the rough walls of her sleeping room with cedar strips and
build a pergola outside the door like the one I had built at Cedar Dale
for Joey.
“We’ll plant some woodbine roots this fall, and set out a crimson
rambler. We may as well have the place blooming like an Eden,” I
said.
“And the wilderness shall blossom like the rose,” murmured Haidee.
“Thank you, Mr. Fixing Man.”
I rode home happier than I had been in many a long day. When I told
Joey of the proposed improvements at Hidden Lake he shouted with
glee, and a few moments later I heard him tooting on his neglected
flute that had lain strangely mute since the day when Haidee had
sung “Bell Brandon” to its accompaniment, and we had seen the
smile die from her curling lips and the light of joy go out in her
sparkling eyes.
After this my days were trances. Through the glowing flame-like
hours I worked to transform the sordid little cabin into a fitting
habitation for my wonder woman. Together we planned the rustic
porch at the rear of the kitchen, and when the foundation was laid I
dug up wild honeysuckle roots and we planted them with a lavish
hand, bending shoulder to shoulder above the sweet, moist earth,
our hands meeting, Haidee’s breath on my face, her unsteady
laughter in my ear, the charm of her rare, compelling personality
stirring my senses to ecstasy.
I labored each day till the sun was well down behind Nigger Head;
and then came a half hour of blissful idleness on the front porch with
Haidee behind a tea tray facing me, Wanza handing around cheese
cakes and sandwiches, and master Joey sitting on a three-legged
stool, the picture of smug, well-fed complacency.
Wanza’s conduct puzzled me sorely during these days. At times she
jested with me in her old bright rollicking way, but oftener her mood
was fitful, and she was hot-tempered, difficult and distrait.
One evening I rode to the village with her in her cart on a special
errand for Haidee. It was a mellow, moonlight evening. The air was
ripe with a frosted sweetness, a tang that only autumn evenings
hold. I was in boisterous spirits; and as Wanza drove I relapsed into
my old way of alternately bantering and teasing and flattering my
companion.
“When you no longer line your umbrella with pink, Wanza,” I said, “I
will know that vanity and you have parted company.”
The blonde head turned restlessly.
“I ain’t half as vain as I used to be.”
“Oh, that’s bad, Wanza—very bad! A pretty girl is naturally vain. And
as for the pink lining—it’s as natural for a fair, pale girl like you to line
her umbrella with pink as it is for a fruit dealer to stretch pink gauze
over his sallow fruit.”
“What do you mean by that?” Wanza demanded fiercely. She
dropped the lines. “Now, what do you mean by that, I say?”
“Dear Wanza,” I said, soothingly, “I don’t mean anything—except that
pink lends a pretty glow to an alabaster skin like yours.”
Her eyes gleamed at me savagely in the moonlight, and she made a
strange sound in her throat that sounded like a sob.
“I don’t understand,” I continued, “why you’re so sensitive, of late.
Why, it’s so hard to talk to you! You’re so difficult I feel like putting on
a mental dress-suit and kid gloves when I converse with you. What’s
come over you, Wanza?”
“Nothing’s come over me. It’s you,” she answered in a low tone.
“Oh, no,” I responded, “Wanza girl, I treat you just the same as I ever
did, my dear!”
“But you don’t treat me the same as you do her—you don’t treat me
just the same—” her voice sounded husky. She turned her head
away.
What could I reply?
I ventured finally: “I don’t know exactly what you mean, child! But I
hope I show by my manner to you how very much you count in my
life,—how dear you are to Joey and me—how fine and staunch a
friend we have ever found you—I hope I show this, Wanza. If I do not
I am sorry indeed.”
There was a slight movement towards me on the girl’s part. Her
hand crept out shyly and touched mine. I heard her whisper
chokingly:
“If I mean a good deal to you and Joey I sure ought to be satisfied. It
oughtn’t to matter—really matter—if you smile different when you
speak to her.”
I took her hand. I was moved. Again I marveled that Wanza had the
power to shake me so. “You have your own place, child,” I said. And
when she questioned, “But what is my place, Mr. Dale?” I asked
myself what indeed was her place. “I shall tell you some time,” I
answered, which was not at all the remark I desired to make, and I
spoke in palpable confusion.
After a short interval she took her hand from mine, and gathered up
the lines, not looking at me as she said: “Mr. Batterly is back in
Roselake.”
I caught her by the shoulder. I drew her quickly to me till I could see
her face in the moonlight.
“When did he come back?” I asked, thickly.
She tugged at my restraining hand and shrugged away from me.
“He’s been back two weeks, I calculate—may be more.”
“Don’t speak to him, Wanza—don’t look at him!” I implored quickly.
She faced me proudly at this. “Do you think I would,” she cried
scornfully, “except to answer him when he speaks to me on the
road?”
“I did not know, Wanza,” I murmured humbly.
“Did not know! It’s little you know me any way, David Dale, I am
thinking. If you know me so little as not to know that, why should I
care indeed how you treat me, or what my place is with you? Why
should I care? Sometimes I think, David Dale, I think that I hate you.
I’m thinking it now. Yes, yes, yes!”
“Please, please, Wanza—”
“Stop! I will ask a few questions, myself. I will put them to you,
although I never—in loyalty to you—put them to myself. But it is not
for you to tell me how to behave—how to walk so and so—say and
do so and so! This is the question I will put: Is it right for you to
spend each and every day at Hidden Lake? Is it? Answer that to
yourself—not to me—before you tell me not even to speak civilly to
Mrs. Batterly’s husband. I don’t want to speak to him! I don’t want
him to speak to me! No, nor look at me. Can you say as much for
her, David Dale?”
“I don’t know what to say,” I stammered, taken by surprise.
“You don’t have to say nothing—not to me. I’m not your judge. But
answer the questions to yourself, quick, before you tell me what to
do and what not, again! Go on, Rosebud, you’re a-getting to be
slower and slower!”
I glanced at her face. It was pale, and her lips were unsteady.
About this time Joey began to take sudden trips down the river in the
flat-bottomed swift-water boat, poling away industriously each
morning with a fine show of mystery—unconsciously admonishing
me to appear indifferent and uninterested. I carried my apathy too
far, I imagine, for one day he said to me:
“Mr. David, do you mind the old hollow stump in the willows on the
river bank—where the flycatcher’s left a funny big nest?”
I answered yes. I had marked it well. The secret waterway which led
to Hidden Lake was close by.
“Well,” Joey continued, looking very important, and puffing out his
chest like a pouter pigeon, “Bell Brandon and me have a post-office
there. She leaves the most things for me there under the flycatcher’s
nest in a box—cut-out pictures, and cookies, and fludge.”
“Fudge, Joey boy.”
“Yes—fludge. And say, Mr. David—any time you’re passing, look in,
won’t you? ’Cause there might be something there would spoil.”
 CHAPTER XIX
BEREFT

I HAD not heard from Janet Jones again and I was beginning to
think that I might never have another letter from her when a missive
came.
Thank you for my cedar chest (she wrote). It reached me
safely, but I have been ill in body and mind and unable to
write sooner. Oh, the joy my bit of cedar wood is to me.
When I look at it, I am transported at once to the heart of
the clean woods. And I shut my eyes and vision the tree
hosts in their tawny brown, like Khaki-clad soldiers
marshalling at the trumpet call of the rushing September
winds. What a sparkle and spirited flavor there is in the
wine-like air. How the leaves swirl in the paths like gilded
cups, and winnow through the air like painted galleons,
and rustle and unroll beneath the tread, like cloth of gold.
Oh, I love the summer. But the fall with its shining
sumptuous days—its melancholy grandeur surpasses it.
Only—the birds are gone—are they not? And the dear
clever nests—“half-way houses on the road to Heaven”—
sway tenantless. While the wood aisles seem hushed and
solemn, I know, like vast cathedral spaces after the organ
has ceased to reverberate.
I read this letter with delight, and I wrote and thanked Janet Jones as
cordially as I knew how for the pleasure it had given me. I began to
look forward to her next missive, and I was beginning to experience
no small satisfaction from our peculiar, unconventional friendship,
when a strange thing happened.
Joey and I were tearing out the straw from his mattress one day,
intent on our usual fall house-cleaning, when my fingers closed over
a bit of cardboard. I drew it forth, unrolled it, and smoothed it in my
hand. It was the small square visiting card that had been attached to
the parcel that Haidee had placed in my saddle-bag for Joey, on the
day that now seemed so long ago, when I had gone to fell the trees
at Hidden Lake and had ridden so ungallantly away.
Joey sprang at me and seized my wrist. “That’s mine! That’s mine!”
he shouted. “Give it here, Mr. David—please.”
But I was staring at the writing on the back of the card. “For the boy
who goes to Sunday school,” Haidee had written in strong, clear
characters. Surely, the hand that had penned that line had more
recently penned other lines to me and beneath them signed the
name of Janet Jones.
I had a letter in my pocket, and later I compared the writing on the
envelope with that on Joey’s card. And I smiled to myself; but
wonderingly. Still a doubt assailed me. I grew wary. And fate favored
me. When Wanza stopped her cart at the meadow bars en route to
Roselake one day, to pick up Joey, I saddled Buttons and rode to the
village in their wake. At the post-office I swung out of my saddle.
“Give me your letters, Wanza,” I suggested. “Don’t get down. I’ll post
them.”
Once inside the office I ran the letters through my fingers. There
were two letters addressed to Miss Janet Jones, Spokane,
Washington, and the writing was that with which I had grown familiar
in Janet Jones’ letters to me.
I was completely mystified. I rode home in a brown study. And then
suddenly I reached a solution. That night I wrote a letter. I took great
pains with its construction. And after Joey was in bed I paddled away
down the river in the light of the moon to the hollow stump among
the willows on the bank. I placed my letter to Haidee within the
recess on a soft bed of ferns and dried grass that I found there; and
then I paddled stealthily home.
I kept an even face when I greeted Haidee the following day, and
she did not betray by word or glance that she had received a
communication from me. But as I opened my lunch pail that night to
give Joey some doughnuts that Wanza had sent him, there on top
was a small white envelope addressed to me.
I read the letter after Joey was in bed and I had built up a fire of pine
cones on the hearth. It was a characteristic Janet Jones letter:
Dear Mr. Craftsman:
Once upon a time—which is the way I begin my fairy tales
to Joey—there was a certain foolish woman, whom we will
call Haidee, who lived all alone in the heart of a forest.
She was a very headstrong young woman, full of whims
and insane impulses, or she never would have gone into
the forest to live alone. But she loved Nature passionately
and she had suffered and known heartache—and she felt
that Nurse Nature could assuage pain.
A big-hearted woodsman lived nearby in this same forest.
He swung his ax, and befriended her. He labored in the
hot sun felling trees that the headstrong woman might be
safe in her flimsy shack. But the woman taunted him, and
when he would have felled every tree that endangered her
habitation she stayed his hand. Then, one day, retribution
overtook her. A tree fell, and she was hewn down in her
conceit and foolhardiness. She was taken to the
woodsman’s cabin by the kind-hearted woodsman who
rescued her. There she was cared for tenderly, and the
coals of fire burned her poor silly head—so much so that,
knowing she was a burden and an expense to the
woodsman, who, like most big-hearted honest woodsmen,
was desperately poor, she lay awake nights planning how
best to recompense him without wounding his proud spirit.
At last, she thought of a plan. And with the connivance of
a dear old-time friend in Spokane, carried it out. Her friend
gave her permission to sign her name to the letters she
wrote the woodsman. After the letters were written, they
were sent to the original Janet Jones, who forthwith
mailed them to the woodsman at Roselake. Janet Jones
also, naturally, received the letters which the woodsman
 wrote, and in due time they were put into envelopes and
addressed to the headstrong woman, whom they did not
fail to reach. The cedar chest was the headstrong
woman’s gift to Janet Jones, who is an invalid, and a
romanticist who enjoys beyond all words any departure
from the commonplace.
Am I forgiven, Mr. Fixing Man? And now, one word more.
You will not receive another letter from Janet Jones. And
—I pray you, come not too often to Hidden Lake—it is
better so.
This was the missive which I read in the firelight. As I finished I
suddenly felt bereft. And I lay back in my chair and stared into the
coals with unseeing eyes, brooding miserably, groping in a misty sea
of doubt and unrest and feeble desire. Then Joey called me in his
sleep. Just as I was sinking utterly, I heard, “Mr. David, Mr. David,”
and the cry of appeal braced me, strengthened the man in me. I
went in to him as a sinner into a sanctuary, and the kiss he gave me
sleepily was a salve that solaced and sustained me throughout the
trying night.
I had finished the improvements on Haidee’s cabin at this time; so I
gave over going to Hidden Lake in prompt obedience to the request
my wonder woman had made in her letter. But I wrote an answer to
the letter and placed it in the old stump. I assured her that I would
respect her wishes, and I begged her to let me know the instant I
could serve her in any way, promising her that never a day should
pass without my going to the secret post-office.
I had advertised my cedar chests in the magazines during the
summer, and orders began to pour in, so that I was kept busy in my
workshop. Those were busy days in the house as well, for, with the
beginning of September, Joey had started to school at Roselake,
and many of the small duties he had taken upon his young shoulders
devolved upon me.
Oh, the day on which Joey started to school!
I dressed him carefully that morning, with all the trepidation of an
over-fond parent, and I admonished him concerning his demeanor in
the school-room until I am sure his small head must have been in a
whirl, and his little heart in a flutter of apprehension.
“I’ll do my best, Mr. David, dear,” he said bravely. “You said yourself
they can’t no one do more.” He hesitated and looked at me,
reddening painfully. “And if the teacher asks me who am I—and
who’s—who’s my father—what am I to tell her?”
My hand closed on his shoulder fiercely. “Tell her you are Mr. Dale’s
boy, from Cedar Dale—tell her your name is Joey Dale,” I cried. The
look on his face had stabbed me.
He considered, looking into my eyes awesomely as I took his chin in
my hand.
“If I have the Dale part, couldn’t I have the David, too?” he
suggested. “Hm! Then we’d be big David and little David.”
“David Dale, the second,” I said, poking him in the ribs.
“But there couldn’t be any David Dale, the second. There couldn’t
never be but one real David Dale. But there could be a little David.”
A little David!
That was a dragging day. I missed the lad which ever way I turned.
And his words to me, when he leaped to my arms from old Buttons’
back that night! “It was fine! I liked it, really and truly. But, oh, Mr.
David, I ’most knew you was lonely and missing me!”
Every morning I walked to the edge of the meadow, let down the
bars for old Buttons, and watched Joey ride away, his sturdy little
figure jouncing up and down in the saddle, his brave, bright face
turned back to me over his shoulder, with rare affection beaming
from big big brown eyes, as he waved and waved to me until a bend
of the road hid him from my sight.
One memorable morning in the latter part of September, as I was
tightening the saddle girths, he bent down to me, and as I lifted my
head he surprised me with a quick shame-faced salute of moist lips
on my forehead.
“You’re a good Mr. David,” he said patronizingly. “And I ain’t yours
either—not blood kin.”
I hugged the little lad to me—a sudden fierce warmth of affection
stirring my sluggish halting heart that had grown weary lately of life’s
complexities.
“You’re my boy, just the same,” I assured him.
“They can’t anybody get me away from you—can they?” he asked
anxiously, and I saw genuine consternation in his eyes.
I laughed and hugged him tighter. “I guess not,” I bragged. “Let them
try. Jingles would eat them up.”
“And we’d hide, wouldn’t we?”
“We surely would.”
“And—and we’d shoot at them from the rushes.”
I know not why Joey’s words should have irked me, but the day
seemed long, and I was glad when I heard the soft thud of Buttons’
hoofs on the turf outside the cabin promptly at the accustomed hour.
I was building the kitchen fire, but I straightened up, stepped to the
door, and threw it wide.
Buttons stood with his bridle over his head, his nose sniffing the
ground, but no Joey sprang from the saddle into my eager arms. The
horse was riderless.
All Roselake joined in the search for Joey, after I had ascertained
that the lad was not with Haidee, and the search was prolonged far
into the night. The school-master had seen Joey ride away at the
close of school, and I argued that Buttons must have come straight
home. At dawn the search was resumed. For miles in each direction
the searching party spread out, but at night, totally disheartened, the
kindly neighbors disbanded, and Joey’s case was left in the hands of
the police.
 CHAPTER XX
“PERHAPS I SHALL GO AWAY”

ALONE the next day I took up the search for Joey, beating back and
forth between Roselake and Cedar Dale, and penetrating to Wallace
and Wardner. It was to Wanza that I spoke my conviction at last,
sitting my cayuse on the river road, while she sat stiff and tearful-
eyed in her cart, pale even beneath the pink-lined umbrella.
“It looks to me, Wanza girl,” I said wearily, “like a plain case of
kidnapping.”
“But who would kidnap him, Mr. Dale?” Wanza queried pitifully.
“Why—that’s the question,” I returned. “Have you ever seen him
talking to any one—any stranger—when you have met him going
and returning from school?”
She shook her head. “Once,” she replied, “Joey was with me, and
Mr. Batterly stopped us. He asked me all about Joey—seeming so
keen! And I told him—thinking it no harm—just how a dying woman
gave him to you, saying he was a waif that had been picked up after
a storm over on the Sound by her dead brother, who had been a
fisherman.”
“Where is Batterly now,” I asked.
“Gone away—this week past.”
“Oh, well,” I sighed, “we’ll acquit him. I’m sure he was not over fond
of Joey.” After a pause I asked brusquely: “Where has he gone?”
“I don’t know—sure I don’t, Mr. Dale. The last I heard of him he was
going to hire a swift-water boat and a poler, and try the swift-water
fishing above St. Joe.”
“Then he hasn’t left the country,” I said. And my heart sank leaden
and my hate of the man boiled up in my veins fiercely, as I pictured
him still skulking about, a menace to Haidee’s peace of mind.
The time went very heavily past. All my days and many nights were
spent in the saddle, and the evenings that I passed at Cedar Dale
were consumed in feverish plans for the scoutings that I made. I did
not even now attempt to visit Haidee at Hidden Lake; but one
morning, at sunrise, hearing a soft tap on my door, I opened to see
Wanza standing there with a covered basket on her arm.
“I saw your light last night,” she quavered. “I have brought you some
good nourishing food. I can see you’re not cooking for yourself.
You’re growing white and thin.”
Her womanly act in coming thus to offer me comfort stirred me
strangely, appealed to the finest fibre in my nature. Her simplicity, her
self-forgetfulness made me falter at her feet.
But at last I gave over my scoutings. I made a cedar chest for Joey’s
room, and in this I placed all his little kickshaws, his few clothes, and
his flute, along with the gay Indian blanket he had reveled in, and the
quilt Wanza had pieced for him. The room thus became to me a sort
of shrine. And finding me here at the close of a long day with tears of
which I was not ashamed in my eyes, Wanza broke down and
sobbed beside me.
“I’d like to kill whoever it is as has taken Joey away,” she cried,
brandishing a resentful fist.
“If we knew any one had taken him,” I said, thoughtfully. “Sometimes
I think—I think, Wanza, that Joey is dead.”
“I don’t think so! No, indeed!” Wanza returned with thrilling
earnestness. “Oh, I feel sure he ain’t dead! He’ll be found—some
day. He sure will, Mr. Dale.”
She helped me by her sturdy optimism.
Soon after this Wanza and I fell into the habit of tramping through the
gleaming golden woods together almost daily, breathing the crisp
sweet autumn air. Wanza in her bright sweater, with her tawny hair,
and the carmine in her cheek flitted in and out of the wood paths like
a forest dryad, exclaiming at every frost-touched leaf, and reveling in
the painted glory about us.
“But the birds are gone,” she said, a tear in her tones, as we looked
into an empty king-bird’s nest one day. “I love the king-birds—they’re
sleek dandies—that’s what they are! Oh, Mr. Dale, what a heartache
an empty nest gives me! The dear little birds are gone—”
“And Joey is not here,” I ended sadly.
After awhile I went on: “Yes, summer has gone. It is the most
evanescent time of the year. It slips and slips away—and just as you
grasp it and thrill to its sweetness it melts into—this—as happiness
merges into sorrow.”
Her face quivered, and her eyes came to mine. “I guess that is so,”
she said in a low tone.
Looking in Wanza’s face lately I always turned away. I did so now.
The look of questioning I found there—the mute appeal—the
suffering—these unmanned me. But it grew to be a strange
satisfaction to be with her, through long crisp daylight hours, in the
hush of pink sunsets, in the gilded autumn twilights, while we rested
after a meagre supper cooked over a camp fire, chatting desultorily,
and watching the big pale stars came out to lie like white-tipped
marguerites on the purple bosom of the sky above our heads.
One day I spoke my thought.
“I am thinking, Wanza—perhaps I shall go away.”
We were in the heart of the woods. A tinkling, sly little brook made
the forest musical, the rustle and purr of the pines sounded about us
like fluty organ notes. Wanza’s eyes were lifted to the sprightly
shivering leaves of a cottonwood, and her face was very still. She did
not move as I spoke, and I repeated my sentence.
“I thought you’d go,” she said. She spoke harshly.
“I can’t stop on here without Joey. I can’t bear it,” I said, haltingly.
“But I’ve got to stay on without either of you—and bear it.”
I saw her eyes. I recoiled at the depth of pain revealed.
“Mr. Dale,” she said gropingly, after a pause, “where are you going?”
“I don’t know, Wanza. But inaction is intolerable. I must be doing
something. I must get away for awhile, at least. It is better.”
Wanza’s eyes were very bright. Her hands that were smoothing a
maple leaf were trembling. Her voice sounded dry and hard as she
asked:
“When do you reckon you’ll go?”
“Why, child, I do not know! Each day I say to myself I cannot bear
another.”
“It’ll be the same wherever you are.”
“Perhaps so, Wanza,” I sighed. And then because I knew the tears
were on her cheeks, I sprang to my feet, saying: “This may be our
last day in the woods together, who knows? Come, let us try to forget
—let us make the best of what we have.”
Wanza rose. She came close to me. When our eyes met she gave a
cry: “If you go you may never come back!”
“Never fear. I have no home but Cedar Dale,” I replied, and I am
afraid my voice was bitter. And when she put her hand on my arm I
shook it off and would have strode away, but again as in the woods
on the occasion of our gipsying I saw her face close to my own, and
caught my breath in marvel. No, there was never such a girl-face!
Such an elf-face! I stooped suddenly and framed the face with my
hands. What were her wonderful eyes saying, back of all the tears,
all the mystery? Why—when I was in love with Haidee—did they
draw me like a lodestar? Why now and then did she stir me in this
strange fashion till I gazed and gazed, and needs must curb my will
to keep from taking her in my arms and crushing her against my
heart?
I had never faced the question. I did not care to face it now. I put it
away for some future time, feeling vaguely that it remained to be
reckoned with.
“I have no home but Cedar Dale,” I repeated.
“And I am glad of that,” she whispered.
She pressed nearer to me, and I released her face, and drew her
slowly within the circle of my arms. But when I held her so, when the
floating hair meshes were just beneath my chin, and her face
brushed my sleeve, I steadied myself.
“Wanza,” I said, “I am almost glad, too, that I have no other home.
When I think of the good friends I have here—you and your father
and Father O’Shan—I realize that I am ungrateful to despise my
humble place among you. Keep it for me, little girl, and I shall come
back. Yes, I shall come back better equipped for the future among
you. If it must be without Joey—” I hesitated and bit my lip—“without
Joey,” I continued more firmly, “I shall at least try to earn your respect
by holding up my head, and forging on to some goal. I shall attain to
something at last, I hope. And I hope I shall be able to serve my
neighbors in many ways, and make myself needed in the
community.”
I held her for a moment after saying this, and then I bent down and
for the first time in my life kissed her. But it was on the brow that I
kissed her. And I am sure no brother could have saluted her more
respectfully.
She drew back. Her head fell against my shoulder. I saw deep into
her splendid eyes,—deep, deep. Back of all the tears and the smiles
and the mystery I read at last what they were saying. I read—and I
was humbled and abashed. I knew the truth at last. Wanza loved
me.
I saw clearly now, indeed. I recalled Father O’Shan’s words: “Be
careful in your dealings with that child.” I had been blind, and a fool. I
blamed myself, and I hated myself. I stood stupidly staring into the
face so near my own until with a sudden wrench Wanza jerked away
from me, and ran on down the purpling wood-aisle before me,
dashing the tears from her eyes as she fled.
I walked home slowly, astounded and perplexed by the revelation I
had had.
 CHAPTER XXI
FATE’S FINAL JAVELIN

THAT night in my lonely cabin I fell ill, and burned with fever, and
shook with ague so that I was unable to drag myself about the cabin,
but lay all the next day and the next in my bunk. The following day
my fever left me magically; and late in the afternoon I arose, fed and
curried my half-starved cayuse and, mounting, rode away beneath
the berry-reddened yews to the trail that led to Haidee.
I dismounted at the rustic pergola at the rear of the cabin, tethered
my cayuse and walked around to the front door. The door was
closed, and a silence that was almost oppressive brooded over the
place. I ran up the steps, and a curious premonition that Haidee had
gone away sickened me as I rapped on the panel. Terrified at
receiving no response, I turned the handle, pressed forward, and
caught at the casement for support in my weakness. I peered in, and
at the sight I saw my knees all but gave way so that I swung about
like a loose sail in a sudden breeze.
On the floor lay Randall Batterly in a ghastly pool of blood. His face
was upturned to the cold October sunlight. His lips were opened in a
half snarl, his full lids were wide apart over his rolled back, terrible
eyes. He was bleeding from a wound in his chest. And Haidee stood
above him, gazing down upon him, gray horror painted on her face.
She heard my step and turned, and I caught the metallic thud as the
revolver she had been holding dropped to the bare floor. She stared
at me, put out her hand as if to thrust me back. I saw fear in her face.
“It is you! It is you!” she breathed.
She continued to stare at me with big gaunt eyes.
“Yes,” I replied, trying to keep the horror out of my tones. “It is I.”
She shuddered and collapsed to her knees, clinging to the door
frame as a drowning man clutches and grips a bulwark. The pupils of
her eyes were dilated with terror and despair until the purple iris was
eclipsed, and they stared black and empty as burnt-out worlds.
“He is dead—dead,” she whispered. “He can’t speak, or move.”
I picked up the revolver and laid it on the table, and then I crossed to
the rigid form on the floor. I knelt and pressed my ear to his heart. I
lifted his hand; it fell back inertly. Yes, it was true. Randall Batterly
was gone past recall, facing the great tribunal above, with who knew
what black secret in his heart.
“We must get a physician,” I murmured dully.
Haidee crept to my side. Her poor face was blanched and twisted till
she looked like a half-dead thing.
“Who could have done this—” I stammered, in a voice that sounded
driveling and uncertain in my own ears.
Again that dumb look of distress in her eyes, and she stood as if
carved in granite.
“My dear—my dear, you must come away—this is too much for you,”
I continued hoarsely. I took her poor cold hands in mine. And then I
turned and faced the door with a curious certainty that some one
was looking at me, and I saw old Lundquist’s rat eyes peering in on
us from the doorway.
He said not one word—only stared and stared at the dead man on
the floor, and at the abject living creatures standing over him; and
then he crept away like a sliding shadow, and the sunlight brightened
the place again. But in that grim room Haidee had fallen face
downward, stark and stiff, and her wild scream as she sank echoed
and re-echoed in my ears for days.
I brought water, I bathed her face, I chafed her hands; but the
moments passed and she did not revive, and twilight fell, as alone, in
the presence of death I wrestled with the stupor that held her. And
there they found me—the sheriff and old Lundquist.
“For God’s sake, lend a hand here,” I cried imploringly. And then I
stood up. “Gentlemen,” I said, “this—dead man is Mrs. Batterly’s
husband. I believe this to be a suicide—I found him lying just as you
see him a short while ago. Mrs. Batterly had just discovered him, I
believe. She is—as you see—in no condition to be questioned.”
The sheriff hesitated. I had known the man for years, and I saw a
swift scepticism darken his keen eyes as they searched my face. He
glanced at Haidee and then at the revolver lying on the table. He
reached over, picked up the weapon and examined it.
“This revolver is loaded in only four of its chambers. The fifth has a
discharged cartridge. Was this lying on the table when you came in,
Dale?”
I spoke hoarsely. “I put it there. It had fallen to the floor.”
Old Lundquist crawled closer. “That ban Mrs. Batterly’s revolver,” he
mumbled, “I see her have it—it ban on the table most o’ the time.
Thar be a letter on it—to mark it like.”
The sheriff’s finger traced the outline of the shining letter on the
polished surface of the weapon. He stood irresolutely, ruminating.
“Come!” I ordered brusquely. “This lady must be seen to.” And as
neither man made a move to assist me, I lifted Haidee in my arms. I
felt her stir. Her eyes opened suddenly. She looked at old Lundquist
and the sheriff, then up at me affrightedly. Her hand clutched my
arm. She cowered, and a tremor shook her from head to foot.
“These men—why are they here?” she asked faintly.
“Gentlemen—” I was beginning, when the sheriff stopped me.
“Mrs. Batterly,” he said, clearing his throat, and speaking raspingly,
“this is your revolver?”
“Why, yes—” Haidee drew in her breath sharply—“why, yes,” she
admitted.
I felt her hand tighten its hold on my arm.
“It is mine, surely,” she continued, as no one spoke. She looked from
one to the other appealingly. “I am fond of shooting at a mark. I used
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and

personal growth!

ebooknice.com