Spark For Python Developers Amit Nandi download

https://ebookbell.com/product/spark-for-python-developers-amit-
nandi-5313210

Explore and download more ebooks at ebookbell.com

Here are some recommended products that we believe you will be
interested in. You can click the link to download.

Databricks Certified Associate Developer For Apache Spark Using Python

Saba Shah

https://ebookbell.com/product/databricks-certified-associate-
developer-for-apache-spark-using-python-saba-shah-57748364

Databricks Certified Associate Developer For Apache Spark Using Python

Saba Shah

https://ebookbell.com/product/databricks-certified-associate-
developer-for-apache-spark-using-python-saba-shah-193885056

Apache Spark 2 For Beginners Develop Largescale Distributed Data

Processing Applications Using Spark 2 In Scala And Python
Rajanarayanan Thottuvaikkatumana

https://ebookbell.com/product/apache-spark-2-for-beginners-develop-
largescale-distributed-data-processing-applications-using-spark-2-in-
scala-and-python-rajanarayanan-thottuvaikkatumana-10833918

Pyspark Cookbook Over 60 Recipes For Implementing Big Data Processing

And Analytics Using Apache Spark And Python Packt Tomasz Drabas

https://ebookbell.com/product/pyspark-cookbook-over-60-recipes-for-
implementing-big-data-processing-and-analytics-using-apache-spark-and-
python-packt-tomasz-drabas-57772432
Spark For The Fire How Youthful Thinking Unlocks Creativity Ian
Wharton

https://ebookbell.com/product/spark-for-the-fire-how-youthful-
thinking-unlocks-creativity-ian-wharton-51386446

Spark For Data Science Srinivas Duvvuri Bikramaditya Singhal

https://ebookbell.com/product/spark-for-data-science-srinivas-duvvuri-
bikramaditya-singhal-5596610

Spark For Data Science Bikramaditya Singhal Srinivas Duvvuri

Bikramaditya Singhal

https://ebookbell.com/product/spark-for-data-science-bikramaditya-
singhal-srinivas-duvvuri-bikramaditya-singhal-7264704

Apache Spark For Java Developers Sumit Kumar Sourav Gulati

https://ebookbell.com/product/apache-spark-for-java-developers-sumit-
kumar-sourav-gulati-50469976

Apache Spark For Data Science 1st Edition Padma Priya Chitturi

https://ebookbell.com/product/apache-spark-for-data-science-1st-
edition-padma-priya-chitturi-11565896
Spark for Python Developers

A concise guide to implementing Spark big data

analytics for Python developers and building a real-time
and insightful trend tracker data-intensive app

Amit Nandi

BIRMINGHAM - MUMBAI
Spark for Python Developers

Copyright © 2015 Packt Publishing

All rights reserved. No part of this book may be reproduced, stored in a retrieval
system, or transmitted in any form or by any means, without the prior written
permission of the publisher, except in the case of brief quotations embedded in
critical articles or reviews.

Every effort has been made in the preparation of this book to ensure the accuracy
of the information presented. However, the information contained in this book is
sold without warranty, either express or implied. Neither the author, nor Packt
Publishing, and its dealers and distributors will be held liable for any damages
caused or alleged to be caused directly or indirectly by this book.

Packt Publishing has endeavored to provide trademark information about all of the
companies and products mentioned in this book by the appropriate use of capitals.
However, Packt Publishing cannot guarantee the accuracy of this information.

First published: December 2015

Production reference: 1171215

Published by Packt Publishing Ltd.

Livery Place
35 Livery Street
Birmingham B3 2PB, UK.

ISBN 978-1-78439-969-6

www.packtpub.com
 Credits

Author Project Coordinator

Amit Nandi Suzanne Coutinho

Reviewers Proofreader
Manuel Ignacio Franco Galeano Safis Editing
Rahul Kavale
Daniel Lemire Indexer
Priya Sane
Chet Mancini
Laurence Welch
Graphics
Kirk D'Penha
Commissioning Editor
Amarabha Banerjee
Production Coordinator
Shantanu N. Zagade
Acquisition Editor
Sonali Vernekar
Cover Work
Shantanu N. Zagade
Content Development Editor
Merint Thomas Mathew

Technical Editor
Naveenkumar Jain

Copy Editor
Roshni Banerjee
 About the Author

Amit Nandi studied physics at the Free University of Brussels in Belgium,

where he did his research on computer generated holograms. Computer generated
holograms are the key components of an optical computer, which is powered by
photons running at the speed of light. He then worked with the university Cray
supercomputer, sending batch jobs of programs written in Fortran. This gave him
a taste for computing, which kept growing. He has worked extensively on large
business reengineering initiatives, using SAP as the main enabler. He focused for the
last 15 years on start-ups in the data space, pioneering new areas of the information
technology landscape. He is currently focusing on large-scale data-intensive
applications as an enterprise architect, data engineer, and software developer.
He understands and speaks seven human languages. Although Python is his
computer language of choice, he aims to be able to write fluently in seven
computer languages too.
 Acknowledgment
I want to express my profound gratitude to my parents for their unconditional love
and strong support in all my endeavors.

This book arose from an initial discussion with Richard Gall, an acquisition
editor at Packt Publishing. Without this initial discussion, this book would never
have happened. So, I am grateful to him. The follow ups on discussions and the
contractual terms were agreed with Rebecca Youe. I would like to thank her for her
support. I would also like to thank Merint Mathew, a content editor who helped me
bring this book to the finish line. I am thankful to Merint for his subtle persistence
and tactful support during the write ups and revisions of this book.

We are standing on the shoulders of giants. I want to acknowledge some of the

giants who helped me shape my thinking. I want to recognize the beauty, elegance,
and power of Python as envisioned by Guido van Rossum. My respectful gratitude
goes to Matei Zaharia and the team at Berkeley AMP Lab and Databricks for
developing a new approach to computing with Spark and Mesos. Travis Oliphant,
Peter Wang, and the team at Continuum.io are doing a tremendous job of keeping
Python relevant in a fast-changing computing landscape. Thank you to you all.
 About the Reviewers

Manuel Ignacio Franco Galeano is a software developer from Colombia. He

holds a computer science degree from the University of Quindío. At the moment of
publication of this book, he was studying to get his MSc in computer science from
University College Dublin, Ireland. He has a wide range of interests that include
distributed systems, machine learning, micro services, and so on. He is looking for
a way to apply machine learning techniques to audio data in order to help people
learn more about music.

Rahul Kavale works as a software developer at TinyOwl Ltd. He is interested in

multiple technologies ranging from building web applications to solving big data
problems. He has worked in multiple languages, including Scala, Ruby, and Java,
and has worked on Apache Spark, Apache Storm, Apache Kafka, Hadoop, and Hive.
He enjoys writing Scala. Functional programming and distributed computing are his
areas of interest. He has been using Spark since its early stage for varying use cases.
He has also helped with the review for the Pragmatic Scala book.
Daniel Lemire has a BSc and MSc in mathematics from the University of Toronto
and a PhD in engineering mathematics from the Ecole Polytechnique and the
Université de Montréal. He is a professor of computer science at the Université du
Québec. He has also been a research officer at the National Research Council of
Canada and an entrepreneur. He has written over 45 peer-reviewed publications,
including more than 25 journal articles. He has held competitive research grants for
the last 15 years. He has been an expert on several committees with funding agencies
(NSERC and FQRNT). He has served as a program committee member on leading
computer science conferences (for example, ACM CIKM, ACM WSDM, ACM SIGIR,
and ACM RecSys). His open source software has been used by major corporations
such as Google and Facebook. His research interests include databases, information
retrieval and high-performance programming. He blogs regularly on computer
science at http://lemire.me/blog/.

Chet Mancini is a data engineer at Intent Media, Inc in New York, where he
works with the data science team to store and process terabytes of web travel data
to build predictive models of shopper behavior. He enjoys functional programming,
immutable data structures, and machine learning. He writes and speaks on topics
surrounding data engineering and information architecture.

He is a contributor to Apache Spark and other libraries in the Spark ecosystem.

Chet has a master's degree in computer science from Cornell University.
 www.PacktPub.com

Support files, eBooks, discount offers, and more

For support files and downloads related to your book, please visit www.PacktPub.com.

Did you know that Packt offers eBook versions of every book published, with PDF
and ePub files available? You can upgrade to the eBook version at www.PacktPub.com
and as a print book customer, you are entitled to a discount on the eBook copy. Get in
touch with us at [email protected] for more details.

At www.PacktPub.com, you can also read a collection of free technical articles, sign
up for a range of free newsletters and receive exclusive discounts and offers on Packt
books and eBooks.

TM

https://www2.packtpub.com/books/subscription/packtlib

Do you need instant solutions to your IT questions? PacktLib is Packt's online digital
book library. Here, you can search, access, and read Packt's entire library of books.

Why subscribe?
• Fully searchable across every book published by Packt
• Copy and paste, print, and bookmark content
• On demand and accessible via a web browser

Free access for Packt account holders

If you have an account with Packt at www.PacktPub.com, you can use this to access
PacktLib today and view 9 entirely free books. Simply use your login credentials for
immediate access.
 Table of Contents
Preface v
Chapter 1: Setting Up a Spark Virtual Environment 1
Understanding the architecture of
data-intensive applications 3
Infrastructure layer 4
Persistence layer 4
Integration layer 4
Analytics layer 5
Engagement layer 6
Understanding Spark 6
Spark libraries 7
PySpark in action 7
The Resilient Distributed Dataset 8
Understanding Anaconda 10
Setting up the Spark powered environment 12
Setting up an Oracle VirtualBox with Ubuntu 13
Installing Anaconda with Python 2.7 13
Installing Java 8 14
Installing Spark 15
Enabling IPython Notebook 16
Building our first app with PySpark 17
Virtualizing the environment with Vagrant 22
Moving to the cloud 24
Deploying apps in Amazon Web Services 24
Virtualizing the environment with Docker 24
Summary 26

[i]
Table of Contents

Chapter 2: Building Batch and Streaming Apps with Spark 27

Architecting data-intensive apps 28
Processing data at rest 29
Processing data in motion 30
Exploring data interactively 31
Connecting to social networks 31
Getting Twitter data 32
Getting GitHub data 34
Getting Meetup data 34
Analyzing the data 35
Discovering the anatomy of tweets 35
Exploring the GitHub world 40
Understanding the community through Meetup 42
Previewing our app 47
Summary 48
Chapter 3: Juggling Data with Spark 49
Revisiting the data-intensive app architecture 50
Serializing and deserializing data 51
Harvesting and storing data 51
Persisting data in CSV 52
Persisting data in JSON 54
Setting up MongoDB 55
Installing the MongoDB server and client 55
Running the MongoDB server 56
Running the Mongo client 57
Installing the PyMongo driver 58
Creating the Python client for MongoDB 58
Harvesting data from Twitter 59
Exploring data using Blaze 63
Transferring data using Odo 67
Exploring data using Spark SQL 68
Understanding Spark dataframes 69
Understanding the Spark SQL query optimizer 72
Loading and processing CSV files with Spark SQL 75
Querying MongoDB from Spark SQL 77
Summary 81

[ ii ]
 Table of Contents

Chapter 4: Learning from Data Using Spark 83

Contextualizing Spark MLlib in the app architecture 84
Classifying Spark MLlib algorithms 85
Supervised and unsupervised learning 86
Additional learning algorithms 88
Spark MLlib data types 90
Machine learning workflows and data flows 92
Supervised machine learning workflows 92
Unsupervised machine learning workflows 94
Clustering the Twitter dataset 95
Applying Scikit-Learn on the Twitter dataset 96
Preprocessing the dataset 103
Running the clustering algorithm 107
Evaluating the model and the results 108
Building machine learning pipelines 113
Summary 114
Chapter 5: Streaming Live Data with Spark 115
Laying the foundations of streaming architecture 116
Spark Streaming inner working 118
Going under the hood of Spark Streaming 120
Building in fault tolerance 124
Processing live data with TCP sockets 124
Setting up TCP sockets 124
Processing live data 125
Manipulating Twitter data in real time 128
Processing Tweets in real time from the Twitter firehose 128
Building a reliable and scalable streaming app 131
Setting up Kafka 133
Installing and testing Kafka 134
Developing producers 137
Developing consumers 139
Developing a Spark Streaming consumer for Kafka 140
Exploring flume 142
Developing data pipelines with Flume, Kafka, and Spark 143
Closing remarks on the Lambda and Kappa architecture 146
Understanding the Lambda architecture 147
Understanding the Kappa architecture 148
Summary 149

[ iii ]
Table of Contents

Chapter 6: Visualizing Insights and Trends 151

Revisiting the data-intensive apps architecture 151
Preprocessing the data for visualization 154
Gauging words, moods, and memes at a glance 160
Setting up wordcloud 160
Creating wordclouds 162
Geo-locating tweets and mapping meetups 165
Geo-locating tweets 165
Displaying upcoming meetups on Google Maps 172
Summary 178
Index 179

[ iv ]
 Preface
Spark for Python Developers aims to combine the elegance and flexibility of Python
with the power and versatility of Apache Spark. Spark is written in Scala and runs
on the Java virtual machine. It is nevertheless polyglot and offers bindings and APIs
for Java, Scala, Python, and R. Python is a well-designed language with an extensive
set of specialized libraries. This book looks at PySpark within the PyData ecosystem.
Some of the prominent PyData libraries include Pandas, Blaze, Scikit-Learn,
Matplotlib, Seaborn, and Bokeh. These libraries are open source. They are developed,
used, and maintained by the data scientist and Python developers community.
PySpark integrates well with the PyData ecosystem, as endorsed by the Anaconda
Python distribution. The book puts forward a journey to build data-intensive apps
along with an architectural blueprint that covers the following steps: first, set up the
base infrastructure with Spark. Second, acquire, collect, process, and store the data.
Third, gain insights from the collected data. Fourth, stream live data and process it in
real time. Finally, visualize the information.

The objective of the book is to learn about PySpark and PyData libraries by building
apps that analyze the Spark community's interactions on social networks. The focus
is on Twitter data.

What this book covers

Chapter 1, Setting Up a Spark Virtual Environment, covers how to create a segregated
virtual machine as our sandbox or development environment to experiment with
Spark and PyData libraries. It covers how to install Spark and the Python Anaconda
distribution, which includes PyData libraries. Along the way, we explain the key
Spark concepts, the Python Anaconda ecosystem, and build a Spark word count app.

[v]
Preface

Chapter 2, Building Batch and Streaming Apps with Spark, lays the foundation of the
Data Intensive Apps Architecture. It describes the five layers of the apps architecture
blueprint: infrastructure, persistence, integration, analytics, and engagement. We
establish API connections with three social networks: Twitter, GitHub, and Meetup.
This chapter provides the tools to connect to these three nontrivial APIs so that you
can create your own data mashups at a later stage.

Chapter 3, Juggling Data with Spark, covers how to harvest data from Twitter and
process it using Pandas, Blaze, and SparkSQL with their respective implementations
of the dataframe data structure. We proceed with further investigations and
techniques using Spark SQL, leveraging on the Spark dataframe data structure.

Chapter 4, Learning from Data Using Spark, gives an overview of the ever expanding
library of algorithms of Spark MLlib. It covers supervised and unsupervised
learning, recommender systems, optimization, and feature extraction algorithms.
We put the Twitter harvested dataset through a Python Scikit-Learn and Spark
MLlib K-means clustering in order to segregate the Apache Spark relevant tweets.

Chapter 5, Streaming Live Data with Spark, lays down the foundation of streaming
architecture apps and describes their challenges, constraints, and benefits. We
illustrate the streaming concepts with TCP sockets, followed by live tweet ingestion
and processing directly from the Twitter firehose. We also describe Flume, a reliable,
flexible, and scalable data ingestion and transport pipeline system. The combination
of Flume, Kafka, and Spark delivers unparalleled robustness, speed, and agility in an
ever-changing landscape. We end the chapter with some remarks and observations
on two streaming architectural paradigms, the Lambda and Kappa architectures.

Chapter 6, Visualizing Insights and Trends, focuses on a few key visualization

techniques. It covers how to build word clouds and expose their intuitive power
to reveal a lot of the key words, moods, and memes carried through thousands of
tweets. We then focus on interactive mapping visualizations using Bokeh. We build
a world map from the ground up and create a scatter plot of critical tweets. Our final
visualization is to overlay an actual Google map of London, highlighting upcoming
meetups and their respective topics.

What you need for this book

You need inquisitiveness, perseverance, and passion for data, software engineering,
application architecture and scalability, and beautiful succinct visualizations. The
scope is broad and wide.

You need a good understanding of Python or a similar language with object-oriented

and functional programming capabilities. Preliminary experience of data wrangling
with Python, R, or any similar tool is helpful.
[ vi ]
 Preface

You need to appreciate how to conceive, build, and scale data applications.

Who this book is for

The target audience includes the following:

• Data scientists are the primary interested parties. This book will help you
unleash the power of Spark and leverage your Python, R, and machine
learning background.
• Software developers with a focus on Python will readily expand their skills
to create data-intensive apps using Spark as a processing engine and Python
visualization libraries and web frameworks.
• Data architects who can create rapid data pipelines and build the famous
Lambda architecture that encompasses batch and streaming processing
to render insights on data in real time, using the Spark and Python rich
ecosystem, will also benefit from this book.

Conventions
In this book, you will find a number of styles of text that distinguish between
different kinds of information. Here are some examples of these styles, and an
explanation of their meaning.

Code words in text, database table names, folder names, filenames, file extensions,
pathnames, dummy URLs, user input, and Twitter handles are shown as follows
"Launch PySpark with IPYNB in directory examples/AN_Spark where the Jupyter or
IPython Notebooks are stored".

A block of code is set as follows:

# Word count on 1st Chapter of the Book using PySpark

# import regex module

import re
# import add from operator module
from operator import add

# read input file

file_in = sc.textFile('/home/an/Documents/A00_Documents/Spark4Py
20150315')

[ vii ]
Preface

Any command-line input or output is written as follows:

# install anaconda 2.x.x
bash Anaconda-2.x.x-Linux-x86[_64].sh

New terms and important words are shown in bold. Words that you see on the
screen, in menus or dialog boxes for example, appear in the text like this: "After
installing VirtualBox, let's open the Oracle VM VirtualBox Manager and click the
New button."

Warnings or important notes appear in a box like this.

Tips and tricks appear like this.

Reader feedback
Feedback from our readers is always welcome. Let us know what you think about
this book—what you liked or may have disliked. Reader feedback is important for us
to develop titles that you really get the most out of.

To send us general feedback, simply send an e-mail to [email protected],

and mention the book title via the subject of your message.

If there is a topic that you have expertise in and you are interested in either writing
or contributing to a book, see our author guide on www.packtpub.com/authors.

Customer support
Now that you are the proud owner of a Packt book, we have a number of things to
help you to get the most from your purchase.

Downloading the example code

You can download the example code files for all Packt books you have purchased
from your account at http://www.packtpub.com. If you purchased this book
elsewhere, you can visit http://www.packtpub.com/support and register to have
the files e-mailed directly to you.

[ viii ]
 Preface

Errata
Although we have taken every care to ensure the accuracy of our content, mistakes
do happen. If you find a mistake in one of our books—maybe a mistake in the text or
the code—we would be grateful if you would report this to us. By doing so, you can
save other readers from frustration and help us improve subsequent versions of this
book. If you find any errata, please report them by visiting http://www.packtpub.
com/submit-errata, selecting your book, clicking on the errata submission form link,
and entering the details of your errata. Once your errata are verified, your submission
will be accepted and the errata will be uploaded on our website, or added to any list of
existing errata, under the Errata section of that title. Any existing errata can be viewed
by selecting your title from http://www.packtpub.com/support.

Piracy
Piracy of copyright material on the Internet is an ongoing problem across all media.
At Packt, we take the protection of our copyright and licenses very seriously. If you
come across any illegal copies of our works, in any form, on the Internet, please
provide us with the location address or website name immediately so that we can
pursue a remedy.

Please contact us at [email protected] with a link to the suspected

pirated material.

We appreciate your help in protecting our authors, and our ability to bring you
valuable content.

Questions
You can contact us at [email protected] if you are having a problem with
any aspect of the book, and we will do our best to address it.

[ ix ]
 Setting Up a Spark Virtual
Environment
In this chapter, we will build an isolated virtual environment for development
purposes. The environment will be powered by Spark and the PyData libraries
provided by the Python Anaconda distribution. These libraries include Pandas,
Scikit-Learn, Blaze, Matplotlib, Seaborn, and Bokeh. We will perform the
following activities:

• Setting up the development environment using the Anaconda Python

distribution. This will include enabling the IPython Notebook environment
powered by PySpark for our data exploration tasks.
• Installing and enabling Spark, and the PyData libraries such as Pandas,
Scikit- Learn, Blaze, Matplotlib, and Bokeh.
• Building a word count example app to ensure that everything is
working fine.

The last decade has seen the rise and dominance of data-driven behemoths such as
Amazon, Google, Twitter, LinkedIn, and Facebook. These corporations, by seeding,
sharing, or disclosing their infrastructure concepts, software practices, and data
processing frameworks, have fostered a vibrant open source software community.
This has transformed the enterprise technology, systems, and software architecture.

This includes new infrastructure and DevOps (short for development and
operations), concepts leveraging virtualization, cloud technology, and
software-defined networks.

[1]
Setting Up a Spark Virtual Environment

To process petabytes of data, Hadoop was developed and open sourced, taking
its inspiration from the Google File System (GFS) and the adjoining distributed
computing framework, MapReduce. Overcoming the complexities of scaling while
keeping costs under control has also led to a proliferation of new data stores.
Examples of recent database technology include Cassandra, a columnar
database; MongoDB, a document database; and Neo4J, a graph database.

Hadoop, thanks to its ability to process huge datasets, has fostered a vast ecosystem
to query data more iteratively and interactively with Pig, Hive, Impala, and Tez.
Hadoop is cumbersome as it operates only in batch mode using MapReduce. Spark
is creating a revolution in the analytics and data processing realm by targeting the
shortcomings of disk input-output and bandwidth-intensive MapReduce jobs.

Spark is written in Scala, and therefore integrates natively with the Java Virtual
Machine (JVM) powered ecosystem. Spark had early on provided Python API and
bindings by enabling PySpark. The Spark architecture and ecosystem is inherently
polyglot, with an obvious strong presence of Java-led systems.

This book will focus on PySpark and the PyData ecosystem. Python is one of the
preferred languages in the academic and scientific community for data-intensive
processing. Python has developed a rich ecosystem of libraries and tools in data
manipulation with Pandas and Blaze, in Machine Learning with Scikit-Learn, and in
data visualization with Matplotlib, Seaborn, and Bokeh. Hence, the aim of this book
is to build an end-to-end architecture for data-intensive applications powered by
Spark and Python. In order to put these concepts in to practice, we will analyze social
networks such as Twitter, GitHub, and Meetup. We will focus on the activities and
social interactions of Spark and the Open Source Software community by tapping
into GitHub, Twitter, and Meetup.

Building data-intensive applications requires highly scalable infrastructure, polyglot

storage, seamless data integration, multiparadigm analytics processing, and efficient
visualization. The following paragraph describes the data-intensive app architecture
blueprint that we will adopt throughout the book. It is the backbone of the book.
We will discover Spark in the context of the broader PyData ecosystem.

Downloading the example code

You can download the example code files for all Packt books you have
purchased from your account at http://www.packtpub.com. If you
purchased this book elsewhere, you can visit http://www.packtpub.
com/support and register to have the files e-mailed directly to you.

[2]
 Chapter 1

Understanding the architecture of

data-intensive applications
In order to understand the architecture of data-intensive applications, the following
conceptual framework is used. The is architecture is designed on the following
five layers:

• Infrastructure layer
• Persistence layer
• Integration layer
• Analytics layer
• Engagement layer

The following screenshot depicts the five layers of the Data Intensive
App Framework:

From the bottom up, let's go through the layers and their main purpose.

[3]
Setting Up a Spark Virtual Environment

Infrastructure layer
The infrastructure layer is primarily concerned with virtualization, scalability,
and continuous integration. In practical terms, and in terms of virtualization, we
will go through building our own development environment in a VirtualBox and
virtual machine powered by Spark and the Anaconda distribution of Python. If
we wish to scale from there, we can create a similar environment in the cloud. The
practice of creating a segregated development environment and moving into test
and production deployment can be automated and can be part of a continuous
integration cycle powered by DevOps tools such as Vagrant, Chef, Puppet, and
Docker. Docker is a very popular open source project that eases the installation and
deployment of new environments. The book will be limited to building the virtual
machine using VirtualBox. From a data-intensive app architecture point of view, we
are describing the essential steps of the infrastructure layer by mentioning scalability
and continuous integration beyond just virtualization.

Persistence layer
The persistence layer manages the various repositories in accordance with data needs
and shapes. It ensures the set up and management of the polyglot data stores. It
includes relational database management systems such as MySQL and PostgreSQL;
key-value data stores such as Hadoop, Riak, and Redis; columnar databases such as
HBase and Cassandra; document databases such as MongoDB and Couchbase; and
graph databases such as Neo4j. The persistence layer manages various filesystems
such as Hadoop's HDFS. It interacts with various storage systems from native hard
drives to Amazon S3. It manages various file storage formats such as csv, json, and
parquet, which is a column-oriented format.

Integration layer
The integration layer focuses on data acquisition, transformation, quality,
persistence, consumption, and governance. It is essentially driven by the
following five Cs: connect, collect, correct, compose, and consume.

The five steps describe the lifecycle of data. They are focused on how to acquire the
dataset of interest, explore it, iteratively refine and enrich the collected information,
and get it ready for consumption. So, the steps perform the following operations:

• Connect: Targets the best way to acquire data from the various data sources,
APIs offered by these sources, the input format, input schemas if they exist,
the rate of data collection, and limitations from providers

[4]
 Chapter 1

• Correct: Focuses on transforming data for further processing and also

ensures that the quality and consistency of the data received are maintained
• Collect: Looks at which data to store where and in what format, to ease data
composition and consumption at later stages
• Compose: Concentrates its attention on how to mash up the various data sets
collected, and enrich the information in order to build a compelling data-
driven product
• Consume: Takes care of data provisioning and rendering and how the right
data reaches the right individual at the right time
• Control: This sixth additional step will sooner or later be required as the
data, the organization, and the participants grow and it is about ensuring
data governance

The following diagram depicts the iterative process of data acquisition and
refinement for consumption:

Analytics layer
The analytics layer is where Spark processes data with the various models,
algorithms, and machine learning pipelines in order to derive insights. For our
purpose, in this book, the analytics layer is powered by Spark. We will delve
deeper in subsequent chapters into the merits of Spark. In a nutshell, what makes
it so powerful is that it allows multiple paradigms of analytics processing in a
single unified platform. It allows batch, streaming, and interactive analytics. Batch
processing on large datasets with longer latency periods allows us to extract patterns
and insights that can feed into real-time events in streaming mode. Interactive and
iterative analytics are more suited for data exploration. Spark offers bindings and
APIs in Python and R. With its SparkSQL module and the Spark Dataframe, it offers
a very familiar analytics interface.

[5]
Setting Up a Spark Virtual Environment

Engagement layer
The engagement layer interacts with the end user and provides dashboards,
interactive visualizations, and alerts. We will focus here on the tools provided by
the PyData ecosystem such as Matplotlib, Seaborn, and Bokeh.

Understanding Spark
Hadoop scales horizontally as the data grows. Hadoop runs on commodity
hardware, so it is cost-effective. Intensive data applications are enabled by scalable,
distributed processing frameworks that allow organizations to analyze petabytes of
data on large commodity clusters. Hadoop is the first open source implementation
of map-reduce. Hadoop relies on a distributed framework for storage called HDFS
(Hadoop Distributed File System). Hadoop runs map-reduce tasks in batch jobs.
Hadoop requires persisting the data to disk at each map, shuffle, and reduce
process step. The overhead and the latency of such batch jobs adversely impact
the performance.

Spark is a fast, distributed general analytics computing engine for large-scale data
processing. The major breakthrough from Hadoop is that Spark allows data sharing
between processing steps through in-memory processing of data pipelines.

Spark is unique in that it allows four different styles of data analysis and processing.
Spark can be used in:

• Batch: This mode is used for manipulating large datasets, typically

performing large map-reduce jobs
• Streaming: This mode is used to process incoming information in near
real time
• Iterative: This mode is for machine learning algorithms such as a gradient
descent where the data is accessed repetitively in order to reach convergence
• Interactive: This mode is used for data exploration as large chunks of data
are in memory and due to the very quick response time of Spark

The following figure highlights the preceding four processing styles:

[6]
 Chapter 1

Spark operates in three modes: one single mode, standalone on a single machine and
two distributed modes on a cluster of machines—on Yarn, the Hadoop distributed
resource manager, or on Mesos, the open source cluster manager developed at
Berkeley concurrently with Spark:

Spark offers a polyglot interface in Scala, Java, Python, and R.

Spark libraries
Spark comes with batteries included, with some powerful libraries:

• SparkSQL: This provides the SQL-like ability to interrogate structured data

and interactively explore large datasets
• SparkMLLIB: This provides major algorithms and a pipeline framework for
machine learning
• Spark Streaming: This is for near real-time analysis of data using micro
batches and sliding widows on incoming streams of data
• Spark GraphX: This is for graph processing and computation on complex
connected entities and relationships

PySpark in action
Spark is written in Scala. The whole Spark ecosystem naturally leverages the JVM
environment and capitalizes on HDFS natively. Hadoop HDFS is one of the many
data stores supported by Spark. Spark is agnostic and from the beginning interacted
with multiple data sources, types, and formats.

PySpark is not a transcribed version of Spark on a Java-enabled dialect of Python

such as Jython. PySpark provides integrated API bindings around Spark and enables
full usage of the Python ecosystem within all the nodes of the cluster with the pickle
Python serialization and, more importantly, supplies access to the rich ecosystem of
Python's machine learning libraries such as Scikit-Learn or data processing such
as Pandas.

[7]
Setting Up a Spark Virtual Environment

When we initialize a Spark program, the first thing a Spark program must do is to
create a SparkContext object. It tells Spark how to access the cluster. The Python
program creates a PySparkContext. Py4J is the gateway that binds the Python
program to the Spark JVM SparkContext. The JVM SparkContextserializes
the application codes and the closures and sends them to the cluster for execution.
The cluster manager allocates resources and schedules, and ships the closures to
the Spark workers in the cluster who activate Python virtual machines as required.
In each machine, the Spark Worker is managed by an executor that controls
computation, storage, and cache.

Here's an example of how the Spark driver manages both the PySpark context and
the Spark context with its local filesystems and its interactions with the Spark worker
through the cluster manager:

The Resilient Distributed Dataset

Spark applications consist of a driver program that runs the user's main function,
creates distributed datasets on the cluster, and executes various parallel operations
(transformations and actions) on those datasets.

Spark applications are run as an independent set of processes, coordinated by a

SparkContext in a driver program.

The SparkContext will be allocated system resources (machines, memory, CPU)

from the Cluster manager.

[8]
 Chapter 1

The SparkContext manages executors who manage workers in the cluster.

The driver program has Spark jobs that need to run. The jobs are split into tasks
submitted to the executor for completion. The executor takes care of computation,
storage, and caching in each machine.

The key building block in Spark is the RDD (Resilient Distributed Dataset). A
dataset is a collection of elements. Distributed means the dataset can be on any node
in the cluster. Resilient means that the dataset could get lost or partially lost without
major harm to the computation in progress as Spark will re-compute from the data
lineage in memory, also known as the DAG (short for Directed Acyclic Graph) of
operations. Basically, Spark will snapshot in memory a state of the RDD in the cache.
If one of the computing machines crashes during operation, Spark rebuilds the RDDs
from the cached RDD and the DAG of operations. RDDs recover from node failure.

There are two types of operation on RDDs:

• Transformations: A transformation takes an existing RDD and leads to a

pointer of a new transformed RDD. An RDD is immutable. Once created, it
cannot be changed. Each transformation creates a new RDD. Transformations
are lazily evaluated. Transformations are executed only when an action
occurs. In the case of failure, the data lineage of transformations rebuilds
the RDD.
• Actions: An action on an RDD triggers a Spark job and yields a value. An
action operation causes Spark to execute the (lazy) transformation operations
that are required to compute the RDD returned by the action. The action
results in a DAG of operations. The DAG is compiled into stages where each
stage is executed as a series of tasks. A task is a fundamental unit of work.

Here's some useful information on RDDs:

• RDDs are created from a data source such as an HDFS file or a DB query.
There are three ways to create an RDD:
°° Reading from a datastore
°° Transforming an existing RDD
°° Using an in-memory collection

• RDDs are transformed with functions such as map or filter, which yield
new RDDs.
• An action such as first, take, collect, or count on an RDD will deliver the
results into the Spark driver. The Spark driver is the client through which
the user interacts with the Spark cluster.

[9]
Setting Up a Spark Virtual Environment

The following diagram illustrates the RDD transformation and action:

Understanding Anaconda
Anaconda is a widely used free Python distribution maintained by Continuum
(https://www.continuum.io/). We will use the prevailing software stack provided
by Anaconda to generate our apps. In this book, we will use PySpark and the
PyData ecosystem. The PyData ecosystem is promoted, supported, and maintained
by Continuum and powered by the Anaconda Python distribution. The Anaconda
Python distribution essentially saves time and aggravation in the installation of
the Python environment; we will use it in conjunction with Spark. Anaconda has
its own package management that supplements the traditional pip install and
easy-install. Anaconda comes with batteries included, namely some of the most
important packages such as Pandas, Scikit-Learn, Blaze, Matplotlib, and Bokeh. An
upgrade to any of the installed library is a simple command at the console:
$ conda update

[ 10 ]
 Chapter 1

A list of installed libraries in our environment can be obtained with command:

$ conda list

The key components of the stack are as follows:

• Anaconda: This is a free Python distribution with almost 200 Python

packages for science, math, engineering, and data analysis.
• Conda: This is a package manager that takes care of all the dependencies
of installing a complex software stack. This is not restricted to Python and
manages the install process for R and other languages.
• Numba: This provides the power to speed up code in Python with
high-performance functions and just-in-time compilation.
• Blaze: This enables large scale data analytics by offering a uniform and
adaptable interface to access a variety of data providers, which include
streaming Python, Pandas, SQLAlchemy, and Spark.
• Bokeh: This provides interactive data visualizations for large and
streaming datasets.
• Wakari: This allows us to share and deploy IPython Notebooks and other
apps on a hosted environment.

The following figure shows the components of the Anaconda stack:

[ 11 ]
Setting Up a Spark Virtual Environment

Setting up the Spark powered

environment
In this section, we will learn to set up Spark:

• Create a segregated development environment in a virtual machine running

on Ubuntu 14.04, so it does not interfere with any existing system.
• Install Spark 1.3.0 with its dependencies, namely.
• Install the Anaconda Python 2.7 environment with all the required libraries
such as Pandas, Scikit-Learn, Blaze, and Bokeh, and enable PySpark, so it can
be accessed through IPython Notebooks.
• Set up the backend or data stores of our environment. We will use MySQL as
the relational database, MongoDB as the document store, and Cassandra as
the columnar database.

Each storage backend serves a specific purpose depending on the nature of the
data to be handled. The MySQL RDBMs is used for standard tabular processed
information that can be easily queried using SQL. As we will be processing a lot of
JSON-type data from various APIs, the easiest way to store them is in a document.
For real-time and time-series-related information, Cassandra is best suited as a
columnar database.

The following diagram gives a view of the environment we will build and use
throughout the book:

[ 12 ]
 Chapter 1

Setting up an Oracle VirtualBox with Ubuntu

Setting up a clean new VirtualBox environment on Ubuntu 14.04 is the safest way to
create a development environment that does not conflict with existing libraries and
can be later replicated in the cloud using a similar list of commands.

In order to set up an environment with Anaconda and Spark, we will create a

VirtualBox virtual machine running Ubuntu 14.04.

Let's go through the steps of using VirtualBox with Ubuntu:

1. Oracle VirtualBox VM is free and can be downloaded from

https://www.virtualbox.org/wiki/Downloads. The installation
is pretty straightforward.
2. After installing VirtualBox, let's open the Oracle VM VirtualBox Manager
and click the New button.
3. We'll give the new VM a name, and select Type Linux and Version Ubuntu
(64 bit).
4. You need to download the ISO from the Ubuntu website and allocate
sufficient RAM (4 GB recommended) and disk space (20 GB recommended).
We will use the Ubuntu 14.04.1 LTS release, which is found here: http://
www.ubuntu.com/download/desktop.
5. Once the installation completed, it is advisable to install the VirtualBox
Guest Additions by going to (from the VirtualBox menu, with the new VM
running) Devices | Insert Guest Additions CD image. Failing to provide the
guest additions in a Windows host gives a very limited user interface with
reduced window sizes.
6. Once the additional installation completes, reboot the VM, and it will be
ready to use. It is helpful to enable the shared clipboard by selecting the VM
and clicking Settings, then go to General | Advanced | Shared Clipboard
and click on Bidirectional.

Installing Anaconda with Python 2.7

PySpark currently runs only on Python 2.7. (There are requests from the community
to upgrade to Python 3.3.) To install Anaconda, follow these steps:

1. Download the Anaconda Installer for Linux 64-bit Python 2.7 from
http://continuum.io/downloads#all.

[ 13 ]
Setting Up a Spark Virtual Environment

2. After downloading the Anaconda installer, open a terminal and navigate to

the directory or folder where the installer has been saved. From here, run the
following command, replacing the 2.x.x in the command with the version
number of the downloaded installer file:
# install anaconda 2.x.x
bash Anaconda-2.x.x-Linux-x86[_64].sh

3. After accepting the license terms, you will be asked to specify the install
location (which defaults to ~/anaconda).
4. After the self-extraction is finished, you should add the anaconda binary
directory to your PATH environment variable:
# add anaconda to PATH
bash Anaconda-2.x.x-Linux-x86[_64].sh

Installing Java 8
Spark runs on the JVM and requires the Java SDK (short for Software Development
Kit) and not the JRE (short for Java Runtime Environment), as we will build apps
with Spark. The recommended version is Java Version 7 or higher. Java 8 is the most
suitable, as it includes many of the functional programming techniques available
with Scala and Python.

To install Java 8, follow these steps:

1. Install Oracle Java 8 using the following commands:

# install oracle java 8
$ sudo apt-get install software-properties-common
$ sudo add-apt-repository ppa:webupd8team/java
$ sudo apt-get update
$ sudo apt-get install oracle-java8-installer

2. Set the JAVA_HOME environment variable and ensure that the Java program is
on your PATH.
3. Check that JAVA_HOME is properly installed:
#
$ echo JAVA_HOME

[ 14 ]
 Chapter 1

Installing Spark
Head over to the Spark download page at http://spark.apache.org/downloads.
html.

The Spark download page offers the possibility to download earlier versions of
Spark and different package and download types. We will select the latest release,
pre-built for Hadoop 2.6 and later. The easiest way to install Spark is to use a Spark
package prebuilt for Hadoop 2.6 and later, rather than build it from source. Move the
file to the directory ~/spark under the root directory.

Download the latest release of Spark—Spark 1.5.2, released on November 9, 2015:

1. Select Spark release 1.5.2 (Nov 09 2015),

2. Chose the package type Prebuilt for Hadoop 2.6 and later,
3. Chose the download type Direct Download,
4. Download Spark: spark-1.5.2-bin-hadoop2.6.tgz,
5. Verify this release using the 1.3.0 signatures and checksums,

This can also be accomplished by running:

# download spark
$ wget http://d3kbcqa49mib13.cloudfront.net/spark-1.5.2-bin-hadoop2.6.tgz

Next, we'll extract the files and clean up:

# extract, clean up, move the unzipped files under the spark directory
$ tar -xf spark-1.5.2-bin-hadoop2.6.tgz
$ rm spark-1.5.2-bin-hadoop2.6.tgz
$ sudo mv spark-* spark

Now, we can run the Spark Python interpreter with:

# run spark
$ cd ~/spark
./bin/pyspark

[ 15 ]
Setting Up a Spark Virtual Environment

You should see something like this:

Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 1.5.2
/_/
Using Python version 2.7.6 (default, Mar 22 2014 22:59:56)
SparkContext available as sc.
>>>

The interpreter will have already provided us with a Spark context object, sc,
which we can see by running:
>>> print(sc)
<pyspark.context.SparkContext object at 0x7f34b61c4e50>

Enabling IPython Notebook

We will work with IPython Notebook for a friendlier user experience than
the console.

You can launch IPython Notebook by using the following command:

$ IPYTHON_OPTS="notebook --pylab inline" ./bin/pyspark

Launch PySpark with IPYNB in the directory examples/AN_Spark where Jupyter or

IPython Notebooks are stored:
# cd to /home/an/spark/spark-1.5.0-bin-hadoop2.6/examples/AN_Spark
# launch command using python 2.7 and the spark-csv package:
$ IPYTHON_OPTS='notebook' /home/an/spark/spark-1.5.0-bin-hadoop2.6/bin/
pyspark --packages com.databricks:spark-csv_2.11:1.2.0

# launch command using python 3.4 and the spark-csv package:

$ IPYTHON_OPTS='notebook' PYSPARK_PYTHON=python3
/home/an/spark/spark-1.5.0-bin-hadoop2.6/bin/pyspark --packages com.
databricks:spark-csv_2.11:1.2.0

[ 16 ]
 Chapter 1

Building our first app with PySpark

We are ready to check now that everything is working fine. The obligatory word
count will be put to the test in processing a word count on the first chapter of
this book.

The code we will be running is listed here:

# Word count on 1st Chapter of the Book using PySpark

# import regex module

import re
# import add from operator module
from operator import add

# read input file

file_in = sc.textFile('/home/an/Documents/A00_Documents/Spark4Py
20150315')

# count lines
print('number of lines in file: %s' % file_in.count())

# add up lengths of each line

chars = file_in.map(lambda s: len(s)).reduce(add)
print('number of characters in file: %s' % chars)

# Get words from the input file

words =file_in.flatMap(lambda line: re.split('\W+', line.lower().
strip()))
# words of more than 3 characters
words = words.filter(lambda x: len(x) > 3)
# set count 1 per word
words = words.map(lambda w: (w,1))
# reduce phase - sum count all the words
words = words.reduceByKey(add)

In this program, we are first reading the file from the directory /home/an/
Documents/A00_Documents/Spark4Py 20150315 into file_in.

We are then introspecting the file by counting the number of lines and the number of
characters per line.

[ 17 ]
Setting Up a Spark Virtual Environment

We are splitting the input file in to words and getting them in lower case. For our
word count purpose, we are choosing words longer than three characters in order to
avoid shorter and much more frequent words such as the, and, for to skew the count
in their favor. Generally, they are considered stop words and should be filtered out
in any language processing task.

At this stage, we are getting ready for the MapReduce steps. To each word, we map a
value of 1 and reduce it by summing all the unique words.

Here are illustrations of the code in the IPython Notebook. The first 10 cells
are preprocessing the word count on the dataset, which is retrieved from the
local file directory.

[ 18 ]
 Chapter 1

Swap the word count tuples in the format (count, word) in order to sort by count,
which is now the primary key of the tuple:
# create tuple (count, word) and sort in descending
words = words.map(lambda x: (x[1], x[0])).sortByKey(False)

# take top 20 words by frequency

words.take(20)

In order to display our result, we are creating the tuple (count, word) and
displaying the top 20 most frequently used words in descending order:

[ 19 ]
Setting Up a Spark Virtual Environment

Let's create a histogram function:

# create function for histogram of most frequent words

% matplotlib inline
import matplotlib.pyplot as plt
#

def histogram(words):
count = map(lambda x: x[1], words)
word = map(lambda x: x[0], words)
plt.barh(range(len(count)), count,color = 'grey')
plt.yticks(range(len(count)), word)

# Change order of tuple (word, count) from (count, word)

words = words.map(lambda x:(x[1], x[0]))
words.take(25)

# display histogram
histogram(words.take(25))

Here, we visualize the most frequent words by plotting them in a bar chart. We have
to first swap the tuple from the original (count, word) to (word, count):

[ 20 ]
 Chapter 1

So here you have it: the most frequent words used in the first chapter are Spark,
followed by Data and Anaconda.

[ 21 ]
Setting Up a Spark Virtual Environment

Virtualizing the environment with Vagrant

In order to create a portable Python and Spark environment that can be easily shared
and cloned, the development environment can be built with a vagrantfile.

We will point to the Massive Open Online Courses (MOOCs) delivered by Berkeley
University and Databricks:

• Introduction to Big Data with Apache Spark, Professor Anthony D. Joseph can
be found at https://www.edx.org/course/introduction-big-data-
apache-spark-uc-berkeleyx-cs100-1x
• Scalable Machine Learning, Professor Ameet Talwalkar can be found at https://
www.edx.org/course/scalable-machine-learning-uc-berkeleyx-
cs190-1x

The course labs were executed on IPython Notebooks powered by PySpark. They can
be found in the following GitHub repository: https://github.com/spark-mooc/
mooc-setup/.

Once you have set up Vagrant on your machine, follow these instructions to get
started: https://docs.vagrantup.com/v2/getting-started/index.html.

Clone the spark-mooc/mooc-setup/ github repository in your work directory

and launch the command $ vagrant up, within the cloned directory:

Be aware that the version of Spark may be outdated as the vagrantfile may not be
up-to-date.

You will see an output similar to this:

C:\Programs\spark\edx1001\mooc-setup-master>vagrant up
Bringing machine 'sparkvm' up with 'virtualbox' provider...
==> sparkvm: Checking if box 'sparkmooc/base' is up to date...
==> sparkvm: Clearing any previously set forwarded ports...
==> sparkvm: Clearing any previously set network interfaces...
==> sparkvm: Preparing network interfaces based on configuration...
sparkvm: Adapter 1: nat
==> sparkvm: Forwarding ports...
sparkvm: 8001 => 8001 (adapter 1)
sparkvm: 4040 => 4040 (adapter 1)
sparkvm: 22 => 2222 (adapter 1)
==> sparkvm: Booting VM...
==> sparkvm: Waiting for machine to boot. This may take a few minutes...
sparkvm: SSH address: 127.0.0.1:2222
sparkvm: SSH username: vagrant
sparkvm: SSH auth method: private key

[ 22 ]
 Chapter 1

sparkvm: Warning: Connection timeout. Retrying...

sparkvm: Warning: Remote connection disconnect. Retrying...
==> sparkvm: Machine booted and ready!
==> sparkvm: Checking for guest additions in VM...
==> sparkvm: Setting hostname...
==> sparkvm: Mounting shared folders...
sparkvm: /vagrant => C:/Programs/spark/edx1001/mooc-setup-master
==> sparkvm: Machine already provisioned. Run `vagrant provision` or use
the `--provision`
==> sparkvm: to force provisioning. Provisioners marked to run always
will still run.

C:\Programs\spark\edx1001\mooc-setup-master>

This will launch the IPython Notebooks powered by PySpark on localhost:8001:

[ 23 ]
Setting Up a Spark Virtual Environment

Moving to the cloud

As we are dealing with distributed systems, an environment on a virtual machine
running on a single laptop is limited for exploration and learning. We can move to
the cloud in order to experience the power and scalability of the Spark distributed
framework.

Deploying apps in Amazon Web Services

Once we are ready to scale our apps, we can migrate our development environment
to Amazon Web Services (AWS).

How to run Spark on EC2 is clearly described in the following page:

https://spark.apache.org/docs/latest/ec2-scripts.html.

We emphasize five key steps in setting up the AWS Spark environment:

1. Create an AWS EC2 key pair via the AWS console http://aws.amazon.com/
console/.
2. Export your key pair to your environment:
export AWS_ACCESS_KEY_ID=accesskeyid
export AWS_SECRET_ACCESS_KEY=secretaccesskey

3. Launch your cluster:

~$ cd $SPARK_HOME/ec2
ec2$ ./spark-ec2 -k <keypair> -i <key-file> -s <num-slaves> launch
<cluster-name>

4. SSH into a cluster to run Spark jobs:

ec2$ ./spark-ec2 -k <keypair> -i <key-file> login <cluster-name>

5. Destroy your cluster after usage:

ec2$ ./spark-ec2 destroy <cluster-name>

Virtualizing the environment with Docker

In order to create a portable Python and Spark environment that can be easily shared
and cloned, the development environment can be built in Docker containers.

We wish capitalize on Docker's two main functions:

• Creating isolated containers that can be easily deployed on different

operating systems or in the cloud.

[ 24 ]
 Chapter 1

• Allowing easy sharing of the development environment image with all its
dependencies using The DockerHub. The DockerHub is similar to GitHub.
It allows easy cloning and version control. The snapshot image of the
configured environment can be the baseline for further enhancements.

The following diagram illustrates a Docker-enabled environment with Spark,

Anaconda, and the database server and their respective data volumes.

Docker offers the ability to clone and deploy an environment from the Dockerfile.

You can find an example Dockerfile with a PySpark and Anaconda setup at the
following address: https://hub.docker.com/r/thisgokeboysef/pyspark-
docker/~/dockerfile/.

[ 25 ]
Setting Up a Spark Virtual Environment

Install Docker as per the instructions provided at the following links:

• http://docs.docker.com/mac/started/ if you are on Mac OS X

• http://docs.docker.com/linux/started/ if you are on Linux
• http://docs.docker.com/windows/started/ if you are on Windows

Install the docker container with the Dockerfile provided earlier with the
following command:
$ docker pull thisgokeboysef/pyspark-docker

Other great sources of information on how to dockerize your environment can be seen
at Lab41. The GitHub repository contains the necessary code:

https://github.com/Lab41/ipython-spark-docker

The supporting blog post is rich in information on thought processes involved in

building the docker environment: http://lab41.github.io/blog/2015/04/13/
ipython-on-spark-on-docker/.

Summary
We set the context of building data-intensive apps by describing the overall
architecture structured around the infrastructure, persistence, integration, analytics,
and engagement layers. We also discussed Spark and Anaconda with their respective
building blocks. We set up an environment in a VirtualBox with Anaconda and
Spark and demonstrated a word count app using the text content of the first chapter
as input.

In the next chapter, we will delve more deeply into the architecture blueprint for
data-intensive apps and tap into the Twitter, GitHub, and Meetup APIs to get a feel
of the data we will be mining with Spark.

[ 26 ]
Building Batch and Streaming
Apps with Spark
The objective of the book is to teach you about PySpark and the PyData libraries
by building an app that analyzes the Spark community's interactions on social
networks. We will gather information on Apache Spark from GitHub, check the
relevant tweets on Twitter, and get a feel for the buzz around Spark in the broader
open source software communities using Meetup.

In this chapter, we will outline the various sources of data and information. We will
get an understanding of their structure. We will outline the data processing pipeline,
from collection to batch and streaming processing.

In this section, we will cover the following points:

• Outline data processing pipelines from collection to batch and stream

processing, effectively depicting the architecture of the app we are planning
to build.
• Check out the various data sources (GitHub, Twitter, and Meetup), their data
structure (JSON, structured information, unstructured text, geo-location,
time series data, and so on), and their complexities. We also discuss the tools
to connect to three different APIs, so you can build your own data mashups.
The book will focus on Twitter in the following chapters.

[ 27 ]
Building Batch and Streaming Apps with Spark

Architecting data-intensive apps

We defined the data-intensive app framework architecture blueprint in the previous
chapter. Let's put back in context the various software components we are going
to use throughout the book in our original framework. Here's an illustration of
the various components of software mapped in the data-intensive architecture
framework:

Spark is an extremely efficient, distributed computing framework. In order to exploit

its full power, we need to architect our solution accordingly. For performance
reasons, the overall solution needs to also be aware of its usage in terms of CPU,
storage, and network.

[ 28 ]
 Chapter 2

These imperatives drive the architecture of our solution:

• Latency: This architecture combines slow and fast processing. Slow

processing is done on historical data in batch mode. This is also called data
at rest. This phase builds precomputed models and data patterns that will
be used by the fast processing arm once live continuous data is fed into the
system. Fast processing of data or real-time analysis of streaming data refers
to data in motion. Data at rest is essentially processing data in batch mode
with a longer latency. Data in motion refers to the streaming computation of
data ingested in real time.
• Scalability: Spark is natively linearly scalable through its distributed in-
memory computing framework. Databases and data stores interacting with
Spark need to be also able to scale linearly as data volume grows.
• Fault tolerance: When a failure occurs due to hardware, software, or network
reasons, the architecture should be resilient enough and provide availability
at all times.
• Flexibility: The data pipelines put in place in this architecture can be adapted
and retrofitted very quickly depending on the use case.

Spark is unique as it allows batch processing and streaming analytics on the same
unified platform.

We will consider two data processing pipelines:

• The first one handles data at rest and is focused on putting together the
pipeline for batch analysis of the data
• The second one, data in motion, targets real-time data ingestion and
delivering insights based on precomputed models and data patterns

Processing data at rest

Let's get an understanding of the data at rest or batch processing pipeline. The
objective in this pipeline is to ingest the various datasets from Twitter, GitHub, and
Meetup; prepare the data for Spark MLlib, the machine learning engine; and derive
the base models that will be applied for insight generation in batch mode or in
real time.

[ 29 ]
Building Batch and Streaming Apps with Spark

The following diagram illustrates the data pipeline in order to enable processing data
at rest:

Processing data in motion

Processing data in motion introduces a new level of complexity, as we are
introducing a new possibility of failure. If we want to scale, we need to consider
bringing in distributed message queue systems such as Kafka. We will dedicate a
subsequent chapter to understanding streaming analytics.

The following diagram depicts a data pipeline for processing data in motion:

[ 30 ]
 Chapter 2

Exploring data interactively

Building a data-intensive app is not as straightforward as exposing a database
to a web interface. During the setup of both the data at rest and data in motion
processing, we will capitalize on Spark's ability to analyse data interactively and
refine the data richness and quality required for the machine learning and streaming
activities. Here, we will go through an iterative cycle of data collection, refinement,
and investigation in order to get to the dataset of interest for our apps.

Connecting to social networks

Let's delve into the first steps of the data-intensive app architecture's integration
layer. We are going to focus on harvesting the data, ensuring its integrity and
preparing for batch and streaming data processing by Spark at the next stage. This
phase is described in the five process steps: connect, correct, collect, compose, and
consume. These are iterative steps of data exploration that will get us acquainted with
the data and help us refine the data structure for further processing.

The following diagram depicts the iterative process of data acquisition and
refinement for consumption:

We connect to the social networks of interest: Twitter, GitHub, and Meetup. We

will discuss the mode of access to the APIs (short for Application Programming
Interface) and how to create a RESTful connection with those services while
respecting the rate limitation imposed by the social networks. REST (short for
Representation State Transfer) is the most widely adopted architectural style on the
Internet in order to enable scalable web services. It relies on exchanging messages
predominantly in JSON (short for JavaScript Object Notation). RESTful APIs and
web services implement the four most prevalent verbs GET, PUT, POST, and DELETE.
GET is used to retrieve an element or a collection from a given URI. PUT updates a
collection with a new one. POST allows the creation of a new entry, while DELETE
eliminates a collection.

[ 31 ]
Random documents with unrelated
content Scribd suggests to you:
"Not in the least; but it was a chance."
"I see!"
He saw also that she was a gentlewoman fitted for more refined
pursuits. How had she reached this pass? he wondered. Would she
volunteer the information, or should he ask her? He failed to
perceive what assistance he could render if he knew; yet if he did
not help her she would go away and die, and he would know that
she was going away to die as he let her out.
"I was introduced to the firm by a very old connection of theirs. I
couldn't find anything to do, and he fancied that as I was—well, that
as I was a lady—it sounds rather odd under the circumstances to
speak of being a lady, doesn't it——?"
"I don't see anything odd about it," he said.
"He fancied I might do rather well. But I think it's a drawback, on
the contrary. It's not easy to me to decline to take 'No' for an
answer; and nobody can do any good at work she's ashamed of."
"But you shouldn't be ashamed," he said; "it's honest enough."
"That's what the manager tells me. Only when la woman has to go
into a stranger's office and bother him, and be snubbed for her
pains, the honesty doesn't prevent her feeling uncomfortable. You
must have found me a nuisance yourself."
"I'm afraid I was rather brusque," he said quickly. "I was busy; I
hope I wasn't rude?"
Her colour rose.
"I didn't mean that at all," she stammered; "I shouldn't be very
grateful to remind you of it even if you had been!"
"I should have thought a book of that sort would have been
tolerably easy to sell. It's a useful work of reference. What's the
price?"
"Two pounds ten altogether. It isn't dear, but people won't buy it, all
the same."
"Yes, it's got up well," he said, taking it from the desk and turning
the leaves. "How many volumes, did you say?"
"Four."
She made a little tentative movement to recover it, but he went on
as if the gesture had escaped him.
"If it's not too late I'll change my mind and subscribe for a copy. Put
my name down, please, will you?"
She clasped her hands tightly in her lap.
"No," she said, "thank you, I'd rather not."
"Why?"
"You don't want the book, I know you don't. You've fed me and done
enough for me already; I won't take your money too; I can't!"
Her bosom began to swell tempestuously. He saw by the widened
eyes fixed upon the fire that she was struggling not to cry again.
"There," he said gently, "don't break down! Let's talk about
something else."
"Oh!"—she sneaked a tear away—"I'm not used ... don't think——"
"No, no," he said, "I know, I understand. Poke it for me, will you?
let's have a blaze."
She took the poker up, and prolonged the task a minute while she
hung her head.
Remarked Kincaid:
"It's awful to be hard up, isn't it? I've been through all the stages;
it's abominable!"
"You have?"
"Oh yes; I know all about it. So I don't tell you that 'money's the
least thing.' Only people who have always had enough say that."
"One wants so little in the world to relieve anxiety," she said; "it
does seem cruel that so few can get enough for ease."
"What do you mean by 'ease'?"
"Oh, I should call employment 'ease' now."
"Did you ask for more once, then?"
"Yes, I used to be more foolish. 'Experience teaches fools.'"
"No, it doesn't," said Kincaid. "Experience teaches intelligent people;
fools go on blundering to the end. 'Once——?' I interrupted you."
"Well, it used to mean a home of my own, and relations to care for
me, and money enough to settle the bills without minding if they
came to five shillings more than I had expected. It's a beautiful
regulation that the less we have, the less we can manage with. But
the horse couldn't live on the one straw."
"How did you come to this?" asked Kincaid; "couldn't you get
different work before the last straw?"
"If you knew how I tried! I haven't any friends here; that was my
difficulty. I wanted a situation as a companion, but I had to give the
idea up at last, and it ended in my going to Pattenden's. Don't think
they know! I mean, don't imagine they guess the straits I'm in: that
would be unfair. They have been very kind to me."
"You've never been a companion, I suppose?"
"No; but I hoped, for all that. Everything has to be done for the first
time; every adept was a novice once.".
"That's true, but there are so many adepts in everything to-day that
the novices haven't much chance."
"Then how are they to qualify?"
"That's the novices' affair. You can't expect people to pay
incompetence when skilled labour is loafing at the street corners."
"I expect nothing," she answered; "my expectations are all dead and
buried. We've only a certain capacity for expectation, I think; under
favourable conditions it wears well and we say, 'While there's life
there's hope;' but; when it's strained too much, it gives out."
"And you drift without a fight in you?"
"A woman can't do more than fight till she's beaten."
"She shouldn't acknowledge to being beaten."
"Theory!" she said between her teeth; "the breakfast-tray is fact!"
"What do you reckon is going to become of; you?"
"I don't anticipate at all."
"Oh, that's all rubbish! Answer straight!"
"I shall starve, then," she said.
"Sss! You know it?"
"I know it, and I'm resigned to it. If I weren't resigned to it, it would
be much harder. There's nothing that can happen to provide for me;
there isn't a soul in the world I can—'will,' to be accurate—appeal to
for help. You've delayed it a little by your kindness, but you can't
prevent its coming. Oh, I've hoped and struggled till I am worn out!"
she went on, her; voice shaking. "If there were a prospect, I could
rouse myself, weak as I am, to reach it; but there isn't a prospect,
not the glimmer of a prospect! I'm not cowardly; I'm only rational. I
admit what is; I've finished duping myself."
She could express her despair, this woman; she had education and
manner. He contemplated her attentively; she interested him.
"You speak like a fatalist, for all that," he said to her.
"I speak like a woman who has reached the lowest rung of
destitution and been fed on charity. I——Oh, don't, don't keep
forcing me to make a child of myself like this; let me go! Perhaps
you're quite right—things 'll improve."
"You shall go presently; not yet—not till I say you may."
There was silence between them once more. He lay back, with his
hands thrust deep into his trouser-pockets, and his feet crossed,
pondering.
"You weren't brought up to anything, of course?" he said abruptly.
"Never been trained to anything? You can't do anything, or make
anything, that has any market value?"
"I lived at home."
"And now you're helpless! What rot it is! Why didn't your father
teach you to use your hands?"
"I think you said you were a doctor?" she returned, lifting her head.
"Eh? Yes, my name is 'Kincaid.'"
"My father was Dr. Anthony Brettan; he never expected his daughter
to be in such want."
"You don't say so—your father was one of us? I'm glad to make your
acquaintance. Is it 'Miss Brettan'?"
She nodded, warming with an impulse to go further and cry, "Also I
have been a nurse: you are a doctor, can't you get me something to
do?" But if she did, he would require corroboration, and, in the
absence of her certificate, institute inquiries at the hospital; and then
the whisper would circulate that "Brettan was no longer living with
her husband"—they would soon ascertain that he had not died—and
from that point to the truth would be the veriest step. "Never
married at all—the disgrace! Of course, an actor, but fancy her!" She
could see their faces, the astonishment of their contempt. Narrow
circle as it was, it had been her world—she could not do it!
"But surely, Miss Brettan," he said, "there must be someone who can
serve you a little—someone who can put you in the way of an
occupation?"
Immediately she regretted having proclaimed so much as she had.
"My father lived very quietly, and socially he was hardly a popular
man. For several reasons I wouldn't like my distress to be talked
about by people who knew him."
"Those people are your credentials, though," he urged; "you can't
afford to turn your back on them. If you'll be guided by advice, you
will swallow your pride."
"I couldn't; I made the resolve to stand alone, and I shall stick to it.
Besides, you are wrong in supposing that any one of them would
exert himself for me to any extent; my father did not have—was not
intimate enough with anybody."
A difficult woman to aid, thought Kincaid pityingly. A notion had
flashed across his mind, at her reference to the kind of employment
she had desired, and the announcement of her parentage was
strengthening it; but there must be something to go upon,
something more than mere assertion.
"If a post turned up, who is there to speak for you?"
"Messrs. Pattenden; I believe they'd speak for me willingly."
"Anybody else?"
"No; but the manager would see anyone who went to him about me,
I'm almost sure."
"You need friends, you know," he said; "you're very awkwardly
placed without any."
"Oh, I do know! To have no friends is a crime; one's helpless without
them. And a woman's helplessness is the best of reasons why no
help should be extended to her. But it sounds a merciless argument,
doctor—horribly merciless, at the beginning!"
"It's a merciless life. Look here, Miss Brettan, I don't want to beat
about the bush: you're in a beastly hole, and if I can pull you out of
it I shall be glad—for your own sake, and for the sake of your dead
father. It's like this, though; the only thing I can see my way to
involves the comfort of someone else. You were talking about a
place as companion; I can't live at home now, and my mother wants
one."
"Doctor!"
She caught her breath.
"If I were to take the responsibility of recommending you, it's
probable she'd engage you; I think you'd suit her, but——Well, it's
rather a large order!"
"Oh, you should never be sorry!" she cried. "You shall never be sorry
for trusting me, if you will!"
"You see, it's not easy. It's not usual to go engaging a lady one
meets for the first time."
"Why, you wouldn't meet anybody else oftener," she pleaded
eagerly; "if you advertised, you'd take the woman after the one
interview. You wouldn't exchange a lot of visits and get friendly
before you engaged her."
He pulled at his moustache again.
"But of course she wouldn't—wouldn't be starving," she added; "she
wouldn't have fainted in your room. It'd be no more judicious, but it
would be more conventional."
"You argue neatly," he said with a smile.
The smile encouraged her. She smiled response. He could not smile
if he were going to refuse her, she felt.
"Dr. Kincaid——"
"One minute," he said; "I hear someone coming, I think. Excuse
me!"
It was Corri; he met him as he turned the handle, and drew him
outside.
"There's a woman in there," he said, "and a breakfast-tray. Come
down on to the next landing; I want to speak to you."
"What on earth——" said Corri. "Are you giving a party? What do
you mean by a woman and a breakfast-tray? Did the woman bring
the breakfast-tray?"
"No, she brought a book. It's serious."
They leant over the banisters conferring, while Mary, in the arm-
chair, remained trembling with suspense. The vista opened by
Kincaid's words had shown her how tenaciously she still clung to life,
how passionately she would clutch at a chance of prolonging it.
Awhile ago her one prayer had been to die speedily; now, with a
possibility of rescue dangled before her eyes, her prayer was only for
the possibility to be fulfilled. Would he be satisfied, or would he send
her away? Her fate swung on the decision. She did not marvel at the
tenacity; it seemed to her so natural that she did not question it at
all. Yet it is of all things the oddest—the love of living which the
most life-worn preserve in their hearts. Every day they long for
sleep, and daily the thought of death alarms them—terrifies their
inconsistent souls, though few indeed believe there is a Hell, and
everybody who is good enough to believe in Heaven believes also
that he is good enough to go to it.
"O God," she whispered, "make him take me! Forgive me what I did;
don't let me suffer any more, God! You know how I loved him—how
I loved him!"
"Well," said Corri, on the landing, "and what are you going to do?"
"I'm thinking," said Kincaid, "of letting my mother go to see her."
"It's wildly philanthropic, isn't it?"
"It looks wild, of course." He mused a moment. "But, after all, one
knows where she comes from; her father was a professional man;
she's a lady."
"What was her father's name, again?"
"Brettan—Anthony."
"Ever heard it before?"
"If there wasn't such a person, one can find it out in five minutes.
Besides, my mother would have to decide for herself. I should tell
her all about it, and if an interview left her content, why——"
"Well," said Corri, "go back to the Bench and sum up! You'll find me
on the bed. By the way, if you could hand my pipe out without
offending the young lady, I should take it as a favour."
"You've smoked enough. Wait! here's a last cigar; go and console
yourself with that!"
Kincaid returned to the room; but he was not prepared to sum up at
the moment. Mary looked at him anxiously, striving to divine, by his
expression, the result of the consultation on the stairs. The person
consulted had been Mr. Corri, she concluded, the man that she had
been sent to importune. Old or young? easy-going or morose? On
which side had he cast the weight of his opinion—this man that she
had never seen?
"We were talking about the companion's place, Miss Brettan," began
Kincaid. "Now, what do you say?"
Instantly she glowed with gratitude towards the unknown
personage, who, in reality, had done nothing.
"Never should you regret it, Dr. Kincaid, never!"
"Understand, I couldn't guarantee the engagement in any case," he
said hastily. "The most I could do would be to mention the matter;
the rest would depend on my mother's own feelings."
"I should be just as thankful to you if she objected. Don't think I
under-estimate my draw-backs—I know that for you even to
consider engaging me is generous. But——Oh, I'd do my best!—I
would indeed! The difficulty's as clear to me as to you," she went on
rapidly, "I see it every bit as plainly. See it? It has barred me from
employment again and again! I'm a stranger, I've no credentials; I
can only look you in the face and say: 'I have told you the truth; if I
were able to take your advice and pocket my pride, I could prove
that I have told you the truth,' And what's that?—anybody might say
it and be lying! Oh yes, I know! Doctor, my lack of references has
made me suspected till I could have cried blood. Doors have been
shut against me, not because I was ineligible in myself, but because
I was a woman who hadn't had employers to say, 'I found her a
satisfactory person.' Things I should have done for have been given
to other women because they had 'characters,' and I hadn't. At the
beginning I thought my tones would carry conviction—I thought I
could say: 'Honestly, this tale is true,' and someone—one in a dozen,
perhaps, one in twenty—would be found to believe me. What a
mistake, to hope to be believed! Why, in all London, there's no
creature so forsaken as a gentleman's daughter without friends. A
servant may be taken on trust; an educated woman, never!"
"She may sometimes," said Kincaid. "Hang it! it isn't so bad as all
that. What I can do for you I will! Very likely my mother will call on
you this afternoon. Where are you staying?"
A hansom had just discharged a fare at one; of the opposite houses,
and he hailed it from the window.
"The best thing you can do now is to go home and rest, and try not
to worry. Cheer up, and hope for the best, Miss Brettan—care killed
a cat!"
She swallowed convulsively.
"That is the address," she said. "God bless you, Dr. Kincaid!"
He led the way down to the passage, and put her into the cab. It
was, perhaps, superfluous to show her that he remembered that
cabs were beyond her means; yet she might be harassed during the
drive by a dread of the man's demand, and he paid him so that she
should see.
The occurrence had swelled his catalogue of calls. He told Corri they
had better drop in at Guy's, and glance at a medical directory; but in
passing a second-hand bookstall they noticed an old copy exposed
for sale, and examined that one. He found Anthony Brettan's name
in the provincial section with gladness, and remarked, moreover, that
Brettan had been a student of his own college.
"'Brettan' is going up!" he observed cheerfully. "Now step it, my
son!"
Mary's arrival at the lodging was an event of local interest. Mrs.
Shuttleworth, who stood at the door conversing with a neighbour,
watched her descent agape. Two children playing on the pavement
suspended their game. She told Mrs. Shuttleworth that a lady might
ask for her during the day, and, mounting to the garret, shut herself
in to wrestle unsuccessfully with her fears of being refused, or
forgotten altogether. Would this mother come or not? If not—she
shivered; she had been so near to ignominious death that the smell
of it had reached her nostrils—if not, the devilish gnawing would be
back again directly, and the faint sick craving would follow it; and
then there would be a fading of consciousness for the last time, and
they would talk about her as "it" and be afraid.
But the mother did come. It seemed so wonderful that, even when
she sat beside her in the attic, and everything was progressing
favourably, Mary could scarcely realise that it was true. She came,
and the engagement was made. There are some women who are
essentially women's women; Mary was one of them. Mrs. Kincaid,
who came already interested, sure that her Philip could make no
mistake and wishful to be satisfied, was charmed with her. The
pleading tones, the repose of manner, the—for so she described it
later—"Madonna face," if they did not go "straight to her heart,"
mightily pleased her fancy. And of course Mary liked her; what more
natural? She was gentle of voice, she had the softest blue eyes that
ever beamed mildly under white hair, and—culminating attraction—
she obviously liked Mary.
"I'm a lonely old woman now my son's been appointed medical
officer at the hospital," she said. "It'll be very quiet for you, but
you'll bear that, won't you? I do think you'll be comfortable with me,
and I'm sure I shall want to keep you."
"Quiet for me!" said Mary. "Oh, Mrs. Kincaid, you speak as if you
were asking a favour of me, but your son must have told you that—
what——I suppose he saved my life!"
"That's his profession," answered the old lady brightly; "that's what
he had to learn to do."
"Ah, but not with hot breakfasts," Mary smiled. "I accept your offer
gratefully; I'll come as soon as you like."
"Can you manage to go back with us the day after to-morrow? Don't
if it inconveniences you; but if you can be ready——"
"I can; I shall be quite ready."
"Good girl!" said Mrs. Kincaid. "Now you must let me advance you a
small sum, or—I daresay you have things to get—perhaps we had
better make it this! There, there! it's your own money, not a present;
there's nothing to thank me for. Good-afternoon, Miss Brettan; I will
write letting you know the train."
"This" was a five-pound note. When she was alone again Mary
picked it up, and smoothed it out, and quivered at the crackle. These
heavenly people! their tenderness, their consideration! Oh, how
beautiful it would be if they knew all about her and there were no
reservations! She did wish she could have, revealed all to them—
they had been so nice and kind.
She sought the landlady and paid her debt—the delight she felt in
paying her debt!—and said that she would be giving up her room
after the next night. She went forth to a little foreign restaurant in
Gray's Inn Road, where she dined wholesomely and well, treating
herself to cutlets, bread-crumbed and brown, and bordered with
tomatoes, to pudding and gruyere, and a cup of black coffee, all for
eighteenpence, after tipping the waiter. She returned to the attic—
glorified attic! it would never appal her any more—and abandoned
herself to meditating upon the "things." There was this, and there
was that, and there was the other. Yes, and she must have a box!
She would have had her initials painted on the box, only the paint
would look so curiously new. Should she have her initials on it? No,
she decided that she would not. Then there were her watch, and the
bag to be redeemed at the pawnbroker's, and she must say good-
bye to Mr. Collins. What a busy day would be the morrow! what a
dawn of new hope, new peace, new life! Her anxieties were left
behind; before her lay shelter and rest. Yet on a sudden the pleasure
faded from her features, and her lips twitched painfully.
"Tony!" she murmured.
She stood still where she had risen. A sob, a second sob, a torrent of
tears. She was on her knees beside the bed, gasping, shuddering,
crying out on God and him:
"O Tony, Tony, Tony!"

CHAPTER VII
The sun shone bright when she met Mrs. Kincaid at Euston. The
doctor was there, lounging loose-limbed and bony, by his mother's
side. He shook Mary's hand and remarked that it was a nice day for
travelling. She had been intending to say something grateful on
greeting him, but his manner did not invite it, so she tried to throw
her thanks into a look instead. She suffered at first from slight
embarrassment, not knowing if she should take her own ticket, nor
what assistance was expected of a companion at a railway-station.
Perhaps she ought to select the compartment, and superintend the
labelling of the luggage? Fortunately the luggage was not heavy, her
own being by far the larger portion; and the tickets, she learnt
directly, he had already got.
Her employer lived in Westport, a town that Mary had never visited,
and a little conversation arose from her questions about it. She did
not say much—she spoke very diffidently, in fact; the consciousness
that she was being paid to talk and be entertaining weighted her
tongue. She was relieved when, shortly after they started, Mrs.
Kincaid imitated her son's example, who lay back in his corner with
his face hidden behind The Lancet.
They travelled second class, and it was not till a stoppage occurred
at some junction that their privacy was invaded. Then a large
woman, oppressed by packages and baskets, entered; and, as the
new-comer belonged to the category of persons who regard a
railway journey as a heaven-sent opportunity to eat an extra meal,
her feats with sandwiches had a fascination that rivalled the interest
of the landscape.
Of course, three hours later, when the train reached Westport, Mary
felt elated. Of course she gazed eagerly from the platform over the
prospect. It was new and pleasant and refreshing. There was a little
winding road with white palings, and a cottage with a red roof. A bell
tolled softly across the meadows, and somebody standing near her
said he supposed "that was for five o'clock service." To have
exchanged the jostle of London for a place where people had time to
remember service on a week-day, to be able to catch the chirp of the
birds between the roll of the wheels, was immediately exhilarating.
Then, too, as they drove to the house she scented in the air the
freshness of tar that bespoke proximity to the sea. Her bosom lifted.
"The blessed peace of it all!" she thought; "how happy I ought to
be!"
But she was not happy. That first evening there came to her the
soreness and sickness of recollection. She was left alone with Mrs.
Kincaid, and in the twilight they sat in the pretty little parlour,
chatting fitfully. How different was this arrival from those that she
was used to! No unpacking of photographs; no landlady to chatter of
the doings of last week's company; no stroll after tea with Tony just
to see where the theatre was. How funny! She said "how funny!" but
presently she meant "how painful!" And then it came upon her as a
shock that the old life was going on still without her. Photographs
were still being unpacked and set forth on mantelpieces; landladies
still waxed garrulous of last week's business; Tony was still strolling
about the towns on Sunday evenings, as he had done when she was
with him. And he was going to be Miss Westland's husband, while
she was here! How hideous, how frightful and unreal, it seemed!
She got up, and went over to the flower-stand in the window.
"Are you tired, Miss Brettan? Perhaps you would like to go to your
room early to-night?"
"No," she said, "thank you; I'm afraid I feel a trifle strange as yet,
that's all."
At the opposite corner was a hoarding, and a comic-opera poster
shone among the local shopkeepers' advertisements. The sudden
sight of theatrical printing was like a welcome to her; she stood
looking at it, thrilling at it, with the past alive and warm again in her
heart.
"You'll soon feel at home," Mrs. Kincaid said after a pause; "I'm sure
I can understand your finding it rather uncomfortable at first."
"Oh, not uncomfortable," Mary explained quickly; "it is queer a little,
just that! I mean, I don't know what I ought to do, and I'm afraid of
seeming inattentive. What is a companion's work, Mrs. Kincaid?"
"Well, I've never had one," the old lady said with a laugh. "I think
you and I will get on best, do you know, if we forget you've come as
companion—if you talk when you like, and keep quiet when you like.
You see, it is literally a companion I want, not somebody to ring the
bell for me, and order the dinner, and make herself useful. This isn't
a big house, and I'm not a fashionable person; I want a woman
who'll keep me from moping, and be nice."
Her answer expressed her requirements, and Mary found little
expected of her in return for the salary; so little that she wondered
sometimes if she earned it, small as it was. Excepting that she was
continually conscious that she must never be out of temper, and was
frequently obliged to read aloud when she would rather have sat in
reverie, she was practically her own mistress. Even, as the days
went by she found herself giving utterance to a thought as it came
to her, without pausing to conjecture its reception; speaking with the
spontaneity which, with the paid companion, is the last thing to be
acquired.
Few pleasures are shorter-lived than the one of being restored to
enough to eat; and in a week her sense of novelty had almost worn
away. They walked together; sometimes to the sea, but more often
in the town, for the approach to the sea tired Mrs. Kincaid. Westport
was not a popular watering-place; and in the summer Mary
discovered that the population of fifty thousand was not very greatly
increased. From Laburnum Lodge it took nearly twenty minutes to
reach the shore, and a hill had to be climbed. At the top of the
incline the better-class houses came to an end; and after some
scattered cottages, an expanse of ragged grass, with a bench or
two, sloped to the beach. Despite its bareness, Mary thought the
spot delightful; its quietude appealed to her. She often wished that
she could go there by herself.
Of the doctor they saw but little. Now and again he came round for
an hour or so, and at first she absented herself on these occasions.
But Mrs. Kincaid commented on her retirement and said it was
unnecessary; and thenceforward she remained.
She did not chance to be out alone until she had been here nearly
three months; and when Mrs. Kincaid inquired one afternoon if she
would mind choosing a novel at the circulating library for her she
went forth gladly. A desire to see The Era and ascertain Carew's
whereabouts, had grown too strong to be subdued.
She crossed the interlying churchyard, and made her way along the
High Street impatiently; and, reaching the railway bookstall, bought
a copy of the current issue. It was with difficulty she restrained
herself from opening it on the platform, but she waited until she had
turned down the little lane at the station's side, and reached the
gate where the coal-trucks came to an end and a patch of green
began. She doubted whether the company would be touring so long,
but the paper would tell her something of his doings anyhow. She
ran her eye eagerly down the titles headed "On the Road." No, The
Foibles evidently was not out now. Had the tour broken up for good,
she wondered, or was there merely a vacation? She could quickly
learn by Tony's professional card. How well she knew the sheet! The
sheet! she knew the column, its very number in the column—knew it
followed "Farrell" and came before "de Vigne." She even recalled the
week when he had abandoned the cheaper advertisements in
alphabetical order; he had been cast for a part in a production. She
remembered she had said,
"Now you're going to create," and, laughing, he had answered, "Oh,
I must have half a crown's worth 'to create'!" He had been lying on
the sofa—how it all came back to her! What was he doing now? She
found the place in an instant:
"MR. SEATON CAREW,
RESTING,
Assumes direction of Miss Olive Westland's Tour, Aug. 4th.
See 'Companies' page."
They were married! She could not doubt it. "Oh," she muttered,
"how he has walked over me, that man! For the sake of two or three
thousand pounds, just for the sake of her money!" She sought
weakly for the company advertisement referred to, but the
paragraphs swam together, and it was several minutes before she
could find it. Yes, here it was: "The Foibles of Fashion and
Répertoire, opening August 4th." Camille, eh? She laughed bitterly.
He was going to play Armand; he had always wanted to play
Armand; now he could do it! "Under the direction of Mr. Seaton
Carew. Artists respectfully informed the company is complete. All
communications to be addressed: Mr. Seaton, Carew, Bath Hotel,
Bournemouth." Oh, my God!
To think that while she had been starving in that attic he had
proceeded with his courtship, to reflect that in one of those terrible
hours that she had passed through he must have been dressing
himself for his wedding, wrung her heart. And now, while she stood
here, he was calling the other woman "Olive," and kissing her. She
gripped the bar with both hands, her breast heaved tumultuously; it
seemed to her that her punishment was more than she had power
to bear. Wasn't his sin worse than her own? she questioned; yet
what price would he ever be called upon to pay for it? At most,
perhaps, occasional discontent! Nobody would-blame him a bit; his
offence was condoned already by a decent woman's hand. In the
wife's eyes she, Mary, was of course an adventuress who had turned
his weakness to account until the heroine appeared on the scene to
reclaim him. How easy it was to be the heroine when one had a few
thousand pounds to offer for a wedding-ring!
She let the paper lie where it had fallen, and went to the library. In
leaving it she met Kincaid on his way to the Lodge. He was rather
glad of the meeting, the man to whom women had been only
patients; he had felt once or twice of late that it was agreeable to
talk to Miss Brettan.
"Hallo," he said, in that voice of his that had so few inflexions; "what
have you been doing? Going home?"
"I've been to get a book for Mrs. Kincaid," she answered. "She was
hoping you'd come round to-day."
"I meant to come yesterday. Well, how are you getting on? Still
satisfied with Westport? Not beginning to tire of it yet?"
"I like it very much," she said, "naturally. It's a great change from
my life three months ago; I shouldn't be very grateful if I weren't
satisfied."
"That's all right. Your coming was a good thing; my mother was
saying the other evening it was a slice of luck."
"Oh, I'm so glad! I have wanted to know whether I—did!"
"You 'do' uncommonly; I haven't seen her so content for a long
while. You don't look very bright; d'ye feel well?"
"It's the heat," she said; "yes, I'm quite well, thank you; I have a
headache this afternoon, that's all."
She was wondering if her path and Carew's would ever cross again.
How horrible if chance brought him to the theatre here and she
came face to face with him in the High Street!
"Hasn't my mother been out to-day herself? She ought to make the
most of the fine weather."
"I left her in the garden; I think she likes that better than taking
walks."
And it might happen so easily! she reflected. Why not that company,
among the many companies that came to Westport? She'd be
frightened to leave the house.
"I suppose when you first heard there was a garden you expected to
see apple-trees and strawberry-beds, didn't you?"
"Oh, I don't know. It's not a bad little garden. We had tea in it last
night."
She might be walking with Mrs. Kincaid, and Tony and his wife would
come suddenly round a corner. And "Miss Westland" would look
contemptuous, and Tony would start, and—and if she turned white,
she'd loathe herself!
"Did you? You must enliven the old lady? a good deal if she goes in
for that sort of thing!"
"Oh, it was stuffy indoors, and we thought that tea outside would be
nicer. I daresay I'm better than no one; it must have been rather dull
for her alone."
"Is that the most you find to say of yourself—'better than no one'?"
"Well, I haven't high spirits; some women are always laughing. We
sit and read, or do needlework; or she talks about you, and——"
"And you're bored? That's a mother's privilege, you know, to bore
everybody about her son; you mustn't be hard on her."
"I am interested; I think it's always interesting to hear of a man's
work in a profession. And, then, Medicine was my father's."
"Were you the only child?"
"Yes. I wasn't much of a child, though! My mother died when I was
very young, and I was taught a lot through that. The practice wasn't
very good—very remunerative, that's to say—and if a girl's father
isn't well off she becomes a woman early. If I had had a brother now
——"
"If you had had a brother—what?"
"I was thinking it might have vmade a difference. Nothing particular.
I don't suppose he'd have been of any monetary assistance; there
wouldn't have been anything to give him a start with. But I should
have liked a brother—one older than I am."
"You'd have made the right kind of man of him, I believe."
"I was thinking of what he'd have made of me. A brother must be
such a help; a boy gets experience, and a girl has only instinct."
"It's a pretty good thing to go on with."
"It needs education, doctor, surely?"
"It needs educating by a mother. Half the women who have children
are no more fit to be mothers than——And one comes across old
maids with just the qualities! Fine material allowed to waste!"
The entrance to a cottage that they were passing stood open, and
she could see into the parlour. There were teacups on the table, and
a mug of wild flowers. On a garden-gate a child in a pink pinafore
was slowly swinging. The brilliance of the day had subsided, and the
town lay soft and yellow in the restfulness of sunset. A certain
liquidity was assumed by the rugged street in the haze that hung
over it; a touch of transparence gilded its flights of steps, the tiles of
the house-tops, and the homely faces of the fisher-folk where they
loitered before their doors. There a girl sat netting among the
hollyhocks, withholding confession from the youth who lounged
beside her, yet lifting at times to him a smile that had not been
wakened by the net. The melody of the hour intensified the discord
in the woman's soul.
"Don't you think——" said Kincaid.
He turned to her, strolling with his hands behind him. He talked to
her, and she answered him, until they reached the house.
 CHAPTER VIII
Slowly there stole into Kincaid's life a new zest. He began to be more
eager to walk round to the Lodge; was often reluctant to rise and
say "good-night"; even found the picture of the little lamplit room
lingering with him after the front door had closed. Formerly the visits
had been rather colourless. Despite her affection for her son, Mrs.
Kincaid was but tepidly interested in the career that engrossed him.
She was vaguely proud to have a doctor for a son, but she felt that
his profession supplied them with little to talk about when he came;
and the man felt that his mother's inquiries about his work were
perfunctory. A third voice had done much for the visits, quickened
the accustomed questions, the stereotyped replies into the vitality of
conversation.
Kincaid did not fail to give Miss Brettan credit for the brighter
atmosphere of the villa. But winter was at hand before he admitted
that much of the pleasure that he took in going there was inspired
by a hearty approval of Miss Brettan. The cosiness of the room, with
two women smiling at him when he entered—always with a little,
surprise, for the time of his coming was uncertain—and getting
things for him, and being sorry when he had to leave had had a
charm that he did not analyse. It was by degrees that he realised
how many of his opinions were directed to her. His one friendship
hitherto had been for Corri; and Corri was not here. The months
when his cordial liking for Mary was clear to him, and possessed of a
fascination due very largely to its unexpectedness, were perhaps the
happiest that he had known.
The development was not so happy; but it was fortunately slow. He
had gone to the house earlier than usual, and the women were
preparing for a walk. Mary stood by the mantelpiece. There was
something they had meant to do; she said she would go alone to do
it. He lay back in the depths of an arm-chair, and watched her while
she spoke to his mother, watched the play of her features and the
quick turn of her cheek. Then—it was the least significant of
trivialities—she plucked a hairpin from her hair, and began to button
her glove. It was revealed to him as he contemplated her that she
was eminently lovable. His eyes dwelt on the tender curve of her
figure, displayed by the flexion of her arm; he remarked the bend of
the head, and the delicate modelling of her ear and neck. These
things were quite new to him. He was stirred abruptly by the magic
of her sex. The admiration did not last ten seconds, and before he
saw her again he recollected it only once, quite suddenly. But the
development had begun.
In his next visit he looked to see these beauties, and found them.
This time, being voluntary, the admiration lasted longer. It was
recurrent all the evening. He discovered a novel excellence in her
performance of the simplest acts, and an additional enjoyment in
talking with her.
Thoughts of her came to him now while he sat at night in his room.
The bare little room witnessed all the phases of the man's love—its
brightness, and then its misgivings. He had no confidant to prose to;
he could never have spoken of the strange thing that had happened
to him, if he had had a confidant. He used to sit alone and think of
her, wondering if God would put it into her heart to care for him,
wondering in all humility if it could be ordained that he should ever
hold this dear woman in his arms and call her "wife."
He would not be in a position to give her luxury, and for a couple of
years certainly he could not marry at all; but he believed primarily
that he could at least make her content; and in reflecting what she
would make of life for him, he smiled. The salary that he drew from
his post was not a very large one, but his mother's means sufficed
for her requirements, and he was able to lay almost the whole of it
aside. He thought that when a couple of years had gone by, he
would be justified in furnishing a small house, and that he might
reasonably expect, through the introductions procured by his
appointment, to establish a practice. It would be rather pinched for
them at first, of course, but she wouldn't mind that much if she were
fond of him. "Fond" of him! Could it be possible? he asked himself—
Miss Brettan fond of him! She was so composed, so quiet, she
seemed such a long way off now that he wanted her for his own.
Would it really ever happen that the woman whose hand had merely
touched him in courtesy would one day be uttering words of love for
him and saying "my husband"?
He wrestled long with his tenderness; the misgivings came quickly.
After all, she was comfortable as she was—she was provided for, she
had no pecuniary cares here. Had he the right to beg her to
relinquish this comparative ease and struggle by his side oppressed
by the worries of a precarious income? Then he told himself that
they might take in patients: that would augment the income. And
she was a dependant now; if she married him she would be her own
mistress.
He weighed all the pros and cons; he was no boy to call the
recklessness of self-indulgence the splendour of devotion. He
balanced the arguments on either side long and carefully. If he
asked her to come to him, it should be with the conviction he was
doing her no wrong. He saw how easy it would be to deceive
himself, to feel persuaded that the fact of her being in a situation
made matrimony an advance to her, whether she married well or ill.
He would not act impatiently and perhaps spoil her life. But he was
very impatient. Through months he used to come away from the
Lodge striving to discern importance in some answer she had made
him, some question she had put to him. It appeared to him that he
had loved her much longer than he had; and he had made no
progress. There were moments when he upbraided himself for being
clumsy and stupid; some men in his place, he thought, would have
divined long ago what her feelings were.
He never queried the wisdom of marrying her on his own account;
the privilege of cherishing her in health and nursing her in sickness,
of having her head pillowed on his breast, and confiding his hopes to
her sympathy; of going through life with her in a union in which she
would give to him all her sacred and withheld identity, looked to him
a joy for which he could never be less than intensely grateful while
life endured. He ceased to marvel at the birth of his love, it looked
natural now; she seemed to belong to Westport so wholly by this
time. He no longer contrasted the present atmosphere of the villa
with the duller atmosphere that she had banished. He had forgotten
that duller atmosphere. She was there—it was as if she had always
been there. To reflect that there had been a period when he had
known no Mary Brettan was strange. He wondered that he had not
felt the want of her. The day that he had met her in Corri's office
appeared to him dim in the mists of at least five years. The exterior
of the man, and the yearnings within him—Kincaid as he knew
himself, and the doctor as he was known to the hospital—were so at
variance that the incongruity would have been ludicrous if it had not
been beautiful.
When Mary saw that he had begun to care for her, it was with the
greatest tremor of insecurity that she had experienced since the
date of her arrival. She had foretasted many disasters in the interval,
been harassed by many fears, but that Dr. Kincaid might fall in love
with her was a contingency that had never entered her head. It was
so utterly unexpected that for a week she had discredited the
evidence of her senses, and when the truth was too palpable to be
blinked any longer, her remaining hope was that he might decide
never to speak. Here the meditations of the man and the woman
were concerned with the same theme—both revolved the claims of
silence; but from different standpoints. His consideration was
whether avowal was unjust to her; she sustained herself by
attributing to him a reluctance to commit himself to a woman of
whom he knew so little. She clung to this haven that she had found;
her refusal, if indeed he did propose to her, would surely necessitate
her relinquishing it. Mrs. Kincaid might not desire to see her
companion marry her son, but still less would she desire to retain a
companion who had rejected him. It had been as peaceful here as
any place could be for her now, felt Mary; the thought of being
driven forth to do battle with the world again terrified her. She
wondered if Mrs. Kincaid had "noticed anything"; it was hard to
believe she could have avoided it; but she had evinced no sign of
suspicion: her manner was the same as usual.
With the complication that had arisen to disturb her, the woman
perceived how prematurely old she was. Her courage had all gone,
she told herself; she said she had passed the capability for any
sustained effort; and it was a fact that the uneventful tenor of the
life that she had been leading, congenial because it demanded no
energy, had done much to render her lassitude permanent. Her pain,
the rawness of it, had dulled—she could touch the wound now
without writhing; but it had left her wearied unto death. To attempt
to forget had been beyond her; recollection continued to be her
secret luxury; and the inertia permitted by her position lent itself so
thoroughly to a dual existence that, to her own mind, she often
seemed to be living more acutely in her reminiscences than in her
intercourse with her employer.
From the commencement of the tour, which had started in the
autumn of the preceding year, she had kept herself posted in
Carew's movements as regularly as was practicable. It was
frequently very difficult for her to gain access to a theatrical paper;
but generally she contrived to see one somehow, if not on the day
that it reached the town, then later. She knew what parts he played,
and where he played them. It was a morbid fascination, but to be
able to see his name mentioned nearly every week made her glad
that he was an actor. If he could have gone abroad or died, without
her being aware of it, she thought her situation would have been too
hideous for words. To steal that weekly glimpse of the paper was her
weekly flicker of sensation; sometimes the past seemed to stir
again; momentarily she was in the old surroundings.
There had been only two tours. After the second, she had watched
his "card" anxiously. Three months had slipped away, and between
his and his agent's name nothing had been added but the "Resting."
At last, after reading from the London paper to Mrs. Kincaid one day,
she derived some further news. A word of the theatrical gossip had
caught her eye, and unperceived by the lady, she started violently.
She had seen "Seaton Carew." For a minute she could not quell her
agitation sufficiently to pick the paper up; she sat staring down at it
and deciphering nothing. Then she learnt that Miss Olive Westland,
and her husband, Mr. Seaton Carew, encouraged by their successes
in the provinces, had completed arrangements to open the Boudoir
Theatre at the end of the following month. It was added that this of
late unfortunate house had been much embellished, and a reference
to an artist, or two already engaged showed Mary that Carew was
playing with big stakes.
Henceforth she had had a new source of information, and one
attainable without trouble, for the London paper was delivered at the
Lodge daily. As the date for the production drew near, her
impatience to hear the verdict had grown so strong that the walls of
the country parlour cooped her; she saw through them into the city
beyond—saw on to a draughty stage where Carew was conducting a
rehearsal.
The piece had failed. On the morning when she learned that it had
failed, she participated dumbly in the chagrin of the failure. "Yes"
and "No" she had answered, and seen with the eyes of her heart the
gloom of a face that used to be pressed against her own. She did
not care, she vowed; her sole feeling with regard to the undertaking
had been curiosity. If it had been more than curiosity she would
despise herself!
But she looked at the Boudoir advertisement every day. And it was
not long before she saw that another venture was in preparation.
And she held more skeins of wool, and watched with veiled
eagerness this advertisement develop like its predecessor. Recently
the play had been; produced, and she had read the notice in Mrs.
Kincaid's presence. When she finished it she guessed that Carew's
hopes were over; unless he had a great deal more money than she
supposed, the experiment at the Boudoir would see; it exhausted.
There was not much said for his performance, either; he was
dismissed in an indifferent sentence, like his wife. High praise of his
Welcome to our website – the perfect destination for book lovers and
knowledge seekers. We believe that every book holds a new world,
offering opportunities for learning, discovery, and personal growth.
That’s why we are dedicated to bringing you a diverse collection of
books, ranging from classic literature and specialized publications to
self-development guides and children's books.

More than just a book-buying platform, we strive to be a bridge

connecting you with timeless cultural and intellectual values. With an
elegant, user-friendly interface and a smart search system, you can
quickly find the books that best suit your interests. Additionally,
our special promotions and home delivery services help you save time
and fully enjoy the joy of reading.

Join us on a journey of knowledge exploration, passion nurturing, and

personal growth every day!

ebookbell.com