100% found this document useful (1 vote)
35 views

A Python Data Analyst’s Toolkit: Learn Python and Python-based Libraries with Applications in Data Analysis and Statistics Gayathri Rajagopalan - Download the full ebook set with all chapters in PDF format

The document promotes a collection of ebooks focused on Python and data analysis, featuring titles such as 'A Python Data Analyst’s Toolkit' and 'Data Analysis from Scratch with Python.' It provides links to download these resources in various formats, emphasizing accessibility and instant downloads. The content covers a range of topics from Python basics to advanced data analysis techniques.

Uploaded by

dusperkilvia
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
100% found this document useful (1 vote)
35 views

A Python Data Analyst’s Toolkit: Learn Python and Python-based Libraries with Applications in Data Analysis and Statistics Gayathri Rajagopalan - Download the full ebook set with all chapters in PDF format

The document promotes a collection of ebooks focused on Python and data analysis, featuring titles such as 'A Python Data Analyst’s Toolkit' and 'Data Analysis from Scratch with Python.' It provides links to download these resources in various formats, emphasizing accessibility and instant downloads. The content covers a range of topics from Python basics to advanced data analysis techniques.

Uploaded by

dusperkilvia
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 55

Explore the full ebook collection and download it now at textbookfull.

com

A Python Data Analyst’s Toolkit: Learn Python and


Python-based Libraries with Applications in Data
Analysis and Statistics Gayathri Rajagopalan

https://textbookfull.com/product/a-python-data-analysts-
toolkit-learn-python-and-python-based-libraries-with-
applications-in-data-analysis-and-statistics-gayathri-
rajagopalan/

OR CLICK HERE

DOWLOAD EBOOK

Browse and Get More Ebook Downloads Instantly at https://textbookfull.com


Click here to visit textbookfull.com and download textbook now
Your digital treasures (PDF, ePub, MOBI) await
Download instantly and pick your perfect format...

Read anywhere, anytime, on any device!

Data Analysis from Scratch with Python Peters Morgan

https://textbookfull.com/product/data-analysis-from-scratch-with-
python-peters-morgan/

textbookfull.com

Data Analysis with Python and PySpark (MEAP V07) Jonathan


Rioux

https://textbookfull.com/product/data-analysis-with-python-and-
pyspark-meap-v07-jonathan-rioux/

textbookfull.com

Python for Data Analysis Data Wrangling with Pandas NumPy


and IPython Wes Mckinney

https://textbookfull.com/product/python-for-data-analysis-data-
wrangling-with-pandas-numpy-and-ipython-wes-mckinney/

textbookfull.com

Python Data Analysis: Perform data collection, data


processing, wrangling, visualization, and model building
using Python 3rd Edition Avinash Navlani
https://textbookfull.com/product/python-data-analysis-perform-data-
collection-data-processing-wrangling-visualization-and-model-building-
using-python-3rd-edition-avinash-navlani/
textbookfull.com
Practical Python Data Visualization: A Fast Track Approach
To Learning Data Visualization With Python Ashwin Pajankar

https://textbookfull.com/product/practical-python-data-visualization-
a-fast-track-approach-to-learning-data-visualization-with-python-
ashwin-pajankar/
textbookfull.com

Python 2 and 3 Compatibility: With Six and Python-Future


Libraries Nanjekye

https://textbookfull.com/product/python-2-and-3-compatibility-with-
six-and-python-future-libraries-nanjekye/

textbookfull.com

Learning Data Mining with Python Layton

https://textbookfull.com/product/learning-data-mining-with-python-
layton/

textbookfull.com

A Tour Of Data Science: Learn R And Python In Parallel


Nailong Zhang

https://textbookfull.com/product/a-tour-of-data-science-learn-r-and-
python-in-parallel-nailong-zhang/

textbookfull.com

Hands-on Scikit-Learn for machine learning applications:


data science fundamentals with Python David Paper

https://textbookfull.com/product/hands-on-scikit-learn-for-machine-
learning-applications-data-science-fundamentals-with-python-david-
paper/
textbookfull.com
A Python Data
Analyst’s Toolkit
Learn Python and Python-based
Libraries with Applications in Data
Analysis and Statistics

Gayathri Rajagopalan
A Python Data
Analyst’s Toolkit
Learn Python and Python-based
Libraries with Applications in Data
Analysis and Statistics

Gayathri Rajagopalan
A Python Data Analyst’s Toolkit: Learn Python and Python-based Libraries with
Applications in Data Analysis and Statistics
Gayathri Rajagopalan

ISBN-13 (pbk): 978-1-4842-6398-3 ISBN-13 (electronic): 978-1-4842-6399-0


https://doi.org/10.1007/978-1-4842-6399-0

Copyright © 2021 by Gayathri Rajagopalan


This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the
material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation,
broadcasting, reproduction on microfilms or in any other physical way, and transmission or information
storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now
known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with
every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an
editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the
trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not
identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to
proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication,
neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or
omissions that may be made. The publisher makes no warranty, express or implied, with respect to the
material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Celestin Suresh John
Development Editor: James Markham
Coordinating Editor: Aditee Mirashi
Cover designed by eStudioCalamar
Cover image designed by Freepik (www.freepik.com)
Distributed to the book trade worldwide by Springer Science+Business Media New York, 1 New York Plaza,
Suite 4600, New York, NY 10004-1562, USA. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail
[email protected], or visit www.springeronline.com. Apress Media, LLC is a California LLC
and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM
Finance Inc is a Delaware corporation.
For information on translations, please e-mail [email protected]; for reprint,
paperback, or audio rights, please e-mail [email protected].
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook versions and
licenses are also available for most titles. For more information, reference our Print and eBook Bulk Sales
web page at http://www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is available to
readers on GitHub via the book’s product page, located at www.apress.com/978-1-4842-6398-3. For more
detailed information, please visit http://www.apress.com/source-code.
Printed on acid-free paper
This book is dedicated to my daughter.
Table of Contents
About the Author��������������������������������������������������������������������������������������������������� xiii

About the Technical Reviewer���������������������������������������������������������������������������������xv


Acknowledgments�������������������������������������������������������������������������������������������������xvii

Introduction������������������������������������������������������������������������������������������������������������xix

Chapter 1: Getting Familiar with Python������������������������������������������������������������������ 1


Technical requirements����������������������������������������������������������������������������������������������������������������� 1
Getting started with Jupyter notebooks���������������������������������������������������������������������������������������� 2
Shortcuts and other features in Jupyter���������������������������������������������������������������������������������� 5
Tab Completion������������������������������������������������������������������������������������������������������������������������ 7
Magic commands used in Jupyter������������������������������������������������������������������������������������������ 7
Python Basics������������������������������������������������������������������������������������������������������������������������������� 8
Comments, print, and input����������������������������������������������������������������������������������������������������� 8
Variables and Constants�������������������������������������������������������������������������������������������������������� 11
Operators������������������������������������������������������������������������������������������������������������������������������� 12
Data types����������������������������������������������������������������������������������������������������������������������������� 15
Working with Strings������������������������������������������������������������������������������������������������������������� 20
Conditional statements���������������������������������������������������������������������������������������������������������� 25
Loops������������������������������������������������������������������������������������������������������������������������������������� 26
Functions������������������������������������������������������������������������������������������������������������������������������� 29
Syntax errors and exceptions����������������������������������������������������������������������������������������������������� 31
Working with files����������������������������������������������������������������������������������������������������������������������� 32
Reading from a file���������������������������������������������������������������������������������������������������������������� 33
Writing to a file���������������������������������������������������������������������������������������������������������������������� 34
Modules in Python���������������������������������������������������������������������������������������������������������������������� 35

v
Table of Contents

Python Enhancement Proposal (PEP) 8 – standards for writing code����������������������������������������� 36


Summary������������������������������������������������������������������������������������������������������������������������������������ 38
Review Exercises������������������������������������������������������������������������������������������������������������������������ 39

Chapter 2: Exploring Containers, Classes, and Objects������������������������������������������ 45


Containers����������������������������������������������������������������������������������������������������������������������������������� 45
Lists��������������������������������������������������������������������������������������������������������������������������������������� 45
Tuples������������������������������������������������������������������������������������������������������������������������������������ 56
Dictionaries��������������������������������������������������������������������������������������������������������������������������� 59
Sets��������������������������������������������������������������������������������������������������������������������������������������� 63
Object-oriented programming���������������������������������������������������������������������������������������������������� 65
Object-oriented programming principles������������������������������������������������������������������������������ 67
Summary������������������������������������������������������������������������������������������������������������������������������������ 70
Review Exercises������������������������������������������������������������������������������������������������������������������������ 71

Chapter 3: Regular Expressions and Math with Python����������������������������������������� 77


Regular expressions������������������������������������������������������������������������������������������������������������������� 77
Steps for solving problems with regular expressions����������������������������������������������������������� 77
Python functions for regular expressions������������������������������������������������������������������������������ 79
Using Sympy for math problems������������������������������������������������������������������������������������������������� 86
Factorization of an algebraic expression������������������������������������������������������������������������������� 86
Solving algebraic equations (for one variable)���������������������������������������������������������������������� 87
Solving simultaneous equations (for two variables)�������������������������������������������������������������� 87
Solving expressions entered by the user������������������������������������������������������������������������������� 88
Solving simultaneous equations graphically������������������������������������������������������������������������� 89
Creating and manipulating sets��������������������������������������������������������������������������������������������� 90
Union and intersection of sets����������������������������������������������������������������������������������������������� 90
Finding the probability of an event���������������������������������������������������������������������������������������� 91
Solving questions in calculus������������������������������������������������������������������������������������������������ 92
Summary������������������������������������������������������������������������������������������������������������������������������������ 94
Review Exercises������������������������������������������������������������������������������������������������������������������������ 95

vi
Table of Contents

Chapter 4: Descriptive Data Analysis Basics�������������������������������������������������������� 101


Descriptive data analysis - Steps��������������������������������������������������������������������������������������������� 101
Structure of data����������������������������������������������������������������������������������������������������������������������� 104
Classifying data into different levels����������������������������������������������������������������������������������������� 104
Visualizing various levels of data���������������������������������������������������������������������������������������������� 106
Plotting mixed data������������������������������������������������������������������������������������������������������������� 110
Summary���������������������������������������������������������������������������������������������������������������������������������� 113
Review Exercises���������������������������������������������������������������������������������������������������������������������� 113

Chapter 5: Working with NumPy Arrays��������������������������������������������������������������� 117


Getting familiar with arrays and NumPy functions������������������������������������������������������������������� 117
Creating an array���������������������������������������������������������������������������������������������������������������������� 118
Reshaping an array������������������������������������������������������������������������������������������������������������� 121
Combining arrays���������������������������������������������������������������������������������������������������������������� 125
Testing for conditions���������������������������������������������������������������������������������������������������������� 127
Broadcasting, vectorization, and arithmetic operations������������������������������������������������������������ 130
Obtaining the properties of an array����������������������������������������������������������������������������������������� 133
Slicing or selecting a subset of data����������������������������������������������������������������������������������������� 136
Obtaining descriptive statistics/aggregate measures�������������������������������������������������������������� 138
Matrices������������������������������������������������������������������������������������������������������������������������������������ 140
Summary���������������������������������������������������������������������������������������������������������������������������������� 140
Review Exercises���������������������������������������������������������������������������������������������������������������������� 141

Chapter 6: Prepping Your Data with Pandas��������������������������������������������������������� 147


Pandas at a glance�������������������������������������������������������������������������������������������������������������������� 147
Technical requirements������������������������������������������������������������������������������������������������������������� 149
Building blocks of Pandas��������������������������������������������������������������������������������������������������������� 149
Examining the properties of a Series���������������������������������������������������������������������������������������� 152
DataFrames������������������������������������������������������������������������������������������������������������������������������� 156
Creating DataFrames by importing data from other formats���������������������������������������������� 158
Accessing attributes in a DataFrame���������������������������������������������������������������������������������� 160
Modifying DataFrame objects���������������������������������������������������������������������������������������������� 161
vii
Table of Contents

Indexing������������������������������������������������������������������������������������������������������������������������������������ 169
Type of an index object�������������������������������������������������������������������������������������������������������� 170
Creating a custom index and using columns as indexes���������������������������������������������������� 171
Indexes and speed of data retrieval������������������������������������������������������������������������������������ 173
Immutability of an index������������������������������������������������������������������������������������������������������ 174
Alignment of indexes����������������������������������������������������������������������������������������������������������� 176
Set operations on indexes��������������������������������������������������������������������������������������������������� 177
Data types in Pandas���������������������������������������������������������������������������������������������������������������� 178
Obtaining information about data types������������������������������������������������������������������������������ 179
Indexers and selection of subsets of data�������������������������������������������������������������������������������� 182
Understanding loc and iloc indexers����������������������������������������������������������������������������������� 183
Other (less commonly used) indexers for data access�������������������������������������������������������� 188
Boolean indexing for selecting subsets of data������������������������������������������������������������������� 192
Using the query method to retrieve data����������������������������������������������������������������������������� 192
Operators in Pandas������������������������������������������������������������������������������������������������������������������ 193
Representing dates and times in Pandas��������������������������������������������������������������������������������� 194
Converting strings into Pandas Timestamp objects������������������������������������������������������������ 195
Extracting the components of a Timestamp object������������������������������������������������������������� 196
Grouping and aggregation�������������������������������������������������������������������������������������������������������� 197
Examining the properties of the groupby object����������������������������������������������������������������� 199
Filtering groups������������������������������������������������������������������������������������������������������������������� 201
Transform method and groupby������������������������������������������������������������������������������������������ 202
Apply method and groupby������������������������������������������������������������������������������������������������� 204
How to combine objects in Pandas������������������������������������������������������������������������������������������� 204
Append method for adding rows����������������������������������������������������������������������������������������� 205
Concat function (adding rows or columns from other objects)������������������������������������������� 207
Join method – index to index���������������������������������������������������������������������������������������������� 210
Merge method – SQL type join based on common columns����������������������������������������������� 211

viii
Table of Contents

Restructuring data and dealing with anomalies����������������������������������������������������������������������� 213


Dealing with missing data��������������������������������������������������������������������������������������������������� 214
Data duplication������������������������������������������������������������������������������������������������������������������ 218
Tidy data and techniques for restructuring data����������������������������������������������������������������������� 220
Conversion from wide to long format (tidy data)����������������������������������������������������������������� 221
Stack method (wide-to-long format conversion)���������������������������������������������������������������� 223
Melt method (wide-to-long format conversion)������������������������������������������������������������������ 226
Pivot method (long-to-wide conversion)����������������������������������������������������������������������������� 228
Summary���������������������������������������������������������������������������������������������������������������������������������� 229
Review Exercises���������������������������������������������������������������������������������������������������������������������� 230

Chapter 7: Data Visualization with Python Libraries�������������������������������������������� 243


Technical requirements������������������������������������������������������������������������������������������������������������� 243
External files������������������������������������������������������������������������������������������������������������������������ 244
Commonly used plots���������������������������������������������������������������������������������������������������������������� 245
Matplotlib���������������������������������������������������������������������������������������������������������������������������������� 248
Approach for plotting using Matplotlib�������������������������������������������������������������������������������� 251
Plotting using Pandas��������������������������������������������������������������������������������������������������������������� 253
Scatter plot�������������������������������������������������������������������������������������������������������������������������� 254
Histogram���������������������������������������������������������������������������������������������������������������������������� 255
Pie charts���������������������������������������������������������������������������������������������������������������������������� 256
Seaborn library������������������������������������������������������������������������������������������������������������������������� 257
Box plots������������������������������������������������������������������������������������������������������������������������������ 258
Adding arguments to any Seaborn plotting function����������������������������������������������������������� 259
Kernel density estimate������������������������������������������������������������������������������������������������������� 259
Violin plot����������������������������������������������������������������������������������������������������������������������������� 260
Count plots�������������������������������������������������������������������������������������������������������������������������� 261
Heatmap������������������������������������������������������������������������������������������������������������������������������ 262
Facet grid���������������������������������������������������������������������������������������������������������������������������� 263
Regplot�������������������������������������������������������������������������������������������������������������������������������� 265

ix
Table of Contents

lmplot���������������������������������������������������������������������������������������������������������������������������������� 266
Strip plot������������������������������������������������������������������������������������������������������������������������������ 267
Swarm plot�������������������������������������������������������������������������������������������������������������������������� 268
Catplot��������������������������������������������������������������������������������������������������������������������������������� 269
Pair plot������������������������������������������������������������������������������������������������������������������������������� 270
Joint plot������������������������������������������������������������������������������������������������������������������������������ 272
Summary���������������������������������������������������������������������������������������������������������������������������������� 273
Review Exercises���������������������������������������������������������������������������������������������������������������������� 274

Chapter 8: Data Analysis Case Studies����������������������������������������������������������������� 279


Technical requirements������������������������������������������������������������������������������������������������������������� 279
Methodology����������������������������������������������������������������������������������������������������������������������������� 280
Case study 8-1: Highest grossing movies in France – analyzing unstructured data���������������� 281
Case study 8-2: Use of data analysis for air quality management�������������������������������������������� 288
Case study 8-3: Worldwide COVID-19 cases – an analysis������������������������������������������������������� 308
Summary���������������������������������������������������������������������������������������������������������������������������������� 320
Review Exercises���������������������������������������������������������������������������������������������������������������������� 321

Chapter 9: Statistics and Probability with Python����������������������������������������������� 325


Permutations and combinations����������������������������������������������������������������������������������������������� 325
Probability��������������������������������������������������������������������������������������������������������������������������������� 327
Rules of probability�������������������������������������������������������������������������������������������������������������� 328
Conditional probability��������������������������������������������������������������������������������������������������������� 330
Bayes theorem�������������������������������������������������������������������������������������������������������������������������� 330
Application of Bayes theorem in medical diagnostics��������������������������������������������������������� 331
Another application of Bayes theorem: Email spam classification�������������������������������������� 333
SciPy library������������������������������������������������������������������������������������������������������������������������������ 334
Probability distributions������������������������������������������������������������������������������������������������������������ 335
Binomial distribution����������������������������������������������������������������������������������������������������������� 335
Poisson distribution������������������������������������������������������������������������������������������������������������� 338
Continuous probability distributions������������������������������������������������������������������������������������ 341

x
Table of Contents

Normal distribution�������������������������������������������������������������������������������������������������������������� 341


Standard normal distribution����������������������������������������������������������������������������������������������� 343
Measures of central tendency��������������������������������������������������������������������������������������������������� 347
Measures of dispersion������������������������������������������������������������������������������������������������������������� 348
Measures of shape�������������������������������������������������������������������������������������������������������������������� 349
Sampling����������������������������������������������������������������������������������������������������������������������������������� 353
Probability sampling������������������������������������������������������������������������������������������������������������ 353
Non-probability sampling���������������������������������������������������������������������������������������������������� 354
Central limit theorem���������������������������������������������������������������������������������������������������������������� 355
Estimates and confidence intervals������������������������������������������������������������������������������������ 356
Types of errors in sampling������������������������������������������������������������������������������������������������������� 357
Hypothesis testing�������������������������������������������������������������������������������������������������������������������� 358
Basic concepts in hypothesis testing���������������������������������������������������������������������������������� 358
Key terminology used in hypothesis testing������������������������������������������������������������������������ 359
Steps involved in hypothesis testing����������������������������������������������������������������������������������� 361
One-sample z-test�������������������������������������������������������������������������������������������������������������������� 362
Two-sample sample z-test�������������������������������������������������������������������������������������������������������� 364
Hypothesis tests with proportions�������������������������������������������������������������������������������������������� 366
Two-sample z-test for the population proportions�������������������������������������������������������������� 368
T-distribution����������������������������������������������������������������������������������������������������������������������������� 370
One sample t-test���������������������������������������������������������������������������������������������������������������� 372
Two-sample t-test��������������������������������������������������������������������������������������������������������������� 372
Two-sample t-test for paired samples�������������������������������������������������������������������������������� 373
Solved examples: Conducting t-tests using Scipy functions���������������������������������������������������� 373
ANOVA��������������������������������������������������������������������������������������������������������������������������������������� 376
Chi-square test of association�������������������������������������������������������������������������������������������������� 379
Summary���������������������������������������������������������������������������������������������������������������������������������� 383
Review Exercises���������������������������������������������������������������������������������������������������������������������� 386

Index��������������������������������������������������������������������������������������������������������������������� 393

xi
About the Author
Gayathri Rajagopalan works for a leading Indian
multinational organization, with ten years of experience
in the software and information technology industry.
She has degrees in computer engineering and business
adminstration, and is a certified Project Management
Professional (PMP). Some of her key focus areas include
Python, data analytics, machine learning, statistics, and
deep learning. She is proficient in Python, Java, and C/C++
programming. Her hobbies include reading, music, and
teaching programming and data science to beginners.

xiii
About the Technical Reviewer
Manohar Swamynathan is a data science practitioner
and an avid programmer, with over 14 years of experience
in various data science related areas that include data
warehousing, Business Intelligence (BI), analytical tool
development, ad hoc analysis, predictive modeling, data
science product development, consulting, formulating
strategy, and executing analytics programs. He’s had a
career covering the life cycle of data across different
domains such as US mortgage banking, retail/ecommerce,
insurance, and industrial IoT. He has a bachelor’s degree
with a specialization in physics, mathematics, and
computers, and a master’s degree in project management. He’s currently living in
Bengaluru, the Silicon Valley of India.

xv
Acknowledgments
This book is a culmination of a year-long effort and would not have been possible
without my family’s support. I am indebted to them for their patience, kindness, and
encouragement.

I would also like to thank my readers for investing their time and money in this book. It is
my sincere hope that this book adds value to your learning experience.

xvii
Introduction
I had two main reasons for writing this book. When I first started learning data science,
I could not find a centralized overview of all the important topics on this subject.
A practitioner of data science needs to be proficient in at least one programming
language, learn the various aspects of data preparation and visualization, and also
be conversant with various aspects of statistics. The goal of this book is to provide
a consolidated resource that ties these interconnected disciplines together and
introduces these topics to the learner in a graded manner. Secondly, I wanted to provide
material to help readers appreciate the practical aspects of the seemingly abstract
concepts in data science, and also help them to be able to retain what they have learned.
There is a section on case studies to demonstrate how data analysis skills can be applied
to make informed decisions to solve real-world challenges. One of the highlights of
this book is the inclusion of practice questions and multiple-choice questions to help
readers practice and apply whatever they have learned. Most readers read a book and
then forget what they have read or learned, and the addition of these exercises will help
readers avoid this pitfall.

The book helps readers learn three important topics from scratch – the Python
programming language, data analysis, and statistics. It is a self-contained introduction
for anybody looking to start their journey with data analysis using Python, as it focuses
not just on theory and concepts but on practical applications and retention of concepts.
This book is meant for anybody interested in learning Python and Python-based libraries
like Pandas, Numpy, Scipy, and Matplotlib for descriptive data analysis, visualization,
and statistics. The broad categories of skills that readers learn from this book include
programming skills, analytical skills, and problem-solving skills.

The book is broadly divided into three parts – programming with Python, data analysis
and visualization, and statistics. The first part of the book comprises three chapters. It
starts with an introduction to Python – the syntax, functions, conditional statements,
data types, and different types of containers. Subsequently, we deal with advanced
concepts like regular expressions, handling of files, and solving mathematical problems

xix
Introduction

with Python. Python is covered in detail before moving on to data analysis to ensure that
the readers are comfortable with the programming language before they learn how to
use it for purposes of data analysis.

The second part of the book, comprising five chapters, covers the various aspects of
descriptive data analysis, data wrangling and visualization, and the respective Python
libraries used for each of these. There is an introductory chapter covering basic concepts
and terminology in data analysis, and one chapter each on NumPy (the scientific
computation library), Pandas (the data wrangling library), and the visualization
libraries (Matplotlib and Seaborn). A separate chapter is devoted to case studies to
help readers understand some real-world applications of data analysis. Among these
case studies is one on air pollution, using data drawn from an air quality monitoring
station in New Delhi, which has seen alarming levels of pollution in recent years. This
case study examines the trends and patterns of major air pollutants like sulfur dioxide,
nitrogen dioxide, and particulate matter for five years, and comes up with insights and
recommendations that would help with designing mitigation strategies.

The third section of this book focuses on statistics, elucidating important principles in
statistics that are relevant to data science. The topics covered include probability, Bayes
theorem, permutations and combinations, hypothesis testing (ANOVA, chi-­squared
test, z-test, and t-test), and the use of various functions in the Scipy library to enable
simplification of tedious calculations involved in statistics.

By the end of this book, the reader will be able to confidently write code in Python, use
various Python libraries and functions for analyzing any dataset, and understand basic
statistical concepts and tests. The code is presented in the form of Jupyter notebooks
that can further be adapted and extended. Readers get the opportunity to test their
understanding with a combination of multiple-choice and coding questions. They
also get an idea about how to use the skills and knowledge they have learned to make
evidence-based decisions for solving real-world problems with the help of case studies.

xx
CHAPTER 1

Getting Familiar
with Python
Python is an open source programming language created by a Dutch programmer
named Guido van Rossum. Named after the British comedy group Monty Python,
Python is a high-level, interpreted, open source language and is one of the most sought-
after and rapidly growing programming languages in the world today. It is also the
language of preference for data science and machine learning.

In this chapter, we first introduce the Jupyter notebook – a web application for running
code in Python. We then cover the basic concepts in Python, including data types,
operators, containers, functions, classes and file handling and exception handling, and
standards for writing code and modules.

The code examples for this book have been written using Python version 3.7.3 and
Anaconda version 4.7.10.

T echnical requirements
Anaconda is an open source platform used widely by Python programmers and data
scientists. Installing this platform installs Python, the Jupyter notebook application, and
hundreds of libraries. The following are the steps you need to follow for installing the
Anaconda distribution.

1. Open the following URL: https://www.anaconda.com/products/


individual

2. Click the installer for your operating system, as shown in Figure 1-1.
The installer gets downloaded to your system.

1
© Gayathri Rajagopalan 2021
G. Rajagopalan, A Python Data Analyst’s Toolkit, https://doi.org/10.1007/978-1-4842-6399-0_1
Chapter 1 Getting Familiar with Python

Figure 1-1. Installing Anaconda

3. Open the installer (file downloaded in the previous step) and run it.

4. After the installation is complete, open the Jupyter application


by typing “jupyter notebook” or “jupyter” in the explorer (search
bar) next to the start menu, as shown in Figure 1-2 (shown for
Windows OS).

Figure 1-2. Launching Jupyter

Please follow the following steps for downloading all the data files used in this book:

• Click the following link: https://github.com/DataRepo2019/


Data-files
• Select the green “Code” menu and click on “Download ZIP” from the
dropdown list of this menu
• Extract the files from the downloaded zip folder and import these
files into your Jupyter application

Now that we have installed and launched Jupyter, let us understand how to use this
application in the next section.

Getting started with Jupyter notebooks


Before we discuss the essentials of Jupyter notebooks, let us discuss what an integrated
development environment (or IDE) is. An IDE brings together the various activities
involved in programming, like including writing and editing code, debugging, and
2
Chapter 1 Getting Familiar with Python

creating executables. It also includes features like autocompletion (completing what


the user wants to type, thus enabling the user to focus on logic and problem-solving)
and syntax highlighting (highlighting the various elements and keywords of the
language). There are many IDEs for Python, apart from Jupyter, including Enthought
Canopy, Spyder, PyCharm, and Rodeo. There are several reasons for Jupyter becoming
a ubiquitous, de facto standard in the data science community. These include ease
of use and customization, support for several programming languages, platform
independence, facilitation of access to remote data, and the benefit of combining output,
code, and multimedia under one roof.

JupyterLab is the IDE for Jupyter notebooks. Jupyter notebooks are web applications that
run locally on a user’s machine. They can be used for loading, cleaning, analyzing, and
modeling data. You can add code, equations, images, and markdown text in a Jupyter
notebook. Jupyter notebooks serve the dual purpose of running your code as well as
serving as a platform for presenting and sharing your work with others. Let us look at the
various features of this application.

1. Opening the dashboard

Type “jupyter notebook” in the search bar next to the start menu.
This will open the Jupyter dashboard. The dashboard can be used
to create new notebooks or open an existing one.

2. Creating a new notebook

Create a new Jupyter notebook by selecting New from the upper


right corner of the Jupyter dashboard and then select Python 3
from the drop-down list that appears, as shown in Figure 1-3.

Figure 1-3. Creating a new Jupyter notebook

3. Entering and executing code

Click inside the first cell in your notebook and type a simple line
of code, as shown in Figure 1-4. Execute the code by selecting Run
Cells from the “Cell” menu, or use the shortcut keys Ctrl+Enter.
3
Chapter 1 Getting Familiar with Python

Figure 1-4. Simple code statement in a Jupyter cell

4. Adding markdown text or headings


In the new cell, change the formatting by selecting Markdown
as shown in Figure 1-5, or by pressing the keys Esc+M on your
keyboard. You can also add a heading to your Jupyter notebook by
selecting Heading from the drop-down list shown in the following
or pressing the shortcut keys Esc+(1/2/3/4).

Figure 1-5. Changing the mode to Markdown

5. Renaming a notebook
Click the default name of the notebook and type a new name, as
shown in Figure 1-6.

Figure 1-6. Changing the name of a file

You can also rename a notebook by selecting File ➤ Rename.


6. Saving a notebook
Press Ctrl+S or choose File ➤ Save and Checkpoint.
7. Downloading the notebook
You can email or share your notebook by downloading your
notebook using the option File ➤ Download as ➤ notebook
(.ipynb), as shown in Figure 1-7.
4
Chapter 1 Getting Familiar with Python

Figure 1-7. Downloading a Jupyter notebook

Shortcuts and other features in Jupyter


Let us look at some key features of Jupyter notebooks, including shortcuts, tab
completions, and magic commands.

Table 1-1 gives some of the familiar icons found in Jupyter notebooks, the corresponding
menu functions, and the keyboard shortcuts.

5
Chapter 1 Getting Familiar with Python

Table 1-1. Jupyter Notebook Toolbar Functions


Icon in Toolbar Function Keyboard shortcut Menu function

Saving a Jupyter notebook Esc+s File ➤ Save as

Adding a new cell to a Esc+b (adding a cell below the Insert ➤ Insert Cell
Jupyter notebook current cell), or Esc+a (adding Above or Insert ➤
a cell above the current cell) Insert Cell Below

Cutting a selected cell Esc+x Edit ➤ Cut Cells

Copying the selected cell Esc+c Edit ➤ Copy Cells

Pasting a cell above or Esc+v Edit ➤ Paste Cells


below another selected cell Above or Edit ➤
Paste Cells Below

Running a given cell Ctrl+Enter (to run selected cell); Cell ➤ Run
Shift+Enter (to run selected cell Selected Cells
and insert a new cell)

Interrupting the kernel Esc+ii Kernel ➤ Interrupt

Rebooting the kernel Esc+00 Kernel ➤ Restart

If you are not sure about which keyboard shortcut to use, go to: Help ➤ Keyboard
Shortcuts, as shown in Figure 1-8.

Figure 1-8. Help menu in Jupyter


6
Chapter 1 Getting Familiar with Python

Commonly used keyboard shortcuts include

• Shift+Enter to run the code in the current cell and move to the next
cell.

• Esc to leave a cell.

• Esc+M changes the mode for a cell to “Markdown” mode.

• Esc+Y changes the mode for a cell to “Code”.

T ab Completion
This is a feature that can be used in Jupyter notebooks to help you complete the code
being written. Usage of tab completions can speed up the workflow, reduce bugs, and
quickly complete function names, thus reducing typos and saving you from having to
remember the names of all the modules and functions.

For example, if you want to import the Matplotlib library but don’t remember the
spelling, you could type the first three letters, mat, and press Tab. You would see a drop-
down list, as shown in Figure 1-9. The correct name of the library is the second name in
the drop-down list.

Figure 1-9. Tab completion in Jupyter

Magic commands used in Jupyter


Magic commands are special commands that start with one or more % signs, followed by
a command. The commands that start with one % symbol are applicable for a single line
of code, and those beginning with two % signs are applicable for the entire cell (all lines
of code within a cell).

7
Chapter 1 Getting Familiar with Python

One commonly used magic command, shown in the following, is used to display
Matplotlib graphs inside the notebook. Adding this magic command avoids the need
to call the plt.show function separately for showing graphs (the Matplotlib library is
discussed in detail in Chapter 7).

CODE:

%matplotlib inline

Magic commands, like timeit, can also be used to time the execution of a script, as shown
in the following.

CODE:

%%timeit
for i in range(100000):
    i*i

Output:

16.1 ms ± 283 μs per loop (mean ± std. dev. of 7 runs, 100 loops each)

Now that you understand the basics of using Jupyter notebooks, let us get started with
Python and understand the core aspects of this language.

P
 ython Basics
In this section, we get familiar with the syntax of Python, commenting, conditional
statements, loops, and functions.

Comments, print, and input


In this section, we cover some basics like printing, obtaining input from the user, and
adding comments to help others understand your code.

C
 omments
A comment explains what a line of code does, and is used by programmers to help others
understand the code they have written. In Python, a comment starts with the # symbol.

8
Chapter 1 Getting Familiar with Python

Proper spacing and indentation are critical in Python. While other languages like Java
and C++ use brackets to enclose blocks of code, Python uses an indent of four spaces
to specify code blocks. One needs to take care of indents to avoid errors. Applications
like Jupyter generally take care of indentation and automatically add four spaces at the
beginning of a block of code.

Printing
The print function prints content to the screen or any other output device.

Generally, we pass a combination of strings and variables as arguments to the print


function. Arguments are the values included within the parenthesis of a function, which
the function uses for producing the result. In the following statement, “Hello!” is the
argument to the print function.

CODE:

print("Hello!")

To print multiple lines of code, we use triple quotes at the beginning and end of the
string, for example:

CODE:

print('''Today is a lovely day.


It will be warm and sunny.
It is ideal for hiking.''')

Output:

Today is a lovely day.


It will be warm and sunny.
It is ideal for hiking.

Note that we do not use semicolons in Python to end statements, unlike some other
languages.

The format method can be used in conjunction with the print method for embedding
variables within a string. It uses curly braces as placeholders for variables that are passed
as arguments to the method.

Let us look at a simple example where we print variables using the format method.
9
Chapter 1 Getting Familiar with Python

CODE:

weight=4.5
name="Simi"
print("The weight of {} is {}".format(name,weight))

Output:

The weight of Simi is 4.5

The preceding statement can also be rewritten as follows without the format method:

CODE:

print("The weight of",name,"is","weight")

Note that only the string portion of the print argument is enclosed within quotes. The name
of the variable does not come within quotes. Similarly, if you have any constants in your
print arguments, they also do not come within quotes. In the following example, a Boolean
constant (True), an integer constant (1), and strings are combined in a print statement.

CODE:

print("The integer equivalent of",True,"is",1)

Output:

The integer equivalent of True is 1

The format fields can specify precision for floating-point numbers. Floating-point
numbers are numbers with decimal points, and the number of digits after the decimal
point can be specified using format fields as follows.

CODE:

x=91.234566
print("The value of x upto 3 decimal points is {:.3f}".format(x))

Output:

The value of x upto 3 decimal points is 91.235

We can specify the position of the variables passed to the method. In this example, we
use position “1” to refer to the second object in the argument list, and position “0” to
specify the first object in the argument list.
10
Chapter 1 Getting Familiar with Python

CODE:

y='Jack'
x='Jill'
print("{1} and {0} went up the hill to fetch a pail of water".format(x,y))

Output:

Jack and Jill went up the hill to fetch a pail of water

I nput
The input function accepts inputs from the user. The input provided by the user is stored
as a variable of type String. If you want to do any mathematical calculations with any
numeric input, you need to change the data type of the input to int or float, as follows.

CODE:

age=input("Enter your age:")


print("In 2010, you were",int(age)-10,"years old")

Output:

Enter your age:76


In 2010, you were 66 years old

Further reading on Input/Output in Python: https://docs.python.org/3/tutorial/


inputoutput.html

V
 ariables and Constants
A constant or a literal is a value that does not change, while a variable contains a value
can be changed. We do not have to declare a variable in Python, that is, specify its data
type, unlike other languages like Java and C/C++. We define it by giving the variable a
name and assigning it a value. Based on the value, a data type is automatically assigned
to it. Values are stored in variables using the assignment operator (=). The rules for
naming a variable in Python are as follows:
• a variable name cannot have spaces

• a variable cannot start with a number

11
Chapter 1 Getting Familiar with Python

• a variable name can contain only letters, numbers, and underscore


signs (_)

• a variable cannot take the name of a reserved keyword (for example,


words like class, continue, break, print, etc., which are predefined
terms in the Python language, have special meanings, and are invalid
as variable names)

Operators
The following are some commonly used operators in Python.

Arithmetic operators: Take two integer or float values, perform an operation, and return
a value.

The following arithmetic operators are supported in Python:

• **(Exponent)

• %(modulo or remainder),

• //(quotient),

• *(multiplication)

• -(subtraction)

• +(addition)

The order of operations is essential. Parenthesis takes precedence over exponents,


which takes precedence over division and multiplication, which takes precedence
over addition and subtraction. An acronym was designed - P.E.D.M.A.S.(Please Excuse
My Dear Aunt Sally) - that can be used to remember the order of these operations to
understand which operator first needs to be applied in an arithmetic expression. An
example is given in the following:

CODE:

(1+9)/2-3

Output:

2.0

12
Chapter 1 Getting Familiar with Python

In the preceding expression, the operation inside the parenthesis is performed first,
which gives 10, followed by division, which gives 5, and then subtraction, which gives the
final output as 2.

Comparison operators: These operators compare two values and evaluate to a true or
false value. The following comparison operators are supported in Python:
• >: Greater than
• < : Less than
• <=: Less than or equal to
• >=: Greater than or equal to
• == : equality. Please note that this is different from the assignment
operator (=)
• !=(not equal to)

Logical (or Boolean) operators: Are similar to comparison operators in that they
also evaluate to a true or false value. These operators operate on Boolean variables or
expressions. The following logical operators are supported in Python:

• and operator: An expression in which this operator is used evaluates


to True only if all its subexpressions are True. Otherwise, if any of
them is False, the expression evaluates to False
An example of the usage of the and operator is shown in the following.
CODE:

(2>1) and (1>3)

Output:

False

• or operator: An expression in which the or operator is used, evaluates


to True if any one of the subexpressions within the expression is True.
The expression evaluates to False if all its subexpressions evaluate to
False.
An example of the usage of the or operator is shown in the following.

CODE:

(2>1) or (1>3)
13
Exploring the Variety of Random
Documents with Different Content
to be fish, which Lord Li said was dried oyster. It seems there is a
particularly large oyster in China which has a sort of bag protrusion.
This bit is cut away and sun-dried, when it makes the flavouring and
decoration for the chicken.
We had not finished yet. Duck was the next course. This came on
a plate and had its bones entire. It was also covered with thick
brown sauce and finely shredded vegetables. His Excellency told us
there were many more vegetables in China than in England, and that
some of them were prepared for export. These appeared to be
shredded in the same way as vegetables are cut for Julienne soup.
With it was also served a great dish of rice, and in ordinary Chinese
households rice is served with every course.
“In the rich homes we eat much meat and little rice, and in the
poor homes much rice and little meat,” said the Minister. This dish I
did not care for at all, besides finding it next to impossible to detach
the meat from the bones with the chopsticks.
Our next course was a very pretty one. On a plate sat a row of
little dumplings, into which lobster, finely shredded with ham, had
been daintily tucked.
I was struck by the fact that with the exception of the duck
everything had been passed through the mincing machine or
chopped. Beef, by the way, is so bad in China that it is rarely eaten.
Then followed the pudding, which was altogether a success,
entitled “Water lily.” The sweet was also served on plates. Lord Li
maintained that the foundation was rice; if so, it had been boiled so
long that it was more like tapioca. Round it were stewed pears and
peaches, and all over it little things that looked like white broad
beans. These had a delicate and delicious flavour, and I guessed a
dozen times what they could be, but in each case was wrong; and
the Minister explained they were the seeds of the lotus flower.
No wonder His Excellency lives on Chinese food at home when it is
so good and so well cooked. The native wine or spirit I did not like;
it rather reminded me of vodka.
Our meal finished we repaired to the drawing-room, where was
set out a silver tray of beautiful Chinese workmanship, with a silver
teapot and silver cups lined with white china and with ordinary
handles.
“You ladies must sit on the sofa,” said Lord Li, “for it is the fashion
in China for the host himself to dispense the tea.”
Accordingly, he lifted the entire table and placed it before us, then
poured out what appeared to be the palest green liquid.
“Surely that is not tea!” I exclaimed.
“Oh yes, it is green tea. Not green tea made for the English
market, but real green tea, uncoloured, such as we drink in China
without sugar or milk.” And, putting the spoon in the pot, he
produced the leaves, very long and broad, each one separate from
the other and absolutely devoid of stalks and dust.
“This I have sent over for me specially from my own estate,” he
said, “and this is the tea of which I drink thirty or forty cups a day.”
It was refreshing, and reminded me of the orange leaves used so
much in tropical Southern Mexico in the same way. With this ended
our quaint Eastern meal.
CHAPTER XVIII

FROM STAGELAND TO SHAKESPEARE-LAND

H OW youth adores the stage! It ever has in all climes and ages,
and probably ever will.
This was amusingly borne in on me just after my boy had gone to
Cambridge. A particular play with a particularly fascinating actress in
the principal part was announced for production there.
Of course, all Cambridge went.
A day or so later, when a lot of “men” were raving over the
beauties of the fascinating actress, buying her photographs, wanting
her autograph, and so on, one of them turned round to my son and
said:
“Isn’t she lovely? I’m just dying to be introduced to her. By Jove,
she is a ripping girl. What did you think of her, Tweedie?”
“I did not go,” he replied.
“Why not?”
“Well, you see, I know her pretty well; she went to school with my
mother.”
A bomb might have fallen.
“Went to school with your mother?”
“Yes, and she has a girl nearly as old as I am.”
Bomb number two.
Charming and pretty as she is, a woman old enough to be their
mother, she stirs the hearts of the undergrads, who, across the
footlights, innocently think she is a girl of eighteen.
So much for the delusions of the stage.
Still, it is marvellous how some actresses seem blessed with
perpetual youth.
There is no doubt about it that Miss Geneviève Ward is one of the
most remarkable women of the age. One morning in March, 1908,
came a knock at the door, and in she walked.
“Out for my constitutional, my dear,” she exclaimed, “so I thought
I would just look you up. I have walked six miles this morning, and
after a little rest and chat with you I shall walk another mile home
and enjoy my luncheon all the better for it.”
“You are a marvel!” I exclaimed. “Seven miles and over seventy. I
saw your ‘Volumnia’ was a great success the other day when you
played it with Benson.” For “Volumnia” is one of the grand old
actress’s chief parts.
“Yes,” she said, “and the next day I started for Rome. I got a
telegram to say one of three old cousins, with whom I was staying in
Rome a few weeks previously, had died suddenly; so four hours after
receiving the message I set out.”
“Were you very tired?” I asked.
“No, not at all. I knitted nearly all the way and talked to my
fellow-passengers, and when I arrived, instead of resting, went at
once to see to some business, for these two old sisters, one of
whom is blind, were absolutely prostrate with grief, and had done
nothing while awaiting my arrival. I stayed a fortnight with them,
settled them up, and arrived back two days ago.”
Miss Ward has one of the most remarkable faces I have ever
known. Her blue-grey eyes are electric. They seem to pierce one’s
very soul. They flash fire or indignation, and yet they literally melt
with love. And this great, majestic tragedienne is full of emotion and
sentiment. Geneviève Ward is the Sarah Siddons of the day. Her
“Lady Macbeth,” “Queen Eleanor,” “Queen Katherine,” and her other
classic rôles, are unrivalled. Her elocution is matchless. Her French is
as perfect as her English; anyone who ever heard her recite in
French will never forget it, and her Italian, for purity of diction, is not
far behind. On the stage her grand manner is superb. She is every
inch a queen, and yet, strange as it may appear, she is only a small
woman, five feet three at most; but so full of activity and courage
that she impresses one with immense power, height, and strength.
I happened to tell her that I had again seen an account of her
marriage in a paper.
“Some new invention,” she laughed. “And yet it is not necessary to
invent, for the romance and tragedy of my life were acute enough.”
And she then told me the following story:
“I was travelling with my mother and brother on the Riviera in
1855, when we met a Russian, Count de Guerbel. He was very tall,
very handsome, very fascinating, very rich, and twenty-eight. I was
seventeen.
“He fell in love with me, and it was settled I should be married at
the Consulate at Nice, which I was; but the Russian law required
that the marriage should be repeated in the Russian Church to make
the ceremony binding, otherwise I was his legal wife, but he was not
my legal husband.
“It was arranged, therefore, that I should go to Paris with my
mother, the Count going on in advance to arrange everything, and
we would be remarried there in the Greek Church. When we arrived
in Paris it was Lent, when no marriage can take place in the Greek
Church; and so time passed on.
“He must have been a thoroughly bad man, because he did his
best at that time to persuade me to run away with him, always
reminding me that I was his legal wife. The whole thing was merely
a trick of this handsome, fascinating rascal. He promised me that, if
I would go to him, he would take me to Russia at once, and there
we should be remarried according to the rules of the Greek Church.
Being positively frightened by his persistence, I told my mother. At
the same time rumours of de Guerbel’s amours and debts reached
her ears, and she wrote to a cousin of ours, then American Minister
in Petersburg, for confirmation of these reports.
“My cousin replied, ‘Come at once.’ We went; I, of course, under
my name of Countess de Guerbel, which I had naturally assumed
from the day of our wedding at Nice, and we stayed at the Embassy
in St. Petersburg. The Count’s brother was charming to me. He told
us my husband was a villain, and I had better leave him alone. That
was impossible, however—I was married to him, but he was not
married to me, and such a state of affairs could not remain. It
became an international matter, and was arranged by the American
Government and the Tzar that we should be officially married at
Warsaw. The Count refused to come. The Tzar therefore sent sealed
orders for his appearance. Wearing a black dress, and feeling
apprehensive and miserably sad, I went to the church, and at the
altar rails, supported by my father and mother, and the Count’s
brother, I met my husband.
“It was a horrible crisis, for I knew my father was armed with a
loaded revolver, and, if de Guerbel refused to give me the last legal
right which was morally already mine, its contents would put an end
to the adventurer’s life. There we stood, husband and wife, knowing
the service was a mere form; but the marriage was lawfully effected.
He had completed his part of the bargain and we had learned his
villainy.
“At the door of the church we parted, and I never saw him again.
We called a cab and drove direct to the railway station, and thence
travelled to Milan.”
Romance, comedy, tragedy! As I sat looking at that beautiful
woman, still beautiful at seventy, it was easy to see how lovely she
must have been at seventeen, and to picture that perfect figure in
her black frock on her bridal morning—a pathetic sight indeed!
She was continuing her story:
“Determined to do something, I at once began studying singing
for the stage on our arrival in Italy, and in a year or two made my
appearance in Paris, London, and New York.
“I made a success in opera; but in Cuba I strained my voice by
continually singing in three octaves, and one fine day discovered it
had gone. Then I took to teaching singing in New York. But,
unfortunately, I hated it; most of my pupils had neither voice nor
talent; it was like beating my head against a stone wall.
“In my operatic days critics had always mentioned my capacity for
acting. Then why not go on the stage? Thus it was at the age of
thirty-five I appeared at Manchester, under my maiden name of
Geneviève Ward, and in the end, having played Forget-me-not some
thousand times, all over the world, I retired from the profession
when I was about sixty. I have occasionally appeared since.”
This gifted tragedienne was going to Stratford to play in the
Shakespeare week in 1908.
She came to have tea with me, and as she sat beside me looking
the picture of strength and dignity, I asked if it took her long to get
up her part.
“Good heavens, no!” she replied. “I have never forgotten a
Shakespearian character in my life. Every word means something. All
I do is to read it through once or twice—perhaps three times—
before the night.”
“I own,” she said, “that sitting here now I do not recall a word of
Forget-me-not, and yet I played that several thousand times. But
then, there is nothing to grip hold of in the modern drama; however,
I could undertake to go on the stage letter-perfect even in that after
a day’s work. I am sure, after reading it through, it would all come
back to me. In Shakespeare I not only know my own part, but most
of the other people’s, and I can both remember things I learnt in my
youth and have played at intervals during my life, and memorise
now more easily than my pupils. I did so last year when I got up
those classical plays for Vedrenne and Barker.”
One cold February day Benson’s Company played Coriolanus at the
“Coronet.”
As Miss Ward had sent me the following note, I was amongst the
pleased spectators.

“Dear Mephisto,
“Here is the Box for Saturday. I hope you will enjoy
‘Volumnia.’ I love her. Come on the stage after the play, and let
me take you home.
“Yours cordially,
“Geneviève Ward.”

Her performance was simply amazing. Well rouged, with a


cheerful smile and sprightly manner, this dear lady of over seventy
looked young, handsome, animated, indeed beautiful, and buoyant
in the first act. As the play proceeded her complexion paled, her
eyes dimmed, the deep black robe and nun-like head-gear helped
the tragedy of the scene, until in the mad scene she was cringing
and yet magnificent; in the last act—thrilling.
Her clear enunciation, magnificent diction, and great repose are
indeed a contrast to the modern young woman of the stage, who
speaks so badly that one cannot hear what she says, and has often
not learnt even the first rules of walking gracefully.
After the play I went behind the scenes, as arranged. Benson was
there standing at Miss Ward’s door thanking her for her
performance.
What a splendid athlete he is in appearance, and though I am not
particularly fond of his performance, Coriolanus is by far his best. I
congratulated him upon it, and his simplicity and almost shyness
were amusing.
“But I am so much below my ideal of the part,” he said; “although
it is strengthening and broadening, I cannot even now get it,” and
then, turning to Miss Ward, added, “However, our ‘Volumnia’ is all
she should be.”
There was Miss Ward, dressed ready to return home, smiling
cheerfully and not in the least tired. As we were driving back to my
house, she told me, in answer to a friendly enquiry, what her day
had been.
“I went for a long walk this morning, had my lunch at a quarter to
one, got to the theatre at two, began at two-thirty, and, as you
know, did not end till five-thirty.”
“I hope you had some tea,” I said.
“Tea, my dear! Certainly not. I shall have a glass of hot milk at six,
when I get in, and then my dinner as usual, a little later.”
Over seventy years of age, she thus had played a strong rôle for
three hours, yet did not even need to be refreshed with a cup of tea.
Geneviève Ward certainly is a great woman.
The three greatest English actresses I have ever seen are Ellen
Terry, Geneviève Ward, and Mrs. Kendal. The latter two are among
the most brilliant women and most charming conversationalists I
know—outside their stage life I mean.
One February day in 1909, Mrs. Kendal walked up Portland Place
to fetch me en route for luncheon with Geneviève Ward.
“Why have you suddenly left the stage like this?” I asked in
banter.
In a serious voice she replied:
“Because we want no farewells. I went on the stage when I was
four, and no one knew I was there. I go off the stage when I am
fifty-five, and I do not see why people should be asked to contribute
to my well-advertised disappearance as to a charity. I’ve worked
hard for fifty years, and have retired to enjoy myself while I can.
Actors have long-drawn-out ‘farewells’ lasting for two or three years.
I don’t wish to do likewise. We’ve worked hard, and we’ve been
thrifty and saved, and now we can retire from a kindly public—as
their friends, I hope. I don’t want to write to the papers, or make
speeches, or call myself their ‘humble servant.’ I’ve given them of my
best, and they’ve paid me for it, as they pay for their hats and
gloves. No gratuities, nothing more than I have rightly earned. Don’t
you think I’m right?”
“Well, it is certainly more dignified, but we should have liked to
give you a farewell cheer.” Then, reverting to others, I asked why
Irving was so poor.
“Ah, because he was so generous. I remember an instance; when
he heard the Duchess of Manchester (afterwards Duchess of
Devonshire) had taken two stalls, he at once sent off to offer her a
private box. She accepted, and then he ordered a two-guinea
bouquet to be placed therein, and invited her to supper. Again she
accepted. He at once asked a party to meet her; that cost him over
twenty-six pounds. He told me so, and he returned the Duchess her
guinea.
“Now do you call that business? Would a dressmaker give material
gratis and entertain a customer to supper? We have never given free
seats. Why should one? If the house does not fill, change the piece,
but don’t pretend it’s a success by paper. Yes—I’m retiring; the
public doesn’t want an actress to-day. It wants a pretty girl. If I was
beginning now, instead of ending, I should be a failure. I was never
really pretty. “Men and women who have never studied acting as an
art are wanted now, young, pretty, well built. But as to acting!—the
old school of acting is a thing of the past, my dear.”
From Stageland to Shakespeare-land is a natural transit. Besides,
there is no space left in this book to describe afresh the many valued
and gifted theatrical friends to whom I devoted an entire volume in
1904, for which a second edition was called two months after
publication.
This book was Behind the Footlights, and it occurred to me to
write in it that “Mrs. Kendal was the most loved and most hated
woman on the stage.” These words might apply almost to Marie
Corelli in literature.
Who could help loving her who saw her as I did on October 6th,
1909, at the opening of Harvard House in Stratford-on-Avon?
It was a wonderful day.
A private train with bowls of flowers on every table, and smilax
hanging in long tendrils from the roof (all this being the offering of
the Railway Company), took us to Stratford at sixty-eight miles an
hour. Our engine was also gaily decked with flags and flowers and
had “HARVARD” painted across its front in big letters.
The sun shone brilliantly on that early autumn day, bestowing, as
it were, his blessing on this scholarly alliance of the Union Jack with
the Stars and Stripes.
A gracious little lady bade us welcome; short and “comely,” with
fluffy brown hair above a round face. As a girl our hostess must have
been a pretty little blonde English type—she owns the sweetest
voice imaginable, a voice to love, to coo a child to sleep, the most
gracious manners, and a delightful smile.
This was Marie Corelli, to whom the work of restoration of Harvard
House had been entrusted; and her guests that day saw it just as
John Harvard himself saw it as a child. In that house where this
most modern of twentieth-century novelists awaited her guests, the
sixteenth-century maiden Katherine Rogers, passed her early days,
and in 1605 went thence as the bride of Robert Harvard the
merchant, to his home in Southwark. Between that place and the
small country town on the Avon their little son spent his childish
years. And just as the river deepened and widened as it joined the
infinity of the ocean, so John Harvard’s youthful intelligence
deepened and widened in the great ocean of learning. Far, far away
it bore fruit—not only in his own generation, but the waves of
scholarly influence have rippled down through successive decades to
the present day, when the College he founded in America—the first
established in the New World—sends forth her men in thousands to
all parts of the globe, and the name of Harvard is an honoured
household word through the length and breadth of the world.
Although I had been twice to America and knew that the best of
the culture and learning in the United States emanated from Boston
and Harvard, I had not then realised that the famous University was
three hundred years old—contemporaneous with our own Will
Shakespeare—nor that its founder had been christened in our little
old English Mecca.
Miss Marie Corelli had a bright word for everyone; flitted hither
and thither like a bee, made speeches charmingly, and yet it must
have been a day of great nervous strain for this little lady. A woman
of taste and refinement, a woman of organisation—as the occasion
revealed, with all its details of a luncheon for a hundred and fifty
people, as well as an opening ceremony—and withal, what a
strangely imaginative mind! Almost a seer, a mystic, a religious
dreamer, a hard worker, a strange but lovable personality—such is
Marie Corelli.
Many men and women who attain great ends are egotistical—and
why not? What others admire they may surely be allowed to
appreciate also.
It is the conceit of ignorance that is so detestable. The assurance
of untutored youth that annoys.
The American Ambassador was, as ever, gentle, persuasive,
eloquent, delightful. We had a long conversation on Harvard, whose
virtues he extolled; but then Mr. Whitelaw Reid is at heart a literary
man and would-be scholar, besides having enough brains to
appreciate brains in others.
Mason Croft is Miss Marie Corelli’s home. Probably no writer of
fiction—not plays, mind you, but pure fiction—ever made so much
money, or has been so widely read, as Marie Corelli. The little girl
without fortune—by pen, ink, and paper and her own imaginative
mind—has won a lovely home. It is a fine old house, charmingly
furnished, and possesses a large meadow (the “croft“) and an
enticing winter garden. The châtelaine keeps four or five horses and
is a Lady Bountiful. Yes, and all this is done by a woman with a tiny
weapon of magic power.
So came the end of a delightful gathering—
But stop!
As Marie Corelli wrote the story of that day in a few pithy words,
let me be allowed to repeat her message to the Evening News:
“To-day, October 6th, America owns for the first time in
history a property of its own in Shakespeare’s native town.
“The ‘Harvard House,’ the gift of Mr. Edward Morris, of
Chicago, to Harvard University, was opened to-day by the
American Ambassador in the presence of a large and
representative gathering of American social magnates amid the
greatest enthusiasm.
“I am proud and glad to know that my dream of uniting the
oldest university in the States to the birth town of the Immortal
Shakespeare has been carried to a successful issue.—Marie
Corelli.”
CHAPTER XIX

ON WOMAN NOWADAYS

W OMAN nowadays. Poor dear! How she is abused, derided, called


this, that, and the other—but she goes steadily on her own
way, and she is forging ahead. This will be woman’s century.
Everything that is new, old age dubs “deterioration.” Because the
modern girl is not early Victorian, does not wear low dresses and
satin slippers by day, shriek at a mouse or faint, she is called
“unwomanly.” Surely this is ridiculous. She is stronger mentally and
physically, she is beginning to take her place in the world; and
because in the transition stage she has forgotten how to make
cordials—which she can buy so much cheaper at any Co-operative
Stores—she is styled “undomesticated.” Every age has its own
manners, and customs and ideals.
No, no, you dear old people, don’t think her unsexed. Woman’s
sphere should be the home; but her horizon must be the world.
In one sense there is nothing new under the sun. In another
everything changes, is renewed continually, and should be new.
Therefore, to call re-arrangement deterioration is absurd. It is more
often advancement. We can no more go back than we can do
without the telephone, telegraph, or taxi-cab. We are all progressing,
improving; the world is improving. Read Society books of a couple of
centuries back, and note the change. Note the coarseness of
Fielding or Smollett, and see the refinement of to-day.
It is a very good world that we live in, but youth must not be
sacrificed to old age, any more than old age must be sacrificed to
youth. Both must stand alone.
All this hue and cry about women’s work is very ridiculous. Since
the world began women have worked. They have borne the greatest
of all burdens—child-bearing; and they have cooked and washed
and mended and made. They have ministered to the wants of man
and home.
Worked? Why, of course they have worked, but they have not
always been paid. Now is their day. They are strong enough to
demand the recognition the world has been ungenerous enough to
withhold.
Equality in all things for the sexes will make happier men and
women, happier homes, and a more prosperous nation.
All women cannot be bread-winners any more than all men can be
soldiers. Women are marching onward in every land, their
advancement and the progress of civilisation are synonymous terms
to-day.
The greater the women, the greater the country.
It is ridiculous to say that women workers oust men. This is hardly
ever the case. In these days of endless change, when a machine is
frequently introduced that does the work of four or five men, labour
is constantly re-arranged. Then again, with increase of work, so
there is incessant all-round shifting of the distribution of
employment. Women do not take the place of men. They merely find
their own footing in the general change. There is a niche for
everyone ready to fill it.
Yes, women do work, and women must work, although a vast
amount of misery might be, and ought to be, alleviated by their
men-folk. The present disastrous state of things is largely due to
men not providing for their wives or equipping their daughters to be
wage-earners.
There are, of course, a few enthusiastic women who work for
work’s sake, but they take the bread out of no man’s mouth. These
are the writers of deep and profound books, who make as many
shillings as they spend pounds in collecting their material—women
who love research work in science; women who labour among the
poor, organise clubs and homes, and devote their lives to charity and
good deeds; but the cases are rare, almost nil, where women work
for salary who do not need the money. Those who do certainly take
the bread from the mouths of men and women alike; but the rich
workers who accept pay are so few they do not count.
Many women with small incomes seek to increase those incomes
in order to clothe their children, pay the butcher, or have more to
spend on little luxuries, but these, again, are a small class. The large
multitude of women who work are those who must do so, and they
are the ones who require help, for theirs is an uphill fight against
great odds. They have to contend with want of general education,
want of special training, want of physical strength, want of positions
open to women, when they enter the already overcrowded field of
labour.
Women must work until men realise the responsibility of thrusting
them unequipped into the sea of life to sink or swim on the tide of
chance.
How bravely women do it too. Aching hearts and throbbing brows
are forgotten in the fight for daily existence. Poor souls, how hard
many of them toil, how lonely are their lives, and what a struggle it
is for them to keep their heads above water. Many of them do so,
however; and to them all honour is due.
Men and women should never be pitted as rivals in anything. Each
sex has its own place to fill; but when the exigencies of fighting for
existence occur, men should nobly help the courageous woman
worker over the difficulties her men-folk have thoughtlessly placed
before her.
I hate sex. Surely, in working, thinking, human beings—it does not
matter whether one wears petticoats or trousers—there should be
no sex as regards bread-earning. There are a million and a quarter
too many women in England, and the gates of independence and
occupation must not be shut in their faces. Personally, I should like
boys and girls to be equal in everything. Forget sex, bring them up
together, educate them together. Send them to public schools and
Universities together, open all the trades and professions to women
the same as to men. Let them stand shoulder to shoulder.
Many people thought that the heavens would descend if a woman
became a doctor. They were wrong. Women are doing well in
medicine and surgery, though they are still excluded from the Bar
and the Church.
Yes, give girls just the same advantages as boys. Divide your
incomes equally amongst all your children when you die, irrespective
of sex. Give them equality in divorce. The world will be all the
happier.
Women will find their own level—just as men do; they will make or
mar their own lives—just as men do. But let men cease shutting
gates of employment in their faces.
A nation’s power depends on the physical strength and character
of its women, and not on its army of men, or its statesmen.
How I envy men with professions. They come down to
comfortable breakfasts, without the least idea of what will be laid
before them. They enjoy it, have a look at the papers, perhaps a
pipe, and then they get into boots and top-coat, go off to their
chambers, offices, studios, or their consulting-rooms, as the case
may be. They throw themselves into their work, knowing that no
interruptions will occur during the whole course of the morning.
They enjoy their luncheon, which they have not had the worry of
ordering beforehand, and so by the time four, or five, or six o’clock
arrives they have done a good day’s work without annoyance from
outside. They have earned so much money, and not far off they see
a tangible reward. Lucky men!
How differently things go with a woman like myself, with a small
income, a house, servants, children, all as important as the daily
round of wage-earning. By the time one gets settled down to one’s
desk at nine-thirty or ten o’clock one has gone through the drudgery
of it all. The orders and wants of cook, housemaid, parlourmaid, and
nurse have all been attended to. The cheques for washing bills and
grocers’ books have to be written, orders sent for coals, the soda-
water telephoned for, with all the endless round of wearying details
which every housekeeper knows. In the midst of one’s morning
work, curtains return from the cleaners, and have to be paid for at
the door, or a man comes to mend the bell, and one has to leave
one’s desk to show him exactly what is wrong. In fact, the
interruptions are incessant even in the best regulated households,
and one has to bring one’s distracted mind back from domestic
details to write important letters or articles for the Press.
A working woman’s life would be endurable were it not for the
interruptions.
Yes! I have lived the ordinary woman’s life and the professional
woman’s life as well, and I always say to myself that the professional
part is a mere bagatelle, because of the larger rewards, in
comparison with the ceaseless worries and endless interruptions that
fall at the feet of every housekeeper.
Men do not half enough appreciate the amount of work (becoming
every year more difficult), the extraordinary number of little details,
necessary to run even the simplest home.
When one covers one’s own furniture, embroiders one’s own
cloths, and trims one’s own hats into the bargain, the daily round
becomes complicated indeed.
I believe in clubs for women. It is so heavenly to get away from
an ordinary dinner. It is really a holiday to have a chop or a fried
sole, that one has not ordered hours beforehand. Besides, at the
club one sometimes learns new dishes, and certainly new ideas from
the newspapers and magazines, all of which one could not afford to
take in at home independently.
For the unmarried woman the club is absolutely indispensable. It
gives her a place where she can receive her friends, and let it be
known that women are more hospitable than men. They are poorer,
but are more generous in giving invitations to tea or a meal. Men’s
clubs are full of old women, and women’s clubs full of young men,
nowadays.
A club is also a boon to the married woman, for there are days
when country relations arrive in town, when, for instance, the sweep
has been ordered at home; then the country or foreign friends can
be taken to the club, and need not know that their hostess’s small
household cannot tackle a luncheon because of the advent of the
sweep.
I believe clubs encourage women to read, and I am sure that
expands their ideas and opens their minds. Women’s clubs are
certainly an advantage, and though I have been an original member
of several, I always float back to my first love, the Albemarle, where
our marble halls, once the Palace of the Bishop of Ely, receive both
men and women members.
I love my own sex. They are the guiding stars of the Universe, and
the modern girl tends to make the world much more interesting than
it used to be. Youth must spread its wings, and if it is sound youth it
will be gently guided by experience. Let the bird fly, or it will fret at
the bars of its cage, break its wings, and languish.
No one ever profited by the experience of another, any more than
any person inherited the learning of an ancestor. Alas and alack, we
must acquire both for ourselves.
To our mothers and grandmothers, with their sweet but secluded
and often sequestered lives, it would have seemed a deed of daring
for a woman to lecture the public. Would they have thought it—
would our grandfathers rather have held it “ladylike”?
It is curious how one acquires a reputation without the least
foundation. For instance, I am always being asked to lecture;
sometimes it is at a People’s Palace, sometimes before a learned
society, or on behalf of various charities, or to address the blind, or
deliver educational discourses; and even the famous Major Pond of
America once tried to persuade me to go on a lecturing tour in the
States.
Tempting as his money offer was, I dared not face that vast
public.
This reputation is a chimera, for I have only lectured a few times
in my life; and these occasions have chiefly been at the People’s
Palace at Vauxhall, where an audience of two or three thousand
persons, paying from one penny to sixpence, eat oranges, smoke
pipes, and otherwise enjoy themselves after their manner, while the
lecturer is doing his (or her) best to amuse them. To keep these
people out of the public-houses and well occupied for an evening
seems worth even the pain and nervousness of standing alone on a
stage, nearly as big as that of Drury Lane, with footlights before,
and a huge white curtain for one’s slides behind.
The first time I ever spoke in public was at a large meeting (seven
or eight hundred) held in the St. Martin’s Town Hall, when at an hour
or two’s notice I took the place of the late Earl of Winchilsea, and, in
reply to his bidding by telegram, discoursed for fifteen minutes on
the position of women in Agriculture, a subject in which I was much
interested at the time. I spoke from notes only, having a horror of a
read paper, which is always exasperating or inaudible. Most
speeches are too low and too long. The fifteen minutes appeared to
be nothing, but the moments of waiting were torture until the first
words had come forth. When one’s knees shake, and one’s tongue
seems to cleave to the roof of the mouth, when the audience dances
like myriads of fireflies before one’s eyes, the misery is so awful that
the result is not worth the effort.
Women are often excellent speakers, both in matter and style, and
those who have an equal amount of practice are quite as good as
the best men. Nevertheless, after-dinner speaking is, alas, far more
often boring than entertaining, and one regrets a bell does not ring
after five minutes, as a gentle hint to sit down. The poor speaker
seldom knows when the right moment to end has arrived.
Everyone is shy about something. The rough-edged shyness of
youth wears away, but we each remain tender somewhere. Shyness
overpowers me when making a speech, or on hearing my name
roared into a room full of people. The first makes me sick, in spite of
having addressed an audience of three thousand people, which I
find easier than thirty; the second makes me wish to run away.
“I’m shy,” is the excuse of youth to cover rudeness. Gauche,
awkward, ill-mannered boys and girls call these delinquencies
shyness. Being shy, however, is no extenuation of being
discourteous. It is merely selfish self-conceit allowed to run rampant
instead of being checked. How much easier it is to form a bad
impression than to destroy one.
We are all imperfect, but the only chance of bettering ourselves is
to realise the fact early and try self-reform.
I have been fighting faults all my life, and although I have
overcome some of them—and I shan’t tell you what they are—a vast
crop still remain to be mowed down by the scythe of Time.
The question of women and the suffrage is now so important that
it is impossible for any thinking man or woman not to have an
opinion on the subject. What a curious thing it is that Liberals who
stand for Progress fear this onward movement. Is it because they
think women in the main are conservative?
On the 6th of February, 1907, at the time when the Women
Suffragists were being marched in scores to prison, and big
processions were being organised, and endless fusses and
excitements were in the air, Punch wrote an amusing article,
sweeping away the House of Lords, and substituting for it a House
of Ladies.
My name happened to be among the half-dozen elected
Peeresses, and a funny crew we were. Miss Christabel Pankhurst was
chosen because she was then considered the only good-looking
suffragette. Madame Zansig because of her thought-reading
propensities. Clara Butt because she could reduce chaos to harmony,
and so on.
Anyway, the article was commented on tremendously in the Press,
and was the subject of much amusement among my friends. It
brought me many quibs, telegrams, and telephones of
congratulation on my elevation to the Peerage.
The following letter is from a notable woman, written about two
years later:

“Edinburgh,
“November 26th, 1909.
“My dear Mrs. Alec Tweedie,
“I am very pleased to hear that you are disposed to take a
more active part than heretofore in demonstrating your support
of Women’s Suffrage. The London Society, of which Lady
Frances Balfour is the President, is non-party in character and is
opposed to stone-throwing, whip-lashing, and other methods of
violence. The London Society is one of more than a hundred
Societies, which together form the National Union of Women’s
Suffrage Societies of which I am President. I have asked Miss
Strachey, the Secretary of the London Society, to send you a
membership form, and if you approve of our methods and
policy, we shall be most grateful if you will join us. I am away
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and


personal growth!

textbookfull.com

You might also like