100% found this document useful (2 votes)
24 views

(Ebook) Advanced Data Analytics Using Python: With Architectural Patterns, Text and Image Classification, and Optimization Techniques [2nd Edition] by Sayan Mukhopadhyay, Pratip Samanta ISBN 9781484280041, 1484280040 - Experience the full ebook by downloading it now

The document provides information about various ebooks available for instant download at ebooknice.com, including titles on advanced data analytics using Python, machine learning, and SAT preparation. It features details such as authors, ISBN numbers, and links to access the ebooks in different formats. The document emphasizes the ease of access and the variety of subjects covered in the available ebooks.

Uploaded by

erpencanasi
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
100% found this document useful (2 votes)
24 views

(Ebook) Advanced Data Analytics Using Python: With Architectural Patterns, Text and Image Classification, and Optimization Techniques [2nd Edition] by Sayan Mukhopadhyay, Pratip Samanta ISBN 9781484280041, 1484280040 - Experience the full ebook by downloading it now

The document provides information about various ebooks available for instant download at ebooknice.com, including titles on advanced data analytics using Python, machine learning, and SAT preparation. It features details such as authors, ISBN numbers, and links to access the ebooks in different formats. The document emphasizes the ease of access and the variety of subjects covered in the available ebooks.

Uploaded by

erpencanasi
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 73

Instant Ebook Access, One Click Away – Begin at ebooknice.

com

(Ebook) Advanced Data Analytics Using Python: With


Architectural Patterns, Text and Image
Classification, and Optimization Techniques [2nd
Edition] by Sayan Mukhopadhyay, Pratip Samanta
ISBN 9781484280041, 1484280040
https://ebooknice.com/product/advanced-data-analytics-using-
python-with-architectural-patterns-text-and-image-
classification-and-optimization-techniques-2nd-
edition-47321334

OR CLICK BUTTON

DOWLOAD EBOOK

Get Instant Ebook Downloads – Browse at https://ebooknice.com


Instant digital products (PDF, ePub, MOBI) ready for you
Download now and discover formats that fit your needs...

Start reading on any device today!

(Ebook) Advanced Data Analytics Using Python : With Architectural Patterns, Text and
Image Classification, and Optimization Techniques by Sayan Mukhopadhyay; Pratip
Samanta ISBN 9781484280058, 1484280059

https://ebooknice.com/product/advanced-data-analytics-using-python-with-
architectural-patterns-text-and-image-classification-and-optimization-
techniques-49612318

ebooknice.com

(Ebook) Biota Grow 2C gather 2C cook by Loucas, Jason; Viles, James ISBN
9781459699816, 9781743365571, 9781925268492, 1459699815, 1743365578, 1925268497

https://ebooknice.com/product/biota-grow-2c-gather-2c-cook-6661374

ebooknice.com

(Ebook) Advanced Data Analytics Using Python: With Machine Learning, Deep Learning
and NLP Examples by Mukhopadhyay, Sayan ISBN 9781484234495, 1484234499

https://ebooknice.com/product/advanced-data-analytics-using-python-with-machine-
learning-deep-learning-and-nlp-examples-55670484

ebooknice.com

(Ebook) SAT II Success MATH 1C and 2C 2002 (Peterson's SAT II Success) by Peterson's
ISBN 9780768906677, 0768906679

https://ebooknice.com/product/sat-ii-success-math-1c-and-2c-2002-peterson-s-sat-
ii-success-1722018

ebooknice.com
(Ebook) Matematik 5000+ Kurs 2c Lärobok by Lena Alfredsson, Hans Heikne, Sanna
Bodemyr ISBN 9789127456600, 9127456609

https://ebooknice.com/product/matematik-5000-kurs-2c-larobok-23848312

ebooknice.com

(Ebook) Cambridge IGCSE and O Level History Workbook 2C - Depth Study: the United
States, 1919-41 2nd Edition by Benjamin Harrison ISBN 9781398375147, 9781398375048,
1398375144, 1398375047

https://ebooknice.com/product/cambridge-igcse-and-o-level-history-
workbook-2c-depth-study-the-united-states-1919-41-2nd-edition-53538044

ebooknice.com

(Ebook) Master SAT II Math 1c and 2c 4th ed (Arco Master the SAT Subject Test: Math
Levels 1 & 2) by Arco ISBN 9780768923049, 0768923042

https://ebooknice.com/product/master-sat-ii-math-1c-and-2c-4th-ed-arco-master-
the-sat-subject-test-math-levels-1-2-2326094

ebooknice.com

(Ebook) Advanced Analytics with Transact-SQL: Exploring Hidden Patterns and Rules in
Your Data by Dejan Sarka ISBN 9781484271728, 1484271726

https://ebooknice.com/product/advanced-analytics-with-transact-sql-exploring-
hidden-patterns-and-rules-in-your-data-33557194

ebooknice.com

(Ebook) Python Data Analytics: Data Analysis and Science Using Pandas, Matplotlib
and the Python Programming Language by Nelli Fabio ISBN 9781484209592, 1484209591

https://ebooknice.com/product/python-data-analytics-data-analysis-and-science-
using-pandas-matplotlib-and-the-python-programming-language-38169124

ebooknice.com
Advanced Data
Analytics Using
Python
With Architectural Patterns,
Text and Image Classification,
and Optimization Techniques
Second Edition

Sayan Mukhopadhyay
Pratip Samanta
Advanced Data Analytics Using Python: With Architectural Patterns, Text
and Image Classification, and Optimization Techniques

Sayan Mukhopadhyay Pratip Samanta


Kolkata, West Bengal, India Kolkota, West Bengal, India

ISBN-13 (pbk): 978-1-4842-8004-1 ISBN-13 (electronic): 978-1-4842-8005-8


https://doi.org/10.1007/978-1-4842-8005-8
Copyright © 2023 by Sayan Mukhopadhyay, Pratip Samanta
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or
part of the material is concerned, specifically the rights of translation, reprinting, reuse of
illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way,
and transmission or information storage and retrieval, electronic adaptation, computer software,
or by similar or dissimilar methodology now known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark
symbol with every occurrence of a trademarked name, logo, or image we use the names, logos,
and images only in an editorial fashion and to the benefit of the trademark owner, with no
intention of infringement of the trademark.
The use in this publication of trade names, trademarks, service marks, and similar terms, even if
they are not identified as such, is not to be taken as an expression of opinion as to whether or not
they are subject to proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of
publication, neither the authors nor the editors nor the publisher can accept any legal
responsibility for any errors or omissions that may be made. The publisher makes no warranty,
express or implied, with respect to the material contained herein.
Managing Director, Apress Media LLC: Welmoed Spahr
Acquisitions Editor: Celestin Suresh John
Development Editor: James Markham
Coordinating Editor: Mark Powers
Copyeditor: Kim Wimpsett
Cover designed by eStudioCalamar
Cover image by David Clode on Unsplash (www.unsplash.com)
Distributed to the book trade worldwide by Apress Media, LLC, 1 New York Plaza, New York, NY
10004, U.S.A. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail [email protected],
or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member
(owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance
Inc is a Delaware corporation.
For information on translations, please e-mail [email protected]; for
reprint, paperback, or audio rights, please e-mail [email protected].
Apress titles may be purchased in bulk for academic, corporate, or promotional use. eBook
versions and licenses are also available for most titles. For more information, reference our Print
and eBook Bulk Sales web page at www.apress.com/bulk-sales.
Any source code or other supplementary material referenced by the author in this book is
available to readers on GitHub (github.com/apress). For more detailed information, please visit
www.apress.com/source-­code.
Printed on acid-free paper
The reason for the success of this book is that it has original
research, so I dedicate it to the person from whom I learned
how to do research: Dr. Debnath Pal, IISc.
—Sayan Mukhopadhyay
Table of Contents
About the Authors��������������������������������������������������������������������������������xi

About the Technical Reviewer�����������������������������������������������������������xiii


Acknowledgments������������������������������������������������������������������������������xv
Introduction��������������������������������������������������������������������������������������xvii

Chapter 1: A Birds Eye View to AI System��������������������������������������������1


OOP in Python�������������������������������������������������������������������������������������������������������1
Calling Other Languages in Python���������������������������������������������������������������������13
Exposing the Python Model as a Microservice���������������������������������������������������14
High-Performance API and Concurrent Programming����������������������������������������17
Choosing the Right Database������������������������������������������������������������������������������21
Summary������������������������������������������������������������������������������������������������������������22

Chapter 2: ETL with Python����������������������������������������������������������������23


MySQL�����������������������������������������������������������������������������������������������������������������24
How to Install MySQLdb?�������������������������������������������������������������������������������24
Database Connection�������������������������������������������������������������������������������������25
INSERT Operation������������������������������������������������������������������������������������������26
READ Operation���������������������������������������������������������������������������������������������26
DELETE Operation������������������������������������������������������������������������������������������27
UPDATE Operation�����������������������������������������������������������������������������������������28
COMMIT Operation�����������������������������������������������������������������������������������������28
ROLL-BACK Operation�����������������������������������������������������������������������������������28

v
Table of Contents

Normal Forms�����������������������������������������������������������������������������������������������������31
First Normal Form�����������������������������������������������������������������������������������������31
Second Normal Form�������������������������������������������������������������������������������������32
Third Normal Form����������������������������������������������������������������������������������������33
Elasticsearch�������������������������������������������������������������������������������������������������������35
Connection Layer API�������������������������������������������������������������������������������������38
Neo4j Python Driver��������������������������������������������������������������������������������������������39
neo4j-rest-client�������������������������������������������������������������������������������������������������39
In-Memory Database������������������������������������������������������������������������������������������40
MongoDB (Python Edition)����������������������������������������������������������������������������������40
Import Data into the Collection����������������������������������������������������������������������41
Create a Connection Using pymongo�������������������������������������������������������������42
Access Database Objects������������������������������������������������������������������������������42
Insert Data�����������������������������������������������������������������������������������������������������43
Update Data���������������������������������������������������������������������������������������������������43
Remove Data�������������������������������������������������������������������������������������������������43
Cloud Databases�������������������������������������������������������������������������������������������������43
Pandas����������������������������������������������������������������������������������������������������������������44
ETL with Python (Unstructured Data)������������������������������������������������������������������45
Email Parsing�������������������������������������������������������������������������������������������������45
Topical Crawling��������������������������������������������������������������������������������������������48
Summary������������������������������������������������������������������������������������������������������������52

Chapter 3: Feature Engineering and Supervised Learning�����������������53


Dimensionality Reduction with Python���������������������������������������������������������������54
Correlation Analysis���������������������������������������������������������������������������������������55
Principal Component Analysis�����������������������������������������������������������������������57
Mutual Information����������������������������������������������������������������������������������������60
Classifications with Python���������������������������������������������������������������������������������64

vi
Table of Contents

Semi-Supervised Learning���������������������������������������������������������������������������������65
Decision Tree�������������������������������������������������������������������������������������������������������66
Which Attribute Comes First?������������������������������������������������������������������������66
Random Forest Classifier������������������������������������������������������������������������������68
Naïve Bayes Classifier�����������������������������������������������������������������������������������������68
Support Vector Machine��������������������������������������������������������������������������������������69
Nearest Neighbor Classifier��������������������������������������������������������������������������������71
Sentiment Analysis���������������������������������������������������������������������������������������������71
Image Recognition����������������������������������������������������������������������������������������������73
Regression with Python���������������������������������������������������������������������������������74
Least Square Estimation��������������������������������������������������������������������������������75
Logistic Regression���������������������������������������������������������������������������������������76
Classification and Regression�����������������������������������������������������������������������������76
Intentionally Bias the Model to Over-Fit or Under-Fit������������������������������������������77
Dealing with Categorical Data�����������������������������������������������������������������������������78
Summary������������������������������������������������������������������������������������������������������������79

Chapter 4: Unsupervised Learning: Clustering�����������������������������������81


K-Means Clustering��������������������������������������������������������������������������������������������82
Choosing K: The Elbow Method���������������������������������������������������������������������������86
Silhouette Analysis����������������������������������������������������������������������������������������������86
Distance or Similarity Measure���������������������������������������������������������������������������88
Properties������������������������������������������������������������������������������������������������������89
General and Euclidean Distance��������������������������������������������������������������������89
Squared Euclidean Distance��������������������������������������������������������������������������91
Distance Between String-Edit Distance��������������������������������������������������������91
Similarity in the Context of a Document�������������������������������������������������������������93
Types of Similarity�����������������������������������������������������������������������������������������94

vii
Table of Contents

Example of K-Means in Images��������������������������������������������������������������������������95


Preparing the Cluster�������������������������������������������������������������������������������������97
Thresholding�������������������������������������������������������������������������������������������������������99
Time to Cluster��������������������������������������������������������������������������������������������100
Revealing the Current Cluster����������������������������������������������������������������������102
Hierarchical Clustering��������������������������������������������������������������������������������������104
Bottom-Up Approach�����������������������������������������������������������������������������������104
Distance Between Clusters�������������������������������������������������������������������������105
Top-Down Approach������������������������������������������������������������������������������������107
Graph Theoretical Approach������������������������������������������������������������������������111
How Do You Know If the Clustering Result Is Good?�����������������������������������112
Summary����������������������������������������������������������������������������������������������������������113

Chapter 5: Deep Learning and Neural Networks�������������������������������115


Backpropagation�����������������������������������������������������������������������������������������������116
Backpropagation Approach�������������������������������������������������������������������������116
Other Algorithms�����������������������������������������������������������������������������������������������119
TensorFlow��������������������������������������������������������������������������������������������������������120
Network Architecture and Regularization Techniques���������������������������������124
Updatable Model and Transfer Learning������������������������������������������������������124
Recurrent Neural Network��������������������������������������������������������������������������������133
LSTM�����������������������������������������������������������������������������������������������������������134
Reinforcement Learning�����������������������������������������������������������������������������������138
TD0��������������������������������������������������������������������������������������������������������������139
TD�������������������������������������������������������������������������������������������������������������145
Example of Dialectic Learning���������������������������������������������������������������������146
Convolution Neural Networks���������������������������������������������������������������������������156
Summary����������������������������������������������������������������������������������������������������������159

viii
Table of Contents

Chapter 6: Time Series���������������������������������������������������������������������161


Classification of Variation����������������������������������������������������������������������������������161
Analyzing a Series Containing a Trend��������������������������������������������������������������161
Curve Fitting������������������������������������������������������������������������������������������������163
Removing Trends from a Time Series����������������������������������������������������������163
Analyzing a Series Containing Seasonality�������������������������������������������������������164
Removing Seasonality from a Time Series��������������������������������������������������������165
By Filtering��������������������������������������������������������������������������������������������������165
By Differencing��������������������������������������������������������������������������������������������166
Transformation��������������������������������������������������������������������������������������������������166
To Stabilize the Variance�����������������������������������������������������������������������������167
To Make the Seasonal Effect Additive���������������������������������������������������������167
To Make the Data Distribution Normal���������������������������������������������������������167
Stationary Time Series��������������������������������������������������������������������������������������168
Stationary Process��������������������������������������������������������������������������������������168
Autocorrelation and the Correlogram����������������������������������������������������������169
Estimating Autocovariance and Autocorrelation Functions�������������������������170
Time-Series Analysis with Python���������������������������������������������������������������������171
Useful Methods��������������������������������������������������������������������������������������������171
Autoregressive Processes���������������������������������������������������������������������������173
Estimating Parameters of an AR Process����������������������������������������������������175
Mixed ARMA Models�����������������������������������������������������������������������������������������177
Integrated ARMA Models�����������������������������������������������������������������������������������179
The Fourier Transform���������������������������������������������������������������������������������������181
An Exceptional Scenario�����������������������������������������������������������������������������������182
Missing Data�����������������������������������������������������������������������������������������������������183
Summary����������������������������������������������������������������������������������������������������������184

ix
Table of Contents

Chapter 7: Analytics at Scale�����������������������������������������������������������185


Hadoop��������������������������������������������������������������������������������������������������������������185
MapReduce Programming���������������������������������������������������������������������������185
Partitioning Function�����������������������������������������������������������������������������������186
Combiner Function��������������������������������������������������������������������������������������187
HDFS File System����������������������������������������������������������������������������������������197
MapReduce Design Pattern�������������������������������������������������������������������������197
A Notes on Functional Programming����������������������������������������������������������������203
Spark�����������������������������������������������������������������������������������������������������������������204
PySpark�������������������������������������������������������������������������������������������������������������207
Updatable Machine Learning and Spark Memory Model����������������������������������209
Analytics in the Cloud���������������������������������������������������������������������������������������221
Internet of Things����������������������������������������������������������������������������������������������231
Essential Architectural Patterns for Data Scientists������������������������������������231
Scenario 1: Hot Potato Anti-Pattern�������������������������������������������������������������232
Scenario 2: Proxy and Layering Patterns�����������������������������������������������������236
Thank You����������������������������������������������������������������������������������������������������������241

Index�������������������������������������������������������������������������������������������������243

x
About the Authors
Sayan Mukhopadhyay has more than 13 years
of industry experience and has been associated
with companies such as Credit Suisse, PayPal,
CA Technologies, CSC, and Mphasis. He has
a deep understanding of applications for
data analysis in domains such as investment
banking, online payments, online advertising,
IT infrastructure, and retail. His area of
expertise is in applying high-performance
computing in distributed and data-driven
environments such as real-time analysis, high-
frequency trading, and so on.
He earned his engineering degree in electronics and instrumentation
from Jadavpur University and his master’s degree in research in
computational and data science from IISc in Bangalore.

Pratip Samanta is a principal AI engineer/


researcher with more than 11 years of
experience. He has worked for several software
companies and research institutions. He has
published conference papers and has been
granted patents in AI and natural language
processing. He is also passionate about
gardening and teaching.

xi
About the Technical Reviewer
Joos Korstanje is a data scientist with more
than five years of industry experience in
developing machine learning tools, of which a
large part is forecasting models. He currently
works at Disneyland Paris where he develops
machine learning for a variety of tools.

xiii
Acknowledgments
Thanks to Labonic Chakraborty (Ripa) and Soumili Chakraborty.

xv
Introduction
We are living in the data science/artificial intelligence era. To thrive in
this environment, where data drives decision-making in everything from
business to government to sports and entertainment, you need the skills
to manage and analyze huge amounts of data. Together we can use this
data to make the world better for everyone. In fact, humans have yet to find
everything we can do using this data. So, let us explore!
Our objective for this book is to empower you to become a leader
in this data-transformed era. With this book you will learn the skills to
develop AI applications and make a difference in the world.
This book is intended for advanced user, because we have incorporated
some advanced analytics topics. Important machine learning models and
deep learning models are explained with coding exercises and real-world
examples.
All the source code used in this book is available for download at
https://github.com/apress/advanced-­data-­analytics-­python-­2e.
Happy reading!

xvii
CHAPTER 1

A Birds Eye View to


AI System
In this book, we assume that you are familiar with Python programming.
In this introductory chapter, we explain why a data scientist should choose
Python as a programming language. Then we highlight some situations
where Python may not be the ideal choice. Finally, we describe some best
practices for application development and give some coding examples that
a data scientist may need in their day-to-day job.

OOP in Python
In this section, we explain some features of object-oriented programming
(OOP) in a Python context.
The most basic element of any modern application is an object. To
a programmer or architect, the world is a collection of objects. Objects
consist of two types of members: attributes and methods. Members can be
private, public, or protected. Classes are data types of objects. Every object
is an instance of a class. A class can be inherited in child classes. Two
classes can be associated using composition.
Python has no keywords for public, private, or protected, so
encapsulation (hiding a member from the outside world) is not implicit in
Python. Like C++, it supports multilevel and multiple inheritance. Like Java,
it has an abstract keyword. Classes and methods both can be abstract.
© Sayan Mukhopadhyay, Pratip Samanta 2023 1
S. Mukhopadhyay and P. Samanta, Advanced Data Analytics Using Python,
https://doi.org/10.1007/978-1-4842-8005-8_1
Chapter 1 A Birds Eye View to AI System

In the following code, we are describing an object-oriented question-­


answering system without any machine learning. The program’s input is a
set of dialogs in input.txt, as shown here:

glob is I
prok is V
pish is X
tegj is L
glob glob Silver is 34 Credits
glob prok Gold is 57800 Credits
pish pish Iron is 3910 Credits
how much is pish tegj glob glob ?
how many Credits is glob prok Silver ?
how many Credits is glob prok Gold ?
how many Credits is glob prok Iron ?
how much wood could a woodchuck chuck if a woodchuck could
chuck wood?
Program has a knowledge base in config.txt.
I,1,roman
V,5,roman
X,10,roman
L,50,roman
C,100,roman
D,500,roman
M,1000,roman

Based on this input and the configuration program, the answer to the
question is given in input.txt in standard output, as shown here:

pish tegj glob glob is 42


glob prok Silver is 68 Credits
glob prok Gold is 57800 Credits
glob prok Iron is 782 Credits
I have no idea what you are talking about

2
Chapter 1 A Birds Eye View to AI System

The parsing logic is in the Observer class.

import operator

#this class verify the validity of input


class Observer(object):
      #store frequecy of symbols
      length = {}
      #most frequent symbol
      symbol = ''
      #count of most frequent symbol
      count = 0
      #calling class
      compiler = None

      def __init__(self,cmpiler):
            self.compiler = cmpiler

      def initialize(self, arr):


            for i in range(len(arr)):
                  self.length[arr[i]] = 0

      #increase count for each occurence of symbol


      def increment(self,symbol):
            self.length[symbol] = self.length[symbol] + 1

      #claculate most frequent symbol and it's count


      def calculate(self):
            self.symbol,self.count = max(self.length.items(),
key=operator.itemgetter(1))

      #verify if wrong symbol is subtracted ie ( V, ..


      def verifySubstract(self, current):
            while current % 10 != 0:
                  current = current / 10

3
Chapter 1 A Birds Eye View to AI System

            if current == 5:
                  raise Exception("Wrong Substraction")

      def evaluate(self):
            #check mximum repeatation is crossing the limit
            if self.count > 3:
                  raise Exception("Repeat more than 3")
            #symbol is proper or not
            if self.symbol not in self.compiler.symbol_map:
                  raise Exception("Wrong Symbol")
            #check if wrong symbol is repeated ie (V, ..
            self.symbol,unit = self.compiler.
evaluateSymbol(self.symbol)
            while self.symbol % 10 != 0:
                  self.symbol = self.symbol / 10
            if self.count > 1 and  self.symbol == 5:
                  raise Exception("Wrong Symbol repeated")

      #checking if input sentence is proper or not


      def evaluateSentence(self, line):
            if "is" not in line:
                  return "I have no idea what you are
talking about"

The compilation logic is in the compiler class, as shown here:

import sys

from observer import Observer

class compilerTrader(object):
    #store mapping of symbols with score and unit
    symbol_map = {}
    #store the list of valid symbol
    valid_values = []

4
Chapter 1 A Birds Eye View to AI System

    #read the config and initialize the class member


    def __init__(self, config_path):
        with open(config_path) as f:
           for line in f:
               if ',' in line :
                   symbol, value, type = line.strip().
split(',')
                   self.symbol_map[symbol] = float(value), type
                   self.valid_values.append(float(value))
           f.close()

    #evaluate the ultimate numerical score with unit for


a symbol
    def evaluateSymbol(self, symbol):
        while symbol not in self.valid_values:
              symbol,unit = self.symbol_map[symbol]
        return float(symbol), unit

    #compiling the info in line


    def compile_super(self, line):
        obs = Observer(self)
        if 'is' in line:
            fields = line.split(' is ')
            value = fields[-1]
            var = fields[0]
            #if one symbol and one value
            if ' ' not in var:
                if value in self.symbol_map:
                   self.symbol_map[var] = int(self.symbol_
map[value][0]) ,'roman'

5
Chapter 1 A Birds Eye View to AI System

                else:
                    #logic for value with unit
                    if ' ' in value:
                        fields = value.split(' ')
                        user_unit = fields[-1]
                        if ' ' not in var:
                            self.symbol_map[var] =
int(fields[0]), user_unit
                        else:
                            #logic for multiple symbols
in input
                            total = int(fields[0])
                            factor = 0
                            arr = var.split(' ')
                            obs.initialize(arr)
                            for i in range(len(arr)):
                                obs.increment(arr[i])
                                if arr[i] in self.symbol_map
and arr[i+1] in self.symbol_map
and i < len(arr) -1:
                                    current, current_
unit = self.
evaluateSymbol(
[arr[i]][0])
                                    next, next_unit = self.
evaluateSymbol(
[arr[i+1]][0])
                                    if current >= next:
                                       factor =  factor
+ current

6
Chapter 1 A Birds Eye View to AI System

                                    else:
                                        obs.verifySubstract(
current)
                                        factor = factor -
current
                                    else:
                                        if arr[i] in self.
symbol_map:
                                        current, current_
unit = self.
evaluateSymbol(
[arr[i]][0])
                                        factor = factor +
current
                                            else:
                                                self.
symbol_map[
arr[i]] =
total/factor,
user_unit
                                                self.valid_
values.append
(total/factor)
                               obs.calculate()
                               obs.evaluate()

The answering logic is in the answer layer, which calls Observer and
compiler. The answering class inherits the compiler class.

import sys

from observer import Observer


from compiler import compilerTrader

7
Chapter 1 A Birds Eye View to AI System

class answeringTrader(compilerTrader):

      def __init__(self, config_path):


            super().__init__(config_path)

      #compiling info in line


      def compile(self, line):
            super().compile_super(line)

      #answering query in line


      def answer(self, line):
            obs = Observer(super())
            if 'is' in line:
                  values = line.split(' is ')[-1]
                  ans = 0
                  arr = values.split(' ')
                  unit = ''
                  obs.initialize(arr)
                  for i in range(len(arr)):
                        if arr[i] in "?.,!;":
                              continue
                        obs.increment(arr[i])
                        if i < len(arr)-2:
                              if arr[i] in super().symbol_
map and arr[i+1] in super().
symbol_map:
                                    current, current_
unit = super().
evaluateSymbol([arr[i]][0])
                                    next, next_unit = super().
evaluateSymbol(
[arr[i+1]][0])

8
Chapter 1 A Birds Eye View to AI System

                                    if current >= next:


                                          ans = ans + current
                                    else:
                                          if next_unit ==
'roman':
                                                obs.verify
Substract(
current)
                                                ans = ans -
current
                                          else:
                                                ans = ans
+ current
                        else:
                              if arr[i] in super().symbol_map:
                                    current,unit = super().
evaluateSymbol([arr[i]][0])
                                    if  unit != 'roman':
                                          ans = ans * current
                                    else:
                                          ans = ans + current

                  obs.calculate()
                  obs.evaluate()
                  values = values.replace("?" , "is ")
                  if unit == 'roman':
                        unit = ''
                  return(values + str(ans) + ' ' + unit)

Finally, the main program calls the answering class and the observer,
and then it performs the task and does unit testing on the logic.

9
Chapter 1 A Birds Eye View to AI System

import sys
import unittest

sys.path.append('./answerLayer')
sys.path.append('./compilerLayer')
sys.path.append('./utilityLayer')

from answer import answeringTrader


from observer import Observer

#client interface for the framework


class ClientTrader(object):
      trader = None
      def __init__(self, config_path):
            self.trader = answeringTrader(config_path)

      #processing an input string


      def process(self, input_string):
            obs = Observer(self.trader)
            valid = obs.evaluateSentence(input_string)
            if valid is not None:
                  return valid
            if input_string.strip()[-1] == '?' :
                  return self.trader.answer(input_string)
            else:
                  return self.trader.compile(input_string)

#unit test cases


class TestTrader(unittest.TestCase):
      trader = None

      def setUp(self):
            pass

10
Chapter 1 A Birds Eye View to AI System

      #test case for non-roman symbol      unit other


than roman
      def test_answer_unit(self):
            ans = self.trader.process("how many Credits is glob
prok Silver ?")
            self.assertEqual(ans.strip(), "glob prok Silver is
68.0 Credits")

      #test case with only roman symbol in unit case


      def test_answer_roman(self):
            ans = self.trader.process("how much is pish tegj
glob glob ?")
            self.assertEqual(ans.strip(), "pish tegj glob glob
is 42.0")

      #test case if repeatation of symbol is exceed max


limit (3)
      def test_exception_over_repeat(self):
            with self.assertRaises(Exception) as context:
                  ans = self.trader.process("how much is pish
tegj glob glob glob glob ?")
                  self.assertTrue("Repeat more than 3" in
context.exception)

      #test case if wrong symbol repeated ie (V ..


      def test_exception_unproper_repeat(self):
            with self.assertRaises(Exception) as context:
                  ans = self.trader.process("how much is pish
tegj D D ?")
                  self.assertTrue("Repeat more than 3" in
context.exception)

11
Chapter 1 A Birds Eye View to AI System

      #test case if wrong symbol substracted ie (V ...


      def test_wrong_substraction(self):
            with self.assertRaises(Exception) as context:
                  ans = self.trader.process("how much
is V X ?")
                  self.assertTrue("Wrong Substraction" in
context.exception)

      #test case if query is not properly formatted


      def test_wrong_format_query(self):
            ans = self.trader.process("how much wood could a
woodchuck chuck if a woodchuck could chuck wood ?")
            self.assertEqual(ans.strip(), "I have no idea what
you are talking about")

if __name__ == '__main__':
      if len(sys.argv) != 3:
            print("Usage is : " + sys.argv[0] + " <intput file
path>  <config file path>")
            exit(0)
      tr = ClientTrader(sys.argv[2])
      f = open(sys.argv[1])
      for line in f:
            response = tr.process(line.strip())
            if response is not None:
                  print(response)

      TestTrader.trader = tr
      unittest.main(argv = [sys.argv[0]], exit = False)

You can run this program with the following command:

python client.py input.txt config.txt

12
Chapter 1 A Birds Eye View to AI System

Calling Other Languages in Python


Now we will describe how to use other languages in Python. There are two
examples here. The first is calling R code from Python. R code is required
for some use cases. For example, if you want a ready-made function for the
Holt-Winters method in a time series, it is difficult to perform in Python,
but it is available in R. So, you can call R code from Python using the rpy2
module, as shown here:

import rpy2.robjects as ro
ro.r('data(input)')
ro.r('x <-HoltWinters(input data frame)')

(You can use example data given in time series chapter.)


Sometimes you need to call Java code from Python. For example, say
you are working on a name-entity recognition problem in the field of
natural language processing (NLP); some text is given as input, and you
have to recognize the names in the text. Python’s NLTK package does have
a name-entity recognition function, but its accuracy is not good. Stanford
NLP is a better choice here, but it is written in Java. You can solve this
problem in two ways.

• You can call Java at the command line using Python


code. You need to install Java with the yum/at-get
install java command before calling it.

• For Windows, it is recommended that you install


the JRE from https://adoptium.net/temurin/
releases/?version=8. You can also install the JRE
from another distribution. The installation will
automatically create JAVA_HOME. If it does not, you
need to set JAVA_HOME as the system variable, and the
value should be the location of Java installation folder,
for example, JAVA_HOME=C:\Program Files\Eclipse
Adoptium\jdk-8.0.345.1-hotspot\.
13
Chapter 1 A Birds Eye View to AI System

import subprocess
subprocess.call(['java','-cp','*','edu.stanford.nlp.sentiment.
SentimentPipeline','-file','foo.txt'])

Please place foo.txt in the same folder where you run the
Python code.

• You can expose Stanford NLP as a web service and call


it as a service. (Before running this code, you’ll need to
download the Stanford nlp JAR file available with the
book’s source code.)

nlp = StanfordCoreNLP('http://127.0.0.1:9000')
output = nlp.annotate(sentence, properties={
"annotators": "tokenize,ssplit,parse,sentiment",
"outputFormat": "json",
# Only split the sentence at End Of Line. We assume that this
method only takes in one single sentence.
"ssplit.eolonly": "true",
# Setting enforceRequirements to skip some annotators and make
the process faster
"enforceRequirements": "false"
})

You will see a more detailed example of Stanford NLP in Chapter 2.

E xposing the Python Model


as a Microservice
You can expose the Python model as a microservice in the same way that
your Python model can be used by others to write their own code. The best
way to do this is to expose your model as a web service. As an example, the
following code exposes a deep learning model using Flask:

14
Chapter 1 A Birds Eye View to AI System

from flask import Flask, request, g


from flask_cors import CORS
import tensorflow as tf
from sqlalchemy import *
from sqlalchemy.orm import sessionmaker
import pygeoip
from pymongo import MongoClient
import json
import datetime as dt
import ipaddress
import math
app = Flask(__name__)
CORS(app)
@app.before_request
def before():
       db = create_engine('sqlite:///score.db')
       metadata = MetaData(db)
       g.scores = Table('scores', metadata, autoload=True)
       Session = sessionmaker(bind=db)
       g.session = Session()
       client = MongoClient()
       g.db = client.frequency
       g.gi = pygeoip.GeoIP('GeoIP.dat')
       sess = tf.Session()
       new_saver = tf.train.import_meta_graph('model.obj.meta')
       new_saver.restore(sess, tf.train.latest_
checkpoint('./'))
       all_vars = tf.get_collection('vars')
       g.dropped_features = str(sess.run(all_vars[0]))
       g.b = sess.run(all_vars[1])[0]
       return

15
Chapter 1 A Birds Eye View to AI System

def get_hour(timestamp):
       return dt.datetime.utcfromtimestamp(timestamp /
1e3).hour
def get_value(session, scores, feature_name, feature_value):
       s = scores.select((scores.c.feature_name == feature_
name) & (scores.c.feature_value == feature_value))
       rs = s.execute()
       row = rs.fetchone()
       if row is not None:
             return float(row['score'])
       else:
             return 0.0
@app.route('/predict', methods=['POST'])
def predict():
       input_json = request.get_json(force=True)
       features = ['size','domain','client_time','device','ad_
position','client_size', 'ip','root']
       predicted = 0
       feature_value = ''
       for f in features:
             if f not in g.dropped_features:
                    if f == 'ip':
                          feature_value = str(ipaddress.
IPv4Address(ipaddress.ip_
address(unicode(request.remote_
addr))))
                    else:
                          feature_value = input_json.get(f)
                    if f == 'ip':
                          if 'geo' not in g.dropped_features:

16
Chapter 1 A Birds Eye View to AI System

                                 geo = g.gi.country_name_by_
addr(feature_value)
                                 predicted = predicted + get_
value(g.session, g.scores,
'geo', geo)

       return str(math.exp(predicted + g.b)-1)


app.run(debug = True, host ='0.0.0.0')

This code exposes a deep learning model as a Flask web service.


A JavaScript client will send the request with web user parameters such
as the IP address, ad size, ad position, and so on, and it will return the
price of the ad as a response. The features are categorical. You will learn
how to convert them into numerical scores in Chapter 3. These scores
are stored in an in-memory database. The service fetches the score from
the database, sums the result, and replies to the client. This score will be
updated real time in each iteration of training of a deep learning model. It
is using MongoDB to store the frequency of that IP address in that site. It is
an important parameter because a user coming to a site for the first time
is really searching for something, which is not true for a user where the
frequency is greater than 5. The number of IP addresses is huge, so they
are stored in a distributed MongoDB database.

 igh-Performance API and


H
Concurrent Programming
Flask is a good choice when you are building a general solution that is
also a graphical user interface (GUI). But if high performance is the most
critical requirement of your application, then Falcon is the best choice.
The following code is an example of the same model shown previously
exposed by the Falcon framework. Another improvement we made in this
code is that we implemented multithreading, so the code will be executed

17
Chapter 1 A Birds Eye View to AI System

in parallel. In addition to the Falcon-specific changes, you should note


the major changes in parallelizing the calling get_score function using a
thread pool class.

import falcon
from falcon_cors import CORS
import json
from sqlalchemy import *
from sqlalchemy.orm import sessionmaker
import pygeoip
from pymongo import MongoClient
import json
import datetime as dt
import ipaddress
import math
from concurrent.futures import *
from sqlalchemy.engine import Engine
from sqlalchemy import event
import sqlite3
@event.listens_for(Engine, "connect")
def set_sqlite_pragma(dbapi_connection, connection_record):
cursor = dbapi_connection.cursor()
cursor.execute("PRAGMA cache_size=100000")
cursor.close()
class Predictor(object):
       def __init__(self,domain):
             db1 = create_engine('sqlite:///score_' + domain +
'0test.db')
             metadata1 = MetaData(db1)
             self.scores = Table('scores', metadata1,
autoload=True)
             client = MongoClient(connect=False,maxPoolSize=1)

18
Chapter 1 A Birds Eye View to AI System

             self.db = client.frequency
             self.gi = pygeoip.GeoIP('GeoIP.dat')
             self.high = 1.2
             self.low = .8
       def get_hour(self,timestamp):
             return dt.datetime.utcfromtimestamp(timestamp /
1e3).hour
       def get_score(self, featurename, featurevalue):
             pred = 0
             s = self.scores.select((self.scores.c.feature_name
== featurename) & (self.scores.c.feature_value ==
featurevalue))
            rs = s.execute()
            row = rs.fetchone()
            if row is not None:
                      pred = pred + float(row['score'])
            res = self.db.frequency.find_one({"ip" : ip})
            freq = 1
            if res is not None:
                    freq = res['frequency']
                    pred2, prob2 = self.get_score('frequency',
str(freq))
                    return (pred1 + pred2), (prob1 + prob2)

            conn = sqlite3.connect('multiplier.db')
                   cursor = conn.execute("SELECT high,low from
multiplier where domain='" + value + "'")
                   row = cursor.fetchone()
                   if row is not None:
                          self.high = row[0]
                          self.low = row[1]
             return self.get_score(f, value)

19
Chapter 1 A Birds Eye View to AI System

       def on_post(self, req, resp):


             input_json = json.loads(req.stream.
read(),encoding='utf-8')
             input_json['ip'] = unicode(req.remote_addr)
             pred = 1
             prob = 1
             with ThreadPoolExecutor(max_workers=8) as pool:
                    future_array = { pool.submit(self.
get_value,f,input_json[f]) : f for f in
input_json}
                    for future in as_completed(future_array):
                          pred1, prob1 = future.result()
                          pred = pred + pred1
                          prob = prob - prob1
             resp.status = falcon.HTTP_200
       res = math.exp(pred)-1
             if res < 0:
                   res = 0
             prob = math.exp(prob)
             if(prob <= .1):
                   prob = .1
             if(prob >= .9):
                   prob = .9
             multiplier = self.low + (self.high -self.low)*prob
             pred = multiplier*pred
             resp.body = str(pred)
cors = CORS(allow_all_origins=True,allow_all_
methods=True,allow_all_headers=True)
wsgi_app = api = falcon.API(middleware=[cors.middleware])
f = open('publishers1.list')

20
Chapter 1 A Birds Eye View to AI System

for domain in f:
       domain = domain.strip()
       p = Predictor(domain)
       url = '/predict/' + domain
       api.add_route(url, p)

Having covered design patterns in Python a bit, let’s now take a look at
some essential architecture patterns for data scientists.

Choosing the Right Database


Before we go, we’ll leave a note for manager on which database is best for
which case.

• A relational database (MySQL, Oracle, SQL Server) is


the preferable choice when data is highly structured
and entities have a clear and strict connection. Mongo,
on the other hand, is a better choice when data is
unstructured and unorganized.

• Elastic Search or Solr is a better choice when data


contains a lengthy textual field and you’re executing
lots of searches in a substring of the text field. With
Elastic Search, you get a free data visualization tool
called Kibana as well as an ETL tool called Logstash,
and full-stack data analytics solutions are fashionable.

• Data must sometimes be represented as a graph. In


that situation, a graph database is required. Neo4j is a
popular graph database that comes with many utility
tools at a low price.

21
Chapter 1 A Birds Eye View to AI System

• We occasionally require a quick application. In that


situation, an in-memory database like SQLite can be
used. However, SQLite does not support updating your
database from a remote host.

You’ll learn more about databases in Chapter 2.

Summary
In this chapter, we discussed fundamental engineering principles for data
scientists, which are covered in separate chapters. The question-answering
example can help you understand how to organize your code. The basic
rule is to not put everything into one class. Divide your code into many
categories and use parent-child relationships where they exist. Then you
learned how to use Python to call other languages’ code. We provided two
instances of R and Java code calls. Then we showed you how to expose
your model as a REST API and make it perform well by using concurrent
programming. Following that, we covered significant architectural patterns
from data scientists.

22
CHAPTER 2

ETL with Python


Every data science professional has to extract, transform, and load (ETL)
data from different data sources. In this chapter, we will discuss how
to perform ETL with Python for a selection of popular databases. For a
relational database, we’ll cover MySQL. As an example of a document
database, we will cover Elasticsearch. For a graph database, we’ll cover
Neo4j, and for NoSQL, we’ll cover MongoDB. We will also discuss the
Pandas framework, which was inspired by R’s data frame concept.
ETL is based on a process in which data is extracted from multiple
sources, transformed into specific formats that involve cleaning
enrichment, and finally loaded into its target destination. The following are
the details of each process:

1. Extract: During data extraction, source data is pulled


from a variety of sources and moved to a staging
area, making the data available to subsequent stages
in the ETL process. After that, the data undergoes
the cleaning and enrichment stage, also known as
data cleansing.

2. Transform: In this stage, the source data is matched


to the format of the target system. This includes
steps such as changing data types, combining fields,
splitting fields, etc.

© Sayan Mukhopadhyay, Pratip Samanta 2023 23


S. Mukhopadhyay and P. Samanta, Advanced Data Analytics Using Python,
https://doi.org/10.1007/978-1-4842-8005-8_2
Chapter 2 ETL with Python

3. Load: This stage is the final ETL stage. Here, data


is loaded into the data warehouse in an automated
manner and can be periodically updated. Once
completed, the data is ready for data analysis.

The previous processes are important in any data analytics work. Once
the data goes through the ETL processes, then it becomes possible to
analysis the data, find insights, and so on.
We will discuss various types of ETL throughout this chapter. We
discussed in Chapter 1 that data is not an isolated thing. We need to load
data from somewhere, which is a database. We need to fetch the data from
some application, which is extraction. In this chapter and the next, we will
discuss various feature engineering that transforms the data from one form
to another.

MySQL
MySQLdb is an API in Python developed to work on top of the MySQL C
interface.

How to Install MySQLdb?


First you need to install the Python MySQLdb module on your machine.
Then run the following script:

#!/usr/bin/python
import MySQLdb

If you get an import error exception, that means the module was not
installed properly.
The following are the instructions to install the MySQL Python module:

$ gunzip MySQL-python-1.2.2.tar.gz

24
Chapter 2 ETL with Python

$ tar -xvf MySQL-python-1.2.2.tar


$ cd MySQL-python-1.2.2
$ python setup.py build
$ python setup.py install

You can download the tar.gz file from https://dev.mysql.com/


downloads/connector/python/. You need to download it to your
working folder.
For Windows, please select the MySQL installer file from https://
dev.mysql.com/downloads/installer/. Once it’s downloaded, double-­
click the file to install it and select MySQL Connector/Python as one of the
products to install. For details, you can visit https://dev.mysql.com/doc/
connector-­python/en/connector-­python-­installation-­binary.html.

Database Connection
Before connecting to a MySQL database, make sure you do the following:

1. You need to access a database called TEST with the


sql "use test" command.

2. In TEST you need a table named STUDENT; use the


command sql "create table student(name
varchar(20), sur_name varchar(20),roll_
no int");.

3. STUDENT needs three fields: NAME, SUR_NAME, and


ROLL_NO.

4. There needs to be a user in TEST that has complete


access to the database.

If you do not do these steps properly, you will get an exception in the
next Python code.

25
Chapter 2 ETL with Python

INSERT Operation
The following code carries out the SQL INSERT statement for the purpose
of creating a record in the STUDENT table:

#!/usr/bin/python
import MySQLdb
# Open database connection
db = MySQLdb.connect("localhost","user","passwd","TEST" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
# Prepare SQL query to INSERT a record into the database.
sql = """INSERT INTO STUDENT(NAME,
         SUR_NAME, ROLL_NO)
         VALUES ('Sayan', 'Mukhopadhyay', 1)"""
try:
   # Execute the SQL command
   cursor.execute(sql)
   # Commit your changes in the database
   db.commit()
except:
   # Rollback in case there is any error
   db.rollback()
# disconnect from server
db.close()

READ Operation
The following code fetches data from the STUDENT table and prints it:

#!/usr/bin/python
import MySQLdb
# Prepare SQL query to INSERT a record into the database.

26
Chapter 2 ETL with Python

sql = "SELECT * FROM STUDENT "


try:
   # Execute the SQL command
   cursor.execute(sql)
   # Fetch all the rows in a list of lists.
   results = cursor.fetchall()
   for row in results:
       fname = row[0]
       lname = row[1]
       id = row[2]
      # Now print fetched result
Print( "name=%s,surname=%s,id=%d" % \
             (fname, lname, id ))
except:
     print "Error: unable to fecth data"

DELETE Operation
The following code deletes a row from TEST with id=1:

#!/usr/bin/python
import MySQLdb
# Prepare SQL query to DELETE required records
sql = "DELETE FROM STUDENT WHERE ROLL_NO =1"
try:
   # Execute the SQL command
   cursor.execute(sql)
   # Commit your changes in the database
   db.commit()
except:
   # Rollback in case there is any error
   db.rollback()

27
Chapter 2 ETL with Python

UPDATE Operation
The following code changes the lastname variable to Mukherjee, from
Mukhopadhyay:

#!/usr/bin/python
import MySQLdb
# Prepare SQL query to UPDATE required records
sql = "UPDATE STUDENT SET SUR_NAME="Mukherjee"
                          WHERE SUR_NAME="Mukhopadhyay"
try:
   # Execute the SQL command
   cursor.execute(sql)
   # Commit your changes in the database
   db.commit()
except:
   # Rollback in case there is any error
   db.rollback()

COMMIT Operation
The commit operation provides its assent to the database to finalize the
modifications, and after this operation, there is no way that this can be
reverted.

ROLL-BACK Operation
If you are not completely convinced about any of the modifications and
you want to reverse them, then you can apply the roll-back() method.
The following is a complete example of accessing MySQL data through
Python. It will give the complete description of the data stored in a CSV file
or MySQL database.

28
Chapter 2 ETL with Python

This code asks for the data source type, either MySQL or text. For
example, if MySQL asks for the IP address, credentials, and database name
and shows all tables in the database, it offers its fields once the table is
selected. Similarly, a text file asks for a path, and in the files it points to, all
the columns are shown to the user.

# importing files and reading config file

import MySQLdb
import sys
out = open('Config1.txt','w')
print ("Enter the Data Source Type:")
print( "1. MySql")
print ("2. Exit")
while(1):
       data1 = sys.stdin.readline().strip()
       if(int(data1) == 1):
             out.write("source begin"+"\n"+"type=mysql\n")

# taking inputs from user

             print ("Enter the ip:")


             ip = sys.stdin.readline().strip()
             out.write("host=" + ip + "\n")
             print ("Enter the database name:")
             db = sys.stdin.readline().strip()
             out.write("database=" + db + "\n")
             print ("Enter the user name:")
             usr = sys.stdin.readline().strip()
             out.write("user=" + usr + "\n")
             print ("Enter the password:")
             passwd = sys.stdin.readline().strip()
             out.write("password=" + passwd + "\n")

29
Chapter 2 ETL with Python

# making connection to and executing query

             connection = MySQLdb.connect(ip, usr, passwd, db)


             cursor = connection.cursor()
             query = ("show tables")
             cursor.execute(query)
             data = cursor.fetchall()
             tables = []

# appending data to the table

             for row in data:


                    for field in row:
                           tables.append(field.strip())
             for i in range(len(tables)):
                    print( i, tables[i])
             tb = tables[int(sys.stdin.readline().strip())]
             out.write("table=" + tb + "\n")
             query = ("describe " + tb)
             cursor.execute(query)
             data = cursor.fetchall()
             columns = []
             for row in data:
                    columns.append(row[0].strip())
             for i in range(len(columns)):
                    print( columns[i])
             print "Not index choose the exact column names
seperated by coma"
             cols = sys.stdin.readline().strip()
             out.write("columns=" + cols + "\n")
             cursor.close()
             connection.close()
             out.write("source end"+"\n")

30
Chapter 2 ETL with Python

             print ("Enter the Data Source Type:")


             print ("1. MySql")             print ("2. Exit")
             out.close()
             sys.exit()

Before we go on to the topic of relational databases, let’s talk about


database normalization.

Normal Forms
Database normal forms are the principles to organize your data in an
optimum way.
Every table in a database can be in one of the normal forms that we’ll
go over next. For the primary key (PK) and foreign key (FK), you want to
have as little repetition as possible. The rest of the information should be
taken from other tables.

• First normal form (1NF)

• Second normal form (2NF)

• Third normal form (3NF)

First Normal Form


There are no repeating groups in the first normal form since only single
values are allowed at the intersection of each row and column.
To get to first normal form, remove the repetitive groups, and establish
two new relations to normalize a connection with a repeating group.
For unique identification, the new connection’s PK is a combination of
the old relation’s PK and a feature from the newly formed relation.

31
Random documents with unrelated
content Scribd suggests to you:
towards our lines. When, however, the advanced guard had reached
the limit of their advance, the Battalion pushed along the bed of the
stony wadi, and were supposed to halt at the junction of the wadi in
front of Jebel Ektief and the main wadi, where they hoped to turn
the corner into their place of deployment. However, the noise had
apparently disturbed the Turk, and he constantly poured machine-
gun fire on to this junction corner. It was therefore found necessary
to find another way into the Jebel Ektief Wadi, and the only way was
to climb the steep, precipice-like sides of the main wadi on to the
high ground held by “D” Company. This was accomplished, but how,
no one can tell; when the Battalion with its Lewis guns, regimental
aid post, and signalling mules reached the crest, dawn was
appearing in the east. It was decided, therefore, to make a hurried
descent into the Jebel Ektief Wadi in spite of the casual fire from the
enemy’s snipers and machine guns. Luckily the going down into the
second wadi was easier and more speedily accomplished than the
climb from the first wadi, and just as daylight arrived the tail of the
Battalion disappeared into the bed of Jebel Ektief Wadi, out of sight
of the Turks. Once in the bottom of the wadi the companies sorted
themselves out and prepared for the actual assault, “A” and “B”
being the attacking companies, “C” in support, and “D” in reserve. It
had been arranged that the 2/13th (Kensingtons) were to assault
the position on our left, having approached the wadi at Jebel Ektief
by a night march on a parallel route to the one we had taken.
Unfortunately, the ground over which the Kensingtons had to pass
had proved too difficult, and when daylight came they found
themselves perched on a precipice unable to cross the deep ravine
before them.

At seven o’clock on the morning of the 20th February, the artillery


opened up a heavy bombardment on the hill. The advance up the
hillside described officially as “trickling forward” should have taken
place under this barrage, while the assault was timed for eight
o’clock when the barrage was to lift.
Unfortunately little or no progress could be made owing to the fact
that the heavies were shelling the “bench mark” some 1,000 yards
behind the enemy’s forward lines thereby leaving the enemy’s
machine guns undisturbed, while at the same time other machine
guns on our left raked us, those, in fact, which should have been
concentrating their attention on our brother battalion, the
Kensingtons, who had been so unfortunate in their advance march.

At eight o’clock, when we should have assaulted, the attack was


therefore hung up for a while. An hour later a Forward Observation
Officer having come up, a second barrage was fired for fifteen
minutes concentrating accurately on the forward Turkish trenches.

At the same time “C” Company was taken from support, and by
means of quick concealed movement succeeded in moving to our
left flank, thereby taking up the position and rôle in the action which
should have been the Kensingtons’.

“C” Company was able to make some ground and also was able to
give great assistance to “A” and “B” Companies by bringing enfilade
fire to bear on the snipers and machine guns that were holding them
up.

As this second barrage lifted, the Battalion assaulted the steep hill
and in spite of heavy frontal fire were successful in capturing the
first line. By this time the two companies of the Queen’s
Westminsters had come up on to our left, and under the gallant
leadership of Captain Flower silenced the machine guns which had
enfiladed our attack. When the first ridge was gained it was found
that a valley beyond led to the second ridge, and here the Turks
were again strongly resisting. After a breather, however, the
“Charge” was given, and the men, led by Captain Wills and Company
Sergeant Major Oldcorn of “C” Company, rushed over the valley and
gained the second ridge only to find a still further ridge strongly
held. Machine-gun fire from the flank was still causing considerable
casualties to us, but after a breather, the Battalion assaulted the
third ridge, and thus completed the capture of Jebel Ektief. The
Turks fled down the precipitous slopes, into the Jordan Valley, and
could be seen collecting in small groups as though preparing for a
counter-attack. The Battalion’s casualties in this action numbered
about one hundred.

Further north of Jebel Ektief, the London Irish of the 180th Brigade
had succeeded in capturing Talat ed Dumm after heavy fighting. The
Turk, however, still had isolated machine guns on the hills between
the recently captured positions of Jebel Ektief and Talat ed Dumm,
and these poured a persistent fire into our flank. In the late
afternoon the Kensingtons came up on our left, and soon cleared the
ground of these irritating machine guns, and by the evening
everything was quiet except for an occasional shell from a Turkish
battery which could be seen some couple of miles away across the
Jordan Plain, well out of the range of our guns. Just before dark we
had a splendid view of the Jericho Plain.

No other valley in the world presents such extraordinary physical


features, none other has been the subject of such various theories
as to its origin and character. From our position on the eastern edge
of Jebel Ektief, the ground sloped steeply for about 2,000 feet to the
flat plain below, which stretched for over twenty miles to the foot-
hills of the Moab Mountains. Winding through the plain the river
Jordan could be seen in places where its steep banks were broken
by small branch ravines; while to the south the dismal flat water of
the Dead Sea was visible. The only town on this extensive plain was
Jericho, and as the light failed it appeared as a dead city of mud
huts.

The following morning, the 21st February, the London Scottish were
moved on to the plain as far as the white ruins of Neba Musa, and
Australian cavalry patrolled the Jericho area. The Turk had crossed
the Jordan, and all was quiet. About midday, however, four enemy
planes flew over our positions, and, but for a remarkable cloud
which suddenly enveloped the hill, would no doubt have poured
bullets into our bivouacs. When the cloud lifted they were nowhere
to be seen. For two days the Battalion remained on the heights of
Jebel Ektief, and although free from fighting it was a most strenuous
business carrying rations and water from the Quartermaster’s Stores
and Transport which were now in the Jebel Ektief Wadi, from which
the attack had started, some 600 or 700 feet below the summit of
the hill.

On the 23rd February, the Battalion was ordered to move as far back
as Jerusalem.
CHAPTER XXXVII
JERUSALEM—MUKMAS—JERICHO—ES SALT—WADI EL JIB

On the 23rd of February the Battalion left Jebel Ektief, and after
some discussion in the wadi near the transport lines, some
companies chose the route via the Wadi Hindi, Abu Dis and Bethany,
while others struck across country over a distinct track, supposed to
be the remains of a pilgrims’ road to the Jordan, and thence along
the main Jerusalem-Jericho Road, via Bethany. Which was the better
route has not been decided, sufficient to say that the troops hated
both. Rain poured down when we were half-way to Jerusalem, and
when Sir John Grey Hills’ house was reached late at night no one
had the heart to argue as to the better way from Jebel Ektief. On the
following day (Sunday) we availed ourselves of the rest. In the
evening many attended Divine Service in the huge chapel of the
German Emperor’s Palace, near to the Mount of Olives. It is a huge
structure, and a wonderful view of the surrounding country could be
obtained from the tower. It was used as a Corps Headquarters
during the operations around Jerusalem; not quite the purpose for
which the Kaiser had intended it. The chapel was wonderfully
decorated inside, but the two things which struck one as being odd,
and even profane, were the panelling of the altar, which was
decorated by the crests of each of the Kaiser’s sons; and the ceiling,
which was divided into two parts; one containing a beautiful painting
of the Son of God, while the other contained, equally well painted, a
picture of the Kaiser and his wife sitting on their thrones!

On the 25th of February the Battalion moved from Jerusalem,


further north to Mukmas; the route was along the Nablous Road as
far as Er Ram, a distance of about five miles, and here we struck off
the main road eastwards over the roughly made military roads as far
as Jeba. Here the route became more difficult and was across rough
mountain paths making it necessary at times for the Battalion to
march in single file. The transport could not travel the whole route,
and was halted near Mukmas, from which point camels and mules
carried stores and rations. The Battalion halted on the hills near
Umm et Talah, and outposts facing the Jericho plain were taken up
by “D” Company. The distance as the crow flies from Jebel Ektief to
Umm et Talah was about five miles, but the country was of such a
nature that the only route between these points entailed a long
march through Jerusalem, a distance of nearly twenty miles. Just in
front of the outpost position was Jebel Kuruntul, reputed to be the
Mount of Temptation. For nearly a month of delightful weather all
was peaceful on our immediate front. Occasionally patrols pushed
out to the front and explored the deep ravines leading on to the
Jericho Plain, but only once were we fired on by the Turks, and then
from a considerable range, probably 1,000 yards. Acting on the
principle, however, that the Devil finds work for idle hands to do,
that wonderful pastime for tired troops was revived, namely road-
making, and large fatigue parties were so employed. It had been a
source of great interest to some of us that nowhere behind the
Turkish lines, away from the four main roads which run north to
Nablous, south to Bethlehem, east to Jericho, and west to Jaffa from
the city of Jerusalem, could any trace be found of ways of
communication, except an occasional mule track; whereas, behind
our own lines, a network of specially constructed tracks was always
to be found. It was a mystery to us how the Turks ever moved their
guns, men and supplies in sufficient quantity to even replace normal
daily wastage, but the fact that after the repulse of the counter
attack on Jerusalem in December, 1917, a Turkish Quartermaster
surrendered to one of our posts because he, a Quartermaster, could
not get sufficient food, may indicate that they suffered from
shortages through lack of roads and means of communication.

We had often gazed at dawn on the gorgeous sunrise over the


Mountains of Moab, and seen the dull silver of the Dead Sea turned
to burnished gold; we had also seen that white streak across the
dark plain of Jericho which denoted the presence of that wonderful
stream, the Jordan, but the thought that the words of the old song
“One more ribber to cross” would ever come literally true never
entered our heads. If it did we gave it no serious thought. We had
heard of the Jordan Valley and its appalling heat in summer (for had
not the Turk left in Jericho a note to the effect that they would
return in the autumn to bury us), and also were familiar with the
expression “go to Jericho,” so, indeed, had no wish to spend any
time there. If the shades of the hereafter are hotter than the scene
of Joshua’s miracle they can have no terrors for the 60th London
Division. “Brass hats,” however, have ever since their creation been
seers of visions and dreamers of dreams, but their decision to give
us an Easter holiday in Moab could only have been the result of a
nightmare.

Accordingly, the 21st March, 1918, saw the 60th Division


concentrated with the Anzac Mounted Division in the Jordan Valley,
ready to carry out a raid on a scale unheard of in the annals of the
war: that is, penetrating over thirty miles into enemy territory of the
wildest nature, with the destruction of the Hedjaz Railway at Annam
as the objective. The Civil Service Rifles crossed the river on the
afternoon of the 23rd of March by a pontoon bridge at Makhadet
Hajlah, which had been gallantly constructed under fire by the
Engineers, and moved up the left bank of the river, passing on our
way the Commander-in-Chief, General Allenby, and the Duke of
Connaught, who was on a tour of inspection well within range of the
Turkish field artillery. Our object to the north was to rout out a nest
of machine guns which was delaying the crossing at the El
Ghoranyieh Ford some five miles to the north, but to our delight this
particular body had realised the threat to their rear, and had
“impshied.”

We bivouacked for the night in the Wadi Nimrin, near Umm Enkhala,
and at 5.0 a.m. the next day moved out in support of the London
Scottish and the Queen’s Westminsters, who attacked the hill of El
Haud which, held in force by the Turk, barred the second means of
approach to Es Salt—that by way of the Wadi Arsinyet.

A Boche “pip squeak” battery, which endeavoured to hinder us, drew


the attention upon itself of one of our own 4·5 howitzer batteries,
and speedily ceased fire. We were not called upon to assist, and
were fortunate to witness a most successful attack in open order
against an extended position. The sight of lines of London infantry
advancing calmly under a heavy enemy field and machine-gun fire,
and also of the final assault with the bayonet was one to be
remembered and treasured. El Haud was taken with little loss to our
forces, and we moved up to the foot-hills and bivouacked in the
Wadi Arsinyet, near El Haud.

The river Jordan at Makhadet Hajlah is 1,200 feet below sea level,
and El Haud is 846 feet above, and the weather changed, becoming
steadily worse. The night was wet and bitterly cold, and we were not
sorry to start at 7.0 a.m. the next day on our advance up the
Arsinyet track to Es Salt (Ramoth Gilead).

In heavy rain up a most slippery, muddy, and stony track, we toiled


upwards and upwards as advanced guard to the Brigade, which,
with a Regiment of Australian Light Horse and Mountain Battery, was
soon strung out in single file. Communication was maintained with
the 181st Brigade on the main Es Salt Road by means of a portable
wireless installation, carried by the Light Horse. After a most
exhausting climb of nearly 3,000 feet, we arrived on the outskirts of
Es Salt about 1.0 p.m., thoroughly fatigued and soaked to the skin.
The weather was bitterly cold, and it was felt all the more, for in a
little over twenty-four hours we had moved from the tropical heat of
the Jordan Valley up some 4,800 feet to the climatic conditions in
winter of the top of a mountain higher than Snowdon. The Turk had
retired hurriedly, and offered little opposition, and further progress
being impossible owing to our exhausted state we bivouacked—and
slept in inches of mud. The night was disturbed by much rifle fire
from Es Salt, and at dawn on the 26th, No. 16 Platoon was detailed
to reconnoitre the town. Under Lieutenant Andrew, who was acting
as O.C. Company, they entered the town to find no trace of the Turk,
and to receive a tumultuous welcome from the population who,
during the night, had kept up a “feu de joie” from their house-tops,
which noise we had interpreted as hostile rifle fire. We were the first
British infantry to enter Es Salt, for it was not until some hours later
that the 181st Brigade, which had been moving up the main road
from Shunet Nimrin, arrived in the town. Supplies were lacking
owing to the great difficulty of getting the camels up the slippery
mountain track, and an officer was detailed to proceed to the 181st
Brigade and bring back some water camels—water being the
pressing need.

The remainder of the Division moved up the Annam road, and the
defence of Es Salt was left to the Civil Service Rifles. The same
evening the Battalion moved out to the north-east of the town to
Kefi Huda (3,597 feet), and took up a position covering the route
from the north to Es Salt.

News came through of a possible attack by some 2,000 Circassian


cavalry, and arrangements were at once made for their proper
reception. Time available was short, but by toiling all night under the
protection of standing patrols of the Australian Light Horse, dawn on
the Maundy Thursday saw the 2/15th snugly settled ready behind
stone sangars. Firing started with the first light of dawn and
continued throughout the day, but no serious attack was attempted
by the Turk.

At dawn on Good Friday, under cover of a thick mist, the Turks


attacked with bombs a post in the centre of the Battalion. An
energetic reply with Mills hand and rifle grenades, and a burst of fire
from a machine gun resulted in their speedy repulse. The mist
suddenly lifted, and opposite a neighbouring post was discovered a
Turkish captain and his batman, with their kit complete. He was
brought in and it transpired that he had set out with 200 men to
assault the hill under cover of the mist, but his left flank got too far
forward and pressed the attack before the remainder were ready,
and they in consequence—like the burglar who heard the noise
overhead—thought all was lost and fled, leaving their officer
stranded on the hillside. In his pocket was found a small diagram
which showed the relative positions of all our Lewis and machine
guns, and this caused no little searching in the mind, for we had, as
we thought, camouflaged them successfully. However, even if he,
the captain, knew, his snipers apparently did not, for, though
throughout the week-end we were continually sniped, one of our
Lewis guns, though continually in action, had not a shot fired at it.

SKETCH MAP OF OPERATIONS AROUND JERUSALEM

There were constant alarms, and on the night of the 30th/31st


March “D” Company repulsed four separate determined attacks on
their position, with no loss to themselves. On the right and left “A”
and “B” Companies were troubled slightly, but the brunt of the
defence of Es Salt fell on the centre of the line.

The situation was nevertheless serious, and the 2/14th and 2/16th
Battalions were brought back to Es Salt.
The Battalion was relieved on the night of the 31st of March, and at
dawn moved down to the south side of Es Salt. The Divisional
withdrawal from Annam, the assault on which place had not been
entirely successful, was now in full swing, and the majority of the
Armenian population of Es Salt, terrified at the prospect of the return
of the terrible Turk, was hastily evacuating the town. The road,
which had been churned up by men, horses, guns, and wheeled
transport, was inches deep in liquid mud, and was packed with
refugees. Old men and boys, women and children of all ages, with
their household goods in large bundles on their backs, staggered
along obsessed with great fear. Their flight was terrible, and not a
few of us hurriedly realised that it is not only troops who suffer in
war time. We could give little assistance; our own baggage wagons
were already overburdened and our own loads were heavy, but we
did what we could.

At 8.0 a.m. on the 1st of April (Easter Monday), after standing in full
marching order in the mud for a solid hour, we started our
retirement, and though hindered and hampered by the crowd of
refugees which surged and swayed either side of us, we marched
steadily until 5.0 a.m. next day, when we emerged from the hills at
Shunet Nimrin, and bivouacked for a few hours, after which we
continued and, crossing the Jordan River by the Ghoraniyeh Ford,
we arrived at a position in the Wadi Nuemiah about 3 p.m., having
covered a distance of about thirty miles since 8.0 p.m. the previous
night.

Every one was footsore, dirty, unshaven, with seven days’ growth of
beard, and thoroughly tired, but we soon settled down to our first
proper night’s rest for nine days, during which time we had
experienced almost the extremes of heat and cold, and had marched
about seventy miles through some of the most atrocious country our
army has ever operated in. The other Brigades had fared even worse
than we had, but the whole Division had covered itself with honour,
and had no reason to be dispirited at the apparent failure of its novel
enterprise. Gains in warfare are not measured by territory alone, and
the influence of this raid—indeed it was almost a campaign—had a
far reaching effect on the strategy of the Palestine campaign.

We were happy, on our return to the Jordan Valley, to welcome back


from leave in England our Commanding Officer, Lieut.-Colonel
Bisdee, and the Battalion, though it had worked well under its
temporary Commander, was delighted to see its Commanding Officer
back once again, and it speaks much for the spirit of comradeship of
the Regiment that no one was perhaps more pleased to be with us
again than Colonel Bisdee himself.

After a day’s rest in the valley, we moved once again up towards


Jerusalem, and halted at Talat ed Dumm for twenty-four hours. At
Talat ed Dumm there is a wayside house reputed to be the Good
Samaritan Inn of Biblical times. After this short rest we again pushed
up the old Jericho road, which we had come to know almost as well
as the Strand. On arriving at Jerusalem we were again billeted on
the Mount of Olives. The road from Jericho goes through particularly
desolate and wild country, and no doubt in years to come, many a
Civil Service Rifleman, when questioned by his offspring concerning
the man who fell among thieves, will truthfully say, “If you had seen
the road you would not be surprised.”

It was not a habit in those days to keep us idle, and we were soon
moved on up the Jerusalem-Nablous road via Bireh and Ram Allah to
the Wadi el Jib, where we temporarily relieved the 10th (Irish)
Division. The Battalion was in reserve, and bivouacked on a terraced
hillside which was covered with fig trees and vines. It was very hot,
and we were glad to resume our summer kit. The camping ground
was one of the most pleasant we had experienced in Palestine, and
we made the most of it. There were flowers and plants in great
profusion, and botanists amongst us spent hours collecting
specimens; one of them, a botanist by profession, collected a
hundred or so specimens which he had never seen before, and
which he could not name.
Of actual war we saw but little, occasional bombing raids were made
by enemy planes, but our chief enemy was just behind us—a gunner
officer in charge of an 18-pounder battery, whose idea of humour
was to give an order for ten rounds gun fire regularly each night at
varying times between midnight and 4.0 a.m. Our considered
opinion of this gentleman cannot be published here.

A week later we moved back as far as Attara, where we gave a most


sincere and regretful God speed to our Colonel, who left us to take
command of the 2/13th Battalion London Regiment. Colonel Bisdee
had joined us at El Shaulth in September, 1917, and no Commanding
Officer was ever so popular, both with his officers and men. He had
instilled into the Regiment a magnificent spirit, and it is no idle boast
to say that the Regiment would have gone into action anywhere and
under any circumstances with Colonel Bisdee, with no misgivings
and fully aware that, whatever happened, he would be their leader.
Photo by Searle, S.W.
LT.-COL. T. E. BISDEE, D.S.O., M.C. (D.C.L.I.)
Commanded 2nd Battalion, 9th October, 1917, to May, 1918.
To face page 338.
CHAPTER XXXVIII
EL HAUD—AIN ARIK—SURAFEND—KANTARA—ALEXANDRIA
—JOURNEY TO ITALY

On the 23rd of April, 1918, the Battalion marched south from Attara
and encamped for the night just north of Ram Allah, near Lake
Ballua. The march was carried out during the evening, and the
bright weather made it enjoyable. On the two following days the
journey was continued towards Jerusalem, and on the night of the
25th we halted at Shafat, a few miles north of the city. Rumours of
another “stunt” across the Jordan were then confirmed, and on the
night of the 27th the Battalion marched down the Jericho road once
more to Talat ed Dumm, arriving there about 11.0 p.m., where the
concentration of transport foreboded another operation on a large
scale. The following evening the Battalion moved to the Jordan
Valley, and crossed over the river at Ghoraniyeh Ford, which was
then held by Indian troops. Although it was only April the heat in the
valley was terrific during the day, and white troops would, no doubt,
have suffered considerable losses from disease had they been
stationed there for any considerable length of time. The next day
was spent amid the thick brushwood on the plain, and every
available piece of shade was used, but even under this protection
the damp heat was nigh unbearable, and only a minimum of clothing
in the shape of drill shorts were worn by most. There was a large
concentration of troops in the area, and preparations were made for
an attack on the foot-hills of the Moab Mountains.

On the 30th the attack was launched by the 179th Brigade, the
Westminsters on the left and the London Scottish on the right, while
the 180th Brigade was operating farther south. Under a heavy
artillery bombardment the troops advanced from their positions of
deployment, which they had taken up during the night. They
attacked the first ridge and, after heavy fighting, obtained a footing.
Many prisoners were captured and casualties were inflicted on the
enemy. The 2/15th were held in reserve at Ghoraniyeh Ford, and
when a crowd of over 200 prisoners appeared we felt that the
operation was so successful that we should not be needed; it
transpired afterwards, however, that these Turks were not captured
in the actual assault, but had been detailed to reconnoitre our
concentration on the plain, and while coming over had walked into
the London Scottish, who were pushing forward for the attack. They
made no resistance, and were soon sent to the rear as prisoners.

After the first ridge had been taken it was found that the whole
position was strongly held by the Turk, and that further advance
over deep ravines swept by machine guns would be very difficult.
Repeated attempts were made to push forward, but only heavy
casualties resulted. The actual objective of the 179th Brigade was a
steep hill called El Haud, which lay some six miles east of the
Jordan, and formed part of a continuous line of steep, rocky hills
that bordered the Jericho Plain on the east.

On the night of the 30th April the London Scottish had only been
able to gain the first ridge of El Haud after heavy fighting, and
during the following day held on to the captured ground with the
idea of pressing forward the attack at dawn next morning. In order
to support this second effort “A” and “B” Companies of the 2/15th
pushed forward to the foot of El Haud, to act as local supports. The
second attack on the 1st of May was also unsuccessful, not only at
this one particular spot but along the whole Divisional front. Heavy
machine-gun fire swept each ridge, both from the front and the
flanks, so cunningly had the Turk arranged his emplacements.
Throughout the day desultory fighting took place and small local
attacks were made, but no general advance could be accomplished.
During the evening of the 1st of May “C” and “D” Companies were
brought up from the Wadi Nimrin, where they had been moved the
previous night, and during the night of the 1st the Battalion relieved
the London Scottish on El Haud. Orders were received on the same
night to attack at dawn. This meant a busy night in reconnoitring.
“A” and “B” Companies were detailed for the attack, with “C”
Company in support, and just before dawn they deployed for the
attack on the right flank of the hill, from which point it appeared the
approach to the summit was easiest. “D” Company remained in
possession of the captured ridge.

The attack was launched forward with great spirit and dash, but was
met with the same murderous fire that had swept down the London
Scottish in their attempts, and the 2/15th could only push forward a
little way, gaining some 500 yards and a small hill, thus forming for
themselves an unpleasant little salient into the enemy’s lines. Even
then casualties were fairly heavy, and the work of carrying back the
wounded was made more hazardous by the Turk, who fired on every
stretcher with his machine guns throughout the day. Captain
Peatfield, “A” Company, was reported missing, but late at night he
returned under cover of darkness, having lain out all day in the sun,
a few yards from a Turkish machine gun, apparently dead.

At dusk the wounded were collected, and “A” and “B” were
withdrawn, “C” Company staying on the captured ground and hastily
organising it for defence by building sangars.

During the afternoon two platoons of “D” Company, under


Lieutenant K. P. Neall, had attempted to push forward from the main
ridge, but the progress was soon arrested by heavy machine-gun
fire, and this proved conclusively that frontal attacks were
impossible. From the result of this local attack it was learnt that
between ourselves and the Turk was a deep ravine with precipitous
banks. The night of the 3rd of May was fairly quiet except for the
intermittent machine-gun and rifle fire from both sides. On the
following morning, however, the Turk made a determined counter
attack on our immediate right, against the London Irish of the 180th
Brigade. Curiously, just before this counter attack took place, four
enemy planes flew over our lines and poured machine-gun fire down
at us. They did little actual damage, and one of them was brought
down by our Lewis-gun fire. Every one was on the alert for a
repetition of the aeroplane attack and sought cover behind the
numerous boulders on the hill. However, the planes sailed out of
sight behind the summit of El Haud, and just as every one thought
of coming from their cover a sentry group in front of the Battalion on
our right shouted “they’re coming over,” at which every one near by
took immediate cover, but were soon surprised to find that no
aeroplanes appeared but that a force of Turks about 250 strong
rushed their position and drove them off. The Turkish “coup,”
however, was short lived, and an immediate counter attack by the
“Irish” sent them hurrying back to their lines with loss. However, the
incident was disastrous to that Company of the 2/15th which was
holding the little salient which had been gained during the morning
attack of the previous day, as the Turk was able to pour machine-
gun fire into the rear of “C” Company, causing casualties. The gallant
action of Private Freer, a signaller, in remaining at his telephone
though severely wounded and under close-range rifle fire, materially
assisted the “Irish” to regain their position, for Captain Wills, of “C”
Company, was thus able to send back information which led to the
“Irish” receiving adequate support from our machine guns and
eighteen pounders.

Later in the day the enemy aeroplanes returned, but this time
devoted their attentions to the transport lines and A.S.C. dumps near
Ghoraniyeh Ford. For the rest of the day the infantry could make no
progress, and spent their time watching the artillery bombardment
of those places where machine guns had been located. The Turk was
clever in concealing his emplacements, and it was only in the
evening when a slight breeze sprang up were we able to detect one
that had been firing at us throughout the whole day. This machine
gun was in a cave in the hill side, and the entrance to the cave had
been cleverly camouflaged with a blanket of the same colour as the
surrounding rocks, and not until the breeze caused the blanket to
move was it realised that the fire had been coming from an
apparently solid rock. However, attentions were soon upon this spot,
and one of our machine guns poured heavy fire into the cave,
tearing the blanket in shreds, and no doubt killing the enemy
gunners.

On the evening of the 4th May orders were received that the whole
of the British Force would be withdrawn to the Ghoraniyeh
Bridgehead as the Turks had sent a considerable force down the
Jordan Valley from the north, and which had been driven back by
our cavalry that afternoon. However, it was realised that should this
attack by the Turk succeed our left flank and our line of retreat
would both be seriously threatened. In the Battalion arrangements
were made for “D” Company to be left as a rear-guard, holding the
hill of El Haud until the rest of the Battalion had got a good start
across the plain. “D” Company kept up an occasional Lewis gun and
rifle fire for nearly an hour and then left the hill to the mercy of the
Turks. Every one was relieved to reach the wire entanglements of
the Ghoraniyeh defences the gaps in which were guarded by Indian
cavalry. The amusing part, however, was that when the Battalion
returned to its bivouac area of a few days previous they found “D”
Company there first. Apparently being a smaller force they were
more mobile and certainly they knew the shortest cut home. Once
inside the wire a few hours’ rest was granted, and at 3 a.m. the
withdrawal was resumed as far as Tel el Sultan, a ruined village near
Jericho. We arrived at our bivouac area early in the morning, and in
the glorious warmth of the day every one enjoyed a thorough rest
after the unsatisfactory hard fighting of the past few days. Before
leaving the subject of El Haud a word of praise is due to the
Divisional R.A.M.C. for the splendid manner in which they evacuated
the wounded during the fighting, and had erected a large field
hospital tent well within range of shell fire under the slopes of El
Haud.

On the 6th of May the Battalion had the surprise of its life, when
orders were received to pack up and move to Jericho, where motor
lorries were to take us as far back as Jerusalem. Never in its
existence had the Battalion partaken of such luxury, but it is
regretted that we all thought that this kindness to tired troops was
to move them to another part of the front where they were needed
in a hurry.

In the late hours of the afternoon the Battalion boarded the lorries
and the convoy started its journey to Jerusalem, along the new
Jericho road, which is a masterpiece in mountain roads, and was
built under German influence. About half-way home we met several
regiments of Indian cavalry proceeding in the direction of the Jordan
Valley, where they were to hold the line during the summer months.
Many of them wore the Mons Star ribbon, and apparently had seen
fighting in France. After passing these troops rain commenced to fall
and those who had greedily sought the seats beside the driver paid
the penalty and were soon drenched through. The lorries pulled up
just north of Jersualem on the Nablous road, from which point each
load of men marched independently to the camping ground near
Shafat, which we had occupied only a week previously. The writer
was detailed to supervise the unloading of the Battalion, and when
they had finally “debussed” he made his way to Shafat probably
about 45 minutes after the first party of the 2/15th had left their
lorry. In spite of the rain and the muddy ground bivouacs had been
erected and each one had a small candle burning inside. To see the
camp from the roadway it would have appeared to a casual observer
that it had been in existence for several weeks, so expert had the
Battalion become in erecting its temporary home. Finding his
bivouac the writer was soon presented with a plate of bread, bully
beef, pickles, cheese and a mug of tea by his batman.

On the 7th, the G.O.C. Division inspected the Battalion and


complimented it on the splendid way in which it had stuck to its
unsuccessful work in the recent visit across the Jordan.

The following day the Battalion moved north a few miles as far as
Ram Allah; the march was not long, the road was fair, and the
weather was glorious, and by 2 p.m. the Battalion had settled down
in its new area, and dinners were being cooked. I have already
referred to the speed with which the Battalion made themselves at
home on the night we arrived at Shafat, but as a further illustration
the following true story may be given. On this day’s march we were
followed by the London Scottish, who were in turn followed by an
officer of our own Battalion and a draft fresh from England who
were on their way to join us. Having missed the Battalion as it left
Shafat this new officer attempted to catch us up. We arrived at our
bivouac area and moved off the road to allow the Scottish to pass to
their camp which was farther north. We immediately put up our
bivouacs and commenced cooking. The new officer and his men
followed the Scottish for some two miles only to find that the
encampment he had passed some 25 minutes before was really that
of his own Battalion. It was hard to make him realise that we were
only just ahead of the Scottish on the march and turning off the
road had immediately set to and erected our camp.

Here news of a long rest reached us, and we were told that after the
next day’s march to Ain Arik we should have a complete rest. On the
way to Ain Arik we were inspected by the Commander-in-Chief,
General Allenby, and on the afternoon of the 9th arrived at our new
camp. The camp was situated on the rocky slopes of a deep wadi;
little ledges were however found for “bivvies,” and each company
made itself at home in its particular area. The slopes were covered
with many small trees, and it was undoubtedly one of the prettiest
spots in Palestine that we had visited. Here the Battalion remained
for 10 days, and everything was done to make the rest enjoyable. At
the top of the wadi near the main roadway a few level places were
found and football competitions, both inter-Battalion and inter-
company were arranged. A Brigade Sports Committee was set up
and a varied programme of games, sports and transport
competitions was arranged. The final placings were as follows:
2/15th Battalion, 10 points; 2/14th Battalion, 10 points; 2/16th
Battalion, 3 points; and 2/13th Battalion, 3 points. Great credit was
due to the excellent turn out of our transport section, and also our
Lewis gunners in their particular competitions. In the Brigade
football final the London Scottish beat the Queen’s Westminsters
after a hard game. The Divisional Concert Party arrived and gave
nightly performances in a large marquee. In the Battalion itself
company concerts were held, a piano being hired all the way from
Jerusalem through the personal efforts of Lieutenant Phelps. The
whist drive held by “B” Company must also be included in the
“mention in dispatches.” Shooting competitions were held at a small
range built at the bed of the wadi, and Battalion sports, limited to
the three-legged, sack, egg and spoon type of race, were held on
the flat bed of the wadi and proved a huge success. There was no
suitable ground for sprinting, and even the course for the comic
races had to be cleared of large stones by a voluntary fatigue party.
One competition of the afternoon which deserves special mention
was a “beauty competition for the best dressed lady.” The originality
of the aspirants was marvellous, and real harem dresses with the aid
of bacon wrappings, etc., were among the prize winners. Canteen
stores were plentiful and “dinner” parties were given throughout the
camp each night. A real happy time was spent here and the
Battalion was loath to leave Ain Arik when orders were issued on the
night of the 19th May.

The Battalion proceeded next morning to Beit Ello by a circuitous


route around the many hills in that part of Palestine. A pleasant
camping ground was found, and after a night’s rest we proceeded
the next morning to Beit Rima, which was near the centre of the
British line in Palestine which then extended from Jaffa, on the west,
to a point just south of Nablous on the east. Here the proximity of
the enemy’s observation posts necessitated us to place our bivouacs
under the numerous olive trees on the terraces round the village. For
the next week or so the Battalion rested by day and each night large
working parties sallied forth north of the village of Beit Rima to dig a
system of trenches for the defence of the hill.

Rumours of France commenced to float about, and these rumours


soon developed into fact. On the 29th of May, 1918, the Battalion,
Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade

Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.

Let us accompany you on the journey of exploring knowledge and


personal growth!

ebooknice.com

You might also like