(Ebook) Advanced Data Analytics Using Python: With Architectural Patterns, Text and Image Classification, and Optimization Techniques [2nd Edition] by Sayan Mukhopadhyay, Pratip Samanta ISBN 9781484280041, 1484280040 - Experience the full ebook by downloading it now
(Ebook) Advanced Data Analytics Using Python: With Architectural Patterns, Text and Image Classification, and Optimization Techniques [2nd Edition] by Sayan Mukhopadhyay, Pratip Samanta ISBN 9781484280041, 1484280040 - Experience the full ebook by downloading it now
com
OR CLICK BUTTON
DOWLOAD EBOOK
(Ebook) Advanced Data Analytics Using Python : With Architectural Patterns, Text and
Image Classification, and Optimization Techniques by Sayan Mukhopadhyay; Pratip
Samanta ISBN 9781484280058, 1484280059
https://ebooknice.com/product/advanced-data-analytics-using-python-with-
architectural-patterns-text-and-image-classification-and-optimization-
techniques-49612318
ebooknice.com
(Ebook) Biota Grow 2C gather 2C cook by Loucas, Jason; Viles, James ISBN
9781459699816, 9781743365571, 9781925268492, 1459699815, 1743365578, 1925268497
https://ebooknice.com/product/biota-grow-2c-gather-2c-cook-6661374
ebooknice.com
(Ebook) Advanced Data Analytics Using Python: With Machine Learning, Deep Learning
and NLP Examples by Mukhopadhyay, Sayan ISBN 9781484234495, 1484234499
https://ebooknice.com/product/advanced-data-analytics-using-python-with-machine-
learning-deep-learning-and-nlp-examples-55670484
ebooknice.com
(Ebook) SAT II Success MATH 1C and 2C 2002 (Peterson's SAT II Success) by Peterson's
ISBN 9780768906677, 0768906679
https://ebooknice.com/product/sat-ii-success-math-1c-and-2c-2002-peterson-s-sat-
ii-success-1722018
ebooknice.com
(Ebook) Matematik 5000+ Kurs 2c Lärobok by Lena Alfredsson, Hans Heikne, Sanna
Bodemyr ISBN 9789127456600, 9127456609
https://ebooknice.com/product/matematik-5000-kurs-2c-larobok-23848312
ebooknice.com
(Ebook) Cambridge IGCSE and O Level History Workbook 2C - Depth Study: the United
States, 1919-41 2nd Edition by Benjamin Harrison ISBN 9781398375147, 9781398375048,
1398375144, 1398375047
https://ebooknice.com/product/cambridge-igcse-and-o-level-history-
workbook-2c-depth-study-the-united-states-1919-41-2nd-edition-53538044
ebooknice.com
(Ebook) Master SAT II Math 1c and 2c 4th ed (Arco Master the SAT Subject Test: Math
Levels 1 & 2) by Arco ISBN 9780768923049, 0768923042
https://ebooknice.com/product/master-sat-ii-math-1c-and-2c-4th-ed-arco-master-
the-sat-subject-test-math-levels-1-2-2326094
ebooknice.com
(Ebook) Advanced Analytics with Transact-SQL: Exploring Hidden Patterns and Rules in
Your Data by Dejan Sarka ISBN 9781484271728, 1484271726
https://ebooknice.com/product/advanced-analytics-with-transact-sql-exploring-
hidden-patterns-and-rules-in-your-data-33557194
ebooknice.com
(Ebook) Python Data Analytics: Data Analysis and Science Using Pandas, Matplotlib
and the Python Programming Language by Nelli Fabio ISBN 9781484209592, 1484209591
https://ebooknice.com/product/python-data-analytics-data-analysis-and-science-
using-pandas-matplotlib-and-the-python-programming-language-38169124
ebooknice.com
Advanced Data
Analytics Using
Python
With Architectural Patterns,
Text and Image Classification,
and Optimization Techniques
Second Edition
Sayan Mukhopadhyay
Pratip Samanta
Advanced Data Analytics Using Python: With Architectural Patterns, Text
and Image Classification, and Optimization Techniques
v
Table of Contents
Normal Forms�����������������������������������������������������������������������������������������������������31
First Normal Form�����������������������������������������������������������������������������������������31
Second Normal Form�������������������������������������������������������������������������������������32
Third Normal Form����������������������������������������������������������������������������������������33
Elasticsearch�������������������������������������������������������������������������������������������������������35
Connection Layer API�������������������������������������������������������������������������������������38
Neo4j Python Driver��������������������������������������������������������������������������������������������39
neo4j-rest-client�������������������������������������������������������������������������������������������������39
In-Memory Database������������������������������������������������������������������������������������������40
MongoDB (Python Edition)����������������������������������������������������������������������������������40
Import Data into the Collection����������������������������������������������������������������������41
Create a Connection Using pymongo�������������������������������������������������������������42
Access Database Objects������������������������������������������������������������������������������42
Insert Data�����������������������������������������������������������������������������������������������������43
Update Data���������������������������������������������������������������������������������������������������43
Remove Data�������������������������������������������������������������������������������������������������43
Cloud Databases�������������������������������������������������������������������������������������������������43
Pandas����������������������������������������������������������������������������������������������������������������44
ETL with Python (Unstructured Data)������������������������������������������������������������������45
Email Parsing�������������������������������������������������������������������������������������������������45
Topical Crawling��������������������������������������������������������������������������������������������48
Summary������������������������������������������������������������������������������������������������������������52
vi
Table of Contents
Semi-Supervised Learning���������������������������������������������������������������������������������65
Decision Tree�������������������������������������������������������������������������������������������������������66
Which Attribute Comes First?������������������������������������������������������������������������66
Random Forest Classifier������������������������������������������������������������������������������68
Naïve Bayes Classifier�����������������������������������������������������������������������������������������68
Support Vector Machine��������������������������������������������������������������������������������������69
Nearest Neighbor Classifier��������������������������������������������������������������������������������71
Sentiment Analysis���������������������������������������������������������������������������������������������71
Image Recognition����������������������������������������������������������������������������������������������73
Regression with Python���������������������������������������������������������������������������������74
Least Square Estimation��������������������������������������������������������������������������������75
Logistic Regression���������������������������������������������������������������������������������������76
Classification and Regression�����������������������������������������������������������������������������76
Intentionally Bias the Model to Over-Fit or Under-Fit������������������������������������������77
Dealing with Categorical Data�����������������������������������������������������������������������������78
Summary������������������������������������������������������������������������������������������������������������79
vii
Table of Contents
viii
Table of Contents
ix
Table of Contents
Index�������������������������������������������������������������������������������������������������243
x
About the Authors
Sayan Mukhopadhyay has more than 13 years
of industry experience and has been associated
with companies such as Credit Suisse, PayPal,
CA Technologies, CSC, and Mphasis. He has
a deep understanding of applications for
data analysis in domains such as investment
banking, online payments, online advertising,
IT infrastructure, and retail. His area of
expertise is in applying high-performance
computing in distributed and data-driven
environments such as real-time analysis, high-
frequency trading, and so on.
He earned his engineering degree in electronics and instrumentation
from Jadavpur University and his master’s degree in research in
computational and data science from IISc in Bangalore.
xi
About the Technical Reviewer
Joos Korstanje is a data scientist with more
than five years of industry experience in
developing machine learning tools, of which a
large part is forecasting models. He currently
works at Disneyland Paris where he develops
machine learning for a variety of tools.
xiii
Acknowledgments
Thanks to Labonic Chakraborty (Ripa) and Soumili Chakraborty.
xv
Introduction
We are living in the data science/artificial intelligence era. To thrive in
this environment, where data drives decision-making in everything from
business to government to sports and entertainment, you need the skills
to manage and analyze huge amounts of data. Together we can use this
data to make the world better for everyone. In fact, humans have yet to find
everything we can do using this data. So, let us explore!
Our objective for this book is to empower you to become a leader
in this data-transformed era. With this book you will learn the skills to
develop AI applications and make a difference in the world.
This book is intended for advanced user, because we have incorporated
some advanced analytics topics. Important machine learning models and
deep learning models are explained with coding exercises and real-world
examples.
All the source code used in this book is available for download at
https://github.com/apress/advanced-data-analytics-python-2e.
Happy reading!
xvii
CHAPTER 1
OOP in Python
In this section, we explain some features of object-oriented programming
(OOP) in a Python context.
The most basic element of any modern application is an object. To
a programmer or architect, the world is a collection of objects. Objects
consist of two types of members: attributes and methods. Members can be
private, public, or protected. Classes are data types of objects. Every object
is an instance of a class. A class can be inherited in child classes. Two
classes can be associated using composition.
Python has no keywords for public, private, or protected, so
encapsulation (hiding a member from the outside world) is not implicit in
Python. Like C++, it supports multilevel and multiple inheritance. Like Java,
it has an abstract keyword. Classes and methods both can be abstract.
© Sayan Mukhopadhyay, Pratip Samanta 2023 1
S. Mukhopadhyay and P. Samanta, Advanced Data Analytics Using Python,
https://doi.org/10.1007/978-1-4842-8005-8_1
Chapter 1 A Birds Eye View to AI System
glob is I
prok is V
pish is X
tegj is L
glob glob Silver is 34 Credits
glob prok Gold is 57800 Credits
pish pish Iron is 3910 Credits
how much is pish tegj glob glob ?
how many Credits is glob prok Silver ?
how many Credits is glob prok Gold ?
how many Credits is glob prok Iron ?
how much wood could a woodchuck chuck if a woodchuck could
chuck wood?
Program has a knowledge base in config.txt.
I,1,roman
V,5,roman
X,10,roman
L,50,roman
C,100,roman
D,500,roman
M,1000,roman
Based on this input and the configuration program, the answer to the
question is given in input.txt in standard output, as shown here:
2
Chapter 1 A Birds Eye View to AI System
import operator
def __init__(self,cmpiler):
self.compiler = cmpiler
3
Chapter 1 A Birds Eye View to AI System
if current == 5:
raise Exception("Wrong Substraction")
def evaluate(self):
#check mximum repeatation is crossing the limit
if self.count > 3:
raise Exception("Repeat more than 3")
#symbol is proper or not
if self.symbol not in self.compiler.symbol_map:
raise Exception("Wrong Symbol")
#check if wrong symbol is repeated ie (V, ..
self.symbol,unit = self.compiler.
evaluateSymbol(self.symbol)
while self.symbol % 10 != 0:
self.symbol = self.symbol / 10
if self.count > 1 and self.symbol == 5:
raise Exception("Wrong Symbol repeated")
import sys
class compilerTrader(object):
#store mapping of symbols with score and unit
symbol_map = {}
#store the list of valid symbol
valid_values = []
4
Chapter 1 A Birds Eye View to AI System
5
Chapter 1 A Birds Eye View to AI System
else:
#logic for value with unit
if ' ' in value:
fields = value.split(' ')
user_unit = fields[-1]
if ' ' not in var:
self.symbol_map[var] =
int(fields[0]), user_unit
else:
#logic for multiple symbols
in input
total = int(fields[0])
factor = 0
arr = var.split(' ')
obs.initialize(arr)
for i in range(len(arr)):
obs.increment(arr[i])
if arr[i] in self.symbol_map
and arr[i+1] in self.symbol_map
and i < len(arr) -1:
current, current_
unit = self.
evaluateSymbol(
[arr[i]][0])
next, next_unit = self.
evaluateSymbol(
[arr[i+1]][0])
if current >= next:
factor = factor
+ current
6
Chapter 1 A Birds Eye View to AI System
else:
obs.verifySubstract(
current)
factor = factor -
current
else:
if arr[i] in self.
symbol_map:
current, current_
unit = self.
evaluateSymbol(
[arr[i]][0])
factor = factor +
current
else:
self.
symbol_map[
arr[i]] =
total/factor,
user_unit
self.valid_
values.append
(total/factor)
obs.calculate()
obs.evaluate()
The answering logic is in the answer layer, which calls Observer and
compiler. The answering class inherits the compiler class.
import sys
7
Chapter 1 A Birds Eye View to AI System
class answeringTrader(compilerTrader):
8
Chapter 1 A Birds Eye View to AI System
obs.calculate()
obs.evaluate()
values = values.replace("?" , "is ")
if unit == 'roman':
unit = ''
return(values + str(ans) + ' ' + unit)
Finally, the main program calls the answering class and the observer,
and then it performs the task and does unit testing on the logic.
9
Chapter 1 A Birds Eye View to AI System
import sys
import unittest
sys.path.append('./answerLayer')
sys.path.append('./compilerLayer')
sys.path.append('./utilityLayer')
def setUp(self):
pass
10
Chapter 1 A Birds Eye View to AI System
11
Chapter 1 A Birds Eye View to AI System
if __name__ == '__main__':
if len(sys.argv) != 3:
print("Usage is : " + sys.argv[0] + " <intput file
path> <config file path>")
exit(0)
tr = ClientTrader(sys.argv[2])
f = open(sys.argv[1])
for line in f:
response = tr.process(line.strip())
if response is not None:
print(response)
TestTrader.trader = tr
unittest.main(argv = [sys.argv[0]], exit = False)
12
Chapter 1 A Birds Eye View to AI System
import rpy2.robjects as ro
ro.r('data(input)')
ro.r('x <-HoltWinters(input data frame)')
import subprocess
subprocess.call(['java','-cp','*','edu.stanford.nlp.sentiment.
SentimentPipeline','-file','foo.txt'])
Please place foo.txt in the same folder where you run the
Python code.
nlp = StanfordCoreNLP('http://127.0.0.1:9000')
output = nlp.annotate(sentence, properties={
"annotators": "tokenize,ssplit,parse,sentiment",
"outputFormat": "json",
# Only split the sentence at End Of Line. We assume that this
method only takes in one single sentence.
"ssplit.eolonly": "true",
# Setting enforceRequirements to skip some annotators and make
the process faster
"enforceRequirements": "false"
})
14
Chapter 1 A Birds Eye View to AI System
15
Chapter 1 A Birds Eye View to AI System
def get_hour(timestamp):
return dt.datetime.utcfromtimestamp(timestamp /
1e3).hour
def get_value(session, scores, feature_name, feature_value):
s = scores.select((scores.c.feature_name == feature_
name) & (scores.c.feature_value == feature_value))
rs = s.execute()
row = rs.fetchone()
if row is not None:
return float(row['score'])
else:
return 0.0
@app.route('/predict', methods=['POST'])
def predict():
input_json = request.get_json(force=True)
features = ['size','domain','client_time','device','ad_
position','client_size', 'ip','root']
predicted = 0
feature_value = ''
for f in features:
if f not in g.dropped_features:
if f == 'ip':
feature_value = str(ipaddress.
IPv4Address(ipaddress.ip_
address(unicode(request.remote_
addr))))
else:
feature_value = input_json.get(f)
if f == 'ip':
if 'geo' not in g.dropped_features:
16
Chapter 1 A Birds Eye View to AI System
geo = g.gi.country_name_by_
addr(feature_value)
predicted = predicted + get_
value(g.session, g.scores,
'geo', geo)
17
Chapter 1 A Birds Eye View to AI System
import falcon
from falcon_cors import CORS
import json
from sqlalchemy import *
from sqlalchemy.orm import sessionmaker
import pygeoip
from pymongo import MongoClient
import json
import datetime as dt
import ipaddress
import math
from concurrent.futures import *
from sqlalchemy.engine import Engine
from sqlalchemy import event
import sqlite3
@event.listens_for(Engine, "connect")
def set_sqlite_pragma(dbapi_connection, connection_record):
cursor = dbapi_connection.cursor()
cursor.execute("PRAGMA cache_size=100000")
cursor.close()
class Predictor(object):
def __init__(self,domain):
db1 = create_engine('sqlite:///score_' + domain +
'0test.db')
metadata1 = MetaData(db1)
self.scores = Table('scores', metadata1,
autoload=True)
client = MongoClient(connect=False,maxPoolSize=1)
18
Chapter 1 A Birds Eye View to AI System
self.db = client.frequency
self.gi = pygeoip.GeoIP('GeoIP.dat')
self.high = 1.2
self.low = .8
def get_hour(self,timestamp):
return dt.datetime.utcfromtimestamp(timestamp /
1e3).hour
def get_score(self, featurename, featurevalue):
pred = 0
s = self.scores.select((self.scores.c.feature_name
== featurename) & (self.scores.c.feature_value ==
featurevalue))
rs = s.execute()
row = rs.fetchone()
if row is not None:
pred = pred + float(row['score'])
res = self.db.frequency.find_one({"ip" : ip})
freq = 1
if res is not None:
freq = res['frequency']
pred2, prob2 = self.get_score('frequency',
str(freq))
return (pred1 + pred2), (prob1 + prob2)
conn = sqlite3.connect('multiplier.db')
cursor = conn.execute("SELECT high,low from
multiplier where domain='" + value + "'")
row = cursor.fetchone()
if row is not None:
self.high = row[0]
self.low = row[1]
return self.get_score(f, value)
19
Chapter 1 A Birds Eye View to AI System
20
Chapter 1 A Birds Eye View to AI System
for domain in f:
domain = domain.strip()
p = Predictor(domain)
url = '/predict/' + domain
api.add_route(url, p)
Having covered design patterns in Python a bit, let’s now take a look at
some essential architecture patterns for data scientists.
21
Chapter 1 A Birds Eye View to AI System
Summary
In this chapter, we discussed fundamental engineering principles for data
scientists, which are covered in separate chapters. The question-answering
example can help you understand how to organize your code. The basic
rule is to not put everything into one class. Divide your code into many
categories and use parent-child relationships where they exist. Then you
learned how to use Python to call other languages’ code. We provided two
instances of R and Java code calls. Then we showed you how to expose
your model as a REST API and make it perform well by using concurrent
programming. Following that, we covered significant architectural patterns
from data scientists.
22
CHAPTER 2
The previous processes are important in any data analytics work. Once
the data goes through the ETL processes, then it becomes possible to
analysis the data, find insights, and so on.
We will discuss various types of ETL throughout this chapter. We
discussed in Chapter 1 that data is not an isolated thing. We need to load
data from somewhere, which is a database. We need to fetch the data from
some application, which is extraction. In this chapter and the next, we will
discuss various feature engineering that transforms the data from one form
to another.
MySQL
MySQLdb is an API in Python developed to work on top of the MySQL C
interface.
#!/usr/bin/python
import MySQLdb
If you get an import error exception, that means the module was not
installed properly.
The following are the instructions to install the MySQL Python module:
$ gunzip MySQL-python-1.2.2.tar.gz
24
Chapter 2 ETL with Python
Database Connection
Before connecting to a MySQL database, make sure you do the following:
If you do not do these steps properly, you will get an exception in the
next Python code.
25
Chapter 2 ETL with Python
INSERT Operation
The following code carries out the SQL INSERT statement for the purpose
of creating a record in the STUDENT table:
#!/usr/bin/python
import MySQLdb
# Open database connection
db = MySQLdb.connect("localhost","user","passwd","TEST" )
# prepare a cursor object using cursor() method
cursor = db.cursor()
# Prepare SQL query to INSERT a record into the database.
sql = """INSERT INTO STUDENT(NAME,
SUR_NAME, ROLL_NO)
VALUES ('Sayan', 'Mukhopadhyay', 1)"""
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
# disconnect from server
db.close()
READ Operation
The following code fetches data from the STUDENT table and prints it:
#!/usr/bin/python
import MySQLdb
# Prepare SQL query to INSERT a record into the database.
26
Chapter 2 ETL with Python
DELETE Operation
The following code deletes a row from TEST with id=1:
#!/usr/bin/python
import MySQLdb
# Prepare SQL query to DELETE required records
sql = "DELETE FROM STUDENT WHERE ROLL_NO =1"
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
27
Chapter 2 ETL with Python
UPDATE Operation
The following code changes the lastname variable to Mukherjee, from
Mukhopadhyay:
#!/usr/bin/python
import MySQLdb
# Prepare SQL query to UPDATE required records
sql = "UPDATE STUDENT SET SUR_NAME="Mukherjee"
WHERE SUR_NAME="Mukhopadhyay"
try:
# Execute the SQL command
cursor.execute(sql)
# Commit your changes in the database
db.commit()
except:
# Rollback in case there is any error
db.rollback()
COMMIT Operation
The commit operation provides its assent to the database to finalize the
modifications, and after this operation, there is no way that this can be
reverted.
ROLL-BACK Operation
If you are not completely convinced about any of the modifications and
you want to reverse them, then you can apply the roll-back() method.
The following is a complete example of accessing MySQL data through
Python. It will give the complete description of the data stored in a CSV file
or MySQL database.
28
Chapter 2 ETL with Python
This code asks for the data source type, either MySQL or text. For
example, if MySQL asks for the IP address, credentials, and database name
and shows all tables in the database, it offers its fields once the table is
selected. Similarly, a text file asks for a path, and in the files it points to, all
the columns are shown to the user.
import MySQLdb
import sys
out = open('Config1.txt','w')
print ("Enter the Data Source Type:")
print( "1. MySql")
print ("2. Exit")
while(1):
data1 = sys.stdin.readline().strip()
if(int(data1) == 1):
out.write("source begin"+"\n"+"type=mysql\n")
29
Chapter 2 ETL with Python
30
Chapter 2 ETL with Python
Normal Forms
Database normal forms are the principles to organize your data in an
optimum way.
Every table in a database can be in one of the normal forms that we’ll
go over next. For the primary key (PK) and foreign key (FK), you want to
have as little repetition as possible. The rest of the information should be
taken from other tables.
31
Random documents with unrelated
content Scribd suggests to you:
towards our lines. When, however, the advanced guard had reached
the limit of their advance, the Battalion pushed along the bed of the
stony wadi, and were supposed to halt at the junction of the wadi in
front of Jebel Ektief and the main wadi, where they hoped to turn
the corner into their place of deployment. However, the noise had
apparently disturbed the Turk, and he constantly poured machine-
gun fire on to this junction corner. It was therefore found necessary
to find another way into the Jebel Ektief Wadi, and the only way was
to climb the steep, precipice-like sides of the main wadi on to the
high ground held by “D” Company. This was accomplished, but how,
no one can tell; when the Battalion with its Lewis guns, regimental
aid post, and signalling mules reached the crest, dawn was
appearing in the east. It was decided, therefore, to make a hurried
descent into the Jebel Ektief Wadi in spite of the casual fire from the
enemy’s snipers and machine guns. Luckily the going down into the
second wadi was easier and more speedily accomplished than the
climb from the first wadi, and just as daylight arrived the tail of the
Battalion disappeared into the bed of Jebel Ektief Wadi, out of sight
of the Turks. Once in the bottom of the wadi the companies sorted
themselves out and prepared for the actual assault, “A” and “B”
being the attacking companies, “C” in support, and “D” in reserve. It
had been arranged that the 2/13th (Kensingtons) were to assault
the position on our left, having approached the wadi at Jebel Ektief
by a night march on a parallel route to the one we had taken.
Unfortunately, the ground over which the Kensingtons had to pass
had proved too difficult, and when daylight came they found
themselves perched on a precipice unable to cross the deep ravine
before them.
At the same time “C” Company was taken from support, and by
means of quick concealed movement succeeded in moving to our
left flank, thereby taking up the position and rôle in the action which
should have been the Kensingtons’.
“C” Company was able to make some ground and also was able to
give great assistance to “A” and “B” Companies by bringing enfilade
fire to bear on the snipers and machine guns that were holding them
up.
As this second barrage lifted, the Battalion assaulted the steep hill
and in spite of heavy frontal fire were successful in capturing the
first line. By this time the two companies of the Queen’s
Westminsters had come up on to our left, and under the gallant
leadership of Captain Flower silenced the machine guns which had
enfiladed our attack. When the first ridge was gained it was found
that a valley beyond led to the second ridge, and here the Turks
were again strongly resisting. After a breather, however, the
“Charge” was given, and the men, led by Captain Wills and Company
Sergeant Major Oldcorn of “C” Company, rushed over the valley and
gained the second ridge only to find a still further ridge strongly
held. Machine-gun fire from the flank was still causing considerable
casualties to us, but after a breather, the Battalion assaulted the
third ridge, and thus completed the capture of Jebel Ektief. The
Turks fled down the precipitous slopes, into the Jordan Valley, and
could be seen collecting in small groups as though preparing for a
counter-attack. The Battalion’s casualties in this action numbered
about one hundred.
Further north of Jebel Ektief, the London Irish of the 180th Brigade
had succeeded in capturing Talat ed Dumm after heavy fighting. The
Turk, however, still had isolated machine guns on the hills between
the recently captured positions of Jebel Ektief and Talat ed Dumm,
and these poured a persistent fire into our flank. In the late
afternoon the Kensingtons came up on our left, and soon cleared the
ground of these irritating machine guns, and by the evening
everything was quiet except for an occasional shell from a Turkish
battery which could be seen some couple of miles away across the
Jordan Plain, well out of the range of our guns. Just before dark we
had a splendid view of the Jericho Plain.
The following morning, the 21st February, the London Scottish were
moved on to the plain as far as the white ruins of Neba Musa, and
Australian cavalry patrolled the Jericho area. The Turk had crossed
the Jordan, and all was quiet. About midday, however, four enemy
planes flew over our positions, and, but for a remarkable cloud
which suddenly enveloped the hill, would no doubt have poured
bullets into our bivouacs. When the cloud lifted they were nowhere
to be seen. For two days the Battalion remained on the heights of
Jebel Ektief, and although free from fighting it was a most strenuous
business carrying rations and water from the Quartermaster’s Stores
and Transport which were now in the Jebel Ektief Wadi, from which
the attack had started, some 600 or 700 feet below the summit of
the hill.
On the 23rd February, the Battalion was ordered to move as far back
as Jerusalem.
CHAPTER XXXVII
JERUSALEM—MUKMAS—JERICHO—ES SALT—WADI EL JIB
On the 23rd of February the Battalion left Jebel Ektief, and after
some discussion in the wadi near the transport lines, some
companies chose the route via the Wadi Hindi, Abu Dis and Bethany,
while others struck across country over a distinct track, supposed to
be the remains of a pilgrims’ road to the Jordan, and thence along
the main Jerusalem-Jericho Road, via Bethany. Which was the better
route has not been decided, sufficient to say that the troops hated
both. Rain poured down when we were half-way to Jerusalem, and
when Sir John Grey Hills’ house was reached late at night no one
had the heart to argue as to the better way from Jebel Ektief. On the
following day (Sunday) we availed ourselves of the rest. In the
evening many attended Divine Service in the huge chapel of the
German Emperor’s Palace, near to the Mount of Olives. It is a huge
structure, and a wonderful view of the surrounding country could be
obtained from the tower. It was used as a Corps Headquarters
during the operations around Jerusalem; not quite the purpose for
which the Kaiser had intended it. The chapel was wonderfully
decorated inside, but the two things which struck one as being odd,
and even profane, were the panelling of the altar, which was
decorated by the crests of each of the Kaiser’s sons; and the ceiling,
which was divided into two parts; one containing a beautiful painting
of the Son of God, while the other contained, equally well painted, a
picture of the Kaiser and his wife sitting on their thrones!
We bivouacked for the night in the Wadi Nimrin, near Umm Enkhala,
and at 5.0 a.m. the next day moved out in support of the London
Scottish and the Queen’s Westminsters, who attacked the hill of El
Haud which, held in force by the Turk, barred the second means of
approach to Es Salt—that by way of the Wadi Arsinyet.
The river Jordan at Makhadet Hajlah is 1,200 feet below sea level,
and El Haud is 846 feet above, and the weather changed, becoming
steadily worse. The night was wet and bitterly cold, and we were not
sorry to start at 7.0 a.m. the next day on our advance up the
Arsinyet track to Es Salt (Ramoth Gilead).
The remainder of the Division moved up the Annam road, and the
defence of Es Salt was left to the Civil Service Rifles. The same
evening the Battalion moved out to the north-east of the town to
Kefi Huda (3,597 feet), and took up a position covering the route
from the north to Es Salt.
The situation was nevertheless serious, and the 2/14th and 2/16th
Battalions were brought back to Es Salt.
The Battalion was relieved on the night of the 31st of March, and at
dawn moved down to the south side of Es Salt. The Divisional
withdrawal from Annam, the assault on which place had not been
entirely successful, was now in full swing, and the majority of the
Armenian population of Es Salt, terrified at the prospect of the return
of the terrible Turk, was hastily evacuating the town. The road,
which had been churned up by men, horses, guns, and wheeled
transport, was inches deep in liquid mud, and was packed with
refugees. Old men and boys, women and children of all ages, with
their household goods in large bundles on their backs, staggered
along obsessed with great fear. Their flight was terrible, and not a
few of us hurriedly realised that it is not only troops who suffer in
war time. We could give little assistance; our own baggage wagons
were already overburdened and our own loads were heavy, but we
did what we could.
At 8.0 a.m. on the 1st of April (Easter Monday), after standing in full
marching order in the mud for a solid hour, we started our
retirement, and though hindered and hampered by the crowd of
refugees which surged and swayed either side of us, we marched
steadily until 5.0 a.m. next day, when we emerged from the hills at
Shunet Nimrin, and bivouacked for a few hours, after which we
continued and, crossing the Jordan River by the Ghoraniyeh Ford,
we arrived at a position in the Wadi Nuemiah about 3 p.m., having
covered a distance of about thirty miles since 8.0 p.m. the previous
night.
Every one was footsore, dirty, unshaven, with seven days’ growth of
beard, and thoroughly tired, but we soon settled down to our first
proper night’s rest for nine days, during which time we had
experienced almost the extremes of heat and cold, and had marched
about seventy miles through some of the most atrocious country our
army has ever operated in. The other Brigades had fared even worse
than we had, but the whole Division had covered itself with honour,
and had no reason to be dispirited at the apparent failure of its novel
enterprise. Gains in warfare are not measured by territory alone, and
the influence of this raid—indeed it was almost a campaign—had a
far reaching effect on the strategy of the Palestine campaign.
It was not a habit in those days to keep us idle, and we were soon
moved on up the Jerusalem-Nablous road via Bireh and Ram Allah to
the Wadi el Jib, where we temporarily relieved the 10th (Irish)
Division. The Battalion was in reserve, and bivouacked on a terraced
hillside which was covered with fig trees and vines. It was very hot,
and we were glad to resume our summer kit. The camping ground
was one of the most pleasant we had experienced in Palestine, and
we made the most of it. There were flowers and plants in great
profusion, and botanists amongst us spent hours collecting
specimens; one of them, a botanist by profession, collected a
hundred or so specimens which he had never seen before, and
which he could not name.
Of actual war we saw but little, occasional bombing raids were made
by enemy planes, but our chief enemy was just behind us—a gunner
officer in charge of an 18-pounder battery, whose idea of humour
was to give an order for ten rounds gun fire regularly each night at
varying times between midnight and 4.0 a.m. Our considered
opinion of this gentleman cannot be published here.
On the 23rd of April, 1918, the Battalion marched south from Attara
and encamped for the night just north of Ram Allah, near Lake
Ballua. The march was carried out during the evening, and the
bright weather made it enjoyable. On the two following days the
journey was continued towards Jerusalem, and on the night of the
25th we halted at Shafat, a few miles north of the city. Rumours of
another “stunt” across the Jordan were then confirmed, and on the
night of the 27th the Battalion marched down the Jericho road once
more to Talat ed Dumm, arriving there about 11.0 p.m., where the
concentration of transport foreboded another operation on a large
scale. The following evening the Battalion moved to the Jordan
Valley, and crossed over the river at Ghoraniyeh Ford, which was
then held by Indian troops. Although it was only April the heat in the
valley was terrific during the day, and white troops would, no doubt,
have suffered considerable losses from disease had they been
stationed there for any considerable length of time. The next day
was spent amid the thick brushwood on the plain, and every
available piece of shade was used, but even under this protection
the damp heat was nigh unbearable, and only a minimum of clothing
in the shape of drill shorts were worn by most. There was a large
concentration of troops in the area, and preparations were made for
an attack on the foot-hills of the Moab Mountains.
On the 30th the attack was launched by the 179th Brigade, the
Westminsters on the left and the London Scottish on the right, while
the 180th Brigade was operating farther south. Under a heavy
artillery bombardment the troops advanced from their positions of
deployment, which they had taken up during the night. They
attacked the first ridge and, after heavy fighting, obtained a footing.
Many prisoners were captured and casualties were inflicted on the
enemy. The 2/15th were held in reserve at Ghoraniyeh Ford, and
when a crowd of over 200 prisoners appeared we felt that the
operation was so successful that we should not be needed; it
transpired afterwards, however, that these Turks were not captured
in the actual assault, but had been detailed to reconnoitre our
concentration on the plain, and while coming over had walked into
the London Scottish, who were pushing forward for the attack. They
made no resistance, and were soon sent to the rear as prisoners.
After the first ridge had been taken it was found that the whole
position was strongly held by the Turk, and that further advance
over deep ravines swept by machine guns would be very difficult.
Repeated attempts were made to push forward, but only heavy
casualties resulted. The actual objective of the 179th Brigade was a
steep hill called El Haud, which lay some six miles east of the
Jordan, and formed part of a continuous line of steep, rocky hills
that bordered the Jericho Plain on the east.
On the night of the 30th April the London Scottish had only been
able to gain the first ridge of El Haud after heavy fighting, and
during the following day held on to the captured ground with the
idea of pressing forward the attack at dawn next morning. In order
to support this second effort “A” and “B” Companies of the 2/15th
pushed forward to the foot of El Haud, to act as local supports. The
second attack on the 1st of May was also unsuccessful, not only at
this one particular spot but along the whole Divisional front. Heavy
machine-gun fire swept each ridge, both from the front and the
flanks, so cunningly had the Turk arranged his emplacements.
Throughout the day desultory fighting took place and small local
attacks were made, but no general advance could be accomplished.
During the evening of the 1st of May “C” and “D” Companies were
brought up from the Wadi Nimrin, where they had been moved the
previous night, and during the night of the 1st the Battalion relieved
the London Scottish on El Haud. Orders were received on the same
night to attack at dawn. This meant a busy night in reconnoitring.
“A” and “B” Companies were detailed for the attack, with “C”
Company in support, and just before dawn they deployed for the
attack on the right flank of the hill, from which point it appeared the
approach to the summit was easiest. “D” Company remained in
possession of the captured ridge.
The attack was launched forward with great spirit and dash, but was
met with the same murderous fire that had swept down the London
Scottish in their attempts, and the 2/15th could only push forward a
little way, gaining some 500 yards and a small hill, thus forming for
themselves an unpleasant little salient into the enemy’s lines. Even
then casualties were fairly heavy, and the work of carrying back the
wounded was made more hazardous by the Turk, who fired on every
stretcher with his machine guns throughout the day. Captain
Peatfield, “A” Company, was reported missing, but late at night he
returned under cover of darkness, having lain out all day in the sun,
a few yards from a Turkish machine gun, apparently dead.
At dusk the wounded were collected, and “A” and “B” were
withdrawn, “C” Company staying on the captured ground and hastily
organising it for defence by building sangars.
Later in the day the enemy aeroplanes returned, but this time
devoted their attentions to the transport lines and A.S.C. dumps near
Ghoraniyeh Ford. For the rest of the day the infantry could make no
progress, and spent their time watching the artillery bombardment
of those places where machine guns had been located. The Turk was
clever in concealing his emplacements, and it was only in the
evening when a slight breeze sprang up were we able to detect one
that had been firing at us throughout the whole day. This machine
gun was in a cave in the hill side, and the entrance to the cave had
been cleverly camouflaged with a blanket of the same colour as the
surrounding rocks, and not until the breeze caused the blanket to
move was it realised that the fire had been coming from an
apparently solid rock. However, attentions were soon upon this spot,
and one of our machine guns poured heavy fire into the cave,
tearing the blanket in shreds, and no doubt killing the enemy
gunners.
On the evening of the 4th May orders were received that the whole
of the British Force would be withdrawn to the Ghoraniyeh
Bridgehead as the Turks had sent a considerable force down the
Jordan Valley from the north, and which had been driven back by
our cavalry that afternoon. However, it was realised that should this
attack by the Turk succeed our left flank and our line of retreat
would both be seriously threatened. In the Battalion arrangements
were made for “D” Company to be left as a rear-guard, holding the
hill of El Haud until the rest of the Battalion had got a good start
across the plain. “D” Company kept up an occasional Lewis gun and
rifle fire for nearly an hour and then left the hill to the mercy of the
Turks. Every one was relieved to reach the wire entanglements of
the Ghoraniyeh defences the gaps in which were guarded by Indian
cavalry. The amusing part, however, was that when the Battalion
returned to its bivouac area of a few days previous they found “D”
Company there first. Apparently being a smaller force they were
more mobile and certainly they knew the shortest cut home. Once
inside the wire a few hours’ rest was granted, and at 3 a.m. the
withdrawal was resumed as far as Tel el Sultan, a ruined village near
Jericho. We arrived at our bivouac area early in the morning, and in
the glorious warmth of the day every one enjoyed a thorough rest
after the unsatisfactory hard fighting of the past few days. Before
leaving the subject of El Haud a word of praise is due to the
Divisional R.A.M.C. for the splendid manner in which they evacuated
the wounded during the fighting, and had erected a large field
hospital tent well within range of shell fire under the slopes of El
Haud.
On the 6th of May the Battalion had the surprise of its life, when
orders were received to pack up and move to Jericho, where motor
lorries were to take us as far back as Jerusalem. Never in its
existence had the Battalion partaken of such luxury, but it is
regretted that we all thought that this kindness to tired troops was
to move them to another part of the front where they were needed
in a hurry.
In the late hours of the afternoon the Battalion boarded the lorries
and the convoy started its journey to Jerusalem, along the new
Jericho road, which is a masterpiece in mountain roads, and was
built under German influence. About half-way home we met several
regiments of Indian cavalry proceeding in the direction of the Jordan
Valley, where they were to hold the line during the summer months.
Many of them wore the Mons Star ribbon, and apparently had seen
fighting in France. After passing these troops rain commenced to fall
and those who had greedily sought the seats beside the driver paid
the penalty and were soon drenched through. The lorries pulled up
just north of Jersualem on the Nablous road, from which point each
load of men marched independently to the camping ground near
Shafat, which we had occupied only a week previously. The writer
was detailed to supervise the unloading of the Battalion, and when
they had finally “debussed” he made his way to Shafat probably
about 45 minutes after the first party of the 2/15th had left their
lorry. In spite of the rain and the muddy ground bivouacs had been
erected and each one had a small candle burning inside. To see the
camp from the roadway it would have appeared to a casual observer
that it had been in existence for several weeks, so expert had the
Battalion become in erecting its temporary home. Finding his
bivouac the writer was soon presented with a plate of bread, bully
beef, pickles, cheese and a mug of tea by his batman.
The following day the Battalion moved north a few miles as far as
Ram Allah; the march was not long, the road was fair, and the
weather was glorious, and by 2 p.m. the Battalion had settled down
in its new area, and dinners were being cooked. I have already
referred to the speed with which the Battalion made themselves at
home on the night we arrived at Shafat, but as a further illustration
the following true story may be given. On this day’s march we were
followed by the London Scottish, who were in turn followed by an
officer of our own Battalion and a draft fresh from England who
were on their way to join us. Having missed the Battalion as it left
Shafat this new officer attempted to catch us up. We arrived at our
bivouac area and moved off the road to allow the Scottish to pass to
their camp which was farther north. We immediately put up our
bivouacs and commenced cooking. The new officer and his men
followed the Scottish for some two miles only to find that the
encampment he had passed some 25 minutes before was really that
of his own Battalion. It was hard to make him realise that we were
only just ahead of the Scottish on the march and turning off the
road had immediately set to and erected our camp.
Here news of a long rest reached us, and we were told that after the
next day’s march to Ain Arik we should have a complete rest. On the
way to Ain Arik we were inspected by the Commander-in-Chief,
General Allenby, and on the afternoon of the 9th arrived at our new
camp. The camp was situated on the rocky slopes of a deep wadi;
little ledges were however found for “bivvies,” and each company
made itself at home in its particular area. The slopes were covered
with many small trees, and it was undoubtedly one of the prettiest
spots in Palestine that we had visited. Here the Battalion remained
for 10 days, and everything was done to make the rest enjoyable. At
the top of the wadi near the main roadway a few level places were
found and football competitions, both inter-Battalion and inter-
company were arranged. A Brigade Sports Committee was set up
and a varied programme of games, sports and transport
competitions was arranged. The final placings were as follows:
2/15th Battalion, 10 points; 2/14th Battalion, 10 points; 2/16th
Battalion, 3 points; and 2/13th Battalion, 3 points. Great credit was
due to the excellent turn out of our transport section, and also our
Lewis gunners in their particular competitions. In the Brigade
football final the London Scottish beat the Queen’s Westminsters
after a hard game. The Divisional Concert Party arrived and gave
nightly performances in a large marquee. In the Battalion itself
company concerts were held, a piano being hired all the way from
Jerusalem through the personal efforts of Lieutenant Phelps. The
whist drive held by “B” Company must also be included in the
“mention in dispatches.” Shooting competitions were held at a small
range built at the bed of the wadi, and Battalion sports, limited to
the three-legged, sack, egg and spoon type of race, were held on
the flat bed of the wadi and proved a huge success. There was no
suitable ground for sprinting, and even the course for the comic
races had to be cleared of large stones by a voluntary fatigue party.
One competition of the afternoon which deserves special mention
was a “beauty competition for the best dressed lady.” The originality
of the aspirants was marvellous, and real harem dresses with the aid
of bacon wrappings, etc., were among the prize winners. Canteen
stores were plentiful and “dinner” parties were given throughout the
camp each night. A real happy time was spent here and the
Battalion was loath to leave Ain Arik when orders were issued on the
night of the 19th May.
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
ebooknice.com