Fr

In this book, you'd learn to use Elasticsearch, Kibana, and

Elasticsearch-Hadoop effectively to analyse and understand
your HDFS and streaming data.

What you will learn from this book

Set up the Elasticsearch-Hadoop environment

You begin with an in-depth understanding of the Hadoop,

Elasticsearch, Marvel, and Kibana setup. Right after this,
you will learn to successfully import Hadoop data into
Elasticsearch by writing MapReduce job with real-world
example. This is then followed by a comprehensive look at
Elasticsearch essentials, such as full-text search analysis,
queries, filters, and aggregations; after which you will look at
creating various visualizations and interactive dashboards
using Kibana. Classifying your real-world streaming data
and identifying trends in it using Storm and Elasticsearch
are some of the other topics that will be covered.
Towards the end, you will learn to integrate Elasticsearch
with other Hadoop ecosystem tools, such as Pig, Hive,
and Spark.

Import HDFS data into Elasticsearch with

MapReduce jobs
Perform full-text search and aggregations
efficiently using Elasticsearch

Elasticsearch for Hadoop

Elasticsearch for
Hadoop

Analysing trends on streaming data using

Storm and Elasticsearch
Inject and classify real-time streaming data
into Elasticsearch

Prices do not include

local sales tax or VAT
where applicable

Visit www.PacktPub.com for books, eBooks,

code, downloads, and PacktLib.

Vishal Shukla

Integrate Elasticsearch with the Hadoop

ecosystem tools such as Pig, Storm, Hive,
and Spark

$ 39.99 US
25.99 UK

P U B L I S H I N G

pl

C o m m u n i t y

Who this book is written for

community experience distilled

Sa
m

Visualize data and create interactive

dashboards using Kibana

Get your Elasticsearch-Hadoop projects ready

for production
This book is targeted at Java developers with basic
knowledge of Hadoop. No prior Elasticsearch experience
is required.

ee

E x p e r i e n c e

D i s t i l l e d

Elasticsearch for
Hadoop
Integrate Elasticsearch into Hadoop to effectively visualize and
analyze your data

Vishal Shukla

In this package, you will find:

The author biography

A preview chapter from the book, Chapter 2 'Getting Started with ES-Hadoop'
A synopsis of the books content
More information on Elasticsearch for Hadoop

About the Author

Vishal Shukla is the CEO of Brevitaz Systems (http://brevitaz.com) and a

technology evangelist at heart. He is a passionate software scientist and a big data

expert. Vishal has extensive experience in designing modular enterprise systems.
Since his college days (more than 11 years), Vishal has enjoyed coding in JVM-based
languages. He also embraces design thinking and sustainable software development.
He has vast experience in architecting enterprise systems in various domains. Vishal
is deeply interested in technologies related to big data engineering, analytics, and
machine learning.
He set up Brevitaz Systems. This company delivers massively scalable and
sustainable big data and analytics-based enterprise applications to their global
clientele. With varied expertise in big data technologies and architectural acumen,
the Brevitaz team successfully developed and re-engineered a number of legacy
systems to state-of-the-art scalable systems. Brevitaz has imbibed in its culture agile
practices, such as scrum, test-driven development, continuous integration, and
continuous delivery, to deliver high-quality products to its clients.
Vishal is a music and art lover. He loves to sing, play musical instruments, draw
portraits, and play sports, such as cricket, table tennis, and pool, in his free time.
You can contact Vishal at [email protected] and on LinkedIn at
https://in.linkedin.com/in/vishalshu. You can also follow Vishal on Twitter
at @vishal1shukla2.

Preface
The core components of Hadoop have been around from 2004-2006 as MapReduce.
Hadoop's ability to scale and process data in a distributed manner has resulted in its
broad acceptance across industries. Very large organizations are able to realize the
value that Hadoop brings in: crunching terabytes and petabytes of data, ingesting
social data, and utilizing commodity hardware to store huge volume of data.
However, big data solutions must fulfill its appetite for the speed, especially when you
query across unstructured data.
This book will introduce you to Elasticsearch, a powerful distributed search and
analytics engine, which can make sense of your massive data in real time. Its rich
querying capabilities can help in performing complex full-text search, geospatial
analysis, and detect anomalies in your data. Elasticsearch-Hadoop, also widely
known as ES-Hadoop, is a two-way connector between Elasticsearch and Hadoop.
It opens the doors to flow your data easily to and from the Hadoop ecosystem and
Elasticsearch. It can flow the streaming data from Apache Storm or Apache Spark to
Elasticsearch and let you analyze it in real time.
The aim of the book is to give you practical skills on how you can harness the
power of Elasticsearch and Hadoop. I will walk you through the step-by-step
process of how to discover your data and find interesting insights out of massive
amount of data. You will learn how to integrate Elasticsearch seamlessly with
Hadoop ecosystem tools, such as Pig, Hive, Cascading, Apache Storm, and Apache
Spark. This book will enable you to use Elasticsearch to build your own analytics
dashboard. It will also enable you to use powerful analytics and the visualization
platform, Kibana, to give different shapes, size, and colors to your data.
I have chosen interesting datasets to give you the real-world data exploration
experience. So, you can quickly use these tools and techniques to build your domainspecific solutions. I hope that reading this book turns out to be fun and a great
learning experience for you.

Preface

What this book covers

Chapter 1, Setting Up Environment, serves as a step-by-step guide to set up your
environment, including Java, Hadoop, Elasticsearch, and useful plugins. Test the
environment setup by running a WordCount job to import the results to Elasticsearch.
Chapter 2, Getting Started With ES-Hadoop, walks you through how the WordCount
job was developed. We will take a look at the real-world problem to solve it better by
introducing Elasticsearch to the Hadoop ecosystem.
Chapter 3, Understanding Elasticsearch, provides you detailed understanding on how
to use Elasticsearch for full text search and analytics. With practical examples, you
will learn indexing, search, and aggregation APIs.
Chapter 4, Visualizing Big Data Using Kibana, introduces Kibana with real-world
examples to show how you can give different shapes and colors to your big data. It
demonstrates how to discover your data and visualize it in dynamic dashboards.
Chapter 5, Real-Time Analytics, shows how Elasticsearch and Apache Storm can
solve your real-time analytics requirements by classifying and streaming tweets
data. We will see how to harness Elasticsearch to solve the data mining problem of
anomaly detection.
Chapter 6, ES-Hadoop In Production, explains how Elasticsearch and the ES-Hadoop
library works in distributed deployments and how to make the best out of it for
your needs. It provides you practical skills to tweak configurations for your specific
deployment needs. We will have a configuration checklist for production and skills
to administer the cluster.
Chapter 7, Integrating with the Hadoop Ecosystem, takes practical examples to show how
you can integrate Elasticsearch with Hadoop ecosystem technologies, such as Pig,
Hive, Cascading, and Apache Spark.
Appendix, Configurations, contains a list of configuration options for any of the
Hadoop ecosystem integrations.

Getting Started
with ES-Hadoop
Hadoop provides you with a batch-oriented distributed storage and a computing
engine. Elasticsearch is a full-text search engine with rich aggregation capabilities.
Getting the data from Hadoop to Elasticsearch can open doors to run some data
discovery tools to find out interesting patterns and perform full-text search or
geospatial analytics. ES-Hadoop is a library that bridges Hadoop with Elasticsearch.
The goal of this book is to get you up-and-running with ES-Hadoop and enable you
to solve real-world analytics problems.
Our goal in this chapter is to develop MapReduce jobs to write/read the data to/from
Elasticsearch. You probably already know how to write basic MapReduce jobs using
Hadoop that writes its output to HDFS. ES-Hadoop is a connector library that provides
a dedicated InputFormat and OutputFormat that you can use to read/write data
from/to Elasticsearch in Hadoop jobs. To take the first step in this direction, we will
start with how to set up Hadoop, Elasticsearch, and the related toolsets, which you will
use throughout the rest of the book.
We encourage you to try the examples in the book to speed up the learning process.
We will cover the following topics in this chapter:

Understanding the WordCount program

Going realnetwork monitoring data

Writing a network logs mapper job

Getting data from Elasticsearch to HDFS

[ 21 ]

Getting Started with ES-Hadoop

Understanding the WordCount program

In the previous chapter, we downloaded and ran our first Hadoop MapReduce job
that used the ES-Hadoop library. Let's get inside the WordCount job to understand
how it is developed.

Understanding Mapper
Here is how WordsMapper.java looks:
package com.packtpub.esh;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import java.io.IOException;
import java.util.StringTokenizer;
public class WordsMapper extends Mapper<Object, Text, Text,
IntWritable> {
private final static IntWritable one = new IntWritable(1);
public void map(Object key, Text value, Context context) throws
IOException, InterruptedException {
StringTokenizer itr = new StringTokenizer(value.toString());
while (itr.hasMoreTokens()) {
Text word = new Text();
word.set(itr.nextToken());
context.write(word, one);
}
}
}

To all the MapReduce developers, this Mapper is very well known and trivial. We
will get the input line in the value and tokenize it with white space to extract word
tokens. For each word, we will then write a count of 1 with the word as a key. This is
no different than other WordsMapper you may have come across so far.

[ 22 ]

Chapter 2

Understanding the reducer

To help you understand the reducer better, let's understand the input that is
provided to it. We also have to understand what it is supposed to populate in
the context.
We are pretty familiar with the Mapper class used for the WordCount program. The
reducer is provided with all the buckets that contains the same word as a key; again
this is no different than the conventional MapReduce program. When the job gets
executed, we want the Elasticsearch index to be populated with the words and its
respective counts.
We have dedicated the next chapter solely to understand
Elasticsearch in depth. What is important for you to know as
of now is that you can index and read data from Elasticsearch.
The unit of these operations is called as Document. This is just
a JSON object that can have fields and values.

We want the Elasticsearch document to look similar to the following JSON structure:
{
"word":"Elasticsearch-hadoop",
"count":14
}

We need to represent this document in Java. This should be written to the Context
object. We can easily map the key/value-based JSON document format to the
MapWritable class. The keys of MapWriteable entries will represent the JSON field
key, and the values will represent the respective JSON field value. In the JSON
example we saw earlier, word and count becomes the keys of MapWritable.
Once the reducer is run and the context output is generated, it is transformed into
the JSON format and passed to the RESTful endpoint of Elasticsearch for indexing as
the bulk indexing request.
Let's take a look at how the WordsReducer class looks in the WordCount job:
package com.packtpub.esh;
import
import
import
import

org.apache.hadoop.io.IntWritable;
org.apache.hadoop.io.MapWritable;
org.apache.hadoop.io.Text;
org.apache.hadoop.mapreduce.Reducer;

import java.io.IOException;

[ 23 ]

Getting Started with ES-Hadoop

public class WordsReducer extends Reducer<Text,IntWritable,Text,MapWr
itable> {
@Override
public void reduce(Text key, Iterable<IntWritable> values,
Context context) throws IOException, InterruptedException {
// This represents our ES document
MapWritable result = new MapWritable();
int sum = 0;
for (IntWritable val : values) {
sum += val.get();
}
// Add "word" field to ES document
result.put(new Text("word"), key);
// Add "count" field to ES document
result.put(new Text("count"), new IntWritable(sum));
context.write(key, result);
}
}

WordsReducer implements the Reducer interface with the input key value types as
<Text, Iterable<IntWritable>> and the output types as <Text, MapWritable>.
To implement the reduce method, we iterated through all the values for key to
derive the final sum value. The final sum value and key are added in the newly
constructed MapWritable object. Finally, the result is written to the context for

further processing.

Understanding the driver

We need one final component to drive the job. Let's call this component as Driver.
This component is responsible for providing job configurations, such as which
Elasticsearch nodes and index to point to. It also performs the plumbing work for the
job, such as which Mapper and Reducer class should be used and what their input
and output specifications are, including formats and paths.
Here is the Driver.java file:
import
import
import
import
import
import
import

org.apache.hadoop.conf.Configuration;
org.apache.hadoop.fs.Path;
org.apache.hadoop.io.IntWritable;
org.apache.hadoop.io.Text;
org.apache.hadoop.mapreduce.Job;
org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
org.elasticsearch.hadoop.mr.EsOutputFormat;

[ 24 ]

Chapter 2
public class Driver {
public static void main(String[] args) throws Exception {
Configuration conf = new Configuration();
// Elasticsearch Server nodes to point to
conf.set("es.nodes", "localhost:9200");
// Elasticsearch index and type name in {indexName}/{typeName}
format
conf.set("es.resource", "eshadoop/wordcount");

First, we created a job instance by providing the Hadoop Configuration object.

Many of the ES-Hadoop library behaviour can be customized as per your needs by
setting the relevant properties in this Configuration object. For your reference,
these configurations are listed in Appendix, Configurations. The Configuration
object in the preceding example is set with a couple of the most basic configurations:
es.nodes and es.resource.
The es.nodes configuration tells the job about the ES nodes that it should connect
to. The value of this setting accepts the <HOSTNAME/IP>:<PORT> format. You can
pass multiple comma-separated values for this setting. The ES-Hadoop library uses
an HTTP REST endpoint to connect with Elasticsearch. It means that we need to use
the HTTP port (by default, 9200) for the connection and not the transport port (by
default, 9300).
It is not required to specify all the cluster nodes in the
Elasticsearch cluster in the es.nodes setting. The ES-Hadoop
library will discover other nodes in the cluster and use
them. However, it's better to provide multiple nodes in this
configuration to ensure that even if some of the Elasticsearch
nodes are down, your job has at least one or a few nodes
running to be able to connect to the cluster.

The es.resource setting specifies the target Elasticsearch index and type. This
means that the data will be read or written to the index and type specified here.
This configuration takes the <index>/<type> format.

[ 25 ]

Getting Started with ES-Hadoop

If you are new to Elasticsearch, you will want to know what

index and type means. You can think of index as a database
schema and type as a table in a relational database. The
Elasticsearch document can be thought of as a row in the
relational table. We will cover Elasticsearch's index, type,
and document in greater detail in the next chapter.
// Create Job instance
Job job = new Job(conf, "word count");
// set Driver class
job.setJarByClass(Driver.class);
job.setMapperClass(WordsMapper.class);
job.setReducerClass(WordsReducer.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
// set OutputFormat to EsOutputFormat provided by
Elasticsearch-Hadoop jar
job.setOutputFormatClass(EsOutputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}
}

Once the job is created with the required configurations, other job properties are
set as typical MapReduce jobs, such as jarByClass, mapperClass, reducerClass,
outputKeyClass, and outputValueClass.
The ES-Hadoop library provides a dedicated EsInputFormat and EsOutputFormat.
These act as an adapter between the Hadoop and JSON document format expected
by Elasticsearch. By default, EsOutputFormat expects the MapWritable object to
be written as the output in the context object. EsOutputFormat then converts this
MapWritable object to the JSON object.
You can write the existing JSON String to Elasticsearch with
EsOutputFormat without applying any kind of transformation.
In order to do this, set the "es.input.json"="yes" parameter.
Setting this configuration makes the ES-Hadoop library look for
the BytesWritable or Text object to be provided as the job's
output. If none of these are found, it expects the class configured
as outputValueClass to provide a toString() method that
returns the JSON String.

[ 26 ]

Chapter 2

Using the old API org.apache.hadoop.mapred

If you are still using the old MapReduce API, the ES-Hadoop library fully supports
the old Hadoop version as well.
Here is a code snippet showing how the Driver program with the old MapReduce
API would look:
JobConf conf = new JobConf();
conf.setSpeculativeExecution(false);
conf.set("es.nodes", "localhost:9200");
conf.set("es.resource", "eshadoop/wordcount");
conf.setOutputFormat(EsOutputFormat.class);
conf.setMapOutputValueClass(MapWritable.class);
conf.setMapperClass(MyMapper.class);
...
JobClient.runJob(conf);

Going real network monitoring data

Now that we have an idea of how to write a job that does some processing with
MapReduce and pushes the data from HDFS to Elasticsearch, let's try out a
real-world example to get the feel of what value we can get by performing this the
ES-Hadoop way.
For illustration purposes, let's consider an example dataset of log files from
a hypothetical network security and a monitoring tool. This tool acts as a
gateway-cum-firewall between the devices connected in the network and the
Internet. The firewall detects viruses or spyware, checks the category of the outgoing
traffic, and blocks or allows the request based on the configured policies.

Getting and understanding the data

You can download the sample data generated by the tool at https://github.com/
vishalbrevitaz/eshadoop/tree/master/ch02. Here is a snippet of the data for
you to take a quick look:

Jan 01 12:26:26 src="10.1.1.89:0" dst="74.125.130.95" id="None"

act="ALLOW" msg="fonts.googleapis.com/css?family=Cabin
Condensed:400,600" (sn=5QMS-CW98-F7Z5-821D, ip=192.168.1.4,
tz=Asia/Kolkata, time=Sat Jan 1 12:26:26 2000)

[ 27 ]

Getting Started with ES-Hadoop

Dec 27 17:51:42 src="10.0.230.100:0" dst="216.163.176.37"
id="InformationTechnology" act="BLOCK"
msg="webres1.qheal.ctmail.com/SpamResolverNG/SpamResolverNG.dll?Do
NewRequest" (sn=S6RY-53CJ-MI1A-A3H1, ip=10.0.230.1,
tz=Asia/Kolkata, time=Sat Dec 27 17:51:42 2014)
Jan 01 14:05:02 src="192.168.23.7:0" dst="74.125.169.50"
id="None" act="ALLOW" msg="r13---snh557snee.c.pack.google.com/crx/blobs/QwAAAHF3InbmKwFIemaY3I3BCN9_1dkhEVaCdgM222vuHjkxK6NBzB_0gL_ZX0viLGk2Oj9RTenoQVq
Ft4t1aRX1UBZFohm5P53pZvgpk0MhUAAMZSmuUMiazmP7QUhdw2GOvyfeovoIJpSQ/extension_1_4_6_
703.crx?cms_redirect=yes&expire=1419697302&ip=117.222.178.31&ipbit
s=0&mm=31&ms=au&mt=1419682885&mv=m&sparams=expire,ip,ipbits,mm,ms,
mv&signature=61E0037BE24C1E66928B6CF9410E3A8F6F845F19.0F (sn=AXN9Z07K-25HR-K712, ip=192.168.1.30, tz=Asia/Kolkata, time=Sat Jan 1
14:05:02 2000)
Dec 27 17:51:42 src="192.168.0.2:0" dst="23.58.43.27"
id="InformationTechnology" act="BLOCK" msg="gtsslocsp.geotrust.com/" (sn=RH9P-5Y51-S4N0-M9JK, ip=192.168.0.162,
tz=Asia/Kolkata, time=Sat Dec 27 17:51:42 2014)
Dec 27 12:21:42 src="10.1.1.5:0" dst="212.56.73.9" id="None"
act="ALLOW" msg="\220L4\001" (sn=VCNY-RD87-A6LT-IXTT, ip=, tz=,
time=Sat Dec 27 12:21:42 2014)
Dec 27 17:51:42 src="10.3.15.7:0" dst="103.243.111.210"
id="InformationTechnology" act="BLOCK"
msg="resolver1.qheal.ctmail.com/SpamResolverNG/SpamResolverNG.dll?
DoNewRequest" (sn=K6KR-U5ST-SQ7R-QV9S, ip=10.3.15.1,
tz=Asia/Kolkata, time=Sat Dec 27 17:51:42 2014)

We can see that the data contains a few useful pieces of information, such as the
date and the source IP address, from which the request originated the destination
IP address. id represents the category of the website, followed by act and msg that
represents the action taken for the request, and the target URL of the request.

Knowing the problems

We have piles of logs being generated every second, and we would like to have an
insight into the following aspects:

Which are the top browsing categories?

Which are the top domains?
Which are the top blocked domains?
Which are the top categories surfed by certain IP ranges?
[ 28 ]

Chapter 2

Which IP addresses have the most blocked requests?

Which are the top domains for certain IP ranges?
Who are the top targets of the attacks?
Who are the top attackers?
Which are the top viruses?

There can be many more such problems you may want to get answered out of
these log files.

Solution approaches
Now, we have got the data and the problems that can be solved by performing
some aggregations and analysis. We also know how to write MapReduce jobs with
Hadoop. Most of the preceding problems are mainly centered around getting the top
results for some specific fieldsfor example, finding the top categories, top domains,
or top attackers.
At a very high level, there are two possible approaches to performing Top N queries:

Preaggregate the results: This is aware of the question beforehand, writes

a job to calculate Top N for a specific field of your interest, and stores the
aggregated result in a file that can be queried later.

Aggregate the results at query-time: This stores the granular data in a

rich analytics engine (such as Elasticsearch) and asks the Top N questions
on-the-fly as it comes to your mind. This performs query-time aggregation
to find the Top N results.

Approach 1 Preaggregate the results

In this approach, we need to know the question in advance. For example, which are
the top 10 categories? Based on the question/s, we will write our MapReduce jobs to
calculate the results that will be stored in the HDFS file.
One might think that the Top N operation can be simply combined to the logic of the
reducer that we developed for the WordCount job. This may work fine if you use a
single reducer. However, a single reducer would not scale well for the large datasets.
This approach would not work with multiple reducers because each reducer would
be working only on a subset of the data.
In order to derive the required results, we need to perform the following two steps:
1. Calculate the count of requests for each browsing category.
2. Sort the results and find the Top N categories.
[ 29 ]

Getting Started with ES-Hadoop

To implement the use case with the MapReduce paradigm, both of these steps will
need a dedicated MapReduce job. This is required because we will perform two
separate aggregation operations when we calculate Top Nthe count occurrences
for each categoryand then find the Top N categories.
First, the MapReduce job takes the raw log files as the input and produces the results
that look similar to our WordCount job. For example, the job to count the categories
may produce an output similar to the following code:
advertisements
500
informationtechnology
newsandmedia
304
none
12316
portals
2945
searchengines
1323
spywaresandp2p 1175

8028

This output can be written to the temporary file in HDFS, is provided as an input to
the second MapReduce job that calculates Top N, and writes the result to the HDFS
file. By performing this, you will obtain the Top N results of one field, which in this
case is category. Great! Similarly, counter jobs have to be created for each field that
you are interested in, such as attacker, IP range, blocked domains, viruses, and so on.
That's not so cool!
The following diagram shows how the solution would look with this approach to
obtain the answers for all the preceding Top N questions:

[ 30 ]

Chapter 2

Approach 2 Aggregate the results at query-time

The same set of questions can be answered very easily if we get the capability
of performing the calculation at query-time. Elasticsearch does exactly that by
providing the query-time aggregation capabilities on the indexed data. All we need
to do is get the granular data indexed in Elasticsearch. Once we have done this, we
can ask questions as they come to our mind, which you may not have thought of
earlier. This makes our life so much easier when it comes to exploring the data.
In order to implement this approach, we will write a single MapReduce job that
parses raw data from the HDFS file and uses the ES-Hadoop library to index the
granular data in the Elasticsearch index.
We can leverage Elasticsearch's querying capabilities to obtain answers to our Top N
questions in a fraction of a second. Elasticsearch will perform these aggregations on
its distributed nodes in the cluster at runtime when we query the data. This gives us
the ability to discover the data and find interesting insights from it.
The following diagram shows the solution approach with Elasticsearch to perform
query-time aggregation:

[ 31 ]

Getting Started with ES-Hadoop

We will gain a lot with Elasticsearch when it comes to

performing discovery on data or performing exploratory
analysis on data; there is a trade-off against storage and
computation power. Depending on the analysis you use for
your full-text search needs and the replication factor you select,
you need additional storage capacity. Similarly, you need a
higher amount of memory in order to compute aggregations at
query-time with high performance. We will touch these topics
in detail in the later part of the book.

Let's see how we can write the NetworkLogsMapper job to get the network logs data
to Elasticsearch.

Writing the NetworkLogsMapper job

Now that we know the approach to getting answers to our problems elegantly, let's
go ahead and write the mapper that we discussed in the previous section.
Here is the relevant part of the NetworkLogsMapper class to map our network logs to
Elasticsearch.

Writing the mapper class

Let's take a look at the mapper class now:
public class NetworkLogsMapper extends Mapper<Object, Text, Text,
MapWritable> {
public void map(Object key, Text value, Context context)
throws IOException, InterruptedException {
MapWritable map = new MapWritable();
String line = value.toString().trim();
String[] parts = line.split(" \\(");
String keyVals = parts[0].substring(15,
parts[0].length()).trim();

The preceding code reads the input log line and divides it into two easily parsable
segments that contain the key/value pairs, parts and keyVals:
int i = 0;
StringTokenizer part1tokenizer = new
StringTokenizer(keyVals);
while (part1tokenizer.hasMoreTokens()) {
String token = part1tokenizer.nextToken();
String keyPart = getKeyValue(token)[0];
[ 32 ]

Chapter 2
String valuePart = getKeyValue(token)[1];
switch (keyPart) {
case "src":
srcIp = valuePart;
break;
case "dst":
destIp = valuePart;
break;
case "id":
category = valuePart;
break;
case "act":
action = valuePart != null ?
valuePart.toUpperCase() : null;
break;
case "msg":
target = valuePart;
break;
}
i++;
}

The preceding code iterates through each of the key/value pair token of keyVals to
extract the sourceIp, destIp, category, action, and msg fields.
i = 0;
if (parts.length > 1) {
StringTokenizer part2Tokenizer = new
StringTokenizer(parts[1], ",");
while (part2Tokenizer.hasMoreTokens()) {
String token = part2Tokenizer.nextToken();
String keyPart = getKeyValue(token)[0];
String valuePart = getKeyValue(token)[1];
switch (keyPart) {
case "sn":
serial = valuePart;
break;
case "ip":
ip = valuePart;
break;
case "tz":
timezone = valuePart;
break;
case "time":
String timeStr = valuePart;
[ 33 ]

Getting Started with ES-Hadoop

timeStr = timeStr.replaceAll("\\)", "");
SimpleDateFormat dateFormat = new
SimpleDateFormat("EEE MMM dd hh:mm:ss YYYY");
try {
time =
dateFormat.parse(timeStr).getTime();
} catch (ParseException e) {
e.printStackTrace();
}
break;
}
i++;
}

Here is the parse parts segment to extract the serial, ip, timezone, and
time fields:
map.put(new Text("srcIp"), getWritableValue(srcIp));
map.put(new Text("destIp"), getWritableValue(destIp));
map.put(new Text("action"), getWritableValue(action));
map.put(new Text("category"), getWritableValue(category));
map.put(new Text("target"), getWritableValue(target));
map.put(new Text("serial"), getWritableValue(serial));
map.put(new Text("timezone"), getWritableValue(timezone));
map.put(new Text("ip"), getWritableValue(ip));
map.put(new
Text("domain"),getWritableValue(getDomainName(target)));
map.put(new Text("@timestamp"), time != null ? new
LongWritable(time) : new LongWritable(new Date().getTime()));
context.write(value, map);

Once we have parsed all the required fields from the log line, we can put all these
values into the MapWritable object. Note that MapWritable needs to have all its keys
and values of the Writable type. The preceding code gets the Writable versions of
each field and writes them to the MapWritable object. Finally, the map is written in
the context object, as shown in the following code:
private static WritableComparable getWritableValue(String
value) {
return value != null ? new Text(value) :
NullWritable.get();
}

[ 34 ]

Chapter 2

The getWritableValue() method takes the String parameter and returns a nullsafe Writable value as follows:
public static String getDomainName(String url) {
if(url==null)
return null;
return DomainUtil.getBaseDomain(url);
}

The getDomainName() method extracts the base domain from the URL that is
provided as a parameter. The indexing domain name can help in knowing the top
domains and their related analysis.
Overall, in the preceding program, we will get the input network logs in the
unstructured format. We just developed a simple program that allows you
to parse these log files and tries to extract the relevant information for our
Elasticsearch documents.
We will also perform the required data type transformations that are expected by
Elasticsearch. We parsed the timestamp from the logs from the end of the input line
and converted it to the LongWritable object.
You may have seen that we never told Elasticsearch about
the fields we will index and the data type of the fields. This is
handled by the automatic type mapping support provided by
Elasticsearch. This automatic type mapping triggers when the
first document is indexed. Elasticsearch automatically detects
and creates the data types for the fields based on the value
that is passed to the object.

To understand how Elasticsearch treats different data types passed from Hadoop,
consider the following table as a reference:
Hadoop Class

Elasticsearch Type

NullWritable, null
BooleanWritable

This class specifies null

Text, MD5Writable, UTF8

ByteWritable

This class specifies string

IntWritable, VInt

This class indicates int

LongWritable, VLongWritable
FloatWritable

This class specifies long

ShortWritable

This class indicates short

This class denotes Boolean

This class denotes byte

[ 35 ]

This class denotes float

Getting Started with ES-Hadoop

Hadoop Class
ArrayWritable

This class specifies array

Elasticsearch Type

AbstractMapWritable

This class denotes object

Text (in ISO8601 dateTime format)

This class indicates date

Writing Driver
The Driver class looks mostly the same as the one in the WordCount job.
The Driver class snippet looks similar to the following code:
// Create configuration object for the jobConfiguration conf = new
Configuration();
// Elasticsearch Server nodes to point to
conf.set("es.nodes", "localhost:9200");
// Elasticsearch index and type name in {indexName}/{typeName}
format
conf.set("es.resource", "esh_network/network_logs_{action}");

We have set the es.resource value to esh_network/network_logs_{action}.

This uses the dynamic or multiresource write feature provided by ES-Hadoop. This
dynamically creates a new index for each new value found for the action field, as
shown in the following code:
// Create Job instance with conf
Job job = new Job(conf, "network monitor mapper");
// set Driver class
job.setJarByClass(Driver.class);
job.setMapperClass(NetworkLogsMapper.class);
// set OutputFormat to EsOutputFormat provided by ElasticsearchHadoop jar
job.setOutputFormatClass(EsOutputFormat.class);
job.setNumReduceTasks(0);
FileInputFormat.addInputPath(job, new Path(args[0]));
System.exit(job.waitForCompletion(true) ? 0 : 1);

Now, let's create the Job instance with the conf object. We no longer need to set
outputKeyClass and outputValueClass. As expected, we didn't declare any
reducer. Also, the number of reducer tasks must be set to 0.

[ 36 ]

Chapter 2

You may have use casessuch as network logswhere you get millions of logs
indexed on a daily basis. In such cases, it may be desirable to have the data split
into several indices. Elasticsearch supports the SimpleDateFormat date patterns
to be provided as a template for the index name or the type name, as shown in the
following code:
es.resource =
esh_network_{@timestamp:YYYY.MM.dd}/network_logs_{action}

This creates a new index whenever a new date is encountered in the document
being indexed.

Building the job

In order to run the Hadoop job JAR file, we need to have the required dependent
JAR files in the Hadoop classpath. We will build a fat JAR file by embedding all the
required dependencies in it. For this job, these dependencies include hadoop-core,
hadoop-hdfs, and elasticsearch-hadoop.
Here is the code snippet from the pom.xml file:
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-core</artifactId>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-hdfs</artifactId>
<version>2.6.0</version>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch-hadoop</artifactId>
<version>2.1.0</version>
<exclusions>
<exclusion>
<artifactId>cascading-hadoop</artifactId>
<groupId>cascading</groupId>
</exclusion>
<exclusion>
<artifactId>cascading-local</artifactId>
<groupId>cascading</groupId>
</exclusion>
[ 37 ]

Getting Started with ES-Hadoop

</exclusions>
</dependency>
</dependencies>

The <dependencies> section declares the required hadoop and elasticsearchhadoop dependencies, as shown in the following code:
<plugin>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.2.1</version>
<executions>
<execution>
<id>make-network-logs-job</id>
<configuration>
<descriptors>
<descriptor>assembly.xml</descriptor>
</descriptors>
<archive>
<manifest>
<mainClass>com.packtpub.esh.nwlogs.
Driver</mainClass>
</manifest>
</archive>
<finalName>${artifactId}-${version}nwlogs</finalName>
</configuration>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>

The preceding code snippet shows the plugin declaration for maven-assembly-plugin.
This plugin provides a way to customize how the JAR file is assembled. The highlighted
text declares the Driver class as the main class for the JAR file. This assembly is
explained in the configured assembly.xml descriptor as follows:
<assembly>
<id>job</id>
<formats>
<format>jar</format>

[ 38 ]

Chapter 2
</formats>
<includeBaseDirectory>false</includeBaseDirectory>
<dependencySets>
<dependencySet>
<unpack>false</unpack>
<scope>runtime</scope>
<outputDirectory>lib</outputDirectory>
<excludes>
<exclude>${groupId}:${artifactId}</exclude>
</excludes>
</dependencySet>
<dependencySet>
<unpack>true</unpack>
<includes>
<include>${groupId}:${artifactId}</include>
</includes>
</dependencySet>
</dependencySets>
</assembly>

The assembly.xml descriptor instructs Maven that the third-party dependency JARs
should be assembled in the JAR under the lib directory.
Alternatively, you can also specify the JAR files to be added in the
classpath by setting HADOOP_CLASSPATH environment variable as shown
below, as shown in the following code:
| HADOOP_CLASSPATH="<colon-separatedpaths-to-custom-jars-including-elasticsearch-hadoop>"

These JARs can also be made available at runtime from the command line
as follows:
| $ hadoop jar your-jar.jar -libjars elasticsearch-hadoop.
jar

Once you have your pom.xml and assembly.xml configured, you are ready to build
the job JAR file. Switch to the parent directory of the pom.xml file and execute the
following command:
$ mvn package

This step assumes that you have the Maven binaries configured in the $PATH
environment variable. This command generates the required JAR file with
the ch02-0.0.1-nwlogs-job.jar name under the target folder of your
project directory.
[ 39 ]

Getting Started with ES-Hadoop

Getting the data into HDFS

Now that we have the data file and our job JAR file ready, let's import the data file to
HDFS with the following commands:
$ hadoop fs -mkdir /input/ch02
$ hadoop fs -put data/network-logs.txt /input/ch02/network-logs.txt

Running the job

Let' s execute the job with the network-logs.txt file using the following command:
$ hadoop jar target/ch02-0.0.1-nwlogs-job.jar /input/ch02/network-logs.
txt

The ES-Hadoop metrics of the output execution looks similar to the following code
(this shows that a total of 31,583 documents were sent from ES-Hadoop and accepted
by Elasticsearch):
Elasticsearch Hadoop Counters
Bulk Retries=0
Bulk Retries Total Time(ms)=0
Bulk Total=32
Bulk Total Time(ms)=3420
Bytes Accepted=10015094
Bytes Received=128000
Bytes Retried=0
Bytes Sent=10015094
Documents Accepted=31583
Documents Received=0
Documents Retried=0
Documents Sent=31583
Network Retries=0
Network Total Time(ms)=3515
Node Retries=0
Scroll Total=0
Scroll Total Time(ms)=0

[ 40 ]

Chapter 2

The following screenshot shows the Elasticsearch Head screen with the
indexed document:

We can see that the job execution generated two types in the index: network_logs_
ALLOW and network_logs_BLOCK. Here, ALLOW and DENY refers to the values of the
action field used in the multiresource naming.

Viewing the Top N results

Now that we have got our granular data indexed in Elasticsearch, let's see how we
can get answers to our Top N queries.
In order to execute Elasticsearch queries, you may either use the Sense interface or
the cURL command-line utility to send REST requests to Elasticsearch. If you
wish to use cURL, verify that you have cURL installed by executing the following
cURL command:
$ curl version

[ 41 ]

Getting Started with ES-Hadoop

To view the top five categories across all the network logs that we have indexed,
execute the following query:
$ curl -XPOST http://localhost:9200/esh_network/_search?pretty -d '{
"aggs": {
"top-categories": {
"terms": {
"field": "category",
"size": 5
}
}
},
"size": 0
}'

Executing the preceding command should show the following response:

{
"took" : 14,
"timed_out" : false,
..
..
},
"aggregations" : {
"top-categories" : {
"doc_count_error_upper_bound" : 26,
"sum_other_doc_count" : 4036,
"buckets" : [ {
"key" : "none",
"doc_count" : 12316
}, {
"key" : "informationtechnology",
"doc_count" : 8028
}, {
"key" : "portals",
"doc_count" : 2945
}, {
"key" : "global_wl",
"doc_count" : 2935
}, {
[ 42 ]

Chapter 2
"key" : "searchengines",
"doc_count" : 1323
} ]
}
}
}

We can just focus on the aggregations part of the result as of now. You will learn
about the Elasticsearch query and its response structures in detail in the next chapter.

Getting data from Elasticsearch to HDFS

So far, you learned how the ES-Hadoop library could help you in getting the data
from HDFS to the Elasticsearch index. There can be use cases when you have already
got your data in Elasticsearch and want to select some specific subset from this data
for complex analysis. This subset can be constrained by some full-text search criteria
as well.

Understanding the Twitter dataset

Twitter provides the REST API to access the Twitter data. Out of the wide range
of data provided by the Twitter API, we will just focus on the tweets data with the
#elasticsearch and #kibana hashtags. The dataset has been dumped in the CSV file in
the following format:
"602491467697881088","RT @keskival: We won #IndustryHack
@CybercomFinland #Elasticsearch #Logstash #Kibana #MarkovChain
#AnomalyDetection https://t.co/Iwes6VVSqk","Sun May 24 20:38:54
IST 2015","Cybercom Finland","<a
href=""http://twitter.com/download/iphone""
rel=""nofollow"">Twitter for iPhone</a>"
"602558729087758336","RT @keskival: We won #IndustryHack
@CybercomFinland #Elasticsearch #Logstash #Kibana #MarkovChain
#AnomalyDetection https://t.co/Iwes6VVSqk","Mon May 25 01:06:10
IST 2015","Ilkka Tengvall","<a
href=""https://about.twitter.com/products/tweetdeck""
rel=""nofollow"">TweetDeck</a>"
"602588512811089920","RT @segnior_david: @doxchile, we are
implementing #RealTimeProgramming @elastic is a pretty awesome
tool, #kibana #kopf #erlang #riak http:","Mon May 25 03:04:31 IST
2015","Leonardo Menezes","<a
href=""http://twitter.com/download/iphone""
rel=""nofollow"">Twitter for iPhone</a>"

[ 43 ]

Getting Started with ES-Hadoop

"602799066527997952","5 Great #Reasons to #Upgrade to #Kibana 4 Logz.io - http://t.co/N65yWeOYEg #log","Mon May 25 17:01:11 IST
2015","Frag","<a href=""http://getfalcon.pro""
rel=""nofollow"">The Real Falcon Pro</a>"
"602821180467322880","RT @elastic: Kurrently in #Kibana: We're on
the road to 4.1 release. PAUSE for a moment to see new features
http://t.co/ZeTvtkxnBy http://t","Mon May 25 18:29:03 IST
2015","Luiz H. Z. Santana","<a href=""http://twitter.com""
rel=""nofollow"">Twitter Web Client</a>"
"602825146878009344","Having fun playing around with geo ip
#elasticsearch #logstash #kibana","Mon May 25 18:44:49 IST
2015","Alex Glser","<a href=""http://twitter.com""
rel=""nofollow"">Twitter Web Client</a>"
"602825343469232128","RT @APGlaeser: Having fun playing around
with geo ip #elasticsearch #logstash #kibana","Mon May 25 18:45:36
IST 2015","search-tech-bot","<a href=""https://roundteam.co""
rel=""nofollow"">RoundTeam</a>"

The data represents the fields, such as id, text, timestamp, user, source, and so on,
in the CSV format.

Trying it yourself
Now, let's implement what you have learned so far with the Twitter dataset. Try
and load the preceding Twitter dataset into Elasticsearch using the MapReduce job.
This is similar to what we did for the WordCount and NetworkLogsMapper job in
the last section.
The Elasticsearch document should look similar to the following code:
{
"text": "RT @elastic: .@PrediktoIoT gives users realtime sensor
data analysis w/ #elasticsearch & #Spark. Here's how
http://t.co/oECqzBWZvh http://t",
"@timestamp": 1420246531000,
"user": "Mario Montag",
"tweetId": "601914318221750272"
}

Creating the MapReduce job to import data

from Elasticsearch to HDFS
I am sure that you already have the Twitter dataset in your Elasticsearch index.
If not, you can always navigate to the source code of the chapter and run the
target/ch02-0.0.1-tweets2es-job.jar job to get the data into Elasticsearch.
[ 44 ]

Chapter 2

You may want to perform complex analysisfor example, sentiment analysis of the
tweets that relate to a specific criteria on the Twitter data. Let's consider that you
want to do so on the data that relates to any two of the elasticsearch, kibana,
analysis, visualize, and realtime terms. To be able to perform such analysis, you
may want to run the MapReduce tasks on the relevant data.
Now, let's create a job that queries the data as required from Elasticsearch and
imports it to HDFS.

Writing the Tweets2Hdfs mapper

Here is the Tweets2HdfsMapper class to map the MapWritable representation of
Elasticsearch documents to the CSV output file to be written to HDFS:
public class Tweets2HdfsMapper extends Mapper<Object, MapWritable,
Text, Text> {
public void map(Object key, MapWritable value, Context
context) throws IOException, InterruptedException {

When we will write a MapReduce job that reads from Elasticsearch, remember that
we will get MapWritable as the input in our Mapper class. Our output will be Text
that will eventually be written to a CSV file in HDFS:
StringBuilder mappedValueBuilder = new StringBuilder();
mappedValueBuilder.append(getQuotedValue(value.get(new
Text("tweetId")))+", ");
mappedValueBuilder.append(getQuotedValue(value.get(new
Text("text")))+", ");
mappedValueBuilder.append(getQuotedValue(value.get(new
Text("user")))+", ");
mappedValueBuilder.append(getQuotedTimeValue(value.get(new
Text("@timestamp"))));
Text mappedValue = new
Text(mappedValueBuilder.toString());
context.write(mappedValue, mappedValue);
}

In the preceding snippet, we will just build a String that represents a single line
of the CSV file. The code appends the quoted tweetId, text, user, and timestamp
fields in the CSV format. Here is an example:
"601078026596687873", "RT @elastic: This week in @Elastic: all
things #Elasticsearch, #Logstash, #Kibana, community & ecosystem
https://t.co/grAEffXek1", "Leslie Hawthorn", "Wed Dec 31 11:02:23
IST 2015"
[ 45 ]

Getting Started with ES-Hadoop

private String getQuotedTimeValue(Writable writable) {
Date timestamp = new
Date(Long.parseLong(writable.toString()));
SimpleDateFormat dateFormat = new SimpleDateFormat("EEE
MMM dd hh:mm:ss zzz YYYY");
return "\""+dateFormat.format(timestamp)+"\"";
}
private String getQuotedValue(Writable value) {
return "\""+value.toString()+"\"";
}
}

The preceding two methods are handy to quickly get the quoted String or Date
values for Writable values.
Now, let's write the Driver class, as shown in the following code:
public static void main(String args[]) throws IOException,
ClassNotFoundException, InterruptedException {
// Create Configuration instance
Configuration conf = new Configuration();
// Elasticsearch Server nodes to point to
conf.set("es.nodes", "localhost:9200");
// Elasticsearch index and type name in
{indexName}/{typeName} format
conf.set("es.resource", "esh/tweets");
conf.set("es.query", query);

First, the Driver class sets the Elasticsearch node and resource to the index where
the tweets are already indexed. Then, the es.query configuration specifies the query
to be used when you fetch the data from the Elasticsearch index. If the es.query
configuration is missing, the ES-Hadoop library queries all the documents from the
index and provides them as the MapWritable input to the Mapper class, as shown in
the following code:
// Create Job instance
Job job = new Job(conf, "tweets to hdfs mapper");
// set Driver class
job.setJarByClass(Driver.class);
job.setMapperClass(Tweets2HdfsMapper.class);
// set IntputFormat to EsInputFormat provided by
Elasticsearch-Hadoop jar

[ 46 ]

Chapter 2
job.setInputFormatClass(EsInputFormat.class);
job.setNumReduceTasks(0);
FileOutputFormat.setOutputPath(job, new Path(args[0]));
System.exit(job.waitForCompletion(true) ? 0 : 1);
}

The important configuration to note here is setInputFormatClass(EsInputForm

at.class). This tells Hadoop to read the input from Elasticsearch. The ES-Hadoop
library reads the documents as mentioned by the es.query configuration and
provides as the MapWritable input to the mapper class.
In the preceding example, the query variable refers to the following Elasticsearch
query in the JSON format (we will get a detailed understanding of such queries in
the next chapter):
{
"query": {
"bool": {
"should": [
{
"term": {
"text": {
"value": "elasticsearch"
}
}
},{
"term": {
"text": {
"value": "kibana"
}
}
},{
"term": {
"text": {
"value": "analysis"
}
}
},{
"term": {
"text": {
"value": "visualize"

[ 47 ]

Getting Started with ES-Hadoop

}
}
},{
"term": {
"text": {
"value": "realtime"
}
}
}
],
"minimum_number_should_match": 2
}
}
}

The es.query configuration can be in one of the following three formats:

As a URI or a parameter query, as shown in the following code:

es.query = "?q=Elasticsearch"

Query DSL in the JSON format, as used in the Twitter2Hdfs example

es.query = { "query" : { "term" : { "text" :
"elasticsearch" } } }

External resource path

es.query = org/queries/Elasticsearch-tweets-query.json
If the URI and the QueryDSL formats are not found for es.query,
ES-Hadoop tries to find the resource in the HDFS, classpath, or the
Hadoop-distributed cache respectively with the same name. The
file must contain the query in the URI or the QueryDSL format.

Running the example

Once you have built the JAR file with dependencies, you can run the job to load the
data from Elasticsearch to HDFS.

[ 48 ]

Chapter 2

Now, execute the following command to run the example:

$ hadoop jar <LOCATION_OF_JOB_JAR_FILE>/ch02-0.0.1-tweets2hdfs-job.jar /
output/ch02/

Make sure that the output directory mentioned as a command-line

argument must not already exist in the HDFS. This will happen if
you try to run the same job twice or more. In such a case, you can
either change the output directory or delete it with the hadoop
fs rm -r /output/ch02/ command.

This command execution should display the output for the provided Twitter sample
data, as shown in the following code:
Elasticsearch Hadoop Counters
Bulk Retries=0
Bulk Retries Total Time(ms)=0
Bulk Total=0
Bulk Total Time(ms)=0
Bytes Accepted=0
Bytes Received=138731
Bytes Retried=0
Bytes Sent=4460
Documents Accepted=0
Documents Received=443
Documents Retried=0
Documents Sent=0
Network Retries=0
Network Total Time(ms)=847
Node Retries=0
Scroll Total=10
Scroll Total Time(ms)=195

The console output for the execution shows that the number of documents received
from Elasticsearch is 443.

Testing the job execution output

First, let's check whether the job successfully generated the CSV file for the expected
tweet data. To verify that the output files are generated successfully, execute the
following command:
$ hadoop fs -ls /output/ch02/

[ 49 ]

Getting Started with ES-Hadoop

Now, the successful execution of the command should print the console output, as
shown in the following code:
Found 6 items
-rw-r--r-1 eshadoop supergroup
ch02/_SUCCESS

0 2015-05-27 23:08 /output/

-rw-r--r-1 eshadoop supergroup

ch02/part-m-00000

36494 2015-05-27 23:08 /output/

-rw-r--r-1 eshadoop supergroup

ch02/part-m-00001

36684 2015-05-27 23:08 /output/

-rw-r--r-1 eshadoop supergroup

ch02/part-m-00002

38326 2015-05-27 23:08 /output/

-rw-r--r-1 eshadoop supergroup

ch02/part-m-00003

34090 2015-05-27 23:08 /output/

-rw-r--r-1 eshadoop supergroup

ch02/part-m-00004

32958 2015-05-27 23:08 /output/

The job generates as many number of output files as the mapper instances. This is
because we didn't have any reducer for the job.
To verify the content of the file, execute the following command:
$ hadoop fs -tail /output/ch02/part-m-00000

Now, verify that the output is similar to the one shown in the following code:
"914318221750272", "RT @elastic: .@PrediktoIoT gives users realtime
sensor data analysis w/ #elasticsearch & #Spark. Here's how http://t.co/
oECqzBWZvh http://t", "Mario Montag", "Sat Jan 03 06:25:31 IST 2015"
"601914318221750272", "RT @elastic: .@PrediktoIoT gives users realtime
sensor data analysis w/ #elasticsearch & #Spark. Here's how http://t.co/
oECqzBWZvh http://t", "Mario Montag", "Sat Jan 03 06:25:31 IST 2015"
"602223267194208256", "Centralized System and #Docker Logging with
#ELK Stack #elasticsearch #logstash #kibana http://t.co/MIn7I52Okl",
"Jol Vimenet", "Sun Dec 28 02:53:10 IST 2015"
"602223267194208256",
"Centralized System and #Docker Logging with #ELK Stack #elasticsearch
#logstash #kibana http://t.co/MIn7I52Okl", "Jol Vimenet", "Sun Dec 28
02:53:10 IST 2015"
"603461899456483328", "#Elasticsearch using in near realtime", "Ilica
Brnadi ", "Wed Dec 31 12:55:03 IST 2015" "603461899456483328",
"#Elasticsearch using in near realtime", "Ilica Brnadi ", "Wed Dec 31
12:55:03 IST 2015"

[ 50 ]

Chapter 2

Summary
In this chapter, we discussed the MapReduce programs by going through the
WordCount program. We checked how to develop the MapReduce jobs with the new
and old map-reduce APIs. We delved into the details of a real-world network logs
monitoring problem. You learned how to solve the problem in a better way by using
the aggregation capabilities of Elasticsearch.
Further, you learned how to write and build the Hadoop MapReduce job that
leverages ES-Hadoop to get the network logs monitoring data to Elasticsearch.
Finally, we explored how to get the data out from Elasticsearch in the MapReduce
job for the Twitter dataset. Overall, we got a complete understanding of how to get
the data in and out between Elasticsearch and Hadoop.
In the next chapter, we will be dive deeper into Elasticsearch to understand
Elasticsearch mappings, how the indexing process works, and how to query the
Elasticsearch data in order to perform full-text search and aggregations.

[ 51 ]

Get more information Elasticsearch for Hadoop

Where to buy this book

You can buy Elasticsearch for Hadoop from the Packt Publishing website.
Alternatively, you can buy the book from Amazon, BN.com, Computer Manuals and most internet
book retailers.
Click here for ordering and shipping details.

www.PacktPub.com

Stay Connected: