Scribd
Upload
52 views

Example - (Map Function in Word Count)

MapReduce is a computation framework that decomposes large data manipulation jobs into individual tasks that can be executed in parallel across a cluster of servers. It consists of two main steps - the Map function that converts input data into key-value pairs, and the Reduce function that combines the outputs of the Map tasks into a smaller set of aggregated results. The example provided demonstrates how a word counting program in Java uses MapReduce by defining Mapper and Reducer classes to implement the map and reduce logic.

Uploaded by

Dileep Reddy
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
52 views

Example - (Map Function in Word Count)

MapReduce is a computation framework that decomposes large data manipulation jobs into individual tasks that can be executed in parallel across a cluster of servers. It consists of two main steps - the Map function that converts input data into key-value pairs, and the Reduce function that combines the outputs of the Map tasks into a smaller set of aggregated results. The example provided demonstrates how a word counting program in Java uses MapReduce by defining Mapper and Reducer classes to implement the map and reduce logic.

Uploaded by

Dileep Reddy
Copyright
© © All Rights Reserved
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 6

In Hadoop, MapReduce is a computation that decomposes large

manipulation jobs into individual tasks that can be executed in parallel


cross a cluster of servers. The results of tasks can be joined together to
compute final results.

MapReduce consists of 2 steps:

 Map Function – It takes a set of data and converts it into another set
of data, where individual elements are broken down into tuples (Key-
Value pair).

Example – (Map function in Word Count)

Set of data Bus, Car, bus, car, train, car, bus, car, train,
Input bus, TRAIN,BUS, buS, caR, CAR, car, BUS,
TRAIN

(Bus,1), (Car,1), (bus,1), (car,1), (train,1),

Convert into (car,1), (bus,1), (car,1), (train,1), (bus,1),


Output another set of data (TRAIN,1),(BUS,1), (buS,1), (caR,1),
(Key,Value) (CAR,1),

(car,1), (BUS,1), (TRAIN,1)

 Reduce Function – Takes the output from Map as an input and


combines those data tuples into a smaller set of tuples.

Example – (Reduce function in Word Count)

Input Set of Tuples (Bus,1), (Car,1), (bus,1), (car,1),


(train,1),
(output of Map
(car,1), (bus,1), (car,1), (train,1),
(bus,1),

function) (TRAIN,1),(BUS,1), (buS,1),


(caR,1), (CAR,1),

(car,1), (BUS,1), (TRAIN,1)

(BUS,7),
Converts into
Output (CAR,7),
smaller set of tuples
(TRAIN,4)

Work Flow of Program


Workflow of MapReduce consists of 5 steps

1. Splitting – The splitting parameter can be anything, e.g. splitting


by space, comma, semicolon, or even by a new line (‘\n’).

2. Mapping – as explained above

3. Intermediate splitting – the entire process in parallel on different


clusters. In order to group them in “Reduce Phase” the similar
KEY data should be on same cluster.

4. Reduce – it is nothing but mostly group by phase

5. Combining – The last phase where all the data (individual result
set from each cluster) is combine together to form a Result

Now Let’s See the Word Count Program in Java

Fortunately we don’t have to write all of the above steps, we only need
to write the splitting parameter, Map function logic, and Reduce function
logic. The rest of the remaining steps will execute automatically.

Make sure that Hadoop is installed on your system with java idk

Steps

Step 1. Open Eclipse> File > New > Java Project >( Name it –
MRProgramsDemo) > Finish

Step 2. Right Click > New > Package ( Name it - PackageDemo) >
Finish

Step 3. Right Click on Package > New > Class (Name it - WordCount)

Step 4. Add Following Reference Libraries –

Right Click on Project > Build Path> Add External Archivals


 /usr/lib/hadoop-0.20/hadoop-core.jar

 Usr/lib/hadoop-0.20/lib/Commons-cli-1.2.jar

Step 5. Type following Program :

package PackageDemo;
import java.io.IOException;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.util.GenericOptionsParser;
public class WordCount {
public static void main(String [] args) throws Exception
{
Configuration c=new Configuration();
String[] files=new GenericOptionsParser(c,args).getRemainingArgs();
Path input=new Path(files[0]);
Path output=new Path(files[1]);
Job j=new Job(c,"wordcount");
j.setJarByClass(WordCount.class);
j.setMapperClass(MapForWordCount.class);
j.setReducerClass(ReduceForWordCount.class);
j.setOutputKeyClass(Text.class);
j.setOutputValueClass(IntWritable.class);
FileInputFormat.addInputPath(j, input);
FileOutputFormat.setOutputPath(j, output);
System.exit(j.waitForCompletion(true)?0:1);
}
public static class MapForWordCount extends Mapper<LongWritable,
Text, Text, IntWritable>{
public void map(LongWritable key, Text value, Context con) throws
IOException, InterruptedException
{
String line = value.toString();
String[] words=line.split(",");
for(String word: words )
{
Text outputKey = new Text(word.toUpperCase().trim());
IntWritable outputValue = new IntWritable(1);
con.write(outputKey, outputValue);
}
}
}
public static class ReduceForWordCount extends Reducer<Text,
IntWritable, Text, IntWritable>
{
public void reduce(Text word, Iterable<IntWritable> values, Context
con) throws IOException, InterruptedException
{
int sum = 0;
for(IntWritable value : values)
{
sum += value.get();
}
con.write(word, new IntWritable(sum));
}
}
}

Explanation

The program consist of 3 classes:


 Driver class (Public void static main- the entry point)

 Map class which extends public class


Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT> and
implements the Map function.

 Reduce class which extends public class


Reducer<KEYIN,VALUEIN,KEYOUT,VALUEOUT> and
implements the Reduce function.

Step 6. Make Jar File

Right Click on Project> Export> Select export destination as Jar File >
next> Finish

You might also like