Scribd
Upload
16 views7 pages

BDA4

This document describes an experiment to analyze weather data using MapReduce. The experiment involves: 1) Downloading weather data files from NOAA and concatenating them into a single file. 2) Creating a Java project in Eclipse containing Mapper, Reducer, and Driver classes to find the maximum temperature by year in the weather data using MapReduce. 3) Configuring the Java project to include the necessary Hadoop libraries and jars.

Uploaded by

nikithakatta0
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
16 views7 pages

BDA4

This document describes an experiment to analyze weather data using MapReduce. The experiment involves: 1) Downloading weather data files from NOAA and concatenating them into a single file. 2) Creating a Java project in Eclipse containing Mapper, Reducer, and Driver classes to find the maximum temperature by year in the weather data using MapReduce. 3) Configuring the Java project to include the necessary Hadoop libraries and jars.

Uploaded by

nikithakatta0
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 7

Big data Analytics LAB

Experiment_no-04

Aim: Write a Map Reduce program that mines weather data. Weather sensors
collecting data every hour at many locations across the globe gather a large volume
of log data, which is a good candidate for analysis with MapReduce, since it is
semi structured and record-oriented.

Description:

1) Open Oracle VM VirtualBox->export cloudera->start

2) Open browser and type “ftp://ftp.ncdc.noaa.gov/pub/data/noaa/”.

3) Download any 3 folders to workspace.

4) There are multiple files in a folder, concatenate those files into a folder as
follows.
5) Concatenate these 3 folders into a single folder as follows.

6) Right click on ncdc.gz ->Extract here->rename->ncdc.txt

7) In eclipse->File->New->Java project->Project name “Weather”->Finish

8) Create three classes.

Right click on Weather-> New->class->Name “MaxTemperature”

Right click on Weather-> New->class->Name “MaxTemperatureMapper”


Right click on Weather -> New->class->Name “MaxTemperatureReducer”

9) Add Hadoop libraries.

Right click on Weather->Build path->Configure Build path->Add external


JARS. (usr\lib\hadoop\hadoop-common-2.6.0-cdh 5.13.0 jar,

usr\lib\hadoop\hadoop-core-2.6.0-cdh 5.13.0 jar)

Program:

MaxTemperature.java

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;

public class MaxTemperature {

public static void main(String[] args)throws Exception{


if(args.length!=2)

System.err.println("error");

System.exit(-1);

Job job=new Job();

job.setJarByClass(MaxTemperature.class);

job.setJobName("Max Temperature");

FileInputFormat.addInputPath(job,new Path(args[0]));

FileOutputFormat.setOutputPath(job,new Path(args[1]));

job.setMapperClass(MaxTemperatureMapper.class);

job.setReducerClass(MaxTemperatureReducer.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);

System.exit(job.waitForCompletion(true)?0:1);

MaxTemperatureMapper.java

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.LongWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Mapper;
public class MaxTemperatureMapper extends

Mapper<LongWritable,Text,Text,IntWritable>

private static final int MISSING=9999;

@Override

public void map(LongWritable key,Text value,Context context)

throws IOException,InterruptedException{

String line=value.toString();

String year=line.substring(15,19);

int airTemperature;

if(line.charAt(87)=='+')

airTemperature=Integer.parseInt(line.substring(88,92));

else

airTemperature=Integer.parseInt(line.substring(87,92));

String quality=line.substring(92,93);

if(airTemperature!=MISSING && quality.matches("[01459]"))

context.write(new Text(year),new IntWritable(airTemperature));

MaxTemperatureReducer.java

import java.io.IOException;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Reducer;

public class MaxTemperatureReducer extends

Reducer<Text,IntWritable,Text,IntWritable>{

@Override

public void reduce(Text key,Iterable<IntWritable> values,Context

context)throws IOException,InterruptedException

int maxValue=Integer.MIN_VALUE;

for(IntWritable Values:values)

maxValue=Math.max(maxValue, Values.get());

context.write(key, new IntWritable(maxValue));

Right click on Weather->Export->java->jar file->JAR file:” Temp”->Finish


OUTPUT:

In Terminal

You might also like