Downloaded from: justpaste.

it/cdbm5

BANK DATA ANALYSIS

You are provided with two datasets(Structured & Semi-structured) containing information about bank deposit details.

The high level activities to be performed are :

1. Load all the structured data to MYSQL

2. Ingest structured data into the HDFS environment using SQOOP (Extract)
3. Ingest Semi-structured data into HDFS environment (Extract)
4. Perform ETL operation on JSON data using PIG scripts (Transform)
5. Perform ETL and Load sqoop output data into HIVE (Load)
6. Analyse data using HQL (Analyse)

INPUT FILES

The bank deposit input files are available in “

~/

Desktop/Project/wingst2-banking-challenge/

” folder.

Structured Data :-

Chase_Bank.csv

Semi structured Data :- Chase_Bank_1.json

Important Instructions to be followed:-

HDFS Directories to store output files.

a. SQOOP output should be stored in hdfs:/user/labuser/

sqoop_bank

directory.
b. PIG output should to be stored in hdfs:/user/labuser/

bank1

directory.

Output files of your assessment (***.txt) should be present in local challenge folder ( /home/labuser/Desktop/Project/wings-xx-challenge).

Follow below steps to complete the assessment:-

Step 1:- Loading Data to MYSQL

Login to MYSQL:

Username: root
Password: labuserbdh

Create new Database and tables using MySQL commands to load structured data.
DB Name:- bank_db
Table:- bank .

Create Table and Load script is given below for you reference

create table bank (Id int , Institution_Name varchar(2000), Branch_Name varchar(2000), Branch_Number int, City varchar(2000),
County varchar(2000), State varchar(2000), Zipcode int, 2010_Deposits int,
2011_Deposits int, 2012_Deposits int, 2013_Deposits int, 2014_Deposits int,
2015_Deposits int, 2016_Deposits int);

Load

Chase_Bank.csv

data into the table

bank

load data local infile '/home/labuser/Desktop/Project/wingst2-banking-challenge/Chase_Bank.csv' into table bank fields terminated by ',' lines terminated by '\n' ignore 1 rows (Id,
Institution_Name, Branch_Name, Branch_Number, City, County, State, Zipcode, 2010_Deposits, 2011_Deposits, 2012_Deposits, 2013_Deposits, 2014_Deposits, 2015_Deposits,
2016_Deposits);

Step 2:- Import Structured data to HDFS using SQOOP

Load data from MYSQL table-

bank

to HDFS (

/user/labuser/sqoop_bank

)using Sqoop based on the following conditions.

Columns to be imported :

Id, City, County, State, Zipcode,

2010_Deposits, 2011_Deposits, 2012_Deposits,

2013_Deposits, 2014_Deposits, 2015_Deposits,

2016_Deposits

Import only the records which are

NOT IN

below mentioned cities.

"
 Rochester", "Austin", "Chicago", "Indianapolis"

Number of mappers should be 1

Run below command to copy sqoop output from HDFS to

sqoop_output.txt

file :

hdfs dfs -cat /user/labuser/sqoop_bank/* > sqoop_output.txt

Note:- Make sure that your output files are available in challenge folder.

You will be loading and analysing Sqoop output data to Hive using HQL in further steps.

Step 3:- Cleansing Semi-structured (Json) data Using PIG

You will be cleansing Json data (

Chase_Bank_1.json

) which is available in challenge input folder.

Load this json data to PIG using PIG Latin Scripts

Note:- You can either load this data directly from challenge input folder , or use required commands to copy to hdfs and then to P

Use Pig Latin scripts to

find Minimum no of deposits in 2016 (ie, MIN(Deposits_2016)) for each county . Assign

minimum_dep”

as the column name

Sort output in descending order based on

minimum_dep”

And read only first 50 records.

 Sample output format:-

{"group":"Gillespie","minimum_dep":212776}{"group":"Imperial","minimum_dep":148284}

STORE result in hdfs ( /user/labuser/pigoutput) directory

Use required commands to copy PIG output to challenge folder in file

pig_output.txt.

Step 4:- Loading and Analysing Data in Hive

Now, you have to load sqoop output to HIVE tables.

Create below database & Tables in Hive

Database

: hive_db
Partition Table

: bank_part.
Columns

Id, City, County, Zipcode, 2010_Deposits, 2011_Deposits, 2012_Deposits, 2013_Deposits, 2014_Deposits, 2015_Deposits, 2016_Depos

Partition

should be based on column

State

Read records which satisfies the below condition & Load to bank_part table
City in Bronx, NewYorkCity, Dallas, Houston, Columbus
State in "NY","OH","TX"
Hint:- Create a temporary table to load Sqoop output and then load data to partitioned table with necessary filters.

Analysing Data using HQL

Use the following command and execute hive query to remove the WARNING messages from the HIVE output

export HIVE_SKIP_SPARK_ASSEMBLY=true

Write a HQL query to fetch the records which satisfies below criteria. 2014_Deposits is greater than 50000, 2015_Deposi
ts is greater than 60000, 2016_Deposits is greater than 70000, City in NewYorkCity, Dallas, Houston.

Columns Required : -City, County, State, 2014_Deposits, 2015_Deposits, 2016_Deposits.

 Save the output in file

hive_output.txt.

Note:- Given below the sample format to copy output to file from terminal.

hive S -e "use hive_db;select count(1) from

bank_part

;" >output.txt

VALIDATION :

Before closing the environment, ensure that all the output files are available in local directory

Desktop/Project/wingst2-banking-challenge/”

sqoop_output.txt

hive_output.txt

pig_output.txt

Click on

SUBMIT

button & validation will take place at backend.