0% found this document useful (0 votes)

80 views18 pages

Sqoop v1.1

Sqoop is a tool used for transferring bulk data between HDFS and relational databases. It can import data from a database into HDFS or export data from HDFS to a database. Sqoop uses MapReduce jobs and JDBC to connect to databases. It supports importing/exporting to file formats like delimited text and SequenceFiles. Common databases supported include MySQL, SQL Server, PostgreSQL and Oracle. Sqoop also integrates with Hive for importing into tables and HBase for importing into column families.

Uploaded by

Saikumar Avanigadda

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

0% found this document useful (0 votes)

80 views18 pages

Sqoop v1.1

Uploaded by

Saikumar Avanigadda

We take content rights seriously. If you suspect this is your content, claim it here.

Available Formats

Download as PDF, TXT or read online on Scribd

You are on page 1/ 18

SQOOP

Karthikeyan
Sanjai
1

Sqoop v1.1

Uploaded by

Sqoop v1.1

Uploaded by

SQOOP

Bulk data transfer tool

Map only job.

Integrates with Hive & Hbase

Support plugins via connector based architecture

Two file formats:

Delimited Text File

Default import format, explicitly as --as-textfile

Sequence File --as-sequencefile

Uses JDBC compatibility layer to talk with the databases

Sample List of DBs

Need to add vendor specific drivers in $SQOOP_HOME/lib

tar txvf sqoop-*.*.tar.gz

Please add the vendor Specific JDBC jar to $SQOOP_HOME/lib

Change to Sqoop Bin folder .

Import tables from a database to HDFS

Import a table from a database to HDFS

Export an HDFS directory to a database table help List available commands

Evaluate a SQL statement and display the results

Import a table definition into Hive

Generate code to interact with database records

List available databases on a server

List available tables in a database version Display version information

mysql>CREATE TABLE patient( pid INT(10),name VARCHAR(20),durg

mysql>insert into patient values(1,'saravanan','avil',100);

mysql>insert into patient values(2,'senthil','metacin',200);

mysql>insert into patient values(3,'Gowtham','paracetamol',300);

mysql>select * from patient;

bin/sqoop eval --connect jdbc:mysql://localhost/test -username root -

password root query "SELECT * FROM patient"

bin/sqoop list-databases --connect jdbc:mysql://localhost/ information_schema

-username root -password root

SQOOP IMPORT - MYSQL TO HDFS

bin/sqoop import --connect jdbc:mysql://localhost/test -username root -

password root --table patient -m 1

Sqoop generates java file(patient.java) for our use

Data imported as text or SequenceFiles

Instead we can use codegen

bin/hadoop dfs cat /user/username/patient/part-00000

All values re displayed

These files can be used as input to MR jobs.

Increasing parallelism (number of mappers)

bin/sqoop import --connect jdbc:mysql://localhost/test -username root -

mysql to hdfs import-all-tables

bin/sqoop import-all-tables --connect jdbc:mysql://localhost/test -

password root --table patient --split-by column name(pid) -m 2

username root -password root -m 1

SQOOP - HIVE IMPORT

bin/sqoop import --connect jdbc:mysql://localhost/test --username root --password root

table patient --hbase-table patienthbase2 --column-family datasqoop [email protected]

key pid --hbase-createtable -m 1

SQOOP HBASE IMPORT

Exports a set of files from HDFS back to an RDBMS

The target table must already exist in the database

Does not export from Hbase

You might also like

tar txvf sqoop-..tar.gz