INTRODUCTION TO SQOOP IN HADOOP

WHAT IS SQOOP AND WHY USE SQOOP?

Sqoop is a tool used to transfer bulk data between Hadoop and external datastores, such as
relational databases (MS SQL Server, MySQL).

To process data using Hadoop, the data first needs to be loaded into Hadoop clusters from
several sources. However, it turned out that the process of loading data from several
heterogeneous sources was extremely challenging. The problems administrators encountered
included:

1. Maintaining data consistency

2. Ensuring efficient utilization of resources

3. Loading bulk data to Hadoop was not possible

4. Loading data using scripts was slow

The solution was Sqoop. Using Sqoop in Hadoop helped to overcome all the challenges of the
traditional approach and it could load bulk data from RDBMS to Hadoop with ease.

Now that we've understood about Sqoop and the need for Sqoop, as the next topic in this
Sqoop tutorial, let's learn the features of Sqoop.
SQOOP FEATURES
Sqoop has several features, which makes it helpful in the Big Data world:

1. Parallel Import/Export

Sqoop uses the YARN framework to import and export data. This provides fault tolerance
on top of parallelism.

2. Import Results of an SQL Query

Sqoop enables us to import the results returned from an SQL query into HDFS.

3. Connectors For All Major RDBMS Databases

Sqoop provides connectors for multiple RDBMSs, such as the MySQL and Microsoft
SQL servers.

4. Kerberos Security Integration

Sqoop supports the Kerberos computer network authentication protocol, which enables
nodes communication over an insecure network to authenticate users securely.

5. Provides Full and Incremental Load

Sqoop can load the entire table or parts of the table with a single command.
After going through the features of Sqoop as a part of this Sqoop tutorial, let us understand
the Sqoop architecture.
SQOOP ARCHITECTURE
Now, let’s dive deep into the architecture of Sqoop, step by step:

1. The client submits the import/ export command to import or export data.

2. Sqoop fetches data from different databases. Here, we have an enterprise data warehouse,
document-based systems, and a relational database. We have a connector for each of these;
connectors help to work with a range of accessible databases.

3. Multiple mappers perform map tasks to load the data on to HDFS.

4. Similarly, numerous map tasks will export the data from HDFS on to RDBMS using the
Sqoop export command.
SQOOP IMPORT

The diagram below represents the Sqoop import mechanism.

1. In this example, a company’s data is present in the RDBMS. All this metadata is sent to
the Sqoop import. Scoop then performs an introspection of the database to gather
metadata (primary key information).

2. It then submits a map-only job. Sqoop divides the input dataset into splits and uses
individual map tasks to push the splits to HDFS.

Few of the arguments used in Sqoop import are shown below:

1. The first step is to gather the metadata through introspection.

2. Sqoop then divides the input dataset into splits and uses individual map tasks to push the
splits to RDBMS.

Let’s now have a look at few of the arguments used in Sqoop export:

After understanding the Sqoop import and export, the next section in this Sqoop tutorial is the
processing that takes place in Sqoop.
SQOOP PROCESSING

Processing takes place step by step, as shown below:

1. Sqoop runs in the Hadoop cluster.

2. It imports data from the RDBMS or NoSQL database to HDFS.

3. It uses mappers to slice the incoming data into multiple formats and loads the data in
HDFS.

4. Exports data back into the RDBMS while ensuring that the schema of the data in the
database is maintained.

Key features of Big Data Sqoop

 Bulk import: Big Data Sqoop facilitates the import of singular tables and comprehensive
databases into HDFS. The information is saved in the native directories and files in the
HDFS file system

 Direct input: Big Data Sqoop can also enable import and map SQL (relational) databases
into Hive and HBase

 Data interaction: Big Data Sqoop is capable of generating Java classes so that you can
interact with the data in the scope of programming

 Data export: Big Data Sqoop can also export information from HDFS into a relational
database with the help of a target table definition based on the specifics of the target
database

Functionality of Sqoop

Sqoop is one of the best Big Data platforms mostly owing to its superior functionalities. It functions
by analyzing the database you want to import and by picking an apt import function required for the
source data. After it identifies the input commands, it checks the metadata for the table (or
database) and creates a class definition of the concerned requirements of the import.

On the other hand, Sqoop can also be very selective so that it aids you with just the columns you
would like to look at before the process of inputting rather than going through the trouble of doing
the entire input and then identifying information. This saves time to a great extent. The actual
import from the external database to HDFS is performed by a MapReduce job created behind the
scenes by Sqoop.
Sqoop is easy enough to be an efficient Big Data tool for amateur programmers too. While it
maybe, it is to be kept in mind that it has a high degree of dependence on underlying technologies
like HDFS and MapReduce.

Benefits

Ease of Use – Sqoop lets connectors to be configured in one place, which can be managed by the
admin role and run by the operator role. This centralized architecture helps in better deployment of
Big Data analytics and solutions.

Ease of Extension – The connectors of Sqoop are not restricted to just the JDBC model. It has the
competencies to extend and define its own vocabulary without having the need to mention a table
name.

Security – The fact that Sqoop operates as server based application that secures access to external
systems and does not allow code generation, makes its security to go by.

Comparison Between Hive and HBase Tables:

Feature Hive Table HBase Table

Storage Stored as files in HDFS Stored in HBase (column-oriented)

Data Model SQL-like, structured (rows/columns) NoSQL, sparse, multidimensional

Use Case Data warehousing, batch processing Real-time data access, NoSQL

Schema Schema-on-read Schema-on-write

Query Language HiveQL (SQL-like) HBase Shell / API

In Hadoop:

 Hive:
o Database: Logical collection of tables.
o Table: Structured data stored in HDFS; can be managed (Hive controls data) or
external (Hive only manages metadata).
 HBase:
o Namespace: Equivalent to a database, groups tables.
o Table: NoSQL, column-family-based storage for real-time access.

Hive is best for batch processing with SQL-like queries, while HBase suits real-time, random
read/write access.