Scribd
Upload
152 views4 pages

Apache Spark Installation

Spark installation

Uploaded by

Harshit Sinha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
152 views4 pages

Apache Spark Installation

Spark installation

Uploaded by

Harshit Sinha
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 4

Jobs

SENDFiles
Whiteboard
Net Meeting
Tools
Articles
Facebook
Google+

Twitter

Linkedin

YouTube

Home
Tutorials Library
Coding Ground
Tutor Connect
Videos
Search

Apache Spark Tutorial


Apache Spark - Home
Apache Spark - Introduction
Apache Spark - RDD
Apache Spark - Installation
Apache Spark - Core Programming
Apache Spark - Deployment
Advanced Spark Programming
Apache Spark Useful Resources
Apache Spark - Quick Guide
Apache Spark - Useful Resources
Apache Spark - Discussion
Selected Reading
Developer's Best Practices
Questions and Answers
Effective Resume Writing
HR Interview Questions
Computer Glossary
Who is Who

Apache Spark - Installation


Advertisements
Previous Page
Next Page
Spark is Hadoops sub-project. Therefore, it is better to install Spark into a
Linux based system. The following steps show how to install Apache Spark.

Step 1: Verifying Java Installation


Java installation is one of the mandatory things in installing Spark. Try the
following command to verify the JAVA version.

$java -version
If Java is already, installed on your system, you get to see the following
response

java version "1.7.0_71" Java(TM) SE Runtime Environment (build 1.7.0_71-b13) Java HotSpot(TM) Client VM (build 25.0-
b02, mixed mode)
In case you do not have Java installed on your system, then Install Java before
proceeding to next step.

Step 2: Verifying Scala installation


You should Scala language to implement Spark. So let us verify Scala
installation using following command.
$scala -version
If Scala is already installed on your system, you get to see the following
response

Scala code runner version 2.11.6 -- Copyright 2002-2013, LAMP/EPFL


In case you dont have Scala installed on your system, then proceed to next
step for Scala installation.

Step 3: Downloading Scala


Download the latest version of Scala by visit the following link Download Scala .
For this tutorial, we are using scala-2.11.6 version. After downloading, you will
find the Scala tar file in the download folder.

Step 4: Installing Scala


Follow the below given steps for installing Scala.

Extract the Scala tar file


Type the following command for extracting the Scala tar file.
$ tar xvf scala-2.11.6.tgz

Move Scala software files


Use the following commands for moving the Scala software files, to respective
directory (/usr/local/scala) .

$ su Password: # cd /home/Hadoop/Downloads/ # mv scala-2.11.6 /usr/local/scala # exit

Set PATH for Scala


Use the following command for setting PATH for Scala.

$ export PATH = $PATH:/usr/local/scala/bin

Verifying Scala Installation


After installation, it is better to verify it. Use the following command for
verifying Scala installation.
$scala -version
If Scala is already installed on your system, you get to see the following
response

Scala code runner version 2.11.6 -- Copyright 2002-2013, LAMP/EPFL

Step 5: Downloading Apache Spark


Download the latest version of Spark by visiting the following link Download
Spark . For this tutorial, we are using spark-1.3.1-bin-hadoop2.6 version.
After downloading it, you will find the Spark tar file in the download folder.

Step 6: Installing Spark


Follow the steps given below for installing Spark.

Extracting Spark tar


The following command for extracting the spark tar file.
$ tar xvf spark-1.3.1-bin-hadoop2.6.tgz

Moving Spark software files


The following commands for moving the Spark software files to respective
directory (/usr/local/spark) .

$ su Password: # cd /home/Hadoop/Downloads/ # mv spark-1.3.1-bin-hadoop2.6 /usr/local/spark # exit

Setting up the environment for Spark


Add the following line to ~ /.bashrc file. It means adding the location, where
the spark software file are located to the PATH variable.
export PATH = $PATH:/usr/local/spark/bin
Use the following command for sourcing the ~/.bashrc file.

$ source ~/.bashrc

Step 7: Verifying the Spark Installation


Write the following command for opening Spark shell.
$spark-shell
If spark is installed successfully then you will find the following output.

Spark assembly has been built with Hive, including Datanucleus jars on classpath Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties 15/06/04 15:25:22 INFO SecurityManager: Changing view acls to: hadoop 15/06/04
15:25:22 INFO SecurityManager: Changing modify acls to: hadoop 15/06/04 15:25:22 INFO SecurityManager:
SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop); users with modify
permissions: Set(hadoop) 15/06/04 15:25:22 INFO HttpServer: Starting HTTP Server 15/06/04 15:25:23 INFO Utils:
Successfully started service 'HTTP class server' on port 43292. Welcome to ____ __ / __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/ /___/ .__/\_,_/_/ /_/\_\ version 1.4.0 /_/ Using Scala version 2.10.4 (Java HotSpot(TM)
64-Bit Server VM, Java 1.7.0_71) Type in expressions to have them evaluated. Spark context available as sc scala>
Previous Page
Print
Next Page
Advertisements

Write for us FAQ's Helping Contact


Copyright 2017. All Rights Reserved.

go

You might also like