Big Data Processing With Spark
2 likes1,672 views
The document outlines a course on Apache Spark and Scala offered by Edureka, detailing the objectives, features, advantages, and architecture of Spark. It emphasizes Spark's capabilities for fast, unified big data processing, including support for various programming languages and integration with Hadoop. Key components include Resilient Distributed Datasets (RDDs), Spark SQL for querying structured data, and the efficiency of in-memory processing.
Big Data Processing With Spark
- 1. View Apache Spark and Scala course details at www.edureka.co/apache-spark-scala-training Apache Spark | Spark SQL
- 2. Slide 2 www.edureka.co/apache-spark-scala-trainingSlide 2 Objectives At the end of this module, you will be able to ïź Introduction of Spark ïź Spark Architecture ïź What is an RDD ïź Demo On Creating RDD and Running sample example ïź Spark SQL
- 3. Slide 3 www.edureka.co/apache-spark-scala-trainingSlide 3 What is Spark? Apache Spark is an open source, parallel data processing framework that complements Apache Hadoop to make it easy to develop fast, unified Big Data applications combining batch, streaming, and interactive analytics. ïź Developed at UC Berkeley ïźWritten in Scala , a Functional Programming Language that runs in a JMV ïźIt generalize the Map Reduce framework
- 4. Slide 4 www.edureka.co/apache-spark-scala-trainingSlide 4 Why Spark ? Speed Run programs up to 100x faster than Hadoop Map Reduce in memory, or 10x faster on disk. Ease of Use Supports different languages for developing applications using Spark Generality Combine SQL, streaming, and complex analytics into one platform Runs Everywhere Spark runs on Hadoop, Mesos, standalone, or in the cloud.
- 5. Slide 5 www.edureka.co/apache-spark-scala-trainingSlide 5 ïźMap Reduce is a great solution for one-pass computations, but not very efficient for use cases that require multi-pass computations and algorithms ( Machine learning etc.) ïźTo run complicated jobs, you would have to string together a series of Map Reduce jobs and execute them in sequence ïź Each of those jobs was high-latency, and none could start until the previous job had finished completely ïźThe Job output data between each step has to be stored in the local file system before the next step can begin ïź Hadoop requires the integration of several tools for different big data use cases (like Mahout for Machine Learning and Storm for streaming data processing) Map Reduce Limitations
- 6. Slide 6 www.edureka.co/apache-spark-scala-trainingSlide 6 Spark Features ïź Spark takes Map Reduce to the next level with less expensive shuffles in the data processing. With capabilities like in- memory data storage ïź Spark has an advanced DAG execution engine that supports cyclic data flow and in-memory computing ïź Itâs designed to be an execution engine that works both in-memory and on-disk ïź Lazy evaluation of big data queries which helps with the optimization of the overall data processing workflow ïź Provides concise and consistent APIs in Scala, Java and Python ïź Offers interactive shell for Scala and Python. This is not available in Java yet ïź Spark support high level APIs to develop applications (Scala, Java, Python, Clojure, R)
- 7. Slide 7 www.edureka.co/apache-spark-scala-trainingSlide 7 Spark Core Spark Streaming Spark Sql Blink DB MLlib Graph X Spark R Spark Architecture
- 8. Slide 8 www.edureka.co/apache-spark-scala-trainingSlide 8 Spark Core Spark Streaming Spark Sql Blink DB MLlib Graph X Spark R Spark Architecture Cluster management ( Native Spark Cluster, YARN, MESOS ) Distributed storage ( HDFS, Cassandra, S3, HBase )
- 9. Slide 9 www.edureka.co/apache-spark-scala-trainingSlide 9 Spark Advantages EASE OF DEVELOPMENT COMBINE WORKFLOWS IN-MEMORY PERFORMANCE ïź Easier APIs ïź Python, Scala, Java ïź RDDs ïź DAGs Unify Processing ïź Shark, ML Streaming, GraphX
- 10. Slide 10 www.edureka.co/apache-spark-scala-trainingSlide 10 UNLIMITED SCALE WIDE RANGE OF APPLICATIONS ENTERPRISE PLATFORM ïź Multiple data sources ïź Multiple applications ïź Multiple users ïź Reliability ïź Multi-tenancy ïź Security ïź Files ïź Databases ïź Semi-structured Hadoop Advantages
- 11. Slide 11 www.edureka.co/apache-spark-scala-trainingSlide 11 Spark + Hadoop UNLIMITED SCALE WIDE RANGE OF APPLICATIONS ENTERPRISE PLATFORM EASE OF DEVELOPMENT COMBINE WORKFLOWS IN-MEMORY PERFORMANCE Operational Applications Augmented by In-Memory Performance
- 12. Slide 12 www.edureka.co/apache-spark-scala-trainingSlide 12 Resilient Distributed Datasets RDD ( Resilient Distributed Data Sets ) Resilient â If data in memory is lost, It can be recreated Distributed â Stored in memory across the cluster Dataset â Initial data can come from a file or created programmatically. RDDs are the fundamental unit of data in spark
- 13. Slide 13 www.edureka.co/apache-spark-scala-trainingSlide 13 Resilient Distributed Datasets Core concept of Spark framework. RDDs can store any type of data. Primitive Types : Integer, Characters, Boolean etc. Files : Text files, SequencFiles etc. RDD is fault tolerance. RDDs are immutable
- 14. Slide 14 www.edureka.co/apache-spark-scala-trainingSlide 14 RDD supports two types of operations: Transformation: Transformations don't return a single value, they return a new RDD. Some of the Transformation functions are map, filter, flatMap, groupByKey, reduceByKey, aggregateByKey, pipe, and coalesce. Action: Action operation evaluates and returns a new value. Some of the Action operations are reduce, collect, count, first, take, countByKey, and foreach. Resilient Distributed Datasets
- 15. Slide 15 www.edureka.co/apache-spark-scala-trainingSlide 15 Spark Sql Spark Core ïź Spark SQL allows relational queries through Spark ïź The backbone for all these operations is SchemaRDD ïź Schema RDDs are mode of row objects along with the metadata information ïź SchemaRDDs are equivalent to RDBMS tables ïź They can be constructed from existing RDDs, JSON data sets, Parquet files or Hive QL queries against the data stored in Apache Hive(*) Spark SQL
- 16. Slide 16 www.edureka.co/apache-spark-scala-training Spark SQL ïźSpark SQL lets you query structured data as a distributed dataset (RDD) in Spark, with integrated APIs in Scala and Java ïź Shark Project is completely closed now Earlier it was Shark but now we will use Spark SQL Shark Spark SQL Hive on Spark Development ending: transitioning to Spark SQL A new SQL engine designed from ground up for Spark Help existing Hive users migrate Spark
- 17. Slide 17 www.edureka.co/apache-spark-scala-trainingSlide 17 Efficient In-Memory Storage ïźSimply caching Hive records as Java objects is inefficient due to high per-object overhead ïźInstead, Spark SQL employs column-oriented storage using arrays of primitive types 1 Column Storage 2 3 john mike sally 4.1 3.5 6.4 Row Storage 1 john 4.1 2 mike 3.5 3 sally 6.4
- 18. Slide 18 www.edureka.co/apache-spark-scala-trainingSlide 18 Demo On Spark RDDs
- 19. Slide 19 www.edureka.co/apache-spark-scala-training LIVE Online Class Class Recording in LMS 24/7 Post Class Support Module Wise Quiz Project Work Verifiable Certificate Course Features
- 21. Slide 21 www.edureka.co/apache-spark-scala-training Course Topics ïź Module 1 » Introduction to Scala ïź Module 2 » Scala Essentials ïź Module 3 » Traits and OOPs in Scala ïź Module 4 » Functional Programming in Scala ïźModule 5 » Introduction to Big Data and Spark ïźModule 6 » Spark Baby Steps ïźModule 7 » Playing with RDDs ïźModule 8 » Spark with SQL- When Spark meets Hive