Scribd
Upload
19 views1 page

Data Set

This document shows how to use Spark SQL to create a Dataset, perform operations like sorting and displaying results, and compute descriptive statistics. It creates a Dataset using spark.range, shows the items, sorts in descending order and shows the first 5, and calculates descriptive stats.

Uploaded by

Sai Gopi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
19 views1 page

Data Set

This document shows how to use Spark SQL to create a Dataset, perform operations like sorting and displaying results, and compute descriptive statistics. It creates a Dataset using spark.range, shows the items, sorts in descending order and shows the first 5, and calculates descriptive stats.

Uploaded by

Sai Gopi
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as TXT, PDF, TXT or read online on Scribd
You are on page 1/ 1

import org.apache.spark.sql.

SparkSession

//val spark = SparkSession


.builder()
.appName("Spark SQL basic example")
.config("spark.some.config.option", "some-value")
.getOrCreate()

//create a Dataset using spark.range starting from 5 to 50, with increments of 5


val numDS = spark.range(5, 50, 5)

//import spark.implicits._
numDS.show()

// reverse the order and display first 5 items


numDS.orderBy(desc("id")).show(5)

//compute descriptive stats and display them


numDs.describe().show()

You might also like