Pyspark-1 6 1
Pyspark-1 6 1
Contents
PySpark is the Python API for Spark.
Public classes:
SparkContext:
Main entry point for Spark functionality.
RDD:
A Resilient Distributed Dataset (RDD), the basic abstraction in Spark.
Broadcast:
A broadcast variable that gets reused across tasks.
Accumulator:
An “addonly” shared variable that tasks can only add values to.
SparkConf:
For configuring Spark.
SparkFiles:
Access files shipped with jobs.
StorageLevel:
Finergrained cache persistence levels.
class pyspark.SparkConf(loadDefaults=True, _jvm=None, _jconf=None )
Configuration for a Spark application. Used to set various Spark parameters as keyvalue pairs.
Most of the time, you would create a SparkConf object with SparkConf(), which will load values from
spark.* Java system properties as well. In this case, any parameters you set directly on the SparkConf
object take priority over system properties.
For unit tests, you can also call SparkConf(false) to skip loading external settings and get the same
configuration no matter what the system properties are.
All setter methods in this class support chaining. For example, you can write
conf.setMaster(“local”).setAppName(“My app”).
Note that once a SparkConf object is passed to Spark, it is cloned and can no longer be modified by the
user.
contains(key)
Does this configuration contain a given key?
get(key, defaultValue=None )
Get the configured value for some key, or return a default otherwise.
getAll()
Get all values as a list of keyvalue pairs.
set(key, value )
Set a configuration property.
setAll(pairs)
Set multiple parameters, passed as a list of keyvalue pairs.
setAppName(value )
Set application name.
setExecutorEnv(key=None, value=None, pairs=None )
Set an environment variable to be passed to executors.
setIfMissing(key, value )
Set a configuration property, if not already set.
setMaster(value )
Set master URL to connect to.
setSparkHome(value )
Set path where Spark is installed on worker nodes.
toDebugString()
Returns a printable version of the configuration, as a list of key=value pairs, one per line.
class pyspark.SparkContext(master=None, appName=None, sparkHome=None, pyFiles=None,
environment=None, batchSize=0, serializer=PickleSerializer(), conf=None, gateway=None, jsc=None,
profiler_cls=<class 'pyspark.profiler.BasicProfiler'>)
Main entry point for Spark functionality. A SparkContext represents the connection to a Spark cluster,
and can be used to create RDD and broadcast variables on that cluster.
PACKAGE_EXTENSIONS = ('.zip', '.egg', '.jar')
accumulator(value, accum_param=None )
Create an Accumulator with the given initial value, using a given AccumulatorParam helper object to
define how to add values of the data type if provided. Default AccumulatorParams are used for
integers and floatingpoint numbers if you do not provide one. For other types, a custom
AccumulatorParam can be used.
addFile(path )
Add a file to be downloaded with this Spark job on every node. The path passed can be either a
local file, a file in HDFS (or other Hadoopsupported filesystems), or an HTTP, HTTPS or FTP URI.
To access the file in Spark jobs, use L{SparkFiles.get(fileName)<pyspark.files.SparkFiles.get>} with
the filename to find its download location.
>>> from pyspark import SparkFiles
>>> path = os.path.join(tempdir, "test.txt")
>>> with open(path, "w") as testFile:
... _ = testFile.write("100")
>>> sc.addFile(path)
>>> def func(iterator):
... with open(SparkFiles.get("test.txt")) as testFile:
... fileVal = int(testFile.readline())
... return [x * fileVal for x in iterator]
>>> sc.parallelize([1, 2, 3, 4]).mapPartitions(func).collect()
[100, 200, 300, 400]
addPyFile(path )
Add a .py or .zip dependency for all tasks to be executed on this SparkContext in the future. The
path passed can be either a local file, a file in HDFS (or other Hadoopsupported filesystems), or
an HTTP, HTTPS or FTP URI.
applicationId
A unique identifier for the Spark application. Its format depends on the scheduler implementation.
in case of local spark app something like ‘local1433865536131’
in case of YARN something like ‘application_1433865536131_34483’
>>> sc.applicationId
u'local-...'
binaryFiles(path, minPartitions=None )
Read a directory of binary files from HDFS, a local file system (available on all nodes), or any
Hadoopsupported file system URI as a byte array. Each file is read as a single record and
returned in a keyvalue pair, where the key is the path of each file, the value is the content of each
file.
Note: Small files are preferred, large file is also allowable, but may cause bad performance.
binaryRecords(path, recordLength )
Load data from a flat binary file, assuming each record is a set of numbers with the specified
numerical format (see ByteBuffer), and the number of bytes per record is constant.
broadcast(value )
Broadcast a readonly variable to the cluster, returning a
L{Broadcast<pyspark.broadcast.Broadcast>} object for reading it in distributed functions. The
variable will be sent to each cluster only once.
cancelAllJobs()
Cancel all jobs that have been scheduled or are running.
cancelJobGroup(groupId )
Cancel active jobs for the specified group. See SparkContext.setJobGroup for more information.
clearFiles()
Clear the job’s list of files added by addFile or addPyFile so that they do not get downloaded to
any new nodes.
defaultMinPartitions
Default min number of partitions for Hadoop RDDs when not given by user
defaultParallelism
Default level of parallelism to use when not given by user (e.g. for reduce tasks)
dump_profiles(path )
Dump the profile stats into directory path
emptyRDD()
Create an RDD that has no partitions or elements.
getLocalProperty(key)
Get a local property set in this thread, or null if it is missing. See setLocalProperty
classmethod getOrCreate(conf=None )
Get or instantiate a SparkContext and register it as a singleton object.
hadoopFile(path, inputFormatClass, keyClass, valueClass, keyConverter=None,
valueConverter=None, conf=None, batchSize=0 )
Read an ‘old’ Hadoop InputFormat with arbitrary key and value class from HDFS, a local file system
(available on all nodes), or any Hadoopsupported file system URI. The mechanism is the same as
for sc.sequenceFile.
A Hadoop configuration can be passed in as a Python dict. This will be converted into a
Configuration in Java.
hadoopRDD(inputFormatClass, keyClass, valueClass, keyConverter=None, valueConverter=None,
conf=None, batchSize=0 )
Read an ‘old’ Hadoop InputFormat with arbitrary key and value class, from an arbitrary Hadoop
configuration, which is passed in as a Python dict. This will be converted into a Configuration in
Java. The mechanism is the same as for sc.sequenceFile.
newAPIHadoopFile(path, inputFormatClass, keyClass, valueClass, keyConverter=None,
valueConverter=None, conf=None, batchSize=0 )
Read a ‘new API’ Hadoop InputFormat with arbitrary key and value class from HDFS, a local file
system (available on all nodes), or any Hadoopsupported file system URI. The mechanism is the
same as for sc.sequenceFile.
A Hadoop configuration can be passed in as a Python dict. This will be converted into a
Configuration in Java
newAPIHadoopRDD(inputFormatClass, keyClass, valueClass, keyConverter=None,
valueConverter=None, conf=None, batchSize=0 )
Read a ‘new API’ Hadoop InputFormat with arbitrary key and value class, from an arbitrary Hadoop
configuration, which is passed in as a Python dict. This will be converted into a Configuration in
Java. The mechanism is the same as for sc.sequenceFile.
parallelize(c, numSlices=None )
Distribute a local Python collection to form an RDD. Using xrange is recommended if the input
represents a range for performance.
pickleFile(name, minPartitions=None )
Load an RDD previously saved using RDD.saveAsPickleFile method.
range(start, end=None, step=1, numSlices=None )
Create a new RDD of int containing elements from start to end (exclusive), increased by step every
element. Can be called the same way as python’s builtin range() function. If called with a single
argument, the argument is interpreted as end, and start is set to 0.
>>> sc.range(5).collect()
[0, 1, 2, 3, 4]
>>> sc.range(2, 4).collect()
[2, 3]
>>> sc.range(1, 7, 2).collect()
[1, 3, 5]
runJob(rdd, partitionFunc, partitions=None, allowLocal=False )
Executes the given partitionFunc on the specified set of partitions, returning the result as an array
of elements.
If ‘partitions’ is not specified, this will run over all partitions.
1. A Java RDD is created from the SequenceFile or other InputFormat, and the
key and value Writable classes
2. Serialization is attempted via Pyrolite pickling
3. If this fails, the fallback is to call ‘toString’ on each key and value
4. PickleSerializer is used to deserialize pickled objects on the Python side
setCheckpointDir(dirName )
Set the directory under which RDDs are going to be checkpointed. The directory must be a HDFS
path if running on a cluster.
setJobGroup(groupId, description, interruptOnCancel=False )
Assigns a group ID to all the jobs started by this thread until the group ID is set to a different value
or cleared.
Often, a unit of execution in an application consists of multiple Spark actions or jobs. Application
programmers can use this method to group all those jobs together and give a group description.
Once set, the Spark web UI will associate such jobs with this group.
The application can use SparkContext.cancelJobGroup to cancel all running jobs in this group.
If interruptOnCancel is set to true for the job group, then job cancellation will result in
Thread.interrupt() being called on the job’s executor threads. This is useful to help ensure that the
tasks are actually stopped in a timely manner, but is off by default due to HDFS1208, where HDFS
may respond to Thread.interrupt() by marking nodes as dead.
setLocalProperty(key, value )
Set a local property that affects jobs submitted from this thread, such as the Spark fair scheduler
pool.
setLogLevel(logLevel)
Control our logLevel. This overrides any userdefined log settings. Valid log levels include: ALL,
DEBUG, ERROR, FATAL, INFO, OFF, TRACE, WARN
classmethod setSystemProperty(key, value )
Set a Java system property, such as spark.executor.memory. This must must be invoked before
instantiating SparkContext.
show_profiles()
Print the profile stats to stdout
sparkUser()
Get SPARK_USER for user who is running SparkContext.
startTime
Return the epoch time when the Spark Context was started.
statusTracker()
Return StatusTracker object
stop()
Shut down the SparkContext.
textFile(name, minPartitions=None, use_unicode=True )
Read a text file from HDFS, a local file system (available on all nodes), or any Hadoopsupported
file system URI, and return it as an RDD of Strings.
If use_unicode is False, the strings will be kept as str (encoding as utf8), which is faster and
smaller than unicode. (Added in Spark 1.2)
This supports unions() of RDDs with different serialized formats, although this forces them to be
reserialized using the default serializer:
version
The version of Spark on which this application is running.
wholeTextFiles(path, minPartitions=None, use_unicode=True )
Read a directory of text files from HDFS, a local file system (available on all nodes), or any
Hadoopsupported file system URI. Each file is read as a single record and returned in a keyvalue
pair, where the key is the path of each file, the value is the content of each file.
If use_unicode is False, the strings will be kept as str (encoding as utf8), which is faster and
smaller than unicode. (Added in Spark 1.2)
For example, if you have the following files:
hdfs://a-hdfs-path/part-00000
hdfs://a-hdfs-path/part-00001
...
hdfs://a-hdfs-path/part-nnnnn
Do rdd = sparkContext.wholeTextFiles(“hdfs://ahdfspath”), then rdd contains:
NOTE: Small files are preferred, as each file will be loaded fully in memory.
class pyspark.SparkFiles
Resolves paths to files added through L{SparkContext.addFile()
<pyspark.context.SparkContext.addFile>}.
SparkFiles contains only classmethods; users should not create SparkFiles instances.
classmethod get(filename )
Get the absolute path of a file added through SparkContext.addFile().
classmethod getRootDirectory()
Get the root directory that contains files added through SparkContext.addFile().
class pyspark.RDD(jrdd, ctx, jrdd_deserializer=AutoBatchedSerializer(PickleSerializer()) )
A Resilient Distributed Dataset (RDD), the basic abstraction in Spark. Represents an immutable,
partitioned collection of elements that can be operated on in parallel.
aggregate(zeroValue, seqOp, combOp )
Aggregate the elements of each partition, and then the results for all the partitions, using a given
combine functions and a neutral “zero value.”
The functions op(t1, t2) is allowed to modify t1 and return it as its result value to avoid object
allocation; however, it should not modify t2.
The first function (seqOp) can return a different result type, U, than the type of this RDD. Thus, we
need one operation for merging a T into an U and one operation for merging two U
aggregateByKey(zeroValue, seqFunc, combFunc, numPartitions=None, partitionFunc=<function
portable_hash at 0x7f1ac7340578>)
Aggregate the values of each key, using given combine functions and a neutral “zero value”. This
function can return a different result type, U, than the type of the values in this RDD, V. Thus, we
need one operation for merging a V into a U and one operation for merging two U’s, The former
operation is used for merging values within a partition, and the latter is used for merging values
between partitions. To avoid memory allocation, both of these functions are allowed to modify and
return their first argument instead of creating a new U.
cache()
Persist this RDD with the default storage level ( MEMORY_ONLY_SER).
cartesian(other )
Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements
(a, b) where a is in self and b is in other.
coalesce(numPartitions, shuffle=False )
Return a new RDD that is reduced into numPartitions partitions.
cogroup(other, numPartitions=None )
For each key k in self or other, return a resulting RDD that contains a tuple with the list of values
for that key in self as well as other.
collect()
Return a list that contains all of the elements in this RDD.
collectAsMap()
Return the keyvalue pairs in this RDD to the master as a dictionary.
combineByKey(createCombiner, mergeValue, mergeCombiners, numPartitions=None, partitionFunc=
<function portable_hash at 0x7f1ac7340578>)
Generic function to combine the elements for each key using a custom set of aggregation
functions.
Turns an RDD[(K, V)] into a result of type RDD[(K, C)], for a “combined type” C. Note that V and C
can be different – for example, one might group an RDD of type (Int, Int) into an RDD of type (Int,
List[Int]).
Users provide three functions:
createCombiner, which turns a V into a C (e.g., creates a oneelement list)
mergeValue, to merge a V into a C (e.g., adds it to the end of a list)
mergeCombiners, to combine two C’s into a single one.
In addition, users can control the partitioning of the output RDD.
>>> x = sc.parallelize([("a", 1), ("b", 1), ("a", 1)])
>>> def add(a, b): return a + str(b)
>>> sorted(x.combineByKey(str, add, add).collect())
[('a', '11'), ('b', '1')]
context
The SparkContext that this RDD was created on.
count()
Return the number of elements in this RDD.
countApprox(timeout, confidence=0.95 )
Approximate version of count() that returns a potentially incomplete result within a timeout, even if
not all tasks have finished.
countApproxDistinct(relativeSD=0.05 )
Return approximate number of distinct elements in the RDD.
The algorithm used is based on streamlib’s implementation of “HyperLogLog in Practice:
Algorithmic Engineering of a State of The Art Cardinality Estimation Algorithm”, available <a
href=”http://dx.doi.org/10.1145/2452376.2452456“>here</a>.
>>> n = sc.parallelize(range(1000)).map(str).countApproxDistinct()
>>> 900 < n < 1100
True
>>> n = sc.parallelize([i % 20 for i in range(1000)]).countApproxDistinct()
>>> 16 < n < 24
True
countByKey()
Count the number of elements for each key, and return the result to the master as a dictionary.
countByValue()
Return the count of each unique value in this RDD as a dictionary of (value, count) pairs.
distinct(numPartitions=None )
Return a new RDD containing the distinct elements in this RDD.
filter(f)
Return a new RDD containing only the elements that satisfy a predicate.
first()
Return the first element in this RDD.
flatMap(f, preservesPartitioning=False )
Return a new RDD by first applying a function to all elements of this RDD, and then flattening the
results.
flatMapValues(f)
Pass each value in the keyvalue pair RDD through a flatMap function without changing the keys;
this also retains the original RDD’s partitioning.
fold(zeroValue, op )
Aggregate the elements of each partition, and then the results for all the partitions, using a given
associative and commutative function and a neutral “zero value.”
The function op(t1, t2) is allowed to modify t1 and return it as its result value to avoid object
allocation; however, it should not modify t2.
This behaves somewhat differently from fold operations implemented for nondistributed collections
in functional languages like Scala. This fold operation may be applied to partitions individually, and
then fold those results into the final result, rather than apply the fold to each element sequentially
in some defined ordering. For functions that are not commutative, the result may differ from that of
a fold applied to a nondistributed collection.
foldByKey(zeroValue, func, numPartitions=None, partitionFunc=<function portable_hash at
0x7f1ac7340578>)
Merge the values for each key using an associative function “func” and a neutral “zeroValue” which
may be added to the result an arbitrary number of times, and must not change the result (e.g., 0
for addition, or 1 for multiplication.).
foreach(f)
Applies a function to all elements of this RDD.
foreachPartition(f)
Applies a function to each partition of this RDD.
fullOuterJoin(other, numPartitions=None )
Perform a right outer join of self and other.
For each element (k, v) in self, the resulting RDD will either contain all pairs (k, (v, w)) for w in
other, or the pair (k, (v, None)) if no elements in other have key k.
Similarly, for each element (k, w) in other, the resulting RDD will either contain all pairs (k, (v, w))
for v in self, or the pair (k, (None, w)) if no elements in self have key k.
Hashpartitions the resulting RDD into the given number of partitions.
getNumPartitions()
Returns the number of partitions in RDD
getStorageLevel()
Get the RDD’s current storage level.
glom()
Return an RDD created by coalescing all elements within each partition into a list.
groupBy(f, numPartitions=None, partitionFunc=<function portable_hash at 0x7f1ac7340578>)
Return an RDD of grouped items.
groupByKey(numPartitions=None, partitionFunc=<function portable_hash at 0x7f1ac7340578>)
Group the values for each key in the RDD into a single sequence. Hashpartitions the resulting
RDD with numPartitions partitions.
Note: If you are grouping in order to perform an aggregation (such as a sum or average) over each
key, using reduceByKey or aggregateByKey will provide much better performance.
groupWith(other, *others)
Alias for cogroup but with support for multiple RDDs.
>>> w = sc.parallelize([("a", 5), ("b", 6)])
>>> x = sc.parallelize([("a", 1), ("b", 4)])
>>> y = sc.parallelize([("a", 2)])
>>> z = sc.parallelize([("b", 42)])
>>> [(x, tuple(map(list, y))) for x, y in sorted(list(w.groupWith(x, y, z).collect()))]
[('a', ([5], [1], [2], [])), ('b', ([6], [4], [], [42]))]
histogram(buckets)
Compute a histogram using the provided buckets. The buckets are all open to the right except for
the last which is closed. e.g. [1,10,20,50] means the buckets are [1,10) [10,20) [20,50], which
means 1<=x<10, 10<=x<20, 20<=x<=50. And on the input of 1 and 50 we would have a histogram
of 1,0,1.
If your histogram is evenly spaced (e.g. [0, 10, 20, 30]), this can be switched from an O(log n)
inseration to O(1) per element(where n = # buckets).
Buckets must be sorted and not contain any duplicates, must be at least two elements.
If buckets is a number, it will generates buckets which are evenly spaced between the minimum
and maximum of the RDD. For example, if the min value is 0 and the max is 100, given buckets as
2, the resulting buckets will be [0,50) [50,100]. buckets must be at least 1 If the RDD contains
infinity, NaN throws an exception If the elements in RDD do not vary (max == min) always returns a
single bucket.
It will return an tuple of buckets and histogram.
id()
A unique ID for this RDD (within its SparkContext).
intersection(other )
Return the intersection of this RDD and another one. The output will not contain any duplicate
elements, even if the input RDDs did.
Note that this method performs a shuffle internally.
isCheckpointed()
Return whether this RDD has been checkpointed or not
isEmpty()
Returns true if and only if the RDD contains no elements at all. Note that an RDD may be empty
even when it has at least 1 partition.
>>> sc.parallelize([]).isEmpty()
True
>>> sc.parallelize([1]).isEmpty()
False
join(other, numPartitions=None )
Return an RDD containing all pairs of elements with matching keys in self and other.
Each pair of elements will be returned as a (k, (v1, v2)) tuple, where (k, v1) is in self and (k, v2) is
in other.
Performs a hash join across the cluster.
keyBy(f)
Creates tuples of the elements in this RDD by applying f.
keys()
Return an RDD with the keys of each tuple.
leftOuterJoin(other, numPartitions=None )
Perform a left outer join of self and other.
For each element (k, v) in self, the resulting RDD will either contain all pairs (k, (v, w)) for w in
other, or the pair (k, (v, None)) if no elements in other have key k.
Hashpartitions the resulting RDD into the given number of partitions.
lookup(key)
Return the list of values in the RDD for key key. This operation is done efficiently if the RDD has a
known partitioner by only searching the partition that the key maps to.
>>> l = range(1000)
>>> rdd = sc.parallelize(zip(l, l), 10)
>>> rdd.lookup(42) # slow
[42]
>>> sorted = rdd.sortByKey()
>>> sorted.lookup(42) # fast
[42]
>>> sorted.lookup(1024)
[]
>>> rdd2 = sc.parallelize([(('a', 'b'), 'c')]).groupByKey()
>>> list(rdd2.lookup(('a', 'b'))[0])
['c']
map(f, preservesPartitioning=False )
Return a new RDD by applying a function to each element of this RDD.
mapPartitions(f, preservesPartitioning=False )
Return a new RDD by applying a function to each partition of this RDD.
mapPartitionsWithIndex(f, preservesPartitioning=False )
Return a new RDD by applying a function to each partition of this RDD, while tracking the index of
the original partition.
mapPartitionsWithSplit(f, preservesPartitioning=False )
Deprecated: use mapPartitionsWithIndex instead.
Return a new RDD by applying a function to each partition of this RDD, while tracking the index of
the original partition.
mapValues(f)
Pass each value in the keyvalue pair RDD through a map function without changing the keys; this
also retains the original RDD’s partitioning.
max(key=None )
Find the maximum item in this RDD.
mean()
Compute the mean of this RDD’s elements.
meanApprox(timeout, confidence=0.95 )
Approximate operation to return the mean within a timeout or meet the confidence.
min(key=None )
Find the minimum item in this RDD.
name()
Return the name of this RDD.
partitionBy(numPartitions, partitionFunc=<function portable_hash at 0x7f1ac7340578>)
Return a copy of the RDD partitioned using the specified partitioner.
persist(storageLevel=StorageLevel(False, True, False, False, 1) )
Set this RDD’s storage level to persist its values across operations after the first time it is
computed. This can only be used to assign a new storage level if the RDD does not have a storage
level set yet. If no storage level is specified defaults to ( MEMORY_ONLY_SER).
pipe(command, env=None, checkCode=False )
Return an RDD created by piping elements to a forked external process.
randomSplit(weights, seed=None )
Randomly splits this RDD with the provided weights.
reduce(f)
Reduces the elements of this RDD using the specified commutative and associative binary
operator. Currently reduces partitions locally.
reduceByKey(func, numPartitions=None, partitionFunc=<function portable_hash at
0x7f1ac7340578>)
Merge the values for each key using an associative reduce function.
This will also perform the merging locally on each mapper before sending results to a reducer,
similarly to a “combiner” in MapReduce.
Output will be partitioned with numPartitions partitions, or the default parallelism level if
numPartitions is not specified. Default partitioner is hashpartition.
reduceByKeyLocally(func)
Merge the values for each key using an associative reduce function, but return the results
immediately to the master as a dictionary.
This will also perform the merging locally on each mapper before sending results to a reducer,
similarly to a “combiner” in MapReduce.
repartition(numPartitions)
Return a new RDD that has exactly numPartitions partitions.
Can increase or decrease the level of parallelism in this RDD. Internally, this uses a shuffle to
redistribute data. If you are decreasing the number of partitions in this RDD, consider using
coalesce, which can avoid performing a shuffle.
repartitionAndSortWithinPartitions(numPartitions=None, partitionFunc=<function
portable_hash at 0x7f1ac7340578>, ascending=True, keyfunc=<function <lambda> at
0x7f1ac7345cf8>)
Repartition the RDD according to the given partitioner and, within each resulting partition, sort
records by their keys.
>>> rdd = sc.parallelize([(0, 5), (3, 8), (2, 6), (0, 8), (3, 8), (1, 3)])
>>> rdd2 = rdd.repartitionAndSortWithinPartitions(2, lambda x: x % 2, 2)
>>> rdd2.glom().collect()
[[(0, 5), (0, 8), (2, 6)], [(1, 3), (3, 8), (3, 8)]]
rightOuterJoin(other, numPartitions=None )
Perform a right outer join of self and other.
For each element (k, w) in other, the resulting RDD will either contain all pairs (k, (v, w)) for v in
this, or the pair (k, (None, w)) if no elements in self have key k.
Hashpartitions the resulting RDD into the given number of partitions.
sample(withReplacement, fraction, seed=None )
Return a sampled subset of this RDD.
sampleByKey(withReplacement, fractions, seed=None )
Return a subset of this RDD sampled by key (via stratified sampling). Create a sample of this RDD
using variable sampling rates for different keys as specified by fractions, a key to sampling rate
map.
sampleStdev()
Compute the sample standard deviation of this RDD’s elements (which corrects for bias in
estimating the standard deviation by dividing by N1 instead of N).
sampleVariance()
Compute the sample variance of this RDD’s elements (which corrects for bias in estimating the
variance by dividing by N1 instead of N).
saveAsHadoopDataset(conf, keyConverter=None, valueConverter=None )
Output a Python RDD of keyvalue pairs (of form RDD[(K, V)]) to any Hadoop file system, using the
old Hadoop OutputFormat API (mapred package). Keys/values are converted for output using
either user specified converters or, by default,
org.apache.spark.api.python.JavaToWritableConverter.
saveAsHadoopFile(path, outputFormatClass, keyClass=None, valueClass=None,
keyConverter=None, valueConverter=None, conf=None, compressionCodecClass=None )
Output a Python RDD of keyvalue pairs (of form RDD[(K, V)]) to any Hadoop file system, using the
old Hadoop OutputFormat API (mapred package). Key and value types will be inferred if not
specified. Keys and values are converted for output using either user specified converters or
org.apache.spark.api.python.JavaToWritableConverter. The conf is applied on top of the base
Hadoop conf associated with the SparkContext of this RDD to create a merged Hadoop
MapReduce job configuration for saving the data.
saveAsNewAPIHadoopDataset(conf, keyConverter=None, valueConverter=None )
Output a Python RDD of keyvalue pairs (of form RDD[(K, V)]) to any Hadoop file system, using the
new Hadoop OutputFormat API (mapreduce package). Keys/values are converted for output using
either user specified converters or, by default,
org.apache.spark.api.python.JavaToWritableConverter.
saveAsNewAPIHadoopFile(path, outputFormatClass, keyClass=None, valueClass=None,
keyConverter=None, valueConverter=None, conf=None )
Output a Python RDD of keyvalue pairs (of form RDD[(K, V)]) to any Hadoop file system, using the
new Hadoop OutputFormat API (mapreduce package). Key and value types will be inferred if not
specified. Keys and values are converted for output using either user specified converters or
org.apache.spark.api.python.JavaToWritableConverter. The conf is applied on top of the base
Hadoop conf associated with the SparkContext of this RDD to create a merged Hadoop
MapReduce job configuration for saving the data.
saveAsPickleFile(path, batchSize=10 )
Save this RDD as a SequenceFile of serialized objects. The serializer used is
pyspark.serializers.PickleSerializer, default batch size is 10.
saveAsSequenceFile(path, compressionCodecClass=None )
Output a Python RDD of keyvalue pairs (of form RDD[(K, V)]) to any Hadoop file system, using the
org.apache.hadoop.io.Writable types that we convert from the RDD’s key and value types. The
mechanism is as follows:
1. Pyrolite is used to convert pickled Python RDD into RDD of Java objects.
2. Keys and values of this Java RDD are converted to Writables and written out.
saveAsTextFile(path, compressionCodecClass=None )
Save this RDD as a text file, using string representations of elements.
Using compressionCodecClass
setName(name )
Assign a name to this RDD.
sortBy(keyfunc, ascending=True, numPartitions=None )
Sorts this RDD by the given keyfunc
>>> tmp = [('a', 1), ('b', 2), ('1', 3), ('d', 4), ('2', 5)]
>>> sc.parallelize(tmp).sortBy(lambda x: x[0]).collect()
[('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]
>>> sc.parallelize(tmp).sortBy(lambda x: x[1]).collect()
[('a', 1), ('b', 2), ('1', 3), ('d', 4), ('2', 5)]
sortByKey(ascending=True, numPartitions=None, keyfunc=<function <lambda> at
0x7f1ac7345de8>)
Sorts this RDD, which is assumed to consist of (key, value) pairs. # noqa
>>> tmp = [('a', 1), ('b', 2), ('1', 3), ('d', 4), ('2', 5)]
>>> sc.parallelize(tmp).sortByKey().first()
('1', 3)
>>> sc.parallelize(tmp).sortByKey(True, 1).collect()
[('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]
>>> sc.parallelize(tmp).sortByKey(True, 2).collect()
[('1', 3), ('2', 5), ('a', 1), ('b', 2), ('d', 4)]
>>> tmp2 = [('Mary', 1), ('had', 2), ('a', 3), ('little', 4), ('lamb', 5)]
>>> tmp2.extend([('whose', 6), ('fleece', 7), ('was', 8), ('white', 9)])
>>> sc.parallelize(tmp2).sortByKey(True, 3, keyfunc=lambda k: k.lower()).collect()
[('a', 3), ('fleece', 7), ('had', 2), ('lamb', 5),...('white', 9), ('whose', 6)]
stats()
Return a StatCounter object that captures the mean, variance and count of the RDD’s elements in
one operation.
stdev()
Compute the standard deviation of this RDD’s elements.
subtract(other, numPartitions=None )
Return each value in self that is not contained in other.
subtractByKey(other, numPartitions=None )
Return each (key, value) pair in self that has no pair with matching key in other.
sum()
Add up the elements in this RDD.
sumApprox(timeout, confidence=0.95 )
Approximate operation to return the sum within a timeout or meet the confidence.
take(num)
Take the first num elements of the RDD.
It works by first scanning one partition, and use the results from that partition to estimate the
number of additional partitions needed to satisfy the limit.
Translated from the Scala implementation in RDD#take().
takeSample(withReplacement, num, seed=None )
Return a fixedsize sampled subset of this RDD.
toDebugString()
A description of this RDD and its recursive dependencies for debugging.
toLocalIterator()
Return an iterator that contains all of the elements in this RDD. The iterator will consume as much
memory as the largest partition in this RDD. >>> rdd = sc.parallelize(range(10)) >>> [x for x in
rdd.toLocalIterator()] [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
top(num, key=None )
Get the top N elements from a RDD.
Note: It returns the list sorted in descending order.
treeAggregate(zeroValue, seqOp, combOp, depth=2 )
Aggregates the elements of this RDD in a multilevel tree pattern.
treeReduce(f, depth=2 )
Reduces the elements of this RDD in a multilevel tree pattern.
union(other )
Return the union of this RDD and another one.
unpersist()
Mark the RDD as nonpersistent, and remove all blocks for it from memory and disk.
values()
Return an RDD with the values of each tuple.
variance()
Compute the variance of this RDD’s elements.
zip(other )
Zips this RDD with another one, returning keyvalue pairs with the first element in each RDD
second element in each RDD, etc. Assumes that the two RDDs have the same number of partitions
and the same number of elements in each partition (e.g. one was made through a map on the
other).
>>> x = sc.parallelize(range(0,5))
>>> y = sc.parallelize(range(1000, 1005))
>>> x.zip(y).collect()
[(0, 1000), (1, 1001), (2, 1002), (3, 1003), (4, 1004)]
zipWithIndex()
Zips this RDD with its element indices.
The ordering is first based on the partition index and then the ordering of items within each
partition. So the first item in the first partition gets index 0, and the last item in the last partition
receives the largest index.
This method needs to trigger a spark job when this RDD contains more than one partitions.
zipWithUniqueId()
Zips this RDD with generated unique Long ids.
Items in the kth partition will get ids k, n+k, 2*n+k, ..., where n is the number of partitions. So there
may exist gaps, but this method won’t trigger a spark job, which is different from zipWithIndex
class pyspark.StorageLevel(useDisk, useMemory, useOffHeap, deserialized, replication=1 )
Flags for controlling the storage of an RDD. Each StorageLevel records whether to use memory,
whether to drop the RDD to disk if it falls out of memory, whether to keep the data in memory in a
serialized format, and whether to replicate the RDD partitions on multiple nodes. Also contains static
constants for some commonly used storage levels, such as MEMORY_ONLY.
DISK_ONLY = StorageLevel(True, False, False, False, 1)
DISK_ONLY_2 = StorageLevel(True, False, False, False, 2)
MEMORY_AND_DISK = StorageLevel(True, True, False, True, 1)
MEMORY_AND_DISK_2 = StorageLevel(True, True, False, True, 2)
MEMORY_AND_DISK_SER = StorageLevel(True, True, False, False, 1)
MEMORY_AND_DISK_SER_2 = StorageLevel(True, True, False, False, 2)
MEMORY_ONLY = StorageLevel(False, True, False, True, 1)
MEMORY_ONLY_2 = StorageLevel(False, True, False, True, 2)
MEMORY_ONLY_SER = StorageLevel(False, True, False, False, 1)
MEMORY_ONLY_SER_2 = StorageLevel(False, True, False, False, 2)
OFF_HEAP = StorageLevel(False, False, True, False, 1)
class pyspark.Broadcast(sc=None, value=None, pickle_registry=None, path=None )
A broadcast variable created with SparkContext.broadcast(). Access its value through value.
Examples:
dump(value, f)
load(path )
unpersist(blocking=False )
Delete cached copies of this broadcast on the executors.
value
Return the broadcasted value
class pyspark.Accumulator(aid, value, accum_param)
A shared variable that can be accumulated, i.e., has a commutative and associative “add” operation.
Worker tasks on a Spark cluster can add values to an Accumulator with the += operator, but only the
driver program is allowed to access its value, using value. Updates from the workers get propagated
automatically to the driver program.
While SparkContext supports accumulators for primitive data types like int and float, users can also
define accumulators for custom types by providing a custom AccumulatorParam object. Refer to the
doctest of this module for an example.
add(term)
Adds a term to this accumulator’s value
value
Get the accumulator’s value; only usable in driver program
class pyspark.AccumulatorParam
Helper object that defines how to accumulate values of a given type.
addInPlace(value1, value2 )
Add two values of the accumulator’s data type, returning a new value; for efficiency, can also
update value1 in place and return it.
zero(value )
Provide a “zero value” for the type, compatible in dimensions with the provided value (e.g., a zero
vector)
class pyspark.MarshalSerializer
Serializes objects using Python’s Marshal serializer:
http://docs.python.org/2/library/marshal.html
This serializer is faster than PickleSerializer but supports fewer datatypes.
dumps(obj)
loads(obj)
class pyspark.PickleSerializer
Serializes objects using Python’s pickle serializer:
http://docs.python.org/2/library/pickle.html
This serializer supports nearly any Python object, but may not be as fast as more specialized serializers.
dumps(obj)
loads(obj, encoding=None )
class pyspark.StatusTracker(jtracker )
Lowlevel status reporting APIs for monitoring job and stage progress.
These APIs intentionally provide very weak consistency semantics; consumers of these APIs should be
prepared to handle empty / missing information. For example, a job’s stage ids may be known but the
status API may not have any information about the details of those stages, so getStageInfo could
potentially return None for a valid stage id.
To limit memory usage, these APIs only provide information on recent jobs / stages. These APIs will
provide information for the last spark.ui.retainedStages stages and spark.ui.retainedJobs jobs.
getActiveJobsIds()
Returns an array containing the ids of all active jobs.
getActiveStageIds()
Returns an array containing the ids of all active stages.
getJobIdsForGroup(jobGroup=None )
Return a list of all known jobs in a particular job group. If jobGroup is None, then returns all known
jobs that are not associated with a job group.
The returned list may contain running, failed, and completed jobs, and may vary across invocations
of this method. This method does not guarantee the order of the elements in its result.
getJobInfo(jobId )
Returns a SparkJobInfo object, or None if the job info could not be found or was garbage collected.
getStageInfo(stageId )
Returns a SparkStageInfo object, or None if the stage info could not be found or was garbage
collected.
class pyspark.SparkJobInfo
Exposes information about Spark Jobs.
class pyspark.SparkStageInfo
Exposes information about Spark Stages.
class pyspark.Profiler(ctx)
Note : DeveloperApi
PySpark supports custom profilers, this is to allow for different profilers to be used as well as outputting
to different formats than what is provided in the BasicProfiler.
A custom profiler has to define or inherit the following methods:
profile will produce a system profile of some sort. stats return the collected stats. dump dumps
the profiles to a path add adds a profile to the existing accumulated profile
The profiler class is chosen when creating a SparkContext
dump(id, path )
Dump the profile into path, id is the RDD id
profile(func)
Do profiling on the function func
show(id )
Print the profile stats to stdout, id is the RDD id
stats()
Return the collected profiling stats (pstats.Stats)
class pyspark.BasicProfiler(ctx)
BasicProfiler is the default profiler, which is implemented based on cProfile and Accumulator
profile(func)
Runs and profiles the method to_profile passed in. A profile object is returned.
stats()