Nitya CloudTech Pvt Ltd.

PySpark Scenario-Based
Interview Questions &
Answers

http://www.nityacloudtech.com/ @nityacloudtech
 Nitya CloudTech Pvt Ltd.

Data Integration & Handling External Data

Interviewer: You need to read data from multiple JSON files stored
in a directory, each with slight variations in structure. How would you
handle schema mismatches in PySpark?

Candidate: I’d use the mergeSchema option when reading JSON files,
which enables PySpark to handle slight schema variations by merging
schemas from each file. This way, fields from different files will be
included in the final schema as optional fields, preventing
mismatches.

Example:
df = spark.read.option("mergeSchema",
"true").json("/path/to/json/files")

Interviewer: Explain how you would read data from a PostgreSQL

database into a PySpark DataFrame, considering options for filtering
data at the source.

Candidate: I’d use jdbc to read data from PostgreSQL, passing filter
criteria as part of the dbtable query to reduce data at the source level.
This minimizes the amount of data transferred, improving
performance.

Example:
url = "jdbc:postgresql://host:port/database"
properties = {"user": "username", "password": "password",
"driver": "org.postgresql.Driver"}

df = spark.read.jdbc(url=url, table="(SELECT * FROM table_name

WHERE condition) AS filtered_table", properties=properties)

http://www.nityacloudtech.com/ @nityacloudtech
 Nitya CloudTech Pvt Ltd.

Interviewer:
You have a DataFrame with raw timestamp data in different formats.
How would you standardize these timestamps into a uniform format?

Candidate: I’d use to_timestamp with custom format strings to

standardize timestamps, handling each format with a conditional
check using when. I’d then use coalesce to merge the different formats
into a single timestamp column.

Example:
from pyspark.sql.functions import to_timestamp, coalesce, when

df = df.withColumn("standard_timestamp", coalesce(
to_timestamp("timestamp_column", "yyyy-MM-dd HH:mm:ss"),
to_timestamp("timestamp_column", "MM/dd/yyyy HH:mm:ss")
))

Interviewer: How would you write a DataFrame to a partitioned

Hive table, with partition columns as year and month?

Candidate: I’d use the partitionBy method with mode("overwrite")

if needed. Specifying year and month in the partitionBy call will
ensure that the data is written to corresponding Hive partitions.

Example:
df.write.mode("overwrite").partitionBy("year",
"month").saveAsTable("database.table_name")

Interviewer: Describe how you would ingest a continuously updating

log file into PySpark and perform real-time transformations.

Candidate: I’d use Spark Structured Streaming to read the log file as
a stream and apply transformations on the streaming DataFrame. The
readStream API can be used to treat a file directory as a streaming
source.

http://www.nityacloudtech.com/ @nityacloudtech
 Nitya CloudTech Pvt Ltd.
Example:
df = spark.readStream.format("text").option("path",
"/path/to/log/files").load()
transformed_df = df.withColumn("processed_column",
some_transformation_function(df["column"]))

Interviewer: Explain how to handle streaming data from Kafka in

PySpark to perform real-time analysis of website traffic data.

Candidate: I’d configure PySpark’s Kafka integration to read from

Kafka topics, then apply structured streaming transformations to
analyze traffic in real-time. Spark’s readStream can connect to Kafka
by specifying kafka as the format and providing
kafka.bootstrap.servers and subscribe options.

Example:
df = spark.readStream.format("kafka") \
.option("kafka.bootstrap.servers", "server:port") \
.option("subscribe", "topic_name") \
.load()

Interviewer: You are given a large XML file that needs to be

processed. Explain how you would convert this data to a DataFrame
in PySpark.

Candidate: I’d use Spark XML library to parse the XML file into a
DataFrame, specifying the root tag and row tag to correctly interpret
the structure.

Example:
from pyspark.sql import SparkSession

df = spark.read.format("com.databricks.spark.xml") \
.option("rowTag", "yourRowTag") \
.load("path/to/file.xml")

http://www.nityacloudtech.com/ @nityacloudtech
 Nitya CloudTech Pvt Ltd.

Interviewer:
Describe how you would integrate data from a NoSQL database, such
as MongoDB, into PySpark for analysis.

Candidate: PySpark’s MongoDB connector enables reading data

directly from MongoDB collections. I’d use read.format("mongodb")
with URI configurations to connect and load data into a DataFrame.

Interviewer: You need to write a DataFrame to an Amazon S3

bucket in a compressed format. Explain how you would do this in
PySpark.

Candidate: I’d use the write method with the format set to parquet
or csv, and specify a compression codec like gzip or snappy.

Example:
df.write.format("parquet").option("compression",
"gzip").save("s3://bucket_name/path")

Interviewer: How would you set up incremental data loading for a

PySpark job that processes data daily?

Candidate: I’d use a date-based filter condition to only read data

added after the last load time. Incremental loading can be achieved
with merge operations if necessary, or simply appending new data to a
base table in an ETL pipeline.

Optimization Techniques

Interviewer: Explain how you would avoid skew in your data if you
observe certain keys appear very frequently in a join operation.

http://www.nityacloudtech.com/ @nityacloudtech
 Nitya CloudTech Pvt Ltd.
Candidate: I’d
use techniques like salting or skew join hints. Salting involves adding
a random value to the skewed key to spread it across multiple
partitions.

Interviewer: You have multiple small DataFrames that need to be

joined. Describe how you would optimize these joins.

Candidate: For joining multiple small DataFrames, I’d use broadcast

joins by calling broadcast on the smaller DataFrames to distribute
them across all nodes, minimizing shuffle operations.

Interviewer: Explain how to optimize PySpark’s memory usage

when performing aggregations on a large DataFrame.

Candidate: I’d reduce memory usage by caching intermediate results,

using persist with an appropriate storage level (like DISK_ONLY) and
ensuring data is serialized efficiently.

Interviewer: How would you use partition pruning to speed up

queries on a large partitioned table in PySpark?

Candidate: Partition pruning allows Spark to only read relevant

partitions, thus reducing I/O. I’d add filters on partition columns to
trigger pruning and skip unnecessary data.

Interviewer: Describe a scenario where you would switch from the

default Tungsten execution engine to the Catalyst optimizer and why.

Candidate: While the Tungsten engine optimizes physical execution,

switching to Catalyst might be more beneficial for complex logical

http://www.nityacloudtech.com/ @nityacloudtech
query Nitya CloudTech Pvt Ltd.
optimizations,
particularly with deeply nested or SQL-heavy transformations.

Interviewer: Explain how to tune PySpark configurations like

spark.executor.memory and spark.driver.memory for large datasets.

Candidate: I’d adjust these based on available cluster resources,

workload characteristics, and data volume, balancing
spark.executor.memory and spark.driver.memory so that executors
have enough memory for computation and don’t overload the driver.

Interviewer: Describe how you would monitor and adjust the number
of shuffle partitions in a PySpark job.

Candidate: I’d use Spark UI to monitor the shuffle size and adjust
spark.sql.shuffle.partitions based on the stage shuffle size and
resource availability to optimize performance.

Interviewer: How would you avoid shuffle operations when

performing a groupBy operation in PySpark?

Candidate: I’d try to partition the data first by the grouping key using
repartition to align data in the same partition, reducing the need for
a shuffle.

Interviewer: You have a DataFrame with an expensive calculation

applied on each row. How would you optimize this calculation using
UDF caching?

http://www.nityacloudtech.com/ @nityacloudtech
 Nitya CloudTech Pvt Ltd.
Candidate: I’d
precompute the result of the UDF if it’s deterministic and cache the
result column using persist to avoid recalculating for each action on
the DataFrame.

Interviewer: Describe how to repartition data in PySpark when using

machine learning algorithms on a large dataset to improve
performance.

Candidate: I’d use repartition to ensure the data is spread evenly

across partitions, considering the number of cores and memory. For
machine learning, balanced partitions improve training performance
by avoiding skew and enhancing parallelism.

If you like the content, please consider supporting us by sharing it

with others who may benefit. Your support helps us continue creating
valuable resources!

http://www.nityacloudtech.com/ @nityacloudtech