Select columns in PySpark dataframe

PySpark DataFrame - Select all except one or a set of columns

Last Updated : 17 Jun, 2021

In this article, we are going to extract all columns except a set of columns or one column from Pyspark dataframe. For this, we will use the select(), drop() functions.

But first, let's create Dataframe for demonestration.

Python3

# importing module
import pyspark

# importing sparksession from pyspark.sql module
from pyspark.sql import SparkSession

# creating sparksession and giving an app name
spark = SparkSession.builder.appName('sparkdf').getOrCreate()

# list  of students  data
data = [["1", "sravan", "vignan"],
        ["2", "ojaswi", "vvit"],
        ["3", "rohith", "vvit"],
        ["4", "sridevi", "vignan"],
        ["1", "sravan", "vignan"],
        ["5", "gnanesh", "iit"]]

# specify column names
columns = ['student ID', 'student NAME', 'college']

# creating a dataframe from the lists of data
dataframe = spark.createDataFrame(data, columns)

print('Actual data in dataframe')
dataframe.show()

Output:

Method 1: Using drop() function

drop() is used to drop the columns from the dataframe.

Syntax: dataframe.drop('column_names')

Where dataframe is the input dataframe and column names are the columns to be dropped

Example: Python program to select data by dropping one column

Python3

# drop student id
dataframe.drop('student ID').show()

Output:

Example 2: Python program to drop more than one column(set of columns)

Python3

# drop student id and college
dataframe.drop('student ID','college').show()

Output:

Method 2: Using select() function

This function is used to select the columns from the dataframe

Syntax: dataframe.select(columns)

Where dataframe is the input dataframe and columns are the input columns

Example 1: Select one column from the dataframe.

Python3

# select student id 
dataframe.select('student ID').show()

Output:

Example 2: Python program to select two columns id and name

Python3

# select student id and student name
dataframe.select('student ID','student NAME').show()

Output:

Select columns in PySpark dataframe

gottumukkalabobby

Improve

Article Tags :

Practice Tags :

python

Similar Reads

Select all columns, except one given column in a Pandas DataFrame

DataFrame Data structure are the heart of Pandas library. DataFrames are basically two dimension Series object. They have rows and columns with rows representing the index and columns representing the content. Now, let's see how to Select all columns, except one given column in Pandas DataFrame in P

PySpark - Select Columns From DataFrame

In this article, we will discuss how to select columns from the pyspark dataframe. To do this we will use the select() function. Syntax: dataframe.select(parameter).show() where,Â dataframe is the dataframe nameparameter is the column(s) to be selectedshow() function is used to display the selected

How to select and order multiple columns in Pyspark DataFrame ?

In this article, we will discuss how to select and order multiple columns from a dataframe using pyspark in Python. For this, we are using sort() and orderBy() functions along with select() function. Methods UsedSelect(): This method is used to select the part of dataframe columns and return a copy

Select columns in PySpark dataframe

In this article, we will learn how to select columns in PySpark dataframe. Function used: In PySpark we can select columns using the select() function. The select() function allows us to select single or multiple columns in different formats.Â Syntax: dataframe_name.select( columns_names ) Note: We

Select specific column of PySpark dataframe with its position

In this article, we will discuss how to select a specific column by using its position from a pyspark dataframe in Python. For this, we will use dataframe.columns() method inside dataframe.select() method. Syntax: dataframe.select(dataframe.columns[column_number]).show() where, dataframe is the data

How to drop all columns with null values in a PySpark DataFrame ?

The pyspark.sql.DataFrameNaFunctions class in PySpark has many methods to deal with NULL/None values, one of which is the drop() function, which is used to remove/delete rows containing NULL values in DataFrame columns. You can also use df.dropna(), as shown in this article. You may drop all rows in

How to add a new column to a PySpark DataFrame ?

In this article, we will discuss how to add a new column to PySpark Dataframe. Create the first data frame for demonstration: Here, we will be creating the sample data frame which we will be used further to demonstrate the approach purpose. Python3 # importing module import pyspark # importing spark

Selecting only numeric or string columns names from PySpark DataFrame

In this article, we will discuss how to select only numeric or string column names from a Spark DataFrame. Methods Used:createDataFrame: This method is used to create a spark DataFrame.isinstance: This is a Python function used to check if the specified object is of the specified type.dtypes: It ret

How to delete columns in PySpark dataframe ?

In this article, we are going to delete columns in Pyspark dataframe. To do this we will be using the drop() function. This function can be used to remove values from the dataframe. Syntax: dataframe.drop('column name') Python code to create student dataframe with three columns: Python3 # importing

How to get name of dataframe column in PySpark ?

In this article, we will discuss how to get the name of the Dataframe column in PySpark.Â To get the name of the columns present in the Dataframe we are using the columns function through this function we will get the list of all the column names present in the Dataframe. Syntax: df.columns We can a