Retry Premium

Search
Home My Network Jobs Messaging Notifications Me Work Free

Picture Credit - Alex Riley

Change the data type of columns in

Pandas
Published on February 25, 2020

Mohit Sharma
DevOps Engineer @SpaceServicesAustralia | (CKA) Kubernetes Certified | 3 9 articles Follow
X AWS Certified | Open Source Contributor

You have three main options for converting types in pandas:

1. to_numeric() - provides functionality to safely convert non-numeric types (e.g. strings)

to a suitable numeric type. (See also to_datetime() and to_timedelta().)

2. astype() - convert (almost) any type to (almost) any other type (even if it's not
necessarily sensible to do so). Also allows you to convert to categorial types (very
useful).

3. infer_objects() - a utility method to convert object columns holding Python objects to a

pandas type if possible.

Read on for more detailed explanations and usage of each of these methods.

1. to_numeric()

The best way to convert one or more columns of a DataFrame to numeric values is to
use pandas.to_numeric().

This function will try to change non-numeric objects (such as strings) into integers or
floating-point numbers as appropriate.

Basic usage

The input to to_numeric() is a Series or a single column of a DataFrame.

>>> s = pd.Series(["8", 6, "7.5", 3, "0.9"]) # mixed string and numeric values

>>> s
0 8
1 6
2 7.5
3 3
4 0.9
dtype: object

>>> pd.to_numeric(s) # convert everything to float values

0 8.0
1 6.0
2 7.5
3 3.0
4 0.9
dtype: float64

As you can see, a new Series is returned. Remember to assign this output to a variable or
column name to continue using it:

# convert Series
my_series = pd.to_numeric(my_series)

# convert column "a" of a DataFrame

df["a"] = pd.to_numeric(df["a"])

You can also use it to convert multiple columns of a DataFrame via the apply() method:

# convert all columns of DataFrame

df = df.apply(pd.to_numeric) # convert all columns of DataFrame

# convert just columns "a" and "b"

df[["a", "b"]] = df[["a", "b"]].apply(pd.to_numeric)

As long as your values can all be converted, that's probably all you need.

Error handling

But what if some values can't be converted to a numeric type?

to_numeric() also takes an errors keyword argument that allows you to force non-numeric
values to be NaN, or simply ignore columns containing these values.

Here's an example using a Series of strings s which has the object dtype:

>>> s = pd.Series(['1', '2', '4.7', 'pandas', '10'])

>>> s
0 1
1 2
2 4.7
3 pandas
4 10
dtype: object

The default behaviour is to raise if it can't convert a value. In this case, it can't cope with the
string 'pandas':

>>> pd.to_numeric(s) # or pd.to_numeric(s, errors='raise')

ValueError: Unable to parse string

Rather than fail, we might want 'pandas' to be considered a missing/bad numeric value. We
can coerce invalid values to NaN as follows using the errors keyword argument:

>>> pd.to_numeric(s, errors='coerce')

0 1.0
1 2.0
2 4.7
3 NaN
4 10.0
dtype: float64

The third option for errors is just to ignore the operation if an invalid value is encountered:

>>> pd.to_numeric(s, errors='ignore')

# the original Series is returned untouched

This last option is particularly useful when you want to convert your entire DataFrame, but
don't know which of our columns can be converted reliably to a numeric type. In that case,
just write:

df.apply(pd.to_numeric, errors='ignore')

The function will be applied to each column of the DataFrame. Columns that can be
converted to a numeric type will be converted, while columns that cannot (e.g. they contain
non-digit strings or dates) will be left alone.

Downcasting

By default, conversion with to_numeric() will give you either an int64 or float64 dtype (or
whatever integer width is native to your platform).

That's usually what you want, but what if you wanted to save some memory and use a more
compact dtype, like float32, or int8?

to_numeric() gives you the option to downcast to either 'integer', 'signed', 'unsigned', 'float'.
Here's an example for a simple series s of integer type:

>>> s = pd.Series([1, 2, -7])

>>> s
0 1
1 2
2 -7
dtype: int64

Downcasting to 'integer' uses the smallest possible integer that can hold the values:

>>> pd.to_numeric(s, downcast='integer')

0 1
1 2
2 -7
dtype: int8

Downcasting to 'float' similarly picks a smaller than normal floating type:

>>> pd.to_numeric(s, downcast='float')

0 1.0
1 2.0
2 -7.0
dtype: float32

2. astype()

The astype() method enables you to be explicit about the dtype you want your DataFrame or
Series to have. It's very versatile in that you can try and go from one type to any other.

Basic usage

Just pick a type: you can use a NumPy dtype (e.g. np.int16), some Python types (e.g. bool),
or pandas-specific types (like the categorical dtype).

Call the method on the object you want to convert and astype() will try and convert it for
you:

# convert all DataFrame columns to the int64 dtype

df = df.astype(int)

# convert column "a" to int64 dtype and "b" to complex type

df = df.astype({"a": int, "b": complex})

# convert Series to float16 type

s = s.astype(np.float16)

# convert Series to Python strings

s = s.astype(str)

# convert Series to categorical type - see docs for more details

s = s.astype('category')

Notice I said "try" - if astype() does not know how to convert a value in the Series or
DataFrame, it will raise an error. For example, if you have a NaN or inf value you'll get an
error trying to convert it to an integer.

As of pandas 0.20.0, this error can be suppressed by passing errors='ignore'. Your original
object will be returned untouched.

Be careful

astype() is powerful, but it will sometimes convert values "incorrectly". For example:

>>> s = pd.Series([1, 2, -7])

>>> s
0 1
1 2
2 -7
dtype: int64

These are small integers, so how about converting to an unsigned 8-bit type to save
memory?

>>> s.astype(np.uint8)
0 1
1 2
2 249
dtype: uint8

The conversion worked, but the -7 was wrapped round to become 249 (i.e. 28 - 7)!

Trying to downcast using pd.to_numeric(s, downcast='unsigned') instead could help prevent

this error.

3. infer_objects()

Version 0.21.0 of pandas introduced the method infer_objects() for converting columns of a
DataFrame that have an object datatype to a more specific type (soft conversions).

For example, here's a DataFrame with two columns of an object type. One holds actual
integers and the other holds strings representing integers:

>>> df = pd.DataFrame({'a': [7, 1, 5], 'b': ['3','2','1']}, dtype='object')

>>> df.dtypes
a object
b object
dtype: object

Using infer_objects(), you can change the type of column 'a' to int64:

>>> df = df.infer_objects()
>>> df.dtypes
a int64
b object
dtype: object

Column 'b' has been left alone since its values were strings, not integers. If you wanted to try
and force the conversion of both columns to an integer type, you could
use df.astype(int) instead.

Do follow or connect with me for articles on AWS and Machine Learning Topics. If you are
interested in wanting particular topic comment below to let me know. Thank you.

Report this

Published by
Mohit Sharma 9 articles Follow
DevOps Engineer @SpaceServicesAustralia | (CKA) Kubernetes Certified |
3 X AWS Certified | Open Source Contributor
Published • 1y

Having a trouble in changing the data type of columns??

Get Started with my article

*" How to change the data type of columns in Pandas? "* or refresh your concept by reading my article.

You have three main options for converting types in pandas:

1. to_numeric() - provides functionality to safely convert non-numeric types (e.g. strings) to a suitable numeric
type.

2. astype() - convert (almost) any type to (almost) any other type (even if it's not necessarily sensible to do so).
Also allows you to convert to categorial types (very useful).

3. infer_objects() - a utility method to convert object columns holding Python objects to a pandas type if possible.

#python #imoisharma #dailycoding #datascience #pandas #linkedin #melbourne #australia #incedge

#sydney

Like Comment Share 54 · 5 comments

Reactions

+42

5 Comments
Most relevant

Add a comment…

Iustina I. • 3rd+ 2mo

Data scientist

this is very helpful, thank you!

Like · 1 Reply · 1 Reply

Mohit Sharma • 3rd+ 2mo

DevOps Engineer @SpaceServicesAustralia | (CKA) Kubernetes Certified | 3 X AWS Certified | Open …

anytime :)

Like Reply

Temitope Adenuga • 3rd+ 1mo

Health Physicist| Educator| Medical Physics Enthusiast

This was very helpful to what I was working on. Thank you

Like · 1 Reply

Load more comments

Mohit Sharma
DevOps Engineer @SpaceServicesAustralia | (CKA) Kubernetes Certified | 3 X AWS Certified | Open Source Contributor

Follow

More from Mohit Sharma

Set up a Continuous Deployment Text Parsing in Python with US- Time Zone Lifecycle of Data Science Projects
Pipeline in less than 15 min Patent Data
Mohit Sharma on LinkedIn Mohit Sharma on LinkedIn
Mohit Sharma on LinkedIn Mohit Sharma on LinkedIn

See all 9 articles

About Accessibility Talent Solutions Questions? Select Language

Visit our Help Center.
Community Guidelines Careers Marketing Solutions English (English)

Privacy & Terms Ad Choices Advertising Manage your account and privacy
Go to your Settings.
Sales Solutions Mobile Small Business

Safety Center

LinkedIn Corporation © 2021

Messaging