BUG: pd.read_sql() fails on SELECT * self join when handling date columns

[RevolutionTech]

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import sqlalchemy as sa
import pandas as pd


username = "..."
password = "..."
host = "..."
port = "..."
database = "..."

conn_string = sa.engine.url.URL.create(
    "postgresql",
    username=username,
    password=password,
    host=host,
    port=port,
    database=database
)
engine = sa.create_engine(conn_string)
conn = engine.connect()

sql_query = 'SELECT * FROM "person" AS p1 INNER JOIN "person" AS p2 ON p1.id = p2.id;'
df = pd.read_sql(sql_query, conn)

Issue Description

When performing a SQL self-join on a table that includes date columns, pd.read_sql() fails if SELECT * is used with ValueError: shape mismatch: value array of shape (0,1) could not be broadcast to indexing result of shape (1,1):

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/io/sql.py", line 628, in read_sql
    return pandas_sql.read_query(
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/io/sql.py", line 1594, in read_query
    frame = _wrap_result(
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/io/sql.py", line 177, in _wrap_result
    frame = _parse_date_columns(frame, parse_dates)
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/io/sql.py", line 158, in _parse_date_columns
    data_frame[col_name] = _handle_date_column(df_col, format=fmt)
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/core/frame.py", line 3612, in __setitem__
    self._set_item(key, value)
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/core/frame.py", line 3797, in _set_item
    self._set_item_mgr(key, value)
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/core/frame.py", line 3756, in _set_item_mgr
    self._iset_item_mgr(loc, value)
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/core/frame.py", line 3746, in _iset_item_mgr
    self._mgr.iset(loc, value)
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/core/internals/managers.py", line 1078, in iset
    blk.set_inplace(blk_locs, value_getitem(val_locs))
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/core/internals/blocks.py", line 360, in set_inplace
    self.values[locs] = values
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py", line 373, in __setitem__
    super().__setitem__(key, value)
  File "/Users/lucas/.virtualenvs/pandas-read-sql-bug/lib/python3.8/site-packages/pandas/core/arrays/_mixins.py", line 183, in __setitem__
    self._ndarray[key] = value
ValueError: shape mismatch: value array of shape (0,1) could not be broadcast to indexing result of shape (1,1)

The tables referenced in the example were created in a PostgreSQL database using the following SQL:

CREATE TABLE person
(
    id serial constraint person_pkey primary key,
    created_dt timestamp with time zone
);

INSERT INTO person
    VALUES (1, '2021-01-01T00:00:00Z');

Expected Behavior

I would expect pd.read_sql() to return a DataFrame here with two id columns and two created_dt columns. This is the behaviour I get when creating the table without a primary key:

CREATE TABLE person
(
    id int not null,
    created_dt timestamp with time zone
);

Installed Versions

INSTALLED VERSIONS

commit : 945c9ed
python : 3.8.2.final.0
python-bits : 64
OS : Darwin
OS-release : 20.6.0
Version : Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:20 PDT 2021; root:xnu-7195.141.6~3/RELEASE_ARM64_T8101
machine : arm64
processor : arm
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.4
numpy : 1.21.4
pytz : 2021.3
dateutil : 2.8.2
pip : 21.1.1
setuptools : 56.2.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : None
html5lib : None
pymysql : None
psycopg2 : 2.9.2 (dt dec pq3 ext lo64)
jinja2 : None
IPython : None
pandas_datareader: None
bs4 : None
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : None
numexpr : None
odfpy : None
openpyxl : None
pandas_gbq : None
pyarrow : None
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : 1.4.27
tables : None
tabulate : None
xarray : None
xlrd : None
xlwt : None
numba : None

[ParfaitG]

When running self-joins (or any joins) in SQL it is important to explicitly rename same named columns and avoid SELECT * FROM. What happens if you use column aliases?

sql_query = """\
    SELECT p1.id AS p1_id
         , p1.created_dt AS p1_created_dt
         , p2.id AS p2_id
         , p2.created_dt AS p2_created_dt
    FROM "person" AS p1 
    INNER JOIN "person" AS p2 
        ON p1.id = p2.id
"""

df = pd.read_sql(sql_query, conn)

[RevolutionTech]

Great callout, @ParfaitG! Indeed the bug does not occur if columns are explicitly selected. I've updated the issue to reflect the fact that the use of SELECT * is key to reproducing the bug.

Unfortunately though, in my use case explicitly listing out columns is insufficient as a workaround because my application does not have control over the incoming SQL.

[ParfaitG]

While this error message can be more clear at the problem of duplicate column names, I am not sure what else pandas can do. With read_sql, pandas cannot control the SQL or ORM query the user submits, so it cannot know in advance duplicate names will occur especially if explicit columns are not declared in SELECT clause.

Consider changing the pipeline of your application processing since SELECT * is really a convenience statement and should not be used for join queries for various reasons. See this discussion: Why is SELECT * considered harmful?. Recall one of the zens of python: explicit is better than implicit.

[RevolutionTech]

I guess the reason why I am perplexed by this issue is because pandas produces a DataFrame with multiple columns of the same name in every other scenario that I've encountered. It is the behaviour when the table does not have a primary key (see example in my original post) and it is also the behaviour when the table doesn't contain any datetimes:

CREATE TABLE person
(
    id serial constraint person_pkey primary key,
    name text
);

INSERT INTO person
    VALUES (1, 'John Smith');

...
>>> conn = engine.connect()
>>> sql_query = 'SELECT * FROM "person" AS p1 INNER JOIN "person" AS p2 ON p1.id = p2.id;'
>>> df = pd.read_sql(sql_query, conn)
>>> df
   id        name  id        name
0   1  John Smith   1  John Smith

Shouldn't the behaviour of pandas be consistent in these three scenario?

[vvilhonen]

This is a significant hindrance to exploratory analysis

[the21st]

I created a bug bounty for this issue: https://gitpay.me/#/task/645/bug-pd-read-sql-fails-on-select-self-join-when-handling-date-columns

[LeGmask]

Was looking at how current version is handling this, and look like no more exception, also, wondering if this issue is still of actuality ?

In fact, with this table (same thing with the first given table)

CREATE TABLE person
(
    id serial constraint person_pkey primary key,
    name text
);

INSERT INTO person
    VALUES (1, 'John Smith');
    

and this same code:

...
>>> conn = engine.connect()
>>> sql_query = 'SELECT * FROM "person" AS p1 INNER JOIN "person" AS p2 ON p1.id = p2.id;'
>>> df = pd.read_sql(sql_query, conn)
>>> df
   id        name
0   1  John Smith

The read_sql output is now a dataframe with single columns (you still can have multiple column by renaming columns).

I'm wondering if we should implement a way to duplicate columns, since the query give us duplicated columns, or, keep the actual way it's handling it ?