Skip to content

REF: Simplify Datetimelike constructor dispatching #23140

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 22 commits into from
Closed

REF: Simplify Datetimelike constructor dispatching #23140

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
f13cc58
Avoid non-public constructors
jbrockmendel Oct 13, 2018
4188ec7
simplify and de-duplicate _generate_range
jbrockmendel Oct 13, 2018
7804f1b
Check for invalid axis kwarg
jbrockmendel Oct 13, 2018
a4775f4
Move some EA properties up to mixins
jbrockmendel Oct 13, 2018
8ee34fa
implement basic TimedeltaArray tests
jbrockmendel Oct 13, 2018
78943c1
clean up PeriodArray constructor, with tests
jbrockmendel Oct 13, 2018
aa71383
make PeriodArray.__new__ more grown-up
jbrockmendel Oct 13, 2018
eae8389
Remove unused kwargs from TimedeltaArray.__new__
jbrockmendel Oct 13, 2018
e871733
revert change that broke tests
jbrockmendel Oct 13, 2018
7840f91
Fixup whitespace
jbrockmendel Oct 14, 2018
ec50b0b
helper function for axis validation
jbrockmendel Oct 14, 2018
eb7a6b6
suggested clarifications
jbrockmendel Oct 14, 2018
32c6391
Merge branch 'dlike8' of https://github.com/jbrockmendel/pandas into …
jbrockmendel Oct 14, 2018
c903917
Merge branch 'master' of https://github.com/pandas-dev/pandas into dl…
jbrockmendel Oct 14, 2018
b97ec96
move axis validation to nv
jbrockmendel Oct 14, 2018
11db555
Merge branch 'master' of https://github.com/pandas-dev/pandas into dl…
jbrockmendel Oct 14, 2018
147de57
revert some removals
jbrockmendel Oct 14, 2018
7c4d281
Merge branch 'master' of https://github.com/pandas-dev/pandas into dl…
jbrockmendel Oct 15, 2018
b90f421
catch too-negative values
jbrockmendel Oct 15, 2018
dc4f474
Roll validate_minmax_axis into existing validate functions
jbrockmendel Oct 15, 2018
46d5e64
fixup typo
jbrockmendel Oct 15, 2018
b5827c7
Merge branch 'master' of https://github.com/pandas-dev/pandas into dl…
jbrockmendel Oct 17, 2018
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Move some EA properties up to mixins
  • Loading branch information
@jbrockmendel
jbrockmendel committed Oct 13, 2018
commit a4775f484249eab4945f64c6be5b89a2425300cb
14 changes: 13 additions & 1 deletion pandas/core/arrays/datetimelike.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@
from pandas._libs.tslibs.period import (
Period, DIFFERENT_FREQ_INDEX, IncompatibleFrequency)

from pandas.util._decorators import deprecate_kwarg
from pandas.errors import NullFrequencyError, PerformanceWarning
from pandas import compat

Expand Down Expand Up @@ -39,7 +40,6 @@
from pandas.core.algorithms import checked_add_with_arr

from .base import ExtensionOpsMixin
from pandas.util._decorators import deprecate_kwarg


def _make_comparison_op(cls, op):
Expand Down Expand Up @@ -143,6 +143,10 @@ def asi8(self):
# ------------------------------------------------------------------
# Array-like Methods

@property
def ndim(self):
return len(self.shape)

@property
def shape(self):
return (len(self),)
Expand All @@ -151,6 +155,10 @@ def shape(self):
def size(self):
return np.prod(self.shape)

@property
def nbytes(self):
return self._ndarray_values.nbytes

def __len__(self):
return len(self._data)

Expand Down Expand Up @@ -211,6 +219,10 @@ def astype(self, dtype, copy=True):
# ------------------------------------------------------------------
# Null Handling

def isna(self):
# EA Interface
return self._isnan
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it needed to have the _isnan concept on the arrays? We use it in some internal methods on the Index class, but for Arrays it seems to me additional complexity compared to simply defining isna appropriately on each Array ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed elsewhere; can we mark as resolved?


@property # NB: override with cache_readonly in immutable subclasses
def _isnan(self):
""" return if each value is nan"""
Expand Down
21 changes: 0 additions & 21 deletions pandas/core/indexes/datetimes.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -380,27 +380,6 @@ def tz(self, value):
raise AttributeError("Cannot directly set timezone. Use tz_localize() "
"or tz_convert() as appropriate")

@property
def size(self):
# TODO: Remove this when we have a DatetimeTZArray
# Necessary to avoid recursion error since DTI._values is a DTI
# for TZ-aware
return self._ndarray_values.size
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you removing those? Those will need to be added back once we do the actual index/array split anyway, as they will be calling in the underlying array?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are you removing those? Those will need to be added back

Because I am OK with needing to add them back in a few days (hopefully)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But can you then try to explain me what the advantage is of moving it now?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. To make it clear what still needs to be moved/implemented at the Array level. e.g. Tom's PeriodArray PR implements some things in PeriodArray that should instead be in DatetimeLikeArrayMixin. Moving these prevents this kind of mixup.

  2. Because there are already a bunch of things that are going to need to be inherited from self.values, its better to get them all in one place and do that all at once.

  3. Because in the next pass I'll be implementing a decorator to do something like:

# TODO: enable this decorator once Datetime/Timedelta/PeriodIndex .values
#   points to a pandas ExtensionArray
# @inherit_from_values(["ndim", "shape", "size", "nbytes",
#                       "asi8", "freq", "freqstr"])
class DatetimeIndexOpsMixin(DatetimeLikeArrayMixin):

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moving these prevents this kind of mixup.

As long as one of the index classes is still inheriting from the ArrayMixin, there will be wrong / strange mixups, that need to be cleaned up

Because in the next pass I'll be implementing a decorator to do something like:

But how would you do that if the underlying values don't yet have those attributes, because it is not yet our internal array class?

And why not move them when implementing such a decorator? Then you actually have overview of the full changes.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You have sufficiently frustrated me into reverting this so we can move this down the field.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jorisvandenbossche if you're still up, can you take a look at the newest push and verify that the parts you have a problem with have been removed?


@property
def shape(self):
# TODO: Remove this when we have a DatetimeTZArray
# Necessary to avoid recursion error since DTI._values is a DTI
# for TZ-aware
return self._ndarray_values.shape

@property
def nbytes(self):
# TODO: Remove this when we have a DatetimeTZArray
# Necessary to avoid recursion error since DTI._values is a DTI
# for TZ-aware
return self._ndarray_values.nbytes

@classmethod
def _cached_range(cls, start=None, end=None, periods=None, freq=None,
name=None):
Expand Down
10 changes: 0 additions & 10 deletions pandas/core/indexes/period.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -314,16 +314,6 @@ def __array_wrap__(self, result, context=None):
# cannot pass _simple_new as it is
return self._shallow_copy(result, freq=self.freq, name=self.name)

@property
def size(self):
# Avoid materializing self._values
return self._ndarray_values.size

@property
def shape(self):
# Avoid materializing self._values
return self._ndarray_values.shape

@property
def _formatter_func(self):
return lambda x: "'%s'" % x
Expand Down