Skip to content

Conversation

natmokval
Copy link
Contributor

xref #53607

enforced deprecation of NDFrame.interpolate with "ffill” / "bfill” / “pad” / “backfill" methods

@natmokval natmokval added Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate Clean labels Mar 17, 2024
@natmokval natmokval marked this pull request as ready for review March 17, 2024 21:16
@@ -7806,14 +7791,9 @@ def interpolate(
fillna_methods = ["ffill", "bfill", "pad", "backfill"]
if method.lower() in fillna_methods:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you just remove this entire if branch? A bad method should raise later on

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for reviewing this PR. I removed this if branch and changed the error message in the if branch below. Just one question: we didn't have a test for the previous ValueError message:

"'fill_value' is not a valid keyword for "
f"{type(self).__name__}.interpolate with method from "
f"{fillna_methods}"

Should I add the test?

obj, should_transpose = (self.T, True) if axis == 1 else (self, False)
# GH#53631
if np.any(obj.dtypes == object):
raise TypeError(
f"{type(self).__name__} cannot interpolate with object dtype."
)

if method in fillna_methods and "fill_value" in kwargs:
else:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this case isn't needed either anymore either. The obj._mgr.interpolate call below should raise on invalid arguments. (Also note the # TODO(3.0): remove this case comment below which you can address in this PR)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I moved this check for the bad methods below and removed the useless check (the one with the note # TODO(3.0): remove this case)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just to be clear, here's the diff of the change I think you can fully clean up (I think we don't need any logic dealing with checking for a fillna_method):

--- a/pandas/core/generic.py
+++ b/pandas/core/generic.py
@@ -7797,30 +7797,11 @@ class NDFrame(PandasObject, indexing.IndexingMixin):
         if not isinstance(method, str):
             raise ValueError("'method' should be a string, not None.")
 
-        fillna_methods = ["ffill", "bfill", "pad", "backfill"]
-        if method.lower() in fillna_methods:
-            # GH#53581
-            warnings.warn(
-                f"{type(self).__name__}.interpolate with method={method} is "
-                "deprecated and will raise in a future version. "
-                "Use obj.ffill() or obj.bfill() instead.",
-                FutureWarning,
-                stacklevel=find_stack_level(),
-            )
-            obj, should_transpose = self, False
-        else:
-            obj, should_transpose = (self.T, True) if axis == 1 else (self, False)
-                    f"{type(self).__name__} cannot interpolate with object dtype."
-                )
-
-        if method in fillna_methods and "fill_value" in kwargs:
-            raise ValueError(
-                "'fill_value' is not a valid keyword for "
-                f"{type(self).__name__}.interpolate with method from "
-                f"{fillna_methods}"
+        obj, should_transpose = (self.T, True) if axis == 1 else (self, False)
+        # GH#53631
+        if np.any(obj.dtypes == object):
+            raise TypeError(
+                f"{type(self).__name__} cannot interpolate with object dtype."
             )
 
         if isinstance(obj.index, MultiIndex) and method != "linear":
@@ -7830,34 +7811,16 @@ class NDFrame(PandasObject, indexing.IndexingMixin):
 
         limit_direction = missing.infer_limit_direction(limit_direction, method)
 
-        if method.lower() in fillna_methods:
-            # TODO(3.0): remove this case
-            # TODO: warn/raise on limit_direction or kwargs which are ignored?
-            #  as of 2023-06-26 no tests get here with either
-            if not self._mgr.is_single_block and axis == 1:
-                # GH#53898
-                if inplace:
-                    raise NotImplementedError()
-                obj, axis, should_transpose = self.T, 1 - axis, True
-
-            new_data = obj._mgr.pad_or_backfill(
-                method=method,
-                axis=self._get_block_manager_axis(axis),
-                limit=limit,
-                limit_area=limit_area,
-                inplace=inplace,
-            )
-        else:
-            index = missing.get_interp_index(method, obj.index)
-            new_data = obj._mgr.interpolate(
-                method=method,
-                index=index,
-                limit=limit,
-                limit_direction=limit_direction,
-                limit_area=limit_area,
-                inplace=inplace,
-                **kwargs,
-            )
+        index = missing.get_interp_index(method, obj.index)
+        new_data = obj._mgr.interpolate(
+            method=method,
+            index=index,
+            limit=limit,
+            limit_direction=limit_direction,
+            limit_area=limit_area,
+            inplace=inplace,
+            **kwargs,
+        )

Copy link
Contributor Author

@natmokval natmokval Mar 20, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahhh, I think I understand now. I removed any logic dealing with checking for a fillna_methods from NDFrame.interpolate.
I think we need to check if method is valid in interpolate in class Block and to do the same check in get_interp_index. I corrected these methods and updated error messages in tests.

@@ -1383,6 +1383,9 @@ def interpolate(
limit_area: Literal["inside", "outside"] | None = None,
**kwargs,
) -> list[Block]:
valid = missing.NP_METHODS + missing.SP_METHODS
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A better place for this check (and the one you added in get_interp_index) in is probably in _interpolate_scipy_wrapper.

We want to make these checks at the lowest level possible

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks, I moved this check from pandas/core/internals/blocks.py to _interpolate_scipy_wrapper in pandas/core/missing.py.

I left the check in get_interp_index, otherwise, several tests fail.

Comment on lines 592 to 594
valid = NP_METHODS + SP_METHODS
if method not in valid:
raise ValueError(f"Can not interpolate with method={method}.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of this, the terp = alt_methods[method] below should be changed to:

terp = alt_methods.get(method, None):
if terp is None:
    raise ValueError(f"Can not interpolate with method={method}.")

@mroeschke mroeschke added this to the 3.0 milestone Mar 22, 2024
@mroeschke mroeschke merged commit 677a4ea into pandas-dev:main Mar 22, 2024
@mroeschke
Copy link
Member

Thanks for sticking with this @natmokval!

@natmokval
Copy link
Contributor Author

Thanks @mroeschke for helping me with this PR. Enforcing this deprecation was more complicated than I expected.

pmhatre1 pushed a commit to pmhatre1/pandas-pmhatre1 that referenced this pull request May 7, 2024
…ad/backfill` methods (pandas-dev#57869)

* enforce deprecation of interpolate with ffill, bfill-pad, backfill methods

* remove redundant if branch

* remove unuseful cheek from interpolate

* move checking for a fillna_method from NDFrame.interpolate to Block.interpolate, correct tests

* remove the check from Block.interpolate

* add a note to v3.0.0

* correct def _interpolate_scipy_wrapper: use alt_methods instead of valid
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Clean Missing-data np.nan, pd.NaT, pd.NA, dropna, isnull, interpolate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants