[SPARK-39314][PS] Respect ps.concat sort parameter to follow pandas behavior #36711
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
also cc @HyukjinKwon @xinrong-databricks ready for review |
HyukjinKwon
approved these changes
Jun 2, 2022
|
Merged to master. |
HyukjinKwon
pushed a commit
that referenced
this pull request
Jun 10, 2022
… changes ### What changes were proposed in this pull request? Add migration guide for pandas 1.4 behavior changes: * SPARK-39054 #36581: In Spark 3.4, if Pandas on Spark API `Groupby.apply`'s `func` parameter return type is not specified and `compute.shortcut_limit` is set to `0`, the sampling rows will be set to 2 (ensure sampling rows always >= 2) to make sure infer schema is accurate. * SPARK-38822 #36168 In Spark 3.4, if Pandas on Spark API `Index.insert` is out of bounds, will rasied IndexError with `index {} is out of bounds for axis 0 with size {}` to follow pandas 1.4 behavior. * SPARK-38857 #36159 In Spark 3.4, the series name will be preserved in Pandas on Spark API `Series.mode` to follow pandas 1.4 behavior. * SPARK-38859 #36142 In Spark 3.4, the Pandas on Spark API `Index.__setitem__` will first to check `value` type is `Column` to avoid raise unexpected error in `is_list_like` like `Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.`. * SPARK-38820 #36357 In Spark 3.4, the Pandas on Spark API `astype('category')` will also refresh `categories.dtype` according to original data `dtype` to follow pandas 1.4 behavior. * SPARK-38947 #36464 In Spark 3.4, the Pandas on Spark API supports groupby positional indexing in `GroupBy.head` and `GroupBy.tail` to follow pandas 1.4. Negative arguments now work correctly and result in ranges relative to the end and start of each group, Previously, negative arguments returned empty frames. * SPARK-39317 #36699 In Spark 3.4, the infer schema process of `groupby.apply` in Pandas on Spark, will first infer the pandas type to ensure the accuracy of the pandas `dtype` as much as possible. * SPARK-39314 #36711 In Spark 3.4, the `Series.concat` sort parameter will be respected to follow pandas 1.4 behaviors. For other test only fixes, I don't add migration doc: SPARK-38821 SPARK-39053 SPARK-38982 ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? Closes #36816 from Yikun/SPARK-39425. Authored-by: Yikun Jiang <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
HyukjinKwon
pushed a commit
that referenced
this pull request
Jul 19, 2022
…low 1.4.3 behavior ### What changes were proposed in this pull request? Respect Series.concat sort parameter when `num_series == 1` to follow 1.4.3 behavior. ### Why are the changes needed? In #36711, we follow the pandas 1.4.2 behaviors to respect Series.concat sort parameter except `num_series == 1` case. In [pandas 1.4.3](https://github.com/pandas-dev/pandas/releases/tag/v1.4.3), fix the issue pandas-dev/pandas#47127. The bug of `num_series == 1` is also fixed, so we add this PR to follow panda 1.4.3 behavior. ### Does this PR introduce _any_ user-facing change? Yes, we already cover this case in: https://github.com/apache/spark/blob/master/python/docs/source/migration_guide/pyspark_3.3_to_3.4.rst ``` In Spark 3.4, the Series.concat sort parameter will be respected to follow pandas 1.4 behaviors. ``` ### How was this patch tested? - CI passed - test_concat_index_axis passed with panda 1.3.5, 1.4.2, 1.4.3. Closes #37217 from Yikun/SPARK-39807. Authored-by: Yikun Jiang <[email protected]> Signed-off-by: Hyukjin Kwon <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Respect ps.concat sort parameter to follow pandas behavior:
num_series != 1for now to follow pandas behaviorWhy are the changes needed?
Since pandas 1.4+ (pandas-dev/pandas@01b8d2a), ps.concat method the sort parameter. We need to follow pandas behavior.
Does this PR introduce any user-facing change?
Yes, but follow pandas 1.4 behavior
How was this patch tested?
test_concat_index_axis, test_concat_multiindex_sort, concat doctest
passed with 1.3/1.4