[SPARK-47151][PYTHON][PS][BUILD] Upgrade to pandas 2.2.1
#45236
Conversation
bjornjorgensen
changed the title
pandas 2.2.1 pandas 2.2.1
|
Merged to master for Apache Spark 4.0.0. |
|
Thank you, @bjornjorgensen . |
| PANDAS_LATEST_VERSION = "2.2.0" | ||
| PANDAS_LATEST_VERSION = "2.2.1" |
There was a problem hiding this comment.
This issue predates this PR, but I note that this version requirement is more strict than our dev requirement:
Line 8 in e46d7d1
One consequence of this is that trying to build the Python API docs locally can fail as follows:
.../spark/python/pyspark/pandas/supported_api_gen.py:115:
UserWarning: Warning: pandas 2.2.1 is required; your version is 2.1.4We should perhaps align the dev requirement with this Pandas version requirement here.
There was a problem hiding this comment.
hmm. there was some discussion on this in
https://github.com/apache/spark/pull/44881/files/9ae857a1b9c47dc12153cbf868e79ca2d0299a1d#diff-95a965e9b4d0ca83ab61f7af36659422910868431d05d68dc21dc8284e1c4b13
but why do you get 2.1.4? are there something else that downgrade it from 2.2.1?
There was a problem hiding this comment.
There are multiple ways to get an older version. Since the requirement only asks for pandas>=1.4.4, then setting up the virtual environment months ago will install a version that remains valid even if you rerun pip install -r ... today. Another way is, yes, if a different requirement specifically prevents 2.2.1 from being installed.
The way we manage Python dependencies leaves a lot to be desired. I tried to fix it in the past (e.g. #27928) but failed to get enough committer support. I am inclined to try again, because I see a constant stream of commits just trying to keep the Python build working, and I think there should be a way to make this easier for everyone.
There was a problem hiding this comment.
yes, but what about dev-container then?
Dev container have been open sourced some months ago Development Containers
There was a problem hiding this comment.
I'm not familiar with Development Containers, but yes, there are probably many ways we can improve the situation.
What I advocated in #27928, and what I still believe is the best option for us today (with some tweaks to my original proposal), is to adopt pip-tools. That's because it's a very conservative approach that builds on our existing use of pip, and lets us focus on the technology-agnostic problem of separating Spark's direct dependencies from our build environment dependencies.
### What changes were proposed in this pull request? Upgrade pandas from 2.2.0 to 2.2.1 ### Why are the changes needed? [Release notes 2.2.1](https://pandas.pydata.org/docs/whatsnew/v2.2.1.html) ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#45236 from bjornjorgensen/pandas2.2.1. Authored-by: Bjørn Jørgensen <[email protected]> Signed-off-by: Dongjoon Hyun <[email protected]>
What changes were proposed in this pull request?
Upgrade pandas from 2.2.0 to 2.2.1
Why are the changes needed?
Release notes 2.2.1
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Pass GA
Was this patch authored or co-authored using generative AI tooling?
No.