Skip to content

Upgrade to Python 3.12 #2072

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 23 commits into from
Oct 23, 2024
Merged

Upgrade to Python 3.12 #2072

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
8f3dce1
Upgrade to 3.12
max-muoto Mar 10, 2024
e4b6dad
Upgrade Pandas
max-muoto Mar 10, 2024
51a4943
Merge branch 'main' into python-3.12
mathbunnyru Mar 27, 2024
4a9db95
Merge branch 'main' into python-3.12
mathbunnyru May 9, 2024
18ee4ad
Merge branch 'main' into python-3.12
mathbunnyru Aug 6, 2024
6c4f7cb
Merge branch 'main' into python-3.12
mathbunnyru Aug 27, 2024
2be2759
Merge branch 'main' into python-3.12
mathbunnyru Aug 31, 2024
70769d9
Merge branch 'main' into python-3.12
mathbunnyru Sep 7, 2024
122eccc
Merge branch 'main' into python-3.12
mathbunnyru Sep 15, 2024
70b2ba6
Update Dockerfile
mathbunnyru Sep 29, 2024
72c166e
Merge branch 'main' into python-3.12
mathbunnyru Sep 30, 2024
7eb512a
Update unit_pandas_version.py
mathbunnyru Sep 30, 2024
d49922f
Merge branch 'main' into python-3.12
mathbunnyru Oct 7, 2024
5a449cd
Merge branch 'main' into python-3.12
mathbunnyru Oct 15, 2024
d86c6d6
Merge branch 'main' into python-3.12
mathbunnyru Oct 20, 2024
392dbe8
Merge branch 'main' into python-3.12
mathbunnyru Oct 21, 2024
779aa11
Merge branch 'main' into python-3.12
mathbunnyru Oct 22, 2024
9abaf2a
Update CHANGELOG.md
mathbunnyru Oct 22, 2024
0f5e3ce
Update CHANGELOG.md
mathbunnyru Oct 22, 2024
f3bfbe0
Merge branch 'main' into python-3.12
mathbunnyru Oct 22, 2024
9d6073a
Update CHANGELOG.md
mathbunnyru Oct 22, 2024
a18bad4
Merge branch 'main' into python-3.12
mathbunnyru Oct 22, 2024
56ff72e
Update README.md
mathbunnyru Oct 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,8 @@ If you want to use the older `Ubuntu` and/or `Python` version, you can use the f
| 2022-10-09 | 22.04 | 3.8 | `7285848c0a11` |
| 2022-10-09 | 22.04 | 3.9 | `ed2908bbb62e` |
| 2023-05-30 | 22.04 | 3.10 | `4d70cf8da953` |
| weekly build | 22.04 | 3.11 | `latest` |
| 2024-03-06 | 22.04 | 3.11 | `07dc2d08734b` |
| weekly build | 22.04 | 3.12 | `latest` |

## Contributing

Expand Down
2 changes: 1 addition & 1 deletion images/docker-stacks-foundation/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,7 @@ RUN echo "auth requisite pam_deny.so" >> /etc/pam.d/su && \
USER ${NB_UID}

# Pin the Python version here, or set it to "default"
ARG PYTHON_VERSION=3.11
ARG PYTHON_VERSION=3.12

# Setup work directory for backward-compatibility
RUN mkdir "/home/${NB_USER}/work" && \
Expand Down
2 changes: 1 addition & 1 deletion images/pyspark-notebook/Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ USER ${NB_UID}
RUN mamba install --yes \
'grpcio-status' \
'grpcio' \
'pandas=2.0.3' \
'pandas=2.2.1' \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you checked the comment above to make sure you’re using the proper version?

Copy link
Contributor Author

@max-muoto max-muoto Mar 10, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They have pandas<=2.2.1 so we should be good here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/apache/spark/blob/master/dev/infra/Dockerfile

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You need to check the latest stable tag, not the current main branch.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah sorry there. Looks like we'll need for their next release, as this commit isn't included in the latest stable tag: ericm-db/spark@98ca3ea

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we'll need to wait on this as well, since we need at least Pandas 2.1.1 to ensure compatibility with 3.12.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bjornjorgensen could you please tell us when the Spark release will include this commit? (at least approximately)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm.. well we are waiting for hadoop 3.4.0 and a new hive release. We haven't started any RC release yet. I build and test my own jupyterlab https://github.com/bjornjorgensen/jupyter-spark-master-docker and I did try python 3.12 but it breake so match so I'm using python 3.11 as debian testing are using.
And Spark 3.5.1 don't support python 3.12 have a look at apache/spark#43922

Copy link

@shreve shreve Jun 11, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we just revert this particular change rather than waiting for the next release of Spark? This seems like an incredibly self-imposed blocker.

Copy link

@flaviomartins flaviomartins Sep 24, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

They have pinned version 2.2.2 apache/spark@7c639a1

'pyarrow' && \
mamba clean --all -f -y && \
fix-permissions "${CONDA_DIR}" && \
Expand Down
2 changes: 1 addition & 1 deletion tests/docker-stacks-foundation/test_python_version.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
from tests.conftest import TrackedContainer

LOGGER = logging.getLogger(__name__)
EXPECTED_PYTHON_VERSION = "3.11"
EXPECTED_PYTHON_VERSION = "3.12"


def test_python_version(container: TrackedContainer) -> None:
Expand Down
2 changes: 1 addition & 1 deletion tests/pyspark-notebook/units/unit_pandas_version.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@
# Distributed under the terms of the Modified BSD License.
import pandas

assert pandas.__version__ == "2.0.3"
assert pandas.__version__ == "2.2.1"