Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix conda version conflict #958

Closed
tomuben opened this issue Sep 19, 2024 · 3 comments
Closed

Fix conda version conflict #958

tomuben opened this issue Sep 19, 2024 · 3 comments
Assignees
@tomuben
Labels
bug Unwanted / harmful behavior

Comments

@tomuben
Copy link
Collaborator

tomuben commented Sep 19, 2024 •
edited
Loading

Background

In exasol/script-languages#450 we removed the dependency in the cuda flavor to:

  • gxx_linux-64==12.4.0
  • gcc_linux-64==12.4.0
    because the conda package cuda-toolkit had a newer version of both packages installed in a previous build step.
    We should investigate the behavior of mamba.
@tomuben tomuben added the task Tasks without changes to the code base label Sep 19, 2024
@tomuben
Copy link
Collaborator Author

tomuben commented Sep 20, 2024

If we move the installation of package cuda-toolkit into the flavor_base_deps buildstep, the old cudatoolkit package will be installed during the language_deps buildstep, because libarrow|16.0.0=he216f8a_1_cuda has a dependency to cudatoolkit:

See the output of conda list in the release image , after moving the package cuda-toolkit from buildstep language deps to flavor_base_deps.

(base) root@479d640e6323:/# conda list
# packages in environment at /opt/conda:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
_sysroot_linux-64_curr_repodata_hack 3                   h69a702a_16    conda-forge
aws-c-auth                0.7.20               h5f1c8d9_0    conda-forge
aws-c-cal                 0.6.12               h2ba76a8_0    conda-forge
aws-c-common              0.9.17               h4ab18f5_0    conda-forge
aws-c-compression         0.2.18               h36a0aea_4    conda-forge
aws-c-event-stream        0.4.2               h161de36_10    conda-forge
aws-c-http                0.8.1               h63f54a0_13    conda-forge
aws-c-io                  0.14.8               h96d4d28_0    conda-forge
aws-c-mqtt                0.10.4               hcc7299c_2    conda-forge
aws-c-s3                  0.5.9                h10bd90f_0    conda-forge
aws-c-sdkutils            0.1.16               h36a0aea_0    conda-forge
aws-checksums             0.1.18               h36a0aea_4    conda-forge
aws-crt-cpp               0.26.8              h4f3a3cc_11    conda-forge
aws-sdk-cpp               1.11.267             h51dfee4_8    conda-forge
binutils                  2.40                 h4852527_7    conda-forge
binutils_impl_linux-64    2.40                 ha1999f0_7    conda-forge
binutils_linux-64         2.40                 hb3c18ed_3    conda-forge
boltons                   24.0.0             pyhd8ed1ab_0    conda-forge
brotli-python             1.1.0           py310hf71b8c6_2    conda-forge
bzip2                     1.0.8                h4bc722e_7    conda-forge
c-ares                    1.33.1               heb4867d_0    conda-forge
c-compiler                1.8.0                h2b85faf_0    conda-forge
ca-certificates           2024.8.30            hbcca054_0    conda-forge
certifi                   2024.8.30          pyhd8ed1ab_0    conda-forge
cffi                      1.17.1          py310h8deb56e_0    conda-forge
charset-normalizer        3.3.2              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
conda                     23.7.4          py310hff52083_0    conda-forge
conda-package-handling    2.3.0              pyh7900ff3_0    conda-forge
conda-package-streaming   0.10.0             pyhd8ed1ab_0    conda-forge
cppzmq                    4.10.0               h2e2a08d_1    conda-forge
cryptography              43.0.1          py310h6c63255_0    conda-forge
cuda-cccl                 12.4.127                      0    nvidia
cuda-command-line-tools   12.4.1                        0    nvidia
cuda-compiler             12.4.1               hbad6d8a_0    conda-forge
cuda-cudart               12.4.127                      0    nvidia
cuda-cudart-dev           12.4.127                      0    nvidia
cuda-cuobjdump            12.4.127                      0    nvidia
cuda-cupti                12.4.127                      0    nvidia
cuda-cuxxfilt             12.4.127                      0    nvidia
cuda-documentation        12.4.127                      0    nvidia
cuda-driver-dev           12.4.127                      0    nvidia
cuda-gdb                  12.4.127                      0    nvidia
cuda-libraries            12.4.1               ha770c72_1    conda-forge
cuda-libraries-dev        12.4.1               ha770c72_1    conda-forge
cuda-nsight               12.4.127                      0    nvidia
cuda-nvcc                 12.4.131                      0    nvidia
cuda-nvdisasm             12.4.127                      0    nvidia
cuda-nvml-dev             12.4.127                      0    nvidia
cuda-nvprof               12.4.127                      0    nvidia
cuda-nvprune              12.4.127                      0    nvidia
cuda-nvrtc                12.4.127                      0    nvidia
cuda-nvrtc-dev            12.4.127                      0    nvidia
cuda-nvtx                 12.4.127                      0    nvidia
cuda-nvvp                 12.4.127                      0    nvidia
cuda-opencl               12.4.127                      0    nvidia
cuda-opencl-dev           12.4.127                      0    nvidia
cuda-profiler-api         12.4.127                      0    nvidia
cuda-sanitizer-api        12.4.127                      0    nvidia
cuda-toolkit              11.8.0                        0    nvidia
cuda-tools                12.4.1               ha770c72_1    conda-forge
cuda-visual-tools         12.4.1               ha770c72_1    conda-forge
cudatoolkit               11.8.0              h4ba93d1_13    conda-forge
cudnn                     8.1.0.77             h90431f1_0    conda-forge
cxx-compiler              1.8.0                h1a2810e_0    conda-forge
fmt                       10.2.1               h00ab1b0_0    conda-forge
gcc                       13.3.0               h9576a4e_1    conda-forge
gcc_impl_linux-64         13.3.0               hfea6d02_1    conda-forge
gcc_linux-64              13.3.0               hc28eda2_3    conda-forge
gds-tools                 1.9.1.3                       0    nvidia
gflags                    2.2.2             h5888daf_1005    conda-forge
glog                      0.7.1                hbabe93e_0    conda-forge
gxx                       13.3.0               h9576a4e_1    conda-forge
gxx_impl_linux-64         13.3.0               hdbfa832_1    conda-forge
gxx_linux-64              13.3.0               h6834431_3    conda-forge
h2                        4.1.0              pyhd8ed1ab_0    conda-forge
hpack                     4.0.0              pyh9f0ad1d_0    conda-forge
hyperframe                6.0.1              pyhd8ed1ab_0    conda-forge
icu                       75.1                 he02047a_0    conda-forge
idna                      3.10               pyhd8ed1ab_0    conda-forge
jsonpatch                 1.33               pyhd8ed1ab_0    conda-forge
jsonpointer               3.0.0           py310hff52083_1    conda-forge
kernel-headers_linux-64   3.10.0              h4a8ded7_16    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
krb5                      1.21.3               h659f571_0    conda-forge
ld_impl_linux-64          2.40                 hf3520f5_7    conda-forge
libabseil                 20240116.2      cxx17_he02047a_1    conda-forge
libarchive                3.7.4                hfca40fe_0    conda-forge
libarrow                  16.0.0          he216f8a_1_cuda    conda-forge
libarrow-acero            16.0.0          h530483c_1_cuda    conda-forge
libarrow-dataset          16.0.0          h530483c_1_cuda    conda-forge
libarrow-substrait        16.0.0          h11d00a1_1_cuda    conda-forge
libblas                   3.9.0           24_linux64_openblas    conda-forge
libbrotlicommon           1.1.0                hb9d3cd8_2    conda-forge
libbrotlidec              1.1.0                hb9d3cd8_2    conda-forge
libbrotlienc              1.1.0                hb9d3cd8_2    conda-forge
libcblas                  3.9.0           24_linux64_openblas    conda-forge
libcrc32c                 1.1.2                h9c3ff4c_0    conda-forge
libcublas                 12.4.5.8                      0    nvidia
libcublas-dev             12.4.5.8                      0    nvidia
libcufft                  11.2.1.3                      0    nvidia
libcufft-dev              11.2.1.3                      0    nvidia
libcufile                 1.9.1.3                       0    nvidia
libcufile-dev             1.9.1.3                       0    nvidia
libcurand                 10.3.5.147                    0    nvidia
libcurand-dev             10.3.5.147                    0    nvidia
libcurl                   8.10.1               hbbe4b11_0    conda-forge
libcusolver               11.6.1.9                      0    nvidia
libcusolver-dev           11.6.1.9                      0    nvidia
libcusparse               12.3.1.170                    0    nvidia
libcusparse-dev           12.3.1.170                    0    nvidia
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 hd590300_2    conda-forge
libevent                  2.1.12               hf998b51_1    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc                    14.1.0               h77fa898_1    conda-forge
libgcc-devel_linux-64     13.3.0             h84ea5a7_101    conda-forge
libgcc-ng                 14.1.0               h69a702a_1    conda-forge
libgfortran               14.1.0               h69a702a_1    conda-forge
libgfortran-ng            14.1.0               h69a702a_1    conda-forge
libgfortran5              14.1.0               hc5f4f2c_1    conda-forge
libgomp                   14.1.0               h77fa898_1    conda-forge
libgoogle-cloud           2.23.0               h9be4e54_1    conda-forge
libgoogle-cloud-storage   2.23.0               hc7a4891_1    conda-forge
libgrpc                   1.62.2               h15f2491_0    conda-forge
libiconv                  1.17                 hd590300_2    conda-forge
liblapack                 3.9.0           24_linux64_openblas    conda-forge
libmamba                  1.5.1                had39da4_2    conda-forge
libmambapy                1.5.1           py310h39ff949_2    conda-forge
libnghttp2                1.58.0               h47da74e_1    conda-forge
libnpp                    12.2.5.30                     0    nvidia
libnpp-dev                12.2.5.30                     0    nvidia
libnsl                    2.0.1                hd590300_0    conda-forge
libnvfatbin               12.4.127                      0    nvidia
libnvfatbin-dev           12.4.127                      0    nvidia
libnvjitlink              12.4.127                      0    nvidia
libnvjitlink-dev          12.4.127                      0    nvidia
libnvjpeg                 12.3.1.117                    0    nvidia
libnvjpeg-dev             12.3.1.117                    0    nvidia
libopenblas               0.3.27          pthreads_hac2b453_1    conda-forge
libparquet                16.0.0          h9348d0c_1_cuda    conda-forge
libprotobuf               4.25.3               h08a7969_0    conda-forge
libre2-11                 2023.09.01           h5a48ba9_2    conda-forge
libsanitizer              13.3.0               heb74ff8_1    conda-forge
libsodium                 1.0.20               h4ab18f5_0    conda-forge
libsolv                   0.7.30               h3509ff9_0    conda-forge
libsqlite                 3.46.1               hadc24fc_0    conda-forge
libssh2                   1.11.0               h0841786_0    conda-forge
libstdcxx                 14.1.0               hc0a3c3a_1    conda-forge
libstdcxx-devel_linux-64  13.3.0             h84ea5a7_101    conda-forge
libstdcxx-ng              14.1.0               h4852527_1    conda-forge
libthrift                 0.19.0               hb90f79a_1    conda-forge
libutf8proc               2.8.0                h166bdaf_0    conda-forge
libuuid                   2.38.1               h0b41bf4_0    conda-forge
libxml2                   2.12.7               he7c6b58_4    conda-forge
libzlib                   1.3.1                h4ab18f5_1    conda-forge
lz4-c                     1.9.4                hcb278e6_0    conda-forge
lzo                       2.10              hd590300_1001    conda-forge
mamba                     1.5.1           py310h51d5547_2    conda-forge
ncurses                   6.5                  he02047a_1    conda-forge
nsight-compute            2024.1.1.4                    0    nvidia
nspr                      4.35                 h27087fc_0    conda-forge
nss                       3.100                hca3bf56_0    conda-forge
numpy                     1.26.2          py310hb13e2d6_0    conda-forge
openssl                   3.3.2                hb9d3cd8_0    conda-forge
orc                       2.0.0                h17fec99_1    conda-forge
packaging                 24.1               pyhd8ed1ab_0    conda-forge
pandas                    2.2.2           py310hf9f9076_1    conda-forge
pip                       24.2               pyh8b19718_1    conda-forge
pluggy                    1.5.0              pyhd8ed1ab_0    conda-forge
pyarrow                   16.0.0          py310h17c5347_0    conda-forge
pyarrow-core              16.0.0          py310h31c9f98_0_cuda    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.6           py310h2372a71_0    conda-forge
pycparser                 2.22               pyhd8ed1ab_0    conda-forge
pyopenssl                 24.2.1             pyhd8ed1ab_2    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
python                    3.10.4          h2660328_0_cpython    conda-forge
python-dateutil           2.9.0              pyhd8ed1ab_0    conda-forge
python-tzdata             2024.1             pyhd8ed1ab_0    conda-forge
python_abi                3.10                    5_cp310    conda-forge
pytz                      2024.2             pyhd8ed1ab_0    conda-forge
re2                       2023.09.01           h7f4b329_2    conda-forge
readline                  8.2                  h8228510_1    conda-forge
reproc                    14.2.4.post0         hd590300_1    conda-forge
reproc-cpp                14.2.4.post0         h59595ed_1    conda-forge
requests                  2.32.3             pyhd8ed1ab_0    conda-forge
ruamel.yaml               0.17.40         py310h2372a71_0    conda-forge
ruamel.yaml.clib          0.2.8           py310h2372a71_0    conda-forge
s2n                       1.4.13               he19d79f_0    conda-forge
setuptools                73.0.1             pyhd8ed1ab_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
snappy                    1.2.1                ha2e4443_0    conda-forge
sqlite                    3.46.1               h9eae976_0    conda-forge
sysroot_linux-64          2.17                h4a8ded7_16    conda-forge
tk                        8.6.13          noxft_h4845f30_101    conda-forge
toolz                     0.12.1             pyhd8ed1ab_0    conda-forge
tqdm                      4.66.5             pyhd8ed1ab_0    conda-forge
tzdata                    2024a                h8827d51_1    conda-forge
urllib3                   2.2.3              pyhd8ed1ab_0    conda-forge
wheel                     0.44.0             pyhd8ed1ab_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml-cpp                  0.8.0                h59595ed_0    conda-forge
zeromq                    4.3.5                ha4adb4c_5    conda-forge
zstandard                 0.23.0          py310ha39cb0e_1    conda-forge
zstd                      1.5.6                ha6fb4c9_0    conda-forge

@tomuben
Copy link
Collaborator Author

tomuben commented Sep 20, 2024

The assumption above was incorrect. cudatoolkit will also be installed if libarrow|16.0.0=he216f8a_1_cuda and cuda-toolkit|11.8.0 are installed together, in build step language_deps.

@tomuben
Copy link
Collaborator Author

tomuben commented Sep 20, 2024 •
edited
Loading

Inverting the order of conda installation and pinning , exasol/script-languages#451, results in a successful build.

  1. In the (final) release image version 13.3.0 of gcc/gxx was installed (dependency of cuda-toolkit)
  2. In the build_deps and build_run image version 12.4.0 was installed (the pinned package in the build description)

@tomuben tomuben self-assigned this Oct 11, 2024
@tomuben tomuben changed the title Investigate conda version conflict Fix conda version conflict Nov 12, 2024
@tomuben tomuben added bug Unwanted / harmful behavior and removed task Tasks without changes to the code base labels Nov 12, 2024
tomuben added a commit to exasol/script-languages that referenced this issue Nov 12, 2024
@tomuben
#958: Fix conda version conflict (#451)
ba95158 
related to exasol/script-languages-release#958
tomuben added a commit that referenced this issue Nov 12, 2024
@tomuben
#958: Fix conda version conflict
fe15115
@tomuben tomuben mentioned this issue Nov 12, 2024
Merged
10 tasks
tomuben added a commit that referenced this issue Nov 12, 2024
@tomuben
#958: Fix conda version conflict
097a3e0
tomuben added a commit that referenced this issue Nov 13, 2024
@tomuben
#958: Fix conda version conflict (#1010)
739c3a9 
closes #958
@tomuben tomuben closed this as completed Nov 13, 2024
tomuben added a commit that referenced this issue Nov 14, 2024
@tomuben
Update master
f2edb41 
Changelist:

- #944: Removed exaudfclient_bin_py3 (#975) 
- #967: Added ctpg script options parser (#968) 
- #972 and #973: Refactorings for Script Options parser (#976) 
- #977: Fixed Trivy update cache workflow (#978) 
- #969: Use new CTPG parser in java vm (#970) 
- #980: Moved flavor template-Exasol-all-java-17 to script-languages repo (#984)
- #971: Removed dead code in Java VM (#985) 
- Refactorings in ScriptOptions parser (#987) 
- #988: Adapted Java integration tests for new ScriptOptions parser (#995) 
- #997: Added documentation for Rollback of Script-Languages-Container (#999)
- #878: Fixed jvmoption and parameters containing a space (#1001) 
- #1002: Use ECR fallback repository for Trivy caching (#1004) 
- #1006: Updated Dependencies on top of 8.4.0 (#1007) 
- #958: Fix conda version conflict (#1010) 
- #1009: Removed NVIDIA GPU Driver Libraries from template-CUDA SLC (#1012)
- Bumped version to 9.0.0 (#1013) 
- #1014: Fixed Push Docker Test Container workflow (#1015)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tomuben tomuben

Labels
bug Unwanted / harmful behavior
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tomuben