Docker-ce 25.0.0 breaks image squashing #236

Chekov2k · 2024-01-22T07:10:31Z

There is an issue with using docker-squash and the latest stable docker release. Using the example Dockefile

ARG BASE_IMAGE="docker"
FROM ${BASE_IMAGE}

ARG VERSION="0.48.3"
ARG SUFFIX="64bit"
RUN wget --no-verbose https://github.com/aquasecurity/trivy/releases/download/v${VERSION}/trivy_${VERSION}_Linux-${SUFFIX}.tar.gz -O - | tar -zxvf - && \
    mv trivy /usr/bin

and building it via

docker build -t test .

I can successfully squash an image with docker-ce version 24.0.7, see
v24.0.7.log

However, if I use docker-ce 25.0.0 I get

docker-squash -v --tmp-dir scratch ${DOCKER_SQUASH_FROM} -t test test
2024-01-22 06:56:48,858 root         DEBUG    Running version 1.1.0
2024-01-22 06:56:48,858 root         DEBUG    Preparing Docker client...
2024-01-22 06:56:48,858 docker.utils.config DEBUG    Trying paths: ['/home/infrastructure/.docker/config.json', '/home/infrastructure/.dockercfg']
2024-01-22 06:56:48,858 docker.utils.config DEBUG    Found file at path: /home/infrastructure/.docker/config.json
2024-01-22 06:56:48,873 urllib3.connectionpool DEBUG    http://localhost:None "GET /version HTTP/1.1" 200 843
2024-01-22 06:56:48,875 urllib3.connectionpool DEBUG    http://localhost:None "GET /v1.44/_ping HTTP/1.1" 200 2
2024-01-22 06:56:48,876 root         DEBUG    Docker client ready
2024-01-22 06:56:48,890 urllib3.connectionpool DEBUG    http://localhost:None "GET /v1.44/version HTTP/1.1" 200 843
2024-01-22 06:56:48,890 root         INFO     docker-squash version 1.1.0, Docker 25.0.0, API 1.44...
2024-01-22 06:56:48,891 root         INFO     Using v2 image format
2024-01-22 06:56:48,891 root         DEBUG    Using scratch as the temporary directory
2024-01-22 06:56:48,894 urllib3.connectionpool DEBUG    http://localhost:None "GET /v1.44/images/test/json HTTP/1.1" 200 None
2024-01-22 06:56:48,900 urllib3.connectionpool DEBUG    http://localhost:None "GET /v1.44/images/sha256:e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5/history HTTP/1.1" 200 None
2024-01-22 06:56:48,901 root         INFO     Old image has 31 layers
2024-01-22 06:56:48,901 root         DEBUG    Old layers: ['<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', 'sha256:e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5']
2024-01-22 06:56:48,901 root         DEBUG    We detected number of layers as the argument to squash
2024-01-22 06:56:48,901 root         INFO     Checking if squashing is necessary...
2024-01-22 06:56:48,901 root         INFO     Attempting to squash last 31 layers...
2024-01-22 06:56:48,901 root         DEBUG    Layers to squash: ['<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', '<missing>', 'sha256:e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5']
2024-01-22 06:56:48,901 root         DEBUG    Layers to move: []
2024-01-22 06:56:48,901 root         INFO     Saving image sha256:e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5 to scratch/old directory...
2024-01-22 06:56:48,901 root         DEBUG    Try #1...
2024-01-22 06:56:53,129 urllib3.connectionpool DEBUG    http://localhost:None "GET /v1.44/images/sha256:e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5/get HTTP/1.1" 200 None
2024-01-22 06:56:53,130 root         DEBUG    Extracting image using iterator over raw data
2024-01-22 06:56:53,939 root         INFO     Image saved!
2024-01-22 06:56:53,939 root         INFO     Squashing image 'test'...
2024-01-22 06:56:53,940 root         DEBUG    Reading 'scratch/old/manifest.json' JSON file...
2024-01-22 06:56:53,940 root         DEBUG    Reading 'scratch/old/blobs/sha256/e58ea5a71f11acb72b3eb06c8a3ab6ffe4cc10bacc8c4db42dd2089b64c271c5' JSON file...
2024-01-22 06:56:53,940 root         INFO     Starting squashing...
2024-01-22 06:56:53,940 root         INFO     Squashing file 'scratch/old/blobs/layer.tar'...
2024-01-22 06:56:53,941 root         DEBUG    Cleaning up scratch temporary directory
2024-01-22 06:56:53,993 root         ERROR    [Errno 2] No such file or directory: 'scratch/old/blobs/layer.tar'
Traceback (most recent call last):
  File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/cli.py", line 126, in run
    squash.Squash(
  File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/squash.py", line 90, in run
    return self.squash(image)
  File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/squash.py", line 125, in squash
    new_image_id = image.squash()
  File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/image.py", line 77, in squash
    ret = self._squash()
  File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/v2_image.py", line 41, in _squash
    self._squash_layers(self.layer_paths_to_squash, self.layer_paths_to_move)
  File "/home/infrastructure/.local/lib/python3.8/site-packages/docker_squash/image.py", line 779, in _squash_layers
    layer_tar: tarfile.TarFile = tarfile.open(
  File "/usr/lib/python3.8/tarfile.py", line 1603, in open
    return func(name, "r", fileobj, **kwargs)
  File "/usr/lib/python3.8/tarfile.py", line 1667, in gzopen
    fileobj = GzipFile(name, mode + "b", compresslevel, fileobj)
  File "/usr/lib/python3.8/gzip.py", line 173, in __init__
    fileobj = self.myfileobj = builtins.open(filename, mode or 'rb')
FileNotFoundError: [Errno 2] No such file or directory: 'scratch/old/blobs/layer.tar'
2024-01-22 06:56:53,996 root         ERROR    Execution failed, consult logs above. If you think this is our fault, please file an issue: https://github.com/goldmann/docker-squash/issues, thanks!

I'm going to look further into this but thought raising an issue already might help

The text was updated successfully, but these errors were encountered:

Chekov2k · 2024-01-22T07:22:56Z

Might be related to moby/moby#44598 The docker image save tarball output is now OCI compliant.

Chekov2k · 2024-01-22T08:43:59Z

Looks like the exported image old/blobs dir has stripped the /layer.tar and looks like this now

ls -l old/blobs/sha256
total 250168
-r--r--r--  1 infrastructure  staff   1.5K  1 Jan  1970 0d979313e9f9262364f038e05249664ce08e9ae049b5f4145a10b45fb64151f3
-r--r--r--  1 infrastructure  staff   1.1K  1 Jan  1970 136d873267a8e1955991a56e76b6bf86f6af0ef234d12de7d73c5f7208e1a50e
-r--r--r--  1 infrastructure  staff   544B  1 Jan  1970 1471633a84a6349cfeb08bf4b9e8879be2fdbdb8d55b489543be73d31bf91c57
-r--r--r--  1 infrastructure  staff   1.9M  1 Jan  1970 1d4959de38143abac793d31e4e8592696721305ff06281f8b9284aee380afcfe
-r--r--r--  1 infrastructure  staff   3.2K  1 Jan  1970 37c9a49ec4c63595dcad7f349573111e79ea86a4345028cfd8f9ea1c4374f044
-r--r--r--  1 infrastructure  staff   3.4K  1 Jan  1970 4993a3d53d15a092f04c547ffeff56f3bd98c746e4981166268602977605aa1d
-r--r--r--  1 infrastructure  staff    91K  1 Jan  1970 4b221b641f720e676dc1fe4410dbb9fbd8a397497ac9a35fa99b2a43cc96f6f9
-r--r--r--  1 infrastructure  staff   566B  1 Jan  1970 4b31d8bab091aef7ea07d8dbc3dcaa8b79dd43beeec9a9073a3aa424e8087dae
-r--r--r--  1 infrastructure  staff   9.1M  1 Jan  1970 4e62c66ee4db0c28bdd0a8d371e3034736195c22a9cfc2221c71fd75403feffe
-r--r--r--  1 infrastructure  staff    32B  1 Jan  1970 4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1
-r--r--r--  1 infrastructure  staff    15M  1 Jan  1970 5598890badaf41ad29ca6e7a62aecf9b18e64ff1308118644d6ecda191bee6ae
-r--r--r--  1 infrastructure  staff    15M  1 Jan  1970 594895876a66f55998b0c3d9c175ec0cfa723898879d7fe8f837de3b0e9e01e6
-r--r--r--  1 infrastructure  staff    13K  1 Jan  1970 5f808b0b65434bc22dee0c2293dada9530071a11efc6208bcc79157591716686
-r--r--r--  1 infrastructure  staff   116B  1 Jan  1970 934f51c79a118bc2c9ef86a286aee9f3c33f0c413bd6c9ffa6c949a71275b217
-r--r--r--  1 infrastructure  staff   856B  1 Jan  1970 9ad316ac6f2cfb87a38894f741f377b29e9313b9def6ad15fd307eca8d145d1c
-r--r--r--  1 infrastructure  staff   167B  1 Jan  1970 9ceb83de43d622e69841979ca0985ea131b55405352bca6e9496dc6753435594
-r--r--r--  1 infrastructure  staff   1.3K  1 Jan  1970 b0686185000f25f48b2caaea5bba0d9a7c01edd8b1e365dd62827707ec488560
-r--r--r--  1 infrastructure  staff    16M  1 Jan  1970 b72e0c5c013142ab0f105384e856690145c739b4c62798a667ba44e8b100815b
-r--r--r--  1 infrastructure  staff   548B  1 Jan  1970 bf613c8d3eefad31faacee38beeedf68069e48ea01d8f083fa0a57ce343ed827
-r--r--r--  1 infrastructure  staff   3.2M  1 Jan  1970 c303524923177661067f7eb378c3dd5277088c2676ebd1cd78e68397bb80fdbf
-r--r--r--  1 infrastructure  staff    13M  1 Jan  1970 d46db1bcf02a720cf4e32fb0c58cc23f2310a263bde464ca9ce928d473c159e9
-r--r--r--  1 infrastructure  staff   1.0K  1 Jan  1970 d64d5b7dddfb9901291c707c8b672a987b74197fe26a2a8c58747cf784a5c08d
-r--r--r--  1 infrastructure  staff    49M  1 Jan  1970 e09e5717c3fbfa466bfb8a152c3718c081fb34db45e79fa281023aace028913d

bMx-Copernicus · 2024-01-22T12:54:43Z

We had the same issue after docker-ce and docker-ce-cli was auto-upgraded from version 24.0.7 to 25.0.0 on our build machine. As a workaround we downgraded to the 24.0.7 docker packages which resolved the issue.

2024-01-22T06:43:20.0552376Z Command: �[33mdocker-squash --output-path CloudCe_docker.tar -t cloudce:MAIN-42257 cloudce_full:MAIN-42257�[39m
2024-01-22T06:43:21.2208160Z 2024-01-22 06:43:21,217 root         INFO     docker-squash version 1.1.0, Docker 25.0.0, API 1.44...
2024-01-22T06:43:21.2210772Z 2024-01-22 06:43:21,218 root         INFO     Using v2 image format
2024-01-22T06:43:21.2298204Z 2024-01-22 06:43:21,227 root         INFO     Old image has 27 layers
2024-01-22T06:43:21.2354938Z 2024-01-22 06:43:21,227 root         INFO     Checking if squashing is necessary...
2024-01-22T06:43:21.2355181Z 2024-01-22 06:43:21,228 root         INFO     Attempting to squash last 27 layers...
2024-01-22T06:43:21.2355901Z 2024-01-22 06:43:21,228 root         INFO     Saving image sha256:0ac958c7ed1f7f38c3dc35b5022891aea77a6bd5e790b4db11bbe427e9198dd9 to /tmp/docker-squash-88vhom9s/old directory...
2024-01-22T06:43:41.3599383Z 2024-01-22 06:43:41,355 root         INFO     Image saved!
2024-01-22T06:43:41.3600393Z 2024-01-22 06:43:41,356 root         INFO     Squashing image 'cloudce_full:MAIN-42257'...
2024-01-22T06:43:41.4089020Z 2024-01-22 06:43:41,394 root         INFO     Starting squashing...
2024-01-22T06:43:41.4237307Z 2024-01-22 06:43:41,394 root         INFO     Squashing file '/tmp/docker-squash-88vhom9s/old/blobs/layer.tar'...
2024-01-22T06:43:41.5180141Z 2024-01-22 06:43:41,510 root         ERROR    [Errno 2] No such file or directory: '/tmp/docker-squash-88vhom9s/old/blobs/layer.tar'
2024-01-22T06:43:41.5181543Z 2024-01-22 06:43:41,511 root         ERROR    Execution failed, consult logs above. If you think this is our fault, please file an issue: https://github.com/goldmann/docker-squash/issues, thanks!

satmandu · 2024-01-26T17:53:33Z

Do we need to manually generate a tarball now? What's the suggestion for compatibility with 25.0.0?

nixsec · 2024-01-26T18:31:05Z

We had the same issue after docker-ce and docker-ce-cli was auto-upgraded from version 24.0.7 to 25.0.0 on our build machine. As a workaround we downgraded to the 24.0.7 docker packages which resolved the issue.

2024-01-22T06:43:20.0552376Z Command: �[33mdocker-squash --output-path CloudCe_docker.tar -t cloudce:MAIN-42257 cloudce_full:MAIN-42257�[39m
2024-01-22T06:43:21.2208160Z 2024-01-22 06:43:21,217 root         INFO     docker-squash version 1.1.0, Docker 25.0.0, API 1.44...
2024-01-22T06:43:21.2210772Z 2024-01-22 06:43:21,218 root         INFO     Using v2 image format
2024-01-22T06:43:21.2298204Z 2024-01-22 06:43:21,227 root         INFO     Old image has 27 layers
2024-01-22T06:43:21.2354938Z 2024-01-22 06:43:21,227 root         INFO     Checking if squashing is necessary...
2024-01-22T06:43:21.2355181Z 2024-01-22 06:43:21,228 root         INFO     Attempting to squash last 27 layers...
2024-01-22T06:43:21.2355901Z 2024-01-22 06:43:21,228 root         INFO     Saving image sha256:0ac958c7ed1f7f38c3dc35b5022891aea77a6bd5e790b4db11bbe427e9198dd9 to /tmp/docker-squash-88vhom9s/old directory...
2024-01-22T06:43:41.3599383Z 2024-01-22 06:43:41,355 root         INFO     Image saved!
2024-01-22T06:43:41.3600393Z 2024-01-22 06:43:41,356 root         INFO     Squashing image 'cloudce_full:MAIN-42257'...
2024-01-22T06:43:41.4089020Z 2024-01-22 06:43:41,394 root         INFO     Starting squashing...
2024-01-22T06:43:41.4237307Z 2024-01-22 06:43:41,394 root         INFO     Squashing file '/tmp/docker-squash-88vhom9s/old/blobs/layer.tar'...
2024-01-22T06:43:41.5180141Z 2024-01-22 06:43:41,510 root         ERROR    [Errno 2] No such file or directory: '/tmp/docker-squash-88vhom9s/old/blobs/layer.tar'
2024-01-22T06:43:41.5181543Z 2024-01-22 06:43:41,511 root         ERROR    Execution failed, consult logs above. If you think this is our fault, please file an issue: https://github.com/goldmann/docker-squash/issues, thanks!

Same issues on our end as well with Docker 25.0.0 and 25.0.1. We've reverted back to 24.0.7 and can confirm that resolves these issues.

rnc · 2024-02-01T16:47:15Z

Thanks for the report. I can confirm that due to Docker 25.x switching to the OCI archive format it is incompatible with docker-squash which currently only supports docker-archive format. Note that podman currently saves by default in docker-archive format but is also planning on switching in its next major release. It would effectively need a completely new v3_image.py implementation to handle the different format.

rnc · 2024-02-06T11:00:50Z

I am currently looking into this and examining the differences between the different formats.

Primarily instead of e.g. 4007b9f4932dd241d3f16a73029e79520a73b1fda673d3b7eeacbab313d343ff/layer.tar one might have blobs/sha256/22694763edf2e6f5beec6a058c8ebc695c8976c2677aba5626f7a684df54bc20. There are also differences between the manifest.json , index.json and potentially the archive might also be compressed (I think - although I haven't seen Docker doing that in any experiments I have done).

satmandu · 2024-02-07T14:26:10Z

Is there any chance of getting a new tagged release so we don't have to install from git to get this fix?

rnc · 2024-02-07T14:33:23Z

@satmandu Yeah @goldmann will be doing a release :-)

Chekov2k · 2024-02-07T16:46:02Z

Thank you for the quick fix :)

goldmann · 2024-02-07T19:30:25Z

We should be now good to go!

https://pypi.org/project/docker-squash/1.2.0/
https://github.com/goldmann/docker-squash/releases/tag/1.2.0

Big ups for @rnc for stepping in to fix this!

rodrigonull mentioned this issue Jan 30, 2024

kie-issues#867: Downgrade Docker to version 24.0.7 to avoid issues with docker-squash cmd apache/incubator-kie-kogito-pipelines#1171

Merged

This was referenced Feb 5, 2024

Sconification failed : [Errno 2] No such file or directory: '/tmp/docker-squash-vagger9i/old/blobs/layer.tar #237

Closed

"Not a directory" issue #223

Closed

rnc mentioned this issue Feb 7, 2024

Handle Docker 25 OCI archive format #241

Merged

rnc closed this as completed in #241 Feb 7, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Docker-ce 25.0.0 breaks image squashing #236

Docker-ce 25.0.0 breaks image squashing #236

Chekov2k commented Jan 22, 2024

Chekov2k commented Jan 22, 2024

Chekov2k commented Jan 22, 2024 •

edited

Loading

bMx-Copernicus commented Jan 22, 2024

satmandu commented Jan 26, 2024

nixsec commented Jan 26, 2024 •

edited

Loading

rnc commented Feb 1, 2024

rnc commented Feb 6, 2024

satmandu commented Feb 7, 2024

rnc commented Feb 7, 2024

Chekov2k commented Feb 7, 2024

goldmann commented Feb 7, 2024 •

edited

Loading

Docker-ce 25.0.0 breaks image squashing #236

Docker-ce 25.0.0 breaks image squashing #236

Comments

Chekov2k commented Jan 22, 2024

Chekov2k commented Jan 22, 2024

Chekov2k commented Jan 22, 2024 • edited Loading

bMx-Copernicus commented Jan 22, 2024

satmandu commented Jan 26, 2024

nixsec commented Jan 26, 2024 • edited Loading

rnc commented Feb 1, 2024

rnc commented Feb 6, 2024

satmandu commented Feb 7, 2024

rnc commented Feb 7, 2024

Chekov2k commented Feb 7, 2024

goldmann commented Feb 7, 2024 • edited Loading

Chekov2k commented Jan 22, 2024 •

edited

Loading

nixsec commented Jan 26, 2024 •

edited

Loading

goldmann commented Feb 7, 2024 •

edited

Loading