About docker HEALTHCHECK

[pwob]

How do you guys enabled healthcheck in multibuild docker using distroless? Currently it is not working for us because obviously curl is not available in distroless image.

[dlorenc]

I'm not sure I understand the question. How is curl related to HEALTHCHECK? What do you mean by "multibuild"? Can you share an example of what you're trying to do without distroless/bazel?

[dlorenc]

Closing. Please reopen with more info.

[Junkern]

I just stumbled upon the distroless docker images and the same question also popped into my mind.

We are deploying our docker images using docker-compose (for exposting ports, configs, secrets and so on) and also utilize the healthcheck, so that our swarm knows whether a replica is healthy or not

Our healthcheck_command is: test: ["CMD", "curl", "-f", "http://localhost:10000"]

I would like to use the distroless images (for several reasons), but how would the healthcheck work with a distroless image, as curl is not available there? How do/would you solve that?

[Junkern]

@dlorenc

[pwob]

@Junkern For now we solved it by making a service (Kubernetes) call an endpoint of the api /healthcheck in a certain interval.

[barmic]

After the reading of this blog post https://blog.sixeyed.com/docker-healthchecks-why-not-to-use-curl-or-iwr/, I create a custom tool write in go to make only my check (and build it with a docker build stage).

[verglor]

@pwob @Junkern You can add 84kB busybox wget to the image and use it instead of curl:

FROM busybox AS builder

ARG BUSYBOX_VERSION=1.31.0-i686-uclibc
ADD https://busybox.net/downloads/binaries/$BUSYBOX_VERSION/busybox_WGET /wget
RUN chmod a+x /wget

FROM gcr.io/distroless/java

COPY --from=builder /wget /usr/bin/wget

@barmic Custom healthcheck in higher language (go, java, ...) even built into native binary is at least several MB large and probably overkill for simple HTTP check.

[barmic]

@verglor My go binary is huge indeed (about 7MB), but comparing to JDK isn't noticeable (~3% of total size) and ensure that can't be used to call another service. If I want reduce the size I can use java, I have already the JDK so one executable class/jar should fit in some KB.

[madduci]

The solution of @verglor unfortunately doesn't work in distroless/java:

docker inspect $container --format "{{ (index (.State.Health.Log) 0).Output }}"
OCI runtime exec failed: exec failed: container_linux.go:370: starting container process caused: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown

In the Dockerfile I have the following added:

HEALTHCHECK --interval=30s \
            --timeout=5s \
            CMD /usr/bin/wget --no-verbose --tries=1 --spider 'http://localhost/health' || exit 1

The CMD expects as Entrypoint /bin/sh probably?

[mabrarov]

Hi @madduci,

Could the way you define HEALTHCHECK in your Dockerfile be the reason Docker trying to run /bin/sh?

Dockerfile reference. HEALTHCHECK:

The command after the CMD keyword can be either a shell command (e.g. HEALTHCHECK CMD /bin/check-running) or an exec array (as with other Dockerfile commands; see e.g. ENTRYPOINT for details).

Can you try this (does it work for your case):

HEALTHCHECK --interval=30s \
            --timeout=5s \
            CMD ["/usr/bin/wget", "--no-verbose", "--tries=1", "--spider", "http://localhost/health"]

?

[mabrarov]

Hi @madduci,

Regarding my previous comment - it looks like HEALTHECHECK should return predefined exit codes:

Dockerfile reference. HEALTHCHECK:

The command’s exit status indicates the health status of the container. The possible values are:

• 0: success - the container is healthy and ready for use
• 1: unhealthy - the container is not working correctly
• 2: reserved - do not use this exit code

This means that we cannot rely on exit code of wget and should treat any non-zero exit code of wget as 1. This definitely requires either a custom binary or a shell script (or a shell style HEALTHCHECK instruction). You can try to create symlinks for the downloaded BusyBox like I do it in mabrarov/docker-compose-init-container.

[madduci]

Hi @mabrarov ,

thank you for your reply, your suggestion works.

It's interesting how with docker-compose, I'm able to write the following:

healthcheck:
  test: [ "CMD", "/usr/bin/wget", "--no-verbose", "--tries=1", "--spider", "http://localhost/health" ]
  interval: 15s
  timeout: 3s
  retries: 5
  start_period: 30s

and it just works.

I guess, as I wrote the command in my previous comment, it is assuming that "CMD-SHELL" (a wrapper operation around /bin/sh) is performed (Documentation CMD-SHELL)

[barmic]

Since my last comment, I change of method. Now I use a custom java tool: https://gist.github.com/barmic/a12c5256f735f4748e3a6f511367407e

This give a very simple tool of 4KB without security issue.

[mabrarov]

@barmic,

There can be issues with custom Java tool:

1. Java application consumes CPU at the start, i.e. Java startup time heavily depends on CPU resources allocated for container.
2. Even with sufficient CPU resources Java application can be slow at startup, taking ~1 sec to start.
3. If one uses JAVA_TOOL_OPTIONS environment variable to pass JVM options into his Java application, then custom Java tool will pick JAVA_TOOL_OPTIONS environment variable too, which seems to be unwanted in most cases (e.g. when JAVA_TOOL_OPTIONS environment variable defines Java heap size).

[barmic]

1s to start a jvm? Never in my case (it's about 10 times more quick). I don't know how can compute the load (part of memory page will be shared between 2 jvm, evaluate CPU load will depende of env and CPU,…) probably it can be possible to disable some startup work.

You can control the env of this tool as you want, it's not a problem.

For my case, the security issue is widely more important than CPU consuming.

[mabrarov]

Hi @barmic,

Never in my case (it's about 10 times more quick)

Have you tried to limit CPU resource for container which uses custom Java tool for health check?

You can control the env of this tool as you want, it's not a problem.

JAVA_TOOL_OPTIONS environment variable is automatically picked by Java. There is no way to define JAVA_TOOL_OPTIONS environment variable for passing JVM options into the main Java application and avoid custom Java tool (which is used just for the health check) to pick the same JVM options.

[mabrarov]

FYI,

With powerman/dockerize 0.14.0+ (refer to dockerize#103) it's possible to use Dockerize for Docker health check this way:

HEALTHCHECK --interval=30s \
            --timeout=5s \
            CMD ["/usr/bin/dockerize", "-timeout", "1s", "-wait", "http://localhost/health", "-exit-code", "1"]

I use Dockerize for Docker health check in mabrarov/docker-compose-init-container where Dockerize fits well into a case when Distroless Java image is used.

[JULIANCHO923]

The strategy to have a healthcheck standalone file to validate my app works very well

I based on this post: https://mflash.dev/post/2021/03/01/java-based-health-check-for-docker/

// myapp/src/main/resources/healthcheck/HealthCheck.java
import java.io.IOException;
import java.net.URI;
import java.net.http.HttpClient;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse.BodyHandlers;

public class HealthCheck {
  public static void main(String[] args) throws InterruptedException, IOException {
    var client = HttpClient.newHttpClient();
    var request = HttpRequest.newBuilder().uri(URI.create("http://localhost/healthcheck")).build();   
    var response = client.send(request, BodyHandlers.ofString());
    if (response.statusCode() != 200) {
      throw new RuntimeException("Healthcheck failed");
    }
  }
}

Here is the snippet to the Dockerfile where I'm using the healthcheck file

FROM gcr.io/distroless/java11-debian11
COPY myapp/src/main/resources/healthcheck/ .
HEALTHCHECK --interval=15s --timeout=10s --start-period=45s --retries=3 CMD ["java", "HealthCheck.java", "||", "exit", "1"]

[refactoriel]

FROM busybox AS builder

ARG BUSYBOX_VERSION=1.31.0-i686-uclibc
ADD https://busybox.net/downloads/binaries/$BUSYBOX_VERSION/busybox_WGET /wget
RUN chmod a+x /wget

FROM gcr.io/distroless/java

COPY --from=builder /wget /usr/bin/wget

For me this shorter version worked for mockserver/mockserver which uses gcr.io/distroless/java. Probably gcr.io/distroless/java itself would work as well

FROM mockserver/mockserver:5.15.0
COPY --from=busybox:1.36.0-musl /bin/wget /usr/bin/wget