OrientDB's available memory estimate ignores cgroup (container) limits

[obi1kenobi]

OrientDB Version: 2.2.29

Java Version: OpenJDK 8

OS: Linux

When calculating the available system memory to use as a disk cache, OrientDB currently uses the formula total system memory - JVM heap size - 2GB.

This ignores the fact that OrientDB may be running on, say, a 64GB RAM system, but inside a docker container that is only allowed to use 10GB maximum. This means that OrientDB will keep using more and more memory, thinking there's plenty of capacity, until it gets terminated by the operating system's OOM killer for exceeding its allocated limit.

As a workaround until this issue can be fixed, if you run OrientDB in a container, I recommend setting the -XX:MaxDirectMemorySize option in the ORIENTDB_OPTS_MEMORY environment variable. Keep in mind that the direct memory consumption is on top of the heap and stack memory, so leave some breathing room between the sum of these three values and the container memory limit.

Expected behavior

OrientDB should be aware of any cgroup limits that might apply to it. Other projects have had similar issues in the past, and have developed fixes. OpenBLAS and YARN are two examples.

Actual behavior

OrientDB is not aware of the cgroup limits, and keeps allocating memory until it gets killed by the OS.

Steps to reproduce

• Get a 64GB machine / instance / VM.
• Start an OrientDB docker container with the --memory option set to significantly less than 64GB, e.g. 6GB.
• Put load on the database until OrientDB gets killed by the operating system for using too much RAM.

[andrii0lomakin]

Some sources to read http://man7.org/linux/man-pages/man7/cgroups.7.html, http://man7.org/linux/man-pages/man2/getrlimit.2.html, https://github.com/java-native-access/jna/blob/master/contrib/platform/src/com/sun/jna/platform/unix/Resource.java , https://fabiokung.com/2014/03/13/memory-inside-linux-containers/ , https://stackoverflow.com/questions/35842/how-can-a-java-program-get-its-own-process-id

[andrii0lomakin]

OK, for now, we will implement only cgroup v1 support. Later we will add cgroup 2, but in a separate issue. Taking in account that very few distr us cgroup v2 version. I think it is completely OK for now. Sad thing is that docker does not use "cgroup" namespace of course but at least /sys/fs/cgroup is provided.

[andrii0lomakin]

Fixed in 2.2.x branch will be in 3.0 soon.

[andrii0lomakin]

Fixed

[obi1kenobi]

Thank you, @Laa!