Coredumps in Docker - How to catch a coredump from a process within a Docker container.

coredumps_docker.md

Coredumps in Docker

This document covers how to configure a Linux host to collect coredump files from processes running inside Docker containers, the different storage options available, and how to work with a coredump file once you have one.

Background

When a process crashes due to a fatal signal (e.g. SIGSEGV, SIGABRT), the kernel can write a snapshot of the process's memory to disk (called "coredump" or "core file") in order to make it easier to debug what went wrong. Two independent settings control whether this happens:

1. ulimit -c: a per-process resource limit that caps the maximum size of a core file. The default on most distros is 0, which silently suppresses all core files. It must be set to unlimited (or a sufficiently large number of 512-byte blocks) for any core to be written.

2. /proc/sys/kernel/core_pattern: a kernel-wide setting (i.e. it is shared between the host and all containers) that controls where the kernel writes the core file. It can be either a file path template or a pipe to a helper program. The default is usually just core, which writes to the current working directory of the crashing process.

Details on exactly how to configure the core_pattern are found in a section below, but first it is helpful to understand how the Docker isolation affects these different methods.

Interaction with Docker

From the kernel's perspective, a containerized process is just a process. When it crashes, the kernel reads core_pattern from the host and acts on it, but there is a significant difference in how it is handled depending on if the pattern is a path or a pipe.

If core_pattern is a pipe handler (e.g. a pipe to systemd-coredump), the kernel executes that binary on the host and not from within the container, which in turn means that any storage paths are based on the host's filesystem.

man 5 core: "The process runs in the initial namespaces (PID, mount, user, and so on) and not in the namespaces of the crashing process."

However, if the core_pattern is a path (like the default core) it will write the file relative to the process's working directory as seen in the mount namespace of that process, which in the case of Docker basically means the container's filesystem. The file would then land at /core inside the container's writable layer, which disappears when the container is removed unless a bind-mount is present at that location. An absolute path is also just absolute within the container.

man 5 core: "Paths are interpreted according to the settings that are active for the crashing process. That means the crashing process's mount namespace, its current working directory, and its root directory."

In both cases it would also fail silently if the process writing the dump file doesn't have write permissions to the target directory. But this is mostly only a problem in the path case, since for this method it is the crashing process's credentials (UID/GID) that are used. For the pipe method it is the credentials of the helper binary ("root" for systemd-coredump), so permissions are usually not an issue here.

So to summarize:

• core_pattern must be configured on the host, not inside the container.
• The ulimit must be raised for the container (not just on the host shell).
• A pipe core_pattern will write the file relative to the host's filesystem, i.e. it is written outside the container.
• A path core_pattern is based on the process's working directory, i.e. it is written inside the container and a bind-mounted volume is necessary.
• Write access is required to the target directory.

Storage Methods

When a coredump happens, and you want to store it, there are two main methods on how to do it. This section explains the benefits of the different methods and how to properly configure them on the system to work well along with Docker. Instructions on how to test if it works are found in the next section.

Option A: Manual Setting of core_pattern (file-based)

This method does not require any additional software to be installed on the system, since the kernel writes the core directly to the path in core_pattern.

This is the simplest approach and gives you full control over the location, but requires manual rotation/cleanup of the directory where the dumps are written, as well as a bind-mount to the location within the container in order to make it persistent. You will also need to make sure that the process UID/GID is allowed to write to the destination folder.

Supported format specifiers for the path:

Specifier	Expands to
%e	Executable name
%p	PID of the crashing process
%u	UID of the crashing process
%t	Unix timestamp of the crash
%h	Hostname

Set the pattern and create the target directory on the host so that all coredumps happening have somewhere to end up:

# Persistent sysctl drop-in.
echo 'kernel.core_pattern = /var/coredumps/core.%e.%p.%t' \
    | sudo tee /etc/sysctl.d/90-coredump.conf
sudo sysctl -p /etc/sysctl.d/90-coredump.conf

Create the directory with permissions such that only "root" may list its content and read all the files. Everyone else may only create and write to files inside the directory without being able to see any other coredump. The sticky bit in the beginning makes so that only "root" and the original owner of the file can delete it.

sudo mkdir -p /var/coredumps
sudo chmod 1773 /var/coredumps

NOTE: Save location is inside /var/coredumps/

Option B: systemd-coredump (pipe-based)

This method requires systemd-coredump to be installed on the host system:

sudo apt-get install systemd-coredump

which should automatically update the core_pattern to a pipe:

cat /proc/sys/kernel/core_pattern
# |/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %h

The kernel pipes the raw core binary to the systemd-coredump helper on the host. The helper stores it according to /etc/systemd/coredump.conf and records metadata in the systemd journal. The coredumpctl tool is then used to list and retrieve cores.

NOTE: systemd-coredump does not need to be present inside the container.

There are three Storage= modes:

Mode	Where the binary core is kept	Managed by
external	/var/lib/systemd/coredump/*.zst	systemd-coredump-vacuum
journal	Embedded in the binary journal as a field	journald retention settings
none	Discarded; only metadata is logged	n/a

Storage=journal is the default on some distros. Avoid it for large processes since the binary core data is embedded in the journal and counts against journald's size limits, which can cause older log entries to be evicted. Storage=external is the recommended choice.

Retention for Storage=external is controlled by the following settings:

Setting	Default	Meaning
MaxUse=	10% of filesystem	Max total space used by all stored cores combined
KeepFree=	15% of filesystem	Minimum free space to maintain on the filesystem
ExternalSizeMax=	512M	Max size of a single core file
MaxRetentionSec=	0 (disabled)	Time-based expiry; e.g. 1week, 3d

It is possible to override the defaults by creating a file in the drop-in directory like this /etc/systemd/coredump.conf.d/storage.conf (create the drop-in directory first if it does not exist):

[Coredump]
Storage=external
ExternalSizeMax=2G
MaxRetentionSec=2weeks

NOTE: Save location is inside /var/lib/systemd/coredump/

Verifying the Setup

To verify that coredumps are properly captured from a Docker container, we are going to simulate a segfault inside one. But in order to produce anything we need to set the ulimit inside the container (in addition to configuring the core_pattern):

docker-compose.yml

services:
  <container_name>:
    ulimits:
      core:
        soft: -1
        hard: -1

docker run

# --ulimit core=<soft>:<hard>
docker run --ulimit core=-1:-1 ...

However, this validation setup is a little bit convoluted because manually sending SIGSEGV to a process with PID 1 (which the main process in a Docker container has) may be ignored and no coredump will be created. This is not an issue for real segfaults, but it means that the simulated crash must be triggered from a second shell within the container.

Step 1: Start a container which won't do anything for a while.

The bind-mount is only necessary if Option A is used.

docker run -d --rm \
    --name segfault_test \
    --ulimit core=-1:-1 \
    -v /var/coredumps:/var/coredumps \
    debian sleep 3600

Step 2: Connect to this container:

docker exec -it segfault_test bash

Step 3: Inside the container, confirm both settings are in effect:

ulimit -c          # should print: unlimited
cat /proc/sys/kernel/core_pattern   # should show the expected path or pipe

Step 4: Crash the current shell (inside the container) with a segfault:

kill -SIGSEGV $$

Then check that the core file appeared in the expected location on the host.

Working with a coredump file

With coredumpctl (option B only)

# List all recorded crashes.
coredumpctl list

# Show metadata and an embedded backtrace for a specific crash.
coredumpctl info <PID>

# Extract the raw core to a file.
coredumpctl dump <PID> -o /tmp/program_name.core

# Open the core directly in GDB without extracting first.
coredumpctl gdb <PID>

# You can also filter by executable name instead of PID.
coredumpctl dump "program_name" -o /tmp/program_name.core

With GDB directly

Once you have the raw core file (from option A, or extracted via coredumpctl dump), load it in GDB together with the matching executable:

gdb /usr/local/sbin/program_name /tmp/program_name.core

Useful GDB commands once inside:

(gdb) bt          # backtrace: show the call stack at the time of the crash
(gdb) bt full     # backtrace with local variables in each frame
(gdb) info registers   # CPU register state
(gdb) frame 3     # switch to frame 3
(gdb) list        # show source lines around the current frame (requires debug info)

Debug symbols

A stripped release binary (built with --strip and --buildtype release) contains no function names or line numbers. GDB will show only memory addresses, which is of limited use. To get a meaningful backtrace you need one of:

• A debug build of the same binary (--buildtype debug, without --strip).

• A separate unstripped binary built from the same source revision, loaded into GDB with symbol-file:

  (gdb) symbol-file /path/to/unstripped/program_name