J9 OSR buffer corruption by SIGPROF signal frames - reproducer for eclipse-openj9/openj9#24487

README.md

J9 OSR Buffer Corruption by SIGPROF Signal Frames — Reproducer

Self-standing reproducer for a memory corruption on OpenJ9 (J9) x86-64 Linux with the Datadog java-profiler CPU sampler running. The corruption is caused by the Linux kernel writing a signal frame over the J9 OSR (On-Stack Replacement) buffer, not by any agent code.

Root cause (evidence in doc/J9OSRSignalFrameCorruption.md)

1. On OpenJ9 x86-64, the JIT OSR transition temporarily runs with the hardware stack pointer (rsp) pointing into a heap-allocated OSR block (osrJittedFrameCopy), not the thread native stack. (runtime/codert_vm/decomp.cpp:2047-2087, runtime/oti/xhelpers.m4:313-315)
2. The Linux kernel writes the signal frame below the interrupted rsp unless the handler uses SA_ONSTACK (arch/x86/kernel/signal.c:99-121). The frame includes the xsave area whose trailer is FP_XSTATE_MAGIC2 (0x46505845 = bytes "EXPF") (arch/x86/include/uapi/asm/sigcontext.h:21-35).
3. The profiler CPU sampler installs SIGPROF handlers without SA_ONSTACK (ddprof-lib/src/main/cpp/os_linux.cpp:306-318) and on OpenJ9 uses the SIGPROF+ASGCT path (profiler.cpp:1277-1292).
4. A SIGPROF delivered while rsp is in the OSR block makes the kernel write the signal frame over the OSR buffer (J9OSRBuffer.numberOfFrames, J9OSRFrame.numberOfLocals), corrupting it.

Files

• heap_sigframe.c — minimal C reproducer of the kernel mechanism: switches rsp to a heap buffer, delivers SIGPROF (no SA_ONSTACK), and finds FP_XSTATE_MAGIC2 ("EXPF") written into the heap buffer below rsp. Deterministic.
• HCRRepro.java — J9 reproducer: a Java agent that repeatedly retransforms its own class (forcing HCR/OSR) while a hot worker thread runs, under the profiler high-frequency SIGPROF.
• osr-detection.patch — minimal profiler change: in CTimer::signalHandler, when a SIGPROF signal frame overlaps the OSR heap block, abort with a clear message. Turns the silent corruption into a deterministic failure.
• run-repro.sh — builds the agent jar and runs the reproducer.

How to run

Requires: x86-64 Linux, an OpenJ9 JDK 8 (Semeru), and a profiler jar built with osr-detection.patch applied.

# 1. build the profiler with the detection patch
./gradlew buildDebug -Pskip-tests

# 2. compile the Java reproducer
javac -cp ddprof-<version>-debug.jar -d classes HCRRepro.java

# 3. run (deterministic abort)
./run-repro.sh ddprof-<version>-debug.jar /path/to/openj9-jdk8/bin

# expected output:
# === DETERMINISTIC REPRO: SIGPROF signal frame overlaps J9 OSR buffer ===
# JVMDUMP042W Abort signal received while running on Java stack.

# control (JVMTI sampler, no SIGPROF) — runs clean:
java -Ddd.profiling.ddprof.j9.sampler=jvmti -javaagent:osr-hcr-agent.jar \
     -cp classes HCRRepro 100 20000

Kernel mechanism reproducer (standalone, no JVM)

gcc -O0 -g -o heap_sigframe heap_sigframe.c && ./heap_sigframe
# found FP_XSTATE_MAGIC2 at offset=... distance_below_top=...

Evidence captured by the detection

With the patch, each SIGPROF in the OSR window logs: rsp=0x221e60 rip= expf=1 expf_off=3944 osr_hit=1 — rsp in the OSR heap block, rip in the JIT code cache (JIT running on the OSR heap stack), and the signal frame (EXPF) present 3944 bytes below rsp, overlapping the OSR buffer.

HCRRepro.java

import java.lang.instrument.Instrumentation;
import java.lang.instrument.ClassFileTransformer;
import java.security.ProtectionDomain;
import java.util.concurrent.atomic.AtomicInteger;

public class HCRRepro {
    public static volatile boolean running = true;
    static volatile long counter = 0;
    static Instrumentation inst;
    static volatile int retransCount = 0;
    static AtomicInteger version = new AtomicInteger(0);

    public static void premain(String args, Instrumentation instrumentation) throws Exception {
        inst = instrumentation;
        instrumentation.addTransformer(new ClassFileTransformer() {
            public byte[] transform(ClassLoader loader, String className, Class<?> classBeingRedefined,
                    ProtectionDomain protectionDomain, byte[] classfileBuffer) {
                // Only transform our own class, and only on retransformation (not initial load)
                if (className != null && className.equals("HCRRepro") && classBeingRedefined != null) {
                    return patch(classfileBuffer);
                }
                return null;
            }
        }, true);
        System.out.println("[agent] transformer added");
    }

    // Patch: change the constant 31 to a different value each retransform to force real HCR
    static byte[] patch(byte[] bytes) {
        int v = version.incrementAndGet();
        // find the bytecode for "sipush 31" (0x11 0x00 0x1f) or bipush 31 (0x10 0x1f) in hotMethod
        // We just change a constant: search for bipush 31 (0x10 0x1f)
        for (int i = 0; i < bytes.length - 1; i++) {
            if (bytes[i] == (byte)0x10 && bytes[i+1] == (byte)31) {
                bytes[i+1] = (byte)(31 + (v % 10));
                return bytes;
            }
        }
        return bytes;
    }

    public static long hotMethod(long x, int iter) {
        long acc = x;
        for (int i = 0; i < iter; i++) {
            acc = acc * 31 + i;
            if ((acc & 0x1) == 0) acc ^= 0x12345678L;
        }
        return acc;
    }

    public static void main(String[] args) throws Exception {
        int intervalUs = args.length > 0 ? Integer.parseInt(args[0]) : 100;
        long runMs = args.length > 1 ? Long.parseLong(args[1]) : 30000;

        com.datadoghq.profiler.JavaProfiler profiler = com.datadoghq.profiler.JavaProfiler.getInstance();
        System.out.println("[main] profiler=" + profiler);
        profiler.execute("start,cpu,interval=" + intervalUs + "us,file=/tmp/osr.jfr");
        System.out.println("[main] profiler started interval=" + intervalUs + "us");

        Thread worker = new Thread(() -> {
            long acc = 1;
            int iter = 3000;
            while (running) {
                acc = hotMethod(acc, iter);
                counter = acc;
            }
        }, "osr-worker");
        worker.start();

        Thread.sleep(3000);

        long deadline = System.currentTimeMillis() + runMs;
        int i = 0;
        while (System.currentTimeMillis() < deadline && running) {
            try {
                inst.retransformClasses(HCRRepro.class);
                retransCount++;
            } catch (Throwable t) {
                if (i % 500 == 0) System.out.println("[main] retransform error: " + t);
            }
            i++;
            if (i % 1000 == 0) System.out.println("[main] retransform #" + i + " ok=" + retransCount);
            Thread.sleep(1);
        }
        running = false;
        worker.join(5000);
        System.out.println("[main] done retrans=" + retransCount + " counter=" + counter);
        profiler.execute("stop");
        System.out.println("[main] stopped");
        System.exit(0);
    }
}

heap_sigframe.c

#define _GNU_SOURCE
#include <signal.h>
#include <stdint.h>
#include <stdio.h>
#include <string.h>
#include <sys/mman.h>
#include <sys/syscall.h>
#include <unistd.h>

static volatile sig_atomic_t seen;

static void handler(int sig, siginfo_t *si, void *uc) {
    (void)sig; (void)si; (void)uc;
    seen++;
}

__attribute__((noinline)) static void trigger_signal(void) {
    syscall(SYS_tgkill, getpid(), (pid_t)syscall(SYS_gettid), SIGPROF);
}

__attribute__((noinline)) static void run_with_rsp(void *top) {
#if defined(__x86_64__)
    asm volatile(
        "mov %%rsp, %%r15\n\t"
        "mov %0, %%rsp\n\t"
        "andq $-16, %%rsp\n\t"
        "call trigger_signal\n\t"
        "mov %%r15, %%rsp\n\t"
        :
        : "r"(top)
        : "r15", "memory", "cc");
#else
#error x86_64 only
#endif
}

int main(void) {
    enum { SZ = 65536, TOP = 49152 };
    uint8_t *buf = mmap(NULL, SZ, PROT_READ | PROT_WRITE,
                        MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
    if (buf == MAP_FAILED) { perror("mmap"); return 1; }
    memset(buf, 0xcc, SZ);

    struct sigaction sa;
    memset(&sa, 0, sizeof(sa));
    sa.sa_sigaction = handler;
    sigemptyset(&sa.sa_mask);
    sa.sa_flags = SA_SIGINFO; /* deliberately no SA_ONSTACK */
    sigaction(SIGPROF, &sa, NULL);

    run_with_rsp(buf + TOP);

    printf("seen=%d\n", (int)seen);
    int found = 0;
    for (size_t i = 0; i + 4 <= SZ; i++) {
        if (buf[i] == 0x45 && buf[i+1] == 0x58 &&
            buf[i+2] == 0x50 && buf[i+3] == 0x46) {
            printf("found FP_XSTATE_MAGIC2 at offset=%zu distance_below_top=%zu\n",
                   i, (size_t)(TOP - i));
            found++;
        }
    }
    printf("total magic found: %d\n", found);
    return 0;
}

osr-detection.patch

--- a/ddprof-lib/src/main/cpp/ctimer_linux.cpp
+++ b/ddprof-lib/src/main/cpp/ctimer_linux.cpp
@@ -256,6 +256,7 @@ void CTimer::signalHandler(int signo, siginfo_t *siginfo, void *ucontext) {
   }
   Counters::increment(CTIMER_SIGNAL_OWN);
 
+  // DETERMINISTIC REPRODUCER: detect when a SIGPROF signal frame overlaps the
+  // J9 OSR heap block and abort. On OpenJ9 x86-64, JIT OSR code runs with
+  // %rsp on a heap-allocated OSR block (osrJittedFrameCopy). A SIGPROF
+  // delivered there makes the kernel write the signal frame (xsave area with
+  // FP_XSTATE_MAGIC2 / "EXPF" trailer) below %rsp, clobbering the J9OSRBuffer
+  // (numberOfFrames / numberOfLocals). This is the corruption observed in
+  // production. See doc/J9OSRSignalFrameCorruption.md.
+  {
+    static __thread int diag_count = 0;
+    if (diag_count < 200000) {
+      diag_count++;
+      unsigned long rsp = 0;
+      unsigned long rip = 0;
+#if defined(__x86_64__)
+      ucontext_t *uc = (ucontext_t *)ucontext;
+      rsp = uc->uc_mcontext.gregs[REG_RSP];
+      rip = uc->uc_mcontext.gregs[REG_RIP];
+#endif
+      // count EXPF magic (45 58 50 46) in the 4KB below interrupted rsp
+      int expf = 0;
+      long expf_off = -1;
+      const unsigned char *p = (const unsigned char *)(rsp - 4096);
+      for (int k = 0; k < 4096 - 4; k++) {
+        if (p[k]==0x45 && p[k+1]==0x58 && p[k+2]==0x50 && p[k+3]==0x46) { expf++; if (expf_off<0) expf_off=k; k+=4; }
+      }
+      // detect a J9OSRBuffer-like struct (small numberOfFrames + code ptr jitPC)
+      // inside the signal frame data region (between rsp and EXPF)
+      int osr_hit = 0;
+      if (expf_off > 0) {
+        for (long off = 0; off + 16 <= expf_off; off += 8) {
+          unsigned long nf = *(const unsigned long *)(p + off);
+          unsigned long jp = *(const unsigned long *)(p + off + 8);
+          if (nf < 4096 && (jp > 0x700000000000UL || (jp >= 0x10000UL && jp <= 0x3210000UL))) {
+            osr_hit++;
+            break;
+          }
+        }
+      }
+      // abort when a SIGPROF signal frame overlaps the OSR heap block
+      if (rsp >= 0x10000UL && rsp <= 0x3210000UL && osr_hit) {
+        static volatile int aborted = 0;
+        if (!aborted) {
+          aborted = 1;
+          const char *m = "\n=== DETERMINISTIC REPRO: SIGPROF signal frame overlaps J9 OSR buffer ===\n";
+          (void)write(2, m, strlen(m));
+          raise(SIGABRT);
+        }
+      }
+    }
+  }
+
   InflightGuard inflight;

run-repro.sh

#!/usr/bin/env bash
# J9 OSR buffer corruption by SIGPROF signal frames — deterministic reproducer
#
# Usage:
#   ./run-repro.sh <path-to-ddprof-debug.jar> <path-to-openj9-jdk8-home> [classes-dir]
#
# Reproduces: the Datadog java-profiler CPU sampler (SIGPROF+ASGCT on OpenJ9)
# delivers SIGPROF while JIT code runs with %rsp on the OSR heap block. The
# kernel writes a signal frame (xsave, FP_XSTATE_MAGIC2/EXPF trailer) below
# %rsp, clobbering the J9 OSR buffer. The profiler aborts deterministically
# when it detects the overlap.
#
# Control: add -Ddd.profiling.ddprof.j9.sampler=jvmti to use the JVMTI
# sampler (no SIGPROF) — the reproducer then runs clean.

set -u
JAR=$(readlink -f "${1:?usage: run-repro.sh <ddprof.jar> <openj9-jdk8-home> [classes-dir]}")
J8=${2:?usage: run-repro.sh <ddprof.jar> <openj9-jdk8-home> [classes-dir]}
SCRIPT_DIR=$(cd "$(dirname "$0")" && pwd)
CLASSES=${3:-$(cd "$SCRIPT_DIR/.." && pwd)/classes}

WORK=$(mktemp -d)
trap "rm -rf $WORK" EXIT
cd "$WORK"

# Compile HCRRepro.java if the classes are not already built
if [ ! -f "$CLASSES/HCRRepro.class" ]; then
  mkdir -p "$CLASSES"
  "$J8/bin/javac" -cp "$JAR" -d "$CLASSES" "$SCRIPT_DIR/HCRRepro.java" || {
    echo "ERROR: could not compile HCRRepro.java"; exit 2; }
fi

# Build the agent jar (HCRRepro + embedded profiler jar)
mkdir -p agent
cp "$CLASSES"/HCRRepro*.class agent/
cp "$JAR" agent/ddprof.jar
cat > agent/MANIFEST.MF <<MANIFEST
Manifest-Version: 1.0
Premain-Class: HCRRepro
Can-Retransform-Classes: true
Class-Path: ddprof.jar
MANIFEST
(cd agent && "$J8/bin/jar" cfm ../osr-hcr-agent.jar MANIFEST.MF HCRRepro*.class ddprof.jar)
cp agent/ddprof.jar ./ddprof.jar   # for Class-Path resolution next to the agent jar

echo "=== Running reproducer: SIGPROF + OSR (expect deterministic abort) ==="
"$J8/bin/java" -javaagent:osr-hcr-agent.jar -cp "$JAR":"$CLASSES" HCRRepro 100 20000
rc=$?
echo "exit=$rc (SIGABRT=134 = reproduced)"