How to Fix Docker Exit Code 137 in Kubernetes Pipelines

 If you've ever watched a pod flip to CrashLoopBackOff in the middle of a deploy, checked kubectl describe pod, and seen Exit Code: 137 staring back at you, you already know the feeling — it's rarely obvious why it happened, and it almost never happens in a convenient, reproducible way. It shows up under load, at 2 a.m., during a traffic spike, or halfway through a CI job.

This post breaks down exactly what exit code 137 means, why it's different from a normal application crash, how to diagnose it inside a Kubernetes pipeline, and walks through an actual production incident — a Node.js API that started getting OOMKilled every few hours after a routine dependency upgrade — from first alert to root cause to permanent fix.

What exit code 137 actually means

Exit code 137 isn't a Kubernetes-specific error code. It's arithmetic: 128 + signal number. Signal 9 is SIGKILL, so 128 + 9 = 137.

That distinction matters because SIGKILL cannot be caught, ignored, or handled by your application. There's no graceful shutdown hook, no "cleanup before exit" — the process is terminated immediately at the kernel level. That's your first diagnostic clue: if you see 137, your app didn't choose to exit. Something external killed it.

In the vast majority of Kubernetes cases, that "something external" is the Linux kernel's cgroup OOM killer — not Kubernetes itself. Here's the actual sequence:

1. Your container's memory usage crosses the resources.limits.memory value set in the pod spec.
2. The kernel's cgroup subsystem, which enforces that limit, invokes the OOM killer.
3. The OOM killer sends SIGKILL to the offending process inside the container.
4. The container runtime reports exit code 137 back to the kubelet.
5. The kubelet restarts the pod according to its restartPolicy, and if it keeps happening, backs off into CrashLoopBackOff.

Kubernetes and the container runtime are just the messengers here. The actual decision is made deep in the kernel, which is why kubectl logs on the crashed container is often empty or unhelpful — the process was killed before it could flush a stack trace or write a final log line.

137 vs. other common exit codes

Worth knowing the neighbors, because teams often misdiagnose one as the other:

Exit code	Meaning	Typical cause
137	SIGKILL (128+9)	OOMKilled, or a manual kill -9
143	SIGTERM (128+15)	Graceful shutdown request, often from a rolling deploy or kubectl delete pod
1	Generic application error	Uncaught exception, unhandled promise rejection, failed startup
139	SIGSEGV (128+11)	Segfault, often native code or a bad binary/library mismatch

If you're seeing 143 instead of 137, that's a different problem — usually an app that isn't handling SIGTERM within terminationGracePeriodSeconds, not a memory issue at all.

Confirming it's actually OOMKilled

Before chasing memory limits, confirm the kernel really did the killing. Two quick checks:

bash

kubectl describe pod <pod-name> -n <namespace>

Look for Last State: Terminated, Reason: OOMKilled, and Exit Code: 137 together. If Reason says something else (like Error), 137 may be coming from a manual kill inside a script or a base-image entrypoint issue, not memory pressure.

Cross-check at the node level, since kubectl's pod events get garbage collected fast:

bash

kubectl get events -n <namespace> --field-selector reason=OOMKilling --sort-by='.lastTimestamp'

And if you have node access (or a DaemonSet log shipper), the kernel log entry is unambiguous:

bash

dmesg -T | grep -i "oom"
# or
journalctl -k | grep -i "killed process"

You'll see a line naming the exact process, its memory footprint at time of death, and the cgroup it belonged to. That process name is often the fastest way to tell whether it's your main application, a sidecar, or an unexpected child process (a common surprise with Node.js cluster mode or JVM subprocess spawning) that actually blew the limit.

Production case: a Node.js API that OOMKilled every 3-6 hours after a dependency bump

Here's a real incident pattern, generalized from a mid-sized SaaS backend running a Node.js/Express API on a managed Kubernetes cluster (GKE, but the mechanics are identical on EKS/AKS).

Symptom: After a routine npm update that bumped a logging library and an image-processing dependency, the deployment started restarting roughly every 3-6 hours in production, always with Reason: OOMKilled, Exit Code: 137. Staging never reproduced it — traffic there was too low to trigger it within a normal testing window.

First response — reactive, not a fix: The on-call engineer's first move was to bump the memory limit from 512Mi to 1Gi. This is the most common (and most common mistake) first response to 137. It bought roughly 10 hours before the crashes resumed, which confirmed this wasn't a fixed-size problem — it was a leak, not a limit that was simply too tight. Raising the ceiling only delays the crash; it doesn't remove the cause.

Diagnosis:

1. Metrics first. Pulling up the pod's memory graph in Grafana (fed by kube-state-metrics and cAdvisor) showed a textbook slow, linear climb from baseline (~180Mi) up to the limit over several hours, with no sawtooth pattern from garbage collection settling back down. A healthy Node.js process shows a sawtooth as V8's GC reclaims memory; a monotonic ramp is the signature of a genuine leak — something is being retained that should have been freed.
2. Reproducing under load. Because staging traffic was too light, the team used kubectl port-forward plus a load-testing tool (autocannon) to hammer a single pod for 30 minutes while capturing heap snapshots:

bash

   kubectl exec -it <pod-name> -- node --inspect=0.0.0.0:9229 server.js
   kubectl port-forward <pod-name> 9229:9229

Connecting Chrome DevTools' memory profiler over that forwarded port and comparing heap snapshots taken 10 minutes apart under sustained load showed a steadily growing count of retained Buffer objects — vastly disproportionate to request volume.

3. Root cause. The image-processing dependency that had just been upgraded changed its default behavior: a stream-based resize function that previously released its internal buffer pool after each call started caching decoded image buffers by default in the new major version, intended for repeated-resize workloads. The API was calling it once per upload and never reusing the cache — so every uploaded image permanently retained a decoded buffer in memory for the life of the process, until the container hit its limit and got SIGKILLed.

Fix:

• Explicitly disabled the library's buffer caching option (a one-line config change once the actual cause was known).
• Added --max-old-space-size to the Node.js start command, sized deliberately below the container's memory limit (roughly 75-80% of the limit), so V8 hits its own heap ceiling and can throw/GC-pressure the app before the kernel OOM killer intervenes with an unrecoverable SIGKILL. This turns an unrecoverable kill into something the app or its process manager can at least log and alert on.
• Right-sized the memory limits (and, importantly, requests) based on the corrected steady-state footprint plus headroom — not the inflated 1Gi band-aid from the initial response.
• Added a memory-usage alert at 80% of the limit sustained over 15 minutes, so the next leak (from any dependency, not just this one) surfaces as a warning instead of a crash loop.

The pattern worth remembering: the memory limit told them something was wrong, but the metrics graph shape told them what kind of problem it was. A flat line that suddenly jumps points to a traffic spike or a single bad request; a slow linear climb points to a leak; a sawtooth that gradually rides higher points to fragmentation or a slow-growing cache. Reading that shape before touching resources.limits saves a lot of wasted iteration.

General fixes, from quickest to most durable

1. Rule out the obvious first — limits set too low for legitimate usage. Not every 137 is a leak. Java (JVM heap plus off-heap overhead), and data-processing jobs with legitimately large working sets are frequently just under-provisioned. Compare actual peak usage (via kubectl top pod or your metrics stack) against the configured limit before assuming a bug.

2. Separate requests from limits deliberately. Setting requests far below limits invites the scheduler to overcommit a node, so many pods hit memory pressure simultaneously under load — the OOM killer then picks a victim by its scoring heuristic, which isn't always the pod actually responsible. Keeping requests reasonably close to real steady-state usage gives more predictable scheduling and OOM behavior.

3. Let the runtime respect the container limit. For JVMs, use -XX:MaxRAMPercentage (or ensure you're on a JDK version that reads cgroup limits natively) rather than an unbounded default heap. For Node.js, set --max-old-space-size below the container limit, as in the case above. The goal is the same in both: let the language runtime hit its own ceiling and possibly log or degrade gracefully, rather than letting the kernel do an unrecoverable SIGKILL with zero application-level warning.

4. Add memory-based alerting before the limit, not at it. An alert firing at 100% of the limit is just a delayed OOMKill notification. Alert at 75-85% sustained over a meaningful window so there's time to react.

5. Check for runaway child processes. Especially relevant for Node.js cluster mode, Python multiprocessing, or any app spawning subprocesses (image/video transcoding, headless browsers) — a leak in a spawned worker counts against the same cgroup memory limit as the parent, and kubectl top pod reports the aggregate, which can mask which specific process is actually responsible until you check dmesg.

6. In CI/build pipelines specifically, 137 often shows up not from application memory but from the build process itself — large npm install/webpack builds, big Docker layer builds, or in-cluster test suites with in-memory databases. If it's a build step exit-coding 137 rather than a running pod, check the build agent/runner's resource limits (Kubernetes-based CI runners have their own pod resource requests/limits, separate from your app's deployment manifest) rather than assuming it's the same class of bug as an application-level leak.

A quick diagnostic checklist

• Confirm Reason: OOMKilled in kubectl describe pod, not just exit code 137 in isolation
• Pull the memory graph — is it flat-then-spike, sawtooth-rising, or linear climb?
• Compare kubectl top pod peak usage against resources.limits.memory
• Check dmesg/kernel logs on the node for the exact process name killed
• If it's a leak: reproduce under sustained load with a profiler before touching limits
• Set the runtime's own memory ceiling (JVM heap %, Node's --max-old-space-size, etc.) below the container limit
• Alert at 75-85% of the limit, not at 100%
• Only raise resources.limits.memory as a stopgap — track it as a follow-up, not the fix

Exit code 137 is ultimately a symptom, not a root cause — it's the kernel telling you a boundary was crossed. The fastest path to a real fix is almost always: confirm it's genuinely OOMKilled, read the shape of the memory graph before changing any YAML, and reproduce under load if the graph says "leak" rather than "under-provisioned."