In brief
A failed container is a lifecycle symptom. Identify whether the failure occurs while pulling the image, creating the container, starting its main process, passing health checks, or serving traffic before changing the host or rebuilding the image.
Key takeaways
- Capture container state, exit code, OOM status, timestamps, and logs before recreating it.
- Verify the effective entrypoint, command, environment, runtime user, mounts, and architecture.
- Separate process startup, container health, published ports, and dependency availability.
- Make one reversible correction and verify the same image through the real request path.
Locate the failing lifecycle stage
Begin with the exact image reference, container name, Docker context, host, command, and UTC timestamp. Determine whether Docker failed while resolving or pulling the image, creating the container configuration, starting the main process, evaluating its health check, or forwarding a request. These stages have different owners and different evidence, so treating every symptom as an application crash wastes time.
List all containers, including stopped ones, and inspect the affected container before removing it. Record its status, exit code, error field, out-of-memory flag, start and finish times, restart count, configured image digest, and health-check output. Recreating the container may restore service, but it also removes the easiest path to the original state unless logs and metadata are retained elsewhere.
- Use `docker ps --all --no-trunc` to preserve the complete command and current state.
- Use `docker inspect <container>` for effective configuration and runtime state.
- Use `docker logs --timestamps --tail 300 <container>` around the first failure.
- Use `docker events --since 30m` to correlate starts, stops, kills, and health changes.
Interpret exit behavior instead of guessing
A container remains alive only while its main process is running. Exit code zero can mean the configured task completed normally even though the operator expected a long-running service. A non-zero code can indicate invalid arguments, a missing executable, an application exception, a permissions failure, an architecture mismatch, or termination by the runtime. Read the application log and the container state together.
Exit code 137 is commonly associated with a forced kill and can accompany memory pressure, but the number alone does not prove an out-of-memory event. Confirm the OOM flag, container memory limit, host memory, application heap settings, and resource trend. Raising the limit without understanding growth can delay rather than solve another production failure.
Verify image, entrypoint, and runtime configuration
Inspect the image reference and digest actually used by the container. Confirm that the image supports the host operating system and CPU architecture, the entrypoint exists and is executable, scripts have compatible line endings and a valid interpreter, and every runtime file copied by a multi-stage build is present. An `exec format error` often points to architecture, shebang, or line-ending problems rather than networking.
Compare the effective entrypoint, command, working directory, environment variables, runtime user, and mounted files with the reviewed deployment definition. A successful image build does not prove that the application has its production configuration. Do not print credentials while comparing configuration, and do not solve a missing secret by baking it into a replacement image.
- Run the exact image locally with a controlled environment when reproduction is safe.
- Use an alternative entrypoint only for diagnosis, not as the permanent application command.
- Confirm the non-root user can read the application and write only required paths.
- Check that bind mounts have not hidden files originally supplied by the image.
Separate health checks from process health
A running process can still be unhealthy, and an unhealthy container can still have an active process. Inspect the health-check command, interval, timeout, start period, retry count, and individual results. Execute the same lightweight check from the relevant network namespace when possible. Verify that the required diagnostic binary exists in the runtime image and that the endpoint does not require authentication or expensive downstream work.
Health semantics should match the deployment platform. Readiness asks whether the instance should receive traffic, while liveness asks whether restarting can recover it. A Docker health check exposes health status, but an external platform may use a different probe to control routing or restart behavior. Diagnose both before changing an application that is healthy from one perspective and unavailable from another.
Trace ports, DNS, dependencies, and storage
When the container is running but unreachable, confirm the application listens on the expected container port and on an address reachable outside its own loopback interface. Compare that with the published host address and port, firewall, reverse proxy, TLS configuration, and upstream health check. Publishing a port does not repair an application that only listens on the wrong port or exits during startup.
Inside a container, `localhost` refers to that container. Reach another container by its service name on a shared user-defined network. For storage failures, compare the runtime user with volume or bind-mount ownership and inspect whether the mount target contains the expected data. Avoid world-writable permissions and broad network exposure as diagnostic shortcuts.
Recover with the smallest evidence-backed change
Correct one confirmed issue: restore the intended image digest, fix the application command, supply the missing configuration through the approved secret mechanism, align mount ownership, change the published port, or adjust a health check whose contract is wrong. Rebuild only when the image content is the failure boundary. Restarting an unchanged container is a temporary mitigation, not proof of root cause.
After recovery, verify the container state, health history, resource use, logs, dependency calls, published endpoint, and a representative user transaction. Observe the service beyond the first successful request. Record the failure stage, decisive evidence, correction, and a preventive control such as an image test, startup gate, resource alert, configuration validation, or immutable deployment digest.
Prevent the next startup incident
Test the built image—not only the source code—in CI. Start it with production-like resource limits, wait for the intended health signal, exercise a representative request, send a termination signal, and confirm graceful shutdown. Scan the final image, retain build provenance according to policy, and promote the same digest rather than rebuilding independently for each environment.
Use restart policies deliberately. They can recover a service after some failures, but they can also hide a crash loop and consume resources. Alert on restart count, health transitions, memory pressure, and repeated startup errors. Keep centralized logs available even when a container is removed or replaced.