Docker Container Health Checks: Testing and Monitoring Container Readiness

Docker Container Health Checks: Testing and Monitoring Container Readiness

Docker HEALTHCHECK defines a command Docker runs periodically to determine container health. This guide covers writing HEALTHCHECK instructions, using health status in Compose depends_on conditions, testing health checks in CI, and debugging containers stuck in "starting" or "unhealthy" states.

A running container is not necessarily a healthy container. Docker's HEALTHCHECK instruction adds an active liveness check: a command Docker runs on a schedule and uses to determine whether the container is actually serving traffic. Without it, orchestrators like Compose and Kubernetes can route traffic to containers that have started but can't handle requests yet.

HEALTHCHECK Instruction Syntax

HEALTHCHECK [OPTIONS] CMD command

Options:

Option Default Meaning
--interval 30s How often to run the check
--timeout 30s How long to wait before marking as failed
--start-period 0s Grace period after container start
--start-interval 5s Check interval during start period
--retries 3 Failures before marking unhealthy

The CMD must exit with code 0 (healthy) or 1 (unhealthy).

Writing Health Check Commands

HTTP API

FROM node:20-alpine
WORKDIR /app
COPY . .
RUN npm ci

HEALTHCHECK \
  --interval=30s \
  --timeout=10s \
  --start-period=15s \
  --retries=3 \
  CMD wget -qO- http://localhost:3000/health || exit 1

EXPOSE 3000
CMD ["node", "server.js"]

Use wget (available in Alpine) or curl:

HEALTHCHECK --interval=30s --timeout=5s \
  CMD curl -f http://localhost:8080/actuator/health || exit 1

The -f flag makes curl exit with code 22 on HTTP errors (4xx, 5xx), which Docker interprets as unhealthy.

Database Readiness

FROM postgres:15

HEALTHCHECK \
  --interval=5s \
  --timeout=5s \
  --start-period=10s \
  --retries=5 \
  CMD pg_isready -U postgres -d mydb || exit 1

pg_isready checks that PostgreSQL accepts connections. It exits 0 when ready, 1 when refusing connections.

For MySQL/MariaDB:

HEALTHCHECK --interval=5s --timeout=5s --retries=5 \
  CMD mysqladmin ping -h localhost -u root --password=$$MYSQL_ROOT_PASSWORD || exit 1

TCP Port Check

When there's no HTTP endpoint or CLI tool available:

HEALTHCHECK --interval=10s --timeout=5s \
  CMD nc -z localhost 6379 || exit 1

nc -z attempts a TCP connection without sending data—useful for Redis, memcached, or custom TCP services.

Custom Application Logic

For complex readiness checks, write a script:

#!/bin/sh
# healthcheck.sh

# Check HTTP endpoint
response=$(curl -sf http://localhost:8080/health)
if [ $? -ne 0 ]; then
  echo "HTTP check failed"
  exit 1
fi

# Verify response body
status=$(echo "$response" | jq -r '.status')
if [ "$status" != "UP" ]; then
  echo "Status is $status, expected UP"
  exit 1
fi

# Check dependent service connectivity
if ! nc -z db 5432; then
  echo "Cannot reach database"
  exit 1
fi

exit 0
COPY healthcheck.sh /usr/local/bin/
RUN chmod +x /usr/local/bin/healthcheck.sh
HEALTHCHECK --interval=15s --timeout=10s CMD /usr/local/bin/healthcheck.sh

Checking Health Status

# View health status
docker inspect --format='{{.State.Health.Status}}' my-container

# View health check log (last 5 runs)
docker inspect --format='{{json .State.Health}}' my-container | jq '.Log[-5:]'

# Watch health status in real time
watch -n 2 'docker inspect --format="{{.State.Health.Status}}" my-container'

Health statuses:

  • starting — within the start period, no result yet
  • healthy — last check succeeded
  • unhealthy — retries exhausted with failures
  • none — no HEALTHCHECK defined

Docker Compose: Conditional Service Dependencies

Health checks unlock condition: service_healthy in Compose:

version: '3.8'
services:
  db:
    image: postgres:15
    environment:
      POSTGRES_PASSWORD: secret
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgres"]
      interval: 5s
      timeout: 5s
      retries: 5
      start_period: 10s

  app:
    build: .
    depends_on:
      db:
        condition: service_healthy  # Wait for db to be healthy
    environment:
      DATABASE_URL: postgresql://postgres:secret@db/mydb
    ports:
      - "8080:8080"

Without condition: service_healthy, Compose starts app as soon as the db container starts—before PostgreSQL finishes initialization. With it, Compose waits until pg_isready returns 0.

Testing Health Checks in CI

Wait for Healthy Before Running Tests

# docker-compose.test.yml
version: '3.8'
services:
  app:
    build: .
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/health"]
      interval: 5s
      timeout: 5s
      retries: 10
      start_period: 15s
    ports:
      - "8080:8080"
# CI script
docker compose -f docker-compose.test.yml up -d app

# Wait for healthy status
timeout 120 bash -c '
  until [ "$(docker inspect --format="{{.State.Health.Status}}" app)" = "healthy" ]; do
    echo "Waiting for app to be healthy..."
    sleep 3
  done
'

echo "App is healthy, running tests"
npm test

GitHub Actions Example

- name: Start services
  run: docker compose up -d

- name: Wait for services to be healthy
  run: |
    echo "Waiting for all services..."
    for service in app db redis; do
      timeout 60 bash -c "
        until docker compose ps $service | grep -q healthy; do
          sleep 2
        done
      "
      echo "$service is healthy"
    done

- name: Run integration tests
  run: npm run test:integration

Kubernetes Equivalents

Kubernetes offers more granular health probes than Docker's single HEALTHCHECK:

apiVersion: apps/v1
kind: Deployment
spec:
  template:
    spec:
      containers:
        - name: app
          image: myapp:latest
          
          # Startup probe: give the container time to boot
          startupProbe:
            httpGet:
              path: /health
              port: 8080
            failureThreshold: 30  # 30 * 10s = 5 min max startup
            periodSeconds: 10
          
          # Liveness probe: restart if unhealthy
          livenessProbe:
            httpGet:
              path: /health/live
              port: 8080
            initialDelaySeconds: 10
            periodSeconds: 15
            failureThreshold: 3
          
          # Readiness probe: remove from load balancer if not ready
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            periodSeconds: 10
            failureThreshold: 3

The key difference: Docker HEALTHCHECK = all-or-nothing container health. Kubernetes separates startup (allow slow boot), liveness (detect deadlocks), and readiness (stop sending traffic during maintenance).

Debugging Failing Health Checks

Check the HEALTHCHECK Command Directly

Run the health check command inside the container to see its output:

# Run the check manually
docker exec my-container curl -f http://localhost:8080/health

# Run with the exact command from HEALTHCHECK
docker exec my-container sh -c 'curl -f http://localhost:8080/health || exit 1'

Inspect Health History

docker inspect my-container | jq '.State.Health.Log'

Output includes stdout, stderr, and exit code for each recent check—far more informative than just the status string.

Common Failure Causes

curl: command not found: The base image doesn't include curl. Use wget instead, or add RUN apk add --no-cache curl (Alpine) or RUN apt-get install -y curl (Debian).

Timeout too short: The app is slow to respond on the health endpoint during startup. Increase --start-period or --timeout.

Wrong port or path: The health check URL doesn't match what the app listens on. Verify with docker exec container curl localhost:PORT/PATH.

Permission denied: The health check script isn't executable. Add chmod +x in the Dockerfile.

Container exits before check runs: The app crashed on startup. Check docker logs container for application errors—health check failures are secondary to the main process dying.

Designing Good Health Endpoints

The /health endpoint should:

  1. Respond fast: < 50ms. Don't run database queries in health checks.
  2. Be distinct from business logic: No authentication required, no business data returned.
  3. Check critical dependencies: Verify database connectivity and any other services the app can't function without.
  4. Return meaningful status: A JSON body with component-level status helps debugging.
// Express.js example
app.get('/health', async (req, res) => {
  const checks = {
    app: 'healthy',
    database: 'unknown',
  };

  try {
    await db.raw('SELECT 1');
    checks.database = 'healthy';
  } catch (err) {
    checks.database = 'unhealthy';
  }

  const allHealthy = Object.values(checks).every(v => v === 'healthy');
  res.status(allHealthy ? 200 : 503).json({
    status: allHealthy ? 'UP' : 'DOWN',
    checks
  });
});

The /health/live and /health/ready split from Kubernetes: liveness just checks the process is alive (always return 200 unless the process is deadlocked), readiness checks whether dependencies are available.

Summary

Docker HEALTHCHECK transforms "container is running" into "container is serving requests." Use it in every service image: HTTP endpoints get curl -f, databases get CLI tools like pg_isready, TCP services get nc -z. In Compose, condition: service_healthy eliminates startup race conditions between dependent services. In CI, wait for healthy status before running integration tests—this prevents false failures from tests starting before the application is ready.

Read more

Start now free