Testing

Testing Container Images: Trivy, Grype, and Structure Tests

HelpMeTest

HelpMeTest

— 6 min read
Testing Container Images: Trivy, Grype, and Structure Tests

Shipping a container image without testing it is like deploying code without running tests. The image might contain known CVEs in its base layer, misconfigured file permissions, unnecessary packages that expand your attack surface, or missing dependencies that cause runtime failures. All of these are detectable before the image ever reaches production — if you add image testing to your pipeline.

Four tools give you comprehensive coverage: Trivy and Grype for vulnerability scanning, Container Structure Tests for verifying image contents and behavior, and Dockle for Dockerfile best practices.

Vulnerability Scanning with Trivy

Trivy is the most widely adopted container scanning tool. It scans OS packages, language-specific dependencies (npm, pip, gem, go modules), and Kubernetes manifests for known CVEs:

# Install
brew install trivy   <span class="hljs-comment"># macOS
<span class="hljs-comment"># or
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh <span class="hljs-pipe">| sh -s -- -b /usr/local/bin

<span class="hljs-comment"># Scan an image
trivy image myapp:latest

<span class="hljs-comment"># Scan with severity filter — only HIGH and CRITICAL
trivy image --severity HIGH,CRITICAL myapp:latest

<span class="hljs-comment"># Output as JSON for programmatic processing
trivy image --format json --output results.json myapp:latest

<span class="hljs-comment"># Fail the command if any HIGH or CRITICAL CVE is found (for CI)
trivy image --exit-code 1 --severity HIGH,CRITICAL myapp:latest

A typical scan output:

myapp:latest (ubuntu 22.04)
================================
Total: 12 (HIGH: 3, CRITICAL: 1)

┌─────────────────────┬────────────────┬──────────┬───────────────────┬───────────────┬──────────────────────────────────────────┐
│       Library       │ Vulnerability  │ Severity │ Installed Version │ Fixed Version │                  Title                   │
├─────────────────────┼────────────────┼──────────┼───────────────────┼───────────────┼──────────────────────────────────────────┤
│ libssl3             │ CVE-2024-0727  │ CRITICAL │ 3.0.2-0ubuntu1.12 │ 3.0.2-0ubuntu1.15 │ OpenSSL: denial of service via null dereference │
│ libc-bin            │ CVE-2023-4527  │ HIGH     │ 2.35-0ubuntu3.3   │ 2.35-0ubuntu3.4   │ glibc: stack read overflow in getaddrinfo │

Trivy also scans your repository directly — not just built images:

# Scan filesystem (finds vulnerabilities in source dependencies)
trivy fs --security-checks vuln,secret .

<span class="hljs-comment"># Scan for secrets committed to the codebase
trivy fs --security-checks secret .

<span class="hljs-comment"># Scan a Dockerfile for misconfigurations
trivy config Dockerfile

Vulnerability Scanning with Grype

Grype from Anchore is Trivy's main alternative. It uses a different vulnerability database (Anchore's), which means running both catches CVEs that only appear in one database:

# Install
curl -sSfL https://raw.githubusercontent.com/anchore/grype/main/install.sh <span class="hljs-pipe">| sh -s -- -b /usr/local/bin

<span class="hljs-comment"># Scan an image
grype myapp:latest

<span class="hljs-comment"># Fail on HIGH or CRITICAL
grype myapp:latest --fail-on high

<span class="hljs-comment"># Output SARIF for GitHub Security tab
grype myapp:latest -o sarif > grype-results.sarif

<span class="hljs-comment"># Scan a tarball (for scanning before pushing to registry)
docker save myapp:latest <span class="hljs-pipe">| grype -

Use a .grype.yaml config file to suppress known false positives:

# .grype.yaml
ignore:
  # Known false positive — not exploitable in our usage
  - vulnerability: CVE-2023-12345
    package:
      name: libcurl
      version: "7.81.0"
    reason: "libcurl is only used for internal mTLS connections, attack vector not applicable"

  # Entire package not present at runtime (build-only dependency)
  - package:
      name: gcc
      type: rpm

Container Structure Tests

The Container Structure Test framework from Google verifies the contents, file permissions, commands, and metadata of built images — not their vulnerability posture, but their correctness.

Install:

curl -LO https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64
chmod +x container-structure-test-linux-amd64
<span class="hljs-built_in">sudo <span class="hljs-built_in">mv container-structure-test-linux-amd64 /usr/local/bin/container-structure-test

Write a structure test file:

# tests/container-structure-test.yaml
schemaVersion: "2.0.0"

commandTests:
  - name: "application starts"
    command: "myapp"
    args: ["--version"]
    expectedOutput:
      - "myapp version \\d+\\.\\d+\\.\\d+"
    exitCode: 0

  - name: "non-root user"
    command: "id"
    args: []
    expectedOutput:
      - "uid=1001"
    excludedOutput:
      - "uid=0"
    exitCode: 0

  - name: "Python version constraint"
    command: "python3"
    args: ["--version"]
    expectedOutput:
      - "Python 3\\.1[0-9]"
    exitCode: 0

fileExistenceTests:
  - name: "config directory exists"
    path: "/etc/myapp"
    shouldExist: true
    isDirectory: true

  - name: "binary is present"
    path: "/usr/local/bin/myapp"
    shouldExist: true
    permissions: "-rwxr-xr-x"
    uid: 1001
    gid: 1001

  - name: "secrets directory must not exist"
    path: "/tmp/secrets"
    shouldExist: false

fileContentTests:
  - name: "environment is production"
    path: "/etc/myapp/app.conf"
    expectedContents:
      - "env=production"
    excludedContents:
      - "debug=true"
      - "DEV_MODE"

metadataTest:
  exposedPorts:
    - "8080/tcp"
  volumes: []
  cmd:
    - "/usr/local/bin/myapp"
    - "serve"
  workdir: "/app"
  user: "1001"
  labels:
    - key: "org.opencontainers.image.source"
      value: "https://github.com/mycompany/myapp"

Run the tests:

container-structure-test test \
  --image myapp:latest \
  --config tests/container-structure-test.yaml

Structure tests are especially valuable for catching:

  • Missing binaries or libraries that would cause runtime failures
  • Wrong file permissions (writable config files, world-readable secrets)
  • Leftover build artifacts (.git, node_modules, build caches)
  • Wrong user (running as root when the container should use a non-root user)
  • Missing required environment variables or configuration files

Dockle: Dockerfile Best Practices

Dockle checks your image against CIS Docker Benchmark and other security best practices:

# Install
brew install goodwithtech/r/dockle   <span class="hljs-comment"># macOS
<span class="hljs-comment"># or
VERSION=$(curl --silent <span class="hljs-string">"https://api.github.com/repos/goodwithtech/dockle/releases/latest" <span class="hljs-pipe">| grep <span class="hljs-string">'"tag_name"' <span class="hljs-pipe">| sed -E <span class="hljs-string">'s/.*"([^"]+)".*/\1/')
curl -L <span class="hljs-string">"https://github.com/goodwithtech/dockle/releases/download/${VERSION}/dockle_<span class="hljs-variable">${VERSION}_Linux-64bit.tar.gz" <span class="hljs-pipe">| tar -xz
<span class="hljs-built_in">sudo <span class="hljs-built_in">mv dockle /usr/local/bin/

<span class="hljs-comment"># Scan an image
dockle myapp:latest

<span class="hljs-comment"># Fail on WARN or higher (for CI)
dockle --exit-code 1 --exit-level warn myapp:latest

<span class="hljs-comment"># Ignore specific checks
dockle --ignore CIS-DI-0010 myapp:latest

Common findings and their Dockerfile fixes:

# CIS-DI-0001: Create a user for the container
# BAD:
FROM node:20-alpine
COPY . /app
CMD ["node", "server.js"]

# GOOD:
FROM node:20-alpine
RUN addgroup -S appgroup && adduser -S appuser -G appgroup
COPY --chown=appuser:appgroup . /app
USER appuser
CMD ["node", "server.js"]

# CIS-DI-0005: Enable Content trust (external concern)
# CIS-DI-0006: Add HEALTHCHECK instruction
HEALTHCHECK --interval=30s --timeout=5s --retries=3 \
  CMD wget -qO- http://localhost:8080/health || exit 1

# DKL-DI-0005: Clear apt-get cache
# BAD:
RUN apt-get install -y curl

# GOOD:
RUN apt-get update && apt-get install -y --no-install-recommends curl \
    && rm -rf /var/lib/apt/lists/*

CI Enforcement: Failing on HIGH/CRITICAL CVEs

A scan that never fails is theater. The value is in blocking deployments when unacceptable vulnerabilities are found:

# .github/workflows/image-security.yaml
name: Container Image Security
on:
  push:
    branches: [main]
  pull_request:

jobs:
  scan:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      security-events: write   # for uploading SARIF results

    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t myapp:${{ github.sha }} .

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:${{ github.sha }}
          format: sarif
          output: trivy-results.sarif
          severity: HIGH,CRITICAL
          exit-code: '1'

      - name: Upload Trivy results to GitHub Security
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: trivy-results.sarif

      - name: Run Grype scanner
        uses: anchore/scan-action@v3
        id: grype-scan
        with:
          image: myapp:${{ github.sha }}
          fail-build: true
          severity-cutoff: high
          output-format: sarif

      - name: Upload Grype results
        if: always()
        uses: github/codeql-action/upload-sarif@v3
        with:
          sarif_file: ${{ steps.grype-scan.outputs.sarif }}

      - name: Run Container Structure Tests
        run: |
          curl -LO https://storage.googleapis.com/container-structure-test/latest/container-structure-test-linux-amd64
          chmod +x container-structure-test-linux-amd64
          ./container-structure-test-linux-amd64 test \
            --image myapp:${{ github.sha }} \
            --config tests/container-structure-test.yaml

      - name: Run Dockle
        run: |
          VERSION=$(curl -s "https://api.github.com/repos/goodwithtech/dockle/releases/latest" \
            | grep '"tag_name"' | sed -E 's/.*"([^"]+)".*/\1/')
          curl -L "https://github.com/goodwithtech/dockle/releases/download/${VERSION}/dockle_${VERSION}_Linux-64bit.tar.gz" \
            | tar -xz dockle
          ./dockle --exit-code 1 --exit-level warn myapp:${{ github.sha }}

Managing Accepted Risks

Not every CVE is actionable. The base image maintainer may not have released a fix yet, or the vulnerable code path may not be reachable in your container's usage. Document accepted risks explicitly:

# .trivyignore
# CVE-2024-12345 — libzstd, no fix available in Ubuntu 22.04
# Risk accepted 2024-03-15, review 2024-06-15
# Justification: compression library not exposed to untrusted input
CVE-2024-12345

The review date in the comment is important — ignored CVEs become permanent when no one remembers why they were ignored. A quarterly review of your ignore files catches CVEs that now have fixes.

Scanning Local Images Before Pushing

Scan before the image reaches your registry:

# Build, scan, push only if clean
docker build -t myapp:latest .

trivy image --exit-code 1 --severity HIGH,CRITICAL myapp:latest && \
  docker push myapp:latest

<span class="hljs-comment"># Or scan the tarball directly (no daemon required)
docker save myapp:latest > /tmp/myapp.tar
trivy image --input /tmp/myapp.tar --severity HIGH,CRITICAL

Scanning before push means vulnerabilities never reach the registry, keeping your image history clean and preventing accidental deployments of scanned-but-not-pushed images.

Container image testing is not optional for production workloads. A single unpatched critical CVE in a widely deployed service has caused more production incidents than most application bugs. Trivy and Grype together catch what either misses alone. Structure tests catch the runtime failures that vulnerability scanners don't cover. Dockle enforces the hygiene practices that prevent the misconfigurations that attackers exploit. Run all four in CI and fail the build on violations — that's the only way to keep your container supply chain honest.

Read more