Security Testing

Sigstore and Cosign: Signing and Verifying Container Images

HelpMeTest

HelpMeTest

— 6 min read
Sigstore and Cosign: Signing and Verifying Container Images

Container image signing closes a critical gap in software supply chain security: verifying that the image running in production is exactly the image your CI pipeline built — and nothing else. Sigstore and its Cosign tool make this practical, with keyless signing that eliminates the burden of managing long-lived private keys.

The Problem: Image Authenticity

When your Kubernetes cluster pulls myapp:v1.2.3, how does it know that image hasn't been tampered with? Image tags are mutable — the same tag can point to different content across pushes. Even content-addressable SHA digests only prove identity, not provenance.

Container image signing answers: "Who built this, and when?"

Sigstore: The Infrastructure

Sigstore is an open-source project (supported by Google, Red Hat, and Chainguard) that provides infrastructure for signing software artifacts:

  • Cosign: CLI and library for signing/verifying container images and other OCI artifacts
  • Rekor: Transparency log — an immutable, append-only ledger of signatures
  • Fulcio: Certificate authority for short-lived code signing certificates
  • Gitsign: Signs git commits using the same infrastructure

Together they enable keyless signing: you sign with your OIDC identity (GitHub Actions, Google accounts, etc.) rather than a long-lived private key. Signatures are stored in the public Rekor transparency log.

Installing Cosign

# macOS
brew install cosign

<span class="hljs-comment"># Linux
curl -O -L <span class="hljs-string">"https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64"
<span class="hljs-built_in">sudo <span class="hljs-built_in">mv cosign-linux-amd64 /usr/local/bin/cosign
<span class="hljs-built_in">sudo <span class="hljs-built_in">chmod +x /usr/local/bin/cosign

<span class="hljs-comment"># Verify installation
cosign version

Keyless signing uses your OIDC token to get a short-lived certificate from Fulcio. The certificate proves your identity at the time of signing without requiring you to manage a private key.

In GitHub Actions

name: Build and Sign Image

on:
  push:
    branches: [main]

permissions:
  contents: read
  packages: write
  id-token: write  # Required for keyless signing

jobs:
  build-and-sign:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install Cosign
        uses: sigstore/cosign-installer@v3

      - name: Login to GHCR
        uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Build and push image
        id: build
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}

      - name: Sign image (keyless)
        run: |
          cosign sign --yes ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}

The --yes flag auto-confirms the Rekor transparency log upload. The id-token: write permission allows the job to get an OIDC token for Fulcio.

Verifying a Keyless Signature

# Verify the signature came from a specific GitHub Actions workflow
cosign verify \
  --certificate-identity-regexp <span class="hljs-string">"https://github.com/myorg/myrepo/.github/workflows/.*" \
  --certificate-oidc-issuer <span class="hljs-string">"https://token.actions.githubusercontent.com" \
  ghcr.io/myorg/myrepo:v1.2.3

<span class="hljs-comment"># Verify with exact workflow path
cosign verify \
  --certificate-identity <span class="hljs-string">"https://github.com/myorg/myrepo/.github/workflows/release.yaml@refs/heads/main" \
  --certificate-oidc-issuer <span class="hljs-string">"https://token.actions.githubusercontent.com" \
  ghcr.io/myorg/myrepo:v1.2.3

Verification output includes the Rekor log entry URL, certificate details, and the image digest.

Key-Based Signing

For environments without OIDC or for additional verification layers, key-based signing is simpler to understand:

Generate a Key Pair

# Generate key pair (prompts for optional password)
cosign generate-key-pair

<span class="hljs-comment"># Files created:
<span class="hljs-comment"># cosign.key  - private key (keep secret!)
<span class="hljs-comment"># cosign.pub  - public key (share this)

<span class="hljs-comment"># Or store in a secret manager
cosign generate-key-pair --kms gcpkms://projects/myproject/locations/global/keyRings/mykeyring/cryptoKeys/mykey

Sign with a Key

# Sign using private key
cosign sign --key cosign.key ghcr.io/myorg/myapp:v1.2.3

<span class="hljs-comment"># Sign a specific digest (safer - tags are mutable)
cosign sign --key cosign.key ghcr.io/myorg/myapp@sha256:abc123...

Verify with the Public Key

# Verify signature
cosign verify --key cosign.pub ghcr.io/myorg/myapp:v1.2.3

<span class="hljs-comment"># Verify with JSON output
cosign verify --key cosign.pub ghcr.io/myorg/myapp:v1.2.3 <span class="hljs-pipe">| jq .

Attestations: Beyond Signatures

Attestations attach metadata to images — not just "I signed this" but "I also ran these tests and here's the evidence":

# Attach an SBOM as an attestation
cosign attest \
  --key cosign.key \
  --predicate sbom.spdx.json \
  --<span class="hljs-built_in">type spdxjson \
  ghcr.io/myorg/myapp:v1.2.3

<span class="hljs-comment"># Attach a SLSA provenance
cosign attest \
  --key cosign.key \
  --predicate provenance.json \
  --<span class="hljs-built_in">type slsaprovenance \
  ghcr.io/myorg/myapp:v1.2.3

<span class="hljs-comment"># Attach test results
cosign attest \
  --key cosign.key \
  --predicate test-results.json \
  --<span class="hljs-built_in">type https://example.com/TestResult/v1 \
  ghcr.io/myorg/myapp:v1.2.3

Verifying Attestations

# Verify and extract SBOM attestation
cosign verify-attestation \
  --key cosign.pub \
  --<span class="hljs-built_in">type spdxjson \
  ghcr.io/myorg/myapp:v1.2.3 <span class="hljs-pipe">| jq <span class="hljs-string">'.payload | @base64d <span class="hljs-pipe">| fromjson'

Policy Enforcement with Cosign

Cosign integrates with Kubernetes admission controllers to enforce signing policies:

Sigstore Policy Controller

# ClusterImagePolicy - require signed images from your registry
apiVersion: policy.sigstore.dev/v1beta1
kind: ClusterImagePolicy
metadata:
  name: require-signed-images
spec:
  images:
    - glob: "ghcr.io/myorg/**"
  authorities:
    - keyless:
        url: https://fulcio.sigstore.dev
        identities:
          - issuer: https://token.actions.githubusercontent.com
            subjectRegExp: "https://github.com/myorg/.*"

Kyverno Policy

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: check-image-signature
spec:
  validationFailureAction: enforce
  rules:
    - name: verify-image
      match:
        any:
          - resources:
              kinds:
                - Pod
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/*"
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/myorg/myrepo/.github/workflows/release.yaml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

The Rekor Transparency Log

Rekor records all signatures publicly. You can search it:

# Install rekor-cli
go install github.com/sigstore/rekor/cmd/rekor-cli@latest

<span class="hljs-comment"># Search for entries by artifact hash
rekor-cli search --sha $(docker inspect --format=<span class="hljs-string">'{{index .RepoDigests 0}}' myapp:v1.2.3 <span class="hljs-pipe">| <span class="hljs-built_in">cut -d@ -f2)

<span class="hljs-comment"># Get a specific log entry
rekor-cli get --uuid <uuid>

<span class="hljs-comment"># Verify an entry is in the log
rekor-cli verify --artifact myapp.tar.gz --signature myapp.tar.gz.sig --public-key cosign.pub

The transparency log provides auditability: anyone can verify that a signature was created at a specific time and hasn't been backdated.

Multi-Stage Signing in CI

A robust pipeline signs at multiple points and includes attestations:

name: Secure Build Pipeline

on:
  push:
    tags: ['v*']

permissions:
  id-token: write
  packages: write
  contents: read

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      digest: ${{ steps.build.outputs.digest }}
    steps:
      - uses: actions/checkout@v4
      - name: Install Cosign
        uses: sigstore/cosign-installer@v3
      - name: Install Syft
        uses: anchore/sbom-action/download-syft@v0
      
      - name: Build and push
        id: build
        uses: docker/build-push-action@v5
        with:
          push: true
          tags: ghcr.io/${{ github.repository }}:${{ github.ref_name }}

      - name: Sign image
        run: |
          cosign sign --yes \
            ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}

      - name: Generate SBOM
        run: |
          syft ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }} \
            -o spdx-json > sbom.spdx.json

      - name: Attest SBOM
        run: |
          cosign attest --yes \
            --predicate sbom.spdx.json \
            --type spdxjson \
            ghcr.io/${{ github.repository }}@${{ steps.build.outputs.digest }}

Testing Your Signing Setup

Before deploying to production, verify the signing workflow end-to-end:

# 1. Build a test image
docker build -t localhost:5000/test:latest .
docker push localhost:5000/test:latest

<span class="hljs-comment"># 2. Sign it
cosign sign --key cosign.key localhost:5000/test:latest

<span class="hljs-comment"># 3. Verify it works
cosign verify --key cosign.pub localhost:5000/test:latest

<span class="hljs-comment"># 4. Tamper with the image and verify detection
docker tag nginx:latest localhost:5000/test:latest
docker push localhost:5000/test:latest

<span class="hljs-comment"># 5. Verify should now fail (wrong digest)
cosign verify --key cosign.pub localhost:5000/test:latest
<span class="hljs-comment"># Error: no matching signatures

Common Pitfalls

Tags vs digests: Always sign digests (@sha256:...), not tags. Tags can be repointed to different content. When verifying, use the digest.

Key management: If using key-based signing, protect the private key with a password and store it in a secrets manager (HashiCorp Vault, AWS KMS, GCP KMS). Cosign has built-in KMS integrations.

Rekor privacy: The public Rekor instance logs all signatures publicly. For private images, you can run a private Rekor instance or use a different storage method.

Clock skew: Keyless certificates are short-lived (10 minutes). Ensure your CI runners have accurate clocks.

Verifying Third-Party Images

You can verify signatures on popular public images:

# Verify a Chainguard base image
cosign verify \
  --certificate-identity <span class="hljs-string">"keyless@distroless.iam.gserviceaccount.com" \
  --certificate-oidc-issuer <span class="hljs-string">"https://accounts.google.com" \
  cgr.dev/chainguard/static:latest

<span class="hljs-comment"># Verify a Sigstore release
cosign verify \
  --certificate-identity <span class="hljs-string">"sigstore-release@sigstore-project.iam.gserviceaccount.com" \
  --certificate-oidc-issuer <span class="hljs-string">"https://accounts.google.com" \
  gcr.io/projectsigstore/cosign:latest

Container image signing with Sigstore and Cosign is now a standard practice in security-conscious organizations. The keyless workflow eliminates key management overhead while the Rekor transparency log provides auditable proof of provenance. Combined with admission controller policies, you can ensure that only signed, verified images ever run in production.

Read more