Testing

Container Image Signing with Sigstore and cosign in CI/CD Pipelines

HelpMeTest

HelpMeTest

— 7 min read
Container Image Signing with Sigstore and cosign in CI/CD Pipelines

In December 2020, attackers inserted a backdoor into a SolarWinds software update. In 2022, a malicious package mimicking a popular npm library was downloaded 300,000 times before removal. In 2024, the XZ Utils backdoor was only caught by accident. The pattern is consistent: software supply chain attacks work by compromising the delivery mechanism, not the source code itself.

Container image signing is a direct defense against this class of attack. When every image in your pipeline carries a cryptographic signature tied to your CI identity, you gain the ability to prove—at deployment time—that a specific image was built by your pipeline, from your code, without tampering. This post covers everything you need to implement it with Sigstore and cosign.

Why Container Signing Matters

Without signing, pulling ghcr.io/myorg/myapp:latest gives you an image—but nothing proves who built it or when. An attacker who compromises your registry credentials, your CI environment, or a third-party image you depend on can push a malicious image with the same tag.

Signing adds a cryptographic assertion: "This image digest sha256:abc123 was signed by the GitHub Actions workflow at github.com/myorg/myapp during a run triggered by commit def456." Any deployment system that verifies signatures before running images will reject unsigned or fraudulently signed images.

Traditional signing with GPG keys suffers from key management problems: keys expire, get lost, or leak. Sigstore's keyless signing model solves this by using short-lived certificates tied to OIDC identity providers—no long-lived keys to manage or protect.

Sigstore Architecture

Sigstore is a set of interoperating services:

Fulcio — a certificate authority that issues short-lived code signing certificates. It accepts OIDC tokens from identity providers (GitHub Actions, Google, Microsoft) and issues a certificate valid for only 10 minutes. The certificate embeds the OIDC identity (e.g., the GitHub Actions workflow URL).

Rekor — a transparency log. Every signature is recorded immutably in Rekor, similar to Certificate Transparency for TLS. This means you can audit all signatures for an image and detect unauthorized signing.

cosign — the client tool. It handles the OIDC flow, certificate issuance, signing, and verification. It stores signatures as OCI artifacts in the same registry as the image being signed.

The flow: CI authenticates to Fulcio with an OIDC token → Fulcio issues a short-lived certificate → cosign signs the image digest and uploads the signature to the registry → Rekor records the event. Verification reverses this: cosign fetches the signature, verifies it against the certificate, and confirms the certificate was issued by Fulcio and recorded in Rekor.

Installing cosign

# macOS
brew install cosign

# Linux
COSIGN_VERSION=$(curl -s https://api.github.com/repos/sigstore/cosign/releases/latest \
  | grep '"tag_name"' | cut -d'"' -f4)
curl -sL "https://github.com/sigstore/cosign/releases/download/${COSIGN_VERSION}/cosign-linux-amd64" \
  -o /usr/local/bin/cosign
chmod +x /usr/local/bin/cosign

# Verify
cosign version

For CI environments, use the official installer action instead of a manual install.

Signing Images in GitHub Actions

Here is a complete workflow that builds, pushes, signs, and verifies a container image:

# .github/workflows/build-and-sign.yml
name: Build, Sign, and Verify Container

on:
  push:
    branches: [main]
  pull_request:

env:
  REGISTRY: ghcr.io
  IMAGE_NAME: ${{ github.repository }}

permissions:
  contents: read
  packages: write
  id-token: write   # CRITICAL: required for OIDC token to sign with Sigstore

jobs:
  build-sign:
    runs-on: ubuntu-latest
    outputs:
      image-digest: ${{ steps.build.outputs.digest }}

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Log in to registry
        uses: docker/login-action@v3
        with:
          registry: ${{ env.REGISTRY }}
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}

      - name: Extract metadata
        id: meta
        uses: docker/metadata-action@v5
        with:
          images: ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}
          tags: |
            type=sha,prefix=sha-
            type=ref,event=branch
            type=semver,pattern={{version}}

      - name: Build and push
        id: build
        uses: docker/build-push-action@v5
        with:
          context: .
          push: ${{ github.event_name != 'pull_request' }}
          tags: ${{ steps.meta.outputs.tags }}
          labels: ${{ steps.meta.outputs.labels }}

      - name: Install cosign
        if: github.event_name != 'pull_request'
        uses: sigstore/cosign-installer@v3

      - name: Sign image (keyless)
        if: github.event_name != 'pull_request'
        run: |
          # Sign using the image digest (not mutable tag)
          cosign sign --yes \
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }}
        env:
          COSIGN_EXPERIMENTAL: 1

      - name: Verify signature
        if: github.event_name != 'pull_request'
        run: |
          cosign verify \
            --certificate-identity-regexp \
              "https://github.com/${{ github.repository }}/.github/workflows/build-and-sign.yml" \
            --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
            ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}@${{ steps.build.outputs.digest }} \
            | jq '.[0].optional | {Issuer, Subject, "GitHub SHA": .["github_sha"]}'
        env:
          COSIGN_EXPERIMENTAL: 1

Key Points About This Workflow

Sign the digest, not the tag. Tags are mutable—latest can point to a different image tomorrow. Signing sha256:abc123 creates a binding to that exact image content, regardless of what tags point to it.

id-token: write permission is mandatory. Without this, GitHub Actions cannot generate an OIDC token, and keyless signing will fail. This is a common footgun.

--yes flag. cosign 2.0+ requires explicit confirmation of transparency log upload. Add --yes to avoid interactive prompts in CI.

Keyless Signing Deep Dive

When cosign runs in a GitHub Actions environment, it detects the ACTIONS_ID_TOKEN_REQUEST_TOKEN environment variable and automatically requests an OIDC token. This token contains claims like:

{
  "iss": "https://token.actions.githubusercontent.com",
  "sub": "repo:myorg/myapp:ref:refs/heads/main",
  "job_workflow_ref": "myorg/myapp/.github/workflows/build.yml@refs/heads/main",
  "sha": "abc123def456",
  "repository": "myorg/myapp",
  "event_name": "push"
}

Fulcio validates this token with GitHub's OIDC endpoint and issues a certificate embedding the sub claim as the Subject Alternative Name. The certificate is valid for 10 minutes—long enough to sign, too short to be stolen and misused later.

The resulting signature stored in the registry contains:

  • The image digest being signed
  • The Fulcio certificate (with the workflow identity)
  • The Rekor transparency log entry ID

Anyone with registry read access can verify this chain.

Verifying Signatures Before Deployment

In a deployment script or pre-deploy check:

#!/bin/bash
IMAGE="ghcr.io/myorg/myapp@sha256:abc123"
WORKFLOW_URL="https://github.com/myorg/myapp/.github/workflows/build-and-sign.yml"
OIDC_ISSUER="https://token.actions.githubusercontent.com"

if cosign verify \
  --certificate-identity-regexp "$WORKFLOW_URL" \
  --certificate-oidc-issuer "$OIDC_ISSUER" \
  "$IMAGE" > /dev/null 2>&1; then
  echo "Signature verified. Proceeding with deployment."
else
  echo "ERROR: Signature verification failed. Refusing to deploy."
  exit 1
fi

Policy Enforcement with Kyverno

Verifying signatures manually in deployment scripts is fragile—someone might forget to add the check. Kyverno enforces image signature policy at the Kubernetes admission controller level: any pod attempting to run an unsigned image is rejected before it starts.

Installing Kyverno

helm repo add kyverno https://kyverno.github.io/kyverno/
helm repo update
helm install kyverno kyverno/kyverno -n kyverno --create-namespace

Signature Verification Policy

# kyverno-verify-images.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: verify-image-signatures
  annotations:
    policies.kyverno.io/title: Verify Image Signatures
    policies.kyverno.io/description: >
      Requires all images from ghcr.io/myorg to have valid
      Sigstore/cosign signatures from the authorized CI workflow.
spec:
  validationFailureAction: Enforce
  background: false
  rules:
    - name: verify-myapp-signature
      match:
        any:
          - resources:
              kinds: [Pod]
              namespaces: [production, staging]
      verifyImages:
        - imageReferences:
            - "ghcr.io/myorg/*"
          attestors:
            - count: 1
              entries:
                - keyless:
                    subject: "https://github.com/myorg/myapp/.github/workflows/build-and-sign.yml@refs/heads/main"
                    issuer: "https://token.actions.githubusercontent.com"
                    rekor:
                      url: https://rekor.sigstore.dev

Apply this policy and test it:

kubectl apply -f kyverno-verify-images.yaml

# This should succeed (signed image)
kubectl run signed \
  --image=ghcr.io/myorg/myapp@sha256:<signed-digest> \
  --restart=Never

# This should fail with a policy violation
kubectl run unsigned \
  --image=nginx:latest \
  --restart=Never
# Error from server: admission webhook "mutate.kyverno.svc" denied the request:
# resource Pod/default/unsigned was blocked due to the following policies:
# verify-image-signatures: verify-myapp-signature: image verification failed

Signing with a Key Pair (Alternative to Keyless)

For environments without OIDC support or where you want explicit key management:

# Generate a key pair (run locally, store private key in secrets)
cosign generate-key-pair

# This produces cosign.key (private) and cosign.pub (public)
# Add cosign.key content to GitHub Secrets as COSIGN_PRIVATE_KEY
# Add COSIGN_PASSWORD as the key encryption password

# Signing in CI with a key
cosign sign --key env://COSIGN_PRIVATE_KEY \
  ghcr.io/myorg/myapp@sha256:abc123

# Verifying with the public key
cosign verify --key cosign.pub \
  ghcr.io/myorg/myapp@sha256:abc123

Store cosign.pub in your repository—it is safe to commit. Use it in Kyverno policies with the keys attestor instead of keyless.

Inspecting the Transparency Log

Every keyless signature is recorded in Rekor. You can look up any image's signatures:

# Get the Rekor log entry for a signed image
cosign triangulate ghcr.io/myorg/myapp@sha256:abc123
# Returns the OCI reference where the signature is stored

# Search Rekor for all entries related to an image
rekor-cli search --sha sha256:abc123

The transparency log provides an audit trail: if someone signs an image with a stolen OIDC token, the event is recorded in Rekor with a timestamp and the claimed identity. You can set up monitoring to alert on unexpected signing events.

Common Pitfalls and Solutions

Signing with a mutable tag gets invalidated. If you sign myapp:latest and then push a new image to :latest, the signature no longer matches. Always sign by digest.

COSIGN_EXPERIMENTAL is deprecated in cosign 2.x. In cosign 2.0+, keyless signing and Rekor integration are enabled by default. You may see deprecation warnings; the flag is still accepted but no longer required.

Registry doesn't support OCI artifacts. cosign stores signatures as OCI artifacts in the same namespace as the image. Some older registries or private ECR configurations may not support this. GHCR, Docker Hub, and GCR all support it. For ECR, ensure OCI artifact support is enabled in the repository settings.

Kyverno policy in Audit mode first. Set validationFailureAction: Audit initially. Check kubectl get policyreport -A to see what would fail before switching to Enforce. Enforce mode on a misconfigured policy can prevent all pods from starting.

The Bigger Picture

Container signing is one layer in a defense-in-depth strategy. By itself, it prevents an attacker who can push to your registry from deploying malicious images. Combined with SBOMs (which document what's in the image) and SLSA provenance (which records how it was built), you have a complete chain of custody from source commit to running container.

Start with signing on your main branch builds. Add Kyverno in audit mode to understand your current unsigned image footprint. Then tighten enforcement namespace by namespace until every workload in production runs only verified images.

Read more

Start now free