Security

Mutual TLS Testing Guide: Client Certificates, Service Mesh & Istio mTLS

HelpMeTest

HelpMeTest

— 9 min read
Mutual TLS Testing Guide: Client Certificates, Service Mesh & Istio mTLS

Mutual TLS (mTLS) requires both client and server to present certificates during the TLS handshake. It's the standard authentication mechanism for service-to-service communication in microservices, and increasingly used for API access in zero-trust architectures. Testing mTLS is distinct from standard TLS testing — you need to verify that certificate requirements are enforced, not just that encryption is configured.

What mTLS Testing Verifies

Standard TLS: server presents a certificate, client verifies it. Client identity: password, token, or session cookie.

Mutual TLS: both sides present certificates. The server verifies the client certificate is signed by a trusted CA. Client identity comes from the certificate itself, not from application-layer credentials.

Testing must cover:

  • Enforcement — requests without a client certificate must be rejected
  • CA validation — only certificates from trusted CAs are accepted
  • Certificate revocation — revoked certificates are rejected
  • Service identity — each service has a unique certificate; a cert valid for Service A can't authenticate as Service B
  • Expiry — expired client certificates are rejected

Testing Client Certificate Authentication

Generate Test Certificates

First, create a test CA and client certificates:

# Generate CA key and certificate
openssl genrsa -out test-ca.key 4096
openssl req -new -x509 -days 365 \
  -key test-ca.key \
  -out test-ca.crt \
  -subj <span class="hljs-string">"/C=US/O=Test CA/CN=Test Root CA"

<span class="hljs-comment"># Generate valid client key and CSR
openssl genrsa -out valid-client.key 2048
openssl req -new \
  -key valid-client.key \
  -out valid-client.csr \
  -subj <span class="hljs-string">"/C=US/O=Test/CN=valid-service"

<span class="hljs-comment"># Sign client cert with test CA
openssl x509 -req -days 30 \
  -<span class="hljs-keyword">in valid-client.csr \
  -CA test-ca.crt \
  -CAkey test-ca.key \
  -CAcreateserial \
  -out valid-client.crt

<span class="hljs-comment"># Generate invalid client cert (different CA)
openssl genrsa -out other-ca.key 4096
openssl req -new -x509 -days 365 \
  -key other-ca.key \
  -out other-ca.crt \
  -subj <span class="hljs-string">"/C=US/O=Other CA/CN=Other Root CA"

openssl genrsa -out invalid-client.key 2048
openssl req -new \
  -key invalid-client.key \
  -out invalid-client.csr \
  -subj <span class="hljs-string">"/C=US/O=Test/CN=invalid-service"

openssl x509 -req -days 30 \
  -<span class="hljs-keyword">in invalid-client.csr \
  -CA other-ca.crt \
  -CAkey other-ca.key \
  -CAcreateserial \
  -out invalid-client.crt

Test with curl

# Test 1: No client cert — should get 400 or 401
curl -v \
  --cacert test-ca.crt \
  https://api.staging.example.com/secure-endpoint

<span class="hljs-comment"># Expected: SSL handshake failure or 400 Bad Request
<span class="hljs-comment"># "SSL_CTX_set_client_CA_list" or "handshake failure" in output = mTLS enforced

<span class="hljs-comment"># Test 2: Valid client cert from trusted CA — should get 200
curl -v \
  --cacert test-ca.crt \
  --cert valid-client.crt \
  --key valid-client.key \
  https://api.staging.example.com/secure-endpoint

<span class="hljs-comment"># Expected: 200 OK with response body

<span class="hljs-comment"># Test 3: Client cert from untrusted CA — should be rejected
curl -v \
  --cacert test-ca.crt \
  --cert invalid-client.crt \
  --key invalid-client.key \
  https://api.staging.example.com/secure-endpoint

<span class="hljs-comment"># Expected: SSL handshake failure

Automated mTLS Test Suite

import ssl
import urllib.request
import subprocess
import pytest

BASE_URL = "https://api.staging.example.com"
CA_CERT = "test-ca.crt"
VALID_CERT = "valid-client.crt"
VALID_KEY = "valid-client.key"
INVALID_CERT = "invalid-client.crt"
INVALID_KEY = "invalid-client.key"


def make_request(url, cert=None, key=None, ca_cert=CA_CERT):
    """Make HTTPS request with optional client cert."""
    context = ssl.create_default_context(cafile=ca_cert)
    if cert and key:
        context.load_cert_chain(cert, key)
    
    try:
        req = urllib.request.Request(url)
        with urllib.request.urlopen(req, context=context) as response:
            return response.status, response.read()
    except urllib.error.HTTPError as e:
        return e.code, e.read()
    except ssl.SSLError as e:
        return None, str(e)


class TestMTLSEnforcement:
    
    def test_rejects_request_without_client_cert(self):
        """mTLS endpoint must reject requests with no client certificate."""
        status, body = make_request(f"{BASE_URL}/secure-endpoint")
        # Should fail at SSL layer (status=None) or return 400/401
        assert status is None or status in (400, 401), \
            f"Expected rejection without client cert, got {status}"
    
    def test_accepts_valid_client_cert(self):
        """mTLS endpoint must accept requests with a valid client certificate."""
        status, body = make_request(
            f"{BASE_URL}/secure-endpoint",
            cert=VALID_CERT,
            key=VALID_KEY
        )
        assert status == 200, f"Expected 200 with valid cert, got {status}"
    
    def test_rejects_cert_from_untrusted_ca(self):
        """mTLS endpoint must reject client certs from non-trusted CAs."""
        status, body = make_request(
            f"{BASE_URL}/secure-endpoint",
            cert=INVALID_CERT,
            key=INVALID_KEY
        )
        assert status is None or status in (400, 401, 403), \
            f"Expected rejection with untrusted CA cert, got {status}"
    
    def test_service_identity_isolation(self):
        """Service A cert must not authenticate as Service B identity."""
        status, body = make_request(
            f"{BASE_URL}/service-b-endpoint",  # B's endpoint
            cert=VALID_CERT,   # A's cert (valid, but wrong service)
            key=VALID_KEY
        )
        assert status in (401, 403), \
            f"Expected rejection of wrong service cert, got {status}"
    
    def test_expired_cert_rejected(self):
        """Expired client certificates must be rejected."""
        status, body = make_request(
            f"{BASE_URL}/secure-endpoint",
            cert="expired-client.crt",
            key="expired-client.key"
        )
        assert status is None or status in (400, 401), \
            f"Expected rejection of expired cert, got {status}"

Service Mesh mTLS Validation

Kubernetes with Istio

Istio manages mTLS automatically for pods in the mesh. Testing ensures policies are actually enforced.

Verify mTLS Mode

# Check PeerAuthentication policies
kubectl get peerauthentication -A

<span class="hljs-comment"># Check if STRICT mode is set namespace-wide
kubectl get peerauthentication -n production -o yaml

<span class="hljs-comment"># Expected output for strict mode:
<span class="hljs-comment"># spec:
<span class="hljs-comment">#   mtls:
<span class="hljs-comment">#     mode: STRICT

Test mTLS from Outside the Mesh

# From a pod outside the mesh, attempt to call a mesh service
<span class="hljs-comment"># This should be blocked in STRICT mode
kubectl run test-client --image=curlimages/curl -it --<span class="hljs-built_in">rm -- \
  curl -v http://my-service.production.svc.cluster.local:8080/api/data

<span class="hljs-comment"># Expected: Connection refused or TLS handshake failure

Test from Within the Mesh

# From a sidecar-injected pod — Istio provides the cert automatically
kubectl <span class="hljs-built_in">exec -n production deployment/my-service -- \
  curl -v http://other-service:8080/api/data

<span class="hljs-comment"># Expected: 200 OK — Istio handles cert presentation transparently

Verify mTLS with istioctl

# Check mTLS status for a specific service
istioctl authn tls-check my-service.production.svc.cluster.local

<span class="hljs-comment"># Output shows mTLS status per service:
<span class="hljs-comment"># HOST                                       STATUS     SERVER                     CLIENT
<span class="hljs-comment"># my-service.production.svc.cluster.local    OK         mTLS                       mTLS

<span class="hljs-comment"># Check all services in a namespace
istioctl authn tls-check -n production

Inspect Certificates in the Mesh

# View the certificate Istio has issued to a pod
istioctl proxy-config secret my-pod-name -n production

<span class="hljs-comment"># View cert expiry
kubectl <span class="hljs-built_in">exec my-pod -n production -c istio-proxy -- \
  openssl s_client -connect other-service:8080 \
  </dev/null 2>/dev/null <span class="hljs-pipe">| openssl x509 -noout -dates

<span class="hljs-comment"># Check the SPIFFE identity in the cert
kubectl <span class="hljs-built_in">exec my-pod -n production -c istio-proxy -- \
  openssl s_client -connect other-service:8080 \
  </dev/null 2>/dev/null <span class="hljs-pipe">| openssl x509 -noout -text <span class="hljs-pipe">| grep -A2 <span class="hljs-string">"Subject Alternative Name"

<span class="hljs-comment"># SPIFFE URI format: spiffe://cluster.local/ns/production/sa/my-service

Testing Istio AuthorizationPolicy

AuthorizationPolicy controls which services can communicate:

# Allow only specific services
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payment-service-policy
  namespace: production
spec:
  selector:
    matchLabels:
      app: payment-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/production/sa/order-service"
              - "cluster.local/ns/production/sa/checkout-service"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/api/payment/*"]

Test the policy is enforced:

# Should succeed — order-service calling payment-service
kubectl <span class="hljs-built_in">exec -n production deployment/order-service -- \
  curl -X POST http://payment-service:8080/api/payment/process \
  -H <span class="hljs-string">"Content-Type: application/json" \
  -d <span class="hljs-string">'{"amount": 100}'

<span class="hljs-comment"># Should fail — inventory-service calling payment-service (not authorized)
kubectl <span class="hljs-built_in">exec -n production deployment/inventory-service -- \
  curl -X POST http://payment-service:8080/api/payment/process \
  -H <span class="hljs-string">"Content-Type: application/json" \
  -d <span class="hljs-string">'{"amount": 100}'
<span class="hljs-comment"># Expected: RBAC: access denied

Cilium mTLS Testing

For clusters using Cilium instead of Istio:

# Check Cilium network policies
kubectl get ciliumnetworkpolicies -A

<span class="hljs-comment"># Verify mTLS enforcement with Hubble
hubble observe --namespace production --protocol http --verdict DROPPED

<span class="hljs-comment"># Test connectivity (Cilium Connectivity Test)
cilium connectivity <span class="hljs-built_in">test --<span class="hljs-built_in">test=pod-to-pod-encryption

Testing mTLS on API Gateways

Kong with mTLS Plugin

# Verify the mTLS plugin is configured
curl -s http://kong-admin:8001/plugins <span class="hljs-pipe">| jq <span class="hljs-string">'.data[] | select(.name == "mtls-auth")'

<span class="hljs-comment"># Test without cert — should get 401
curl -v https://api.gateway.example.com/protected-route

<span class="hljs-comment"># Test with valid cert
curl -v \
  --cert valid-client.crt \
  --key valid-client.key \
  --cacert gateway-ca.crt \
  https://api.gateway.example.com/protected-route

Nginx with client_certificate

# Test Nginx mTLS config
nginx -t -c /etc/nginx/nginx.conf

<span class="hljs-comment"># Verify ssl_verify_client is on or optional_no_ca
grep -r <span class="hljs-string">"ssl_verify_client" /etc/nginx/

<span class="hljs-comment"># Test with curl
curl -v \
  --cacert /etc/nginx/ssl/ca.crt \
  --cert /tmp/test-client.crt \
  --key /tmp/test-client.key \
  https://staging.example.com/api/protected

Automated mTLS Validation in CI

# .github/workflows/mtls-test.yml
name: mTLS Enforcement Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  mtls-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Generate test PKI
        run: |
          # CA
          openssl genrsa -out test-ca.key 4096
          openssl req -new -x509 -days 1 \
            -key test-ca.key -out test-ca.crt \
            -subj "/CN=Test CA"
          
          # Valid client cert
          openssl genrsa -out valid-client.key 2048
          openssl req -new -key valid-client.key \
            -out valid-client.csr -subj "/CN=valid-service"
          openssl x509 -req -days 1 \
            -in valid-client.csr \
            -CA test-ca.crt -CAkey test-ca.key -CAcreateserial \
            -out valid-client.crt
          
          # Untrusted CA cert
          openssl genrsa -out other-ca.key 4096
          openssl req -new -x509 -days 1 \
            -key other-ca.key -out other-ca.crt \
            -subj "/CN=Other CA"
          openssl genrsa -out bad-client.key 2048
          openssl req -new -key bad-client.key \
            -out bad-client.csr -subj "/CN=bad-service"
          openssl x509 -req -days 1 \
            -in bad-client.csr \
            -CA other-ca.crt -CAkey other-ca.key -CAcreateserial \
            -out bad-client.crt

      - name: Setup test server
        run: docker-compose -f docker-compose.test.yml up -d

      - name: Run mTLS tests
        run: |
          pip install pytest
          pytest tests/mtls/ -v

      - name: Verify Istio mTLS (staging)
        if: github.ref == 'refs/heads/main'
        run: |
          # Check all services are in STRICT mode
          PERMISSIVE=$(kubectl get peerauthentication -A -o json | \
            jq -r '.items[] | select(.spec.mtls.mode == "PERMISSIVE") | .metadata.name')
          
          if [ -n "$PERMISSIVE" ]; then
            echo "::warning::Services in PERMISSIVE mTLS mode: $PERMISSIVE"
          fi
          
          # Check for missing PeerAuthentication
          NAMESPACES="production staging"
          for NS in $NAMESPACES; do
            POLICY=$(kubectl get peerauthentication -n $NS -o json | jq '.items | length')
            if [ "$POLICY" -eq 0 ]; then
              echo "::error::No PeerAuthentication policy in namespace $NS"
              exit 1
            fi
          done

Common mTLS Issues

Issue Symptom Fix
Certificate not sent SSL handshake error on server Check client cert path and key match
CA cert mismatch certificate verify failed Ensure client cert signed by the CA the server trusts
Wrong SAN hostname doesn't match Add correct hostname to SubjectAltName in cert
Cert expired certificate expired Renew; automate with cert-manager in Kubernetes
Istio in PERMISSIVE mode mTLS not enforced Set PeerAuthentication to STRICT
Missing namespace policy Pods bypass mTLS Add namespace-scoped PeerAuthentication

Certificate Rotation Testing

Testing that cert rotation doesn't cause downtime:

# Generate new cert (same CA)
openssl genrsa -out new-client.key 2048
openssl req -new -key new-client.key \
  -out new-client.csr -subj <span class="hljs-string">"/CN=valid-service"
openssl x509 -req -days 30 \
  -<span class="hljs-keyword">in new-client.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out new-client.crt

<span class="hljs-comment"># Hot-swap test: call endpoint continuously while rotating
<span class="hljs-keyword">for i <span class="hljs-keyword">in {1..20}; <span class="hljs-keyword">do
  curl -s \
    --cert new-client.crt \
    --key new-client.key \
    --cacert ca.crt \
    https://api.staging.example.com/health
  <span class="hljs-built_in">sleep 0.5
<span class="hljs-keyword">done

<span class="hljs-comment"># All requests should return 200 — no interruption during rotation

In Kubernetes, cert-manager handles rotation automatically. Verify it's working:

# Check cert-manager certificates
kubectl get certificates -A
kubectl describe certificate my-service-cert -n production

<span class="hljs-comment"># Check for expiring certs
kubectl get certificates -A -o json <span class="hljs-pipe">| \
  jq -r <span class="hljs-string">'.items[] | select(.status.notAfter) <span class="hljs-pipe">| 
    {name: .metadata.name, namespace: .metadata.namespace, expires: .status.notAfter}'

Read more