Security

Mutual TLS Testing Guide: Client Certificates, Service Mesh & Istio mTLS

HelpMeTest

18 May 2026 — 7 min read

Mutual TLS (mTLS) requires both client and server to present certificates during the TLS handshake. It's the standard authentication mechanism for service-to-service communication in microservices, and increasingly used for API access in zero-trust architectures. Testing mTLS is distinct from standard TLS testing — you need to verify that certificate requirements are enforced, not just that encryption is configured.

What mTLS Testing Verifies

Standard TLS: server presents a certificate, client verifies it. Client identity: password, token, or session cookie.

Mutual TLS: both sides present certificates. The server verifies the client certificate is signed by a trusted CA. Client identity comes from the certificate itself, not from application-layer credentials.

Testing must cover:

Enforcement — requests without a client certificate must be rejected
CA validation — only certificates from trusted CAs are accepted
Certificate revocation — revoked certificates are rejected
Service identity — each service has a unique certificate; a cert valid for Service A can't authenticate as Service B
Expiry — expired client certificates are rejected

Testing Client Certificate Authentication

Generate Test Certificates

First, create a test CA and client certificates:

# Generate CA key and certificate
openssl genrsa -out test-ca.key 4096
openssl req -new -x509 -days 365 \
  -key test-ca.key \
  -out test-ca.crt \
  -subj "/C=US/O=Test CA/CN=Test Root CA"

# Generate valid client key and CSR
openssl genrsa -out valid-client.key 2048
openssl req -new \
  -key valid-client.key \
  -out valid-client.csr \
  -subj "/C=US/O=Test/CN=valid-service"

# Sign client cert with test CA
openssl x509 -req -days 30 \
  -in valid-client.csr \
  -CA test-ca.crt \
  -CAkey test-ca.key \
  -CAcreateserial \
  -out valid-client.crt

# Generate invalid client cert (different CA)
openssl genrsa -out other-ca.key 4096
openssl req -new -x509 -days 365 \
  -key other-ca.key \
  -out other-ca.crt \
  -subj "/C=US/O=Other CA/CN=Other Root CA"

openssl genrsa -out invalid-client.key 2048
openssl req -new \
  -key invalid-client.key \
  -out invalid-client.csr \
  -subj "/C=US/O=Test/CN=invalid-service"

openssl x509 -req -days 30 \
  -in invalid-client.csr \
  -CA other-ca.crt \
  -CAkey other-ca.key \
  -CAcreateserial \
  -out invalid-client.crt

Test with curl

# Test 1: No client cert — should get 400 or 401
curl -v \
  --cacert test-ca.crt \
  https://api.staging.example.com/secure-endpoint

# Expected: SSL handshake failure or 400 Bad Request
# "SSL_CTX_set_client_CA_list" or "handshake failure" in output = mTLS enforced

# Test 2: Valid client cert from trusted CA — should get 200
curl -v \
  --cacert test-ca.crt \
  --cert valid-client.crt \
  --key valid-client.key \
  https://api.staging.example.com/secure-endpoint

# Expected: 200 OK with response body

# Test 3: Client cert from untrusted CA — should be rejected
curl -v \
  --cacert test-ca.crt \
  --cert invalid-client.crt \
  --key invalid-client.key \
  https://api.staging.example.com/secure-endpoint

# Expected: SSL handshake failure

Automated mTLS Test Suite

import ssl
import urllib.request
import subprocess
import pytest

BASE_URL = "https://api.staging.example.com"
CA_CERT = "test-ca.crt"
VALID_CERT = "valid-client.crt"
VALID_KEY = "valid-client.key"
INVALID_CERT = "invalid-client.crt"
INVALID_KEY = "invalid-client.key"


def make_request(url, cert=None, key=None, ca_cert=CA_CERT):
    """Make HTTPS request with optional client cert."""
    context = ssl.create_default_context(cafile=ca_cert)
    if cert and key:
        context.load_cert_chain(cert, key)
    
    try:
        req = urllib.request.Request(url)
        with urllib.request.urlopen(req, context=context) as response:
            return response.status, response.read()
    except urllib.error.HTTPError as e:
        return e.code, e.read()
    except ssl.SSLError as e:
        return None, str(e)


class TestMTLSEnforcement:
    
    def test_rejects_request_without_client_cert(self):
        """mTLS endpoint must reject requests with no client certificate."""
        status, body = make_request(f"{BASE_URL}/secure-endpoint")
        # Should fail at SSL layer (status=None) or return 400/401
        assert status is None or status in (400, 401), \
            f"Expected rejection without client cert, got {status}"
    
    def test_accepts_valid_client_cert(self):
        """mTLS endpoint must accept requests with a valid client certificate."""
        status, body = make_request(
            f"{BASE_URL}/secure-endpoint",
            cert=VALID_CERT,
            key=VALID_KEY
        )
        assert status == 200, f"Expected 200 with valid cert, got {status}"
    
    def test_rejects_cert_from_untrusted_ca(self):
        """mTLS endpoint must reject client certs from non-trusted CAs."""
        status, body = make_request(
            f"{BASE_URL}/secure-endpoint",
            cert=INVALID_CERT,
            key=INVALID_KEY
        )
        assert status is None or status in (400, 401, 403), \
            f"Expected rejection with untrusted CA cert, got {status}"
    
    def test_service_identity_isolation(self):
        """Service A cert must not authenticate as Service B identity."""
        status, body = make_request(
            f"{BASE_URL}/service-b-endpoint",  # B's endpoint
            cert=VALID_CERT,   # A's cert (valid, but wrong service)
            key=VALID_KEY
        )
        assert status in (401, 403), \
            f"Expected rejection of wrong service cert, got {status}"
    
    def test_expired_cert_rejected(self):
        """Expired client certificates must be rejected."""
        status, body = make_request(
            f"{BASE_URL}/secure-endpoint",
            cert="expired-client.crt",
            key="expired-client.key"
        )
        assert status is None or status in (400, 401), \
            f"Expected rejection of expired cert, got {status}"

Service Mesh mTLS Validation

Kubernetes with Istio

Istio manages mTLS automatically for pods in the mesh. Testing ensures policies are actually enforced.

Verify mTLS Mode

# Check PeerAuthentication policies
kubectl get peerauthentication -A

# Check if STRICT mode is set namespace-wide
kubectl get peerauthentication -n production -o yaml

# Expected output for strict mode:
# spec:
#   mtls:
#     mode: STRICT

Test mTLS from Outside the Mesh

# From a pod outside the mesh, attempt to call a mesh service
# This should be blocked in STRICT mode
kubectl run test-client --image=curlimages/curl -it --rm -- \
  curl -v http://my-service.production.svc.cluster.local:8080/api/data

# Expected: Connection refused or TLS handshake failure

Test from Within the Mesh

# From a sidecar-injected pod — Istio provides the cert automatically
kubectl exec -n production deployment/my-service -- \
  curl -v http://other-service:8080/api/data

# Expected: 200 OK — Istio handles cert presentation transparently

Verify mTLS with istioctl

# Check mTLS status for a specific service
istioctl authn tls-check my-service.production.svc.cluster.local

# Output shows mTLS status per service:
# HOST                                       STATUS     SERVER                     CLIENT
# my-service.production.svc.cluster.local    OK         mTLS                       mTLS

# Check all services in a namespace
istioctl authn tls-check -n production

Inspect Certificates in the Mesh

# View the certificate Istio has issued to a pod
istioctl proxy-config secret my-pod-name -n production

# View cert expiry
kubectl exec my-pod -n production -c istio-proxy -- \
  openssl s_client -connect other-service:8080 \
  </dev/null 2>/dev/null | openssl x509 -noout -dates

# Check the SPIFFE identity in the cert
kubectl exec my-pod -n production -c istio-proxy -- \
  openssl s_client -connect other-service:8080 \
  </dev/null 2>/dev/null | openssl x509 -noout -text | grep -A2 "Subject Alternative Name"

# SPIFFE URI format: spiffe://cluster.local/ns/production/sa/my-service

Testing Istio AuthorizationPolicy

AuthorizationPolicy controls which services can communicate:

# Allow only specific services
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: payment-service-policy
  namespace: production
spec:
  selector:
    matchLabels:
      app: payment-service
  action: ALLOW
  rules:
    - from:
        - source:
            principals:
              - "cluster.local/ns/production/sa/order-service"
              - "cluster.local/ns/production/sa/checkout-service"
      to:
        - operation:
            methods: ["POST"]
            paths: ["/api/payment/*"]

Test the policy is enforced:

# Should succeed — order-service calling payment-service
kubectl exec -n production deployment/order-service -- \
  curl -X POST http://payment-service:8080/api/payment/process \
  -H "Content-Type: application/json" \
  -d '{"amount": 100}'

# Should fail — inventory-service calling payment-service (not authorized)
kubectl exec -n production deployment/inventory-service -- \
  curl -X POST http://payment-service:8080/api/payment/process \
  -H "Content-Type: application/json" \
  -d '{"amount": 100}'
# Expected: RBAC: access denied

Cilium mTLS Testing

For clusters using Cilium instead of Istio:

# Check Cilium network policies
kubectl get ciliumnetworkpolicies -A

# Verify mTLS enforcement with Hubble
hubble observe --namespace production --protocol http --verdict DROPPED

# Test connectivity (Cilium Connectivity Test)
cilium connectivity test --test=pod-to-pod-encryption

Testing mTLS on API Gateways

Kong with mTLS Plugin

# Verify the mTLS plugin is configured
curl -s http://kong-admin:8001/plugins | jq '.data[] | select(.name == "mtls-auth")'

# Test without cert — should get 401
curl -v https://api.gateway.example.com/protected-route

# Test with valid cert
curl -v \
  --cert valid-client.crt \
  --key valid-client.key \
  --cacert gateway-ca.crt \
  https://api.gateway.example.com/protected-route

Nginx with client_certificate

# Test Nginx mTLS config
nginx -t -c /etc/nginx/nginx.conf

# Verify ssl_verify_client is on or optional_no_ca
grep -r "ssl_verify_client" /etc/nginx/

# Test with curl
curl -v \
  --cacert /etc/nginx/ssl/ca.crt \
  --cert /tmp/test-client.crt \
  --key /tmp/test-client.key \
  https://staging.example.com/api/protected

Automated mTLS Validation in CI

# .github/workflows/mtls-test.yml
name: mTLS Enforcement Tests

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  mtls-tests:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - name: Generate test PKI
        run: |
          # CA
          openssl genrsa -out test-ca.key 4096
          openssl req -new -x509 -days 1 \
            -key test-ca.key -out test-ca.crt \
            -subj "/CN=Test CA"
          
          # Valid client cert
          openssl genrsa -out valid-client.key 2048
          openssl req -new -key valid-client.key \
            -out valid-client.csr -subj "/CN=valid-service"
          openssl x509 -req -days 1 \
            -in valid-client.csr \
            -CA test-ca.crt -CAkey test-ca.key -CAcreateserial \
            -out valid-client.crt
          
          # Untrusted CA cert
          openssl genrsa -out other-ca.key 4096
          openssl req -new -x509 -days 1 \
            -key other-ca.key -out other-ca.crt \
            -subj "/CN=Other CA"
          openssl genrsa -out bad-client.key 2048
          openssl req -new -key bad-client.key \
            -out bad-client.csr -subj "/CN=bad-service"
          openssl x509 -req -days 1 \
            -in bad-client.csr \
            -CA other-ca.crt -CAkey other-ca.key -CAcreateserial \
            -out bad-client.crt

      - name: Setup test server
        run: docker-compose -f docker-compose.test.yml up -d

      - name: Run mTLS tests
        run: |
          pip install pytest
          pytest tests/mtls/ -v

      - name: Verify Istio mTLS (staging)
        if: github.ref == 'refs/heads/main'
        run: |
          # Check all services are in STRICT mode
          PERMISSIVE=$(kubectl get peerauthentication -A -o json | \
            jq -r '.items[] | select(.spec.mtls.mode == "PERMISSIVE") | .metadata.name')
          
          if [ -n "$PERMISSIVE" ]; then
            echo "::warning::Services in PERMISSIVE mTLS mode: $PERMISSIVE"
          fi
          
          # Check for missing PeerAuthentication
          NAMESPACES="production staging"
          for NS in $NAMESPACES; do
            POLICY=$(kubectl get peerauthentication -n $NS -o json | jq '.items | length')
            if [ "$POLICY" -eq 0 ]; then
              echo "::error::No PeerAuthentication policy in namespace $NS"
              exit 1
            fi
          done

Common mTLS Issues

Issue	Symptom	Fix
Certificate not sent	SSL handshake error on server	Check client cert path and key match
CA cert mismatch	`certificate verify failed`	Ensure client cert signed by the CA the server trusts
Wrong SAN	`hostname doesn't match`	Add correct hostname to SubjectAltName in cert
Cert expired	`certificate expired`	Renew; automate with cert-manager in Kubernetes
Istio in PERMISSIVE mode	mTLS not enforced	Set PeerAuthentication to STRICT
Missing namespace policy	Pods bypass mTLS	Add namespace-scoped PeerAuthentication

Certificate Rotation Testing

Testing that cert rotation doesn't cause downtime:

# Generate new cert (same CA)
openssl genrsa -out new-client.key 2048
openssl req -new -key new-client.key \
  -out new-client.csr -subj "/CN=valid-service"
openssl x509 -req -days 30 \
  -in new-client.csr \
  -CA ca.crt -CAkey ca.key -CAcreateserial \
  -out new-client.crt

# Hot-swap test: call endpoint continuously while rotating
for i in {1..20}; do
  curl -s \
    --cert new-client.crt \
    --key new-client.key \
    --cacert ca.crt \
    https://api.staging.example.com/health
  sleep 0.5
done

# All requests should return 200 — no interruption during rotation

In Kubernetes, cert-manager handles rotation automatically. Verify it's working:

# Check cert-manager certificates
kubectl get certificates -A
kubectl describe certificate my-service-cert -n production

# Check for expiring certs
kubectl get certificates -A -o json | \
  jq -r '.items[] | select(.status.notAfter) | 
    {name: .metadata.name, namespace: .metadata.namespace, expires: .status.notAfter}'