HSM Integration Testing: PKCS#11, Key Management, and Failover Validation

HSM Integration Testing: PKCS#11, Key Management, and Failover Validation

Hardware Security Modules are the trust anchors of PKI, payment systems, and high-security key management. But they're notoriously difficult to test: physical HSMs are expensive, often shared across teams, and don't reset cleanly between test runs. A poorly written test that leaves orphan key objects can fill your NVRAM and cause production failures.

This post covers how to build a proper HSM test suite using SoftHSM2 as your local stand-in, with python-pkcs11 as the API layer. The patterns apply to real HSMs (Thales Luna, Entrust nShield, AWS CloudHSM) with only configuration changes.

Why SoftHSM2

SoftHSM2 is a software PKCS#11 implementation. It exposes the same PKCS#11 interface as a physical HSM, runs on any Linux/macOS machine, and resets cleanly between tests. It doesn't provide tamper resistance or secure key storage, but for testing your PKCS#11 integration code, it's perfect.

# Install SoftHSM2
brew install softhsm  # macOS
apt-get install softhsm2  # Ubuntu

# Initialize a token
softhsm2-util --init-token \
  --slot 0 \
  --label "TestHSM" \
  --so-pin 1234567890 \
  --pin 12345678

# Verify the token is visible
softhsm2-util --show-slots

# Install python-pkcs11
pip install python-pkcs11

Find the SoftHSM2 library path:

# macOS with Homebrew
find /opt/homebrew -name "libsofthsm2.so" 2>/dev/null
# Linux
find /usr/lib -name "libsofthsm2.so" 2>/dev/null

Test Infrastructure: Fixtures and Teardown

The most important rule in HSM testing: always clean up key objects. HSM NVRAM is finite. A failed test that doesn't delete its keys will eventually fill it.

import pkcs11
import pytest
import os
from pkcs11 import Attribute, KeyType, ObjectClass, Mechanism, MGF

PKCS11_LIB = os.environ.get(
    "PKCS11_LIB",
    "/opt/homebrew/lib/softhsm/libsofthsm2.so"
)
TOKEN_LABEL = "TestHSM"
USER_PIN = "12345678"

@pytest.fixture(scope="session")
def hsm_lib():
    lib = pkcs11.lib(PKCS11_LIB)
    yield lib

@pytest.fixture(scope="session")
def hsm_token(hsm_lib):
    token = hsm_lib.get_token(token_label=TOKEN_LABEL)
    yield token

@pytest.fixture
def hsm_session(hsm_token):
    """Open a read-write session; close and clean up after each test."""
    session = hsm_token.open(user_pin=USER_PIN, rw=True)
    yield session
    # Clean up any objects created in this test
    try:
        for obj in session.get_objects():
            try:
                obj.destroy()
            except pkcs11.exceptions.ActionProhibited:
                pass  # Some token objects can't be destroyed (e.g., built-in)
    except Exception:
        pass
    finally:
        session.close()

The session-scoped fixtures reuse a single library/token handle; the function-scoped hsm_session gives each test a fresh session and cleans up key objects after.

Key Generation Testing

from pkcs11 import KeyType, Attribute, Mechanism

def test_aes_key_generation(hsm_session):
    """HSM must generate a 256-bit AES key with correct attributes."""
    key = hsm_session.generate_key(
        KeyType.AES,
        256,
        label="test-aes-key",
        id=b"\x01",
        store=True,
        capabilities=pkcs11.MechanismFlag.ENCRYPT | pkcs11.MechanismFlag.DECRYPT,
    )

    assert key[Attribute.KEY_TYPE] == KeyType.AES
    assert key[Attribute.VALUE_LEN] == 32  # 256 bits = 32 bytes
    assert key[Attribute.LABEL] == "test-aes-key"
    assert key[Attribute.TOKEN] is True  # Stored on token, not session-only
    assert key[Attribute.EXTRACTABLE] is False  # Must not be extractable

def test_rsa_keypair_generation(hsm_session):
    """HSM must generate a 2048-bit RSA keypair."""
    pub_key, priv_key = hsm_session.generate_keypair(
        KeyType.RSA,
        2048,
        label="test-rsa",
        id=b"\x02",
        store=True,
    )

    assert pub_key[Attribute.KEY_TYPE] == KeyType.RSA
    assert pub_key[Attribute.MODULUS_BITS] == 2048
    assert priv_key[Attribute.SENSITIVE] is True
    assert priv_key[Attribute.EXTRACTABLE] is False

def test_ec_keypair_generation_p256(hsm_session):
    """HSM must generate P-256 EC keypair for ECDSA signing."""
    from pkcs11.util.ec import encode_named_curve_parameters
    import pkcs11.util.ec

    # P-256 OID: 1.2.840.10045.3.1.7
    ec_params = encode_named_curve_parameters("prime256v1")

    pub_key, priv_key = hsm_session.generate_keypair(
        KeyType.EC,
        key_length=None,
        label="test-ec-p256",
        id=b"\x03",
        store=True,
        public_template={
            Attribute.EC_PARAMS: ec_params,
        },
    )

    assert pub_key[Attribute.KEY_TYPE] == KeyType.EC
    assert pub_key[Attribute.EC_PARAMS] == ec_params

def test_key_persistence_across_sessions(hsm_token):
    """Keys stored with TOKEN=True must persist across sessions."""
    label = "persist-test-key"
    key_id = b"\x10"

    # Session 1: Create key
    with hsm_token.open(user_pin=USER_PIN, rw=True) as s1:
        key = s1.generate_key(
            KeyType.AES, 128,
            label=label, id=key_id, store=True
        )
        assert key is not None

    # Session 2: Find the key
    with hsm_token.open(user_pin=USER_PIN) as s2:
        found = list(s2.get_objects({
            Attribute.LABEL: label,
            Attribute.ID: key_id,
        }))
        assert len(found) == 1, "Persistent key not found in new session"

    # Cleanup
    with hsm_token.open(user_pin=USER_PIN, rw=True) as s_cleanup:
        for obj in s_cleanup.get_objects({Attribute.ID: key_id}):
            obj.destroy()

Cryptographic Operation Validation

def test_aes_gcm_encrypt_decrypt(hsm_session):
    """AES-GCM encrypt+decrypt roundtrip must recover original plaintext."""
    key = hsm_session.generate_key(
        KeyType.AES, 256,
        label="gcm-test", id=b"\x20",
        capabilities=pkcs11.MechanismFlag.ENCRYPT | pkcs11.MechanismFlag.DECRYPT,
    )

    plaintext = b"Sensitive data that must not leak"
    iv = os.urandom(12)  # 96-bit IV for GCM

    ciphertext, tag = key.encrypt(
        plaintext,
        mechanism=Mechanism.AES_GCM,
        mechanism_param=pkcs11.GCMParameters(iv=iv, tag_length=128),
    )

    recovered = key.decrypt(
        ciphertext + tag,
        mechanism=Mechanism.AES_GCM,
        mechanism_param=pkcs11.GCMParameters(iv=iv, tag_length=128),
    )

    assert recovered == plaintext

def test_aes_gcm_tampered_ciphertext_rejected(hsm_session):
    """GCM authentication must reject tampered ciphertext."""
    key = hsm_session.generate_key(
        KeyType.AES, 256, label="gcm-tamper-test", id=b"\x21",
        capabilities=pkcs11.MechanismFlag.ENCRYPT | pkcs11.MechanismFlag.DECRYPT,
    )

    plaintext = b"authentic message"
    iv = os.urandom(12)

    ciphertext, tag = key.encrypt(
        plaintext,
        mechanism=Mechanism.AES_GCM,
        mechanism_param=pkcs11.GCMParameters(iv=iv, tag_length=128),
    )

    # Flip a byte in the ciphertext
    tampered = bytearray(ciphertext)
    tampered[0] ^= 0xFF

    with pytest.raises(pkcs11.exceptions.FunctionFailed):
        key.decrypt(
            bytes(tampered) + tag,
            mechanism=Mechanism.AES_GCM,
            mechanism_param=pkcs11.GCMParameters(iv=iv, tag_length=128),
        )

def test_rsa_sign_verify(hsm_session):
    """RSA-PSS sign/verify roundtrip."""
    pub_key, priv_key = hsm_session.generate_keypair(
        KeyType.RSA, 2048,
        label="rsa-sign-test", id=b"\x30",
        store=True,
    )

    message = b"Document to sign"

    signature = priv_key.sign(
        message,
        mechanism=Mechanism.SHA256_RSA_PKCS_PSS,
        mechanism_param=pkcs11.RSAPSSParameters(
            hash_alg=Mechanism.SHA256,
            mask_gen=MGF.SHA256,
            salt_length=32,
        ),
    )

    assert pub_key.verify(
        message,
        signature,
        mechanism=Mechanism.SHA256_RSA_PKCS_PSS,
        mechanism_param=pkcs11.RSAPSSParameters(
            hash_alg=Mechanism.SHA256,
            mask_gen=MGF.SHA256,
            salt_length=32,
        ),
    )

def test_rsa_sign_rejects_tampered_signature(hsm_session):
    """Tampered signature must fail verification."""
    pub_key, priv_key = hsm_session.generate_keypair(
        KeyType.RSA, 2048,
        label="rsa-tamper-test", id=b"\x31",
        store=True,
    )

    message = b"Original message"
    signature = priv_key.sign(message, mechanism=Mechanism.SHA256_RSA_PKCS)

    tampered_sig = bytearray(signature)
    tampered_sig[-1] ^= 0xFF

    with pytest.raises(pkcs11.exceptions.SignatureInvalid):
        pub_key.verify(
            message,
            bytes(tampered_sig),
            mechanism=Mechanism.SHA256_RSA_PKCS,
        )

Key Import Testing

Many workflows require importing externally generated keys (e.g., a KMS-generated key wrapped for HSM import).

from pkcs11.util.rsa import encode_rsa_public_key
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.backends import default_backend

def test_external_rsa_public_key_import(hsm_session):
    """Import an externally generated RSA public key and use it to verify."""
    # Generate RSA key externally (e.g., via cryptography library)
    ext_key = rsa.generate_private_key(
        public_exponent=65537,
        key_size=2048,
        backend=default_backend(),
    )
    ext_pub = ext_key.public_key()

    # Import public key into HSM
    from pkcs11.util.rsa import decode_rsa_public_key
    pub_attrs = decode_rsa_public_key(
        ext_pub.public_bytes(
            encoding=serialization.Encoding.DER,
            format=serialization.PublicFormat.SubjectPublicKeyInfo,
        )
    )
    pub_attrs.update({
        Attribute.LABEL: "imported-rsa-pub",
        Attribute.TOKEN: True,
        Attribute.ID: b"\x40",
    })
    hsm_pub = hsm_session.create_object(pub_attrs)

    assert hsm_pub[Attribute.KEY_TYPE] == KeyType.RSA
    assert hsm_pub[Attribute.MODULUS_BITS] == 2048

HSM Failover and HA Testing

For production HSMs in HA configurations (Luna Network HSM HA group, nShield Security World cluster), test that your application handles HSM failover without dropping cryptographic operations.

import threading
import time
import queue

def test_concurrent_operations_no_corruption(hsm_token):
    """Concurrent sign operations must each return valid, independent signatures."""
    errors = queue.Queue()
    signatures = queue.Queue()

    message = b"Concurrent signing test"

    def sign_in_thread(thread_id: int):
        try:
            with hsm_token.open(user_pin=USER_PIN) as session:
                pub_key, priv_key = session.generate_keypair(
                    KeyType.RSA, 2048,
                    label=f"concurrent-{thread_id}",
                    id=bytes([0x50 + thread_id]),
                    store=True,
                )
                sig = priv_key.sign(message, mechanism=Mechanism.SHA256_RSA_PKCS)
                is_valid = pub_key.verify(message, sig, mechanism=Mechanism.SHA256_RSA_PKCS)
                signatures.put((thread_id, is_valid))
                # Cleanup
                priv_key.destroy()
                pub_key.destroy()
        except Exception as e:
            errors.put((thread_id, str(e)))

    threads = [threading.Thread(target=sign_in_thread, args=(i,)) for i in range(5)]
    for t in threads:
        t.start()
    for t in threads:
        t.join(timeout=30)

    assert errors.empty(), f"Thread errors: {list(errors.queue)}"
    results = list(signatures.queue)
    assert len(results) == 5
    assert all(valid for _, valid in results), "Some signatures failed verification"

def test_session_recovery_after_disconnect(hsm_token):
    """Application must recover from dropped HSM session."""
    with hsm_token.open(user_pin=USER_PIN, rw=True) as session:
        key = session.generate_key(
            KeyType.AES, 256,
            label="recovery-test", id=b"\x60",
        )
        # Simulate session disruption by closing mid-operation
        # In real HA testing, you'd pull the network cable or kill the HSM process

    # After disruption, open a new session and verify token state is intact
    with hsm_token.open(user_pin=USER_PIN) as new_session:
        # Session-only key should be gone; verify graceful state
        found = list(new_session.get_objects({Attribute.ID: b"\x60"}))
        assert len(found) == 0, "Session-only key must not persist after session close"

Performance Under Load

import time
import statistics

def test_hsm_signing_throughput(hsm_session):
    """Baseline: HSM must sustain >10 RSA-2048 signs/sec under load."""
    pub_key, priv_key = hsm_session.generate_keypair(
        KeyType.RSA, 2048,
        label="perf-test", id=b"\x70",
        store=True,
    )

    message = b"Performance test message"
    iterations = 50
    times = []

    for _ in range(iterations):
        t0 = time.perf_counter()
        sig = priv_key.sign(message, mechanism=Mechanism.SHA256_RSA_PKCS)
        times.append(time.perf_counter() - t0)

    median_ms = statistics.median(times) * 1000
    ops_per_sec = 1.0 / statistics.median(times)

    print(f"RSA-2048 sign: p50={median_ms:.1f}ms, throughput={ops_per_sec:.1f} ops/sec")

    # SoftHSM2 on modern hardware: >100 ops/sec is normal
    # Physical HSM: 10-1000 ops/sec depending on model
    assert ops_per_sec > 10, f"Signing too slow: {ops_per_sec:.1f} ops/sec"

AWS CloudHSM Testing Differences

AWS CloudHSM uses the same PKCS#11 interface but with a custom provider (cloudhsm_pkcs11.so). The main differences in your test configuration:

# AWS CloudHSM
PKCS11_LIB = "/opt/cloudhsm/lib/libcloudhsm_pkcs11.so"

# CloudHSM requires cluster password, not PIN
# Set via: /opt/cloudhsm/bin/configure-pkcs11 --hsm-ca-cert cluster-ca.crt

# CloudHSM doesn't support all mechanisms — check before testing
# Unsupported: AES-GCM with IV generated by HSM (must provide IV)
# Supported: AES-CBC, RSA-PKCS, ECDSA, HMAC-SHA256

For Thales Luna HSM, the library is /usr/safenet/lunaclient/lib/libCryptoki2_64.so. The PKCS#11 interface is identical; your test code runs unchanged.

What to Run in CI vs Pre-Release

In CI (every commit):

  • Key generation for each key type you use
  • Encrypt/decrypt roundtrip with tamper detection
  • Sign/verify roundtrip with tamper rejection
  • Key persistence check
  • Performance baseline (with tight thresholds that fail the build)

Pre-release only (slow, needs real HSM):

  • Full failover test (requires HA HSM cluster)
  • Concurrent load test (50+ threads)
  • Full KAT vector validation for all mechanisms
  • NVRAM exhaustion recovery test

Keep your CI suite under 60 seconds by using SoftHSM2 and limiting to the mechanisms your application actually uses. Don't test every PKCS#11 mechanism — test the ones in your critical path.

Read more

Start now free