Testing

Event-Driven Serverless Testing Patterns

HelpMeTest

HelpMeTest

— 7 min read
Event-Driven Serverless Testing Patterns

Event-driven serverless architectures are difficult to test for a specific reason: the chain of cause and effect spans multiple services, most of which are asynchronous. An HTTP request triggers a Lambda that publishes to SNS, which fans out to three SQS queues, each of which triggers a different Lambda that writes to DynamoDB and maybe emits another event.

Testing each Lambda in isolation is straightforward. Testing that the whole chain works correctly — and catches every failure mode — requires deliberate strategy.

The Testing Challenges

Asynchrony. Events don't complete synchronously. After your function publishes a message, you don't know when downstream handlers will process it. Tests need to wait, poll, or observe eventual state.

Distributed state. The result of an event chain is spread across multiple services. To verify an order was processed correctly, you need to check DynamoDB, S3, and maybe a third service.

Non-linear flow. Fan-out patterns (one event → multiple handlers) mean multiple things happen "at once." Fan-in patterns (multiple events → one aggregator) create coordination complexity.

Partial failures. When one step in a chain fails, you need to verify that the right retry/rollback behavior occurs — not just that the failure is logged.

Idempotency. Events can be delivered more than once (at-least-once delivery). Your handlers must produce the same result whether called once or three times.

Strategy 1: Test Each Lambda in Isolation First

Before testing event chains, verify each Lambda handles all event shapes correctly in isolation. This is standard unit testing:

// order-processor.handler.test.js
const { handler } = require('./order-processor.handler');
const { DynamoDBDocumentClient } = require('@aws-sdk/lib-dynamodb');
const { mockClient } = require('aws-sdk-client-mock');

describe('order-processor handler', () => {
  test('processes valid order event', async () => {
    const event = buildSQSEvent([
      { orderId: 'ord-1', customerId: 'cust-1', items: [{ sku: 'ITEM-A', qty: 2 }] }
    ]);
    
    const result = await handler(event);
    
    expect(result.batchItemFailures).toEqual([]);
  });
  
  test('marks malformed messages as batch failures', async () => {
    const event = buildSQSEvent([
      'not-json',
      { orderId: 'valid-order', customerId: 'cust-1', items: [] },
    ]);
    
    const result = await handler(event);
    
    expect(result.batchItemFailures).toHaveLength(1);
    expect(result.batchItemFailures[0].itemIdentifier).toBe(event.Records[0].messageId);
  });
});

function buildSQSEvent(messages) {
  return {
    Records: messages.map((msg, i) => ({
      messageId: `msg-${i}`,
      receiptHandle: `receipt-${i}`,
      body: typeof msg === 'string' ? msg : JSON.stringify(msg),
      attributes: { ApproximateReceiveCount: '1', SentTimestamp: Date.now().toString() },
      eventSource: 'aws:sqs',
      awsRegion: 'us-east-1',
    })),
  };
}

Strategy 2: Contract Testing for Event Schemas

Event-driven systems break when a producer changes the event schema without updating consumers. Contract testing catches schema drift.

Define your event contracts explicitly:

// contracts/order-placed.contract.ts
import { z } from 'zod';

export const OrderPlacedEventSchema = z.object({
  eventType: z.literal('order.placed'),
  version: z.string().regex(/^\d+\.\d+$/),
  orderId: z.string().uuid(),
  customerId: z.string(),
  items: z.array(z.object({
    sku: z.string(),
    quantity: z.number().positive(),
    priceAtTime: z.number().positive(),
  })).nonempty(),
  placedAt: z.number(), // Unix timestamp
  metadata: z.record(z.string()).optional(),
});

export type OrderPlacedEvent = z.infer<typeof OrderPlacedEventSchema>;

Test that producers emit valid events:

// order.service.contract.test.ts
import { OrderPlacedEventSchema } from '../contracts/order-placed.contract';
import { OrderService } from './order.service';

describe('OrderService event contracts', () => {
  test('placeOrder emits event conforming to contract', async () => {
    const capturedEvents: unknown[] = [];
    const mockPublisher = { publish: jest.fn((event) => capturedEvents.push(event)) };
    
    const service = new OrderService(mockPublisher, mockOrderRepo);
    await service.placeOrder({
      customerId: 'cust-123',
      items: [{ sku: 'ITEM-A', quantity: 1, priceAtTime: 29.99 }],
    });
    
    expect(capturedEvents).toHaveLength(1);
    
    // This will throw if the event doesn't match the contract
    const parsed = OrderPlacedEventSchema.safeParse(capturedEvents[0]);
    expect(parsed.success).toBe(true);
    
    if (parsed.success) {
      expect(parsed.data.eventType).toBe('order.placed');
      expect(parsed.data.items).toHaveLength(1);
    }
  });
  
  test('consumer accepts all valid contract versions', () => {
    const validEvent: OrderPlacedEvent = {
      eventType: 'order.placed',
      version: '1.0',
      orderId: '550e8400-e29b-41d4-a716-446655440000',
      customerId: 'cust-1',
      items: [{ sku: 'ITEM-A', quantity: 1, priceAtTime: 9.99 }],
      placedAt: Date.now(),
    };
    
    const result = OrderPlacedEventSchema.safeParse(validEvent);
    expect(result.success).toBe(true);
  });
});

Strategy 3: Testing Idempotency

At-least-once delivery means your handlers will process the same event multiple times under failure conditions. Test that this doesn't cause duplicate work:

// order-processor.idempotency.test.ts
describe('order processor idempotency', () => {
  test('processing same order twice produces same result', async () => {
    const event = buildSQSEvent([
      { orderId: 'ord-duplicate', customerId: 'cust-1', items: [{ sku: 'ITEM-A', qty: 1 }] }
    ]);
    
    // Process same event twice
    await handler(event);
    await handler(event); // This is the duplicate delivery
    
    // Should result in exactly one order in the database
    const orders = await orderRepo.findByOrderId('ord-duplicate');
    expect(orders).toHaveLength(1);
    
    // Should charge customer exactly once
    expect(mockPaymentService.charge).toHaveBeenCalledTimes(1);
  });
  
  test('idempotency key expires after retention period', async () => {
    const event = buildSQSEvent([{ orderId: 'ord-old', ... }]);
    
    // Process and mark as processed 25 hours ago
    await idempotencyStore.markProcessed('ord-old', { expiresAt: Date.now() - 25 * 3600 * 1000 });
    
    // Processing again after expiry should re-process
    await handler(event);
    
    expect(mockPaymentService.charge).toHaveBeenCalledTimes(1); // Processed again
  });
});

Strategy 4: End-to-End Event Chain Testing

Test that a complete event chain produces the expected final state. Use polling to wait for eventual consistency:

// order-flow.e2e.test.ts
async function waitForCondition(
  check: () => Promise<boolean>,
  timeout: number = 30000,
  interval: number = 500
): Promise<void> {
  const start = Date.now();
  while (Date.now() - start < timeout) {
    if (await check()) return;
    await sleep(interval);
  }
  throw new Error(`Condition not met within ${timeout}ms`);
}

describe('Order flow end-to-end', () => {
  test('order placed → inventory reserved → payment charged → confirmation sent', async () => {
    // 1. Trigger: place order via API
    const response = await fetch('http://localhost:3000/orders', {
      method: 'POST',
      body: JSON.stringify({
        customerId: 'cust-e2e',
        items: [{ sku: 'ITEM-A', qty: 1 }],
      }),
      headers: { 'Content-Type': 'application/json' },
    });
    
    expect(response.status).toBe(202); // Accepted, processing async
    const { orderId } = await response.json();
    
    // 2. Wait for inventory reservation
    await waitForCondition(
      async () => {
        const reservation = await inventoryService.getReservation(orderId);
        return reservation?.status === 'reserved';
      },
      10000 // 10 second timeout
    );
    
    // 3. Wait for payment processing
    await waitForCondition(
      async () => {
        const payment = await paymentService.getByOrderId(orderId);
        return payment?.status === 'charged';
      },
      15000
    );
    
    // 4. Verify confirmation email queued
    await waitForCondition(
      async () => {
        const emails = await emailQueue.getByOrderId(orderId);
        return emails.some(e => e.type === 'order_confirmation');
      },
      5000
    );
    
    // 5. Final state check
    const order = await orderRepo.getById(orderId);
    expect(order.status).toBe('confirmed');
  });
});

Strategy 5: Testing Fan-Out Patterns

When one event triggers multiple downstream handlers, test that all handlers receive the event and process it correctly:

// notification-fanout.test.ts
describe('notification fan-out', () => {
  test('order.placed triggers all notification channels', async () => {
    const orderId = 'ord-fanout-test';
    
    // Publish the event
    await eventBus.publish({
      eventType: 'order.placed',
      orderId,
      customerId: 'cust-fanout',
      items: [{ sku: 'ITEM-A', qty: 1, priceAtTime: 10 }],
      placedAt: Date.now(),
    });
    
    // Wait for all downstream handlers
    await Promise.all([
      // Email notification handler
      waitForCondition(
        async () => (await emailLogs.findByOrderId(orderId)).length > 0,
        5000
      ),
      // Push notification handler
      waitForCondition(
        async () => (await pushLogs.findByOrderId(orderId)).length > 0,
        5000
      ),
      // Analytics handler
      waitForCondition(
        async () => (await analyticsStore.getEvent('order_placed', orderId)) !== null,
        5000
      ),
    ]);
    
    // Verify each handler processed correctly
    const emails = await emailLogs.findByOrderId(orderId);
    expect(emails[0].templateId).toBe('order-confirmation');
    
    const pushNotifications = await pushLogs.findByOrderId(orderId);
    expect(pushNotifications[0].title).toContain('Order placed');
    
    const analyticsEvent = await analyticsStore.getEvent('order_placed', orderId);
    expect(analyticsEvent.properties.orderId).toBe(orderId);
  });
});

Strategy 6: Failure and Retry Testing

Test that failures are retried correctly and eventually reach the dead letter queue:

// retry-behavior.test.ts
describe('retry and DLQ behavior', () => {
  test('transient failure retries and eventually succeeds', async () => {
    let attempts = 0;
    mockPaymentService.charge.mockImplementation(() => {
      attempts++;
      if (attempts < 3) throw new Error('Payment gateway timeout');
      return { transactionId: 'txn-success' };
    });
    
    await publishEvent({ eventType: 'order.placed', orderId: 'ord-retry' });
    
    await waitForCondition(
      async () => {
        const order = await orderRepo.getById('ord-retry');
        return order?.paymentStatus === 'charged';
      },
      30000 // Allow time for retries
    );
    
    expect(attempts).toBe(3);
  });
  
  test('persistent failure routes to DLQ after max retries', async () => {
    mockInventoryService.reserve.mockRejectedValue(
      new Error('Inventory service unavailable')
    );
    
    await publishEvent({ eventType: 'order.placed', orderId: 'ord-dlq' });
    
    // Wait for DLQ delivery (after all retries exhaust)
    await waitForCondition(
      async () => {
        const dlqMessages = await sqs.receiveMessages(dlqUrl);
        return dlqMessages.some(m => JSON.parse(m.body).orderId === 'ord-dlq');
      },
      120000 // Long timeout — max retries take time
    );
    
    const order = await orderRepo.getById('ord-dlq');
    expect(order.status).toBe('failed');
  });
});

Strategy 7: Chaos Testing for Event Systems

Introduce failures to verify your event system handles them gracefully:

// chaos.test.ts
describe('chaos scenarios', () => {
  test('partial DynamoDB outage does not lose events', async () => {
    // Make DynamoDB fail 50% of the time
    let callCount = 0;
    ddbMock.on(PutCommand).callsFake(() => {
      callCount++;
      if (callCount % 2 === 0) throw new ProvisionedThroughputExceededException({ message: '' });
      return {};
    });
    
    // Publish 10 events
    const orderIds = Array.from({ length: 10 }, (_, i) => `chaos-ord-${i}`);
    await Promise.all(orderIds.map(orderId => publishEvent({ eventType: 'order.placed', orderId })));
    
    // Eventually all 10 should be in DynamoDB (retries handle the failures)
    await waitForCondition(
      async () => {
        const count = await orderRepo.countByPrefix('chaos-ord-');
        return count === 10;
      },
      60000
    );
  });
  
  test('message broker restart does not lose in-flight events', async () => {
    const orderId = 'restart-test';
    await publishEvent({ eventType: 'order.placed', orderId });
    
    // Restart the message broker mid-flight
    await restartSQS();
    
    // Event should eventually process
    await waitForCondition(
      async () => (await orderRepo.getById(orderId)) !== null,
      30000
    );
  });
});

Monitoring Event Chains in Production

Unit tests verify isolated handler logic. E2E tests verify the chain works end-to-end. Production monitoring verifies it keeps working:

*** Test Cases ***
Event Chain Health Check
    [Documentation]    Trigger an event and verify full chain completes within SLA
    ${orderId}=    Generate Order ID
    POST    /api/test/orders    {"orderId": "${orderId}", "type": "health-check"}
    
    Wait Until Keyword Succeeds    12x    5s
    ...    Order Status Should Be    ${orderId}    confirmed
    
    Verify DynamoDB Record Exists    orders    ${orderId}
    Verify Email Was Queued    ${orderId}

Run this every 5 minutes. Event chains can break silently: a Lambda may fail but not affect the HTTP response. Continuous monitoring catches stuck event chains before users notice their orders are in limbo.

What to Test at Each Layer

Layer What to Test Tools
Handler unit tests Each handler in isolation, all event shapes, error paths Jest + aws-sdk-mock
Contract tests Event schema matches between producer and consumer Zod schemas + Jest
Idempotency tests Duplicate events don't cause duplicate work Integration tests
Chain tests Full event flow end-to-end LocalStack + polling
Failure tests Retry behavior, DLQ routing LocalStack + fault injection
Production monitoring SLA compliance, chain completion HelpMeTest

The most common testing gap: teams write excellent unit tests for individual Lambdas and skip chain tests entirely. That gap is exactly where production incidents happen — not in individual handlers, but in the handoffs between them.

Read more

Start now free