Testing Pub/Sub Messaging Patterns: From Redis to Kafka

Testing Pub/Sub Messaging Patterns: From Redis to Kafka

Pub/sub messaging is everywhere in modern applications: chat systems, live notifications, event-driven microservices, real-time dashboards. But testing pub/sub is tricky — messages are asynchronous, delivery isn't guaranteed in all systems, ordering matters in some but not others, and consumer failures create subtle bugs that only appear under load.

This guide covers pub/sub testing strategies for JavaScript/TypeScript applications, from simple in-memory brokers through Redis Pub/Sub and Kafka.

The Core Challenge: Asynchronous Message Flow

In a synchronous system, doThing() produces a result you can assert immediately. In pub/sub, the sequence is:

  1. Publisher publishes a message
  2. Broker receives it
  3. Broker delivers it to subscribers
  4. Subscribers process it

Steps 2-4 happen asynchronously, sometimes across network hops. Your test can't just assert after publish() — it needs to wait for downstream processing to complete.

The key pattern: subscribe before publishing, collect messages into a buffer, and use a Promise that resolves when the expected messages arrive.

Testing with an In-Memory Broker

Start with an in-memory pub/sub for unit tests. No external dependencies, fast, deterministic:

// message-bus.js
class MessageBus {
  constructor() {
    this._subscriptions = new Map();
  }

  subscribe(channel, handler) {
    if (!this._subscriptions.has(channel)) {
      this._subscriptions.set(channel, new Set());
    }
    this._subscriptions.get(channel).add(handler);

    return () => this._subscriptions.get(channel).delete(handler);
  }

  async publish(channel, message) {
    const handlers = this._subscriptions.get(channel) || new Set();
    await Promise.all([...handlers].map(h => h(message)));
  }
}

module.exports = { MessageBus };
// message-bus.test.js
const { MessageBus } = require('./message-bus');

test('subscriber receives published message', async () => {
  const bus = new MessageBus();
  const received = [];

  bus.subscribe('orders', (msg) => received.push(msg));
  await bus.publish('orders', { id: '001', amount: 99.99 });

  expect(received).toHaveLength(1);
  expect(received[0].id).toBe('001');
});

test('multiple subscribers all receive the message', async () => {
  const bus = new MessageBus();
  const received1 = [];
  const received2 = [];

  bus.subscribe('events', (msg) => received1.push(msg));
  bus.subscribe('events', (msg) => received2.push(msg));

  await bus.publish('events', { type: 'user.created' });

  expect(received1).toHaveLength(1);
  expect(received2).toHaveLength(1);
});

test('unsubscribed handler no longer receives messages', async () => {
  const bus = new MessageBus();
  const received = [];

  const unsubscribe = bus.subscribe('news', (msg) => received.push(msg));
  await bus.publish('news', { headline: 'first' });
  unsubscribe();
  await bus.publish('news', { headline: 'second' });

  expect(received).toHaveLength(1);
  expect(received[0].headline).toBe('first');
});

Testing Application Logic with a Mocked Broker

Your application code shouldn't care whether the broker is Redis or Kafka. Inject a mock:

// order-service.js
class OrderService {
  constructor(bus) {
    this.bus = bus;
  }

  async placeOrder(order) {
    // Validate, persist, etc.
    const confirmed = { ...order, confirmedAt: Date.now(), status: 'confirmed' };
    await this.bus.publish('orders.confirmed', confirmed);
    return confirmed;
  }

  subscribeToPayments(handler) {
    return this.bus.subscribe('payments.received', handler);
  }
}

module.exports = { OrderService };
// order-service.test.js
const { OrderService } = require('./order-service');
const { MessageBus } = require('./message-bus');

test('placeOrder publishes to orders.confirmed channel', async () => {
  const bus = new MessageBus();
  const published = [];
  bus.subscribe('orders.confirmed', (msg) => published.push(msg));

  const service = new OrderService(bus);
  await service.placeOrder({ id: 'order-1', total: 150 });

  expect(published).toHaveLength(1);
  expect(published[0].id).toBe('order-1');
  expect(published[0].status).toBe('confirmed');
  expect(published[0].confirmedAt).toBeDefined();
});

test('payment triggers order fulfillment', async () => {
  const bus = new MessageBus();
  const service = new OrderService(bus);
  const fulfilled = [];

  service.subscribeToPayments(async (payment) => {
    fulfilled.push(payment.orderId);
  });

  await bus.publish('payments.received', { orderId: 'order-1', amount: 150 });

  expect(fulfilled).toContain('order-1');
});

Integration Testing with Redis Pub/Sub

For integration tests, use a real Redis instance (via Docker in CI):

# docker-compose.test.yml
services:
  redis:
    image: redis:7-alpine
    ports:
      - "6379:6379"
// redis-bus.test.js
const { createClient } = require('redis');

let publisher;
let subscriber;

beforeAll(async () => {
  publisher = createClient({ url: 'redis://localhost:6379' });
  subscriber = createClient({ url: 'redis://localhost:6379' });
  await Promise.all([publisher.connect(), subscriber.connect()]);
});

afterAll(async () => {
  await Promise.all([publisher.disconnect(), subscriber.disconnect()]);
});

function waitForMessage(channel, predicate, timeoutMs = 5000) {
  return new Promise((resolve, reject) => {
    const timer = setTimeout(() => {
      reject(new Error(`Timeout waiting for message on ${channel}`));
    }, timeoutMs);

    subscriber.subscribe(channel, (message) => {
      const parsed = JSON.parse(message);
      if (!predicate || predicate(parsed)) {
        clearTimeout(timer);
        subscriber.unsubscribe(channel);
        resolve(parsed);
      }
    });
  });
}

test('message published to Redis is received by subscriber', async () => {
  const receivePromise = waitForMessage(
    'test:orders',
    (msg) => msg.id === 'test-001'
  );

  // Small delay to ensure subscription is active
  await new Promise(r => setTimeout(r, 50));

  await publisher.publish('test:orders', JSON.stringify({
    id: 'test-001',
    amount: 200,
  }));

  const msg = await receivePromise;
  expect(msg.id).toBe('test-001');
  expect(msg.amount).toBe(200);
}, 10000);

test('pattern subscription receives matching channels', async () => {
  const received = [];

  await subscriber.pSubscribe('test:events:*', (message, channel) => {
    received.push({ channel, data: JSON.parse(message) });
  });

  await new Promise(r => setTimeout(r, 50));

  await publisher.publish('test:events:user', JSON.stringify({ type: 'login' }));
  await publisher.publish('test:events:order', JSON.stringify({ type: 'created' }));
  await publisher.publish('test:other', JSON.stringify({ type: 'ignored' }));

  await new Promise(r => setTimeout(r, 200));
  await subscriber.pUnsubscribe('test:events:*');

  expect(received).toHaveLength(2);
  expect(received.map(r => r.data.type)).toEqual(
    expect.arrayContaining(['login', 'created'])
  );
});

Testing Kafka Consumer Groups

Kafka introduces consumer groups — multiple consumers sharing message processing. Testing consumer groups requires verifying that messages are partitioned correctly and each message is processed exactly once.

// kafka-consumer.test.js
const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'test-client',
  brokers: ['localhost:9092'],
});

const TOPIC = `test-topic-${Date.now()}`; // Unique topic per test run

let producer;
let consumer;

beforeAll(async () => {
  const admin = kafka.admin();
  await admin.connect();
  await admin.createTopics({
    topics: [{ topic: TOPIC, numPartitions: 3 }],
  });
  await admin.disconnect();

  producer = kafka.producer();
  await producer.connect();

  consumer = kafka.consumer({ groupId: `test-group-${Date.now()}` });
  await consumer.connect();
});

afterAll(async () => {
  await consumer.disconnect();
  await producer.disconnect();
});

function consumeN(n, timeoutMs = 10000) {
  return new Promise(async (resolve, reject) => {
    const messages = [];
    const timer = setTimeout(() => {
      reject(new Error(`Received only ${messages.length}/${n} messages`));
    }, timeoutMs);

    await consumer.subscribe({ topic: TOPIC, fromBeginning: true });

    await consumer.run({
      eachMessage: async ({ message }) => {
        messages.push(JSON.parse(message.value.toString()));
        if (messages.length >= n) {
          clearTimeout(timer);
          resolve(messages);
        }
      },
    });
  });
}

test('all published messages are consumed', async () => {
  const consumePromise = consumeN(3);

  await producer.sendBatch({
    topicMessages: [
      {
        topic: TOPIC,
        messages: [
          { value: JSON.stringify({ id: 1, data: 'a' }) },
          { value: JSON.stringify({ id: 2, data: 'b' }) },
          { value: JSON.stringify({ id: 3, data: 'c' }) },
        ],
      },
    ],
  });

  const consumed = await consumePromise;
  expect(consumed).toHaveLength(3);
  const ids = consumed.map(m => m.id).sort();
  expect(ids).toEqual([1, 2, 3]);
}, 15000);

Testing Message Ordering and Idempotency

Two critical behaviors to test explicitly:

Ordering: Within a Kafka partition, messages are ordered. Across partitions, they're not. Test that your application doesn't assume cross-partition ordering:

test('handles out-of-order messages across partitions', async () => {
  const processed = [];

  const bus = new MessageBus();
  bus.subscribe('events', (msg) => processed.push(msg.seq));

  // Simulate out-of-order delivery
  await bus.publish('events', { seq: 3 });
  await bus.publish('events', { seq: 1 });
  await bus.publish('events', { seq: 2 });

  // Your application should handle this gracefully
  // (reorder, use timestamps, or be order-independent)
  expect(processed).toContain(1);
  expect(processed).toContain(2);
  expect(processed).toContain(3);
});

Idempotency: At-least-once delivery means your consumers will see duplicates. Test that duplicate processing is safe:

test('duplicate message processing is idempotent', async () => {
  const db = new Map(); // Simulated database

  const handler = async (msg) => {
    if (db.has(msg.id)) return; // Idempotency check
    db.set(msg.id, msg);
  };

  const bus = new MessageBus();
  bus.subscribe('orders', handler);

  const order = { id: 'order-duplicate', amount: 50 };
  await bus.publish('orders', order);
  await bus.publish('orders', order); // Duplicate
  await bus.publish('orders', order); // Another duplicate

  expect(db.size).toBe(1); // Stored exactly once
  expect(db.get('order-duplicate').amount).toBe(50);
});

Testing the Full Stack

Unit and integration tests verify your pub/sub logic in isolation. But in production, pub/sub feeds real-time UI updates — does the notification actually appear in the browser when a message arrives? That's where end-to-end testing matters.

HelpMeTest can run multi-step browser tests against your staging environment: trigger an action that publishes a message, then verify the subscriber's UI updates within your expected latency window. Because HelpMeTest runs in the cloud and supports concurrent browser sessions, you can test multi-user scenarios — two users in the same chat room, admin publishes announcement and all subscribers see it — without managing parallel browser infrastructure yourself.

Summary

Pub/sub testing has three layers:

  1. Unit tests with in-memory brokers: Test application logic — what does your service publish? What does it do when it receives a message? Fast, no external dependencies.
  2. Integration tests with real brokers: Verify Redis pattern subscriptions, Kafka partition behavior, consumer group semantics. Use Docker Compose to spin up real brokers in CI.
  3. Behavioral tests for ordering and idempotency: Explicitly test out-of-order delivery and duplicate messages — at-least-once delivery will guarantee you see these in production.

The pattern that saves the most debugging time: subscribe before publishing in tests, and always use Promises with explicit timeouts rather than sleep. Deterministic test timing is the difference between tests you trust and tests you re-run to see if they pass.

Read more

Start now free