Testing

Testing Conversational Flows in Chatbots and Voice Assistants

HelpMeTest

HelpMeTest

— 8 min read
Testing Conversational Flows in Chatbots and Voice Assistants

Conversational AI systems fail in ways that traditional software doesn't. A REST API either returns a 200 or it doesn't. A chatbot might misunderstand the user, lose context mid-conversation, fill a slot with the wrong value, or route to an escalation path when it should have handled the intent itself. Testing these systems requires a fundamentally different mental model — one built around dialog flows, state transitions, and user utterance variation.

This guide walks through the core testing strategies for conversational AI: state machine testing, happy path and error path coverage, slot-filling validation, context carryover, fallback and escalation paths, and end-to-end dialog simulation using Botium and Rasa Test.

Modeling Conversations as State Machines

Every conversation is a state machine. The bot starts in an initial state, transitions between states based on user input and extracted entities, and eventually reaches a terminal state (task complete, escalated, or abandoned). Before writing a single test, draw the state machine.

For a simple flight booking bot:

IDLE → [intent: book_flight] → COLLECT_ORIGIN
COLLECT_ORIGIN → [entity: city] → COLLECT_DESTINATION
COLLECT_DESTINATION → [entity: city] → COLLECT_DATE
COLLECT_DATE → [entity: date] → CONFIRM
CONFIRM → [intent: affirm] → BOOKED
CONFIRM → [intent: deny] → IDLE
* → [intent: cancel] → IDLE
* → [no match] → FALLBACK
FALLBACK → [intent: escalate] → ESCALATION

Each arrow is a test case. Each state is an assertion point. This diagram tells you exactly which paths must be covered before the bot ships.

State coverage checklist:

  • Every state reachable from IDLE? (reachability)
  • Every transition exercised at least once? (transition coverage)
  • Every terminal state reachable? (completeness)
  • Every state reachable via at least two different utterance phrasings? (NLU robustness)

Testing with Botium (JavaScript)

Botium is the de-facto standard for automated chatbot testing. It speaks to your bot over its native channel (REST, WebSocket, Dialogflow, Rasa, etc.) and lets you write tests as conversation scripts.

Install the core and a connector:

npm install --save-dev botium-core botium-connector-rasa

A basic Botium conversation test script (.convo format):

#me
I want to book a flight

#bot
Where would you like to fly from?

#me
From New York

#bot
And where are you flying to?

#me
London

#bot
What date would you like to travel?

#me
Next Friday

#bot
Got it. New York to London on Friday, May 31st. Shall I confirm this booking?

#me
Yes please

#bot
Your flight has been booked!

That covers the happy path. Now test the cancel transition mid-flow:

#me
Book a flight

#bot
Where would you like to fly from?

#me
Actually cancel that

#bot
No problem. Is there anything else I can help you with?

For programmatic test generation in JavaScript, Botium exposes a full API:

const { BotDriver } = require('botium-core');

async function testSlotFillingInterruption() {
  const driver = new BotDriver();
  const container = await driver.Build();
  await container.Start();

  const msg = await container.pluginInstance;

  // Start flow
  await msg.UserSays('I need to book a flight');
  const step1 = await msg.BotSays();
  console.assert(
    step1.messageText.includes('fly from'),
    `Expected origin prompt, got: ${step1.messageText}`
  );

  // Provide origin
  await msg.UserSays('From Chicago');
  const step2 = await msg.BotSays();
  console.assert(
    step2.messageText.includes('flying to'),
    `Expected destination prompt, got: ${step2.messageText}`
  );

  // Inject out-of-scope utterance — bot should clarify, not crash
  await msg.UserSays('What is the weather like?');
  const step3 = await msg.BotSays();
  console.assert(
    step3.messageText.includes('destination') ||
    step3.messageText.includes('still booking'),
    `Expected context preservation, got: ${step3.messageText}`
  );

  await container.Stop();
}

testSlotFillingInterruption().catch(console.error);

Utterance Variation Testing

One of Botium's most powerful features is utterance expansion. Instead of hardcoding "From New York", you define an utterances file:

CITY_UTTERANCES
From New York
New York
NYC
Flying from New York
I'll be leaving from New York
New York City

Then reference it in the conversation script:

#me
UTT_CITY_UTTERANCES

#bot
And where are you flying to?

Botium runs the test once per utterance variant. This gives you NLU coverage across paraphrase diversity without writing dozens of test files by hand.

Testing with Rasa Test (Python)

Rasa ships with a built-in test runner (rasa test) that evaluates your stories and NLU pipeline against a test set. The key artifact is the test stories file.

A test story in Rasa YAML format:

# tests/test_stories.yml
version: "3.1"

stories:
  - story: happy path - full booking
    steps:
      - user: |
          I want to book a flight
        intent: book_flight
      - action: utter_ask_origin
      - user: |
          From New York
        intent: inform
        entities:
          - city: "New York"
      - action: utter_ask_destination
      - user: |
          To London
        intent: inform
        entities:
          - city: "London"
      - action: utter_ask_date
      - user: |
          Next Friday
        intent: inform
        entities:
          - date: "2026-05-31"
      - action: utter_confirm_booking
      - user: |
          Yes
        intent: affirm
      - action: action_create_booking
      - action: utter_booking_confirmed

  - story: user cancels mid-flow
    steps:
      - user: |
          Book me a flight
        intent: book_flight
      - action: utter_ask_origin
      - user: |
          Never mind, cancel
        intent: cancel
      - action: action_cancel_booking
      - action: utter_cancelled

Run the tests:

rasa test --stories tests/test_stories.yml --out results/

Rasa produces a failed_test_stories.yml file and a confusion matrix for NLU. Integrate this into CI:

# scripts/check_rasa_results.py
import json
import sys

with open('results/story_report.json') as f:
    report = json.load(f)

failed = report.get('failed_stories', [])
if failed:
    print(f"FAILED: {len(failed)} stories failed")
    for story in failed:
        print(f"  - {story['name']}: {story['failure_reason']}")
    sys.exit(1)

print(f"All {report['total_stories']} stories passed.")

Testing Slot Filling and Form Validation

Rasa's form abstraction handles multi-turn slot collection. Test that the form loops correctly when slots are invalid:

stories:
  - story: form - invalid date rejected
    steps:
      - user: |
          Book a flight from Paris to Berlin
        intent: book_flight
        entities:
          - city: "Paris"
            role: "origin"
          - city: "Berlin"
            role: "destination"
      - action: booking_form
      - active_loop: booking_form
      - user: |
          Yesterday
        intent: inform
        entities:
          - date: "2026-05-28"
      - action: utter_invalid_date
      - action: booking_form
      - active_loop: booking_form
      - user: |
          June 5th
        intent: inform
        entities:
          - date: "2026-06-05"
      - action: booking_form
      - active_loop: null
      - action: utter_confirm_booking

This test verifies that the form does not accept a past date, re-prompts, and only exits the loop when a valid future date is provided. Testing this path manually is tedious; automated story tests make it repeatable.

Context Carryover Testing

Context carryover is where many bots break. The user provides information in turn 3 and references it in turn 7 — does the bot still have it?

Test this explicitly in Python using the Rasa REST channel:

import requests

BASE_URL = "http://localhost:5005"
SENDER_ID = "test-session-context-001"

def send(text):
    resp = requests.post(
        f"{BASE_URL}/webhooks/rest/webhook",
        json={"sender": SENDER_ID, "message": text}
    )
    resp.raise_for_status()
    return [m["text"] for m in resp.json()]

# Turn 1: establish name
replies = send("My name is Alex")
assert any("Alex" in r or "hello" in r.lower() for r in replies), \
    f"Bot did not acknowledge name. Got: {replies}"

# Turn 2: pivot to task
replies = send("I need help with my order")
assert replies, "Bot did not respond to order query"

# Turn 3: reference earlier context — bot should remember name
replies = send("Can you check the status?")
# Bot should use the name in its response OR have the slot set
tracker = requests.get(f"{BASE_URL}/conversations/{SENDER_ID}/tracker").json()
name_slot = tracker["slots"].get("user_name")
assert name_slot == "Alex", \
    f"Context carryover failed: expected slot user_name='Alex', got '{name_slot}'"

print("Context carryover test passed.")

This pattern — asserting the tracker state directly — is far more reliable than parsing bot text for the presence of a name.

Fallback and Escalation Testing

Fallback paths are the most under-tested part of any conversational system. They activate when NLU confidence is below threshold or when the user has been in a loop too long. Test them deliberately.

Botium — low-confidence fallback:

#me
qwerty asdf random nonsense

#bot BUTTON
Sorry, I didn't catch that. Could you rephrase?

The #bot BUTTON assertion verifies the bot offered quick-reply buttons (a common fallback UX pattern). Without this, low-confidence inputs drop the user into a dead end.

Rasa — repeated fallback triggers escalation:

stories:
  - story: two fallbacks trigger escalation
    steps:
      - user: |
          aaabbbccc
        intent: nlu_fallback
      - action: action_default_fallback
      - user: |
          I don't understand what to do
        intent: nlu_fallback
      - action: utter_offer_escalation
      - user: |
          Yes connect me to an agent
        intent: affirm
      - action: action_escalate_to_human
      - action: utter_escalation_confirmed

JavaScript — test the escalation API call fires:

const nock = require('nock');
const { BotDriver } = require('botium-core');

async function testEscalationHandoff() {
  // Mock the human handoff endpoint
  const handoffCall = nock('https://crm.example.com')
    .post('/api/escalations')
    .reply(200, { ticket_id: 'TKT-9001' });

  const driver = new BotDriver();
  const container = await driver.Build();
  await container.Start();
  const msg = await container.pluginInstance;

  await msg.UserSays('zzzzz gibberish');
  await msg.BotSays(); // fallback 1

  await msg.UserSays('help me I am stuck');
  const offer = await msg.BotSays();
  console.assert(
    offer.messageText.toLowerCase().includes('agent') ||
    offer.messageText.toLowerCase().includes('human'),
    `Expected escalation offer, got: ${offer.messageText}`
  );

  await msg.UserSays('Yes please');
  await msg.BotSays();

  console.assert(handoffCall.isDone(), 'Escalation API was not called');
  await container.Stop();
}

testEscalationHandoff().catch(console.error);

Generating Test Cases from Flow Diagrams

If your team uses a visual dialog design tool (Voiceflow, Botmock, Lucidchart), export the flow as JSON and generate Botium or Rasa stories programmatically. This eliminates the gap between design and test.

import json
import yaml

def flow_to_rasa_stories(flow_json_path: str, output_path: str):
    with open(flow_json_path) as f:
        flow = json.load(f)

    stories = []
    for path in enumerate_paths(flow):
        steps = []
        for node in path:
            if node["type"] == "user":
                steps.append({
                    "user": node["sample_utterance"],
                    "intent": node["intent"]
                })
                if node.get("entities"):
                    steps[-1]["entities"] = node["entities"]
            elif node["type"] == "bot":
                steps.append({"action": node["action"]})
        stories.append({"story": path_name(path), "steps": steps})

    with open(output_path, "w") as f:
        yaml.dump({"version": "3.1", "stories": stories}, f, allow_unicode=True)

def enumerate_paths(flow: dict) -> list:
    """DFS traversal of the flow graph, returns all root-to-leaf paths."""
    paths = []
    def dfs(node_id, current_path):
        node = flow["nodes"][node_id]
        current_path = current_path + [node]
        children = flow["edges"].get(node_id, [])
        if not children:
            paths.append(current_path)
        for child_id in children:
            dfs(child_id, current_path)
    dfs(flow["start_node"], [])
    return paths

def path_name(path: list) -> str:
    intents = [n["intent"] for n in path if n.get("intent")]
    return " → ".join(intents) if intents else "unnamed path"

Run this in your CI pipeline against the exported flow JSON, and every new branch in the dialog graph automatically becomes a test case.

End-to-End Dialog Simulation

Unit-level story tests catch individual flow failures. End-to-end simulation catches integration failures — the NLU model misclassifying an entity, an action server returning unexpected data, a database lookup timing out mid-form.

Structure E2E tests around user personas with realistic, messy inputs:

# tests/e2e_personas.yml
stories:
  - story: e2e - power user, fast input, abbreviations
    steps:
      - user: |
          book nyc to lhr fri
        intent: book_flight
        entities:
          - city: "New York"
            role: origin
          - city: "London Heathrow"
            role: destination
          - date: "2026-05-31"
      - action: utter_confirm_booking
      - user: |
          yep
        intent: affirm
      - action: action_create_booking
      - action: utter_booking_confirmed

  - story: e2e - hesitant user, multiple corrections
    steps:
      - user: |
          I think I want to book a flight maybe
        intent: book_flight
      - action: utter_ask_origin
      - user: |
          Wait how much does it cost?
        intent: ask_price
      - action: utter_price_info
      - action: utter_ask_origin
      - user: |
          OK from Dallas
        intent: inform
        entities:
          - city: "Dallas"
      - action: utter_ask_destination

Maintain a set of persona-based E2E tests and run them nightly against your staging environment. They are slower than unit tests but catch the class of bugs that only emerge when the full pipeline is connected.

CI Integration

Put all of this in your pipeline:

# .github/workflows/bot-tests.yml
name: Conversational Flow Tests

on: [push, pull_request]

jobs:
  rasa-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Set up Python
        uses: actions/setup-python@v4
        with:
          python-version: "3.10"
      - name: Install dependencies
        run: pip install rasa
      - name: Train model
        run: rasa train --fixed-model-name ci-model
      - name: Run story tests
        run: rasa test --stories tests/ --model models/ci-model.tar.gz --out results/
      - name: Check results
        run: python scripts/check_rasa_results.py
      - name: Upload results
        uses: actions/upload-artifact@v3
        if: always()
        with:
          name: rasa-test-results
          path: results/

  botium-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install Node
        uses: actions/setup-node@v3
        with:
          node-version: "18"
      - run: npm ci
      - run: npx botium-cli run --convos tests/convos/

What to Measure

Track these metrics per release:

  • Flow completion rate — percentage of test conversations that reach a terminal success state
  • Fallback rate — how often the NLU fires the fallback intent in test runs
  • Slot error rate — percentage of forms where at least one slot was filled incorrectly
  • Context drop rate — percentage of multi-turn tests where a previously set slot was lost
  • Escalation rate — how often tests that should self-serve reach the human handoff action

A regression in any of these metrics between releases is a signal that a specific flow or intent was broken. Attach these metrics to your CI run artifacts and diff them against the last green build.

Wrapping Up

Testing conversational flows is not optional — it is the difference between a bot that works in demos and one that works in production. The state machine model gives you a systematic way to enumerate paths. Botium and Rasa Test give you the tooling to automate them. Context carryover and fallback tests catch the failures that happy path coverage misses entirely.

Start with the flow diagram. Enumerate every transition. Write a test for each one. Then add persona-based E2E tests for the messy, realistic inputs your users will actually send. That's a complete coverage strategy for conversational AI.

Read more

Start now free