Testing

Testing Voice Assistants: Alexa Skills and Google Actions QA

HelpMeTest

HelpMeTest

— 7 min read
Testing Voice Assistants: Alexa Skills and Google Actions QA

Voice UI testing is conversation graph testing. Every intent is a node, every response is an edge. Test happy paths through the conversation tree, test slot validation for all entity types, test session persistence across turns, and test what happens when users say unexpected things at every state. Use ask-cli and Actions SDK for local testing, then run utterance regression suites before every release.

Voice interfaces have no visual affordances. Users can't see what's possible — they have to discover it by speaking. This makes the failure modes fundamentally different from GUI applications. A broken button is visible. A broken intent handler is invisible until a user runs into it and gives up. QA for voice UIs means exhaustively testing the conversation graph before users discover the gaps.

The Voice Testing Mental Model

A voice application is a state machine. Each state is a point in a conversation. Each user utterance is a transition trigger. Testing means:

  1. Verifying that utterances route to the correct intent
  2. Verifying that required information (slots/entities) is correctly extracted
  3. Verifying that the response is appropriate for the current state
  4. Verifying that the application moves to the correct next state
  5. Verifying that errors and unexpected inputs are handled at every state

Most voice QA focuses only on #1 and #3. Teams ship with no testing for slot edge cases (#2), broken session management (#4), or error handling in non-initial states (#5).

Alexa Skills Testing

Setup: ask-cli Local Testing

The Alexa Skills Kit CLI allows local testing without deploying to AWS Lambda:

# Install ask-cli
npm install -g ask-cli

# Initialize (first time)
ask configure

# Start local debugging session
ask run --debug

# Run unit tests
ask util run-tests

Unit Testing with ask-sdk-test

The ask-sdk-test library provides a testing framework that simulates Alexa's request/response cycle:

const { AlexaTest, IntentRequestBuilder, LaunchRequestBuilder } = require('ask-sdk-test');
const skillHandler = require('../lambda/index');

const alexaTest = new AlexaTest(skillHandler, {
    appId: 'amzn1.ask.skill.test-skill-id',
    userId: 'amzn1.ask.account.test-user',
    deviceId: 'amzn1.ask.device.test-device',
    locale: 'en-US'
});

describe('BookingIntent', () => {
    it('confirms booking when all slots provided', async () => {
        await alexaTest.test([
            {
                request: new IntentRequestBuilder('BookingIntent')
                    .withSlot('date', '2026-06-15')
                    .withSlot('time', '14:00')
                    .withSlot('partySize', '4')
                    .build(),
                says: /Booking confirmed for June 15th at 2 PM for 4 people/,
                repromptsNothing: true,
                shouldEndSession: true
            }
        ]);
    });
    
    it('elicits missing date slot', async () => {
        await alexaTest.test([
            {
                request: new IntentRequestBuilder('BookingIntent')
                    .withSlot('time', '14:00')
                    .withSlot('partySize', '2')
                    .build(),
                says: /What date would you like to book/,
                elicitsSlot: 'date',
                shouldEndSession: false
            }
        ]);
    });
});

Testing Multi-Turn Conversations

Multi-turn tests are the most critical and most commonly skipped. They validate that session attributes persist correctly and that follow-up intents receive the right context:

describe('Multi-turn order flow', () => {
    it('completes order across 3 turns', async () => {
        await alexaTest.test([
            // Turn 1: User starts order
            {
                request: new IntentRequestBuilder('StartOrderIntent').build(),
                says: /What would you like to order/,
                shouldEndSession: false,
                // Verify session attribute is set
                sessionAttributes: {
                    orderState: 'awaiting_item'
                }
            },
            // Turn 2: User provides item
            {
                request: new IntentRequestBuilder('SelectItemIntent')
                    .withSlot('item', 'pepperoni pizza')
                    .build(),
                says: /What size would you like/,
                shouldEndSession: false,
                sessionAttributes: {
                    orderState: 'awaiting_size',
                    selectedItem: 'pepperoni pizza'
                }
            },
            // Turn 3: User provides size and confirms
            {
                request: new IntentRequestBuilder('SelectSizeIntent')
                    .withSlot('size', 'large')
                    .build(),
                says: /Large pepperoni pizza added to your order/,
                shouldEndSession: false,
                sessionAttributes: {
                    orderState: 'item_added'
                }
            }
        ]);
    });
    
    it('handles AMAZON.CancelIntent at any turn', async () => {
        await alexaTest.test([
            {
                request: new IntentRequestBuilder('StartOrderIntent').build(),
                says: /What would you like to order/,
                shouldEndSession: false
            },
            // Cancel mid-flow
            {
                request: new IntentRequestBuilder('AMAZON.CancelIntent').build(),
                says: /Order cancelled/,
                shouldEndSession: true,
                // Verify session is clean
                sessionAttributes: {}
            }
        ]);
    });
});

Slot Validation Testing

Slots have types (dates, numbers, custom enumerations, free-form strings). Each type needs boundary testing:

describe('DateSlot validation', () => {
    const dateTestCases = [
        { input: '2026-06-15', valid: true, description: 'explicit date' },
        { input: '2026-W24', valid: true, description: 'week reference' },
        { input: 'PRESENT_REF', valid: false, description: 'today reference' },
        { input: '????-06-15', valid: false, description: 'year unknown' },
    ];
    
    dateTestCases.forEach(({ input, valid, description }) => {
        it(`handles ${description}: ${input}`, async () => {
            await alexaTest.test([{
                request: new IntentRequestBuilder('BookingIntent')
                    .withSlot('date', input)
                    .build(),
                says: valid 
                    ? /confirmed/ 
                    : /What date did you have in mind/,
                shouldEndSession: !valid
            }]);
        });
    });
});

Utterance Testing

Your interaction model defines which phrases map to which intents. Test that your sample utterances correctly route to their intended intents:

# Use ask-cli to test utterance routing
ask dialog --locale en-US --skill-id amzn1.ask.skill.your-skill

# Or use the NLU evaluation tool
ask smapi invoke-skill-end-point \
  --skill-id amzn1.ask.skill.your-skill \
  --endpoint-region default \
  --payload '{
    "version": "1.0",
    "session": {"new": true},
    "request": {
      "type": "IntentRequest",
      "intent": {
        "name": "BookingIntent",
        "slots": {}
      }
    }
  }'

For bulk utterance testing, maintain a CSV of utterances and expected intents, then run them through the Alexa NLU:

// bulk-utterance-test.js
const { NLUTest } = require('./test-utils');

const utteranceTests = [
    { utterance: "book a table for two", expectedIntent: "BookingIntent" },
    { utterance: "make a reservation", expectedIntent: "BookingIntent" },
    { utterance: "cancel my booking", expectedIntent: "CancelBookingIntent" },
    { utterance: "what time do you open", expectedIntent: "HoursIntent" },
    { utterance: "goodbye", expectedIntent: "AMAZON.StopIntent" },
];

async function runUtteranceTests() {
    const results = await Promise.all(
        utteranceTests.map(async ({ utterance, expectedIntent }) => {
            const actual = await NLUTest.classifyUtterance(utterance, 'en-US');
            return {
                utterance,
                expectedIntent,
                actualIntent: actual.intent,
                passed: actual.intent === expectedIntent
            };
        })
    );
    
    const failures = results.filter(r => !r.passed);
    if (failures.length > 0) {
        console.error('Intent routing failures:');
        failures.forEach(f => {
            console.error(`  "${f.utterance}" → ${f.actualIntent} (expected ${f.expectedIntent})`);
        });
        process.exit(1);
    }
    console.log(`All ${results.length} utterance tests passed`);
}

Google Actions Testing

Setup: Actions SDK and Local Testing

# Install gactions CLI
npm install -g @assistant/cli

# Initialize project
gactions init

# Start local development server
gactions deploy preview

# Run local fulfillment
node index.js

Testing with @assistant/conversation-testing

const { ActionsOnGoogleTestManager } = require('@assistant/conversation-testing');

describe('Recipe Assistant', () => {
    let test;
    
    beforeEach(async () => {
        test = new ActionsOnGoogleTestManager({ projectId: 'your-project-id' });
        await test.setupSuite();
    });
    
    afterEach(async () => {
        await test.cleanupSuite();
    });
    
    it('handles recipe search intent', async () => {
        await test.sendQuery('find me a pasta recipe');
        test.assertSpeech(/Here's a recipe for/);
        test.assertScene('RecipeDisplay');
    });
    
    it('handles ingredient substitution request', async () => {
        await test.sendQuery('find me a pasta recipe');
        await test.sendQuery('what can I use instead of eggs');
        test.assertSpeech(/You can substitute eggs with/);
    });
    
    it('handles no-match gracefully', async () => {
        await test.sendQuery('play jazz music');  // out of scope
        test.assertSpeech(/I can help you find recipes/);  // graceful redirect
    });
});

Scene and Transition Testing

Google Actions uses a scene-based conversation model. Test that scene transitions happen correctly:

it('transitions from search to detail scene', async () => {
    await test.sendQuery('find chicken recipes');
    
    // Should be in RecipeList scene
    test.assertScene('RecipeList');
    test.assertSpeech(/I found \d+ chicken recipes/);
    
    // Select first result
    await test.sendQuery('tell me more about the first one');
    
    // Should transition to RecipeDetail
    test.assertScene('RecipeDetail');
    test.assertSpeech(/Here are the ingredients/);
    
    // Verify back navigation
    await test.sendQuery('go back');
    test.assertScene('RecipeList');
});

Webhook Fulfillment Testing

Your fulfillment webhook handles intent processing. Test it directly without the Actions SDK overhead:

// test/fulfillment.test.js
const { createConversation } = require('@assistant/conversation');
const app = require('../fulfillment');

function createMockRequest(intent, parameters = {}) {
    return {
        handler: { name: intent },
        intent: { name: intent, params: parameters },
        scene: { name: 'Main', slots: {} },
        session: { id: 'test-session', params: {} },
        user: { locale: 'en-US' },
        home: { params: {} },
        device: { capabilities: ['SPEECH', 'RICH_RESPONSE'] }
    };
}

describe('RecipeSearchHandler', () => {
    it('returns results for valid query', async () => {
        const mockReq = createMockRequest('recipe_search', {
            ingredient: { resolved: 'chicken', original: 'chicken' }
        });
        
        const mockRes = {
            json: jest.fn()
        };
        
        await app(mockReq, mockRes);
        
        const response = mockRes.json.mock.calls[0][0];
        expect(response.prompt.override).toBe(false);
        expect(response.prompt.firstSimple.speech).toMatch(/chicken/i);
    });
    
    it('handles empty results gracefully', async () => {
        const mockReq = createMockRequest('recipe_search', {
            ingredient: { resolved: 'unobtainium', original: 'unobtainium' }
        });
        
        const mockRes = { json: jest.fn() };
        await app(mockReq, mockRes);
        
        const response = mockRes.json.mock.calls[0][0];
        expect(response.prompt.firstSimple.speech).toMatch(/couldn't find/i);
    });
});

Session Management Testing

Session state is the most common source of subtle voice bugs. Test:

Session Persistence

describe('Session attribute persistence', () => {
    it('maintains cart across turns', async () => {
        const session = new TestSession('en-US');
        
        // Add first item
        await session.send('add milk to my cart');
        expect(session.attributes.cart).toContain('milk');
        
        // Add second item
        await session.send('also add bread');
        expect(session.attributes.cart).toContain('milk');
        expect(session.attributes.cart).toContain('bread');
        expect(session.attributes.cart).toHaveLength(2);
    });
    
    it('clears session on AMAZON.StopIntent', async () => {
        const session = new TestSession('en-US');
        
        await session.send('add milk to cart');
        await session.send('stop');
        
        expect(session.ended).toBe(true);
        // New session should have no cart
        const newSession = new TestSession('en-US');
        await newSession.send('what is in my cart');
        expect(newSession.lastResponse).toMatch(/your cart is empty/i);
    });
});

Timeout and Re-prompt Behavior

Most voice platforms send a SessionEndedRequest after 8-10 seconds of no user response. Test that your re-prompt fires correctly:

it('re-prompts when user does not respond', async () => {
    await alexaTest.test([
        {
            request: new IntentRequestBuilder('StartOrderIntent').build(),
            says: /What would you like to order/,
            reprompts: /I can help you place an order — what would you like/,
            shouldEndSession: false
        }
    ]);
});

it('ends session after re-prompt timeout', async () => {
    await alexaTest.test([
        {
            request: new SessionEndedRequestBuilder()
                .withReason('EXCEEDED_MAX_REPROMPTS')
                .build(),
            saysNothing: true,
            shouldEndSession: true
        }
    ]);
});

Regression Testing Strategy

Voice applications break in non-obvious ways after interaction model changes. Build a regression suite:

Utterance Regression Corpus

Maintain a versioned corpus of utterances with expected outcomes:

// test/corpus/regression-utterances.json
[
    {
        "id": "booking-001",
        "utterance": "book a table",
        "expectedIntent": "BookingIntent",
        "expectedSlots": {},
        "expectedElicits": "date",
        "addedVersion": "1.0.0",
        "failedVersions": []
    },
    {
        "id": "booking-002", 
        "utterance": "reserve a spot for saturday at 7",
        "expectedIntent": "BookingIntent",
        "expectedSlots": {
            "date": "AMAZON.DATE",
            "time": "AMAZON.TIME"
        },
        "addedVersion": "1.2.0",
        "failedVersions": []
    }
]

Run this corpus on every interaction model change. Track which version broke which utterance. Protect against NLU regressions as aggressively as code regressions.

CI Integration

# .github/workflows/voice-tests.yml
name: Voice Skill Tests

on: [push, pull_request]

jobs:
  unit-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - run: npm test
      
  utterance-regression:
    needs: unit-tests
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: node test/run-utterance-regression.js
        env:
          ASK_CLI_CLIENT_ID: ${{ secrets.ASK_CLI_CLIENT_ID }}
          ASK_CLI_CLIENT_SECRET: ${{ secrets.ASK_CLI_CLIENT_SECRET }}
          SKILL_ID: ${{ secrets.SKILL_ID }}
      - name: Archive regression results
        uses: actions/upload-artifact@v4
        with:
          name: utterance-regression-${{ github.sha }}
          path: test/results/

Error Handling Test Scenarios

Every intent handler must have error handling tests. Common scenarios:

describe('Error handling', () => {
    it('handles API timeout gracefully', async () => {
        jest.spyOn(externalApi, 'search').mockRejectedValue(new Error('TIMEOUT'));
        
        await alexaTest.test([{
            request: new IntentRequestBuilder('SearchIntent')
                .withSlot('query', 'pizza')
                .build(),
            says: /I'm having trouble connecting right now/,
            shouldEndSession: false
        }]);
    });
    
    it('handles unrecognized custom slot value', async () => {
        await alexaTest.test([{
            request: new IntentRequestBuilder('LocationIntent')
                .withSlot('city', 'ZZZZNOTACITY')
                .build(),
            says: /I didn't recognize that location/,
            elicitsSlot: 'city',
            shouldEndSession: false
        }]);
    });
    
    it('handles AMAZON.FallbackIntent', async () => {
        await alexaTest.test([{
            request: new IntentRequestBuilder('AMAZON.FallbackIntent').build(),
            says: /I'm not sure I understood that/,
            reprompts: /Can you try rephrasing/,
            shouldEndSession: false
        }]);
    });
});

Summary

Voice UI QA is conversation graph validation. Map out every state your application can be in, every valid utterance at that state, and every error condition. Test multi-turn flows explicitly — they're where most production bugs hide. Build an utterance regression corpus and run it on every interaction model change. The investment in a thorough regression suite compounds: each new feature adds utterances to the corpus, and the corpus protects every previous feature from silent breakage.

Read more

Start now free