AI Testing

Amazon Q Developer Testing Workflow: AWS-Specific Test Generation and Validation

HelpMeTest

HelpMeTest

— 6 min read
Amazon Q Developer Testing Workflow: AWS-Specific Test Generation and Validation

Amazon Q Developer is AWS's AI coding assistant, and it has a specific advantage over general-purpose tools like Copilot: it knows AWS. It understands IAM policies, CloudFormation templates, CDK constructs, Lambda handlers, and the AWS SDK. This makes it genuinely useful for AWS-heavy projects — and it makes validating its output more nuanced than just running unit tests.

This guide covers how to test Q Developer's output across the areas where it's most useful: Lambda functions, CDK infrastructure, and IAM configuration.

Why AWS-Specific Testing Matters

Generic coding assistants generate syntactically valid code. Amazon Q generates syntactically valid code that also has AWS-specific failure modes:

  • IAM policies that are too permissive — Q might generate "Action": "*" where you need specific actions
  • Lambda handlers missing error handling — Functions that throw unhandled exceptions produce opaque error messages in CloudWatch
  • CDK constructs with insecure defaults — S3 buckets without versioning, security groups with 0.0.0.0/0 ingress
  • DynamoDB queries missing error paths — ConditionalCheckFailedException not handled
  • Missing retry logic — AWS services are eventually consistent; Q often generates happy-path code

Testing Q-generated code means testing against these AWS-specific failure modes, not just general correctness.

Setting Up the Test Environment

For testing Q Developer output locally, you need:

# AWS SAM CLI for Lambda testing
pip install aws-sam-cli

# LocalStack for local AWS service simulation
docker run -d -p 4566:4566 localstack/localstack

# AWS CDK
npm install -g aws-cdk

# Configure AWS CLI for LocalStack
aws configure set aws_access_key_id test
aws configure set aws_secret_access_key test
aws configure set region us-east-1
export AWS_ENDPOINT_URL=http://localhost:4566

Create a test configuration that points at LocalStack:

// test/aws-config.js
const { DynamoDBClient } = require('@aws-sdk/client-dynamodb');

const client = new DynamoDBClient({
  endpoint: process.env.AWS_ENDPOINT_URL || 'http://localhost:4566',
  region: 'us-east-1',
  credentials: {
    accessKeyId: 'test',
    secretAccessKey: 'test',
  },
});

module.exports = { client };

Testing Q Developer Lambda Functions

Q Developer is particularly good at generating Lambda handlers. The output is usually valid — but it needs validation against real AWS behavior.

Example: Q generates this Lambda handler for processing SQS messages:

// Q Developer generated handler
exports.handler = async (event) => {
  const results = [];

  for (const record of event.Records) {
    const body = JSON.parse(record.body);

    await processMessage(body);
    results.push({ messageId: record.messageId, status: 'success' });
  }

  return { batchItemFailures: [] };
};

This looks reasonable but has issues. Write tests that catch them:

const { handler } = require('./handler');

describe('SQS handler - Q Developer generated', () => {
  // Test 1: Basic happy path
  it('processes valid messages', async () => {
    const event = {
      Records: [{
        messageId: 'msg-001',
        body: JSON.stringify({ userId: '123', action: 'process' }),
      }],
    };

    const result = await handler(event);
    expect(result.batchItemFailures).toHaveLength(0);
  });

  // Test 2: Individual message failure — Q's version fails the entire batch
  it('returns failed message ID on partial failure', async () => {
    // Make processMessage fail for the second message
    jest.spyOn(global, 'processMessage')
      .mockResolvedValueOnce(undefined)  // First succeeds
      .mockRejectedValueOnce(new Error('Processing failed'));  // Second fails

    const event = {
      Records: [
        { messageId: 'msg-001', body: JSON.stringify({ userId: '1' }) },
        { messageId: 'msg-002', body: JSON.stringify({ userId: '2' }) },
      ],
    };

    const result = await handler(event);

    // Q's generated code returns empty batchItemFailures even on failure
    // Correct behavior: failed messages should be in batchItemFailures
    expect(result.batchItemFailures).toContainEqual({
      itemIdentifier: 'msg-002',
    });
  });

  // Test 3: Malformed JSON in message body
  it('handles malformed JSON without crashing the batch', async () => {
    const event = {
      Records: [{
        messageId: 'msg-bad',
        body: 'not-valid-json{',
      }],
    };

    // Q's handler throws here: JSON.parse throws, fails entire Lambda invocation
    // Correct behavior: return this messageId in batchItemFailures
    const result = await handler(event);
    expect(result.batchItemFailures).toContainEqual({
      itemIdentifier: 'msg-bad',
    });
  });
});

These tests reveal that Q's generated handler has the partial batch failure bug. The correct implementation:

exports.handler = async (event) => {
  const batchItemFailures = [];

  for (const record of event.Records) {
    try {
      const body = JSON.parse(record.body);
      await processMessage(body);
    } catch (err) {
      console.error(`Failed to process message ${record.messageId}:`, err);
      batchItemFailures.push({ itemIdentifier: record.messageId });
    }
  }

  return { batchItemFailures };
};

Testing Q Developer CDK Infrastructure Code

Q Developer generates CDK constructs, and these need a different kind of testing: infrastructure assertions.

// Q Developer generated S3 bucket construct
import * as s3 from 'aws-cdk-lib/aws-s3';

const dataBucket = new s3.Bucket(this, 'DataBucket', {
  bucketName: 'my-app-data',
  versioned: true,
});

Test the CDK construct with assertions:

import { App, Stack } from 'aws-cdk-lib';
import { Template } from 'aws-cdk-lib/assertions';
import { DataStack } from '../lib/data-stack';

describe('DataStack CDK construct - Q Developer generated', () => {
  let template: Template;

  beforeAll(() => {
    const app = new App();
    const stack = new DataStack(app, 'TestStack');
    template = Template.fromStack(stack);
  });

  // Test 1: Bucket is encrypted
  it('S3 bucket has server-side encryption', () => {
    template.hasResourceProperties('AWS::S3::Bucket', {
      BucketEncryption: {
        ServerSideEncryptionConfiguration: [{
          ServerSideEncryptionByDefault: {
            SSEAlgorithm: 'aws:kms',
          },
        }],
      },
    });
  });

  // Test 2: Public access is blocked
  it('S3 bucket blocks all public access', () => {
    template.hasResourceProperties('AWS::S3::Bucket', {
      PublicAccessBlockConfiguration: {
        BlockPublicAcls: true,
        BlockPublicPolicy: true,
        IgnorePublicAcls: true,
        RestrictPublicBuckets: true,
      },
    });
  });

  // Test 3: Versioning is enabled
  it('S3 bucket has versioning enabled', () => {
    template.hasResourceProperties('AWS::S3::Bucket', {
      VersioningConfiguration: {
        Status: 'Enabled',
      },
    });
  });

  // Test 4: Q might have missed lifecycle rules
  it('S3 bucket has lifecycle rules for cost management', () => {
    template.hasResourceProperties('AWS::S3::Bucket', {
      LifecycleConfiguration: {
        Rules: expect.arrayContaining([
          expect.objectContaining({ Status: 'Enabled' }),
        ]),
      },
    });
  });
});

Run CDK tests:

npx cdk synth
npx jest --testPathPattern="data-stack"

Q often generates CDK code without encryption or public access blocking on S3 buckets. These tests catch those omissions before they reach production.

Testing Q Developer IAM Policies

IAM is where Q Developer generates the most dangerous code. An overly permissive policy is a security incident waiting to happen.

Q might generate:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": "s3:*",
    "Resource": "*"
  }]
}

Test IAM policies with the principle of least privilege:

# test_iam_policy.py
import json
import boto3

def test_lambda_execution_role_not_overly_permissive():
    """Verify Q-generated IAM policy follows least privilege."""
    iam = boto3.client('iam', endpoint_url='http://localhost:4566')

    # Get the policy document
    policy = iam.get_policy_version(
        PolicyArn='arn:aws:iam::000000000000:policy/LambdaExecutionPolicy',
        VersionId='v1'
    )

    document = policy['PolicyVersion']['Document']

    for statement in document['Statement']:
        if statement['Effect'] == 'Allow':
            # Wildcard actions are dangerous
            actions = statement.get('Action', [])
            if isinstance(actions, str):
                actions = [actions]

            assert '*' not in actions, (
                f"Wildcard action found in statement: {statement}. "
                "This violates least privilege. Use specific actions."
            )

            # Wildcard resources require justification
            resources = statement.get('Resource', [])
            if isinstance(resources, str):
                resources = [resources]

            if '*' in resources:
                # Must be an action that legitimately needs resource-level wildcard
                allowed_wildcard_actions = ['s3:ListAllMyBuckets', 'cloudwatch:PutMetricData']
                unexpected = [a for a in actions if a not in allowed_wildcard_actions]
                assert not unexpected, (
                    f"Resource wildcard with unexpected actions: {unexpected}"
                )

def test_s3_bucket_policy_restricts_to_specific_prefix():
    """Verify Q-generated bucket policy restricts access by prefix."""
    # ...

Testing Q Developer's Test Generation Feature

Q Developer has a built-in test generation feature (/test command in the IDE). Like Copilot, it generates tests that reflect the code, not the spec.

Validate Q-generated tests with mutation testing:

# Install mutation testing for Python (if you're using Python for Lambda)
pip install mutmut

# Run mutation testing on Q-generated tests
mutmut run --paths-to-mutate src/lambda/ --tests-dir tests/

# Check results
mutmut results

# See what mutations survived (these are gaps in Q's generated tests)
mutmut show

A mutation score below 65% means Q's generated tests aren't assertive enough. Common gaps in Q-generated tests:

  • Tests that don't assert on error paths
  • Tests that use assertIsNotNone instead of checking specific values
  • Tests that mock the AWS SDK but don't verify the mock was called with correct parameters

AWS-Specific Integration Tests

For code that interacts with real AWS services, use LocalStack for integration testing:

# test_dynamo_integration.py
import pytest
import boto3
from moto import mock_dynamodb  # Alternative to LocalStack

@mock_dynamodb
def test_q_generated_create_user_stores_correctly():
    """Test that Q Developer's DynamoDB code stores and retrieves correctly."""
    # Create table
    dynamodb = boto3.resource('dynamodb', region_name='us-east-1')
    table = dynamodb.create_table(
        TableName='Users',
        KeySchema=[
            {'AttributeName': 'userId', 'KeyType': 'HASH'},
        ],
        AttributeDefinitions=[
            {'AttributeName': 'userId', 'AttributeType': 'S'},
        ],
        BillingMode='PAY_PER_REQUEST',
    )

    # Test Q-generated function
    from src.users import create_user, get_user

    create_user({'userId': 'u001', 'email': 'test@example.com', 'role': 'admin'})

    result = get_user('u001')

    assert result['userId'] == 'u001'
    assert result['email'] == 'test@example.com'
    assert result['role'] == 'admin'

    # Q often forgets to store timestamps
    assert 'createdAt' in result, "Q forgot to add createdAt timestamp"

@mock_dynamodb
def test_q_generated_handles_conditional_check_failure():
    """Test that Q's code handles DynamoDB conditional check failures."""
    from src.users import create_user
    from botocore.exceptions import ClientError

    # Create user first
    create_user({'userId': 'u001', 'email': 'first@example.com'})

    # Q often doesn't handle ConditionalCheckFailedException
    with pytest.raises((ClientError, ValueError)) as exc_info:
        create_user({'userId': 'u001', 'email': 'second@example.com'})

    # Verify the error is handled gracefully, not propagated raw
    assert 'already exists' in str(exc_info.value).lower(), (
        "Q's error handling doesn't produce useful error messages"
    )

Running the Full Validation Workflow

# 1. Validate generated Lambda handlers
sam local invoke -e events/sqs-event.json LambdaFunction

# 2. Run unit tests
pytest tests/unit/ -v

# 3. Run integration tests with LocalStack
docker start localstack  # or start fresh
pytest tests/integration/ -v

# 4. CDK synth and assertions
npx cdk synth
npx jest tests/cdk/

# 5. Mutation testing
mutmut run
mutmut results

# 6. Security checks on IAM
python tests/security/test_iam_policies.py

Summary: What to Test in Q Developer Output

Code type What Q gets right What to test
Lambda handlers Basic structure, SDK usage Partial batch failures, error handling, cold start
CDK constructs Resource creation Encryption, public access blocking, lifecycle rules
IAM policies Correct JSON structure Least privilege, resource scoping, wildcard actions
DynamoDB Basic CRUD Condition expressions, error handling, attribute validation
SQS/SNS Message format Dead letter queues, retry logic, visibility timeout

Q Developer accelerates AWS development significantly. The test workflow above ensures that acceleration doesn't come with hidden correctness or security costs.

HelpMeTest integrates with CI pipelines to run behavioral tests against deployed AWS applications. Use it to verify Q Developer output works end-to-end in your actual infrastructure. Start free →

Read more

Start now free