AI Testing

How to Test Google Genkit AI Applications

HelpMeTest

HelpMeTest

— 6 min read
How to Test Google Genkit AI Applications

Google Genkit is an open-source framework for building AI-powered applications in TypeScript and Go. It provides typed flows, reusable prompts, tool definitions, and observability built-in. The testing story is practical — Genkit flows are just async functions with well-defined inputs and outputs, and the framework provides test utilities that let you run flows without calling a real model.

This guide covers unit testing Genkit flows with mocked models, testing tool implementations, running Genkit evaluations, and integration testing against live models.

Genkit Fundamentals for Testing

A Genkit application has three main testable units:

// Flows — typed async functions with AI calls inside
const summarizeFlow = ai.defineFlow(
  { name: "summarize", inputSchema: z.string(), outputSchema: z.string() },
  async (text) => { /* ... */ }
);

// Tools — functions the model can call
const getWeatherTool = ai.defineTool(
  { name: "getWeather", inputSchema: z.object({ city: z.string() }) },
  async ({ city }) => { /* call weather API */ }
);

// Prompts — reusable prompt templates
const summaryPrompt = ai.definePrompt(
  { name: "summary", inputSchema: z.string() },
  "Summarize the following text in 2 sentences: {{input}}"
);

Each is independently testable, which is Genkit's main advantage for testing.

Project Setup

npm install genkit @genkit-ai/googleai
npm install -D vitest

Basic Genkit initialization for tests:

// src/ai/genkit.ts
import { genkit } from "genkit";
import { googleAI } from "@genkit-ai/googleai";

export const ai = genkit({
  plugins: [googleAI()],
  model: "googleai/gemini-2.0-flash",
});

Unit Testing Flows with Mocked Models

Genkit's runFlow can be used in tests after you configure a test model that returns deterministic responses:

// src/ai/flows/summarize.ts
import { z } from "zod";
import { ai } from "../genkit";

export const summarizeFlow = ai.defineFlow(
  {
    name: "summarizeFlow",
    inputSchema: z.string(),
    outputSchema: z.string(),
  },
  async (text) => {
    const { text: summary } = await ai.generate({
      prompt: `Summarize in 2 sentences: ${text}`,
    });
    return summary;
  }
);

Test with a mock model plugin:

// src/ai/flows/__tests__/summarize.test.ts
import { describe, it, expect } from "vitest";
import { genkit } from "genkit";
import { z } from "zod";

// Create a test instance with a mock model
function createTestAi(mockResponse: string) {
  return genkit({
    plugins: [],
    model: {
      name: "test-model",
      generate: async () => ({
        candidates: [
          {
            index: 0,
            finishReason: "stop",
            message: {
              role: "model",
              content: [{ text: mockResponse }],
            },
          },
        ],
        usage: { inputTokens: 10, outputTokens: 5 },
      }),
    },
  });
}

describe("summarizeFlow", () => {
  it("returns the model output as the flow result", async () => {
    const testAi = createTestAi("This is a two-sentence summary. It covers the main points.");

    const summarizeFlowUnderTest = testAi.defineFlow(
      {
        name: "summarizeFlow",
        inputSchema: z.string(),
        outputSchema: z.string(),
      },
      async (text) => {
        const { text: summary } = await testAi.generate({
          prompt: `Summarize in 2 sentences: ${text}`,
        });
        return summary;
      }
    );

    const result = await summarizeFlowUnderTest("Long article text here...");

    expect(result).toBe(
      "This is a two-sentence summary. It covers the main points."
    );
  });
});

A Better Pattern: Inject the AI Instance

For easier testing, inject the ai instance instead of importing it directly:

// src/ai/flows/summarize.ts
import { Genkit, z } from "genkit";

export function createSummarizeFlow(ai: Genkit) {
  return ai.defineFlow(
    {
      name: "summarizeFlow",
      inputSchema: z.string(),
      outputSchema: z.string(),
    },
    async (text) => {
      const { text: summary } = await ai.generate({
        prompt: `Summarize in 2 sentences: ${text}`,
      });
      return summary;
    }
  );
}

Now tests can inject a test AI instance:

// src/ai/flows/__tests__/summarize.test.ts
import { describe, it, expect, vi } from "vitest";
import { createSummarizeFlow } from "../summarize";

function mockAi(responseText: string) {
  return {
    defineFlow: (config: any, handler: any) => handler,
    generate: vi.fn().mockResolvedValue({ text: responseText }),
  } as any;
}

describe("summarizeFlow handler", () => {
  it("calls generate with the correct prompt", async () => {
    const ai = mockAi("Short summary here.");
    const handler = createSummarizeFlow(ai);

    const result = await handler("This is the input text");

    expect(ai.generate).toHaveBeenCalledWith(
      expect.objectContaining({
        prompt: expect.stringContaining("This is the input text"),
      })
    );
    expect(result).toBe("Short summary here.");
  });

  it("propagates errors from the model", async () => {
    const ai = {
      defineFlow: (_: any, handler: any) => handler,
      generate: vi.fn().mockRejectedValue(new Error("Rate limit exceeded")),
    } as any;

    const handler = createSummarizeFlow(ai);
    await expect(handler("any input")).rejects.toThrow("Rate limit exceeded");
  });
});

Testing Tools

Tools are regular async functions — test them directly without involving Genkit:

// src/ai/tools/weather.ts
import { z } from "zod";
import { ai } from "../genkit";

export const weatherTool = ai.defineTool(
  {
    name: "getWeather",
    description: "Get current weather for a city",
    inputSchema: z.object({ city: z.string() }),
    outputSchema: z.object({
      temperature: z.number(),
      condition: z.string(),
    }),
  },
  async ({ city }) => {
    const response = await fetch(
      `https://api.weather.example.com/current?q=${city}`
    );
    if (!response.ok) throw new Error(`Weather API error: ${response.status}`);
    return response.json();
  }
);

Test the tool handler:

// src/ai/tools/__tests__/weather.test.ts
import { describe, it, expect, vi, beforeEach } from "vitest";

// Mock fetch
global.fetch = vi.fn();

describe("getWeather tool", () => {
  beforeEach(() => vi.clearAllMocks());

  it("returns temperature and condition for valid city", async () => {
    vi.mocked(fetch).mockResolvedValue({
      ok: true,
      json: () =>
        Promise.resolve({ temperature: 22, condition: "Sunny" }),
    } as Response);

    // Test the tool handler by extracting the handler function
    // In Genkit, defineTool returns an object with a .run() method
    const { weatherTool } = await import("../weather");
    const result = await weatherTool({ city: "London" });

    expect(result).toEqual({ temperature: 22, condition: "Sunny" });
    expect(fetch).toHaveBeenCalledWith(
      expect.stringContaining("q=London")
    );
  });

  it("throws when the weather API returns an error", async () => {
    vi.mocked(fetch).mockResolvedValue({
      ok: false,
      status: 503,
    } as Response);

    const { weatherTool } = await import("../weather");
    await expect(weatherTool({ city: "InvalidCity" })).rejects.toThrow(
      "Weather API error: 503"
    );
  });
});

Testing Flows That Use Tools

When a flow uses tools, test the tool invocation path:

// src/ai/flows/weatherFlow.ts
export function createWeatherFlow(ai: Genkit, weatherFn: (city: string) => Promise<WeatherData>) {
  return ai.defineFlow(
    { name: "weatherFlow", inputSchema: z.string(), outputSchema: z.string() },
    async (city) => {
      const weather = await weatherFn(city);
      const { text } = await ai.generate({
        prompt: `The weather in ${city} is ${weather.condition} at ${weather.temperature}°C. Write a friendly one-sentence summary.`,
      });
      return text;
    }
  );
}

Test:

describe("weatherFlow", () => {
  it("fetches weather and generates a summary", async () => {
    const mockWeatherFn = vi.fn().mockResolvedValue({
      temperature: 18,
      condition: "Cloudy",
    });
    const ai = mockAi("It's a cool, cloudy day in Paris today!");

    const handler = createWeatherFlow(ai, mockWeatherFn);
    const result = await handler("Paris");

    expect(mockWeatherFn).toHaveBeenCalledWith("Paris");
    expect(ai.generate).toHaveBeenCalledWith(
      expect.objectContaining({
        prompt: expect.stringContaining("Paris"),
      })
    );
    expect(result).toBe("It's a cool, cloudy day in Paris today!");
  });
});

Running Genkit Evaluations

Genkit has a built-in evaluation framework for measuring LLM output quality. Define datasets and metrics:

// src/ai/evals/summarize.eval.ts
import { ai } from "../genkit";
import { createSummarizeFlow } from "../flows/summarize";

const summarizeFlow = createSummarizeFlow(ai);

// Define the evaluation dataset
export const summarizeEvalDataset = [
  {
    input:
      "The Eiffel Tower was built in 1889 for the World's Fair in Paris. It stands 330 meters tall.",
    reference:
      "The Eiffel Tower, constructed in 1889 for the Paris World's Fair, is a 330-meter tall landmark.",
  },
  {
    input:
      "JavaScript was created in 10 days by Brendan Eich in 1995. It was originally called Mocha.",
    reference:
      "Brendan Eich created JavaScript in just 10 days in 1995, initially naming it Mocha.",
  },
];

// Run evaluation
export async function runSummarizeEval() {
  const results = await Promise.all(
    summarizeEvalDataset.map(async ({ input, reference }) => {
      const output = await summarizeFlow(input);
      return { input, reference, output };
    })
  );

  return results;
}

Add automated quality assertions:

// src/ai/evals/__tests__/summarize.eval.test.ts
import { describe, it, expect } from "vitest";
import { runSummarizeEval } from "../summarize.eval";

describe.skipIf(!process.env.RUN_EVALS)("Summarize flow evaluation", () => {
  it("produces summaries within reasonable length", async () => {
    const results = await runSummarizeEval();

    for (const { input, output } of results) {
      // A 2-sentence summary should be shorter than the input
      expect(output.length).toBeLessThan(input.length);
      // Should produce something non-empty
      expect(output.trim().length).toBeGreaterThan(10);
    }
  }, 60_000);
});

Testing with the Genkit Dev UI

During development, use the Genkit Dev UI to test flows interactively:

npx genkit start -- tsx src/index.ts

Open http://localhost:4000 to run flows manually, inspect traces, and view evaluation results. This complements automated tests — use it to debug why a flow returns unexpected output before writing a test to prevent regression.

Integration Testing Against Live Models

// src/ai/flows/__tests__/summarize.integration.test.ts
import { describe, it, expect } from "vitest";
import { ai } from "../../genkit";
import { createSummarizeFlow } from "../summarize";

const LIVE = process.env.GOOGLE_GENAI_API_KEY && process.env.RUN_LIVE_TESTS;

describe.skipIf(!LIVE)("summarizeFlow (live)", () => {
  it("produces a shorter summary than the input", async () => {
    const flow = createSummarizeFlow(ai);
    const input =
      "Machine learning is a subset of artificial intelligence that gives systems the ability to automatically learn and improve from experience without being explicitly programmed. It focuses on developing computer programs that can access data and use it to learn for themselves.";

    const summary = await flow(input);

    expect(summary.length).toBeLessThan(input.length);
    expect(summary.trim().length).toBeGreaterThan(20);
  }, 30_000);
});

What to Test

Layer What to Test How
Tools Input validation, external API calls, error handling Unit test the handler directly
Flow handlers Business logic, prompt construction, output parsing Inject mock AI instance
Flow integration Tool invocation, model call sequence Mock individual dependencies
Evaluations Output quality metrics Run against live model, gate on threshold
E2E Full user scenario through your app Playwright / integration test

Genkit's design makes the unit test layer genuinely useful because flows are just functions. Mock the generate call, test your business logic, and reserve live API calls for evaluation tests that run on a schedule. That balance gives you fast feedback in CI without burning API budget on every commit.

Read more

Start now free