CI/CD

Checkly in CI/CD: Monitoring as Code with the Checkly CLI

HelpMeTest

HelpMeTest

— 6 min read
Checkly in CI/CD: Monitoring as Code with the Checkly CLI

Checkly checks defined through the UI are fine for small setups. They break down when you have a team: no version control, no code review, no way to track what changed when, and manual effort to keep checks in sync with your application.

The Checkly CLI solves this. You define checks as TypeScript or JavaScript files, commit them to your repo, and deploy them with checkly deploy. This guide covers the full workflow: setting up the CLI, defining checks as code, integrating with GitHub Actions, and using checks as deployment gates.

Monitoring as Code: The Concept

The idea is simple: treat your synthetic monitors the same way you treat your application code.

  • Checks live in your repository alongside the code they test
  • Changes go through pull request review
  • CI deploys the checks when code merges
  • Before deploying to production, CI runs the checks against staging and blocks the deploy if they fail

This means a broken login flow that your checks catch in CI never reaches production. It also means when you change your login page, you update the check in the same PR — the two stay in sync by default.

Setting Up the Checkly CLI

Installation

npm install --save-dev checkly
# or
npx checkly --version  # Run without installing

Authentication

npx checkly login

This opens a browser window for OAuth. After authenticating, the CLI stores credentials in ~/.checkly/credentials.json.

For CI environments (no browser), use API key auth:

export CHECKLY_API_KEY=your_api_key
export CHECKLY_ACCOUNT_ID=your_account_id

Get both from Account Settings in the Checkly dashboard. Never commit these to your repo.

Initialize a Project

npx checkly init

This creates:

  • checkly.config.ts — project configuration
  • __checks__/ — directory for your check definitions

Project Structure

A typical Checkly-as-code project looks like this:

my-app/
├── src/                    # Application code
├── __checks__/
│   ├── api/
│   │   ├── health.check.ts
│   │   ├── auth.check.ts
│   │   └── products.check.ts
│   ├── browser/
│   │   ├── login.spec.ts
│   │   └── checkout.spec.ts
│   └── alert-channels.ts
├── checkly.config.ts
└── package.json

Checks live in __checks__/ by default (configurable). The naming convention *.check.ts for API checks and *.spec.ts for browser checks is recommended but not required.

checkly.config.ts

The project configuration file:

import { defineConfig } from 'checkly'
import { Frequency } from 'checkly/constructs'

export default defineConfig({
  projectName: 'My App Monitoring',
  logicalId: 'my-app-monitoring',
  repoUrl: 'https://github.com/my-org/my-app',
  checks: {
    // Default settings inherited by all checks
    activated: true,
    muted: false,
    runtimeId: '2024.02',
    frequency: Frequency.EVERY_5M,
    locations: ['us-east-1', 'eu-west-1'],
    tags: ['my-app'],
    alertChannels: [],  // Set globally or per check
    environmentVariables: [],
    checkMatch: '**/__checks__/**/*.check.ts',
    browserChecks: {
      frequency: Frequency.EVERY_10M,
      testMatch: '**/__checks__/**/*.spec.ts',
    },
  },
  cli: {
    runLocation: 'eu-west-1',
    reporters: ['list'],
  },
})

Defining API Checks

// __checks__/api/health.check.ts
import { ApiCheck, AssertionBuilder } from 'checkly/constructs'

new ApiCheck('health-check', {
  name: 'Health Endpoint',
  activated: true,
  frequency: 1, // Every 1 minute
  locations: ['us-east-1', 'eu-west-1', 'ap-southeast-1'],
  request: {
    method: 'GET',
    url: '{{BASE_URL}}/health',
    assertions: [
      AssertionBuilder.statusCode().equals(200),
      AssertionBuilder.responseTime().lessThan(2000),
      AssertionBuilder.jsonBody('$.status').equals('ok'),
    ],
  },
})
// __checks__/api/auth.check.ts
import { ApiCheck, AssertionBuilder } from 'checkly/constructs'

new ApiCheck('auth-token-check', {
  name: 'Auth - Token Exchange',
  frequency: 5,
  locations: ['us-east-1', 'eu-west-1'],
  request: {
    method: 'POST',
    url: '{{BASE_URL}}/api/auth/token',
    headers: [
      { key: 'Content-Type', value: 'application/json' },
    ],
    body: JSON.stringify({
      grant_type: 'client_credentials',
      client_id: '{{CLIENT_ID}}',
      client_secret: '{{CLIENT_SECRET}}',
    }),
    assertions: [
      AssertionBuilder.statusCode().equals(200),
      AssertionBuilder.responseTime().lessThan(1000),
      AssertionBuilder.jsonBody('$.access_token').isNotNull(),
      AssertionBuilder.jsonBody('$.expires_in').greaterThan(0),
    ],
  },
})

Defining Browser Checks

Browser checks reference Playwright spec files:

// __checks__/browser/login.spec.ts
import { test, expect } from '@playwright/test'

test('User can log in and reach dashboard', async ({ page }) => {
  await page.goto(process.env.BASE_URL + '/login')

  await page.fill('input[name="email"]', process.env.TEST_EMAIL!)
  await page.fill('input[name="password"]', process.env.TEST_PASSWORD!)
  await page.click('button[type="submit"]')

  await page.waitForURL('**/dashboard')
  await expect(page.locator('[data-testid="dashboard-header"]')).toBeVisible()
})
// __checks__/browser/checkout.spec.ts
import { test, expect } from '@playwright/test'

test('User can add item to cart and reach checkout', async ({ page }) => {
  // Assume user is logged in via stored auth state
  await page.goto(process.env.BASE_URL + '/products')

  // Add first product to cart
  await page.click('[data-testid="product-card"]:first-child [data-testid="add-to-cart"]')

  // Go to cart
  await page.click('[data-testid="cart-icon"]')
  await page.waitForURL('**/cart')

  // Verify item in cart
  await expect(page.locator('[data-testid="cart-item"]')).toHaveCount(1)

  // Proceed to checkout
  await page.click('[data-testid="checkout-button"]')
  await page.waitForURL('**/checkout')
  await expect(page.locator('[data-testid="order-summary"]')).toBeVisible()
})

Local Development Workflow

Before deploying, test your checks locally:

# Run all checks against your local environment
npx checkly test --env BASE_URL=http://localhost:3000

# Run a specific check
npx checkly test __checks__/api/health.check.ts

# Dry run: validate check definitions without running them
npx checkly test --dry-run

# Run from a specific location
npx checkly test --location eu-west-1

The test command runs checks once and reports results without affecting your live Checkly configuration. Use it during development to iterate on check logic.

Deploying Checks

Once checks are working locally:

# Deploy all checks to Checkly
npx checkly deploy

# Preview what would be deployed (no changes made)
npx checkly deploy --preview

# Force deploy even if there are conflicts
npx checkly deploy --force

checkly deploy is idempotent. Run it as many times as you want. Existing checks are updated, new ones are created, and checks not present in code are left untouched (they're not deleted automatically — use checkly destroy if you want to remove checks).

GitHub Actions Integration

Basic CI Workflow

Add Checkly credentials as GitHub Actions secrets: CHECKLY_API_KEY and CHECKLY_ACCOUNT_ID.

# .github/workflows/checkly.yml
name: Checkly Monitoring

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  deploy-checks:
    if: github.ref == 'refs/heads/main'
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - name: Deploy Checkly checks
        run: npx checkly deploy
        env:
          CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
          CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}

This deploys updated checks every time code merges to main.

Deployment Gate: Run Checks Before Promoting

The more powerful pattern: run your E2E checks against staging, and only promote to production if they pass.

# .github/workflows/deploy.yml
name: Deploy

on:
  push:
    branches: [main]

jobs:
  deploy-staging:
    runs-on: ubuntu-latest
    outputs:
      staging-url: ${{ steps.deploy.outputs.url }}
    steps:
      - uses: actions/checkout@v4
      - name: Deploy to staging
        id: deploy
        run: |
          # Your deploy command here
          echo "url=https://staging.myapp.com" >> $GITHUB_OUTPUT

  run-e2e-checks:
    needs: deploy-staging
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'
      - run: npm ci
      - name: Run Checkly E2E checks against staging
        run: npx checkly test --env BASE_URL=${{ needs.deploy-staging.outputs.staging-url }}
        env:
          CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
          CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
          TEST_EMAIL: ${{ secrets.TEST_EMAIL }}
          TEST_PASSWORD: ${{ secrets.TEST_PASSWORD }}

  deploy-production:
    needs: [deploy-staging, run-e2e-checks]
    runs-on: ubuntu-latest
    steps:
      - name: Deploy to production
        run: echo "Promoting staging to production..."
        # Your production deploy command here

If run-e2e-checks fails, deploy-production never runs. Your E2E checks are now the literal gate between staging and production. A failing login flow blocks the deploy automatically.

Deploy Checks Alongside Code

The cleanest setup: keep check deployment in your main deploy workflow, not a separate one:

- name: Deploy application
  run: ./deploy.sh production

- name: Deploy Checkly checks
  run: npx checkly deploy
  env:
    CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
    CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}
    BASE_URL: https://myapp.com

- name: Smoke test production with Checkly
  run: npx checkly test --location us-east-1 --env BASE_URL=https://myapp.com
  env:
    CHECKLY_API_KEY: ${{ secrets.CHECKLY_API_KEY }}
    CHECKLY_ACCOUNT_ID: ${{ secrets.CHECKLY_ACCOUNT_ID }}

This pattern: deploy the app, deploy the updated checks, then run a smoke test against production using the checks. If the smoke test fails, you know immediately post-deploy and can roll back.

Alert Channels as Code

Define alert channels in code too:

// __checks__/alert-channels.ts
import { SlackAlertChannel, EmailAlertChannel } from 'checkly/constructs'

export const slackChannel = new SlackAlertChannel('slack-alerts', {
  name: 'Engineering Slack',
  url: process.env.SLACK_WEBHOOK_URL!,
  sendRecovery: true,
  sendFailure: true,
  sendDegraded: false,
})

export const emailChannel = new EmailAlertChannel('email-alerts', {
  name: 'Engineering Email',
  address: 'oncall@myapp.com',
  sendRecovery: true,
  sendFailure: true,
})

Then reference them in checks:

import { slackChannel } from '../alert-channels'

new ApiCheck('critical-check', {
  name: 'Critical API',
  alertChannels: [slackChannel],
  // ...
})

Everything is code. Everything is in version control. Changes to alert routing go through pull requests.

Verifying Your Setup

After deploying:

# List all deployed checks
npx checkly ls

# Show check details
npx checkly ls --verbose

# Destroy all checks (careful — this deletes from Checkly)
npx checkly destroy

In the Checkly dashboard, you'll see all checks created by your CLI project labeled with the project name. They behave identically to UI-created checks — same scheduling, same alerts, same dashboards — but now they're version-controlled.

Common Pitfalls

Don't mix UI checks and code checks for the same service. Pick one approach per service. Mixing creates confusion about which is authoritative.

Always set logicalId uniquely. The logicalId in checkly.config.ts uniquely identifies your project. If two repos use the same ID, they'll conflict.

Environment variables in CI need to match what your checks expect. If your check references {{BASE_URL}}, make sure BASE_URL is set in your CI environment variables. Missing variables cause checks to fail with confusing errors.

checkly test is not checkly deploy. test runs checks once without saving them. deploy saves check definitions to Checkly. You need both in your workflow: test in CI as a gate, deploy on merge to update the live monitors.

Read more

Start now free