No-Code Testing

Reflect.io CI/CD Integration and Test Maintenance: Keeping No-Code Tests Up to Date

HelpMeTest

HelpMeTest

— 6 min read
Reflect.io CI/CD Integration and Test Maintenance: Keeping No-Code Tests Up to Date

A no-code testing tool is only valuable if the tests run automatically and stay up to date as your application changes. Reflect.io's CI/CD integration and AI-assisted maintenance solve both challenges.

This guide covers deploying Reflect tests in your pipeline and managing test health over time.

CI/CD Integration Options

Reflect integrates with CI/CD systems through its REST API. When you trigger a test run via the API, Reflect returns a run ID that you poll for completion.

The Integration Pattern

CI pipeline starts
   ↓
POST /v1/runs (start test run)
   ↓
GET /v1/runs/{id} (poll for completion)
   ↓
Check status: passed | failed | error
   ↓
Fail or pass the CI build accordingly

Getting Your API Key

  1. Log in to the Reflect dashboard
  2. Go to Settings → API Keys
  3. Click Generate New Key
  4. Copy the key — store it as a CI secret, never in code

Get Your Suite and Test IDs

From the Reflect dashboard URL when viewing a suite: https://app.reflect.run/projects/{projectId}/suites/{suiteId}

Note both IDs — you'll use suiteId in API calls.

GitHub Actions Integration

Basic Integration

name: E2E Tests

on:
  push:
    branches: [main]
  pull_request:

jobs:
  e2e-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Run Reflect test suite
        id: run-tests
        run: |
          RESPONSE=$(curl -s -X POST https://api.reflect.run/v1/runs \
            -H "Authorization: Bearer ${{ secrets.REFLECT_API_KEY }}" \
            -H "Content-Type: application/json" \
            -d '{
              "suiteId": "${{ vars.REFLECT_SUITE_ID }}"
            }')
          RUN_ID=$(echo $RESPONSE | jq -r '.id')
          echo "run_id=$RUN_ID" >> $GITHUB_OUTPUT

      - name: Wait for test completion
        run: |
          RUN_ID="${{ steps.run-tests.outputs.run_id }}"
          MAX_WAIT=300  # 5 minutes
          ELAPSED=0
          
          while [ $ELAPSED -lt $MAX_WAIT ]; do
            RESULT=$(curl -s "https://api.reflect.run/v1/runs/$RUN_ID" \
              -H "Authorization: Bearer ${{ secrets.REFLECT_API_KEY }}")
            STATUS=$(echo $RESULT | jq -r '.status')
            
            if [ "$STATUS" = "passed" ]; then
              echo "Tests passed"
              exit 0
            elif [ "$STATUS" = "failed" ] || [ "$STATUS" = "error" ]; then
              echo "Tests failed"
              echo $RESULT | jq '.results'
              exit 1
            fi
            
            sleep 10
            ELAPSED=$((ELAPSED + 10))
          done
          
          echo "Tests timed out after ${MAX_WAIT}s"
          exit 1

Running Against PR Preview URLs

For teams using preview deployments (Vercel, Netlify, Railway):

name: E2E Tests on Preview

on:
  pull_request:

jobs:
  e2e:
    runs-on: ubuntu-latest
    needs: deploy-preview  # wait for deployment job

    steps:
      - name: Get preview URL
        id: preview
        run: echo "url=${{ needs.deploy-preview.outputs.preview_url }}" >> $GITHUB_OUTPUT

      - name: Run Reflect tests against preview
        run: |
          curl -s -X POST https://api.reflect.run/v1/runs \
            -H "Authorization: Bearer ${{ secrets.REFLECT_API_KEY }}" \
            -H "Content-Type: application/json" \
            -d "{
              \"suiteId\": \"${{ vars.REFLECT_SUITE_ID }}\",
              \"overrides\": {
                \"baseUrl\": \"${{ steps.preview.outputs.url }}\"
              }
            }"

The overrides.baseUrl replaces the recorded base URL, so tests recorded against production run against your PR preview.

Parallel Suite Execution

For large test suites, run multiple suites in parallel:

jobs:
  test-auth:
    steps:
      - name: Run auth suite
        run: |
          curl -X POST https://api.reflect.run/v1/runs \
            -H "Authorization: Bearer ${{ secrets.REFLECT_API_KEY }}" \
            -d '{"suiteId": "${{ vars.AUTH_SUITE_ID }}"}'

  test-checkout:
    steps:
      - name: Run checkout suite
        run: |
          curl -X POST https://api.reflect.run/v1/runs \
            -H "Authorization: Bearer ${{ secrets.REFLECT_API_KEY }}" \
            -d '{"suiteId": "${{ vars.CHECKOUT_SUITE_ID }}"}'

GitHub Actions runs these jobs in parallel, reducing total CI time.

Jenkins Integration

Declarative Pipeline

pipeline {
    agent any

    environment {
        REFLECT_API_KEY = credentials('reflect-api-key')
        REFLECT_SUITE_ID = '${env.REFLECT_SUITE_ID}'
    }

    stages {
        stage('Deploy to Staging') {
            steps {
                sh './deploy-staging.sh'
            }
        }

        stage('E2E Tests') {
            steps {
                script {
                    def response = sh(
                        script: """
                            curl -s -X POST https://api.reflect.run/v1/runs \
                                -H 'Authorization: Bearer ${REFLECT_API_KEY}' \
                                -H 'Content-Type: application/json' \
                                -d '{"suiteId": "${REFLECT_SUITE_ID}"}'
                        """,
                        returnStdout: true
                    ).trim()

                    def json = readJSON text: response
                    def runId = json.id

                    // Poll for completion
                    def status = 'running'
                    def attempts = 0
                    while (status == 'running' && attempts < 60) {
                        sleep(10)
                        def result = sh(
                            script: """
                                curl -s https://api.reflect.run/v1/runs/${runId} \
                                    -H 'Authorization: Bearer ${REFLECT_API_KEY}'
                            """,
                            returnStdout: true
                        ).trim()
                        def resultJson = readJSON text: result
                        status = resultJson.status
                        attempts++
                    }

                    if (status != 'passed') {
                        error("Reflect tests ${status}")
                    }
                }
            }
        }
    }
}

Managing Test Maintenance

Understanding Why Tests Break

Reflect tests break for two reasons:

  1. Real failures — your application has a bug and the flow doesn't work
  2. UI changes — the application works, but the UI changed (new selector, new text, redesigned flow)

Before fixing a broken test, determine which type it is:

  • Check if the user flow still works manually in the browser
  • If it works → UI change, update the test
  • If it doesn't work → real bug, log it and fix the code

Using Reflect's AI Suggestions

When a test fails, Reflect analyzes the failure and suggests fixes:

  1. Open the failed test run
  2. Click View Failure
  3. See the step that failed with a screenshot
  4. Check the AI Suggestion panel — Reflect may suggest an updated selector or step

Accept the suggestion if it looks correct. Reflect applies it and reruns the test to confirm.

When to Re-Record vs. Edit

Re-record when:

  • The entire flow changed (new steps, different navigation path)
  • More than 3-4 steps need updating
  • The page was redesigned significantly

Edit individual steps when:

  • A button text changed
  • An element moved to a different location
  • An additional step was added to an existing flow

Re-recording is faster than editing when the scope of changes is large.

Building Resilient Tests

Reduce maintenance burden by writing resilient tests from the start:

Use text-based selectors over CSS: When recording, prefer clicking elements by their visible text. "Click 'Sign In' button" is more stable than "click .btn-primary.header-cta." If the button text stays the same, the selector stays valid even if CSS classes change.

Avoid position-based selectors: "Click the third item in the list" breaks when list order changes. "Click the item with text 'Premium Plan'" is more stable.

Assert meaningful content: Assert "Order confirmed" text rather than "check that an element with class .success-banner is visible." Text assertions survive CSS changes; class assertions don't.

Avoid asserting dynamic values: Don't assert that an order ID equals "ORD-12345" — it changes every run. Assert that the order ID exists and matches a pattern.

Test Organization for Maintainability

Modular Test Design

Instead of one giant test that does everything, break flows into focused tests:

Login test → tests only login
Dashboard test → starts logged in, tests dashboard
Checkout test → starts logged in with items in cart, tests checkout

This way, when the login flow changes, you update only the login test. The checkout test doesn't need to change.

Shared Setup Steps

For tests requiring authentication, create a Login suite setup:

  1. Create one test: "Authenticated Session Setup"
  2. Record the login steps
  3. Mark it as the suite setup step
  4. All other tests in the suite start authenticated

When the login flow changes, update only the setup test.

Test Naming Conventions

Good naming helps understand what breaks:

Bad: Test1, Test_002, Homepage check Good: User logs in with valid credentials, User sees error on invalid login, Guest adds product to cart

When User logs in with valid credentials fails, you immediately know: the login flow is broken.

Handling Flaky Tests

No-code tests can be flaky due to:

  • Timing issues (page loads slowly, test doesn't wait)
  • Network variability
  • Environment-specific issues

Adding Wait Steps

If a test fails intermittently on a step:

  1. Edit the test
  2. Before the failing step, add a Wait step:
    • Wait for element to be visible: "Wait for 'Dashboard' heading"
    • Wait for URL: "Wait for URL to contain /dashboard"
    • Fixed delay: "Wait 2 seconds" (last resort)

Waiting for specific elements is more reliable than fixed delays — the test proceeds as soon as the element appears rather than always waiting the full duration.

Marking Known Flaky Tests

For tests that are flaky due to known infrastructure issues:

  1. Open the test
  2. Click Settings → Retry on failure
  3. Set retry count (1-2 retries for flaky tests)

Reflect retries the full test before marking it as failed. Use sparingly — retries mask real failures and slow down CI.

Monitoring Dashboard Health

Set up proactive monitoring of test health:

Failure rate tracking: Reflect's dashboard shows pass rates over time. A test with 70% pass rate over the last 30 runs is flaky and needs attention.

Run duration trending: If a test's average duration is increasing, it might be testing a degrading feature or the test has accumulated too many steps.

Weekly review cadence: Schedule 30 minutes weekly to review:

  • Tests failing consistently (real failures or stale tests)
  • Tests with declining pass rates (flakiness)
  • Tests not run recently (may be outdated)

Deleting vs. Disabling Tests

When a feature is removed or significantly changed:

Disable the test if:

  • The feature is temporarily disabled
  • You plan to update the test soon
  • You need the test as documentation of the old behavior

Delete the test if:

  • The feature is permanently removed
  • The test was never reliable and isn't worth fixing
  • The test has been replaced by a better one

Keeping too many disabled tests creates confusion. Audit disabled tests monthly.

Summary

Reflect.io CI/CD integration keeps your no-code tests running on every deployment. The API-based approach works with any CI system. Combined with AI-assisted maintenance and resilient test design practices, you can maintain a healthy test suite without dedicated automation engineers.

Production Monitoring While CI Is Idle

Reflect tests run when you deploy. HelpMeTest runs tests against your production app every 5 minutes, 24/7 — alerting you the moment a user flow breaks.

Start monitoring production →

Read more

Start now free