Testing

GitLab Runner & Argo Workflows for Continuous Testing

HelpMeTest

HelpMeTest

— 5 min read
GitLab Runner & Argo Workflows for Continuous Testing

GitLab Runner and Argo Workflows serve different parts of the CI/CD testing problem. GitLab Runner is a general-purpose CI executor tightly integrated with GitLab. Argo Workflows is a Kubernetes-native workflow engine better suited for complex, long-running, or data-heavy pipelines.

Used together — or separately depending on your stack — they cover most continuous testing scenarios for teams on Kubernetes.

GitLab Runner for Testing

What GitLab Runner Does

GitLab Runner is the agent that executes jobs defined in .gitlab-ci.yml. It connects to your GitLab instance (GitLab.com or self-managed), picks up jobs, runs them, and reports results back.

Runners can execute jobs in several modes:

  • Shell — runs commands directly on the host
  • Docker — runs each job in a fresh container (most common for testing)
  • Kubernetes — creates a pod per job in your cluster
  • Docker Machine — auto-scales EC2/GCE instances (deprecated, being replaced)

For testing pipelines, Docker and Kubernetes executors are the right choice. Shell executor is fast but creates isolation and dependency problems.

Installing a GitLab Runner

# macOS
brew install gitlab-runner
gitlab-runner install
gitlab-runner start

# Linux
curl -L "https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh" | sudo bash
sudo apt-get install gitlab-runner

Register the runner with your GitLab instance:

gitlab-runner register \
  --url https://gitlab.com \
  --registration-token YOUR_REGISTRATION_TOKEN \
  --executor docker \
  --docker-image node:20-alpine \
  --description "test-runner" \
  --tag-list "testing,docker"

Or use the Kubernetes executor for cluster-native execution:

gitlab-runner register \
  --url https://gitlab.com \
  --registration-token YOUR_REGISTRATION_TOKEN \
  --executor kubernetes \
  --kubernetes-image node:20-alpine \
  --description "k8s-test-runner"

Writing Test Jobs

A basic .gitlab-ci.yml for a Node.js project:

stages:
  - test
  - integration

variables:
  NODE_ENV: test

cache:
  paths:
    - node_modules/

unit-tests:
  stage: test
  image: node:20-alpine
  script:
    - npm ci
    - npm test
  artifacts:
    reports:
      junit: junit.xml
    when: always

lint:
  stage: test
  image: node:20-alpine
  script:
    - npm ci
    - npm run lint

The artifacts: reports: junit line uploads test results to GitLab, which shows per-test pass/fail status in the merge request UI.

Parallel Test Execution

Split tests across multiple jobs using parallel:

unit-tests:
  stage: test
  image: node:20-alpine
  parallel: 4
  script:
    - npm ci
    - npx jest --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
  artifacts:
    reports:
      junit: junit.xml

CI_NODE_INDEX and CI_NODE_TOTAL are injected automatically. Jest's --shard flag divides the test suite by file count. For timing-based splitting, use Knapsack Pro or a custom script that reads historical timing data.

Matrix Builds

Test against multiple versions or configurations:

test:
  stage: test
  parallel:
    matrix:
      - NODE_VERSION: ["18", "20", "22"]
        DATABASE: ["postgres", "mysql"]
  image: node:${NODE_VERSION}-alpine
  services:
    - name: ${DATABASE}:latest
      alias: db
  script:
    - npm ci
    - DATABASE_URL=${DATABASE}://db/testdb npm test

This generates 6 jobs (3 Node versions × 2 databases) and runs them in parallel.

Services for Integration Tests

GitLab Runner can spin up service containers alongside the test container:

integration-tests:
  stage: integration
  image: node:20-alpine
  services:
    - postgres:15-alpine
    - redis:7-alpine
  variables:
    POSTGRES_DB: test_db
    POSTGRES_USER: testuser
    POSTGRES_PASSWORD: testpassword
    DATABASE_URL: postgresql://testuser:testpassword@postgres/test_db
    REDIS_URL: redis://redis:6379
  script:
    - npm ci
    - npm run test:integration

Services are reachable by their image name (or alias). The test container waits for services to be healthy via health check scripts or wait-for-it.sh.

Argo Workflows for Continuous Testing

When Argo Makes Sense

Argo Workflows is overkill for simple "run tests on commit" workflows — use GitLab CI for that. Argo shines when:

  • Tests take hours and need checkpoint/resume
  • You're running data-heavy tests that produce large artifacts
  • You need complex DAG (directed acyclic graph) dependencies between test steps
  • Tests need to fan out to hundreds of parallel workers dynamically
  • You're already using Argo for other workflows and want consistency

Common use cases: nightly full regression suites, load testing pipelines, data pipeline validation, and ML model test pipelines.

Installing Argo Workflows

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/latest/download/install.yaml

Install the Argo CLI:

# macOS
brew install argo

# Linux
curl -sLO https://github.com/argoproj/argo-workflows/releases/latest/download/argo-linux-amd64.gz
gunzip argo-linux-amd64.gz
chmod +x argo-linux-amd64
mv argo-linux-amd64 /usr/local/bin/argo

Access the UI:

kubectl -n argo port-forward svc/argo-server 2746:2746

Open https://localhost:2746.

A Basic Test Workflow

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: test-workflow-
spec:
  entrypoint: run-tests
  templates:
    - name: run-tests
      dag:
        tasks:
          - name: unit-tests
            template: jest-tests
            arguments:
              parameters:
                - name: test-pattern
                  value: "unit/**"
          - name: integration-tests
            template: jest-tests
            dependencies: [unit-tests]
            arguments:
              parameters:
                - name: test-pattern
                  value: "integration/**"

    - name: jest-tests
      inputs:
        parameters:
          - name: test-pattern
      container:
        image: node:20-alpine
        command: [sh, -c]
        args:
          - |
            git clone https://github.com/your-org/your-repo .
            npm ci
            npx jest $(inputs.parameters.test-pattern)

Run it:

argo submit --watch test-workflow.yaml

Parallel Test Fan-Out with withItems

- name: parallel-test-shards
  template: test-shard
  arguments:
    parameters:
      - name: shard
        value: "{{item}}"
  withItems:
    - "0"
    - "1"
    - "2"
    - "3"
    - "4"
    - "5"
    - "6"
    - "7"

This creates 8 parallel pods. Combine with withParam to generate the shard list dynamically from a previous step.

Cron-Based Test Schedules

Argo CronWorkflows run workflows on a schedule — useful for nightly regression suites:

apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: nightly-tests
spec:
  schedule: "0 2 * * *"  # 2 AM UTC daily
  timezone: "UTC"
  workflowSpec:
    entrypoint: full-regression
    templates:
      - name: full-regression
        dag:
          tasks:
            - name: smoke-tests
              template: run-smoke
            - name: regression-suite
              template: run-regression
              dependencies: [smoke-tests]

Artifact Passing Between Steps

Argo can pass files between workflow steps via artifact repositories (S3, GCS, Minio):

templates:
  - name: build-and-test
    dag:
      tasks:
        - name: build
          template: build-app
        - name: test
          template: run-tests
          dependencies: [build]
          arguments:
            artifacts:
              - name: app-binary
                from: "{{tasks.build.outputs.artifacts.app-binary}}"

This is useful for compile-heavy projects where you want to build once and test in parallel across multiple environments.

Combining GitLab Runner and Argo

A common pattern: GitLab CI handles per-commit testing, Argo handles longer-running scheduled suites.

GitLab CI job that triggers an Argo Workflow:

nightly-regression:
  stage: test
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
  image: argoproj/argocli:latest
  script:
    - argo submit --wait nightly-regression-workflow.yaml
      --server https://argo.internal
      --token $ARGO_TOKEN

GitLab's scheduled pipelines trigger nightly; the CI job submits the workflow to Argo and waits for completion. Results flow back to GitLab via artifact uploads or API calls.

Adding Continuous Monitoring

Both GitLab Runner and Argo run tests reactively — on commits or schedules. Neither provides 24/7 monitoring of your live application.

HelpMeTest fills that gap. It runs your end-to-end tests against staging and production continuously (every 5 minutes), independent of your CI pipeline. When a deployment breaks something, HelpMeTest catches it within minutes — not at the next commit.

The integration is simple: write tests once in plain English, and HelpMeTest handles scheduling, execution, and alerting. Usage-based pricing ($0.003/run, no base fee) covers unlimited tests and health checks.

Summary

GitLab Runner is the right tool for per-commit testing integrated with GitLab. Easy to set up, good parallel support, native test reporting in merge requests.

Argo Workflows is the right tool for complex, long-running, or Kubernetes-native test pipelines that need DAG dependencies, fan-out parallelism, or cron scheduling at scale.

Use them together when you have both needs: GitLab CI for fast per-commit feedback, Argo for comprehensive nightly or scheduled suites. Add HelpMeTest on top for continuous production monitoring that neither CI tool handles.

Read more

Start now free