TestProject CI/CD Integration: Free Automation in Your DevOps Pipeline

TestProject CI/CD Integration: Free Automation in Your DevOps Pipeline

TestProject's CI/CD integration was built around one constraint: tests ran on an agent you controlled, not on TestProject's infrastructure (on the free plan). That constraint shaped everything — how you configured the agent in Docker, how you triggered jobs via API, and how you pulled results back into your pipeline.

This guide covers the complete integration patterns TestProject supported, with working configurations for the most common CI systems.

How the CI Integration Architecture Worked

TestProject's CI model had three components:

  1. The Agent — a Java process (packaged as a Docker image or binary) that executed tests locally
  2. The REST API — used to trigger test jobs and poll for results
  3. The Dashboard — where reports appeared after execution

Your CI pipeline called the API to start a job, waited for the agent to pick it up, then polled until completion. The agent needed a persistent connection to TestProject's cloud service to receive job assignments, but test execution happened entirely on your infrastructure.

Setting Up the Agent in Docker

The official TestProject agent Docker image was the standard approach for CI:

# docker-compose.yml for CI
version: '3'
services:
  testproject-agent:
    image: testproject/agent:latest
    environment:
      TP_API_KEY: ${TP_API_KEY}
      TP_AGENT_ALIAS: "ci-agent-${BUILD_NUMBER}"
    ports:
      - "8585:8585"
      - "8686:8686"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

The agent required two environment variables:

  • TP_API_KEY — your TestProject API key (found in account settings)
  • TP_AGENT_ALIAS — a unique name for this agent instance (critical in CI to avoid conflicts between parallel builds)

The agent took 30-60 seconds to register with TestProject's cloud before it was ready to accept jobs. Your pipeline had to wait for readiness before triggering tests.

GitHub Actions Integration

# .github/workflows/test.yml
name: Run TestProject Tests

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    
    services:
      testproject-agent:
        image: testproject/agent:latest
        env:
          TP_API_KEY: ${{ secrets.TP_API_KEY }}
          TP_AGENT_ALIAS: "gh-actions-${{ github.run_id }}"
        ports:
          - 8585:8585
          - 8686:8686
        options: >-
          --health-cmd "curl -f http://localhost:8585/api/status"
          --health-interval 10s
          --health-timeout 5s
          --health-retries 10

    steps:
      - uses: actions/checkout@v3
      
      - name: Wait for agent registration
        run: |
          echo "Waiting for agent to register with TestProject cloud..."
          sleep 30
          
      - name: Trigger test job
        id: trigger
        run: |
          JOB_ID=$(curl -s -X POST \
            "https://api.testproject.io/v2/projects/${TP_PROJECT_ID}/jobs/${TP_JOB_ID}/run" \
            -H "Authorization: ${TP_API_KEY}" \
            -H "Content-Type: application/json" \
            -d '{"agentId": "gh-actions-${{ github.run_id }}"}' \
            | jq -r '.id')
          echo "execution_id=$JOB_ID" >> $GITHUB_OUTPUT
        env:
          TP_API_KEY: ${{ secrets.TP_API_KEY }}
          TP_PROJECT_ID: ${{ secrets.TP_PROJECT_ID }}
          TP_JOB_ID: ${{ secrets.TP_JOB_ID }}
          
      - name: Wait for execution to complete
        run: |
          EXECUTION_ID="${{ steps.trigger.outputs.execution_id }}"
          for i in $(seq 1 60); do
            STATE=$(curl -s \
              "https://api.testproject.io/v2/reports/project/${TP_PROJECT_ID}/jobs/${TP_JOB_ID}/executions/${EXECUTION_ID}" \
              -H "Authorization: ${TP_API_KEY}" \
              | jq -r '.state')
            echo "State: $STATE"
            if [ "$STATE" = "Passed" ]; then exit 0; fi
            if [ "$STATE" = "Failed" ]; then exit 1; fi
            sleep 10
          done
          echo "Timeout waiting for tests"
          exit 1
        env:
          TP_API_KEY: ${{ secrets.TP_API_KEY }}
          TP_PROJECT_ID: ${{ secrets.TP_PROJECT_ID }}
          TP_JOB_ID: ${{ secrets.TP_JOB_ID }}

Jenkins Pipeline Configuration

// Jenkinsfile
pipeline {
    agent any
    
    environment {
        TP_API_KEY = credentials('testproject-api-key')
        TP_PROJECT_ID = 'your-project-id'
        TP_JOB_ID = 'your-job-id'
    }
    
    stages {
        stage('Start TestProject Agent') {
            steps {
                sh '''
                    docker run -d \
                        --name tp-agent-${BUILD_NUMBER} \
                        -e TP_API_KEY=${TP_API_KEY} \
                        -e TP_AGENT_ALIAS="jenkins-${BUILD_NUMBER}" \
                        -p 8585:8585 \
                        testproject/agent:latest
                    
                    # Wait for agent to register
                    sleep 45
                '''
            }
        }
        
        stage('Run Tests') {
            steps {
                script {
                    def response = sh(
                        script: '''
                            curl -s -X POST \
                                "https://api.testproject.io/v2/projects/${TP_PROJECT_ID}/jobs/${TP_JOB_ID}/run" \
                                -H "Authorization: ${TP_API_KEY}" \
                                -H "Content-Type: application/json" \
                                -d "{\\"agentId\\": \\"jenkins-${BUILD_NUMBER}\\"}"
                        ''',
                        returnStdout: true
                    )
                    def executionId = readJSON(text: response).id
                    
                    // Poll for completion
                    timeout(time: 30, unit: 'MINUTES') {
                        waitUntil {
                            def state = sh(
                                script: """
                                    curl -s \
                                        "https://api.testproject.io/v2/reports/project/${TP_PROJECT_ID}/jobs/${TP_JOB_ID}/executions/${executionId}" \
                                        -H "Authorization: ${TP_API_KEY}" \
                                    | jq -r '.state'
                                """,
                                returnStdout: true
                            ).trim()
                            
                            if (state == 'Failed') error("Tests failed")
                            return state == 'Passed'
                        }
                    }
                }
            }
        }
    }
    
    post {
        always {
            sh 'docker stop tp-agent-${BUILD_NUMBER} || true'
            sh 'docker rm tp-agent-${BUILD_NUMBER} || true'
        }
    }
}

GitLab CI Configuration

# .gitlab-ci.yml
stages:
  - test

variables:
  TP_AGENT_IMAGE: "testproject/agent:latest"

run-tests:
  stage: test
  image: curlimages/curl:latest
  services:
    - name: $TP_AGENT_IMAGE
      alias: testproject-agent
      variables:
        TP_API_KEY: $TP_API_KEY
        TP_AGENT_ALIAS: "gitlab-$CI_PIPELINE_ID"
  script:
    - sleep 40  # Wait for agent registration
    - |
      EXECUTION_ID=$(curl -sf -X POST \
        "https://api.testproject.io/v2/projects/$TP_PROJECT_ID/jobs/$TP_JOB_ID/run" \
        -H "Authorization: $TP_API_KEY" \
        -H "Content-Type: application/json" \
        -d "{\"agentId\": \"gitlab-$CI_PIPELINE_ID\"}" \
        | jq -r '.id')
    - |
      for i in $(seq 1 60); do
        STATE=$(curl -sf \
          "https://api.testproject.io/v2/reports/project/$TP_PROJECT_ID/jobs/$TP_JOB_ID/executions/$EXECUTION_ID" \
          -H "Authorization: $TP_API_KEY" | jq -r '.state')
        [ "$STATE" = "Passed" ] && exit 0
        [ "$STATE" = "Failed" ] && exit 1
        sleep 10
      done

The REST API Reference

TestProject's v2 API covered the full automation lifecycle:

Endpoint Method Purpose
/v2/projects GET List all projects
/v2/projects/{id}/jobs GET List jobs in project
/v2/projects/{id}/jobs/{jobId}/run POST Trigger a job
/v2/reports/project/{id}/jobs/{jobId}/executions/{execId} GET Get execution status
/v2/agents GET List registered agents
/v2/agents/{id} DELETE Deregister an agent

All requests required the Authorization header with your API key (no Bearer prefix — just the raw key).

Common CI Integration Problems

Agent alias conflicts: If two pipeline runs start simultaneously with the same alias, one agent registration overwrites the other. Always include a unique build identifier ($BUILD_NUMBER, $CI_PIPELINE_ID, $GITHUB_RUN_ID) in the alias.

Agent startup timing: The 30-second sleep is a hack. The correct approach is to poll the agent status endpoint:

until curl -sf http://localhost:8585/api/status | jq -e '.registered == true' > /dev/null; do
  sleep 5
done

Test result artifacts: TestProject's dashboard showed results, but CI systems expect JUnit XML. The API could generate reports but exporting to JUnit XML required a paid plan feature. On the free plan, you relied on pass/fail state only — no detailed test results in CI artifacts.

The Infrastructure Reality

Every approach described above requires you to manage a Docker container per pipeline run, handle registration timing, clean up agents after runs, and deal with networking between the agent container and test target. For teams already running Docker-based CI, this is manageable. For smaller teams, it's meaningful overhead.

HelpMeTest eliminates this layer entirely. Tests are defined in plain English, run on cloud infrastructure, and integrate with CI via a simple API call — no agent, no Docker, no registration timing. The free plan includes CI/CD integration out of the box. For teams evaluating how much infrastructure complexity is acceptable in their test automation stack, that tradeoff is worth examining.


Key takeaways:

  • TestProject's free CI integration required running a Docker agent in your pipeline
  • Each CI run needed a unique agent alias to avoid conflicts
  • The REST API triggered jobs and polled for completion — no native CI plugins existed
  • Agent startup added 30-60 seconds of overhead per pipeline run
  • JUnit XML export required a paid plan; free users got pass/fail state only

Read more

Start now free