TestProject CI/CD Integration: Free Automation in Your DevOps Pipeline
TestProject's CI/CD integration was built around one constraint: tests ran on an agent you controlled, not on TestProject's infrastructure (on the free plan). That constraint shaped everything — how you configured the agent in Docker, how you triggered jobs via API, and how you pulled results back into your pipeline.
This guide covers the complete integration patterns TestProject supported, with working configurations for the most common CI systems.
How the CI Integration Architecture Worked
TestProject's CI model had three components:
- The Agent — a Java process (packaged as a Docker image or binary) that executed tests locally
- The REST API — used to trigger test jobs and poll for results
- The Dashboard — where reports appeared after execution
Your CI pipeline called the API to start a job, waited for the agent to pick it up, then polled until completion. The agent needed a persistent connection to TestProject's cloud service to receive job assignments, but test execution happened entirely on your infrastructure.
Setting Up the Agent in Docker
The official TestProject agent Docker image was the standard approach for CI:
# docker-compose.yml for CI
version: '3'
services:
testproject-agent:
image: testproject/agent:latest
environment:
TP_API_KEY: ${TP_API_KEY}
TP_AGENT_ALIAS: "ci-agent-${BUILD_NUMBER}"
ports:
- "8585:8585"
- "8686:8686"
volumes:
- /var/run/docker.sock:/var/run/docker.sockThe agent required two environment variables:
TP_API_KEY— your TestProject API key (found in account settings)TP_AGENT_ALIAS— a unique name for this agent instance (critical in CI to avoid conflicts between parallel builds)
The agent took 30-60 seconds to register with TestProject's cloud before it was ready to accept jobs. Your pipeline had to wait for readiness before triggering tests.
GitHub Actions Integration
# .github/workflows/test.yml
name: Run TestProject Tests
on: [push, pull_request]
jobs:
test:
runs-on: ubuntu-latest
services:
testproject-agent:
image: testproject/agent:latest
env:
TP_API_KEY: ${{ secrets.TP_API_KEY }}
TP_AGENT_ALIAS: "gh-actions-${{ github.run_id }}"
ports:
- 8585:8585
- 8686:8686
options: >-
--health-cmd "curl -f http://localhost:8585/api/status"
--health-interval 10s
--health-timeout 5s
--health-retries 10
steps:
- uses: actions/checkout@v3
- name: Wait for agent registration
run: |
echo "Waiting for agent to register with TestProject cloud..."
sleep 30
- name: Trigger test job
id: trigger
run: |
JOB_ID=$(curl -s -X POST \
"https://api.testproject.io/v2/projects/${TP_PROJECT_ID}/jobs/${TP_JOB_ID}/run" \
-H "Authorization: ${TP_API_KEY}" \
-H "Content-Type: application/json" \
-d '{"agentId": "gh-actions-${{ github.run_id }}"}' \
| jq -r '.id')
echo "execution_id=$JOB_ID" >> $GITHUB_OUTPUT
env:
TP_API_KEY: ${{ secrets.TP_API_KEY }}
TP_PROJECT_ID: ${{ secrets.TP_PROJECT_ID }}
TP_JOB_ID: ${{ secrets.TP_JOB_ID }}
- name: Wait for execution to complete
run: |
EXECUTION_ID="${{ steps.trigger.outputs.execution_id }}"
for i in $(seq 1 60); do
STATE=$(curl -s \
"https://api.testproject.io/v2/reports/project/${TP_PROJECT_ID}/jobs/${TP_JOB_ID}/executions/${EXECUTION_ID}" \
-H "Authorization: ${TP_API_KEY}" \
| jq -r '.state')
echo "State: $STATE"
if [ "$STATE" = "Passed" ]; then exit 0; fi
if [ "$STATE" = "Failed" ]; then exit 1; fi
sleep 10
done
echo "Timeout waiting for tests"
exit 1
env:
TP_API_KEY: ${{ secrets.TP_API_KEY }}
TP_PROJECT_ID: ${{ secrets.TP_PROJECT_ID }}
TP_JOB_ID: ${{ secrets.TP_JOB_ID }}Jenkins Pipeline Configuration
// Jenkinsfile
pipeline {
agent any
environment {
TP_API_KEY = credentials('testproject-api-key')
TP_PROJECT_ID = 'your-project-id'
TP_JOB_ID = 'your-job-id'
}
stages {
stage('Start TestProject Agent') {
steps {
sh '''
docker run -d \
--name tp-agent-${BUILD_NUMBER} \
-e TP_API_KEY=${TP_API_KEY} \
-e TP_AGENT_ALIAS="jenkins-${BUILD_NUMBER}" \
-p 8585:8585 \
testproject/agent:latest
# Wait for agent to register
sleep 45
'''
}
}
stage('Run Tests') {
steps {
script {
def response = sh(
script: '''
curl -s -X POST \
"https://api.testproject.io/v2/projects/${TP_PROJECT_ID}/jobs/${TP_JOB_ID}/run" \
-H "Authorization: ${TP_API_KEY}" \
-H "Content-Type: application/json" \
-d "{\\"agentId\\": \\"jenkins-${BUILD_NUMBER}\\"}"
''',
returnStdout: true
)
def executionId = readJSON(text: response).id
// Poll for completion
timeout(time: 30, unit: 'MINUTES') {
waitUntil {
def state = sh(
script: """
curl -s \
"https://api.testproject.io/v2/reports/project/${TP_PROJECT_ID}/jobs/${TP_JOB_ID}/executions/${executionId}" \
-H "Authorization: ${TP_API_KEY}" \
| jq -r '.state'
""",
returnStdout: true
).trim()
if (state == 'Failed') error("Tests failed")
return state == 'Passed'
}
}
}
}
}
}
post {
always {
sh 'docker stop tp-agent-${BUILD_NUMBER} || true'
sh 'docker rm tp-agent-${BUILD_NUMBER} || true'
}
}
}GitLab CI Configuration
# .gitlab-ci.yml
stages:
- test
variables:
TP_AGENT_IMAGE: "testproject/agent:latest"
run-tests:
stage: test
image: curlimages/curl:latest
services:
- name: $TP_AGENT_IMAGE
alias: testproject-agent
variables:
TP_API_KEY: $TP_API_KEY
TP_AGENT_ALIAS: "gitlab-$CI_PIPELINE_ID"
script:
- sleep 40 # Wait for agent registration
- |
EXECUTION_ID=$(curl -sf -X POST \
"https://api.testproject.io/v2/projects/$TP_PROJECT_ID/jobs/$TP_JOB_ID/run" \
-H "Authorization: $TP_API_KEY" \
-H "Content-Type: application/json" \
-d "{\"agentId\": \"gitlab-$CI_PIPELINE_ID\"}" \
| jq -r '.id')
- |
for i in $(seq 1 60); do
STATE=$(curl -sf \
"https://api.testproject.io/v2/reports/project/$TP_PROJECT_ID/jobs/$TP_JOB_ID/executions/$EXECUTION_ID" \
-H "Authorization: $TP_API_KEY" | jq -r '.state')
[ "$STATE" = "Passed" ] && exit 0
[ "$STATE" = "Failed" ] && exit 1
sleep 10
doneThe REST API Reference
TestProject's v2 API covered the full automation lifecycle:
| Endpoint | Method | Purpose |
|---|---|---|
/v2/projects |
GET | List all projects |
/v2/projects/{id}/jobs |
GET | List jobs in project |
/v2/projects/{id}/jobs/{jobId}/run |
POST | Trigger a job |
/v2/reports/project/{id}/jobs/{jobId}/executions/{execId} |
GET | Get execution status |
/v2/agents |
GET | List registered agents |
/v2/agents/{id} |
DELETE | Deregister an agent |
All requests required the Authorization header with your API key (no Bearer prefix — just the raw key).
Common CI Integration Problems
Agent alias conflicts: If two pipeline runs start simultaneously with the same alias, one agent registration overwrites the other. Always include a unique build identifier ($BUILD_NUMBER, $CI_PIPELINE_ID, $GITHUB_RUN_ID) in the alias.
Agent startup timing: The 30-second sleep is a hack. The correct approach is to poll the agent status endpoint:
until curl -sf http://localhost:8585/api/status | jq -e '.registered == true' > /dev/null; do
sleep 5
doneTest result artifacts: TestProject's dashboard showed results, but CI systems expect JUnit XML. The API could generate reports but exporting to JUnit XML required a paid plan feature. On the free plan, you relied on pass/fail state only — no detailed test results in CI artifacts.
The Infrastructure Reality
Every approach described above requires you to manage a Docker container per pipeline run, handle registration timing, clean up agents after runs, and deal with networking between the agent container and test target. For teams already running Docker-based CI, this is manageable. For smaller teams, it's meaningful overhead.
HelpMeTest eliminates this layer entirely. Tests are defined in plain English, run on cloud infrastructure, and integrate with CI via a simple API call — no agent, no Docker, no registration timing. The free plan includes CI/CD integration out of the box. For teams evaluating how much infrastructure complexity is acceptable in their test automation stack, that tradeoff is worth examining.
Key takeaways:
- TestProject's free CI integration required running a Docker agent in your pipeline
- Each CI run needed a unique agent alias to avoid conflicts
- The REST API triggered jobs and polled for completion — no native CI plugins existed
- Agent startup added 30-60 seconds of overhead per pipeline run
- JUnit XML export required a paid plan; free users got pass/fail state only