Running Tests in TeamCity: Complete Setup Guide

TeamCity is JetBrains' CI/CD server, widely used in Java and .NET shops but capable of building anything. Its test reporting goes deeper than most CI tools — it tracks test history, identifies flaky tests, and shows failure trends across builds without plugins.

This guide covers configuring TeamCity for reliable test execution with JUnit XML reports, parallel builds, test retry, and the Kotlin DSL for version-controlled build configurations.

TeamCity Architecture

Before configuring builds, understand the key concepts:

• Build configurations define what to build and how. Each configuration has steps, triggers, and parameters.
• Build agents execute configurations. TeamCity supports multiple agents for parallelism.
• Projects group related build configurations.
• VCS roots connect configurations to source code repositories.

Connecting Your Repository

1. Navigate to Administration → Projects → Create project
2. Choose From VCS URL and paste your repository URL
3. TeamCity detects the VCS type and creates a VCS root
4. Create a build configuration within the project

For GitHub repositories, set up an OAuth connection in Administration → OAuth Connections for automatic webhook triggers.

Basic Test Step Configuration

Maven (Java)

Add a Maven build step:

Runner: Maven
Goals: clean test
Maven version: (select installed version or use bundled)
Pom.xml location: pom.xml
JVM command line parameters: -Xmx1g

TeamCity automatically parses Maven's Surefire XML reports. Test results appear in the Tests tab without any additional configuration.

Gradle (Java/Kotlin)

Runner: Gradle
Gradle tasks: clean test
Build file: build.gradle
JVM command line parameters: -Xmx1g

Gradle tasks that run tests automatically generate XML reports in build/test-results. TeamCity picks these up.

Node.js (Jest)

TeamCity doesn't have a native Jest runner, but Jest outputs JUnit XML:

Runner: Command Line
Command executable: npx
Command parameters: jest --ci --reporters=default --reporters=jest-junit

Add the jest-junit package:

npm install --save-dev jest-junit

Configure output path in package.json:

{
  "jest": {
    "reporters": [
      "default",
      ["jest-junit", { "outputDirectory": "test-results", "outputName": "junit.xml" }]
    ]
  }
}

Add a second build step to import the XML report:

Runner: XML Report Processing
Report type: JUnit
Monitoring rules: test-results/junit.xml

pytest (Python)

Runner: Command Line
Command executable: pytest
Command parameters: --junitxml=test-results/junit.xml -v

Add XML report processing step pointing to test-results/junit.xml.

JUnit XML Reports

TeamCity's test reporting relies on JUnit XML format. Regardless of your test framework, output in this format and configure TeamCity to import it.

Service messages (alternative approach)

For frameworks that don't output JUnit XML, use TeamCity's service messages — special strings printed to stdout that TeamCity interprets:

##teamcity[testSuiteStarted name='MyTests']
##teamcity[testStarted name='should add numbers']
##teamcity[testFinished name='should add numbers' duration='5']
##teamcity[testSuiteFinished name='MyTests']

For a failure:

##teamcity[testFailed name='should add numbers' message='Expected 4 but got 5' details='AssertionError...']

This approach works with any language or framework.

Parallel Builds

TeamCity supports several forms of parallelism:

Parallel builds across agents

Run multiple build configurations simultaneously on different agents. Create a configuration for each test group and add them as dependencies or parallel builds in a Build Chain.

Matrix builds

For testing against multiple environments (Node versions, databases, OS variants), use a Matrix build configuration:

1. In build configuration settings, go to Parameters
2. Add a parameter: env.NODE_VERSION with values 16,18,20 (comma-separated)
3. Enable Matrix build in build configuration options
4. TeamCity creates separate builds for each parameter value

Parallel test execution within a build

For test frameworks that support parallelism, configure it in the test runner:

Jest:

{
  "jest": {
    "maxWorkers": 4
  }
}

pytest:

pytest -n 4  # requires pytest-xdist

Maven Surefire:

<configuration>
  <forkCount>4</forkCount>
  <reuseForks>true</reuseForks>
</configuration>

Build grid for test splitting

For large test suites, split tests across multiple agents:

1. Create a composite build configuration as the parent
2. Create multiple "shard" configurations with parameters like SHARD_INDEX=1 and TOTAL_SHARDS=4
3. Each shard runs a subset: jest --testPathPattern="shard-${SHARD_INDEX}"
4. The composite configuration waits for all shards to complete

Test Retry

TeamCity supports automatic test retry to handle intermittent failures:

Build-level retry

In build configuration settings:

1. Go to Build Failure Conditions
2. Enable Rerun build on test failure
3. Set maximum retry count

Framework-level retry (recommended)

Configure retry in the test framework itself for faster feedback:

Jest:

// jest.config.js
module.exports = {
  testRunner: 'jest-circus/runner',
  testRetries: 2, // retry each failing test up to 2 times
}

pytest:

pip install pytest-rerunfailures
pytest --reruns 3 --reruns-delay 1

Maven Surefire:

<plugin>
  <groupId>org.apache.maven.plugins</groupId>
  <artifactId>maven-surefire-plugin</artifactId>
  <configuration>
    <rerunFailingTestsCount>2</rerunFailingTestsCount>
  </configuration>
</plugin>

Kotlin DSL

TeamCity's build configurations can be defined as code using the Kotlin DSL. This enables version control, code review, and programmatic configuration of builds.

Enabling versioned settings

1. Go to Project → Versioned Settings
2. Enable versioned settings and choose your VCS root
3. Select Use settings from VCS and format Kotlin
4. TeamCity generates a .teamcity/ directory with your current settings

Writing build configurations in Kotlin

// .teamcity/settings.kts
import jetbrains.buildServer.configs.kotlin.*
import jetbrains.buildServer.configs.kotlin.buildSteps.maven
import jetbrains.buildServer.configs.kotlin.buildSteps.script
import jetbrains.buildServer.configs.kotlin.triggers.vcs

version = "2024.07"

project {
    buildType(UnitTests)
    buildType(IntegrationTests)
}

object UnitTests : BuildType({
    name = "Unit Tests"
    
    vcs {
        root(DslContext.settingsRoot)
    }
    
    steps {
        maven {
            goals = "clean test"
            pomLocation = "pom.xml"
            jvmArgs = "-Xmx1g"
        }
    }
    
    triggers {
        vcs {
            branchFilter = "+:*"
        }
    }
    
    failureConditions {
        testFailure = true
        executionTimeoutMin = 30
    }
})

object IntegrationTests : BuildType({
    name = "Integration Tests"
    
    vcs {
        root(DslContext.settingsRoot)
    }
    
    steps {
        script {
            scriptContent = """
                docker-compose up -d
                npm run test:integration
                docker-compose down
            """.trimIndent()
        }
        
        step {
            type = "xml-report-plugin"
            param("xmlReportParsing.reportType", "junit")
            param("xmlReportParsing.reportDirs", "test-results/*.xml")
        }
    }
    
    dependencies {
        snapshot(UnitTests) {
            onDependencyFailure = FailureAction.FAIL_TO_START
        }
    }
})

Parameterized build configurations

object Tests : BuildType({
    name = "Tests"
    
    params {
        param("env.NODE_VERSION", "18")
        param("env.TEST_SUITE", "all")
        password("env.API_KEY", "credentialsJSON:your-credentials-id")
    }
    
    steps {
        script {
            scriptContent = """
                nvm use %env.NODE_VERSION%
                npm ci
                npm test -- --suite=%env.TEST_SUITE%
            """.trimIndent()
        }
    }
})

Test History and Flaky Test Detection

TeamCity tracks test history automatically. Navigate to Tests tab in any build to see:

• Test duration trends
• Pass/fail history across recent builds
• Investigation assignments (assign failing tests to developers)

Flaky test identification

TeamCity automatically marks tests as flaky when they pass and fail on the same code revision. In Project → Flaky Tests, view all tests with inconsistent results.

For flaky tests:

1. Click the test name to see its history
2. Use Assign Investigation to track who is fixing it
3. Configure retry (see above) as a temporary mitigation

Muting tests

For known broken tests that you don't want to fail builds:

1. Click the test in the Tests tab
2. Select Mute with an optional expiration date
3. The test still runs but doesn't fail the build

Use muting sparingly — it masks real failures.

Build Artifacts

Publish test reports, coverage data, and other artifacts:

object Tests : BuildType({
    artifactRules = """
        test-results/** => test-results.zip
        coverage/** => coverage.zip
        target/surefire-reports/** => surefire-reports.zip
    """.trimIndent()
})

Artifacts appear in the Artifacts tab of each build and can be downloaded or passed to dependent builds.

Notifications and Failure Policies

Configure build failure notification via email, Slack, or other channels:

In Kotlin DSL:

object Tests : BuildType({
    failureConditions {
        testFailure = true
        nonZeroExitCode = true
        executionTimeoutMin = 60
        
        failOnMetricChange {
            metric = BuildFailureOnMetric.MetricType.TEST_FAILED_COUNT
            threshold = 5.0
            units = BuildFailureOnMetric.MetricUnit.DEFAULT_UNIT
            comparison = BuildFailureOnMetric.MetricComparison.MORE
        }
    }
})

Conclusion

TeamCity's test reporting capabilities set it apart from simpler CI tools. Automatic flaky test detection, investigation tracking, and historical trend analysis reduce the overhead of maintaining a large test suite.

The Kotlin DSL brings build configurations into version control, making them reviewable and reproducible. Combined with matrix builds and parallel execution, TeamCity can handle test suites of any scale.

Start with the basic configuration for your build tool (Maven, Gradle, Jest, pytest), add JUnit XML reporting, and then layer in parallelism and retry policies as your test suite grows.