Microservices Integration Testing: Strategies That Actually Work
Integration testing for microservices is where good intentions go to die. Teams start with aspirations of comprehensive test coverage across service boundaries, then hit the reality: standing up five services, three databases, a message queue, and a cache just to run a single test is slow, fragile, and expensive to maintain. So they scale back to pure unit tests and hope for the best. Then production breaks in exactly the ways integration tests would have caught.
The mistake is treating integration testing as an all-or-nothing proposition. The services that need to be tested together are not all of them — they're the ones that share a boundary. The infrastructure that needs to be real is not all of it — only the parts where mock behavior diverges meaningfully from real behavior. Get these distinctions right and you can build a fast, reliable integration test suite that actually runs in CI.
This guide covers the strategies that work in practice: the integration test pyramid for microservices, Testcontainers for realistic dependency management, service boundary testing, API gateway testing, and data consistency verification.
The Microservices Integration Test Pyramid
The classic test pyramid (many unit tests, fewer integration tests, few E2E tests) needs adaptation for microservices. The key insight is that service boundaries are the primary integration points that matter.
For each service, structure tests as follows:
Level 1 — In-process integration tests (fast, numerous): Test each service with real implementations of its own dependencies (database, cache) but mock external services. These run in under a second each and cover the majority of your integration surface.
Level 2 — Boundary integration tests (medium speed, targeted): Test the interface between two specific services — real implementations of both, mocked everything else. These are your contract-verification tests that confirm two services can actually communicate.
Level 3 — Slice integration tests (slower, few): Test a meaningful vertical slice end-to-end: one user journey through three or four services. Not a full E2E test — just enough to verify the critical path works when the pieces are assembled.
Here's the distribution target: for every 1 slice integration test, write 5-10 boundary tests, and 20-50 in-process integration tests. This gives comprehensive coverage without the overhead of spinning up your full system for every test.
Testcontainers: Real Dependencies Without Infrastructure Overhead
Testcontainers is the most important tool in the microservices integration testing toolkit. It spins up Docker containers for your actual dependencies — PostgreSQL, Redis, Kafka, Elasticsearch — on demand, isolated per test run, and tears them down automatically. No shared test databases, no state leaking between runs, no "works on my machine" problems.
Here's a complete in-process integration test for an order service with a real PostgreSQL database:
// OrderServiceIntegrationTest.java
@SpringBootTest
@Testcontainers
@Transactional
class OrderServiceIntegrationTest {
@Container
static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:15")
.withDatabaseName("orders_test")
.withUsername("test")
.withPassword("test")
.withInitScript("db/schema.sql"); // Run schema migrations on startup
@DynamicPropertySource
static void configureProperties(DynamicPropertyRegistry registry) {
registry.add("spring.datasource.url", postgres::getJdbcUrl);
registry.add("spring.datasource.username", postgres::getUsername);
registry.add("spring.datasource.password", postgres::getPassword);
}
@Autowired
private OrderService orderService;
@Autowired
private OrderRepository orderRepository;
// Mock the external payment service — we're testing order logic, not payment
@MockBean
private PaymentServiceClient paymentServiceClient;
@MockBean
private InventoryServiceClient inventoryServiceClient;
@BeforeEach
void setUp() {
// Set up consistent mock behavior
when(paymentServiceClient.charge(any(), any()))
.thenReturn(new PaymentResult("txn_test_123", "success"));
when(inventoryServiceClient.reserve(any(), anyInt()))
.thenReturn(new ReservationResult(true, "res_test_456"));
}
@Test
void createOrder_persistsOrderAndReturnsId() {
CreateOrderRequest request = CreateOrderRequest.builder()
.customerId("cust_123")
.items(List.of(
new OrderItemRequest("PROD-1", 2, new BigDecimal("29.99")),
new OrderItemRequest("PROD-2", 1, new BigDecimal("49.99"))
))
.shippingAddress(new Address("123 Main St", "Springfield", "IL", "62701"))
.build();
Order created = orderService.createOrder(request);
assertThat(created.getId()).isNotNull();
assertThat(created.getStatus()).isEqualTo(OrderStatus.PENDING);
assertThat(created.getTotalAmount()).isEqualByComparingTo(new BigDecimal("109.97"));
// Verify persistence — reload from database
Order persisted = orderRepository.findById(created.getId()).orElseThrow();
assertThat(persisted.getCustomerId()).isEqualTo("cust_123");
assertThat(persisted.getItems()).hasSize(2);
}
@Test
void createOrder_rollsBackOnPaymentFailure() {
when(paymentServiceClient.charge(any(), any()))
.thenThrow(new PaymentFailedException("Card declined"));
CreateOrderRequest request = CreateOrderRequest.builder()
.customerId("cust_declined")
.items(List.of(new OrderItemRequest("PROD-1", 1, new BigDecimal("29.99"))))
.build();
assertThatThrownBy(() -> orderService.createOrder(request))
.isInstanceOf(PaymentFailedException.class);
// Critical: verify the order was NOT persisted (transaction rolled back)
long orderCount = orderRepository.countByCustomerId("cust_declined");
assertThat(orderCount).isZero();
// And inventory reservation should have been released
verify(inventoryServiceClient).releaseReservation(any());
}
@Test
void getOrders_returnsPagedResults_sortedByCreatedAtDesc() {
// Create 15 orders with distinct timestamps
for (int i = 0; i < 15; i++) {
orderRepository.save(Order.builder()
.customerId("cust_paged")
.status(OrderStatus.PENDING)
.totalAmount(new BigDecimal("10.00"))
.createdAt(Instant.now().minusSeconds(i * 60))
.build());
}
Page<Order> page1 = orderService.getOrdersByCustomer("cust_paged", 0, 10);
Page<Order> page2 = orderService.getOrdersByCustomer("cust_paged", 1, 10);
assertThat(page1.getContent()).hasSize(10);
assertThat(page2.getContent()).hasSize(5);
assertThat(page1.getTotalElements()).isEqualTo(15);
// Verify ordering: most recent first
List<Instant> timestamps = page1.getContent().stream()
.map(Order::getCreatedAt)
.toList();
for (int i = 0; i < timestamps.size() - 1; i++) {
assertThat(timestamps.get(i)).isAfterOrEqualTo(timestamps.get(i + 1));
}
}
}The @Transactional annotation rolls back all database changes after each test, keeping the database clean without truncating tables between tests. This is significantly faster than setup/teardown scripts.
For tests that need Redis (caching, rate limiting, session storage):
@Container
static GenericContainer<?> redis = new GenericContainer<>(
DockerImageName.parse("redis:7-alpine")
)
.withExposedPorts(6379)
.withCommand("redis-server", "--maxmemory", "256mb", "--maxmemory-policy", "allkeys-lru");
@DynamicPropertySource
static void redisProperties(DynamicPropertyRegistry registry) {
registry.add("spring.data.redis.host", redis::getHost);
registry.add("spring.data.redis.port", () -> redis.getMappedPort(6379));
}Testing Service Boundaries
Boundary tests verify that two specific services communicate correctly. The key challenge is schema compatibility — the data one service sends must be parseable and meaningful to the other.
Here's a boundary test between the API gateway and the order service:
# test_gateway_order_boundary.py
import pytest
import requests
from testcontainers.compose import DockerCompose
@pytest.fixture(scope="session")
def services():
"""Start API gateway + order service with a test database."""
compose = DockerCompose(
"tests/fixtures/",
compose_file_name="docker-compose.boundary-test.yml",
pull=False,
)
with compose:
compose.wait_for("http://localhost:8080/health")
compose.wait_for("http://localhost:8081/health")
yield {"gateway": "http://localhost:8080", "order_service": "http://localhost:8081"}
class TestGatewayOrderBoundary:
def test_gateway_correctly_translates_request_to_order_service(self, services):
"""
The gateway accepts a public API request and forwards it to the
order service with the correct internal format.
"""
# Public API request (what clients send to the gateway)
public_request = {
"cart": {
"items": [
{"productId": "prod-abc-123", "qty": 2}
]
},
"deliveryAddress": {
"street": "123 Main St",
"city": "Springfield"
}
}
resp = requests.post(
f"{services['gateway']}/v1/orders",
json=public_request,
headers={"Authorization": "Bearer test-token"}
)
assert resp.status_code == 201, f"Expected 201, got {resp.status_code}: {resp.text}"
body = resp.json()
# Gateway should return public-facing order representation
assert "orderId" in body
assert "status" in body
assert "estimatedDelivery" in body
# Should NOT expose internal fields
assert "internalOrderId" not in body
assert "paymentGatewayRef" not in body
def test_gateway_maps_order_service_errors_to_public_error_format(self, services):
"""
Order service 422 (validation error) should be translated to a
public-friendly error response, not leaked as-is.
"""
invalid_request = {
"cart": {"items": []}, # Empty cart — invalid
"deliveryAddress": {"street": "123 Main St", "city": "Springfield"}
}
resp = requests.post(
f"{services['gateway']}/v1/orders",
json=invalid_request,
headers={"Authorization": "Bearer test-token"}
)
assert resp.status_code == 400
error = resp.json()
assert "error" in error
assert "message" in error
# Public error format, not internal Spring validation format
assert "timestamp" in error
assert "path" in error
# Should NOT expose internal service details
assert "stackTrace" not in str(error)
assert "internal" not in str(error).lower()
def test_gateway_enforces_authentication(self, services):
"""Requests without valid tokens must be rejected at the gateway."""
resp = requests.post(
f"{services['gateway']}/v1/orders",
json={"cart": {"items": [{"productId": "prod-1", "qty": 1}]}}
# No Authorization header
)
assert resp.status_code == 401
def test_gateway_rate_limiting_per_customer(self, services):
"""Gateway should rate-limit individual customers, not block all traffic."""
headers = {"Authorization": "Bearer rate-limit-test-token"}
# Send 20 rapid requests from the same customer
responses = [
requests.post(
f"{services['gateway']}/v1/orders",
json={"cart": {"items": [{"productId": "prod-1", "qty": 1}]}},
headers=headers
)
for _ in range(20)
]
status_codes = [r.status_code for r in responses]
# Some should succeed, some should be rate-limited (429)
assert 429 in status_codes, "Rate limiting not enforced"
# But not all should fail — other customers should still work
assert 201 in status_codes or 202 in status_codes, "All requests blocked (too aggressive)"Data Consistency Testing Across Services
Distributed data consistency is one of the hardest problems in microservices. When an order is created, the inventory must be reserved, the payment must be charged, and the notification must be sent — all consistently. Testing this end-to-end requires verifying the state across multiple data stores.
# test_order_data_consistency.py
import pytest
import time
import psycopg2
import redis
import requests
class TestOrderDataConsistency:
"""
Tests that verify data consistency across the order, inventory, and payment
services after a completed order workflow.
"""
def test_successful_order_consistent_across_all_services(
self, gateway_url, order_db, inventory_db, redis_client
):
"""
After a successful order:
- order-service DB: order exists with status 'confirmed'
- inventory-service DB: stock reduced by ordered quantity
- Redis: order summary cached with correct data
"""
initial_stock = self._get_stock(inventory_db, "PROD-1")
# Create an order
resp = requests.post(
f"{gateway_url}/v1/orders",
json={
"cart": {"items": [{"productId": "PROD-1", "qty": 2}]},
"deliveryAddress": {"street": "123 Main St", "city": "Springfield"}
},
headers={"Authorization": "Bearer test-user-token"}
)
assert resp.status_code == 201
order_id = resp.json()["orderId"]
# Wait for async processing to complete (inventory reservation, etc.)
self._wait_for_order_status(order_db, order_id, "confirmed", timeout=15)
# Verify order service state
order_row = self._get_order(order_db, order_id)
assert order_row["status"] == "confirmed"
assert order_row["total_amount"] is not None
# Verify inventory service state
final_stock = self._get_stock(inventory_db, "PROD-1")
assert final_stock == initial_stock - 2, (
f"Expected stock to decrease by 2: {initial_stock} -> {final_stock}"
)
# Verify cache is consistent with database
cached = redis_client.hgetall(f"order:{order_id}")
if cached: # Cache is eventually consistent — may not exist yet
assert cached[b"status"] == b"confirmed"
assert cached[b"order_id"] == order_id.encode()
def test_failed_payment_leaves_no_partial_state(
self, gateway_url, order_db, inventory_db
):
"""
When payment fails, the system must be in a consistent state:
- No order record in order-service DB
- Inventory reservation released (stock unchanged)
- No partial payment record
"""
initial_stock = self._get_stock(inventory_db, "PROD-1")
resp = requests.post(
f"{gateway_url}/v1/orders",
json={
"cart": {"items": [{"productId": "PROD-1", "qty": 1}]},
"deliveryAddress": {"street": "123 Main St", "city": "Springfield"}
},
headers={"Authorization": "Bearer declined-card-token"} # Token mapped to failing payment
)
assert resp.status_code in [402, 400], f"Expected payment failure, got {resp.status_code}"
# Wait for any async cleanup
time.sleep(3)
# No order should exist
order_id = resp.json().get("orderId")
if order_id:
order = self._get_order(order_db, order_id)
assert order is None or order["status"] == "failed"
# Stock must be restored
final_stock = self._get_stock(inventory_db, "PROD-1")
assert final_stock == initial_stock, (
f"Inventory not restored after payment failure: {initial_stock} -> {final_stock}"
)
def _wait_for_order_status(self, db_conn, order_id, expected_status, timeout):
deadline = time.time() + timeout
while time.time() < deadline:
order = self._get_order(db_conn, order_id)
if order and order["status"] == expected_status:
return
time.sleep(0.5)
raise AssertionError(
f"Order {order_id} did not reach status '{expected_status}' within {timeout}s"
)
def _get_order(self, db_conn, order_id):
with db_conn.cursor() as cur:
cur.execute("SELECT * FROM orders WHERE id = %s", (order_id,))
row = cur.fetchone()
if row:
return dict(zip([desc[0] for desc in cur.description], row))
return None
def _get_stock(self, db_conn, sku):
with db_conn.cursor() as cur:
cur.execute("SELECT available_quantity FROM inventory WHERE sku = %s", (sku,))
row = cur.fetchone()
return row[0] if row else 0API Gateway Testing
The API gateway deserves its own test focus because it's the single point through which all external traffic passes. Beyond the boundary tests above, there are gateway-specific concerns worth testing independently:
// api-gateway.test.js
const supertest = require('supertest');
const { createGateway } = require('../src/gateway');
describe('API Gateway', () => {
let gateway;
beforeAll(async () => {
gateway = await createGateway({
orderServiceUrl: 'http://order-service-mock:8080',
inventoryServiceUrl: 'http://inventory-service-mock:8080',
});
});
describe('Request routing', () => {
it('routes /v1/orders to order service', async () => {
const resp = await supertest(gateway)
.post('/v1/orders')
.set('Authorization', 'Bearer valid-token')
.send({ cart: { items: [] } });
// Even an empty cart validation error confirms routing worked
expect([400, 422]).toContain(resp.status);
});
it('routes /v1/products to product service, not order service', async () => {
const resp = await supertest(gateway)
.get('/v1/products/prod-123')
.set('Authorization', 'Bearer valid-token');
// Response should come from product service (different response shape)
if (resp.status === 200) {
expect(resp.body).toHaveProperty('sku');
expect(resp.body).not.toHaveProperty('orderId');
}
});
});
describe('Timeout handling', () => {
it('returns 504 when upstream service exceeds timeout', async () => {
// order-service-mock configured to delay 10s for /orders/slow
const resp = await supertest(gateway)
.get('/v1/orders/slow-endpoint')
.set('Authorization', 'Bearer valid-token')
.timeout(15000);
expect(resp.status).toBe(504);
expect(resp.body.error).toBe('Gateway Timeout');
});
});
describe('Response transformation', () => {
it('removes internal fields from upstream responses', async () => {
const resp = await supertest(gateway)
.get('/v1/orders/ord_public_123')
.set('Authorization', 'Bearer valid-token');
if (resp.status === 200) {
// These internal fields must never reach external clients
expect(resp.body).not.toHaveProperty('internalId');
expect(resp.body).not.toHaveProperty('paymentGatewayRef');
expect(resp.body).not.toHaveProperty('_internal');
}
});
it('adds correlation ID to all responses', async () => {
const resp = await supertest(gateway)
.get('/v1/orders/ord_123')
.set('Authorization', 'Bearer valid-token');
expect(resp.headers['x-correlation-id']).toBeDefined();
expect(resp.headers['x-correlation-id']).toMatch(/^[0-9a-f-]{36}$/);
});
});
describe('Circuit breaker', () => {
it('returns 503 when order service circuit is open', async () => {
// Trigger circuit breaker by making many failing requests
for (let i = 0; i < 10; i++) {
await supertest(gateway)
.get('/v1/orders/force-error')
.set('Authorization', 'Bearer valid-token');
}
// Circuit should now be open
const resp = await supertest(gateway)
.get('/v1/orders/ord_normal')
.set('Authorization', 'Bearer valid-token');
expect(resp.status).toBe(503);
expect(resp.body.error).toContain('Service temporarily unavailable');
});
});
});Keeping Integration Tests Fast
The biggest killer of integration test adoption is speed. Tests that take 20 minutes to run get disabled. Some techniques that keep Testcontainers-based tests fast:
Shared containers across tests — Use @ClassRule (JUnit 4) or static @Container (JUnit 5) to start containers once per test class, not once per test. The overhead of starting a PostgreSQL container (2-5 seconds) multiplied by 100 tests becomes significant.
Parallel test execution — Testcontainers allocates separate ports per container instance, so parallel test classes don't conflict. Configure Maven Surefire or Gradle to run test classes in parallel.
Reuse containers across test runs (development only) — Testcontainers supports withReuse(true) for development, where the container persists between runs and isn't torn down. This cuts startup time to near zero for local development. Disable in CI to ensure isolation.
@Container
static PostgreSQLContainer<?> postgres = new PostgreSQLContainer<>("postgres:15")
.withReuse(Boolean.parseBoolean(System.getenv().getOrDefault("TC_REUSE", "false")));Seed data via SQL scripts — Prefer withInitScript("schema.sql") over application-layer data setup. SQL scripts run at container startup and are an order of magnitude faster than ORM-level inserts.
What to Test, What to Skip
Integration tests are expensive per-test compared to unit tests. Make deliberate choices about what earns integration test coverage:
Worth testing at the integration level:
- Database query correctness (indexes, joins, transactions)
- Cache interaction patterns (cache hits, misses, eviction)
- Message serialization/deserialization round-trips
- External API response parsing (mock the external API, but test the parsing code)
- Transaction rollback behavior
- Data consistency across service boundaries
Better tested at the unit level:
- Business logic (calculations, state machines, validation rules)
- Error handling branches
- Data transformation logic
Better tested at the E2E level:
- Full user journeys
- Browser-specific behavior
- Performance under load
The integration test pyramid for microservices is not about coverage percentage — it's about testing the right things at the right level. Get the layer right, and your tests will be fast enough to run on every commit and reliable enough to trust when they pass.
