Testing

Split.io Feature Flag Testing: Strategies, Mocks, and CI/CD Patterns

HelpMeTest

HelpMeTest

— 5 min read
Split.io Feature Flag Testing: Strategies, Mocks, and CI/CD Patterns

Split.io combines feature flags, experimentation, and observability in a single platform. Its treatment-based model — where flags return named treatments like "on", "off", or custom variant names — requires test strategies that go beyond simple boolean checks. This guide covers Split.io-specific testing patterns from unit tests through production monitoring.

Understanding Split.io's Testing Model

Split.io uses "treatments" rather than simple boolean flags. A split (flag) returns a string treatment based on targeting rules:

const treatment = client.getTreatment('user-123', 'new-checkout');
// Returns: 'on', 'off', 'control', or a custom treatment name

This means your tests must cover:

  • Each named treatment, not just on/off
  • The control treatment (returned when SDK is uninitialized or the split doesn't exist)
  • Treatment configurations (JSON payloads attached to treatments)
  • Impression logging (for experiment tracking)

Unit Testing with the Split SDK Mock

JavaScript: In-Memory Client

const SplitFactory = require('@splitsoftware/splitio').SplitFactory;

// Initialize with localhost mode for unit tests
const factory = SplitFactory({
  core: {
    authorizationKey: 'localhost',
  },
  features: {
    'new-checkout': 'on',
    'pricing-experiment': 'variant-b',
    'beta-dashboard': 'off',
  },
});

const client = factory.client();

describe('CheckoutService', () => {
  it('shows new checkout UI for "on" treatment', () => {
    const service = new CheckoutService(client);
    const ui = service.getCheckoutUI('user-123');
    expect(ui).toBe('new-checkout-v2');
  });

  it('shows legacy checkout for "off" treatment', () => {
    // Override treatment for this test
    const localFactory = SplitFactory({
      core: { authorizationKey: 'localhost' },
      features: { 'new-checkout': 'off' },
    });
    const localClient = localFactory.client();

    const service = new CheckoutService(localClient);
    const ui = service.getCheckoutUI('user-123');
    expect(ui).toBe('legacy-checkout');
  });
});

Python: Localhost Mode

import splitio
from splitio import get_factory

def create_test_client(treatments):
    """Create a Split client with predefined treatments for testing."""
    factory = get_factory(
        'localhost',
        config={
            'splitFile': None,
            'localhostModeEnabled': True,
        }
    )
    # In localhost mode, set treatments programmatically
    return factory.client()

Java: Mocking the SplitClient

import io.split.client.SplitClient;
import static org.mockito.Mockito.*;

@ExtendWith(MockitoExtension.class)
class CheckoutServiceTest {
    @Mock
    private SplitClient splitClient;

    @Test
    void showsNewCheckout_whenTreatmentIsOn() {
        when(splitClient.getTreatment("user-123", "new-checkout"))
            .thenReturn("on");

        CheckoutService service = new CheckoutService(splitClient);
        assertEquals("new-checkout-v2", service.getCheckoutUI("user-123"));
    }

    @Test
    void showsLegacyCheckout_whenTreatmentIsOff() {
        when(splitClient.getTreatment("user-123", "new-checkout"))
            .thenReturn("off");

        CheckoutService service = new CheckoutService(splitClient);
        assertEquals("legacy-checkout", service.getCheckoutUI("user-123"));
    }

    @Test
    void showsLegacyCheckout_whenTreatmentIsControl() {
        // 'control' is returned when SDK is uninitialized or split doesn't exist
        when(splitClient.getTreatment("user-123", "new-checkout"))
            .thenReturn("control");

        CheckoutService service = new CheckoutService(splitClient);
        // control should degrade gracefully to legacy experience
        assertEquals("legacy-checkout", service.getCheckoutUI("user-123"));
    }
}

The control treatment test is critical. If your code only handles "on" and "off", a control treatment will fall through to undefined behavior.

Testing Treatment Configurations

Split.io treatments can carry JSON configuration payloads:

describe('Treatment configuration', () => {
  it('uses button text from treatment config', async () => {
    const client = createTestClient({
      'cta-experiment': {
        treatment: 'variant-b',
        config: JSON.stringify({ buttonText: 'Get Started Free', color: 'green' }),
      },
    });

    const result = client.getTreatmentWithConfig('user-123', 'cta-experiment');
    expect(result.treatment).toBe('variant-b');

    const config = JSON.parse(result.config);
    expect(config.buttonText).toBe('Get Started Free');
  });

  it('handles null config gracefully', () => {
    const result = client.getTreatmentWithConfig('user-456', 'cta-experiment');
    // treatment = 'off', config = null
    expect(result.config).toBeNull();

    // Code must handle null config without crashing
    const config = result.config ? JSON.parse(result.config) : {};
    expect(config.buttonText).toBeUndefined();
  });
});

Always test the null config case — Split returns null for configurations when a treatment has no config defined.

Testing Multiple Evaluations (getTreatments)

For pages that check multiple flags at once, use getTreatments and test it:

it('evaluates multiple treatments at once', () => {
  const treatments = client.getTreatments('user-123', [
    'new-checkout',
    'pricing-experiment',
    'beta-dashboard',
  ]);

  expect(treatments['new-checkout']).toBe('on');
  expect(treatments['pricing-experiment']).toBe('variant-b');
  expect(treatments['beta-dashboard']).toBe('off');
});

it('returns control for all splits when client is not ready', () => {
  const uninitializedClient = SplitFactory({
    core: { authorizationKey: 'localhost' },
    features: {},
  }).client();

  const treatments = uninitializedClient.getTreatments('user-123', [
    'new-checkout',
    'beta-dashboard',
  ]);

  // All unknown splits return 'control'
  expect(treatments['new-checkout']).toBe('control');
  expect(treatments['beta-dashboard']).toBe('control');
});

Testing Impression Logging

Split.io logs impressions for each treatment evaluation — this powers your experiment analysis. Test that impressions are fired correctly:

describe('Impression logging', () => {
  it('logs impression on treatment evaluation', async () => {
    const impressions = [];
    const factory = SplitFactory({
      core: { authorizationKey: 'localhost' },
      features: { 'new-checkout': 'on' },
      impressionListener: {
        logImpression: (data) => impressions.push(data),
      },
    });

    const client = factory.client();
    await client.ready();

    client.getTreatment('user-123', 'new-checkout');
    await new Promise(r => setTimeout(r, 100)); // wait for async flush

    expect(impressions).toHaveLength(1);
    expect(impressions[0]).toMatchObject({
      impression: {
        feature: 'new-checkout',
        treatment: 'on',
        keyName: 'user-123',
      },
    });
  });
});

If impressions aren't logged, your experiment results will be incomplete.

Integration Testing Against Split.io API

For integration tests, use Split.io's API with a dedicated API key in a non-production environment:

// jest.setup.js
const SplitFactory = require('@splitsoftware/splitio').SplitFactory;

let splitFactory;
let splitClient;

beforeAll(async () => {
  splitFactory = SplitFactory({
    core: {
      authorizationKey: process.env.SPLIT_TEST_API_KEY,
    },
    startup: {
      readyTimeout: 10,
    },
  });

  splitClient = splitFactory.client();
  await splitClient.ready();
});

afterAll(async () => {
  await splitFactory.destroy();
});

Pre-setting Treatments via Split Admin API

# Use the Split Admin API to set treatments for integration tests
curl -X PUT \
  <span class="hljs-string">"https://api.split.io/internal/api/v2/splits/ws/{workspace_id}/environments/Integration/definitions/new-checkout" \
  -H <span class="hljs-string">"Authorization: Bearer $SPLIT_ADMIN_API_KEY" \
  -H <span class="hljs-string">"Content-Type: application/json" \
  -d <span class="hljs-string">'{
    "treatments": [{"name": "on"}, {"name": "off"}],
    "defaultTreatment": "off",
    "rules": []
  }'

Testing Attribute-Based Targeting

Split.io evaluates targeting rules based on user attributes. Test that your application passes attributes correctly:

describe('Attribute targeting', () => {
  it('passes user attributes to treatment evaluation', () => {
    const capturedAttributes = {};
    
    const mockClient = {
      getTreatment: (key, split, attributes) => {
        Object.assign(capturedAttributes, attributes);
        return 'on';
      }
    };

    const service = new FeatureService(mockClient);
    service.getFeatureForUser({
      id: 'user-123',
      plan: 'pro',
      country: 'US',
      accountAge: 365,
    });

    expect(capturedAttributes).toMatchObject({
      plan: 'pro',
      country: 'US',
      accountAge: 365,
    });
  });
});

If attributes aren't passed, targeting rules that depend on them will silently evaluate using the default rule — often returning the wrong treatment.

End-to-End Testing

Deterministic E2E Test Users

Create dedicated test users with fixed treatments in Split.io:

  1. In Split.io, go to your split definition
  2. Add an "Individual Targets" rule: user key e2e-test-user → treatment on
  3. Your E2E tests always authenticate as e2e-test-user
As e2e-test-user
Navigate to /checkout
Verify the new checkout form is displayed (treatment: on)
Complete a test purchase
Verify order confirmation page loads

HelpMeTest for Continuous E2E Validation

Run your critical user journeys on a schedule with HelpMeTest to catch split-related regressions:

  • If a Split.io SDK update breaks treatment evaluation, you'll know within minutes
  • If a targeting rule change accidentally affects your E2E test user, the test will fail immediately
  • 24/7 monitoring means you catch issues at 3am, not when customers complain at 9am

CI/CD Integration

Localhost Mode for CI

For CI pipelines, always use localhost mode to avoid network dependencies:

# .github/workflows/test.yml
env:
  SPLIT_API_KEY: localhost
  SPLIT_FEATURES_FILE: ./test/fixtures/splits.yaml

steps:
  - name: Run tests
    run: npm test

Create test/fixtures/splits.yaml with your test flag states:

# Split.io localhost features file
new-checkout: on
pricing-experiment: variant-b
beta-dashboard: off

Validating Split Definitions in CI

// scripts/validate-splits.js
const usedSplits = extractSplitRefs('./src'); // grep getTreatment calls
const definedSplits = await fetchSplitDefinitions(); // Split Admin API

const missingInSplit = usedSplits.filter(s => !definedSplits.includes(s));
if (missingInSplit.length > 0) {
  console.error('Splits referenced in code but not defined:', missingInSplit);
  process.exit(1);
}

Testing SDK Readiness

describe('SDK readiness', () => {
  it('returns control when SDK is not ready', () => {
    const factory = SplitFactory({
      core: { authorizationKey: 'bad-key' },
      startup: { readyTimeout: 0 },
    });
    const client = factory.client();
    
    // Don't await ready() — simulate uninitialized state
    const treatment = client.getTreatment('user-123', 'new-checkout');
    
    // Must return 'control', not throw
    expect(treatment).toBe('control');
    
    factory.destroy();
  });

  it('resolves ready promise on initialization', async () => {
    const factory = SplitFactory({
      core: { authorizationKey: 'localhost' },
      features: { 'test-split': 'on' },
    });
    const client = factory.client();
    
    await expect(client.ready()).resolves.toBeUndefined();
    
    factory.destroy();
  });
});

Summary

Testing Split.io integrations requires covering all named treatments including control, validating treatment configuration JSON including null cases, testing getTreatments for multi-flag evaluations, verifying impression logging fires correctly, testing attribute propagation for targeting rules, and using localhost mode for deterministic CI tests. The control treatment handling is the most commonly missed test — always add it, because that's what users see when something goes wrong.

Read more