Mobile Testing

Advanced BitBar Features: Real Device Testing at Scale

HelpMeTest

HelpMeTest

— 6 min read
Advanced BitBar Features: Real Device Testing at Scale

Once you've run your first tests on BitBar and confirmed the basic integration works, the interesting questions become operational: how do you scale to hundreds of tests without spending a fortune, how do you get useful performance data out of device sessions, and how do you manage a growing suite across multiple device configurations? This post covers BitBar's advanced capabilities with the detail that gets omitted from introductory guides.

Device Groups and Private Device Pools

The default BitBar experience uses shared public devices — devices that multiple customers access sequentially. For teams at scale, this creates two problems: unpredictable availability during peak hours, and devices that accumulate state between sessions despite the cleaning process.

Private device pools solve both problems. A private pool reserves physical devices exclusively for your organization. No queuing, no shared state concerns, and consistent device condition across runs.

Private pools are an enterprise add-on and priced accordingly — you're effectively renting dedicated hardware. The economics make sense when you're consuming hundreds of device-hours per month; they don't make sense for occasional testing.

Device groups are the more accessible alternative. You define a logical group of devices by filter criteria — OS version range, manufacturer, screen size class — and BitBar routes your test runs to any available device matching those criteria within your subscription tier.

Creating a device group via API:

curl -X POST \
  -H "Authorization: Bearer $BITBAR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Android 12-13 High-End",
    "deviceFilters": [
      {"type": "OS_VERSION", "value": "12.0", "operator": "GTE"},
      {"type": "OS_VERSION", "value": "13.99", "operator": "LTE"},
      {"type": "MEMORY", "value": "6144", "operator": "GTE"}
    ]
  }' \
  https://cloud.bitbar.com/api/me/device-groups

Parallel Test Sharding

Running 500 tests sequentially on a single device takes hours. Sharding distributes tests across parallel device sessions to cut wall-clock time.

BitBar supports two sharding approaches:

Framework-level sharding: Use your test framework's native sharding capabilities. For Espresso, this means using --numShards and --shardIndex in the runner arguments. Each parallel BitBar session gets different shard parameters.

# BitBar run configuration with sharding
testRunParameters:
  - key: "--numShards"
    value: "4"
  - key: "--shardIndex"
    value: "0"  # Each parallel session gets a different index

Test list sharding: Extract your test list, split it into N chunks, and pass each chunk to a separate BitBar run. More control, more setup:

#!/bin/bash
# Get all test names
TEST_LIST=$(./gradlew listAllTests --no-daemon -q)
TOTAL=$(echo "$TEST_LIST" | wc -l)
SHARDS=8
SHARD_SIZE=$((TOTAL / SHARDS))

# Split into chunks and launch parallel runs
for i in $(seq 0 $((SHARDS - 1))); do
  START=$((i * SHARD_SIZE))
  SHARD_TESTS=$(echo "$TEST_LIST" | tail -n +$((START + 1)) | head -n $SHARD_SIZE)
  
  # Upload test list and trigger BitBar run
  launch_bitbar_run "$SHARD_TESTS" &
done

wait  # Wait for all parallel runs to complete

What to watch: Sharding improves wall time but can hide flakiness. A test that fails intermittently will appear in only one shard, making the failure pattern harder to see across runs. Monitor per-shard pass rates, not just overall.

Performance Profiling During Tests

BitBar captures device performance metrics during test sessions on physical devices. This is one area where real devices provide data that emulators cannot reliably reproduce.

Available metrics:

  • CPU utilization (system-wide and per-process)
  • Memory usage and garbage collection events (Android)
  • Battery drain rate
  • Network throughput
  • GPU utilization (on supported devices)
  • Frame rate and dropped frames

Access performance data via the API after a run completes:

# Get performance data for a device session
curl -H "Authorization: Bearer $BITBAR_API_KEY" \
  "https://cloud.bitbar.com/api/me/runs/$RUN_ID/device-sessions/$SESSION_ID/performance"

Practical use: Set up a dedicated performance test suite that exercises key user flows — app launch, list scrolling, image loading, purchase flow. Run this suite against your release builds and track the metrics over time. A regression in startup time or memory usage during the purchase flow is worth catching before the release.

The data comes back as time-series JSON that you can feed into your own monitoring or visualization tools.

Network Condition Simulation

Real users don't all have 5G in a major city. BitBar supports network condition simulation to test your app's behavior under constrained or unreliable connections.

Available simulation profiles vary by device OS and model, but generally include:

  • 2G (250 kbps download, 50 kbps upload, 300ms latency)
  • 3G (1.5 Mbps down, 384 kbps up, 150ms latency)
  • 4G/LTE (15 Mbps down, 5 Mbps up, 50ms latency)
  • Custom profiles with specified bandwidth and latency

Configure via run capabilities:

DesiredCapabilities caps = new DesiredCapabilities();
caps.setCapability("bitbar_apiKey", apiKey);
caps.setCapability("bitbar_device", "Samsung Galaxy A52");
caps.setCapability("bitbar_networkConditions", "2G");
// or custom:
caps.setCapability("bitbar_networkProfile", "{\"downBandwidth\": 500, \"upBandwidth\": 100, \"latency\": 200}");

Testing strategy: Use network simulation for a targeted subset of tests, not your entire suite. Focus on:

  • App launch and initial data load
  • Image-heavy screens
  • Forms that submit data
  • Features that should work offline or degrade gracefully

Running your full 500-test suite under 2G conditions isn't useful — most of those tests aren't affected by network speed. Pick the 20-30 tests where network behavior matters.

Screenshot and Video Analysis at Scale

For large parallel runs, manual review of video recordings per test isn't practical. Build analysis workflows around the artifacts BitBar produces.

Automated screenshot comparison: Use the screenshots BitBar captures on failure as inputs to a comparison pipeline. Tools like pixelmatch or resemble.js can flag visual regressions automatically:

// Simplified screenshot comparison pipeline
const { createCanvas, loadImage } = require('canvas');
const pixelmatch = require('pixelmatch');

async function compareScreenshots(baselineUrl, currentUrl, threshold = 0.1) {
  const [baseline, current] = await Promise.all([
    loadImage(baselineUrl),
    loadImage(currentUrl)
  ]);
  
  const diff = createCanvas(baseline.width, baseline.height);
  const diffPixels = pixelmatch(
    baseline.data, current.data, diff.getContext('2d').createImageData(baseline.width, baseline.height).data,
    baseline.width, baseline.height,
    { threshold }
  );
  
  return {
    passed: diffPixels < (baseline.width * baseline.height * 0.01),
    diffPixelCount: diffPixels
  };
}

Failure categorization: Download the device logs from failed sessions and build a categorization pipeline. Common failure categories — network timeouts, ANRs, test framework timeouts, app crashes — each have distinct log signatures. Automating this categorization reduces the time spent triaging failures.

Managing Multiple Apps and Versions

Teams testing multiple apps or maintaining multiple versions in parallel need organizational discipline in BitBar.

Project structure: BitBar's project concept groups test runs. Use one project per app, with descriptive run naming that includes build version and branch:

def create_bitbar_run(app_id, version, branch, device_group_id):
    run_name = f"{app_id}-v{version}-{branch}-{datetime.now().strftime('%Y%m%d-%H%M')}"
    
    payload = {
        "name": run_name,
        "projectId": BITBAR_PROJECT_ID,
        "deviceGroupId": device_group_id,
        "files": [{"id": app_file_id}],
        "tags": [f"version:{version}", f"branch:{branch}"]
    }
    
    response = requests.post(
        "https://cloud.bitbar.com/api/me/runs",
        headers={"Authorization": f"Bearer {BITBAR_API_KEY}"},
        json=payload
    )
    return response.json()

File storage management: BitBar stores uploaded APKs/IPAs in your account. Without cleanup, storage accumulates. Set up automated cleanup of old builds:

# Delete files older than 30 days
curl -H "Authorization: Bearer $BITBAR_API_KEY" \
  "https://cloud.bitbar.com/api/me/files?limit=100&sort=createTime+asc" \
  | jq -r '.data[] | select(.createTime < (now - 2592000) * 1000) | .id' \
  | xargs -I{} curl -X DELETE \
    -H "Authorization: Bearer $BITBAR_API_KEY" \
    "https://cloud.bitbar.com/api/me/files/{}"

Test Retry and Flakiness Management

Flakiness in cloud device testing is real. Network latency, device allocation variability, and timing-sensitive tests all contribute. Rather than accepting flakiness as noise, build systems to measure and manage it.

Automatic retries: Configure BitBar to retry failed test cases automatically. Most teams use 2-3 retries before marking a test as permanently failed. This reduces noise from transient failures while still catching real regressions.

Flakiness tracking: Collect results across runs and identify tests that pass sometimes and fail other times with identical code. These tests need investment — either fixing the underlying flakiness or marking them as non-blocking while the fix is in progress.

# Simplified flakiness detection
def calculate_flakiness_score(test_name, recent_runs, window=20):
    results = [run['tests'][test_name] for run in recent_runs[-window:] 
               if test_name in run['tests']]
    
    if not results:
        return None
    
    failures = sum(1 for r in results if r == 'FAILED')
    passes = sum(1 for r in results if r == 'PASSED')
    
    if passes > 0 and failures > 0:
        return failures / len(results)  # Flakiness rate
    return 0  # Consistently passing or failing

Reporting and Metrics

At scale, anecdotal awareness of test health isn't enough. Build metrics that give you continuous visibility:

  • Pass rate by device: Which specific devices are failing most? Sometimes a single device model has a hardware or software issue that manifests as widespread test failures.
  • Pass rate by test: Which tests fail most frequently? High-failure tests need investment.
  • Run duration trends: Are runs getting slower? Growing suites without pruning can accumulate dead weight.
  • Device availability wait time: How long are jobs queuing? If consistently high, you may need more parallel slots.

Pull these from the BitBar API into your own dashboards or existing monitoring tools. BitBar's built-in dashboard shows this data, but integrating it with your existing observability stack usually provides more context.

Conclusion

BitBar's advanced features — private device pools, performance profiling, network simulation, and sharding — are genuinely useful at scale. The teams that get the most value out of cloud device testing are the ones that invest in the tooling around the platform: automated failure analysis, flakiness tracking, and metrics pipelines that make the data actionable. The platform provides the raw capabilities; the value comes from building operational discipline on top of them.

Read more

Start now free