AWS Device Farm: Complete Guide to Mobile App Testing

AWS Device Farm lets you run tests on real Android and iOS devices hosted in AWS data centers. Instead of managing a physical device lab, you upload your app and test scripts, pick devices, and run tests in parallel.

This guide covers how Device Farm works, how to set it up, and how to integrate it into your CI/CD pipeline.

What AWS Device Farm Offers

Real device testing: Tests run on physical hardware — not simulators. This matters for GPS, Bluetooth, camera, biometrics, and real-world performance.

Device selection: 200+ device configurations covering Android phones and tablets, iPhones, and iPads across multiple OS versions and manufacturers.

Two test modes:

• Automated testing — run your test framework scripts (Appium, XCTest, Espresso, Calabash, built-in fuzz)
• Remote access — interact with a real device via browser in real time for manual testing and debugging

Parallel execution: Run the same test suite on 10+ devices simultaneously. A suite that would take 4 hours on one device completes in 20 minutes.

Supported Test Frameworks

If you're starting fresh, Appium is the most portable choice — same test code runs on Android and iOS.

Setting Up Your First Test Run

Prerequisites

• AWS account with Device Farm enabled (available in us-west-2 region)
• Android APK or iOS IPA file
• Test package (test code + dependencies bundled as ZIP)

Using the AWS Console

1. Go to AWS Device Farm > Mobile Testing
2. Create a project
3. Click Create a new run
4. Upload your app (APK/IPA)
5. Select test framework and upload test package
6. Choose devices from the device pool
7. Configure execution settings (test timeout, device state)
8. Run

Packaging Appium Tests

Device Farm requires tests packaged in a specific ZIP structure:

For Appium + Python:

# Install dependencies locally
pip install appium-python-client pytest -t ./vendor

# Create requirements file
pip freeze > requirements.txt

# Package
zip -r test-package.zip tests/ vendor/ requirements.txt

For Appium + Java (Maven):

mvn clean package -DskipTests
# Upload target/zip-with-dependencies.zip

Device Farm requires the Maven dependency ZIP plugin:

<plugin>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
        <execution>
            <id>copy-dependencies</id>
            <phase>package</phase>
            <goals><goal>copy-dependencies</goal></goals>
        </execution>
    </executions>
</plugin>

Test File Structure (Appium Python)

# tests/test_login.py
import pytest
from appium import webdriver
from appium.options import AppiumOptions

@pytest.fixture(scope="function")
def driver():
    options = AppiumOptions()
    # Device Farm injects capabilities automatically
    # Just specify your app-specific caps
    options.set_capability("appActivity", "com.yourapp.MainActivity")
    options.set_capability("appPackage", "com.yourapp")
    
    driver = webdriver.Remote(
        command_executor="http://localhost:4723/wd/hub",
        options=options
    )
    yield driver
    driver.quit()

def test_login_success(driver):
    username = driver.find_element("id", "com.yourapp:id/username")
    username.send_keys("testuser@example.com")
    
    password = driver.find_element("id", "com.yourapp:id/password")
    password.send_keys("TestPassword123")
    
    driver.find_element("id", "com.yourapp:id/login_button").click()
    
    assert driver.find_element("id", "com.yourapp:id/dashboard_title").is_displayed()

CI/CD Integration

Using AWS CLI

# Create a run
aws devicefarm create-upload \
  --project-arn "arn:aws:devicefarm:us-west-2:123456789:project:abc123" \
  --name "my-app.apk" \
  --type ANDROID_APP \
  --region us-west-2

# Upload the file (use the presigned URL from the response)
curl -T my-app.apk "https://presigned-upload-url..."

# Create test package upload
aws devicefarm create-upload \
  --project-arn "arn:aws:devicefarm:us-west-2:123456789:project:abc123" \
  --name "test-package.zip" \
  --type APPIUM_PYTHON_TEST_PACKAGE \
  --region us-west-2

# Schedule a run
aws devicefarm schedule-run \
  --project-arn "arn:..." \
  --app-arn "arn:..." \
  --device-pool-arn "arn:..." \
  --test "type=APPIUM_PYTHON,testPackageArn=arn:..." \
  --region us-west-2

GitHub Actions

name: Mobile Testing - AWS Device Farm
on:
  push:
    branches: [main]

jobs:
  device-farm:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v2
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: us-west-2
      
      - name: Build APK
        run: ./gradlew assembleDebug
      
      - name: Package Tests
        run: |
          pip install appium-python-client pytest -t vendor/
          zip -r test-package.zip tests/ vendor/ requirements.txt
      
      - name: Run Device Farm Tests
        uses: aws-actions/aws-devicefarm-mobile-testing@v1
        with:
          project-arn: ${{ secrets.DEVICE_FARM_PROJECT_ARN }}
          app-file: app/build/outputs/apk/debug/app-debug.apk
          app-type: ANDROID_APP
          test-type: APPIUM_PYTHON
          test-package-file: test-package.zip
          device-pool-arn: ${{ secrets.DEVICE_FARM_DEVICE_POOL_ARN }}
          timeout: 1800

Device Pools

Device pools determine which devices your tests run on. Device Farm has built-in pools:

• Top Devices — most popular Android and iOS devices
• All Devices — every available device

For most teams, custom pools are better:

# Create a custom pool via AWS CLI
aws devicefarm create-device-pool \
  --project-arn "arn:..." \
  --name "Critical Devices" \
  --rules '[
    {"attribute": "PLATFORM", "operator": "EQUALS", "value": "ANDROID"},
    {"attribute": "OS_VERSION", "operator": "GREATER_THAN_OR_EQUALS", "value": "11"},
    {"attribute": "MANUFACTURER", "operator": "IN", "value": "[\"Samsung\", \"Google\"]"}
  ]' \
  --region us-west-2

Recommended device selection strategy:

• Latest 2 Android OS versions × top 3 manufacturers (Samsung, Google Pixel, OnePlus)
• Latest 2 iOS versions × iPhone 13/14/15
• One older mid-range Android (to catch performance issues)

Reading Test Results

After a run completes, Device Farm provides:

Artifacts for each device:

• Screenshots at each test step
• Video recording of the full test run
• Device logs (logcat for Android, device console for iOS)
• Test framework logs
• Performance data (CPU, memory, network, FPS)

Common failure patterns:

• NoSuchElementException — element locators are wrong or timing issue (add explicit waits)
• SessionNotCreatedException — app failed to install (check APK target API level vs. device OS)
• Timeout failures — tests taking too long (reduce scope or increase timeout)

Remote Access for Debugging

When automated tests fail, use Remote Access to debug:

1. In Device Farm console: Remote Access > Create Session
2. Select a specific device
3. Access via browser — real-time screen, touch input, keyboard
4. Install your APK directly, run through the failure manually
5. Access device logs in real time

Remote access sessions are billed per minute (typically $0.17/minute).

Cost Optimization

Device Farm pricing:

• Metered: ~$0.17/device-minute
• Unmetered plans: flat monthly fee for unlimited minutes on a fixed device pool

Cost reduction tactics:

• Run full device matrix weekly; run critical device subset on every PR
• Use device slots (reserved capacity) if you run tests multiple times daily
• Set aggressive test timeouts to avoid paying for hung tests
• Use simulators (free in Device Farm) for smoke tests; real devices for release candidates

AWS Device Farm vs. Local Simulators

For most teams: simulators for fast feedback during development, Device Farm for pre-release validation on real hardware.

Summary

AWS Device Farm removes the need to manage a physical device lab. The trade-off is per-minute cost, but for teams that run tests a few times per week rather than continuously, it's usually cheaper and simpler than maintaining hardware.

Start with a small device pool (5–8 devices), integrate with your CI/CD pipeline, and expand coverage as your test suite matures.