How Repeato Uses Computer Vision for Mobile Test Automation

Traditional mobile test automation — Appium, XCUITest, Espresso — works by querying the app's accessibility tree. Every interaction targets an element by ID, XPath, or accessibility label. This works well until the app changes: rename a resource ID, restructure a layout, and tests start throwing NoSuchElementException.

Repeato takes a different approach. Instead of querying the accessibility tree, it captures what the screen looks like and uses computer vision to recognize where to interact. This changes the fundamental contract between your tests and your app.

How Visual Matching Works

When you record a test step in Repeato — say, tapping the "Login" button — Repeato captures:

1. A screenshot of the screen at that moment
2. The coordinates of your tap relative to what's visible

At replay time, Repeato doesn't query the app for an element named "Login". Instead, it takes a fresh screenshot and compares it to the recorded one, looking for the visual region that matches. It calculates a similarity score. If the score exceeds the configured threshold, it determines the target location and performs the tap.

This is fundamentally different from selector-based automation. The test doesn't know or care what the element is called internally — it only knows what it looks like.

Template Matching vs. Feature Detection

Repeato uses template matching as its primary visual recognition technique. In simple terms:

1. Take the recorded screenshot region (the "template")
2. Scan the live screen for areas that look like the template
3. Find the best match above the similarity threshold
4. Interact at the matched location

This is fast and computationally cheap, which is why Repeato can replay tests in near-real-time without significant overhead.

For more complex scenarios — when UI elements can appear anywhere on screen, or when content shifts significantly — Repeato also uses feature-point detection, which identifies distinctive visual features in the template and looks for those features in the live screen, regardless of exact position.

Similarity Thresholds

Every Repeato step has a configurable similarity threshold (default: ~0.8 on a 0-1 scale). This controls how strict the visual matching is:

• High threshold (0.9+): The screen must look almost identical to the recording. More reliable for stable UIs, more brittle if the app updates.
• Low threshold (0.6-0.7): Looser matching tolerates more visual change. Useful for apps with dynamic content, but can cause false positives.

When a test fails, Repeato shows you the similarity score for the failing step alongside the recorded vs. actual screenshots — you can immediately see whether the match was close (threshold issue) or completely off (the UI changed significantly).

Assertion Types

Repeato supports several visual assertion modes:

Region Existence Check

Verify that a visual region matching the recorded template appears somewhere on screen. This is the most common assertion type — confirm a success message, a specific button, or a navigation element is visible.

Text Assertions

Repeato integrates OCR (optical character recognition) to extract text from screen regions. You can assert:

• A region contains specific text
• A region contains text matching a pattern
• A region is empty (no text)

This is useful for verifying dynamic content — order totals, user names, error messages — without hard-coding expected values.

Full-Screen Comparison

Compare the entire screen against a recorded baseline. This is stricter and more useful for visual regression testing — confirming that a screen looks exactly right after a code change.

Handling Dynamic Content

Computer vision struggles with truly dynamic content — screens where the visual appearance changes on every run (timestamps, generated IDs, live data). Repeato handles this through:

Masking: Define regions of the screen to ignore during comparison. If your app shows a timestamp in the header, mask that region — the visual match proceeds on the stable parts of the screen.

Text variable extraction: Extract text from a screen region into a variable and use it in subsequent steps. This lets you capture a dynamic value (like an order number) and reference it later in the test.

Pattern matching in OCR assertions: Instead of asserting exact text, assert text that matches a regex pattern — useful for dates, IDs, and other generated values.

Advantages Over Selector-Based Automation

Resilient to code refactoring: If developers rename a resource ID, restructure a layout, or change the accessibility tree, Repeato tests keep passing as long as the UI still looks the same.

Works on compiled apps: You don't need source code access. Repeato can test any installed .apk or .app — including apps you don't own the source for.

Accessible to non-engineers: QA engineers and product managers who can't read accessibility IDs can write, maintain, and debug tests.

Limitations to Understand

Computer vision isn't magic — it has real limitations:

Significant redesigns break tests: If a screen is completely reimagined, the visual templates no longer match and tests need re-recording. This is similar effort to rewriting Appium locators for a redesign.

Low contrast and small elements: Very small UI elements or elements with low contrast against the background can be difficult to match reliably. Increase threshold sensitivity or use OCR assertions for text-heavy elements.

Performance: Visual matching is slightly slower than direct accessibility tree queries. For most test suites this is imperceptible, but very high-volume test runs may notice the difference.

Simulator dependency: Repeato requires a running simulator or connected device to capture and replay screens. There's no execution layer that works purely from app binaries.

When Computer Vision Wins

Computer vision-based testing earns its place when:

• UI stability is reasonable — the app doesn't change layout completely on every sprint
• Selector maintenance is the bottleneck — your Appium tests spend more time broken than passing
• Non-technical stakeholders need to own tests — PMs or QA leads who can't write XPath
• Speed of authoring matters — you need tests on new features immediately, not after a locator investigation

For teams fitting this profile, Repeato's computer vision approach isn't a compromise — it's a genuine advantage.