Test Automation

Eggplant Image-Based Testing: How Screen-Level Automation Works

HelpMeTest

HelpMeTest

— 7 min read
Eggplant Image-Based Testing: How Screen-Level Automation Works

Eggplant's defining characteristic is that it tests applications the way a human would: by looking at the screen. While most modern test automation tools interact with DOM elements, API responses, or accessibility trees, Eggplant takes screenshots and finds UI elements by visual pattern matching.

This is called image-based testing, and it's simultaneously Eggplant's greatest strength and most significant limitation.

The Core Mechanism: VNC + Image Recognition

Eggplant connects to the System Under Test (SUT) via VNC (Virtual Network Computing). VNC is a remote desktop protocol — it streams a live view of a computer's screen and allows remote input (mouse clicks, keyboard events, touch gestures).

When Eggplant connects to an SUT via VNC:

  1. It receives a continuous stream of screenshots from the SUT
  2. Test scripts specify UI elements by image templates or OCR text
  3. Eggplant searches each screenshot for matching elements
  4. When found, Eggplant sends mouse/keyboard events to the VNC server to interact with the element
  5. The SUT reacts, the screen updates, and Eggplant captures the next screenshot

The entire interaction is at the screen level — no DOM access, no JavaScript execution, no API calls. Just pixels in, events out.

Why VNC?

VNC works with almost any operating system and UI framework. If a computer has a screen, you can connect to it via VNC. This is why Eggplant can test:

  • Windows desktop applications
  • macOS applications
  • Linux desktop GUIs
  • Embedded Linux devices with graphical interfaces
  • Web browsers (via VNC connection to a machine running the browser)
  • ATM and kiosk interfaces
  • Medical device displays
  • Industrial control system HMIs

No other mainstream testing tool has this breadth of platform support.

For specialized hardware (set-top boxes, game consoles, specialized embedded devices), Eggplant also supports proprietary connection protocols beyond standard VNC.

Image Templates: The Foundation

An image template is a screenshot crop of a UI element that Eggplant will search for during test execution. When your script says Click "Submit Button", Eggplant looks for an image in its library named "Submit Button" and finds it on screen.

Capturing Templates

Templates are typically captured during test development:

  1. Connect Eggplant to your SUT
  2. Navigate to the screen with the element you want to capture
  3. Use Eggplant's capture tool to draw a rectangle around the element
  4. Name the template
  5. Eggplant saves the cropped image as a template file (.png in your suite's Images folder)

The template should capture the element clearly with minimal surrounding context — enough to uniquely identify it, but not so much that minor screen changes break the match.

Template Matching Algorithm

Eggplant uses a normalized cross-correlation algorithm (similar to what's used in computer vision for template matching). It slides the template image across the screenshot looking for the position with the highest correlation score.

Key parameters you can tune:

MinScore (default ~0.7): Minimum correlation score to consider a match. Higher values require closer visual similarity but reduce false positives.

SearchRectangle: Constrain the search to a specific region of the screen, improving speed and reducing false matches.

ImageTolerance: How much color variation to allow. Useful for anti-aliasing differences between displays.

HotSpot: Override the click point within the template. Default is the center; you can specify a different point for elements where you need to click at a precise location.

When Template Matching Breaks

Image templates are fragile when:

Resolution changes: A template captured at 1080p won't match at 4K or 720p. Eggplant has scaling options, but they're not always reliable.

Theme or skin changes: Dark mode, high contrast mode, or UI reskins change colors. Your templates need to be recaptured.

Partial rebrands: If a button changes text or icon, the template no longer matches. Common source of maintenance work.

Anti-aliasing differences: Rendering engines vary slightly between OS versions, graphics drivers, and display hardware. Usually handled by ImageTolerance tuning.

Overlapping elements: Tooltips, dialogs, and pop-unders can obscure elements and cause false negatives.

OCR: Text-Based Element Finding

For text elements (buttons with labels, form fields, menu items), Eggplant can use OCR (Optical Character Recognition) instead of image templates.

Click Text "Submit"
Click Text "Cancel Order"
TypeText "user@example.com" into text field "Email Address"

OCR-based finding is more resilient than image templates because it doesn't depend on exact visual appearance — only the text content. It handles font changes, color changes, and minor UI restyling as long as the text remains legible.

Eggplant's OCR engine supports multiple languages and can be configured for specific font characteristics. For mixed-language UIs or specialized text (numbers on a dashboard, technical codes), tuning OCR parameters is sometimes necessary.

OCR is slower than image matching because it has to process and interpret pixel regions as text. For performance-sensitive test suites, use image templates for stable elements and OCR for text-heavy or frequently-changing elements.

SenseTalk: The Scripting Language

Eggplant Functional's scripting language is SenseTalk, designed to be readable by non-programmers. It has an English-like syntax:

-- Basic click and type
Click "Username Field"
TypeText "john.doe@company.com"

-- Wait for element to appear
WaitFor 10, "Loading Complete"

-- Conditional logic
if ImageFound("Error Message") then
  LogError "Login failed"
  throw "LoginFailure"
end if

-- Capture screenshot evidence
CaptureScreen name:"login_attempt"

-- Assertions
Assert that ImageFound("Dashboard") is true

Working with Dynamic Content

For dynamic content like form field values, user-specific data, or content that changes between test runs, SenseTalk can capture screen regions as text and make assertions on the captured value:

-- Read text from screen region
set orderTotal to ReadText from (200, 450, 400, 480)
Assert that orderTotal contains "$"

-- Capture and compare against stored value
set productName to ReadText from nameRegion
Assert that productName is equal to expectedProductName

The ReadText function extracts text from a screen region using OCR, letting you make data-driven assertions without image templates.

Parameterization and Data-Driven Testing

SenseTalk supports data-driven testing through Excel/CSV integration and built-in data structures:

-- Read test data from Excel
repeat with each row of sheet "TestData" from "test-data.xlsx"
  Click "Username Field"
  TypeText row.username
  Click "Password Field"  
  TypeText row.password
  Click "Login Button"
  WaitFor 5, "Dashboard"
  Click "Logout"
end repeat

This allows the same test script to run against multiple test cases defined in a spreadsheet — a pattern familiar to teams coming from data-driven frameworks like Robot Framework.

Managing an Image Library

For large test suites, image library management becomes a significant operational concern.

Organizing Templates

Eggplant recommends organizing templates by application area:

MyTestSuite/
  Images/
    Login/
      username_field.png
      password_field.png
      login_button.png
      login_error.png
    Dashboard/
      nav_menu.png
      user_profile_icon.png
      ...
    Checkout/
      ...

Scripts reference templates by name, and Eggplant searches through all image libraries in the suite. Consistent naming conventions prevent conflicts between templates with similar appearances.

Template Versioning

Image templates are binary files (.png) — they don't diff well in version control. Teams typically store them in Git with LFS (Large File Storage) or use Eggplant's own suite management to handle versioning.

When a UI redesign requires mass template updates, teams need to recapture and replace templates systematically. Eggplant DAI's self-healing capability can suggest updates for minor changes, but significant redesigns require manual work.

Cross-Platform Template Sets

Applications that run on multiple platforms often need platform-specific image templates because fonts render differently, UI controls look different, and color profiles vary. Best practice is to maintain separate image libraries per platform:

Images/
  Windows/
    submit_button.png
  macOS/
    submit_button.png  
  Linux/
    submit_button.png

Scripts use conditional logic to select the appropriate library based on the SUT's platform.

Performance Characteristics

Image-based testing is inherently slower than DOM-based testing:

  • Screenshot capture: 50–200ms per frame depending on network and SUT performance
  • Template matching: 10–100ms depending on template complexity and search area
  • OCR: 200–500ms for typical text regions
  • Click-to-response cycle: 300–1000ms typical (vs. ~50ms for DOM-based tools)

A test that takes 5 seconds in Playwright might take 30–60 seconds in Eggplant. For individual tests this is manageable; for test suites with hundreds of tests, it becomes significant.

Mitigation strategies:

  • SearchRectangle: Constrain searches to expected regions, avoiding full-screen scans
  • WaitFor with short intervals: Don't sleep; poll with WaitFor to continue as soon as elements appear
  • Parallel execution: Run tests on multiple SUTs simultaneously (requires additional Eggplant Functional licenses)

Connecting to Different Device Types

Beyond standard VNC, Eggplant supports several specialized connection types:

RDP (Remote Desktop Protocol): For Windows-only connections, RDP can be faster and more reliable than VNC.

iOS/Android: Eggplant Mobile supports connecting to mobile devices via its proprietary protocol layered on top of Appium.

Web (via VNC): Launch a browser on a VNC-connected machine. Eggplant sees the rendered browser output. This is less efficient than Playwright/Selenium for web testing but allows the same Eggplant tests to cover web alongside other UI types.

Custom protocols: For specialized hardware (ATMs, medical devices, industrial HMIs), Eggplant provides extension points for custom connection protocols.

When Image-Based Testing Is the Right Choice

Use Eggplant's image-based approach when:

  • You can't access the DOM: Legacy thick-client apps, embedded systems, mainframes
  • Cross-platform consistency is critical: Same tests across different OS/device types
  • Source code access is restricted: Third-party applications or vendor-supplied systems
  • Visual layout verification is a primary concern: Checking that UI looks correct, not just functions correctly
  • Regulatory requirements mandate black-box testing: Some compliance frameworks require testing without application instrumentation

Don't use Eggplant's image-based approach when:

  • You're testing web applications where DOM access is available
  • Test speed matters (it will be 5–10x slower than DOM-based tools)
  • Your UI changes frequently (image template maintenance becomes expensive)
  • Team size is small (the tooling overhead isn't justified)
  • Budget is limited (Eggplant licensing is enterprise-priced)

Alternatives to Image-Based Testing

For web applications, Playwright or Cypress provide DOM-based automation that's faster, cheaper, and easier to maintain. These tools can access every element via CSS selectors, data attributes, or ARIA labels — far more reliable than image matching.

For AI-powered test creation without the image template overhead, HelpMeTest lets you describe tests in plain English and handles the underlying automation automatically. No templates to capture, no image library to maintain.

For mobile-specific testing, Appium or native tools (XCTest, Espresso) interact with platform accessibility layers — more reliable than image matching for mobile UIs.

Summary

Eggplant's image-based testing via VNC is a powerful but specialized approach. It unlocks testing scenarios that are simply impossible with other tools — legacy systems, embedded devices, cross-platform consistency across disparate UI technologies.

The cost is real: slower execution, higher maintenance burden, significant tool cost. For the right use case — testing complex enterprise systems with legacy components — the trade-off is worth it. For standard web and mobile development, you're better served by tools built for the modern stack.

Image-based testing isn't obsolete. It's a precision instrument for a specific set of problems. Use it when your problem matches its strengths.

Read more

Start now free