Sikuli: Image-Based UI Testing for Desktop and Web Applications

Most test automation frameworks interact with applications through their DOM, accessibility tree, or native platform APIs. Sikuli takes a fundamentally different approach: it finds and interacts with UI elements by recognizing screenshots of them.

If you can see it on screen, Sikuli can click it — regardless of whether the application exposes any programmatic interface.

What Is Sikuli?

Sikuli (now maintained as SikuliX) is an open-source automation tool that uses OpenCV image recognition to find UI elements on screen. You provide screenshots of what you're looking for, and Sikuli locates matching regions and simulates mouse clicks, keyboard input, and other interactions.

This approach works for:

• Desktop applications with no accessibility API
• Legacy enterprise apps (Java Swing, WinForms, old web apps)
• Applications inside virtual machines or remote desktops
• Browser extensions and browser UI (not page content)
• Games and any application where DOM access isn't possible

Installation

SikuliX requires Java 11+. Download the IDE and/or API jar from the SikuliX releases page.

IDE Setup

1. Download sikulixide-2.0.5.jar
2. Run: java -jar sikulixide-2.0.5.jar

The IDE provides a visual environment where you can capture screenshots and write scripts.

Script-Only Setup

For scripting without the IDE:

# Download the API jar
wget https://github.com/RaiMan/SikuliX1/releases/download/v2.0.5/sikulixapi-2.0.5.jar

# Run a script
java -jar sikulixapi-2.0.5.jar -r myscript.sikuli

Writing Your First Script

Sikuli scripts are Python (using Jython) by default, though Java and Ruby bindings exist.

Capture Screenshots

In the Sikuli IDE, use the Camera icon to capture screenshots of UI elements you want to interact with. The IDE saves these as PNG files alongside your script.

Basic Script

# Find and click a button by its image
click("submit-button.png")

# Type into a field (click first to focus it)
click("email-field.png")
type("user@example.com")

# Find text and verify it exists
if exists("success-message.png"):
    print("Login succeeded")
else:
    print("Login failed")

Image Similarity Threshold

By default, Sikuli requires ~70% image similarity. Adjust for tolerance:

# More tolerant matching (0.0 to 1.0)
Settings.MinSimilarity = 0.7

# Strict matching
Settings.MinSimilarity = 0.95

# Per-operation similarity
click(Pattern("button.png").similar(0.8))

Core Operations

Mouse Operations

# Click
click("element.png")

# Double-click
doubleClick("element.png")

# Right-click
rightClick("element.png")

# Hover
hover("element.png")

# Drag and drop
drag("source.png")
dropAt("target.png")

# Click at an offset from a found image
click(Pattern("icon.png").targetOffset(50, 0))  # 50px to the right

Keyboard Operations

# Type text
type("Hello, World!")

# Type with modifiers
type("a", KeyModifier.CTRL)  # Ctrl+A (select all)
type(Key.ENTER)

# Special keys
type(Key.TAB)
type(Key.F5)
type(Key.ESCAPE)

Waiting and Assertions

# Wait for element to appear (up to 10 seconds)
wait("loading-spinner.png", 10)

# Wait for element to disappear
waitVanish("loading-spinner.png", 30)

# Assert element is present
assert exists("success-banner.png"), "Success banner not found"

# Find all instances
matches = findAll("list-item.png")
for match in matches:
    click(match)

Region-Based Operations

Working with specific screen regions avoids false matches:

# Define a region: (x, y, width, height)
sidebar = Region(0, 100, 300, 800)

# Find within region only
sidebar.click("settings-icon.png")

# Relative regions
dialog = find("dialog-header.png")
dialog.below(200).click("ok-button.png")

Practical Example: Testing a Desktop Application

import sys
from sikuli import *

def test_file_save():
    # Open the application
    App.open("notepad.exe")
    wait("notepad-window.png", 5)
    
    # Type some content
    click("text-area.png")
    type("Test content")
    
    # Save via keyboard shortcut
    type("s", KeyModifier.CTRL)
    
    # Wait for Save dialog
    wait("save-dialog.png", 5)
    
    # Enter filename
    click("filename-field.png")
    type("test-file.txt")
    type(Key.ENTER)
    
    # Verify save succeeded
    if exists("title-bar-saved.png"):
        print("PASS: File saved successfully")
    else:
        print("FAIL: Save dialog did not close")
        sys.exit(1)

test_file_save()

Web Application Testing

For web apps, Sikuli can interact with browser UI and handle cases where WebDriver struggles:

# Handle browser file upload dialog (not accessible via WebDriver)
click("upload-button.png")
wait("file-dialog.png", 5)
type("/path/to/file.pdf")
type(Key.ENTER)

# Handle browser permission dialogs
if exists("allow-camera-dialog.png"):
    click("allow-button.png")

# Interact with Flash or canvas elements
click("flash-player.png")

CI Integration

Running Sikuli in CI requires a real display. Use Xvfb on Linux:

name: Sikuli Tests

on: [push]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Java
        uses: actions/setup-java@v3
        with:
          java-version: '11'
      
      - name: Install Xvfb
        run: sudo apt-get install -y xvfb
      
      - name: Run tests with virtual display
        run: |
          Xvfb :99 -screen 0 1920x1080x24 &
          export DISPLAY=:99
          java -jar sikulixapi.jar -r tests/login_test.sikuli

Important: Sikuli in CI is fragile. Screen resolution, DPI settings, font rendering, and OS-level theme differences all affect image matching. Capture screenshots in the same environment where tests will run.

When Image-Based Testing Makes Sense

Do use Sikuli for:

• Legacy applications with no automation API (old Java Swing apps, Access-based tools)
• Testing across the full OS UI stack (desktop apps, system dialogs, tray icons)
• Applications inside RDP/VNC sessions
• Verifying that a UI element is visually rendered correctly, not just present in DOM
• Automation of applications you don't own and can't modify

Don't use Sikuli for:

• Modern web applications (use Playwright, Cypress, or Selenium)
• Mobile apps (use Appium)
• Applications with accessible automation APIs (use those APIs)
• High-volume CI testing (too slow, too fragile)

Limitations

Brittle on UI changes. Any visual change — new button style, moved element, different font rendering — breaks image matches. Maintenance burden is high.

Resolution-dependent. Screenshots captured at 1920x1080 won't match on a 1280x800 CI runner. Managing screen resolution consistency is a real problem.

Slow. Image recognition is computationally expensive. Tests run slower than DOM-based automation.

No debugging information. When a test fails, you get "image not found" — not "element not visible because modal is blocking it." Debugging takes longer.

Not suitable for dynamic content. Any content that changes (timestamps, user names, avatars) needs masking or separate handling.

SikuliX vs Alternatives

For Windows desktop applications specifically, Microsoft's WinAppDriver (based on WebDriver) and Appium are generally more reliable than image-based approaches when accessibility APIs are available.

Summary

Sikuli fills a genuine gap: automating applications that have no programmatic interface. For legacy desktop apps, custom in-house tools, or any scenario where you can only interact with the screen output, image-based testing is sometimes the only option.

The trade-off is maintenance cost. Image-based tests require careful screenshot management, tight environment control, and regular updates when UI changes. For applications that rarely change, this is manageable. For actively developed software, DOM-based approaches are almost always preferable.

If you're inheriting a legacy enterprise application with no API and need automated testing, SikuliX is a pragmatic solution.