looktap

Let your AI agent see, use and control the apps on your phone

It reads the screen as text and taps by label, on iPhone and Android devices. Then what worked becomes a script or a test your CI runs. No server to start and no dependencies.

Website · Quick start · Commands · For AI agents · Benchmark · Troubleshooting

---

looktap lets your AI agent see, use and control the apps on your phone: an iPhone or Android device, real or simulated. It reads the screen as text and taps by label. Each command does one thing and prints a few lines of plain text:

Your agent sees and uses the phone. It's made for AI coding agents (Claude Code, Cursor, Codex, opencode, …) that need to see and use the app they're building, and it controls any app on the device, not only yours: system apps, apps you didn't build, and web pages, with nothing added to the app.

• Small output. A screen read is about 1 KB of text, where other agent tools return 4–125 KB. A screenshot is a file path, where most MCP servers put 40–270 KB of image into the context.
• One call per step. Add --ls to an action and its result comes back in the same call.
• Fast. Screen reads take about 0.1 s and taps 0.1–0.2 s on both platforms, and the first command comes back in under a second, with no session to set up. Against eight other agent tools on the same devices, it had the lowest agent time on both iPhone and Android (How it compares).

Then what worked becomes a script or a test your CI runs. Every command is a plain shell call with an exit code, so the steps that worked with the agent run on their own afterwards:

• Tests in CI. Save the steps as flow files: expect checks the screen, and looktap test runs the flows and reports each one, with a JUnit report (Test flows in CI). On Android, looktap codegen writes a flow while you use the phone (Record a flow).
• Screenshot sweeps, on one device or several (A repeatable screenshot sweep, Same screen on several devices).
• Bug repros that collect their own logs and crash reports (Debug).

---

Contents

• What you get: everything looktap does today
• How it compares: the measured comparison with every other agent mobile tool
• Quick start: install, boot a device, first commands
• 🤖 Kickstart with your AI agent: one prompt and your agent takes it from there
• Requirements
• How it works
• Examples: tap by label, forms, waiting, scrolling, debugging, performance, bursts and video
• Tests and automation: flow tests in CI, recording a flow, screenshot sweeps, several devices
• For AI agents
• Commands · Platform support · Options & config
• Troubleshooting · FAQ · Roadmap
• Safety rules · Contributing · License

What you get

How it compares

Every agent mobile tool we could run on one Mac was given the same 12 steps in the Settings app, 10 runs each, on an iPhone simulator and an Android emulator: looktap (CLI and MCP), mobile-mcp, Maestro MCP, Appium MCP, raw Appium, ios-simulator-mcp, MobileBuildMCP, raw idb and raw adb. A neutral checker decided whether each step worked. These are the totals over the steps every tool can do:

¹ The device's reaction time plus 2 s per tool call for the model's round trip.

On real phones (iPhone 14 Pro and Samsung Galaxy A06 over USB, looktap 0.4.0, all 12 steps), the looktap CLI again had the lowest agent time (87 s and 38 s, against 104–114 s and 45–89 s) and returned the least text (1.7 KB and 1.0 KB, against 8.9–191 KB). Four of the iOS tools can't drive a real iPhone at all: ios-simulator-mcp, MobileBuildMCP, idb and Maestro.

Where looktap loses:

• On raw device time, others are faster in total: ios-simulator-mcp on the iOS simulator (4.5 s against 5.2 s, with 20 calls to looktap's 8), mobile-mcp and raw Appium on the real iPhone (60 s and 62 s against 63 s), and raw Appium on the Samsung (11 s against 12 s).
• The looktap MCP waits for the screen to settle after every action, which makes it slower per action than the CLI. On the real iPhone that takes 6–11 s a tap and puts it last for agent time.
• In the simulator and emulator runs, looktap 0.2.1 couldn't stop an app or press Home on iOS; 0.3.0 added stop and home.

Method, every device's tables and what else they show are in BENCHMARK.md.

Where Appium is still the better choice: it plugs into existing WebDriver test suites and CI grids, and supports many more platforms. looktap's tests are flow files of plain commands: good for smoke tests and agent-written flows, not a replacement for a large suite in code.

Quick start

# 1. install (Node 18+; zero dependencies)
npm install -g @looktap/cli

# 2. check the toolchain
looktap doctor

# 3. boot a device: an iOS simulator or an Android emulator, by name
looktap devices --all                         # what's running, and what you can boot
looktap boot "iPhone 16 Pro"                  # or an AVD: looktap boot Pixel_9a

# 4. drive it
looktap launch com.apple.Preferences --ls     # (Android: com.android.settings)
looktap tap "General" --ls
looktap shot general

Or try it without installing anything: npx @looktap/cli doctor (any command works the same way).

To teach Claude Code how to use looktap, install the plugin. In Claude Code:

/plugin install looktap --marketplace lakpriya1s/looktap

It adds the looktap skill. Then run /looktap:setup: it installs the CLI, checks your toolchain and gets a device ready.

Any other coding agent (Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, Kiro, …): install the skill for every agent on your machine in one go (which agents):

looktap skill --install      # or, without installing looktap: npx -y @looktap/cli skill --install

Other ways to install

The latest from GitHub, ahead of the npm release:

npm install -g github:lakpriya1s/looktap

From a checkout:

git clone https://github.com/lakpriya1s/looktap.git
cd looktap && npm install -g .

As a tarball you can hand to someone (it includes both prebuilt helpers):

npm pack                                  # → looktap-cli-<version>.tgz
npm install -g looktap-cli-<version>.tgz  # on their machine

Without installing: node bin/looktap.js --help.

Uninstall: npm uninstall -g @looktap/cli && rm -rf ~/.looktap.

🤖 Kickstart with your AI agent

Paste this into your AI coding agent (Claude Code, Cursor, Codex, opencode, …) and let it set things up and start using the app with you:

Install looktap (https://github.com/lakpriya1s/looktap) so you can drive
my app in the iOS simulator / Android emulator:

1. Run `npm install -g @looktap/cli`, then `looktap doctor`,
   and fix anything it reports. If no device is up, boot one: `looktap devices --all` lists them, `looktap boot "<name>"` starts one.
2. Run `looktap skill --install` and read the skill it installs.
3. Find my app's bundle id / package name in this repo and write a
   looktap.config.json with it at the repo root.
4. Launch the app with `looktap launch --ls` and tell me what screen it's on.

Rules: never type real credentials, OTPs or card numbers. When the app needs a
sign-in, stop and let me do it, then continue from there.

Requirements

Tested with Xcode 26 and 27.1 / iOS 26, 27 and 27.1 (iPhone 14, iPhone 18 Pro and the foldable iPhone Duo simulators), a real iPhone 14 Pro on iOS 26, and Android 16 / API 36 (Pixel 9a emulator).

How it works

looktap ships its own small helper for each platform. Nothing is installed on simulators, emulators or Android devices. A real iPhone or iPad is the exception: it gets a small test runner (Using a real iPhone).

Each helper starts on first use, keeps running in the background, and is replaced automatically when looktap is upgraded. looktap setup starts it ahead of time, and looktap teardown stops it. The CLI talks to it over a local socket, using JSON lines.

iOS: private frameworks and Xcode versions

The iOS helper uses Apple's private simulator frameworks, the same ones idb uses. They can change between Xcode releases. looktap doctor checks they're present and shows which input path applies:

• Xcode 26 and earlier (CoreSimulator below 1155.4): touches, keys and the Home button go through SimulatorKit's HID client ("indigo"). Tested.
• Xcode 27 (CoreSimulator 1155.4 and later): touches, keys and the Home button go as XPC messages to the simulator's dtuhidd service ("dtuhid"), the same way idb does it. Tested on Xcode 27.1 with iOS 26 and iOS 27 simulators. Reading the screen and screenshots don't depend on it. Xcode 27.1 moved SimulatorKit to Contents/SharedFrameworks, which looktap 0.7.1 and later finds; it also replaced the Simulator app with Device Hub (see boot).

LOOKTAP_IOS_HID=indigo|dtuhid overrides the choice. It applies when the helper starts, so run looktap teardown first.

Using a real iPhone

A simulator can be driven from the Mac with nothing installed in it; a real iPhone or iPad can't. So the first command aimed at one builds a small XCUITest runner (native/ios-device), signs it with your Apple ID, and installs it. It reads the screen and taps, types and swipes through XCUITest, and looktap reaches it through usbmuxd, over USB or Wi-Fi. Every looktap user signs the runner with their own Apple ID. A free one is enough; the paid Developer Program isn't needed.

Once per Mac and device:

1. In Xcode → Settings → Accounts, sign in with your Apple ID. That gives you a "Personal Team".
2. Pair the device with the Mac (connect it with a cable once and tap Trust; Wi-Fi works after that), and turn on Developer Mode on it: Settings → Privacy & Security → Developer Mode.
3. Unlock the device and run any command against it, for example looktap -d <hardware UDID> ls (looktap devices lists the UDID, and says what to do if the device isn't ready yet: Trust, Developer Mode). The first run builds the runner, which takes about a minute.
4. With a free Apple ID, the device asks you to trust the developer the first time: Settings → General → VPN & Device Management → your Apple ID → Trust. Then run the command again.

If Xcode is signed in to more than one team, choose the one to sign with: LOOKTAP_TEAM=<team id>, or {"ios": {"team": "<team id>"}} in looktap.config.json. The error message lists your teams and their ids.

Picking the device: -d <hardware UDID> always picks it. Without -d, a paired device is used only when no simulator or emulator is running, so nothing changes for simulator workflows.

Signing expiry: a runner signed with a free Apple ID stops launching after 7 days, and one signed with a paid team after a year. looktap reads the expiry date and builds and signs the runner again before it lapses. If iOS refuses to launch it anyway (for example, after a certificate was revoked), looktap rebuilds it once and retries. Free Apple IDs have Apple's own limits too: about 3 devices and 10 app ids per 7 days, and at most 3 apps signed by the free account installed on one device at a time. The runner counts as one, so if the device is full, looktap names the apps it found and you delete one.

What works on a device: everything except location, network, push, statusbar, textsize, bio, record, permission, reset, logs and crashes, which say so; zoom works around the screen's centre only. perf reads CPU, memory and threads through Instruments (xcrun xctrace, part of Xcode), 10–25 s a reading, and perf --frames reads the screen over the USB cable, so it needs the phone plugged in rather than on Wi-Fi. Platform support has the reasons, and looktap capabilities prints them. A command takes about 1–1.5 s, slower than on a simulator, because XCUITest takes longer to read the screen.

The device must be unlocked while the runner starts. It keeps running until looktap teardown, or until the device is restarted. While the device is locked, commands that need the screen stop with "the device is locked" (stop still works). A runner can lose its permission to drive the screen while it keeps running (seen after the device sat idle); looktap then starts a fresh one, once, and carries on.

Android: the fallback path

If the helper can't start (for example, another tool holds UiAutomation), looktap falls back to plain adb + uiautomator dump. That works everywhere but takes about 2.5 s per screen read. LOOKTAP_ANDROID_HELPER=0 forces that path.

Examples

Tests in CI, recording a flow, screenshot sweeps and running on several devices are in Tests and automation.

Look, then act

looktap ls

@1   Button           16, 285 358x54  General
@2   Button           16, 339 358x54  Accessibility
@3   TextField        33, 773 324x38  Search  = Search

Each row shows a ref (@1), the element type, its x,y position and width×height (iOS points / Android dp, the units tapxy and swipe take), its label, and = value for fields that hold text and for switches and checkboxes (1 on, 0 off). Unlabelled buttons are listed too, as (no label), so you can tap them by ref. The on-screen keyboard is a single row.

Tap by label, not by coordinates

looktap tap "Open menu"              # exact label (or identifier on iOS)
# tapped "Open menu" at 30,62
looktap tap "Messages"               # no exact match → the one label starting with it
# tapped "Messages, 2 unread" at 342,62
looktap tap "Wi"                     # matches two different labels → refuses
# looktap: "Wi" is ambiguous — matches 2 labels: "Wi-Fi", "Wi-Fi hotspot"
looktap tap --exact "Messages"       # exact only
looktap tap @7                       # row @7 of the last ls
looktap tapxy 205 874                # when ls doesn't list it at all

• A ref works only while its element is still where ls saw it. If the screen changed, tap @7 fails ("no longer on screen — run: looktap ls") rather than tapping something else.
• If another element covers the tap point (a toast, a dev-warning banner, a floating bar), tap prints a warning, because the tap probably landed on the cover instead.

Long press and pinch-zoom

looktap longpress "Photo"               # context menus, drag handles, "edit" modes (default 800 ms)
looktap longpress @4 1500               # by ref, held 1.5 s
looktap longpress 120 470               # a point, e.g. to drop a pin on a map
looktap zoom in                         # two fingers spreading at the screen centre (maps, photos, PDFs)
looktap zoom out 200 400 --scale 4      # pinch around a point; --scale is how far the fingers travel

zoom is a real two-finger gesture on both platforms. On Android it needs looktap's helper, because adb input has no multi-touch.

Fill a form and capture its validation errors

looktap fill "Email" not-an-email
looktap fill "Name" ""               # empty: just clears the field
looktap hidekb
looktap shot profile__validation

fill finds the field by its label (on iOS also its identifier), or else the first field at or below a matching label. It clears the field and types, then reads the field back. Keys can get lost (a field that was still clearing swallows the first letters; a web field drops keys while WebKit autocorrects), so on a miss it empties the field and types again key by key, then, on the iOS simulator, pastes the text, and fails rather than report a value the field doesn't hold. A field that formats its input (spaces, a currency sign, a capital) passes. A password field shows dots, so its length is what's checked. On Android it also presses Enter, because React Native fields only show validation on blur.

A one-time code split across boxes goes in with code:

looktap code 540390

It finds the row of boxes, taps only the first and types the digits one at a time, pausing while the boxes move focus along (tapping every box too puts digits a box off). Then it reads every box. If they don't hold the code, it taps each box before its digit, then pastes the code into the first; it fails listing what each box holds if none of that works.

On iOS, hidekb tries, in order:

1. the keyboard's Done/Return key;
2. the ✓ on a web form's Previous/Next bar;
3. a tap on plain text above the keyboard.

On Android it takes focus from the field, as a tap outside it would (so a form that validates on blur shows its errors), and presses Back only if the keyboard is still up. It checks after each. If the keyboard is still up, it exits 2 rather than reporting success.

Wait instead of sleeping, and scroll

looktap wait "Welcome"                  # until it's on screen (default up to 10 s; --wait 30)
looktap wait "Loading" --gone           # until it's gone
looktap scrollto "Privacy & Security"   # swipe until fully on screen; fails at the end of the list
looktap scrollto "General" --up
looktap swipe 200 700 200 250           # scroll down (finger moves up); optional 5th arg: ms
looktap back                            # Android: system Back · iOS: the navigation bar's back button
looktap home                            # the Home button: home screen, the app keeps running

Debug: logs, crashes, "where did my app go?"

looktap tap "Pay"                       # something goes wrong…
looktap logs --errors                   # the app's errors since the last `logs`
# 15:41:06.700 E [com.example:api] POST /payments failed: 422 amount_too_small
looktap logs --grep payment             # everything it logged that matches
looktap app --app com.example           # is it still running?
# front: com.android.launcher3  pid 1205
# com.example: not running — if it was, check: looktap crashes
looktap crashes
# com.example (pid 14001) exited at 14:56:06.724 while in the background: LOW_MEMORY — killed by the system to free memory, not a crash

When the app dies, looktap says so. After an action, and at the next ls, looktap checks whether the app that was in front still has its process. If not, it prints:

looktap: warning: com.example is no longer running — it crashed or was killed. See: looktap crashes com.example

• It also catches an Android app that crashed and was restarted at once, because the check is by process id.
• Switching to another app, going home, or uninstalling doesn't trigger it.
• On iOS the crash report is written up to a minute after the crash. Until then, crashes already shows the signal (e.g. SIGTRAP) and says the full report is coming.

Each logs call starts where the previous one ended, so a loop of act → logs only ever shows what's new. --since 5m sets a fixed window instead.

What's in the log:

• iOS: what the app's own code logged, plus errors and faults from anything in its process. The simulator's own noise (low-level networking, the keyboard) is hidden. --all shows everything, Apple frameworks included. A read takes about 1–2 s.
• Android: everything logcat has for the app's uid, so restarts are covered. A read takes about 0.1 s.

What crashes reports:

• iOS: each crash report as exception, reason (such as an uncaught NSException), the crashed thread's top frames, and the path to the full .ips report.
• Android: Java crashes, native tombstones, ANRs, and exits that leave no crash report, low-memory kills above all.

React Native: in development builds, JS console.* output and uncaught JS errors are in logs on both platforms:

• iOS: as [com.facebook.react.log:javascript] lines.
• Android: as ReactNativeJS lines.

A red box is also on screen, so ls reads its message and call stack.

Swift: print() writes to stdout, not to the log. Use os.Logger or NSLog for messages logs should see.

Performance: CPU, memory, frames, launch time

looktap launch com.example --time       # a cold start, timed
# launched com.example: in front after 0.44 s, first screen settled by 0.85 s · Android TotalTime 300 ms
looktap perf                            # the app in front: one reading
# com.example  pid 4127
# cpu      14%  (over 0.5 s; 100% is one core)
# memory   pss 78.4 MB · rss 171.2 MB · java heap 12.1 MB · native heap 18.6 MB
# threads  21
looktap perf --for 20s                  # a reading each second while you (or a flow) use the app, then avg/max
looktap perf --frames                   # frames since the last --frames, janky %, p50–p99 frame time (iOS simulator:
                                        # the first call starts counting)
looktap swipe 200 700 200 200; looktap perf --frames   # how smooth was that scroll?

• Memory: on the iOS simulator, the footprint Xcode's memory gauge shows, and its peak. On Android, PSS (the app's share of RAM) plus RSS and the Java and native heaps.
• CPU is a share of one core over the reading, like top: a busy app on two cores reads 200%.
• Simulator numbers come from the Mac. An iOS simulator app runs on the Mac's CPU and memory, so compare builds and screens against each other, not against a phone. An emulator is closer, but still not a phone: for release numbers, use a real Android device.
• launch --time starts the clock at the start command (stopping the old copy comes first, outside it). Right after a launch an iOS screen read can take a second, so it says how much sooner the screen may have settled.
• Real iPhone: perf reads it through Instruments' Activity Monitor (xcrun xctrace, part of Xcode, so nothing to install). Starting Instruments takes a few seconds, so a reading takes 10–25 s and --for gives one about every 8 s. Its CPU matches Instruments' own figure to within about 10%.
• Frames on the iOS simulator come from the screen itself. Instruments' Animation Hitches doesn't support the simulator, so looktap counts how often the simulator's screen actually changes (its framebuffer), snapped to the 60 Hz display. A frame that skips a refresh (33 ms or more) is janky, and fps is measured while something moves. It's the whole screen as you'd see it, not the app's own render time, which only Instruments has. A gap over 0.1 s counts as "nothing moving", so a pause between two actions doesn't count as jank, and is reported as a pause. The first perf --frames starts counting; each one after reports what happened since the last.
• Frames on a real iPhone come from its USB screen feed, the capture device QuickTime uses, compared frame by frame in the same way. It needs a USB cable (not Wi-Fi), an unlocked phone, and QuickTime not showing the phone. The feed runs at 60 Hz at most, so a 120 Hz screen is measured at half its rate, and during fast motion the phone sometimes skips a frame of the feed: those refreshes are reported as unknown, never as jank. (Instruments' Animation Hitches would measure the app's own frames, but its recordings hang while looktap's device runner is up.)

Catch a spinner, toast, or transition (burst)

Start the burst in the background, then trigger the action:

looktap burst ./frames 5 &              # 5 s: a frame each time the screen changes
looktap swipe 200 250 200 700           # pull to refresh
wait
# 61 frames, 23 distinct, in ./frames

For a video of a whole flow, record it. The recording runs in the background between the two commands:

looktap record start checkout           # → ./shots/checkout.mp4
looktap tap "Add to cart" --ls
looktap tap "Checkout" --ls
looktap record stop                     # prints the path and the length

• Android records with the device's screenrecord: at most 3 minutes, and FLAG_SECURE screens come out black.
• iOS records with simctl io recordVideo. A recording started in landscape is marked to play turned.

Start from a clean state: install, reset, permissions

looktap install build/MyApp.app          # iOS: a simulator build · Android: an .apk (keeps the app's data)
# installed com.example.myapp
looktap reset com.example.myapp          # as if freshly installed: data and permissions cleared
looktap permission grant location com.example.myapp    # no prompt
looktap permission revoke photos com.example.myapp     # test the "denied" path
looktap permission reset camera com.example.myapp      # it asks again on next use
looktap stop com.example.myapp          # end it; prints "stopped" once its process is gone
looktap uninstall com.example.myapp

• reset:
  • iOS: looktap copies the app's own bundle aside, uninstalls it and installs the copy, so documents, caches, app groups and permissions all go. The keychain survives, just as it does on a real iPhone, so an app that keeps its sign-in token there may still be signed in.
  • Android: pm clear.
• Permission names are the same on both platforms: camera, microphone, location, location-always, photos, contacts, calendar, notifications, motion, all.
  • Android also takes any android.permission.*, and acts only on permissions the app declares. A reset also clears "don't ask again".
  • The iOS simulator can't set camera or notifications; looktap says so.
• These commands never guess the app from the screen. stop, uninstall, reset and permission take the app id as an argument, or from --app / looktap.config.json.

Simulated conditions: location, dark mode, text size, orientation, network, push

looktap location 51.5072 -0.1276        # GPS (Android: emulators) · `location clear` on iOS
looktap appearance dark                 # or light; no argument prints the current one
looktap textsize ax2                    # Dynamic Type / font scale: xs s m l xl xxl xxxl ax1…ax5
looktap statusbar clean                 # 9:41, full battery and Wi-Fi, for screenshots · `statusbar live`
looktap rotate landscape --ls           # or left, right, upsidedown, portrait; no argument prints it
looktap fold open --ls                  # a foldable (iPhone Duo): closed, half or open; no argument prints it
looktap network offline                 # Android: airplane mode · `network online`
looktap push com.example.myapp "Your order has shipped"          # iOS: an alert
looktap push com.example.myapp '{"aps":{"alert":"Hi"},"orderId":7}'   # or any payload / a .json file

• textsize uses the iOS Dynamic Type sizes. On Android each size maps to the nearest font scale, from 0.8 to 2.0, the default being l / large on both.
• statusbar clean uses iOS status-bar overrides, and Android's System UI demo mode.
• rotate turns the device the way Simulator's Rotate Left/Right does, or locks Android's rotation. landscape is left: the device turned counter-clockwise. ls coordinates, taps and screenshots follow the turned screen, so read the screen again afterwards. An app that doesn't support the orientation stays as it is, and rotate says so. On Android it leaves auto-rotate off.
• fold folds or opens a foldable simulator, the iPhone Duo (Xcode 27.1), the way Device Hub's fold buttons do: closed (0°) shows the outer screen, half (130°) and open (180°) the inner one. ls, taps, screenshots and recordings follow whichever screen it shows, so read the screen again afterwards. Closing takes the simulator a few seconds.
• network offline turns on Android's airplane mode and waits until the device has no route out. The iOS simulator shares the Mac's network, so it can't go offline or slow by itself.
• push delivers a remote notification the way APNs would. The app needs notification permission, as on a device: iOS 27 refuses the push outright without it (iOS 26 delivered it anyway), and push says so.
  • A payload without "aps" is sent as a silent (background) push carrying that data.
  • Android pushes can't be simulated locally. They come through Firebase (FCM), so send them from your backend or the Firebase console.

Biometrics

looktap bio match       # iOS: Face ID success · Android: enrolled finger 1
looktap bio nomatch     # iOS: Face ID failure · Android: unknown finger

On Android, enrol a fingerprint once first: Settings → Security → Fingerprint, using a screen-lock PIN, and touch with looktap bio match when it asks.

Android screenshots of secure screens

Apps that set FLAG_SECURE (common in banking and payment apps) make normal Android screenshots come out black. On emulators, looktap shot takes the frame from the emulator itself, which gets past that. --plain takes a normal on-device screenshot, for example to check that a screen is secure.

Tests and automation

What worked with your agent runs on its own afterwards. Every command is a plain shell call with an exit code, so a shell script or a CI job can run the same steps: as flow files that looktap test checks, or as plain scripts.

Test flows in CI

A flow file is a run script: one looktap command per line, # for comments. Use wait where the app takes time and expect where the flow should check something:

# flows/sign-in.txt
launch com.example.myapp
wait "Sign in"
fill Email sam@example.com
tap "Continue"
wait Welcome
expect "Sign in" --gone

$ looktap test flows/ -o results --junit results/junit.xml
✓ flows/browse.txt  9 steps · 6.1 s
✗ flows/checkout.txt  3.4 s
    step 5 (expect Total = €4.20): expect failed: "Total" is "€0.00", expected "€4.20" (waited 3 s)
    │ $ tap "Add to bag"
    │ tapped "Add to bag" at 195,612
    │ …
    evidence: results/failures/flows-checkout
✓ flows/sign-in.txt  6 steps · 4.8 s
looktap: 3 flows: 2 passed, 1 failed · 14.3 s · results/junit.xml

Flows run in order on one device, each from wherever the last one left the app, so start each with launch (or reset <app> and launch for a fresh install). Steps wait for their targets, but a flow that depends on a backend can still be slow: raise --wait rather than adding sleeps.

Flow files live in your repo, so never put real credentials in one. Sign in with a test-only account, and keep its password out of the file: test replaces ${NAME} with that environment variable (a CI secret), e.g. fill Password "${PASSWORD}".

In GitHub Actions, iOS runs on a macOS runner and Android on a Linux runner with KVM:

jobs:
  ios:
    runs-on: macos-15
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm install -g @looktap/cli
      - run: xcodebuild -scheme MyApp -sdk iphonesimulator -derivedDataPath build build
      - run: looktap boot "iPhone 16" --headless
      - run: looktap install build/Build/Products/Debug-iphonesimulator/MyApp.app
      - run: looktap test flows/ -o results --junit results/junit.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: ios-results, path: results }

  android:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm install -g @looktap/cli
      - run: ./gradlew assembleDebug
      - name: Enable KVM
        run: |
          echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666", OPTIONS+="static_node=kvm"' | sudo tee /etc/udev/rules.d/99-kvm4all.rules
          sudo udevadm control --reload-rules
          sudo udevadm trigger --name-match=kvm
      - uses: reactivecircus/android-emulator-runner@v2
        with:
          api-level: 34
          arch: x86_64
          script: |
            looktap -p android install app/build/outputs/apk/debug/app-debug.apk --grant
            looktap -p android test flows/ -o results --junit results/junit.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: android-results, path: results }

Upload results/ even when the job fails: that's where the failed flows' screenshots and logs are.

Record a flow (codegen)

On Android you can write a flow by using the app. codegen restarts the app, then turns each touch into the step that replays it:

$ looktap codegen flows/about.txt com.android.settings
+ stop com.android.settings
+ launch com.android.settings
+ wait Settings
recording com.android.settings to flows/about.txt — use the phone. …
+ scrollto "About phone"
+ tap "About phone"
+ back
+ tap "Search settings"
+ fill Search wifi
expect "Wi-Fi"
ok: "Wi-Fi" at 107,148
+ expect Wi-Fi
saved 9 steps to flows/about.txt — run it: looktap test flows/about.txt

• Taps are saved by label: a tap anywhere on a row is a tap on the row's label, when tap with that label finds the same row again. Where no label does (an icon with no accessibility label), it's saved as tapxy, with a note.
• Typing is one fill, with the field's name and its final text. Text typed into a password field isn't saved: the step reads it from an environment variable instead (fill Password "${PASSWORD}").
• Scrolling that brought a row into view becomes scrollto, which replays on any screen size. Other swipes (a carousel, a map) stay swipes. The navigation bar's Back and Home, and gesture navigation, become back and home.
• Checks: type a command into the terminal while it records, e.g. expect "Welcome". It's run, and saved if it passes. undo drops the last step.

It reads the touchscreen through adb shell getevent, so it records real fingers (not taps sent by looktap itself). It isn't available on iOS yet: write the flow by hand there, or record it on Android if the app is the same.

A repeatable screenshot sweep

#!/bin/sh
set -e
looktap launch
for route in home search messages profile settings; do
  looktap open "myapp://$route" --ls > /dev/null
  looktap shot "$route"
  looktap swipe 200 700 200 250
  looktap shot "${route}__scrolled"
done

Same screen on several devices

looktap boot "iPhone 14" --headless && looktap boot "iPhone 16 Pro" --headless
looktap devices
# ios      42E41FA7-…  iPhone 14 · simulator · iOS 26.0 · headless
# ios      F9213EDC-…  iPhone 16 Pro · simulator · iOS 26.0 · headless
# android  emulator-5554  Pixel_9a · emulator · Android 16 · window

for d in 42E41FA7-EA40-45C1-8C6E-13A70BAF1FD6 F9213EDC-180D-412B-B24C-2FD69A37D3AC; do
  looktap -d $d open myapp://home
  looktap -d $d wait "Home"
  looktap -d $d shot -o shots/$d home
done

Passing both -p and -d (or exporting LOOKTAP_PLATFORM and LOOKTAP_DEVICE) makes every command about 0.2 s faster, because looktap then skips listing devices.

For AI agents

Pick the route that fits your agent:

Every coding agent

looktap skill --install finds the coding agents on your machine (by their config folder or command) and writes the skill into the folder each one reads, with the frontmatter that agent accepts: Codex, for one, rejects fields it doesn't know. Each copy carries looktap's version: run it again after upgrading, and looktap doctor says when a copy is out of date.

looktap skill --install                          # every agent found here, user-wide
looktap skill --install --agents codex,gemini    # just these (or --agents all)
looktap skill --install --project                # into this repo, to commit for the team
looktap skill --list                             # what's supported, found and installed where
looktap skill --uninstall

Claude Code's folder follows CLAUDE_CONFIG_DIR when it's set. If Claude Code already has the looktap plugin, it's skipped: the plugin carries the skill and updates with it, and a second copy would show Claude the skill twice.

--project writes .claude/skills and .agents/skills unless you name agents: between them, Claude Code, Codex, Cursor, Gemini CLI, Copilot, OpenCode, Pi, Mistral Vibe, Antigravity and Grok Build all find it. Some agents need skills switched on first (Cursor, Gemini CLI); Hermes asks you to trust a repo's skills once (hermes skills trust). An agent without skills can read the guide from looktap skill, which prints it.

Already manage skills with npx skills? npx skills add lakpriya1s/looktap finds the skill in this repo and installs it too. It copies the guide as it is, without the per-agent frontmatter or the version that doctor checks, so looktap skill --install is the one to rerun after upgrading looktap.

An agent spends seconds per turn, so turns matter more than milliseconds. looktap is built to save them: in the benchmark the CLI needed 8 calls for a scenario that took other tools 9–27 on iOS, and 12 where they took 11–19 on Android.

• --ls on any action prints the screen once it has settled, so acting and looking take one turn instead of two:

  looktap tap @3 --ls
  looktap fill "Email" a@b.co --ls

• wait instead of guessing sleeps. After an action that loads data, --ls can return a loading skeleton, so wait for the text you expect.
• run batches steps you already know into one call and one process. It prints each step as $ <step>, stops at the first failure, and names it:

  looktap run 'tap General; wait About; tap About --ls'
  looktap run - < steps.txt          # one step per line, # for comments

• logs / crashes after a step that misbehaved. Each logs shows only what's new since the last one.
• perf / launch --time when the question is speed or memory: numbers instead of impressions. perf --for while the screen does the thing being measured, and perf --frames before and after it.
• ls --json for structured output: {screen, elements: [{ref, label, type, x, y, w, h, value}]}.
• shot --scale 0.5 costs a quarter of the image tokens when the agent needs to see the screen (layout, colours, images). Usually ls is enough and the PNG is never opened.
• Hints when the screen isn't what it seems. When a passcode or permission prompt covers the app, ls prints a hint rather than an empty list.

Claude Code plugin

In Claude Code, run:

/plugin install looktap --marketplace lakpriya1s/looktap

• What it adds:
  • the looktap skill, which teaches Claude the look-act loop, how to pick targets, when to wait, when a screenshot is worth it, how to debug a crash, and the safety rules;
  • /looktap:setup, which gets the machine ready: it installs the CLI (asking first), runs looktap doctor and explains each fix, offers to boot a simulator or emulator, walks you through a real iPhone's one-time steps, and finishes with a working ls. Add ios, android or iphone to set up just that.
• What it doesn't install: Xcode, the Android SDK and the Apple ID sign-in are yours to do; /looktap:setup tells you exactly what's missing.
• What it costs: about 150 tokens per session for the skill's description, and about 4.5k tokens each time Claude loads the skill.
• Staying current: every push to this repo is a new plugin version. Claude Code doesn't auto-update third-party plugins unless you turn it on (/plugin → Marketplaces → looktap → Enable auto-update); otherwise run /plugin marketplace update looktap now and then.
• The CLI: the skill runs looktap commands. Install them with npm install -g @looktap/cli, or the skill falls back to npx @looktap/cli.
• No MCP server inside, on purpose: with a shell, the CLI and the skill do the same job with fewer tokens, and an MCP server would add its 16 tool definitions to every session. Add @looktap/mcp separately if you want it anyway.

MCP server, for clients without a shell

Agents with a shell (Claude Code, Codex, Cursor's agent, …) are best served by the CLI and the skill above. For chat apps and other clients without a shell, such as Claude Desktop, looktap is also an MCP server, published as @looktap/mcp and listed in the MCP Registry as io.github.lakpriya1s/looktap. Nothing to install first:

{ "mcpServers": { "looktap": { "command": "npx", "args": ["-y", "@looktap/mcp"] } } }

If your client can't find npx (desktop apps start servers with a minimal PATH), install the CLI and let it write the config with absolute paths:

looktap mcp --config      # prints a ready-to-paste client config with absolute paths

{
  "mcpServers": {
    "looktap": {
      "command": "/path/to/node",
      "args": ["/path/to/looktap/bin/looktap.js", "mcp"],
      "env": { "PATH": "/path/to/node/bin:/path/to/Android/sdk/platform-tools:/usr/bin:/bin" }
    }
  }
}

• Where to paste it:
  • Claude Desktop: ~/Library/Application Support/Claude/claude_desktop_config.json
  • Cursor: ~/.cursor/mcp.json
  • Claude Code: claude mcp add looktap -- npx -y @looktap/mcp (no install needed: it runs the latest version from npm). Or the absolute paths: claude mcp add looktap -- <node> <looktap.js> mcp
• Why absolute paths: desktop apps start servers with a minimal PATH. looktap also finds adb in the Android SDK by itself.
• The tools:
  • ls, tap, tap_xy, long_press, fill, paste, code, type, key, hide_keyboard, swipe, scroll_to, back, home and wait;
  • screenshot, which returns the image itself;
  • logs and crashes;
  • perf: CPU, memory and threads, or with frames the frames and jank since the last call (for: seconds to sample);
  • looktap, for any other command line (apps, perf, launch --time, boot …), or several separated by ;.
• Actions return the settled screen unless ls: false. That saves the agent a separate ls, but each action then waits for the screen to stop moving, 0.7–1.9 s in the benchmark. Pass ls: false when you don't need to see the result.
• The device is remembered: name a platform (or device) once when both an iPhone and an Android are booted.
• Speed: the server keeps each device's helper connection open, so a warm ls comes back in 40–70 ms.

Why a CLI first? Any agent with a shell can use it with no configuration, and its output is already as small as it gets. In the benchmark the CLI beat the MCP server on both platforms, mostly because it doesn't wait for the screen to settle unless asked (--ls).

Commands

--ls works on tap, longpress, zoom, tapxy, fill, type, key, paste, code, hidekb, swipe, back, home, open, launch, bio, rotate, fold and scrollto. After the action it waits for the screen to change (up to 1.5 s, or 6 s for launch/open), then for it to stop changing, then prints it like ls.

Exit codes: 0 means OK. 2 means a usage problem: bad arguments, element not found, no device, or the helper can't start. 1 means anything else failed, including an expect that didn't hold and a test run with a failed flow. A script can set -e and stop at the first missing element.

Platform support

Every command works on all four kinds of device except the ones below. ⚠️ means it works with the limit shown; ❌ means it doesn't run there and exits 2 with that reason. looktap capabilities prints the same for the device in front of you (--json for every command).

Options, environment variables, config file

Config file. looktap uses the nearest looktap.config.json, walking up from the current folder. Put it at the root of your app's repo:

{
  "out": "shots",
  "wait": 3,
  "headless": false,
  "ios": { "app": "com.example.myapp" },
  "android": { "app": "com.example.myapp" }
}

Precedence: command-line option, then environment variable, then config file, then built-in default.

Files looktap writes

Troubleshooting

FAQ

Is this a test framework?

Not a framework, but it runs tests. expect checks the screen, and looktap test runs flow files (plain looktap commands, one per line) and reports them, with a JUnit report and the evidence of each failure. That covers smoke tests on every build, bug repros, and flows an agent wrote while building a feature: see Test flows in CI.

Flow files have no variables, loops or shared setup, and looktap doesn't mock the network or shard across devices. For a large suite that needs those, use XCTest, Espresso, Detox or Maestro.

Does it work on a real iPhone?

Yes. looktap installs a small XCUITest runner on it, signed with your own Apple ID (a free one works). See Using a real iPhone for the one-time setup and what works there.

On Android it works with emulators and with USB-debugging devices. (On a device, screenshots go through the device, so FLAG_SECURE screens are black.)

Does it install anything on my device or in my app?

Nothing on the iOS simulator. On Android it copies an 8 KB helper to /data/local/tmp and runs it with app_process, the way scrcpy does. No APK is installed, no app is modified, and there's no SDK to add to your app. looktap teardown stops the helper.

Does it work with React Native, Flutter, SwiftUI, Compose, web views?

It reads the platform accessibility tree, so anything that exposes accessibility labels works. That includes React Native (it was built while driving RN apps), SwiftUI, UIKit, Compose and Views. It also reads web content in Safari, in-app browsers and web views. Flutter isn't tested yet: it exposes elements through its semantics tree, so it should work wherever that is enabled. Elements without a label show as (no label) and can be tapped by @ref.

Linux or Windows?

iOS needs a Mac. The Android side is plain Node + adb and should work anywhere they do. It's developed and tested on macOS.

Will an Xcode update break it?

It can. The iOS helper uses private simulator frameworks, like idb does. looktap doctor tells you if something it needs has moved. Reading the screen and screenshots have held steady. Input changed in Xcode 27, and Xcode 27.1 moved a framework and replaced the Simulator app with Device Hub: looktap 0.7.1 handles both, tested on Xcode 27.1.

Does it handle foldables?

The iPhone Duo simulator (Xcode 27.1), yes: looktap fold closed|half|open folds it the way Device Hub's buttons do, and ls, taps, screenshots, recordings and rotate follow whichever screen it shows, the outer one closed and the inner one open. Android foldable emulators aren't supported yet (see the Roadmap).

Can my agent sign in to my app with it?

It can type, but it shouldn't type real credentials. Have a person sign in (and enter OTPs), then let the agent continue. The skill tells agents exactly that. See Safety rules.

With a test-only account it can: fill checks each field holds what it typed (a password by its length), and code enters a test account's fixed or staging one-time code into its boxes and checks each one. That's also how to test the sign-in form itself.

Roadmap

• Rotation verified on iPad simulators
• fold on Android foldable emulators (the emulator's own fold and unfold)
• codegen on the iOS simulator, from the Mac's clicks and keys in the Simulator window (a real iPhone gives no way to observe touches)
• Android elements by resource-id (React Native's testID), as iOS ones already are by accessibility identifier, so flows survive copy changes

Ideas and votes welcome in Issues.

Safety rules

• Never script a sign-in with real credentials, whether from a shell script or an AI agent. A person signs in and enters OTPs, then looktap takes over.
• Never burst or record a sign-in or passcode screen. The frames capture what's typed.
• Against a live backend, every tap is real. Use the smallest amounts, then undo what you created.
• Logs can contain tokens and personal data. Quote only the lines that matter, and don't commit screenshots of real customer data.

Contributing

git clone https://github.com/lakpriya1s/looktap.git
cd looktap
npm test                          # unit tests, no device needed
native/ios/build.sh               # rebuild the iOS helper (needs Xcode)
native/android/build.sh           # rebuild the Android helper (needs the Android SDK + a JDK)
node bin/looktap.js --help      # run from source
npm run site:serve                # build the website and serve it on http://127.0.0.1:4173

bin/looktap.js        CLI: arguments, device selection, commands
lib/ios.js              iOS driver: helper client, simctl, logs, crash reports
lib/android.js          Android driver: helper client, adb fallback, logcat, exits
lib/screen.js           ls rows, @refs, wait, scrollto, settle (platform-neutral)
lib/logs.js             logs/crashes windows and formatting
lib/helper.js           the JSON-lines socket protocol both helpers speak
lib/emulator.js         Android emulator gRPC frames, no dependencies
lib/mcp.js              the MCP server (stdio)
lib/png.js · lib/mp4.js · lib/util.js · lib/conditions.js · lib/permissions.js
lib/assets/             built helpers (looktap-sim, looktap-server.jar)
native/ios/             iOS helper source (Objective-C)
native/android/         Android helper source (Java)
skills/looktap/       the AI-agent guide
site/                   the website: landing page (site/src) and docs generated from this README,
                        SKILL.md and BENCHMARK.md (site/build.mjs); published by .github/workflows/pages.yml

Notes for contributors:

• Zero dependencies, on purpose. Use only Node built-ins. The helpers use only what ships with macOS/Xcode and Android.
• Helpers are versioned by content hash. After you rebuild one, the next command replaces the running copy. Commit the rebuilt file in lib/assets/.
• A new command needs:
  • a method on both Ios and Android, or, if it only reads the screen, code in lib/screen.js;
  • a case in bin/looktap.js;
  • a row in this README;
  • a line in skills/looktap/SKILL.md if agents should use it;
  • a scene that shows it in site/src/scenes.js (its covers list). npm run site:build fails until the README row and the scene are there.

Issues and PRs welcome. If looktap saved your agent some tokens, a ⭐ helps others find it.

License

AGPL-3.0 © Lakpriya Senevirathna. You're free to use, modify and self-host it. If you run a modified version as a service for others, you must publish your changes under the same license.

---

looktap is an independent open-source project, not affiliated with, endorsed by or sponsored by Apple Inc., Google LLC or the Appium project. iOS, iPhone, Xcode and Face ID are trademarks of Apple Inc. Android is a trademark of Google LLC.

Text in · taps out · looktap