karagoz ui-tree

Returns the accessibility tree of the screen: the focused window on Android, the frontmost app on iOS.

Android emulator Done
Android device Done
iOS simulator Done
iOS device Done

Usage

karagoz ui-tree [--device <id>]

Before you start

On Android: the screen on, and no other UiAutomation client connected (Appium, Maestro, uiautomator events).
On an iOS simulator: Xcode, opened once. The Command Line Tools alone have no swiftc for the helper.
On an iPhone: paired with the Mac, Developer Mode on, connected by USB cable, Settings > Developer > Enable UI Automation on, and Karagöz’s runner installed with install-runner.

Examples

Read the tree on Android

Children cut short here; the real tree nests the Chrome icon several levels deep.

$ karagoz ui-tree
{"device":"emulator-5554","rotation":0,"root":{"class":"android.widget.FrameLayout","package":"com.google.android.apps.nexuslauncher","bounds":[0,0,1080,2400],"children":[…]}}

Output

{
  "device": "emulator-5554",
  "rotation": 0,
  "root": {
    "class": "android.widget.FrameLayout",
    "package": "com.google.android.apps.nexuslauncher",
    "bounds": [0, 0, 1080, 2400],
    "children": […]
  }
}

device string: Serial, or the simulator's or iPhone's UDID.
rotation number or null: One of 0, 90, 180 or 270. On iOS also null, as in screenshot.
root object: The top node.

On Android, root and every item of children are nodes with the fields below, in this order. A node from the same tree:

{
  "class": "android.widget.TextView",
  "text": "Chrome",
  "contentDesc": "Chrome",
  "clickable": true,
  "longClickable": true,
  "focusable": true,
  "bounds": [581, 1905, 754, 2100]
}

class string: Android class, e.g. android.widget.Button. Can be empty.
package string: Only on the root, and on any node whose package differs from its parent's: the app package.
text string: Only when not empty. Jetpack Compose puts text in text, Flutter puts it in contentDesc.
contentDesc string: Only when not empty. Jetpack Compose puts text in text, Flutter puts it in contentDesc.
resourceId string: Only when not empty: the full form, com.example:id/save.
hint string: Only when not empty. Exists from Android 15 QPR2 (some API 35 images) on.
checkable boolean: Only when true.
checked boolean: Only when true.
clickable boolean: Only when true.
longClickable boolean: Only when true.
focusable boolean: Only when true.
focused boolean: Only when true.
scrollable boolean: Only when true.
selected boolean: Only when true.
password boolean: Only when true.
enabled boolean: Only when false.
bounds array: [left, top, right, bottom] in screen pixels. Clipped to the screen: a node wholly off screen is [0,0,0,0], and bottom can be above top. See below.
children array: Only when the node has any: nodes, same shape.

Up to Android 13 the clip of bounds also leaves out the status and navigation bars, so a node in the bottom band can be [0,0,0,0] while it is visible.

Other attributes that uiautomator prints (index, drawing-order, NAF) are dropped. Every node uiautomator returns is kept; there is no filtering.

The tree is read with uiautomator dump, so it covers what uiautomator covers: the focused window, and nodes visible to the user.

WebView

A WebView’s content often arrives only on the second read. When the tree has a WebView with no children, Karagöz reads once more and returns the second tree, or the first if the second read fails.

Errors

Code	When
`CAPTURE_FAILED`	The screen kept changing for uiautomator's 10 s idle wait (an animation or live content).
`CAPTURE_FAILED`	No focused window: the screen is off, an app is still starting, or the app is in a work profile or Secure Folder, which adb cannot read.
`CAPTURE_FAILED`	uiautomator was killed; the message names `adb -s <id> logcat -b crash`. A lone UTF-16 surrogate in on-screen text does this.
`CAPTURE_FAILED`	The output could not be parsed.
`AUTOMATION_BUSY`	Another UiAutomation client is connected. Only one can be at a time.
`ADB_TIMEOUT`	After 20 s.

Each CAPTURE_FAILED gives its reason in the message. Plus the device selection errors, INVALID_ARGS and the adb errors.

Notes

On Android one read takes 2.4 to 3.3 s; a screen with a fresh WebView about 5.3 s. The idle failure arrives after about 12 s.
Output size on the screens measured was about 900 to 6,200 tokens.
While a read runs, accessibility services such as TalkBack are unbound, and apps see accessibility as enabled. A service that requests the accessibility button is taken off the button and shortcut and stays off; add it back in the accessibility settings.

On an iOS simulator

The tree is the frontmost app’s, read by a helper Karagöz compiles. An iPhone 17 home screen (macOS, Xcode 27; children cut short):

{
  "device": "27024DC1-EDB0-4BF2-8010-08FE3936BAB6",
  "rotation": 0,
  "root": {
    "role": "Application",
    "label": " ",
    "bounds": [0, 0, 1206, 2622],
    "children": […]
  }
}

Two nodes from the same tree:

{
  "role": "Button",
  "label": "Settings",
  "identifier": "Settings",
  "bounds": [917, 1167, 1122, 1439]
}
{
  "role": "Slider",
  "label": "Search",
  "identifier": "spotlight-pill",
  "value": "Page 1 of 2",
  "bounds": [486, 2111, 720, 2201]
}

Node fields, in this order:

role string: The accessibility subrole when there is one, else the role, without the AX prefix: Application, Button, StaticText, Heading, TextField, SecureTextField, SearchField, Switch, Image, Group, Slider, ... Can be empty.
label string: Only when not empty: what VoiceOver reads. The home screen's root label is a single space.
identifier string: Only when not empty: the app's accessibilityIdentifier.
value string: Only when not empty: a text field's content, or its placeholder when empty; a switch's 0 or 1; a slider's text.
focused boolean: Only when true.
selected boolean: Only when true.
enabled boolean: Only when false.
bounds array: [left, top, right, bottom] in screen pixels: the frame in points times screenshot's scale, rounded to 0.01. Not clipped to the screen: an element below the fold has a bottom past the screen's height.
children array: Only when the node has any: nodes, same shape.

The root is the frontmost app, or SpringBoard on the home screen.
An alert, a system alert included, is in the tree in place of what it covers.
Only elements on screen are listed. Keyboard keys and WebView content have not been measured.
An app stopped in a debugger (an Xcode breakpoint or pause) reads as a root Application with bounds [0,0,0,0] and no label or children. The call still exits 0.

The first call compiles the helper and takes a few seconds; later calls take about 0.25 s. The helper only reads: it installs nothing in the simulator, needs no macOS permission and leaves no process running.

Errors on iOS

Code	When
`HELPER_FAILED`	This Mac cannot read iOS trees: no Xcode selected, or no `swiftc` at the selected Xcode's toolchain path.
`HELPER_FAILED`	A failed compile (swiftc's first error lines), or a crashed helper.
`HELPER_FAILED`	An Xcode that changed a private interface Karagöz uses (`update karagoz`).
`CAPTURE_FAILED`	No frontmost application, or helper output that could not be parsed.
`CAPTURE_FAILED`	An accessibility request took over 5 s, or no answer came within 30 s. With Control Center open, a read ends this way after 30 s.
`CAPTURE_FAILED`	The app in front opened while the simulator's app accessibility setting was off, so it reads as an empty root. An Xcode UI test, or another XCUI client, turns the setting off when it ends. Karagöz turns the setting back on and fails with `the app in front opened while the simulator's app accessibility was off ...: restart the app (terminate, then launch) and read again`. After `terminate` and `launch` the tree is full.
`DEVICE_NOT_READY`	The simulator is still booting. This comes before anything is compiled.

Plus the device selection errors and the simctl and devicectl errors from listing simulators and iPhones.

On an iPhone

The tree is the frontmost app’s, read by Karagöz’s own runner on the phone: a small XCUITest bundle that install-runner builds with your Apple team and installs there.

When nothing answers, ui-tree starts the runner with xcodebuild test-without-building.
The runner stays up between calls and stops itself after 10 minutes without a read or an input. No daemon is left behind: the only process is that xcodebuild, which exits with the runner.
ui-tree never installs the runner: without it the error is RUNNER_NOT_INSTALLED.

The node fields are the simulator’s, with these differences:

role is XCTest’s element type: Application, Window, Group (XCTest’s Other), Button, StaticText, Heading (a static text with the header trait), Cell, Icon, Image, Switch (also XCTest’s Toggle), TextField, SearchField, Keyboard, Key, NavigationBar, CollectionView, … A type Karagöz does not know is Element<number>.
XCTest’s layers stay: windows and unlabeled groups.
Elements off screen are listed; icons on another home screen page have bounds [0,0,0,0].
In landscape XCTest reports the keyboard in portrait coordinates; Karagöz turns it into the screen’s. Only the phone turned clockwise (rotation 270) was measured; the other direction is turned by symmetry.
rotation and the scale come from devicectl device info displays, as in screenshot, so bounds are that screenshot’s pixels.
The root is the app in front, or SpringBoard on the home screen. A system alert, when one is up, is the root’s only child.
Not measured on a phone: an alert, Control Center, Notification Center, the lock screen.

Starting the runner and reading the first tree takes 3 to 4.4 s; later reads take about 0.7 s for the whole command.

Errors on an iPhone

Code	When
`RUNNER_NOT_INSTALLED`	No runner to start. The message names `karagoz install-runner --device <UDID>` and the MCP tool `install_runner`.
`DEVICE_NOT_READY`	The iPhone is not connected by USB cable.
`DEVICE_NOT_READY`	The iPhone is locked. The start ends at once.
`DEVICE_NOT_READY`	The iPhone does not trust the runner's developer yet (free accounts: Settings > General > VPN & Device Management).
`DEVICE_NOT_READY`	The runner did not start within 60 s. Look at the phone for a passcode prompt to enable UI automation, and check Settings > Developer > Enable UI Automation.
`DEVICE_NOT_READY`	Something else answers on the runner's port without this Mac's key: another app, or a runner another user of this Mac started, which stops within 10 minutes. Restart the iPhone if it stays.
`DEVICE_NOT_READY`	Another app holds the runner's port.
`HELPER_FAILED`	The selected Xcode has no `xcodebuild`: `xcodebuild not found in <developer dir>`, as `install-runner` says.
`HELPER_FAILED`	`xcodebuild` could not start the runner for a reason Karagöz does not know. The message has its last error line and the log path.
`HELPER_FAILED`	This iOS changed an interface the runner uses (`update karagoz`).
`CAPTURE_FAILED`	XCTest could not take the snapshot (its message), no answer within 30 s, or the runner closed the connection. A snapshot stuck for 20 s ends the runner; the next call starts a new one.

Plus the device selection errors, the pairing states among them, and the devicectl errors.

Notes on an iPhone

An iPhone on Wi-Fi only is not supported: Karagöz reaches the runner over the USB cable.
The runner plays one input or ui-tree read at a time. A call that waits more than 8 s behind another fails with HELPER_FAILED (CAPTURE_FAILED for ui-tree) and plays nothing.
With a passcode set, a start can wait up to 60 s. Before UI automation can start, iOS asks for the passcode on the phone (“Enter iPhone Passcode for XCTest”), and xcodebuild says nothing about it. The prompt comes:
- at the first start after a restart
- about once a day
- at the first start after the phone was locked
xcodebuild’s output for the last start is in ~/Library/Caches/karagoz/runner/<UDID>.log.

How the runner is reached and secured

The runner listens on 127.0.0.1 port 27753 on the phone. Karagöz reaches it through macOS’s usbmuxd, so nothing on the phone’s network can.
Other apps on the phone can reach that port, so each start gives the runner a new secret. It is kept in ~/Library/Caches/karagoz/runner/<UDID>.token (mode 0600) and handed over in a 0600 copy of the build’s .xctestrun, never on a command line.
Every request and every answer carries an HMAC-SHA256 under the secret, so the secret never crosses the connection. The runner refuses anything unsigned (HTTP 401).
Two first calls at once start one runner: a lock file in ~/Library/Caches/karagoz/locks/ lines them up, and the kernel releases it if a call crashes.
The runner build is in ~/Library/Caches/karagoz/runner/, one folder per Karagöz version, Xcode and team.

MCP equivalent

MCP equivalentui_tree {}MCP tools

Arguments: device. The names are the CLI's option and positional names.

karagoz screenshot: On iOS, rotation reads as in its output; on an iPhone, the tree's bounds are that screenshot's pixels.
karagoz install-runner: Build and install the runner that reads the tree on an iPhone.