# karagoz ui-tree Returns the accessibility tree of the screen: the focused window on Android, the frontmost app on iOS. - Android emulator Done - Android device Done - iOS simulator Done - iOS device Done ## Usage ```console title="~/app" karagoz ui-tree [--device ] ``` ## Before you start - On Android: the screen on, and no other UiAutomation client connected (Appium, Maestro, `uiautomator events`). - On an iOS simulator: Xcode, opened once. The Command Line Tools alone have no `swiftc` for the helper. - On an iPhone: paired with the Mac, Developer Mode on, connected by USB cable, Settings > Developer > Enable UI Automation on, and Karagöz’s runner installed with [`install-runner`](https://karagozmcp.com/docs/commands/install-runner/). ## Examples ### Read the tree on Android Children cut short here; the real tree nests the Chrome icon several levels deep. ```console title="~/app" $ karagoz ui-tree {"device":"emulator-5554","rotation":0,"root":{"class":"android.widget.FrameLayout","package":"com.google.android.apps.nexuslauncher","bounds":[0,0,1080,2400],"children":[…]}} ``` ## Output ```json { "device": "emulator-5554", "rotation": 0, "root": { "class": "android.widget.FrameLayout", "package": "com.google.android.apps.nexuslauncher", "bounds": [0, 0, 1080, 2400], "children": […] } } ``` - `device` string: Serial, or the simulator's or iPhone's UDID. - `rotation` number or null: One of `0`, `90`, `180` or `270`. On iOS also `null`, as in [`screenshot`](https://karagozmcp.com/docs/commands/screenshot/). - `root` object: The top node. On Android, `root` and every item of `children` are nodes with the fields below, in this order. A node from the same tree: ```json { "class": "android.widget.TextView", "text": "Chrome", "contentDesc": "Chrome", "clickable": true, "longClickable": true, "focusable": true, "bounds": [581, 1905, 754, 2100] } ``` - `class` string: Android class, e.g. `android.widget.Button`. Can be empty. - `package` string: Only on the root, and on any node whose package differs from its parent's: the app package. - `text` string: Only when not empty. Jetpack Compose puts text in `text`, Flutter puts it in `contentDesc`. - `contentDesc` string: Only when not empty. Jetpack Compose puts text in `text`, Flutter puts it in `contentDesc`. - `resourceId` string: Only when not empty: the full form, `com.example:id/save`. - `hint` string: Only when not empty. Exists from Android 15 QPR2 (some API 35 images) on. - `checkable` boolean: Only when `true`. - `checked` boolean: Only when `true`. - `clickable` boolean: Only when `true`. - `longClickable` boolean: Only when `true`. - `focusable` boolean: Only when `true`. - `focused` boolean: Only when `true`. - `scrollable` boolean: Only when `true`. - `selected` boolean: Only when `true`. - `password` boolean: Only when `true`. - `enabled` boolean: Only when `false`. - `bounds` array: `[left, top, right, bottom]` in [screen pixels](https://karagozmcp.com/docs/output/#coordinates). Clipped to the screen: a node wholly off screen is `[0,0,0,0]`, and bottom can be above top. See below. - `children` array: Only when the node has any: nodes, same shape. Up to Android 13 the clip of `bounds` also leaves out the status and navigation bars, so a node in the bottom band can be `[0,0,0,0]` while it is visible. Other attributes that uiautomator prints (`index`, `drawing-order`, `NAF`) are dropped. Every node uiautomator returns is kept; there is no filtering. The tree is read with `uiautomator dump`, so it covers what uiautomator covers: the focused window, and nodes visible to the user. ## WebView A WebView’s content often arrives only on the second read. When the tree has a WebView with no children, Karagöz reads once more and returns the second tree, or the first if the second read fails. ## Errors | Code | When | | ----------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------- | | [`CAPTURE_FAILED`](https://karagozmcp.com/docs/error-codes/#capture-failed) | The screen kept changing for uiautomator's 10 s idle wait (an animation or live content). | | [`CAPTURE_FAILED`](https://karagozmcp.com/docs/error-codes/#capture-failed) | No focused window: the screen is off, an app is still starting, or the app is in a work profile or Secure Folder, which adb cannot read. | | [`CAPTURE_FAILED`](https://karagozmcp.com/docs/error-codes/#capture-failed) | uiautomator was killed; the message names `adb -s logcat -b crash`. A lone UTF-16 surrogate in on-screen text does this. | | [`CAPTURE_FAILED`](https://karagozmcp.com/docs/error-codes/#capture-failed) | The output could not be parsed. | | [`AUTOMATION_BUSY`](https://karagozmcp.com/docs/error-codes/#automation-busy) | Another UiAutomation client is connected. Only one can be at a time. | | [`ADB_TIMEOUT`](https://karagozmcp.com/docs/error-codes/#adb-timeout) | After 20 s. | Each `CAPTURE_FAILED` gives its reason in the message. Plus the [device selection errors](https://karagozmcp.com/docs/device-selection/#errors), [`INVALID_ARGS`](https://karagozmcp.com/docs/error-codes/#invalid-args) and the adb errors. ## Notes - On Android one read takes 2.4 to 3.3 s; a screen with a fresh WebView about 5.3 s. The idle failure arrives after about 12 s. - Output size on the screens measured was about 900 to 6,200 tokens. - While a read runs, accessibility services such as TalkBack are unbound, and apps see accessibility as enabled. A service that requests the accessibility button is taken off the button and shortcut and stays off; add it back in the accessibility settings. ## On an iOS simulator The tree is the frontmost app’s, read by a helper Karagöz compiles. An iPhone 17 home screen (macOS, Xcode 27; children cut short): ```json { "device": "27024DC1-EDB0-4BF2-8010-08FE3936BAB6", "rotation": 0, "root": { "role": "Application", "label": " ", "bounds": [0, 0, 1206, 2622], "children": […] } } ``` Two nodes from the same tree: ```json { "role": "Button", "label": "Settings", "identifier": "Settings", "bounds": [917, 1167, 1122, 1439] } { "role": "Slider", "label": "Search", "identifier": "spotlight-pill", "value": "Page 1 of 2", "bounds": [486, 2111, 720, 2201] } ``` Node fields, in this order: - `role` string: The accessibility subrole when there is one, else the role, without the `AX` prefix: `Application`, `Button`, `StaticText`, `Heading`, `TextField`, `SecureTextField`, `SearchField`, `Switch`, `Image`, `Group`, `Slider`, ... Can be empty. - `label` string: Only when not empty: what VoiceOver reads. The home screen's root label is a single space. - `identifier` string: Only when not empty: the app's `accessibilityIdentifier`. - `value` string: Only when not empty: a text field's content, or its placeholder when empty; a switch's `0` or `1`; a slider's text. - `focused` boolean: Only when `true`. - `selected` boolean: Only when `true`. - `enabled` boolean: Only when `false`. - `bounds` array: `[left, top, right, bottom]` in screen pixels: the frame in points times `screenshot`'s `scale`, rounded to 0.01. Not clipped to the screen: an element below the fold has a bottom past the screen's height. - `children` array: Only when the node has any: nodes, same shape. * The root is the frontmost app, or SpringBoard on the home screen. * An alert, a system alert included, is in the tree in place of what it covers. * Only elements on screen are listed. Keyboard keys and WebView content have not been measured. * An app stopped in a debugger (an Xcode breakpoint or pause) reads as a root `Application` with `bounds` `[0,0,0,0]` and no label or children. The call still exits 0. The first call compiles the [helper](https://karagozmcp.com/docs/requirements/#cache-folder) and takes a few seconds; later calls take about 0.25 s. The helper only reads: it installs nothing in the simulator, needs no macOS permission and leaves no process running. ### Errors on iOS | Code | When | | ------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`HELPER_FAILED`](https://karagozmcp.com/docs/error-codes/#helper-failed) | This Mac cannot read iOS trees: no Xcode selected, or no `swiftc` at the selected Xcode's toolchain path. | | [`HELPER_FAILED`](https://karagozmcp.com/docs/error-codes/#helper-failed) | A failed compile (swiftc's first error lines), or a crashed helper. | | [`HELPER_FAILED`](https://karagozmcp.com/docs/error-codes/#helper-failed) | An Xcode that changed a private interface Karagöz uses (`update karagoz`). | | [`CAPTURE_FAILED`](https://karagozmcp.com/docs/error-codes/#capture-failed) | No frontmost application, or helper output that could not be parsed. | | [`CAPTURE_FAILED`](https://karagozmcp.com/docs/error-codes/#capture-failed) | An accessibility request took over 5 s, or no answer came within 30 s. With Control Center open, a read ends this way after 30 s. | | [`CAPTURE_FAILED`](https://karagozmcp.com/docs/error-codes/#capture-failed) | The app in front opened while the simulator's app accessibility setting was off, so it reads as an empty root. An Xcode UI test, or another XCUI client, turns the setting off when it ends. Karagöz turns the setting back on and fails with `the app in front opened while the simulator's app accessibility was off ...: restart the app (terminate, then launch) and read again`. After `terminate` and `launch` the tree is full. | | [`DEVICE_NOT_READY`](https://karagozmcp.com/docs/error-codes/#device-not-ready) | The simulator is still booting. This comes before anything is compiled. | Plus the device selection errors and the simctl and devicectl errors from listing simulators and iPhones. ## On an iPhone The tree is the frontmost app’s, read by Karagöz’s own runner on the phone: a small XCUITest bundle that `install-runner` builds with your Apple team and installs there. - When nothing answers, `ui-tree` starts the runner with `xcodebuild test-without-building`. - The runner stays up between calls and stops itself after 10 minutes without a read or an input. No daemon is left behind: the only process is that `xcodebuild`, which exits with the runner. - `ui-tree` never installs the runner: without it the error is [`RUNNER_NOT_INSTALLED`](https://karagozmcp.com/docs/error-codes/#runner-not-installed). The node fields are the simulator’s, with these differences: - `role` is XCTest’s element type: `Application`, `Window`, `Group` (XCTest’s `Other`), `Button`, `StaticText`, `Heading` (a static text with the header trait), `Cell`, `Icon`, `Image`, `Switch` (also XCTest’s `Toggle`), `TextField`, `SearchField`, `Keyboard`, `Key`, `NavigationBar`, `CollectionView`, … A type Karagöz does not know is `Element`. - XCTest’s layers stay: windows and unlabeled groups. - Elements off screen are listed; icons on another home screen page have `bounds` `[0,0,0,0]`. - In landscape XCTest reports the keyboard in portrait coordinates; Karagöz turns it into the screen’s. Only the phone turned clockwise (`rotation` `270`) was measured; the other direction is turned by symmetry. - `rotation` and the scale come from `devicectl device info displays`, as in `screenshot`, so `bounds` are that screenshot’s pixels. - The root is the app in front, or SpringBoard on the home screen. A system alert, when one is up, is the root’s only child. - Not measured on a phone: an alert, Control Center, Notification Center, the lock screen. Starting the runner and reading the first tree takes 3 to 4.4 s; later reads take about 0.7 s for the whole command. ### Errors on an iPhone | Code | When | | --------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | [`RUNNER_NOT_INSTALLED`](https://karagozmcp.com/docs/error-codes/#runner-not-installed) | No runner to start. The message names `karagoz install-runner --device ` and the MCP tool `install_runner`. | | [`DEVICE_NOT_READY`](https://karagozmcp.com/docs/error-codes/#device-not-ready) | The iPhone is not connected by USB cable. | | [`DEVICE_NOT_READY`](https://karagozmcp.com/docs/error-codes/#device-not-ready) | The iPhone is locked. The start ends at once. | | [`DEVICE_NOT_READY`](https://karagozmcp.com/docs/error-codes/#device-not-ready) | The iPhone does not trust the runner's developer yet (free accounts: Settings > General > VPN & Device Management). | | [`DEVICE_NOT_READY`](https://karagozmcp.com/docs/error-codes/#device-not-ready) | The runner did not start within 60 s. Look at the phone for a passcode prompt to enable UI automation, and check Settings > Developer > Enable UI Automation. | | [`DEVICE_NOT_READY`](https://karagozmcp.com/docs/error-codes/#device-not-ready) | Something else answers on the runner's port without this Mac's key: another app, or a runner another user of this Mac started, which stops within 10 minutes. Restart the iPhone if it stays. | | [`DEVICE_NOT_READY`](https://karagozmcp.com/docs/error-codes/#device-not-ready) | Another app holds the runner's port. | | [`HELPER_FAILED`](https://karagozmcp.com/docs/error-codes/#helper-failed) | The selected Xcode has no `xcodebuild`: `xcodebuild not found in `, as `install-runner` says. | | [`HELPER_FAILED`](https://karagozmcp.com/docs/error-codes/#helper-failed) | `xcodebuild` could not start the runner for a reason Karagöz does not know. The message has its last error line and the log path. | | [`HELPER_FAILED`](https://karagozmcp.com/docs/error-codes/#helper-failed) | This iOS changed an interface the runner uses (`update karagoz`). | | [`CAPTURE_FAILED`](https://karagozmcp.com/docs/error-codes/#capture-failed) | XCTest could not take the snapshot (its message), no answer within 30 s, or the runner closed the connection. A snapshot stuck for 20 s ends the runner; the next call starts a new one. | Plus the device selection errors, the pairing states among them, and the devicectl errors. ### Notes on an iPhone - An iPhone on Wi-Fi only is not supported: Karagöz reaches the runner over the USB cable. - The runner plays one input or `ui-tree` read at a time. A call that waits more than 8 s behind another fails with [`HELPER_FAILED`](https://karagozmcp.com/docs/error-codes/#helper-failed) ([`CAPTURE_FAILED`](https://karagozmcp.com/docs/error-codes/#capture-failed) for `ui-tree`) and plays nothing. - With a passcode set, a start can wait up to 60 s. Before UI automation can start, iOS asks for the passcode on the phone (“Enter iPhone Passcode for XCTest”), and xcodebuild says nothing about it. The prompt comes: - at the first start after a restart - about once a day - at the first start after the phone was locked - `xcodebuild`’s output for the last start is in `~/Library/Caches/karagoz/runner/.log`. How the runner is reached and secured - The runner listens on 127.0.0.1 port 27753 on the phone. Karagöz reaches it through macOS’s usbmuxd, so nothing on the phone’s network can. - Other apps on the phone can reach that port, so each start gives the runner a new secret. It is kept in `~/Library/Caches/karagoz/runner/.token` (mode 0600) and handed over in a 0600 copy of the build’s `.xctestrun`, never on a command line. - Every request and every answer carries an HMAC-SHA256 under the secret, so the secret never crosses the connection. The runner refuses anything unsigned (HTTP 401). - Two first calls at once start one runner: a lock file in `~/Library/Caches/karagoz/locks/` lines them up, and the kernel releases it if a call crashes. - The runner build is in `~/Library/Caches/karagoz/runner/`, one folder per Karagöz version, Xcode and team. ## MCP equivalent MCP equivalent `ui_tree {}` [MCP tools](https://karagozmcp.com/docs/mcp/#tools) Arguments: `device`. The names are the CLI's option and positional names. ## Related commands - [karagoz screenshot](https://karagozmcp.com/docs/commands/screenshot/): On iOS, rotation reads as in its output; on an iPhone, the tree's bounds are that screenshot's pixels. - [karagoz install-runner](https://karagozmcp.com/docs/commands/install-runner/): Build and install the runner that reads the tree on an iPhone.