DSL Reference

Overview

A DSL script is a list of steps that MobAI runs on one device in a single request. Send it as JSON to POST /api/v1/devices/{id}/dsl/execute on the local API at http://127.0.0.1:8686. The execute_dsl tool of MobAI's MCP server takes the same JSON. The same steps can also be written as .mob files, one step per line, which is how saved tests are stored. Every action on this page shows the JSON step and, where one exists, the .mob line.

JSON

{
  "version": "0.2",
  "steps": [
    {"action": "open_app", "bundle_id": "com.apple.Preferences"},
    {"action": "tap", "predicate": {"text_contains": "General"}},
    {"action": "wait_for", "predicate": {"text": "About"}, "timeout_ms": 5000},
    {"action": "observe", "include": ["ui_tree"]}
  ]
}

.mob

app "com.apple.Preferences"
tap ~"General"
wait_for "About" timeout:5000
observe

Script format

A script is a JSON object with a steps array. Everything else is optional.

Script keys

Step keys

Every step takes these keys, next to the keys of its action.

Coordinates

Coordinates are {"x": 120, "y": 640} in screen units, the same units as the bounds in the UI tree. Use {"x_percent": 50, "y_percent": 30} (0 to 100) for points that work on any screen size. One axis can be a percent and the other absolute. A plain pair [120, 640] is read as x and y.

Forgiving input

Action names ignore case, underscores, hyphens and spaces, so pressKey and Press-Key both mean press_key. Keys of a step, a predicate, near and coordinates can be camelCase (bundleId, clearFirst, textContains). Keys inside other objects, such as on_fail, to_element and filter, must be snake_case. coordinate, coordinates, point and position are read as coords, selector as predicate, and ms and milliseconds as timeout_ms. observe also takes "ui_tree": true style flags in place of include.

A step key MobAI does not know is ignored, so check the spelling when a setting seems to have no effect. A predicate key it does not know fails the step with VALIDATION_ERROR, because a predicate that matches nothing specific would match every element.

Action aliases

These names run the action on the left. wait is not an alias, because it could mean delay or wait_for.

.mob files

A .mob file holds one step per line. Text goes in double quotes, with \" for a quote inside. Settings after the target are key:value pairs, such as timeout:5000 or on_fail:skip. Lines that start with # are comments, except for these:

run "./login.mob" email="[email protected]" exists only in .mob: it pastes another file's steps in place, with values for its # Param: lines. script "./seed.js" is the .mob form of run_script. Test runs of .mob files use deterministic mode with settle on, and on iOS a step that finds no element in the UI tree also looks at the text on screen.

All actions

Every action MobAI accepts in a script. Native actions work on app screens through the UI tree. The second column is the .mob keyword.

request_user_input is also an action name, but it always fails in a script run.

Apps and links

open_app

Launches an app by its bundle ID (iOS) or package name (Android). With "alerts" set, a system alert that shows right after launch is cleared.

Also accepted as: launch_app, start_app

JSON

{"action": "open_app", "bundle_id": "com.example.app"}
{"action": "open_app", "bundle_id": "com.example.app", "fresh": true, "arguments": ["-UITesting"], "environment": {"API_URL": "http://localhost:3000"}}

.mob

app "com.example.app"
app "com.example.app" fresh arg:"-UITesting" env:"API_URL=http://localhost:3000"

The result echoes what reached the app under launch. Launching an app that is not installed fails with a message that says so.

kill_app

Ends a running app.

Also accepted as: terminate_app, close_app, stop_app, force_stop

JSON

{"action": "kill_app", "bundle_id": "com.example.app"}

.mob

kill_app "com.example.app"

In .mob, kill_app never fails the run: it is skipped when it fails.

open_link

Opens a URL or a deep link. The system hands a custom scheme such as myapp:// to the app that registered it, so you can jump straight to a screen.

Also accepted as: open_url, openurl, deeplink, open_deeplink, open_deep_link

JSON

{"action": "open_link", "url": "myapp://profile/42"}
{"action": "open_link", "url": "https://example.com"}

.mob

open_link "myapp://profile/42"

Touch and gestures

Touch actions take a predicate (see Predicates) or coords. A predicate taps the center of the element it finds. After a gesture, MobAI checks the screen, and when nothing changed the step still passes but carries a warning that the target may be covered or outside its hit area.

tap

Taps an element or a point. When several elements match and exactly one of them is a control (a button among labels with the same text), that one is tapped. On Android, an element covered by a sticky bar is scrolled into the open first.

With the script's settle on, the tap waits for a moving target to stop. In the web context, tap clicks the element that css_selector finds.

Also accepted as: tap_on, click, touch

JSON

{"action": "tap", "predicate": {"text_contains": "Settings"}}
{"action": "tap", "coords": {"x": 120, "y": 640}}
{"action": "tap", "coords": {"x_percent": 50, "y_percent": 30}}

.mob

tap ~"Settings"
tap 120,640
tap 50%,30%

double_tap

Taps twice in quick succession.

Also accepted as: double_tap_on

JSON

{"action": "double_tap", "predicate": {"text": "Photo"}}

.mob

double_tap "Photo"

long_press

Presses and holds. Picks the control among same-text matches the way tap does.

Also accepted as: long_press_on, long_tap

JSON

{"action": "long_press", "predicate": {"text": "Message"}, "duration_ms": 1500}

.mob

long_press "Message" duration:1500

two_finger_tap

Taps with two fingers at once, for maps and other views that zoom out on it.

JSON

{"action": "two_finger_tap", "predicate": {"text_contains": "Map"}}

.mob

two_finger_tap ~"Map"

pinch

Pinches around the center of an element or a point.

JSON

{"action": "pinch", "predicate": {"text_contains": "Map"}, "scale": 0.5}

.mob

pinch ~"Map" scale:0.5

swipe

Swipes like a finger. direction is the way the finger moves, so "up" moves content up. To bring content into view, scroll is usually simpler.

JSON

{"action": "swipe", "direction": "up", "distance": "medium"}
{"action": "swipe", "direction": "left", "from": {"predicate": {"text": "Card 1"}}}
{"action": "swipe", "from_coords": {"x": 200, "y": 700}, "to_coords": {"x": 200, "y": 200}, "duration_ms": 400}

.mob

swipe up distance:medium
swipe from:200,700 to:200,200 duration:400

A swipe by direction stops at least 50 screen units from the edge.

scroll

Scrolls to see more content. direction is where you want to look, so "down" shows what is below. With to_element, it scrolls until that element shows.

The search stops with NO_MATCH and the message reached end of scrollable content when the screen stops moving, or element not found after scrolling when max_scrolls runs out while the list still moves.

JSON

{"action": "scroll", "direction": "down", "to_element": {"predicate": {"text": "Privacy"}}, "max_scrolls": 10}
{"action": "scroll", "direction": "down", "to_element": {"predicate": {"text": "Delete account"}, "fully_visible": true}}
{"action": "scroll", "direction": "right", "predicate": {"type": "scrollview"}, "amount": "small"}

.mob

scroll down to "Privacy" max_scrolls:10
scroll right in type:scrollview amount:small

drag

Drags from one element or point to another. Each end can be an element or coordinates, and the two can be mixed in JSON. When a predicate matches several elements, the first one is used.

JSON

{"action": "drag", "from": {"predicate": {"text": "Item 1"}}, "to_element": {"predicate": {"text": "Trash"}}}
{"action": "drag", "from_coords": {"x": 100, "y": 200}, "to_coords": {"x": 300, "y": 400}, "duration_ms": 800, "press_duration_ms": 500, "hold_duration_ms": 200}

.mob

drag "Item 1" to "Trash"
drag 100,200 to 300,400 press_duration:500 hold_duration:200 duration:800

drag_path

Drags one finger along a path of points, for unlock patterns, drawing or curved moves.

JSON

{"action": "drag_path", "points": [{"x": 100, "y": 400}, {"x": 150, "y": 300, "duration_ms": 150}, {"x": 300, "y": 500, "duration_ms": 300}]}

.mob

drag_path 100,400 150,300:150 300,500:300

perform_actions

Sends raw pointer and key input in the W3C Actions format, for gestures no other action covers: several fingers, exact timing, or pointer and keys together. Two pointer sources with different ids move as two fingers at once. In pointerMove items, x_percent and y_percent (0 to 100) can replace x and y.

Also accepted as: w3c_actions, perform_gesture

JSON

{"action": "perform_actions", "actions": [{"type": "pointer", "id": "finger1", "parameters": {"pointerType": "touch"}, "actions": [{"type": "pointerMove", "duration": 0, "x_percent": 50, "y_percent": 50}, {"type": "pointerDown", "button": 0}, {"type": "pause", "duration": 300}, {"type": "pointerMove", "duration": 500, "x_percent": 50, "y_percent": 20}, {"type": "pointerUp", "button": 0}]}]}

Fails with a clear message on a device that cannot take raw input.

Text and keys

type

Types text. With a predicate, MobAI taps the field first unless it already has focus. Without one, the keyboard must be open or a field must have focus, otherwise the step fails.

In the web context, predicate needs a css_selector.

Also accepted as: input_text, type_text, enter_text, input, set_text

JSON

{"action": "type", "predicate": {"text_contains": "Email"}, "text": "[email protected]", "clear_first": true}
{"action": "type", "text": "hello"}

.mob

type ~"Email" → "[email protected]" clear
type "hello"

In .mob, the field and the text are joined by → or ->.

type_secret

Types a credential saved in MobAI, by its id. The value is typed on the device and never appears in the script, the logs or the result. Targets the field the same way as type.

Also accepted as: type_password, input_secret, type_credential

JSON

{"action": "type_secret", "predicate": {"text_contains": "Password"}, "secret_id": "qa-password"}

.mob

type_secret ~"Password" → "qa-password"

clear

Clears a text field without typing anything. With a predicate it focuses that field first, without one it clears the field that has focus.

Also accepted as: clear_text, clear_field, erase_text, erase

JSON

{"action": "clear", "predicate": {"text_contains": "Search"}}
{"action": "clear"}

.mob

clear ~"Search"
clear

hide_keyboard

Closes the on-screen keyboard.

Also accepted as: dismiss_keyboard

JSON

{"action": "hide_keyboard"}

.mob

hide_keyboard

copy_text

Copies an element's text to the device clipboard.

Also accepted as: copy_text_from

JSON

{"action": "copy_text", "predicate": {"text_contains": "Code"}, "store_as": "code"}

.mob

copy_text ~"Code"

paste_text

Types the clipboard's text into the field that has focus. It takes no target: focus the field first.

Also accepted as: paste

JSON

{"action": "paste_text"}

.mob

paste_text

press_key

Presses a keyboard key or a hardware button.

Also accepted as: key_press

JSON

{"action": "press_key", "key": "enter"}
{"action": "press_key", "key": "back"}

.mob

press_key enter

Android: back, home, recent_apps, enter, tab, delete, escape, volume_up, volume_down, mute, power, play_pause, next, previous

iOS: enter, tab, delete, home, volume_up, volume_down. There is no back or recent apps key on iOS: tap the app's back button or swipe.

Web context: enter, tab, delete, escape. Enter also submits the form of the field that has focus.

navigate

Goes to the home screen, back, or to recent apps. A target that starts with http:// or https:// opens that URL, in the selected web page when there is one, else on the device.

Platforms: back and recent_apps are Android only. home works on both.

JSON

{"action": "navigate", "target": "home"}

.mob

navigate home
navigate back

toggle

Turns a switch, toggle button or checkbox to a state. If it is already there, nothing is tapped. Fails when the element found is not a switch: find the switch by its label with near.

JSON

{"action": "toggle", "predicate": {"type": "switch", "near": {"text_contains": "Wi-Fi"}}, "state": "on"}

.mob

toggle type:switch near ~"Wi-Fi" on

The result has toggled: true when MobAI tapped, false when the switch was already in that state.

select_picker

Turns a picker wheel to the row with the given text: an iOS picker wheel (date, time and custom pickers) or an Android NumberPicker (time and date pickers in spinner mode). One step replaces a loop of swipes and checks.

Target the wheel itself, with index when a picker has several wheels, not the picker around it. On iOS, "7" matches a row that reads 7 or starts with the word 7, such as 7 o'clock. On Android, "7" also matches 07. Fails when there is no wheel at the target or no row matches. Not for drop-down menus: tap those and tap the option.

Also accepted as: pick, set_picker, picker_select, select_wheel

JSON

{"action": "select_picker", "predicate": {"type": "PickerWheel", "index": 1}, "value": "March"}
{"action": "select_picker", "predicate": {"type": "NumberPicker", "index": 0}, "value": "7"}

.mob

select_picker type:PickerWheel [1] "March"
select_picker 200,700 "7"

Screen, waits and recording

observe

Reads the screen. In the default explore mode only the last observe of a script runs, so put one at the end to see the result of the steps before it.

A ui_tree is a text list of elements, one per line, such as [4] Button "Settings" #settings (16,120 358x44): index, type, text, id, state and bounds. On iOS, a tree too large to return comes back cut down to its controls, with the text on screen added from OCR.

JSON

{"action": "observe", "include": ["ui_tree"]}
{"action": "observe", "include": ["ui_tree"], "filter": {"text_regex": "Settings|Wi-Fi"}}
{"action": "observe", "include": ["ui_tree"], "compact": true}
{"action": "observe", "include": ["screenshot"]}

.mob

observe

In .mob, observe reads the UI tree, a screenshot, the installed apps and the activity.

On iOS, a screenshot without ui_tree also returns ocr_elements and icons: text and unlabeled icons found on the image, each with its center point to tap. Icons carry a short list of likely names, closest first, with a distance where lower means a closer match.

Result

{
  "observations": {
    "native": {
      "ui_tree": "[3] StaticText \"General\" (16,98 120x22)\n[4] Button \"Settings\" #settings (16,120 358x44)\n",
      "screenshot": "/tmp/mobai/screenshots/observe_1767000000000.png",
      "screenshot_saved": true
    }
  }
}

The screenshot is saved on this computer and the result holds its path. screenshot_scale appears when the image is smaller than the screen: multiply a point read off the image by it before tapping there. Other keys, when asked for: installed_apps (bundleId, name, version, siri), ocr_elements (text, confidence, centerX, centerY, width, height), icons and page_info (id, title, url).

audit

Checks the current screen for touch targets that are too small, clipped text, alignment, spacing, safe area and accessibility labels. Returns the findings and a screenshot with the flagged elements boxed. Findings are measurements, not verdicts: look at the screenshot to decide which ones are real problems.

JSON

{"action": "audit"}

The findings are in observations.native.audit.findings, each with analyzer, type, elements, measured, expected, confidence and note.

screenshot

Saves a full-quality PNG of the screen on this computer.

Also accepted as: take_screenshot, capture_screenshot

JSON

{"action": "screenshot", "file_path": "~/mobai-shots", "name": "login_screen"}

.mob

screenshot "~/mobai-shots"

The result has screenshot_path, the full path of the file.

record_start

Starts recording the screen as a series of frames, for checking animations and transitions that a single screenshot misses. Only one recording runs at a time. A recording still running when the script ends is stopped and saved, and appears as a record_stop entry in the results.

JSON

{"action": "record_start"}

.mob

record_start

record_stop

Stops the recording and saves the frames. Frames that did not change are left out.

JSON

{"action": "record_stop", "file_path": "~/mobai-recordings/login"}

.mob

record_stop

The result has recording_path, frame_count, contact_sheets (images that tile the frames in order, each tile labeled with its frame number) and transition_hints: frame pairs with a suspicious jump, each with from_frame, to_frame, type (jump, flash, stutter, incoherent_motion or structural), delta_percent, region and message. When transition_hints is empty, nothing stood out.

wait_for

Waits until an element matching the predicate is on screen. Fails with TIMEOUT when it does not show in time.

With "stable": true, or with no predicate, it waits until the screen stops changing, for example after an animation. A screen that never settles does not fail the step unless the step has its own on_fail.

Also accepted as: extended_wait_until, wait_for_element, wait_until_visible

JSON

{"action": "wait_for", "predicate": {"text": "Welcome"}, "timeout_ms": 5000}
{"action": "wait_for", "stable": true, "timeout_ms": 3000}

.mob

wait_for "Welcome" timeout:5000
wait_stable timeout:3000

In the web context it waits for css_selector on the page.

delay

Waits a fixed time. Prefer wait_for when you know what you are waiting for.

Also accepted as: sleep, pause

JSON

{"action": "delay", "duration_ms": 1000}

.mob

delay 1000

checkpoint

Marks a point in the run. Checkpoints come back in the response's checkpoints list.

JSON

{"action": "checkpoint", "name": "logged_in", "metadata": {"user": "qa"}}

.mob

checkpoint "logged_in"

Device controls

set_location

Sets a simulated GPS location.

Platforms: iOS. Android emulators, and Android phones on Android 12 or later.

JSON

{"action": "set_location", "lat": 40.7128, "lon": -74.006}

.mob

set_location 40.7128,-74.006

A location of exactly 0, 0 is rejected.

reset_location

Stops the simulated location so the device uses its real one again.

Platforms: Same as set_location.

JSON

{"action": "reset_location"}

.mob

reset_location

set_permissions

Grants, revokes or resets an app's permissions without a prompt, so a test starts from a known state. Run it before open_app: changing a permission the app uses ends the app.

Names: all, bluetooth, calendar, camera, contacts, health, homekit, location, medialibrary, microphone, motion, notifications, phone, photos, reminders, siri, sms, speech, storage, userTracking. all is applied first, so {"all": "deny", "camera": "allow"} leaves only the camera granted.

Platforms: iOS simulators and Android. On a simulator only all, calendar, contacts, location, medialibrary, microphone, motion, photos, reminders, siri can be set, and other names are skipped. On Android, a permission the app does not ask for is skipped, as are homekit, reminders, siri, speech, userTracking.

JSON

{"action": "set_permissions", "bundle_id": "com.example.app", "permissions": {"location": "allow", "photos": "deny"}}

.mob

set_permissions com.example.app location=allow photos=deny

biometrics

Simulates the biometric sensor. Enroll a face or finger before open_app, then send match or nomatch while the app shows its prompt.

Platforms: iOS simulators: all four states for faceid and touchid. Android: match and nomatch with fingerprint, with a finger already enrolled in Settings. Not available on a physical iPhone.

Also accepted as: biometric, face_id, touch_id, fingerprint

JSON

{"action": "biometrics", "state": "enroll", "sensor": "faceid"}
{"action": "biometrics", "state": "match"}

.mob

biometrics enroll faceid
biometrics match

fold

Folds or unfolds a foldable iOS simulator. The step returns once the device has settled, and later steps measure against the screen now in use. Observe again after folding: positions from before the fold no longer apply.

Platforms: Foldable iOS simulators only.

Also accepted as: set_fold, fold_device

JSON

{"action": "fold", "state": "closed"}
{"action": "fold", "angle": 90}

.mob

fold closed
fold 90

siri

Sends a command to Siri as text. MobAI taps a consent button if Siri asks (Yes, Turn On, Allow, OK, Continue, Enable or Confirm), reads Siri's reply, then closes Siri. The reply comes back as js_value and as the variable siri_response. If Siri asks a follow-up question, send a more specific command.

Platforms: iOS only.

JSON

{"action": "siri", "prompt": "Set a timer for 5 minutes"}

.mob

siri "Set a timer for 5 minutes"

Web actions

Web steps work inside a web page: Safari tabs and app web views on an iPhone, Chrome tabs on Android. Use native actions for the browser's own controls (address bar, tabs) and web actions for the page's content.

select_web_context

Picks the web page that later web steps work on. Run it first: web steps fail until a page is selected. With no filter it takes the page in front, else the first one. The result lists every page under pages (id, title, url, app_id, is_active).

JSON

{"action": "select_web_context", "url_contains": "example.com"}

A page that has just started loading may not be listed yet. Give the step an on_fail retry, as in the web example below.

execute_js

Runs a JavaScript expression in the selected page and returns its value as js_value. An exception comes back as was_thrown and js_error while the step still passes. Write an expression, not a function body: document.title, not return document.title. Return strings, numbers or booleans, or wrap objects in JSON.stringify.

JSON

{"action": "execute_js", "script": "document.title", "store_as": "title"}
{"action": "execute_js", "script": "location.href = 'https://example.com/pricing'"}

Web forms of native actions

Add "context": "web" and put a CSS selector in the predicate as css_selector.

JSON

{"action": "tap", "context": "web", "predicate": {"css_selector": "button[type='submit']"}}
{"action": "type", "context": "web", "predicate": {"css_selector": "input#email"}, "text": "[email protected]", "clear_first": true}
{"action": "clear", "context": "web", "predicate": {"css_selector": "input#email"}}
{"action": "press_key", "context": "web", "key": "enter"}
{"action": "assert_exists", "context": "web", "predicate": {"css_selector": "div.loaded"}, "timeout_ms": 5000}
{"action": "observe", "context": "web", "include": ["ui_tree", "page_info"]}

These work on the page: tap, type, type_secret, clear, press_key, observe and execute_js, and with a css_selector, assert_exists, assert_not_exists, wait_for, if_exists and the while and until forms of repeat. Every other action reads the app screen, where a css_selector fails with VALIDATION_ERROR.

Assertions

An assertion that does not hold fails with ASSERTION_FAILED. Each takes an optional message that replaces the default failure message.

assert_exists

Fails unless an element matching the predicate is on screen. By default it checks once. With timeout_ms it keeps checking until the element shows or the time runs out. When the element exists but is scrolled out of view, the error says so and lists it under candidates. In the web context it checks the page for css_selector.

Also accepted as: assert, assert_visible, assert_present

JSON

{"action": "assert_exists", "predicate": {"text_contains": "Success"}, "timeout_ms": 3000}

.mob

assert_exists ~"Success" timeout:3000

assert_not_exists

Fails if an element matching the predicate is on screen. With timeout_ms it waits up to that long for the element to go away.

Also accepted as: assert_not_visible, assert_absent, assert_hidden

JSON

{"action": "assert_not_exists", "predicate": {"text": "Error"}, "timeout_ms": 2000}

.mob

assert_not_exists "Error" timeout:2000

assert_count

Fails unless exactly expected elements match. It checks once, without waiting. The result has match_count and matched_elements.

JSON

{"action": "assert_count", "predicate": {"type": "cell"}, "expected": 5}

.mob

assert_count type:cell expected:5

assert_property

Fails unless a property of the element equals expected_value. The result has property_value, the value found.

JSON

{"action": "assert_property", "predicate": {"text": "Submit", "type": "button"}, "property": "enabled", "expected_value": true}
{"action": "assert_property", "predicate": {"type": "input", "text_contains": "Email"}, "property": "value", "expected_value": "[email protected]"}

assert_screen_changed

Fails unless enough of the screen changed since the last observe that read the UI tree. The change is the share of elements on the screen now that were not there at that observe. The result has screen_change_percent. Put the observe before the action you check, and no other observe between them.

JSON

{"action": "assert_screen_changed", "threshold_percent": 20}

JSON

{
  "version": "0.2",
  "steps": [
    {"action": "observe", "include": ["ui_tree"]},
    {"action": "tap", "predicate": {"text": "Next"}},
    {"action": "wait_for", "stable": true, "timeout_ms": 3000},
    {"action": "assert_screen_changed", "threshold_percent": 20}
  ]
}

Performance metrics

Collect CPU, memory, frame rate, network and battery metrics while a script runs. The metrics steps are JSON only.

metrics_start

Starts collecting metrics in the background. Collection goes on after the script ends, until metrics_stop in this script or a later one. A device has one session at a time: starting another one stops the first. The result has metrics_started with session_id, device_id and started_at.

JSON

{"action": "metrics_start", "types": ["system_cpu", "system_memory", "fps"], "label": "app_launch"}
{"action": "metrics_start", "bundle_id": "com.example.app", "capture_logs": true, "label": "checkout"}

metrics_stop

Stops the session and returns metrics_summary. Fails when no session is running on the device.

JSON

{"action": "metrics_stop"}

Result

{
  "metrics_summary": {
    "session": {"label": "app_launch", "duration_seconds": 12.4, "sample_count": 12, "session_id": "a1b2c3", "data_file": "/path/to/samples.jsonl", "logs_available": false},
    "overall_health": "healthy",
    "health_score": 96,
    "system_cpu": {"avg": 41.2, "max": 72.5, "min": 12.0, "p95": 68.1, "status": "ok"},
    "system_memory": {"avg_bytes": 2147483648, "max_bytes": 2252341248, "total_bytes": 6442450944, "avg_percent": 33.3, "max_percent": 35.0, "growth_mb": 12.5, "trend": "stable", "status": "ok"},
    "fps": {"avg": 57.8, "min": 44.0, "max": 60.0, "p5": 48.0, "jank_percent": 0, "status": "ok"}
  }
}

Other summary keys, present when collected: network (total_rx_mb, total_tx_mb, status), battery (start_level, end_level, drain_percent, drain_per_hour, status), app (bundle_id, cpu_avg, cpu_max, memory_avg_bytes, memory_max_bytes, memory_growth_mb, status), and session.logs_file with capture_logs. When limits were crossed, anomalies lists them under cpu_spikes, fps_drops, memory_issues and battery_drain, with recommendations in plain words. overall_health is healthy, warning or critical, and health_score runs from 0 to 100.

JSON

{
  "version": "0.2",
  "steps": [
    {"action": "metrics_start", "types": ["system_cpu", "system_memory", "fps"], "label": "app_launch"},
    {"action": "open_app", "bundle_id": "com.example.app", "fresh": true},
    {"action": "wait_for", "predicate": {"text": "Home"}, "timeout_ms": 10000},
    {"action": "delay", "duration_ms": 2000},
    {"action": "metrics_stop"}
  ]
}

Predicates

A predicate describes the element a step acts on. MobAI looks for it in the current UI tree, the same tree observe returns.

Native predicate keys

All keys in a predicate must match. Prefer text_contains or text_regex over text, since exact text breaks with state, locale and counts. Add type when a label and a control share the same text.

Types

A raw type from the UI tree works too, such as StaticText, SearchField, PickerWheel or android.widget.EditText. A word MobAI does not recognize as a type fails with VALIDATION_ERROR, which lists the valid types.

near

JSON

{"action": "tap", "predicate": {"type": "input", "near": {"text": "Email", "direction": "below"}}}
{"action": "tap", "predicate": {"type": "button", "parent_of": {"text_contains": "Remove"}, "index": 0}}
{"action": "tap", "predicate": {"text": "Edit", "bounds_hint": "top_half"}}

When several elements match

Actions that need one element (tap, type, toggle and the like) narrow the matches down in this order:

Waits and checks (wait_for, assert_exists, if_exists, repeat) only need one match and never fail on several. drag uses the first match. Only elements on screen count, unless an action says otherwise.

Predicates in .mob

A line can use settings alone, with no text: tap type:button bounds:bottom_half. text_starts_with, label, label_contains, near.max_distance and css_selector have no .mob form.

Control flow

if_exists

Checks the screen once, without waiting, and runs then when an element matches, else else. Steps inside run in order, and the first one that fails ends the branch.

The if_exists step itself still passes when a step inside fails: read then_results or else_results, and condition_met. Steps inside ignore their own on_fail, and the whole block shares the time limit of the if_exists step.

JSON

{"action": "if_exists", "predicate": {"text": "Allow"}, "then": [{"action": "tap", "predicate": {"text": "Allow"}}], "else": [{"action": "checkpoint", "name": "no_prompt"}]}

.mob

if_exists "Allow" {
  tap "Allow"
} else {
  checkpoint "no_prompt"
}

repeat

Runs body in a loop. Set exactly one of times, while, until or condition. Inside the body, ${repeat_index} is the loop count from 0. Loops can hold if_exists blocks and other loops.

A step in the body that fails ends the loop and fails the repeat step, so the repeat step's own on_fail applies. Body steps ignore their own on_fail. The whole loop shares the repeat step's time limit, 2 minutes unless you set timeout_ms. The result has iterations and body_results.

JSON

{"action": "repeat", "times": "3", "body": [{"action": "tap", "predicate": {"text": "Add"}}]}
{"action": "repeat", "while": {"text_contains": "Loading"}, "max_iterations": 20, "body": [{"action": "delay", "duration_ms": 500}]}
{"action": "repeat", "until": {"text": "Done", "type": "button"}, "body": [{"action": "scroll", "direction": "down"}]}
{"action": "repeat", "condition": "(vars.n || 0) < 3", "body": [{"action": "eval_script", "script": "vars.n = (vars.n || 0) + 1"}]}

.mob

repeat 3 times {
  tap "Add"
}
repeat while ~"Loading" max:20 {
  delay 500
}
repeat until "Done" type:button {
  scroll down
}
repeat while js:"(vars.n || 0) < 3" {
  eval "vars.n = (vars.n || 0) + 1"
}

Variables

Write ${name} in a step to use a variable. It is filled in when the step runs, so a value saved by an earlier step is ready for later ones. A name with no value fails the step with unknown variable: ${name}. At the end of the run, every variable comes back in the response under storage.

Variables are filled in these keys: bundle_id, text, name, key, target, file_path, log_path, message, prompt, direction, distance, amount, state, store_as, extract_regex, url_contains, title_contains, times, script for execute_js, and the predicate keys text, text_contains, text_starts_with, text_regex, accessibility_id, label, label_contains, css_selector, bounds_hint and near's text and text_contains. Other keys, such as url, value, secret_id, numbers and coordinates, take the text as written.

extract

Saves a value as a variable, under extract_key. Three kinds:

JSON

{"action": "extract", "extract_key": "order_id", "extract_kind": "text", "predicate": {"text_contains": "Order #"}, "extract_regex": "#([0-9]+)"}
{"action": "extract", "extract_key": "proof", "extract_kind": "screenshot", "file_path": "~/mobai-shots"}
{"action": "extract", "extract_key": "user", "extract_kind": "literal", "text": "qa-user"}

.mob

extract order_id from ~"Order #" regex:"#([0-9]+)"
extract proof screenshot
extract user = "qa-user"

eval_script

Runs JavaScript on this computer, not on the device, to compute values, seed test data or call a backend between steps. The value of the last expression comes back as js_value, and console.log output as logs.

JSON

{"action": "eval_script", "script": "vars.user_id = 'u-' + Date.now()"}
{"action": "eval_script", "script": "1 + 1", "store_as": "two"}

.mob

eval "vars.user_id = 'u-' + Date.now()"
eval "1 + 1" store_as:two

Available in the script: vars (also output), the variables. http.get, http.post, http.put, http.delete and http.request(method, url, opts), which wait for the answer and return {status, ok, headers, body}. opts takes headers, body (an object is sent as JSON) and timeout_ms. console.log, console.info, console.warn and console.error, shown under logs. There is no file access and no require. ${...} is not filled in inside script, so JavaScript template strings work as usual.

run_script

Runs a JavaScript file on this computer, with the same globals as eval_script. In .mob, the path is relative to the .mob file and the file is read when the test is loaded.

JSON

{"action": "run_script", "script_path": "/Users/me/tests/seed.js"}
{"action": "run_script", "script": "var r = http.get('https://api.example.com/seed')\nvars.seed_id = JSON.parse(r.body).id"}

.mob

script "./seed.js"

Failure strategies

A step's on_fail decides what happens when it fails.

JSON

{"action": "tap", "predicate": {"text": "Continue"}, "on_fail": {"strategy": "retry", "max_retries": 3, "retry_delay_ms": 1000, "fallback_strategy": {"strategy": "skip"}}}
{"action": "tap", "predicate": {"text": "Not now"}, "on_fail": {"strategy": "skip"}}

.mob

tap "Continue" on_fail:retry retries:3 retry_delay:1000
tap "Not now" on_fail:skip

Put on_fail at the top of the script to set it for every step. A step's own on_fail wins. Steps inside if_exists and repeat ignore their own on_fail.

Automatic recovery

If the bridge stops during a step, MobAI waits up to 60 seconds for it to come back and runs the step again once. With "alerts": "accept" or "dismiss", a failed step is checked for a system alert in the way, such as a permission prompt or a crash dialog. MobAI taps the allow-style or deny-style button, up to three alerts in a row, and runs the step again once. Alerts inside the app are never touched. Both happen before on_fail is applied.

Error codes

The error object holds code, message and, when they apply, predicate, candidates (index, type, text, bounds), a one-line hint, valid_predicate_keys and valid_types.

Response format

dsl/execute returns one JSON object for the whole run.

Response

{
  "success": false,
  "step_results": [
    {"success": true, "step_index": 0, "action": "open_app", "started_at_ms": 1767000000000, "duration_ms": 1830, "result": {}},
    {
      "success": false,
      "step_id": "save",
      "step_index": 1,
      "action": "tap",
      "started_at_ms": 1767000001830,
      "duration_ms": 412,
      "error": {
        "code": "AMBIGUOUS_MATCH",
        "message": "2 elements matched",
        "predicate": {"text": "Save"},
        "candidates": [
          {"index": 12, "type": "Button", "text": "Save", "bounds": {"x": 16, "y": 120, "width": 80, "height": 44}},
          {"index": 31, "type": "Button", "text": "Save", "bounds": {"x": 16, "y": 700, "width": 80, "height": 44}}
        ],
        "hint": "disambiguate with one of: index (0-based), near {text, direction}, bounds_hint (top_half/bottom_half/...), or a more specific type/text. See candidates for the choices."
      },
      "debug": {"ui_tree": "[12] Button \"Save\" (16,120 80x44)\n..."}
    }
  ],
  "total_time_ms": 2310
}

Top level

Each step result

matched_element has index, type, text and bounds (x, y, width, height).

The endpoint answers 200 with this body whether the run passed or failed. A body that is not valid JSON, or a step whose keys have the wrong type, gets 400 with {"code": "EXECUTION_ERROR", "message": "invalid DSL script: ..."} and no step runs. An unknown alerts value gets 500 with the same shape. An unknown device, or a bridge that cannot start, is reported before any step runs.

Complete examples

Sign in with a parameter and a saved password

Clears permission prompts on the way, waits for each screen, and ends with the UI tree so you can see where the run landed.

JSON

{
  "version": "0.2",
  "params": {"email": "[email protected]"},
  "alerts": "accept",
  "steps": [
    {"action": "open_app", "bundle_id": "com.example.app", "fresh": true},
    {"action": "wait_for", "predicate": {"text_contains": "Sign in"}, "timeout_ms": 10000},
    {"action": "type", "predicate": {"text_contains": "Email"}, "text": "${email}", "clear_first": true},
    {"action": "type_secret", "predicate": {"text_contains": "Password"}, "secret_id": "qa-password"},
    {"action": "tap", "predicate": {"text": "Sign in", "type": "button"}},
    {"action": "assert_exists", "predicate": {"text_contains": "Welcome"}, "timeout_ms": 5000},
    {"action": "wait_for", "stable": true, "timeout_ms": 3000},
    {"action": "observe", "include": ["ui_tree"]}
  ]
}

.mob

# Param: email = [email protected]
# Alerts: accept

app "com.example.app" fresh
wait_for ~"Sign in" timeout:10000
type ~"Email" → "${email}" clear
type_secret ~"Password" → "qa-password"
tap "Sign in" type:button
assert_exists ~"Welcome" timeout:5000
wait_stable timeout:3000
observe

Get past onboarding and read a total

Taps Next while it is on screen, closes an optional prompt, scrolls to the total, saves the amount as total and keeps a screenshot.

JSON

{
  "version": "0.2",
  "steps": [
    {"action": "open_app", "bundle_id": "com.example.shop"},
    {"action": "repeat", "while": {"text": "Next", "type": "button"}, "max_iterations": 5, "body": [
      {"action": "tap", "predicate": {"text": "Next", "type": "button"}},
      {"action": "wait_for", "stable": true, "timeout_ms": 2000}
    ]},
    {"action": "if_exists", "predicate": {"text": "Not now"}, "then": [
      {"action": "tap", "predicate": {"text": "Not now"}}
    ]},
    {"action": "scroll", "direction": "down", "to_element": {"predicate": {"text_contains": "Total"}}, "max_scrolls": 5,
     "on_fail": {"strategy": "retry", "max_retries": 2}},
    {"action": "extract", "extract_key": "total", "extract_kind": "text", "predicate": {"text_contains": "Total"}, "extract_regex": "([0-9]+[.][0-9]{2})"},
    {"action": "screenshot", "file_path": "~/mobai-shots"}
  ]
}

.mob

app "com.example.shop"
repeat while "Next" type:button max:5 {
  tap "Next" type:button
  wait_stable timeout:2000
}
if_exists "Not now" {
  tap "Not now"
}
scroll down to ~"Total" max_scrolls:5 on_fail:retry retries:2
extract total from ~"Total" regex:"([0-9]+[.][0-9]{2})"
screenshot "~/mobai-shots"

The response's storage holds total.

Fill a web form on a phone

Opens the page, retries until the browser lists it, waits for the field, submits, and reads the title of the next page. JSON only, since .mob has no web steps.

JSON

{
  "version": "0.2",
  "steps": [
    {"action": "open_link", "url": "https://example.com/signup"},
    {"action": "select_web_context", "url_contains": "example.com/signup",
     "on_fail": {"strategy": "retry", "max_retries": 5, "retry_delay_ms": 1000}},
    {"action": "assert_exists", "context": "web", "predicate": {"css_selector": "input[type='email']"}, "timeout_ms": 8000},
    {"action": "type", "context": "web", "predicate": {"css_selector": "input[type='email']"}, "text": "[email protected]", "clear_first": true},
    {"action": "press_key", "context": "web", "key": "enter"},
    {"action": "assert_exists", "context": "web", "predicate": {"css_selector": ".signup-done"}, "timeout_ms": 8000},
    {"action": "execute_js", "script": "document.title", "store_as": "page_title"}
  ]
}