Disclosure: I build MobAI, which also works with mobile device automation, so there is some overlap here. Everything below can be checked on your own machine, and I included the commands I used. Most of this was inferred from the compiled app, linked frameworks, strings, and sandbox profile, so some details about the internal implementation may be wrong.
Recently Claude Code team added an iOS Simulator pane to their desktop and CLI apps.
The agent can build an app, launch it in a simulator, interact with it, and show the device screen next to the conversation. The feature got a lot of attention.
What I was more interested in was how it worked. There are several ways to build something like this, so what did the Anthropic team Mythos choose?
It doesn't use computer use
This is probably the first thing that comes to mind.
The obvious way for an agent to control a simulator is through normal desktop automation: take screenshots, move the mouse, and click on the Simulator window.
Claude already has computer-use tooling for that. But the simulator pane in the Desktop app works differently.
You can test this by requesting a simulator screenshot while the panel is detached and no Simulator window is visible. The helper still returns a live frame from the device.
The input coordinates are also device coordinates, not desktop pixels. They don't depend on where the Simulator window is placed, whether it is covered by another window, or whether it is open at all.
There is one important difference between the Desktop app and the CLI.
Claude Code CLI does use computer use to interact with the simulator. Anthropic says this directly in its documentation. The direct simulator integration only exists in the Desktop app.
So the two versions of Claude Code use different mechanisms for the same task.
The simulator code runs in a separate helper
You can inspect its bundle with:
plutil -p /Applications/Claude.app/Contents/Helpers/Claude\ iOS\ Sim.app/Contents/Info.plistThe output includes:
"CFBundleIdentifier" => "com.anthropic.claude.ios-sim"
"LSBackgroundOnly" => true
"LSMinimumSystemVersion" => "14.0"The helper is a universal Mach-O with both x86_64 and arm64 slices. It uses the hardened runtime and is signed with the same Developer ID as the main Claude app, but it lives in a separate app bundle.
LSBackgroundOnly means it has no window and no Dock icon, even though it is packaged as an app.
There is also an ElectronAsarIntegrity key in the helper's Info.plist, which looks like leftover packaging metadata. The helper itself is a native binary, not an Electron process.
It includes parts of Facebook's idb
List the helper's frameworks:
ls /Applications/Claude.app/Contents/Helpers/Claude\ iOS\ Sim.app/Contents/Frameworks/You will find:
FBControlCore.framework
FBSimulatorControl.frameworkThese frameworks come from the facebook/idb project.
idb_companion is built on top of them. The project has been around for years and is used by a lot of iOS testing and CI tooling.
Claude does not appear to ship the normal idb setup.
There is no idb_companion process, no gRPC layer, and no Python client. Instead, the frameworks are linked directly into the helper process.
So Anthropic took the underlying library and built its own interface around it.
How it communicates with the simulator
It talks directly to Apple's private CoreSimulator API.
You can see references to CoreSimulator classes in the framework:
strings -a /Applications/Claude.app/Contents/Helpers/Claude\ iOS\ Sim.app/Contents/Frameworks/FBSimulatorControl.framework/FBSimulatorControl | grep -E "SimDevice"This returns names such as:
SimDevice
SimDeviceSet
SimDeviceLegacyClientThese are Objective-C classes from CoreSimulator.
The framework loads CoreSimulator with dlopen and communicates with it through XPC. Apple's simctl tool uses the same underlying API, so this is not simply a wrapper around simctl.
They both operate at roughly the same layer.
The implementation is not entirely direct, though.
The framework also contains classes such as FBAppleSimctlCommandExecutor and FBSimulatorVideoSimCtl, along with a hardcoded path to the simctl binary inside the CoreSimulator bundle.
That means some operations, including parts of video capture, still call simctl.
Builds are handled through an xcodebuild wrapper.
One observation method, despite the engine supporting more
The integration exposes two MCP tools with thirteen actions.
control
attachdetachlaunchscreenshottapswipetouch_pathtouch2_pathtextbuttonopen_url
build
buildbuild_status
Everything the model knows about the app's state comes from a screenshot.
There is no element query, text extraction, UI hierarchy, identifier lookup, or wait-for-element operation.
That leaves the agent doing several things through screenshots alone:
Finding targets visually and tapping coordinates. This is usually fine for a large button. It gets less reliable with small controls, dense lists, similar-looking elements, and custom-drawn interfaces.
Checking results from pixels. “There is a button labelled Continue” becomes “the screenshot looks like it contains one.” When the model misreads something, there is no failed query or missing-element error to tell it that.
Scrolling without knowing what is off-screen. There is no element search or scroll-to-element operation. The agent has to scroll, take another screenshot, inspect it, and repeat.
Handling timing by polling screenshots. Without wait-for-element, the agent has to keep taking screenshots until the expected state appears or the screen seems to stop changing.
Accessibility data would not solve every case. Custom controls can expose incomplete or bad accessibility information. But for normal native interfaces, it gives the agent structured text, element roles, bounds, hierarchy, and identifiers instead of asking it to infer all of that from an image.
That matters because this feature is supposed to let an agent check its own work. A verification step that silently accepts the wrong screen or misreads a label can be worse than no verification, because the result looks confirmed when it is not.
Screenshots are not the cheaper option either.
Image input costs roughly:
(width × height) / 750tokens, with the long edge capped at 1568 pixels. That puts a large iPad frame at around 2,500 tokens.
A twenty-step loop where the agent taps and then checks a screenshot can therefore consume around 50,000 image tokens in a worst-case scenario. A structured accessibility dump of a simple screen would be much smaller and would preserve text and element relationships directly.
So screenshot-only control is less reliable and potentially more expensive, even though the bundled framework appears to contain the foundations for ax tree observation.
The imbalance
The action side is much more capable.
touch_path accepts an array of {x, y, dt_ms} values. That supports timed paths, variable-speed swipes, eased curves, and gestures such as long-press-and-drag.
touch2_path accepts {x1, y1, x2, y2, dt_ms} for each step, allowing two-finger gestures such as pinch and rotation.
This is HID-level path injection inherited from idb.
That is the imbalance in the current interface: the agent can perform a two-finger rotation using a precisely timed path, but it cannot ask for a structured description of what is currently on screen.
The control side gets much of idb's expressiveness. The observation side gets pixels.
The 4-point trap
The full MCP schema is included in the Electron bundle, so you can read the tool descriptions yourself:
strings -a /Applications/Claude.app/Contents/Resources/app.asar \
| grep -o "within 4pt of an edge[^\"]*"A swipe or touch_path starting within four points of a screen edge does not behave like a normal drag. It triggers the corresponding system edge gesture: back navigation from the left, Notification Center from the top, Home or the app switcher from the bottom, and Control Center from the right, adjusted for the current orientation.
So a scroll that begins slightly too close to the bezel can navigate away from the screen being tested. Nothing in the return value says that this happened. The agent just receives a screenshot of a different screen and has to work out where it ended up.
For a model choosing coordinates by eye from a screenshot, using device dimensions returned by launch, a four-point margin does not leave much room for error.
It can also fail in the opposite direction. The helper contains this log line:
[ios-sim] SimulatorKit mouse factory unavailable; edge gestures degrade to plain dragsEdge gestures depend on a private SimulatorKit entry point. When it is unavailable, they silently turn into normal drags.
Both failure modes return success. A drag can unexpectedly become navigation, or an intended navigation gesture can become a drag. An agent asked to swipe back from the left edge may stay on the same screen with no indication that the gesture behaved differently.
Conclusions
The overall design is fairly straightforward.
Anthropic uses an existing simulator-control library in a separate native helper instead of implementing its own HID injection layer.
Simulator automation itself is not new, and there is no new simulator runtime hidden inside Claude.
The feature is built on Meta's open-source idb frameworks and Apple's existing CoreSimulator infrastructure. It still requires Xcode, still runs Apple's simulators on the user's Mac, and those simulators still consume the same CPU, RAM, and disk space as they normally would.
That is the main reason I wanted to look into it. From the outside, the feature can seem like a completely new simulator stack, but it is really an integration built on top of existing open-source projects and Apple's simulator tooling.