Image: GitHub
Why it mattersA team that wanted an agent to run tasks on a real iPhone had to buy a Mac, join Apple's developer programme and ship signed code; this project replaces the whole chain with a $7 chip and a vision model.
Driving an iPhone from an AI model used to need a Mac, a paid Apple developer account, and signed code that Apple's build chain accepts, which is a stack most people building an agent do not want to assemble. The esp32-iphone-agent project collapses that stack to a $7 ESP32-S3 microcontroller and a running vision model, and the GitHub repository has 1,418 stars in 30 days. The agent sees through AirPlay mirroring from the phone, and acts through the ESP32 presenting itself as a USB keyboard and mouse to the iPhone's USB-C port.
The project is MIT-licensed and runs the control loop on any computer, including a Mac or a Linux machine. The vision model is any OpenAI-compatible endpoint, so a developer can plug in GPT-6 Astra, Opus 5.5, a Gemini vision model, or a local model served through llama.cpp, and compare how each one does on the same screen tasks.
What the setup actually does
AirPlay mirroring gives the model an image of the current iPhone screen every few hundred milliseconds. The model decides on an action, which is one of tap at a coordinate, swipe between two coordinates, or type a short string, and the control server sends that action over serial to the ESP32. The ESP32 looks to the iPhone like a plain USB mouse and keyboard, so the iPhone accepts the input without any software installed on it. The phone stays unmodified for every step of this.
The three constraints the README names are important. Some screens cannot be mirrored over AirPlay, including the lock screen passcode, Apple Pay, and content protected by DRM, so the agent cannot act on them. The phone has to stay unlocked for the model to see the screen. And text input is limited to ASCII characters, because the keyboard the ESP32 presents is a standard US HID keyboard without a composition layer, so a model that needs to type an emoji or a non-Latin character has to drive the iPhone's input method through tap sequences instead.
Portrait apps work; landscape apps are not supported yet. The repository is at an early stage of a short list of demos, including browsing a shopping site, filling a form, and ordering food, which the project uses as its own regression set.
Who this is actually for
The project opens two kinds of work a developer could not do cheaply before. The first is automated testing on a real iPhone without Apple's developer flow. A QA team that was running tests on an emulator now has a path to drive a real device, which catches a class of bug an emulator misses, including Face ID flows, specific sensor behaviour, and apps that detect the simulator and refuse to run. The second is small personal automation, such as a parent who wants an AI model to buy a specific product from a specific shop on a phone that is not the parent's main phone.
The traction, measured by stars, is at a point where developers have found the demo but not yet at a scale where production use is realistic. 1,418 stars in a month is the kind of signal a maintainer looks for before investing in landscape mode, non-ASCII input, or the lock screen gap. For a team planning around this project, the useful move is to try the three demos the repository ships with on a spare iPhone, and judge from the actual experience.
Source
onehumanbeing/esp32-iphone-agent, on GitHub.
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
