The open-source ESP32 iPhone Agent project turns an ESP32 (priced at just over 40 yuan) into an iPhone version of Doubao mobile assistant.

Beating AI Express News: Developer Henry has open-sourced the ESP32 iPhone Agent. Using just a XIAO ESP32S3 development board priced at around 40–50 yuan, paired with a mobile app, this tool enables AI to operate a physical iPhone—opening apps, tapping buttons, and swiping the screen. It requires no jailbreaking and does not rely on Apple’s XCTest testing permissions; its implementation is clever. The ESP32 creates its own Wi-Fi hotspot to act as an AirPlay screen-casting receiver. The iPhone streams its screen to the development board, which does not handle decoding. Instead, it sends compressed video back to the Agent app on the same phone via Bluetooth. The app calls a vision model to identify the interface and determine the next action, then sends commands via Bluetooth to the ESP32, which simulates USB mouse and keyboard operations on the iPhone. To avoid tap position offsets, the system automatically calibrates the mouse pointer and adjusts its position based on the screen when needed. This solution combines screen reception and input simulation using a standard ESP32-S3, eliminating the need for dedicated video capture hardware like the NanoKVM Go. The Agent runs directly on the iPhone, while model inference relies on external APIs or self-hosted servers. Since the phone’s Wi-Fi is occupied by the development board’s hotspot, model access defaults to cellular data. Initial firmware flashing and compilation/installation of the iPhone app still require a Mac and Xcode. Limited by Bluetooth bandwidth, the streamed screen has a resolution of only 480×1040, with a maximum frame rate of 5 FPS. Each operation typically takes several seconds. Currently, touch coordinates only work with vertical interfaces, keyboard input is limited to ASCII characters, and screen casting cannot continue after the iPhone is locked.