Cua Driver adds "eyes" to Agent, enabling button and text recognition even on pure pixel-based interfaces.

Beating AI Express (from Dongcha) reports that open-source Computer Use tool Cua has added an optional Perception visual extension to its Cua Driver. The Driver originally prioritized locating buttons via web structures and system accessibility trees, but these structures may be empty when dealing with Canvas, games, remote desktops, or custom-drawn applications. Perception directly reads screenshots, splitting the screen into regions with text, type, and position for the Agent to operate on. The first version uses OmniParser to identify icons and controls, and PP-OCR to read text. The entire parsing process runs on the local CPU, no GPU required, and screenshots are not transmitted over the network. Official tests show processing a single screenshot takes roughly 2–2.5 seconds on macOS, 3.5–4 seconds on Linux, and 8–9 seconds on Windows. It acts as a fallback when page structures and accessibility trees fail, and does not run visual parsing by default for every step. Cua also restricts the reuse of old screenshots for visual operations: each visual recognition is bound to the current screenshot’s `capture_id`, and subsequent clicks must originate from the same screenshot. If the screenshot is over 60 seconds old, or an operation has already been performed, the Driver will refuse to continue using it, preventing accidental clicks with outdated coordinates after the interface changes. Cua was already capable of visual positioning via `cua-som` and OmniParser in the past; now this functionality is directly integrated into Cua Driver, allowing upper-layer Agents to obtain recognition results via a unified interface. Even if the model itself cannot process images, it can decide on the next step based on these text and region data. Perception is not a default component: the OmniParser detector uses the AGPL-3.0 license, while Cua Driver remains under the MIT license.