Image: GitHub
Why it mattersMost computer use agents cover one surface only. One agent across the desktop, browser, terminal and phone lets a team script jobs that cross all four.
A real workflow crosses screens. The agent opens a PDF on your desktop. It copies a number into a web form. Then it checks a notification on your phone. Most agent tools cover one screen. AutoCua is an MIT-licensed Python agent that drives all four. The repository has 642 stars. It was created on 8 October 2026, 18 days ago.
What one call drives
The agent picks a surface from one string. computer use drives the desktop with mouse and keyboard. shell use runs a coding agent in a real terminal. web use drives your own Chrome through a Chrome extension. mobile use, ios drives a paired iPhone, iPad or an iOS Simulator. The host must be macOS for the phone path. The README says Android mobile use is not available yet. Computer use works on macOS, Windows 10 or 11, and Linux with a GNOME desktop. Linux computer use was tested on Wayland.
How it sees and acts
The model never sees pixel coordinates. The maintainers write that AutoCua fuses the accessibility tree with OCR. The model picks from a numbered view. A number that was never shown is rejected before anything moves. Input goes in at the operating system's own input layer. The README names kCGHIDEventTap on macOS and pywinauto on Windows. An Interception kernel driver takes over when Windows UAC blocks user-mode input. The driver is not bundled. Its licence is dual LGPL-3.0 non-commercial or paid commercial.
The browser agent
A normal web run drives your own Chrome through the AutoCua extension. Chrome shows its own "AutoCua started debugging this browser" bar while the agent works. The Cancel button on the bar hands the browser back to you. A debugging port is opened only for headless runs. Those use a browser the agent brings itself. The page scanner never calls Runtime.enable. The numbered boxes are painted into the screenshot, not into your DOM.
Models and limits
The README says AutoCua works with 48 models across 9 providers. Model names come from a model_list.txt file and are case sensitive. A name that is not on the list is forwarded to the provider. It comes back as a 404. The README says AutoCua sends nothing anywhere except to the chosen model provider. One exception: Telegram, if you connect it yourself. A separate compression agent splices a handoff summary into the transcript mid-run. A long task finishes instead of hitting the context limit.
A team moving a daily job off manual clicks has one hard decision. Can it trust an agent with keyboard and mouse? AutoCua states what it does. It reads the accessibility tree. It posts input at the operating system's input layer. It shows the user a visible bar in Chrome. The maintainers also say what has not been built yet. Android mobile use is missing. X11 Linux is not tested. pip install cannot install the Windows kernel driver. Those are honest limits, and they belong in the decision.
Source
- AutoCua: ComputerUseAgent/AutoCua on GitHub
This item was written by an AI system from the linked source. Reveneau is responsible for what it publishes.
Get AI News in your inbox
New developer tools, model and agent releases, and how teams are actually using them to release software. Short, and only when there is something worth reading.
