v1.22.52see what's new →

Computer

Let an agent click, type, and take screenshots inside any Mac app — the same way assistive technology does, without needing the app to have a web interface.

When to use it

agents computer drives macOS applications through a small helper app called Computer Helper. Use it when you want to automate a desktop app that has no web interface — a native finance tool, a system preferences pane, or anything a browser cannot reach.

This is macOS-only.

How the allow-list works

By default the helper is allowed to drive zero apps. You name each one you want to automate. The agent cannot click inside Mail unless you explicitly say Mail is allowed — nothing runs silently in the background.

  1. Agent asks
    “click Send in Mail”
  2. Request goes to Computer Helper
    small allow-listed helper app
  3. Helper checks the allow-list
    Mail is allowed → proceed
  4. Helper clicks Send, returns confirmation
    agent gets a structured ack

In the wild: triage a native chat app

WhatsApp is a desktop app with no web interface a browser could reach. Asked “who's coming to spikeball this week?”, an agent drove it end to end — enumerate the app's windows as JSON, capture the chat, read the accessibility tree for stable @e element refs, click the group by ref, scroll, and summarize the replies.

# Enumerate WhatsApp's windows as JSON
agents computer screenshot --bundle net.whatsapp.WhatsApp --list --json

# Capture the window so the agent can read it
agents computer screenshot --bundle net.whatsapp.WhatsApp --window-id 188765 --out /tmp/wa.jpg

# Read the accessibility tree — every element gets a stable @e ref
agents computer describe --bundle net.whatsapp.WhatsApp --depth 20

# Open the group chat by ref, then scroll and re-read the thread
agents computer click  --bundle net.whatsapp.WhatsApp --id @e17
agents computer scroll --bundle net.whatsapp.WhatsApp --x 550 --y 450 --dy 600

Addressing elements by @e ref is element mode: it uses macOS Accessibility actions, so it never moves your cursor or steals the foreground while it works — you can keep using the machine.

Sensitive actions stop at your fingerprint

Driving Bambu Studio to resume a stalled 3D print, an agent enumerated the app's windows (screenshot --list --json), did window-bounds coordinate math, and clicked through the print dialog — but the final Send raised a macOS Touch ID prompt. The agent could not satisfy the biometric, and correctly handed it back to the human rather than trying to bypass it. That is by design: actions gated behind Touch ID or a system security prompt remain the user's to approve.

One-time setup

Four steps, done once per machine:

Step 1 — Install the helper app:

agents computer install-helper

The helper is not bundled in the npm package — it is downloaded on demand from its own signed GitHub release, then verified before install: sha256 against the published checksum, Developer ID team, and Apple notarization. So the first run of this command needs network access, and an offline machine will say so rather than install something unverified.

Step 2 — Grant permissions in System Settings (manual):

System Settings > Privacy & Security > Accessibility     — add Computer Helper.app
System Settings > Privacy & Security > Screen Recording  — add Computer Helper.app

Step 3 — Name the apps you want to drive. Create a file at ~/.agents/permissions/groups/computer.yaml:

name: computer
allow:
  - "Computer(com.apple.mail)"
  - "Computer(com.apple.finder)"
  - "Computer(com.apple.notes)"

To find an app's identifier: defaults read /Applications/SomeApp.app/Contents/Info CFBundleIdentifier

Step 4 — Start the helper and confirm it works:

agents computer start
agents computer status            # look for trust: granted
agents computer screenshot --out /tmp/snapshot.jpg

If trust: denied appears, the System Settings step did not complete. Add the app there, then run agents computer start again.

Add an app without restarting

# Append to the permissions file, then reload
echo '  - "Computer(com.apple.calendar)"' >> ~/.agents/permissions/groups/computer.yaml
agents computer reload
# Output: policy: 4 apps allowed (com.apple.mail, ...)

Stop when you are done

Accessibility and Screen Recording are sensitive permissions. Stop the helper when you are not using it.

agents computer stop

Command reference

agents computer install-helper    # download + verify helper, copy to /Applications/, write config
agents computer start             # start the helper
agents computer stop              # stop the helper
agents computer reload            # reload the allow list without restarting
agents computer status            # install state, running state, permission trust
agents computer screenshot        # capture a JPEG (--list --json to enumerate windows)
agents computer describe          # dump the accessibility tree with stable @e refs
agents computer click             # click by --id @e12 (element mode) or --x/--y
agents computer scroll            # scroll at a point by --dx/--dy
agents computer type / key        # type text, or send keys like esc / enter

Related