Computer use (opt-in)

Cluster Build can observe your desktop and drive it — clicking, scrolling, typing, and pressing keys — through a single native computer tool, available in interactive sessions on Linux GNOME Wayland and on macOS 26 or newer. It's off by default and requires two separate consent prompts before it can see or touch anything.

Computer use is not browser control — the built-in Playwright provider still owns web pages. The computer tool is for native applications that can't be reached through the browser.

Enabling it

cluster config set computer_enabled true

or set it directly in ~/.cluster/config.toml:

[computer]
enabled = true

The interactive TUI only advertises computer when this is enabled and a supported native backend is available. Headless surfaces — cluster exec, cron runs, local child agents, and app-server sessions — never advertise it, regardless of this setting.

Linux setup

The Linux backend targets an already logged-in GNOME Wayland session. It needs gnome-screenshot for capture and a readable, writable /dev/uinput for pointer input, which a one-time machine setup installs:

version="$(cluster --version | awk '{print $2}')"
base="https://assets.cluster.app/serve/cstatic-assets/releases/cli/${version}"
curl -fLO "${base}/setup-linux-computer.sh"
curl -fLO "${base}/70-cluster-uinput.rules"
chmod +x setup-linux-computer.sh
./setup-linux-computer.sh

Keep both files together — the script installs the udev uaccess rule next to it. It currently supports Debian/Ubuntu. Capture is prompt-free: no XDG portal picker is opened, and nothing is shared with the compositor beyond the session Cluster is already running in. Without the helper or the input-device grant, the backend stays unavailable and the tool is never advertised.

macOS setup

macOS attributes Screen Recording and Accessibility to an application launched through Launch Services, not reliably to a terminal child process. The release therefore ships the cluster executable inside a background-only Cluster.app, with ~/.local/bin/cluster symlinked to it — one binary, not a second CLI. The foreground CLI launches that same binary through Launch Services as a short-lived local broker and talks to it over a private, owner-scoped Unix socket under ~/.cluster/computer/. Running a cluster binary from outside the app bundle leaves computer use unavailable.

Two macOS privacy grants are required, both in System Settings → Privacy & Security:

• Screen & System Audio Recording — for observe and every post-action screenshot.
• Accessibility — for click, scroll, type, and key.

They're reported as separate errors, so a missing grant names the one to fix. Grant it and restart cluster. The desktop session must be unlocked — Cluster holds the display awake for the run but never bypasses the login window.

What's implemented

• observe captures the desktop and returns a screenshot plus a frame_id. Linux captures the full desktop; macOS captures the primary display, and secondary displays aren't supported yet.
• click performs a left click at image coordinates grounded against a specific frame_id, then returns a fresh screenshot. An optional click_count of 1 (default) or 2 requests a single or double click; other values are rejected.
• scroll scrolls at a grounded point, up to 10 wheel steps per axis.
• type types literal printable ASCII text (tab and newline included, up to 1000 characters) into the application focused in that frame.
• key presses one bounded key — a letter, digit, f1–f12, or a named navigation key such as enter, escape, or arrow_up — optionally with ctrl, alt, shift, or meta held.
• wait delays up to 10 seconds (duration_ms in 1..=10000) before capturing a fresh screenshot, with no input performed. It's for applications whose visual state changes asynchronously — not a background task or an unbounded polling primitive. Cancelling a wait mid-delay performs no capture; the next action must observe again.

Drag and free pointer movement are not exposed. Windows has no backend.

Every action is validated against the frame it was grounded on: a stale frame_id, out-of-bounds coordinates, or display geometry that changed since the observation is rejected rather than acted on.

Consent

Two prompts gate this feature independently, each scoped to the current process — separate from the OS-level grants above, which are machine-level and persist:

• Capture — approves sending a screenshot of the desktop to the model. Required before the first observe.
• Control — approves sending pointer and keyboard input to the desktop. Required before the first click, scroll, type, or key, separately from capture.

wait only requires capture consent, like observe.

Approving one does not approve the other, and neither persists past the process — restarting cluster asks again. Set computer_unattended to treat your machine-level setup plus that explicit opt-in as standing authorization and skip both prompts; leave it off for interactive use. Screenshots and action results follow the same private, local session storage as other tool results; nothing is uploaded except as model request content for the active turn.

Plan mode

observe is read-only and allowed in plan mode. wait is likewise allowed in plan mode. click, scroll, type, and key are mutating actions and are denied in plan mode like any other mutating tool.

Next steps

• Advanced configuration — the rest of config.toml.
• Interactive sessions — plan mode and workspace trust, which govern when tools may run at all.