🚫 The claim: no screenshot at all 0:00
Every mainstream computer-use stack works the same way: take a screenshot, send it to a vision model, ask the model to infer coordinates, click there, repeat. It is general — it can see anything a human can see — and it is slow, because a frontier model has to look at an image and reason about geometry on every single step.
Vercel Labs' agent-desktop does not take a screenshot at all. It asks the operating system for the live structure of a window — the same accessibility layer screen readers use — and gets back JSON with a reference for every element an agent can act on.
That sentence is the whole architecture. If the interface can describe itself, there is no reason to make a model guess from a picture of it.
🏢 Who built it, and when 0:26
| Fact | Detail |
|---|---|
| Origin | Vercel Labs, the experimental arm of the company behind Next.js — ~10,000 followers and 396 public repositories on GitHub |
| Announced | 31 October 2026 by Chris Tate, with one line: "agent browser for your whole computer" |
| Licence | Apache-2.0, free |
| Relation to agent-browser | agent-browser already applied this philosophy to websites; agent-desktop extends it to every app on the machine |
The naming is the strategy. agent-browser (Apache-2.0, Rust, ~45k stars) became the token-efficient way for agents to drive web pages: structured text in, actions out. agent-desktop is that same contract, one layer down the stack.
🔄 Observe, act, verify 0:59
The loop in three moves, and the third one is the point:
| Step | What happens |
|---|---|
| Observe | snapshot --app Finder -i returns JSON: every interactive element, labelled by role, each with a ref |
| Act | click @s8f3k2p9:e8 — the agent targets a ref, not a coordinate |
| Verify | Snapshot again and check what changed. Nothing is inferred from a picture |
If you have watched a vision agent click two pixels left of a button, the value is obvious: a ref either exists or it does not. There is no near-miss, no drifting coordinate space, and no need to re-derive where the button was.
⚡ The speed gap 1:39
The video's headline numbers, as it presents them:
| Operation | Time | Why |
|---|---|---|
| Accessibility tree read | 50–200 ms (typical ~120 ms) | Local OS call; no model in the path |
| Single call via the native binary | ~21 ms | Process overhead only |
| Screenshot-vision computer use, per observation | 3–8 s | Model must look at an image and reason about coordinates every step |
Multiply by the number of observations a task needs — he uses 50 as an illustration — and the difference becomes seconds versus minutes. The second saving is financial: a snapshot is plain JSON of a few hundred tokens, not an image sent to a vision model on every glance.
📋 What a snapshot actually looks like 2:45
The contract is small enough to read in full. A trimmed snapshot from the project's own quick start:
| Field | Meaning |
|---|---|
snapshot_id | Identifies this observation — carried inside every ref from it |
ref_count / complete | How many actionable elements were found, and whether the tree was fully traversed |
tree | Nested roles and names: a window, its children, their children |
ref_id | The handle the agent acts on: @s8f3k2p9:e8 |
available_actions | What may be done to that element — Click, and so on |
No bounding boxes to estimate, no OCR, no coordinate space to calibrate, no pixel matching. Every command prints one JSON envelope and exits, so an agent can parse it with the same tooling it uses for any other CLI.
🔖 Refs, and the stale-ref design 4:40
The detail that separates this from a wrapper around accessibility APIs: the ref carries the id of the snapshot it came from. A stale ref from an old observation is rejected rather than silently clicking whatever now occupies that position.
| Situation | What the tool does |
|---|---|
A bare ref like @e8 with no snapshot | Fails with INVALID_ARGS |
| The UI changed since that snapshot | STALE_REF — take a new snapshot |
| Two elements could match | AMBIGUOUS_TARGET — disambiguate, do not guess |
| Action succeeded | Confirmed before returning, so the agent's next step starts from truth |
This kills the classic computer-use failure mode — the screen changed and the agent did not notice — by making the mistake impossible rather than detectable. It is the same principle as a database's optimistic concurrency check, applied to a user interface.
🛡 Safety by refusal 3:45
The safety posture is unusually conservative for a developer preview, and the video is right to call it refreshing:
| Behaviour | What it means |
|---|---|
| Headless by default | No pointer movement, no focus stealing. Physical input requires --headed |
| Raw input primitives refuse to run | key-down, key-up, mouse-down, mouse-up are four of the sixty commands and will not execute until a daemon can release a held key or button |
| Protected OS processes | Cannot be closed |
| Elevated apps on Windows | Reject standard callers with a permission error instead of letting an agent escalate |
That is the correct priority order for a tool that has hands on your whole machine. A predictable agent can be retried and composed; a powerful one that occasionally does something unexpected cannot be trusted with a task loop.
📦 Inside the repository 5:22
| Property | Detail |
|---|---|
| Language | Rust, with a platform adapter per OS |
| Accessibility layers | macOS AX API (13+), Windows UI Automation (10 1809+), Linux AT-SPI (Ubuntu 24.04+, X11 or Wayland) |
| Commands | 60 command names, 56 operational — observe, act on a ref, keyboard/mouse, apps and windows, notifications, clipboard, wait, sessions, system |
| Install | npm install -g agent-desktop, Homebrew on macOS/Linux, or native archives from GitHub Releases |
| Embedding | A C library (libagent_desktop_ffi) for in-process calls, with a Python ctypes example in the README |
| Interfaces beyond the CLI | An MCP server mode, and a skill installable into agent harnesses |
| Telemetry of its own | Optional traces and a cursor overlay for debugging |
The consequence is neutrality. Anything that can run a shell command can drive a desktop: Claude Code, Codex, Cursor, or your own script — which is genuinely rare in a category where most computer-use stacks assume you have married one vendor's agent.
⚖️ The honest debate: what trees cannot see 5:58
Here is the weakness the video promises at the start, and it is real and structural: accessibility trees are only as good as the labels developers wrote.
| Case | What happens | Mitigation |
|---|---|---|
| Canvas-rendered apps, games, most design tools | Invisible to the approach — no tree to read | Screenshot vision still exists as a fallback |
| Electron apps with thin or missing labels | Half a UI: elements exist but cannot be identified | Pull the web content out to agent-browser over a DevTools port |
| Apps that publish no accessibility tree at all | Not automatable by this tool, full stop | Use a different approach for that app |
The Electron bridge is the clever part, and it is documented rather than improvised: for a Chromium or Electron app you launch, agent-desktop can open a DevTools port that agent-browser connects to, so the native chrome stays on the desktop tool and the web contents go to the browser tool. The two projects are designed to meet in the middle.
🗓 Eight months to 1.0, then a sprint 6:30
| Date | Event |
|---|---|
| 19 February 2026 | Repository created — quiet start |
| Spring–summer | Iteration on the ref model and the per-platform adapters |
| 28 October | v1.0.0 released; v1.0.1 patched the same day |
| 30 October | v1.0.2 |
| 31 October | Public announcement — after the stable release, not before it |
| 3 November | v1.1.0, and ~13,200 stars / ~910 forks |
The video's read is the right one: shipping the software before the hype is a credibility signal, and in a category this failure-prone the ordering is the strongest evidence you get from outside. The star count moved fast too — the video quotes 8,778 stars and 609 forks within days of the announcement; the repository was at roughly 13,200 stars and 910 forks when checked for this article, so the trajectory continued.
✅ Claims checked
| Claim | Verdict | Evidence |
|---|---|---|
| Vercel Labs shipped Agent Desktop, free and open source | Correct | vercel-labs/agent-desktop, Apache-2.0, Rust, created 19 Feb 2026 |
| It reads the OS accessibility tree instead of screenshots | Correct | README: "reads the accessibility tree of a desktop app and returns it as JSON, with a ref for every element" |
| macOS, Windows and Linux support | Correct | AX API (macOS 13+), UI Automation (Windows 10 1809+), AT-SPI (Ubuntu 24.04+) |
| Refs carry the snapshot id; stale refs are rejected | Correct | Bare @e8 fails INVALID_ARGS; changed UI returns STALE_REF or AMBIGUOUS_TARGET |
Headless by default; pointer and focus untouched unless --headed | Correct | README: "Actions are headless by default: no pointer movement, no focus change" |
| Raw key-down/mouse-down primitives refuse to run | Correct | Four of 60 commands refuse until a daemon can release a held key or button |
| 60+ commands across all three platforms | Correct, precisely | 60 command names, 56 operational, grouped into observe / act / keyboard+mouse / apps+windows / notifications / clipboard / wait / sessions / system |
| Ships as CLI, installable skill, and a C library with Python bindings | Correct | npm/Homebrew CLI, skills command, libagent_desktop_ffi with a ctypes example; also an MCP server mode |
| Electron/Chromium handoff to agent-browser over a DevTools port | Correct | Documented: launch "App" --cdp opens a DevTools endpoint for agent-browser |
| Canvas apps and unlabelled UIs are blind spots | Correct — stated by the project | Docs: "It reads accessibility trees and does not match pixels, so it cannot see a UI that publishes no accessibility tree" |
| Announced 31 October by Chris Tate; v1.0.0 on 28 October | Consistent | Releases: v1.0.0 and v1.0.1 on 28 Oct, v1.0.2 on 30 Oct, v1.1.0 on 3 Nov |
| 8,778 stars / 609 forks within days of 1.0 | Was accurate; now higher | ~13,200 stars / ~910 forks on 4 November 2026 |
| Tree read 50–200 ms (typical ~120 ms); ~21 ms per native call; screenshot vision 3–8 s | Unverified | Attributed in the video to third-party testing; no performance figures in the project's own docs or README |
| Vercel Labs ~10,000 followers, 395 repos | Correct | 10,082 followers, 396 public repositories |
| Independent reproduction on Linux, no pixel matching in the loop | Second-hand | Reported as Flavio Copes' end-to-end test in the video; not independently re-run here |
Basis note. Built from the full 7:16 transcript, then verified against the vercel-labs/agent-desktop README, the project documentation at agent-browser.dev/desktop, and the GitHub releases API on 4 November 2026. The structural claims hold; the performance numbers do not appear in the project's own material and are marked as reported. Star counts move quickly and are dated.
🔑 Key takeaways
- Read the interface, do not photograph it. The accessibility tree is a structured description the OS already maintains — and the model is far better at text than at estimating coordinates.
- No model inside the tool. agent-desktop is deterministic plumbing the agent calls; the loop stays in your harness, which is why it works with Claude Code, Codex, Cursor or a shell script.
- Refs beat coordinates. A ref either exists or it does not — no near-misses, no drifting coordinate space, no pixel matching.
- Put the snapshot id inside the ref. A stale handle is rejected instead of clicking whatever moved into that position: optimistic concurrency, applied to a GUI.
- Verify by observing again. Observe, act, observe — the third step is what turns actions into a loop that can be trusted.
- Predictable beats powerful. Headless by default, no focus stealing, raw input primitives disabled, protected processes uncloseable, elevated apps refusing standard callers.
- Speed and cost move together: a local JSON read has no model and no image, so it saves both seconds and tokens on every observation.
- The blind spot is structural: accessibility trees are as good as the labels developers wrote. Canvas is invisible; thin Electron UIs are half-visible.
- The right answer is a router, not a winner: trees for labelled surfaces, pixels as fallback, and a DevTools handoff to agent-browser for Chromium and Electron web content.
- Neutrality is the moat. Shipping as CLI, skill, C library and MCP server means no vendor lock-in — rare in computer use, where most stacks assume one agent.
- Release order is a credibility signal. Stable versions first, announcement second, patch same night: eight quiet months, then a sprint.
- Treat snapshots as sensitive. An accessibility tree read of a window can contain private content; keep snapshots, clipboard reads and traces out of logs and uploads.
🔗 Resources
- The source video — Prism Labs, "Agent Desktop: Free Open Source Computer Use, 40x Faster", 7:16, 2 November 2026
- vercel-labs/agent-desktop — Apache-2.0, Rust, the CLI and C library
- agent-desktop documentation — the loop, snapshots and refs, errors, skills, MCP, platform pages
- Snapshots and refs — the contract between agent and desktop
- Chromium and Electron apps — the DevTools handoff to agent-browser
- vercel-labs/agent-browser — the same philosophy applied to web pages (Apache-2.0, Rust)
- agent-desktop on npm — global install with native binaries for x64 and ARM64