gridhand

Formerly published as gui-tool / agent-desktop-interface.

Let an AI agent click any desktop app by naming a grid cell — not guessing pixels.

A cross-platform Rust CLI for GUI automation: screenshots, window management, mouse and keyboard control, strict JSON in / JSON out. Zero dependencies — no crates, one small binary, direct OS APIs.

Real gridhand output — orient to a labeled grid → zoom into a cell → click the crosshair → verify.

---

Built for AI desktop agents (Claude Code, Codex, Gemini CLI, and friends), but it works fine as a general-purpose GUI automation tool.

Why it's different: agents are bad at guessing pixel coordinates from a screenshot, and every other automation tool makes them try. gridhand removes pixels entirely — it overlays a labeled grid with a crosshair at each cell's center, the agent names a cell, and the click lands exactly on that crosshair. Need more precision? Zoom into the cell for a sub-grid and name a sub-cell (C7.F3). It's the whole workflow in one line: orient → zoom → click → verify.

It sees pixels, not an accessibility tree. Most desktop-automation tools for agents find elements by reading the OS accessibility tree — which only helps when that tree exists, is complete, and is correct. gridhand never touches it. It works from the actual rendered screen, so it drives what tree-based tools can't see: games, <canvas> / WebGL apps, Flutter and other custom-drawn UIs, remote desktops and VNC, video, and any app with missing, mislabeled, or wrong accessibility data. If a human can see it, gridhand can click it.

Everything is hand-rolled against raw OS APIs — its own PNG encoder, D-Bus client, JSON output, and DEFLATE — so there are no crates to audit, no build surprises, and it runs natively on GNOME/Wayland where xdotool and pyautogui give up.

End-to-end in a real browser — navigate, click a field by cell, type, and submit, no pixel coordinates anywhere.

What it's for

Concrete things it's been used for:

• Automating apps with no CLI or API — drive GUI-only software (legacy tools, proprietary apps, installers, settings panels) the same way a person would, so an agent can script a workflow that otherwise has no headless path.
• Debugging GUI applications — let an agent open the app, reproduce a bug by clicking through it, screenshot each state, and read what's actually on screen — hands and eyes for the parts a log file doesn't capture.
• Reaching what the accessibility tree can't — pixel-precise grid zoom clicks targets in apps whose accessibility data is missing, incomplete, or wrong: games, <canvas>/WebGL, Flutter and other custom-drawn UIs, remote desktops, and video.

Features