SwiftAutoGUI

A Swift library for macOS automation — mouse, keyboard, screenshots, image recognition, and AI-powered agents.

This repository is inspired by pyautogui.

Demo

AI Agent that autonomously observes the screen and executes actions to achieve a goal.

sagui agent "Open Safari and search for Swift"

Requirements

• macOS 27.0+
• Swift 6.2+

Installation

Swift Package Manager

SwiftAutoGUI is available through Swift Package Manager.

in Package.swift add the following:

dependencies: [
    // Dependencies declare other packages that this package depends on.
    .package(url: "https://github.com/NakaokaRei/SwiftAutoGUI", branch: "master")
],
targets: [
    .target(
        name: "MyProject",
        dependencies: [..., "SwiftAutoGUI"]
    )
    ...
]

Homebrew (sagui CLI)

The sagui command-line tool is available via the Homebrew tap:

brew install NakaokaRei/tap/sagui

After install, grant Accessibility permission to your terminal in System Settings → Privacy & Security → Accessibility.

sagui --version
sagui key list
sagui key shortcut return
sagui mouse click --x 100 --y 200
sagui mouse click --right --x 100 --y 200

Optional Chromium CDP backend

Quick start: Browser Agent guide

SwiftAutoGUIBrowser adds semantic automation for an existing Chrome, Edge, or other Chromium debugging session. It is a separate product, so applications that use only native macOS automation do not acquire browser-specific code.

Add the optional product to the client target:

.target(
    name: "MyProject",
    dependencies: [
        .product(name: "SwiftAutoGUI", package: "SwiftAutoGUI"),
        .product(name: "SwiftAutoGUIBrowser", package: "SwiftAutoGUI")
    ]
)

Start Chromium with a dedicated profile and a loopback debugging port. Chrome 136 and later do not honor remote debugging against the default profile.

"/Applications/Google Chrome for Testing.app/Contents/MacOS/Google Chrome for Testing" \
  --remote-debugging-port=9222 \
  --user-data-dir=/tmp/swift-auto-gui-browser-profile

Connect with an explicit navigation allowlist:

import SwiftAutoGUI
import SwiftAutoGUIBrowser

let browser = try await BrowserSession.connect(
    endpoint: URL(string: "http://127.0.0.1:9222")!,
    securityPolicy: BrowserSecurityPolicy(
        allowedDomains: ["example.com", "*.example.org"]
    )
)

let observation = try await browser.observe()
print(observation.formattedContext)

let apiKey = ProcessInfo.processInfo.environment["OPENAI_API_KEY"]!
let browserAgent = Agent(
    model: AutomationModels.openAI(apiKey: apiKey),
    automationBackend: browser
)

The browser Agent is intentionally browser-only: native app, window, AX-label, and coordinate mouse actions fail as unsupported instead of falling back to Accessibility or CGEvent. Stale DOM elements also fail safely without clicking a saved coordinate. Cross-origin navigation and downloads require a BrowserActionAuthorizing implementation; without one they are denied.

The sagui CLI supports deterministic CDP commands as well as the browser-only Agent: