Openreality

The Agentic OS for any hardware Open Reality gives your AI agent the ability to use any hardware. Vision + Action.

Website · Get started · Things you can ask · Self-host · MCP docs · Contributing

---

Demo v1: a real Claude Code (Opus) session against the built-in offline simulator, beside a synchronized scene panel. Video: docs/demo/demo-video-v1.mp4. Rendered from docs/demo/ with VHS.

Live reconstruction from a handheld video: the camera path and the 3D scene build up together. Scan demo and the initial technical idea: VGGT-SLAM by Dominic Maggio, Hyungtae Lim and Luca Carlone (MIT SPARK Lab). Open Reality's reconstruction core builds on their work.

Utilities

🎥  Video in, 3D scene out	Upload a phone video. A few minutes later you have a persistent 3D scene.
📏  Measurement	Distances and angles between any points. Numbers are only called metres after you calibrate with real distance; otherwise relative.
🧭  Path planning	Plan a route through the scanned free space to an object or a point.
🤖  Robot-training exports	Turns a scan into LeRobot / GR00T style datasets or an Isaac Sim scene (hosted service).
🕵️  Scene agents	Server-side agents that survey, label, and answer questions about a scene.
🛠️  41 tools for your AI	Everything is exposed through MCP.
🧪  Offline simulator	A mock backend fakes the entire workflow with fixture data, so you can develop and demo with no account and no GPU.
🏠  Self-hostable	The full server runs on your own GPU box or your own Modal account, no account with us needed.

Quickstart

Add the tools to your AI assistant:

claude mcp add openreality -- npx -y openreality-mcp serve

codex mcp add openreality -- npx -y openreality-mcp serve

Claude desktop / Cursor (click to expand)

Add this to claude_desktop_config.json (Settings → Developer → Edit Config) or ~/.cursor/mcp.json:

{
  "mcpServers": {
    "openreality": {
      "command": "npx",
      "args": ["-y", "openreality-mcp", "serve"]
    }
  }
}

Claude Code plugin / Claude desktop one-click extension (click to expand)

Claude Code plugin (the MCP server plus the workflow skill in one step):

/plugin marketplace add reality-opened/openreality
/plugin install openreality@openreality

Claude desktop extension: download openreality-mcp-<version>.mcpb from the latest release and double-click it, or drag it into the Claude desktop window. No terminal needed.

Then sign in once (opens your browser, stores a revocable API key on your machine):

npx -y openreality-mcp login

Scan a room with your phone at open-reality.io, or just ask your assistant to upload a video file. Full per-client setup: open-reality.io/mcp.

Examples

Once connected, talk to your assistant like this:

"Upload ~/Videos/kitchen.mp4 and reconstruct it."

"What objects are in my latest scan, and how big is the room?"

"The counter edge to the window is 2.4 m. Calibrate the scene, then measure the couch."

"Plan a path from the door to the desk and describe it."

"Export this scan as robot-training data and save the zip locally."

Self Hosting