GPUI Agentation

[![Contributors][contributors-shield]][contributors-url] [![Forks][forks-shield]][forks-url] [![Stargazers][stars-shield]][stars-url] [![Issues][issues-shield]][issues-url] [![MIT License][license-shield]][license-url]

gpui-agentation

🎯 Point at a running GPUI app, leave a note, and your coding agent gets the element, its bounds, and its Rust source location.

Explore the docs »

View Demo · Report Bug · Request Feature

Version: 0.5.1. Pre-1.0: the API can change between minor releases, so pin the version. Debug builds only — release builds expose no annotation API. Pinned to gpui-pre =0.3.8.

Table of Contents

About The Project

Built With

Getting Started

Prerequisites Installation

Usage

Inspect and annotate Capture feedback Connect your agent Agent skills

Try the Demo Roadmap Contributing License Contact Acknowledgments

About The Project

[![gpui-agentation dashboard demo][product-screenshot]](https://emmanuel-defreitas.github.io/gpui-agentation/)

Agentation annotates web pages through the DOM. A native GPUI window has no DOM, so gpui-agentation registers screens (and optional finer targets) with an ID, label, bounds, and Rust call site, then attaches your note to that.

Use it when you would otherwise describe a UI bug to your agent in words or paste a screenshot. Notes persist to a local JSON file, carry a status (pending, acknowledged, resolved, dismissed) and a reply thread, and an agent reads them over a loopback MCP server or REST/SSE.

It is a development tool. It annotates registered targets, not arbitrary pixels (by default, the whole screen), and release builds do not include it. For production telemetry or session replay, use something else.

(back to top)

Built With

[![Rust][Rust-shield]][Rust-url] [![GPUI][GPUI-shield]][GPUI-url] [![MCP][MCP-shield]][MCP-url]

(back to top)

Getting Started

Add gpui-agentation to your own GPUI app in three steps: install the crate, initialize once at launch, then mount and wrap your screen.

Prerequisites

• macOS with Xcode command-line tools
• Rust via rustup (the repository pins the toolchain)
• Python 3 (only needed to run the bundled demo)

Ely is bundled, so there is nothing else to install.

Installation

1. Add the crate with the native feature set:

    cargo add gpui-agentation --no-default-features --features native

2. The crate does not exist in release builds, so every call sits behind #[cfg(debug_assertions)] and release returns your content unchanged:

    // Launch: serve Agentation's assets, then initialize.
    let app = gpui_platform::application();
    #[cfg(debug_assertions)]
    let app = app.with_assets(gpui_agentation::Assets);
    app.run(|cx| {
        #[cfg(debug_assertions)]
        gpui_agentation::init(cx).expect("initialize Agentation");
        // ...open your window
    });
   
    // Window setup: app name, route, notes file, local port. Keep `feedback`
    // in a field of your root view that carries the same cfg.
    #[cfg(debug_assertions)]
    let feedback = gpui_agentation::Agentation::mount(
        "My app", "gpui://my-app/home", "target/agentation/feedback.json", 4748, window, cx,
    )?;
   
    // Render: wrap the screen in debug builds; release keeps `content` as is.
    #[cfg(debug_assertions)]
    let content = self.feedback.root("home", "Home screen", content, window, cx);
    content

3. Add target(id, label, child) only where you want finer inspection. Ordinary controls need no wrappers. Agentation handles its Ely setup, assets, and focus internally.

The consumer example keeps all of this in one devtools module. More: setup guide, setup skill for agents, and customization.

(back to top)

Usage

Inspect and annotate

Choose Inspect, Box, Arrow, or Pen, then select an element or draw on the screen. Inspect surfaces the element's ID, label, bounds, and Rust call site.

Draw an arrow (or a box or freehand pen stroke) to point your agent at exactly the right spot.

Capture feedback

Write a comment and save it. Open Comments → Copy to copy your feedback as Markdown, ready to paste to your agent.

Move the toolbar out of the way whenever it overlaps the UI you are inspecting.

Connect your agent

1. Keep the debug app running.
2. Open Settings → MCP and turn the server on.
3. Copy the config into your MCP client.

The endpoint runs on loopback. Clients that require stdio can use the MCP bridge.

• MCP Registry name: mcp-name: io.github.emmanuel-defreitas/gpui-agentation

For more examples, please refer to the Reference.

Agent skills

Give your coding agent a reusable workflow for installation or exploring a running app's feedback. Both skills live under skills/:

From a clone of this repository, copy either skill directory into your host project's .agents/skills/ directory, or into ~/.agents/skills/ for personal use across projects. For example, to install the exploration skill in a host:

mkdir -p /path/to/your-app/.agents/skills
cp -R skills/gpui-agentation-explore /path/to/your-app/.agents/skills/

In Zed, start a new conversation after copying, then invoke /gpui-agentation-explore or /gpui-agentation-setup. Other agents can load the linked SKILL.md directly or use their supported skill directory. Try: “Use gpui-agentation-explore to discover my app's MCP tools and summarize pending feedback without changing notes.” Skills provide instructions; connect the running debug app's MCP server separately using the steps above.

(back to top)

Try the Demo

Clone the repository and run the bundled demo app:

git clone https://github.com/emmanuel-defreitas/gpui-agentation
cd gpui-agentation
make dev

1. Choose Inspect, Box, Arrow, or Pen.
2. Select an element or draw, then write a comment and save.
3. Open Comments → Copy to copy your feedback as Markdown.

For rebuilds on save, enable Settings → Hot reload → Server status. It rebuilds and restarts the app after a successful build; in-memory state resets, saved annotations survive. Demo details

(back to top)

Roadmap

• [x] Inspect, Box, Arrow, and Pen annotation tools
• [x] Loopback MCP server plus REST/SSE access
• [x] Hot reload for rebuild-on-save
• [ ] Add text annotation tool
• [ ] Update Comment annotation UX
• [ ] Add gpui-devtools features from joris-gallot/gpui-devtools
• [ ] Extends AI features from hexuria/gpui-agent
• [ ] Stabilize the API toward 1.0
• [ ] Port MCP tools to a standalone macOS companion app

See the open issues for a full list of proposed features (and known issues).

(back to top)

Contributing

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement". Don't forget to give the project a star! Thanks again!

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/123-amazing-feature)
3. Commit your Changes (git commit -m 'feat: add amazing feature')
4. Push to the Branch (git push origin feature/123-amazing-feature)
5. Open a Pull Request

Changes target dev, then promote through next to main. Run make ci for checks or make help for available commands.

Before promotion, run make release-prepare to calculate the next stable version from conventional commits and synchronize Cargo, the lockfile, Registry metadata, documentation, skills, and site references. Settings, CLI, and MCP versions derive from Cargo at build time. Review that diff and update CHANGELOG.md before committing through the promotion PRs. CI and release tagging reject mismatched versions. See the release policy.

(back to top)

License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

Contact

Emmanuel De Freitas — @emmanuel-defreitas

Project Link: https://github.com/emmanuel-defreitas/gpui-agentation

(back to top)

Acknowledgments

• Inspired by Agentation. Independent GPUI implementation. Original MIT attribution is preserved in ORIGIN.md.
• Built on GPUI and the Model Context Protocol.
• Best-README-Template

(back to top)

[contributors-shield]: https://img.shields.io/github/contributors/emmanuel-defreitas/gpui-agentation.svg?style=for-the-badge [contributors-url]: https://github.com/emmanuel-defreitas/gpui-agentation/graphs/contributors [forks-shield]: https://img.shields.io/github/forks/emmanuel-defreitas/gpui-agentation.svg?style=for-the-badge [forks-url]: https://github.com/emmanuel-defreitas/gpui-agentation/network/members [stars-shield]: https://img.shields.io/github/stars/emmanuel-defreitas/gpui-agentation.svg?style=for-the-badge [stars-url]: https://github.com/emmanuel-defreitas/gpui-agentation/stargazers [issues-shield]: https://img.shields.io/github/issues/emmanuel-defreitas/gpui-agentation.svg?style=for-the-badge [issues-url]: https://github.com/emmanuel-defreitas/gpui-agentation/issues [license-shield]: https://img.shields.io/github/license/emmanuel-defreitas/gpui-agentation.svg?style=for-the-badge [license-url]: https://github.com/emmanuel-defreitas/gpui-agentation/blob/main/LICENSE [product-screenshot]: https://emmanuel-defreitas.github.io/gpui-agentation/assets/motion/dashboard-demo.gif?v=35d6cdd052b5 [Rust-shield]: https://img.shields.io/badge/Rust-000000?style=for-the-badge&logo=rust&logoColor=white [Rust-url]: https://www.rust-lang.org/ [GPUI-shield]: https://img.shields.io/badge/GPUI-084CCF?style=for-the-badge&logoColor=white [GPUI-url]: https://www.gpui.rs/ [MCP-shield]: https://img.shields.io/badge/MCP-000000?style=for-the-badge&logoColor=white [MCP-url]: https://modelcontextprotocol.io/