The Agent panel: a chat that edits the scene¶

View → Agent (F8) opens G-Compose AI Assist, a chat with an LLM agent that acts on the scene in front of you. You can ask it what a class does, have it build a detector setup, set up a voxel image, or write and run a recipe. Its changes show up in the window as it makes them.

It gets its tools from the MCP stack documented elsewhere:

AgentPanel (C++ dock)  ── JSON lines ──  python -m uLib.agentchat  (worker process)
                                                │
                     Claude / Gemini ◄──────────┤  tools: uLib.mcp.MCPServer, in-process
                                                │         static tools + uLib.pyagent's
                                                │         per-class tools
                     OpenCode ◄─────────────────┘  given `python -m uLib.mcp` as its MCP server
                                                │
                              gcompose's own remote-control port (127.0.0.1)

The agent knows what this build and its loaded plugins provide. At the start of each conversation it is told which plugin libraries are loaded and is given the live class catalog, so a plugin's classes are part of its vocabulary exactly while the plugin is loaded. During the conversation it gets ulib__<Class>__configure and ulib__<Class>__<Action> tools for every class in the scene. These are generated by uprops::McpContext from the class's own property declarations, with their ranges, enums and units (see pyagent).

Back ends¶

Back end	What runs the loop	Credentials	Needs
Claude	the worker, with the Messages API	key in Preferences, or `ANTHROPIC_API_KEY`, or the browser login	`pip install anthropic`
Gemini	the worker, with the Gemini API	key in Preferences, or `GEMINI_API_KEY`	`pip install google-genai`
OpenCode	a local `opencode serve`	OpenCode's own configuration	`opencode` on `PATH`, or a server URL in Preferences

Claude defaults to claude-opus-5-5 with adaptive thinking at high effort and the server-side refusal fallback. Gemini defaults to gemini-3.8-flash. The model box can be edited, and ↻ lists the models available to your account. OpenCode takes provider/model names, or nothing to use its own default.

Settings: Preferences → AI Assist¶

Open them from File → Preferences → AI Assist, or with the panel's ⚙ button. The tab has:

Claude: the API key; Get an API key ↗ opens the Claude Console key page; Log in with browser runs ant auth login (the ant CLI), which signs you in through the browser and leaves a profile the SDK uses by itself; Check login shows ant auth status. A key in Preferences wins over the login.
Gemini: the API key; Log in to Google AI Studio and get a key ↗ opens the AI Studio key page.
OpenCode: the server URL; empty starts opencode serve locally. OpenCode keeps its own model logins (opencode auth login).
External MCP clients: serves this session over MCP on HTTP, so Antigravity, Claude Desktop, VS Code and other clients can drive gcompose with the same tools as the panel. You set the port, whether other machines may connect, and the access token, and Copy Antigravity config copies a ready mcp_config.json entry. The status line says where it is serving. See the MCP server doc.
Agent worker: the Python that runs the worker, and the endpoint above.

An empty key falls back to the environment. The line under the panel's controls says which credential the current back end is using.

The keys are saved in the gcompose settings file (~/.config/cmt/gcompose.ini on Linux), which is made readable only by you once it holds a key. They are never part of the preferences snapshot a scene document carries, and a document that contains them anyway is ignored (Settings::IsMachineLocalKey). The worker receives a key on its stdin, not on its command line or in its environment.

Which Python runs the worker¶

In order of preference: Preferences → AI Assist → Python, then the Python interpreter from the Coding tab, then $GCOMPOSE_AGENT_PYTHON, then $CONDA_PREFIX/bin/python, then python3 on PATH. That Python must be able to import uLib. The panel adds the build tree and the source checkout to PYTHONPATH, as the embedded interpreter does, and an installed package puts uLib in site-packages. The model SDKs only need to be installed in that Python, not in gcompose.

The worker runs in a separate process for two reasons: a slow or failing API must never block the editor, and the SDKs don't belong in the embedded interpreter. If it dies, the panel shows the end of its stderr, and the next message starts a new worker.

The port¶

The tools act through gcompose's remote-control port. If gcompose was started with --port, the panel uses that one. Otherwise it opens one the first time the agent needs it, on a free port and bound to 127.0.0.1 only (PropertyServer::SetBindAddress), because nobody chose to expose it and the protocol includes eval_python. The window title shows the port either way.

What the agent is not given¶

session_start, session_attach and session_stop are removed from its tool list. Inside gcompose the session is already fixed: calling session_start would launch a second gcompose and leave the scene you are looking at. Every other uLib.mcp tool is available, including eval_python, recipe_set / recipe_run and screenshot. Claude also receives screenshots as images.

Theme¶

The panel follows the dark and bright themes: its transcript is kept as a list of entries and redrawn in the new palette when the theme changes.

For developers¶

uLib.agentchat has four modules:

toolbox.py: the MCP server run in-process. It maps tool names to names model APIs accept (ulib.ContainerBox.configure → ulib__ContainerBox__configure) and property names to argument names (Show Grid → Show_Grid), and maps both back on every call.
knowledge.py: the system prompt. It is built once per conversation so the request prefix stays cacheable.
providers.py: one class per back end, each with run(prompt, emit, cancel), reset() and models().
__main__.py: the worker and its JSON-lines protocol, documented in the module docstring.

Test: src/Python/testing/agentchat_test.py (ctest AgentChatTest). It runs a whole Claude turn against a local stand-in for the Messages API and a real gcompose, and never calls a real model API.