# Agent API

> Drive the Photo Slop editor from an AI agent — tool schemas for Claude and OpenAI, plus a look-at-the-result preview loop.

**Site index:** [https://photoslop.2plot.dev/llms.txt](https://photoslop.2plot.dev/llms.txt) — every page on this site, as Markdown.  
**Network index:** [https://2plot.dev/llms.txt](https://2plot.dev/llms.txt) — The 2plot network; start here to discover sibling sites.  
**Sibling sites:** 14 more in The 2plot network — listed in the site index above.  
**Sitemap:** https://photoslop.2plot.dev/sitemap.xml  


---



## Why it works

The editor's whole surface is the [command bus](/guide/commands), and every command has a JSON Schema. An agent needs no screen-scraping and no clicks: it emits a tool call, the browser tab running the editor executes it, and the result comes back as data.

## Published schemas

| URL | What |
|-----|------|
| `/photo/agent/tools.json` | The tool definitions, versioned |
| `/photo/agent/llms.txt` | The same, as markdown an LLM can read directly |

In Python, `dash_photo_canvas.ops` is the single source for all of these:

```python
from dash_photo_canvas import ops

ops.TOOLS                  # list of {name, description, input_schema}
ops.to_anthropic_tools()   # Anthropic Messages API `tools=`
ops.to_openai_tools()      # OpenAI function-calling `tools=`
ops.from_tool_call("layer_add_shape", {"shape": "rect", "fill": "#f00"})  # -> a command dict
```

Tool names use underscores (`layer_add_shape`); the commands they map to use dots (`layer.addShape`).

### The tools, live

Generated from the same list the server publishes, so this table and `/photo/agent/tools.json` cannot disagree.


How it is generated:


```python
# File: docs/photo-agent/tool_table.py

"""The agent tool surface, rendered live from `dash_photo_canvas.ops.TOOLS`.

The Agent API guide (docs/photo-agent/photo-agent.md) embeds this module live, so the table a reader
sees is generated from the same list the server publishes at /photo/agent/tools.json — the two cannot
disagree.
"""
import dash_mantine_components as dmc

from dash_photo_canvas import ops


def tool_rows():
    """One row per tool: the tool name an agent calls, the engine command it maps to, its arguments."""
    rows = []
    for tool in ops.TOOLS:
        schema = tool.get("input_schema") or {}
        props = sorted((schema.get("properties") or {}).keys())
        required = set(schema.get("required") or [])
        args = ", ".join(f"{p}*" if p in required else p for p in props) or "—"
        rows.append(
            dmc.TableTr(
                [
                    dmc.TableTd(dmc.Code(tool["name"])),
                    dmc.TableTd(dmc.Code(tool["command"])),
                    dmc.TableTd(dmc.Text(args, size="xs", ff="monospace")),
                ]
            )
        )
    return rows


component = dmc.Stack(
    [
        dmc.Text(
            f"{len(ops.TOOLS)} tools. An asterisk marks a required argument. The full JSON Schema for each is at "
            "/photo/agent/tools.json.",
            size="sm",
            c="dimmed",
        ),
        dmc.ScrollArea(
            dmc.Table(
                [
                    dmc.TableThead(
                        dmc.TableTr([dmc.TableTh("Tool"), dmc.TableTh("Command"), dmc.TableTh("Arguments")])
                    ),
                    dmc.TableTbody(tool_rows()),
                ],
                striped=True,
                highlightOnHover=True,
                withTableBorder=True,
                fz="sm",
            ),
            type="auto",
        ),
    ],
    gap="xs",
)
```

    :defaultExpanded: false


## The shipped loop

The editor's **Agent tab** is a working agent, and `lib/photo/agent.py` is the code behind it. Set a key on the server and it is on; without one the tab says so and stays disabled:

```bash
export ANTHROPIC_API_KEY=sk-ant-…          # or ANTHROPIC_AUTH_TOKEN
python run.py                               # /editor → Agent tab
```

| Variable | Default | Meaning |
|----------|---------|---------|
| `PHOTO_AGENT_MODEL` | `haiku` | The default model — a key (`haiku` · `sonnet` · `opus`) or a model id; Sonnet and Opus apply only to the owner |
| `PHOTO_AGENT_MAX_ROUNDS` | `8` | Model turns per request; the run stops with a message when it is reached |
| `PHOTO_AGENT_EFFORT` | unset | `low` · `medium` · `high` · `xhigh` · `max` (sent as `output_config.effort`) |
| `PHOTO_AGENT_PREVIEW_PX` | `768` | Longest side of the preview the model is shown after each turn |
| `PHOTO_AGENT_FALLBACKS` | `1` | Server-side refusal fallback (`fallbacks: "default"`); `0` disables it |

### Models, accounts and the cap

The Agent tab offers **Haiku 5.5** to every account; **Sonnet 5.5** and **Opus 5.5** are **owner-only** — the menu is built from what the request's account may use and the choice is validated again in `agent.start`/`agent.resume`, so a member naming either is refused before any request. The agent is disabled for signed-out visitors, because every response is priced (`lib/photo/ledger.py`, the public per-million-token table) and written against the signed-in account, and a **hard cap** ($10 per account, `PHOTO_SPEND_CAP_USD`; the owner's `PHOTO_SPEND_CAP_OWNER_USD`) is checked before every call. At the cap the run stops with a message. The tab shows the request's cost in the finished line and the account's running total under the prompt; `/usage` is the full board. See [Usage, limits & costs](/guide/usage).

### How a request runs

The canvas lives in the browser, so the loop is a state machine split across two places rather than a `while` loop in one process. One Dash callback drives it:

1. **Start.** The prompt and a compact summary of the open document (canvas size, every layer with its real id, position, scale, effects) go to the model with the tool list from `ops.to_anthropic_tools()`.
2. **Tool calls → one `batch`.** Every `tool_use` block becomes a command via `ops.from_tool_call`; the batch is written to the component's `command` prop, so the turn is **one undo step**. If the model did not ask for `export_preview` itself, a preview render is appended to the batch.
3. **Results → tool results + a picture.** (`export_preview` takes `region: "selection"` or a `layerId` to look at just the selected layers, cropped to their bounds.) The `commandResult` for that batch comes back through the same callback. Each step's result becomes a `tool_result` block (failures are `is_error`, steps after a failure are reported as not executed), and the preview JPEG is attached as an **image block**, so the model *looks* at the canvas before deciding what to do next.
4. **Repeat** until the model answers without tool calls, the round limit is hit, Stop is pressed, or the model declines.
5. **Reply.** The tab is a conversation. When the model stops — to ask for an image, offer a choice, or report it is done — type in the same box and Send (or ⌘/Ctrl+Enter). `agent.resume` appends your reply to the **same** message history, so the model keeps everything it said, did and saw, and it gets a **fresh document summary** because you may have dropped images or moved layers in between. The round limit starts over for each reply. **New chat** forgets the conversation (the canvas keeps its edits).

```python
from lib.photo import agent

run = agent.start("make the top layer black and white", document,
                  owner="user_123", model="sonnet", privileged=False)   # → run.next_command (a batch)
# … the browser executes it; its commandResult comes back …
run = agent.feed(run, command_result)                                 # → the next batch, or run.done
print(run.transcript)   # user / assistant / tool / image entries, what the Agent tab renders

run = agent.resume(run, "now add a white title near the top", document)   # the user's reply, same conversation
```

Gradients: the model can give any shape or text fill as a CSS gradient string (`linear-gradient(180deg, #000 0%, #fff 100%)`, `radial-gradient(…)`), so "a black-to-white gradient" is one `layer_add_shape` call rather than a stack of grey bands. The canvas background itself stays a flat colour; the system prompt tells the model to use a full-size rect at index 0 instead.

`tests/test_photo_agent.py` runs this against a scripted fake model; `tests/test_photo_browser.py::test_agent_loop_end_to_end_through_the_real_callbacks` runs the real SDK against a local fake API through the real page, so the whole round trip is exercised without a key.

### Limits worth knowing

- Run state is **per process** (a bounded in-memory table), like the `memory` asset backend. One gunicorn worker is fine; several need a shared store.
- A model call is synchronous inside the callback. While it is in flight the Send button spins (Dash's `running=` drives Mantine's `loading`), the log dims and reads "Thinking…"; between rounds the status line says which round is being applied. The SDK timeout is 120 s.
- Only the agent's own batch results reach the server: a clientside filter forwards a `commandResult` whose id starts with `agent_` and drops every other one (toolbar, console), so a toolbar click never costs a round trip or flashes the button.
- The agent can only do what the toolbar can: unknown tool names are answered to the model as errors and never reach the canvas, payload values are validated by the engine, and the upload/export guards apply unchanged.

## Writing your own

Everything above composes from three public pieces — `ops.to_anthropic_tools()`, `ops.from_tool_call()` and the `command` / `commandResult` props — so a different provider or loop shape is a small file:

```python
import anthropic
from dash import Input, Output, State, callback, no_update
from dash_photo_canvas import ops

client = anthropic.Anthropic()

@callback(Output("pc-canvas", "command"), Input("ask", "n_clicks"), State("prompt", "value"),
          prevent_initial_call=True)
def run_agent(_, prompt):
    msg = client.messages.create(
        model="claude-opus-5-5", max_tokens=4096,
        tools=ops.to_anthropic_tools(),
        messages=[{"role": "user", "content": prompt}],
    )
    calls = [ops.from_tool_call(b.name, b.input) for b in msg.content if b.type == "tool_use"]
    if not calls:
        return no_update
    # Run everything the model asked for as ONE undo step.
    return ops.cmd("batch", commands=[{"type": c["type"], "payload": c["payload"]} for c in calls])
```

To let the model **see** what it made, send `ops.cmd("export.render", preview=True)` and read `lastExport["dataUrl"]` (a small JPEG) from the component; pass it back as an image block and loop — which is what `lib/photo/agent.py` does.

## Design notes for agents

- Prefer `batch` for related edits — one undo step, one commit.
- `layer.addImage` needs a URL the *browser* can load. Upload first (`POST /photo/api/assets` returns a stable `url`), or use a CORS-enabled public URL.
- `document.get` returns the full state; use it to find layer ids rather than guessing.
- `layer.update` with `live: true` is for interactive dragging; agents should not need it.

## Security

The editor trusts commands, so **expose the command surface only to code you control.** If an end user can write arbitrary JSON into the `command` prop (for example through a chat box that forwards raw model output), constrain it to an allow-list of command types (`ops.validate` rejects types that are not published agent commands) and validate payload values yourself. The upload and export routes have their own guards — see [Storage & Export](/guide/storage-export#safety).

The shipped loop follows that rule: the key lives only in the server's environment, the status endpoint never echoes it, and the model's calls are mapped through the published tool list before anything reaches the canvas.


---

*Source: /guide/agent-api*