Agent API

Drive the Photo Slop editor from an AI agent — tool schemas for Claude and OpenAI, plus a look-at-the-result preview loop.

Why it works

The editor's whole surface is the command bus, and every command has a JSON Schema. An agent needs no screen-scraping and no clicks: it emits a tool call, the browser tab running the editor executes it, and the result comes back as data.

Published schemas

In Python, dash_photo_canvas.ops is the single source for all of these:

from dash_photo_canvas import ops

ops.TOOLS                  # list of {name, description, input_schema}
ops.to_anthropic_tools()   # Anthropic Messages API `tools=`
ops.to_openai_tools()      # OpenAI function-calling `tools=`
ops.from_tool_call("layer_add_shape", {"shape": "rect", "fill": "#f00"})  # -> a command dict

Tool names use underscores (layer_add_shape); the commands they map to use dots (layer.addShape).

The tools, live

Generated from the same list the server publishes, so this table and /photo/agent/tools.json cannot disagree.

How it is generated:

# File: docs/photo-agent/tool_table.py

"""The agent tool surface, rendered live from `dash_photo_canvas.ops.TOOLS`.

The Agent API guide (docs/photo-agent/photo-agent.md) embeds this module live, so the table a reader
sees is generated from the same list the server publishes at /photo/agent/tools.json — the two cannot
disagree.
"""
import dash_mantine_components as dmc

from dash_photo_canvas import ops


def tool_rows():
    """One row per tool: the tool name an agent calls, the engine command it maps to, its arguments."""
    rows = []
    for tool in ops.TOOLS:
        schema = tool.get("input_schema") or {}
        props = sorted((schema.get("properties") or {}).keys())
        required = set(schema.get("required") or [])
        args = ", ".join(f"{p}*" if p in required else p for p in props) or "—"
        rows.append(
            dmc.TableTr(
                [
                    dmc.TableTd(dmc.Code(tool["name"])),
                    dmc.TableTd(dmc.Code(tool["command"])),
                    dmc.TableTd(dmc.Text(args, size="xs", ff="monospace")),
                ]
            )
        )
    return rows


component = dmc.Stack(
    [
        dmc.Text(
            f"{len(ops.TOOLS)} tools. An asterisk marks a required argument. The full JSON Schema for each is at "
            "/photo/agent/tools.json.",
            size="sm",
            c="dimmed",
        ),
        dmc.ScrollArea(
            dmc.Table(
                [
                    dmc.TableThead(
                        dmc.TableTr([dmc.TableTh("Tool"), dmc.TableTh("Command"), dmc.TableTh("Arguments")])
                    ),
                    dmc.TableTbody(tool_rows()),
                ],
                striped=True,
                highlightOnHover=True,
                withTableBorder=True,
                fz="sm",
            ),
            type="auto",
        ),
    ],
    gap="xs",
)

:defaultExpanded: false

The shipped loop

The editor's Agent tab is a working agent, and lib/photo/agent.py is the code behind it. Set a key on the server and it is on; without one the tab says so and stays disabled:

export ANTHROPIC_API_KEY=sk-ant-…          # or ANTHROPIC_AUTH_TOKEN
python run.py                               # /editor → Agent tab

Models, accounts and the cap

The Agent tab offers Haiku 5.5 to every account; Sonnet 5.5 and Opus 5.5 are owner-only — the menu is built from what the request's account may use and the choice is validated again in agent.start/agent.resume, so a member naming either is refused before any request. The agent is disabled for signed-out visitors, because every response is priced (lib/photo/ledger.py, the public per-million-token table) and written against the signed-in account, and a hard cap ($10 per account, PHOTO_SPEND_CAP_USD; the owner's PHOTO_SPEND_CAP_OWNER_USD) is checked before every call. At the cap the run stops with a message. The tab shows the request's cost in the finished line and the account's running total under the prompt; /usage is the full board. See Usage, limits & costs.

How a request runs

The canvas lives in the browser, so the loop is a state machine split across two places rather than a while loop in one process. One Dash callback drives it:

1. Start. The prompt and a compact summary of the open document (canvas size, every layer with its real id, position, scale, effects) go to the model with the tool list from ops.to_anthropic_tools().
2. Tool calls → one batch. Every tool_use block becomes a command via ops.from_tool_call; the batch is written to the component's command prop, so the turn is one undo step. If the model did not ask for export_preview itself, a preview render is appended to the batch.
3. Results → tool results + a picture. (export_preview takes region: "selection" or a layerId to look at just the selected layers, cropped to their bounds.) The commandResult for that batch comes back through the same callback. Each step's result becomes a tool_result block (failures are is_error, steps after a failure are reported as not executed), and the preview JPEG is attached as an image block, so the model looks at the canvas before deciding what to do next.
4. Repeat until the model answers without tool calls, the round limit is hit, Stop is pressed, or the model declines.
5. Reply. The tab is a conversation. When the model stops — to ask for an image, offer a choice, or report it is done — type in the same box and Send (or ⌘/Ctrl+Enter). agent.resume appends your reply to the same message history, so the model keeps everything it said, did and saw, and it gets a fresh document summary because you may have dropped images or moved layers in between. The round limit starts over for each reply. New chat forgets the conversation (the canvas keeps its edits).

from lib.photo import agent

run = agent.start("make the top layer black and white", document,
                  owner="user_123", model="sonnet", privileged=False)   # → run.next_command (a batch)
# … the browser executes it; its commandResult comes back …
run = agent.feed(run, command_result)                                 # → the next batch, or run.done
print(run.transcript)   # user / assistant / tool / image entries, what the Agent tab renders

run = agent.resume(run, "now add a white title near the top", document)   # the user's reply, same conversation

Gradients: the model can give any shape or text fill as a CSS gradient string (linear-gradient(180deg, #000 0%, #fff 100%), radial-gradient(…)), so "a black-to-white gradient" is one layer_add_shape call rather than a stack of grey bands. The canvas background itself stays a flat colour; the system prompt tells the model to use a full-size rect at index 0 instead.

tests/test_photo_agent.py runs this against a scripted fake model; tests/test_photo_browser.py::test_agent_loop_end_to_end_through_the_real_callbacks runs the real SDK against a local fake API through the real page, so the whole round trip is exercised without a key.

Limits worth knowing

• Run state is per process (a bounded in-memory table), like the memory asset backend. One gunicorn worker is fine; several need a shared store.
• A model call is synchronous inside the callback. While it is in flight the Send button spins (Dash's running= drives Mantine's loading), the log dims and reads "Thinking…"; between rounds the status line says which round is being applied. The SDK timeout is 120 s.
• Only the agent's own batch results reach the server: a clientside filter forwards a commandResult whose id starts with agent_ and drops every other one (toolbar, console), so a toolbar click never costs a round trip or flashes the button.
• The agent can only do what the toolbar can: unknown tool names are answered to the model as errors and never reach the canvas, payload values are validated by the engine, and the upload/export guards apply unchanged.

Writing your own

Everything above composes from three public pieces — ops.to_anthropic_tools(), ops.from_tool_call() and the command / commandResult props — so a different provider or loop shape is a small file:

import anthropic
from dash import Input, Output, State, callback, no_update
from dash_photo_canvas import ops

client = anthropic.Anthropic()

@callback(Output("pc-canvas", "command"), Input("ask", "n_clicks"), State("prompt", "value"),
          prevent_initial_call=True)
def run_agent(_, prompt):
    msg = client.messages.create(
        model="claude-opus-5-5", max_tokens=4096,
        tools=ops.to_anthropic_tools(),
        messages=[{"role": "user", "content": prompt}],
    )
    calls = [ops.from_tool_call(b.name, b.input) for b in msg.content if b.type == "tool_use"]
    if not calls:
        return no_update
    # Run everything the model asked for as ONE undo step.
    return ops.cmd("batch", commands=[{"type": c["type"], "payload": c["payload"]} for c in calls])

To let the model see what it made, send ops.cmd("export.render", preview=True) and read lastExport["dataUrl"] (a small JPEG) from the component; pass it back as an image block and loop — which is what lib/photo/agent.py does.

Design notes for agents

• Prefer batch for related edits — one undo step, one commit.
• layer.addImage needs a URL the browser can load. Upload first (POST /photo/api/assets returns a stable url), or use a CORS-enabled public URL.
• document.get returns the full state; use it to find layer ids rather than guessing.
• layer.update with live: true is for interactive dragging; agents should not need it.

Security

The editor trusts commands, so expose the command surface only to code you control. If an end user can write arbitrary JSON into the command prop (for example through a chat box that forwards raw model output), constrain it to an allow-list of command types (ops.validate rejects types that are not published agent commands) and validate payload values yourself. The upload and export routes have their own guards — see Storage & Export.

The shipped loop follows that rule: the key lives only in the server's environment, the status endpoint never echoes it, and the model's calls are mapped through the published tool list before anything reaches the canvas.

Source: /guide/agent-api