> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gomajordomo.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent Run Tracking

> Group the many LLM calls of one conversation or agent run into a single run — with a rolled-up cost and a nested waterfall of which step drove which calls.

Conversational agents and workflow agents make **many LLM calls per task** — a planning
call, tool calls, follow-ups. By default the gateway logs each as an independent request.
Agent run tracking lets you stamp those calls with a shared id so the dashboard shows them
as **one run**: a rolled-up total cost, plus a nested **waterfall** of which tool/agent step
drove which calls and what each cost.

Because the gateway is a proxy, it sees your LLM calls but never the tool/agent code that
runs between them. So the tree is reconstructed from headers **your client sends** — no SDK
or in-app tracer required. Any client that can set request headers works.

## The headers

| Header                  | Required      | Value                                                                                                                                                                      |
| ----------------------- | ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `X-Majordomo-Trace-Id`  | To join a run | One id per conversation / agent run. Generate it once when the run starts and send it on **every** LLM call in that run (any opaque string).                               |
| `X-Majordomo-Span-Path` | No            | `/`-joined names of the ancestor **steps** from the run root down to this call's parent, e.g. `planner/tool:search_db`. Omit to hang the call directly under the run root. |
| `X-Majordomo-Span-Name` | No            | Label for this call in the waterfall. Defaults to the model name.                                                                                                          |

**Graceful degradation.** A trace id alone gives you a **flat run rollup** — every call in
the run plus its total cost. Adding a span path gives you the **waterfall** — the calls
grouped under named tool/agent steps, with cost rolled up each step. Start with just a trace
id; add paths when you want the tree.

<Note>
  `/` is the reserved separator between step names. If a step name itself contains a `/`,
  percent-encode it so it doesn't split into extra nodes.
</Note>

## Modeling the tree

Send, on each LLM call, the **path of the steps above it**. Interior tool/agent nodes don't
make their own request — they're inferred from the paths of the calls beneath them.

For an agent that plans, then calls a `search_db` tool that makes two calls:

```
planner                          # X-Majordomo-Span-Path: planner
├─ route          $0.004         #   Span-Name: route
└─ tool:search_db                # X-Majordomo-Span-Path: planner/tool:search_db
   ├─ summarize   $0.021         #   Span-Name: summarize
   └─ rank        $0.027         #   Span-Name: rank
```

The `tool:search_db` node's cost (`$0.048`) is the sum of the calls beneath it; the `planner`
node totals the whole run.

## Sending the headers

<CodeGroup>
  ```python Python (majordomo-llm) theme={null}
  import os, uuid
  from majordomo_llm import get_llm_instance

  llm = get_llm_instance(
      "anthropic", "claude-sonnet-4-6",
      base_url=os.environ["MAJORDOMO_GATEWAY_URL"],
      default_headers={"X-Majordomo-Key": os.environ["MAJORDOMO_API_KEY"]},
  )

  # One id per conversation / agent run, reused across every call in the run.
  trace_id = str(uuid.uuid4())

  await llm.get_response(
      user_prompt="Which tool should I use?",
      extra_headers={
          "X-Majordomo-Trace-Id": trace_id,
          "X-Majordomo-Span-Path": "planner",
          "X-Majordomo-Span-Name": "route",
      },
  )

  await llm.get_response(
      user_prompt="Summarize these rows: ...",
      extra_headers={
          "X-Majordomo-Trace-Id": trace_id,
          "X-Majordomo-Span-Path": "planner/tool:search_db",
          "X-Majordomo-Span-Name": "summarize",
      },
  )
  ```

  ```python Python (OpenAI SDK) theme={null}
  from openai import OpenAI

  client = OpenAI(
      base_url="https://gateway.gomajordomo.com/v1",
      api_key="your-openai-key",
      default_headers={"X-Majordomo-Key": "mdm_sk_your_key_here"},
  )

  client.chat.completions.create(
      model="gpt-4o",
      messages=[...],
      extra_headers={
          "X-Majordomo-Trace-Id": "run_7f3a",
          "X-Majordomo-Span-Path": "planner/tool:search_db",
          "X-Majordomo-Span-Name": "summarize",
      },
  )
  ```

  ```bash curl theme={null}
  curl -X POST https://gateway.gomajordomo.com/v1/chat/completions \
    -H "Authorization: Bearer your-openai-key" \
    -H "X-Majordomo-Key: mdm_sk_your_key_here" \
    -H "X-Majordomo-Trace-Id: run_7f3a" \
    -H "X-Majordomo-Span-Path: planner/tool:search_db" \
    -H "X-Majordomo-Span-Name: summarize" \
    -H "Content-Type: application/json" \
    -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "..."}]}'
  ```
</CodeGroup>

### Frameworks that hide per-request headers

Pydantic-AI, Mastra, and similar frameworks don't expose a per-request header hook on
`agent.run()` — they own the model-call loop internally. Two options:

* **A client per conversation** — build the model/client with the trace id in its default
  headers and use it for that conversation's calls.
* **Context-scoped injection** — keep one client and inject the current run's headers from a
  context store, so every model request the framework makes picks them up automatically. This
  is usually the cleaner option, shown below.

The idea is the same in both languages: hold the current run/step headers in a context store
(`contextvars` in Python, `AsyncLocalStorage` in Node), and give the model client a
custom HTTP layer that reads that store on every request. You then set the store once around
each `agent.run()` — every internal model call in that run (including tool-loop calls) inherits
the headers. The natural granularity is **per agent/tool step**, which is exactly the
`span_path` you want: set the store before each agent or tool boundary.

<CodeGroup>
  ```python Pydantic-AI (Python) theme={null}
  import contextvars, os, uuid
  import httpx
  from pydantic_ai import Agent
  from pydantic_ai.models.openai import OpenAIChatModel
  from pydantic_ai.providers.openai import OpenAIProvider

  # Holds the headers for whichever run/step is currently executing.
  _run_ctx: contextvars.ContextVar[dict[str, str]] = contextvars.ContextVar("majordomo_run", default={})

  async def _inject_majordomo_headers(request: httpx.Request) -> None:
      for key, value in _run_ctx.get().items():
          if value:
              request.headers[key] = value

  # One client; the X-Majordomo-Key is constant, the run/step headers come from the hook.
  http_client = httpx.AsyncClient(
      headers={"X-Majordomo-Key": os.environ["MAJORDOMO_API_KEY"]},
      event_hooks={"request": [_inject_majordomo_headers]},
  )

  model = OpenAIChatModel(
      "gpt-4o",
      provider=OpenAIProvider(
          base_url=f"{os.environ['MAJORDOMO_GATEWAY_URL']}/v1",
          api_key=os.environ["OPENAI_API_KEY"],  # provider key, passed through Steward
          http_client=http_client,
      ),
  )
  agent = Agent(model)

  def run_headers(trace_id: str, span_path: str | None = None, span_name: str | None = None) -> dict[str, str]:
      h = {"X-Majordomo-Trace-Id": trace_id}
      if span_path:
          h["X-Majordomo-Span-Path"] = span_path
      if span_name:
          h["X-Majordomo-Span-Name"] = span_name
      return h

  trace_id = str(uuid.uuid4())  # one id per conversation / agent run

  # Set the store before each step. All of this run's model calls inherit the headers.
  _run_ctx.set(run_headers(trace_id, "planner/tool:search_db", "summarize"))
  result = await agent.run("Summarize these rows: ...")

  # Under concurrency (asyncio.gather of multiple runs), call _run_ctx.set(...) *inside*
  # each task — contextvars are copied per task, so each run keeps its own headers.
  ```

  ```typescript Mastra (TypeScript) theme={null}
  import { AsyncLocalStorage } from 'node:async_hooks';
  import { createOpenAI } from '@ai-sdk/openai';
  import { Agent } from '@mastra/core/agent';

  // Holds the headers for whichever run/step is currently executing.
  const runContext = new AsyncLocalStorage<Record<string, string>>();

  // Custom fetch: adds the constant key plus the current run/step headers to every request.
  const majordomoFetch = (input: RequestInfo | URL, init: RequestInit = {}): Promise<Response> => {
    const headers = new Headers(init.headers);
    headers.set('X-Majordomo-Key', process.env.MAJORDOMO_API_KEY!);
    for (const [key, value] of Object.entries(runContext.getStore() ?? {})) {
      headers.set(key, value);
    }
    return fetch(input, { ...init, headers });
  };

  const openai = createOpenAI({
    baseURL: `${process.env.MAJORDOMO_GATEWAY_URL}/v1`,
    apiKey: process.env.OPENAI_API_KEY, // provider key, passed through Steward
    fetch: majordomoFetch,
  });

  const agent = new Agent({
    name: 'support-copilot',
    instructions: '...',
    model: openai('gpt-4o'),
  });

  const traceId = crypto.randomUUID(); // one id per conversation / agent run

  // Run each step inside the store; every model call in the run inherits the headers.
  await runContext.run(
    {
      'X-Majordomo-Trace-Id': traceId,
      'X-Majordomo-Span-Path': 'planner/tool:search_db',
      'X-Majordomo-Span-Name': 'summarize',
    },
    () => agent.generate('Summarize these rows: ...'),
  );
  ```
</CodeGroup>

`AsyncLocalStorage` isolates concurrent runs automatically; in Python, set the `contextvars`
value inside each task so parallel runs don't share one store.

## Viewing runs

In the dashboard, open **Observe → Runs**:

* The **runs list** shows one row per run with its rolled-up cost, token totals, call count,
  and duration.
* Opening a run shows the **waterfall**: the nested steps and LLM calls with per-node cost.
  Click any LLM node to open its full request detail.

Runs and **Logs** are two views of the same requests. Every request still appears in Logs;
a request that belongs to a run gets a **Run** link there, and a quick **Exclude runs**
toggle hides run requests when you want only standalone traffic.

## Querying runs directly

The headers are stored as first-class columns on `llm_requests`, so a run's cost is a plain
aggregate (see the [data model](/reference/schema)):

```sql theme={null}
-- Cost and calls per run in the last 7 days
SELECT
    trace_id,
    COUNT(*)          AS calls,
    SUM(total_cost)   AS total_cost,
    MIN(requested_at) AS started_at,
    MAX(responded_at) AS ended_at
FROM llm_requests
WHERE trace_id IS NOT NULL
  AND requested_at > now() - interval '7 days'
GROUP BY trace_id
ORDER BY total_cost DESC;
```

```sql theme={null}
-- Cost per step within one run
SELECT
    split_part(span_path, '/', 1) AS top_step,
    span_path,
    COUNT(*)        AS calls,
    SUM(total_cost) AS step_cost
FROM llm_requests
WHERE trace_id = 'run_7f3a'
GROUP BY span_path
ORDER BY span_path;
```

## Runs vs. metadata

Use **agent run tracking** to group the calls of one execution into a tree. Use
[metadata headers](/guides/cost-attribution) (feature, team, environment, user) to attribute
cost across dimensions. They compose: stamp a run with a trace id **and** a
`X-Majordomo-Feature` so you can both drill into a single run and roll runs up by feature.