Agent Run Tracking

Conversational agents and workflow agents make many LLM calls per task — a planning call, tool calls, follow-ups. By default the gateway logs each as an independent request. Agent run tracking lets you stamp those calls with a shared id so the dashboard shows them as one run: a rolled-up total cost, plus a nested waterfall of which tool/agent step drove which calls and what each cost. Because the gateway is a proxy, it sees your LLM calls but never the tool/agent code that runs between them. So the tree is reconstructed from headers your client sends — no SDK or in-app tracer required. Any client that can set request headers works.

The headers

Header	Required	Value
`X-Majordomo-Trace-Id`	To join a run	One id per conversation / agent run. Generate it once when the run starts and send it on every LLM call in that run (any opaque string).
`X-Majordomo-Span-Path`	No	`/`-joined names of the ancestor steps from the run root down to this call’s parent, e.g. `planner/tool:search_db`. Omit to hang the call directly under the run root.
`X-Majordomo-Span-Name`	No	Label for this call in the waterfall. Defaults to the model name.

Graceful degradation. A trace id alone gives you a flat run rollup — every call in the run plus its total cost. Adding a span path gives you the waterfall — the calls grouped under named tool/agent steps, with cost rolled up each step. Start with just a trace id; add paths when you want the tree.

/ is the reserved separator between step names. If a step name itself contains a /, percent-encode it so it doesn’t split into extra nodes.

Modeling the tree

Send, on each LLM call, the path of the steps above it. Interior tool/agent nodes don’t make their own request — they’re inferred from the paths of the calls beneath them. For an agent that plans, then calls a search_db tool that makes two calls:

planner                          # X-Majordomo-Span-Path: planner
├─ route          $0.004         #   Span-Name: route
└─ tool:search_db                # X-Majordomo-Span-Path: planner/tool:search_db
   ├─ summarize   $0.021         #   Span-Name: summarize
   └─ rank        $0.027         #   Span-Name: rank

The tool:search_db node’s cost ($0.048) is the sum of the calls beneath it; the planner node totals the whole run.

Sending the headers

import os, uuid
from majordomo_llm import get_llm_instance

llm = get_llm_instance(
    "anthropic", "claude-sonnet-4-6",
    base_url=os.environ["MAJORDOMO_GATEWAY_URL"],
    default_headers={"X-Majordomo-Key": os.environ["MAJORDOMO_API_KEY"]},
)

# One id per conversation / agent run, reused across every call in the run.
trace_id = str(uuid.uuid4())

await llm.get_response(
    user_prompt="Which tool should I use?",
    extra_headers={
        "X-Majordomo-Trace-Id": trace_id,
        "X-Majordomo-Span-Path": "planner",
        "X-Majordomo-Span-Name": "route",
    },
)

await llm.get_response(
    user_prompt="Summarize these rows: ...",
    extra_headers={
        "X-Majordomo-Trace-Id": trace_id,
        "X-Majordomo-Span-Path": "planner/tool:search_db",
        "X-Majordomo-Span-Name": "summarize",
    },
)

from openai import OpenAI

client = OpenAI(
    base_url="https://gateway.gomajordomo.com/v1",
    api_key="your-openai-key",
    default_headers={"X-Majordomo-Key": "mdm_sk_your_key_here"},
)

client.chat.completions.create(
    model="gpt-4o",
    messages=[...],
    extra_headers={
        "X-Majordomo-Trace-Id": "run_7f3a",
        "X-Majordomo-Span-Path": "planner/tool:search_db",
        "X-Majordomo-Span-Name": "summarize",
    },
)

curl -X POST https://gateway.gomajordomo.com/v1/chat/completions \
  -H "Authorization: Bearer your-openai-key" \
  -H "X-Majordomo-Key: mdm_sk_your_key_here" \
  -H "X-Majordomo-Trace-Id: run_7f3a" \
  -H "X-Majordomo-Span-Path: planner/tool:search_db" \
  -H "X-Majordomo-Span-Name: summarize" \
  -H "Content-Type: application/json" \
  -d '{"model": "gpt-4o", "messages": [{"role": "user", "content": "..."}]}'

Frameworks that hide per-request headers

Pydantic-AI, Mastra, and similar frameworks don’t expose a per-request header hook on agent.run() — they own the model-call loop internally. Two options:

A client per conversation — build the model/client with the trace id in its default headers and use it for that conversation’s calls.
Context-scoped injection — keep one client and inject the current run’s headers from a context store, so every model request the framework makes picks them up automatically. This is usually the cleaner option, shown below.

The idea is the same in both languages: hold the current run/step headers in a context store (contextvars in Python, AsyncLocalStorage in Node), and give the model client a custom HTTP layer that reads that store on every request. You then set the store once around each agent.run() — every internal model call in that run (including tool-loop calls) inherits the headers. The natural granularity is per agent/tool step, which is exactly the span_path you want: set the store before each agent or tool boundary.

import contextvars, os, uuid
import httpx
from pydantic_ai import Agent
from pydantic_ai.models.openai import OpenAIChatModel
from pydantic_ai.providers.openai import OpenAIProvider

# Holds the headers for whichever run/step is currently executing.
_run_ctx: contextvars.ContextVar[dict[str, str]] = contextvars.ContextVar("majordomo_run", default={})

async def _inject_majordomo_headers(request: httpx.Request) -> None:
    for key, value in _run_ctx.get().items():
        if value:
            request.headers[key] = value

# One client; the X-Majordomo-Key is constant, the run/step headers come from the hook.
http_client = httpx.AsyncClient(
    headers={"X-Majordomo-Key": os.environ["MAJORDOMO_API_KEY"]},
    event_hooks={"request": [_inject_majordomo_headers]},
)

model = OpenAIChatModel(
    "gpt-4o",
    provider=OpenAIProvider(
        base_url=f"{os.environ['MAJORDOMO_GATEWAY_URL']}/v1",
        api_key=os.environ["OPENAI_API_KEY"],  # provider key, passed through Steward
        http_client=http_client,
    ),
)
agent = Agent(model)

def run_headers(trace_id: str, span_path: str | None = None, span_name: str | None = None) -> dict[str, str]:
    h = {"X-Majordomo-Trace-Id": trace_id}
    if span_path:
        h["X-Majordomo-Span-Path"] = span_path
    if span_name:
        h["X-Majordomo-Span-Name"] = span_name
    return h

trace_id = str(uuid.uuid4())  # one id per conversation / agent run

# Set the store before each step. All of this run's model calls inherit the headers.
_run_ctx.set(run_headers(trace_id, "planner/tool:search_db", "summarize"))
result = await agent.run("Summarize these rows: ...")

# Under concurrency (asyncio.gather of multiple runs), call _run_ctx.set(...) *inside*
# each task — contextvars are copied per task, so each run keeps its own headers.

import { AsyncLocalStorage } from 'node:async_hooks';
import { createOpenAI } from '@ai-sdk/openai';
import { Agent } from '@mastra/core/agent';

// Holds the headers for whichever run/step is currently executing.
const runContext = new AsyncLocalStorage<Record<string, string>>();

// Custom fetch: adds the constant key plus the current run/step headers to every request.
const majordomoFetch = (input: RequestInfo | URL, init: RequestInit = {}): Promise<Response> => {
  const headers = new Headers(init.headers);
  headers.set('X-Majordomo-Key', process.env.MAJORDOMO_API_KEY!);
  for (const [key, value] of Object.entries(runContext.getStore() ?? {})) {
    headers.set(key, value);
  }
  return fetch(input, { ...init, headers });
};

const openai = createOpenAI({
  baseURL: `${process.env.MAJORDOMO_GATEWAY_URL}/v1`,
  apiKey: process.env.OPENAI_API_KEY, // provider key, passed through Steward
  fetch: majordomoFetch,
});

const agent = new Agent({
  name: 'support-copilot',
  instructions: '...',
  model: openai('gpt-4o'),
});

const traceId = crypto.randomUUID(); // one id per conversation / agent run

// Run each step inside the store; every model call in the run inherits the headers.
await runContext.run(
  {
    'X-Majordomo-Trace-Id': traceId,
    'X-Majordomo-Span-Path': 'planner/tool:search_db',
    'X-Majordomo-Span-Name': 'summarize',
  },
  () => agent.generate('Summarize these rows: ...'),
);

AsyncLocalStorage isolates concurrent runs automatically; in Python, set the contextvars value inside each task so parallel runs don’t share one store.

Viewing runs

In the dashboard, open Observe → Runs:

The runs list shows one row per run with its rolled-up cost, token totals, call count, and duration.
Opening a run shows the waterfall: the nested steps and LLM calls with per-node cost. Click any LLM node to open its full request detail.

Runs and Logs are two views of the same requests. Every request still appears in Logs; a request that belongs to a run gets a Run link there, and a quick Exclude runs toggle hides run requests when you want only standalone traffic.

Querying runs directly

The headers are stored as first-class columns on llm_requests, so a run’s cost is a plain aggregate (see the data model):

-- Cost and calls per run in the last 7 days
SELECT
    trace_id,
    COUNT(*)          AS calls,
    SUM(total_cost)   AS total_cost,
    MIN(requested_at) AS started_at,
    MAX(responded_at) AS ended_at
FROM llm_requests
WHERE trace_id IS NOT NULL
  AND requested_at > now() - interval '7 days'
GROUP BY trace_id
ORDER BY total_cost DESC;

-- Cost per step within one run
SELECT
    split_part(span_path, '/', 1) AS top_step,
    span_path,
    COUNT(*)        AS calls,
    SUM(total_cost) AS step_cost
FROM llm_requests
WHERE trace_id = 'run_7f3a'
GROUP BY span_path
ORDER BY span_path;

Runs vs. metadata

Use agent run tracking to group the calls of one execution into a tree. Use metadata headers (feature, team, environment, user) to attribute cost across dimensions. They compose: stamp a run with a trace id and a X-Majordomo-Feature so you can both drill into a single run and roll runs up by feature.

Getting Started

How It Works

Connect Your Code

Track & Analyze

Control

Enterprise

Reference

Agent Run Tracking

The headers

Modeling the tree

Sending the headers

Frameworks that hide per-request headers

Viewing runs

Querying runs directly

Runs vs. metadata

​The headers

​Modeling the tree

​Sending the headers

​Frameworks that hide per-request headers

​Viewing runs

​Querying runs directly

​Runs vs. metadata

The headers

Modeling the tree

Sending the headers

Frameworks that hide per-request headers

Viewing runs

Querying runs directly

Runs vs. metadata