What is an Agent Harness? Running Governed Managed Agents in Production

Choosing a model and connecting it to tools can get an AI agent working. Getting that agent ready for production is a different challenge.

You need a reliable execution loop, a secure environment for running code, approval gates for sensitive actions, and traces that show what happened during each run. These capabilities sit in a layer called the agent harness. And once you're building agents at scale, you have to decide whether to build this layer yourself or use a managed solution.

Here’s an example: Sofia, a platform engineer, inherited three teams' worth of agents and a request to make them production-ready. 

Each team had built its own runtime. One wrote an orchestration loop in Python, another wrapped an agent framework, and the third called a model directly from a cron job. API keys were stored in agent configurations and committed to repositories. Approval processes varied between teams, and two had no reliable way to trace an agent's actions.

The agents weren't necessarily missing better models or more tools. They were missing a common runtime to execute tasks, manage credentials, enforce approvals, and track what happened. Sofia needed an agent harness.

This is a common challenge once the initial demo works. Building an agent is one problem; operating it reliably and securely is another. Without a shared harness, every team ends up rebuilding the same capabilities, often with different gaps in security and observability. In this post, we'll explain what an agent harness does, which capabilities make it production-ready, and what to consider when deciding whether to build your own or adopt a managed harness.

Bring Your AI Agents to Production with TrueFoundry

Build, deploy, and govern AI agents with centralized credentials, controlled tool access, and end-to-end observability.

Book a Demo

What is an Agent Harness?

An agent harness is the runtime layer around a large language model (LLM) that turns it from a text generator into an agent that can execute tasks. It manages the execution loop: the model decides what to do, the harness calls the required tool, observes the result, and feeds that result back to the model. The loop continues until the task is complete or a configured limit is reached.

The harness also handles the components needed to run agents reliably, including tool execution, memory, context management, security controls, human approval, and observability.

Think of the harness as the infrastructure that keeps an agent's work on track. The model provides the reasoning, and tools provide external capabilities. The harness connects them, manages execution, and enforces the boundaries that make the workflow usable in production.

How does an agent harness work?

An agent harness coordinates the steps between receiving a task and producing a result. Here’s how it works: 

1. The application or agent interface receives the user's request and passes it to the agent runtime. The harness creates a run and combines the request with the agent's instructions, relevant conversation history, available tools, and any retrieved context. It prepares this information in the format the model expects.
2. The harness assembles the context the model needs to respond. This can include system instructions, the user's request, previous messages, retrieved documents, tool descriptions, and outputs from earlier tool calls. A context-management component may retrieve, filter, summarize, or compact this information to keep it relevant. The harness then sends the assembled context to the LLM through the model API.
3. The LLM processes the request and context, then returns either a response or a proposed tool call. For example, a coding agent might decide it needs to inspect a file before fixing a bug. The model specifies the tool it wants to use and the arguments required. The harness receives this output and determines whether the proposed action is allowed to execute.
4. When the model requests a tool, the harness routes the call to the appropriate API, MCP server, database, or code execution environment. Before execution, it can check permissions, validate arguments, apply guardrails, and request human approval for sensitive actions. The tool performs the requested operation and returns its result to the harness. The model proposes the action; the harness and tool infrastructure handle its controlled execution.
5. The harness passes the tool's output back into the agent's context, typically as a tool-result message. It sends the updated context to the model again, allowing the model to interpret the result and choose the next action. The agent can repeat this loop across multiple tool calls until it has enough information or has completed the task. If the context grows too large, the harness may summarize or compact earlier information before continuing.
6. When the model returns a final response instead of requesting another action, the harness ends the run and passes the result back to the application for delivery to the user. It also records execution details such as model calls, tool activity, latency, token usage, and cost, depending on the configured observability system. The harness can also stop execution if it reaches a step limit, timeout, budget limit, or other configured constraint.

For example, a coding agent asked to fix a bug might inspect a repository, run tests, modify a file, and rerun the tests. The agent harness coordinates these actions and controls how the agent accesses the code and execution environment.

What Does an Agent Harness Manage?

An agent harness brings together several components that would otherwise need to be implemented and maintained separately.

• Tool routing and execution connect the model to APIs, MCP servers, databases, and code execution environments. When the model requests a tool, the harness routes the call, receives the result, and passes it back to the model to determine the next action.
• Memory and context management maintain conversation history, task state, and relevant information across multiple steps. The harness can retrieve information, summarize previous interactions, and compact context to keep it relevant during longer workflows.
• Security and permissions control which tools, credentials, files, and systems an agent can access. The harness can enforce access policies and use sandboxes to isolate code execution and limit access to sensitive resources.
• Human approval workflows pause execution when an action requires review, such as modifying production data or deleting files. The harness resumes or stops the workflow based on the approval decision and configured policies.
• Reliability and execution limits handle tool errors, retries, timeouts, and iteration limits. These controls help prevent agents from repeating failed actions or running indefinitely.
• Observability and tracing track model requests, tool calls, errors, latency, token usage, and costs. End-to-end traces help teams understand each step of a run and identify performance issues or failures.

These components help teams move beyond simple model calls to agents that can perform multi-step work. However, a harness does not automatically make an agent secure or reliable. Its effectiveness depends on how its permissions, runtime controls, and execution policies are configured.

Agent Harness vs Agent Framework vs Agent Runtime

An agent harness, agent framework, and agent runtime play different roles in building and running AI agents. 

A framework provides the tools to build an agent, a harness manages how it carries out tasks, and a runtime provides the environment where it runs. 

An agent harness manages how an AI agent carries out a task. It coordinates model calls, tool execution, and the processing of results until the task is complete. It can also manage permissions, retries, execution limits, and logging to keep the agent's actions controlled and observable.

An agent framework provides the components developers need to build AI agents. These include model integrations, tool-calling interfaces, and workflow orchestration. For example, LangGraph helps developers define how an agent moves between tasks and tools. The harness manages the execution process, although some frameworks provide both capabilities.

An agent runtime provides the environment where an AI agent executes its tasks. It manages computing resources, process isolation, and deployment. For example, when a coding agent needs to run a test, the harness coordinates the action, while the runtime provides the environment in which the test runs.

Where Does an Agent Harness Fit in the AI Infrastructure Stack? 

The harness is the capstone of the control-plane story this series has been telling. The AI Gateway governs model traffic; the MCP Gateway governs tool traffic; the Agent Gateway governs agent-to-agent traffic; the Skills Registry governs reusable, versioned skills; and the harness is the runtime that ties them together into a managed agent, running in the same plane so orchestration, governance, and observability stay in one system rather than scattered across layers.

That co-location is the reason the harness can keep credentials out of definitions, gate destructive tools once, and trace a run end-to-end: it's not bolting governance onto an agent runtime after the fact; it's running the agent runtime inside the governance plane that was already there. For Sofia, that's the difference between standardizing three teams onto one governed runtime and continuing to babysit three bespoke ones.

How Does an Agent Harness Compare with Managed Agent Platforms? 

TrueFoundry isn't the only managed harness, and the alternatives are good. Anthropic's Claude Managed Agents and LangChain's LangSmith Managed Deep Agents are both strong hosted runtimes, and for many teams they're an excellent fit. The useful way to compare them isn't a scoreboard; it's a difference in design philosophy, and which philosophy fits depends on your situation.

As documented at the time of writing, many managed-cloud runtimes lean pro-code and API-first: you define agents, sessions, and environments through an SDK or REST API, register credentials per agent or workspace (vault IDs, header arrays), set tool-approval policies per agent, and run in the provider's managed cloud. 

That's a clean, tightly integrated model — especially if you're already standardized on one model provider or one orchestration framework, and your team is comfortable defining agents in code. TrueFoundry's philosophy is different: a no-code builder first (with the same definition available via SDK and REST), credentials centralized in the gateway control plane rather than registered per agent, destructive-tool approval flagged once org-wide, model access governed with RBAC and budgets across any provider, and deployment available as SaaS, self-hosted, or on-prem. The trade is integration simplicity within one ecosystem versus centralized governance across many.

Managed harness vs building your own

Building your own agent harness gives you control over tool execution, context management, permissions, and workflows. But your team must also build and maintain retries, failure handling, security controls, and observability.

A managed harness provides many of these capabilities out of the box, reducing infrastructure work so your team can focus on agent logic. Building makes sense when you need specialized behavior and can maintain it. A managed option is often better when you want to deploy faster and reduce operational overhead.

Why "Build vs. Buy" the Harness is the Real Decision

Here's the uncomfortable arithmetic of agentic AI: the model is a few lines of API call, the tools are a registry away, and the harness is the rest of the system. The orchestration loop with its stopping conditions, the sandbox with its lifecycle, the credential handling, the approval gates, the per-step tracing and cost accounting — that's where the engineering months go, and almost none of it is specific to your use case. It's undifferentiated heavy lifting, rebuilt every time a team starts an agent from scratch.

That's the trap Sofia's three teams fell into independently. Each spent real effort on runtime plumbing instead of on the agent's actual job, and each produced a different, partial version with its own gaps — the committed API key, the missing approval gate, the absent trace. Buying the harness (or adopting a managed one) is the decision to stop rebuilding the plumbing and standardize it once, so teams spend their effort on instructions, tools, and skills — the parts that are actually theirs.

Govern Every AI Agent from One Platform

Manage model access, MCP tools, credentials, and approval policies centrally without rebuilding runtime infrastructure for every agent.

Book a Demo

The TrueFoundry Agent Harness: Model + MCP + Skills + Instructions

TrueFoundry's Agent Harness is a managed harness built on top of the AI Gateway and MCP Gateway. The builder's mental model is deliberately small: you choose a model, connect MCP servers, add skills, and write instructions. 

TrueFoundry provides the rest as managed capabilities — orchestration, sandbox lifecycle, tool execution, approval flows, governance, and observability — while teams still own the agent's instructions, tools, policies, and deployment posture. There's a no-code builder in the console for non-developers and a Python SDK and REST API for the same agent definition, so the path from "pick a model" to "working agent" is short, and the same definition can be driven from code.

An agent defined declaratively — model, tools, skills, instructions (illustrative)

name: refund-assistant
model: claude-sonnet-4-6          # a name, not a key — credentials live in the gateway
mcp_servers:
  - zendesk                       # governed tools, referenced by name
  - payments
skills:
  - refund-policy@v4              # versioned SKILL.md from the Skills Registry
instructions: |
  Resolve refund requests within policy. Confirm any refund over $100 with the user.

Notice what's not in that definition: any secret. The model is a name, the MCP servers are names, the skill is a versioned reference. That absence is the whole point of the next section, and it's the difference between an agent definition you can safely commit to a repo and one you can't.

Why Your Agents Shouldn't Handle API Keys and Credentials

The single most consequential design choice in a harness is where credentials live. The tempting answer — paste the provider key and the tool tokens into the agent definition — is exactly what put Sofia's keys in a repo. It doesn't scale (every agent and every user re-registers secrets), it doesn't rotate cleanly (a key change touches every definition), and it doesn't stay secure (secrets spread to wherever definitions are stored).

TrueFoundry's harness takes the other path: no API keys or credentials are ever pasted into agent definitions. Provider credentials live in the AI Gateway, and agents reference model names while RBAC, budgets, and guardrails are enforced at the gateway. 

MCP authentication — OAuth tokens, API keys — lives in the MCP Gateway, which handles credential injection, token refresh, and per-user delegation, so users authenticate inline and the agent calls tools by name. Skills are published in the Skills Registry with versioning and RBAC, so agents pick from a governed catalog. Platform teams configure access once; agent builders never handle secrets.

The Capabilities That Make it Production-Ready

An agent harness needs more than a loop that sends requests to a model and executes tool calls. To run reliably in production, it must manage context, control tool execution, handle failures, and know when to ask for human input. TrueFoundry's harness brings these capabilities together to address common problems that arise when agents handle real tasks.

Context and State Management

Long-running agents can fill their context windows with tool results, conversation history, and intermediate outputs. Context engineering helps keep that information manageable. TrueFoundry's harness supports subagents to isolate subtasks, code mode to process data programmatically, large-result offloading to prevent oversized tool outputs from overwhelming the context window, and compaction for longer runs. This complements gateway-side session management covered in our context engineering guide.

Tool Execution and Sandboxing

A sandbox gives an agent a controlled environment to run code, handle files, and execute tasks without directly exposing the host or sensitive systems. This is especially important for coding agents and workflows that process untrusted inputs. The harness also needs to manage tool calls and their results so the agent can continue working within defined limits.

Human Approval and Guardrails

Some tool calls should not execute without explicit approval. Human-in-the-loop controls let the harness pause a run until an authorized person approves a sensitive action. TrueFoundry can flag destructive tools centrally at the MCP Gateway, so the approval requirement applies to every agent that uses those tools instead of relying on each agent builder to configure it correctly.

Error Handling and Long-Running Tasks

Production agents need to handle failed tool calls, retries, execution limits, and interruptions without losing track of the task. Run tracing also helps teams understand what happened at each step and investigate failures. These controls are particularly important when a task involves several tools or takes a long time to complete.

Generative UI and Agent Interaction

Agents sometimes need more than a text response to complete a task. The harness can support interactions in which an agent asks the user for clarification or presents choices before continuing. Generative UI takes this further by allowing the agent to stream structured content, such as cards, tables, and charts, that the client can render. This makes it easier for users to review information and respond while a task is in progress.

Each capability addresses a specific failure mode. The table below shows how they contribute to a production-ready agent harness.

session = harness.sessions.create(
    agent="refund-assistant",
    input="Refund order #8842, $240"
)

for event in session.stream():
    if event.type == "tool_call":
        log(event.tool, event.args)
    elif event.type == "approval_required":
        decision = ask_human(event)
        session.respond(event.id, decision)
    elif event.type == "final":
        return event.output

In this example, the harness streams run events, logs tool calls, and pauses when a tool requires approval. The code is illustrative; the exact event names and API methods depend on the implementation.

The approval gate deserves particular attention because it determines when an agent can act independently. Instead of asking every agent builder to configure approval for every sensitive action, platform teams can flag destructive tools centrally.

Illustrative policy configured at the MCP Gatewaytools:  payments.issue_refund:    destructive: true    approval: require_user  payments.read_balance:    destructive: false

Here, issuing a refund requires approval, while reading a balance does not. Centralizing this policy helps apply the same rule across agents that use the tool.

However, requiring approval is only part of the control. A useful policy should specify which tools require approval, who can approve, whether approval applies to an individual call or an entire session, what information the approver sees, and how long the approval remains valid. For example, a refund request should show the amount and recipient before approval. Without that context, a person cannot make an informed decision.

By enforcing these rules at the platform level, an agent harness helps teams maintain consistent controls as they build and deploy more agents.

Observability: One Pane Across Model, Tool, and Agent Traffic

Two of Sofia's three teams couldn't reconstruct what an agent did on a given run, which is the observability gap this whole series keeps returning to — now at the agent level. A harness has to emit an end-to-end trace per run: every LLM call, tool call, sandbox execution, and subagent, with cost, tokens, and latency attributed per step. Without that, a misbehaving agent is a black box, and the only signal you get is the bill or the complaint.

What the trace carries matters as much as that it exists. A useful per-step record ties together the run and step identity, the model and tool involved, the approval decision, the sandbox session, and the cost, tokens, and latency of the step — enough to reconstruct, attribute, and cost any run after the fact:

An illustrative per-step trace record (shape is gateway-specific)

{  "agent.run_id": "run_abc123",  "agent.name": "refund-assistant",  "agent.step.type": "tool_call",  "agent.step.name": "payments.issue_refund",  "model": "claude-sonnet-4-6",  "mcp.server": "payments",  "approval.required": true,  "approval.status": "approved",  "sandbox.session_id": "sbx_7f1c",  "tokens.input": 1842,  "tokens.output": 391,  "latency_ms": 2200,  "cost_usd": 0.0142}‍

A run is then just the ordered set of these records under one run_id — which is what turns "the agent did something expensive" into "step 4 called issue_refund after an approved gate, costing this much."

Because TrueFoundry's harness runs on the same gateway plane as model and MCP traffic, it inherits the AI Gateway's analytics, request logs, OpenTelemetry export, and Prometheus and Grafana integration — one pane of glass across model, MCP, and agent traffic rather than three disconnected dashboards. 

Sign up to see TrueFoundry in action.