This article was originally published on the Dataiku medium publication data from the trenches.

Harness engineering has become a hot topic and seems to be ever evolving. What framework or architecture should it use? What capabilities should it include? How do I know if it is good enough? How do I make it safe and reliable? What even is an agent harness?

At Dataiku, we worked with 60+ enterprise customers across multiple industry verticals to help them deliver agents in production from which we’ve been able to identify AI agent building patterns.

We distilled all these agent building learnings into this article, splitting the agent building process into three layers: the model, the agent harness, and the agent harness configuration.

Layer 1: The model

The first thing to remind ourselves is that an agent is a powerful system and, at its core, it boils down to a smart orchestration of LLM calls. The system that handles and orchestrates LLM calls within an agent is what we will call the agent harness.

LLM output generations can be structured to enable tool calling, the ability of an LLM to request for an action to be performed, which has become the backbone of all agentic systems.

The challenge is that a single LLM call is only able to “request” for a tool to be called, it doesn’t directly have the ability to execute it. That means that we still have to call the requested tool on behalf of the model for the action to be performed. This motivates the need for the second layer: the agent harness.

Layer 2: The agent harness

Historically, agentic systems were limited by their underlying model’s performance. Recently, the limiting factor has shifted to the agent harness. More real-world agent harness benchmarks like Terminal-Bench showcases how significant performance improvement can be achieved and attributed solely to agent harness optimizations.

That being said, with new agent harness features and capabilities being released on an almost daily basis, it can get complicated to focus on the ones that will survive long term versus the brittle features destined to be eaten by the next big model release feature, commonly known as “agent scaffolding.”

Agent harness core model loop

To give more agency to a model, we want to build a mechanism to inform the model of their previous action’s result so it can decide to perform another action OR give its final answer. This looping mechanism typically follows the ReAct Framework and is a foundation feature of any agent harness.

We found that the agent harness’s core loop can be reduced to a repeating cycle of five steps:

1. Input Request: This can be a new user request, a user clarification, or the result of the previous core model loop iteration.

2. Context Engineering: This step focused on assembling all the context the model needs for its next generation including general LLM guidelines saved in the system prompt, user preferences, historical conversation & steps taken, current plan, expertise instructions, and retrieved data.

3. Equip Relevant Tools: Select and attach the tools available to the model for this turn. These are generally kept the same but could be updated based on the task requirements.

4. Run the Model: Choose the right model to run based on task complexity/cost requirements and execute it with the assembled context and tools that handle model errors and retries.

5. Harness Housekeeping (where most of the complexity lies): Also known as hooks or middleware, this step parses the model outputs; executes any requested tool or sub-agent call while handling errors and retries; applies guardrails; provides observability and logging; manages memory; tests the agent’s work and calls for user intervention when needed (human in the loop).

At the end of each cycle, a decision is made to return a final response OR feeds the result of the current cycle into the start of the next one and kicks it off.

Most of the core agent harness features we will cover are built on top of this core model loop and apply to multiple steps of the harness cycle. They can shape context in step 2, provide tools in step 3, and enforce rules in step 5.

Simple agent harness limitations

We started implementing agents with the simplest harness possible: a core loop. In theory, an agent harness equipped with a core model loop, with the right tools to interact with its environment and a capable model should be enough to complete any task.

In practice, we found that it was not the case. As tasks get longer and more complex, a bare core model loop harness runs into problems like:

• Context Rot when information in the context gets out of date or irrelevant.

• Context Anxiety when the model ends early when it approaches its context window.

• Tool Pollution when tool descriptions drown the context.

• Infinite loops when getting stuck in a local minima.

• Lack of visibility and trust as harness reasoning can become an unconstrained blackbox.

Agent harness design principles

This motivated the need to add more features to our harness to remediate the issues listed above. Through trial and error, we noticed that to avoid building unnecessary “agent scaffolding,” the following design principles have reliably helped produce versatile and robust agent harness features.

• Progressive disclosure: As agent working environments start getting more and more complex, agent harnesses should be designed to dynamically allow discovery, navigation and deep dive into knowledge about that environment based on the assigned task.

• Design for reliability not capability: You can have all the capabilities in the world, but if you can’t trust the agent, its value and usability drops to zero.

• Build for the models of tomorrow: Build features that work for the model today and that will allow future better model versions to grow into the harness and not be restricted by it.

Agent harness feature implementation patterns

When implementing a harness feature, we found that we needed to choose the level of autonomy which ended up falling into one of four implementation patterns. Let’s take the agent memory feature ordered by increasing level of model agency as an example:

1. Fully handled by the harness: The harness manages the feature end to end, invisible to the model. Ex: At the start of each turn, relevant user preferences & memory fragments are injected automatically into the context by the harness.

2. Specialized tool-based capability: The model is given access to the capability via hyper-specific tools, with some light agent harness housekeeping. Ex: The agent is given specific memorizing and remembering tools that include deduplication of memory fragments.

3. High-level instruction: The model is instructed at a high level how to implement and run the feature via core multi-purpose generic tools. Ex: The harness is instructed to save and load any relevant user information in a user/memory.md file, segmenting the memory by type.

4. Full autonomy: The model just knows that the feature is required, decides on how to implement it, and how to use it. Ex: The model decides to build a memory system and leverages the generic tools available to spin up a database or file systems structure as appropriate.

As models get better, harness builders naturally start to favor implementing harness features in the 3–4 buckets. This naturally increases agent entropy, making controlling, standardizing, and monitoring agent behaviors more challenging. Working mainly with enterprise agents, all the harness features we implemented landed in the 1–3 range for control and auditability reasons, with any feature related to access control and security kept in the first bucket.

Agent harness core features

It is finally time to dive into the features we ended up equipping our enterprise agents.

We can divide these harness features into two categories. Let’s use a car as an analogy to motivate both categories. The car’s purpose is to get you from A to B, and the agent’s purpose is to execute (hopefully) valuable tasks delegated to it. The car, same as the agent harness, has two conflicting missions to achieve:

1. Performance: Get from A to B in the fastest way possible by converting as much raw motor power into speed, with the motor being the model and the car being the harness capabilities.

2. Safety: Keep the car from crashing or breaking down by providing reliable brakes, well maintained parts, etc. Here, the brakes would be the safety, reliability, and observability features within the agent harness.

Mission 1: Performance

A performant agent harness does two things well:

1. Gives the model access to the right context at the right time

2. Gives the model powerful and versatile tools to explore and interact with its environment.

Intuition tells us that if we want an agent to be as versatile as a human when assigned tasks, we need to equip it with the same tools that a human would use. Nowadays, most knowledge work can be done remotely via a computer and an internet connection. The idea therefore is, what if we give the agent access to a computer and an internet connection?

For the last 80+ years, the terminal has been the main text-based entrypoint into any computer, a medium that LLMs are known to be very good at. Terminal access would then allow agents to:

1. Progressively disclose information via the file system: This is done by leveraging commands like cat, grep, find, etc.

2. Persist information and work over time by writing/editing files to the filesystem: This allows the agent to progressively save data and decompose and perform complex tasks step by step.

3. Execute code via runtime environments: This allows the agent to run previously written or pre-existing scripts to execute complex predefined logic or deploy complete applications.

4. Call third-party applications: This allows the agent to call any existing tool, software, or application, provided they have a command line interface (CLI) like curl, ssh, run cron jobs, etc. to further interact with their environment to perform their test.

5. Leverage built-in Operating System (OS) features: This allows the use of the underlying hardware, access control & security, and networking native features.

Terminal access is necessary but not sufficient, as the models of today still need a bit more structure and guidance from the agent harness to efficiently use these low-level, powerful primitives provided by the terminal to reliably perform requested tasks. This is why we still equipped our agent harnesses with higher-level functionalities like:

• General system prompt providing guidance agnostic to the task at hand.

• Planning to decompose any task into manageable subtasks and track their progress.

• Skills to provide structured instruction & tribal knowledge discoverability to perform specific tasks.

• RAG to dynamically enrich the context with relevant data.

• Context management/reduction, a subsection of context engineering that allows the agent to keep their context as relevant as possible as the task horizon increases.

• Memory allows the agent to simulate remembering preference, facts, and procedures between turns and sessions.

• Self evaluation/correction loops via systems like the ralph loop that systematically reviews the work of the main agent to validate that the job is completed.

Mission 2: Safety

The faster and longer agents can do things, the faster mistakes can happen and compound. The more freedom and access the agents are given, the more dangerous those mistakes can become, which is why equipping the right agent “brakes” is equally, if not more important.

When it came time to build our agent harnesses, we made sure to provide the right level of security, reliability, and observability.

Security involves access control and prevention from malicious intent:

• User-level access control should be applied for every action performed by the model on behalf of the calling user, respecting that user’s permissions.

• A secure sandbox for code executions provides a safe environment to run agent-triggered code, separated from the existing agent environment following least privilege principles.

• Prompt injection prevention is needed for ingestion of uncontrolled or unverified data sources such as from web scraping or from external emails that should be vetted for any potential malicious intent, especially when in the lethal trifecta situation.

Reliability involves giving confidence in the agent abilities and maintainability via versioning and rollback of work capabilities and resilience in runtime execution failures:

• Versioning & rollbacks allows the user to see the history of changes and actions taken and rollback to an older state if needed, erasing all actions performed in between.

• Core harness cycle error management is the capability that allows the harness to handle errors within its cycle like LLM call errors, LLM output parsing error, tool calling errors, infinite loop detection, or context limit management.

Observability provides transparency and auditability to the overall system, all while delivering the right level of agency to the agent user:

• Agent interaction logging has two components, the agent execution logs captured in the form of a trace that contains all the steps taken by the agent and the agent runtime logs emitted by the agent’s runtime environment throughout a run.

• Live async user feedback informs the user on what is currently being worked on, what is the current plan and where human sign off is required. The right level of feedback loop can make or break your harness.

Each of these categories and sub topics could warrant their own article but it is important to know that they should be considered when designing the agent harness.

In practice, building a harness from scratch can be technically challenging, time consuming, and with the AI space rapidly changing, it is easy to quickly accumulate technical debt. Agent harnesses will also most likely end up getting standardized and commoditized in the near- to medium-term, making it an overall low return on investment to build one from scratch if it is not core to the business.

It is much more worthwhile to focus time and effort on the real agent-building value generator: the third layer of the agent building process, the configuration of the agent harnesses.

Layer 3: The agent harness configuration

The agent harness provides a fast car with adequate brakes, but we are still missing the wheels, so to speak. Only when the agent is given access to the right use case/environment-specific context and tools will it actually be able to hit the road and showcase that valuable performance.

To configure an agent harness, you first need to identify and define the use case that would benefit from an agent. This exercise can end up being more challenging than it seems, so we put together a simple and intuitive framework to help answer the question, “How can you identify an agentic use case?”

Once the agent’s core five characteristics (from the framework above) have been identified and documented, the agent harness can be configured following the next six steps.

1. Giving the agent access to context

Context is king. Provide the right context and the rest will follow. This can be done by giving general guidance via the system prompt (ex: AGENT.md) or providing procedural knowledge via skills, ranging from tribal knowledge to internal tool documentation to company-approved formats for building slide decks or reports.

2. Giving the agent access to use case/environment-specialized tools

On top of the built-in tools to the harness, the agent needs to be given access to the right read and write tools to properly interact with its environment. READ tools are needed to query CRM data, news data, shared emails, semantic data models or unstructured shared drive folders. WRITE (action) tools allow the agent to not only inform but do things for the user like submit draft email, generate pptx/doc, build data visualizations, submit a transaction, write/run code scripts, etc.

Tools implementations can range from MCP servers to CLIs. As a rule of thumb, the more generic tools, the better. Tools can be made more or less specific based on model performance and requirements for strict guardrails or auditability. As an example, a supply chain agent could have full generic access to a CRM to “create” a PO for the user OR could be required to explicitly call a “submit PO” tool which automatically triggers the proof-reading workflow before submission.

3. Set up the right level of permissioning

The moment the agent is user facing and is connected to internal/external systems, role-based permissioning should become top of mind. For each of the previously defined tool calls, the agent should act on behalf of the calling user and respect the Role Based Access Control (RBAC) policies set on the underlying system via Oauth or at the agent harness level via pre-tool call hooks to avoid any permission leaks.

4. Human-in-the-loop requirements

Once you have your tools defined and set up with the right permissions, deciding on the right “intervention threshold” is crucial. Too many human interventions can cause decision fatigue and kill the calling user’s productivity, but too little can risk irreversible actions being made, non-compliance with company and legal regulations, and lost confidence in the system as a whole.

5. Equipping sub-agents

An agent can be powerful, but give it a team to delegate to, allowing it to handle much more complex tasks and run for longer. Equipped sub-agents can either be generalist sub-agents to perform a specific delegatable subtask or hyper specific sub-agents for specific tasks like running deep research or a full review of the main agent’s work via a curated test suite following a testing framework like Test-Driven Development (TDD).

6. Integrating the agent into existing processes

Your agent might have all the functionalities in the world, but if it does not fit into core processes and doesn’t have an appropriate interface, it won’t have the expected value add. Chat interfaces, albeit intuitive interfaces for agents, are only one way of embedding an agent and rarely the most efficient to interface with an agent. Agents can also be kept running in the background as batch automation or live triggered by events like email receipts, alerts, or other notifications.

A key takeaway is that once our agent harness was built, we didn’t really need to touch it anymore. Value starts being unlocked once the agent harness actually gets configured to fit the use case. This means, giving the agent access to the relevant context, integrations and systems required for the tasks you plan to delegate to it.

Conclusion

We rely on the agent harness to bridge the gap between the model and its environment allowing it to perform meaningful and valuable tasks the same way that we rely on the computer’s OS and programming languages to bridge the gap between high level business applications and the underlying CPU and RAM.

It is looking like LLMs will soon become a commodity (if not already) thanks to the open-source community publishing open weights of very powerful models and fierce competition between the model vendors bringing the cost of models down. Agent harnesses are the next in line as agent harness architectures will start converging, driven again by the open-source community projects like OpenClaw or Langchain’s Deep Agents, industry-leading agent harnesses like Anthropic’s Claude Code, Google’s Antigravity, OpenAI’s Codex and incentives for labs to post-train models to work well with specific harness architectures.

One thing that only time will tell is, with models starting to support more and more modalities and possibly transition over to world models, would the core harness primitives we’ve built still hold or would a new harness paradigm emerge?

Either way, at least in an enterprise setting, the best use of time and effort remains on the last mile configuration of the agent harnesses as it allows a model to effectively interact and understand your one-of-a-kind environment, your data, and your processes; all in a reliable and safe way. This is possible via solutions and platforms like Dataiku that offer prebuilt agent-harnesses-as-a-service, considerably reducing technical debt and empowering developers to focus on the last mile agent configuration to translate their business and industry knowledge into an agentic system in a centralized agent control plane.