This article was originally published on the Dataiku medium publication data from the trenches.

The future of AI is about empowering agents with the right tools for the job

It all started with a business problem so painful that it has haunted office workers for decades: responding to Requests for Proposal (RFPs) and Requests for Information (RFIs). These massive documents are a gauntlet of complex, repetitive, and interconnected questions.

A single question might require you to dig up info from three different internal documents, and the question on page 20 is probably a follow-up to one you answered back on page five. Working on this problem using an agentic RAG framework, I stumbled upon some hard-won lessons and insights that I believe can apply to a whole lot of RAG systems out there.

What exactly makes this task difficult?

Automating RFP and RFI completion means that your solution can handle these two main challenges:

• First, questions may require finding and combining information from multiple, separate knowledge sources to form a single answer. But just that would be playing on easy mode! What's more, each retrieval often depends on previous parts of the answer, breaking any hope for parallel processing. This process of performing a series of intricate reasoning "hops" between documents to synthesize a complete response from scattered evidence is what's known as a Multi-Hop challenge.

• Second, a later question might be a direct follow-up to a much earlier answer. This makes it necessary for the agent to maintain a conversational “thread” to handle the dependencies between questions, and breaks question answering parallelism. It is referred to as Multi-Turn questions (as this feed can be seen as a conversation between a user asking questions and an assistant answering, each speaking in turns)

Typical instance of multi-hop question: The agent must first find the information about a GDPR solution in one data source, before searching for the date of last internal audit on it, very likely on another source.

Now, naive RAGs (1 retrieval + 1 generation) obviously cannot handle it, because their rigid, single-pass design lacks the necessary dynamic reasoning and memory capabilities. So, how can we do this?

Assembly line vs. master chef: a tale of two AIs

When you build an advanced RAG system, you're faced with a fundamental choice in its design philosophy, to define how it will reason and interact with data. This choice boils down to two competing paradigms: workflow vs. agentic.

1. Workflow: the assembly line

Think of this as the classic, industrial approach. The system is a rigid assembly line with a fixed, hard-coded sequence of steps, functioning as a predefined system. The AI is a highly-skilled worker on this line, performing its specific task perfectly, but it has no say in the overall process. The path is always the same, for instance: first, rewrite the query; second, retrieve the documents; third, re-rank them; fourth, generate the answer. No deviations allowed.

This is the "System 1" of AI: controllable, tunable, interpretable, but completely inflexible. In the present case, you would typically include:

• A question decomposer that breaks the complex initial query into a fixed sequence of simpler sub-questions.

• A query builder that rephrases each sub-question, often by integrating answers from previous turns to handle the multi-turn context.

• A retriever called sequentially, that execute in a predetermined order to fetch the necessary information for each sub-question.

• One or several evaluators, at various stages, to check that the model's on track.

• A memory handler, so that you save past answers, to be used in later questions.

• A synthesizer, a final module that takes all the separately retrieved pieces of information and combines them into a single, coherent answer.

And these are just the required blocks, nothing fancy yet…

2. Agentic system: the master chef

This is where things get exciting. In the agentic paradigm, the AI is no longer a worker on the line; it's the master chef running the kitchen.

The agent has complete autonomy. It looks at the user's request, assesses the available information and tools, and dynamically creates its own recipe for solving the problem. It decides if it needs a tool, which tool to use from its toolkit, and in what order. It might start with a web search, then decide based on the results that it needs to query an internal database, all while "thinking" and adapting its plan on the fly.

This is the "System 2" of AI: deliberate, flexible, and powerful enough to tackle the messy, unpredictable problems of the real world. However, you must trust the AI ability to effectively handle those tools and reason efficiently. Expecting a master chef but getting a chaotic kitchen hand? That's bad news for your restaurant…

So, which paradigm should you use? Luckily for you, I tested them myself, and here are the results.

Cages vs. playgrounds: choosing how your AI thinks

We have our two competing philosophies. Here's what I did: I tested several instances of each kind, including the modular frameworks in the predefined setting, and different ReAct modules for the agentic paradigm. More specifically, I replicated the ComposeRAG architecture from this recent paper, an approach noted for its strong performance in business settings on this kind of Multi-Hop challenges, and then benchmarked it against several relevant datasets.

Then, I compared it with a ReAct agent equipped with a tool to access the same database. I won't delve into anymore details, but rather stick to results and general conclusions that I think are safe to draw at this point.

First: Original figure from the ComposeRAG paper.

Second: Generic ReAct architecture from Daily Dose of Data Science.

The assembly line on trial

As you've probably already guessed, I am not a big fan of this paradigm. The workflow approach is fundamentally rooted in a mistrust of the model's own capabilities. It operates like a micromanager, forcing the AI down a strict, unchangeable path because it doesn't trust it to figure out the best steps on its own. This philosophy leads to these critical flaws:

• It is inherently inefficient. This paradigm forces every problem, regardless of its complexity, through the same rigid, multi-step pipeline. This "one-size-fits-all" approach prevents the model from adapting its effort, often wasting significant computational resources on simple tasks, and lacking budget for the hardest ones. Sure, you can try to multiply the pipes, discriminating between cases, but then…

• It requires significant upfront engineering. Designing a robust, fixed pipeline involves meticulously planning a complex web of inter-module dependencies to prevent errors and optimize cooperation. Thus, complexity (hence time) grows exponentially with the number of blocks. Building this sophisticated support structure to guide the model is a time-consuming process that may become obsolete as models evolve.

• It cannot adapt easily through time. Like an assembly line built for one specific product, it lacks the flexibility to adapt to small changes in the goal or nature of the task over time. You have the right to mistrust AI's ability to reason, but then you ought to do it yourself, and anticipate each and every corner case that could pop up. Otherwise, you may be forcing your system to put a square peg in a round hole. And if you have been engineering solutions for some time, you know that unexpected cases are bound to happen. Even worse, a single unexpected event can cause the entire pipeline to fail but take some time to debug, retracing all generations, from all modules…

The agentic approach takes the stage

This approach is the exact opposite. It's built on trusting the model's intelligence and giving it the autonomy to be a maverick. Give the agent its goals, the tools it can use, and let him handle the rest.

• It simplifies development and debugging. Instead of engineering a complex, rigid pipeline with numerous inter-dependencies, the developer's role shifts to providing the AI with a clear set of tools. The agent itself handles the complex task of planning and orchestration, which can make the underlying code far simpler to write and maintain.

• It leverages the model's superior decision-making. AI (with all due respect) is already smarter than you when it comes to tool handling, which often leads to more effective and efficient solutions than a human can predict and hard-code, as the model can adapt its strategy in real-time.

• It's a future-proof bet. Building agentic systems aligns with the primary focus of major AI labs, as the latest and future models are being explicitly trained for autonomous tool use.

A few conclusions

Ultimately, this choice between the Micromanager and the Maverick is the very heart of what I call the Crutch Dilemma. A crutch is an indispensable tool for someone with a broken leg, providing the necessary support to overcome a temporary weakness. However, once the leg has healed and recovered its strength, continuing to rely on it becomes counterproductive. It actively hinders natural movement and slows the person down.

Similarly, many of the complex, engineered solutions we build to guide LLMs are essentially "cognitive crutches". They are sophisticated systems designed to compensate for the specific, temporary weaknesses of a particular model generation.

The dilemma arises because these models are "healing" at an astonishing rate. The carefully designed crutch that helps today's model may become a liability just a few months later. This creates a difficult choice: Is it worth investing significant effort to build a complex support structure for a problem that a simple model update might solve organically in the near future?

Yes, that is ridiculous. But are you sure that you aren't forcing your agent to run with crutches? Source:theRandomAdventureMan Youtube channel.

Back to our example: I was working on these frameworks based on OpenAI's o4-mini and o3 when gpt-5 was released in August. So this was the perfect test to evaluate how a more powerful model would perform. Guess what: When placed into a workflow, the new model's performance stagnated and sometimes even degraded, whereas the agentic framework allowed gpt-5 to flourish, showing massive performance gains on top of its already strong baseline.

So, when it comes to the future of AI, will you bet on control or autonomy?

The elephant in the room: it's always been about the data

We've spent a lot of time looking at the AI's "brain," but lurking behind every success and failure is the real kingmaker of any RAG system: the data itself!

The true bottleneck

Let's be clear: In the world of RAG, the bottleneck has almost always been the data. The most common reason these systems fail is not that the AI misunderstands the information it's given, but that the retrieval system failed to provide the right information in the first place. The principle of "garbage in, garbage out" is paramount. If your data handling is flawed, even the most brilliant AI agent is doomed to fail. So was it worth discussing on reasoning frameworks if the bottleneck is elsewhere? Well, everything's linked.

A new paradigm: from retrieval to navigation

How your knowledge base is structured and accessed is the most critical question. The traditional approach is to take structured documents and flatten them into a collection of unstructured text chunks. This is a massive problem because it severs the inherent relationships between sections, destroying the document's original, logical structure.

Think about how a human researcher works. They don't search through a million disconnected sentences. They navigate hierarchically: skimming a table of contents, identifying relevant chapters, scanning section headings, and only then reading a specific passage. This navigational paradigm proposes giving AI agents a similar set of tools to traverse the hierarchical structure of a knowledge base.

This is where everything comes together. This advanced, navigational approach to data only works symbiotically with an agentic system. A workflow, simply cannot perform this kind of dynamic exploration natively. It can't decide to "skim a table of contents" and then "read a specific section" based on what it finds, unless it relies on numerous, complex and brittle modules that will not fit well in this setting.

An agentic system, however, is perfectly suited for this, because it was created for this, based on an LLM trained to do so. Its ability to autonomously choose which tool to use and what to do next is exactly what's needed to intelligently navigate a complex document structure, just like a human would.

The future of data handling doesn't just prefer the agentic approach; it demands it.

Your takeaway: stop building pipelines, start empowering agents

The future of advanced, useful AI lies not in forcing models into rigid pipelines, but in empowering them with the autonomy, and ad-hoc tools to navigate problems on their own terms.

So if you're building a system to tackle complex Q&A, the recommendations are simple:

1. Structure your data for an AI system. Make your knowledge base easy for an agent to interact with, ideally through a well-documented API.

2. Give it the right tools. Provide the agent with simple, clear functions to access that data.

In the age of AI, the ones who truly understand the technology are the ones who will lead the race. And that means understanding that the best systems don't constrain AI — they set it free.