This article was originally published on the Dataiku medium publication data from the trenches.

A guide to test-time compute scaling to unlock smarter, more accurate AI without breaking the bank on retraining

For decades, the playbook in AI has been simple: Bigger is better. Got a problem? Build a bigger model. Need more accuracy? Throw more data and a few million dollars at it, retrain through backpropagation, and there you are. This relentless scaling gave us incredible tools, but we’ve started to hit a massive, expensive, and frustrating “Scaling Wall.”

Training the next generation of trillion-parameter models now costs more than a Hollywood blockbuster (we’re talking >$100 million), requires more high-quality data than we can find on the whole internet, and is often plagued by instability.

Post-training helped for some time (you know, fine-tuning the model so that it aligns with our goals, our way of thinking, etc.), but eventually hit a wall as well (see figure below). So, researchers and tech firms thought, “What if, instead of building a bigger brain, we could just teach the brains we have to think harder?”

This is the promise of Test-Time Compute (TTC), the next frontier in AI scaling that blossomed in 2025. Get ready to understand how it works (and whether you should use it or not).

What on earth is Test-Time Compute?

Let’s break it down with a simple analogy: Imagine that your AI model is a brilliant student.

• Training Time is their entire education — years of learning, reading books, and taking exams. This is where the model learns its skills. It’s incredibly expensive and time-consuming.

• Test-Time (or “inference”) is … test time! When you give that student a single, tough question, and he/she must now use their knowledge to understand the question, its structure and underlying ins and outs, and answer it in a comprehensive way.

The old way was to just hope the student’s education was good enough to blurt out the right answer instantly.

A (hopefully) better way is giving that student a piece of scratch paper and telling them, “Hey, take your time. Think about this step by step. Write down your thoughts before you give me the final answer”. This is one possible technique based on TTC scaling, usually called Chain-of-Thoughts (CoT), but there are others. We’ll delve into the details in a minute.

Before this, let’s give TTC scaling a proper definition, breaking it down term-by-term:

• Test-Time: It’s the phase where training is done, and weights of the model are frozen.

• Compute: The amount of computing power needed, usually expressed in FLOPs (floating-point operations per second). Remember: LLMs are basically huge neural networks, composed of thousands of matrices undergoing basic operations (multiplication, addition, etc.). So, in the context of LLMs at test-time, compute is directly related to the number of matrix calculations you need. As each new generated token implies a pass in the model, longer generation (more tokens) means more compute.

• Scaling: As you’d imagine, tech firms do not want to spoil money on supplementary compute if it does not lead to increased performance. So, the goal is to find techniques that let us continue riding these nice scaling laws (see figure below).

So, TTC scaling is just this: using more compute at inference to boost accuracy.

So, does it work? Short answer is … yes, and quite dramatically!

The verdict is in: TTC scaling works

The concept has been discussed for several years, initially in the context of board games and AlphaZero algorithms. However, it gained significant traction in the LLM space following two key events in late 2024 and early 2025:

• OpenAI’s o1: In September 2024, OpenAI’s release of o1 marked a major breakthrough for the reasoning paradigm.

• Deepseek’s R1: In January 2025, the open-sourcing of the Deepseek R1 model enabled wider community experimentation with these techniques.

For years, poor performance on the ARC-AGI benchmark — a test of abstract reasoning using visual puzzles, specifically designed to test raw reasoning abilities — fueled the argument that LLMs couldn’t truly “think.” The consistent failure of even advanced models like GPT-4 suggested they were merely sophisticated mimics incapable of genuine generalization.

This long-held belief was shattered by OpenAI’s “o1” and then “o3” models (see figure above). They achieved a sudden, massive jump in performance on ARC-AGI, directly challenging the perceived limits of AI reasoning and reigniting the debate about their potential for true intelligence. And guess what, the key change was the use of TTC scaling!

These models also outperformed any previous ones in a series of very challenging benchmarks (competition-level maths, Ph.D.-level science questions, etc.). As a consequence, all major LLM providers propose reasoning models (we will see in a minute what they are).Because they’ve proved so effective, we’ll now look at how these reasoning models have been trained.

Learning to reason

Training an LLM to reason goes beyond simply showing it a massive library of text (which had basically been the training paradigm until then). The process is a deliberate, multi-stage effort to teach the model how to construct logical arguments and solve problems step by step.At its core, the training relies on specialized datasets and sophisticated reward mechanisms.

• Curated datasets with explicit reasoning: The model isn't just trained on the final answers to problems; it's fine-tuned on high-quality, step-by-step annotated data. These datasets contain problems (in mathematics, logic, or code) paired with detailed, human-verified reasoning chains. This teaches the model the structure of a logical argument, not just the correlation between a question and its answer. For example, the GSM8K dataset, a popular benchmark, includes grade-school math problems with fully worked-out solutions.

• Process supervision in reinforcement learning: Standard Reinforcement Learning from Human Feedback (RLHF) might only reward a model for getting the final answer correct (outcome supervision). This is risky, as the model could learn flawed logic that happens to work for some problems. Reasoning models, however, are often trained using process supervision. In this method, a human reviewer evaluates and rewards each individual step in the model's generated reasoning chain. This is far more labor-intensive but ensures the model learns a robust and reliable reasoning process, penalizing logical fallacies even if the final answer is accidentally correct.

• Synthetic data generation and self-correction: To scale the creation of reasoning data, techniques like Self-Taught Reasoner (STaR) are employed. Here, an LLM is used to generate its own reasoning chains for a set of problems. If the final answer is correct, the generated chain is added back into the training data as a positive example. This creates a powerful self-improvement loop.

• Architectural and algorithmic enhancements: The training process is designed to support more complex inference-time algorithms. While a standard LLM generates text token-by-token in a linear fashion, reasoning models are trained to be amenable to more structured exploration. This enables techniques like Tree of Thoughts (ToT), where the model can explore multiple parallel reasoning paths (branches of a tree), evaluate their potential, and backtrack from unpromising ones. This requires the model to be proficient at generating self-evaluations and scoring intermediate "thoughts."

These are the main building blocks when training a powerful "native" thinker: At inference, the model will spontaneously produce CoT detailing its reasoning path before generating a final answer.

But keep in mind that reasoning models are just one way to leverage CoT, a specific technique based on TTC scaling. The main principle is increasing the number of generated tokens at inference: There are multiple approaches to this, and some can even be combined. We will see now a taxonomy of the main techniques.

I pulled the emphasis from the live Medium post, so the italics on structure, each individual step, and CoT are in there too. Only the four bullet lead-ins changed case; the bolded terms inside the bullets are proper names or defined terms, so they kept their caps.

The TTC toolkit: How to make your AI “Think”

TTC isn’t a single method; it’s a whole galaxy of clever techniques. Most of them fall into three main families, but keep in mind that this is not exhaustive.

1. CoT: The “show your work” method

This is the most popular TTC technique and also the way it emerged historically. Instead of letting the model jump straight to an answer, you prompt it to generate a “Chain of thought” — a series of reasoning steps that lead to the solution.

It can be done by giving it examples in your prompt (in-context learning) or by simply adding a magic phrase like “Let’s think step by step.”

This is at the core of what we call reasoning models, including OpenAI’s “o” series: models that have been specially trained on thousands of CoT, to replicate this behavior. To the point that, at inference, they resort automatically to this drafting: They “think” natively before answering, without the need to nudge them.

2. Sampling: The “phone a friend … or 50” method

Why get one opinion when you can get many? Sampling techniques involve asking the model the same question multiple times to generate several different answers, which are then combined to find the best one.

• Self-Consistency: You generate, say, 10 answers and pick the one that appears most often. It’s a simple majority vote that dramatically boosts reliability.

• Best-of-N: You generate N answers and then use a second, smaller model (a “verifier” or “reward model” trained specifically for this task) to score each one and pick the winner.

• Beam Search: A most advanced way to navigate into the tree of thoughts (where branches are feeds of successive thoughts) while only keeping most promising leads.

3. Criticism & feedback: the “Draft, revise, repeat” method

This is my personal favorite. It turns the generation process into an iterative loop of refinement.

1. The LLM generates an initial draft of the answer.

2. The LLM (or another model) critiques that draft, pointing out flaws or areas for improvement.

3. The LLM takes the feedback and generates a new, better version.

You can loop this process until the answer is perfect, or you’ve hit your budget limit. A variation is using a powerful “teacher” LLM to guide a smaller, cheaper “student” model, correcting its reasoning only when it starts to go off track.

Now let’s talk about when to use these techniques.

To TTC or not to TTC?

TTC techniques are more suited for certain tasks and are unlikely to help for others. Choosing the right technique is all about diagnosing the type of difficulty you’re facing, as each family aims at solving a different kind of failure. Here are basic heuristics:

CoT-based techniques, including the use of reasoning models, are adapted for tasks within the model’s known capabilities but with intricate and multiple steps. Here’s a simple rule of thumb I call The Human Expert Test.

Ask yourself: If I gave a human expert all the necessary information to perform a task, could they solve this task instantly, on the fly?

• YES? (e.g., live translation, answering a simple factual question, writing a basic code snippet). Then TTC probably won’t help much. If your LLM is failing, the problem is likely somewhere else — it probably doesn’t have the right information.

• NO? (e.g., summarizing a messy 50-page document, solving a multi-step math problem, planning a complex project). If the expert needs to pause, think, and organize their thoughts, that’s a perfect use case for TTC! Letting the LLM “think” will allow it to break down the problem and deliver a much more accurate result.

When the model is generally capable of a task but suffers from a lack of consistency, sometimes succeeding, other times hallucinating facts or getting stuck in a suboptimal reasoning path, Sampling techniques may help.

Finally, feedback can be used for novel tasks, likely falling outside the model’s training distribution. If it involves novel concepts, or requires adhering to very strict and unusual constraints, then checks and corrections can improve your results.

With all that said, a question remains: Should you DIY your own TTC techniques? The answer is unclear and depends greatly on your level of expertise, the task you are working on, the model you use, the budget you have, etc.

Keep in mind that the most powerful, mind-bending reasoning techniques are already being baked directly into the flagship models from the big leagues. They’re serving you five-star models, and they are getting better with every update.

So you could spend days or weeks of your time (and money) building a brilliant custom “thinking” engine for your AI, just in time for the next big model to be released, offering even better reasoning right out of the box and turning your masterpiece into a museum piece overnight. Furthermore, LLMs training processes are notoriously hard to predict, and the clever trick that boosts your accuracy today could become completely useless when the model’s mood shifts tomorrow.

Another point is that generation is not that often the real bottleneck in business use cases, not so much as data quality and availability. Most benchmarks build on very clean environments where models have access to every information they need, under the right format, in controlled settings, etc. This is not always (never?) the case in real life! So before any expensive experimentation, it is worth checking where the real bottleneck of your system lies.

Conclusion

So, did we go through all of this just to learn that the best move is to … wait for the next model? Was it even worth it?

Well, first, there are still a lot of cases where TTC scaling can be applied successfully, so it might be worth trying depending on factors we highlighted below.

But more importantly, TTC isn’t just a temporary hack; it’s part of the new playbook for cutting-edge AI. Understanding how it works is your ticket to keeping up with the next wave of tech breakthroughs. Even if you let the big providers handle the complex implementation, knowing the blueprint is how you’ll be ready for what comes next.

Finally, it is a massive leap forward, but it is not a silver bullet. Models with reasoning abilities still struggle with things like physical common sense, recognizing when a task is impossible, or adapting to new procedures on the fly. So, having a feeling of how all this works let you use these models accordingly, understanding their strengths and weaknesses.

In the age of AI, the ones who truly understand the technology are most probably the ones who will lead the race.

Sources

All three illustrations without indicated source are from A Visual Guide to Reasoning LLMs. Maarten Grootendorst, 2025.

Additional resources:

• What is test-time compute and how to scale it? Ksenia Se, Alyona Vert, 2024.

• Scaling test-time compute with open models. Edward Beeching, Lewis Tunstall, Sasha Rush, 2024.

• Learning to reason with LLMs. OpenAi team, 2024.

• A Visual Guide to Reasoning LLMs. Maarten Grootendorst, 2025.

• Why We Think. Lilian Weng, 2025.