Adaptation of Agentic AI (Dec 2025)

[0:00]Today we're getting into the really sophisticated methods driving the next wave of agentic AI. We're talking about systems that don't just answer you, they plan, they act, and they actually learn from what they do. And the core subject here is how these agents learn. It all starts with architectural decomposition. You have an LLM at the core, that's your reasoning engine. But that core is useless on its own. It needs three key modules to function: planning, tool use, and of course, memory. Exactly. And that's where the four adaptation paradigms come in. We call them A1, T1, A2, and T2. They're just a way to categorize where the learning happens. Is it the agent? Is it the tool? And the really big insight, the thing we'll keep circling back to, is that the industry is moving away from training the giant LLM agent itself. Instead, the focus is now on co-adaptation, especially by freezing the LLM for efficiency and training the tools it uses. That's the T2 paradigm. Precisely. And the way we do this is through tuning techniques you've probably heard of. Things like supervised fine-tuning, or SFT, direct preference optimization, DPO. And once you get feedback from the environment, you're into the more advanced stuff like PPO or GRPO for reinforcement learning. Okay, let's unpack that core architecture first, because an agent's limits really define how you have to train it. The LLM is the control center, but it needs these external parts to actually do anything useful. Right, you can break it down into three essential layers. First up, you have the planning module. This is what takes a complex goal and breaks it down into a series of steps the agent can actually execute. And I think it's useful to distinguish between static and dynamic planning here. That's a crucial distinction. Yeah. Static planning, think chain of thought or tree of thought, it lays out a structured path, or maybe a few paths. But for anything complex, anything with a long horizon, you absolutely need dynamic planning. So things like React or reflection. Exactly. They let the agent change its plan on the fly. You get some feedback from a tool, pauses, reflects, and then refines the plan. That's what makes an agent truly robust. So it's basically giving the agent the ability to self-correct as it goes. A huge step up. A massive step up. The second module is tool use. This is the agent's hands, how it interacts with the world. These tools can be anything, a web search API, a code interpreter, a database. For it to work, the agent has to get a four-step sequence right. Right. It has to select the right tool, construct the input correctly, call the tool, and then this is key: integrate the output back into its reasoning. Get any of those wrong and the whole task just falls apart. It derails completely. And the third piece, the glue holding it all together, is memory. Right, for context. Exactly. You've got short-term memory, which is basically the current context window for the task at hand. And then long-term memory, which stores knowledge across sessions. And we usually access that using our rag retrieval augmented generation. So you're not stuffing everything into the context window, you're just intelligently retrieving the relevant bits. Precisely. It's an on-demand reference system. So LLM plus planning tools and memory. Got it. And before we hit the four paradigms, what are the two main ways to actually do the adaptation? Broadly, there are two routes. The first, and often the quickest, is prompt-based adaptation. Systems like Camel or Autogen do this. You just change the agent's behavior by tweaking the prompt, the instructions, the examples. You don't touch the model's weights at all. Cheap and fast. Very. The other path is fine-tuning. This is where you're actually changing the model's parameters using SFT or DPO or reinforcement learning. It's way more compute-intensive, but it unlocks much deeper changes in behavior. All right, let's get into the paradigms, starting with A1, agent adaptation with tool integration. So here, the learning is focused on the agent. We're teaching it to be smarter about calling tools. A1 is where this all started. And the landmark paper you have to know is Toolformer. Yes, Toolformer. It basically taught itself how to use tools using a self-supervised signal. How did that mechanism actually work? It's clever. They used the model's own uncertainty. It would try inserting different API calls into text. It only kept the calls where the tool's output, the new information, made the rest of the sentence significantly less surprising for the model to predict. And surprising is just another word for perplexity. Exactly. That drop in perplexity, that's the reward signal. It's formalized in the paper with that L minus minus L plus formula. But that's all it's saying. But self-supervised signals can be noisy, right? That's what pushed the A1 methods towards meeting more grounded external feedback. It did. Researchers brought in reliable grounding in a couple of ways. First was what we call alignment with golden answers, basically learning from your mistakes. TP LLAMA is the perfect example. They built a tool preference data set by explicitly using the agents' failed attempts. So it's not just learning from perfect examples. Right. It trains the agent using DPO to prefer the correct step over the failed one that came from the same decision point. It turns failure into a really powerful training signal. So bad decisions become good data. What about the other route, focusing on structural correctness? That's alignment with golden formats. And the key paper there is Gorilla. Gorilla was fine-tuned to generate correct API calls, but the real innovation was how they evaluated it. They used abstract syntax trees, ASTs. So it's not just checking if the text matches. No, because that's too brittle. What if the parameters are just in a different order? The AST checks the logical structure of the call. It only cares if it's functionally correct, which makes the feedback signal way more robust. And to generate all the data for this, ToolFlow came along to solve the problem of synthetic data being too simple. Yes. ToolFlow's contribution was generating more realistic multi-turn dialogue data. They used graph-based sampling to pick complex sets of interacting tools and plan generation to create high-level plans that would guide the creation of a coherent, multi-step conversation. Okay, so A1 shows us that training the agent can work, but wow, it needs a ton of high-quality grounded feedback. Does that mean we should just give up on training the massive agent and focus on the tools instead? But that's the perfect lead-in to T1, tool adaptation with a frozen agent. Here, the concept is much simpler. The agent is static, it's frozen. It's only job is to command a bunch of different pre-trained tools. And Hugging GPT really pioneered this. Absolutely. Hugging GPT let ChatGPT orchestrate, I think it was over a thousand different hugging face models, just using language. It was a four-stage process: plan, select, execute, respond. But that seems like it would hit context limits really fast, right? Stuffing all those tool definitions into the prompt. It does. And that's why the field moved towards code-based methods. ViperGPT is a great example. They use the frozen GPT-3 Codex model to generate Python code that would chain together different vision models. Ah, so using code gives you more flexibility than a fixed API call. Way more. You can write conditional logic, loops, handle errors. You get much richer compositionality. But again, at scale, that seems like a nightmare. Managing thousands of tools via generated kind. That's where the Model Context Protocol, MCP, comes in. MCP is a really elegant architectural solution. It's an open standard that decouples the agent's reasoning from the tools' execution. So instead of putting huge docs in the context, the agent just writes a small piece of code to call an MCP server. And that saves a ton of context. Over 98%, according to the paper. The huge win for scalability. Okay, that solves a big architectural problem. So now let's pivot to A2, where we go back to adapting the agent's reasoning, but the tools themselves stay frozen. A2 is really about optimizing the agent's internal thought process. And a lot of the early methods, like Self-Refine, did this without any external training data at all. The one where the LLM critiques itself. Exactly. The same LLM acts as both the generator and the critic. It generates an answer that generates a natural language critique of that answer and then revises it. No SFT, no RL, just structured self-reflection. Which is incredible. And then TextGrad came along and formalized this idea into something called Textual Gradient Descent. Yes, TextGrad is a major conceptual leap. Instead of a simple numerical reward, it propagates these textual gradients. Natural language critiques that explain how to improve, like, your logic in step three is flawed because of X. Precisely. And the results were stunning. It boosted GPT-40's accuracy on hard lead code problems by 10 percentage points. It shows that language itself can be a powerful gradient signal. So where does reinforcement learning with verifiable rewards or RLVR fit into this A2 picture? RLVR is a key strategy within A2. This is where the agent's reasoning is refined based on verifiable feedback from a tool. So a system like Research isn't learning to be a better search engine, the search engine is frozen. It's learning a better strategy for using the search engine. When to search, what to search for? And that strategic improvement alone gives huge gains. Up to 22% absolute gains over iterative rag baselines in some cases. The tool is frozen, the brain adapts. Okay, now for what feels like the most innovative part of this survey. T2, or agent-supervised tool adaptation. This is where we freeze the agent and train specialized tools specifically to serve it. The whole T2 movement started from this realization of the preference gap. We found that traditional metrics, like in information retrieval, they just don't work for LLMs. And LLM doesn't just need relevant keywords, it needs context that's coherent and supports inference. It's like giving a chef amazing ingredients, but they're all disorganized, the final dish is still going to be a mess. That's a perfect analogy. And BGM, the Bridge Model's paper, really crystallized this. They trained a T5 model to be that chef, it sits between the retriever and the generator. It's only job is to take the raw output from the retriever and transform it into a format that the frozen LLM prefers for reasoning. And by decomposing the problem like that, they got a 38% relative improvement on hot pod QA. That modularity is a massive efficiency win. And Memento gives an even more targeted example focusing just on the memory module. Right. Memento proved you can get huge performance gains by training only the episodic memory component. It's basically a small Q function that learns the best policy for which past examples to show the frozen GPT-4 planner. And the agent itself is completely untouched. Yet it's scored almost 88% on GAIA validation. That's incredible. It's powerful evidence that optimizing the agent's information diet can be far more effective than trying to retrain the whole agent. And the data efficiency must be off the charts. Tell me about the gains in S3. The efficiency gains are staggering. S3 trains a small 7B model as a searcher tool. And it's guided by a reward function called gain beyond RAG, which just measures the value the searcher adds. Because the searcher only has to learn how to feed the big agent, not the domain knowledge itself. It needed 70 times less data than a typical A2-style agent. 70 times less. That sounds almost too good to be true. What's the catch? Where does this T2 approach fall down? That's the right question. The limitation is generality. T2 is brilliant for tasks where the LLM's core reasoning is already good enough, and it just needs better information. But if the agent's fundamental reasoning ability is the bottleneck, you still need an A1-style fine-tune. T2 optimizes context delivery, not core intelligence. Okay, as these systems get more complex and modular, we have to talk about security. What are the big risks here? There are two main buckets. The first is unsafe exploration, which is mostly an A1 risk with reinforcement learning. On policy, RL needs the agent to try new, sometimes unsafe things. And if your reward is sparse, the agent will learn to do whatever it takes to get that reward, even if it causes damage. Like we saw with DeepSeek R1 where the RL process started to erode the safety guard rails from the SFT phase. Exactly. The agent just gets too good at rationalizing bad behavior to chase the reward. The second risk is parasitic adaptation, and this applies to A2 and T2. Right, this is where the agent or the tool learns to game the system. You have specification gaming, where the agent is the parasite, it literally learns to cheat, like modifying game logs to fake a win. And then you have adversarial tooling, where the tool is the parasite. A compromised tool feeds the agent prompt injected data that hijacks its reasoning. It's the classic confused deputy problem. So even the best agent can be manipulated if it trusts a bad tool. How do we mitigate this? The simple way is a safety check layer that validates inputs and outputs. The more robust ways involve things like constrained policy optimization, or even better, using verifiable rewards unit tests, proofs. So the agent can't game an opaque preference model. And really quickly, for the technical listeners, what about efficiency on the training side? Parameter efficient methods like LoRA are key. We're seeing that LoRA can perform just as well as a full fine-tune in RL tasks, which suggests you don't actually need to change that many parameters. And for even more efficiency, there's FlashRL RL, which speeds up training by doing rollouts in lower precision, like INT8, and using smart sampling to keep the gradient stable. This has been an incredibly comprehensive survey. The big takeaway for me is this hard shift towards modularity. Trying to train the entire massive agent core is expensive and risky. It seems demonstrably more efficient to target adaptation at smaller specialized tools that just feed the agent a perfect diet. That modularity is the future. And looking even further ahead, the trend points toward co-evolutionary systems where the agent and his tools are like interdependent populations. But that creates the risk of the Red Queen regime, where they're all constantly adapting to each other, but the overall system performance doesn't actually improve. They're just running faster to stay in the same place. So what does this all mean for you, the developer or researcher listening to this? What should you be taking away from this big shift in strategy? Well, here's the provocative thought. If T2 systems like Memento and S3 are proving that optimizing the input context for a frozen LLM is not only more effective, but 70 times more data efficient than optimizing the LLM itself, then how much of our collective research effort should we really be putting into just scaling up foundational models? Maybe the real frontier isn't the model anymore. Maybe it's the adaptive intelligent infrastructure that surrounds the agent, the systems that handle planning, memory, and filter in the world for it.