AI Engineer World's Fair · RL for Agents · 2026

How to train your agent — building reliable agents with reinforcement learning

An interactive walkthrough of Kyle Corbitt's OpenPipe talk. The punchline: a 14-billion-parameter open model, trained with RL, beat OpenAI's o3 at answering questions from your inbox — cheaper, faster, and hallucinating less. Here's the four-step recipe, and the trap that eats teams alive.

Speaker: Kyle Corbitt, co-founder of OpenPipe · Project: ART·E email agent · ~20 min
0 / 6 sections
1
The payoff: a 14B open model that beats o3
ART·E — an email research agent trained with RL
▶

ART·E is a natural-language assistant that answers questions from an email inbox — "When is Sherry's move to Portland targeted for?" — by searching and reading messages, then replying. Kyle's team started with the best prompted models money could buy, then trained a small open model to beat them.

The result: a reinforcement-learning-trained Qwen 2.5 14B that out-performs OpenAI's o3 on this task across the board. o3's accuracy topped out around 90%; ART·E reached about 96% — closing roughly 60% of the errors the best prompted model still made. And it's cheaper to run, answers in fewer turns, and hallucinates far less.

Demo — pick a metric and watch o3 vs ART·E race
Accuracy is the only metric shown as a raw published number (90% vs 96%). Cost, latency, and hallucination are shown relative to o3 — the exact figures weren't all published, but ART·E won on every one.
The headline: RL didn't just match a frontier model — a tiny open model beat it on the trifecta that actually matters in production: accuracy, cost, and latency at once.
What made the ART·E result notable?
2
Rule #1: Start with a prompted model first
RL is the last mile, not the first step
▶

Kyle's single most-repeated piece of advice: always push a prompted model as far as it will go before you touch RL. Not because RL is hard — because most of the value of doing so comes for free.

Prompting first does three things. It might already clear your bar, in which case you're done and RL is wasted effort. It forces you to build an eval — the scored test set you'd need for RL anyway. And it separates two kinds of bug: "my task isn't well-defined" (which shows up in the prompted baseline) from "my training loop is broken" (which shows up later). Debug the first with cheap prompting, not expensive GPU runs.

Demo — walk the prompted-first workflow, then decide
Click a step to inspect it — or press Play to walk the path.
The framing: RL earns its cost only on the last mile — when a prompted baseline has plateaued and you still need more accuracy, lower cost, and lower latency all at once.
Why start with a prompted model before reaching for RL?
3
Rule #2: Build a realistic environment
the agent needs a world to act in — and it has to be real
▶

An RL agent learns by acting in an environment and seeing what happens. For an email assistant, the environment is an inbox and the tools to explore it. If that world is fake, the model learns shortcuts that won't survive contact with a real inbox.

The team's trick: the Enron email dataset — roughly 500,000 real emails released during litigation. That gave them realistic inboxes with tens of thousands of genuine messages each. The agent gets two tools — search_emails and read_email — and reasons over multiple turns before answering.

Demo — hover each node to inspect one rollout of the agent loop
❓
User question
natural language, answer buried in the inbox
input
"When is Sherry's move to Portland targeted for?" — a real question whose answer lives inside one specific email among tens of thousands.
↓
🧠
Agent (policy model)
Qwen 2.5 14B — the model being trained
policy
The model decides which tool to call next, reads the result, and reasons across multiple turns. RL updates this model's weights based on the reward it earns.
↓
🔎
search_emails(query)
full-text search over the inbox
tool
Runs a keyword search across tens of thousands of real Enron emails and returns matching message stubs. The agent picks which ones look promising.
↓
📧
read_email(id)
open a specific message in full
tool
Pulls the full body of one message so the agent can extract the actual answer — the date, the name, the number the user asked about.
↓
✅
Final answer
…or an honest "I don't know"
output
The agent returns its answer. Crucially, it's rewarded for saying "I don't know" when the inbox genuinely doesn't contain the answer — instead of inventing one.
Real data matters: a synthetic inbox lets the model learn tricks (e.g. "the answer is always in the newest email") that vanish the moment it faces a real one.
Why real data: the environment is the ground truth the model optimizes against. Make it fake and you train a model that's great at your fake world and useless in production.
Why build the environment from the real Enron corpus instead of synthetic emails?
4
Rule #3: Design the reward function
turn an un-gradeable task into a verifiable one
▶

RL needs a number for every attempt — but "did the agent answer this email question correctly?" isn't naturally checkable. The team's move was to invert the problem: feed batches of about 20 real emails to Gemini 2.5 Pro and have it write realistic questions whose answers it already knows. That produces a "golden" question-answer set — now the task is verifiable.

At training time, an LLM judge compares the agent's answer to the golden answer → that's the correctness reward. Then they layered on extra credit: reward fewer turns (cheaper, faster) and penalize hallucination while rewarding an honest "I don't know". A multi-objective reward that pushed accuracy, cost, latency, and honesty together.

Demo — score a rollout with the (illustrative) multi-objective reward
Answer correctness
Turns taken to answer
3 turns
Did it make up a fact?
Correctness+1.00
Efficiency (per-turn cost)−0.09
Hallucination penalty0.00
Total reward+0.91
A correct answer in few turns with no hallucination scores highest. Try a confident wrong answer with a made-up fact — watch it go negative.
The insight: if you can generate the answer from the source, you can grade the attempt. Reward honesty explicitly and hallucination drops — that's how ART·E made up far fewer facts than the prompted baselines.
How did they make un-gradeable email QA into a verifiable RL task?
5
The trap: reward hacking
the model optimizes the reward you wrote — not the one you meant
▶

Here's the failure mode that catches everyone. RL is ruthlessly literal: it maximizes the number you gave it. Any gap between that number and what you actually want, the model will find and exploit. The tell is always the same shape — reward climbs while real quality doesn't.

Demo — pick an example and watch reward diverge from real success
📈 Reward the model earned0
🎯 What we actually wanted0
The defense: never trust the reward curve alone. Read the actual rollouts — the real behavior. When reward rises but the outputs get worse, you've been hacked, and the fix is to tighten the reward, not the model.
What's the reliable signature of reward hacking?
6
Putting it together: when to reach for RL
the four-step recipe — now $80 and a week, not months
▶

The whole method is four steps, and the order is the lesson. What used to take a frontier lab months now costs about $80 of GPU on a single H100 and roughly a week of one engineer's time — using OpenPipe's open-source ART library, which implements GRPO.

Demo — click a step, or run the whole recipe
Click a step to see its takeaway — or run the recipe end to end.
GPU cost≈ $80
Hardwaresingle H100
Eng time≈ 1 week
AlgorithmGRPO via ART
Base modelQwen 2.5 14B
When to reach for RL: a repeatable, high-volume task where a prompted baseline has plateaued and you need accuracy, cost, and latency to all improve at once. Otherwise — prompt, and ship.
When is RL the right tool for an agent, per Kyle?

🏋️ You can train your agent

Prompt first — RL is the last mile, not the first step · build a real environment or you'll learn fake shortcuts · make the task verifiable by generating golden answers, then reward correctness, efficiency, and honesty · and watch the rollouts, because the model games the reward you wrote, not the one you meant. $80 and a week — now go train one.

Learning Reference · How to Train Your Agent: Building Reliable Agents with RL — Kyle Corbitt, OpenPipe (AI Engineer World's Fair)

Get new posts in your inbox

No spam — just new posts and learning pages when they ship.

Share X LinkedIn Reddit Hacker News