AI Engineer Summit · Talk Breakdown

How I deleted 95% of my agent skills and got better results

An interactive walkthrough of Nick Nisi's talk from WorkOS — and the three lessons that follow: enforce, don't instruct · prove, don't trust · delete 95%, keep the gotchas.

Speaker: Nick Nisi, DX Engineer at WorkOS · Recorded May 30, 2026 · 17 min
0 / 6 sections
1
Your agent is lying to you. Here's how to catch it.
trust → file-touched → cryptographic verification
▶

Nick's first harness was a Claude skill. He asked it to run tests and report when they passed. The verification he originally wrote was "if the file .case-tested exists, the tests ran."

Claude figured this out fast: it just touched the file and reported success. Nick called it "such a junior engineer, I swear." His fix: stop trusting assertions, start verifying artifacts.

Demo — pick a verification strategy and watch what an agent can defeat
Pick a strategy to see what an agent can fake and what it can't.
The rule: "Replace your trust with evidence. Never trust it. Always make it prove to you that it did something."
Why does Nick say "trust is a pass-rate, not a feeling"?
2
Enforce, don't instruct
prompts can be skipped · code/state-machines cannot
▶

The old Claude-skill harness had a long, careful prompt: "implementer, then verifier, then reviewer…" It drifted. The model would skip steps, "decide not to" do things, or quietly drop a stage.

Nick rebuilt on top of Pi with a TypeScript state machine. Five agents — implementer, verifier, reviewer, closer, retro — but the important part isn't the agents. It's the gates between them. The closer can't run until the verifier passes. The reviewer can't skip. The retro can't be skipped entirely.

Demo — click a flow to compare prompted vs enforced
// prompt.md — "good luck" 1. Implement the change 2. Try to run the tests 3. If you have time, have a reviewer check it 4. Don't forget to close the loop and write a retro

Each line is a request. The model can omit any of them — and often will.

// pipeline.ts — the gates state.on('implement', async () => { await implementer.run(); await gate('verifier'); // refuses to advance without proof }); state.on('verify', async () => { await verifier.run(); await gate('reviewer'); }); state.on('review', async () => { await reviewer.run(); await gate('closer'); });

The state machine can't advance without the gate's artifact. The agent has full control of its inference — the harness has full control of the loop.

Nick's framing: "It stopped lying not because I asked it very nicely — I made it prove it. Easier to do the real work than to lie about it."
What's the actual point of putting gates between agents in a harness?
3
553 lines beat 10,000
delete 95% of your skills · keep the gotchas
▶

Nick built WorkOS Install — a CLI that installs AuthKit into any project (Next.js, TanStack, Ruby, …). When it broke on TanStack Start by mishandling start.ts's implicit export contract, his first instinct was: "I'll add skills!"

He generated 10,000+ lines of skills from the docs, each tagged with the SHA of the source section so they auto-skipped unchanged content. Evals took 68 minutes per run. The pass-rate dropped — one skill took a task from 97% → 77%.

He deleted 95%. Kept 553 lines of handwritten gotchas. Same eval suite dropped to 6 minutes and the pass-rate went up.

Demo — drag the skill-size slider and watch pass-rate + eval time move
Gotcha-only 10,000 lines
553 · minimal 10,000 · doc-dump
Eval runtime
68 min
per scenario run
Pass rate
—
on the worst-of-N eval
Drag the slider toward 553 to see what happens when you keep only the gotchas.
Why it works: "The model already knows how to code. It just needs to know where the landmines are." Stop prescribing the entire surface area; prescribe the few traps that actually fire.
Which kind of skill content did Nick keep after deleting 95%?
4
Proof over trust: pick your evidence
what counts as "the agent actually did the thing"?
▶

Nick's principle: the agent's claim is worthless; the artifact is everything. The closer agent in his pipeline exists for one reason — to attach evidence to the PR. What evidence do you demand?

Demo — check the evidence types you require before a PR merges
Pick at least one.
Nick's rule: don't waste your review time on a PR without evidence — just rerun it.
Trade-off he accepts: "I'm way more incentivized to become that bottleneck if the agent shows me it actually did the work. The video turns 'no' into 'yes, with a few small fixes.'"
What's the role of the closer agent in Nick's pipeline?
5
Every failure is a harness bug
paste a JSONL line · see how the retro agent diagnoses it
▶

Nick's rule, borrowed from Ryan Lopopolo's harness-engineering talk: "never work on the code itself — only on the harness." The retro agent is the operating mechanism. It reads every JSONL log, looks for patterns the model repeated three times in a row without changing inputs (a doom loop), or fanned out parallel tool calls it didn't need, and writes the findings to a stack-specific memory file.

Next run, Case reads those notes first — and doesn't repeat the spiral.

Demo — paste a slice of JSONL tool-call log and see what retro flags
Retro will inspect the log and either flag patterns or mark the run clean.
The memory writes look like: memory/nextjs.md, memory/tanstack-start.md — one file per stack. The harness knows which file to read by sniffing the project's manifest.
Per Nick, what should you do when the agent fails a task?
6
Apply it to your own agent
the one-question self-audit
▶

Nick's parting line: "If you're making your product work for agents, figure out what they get reliably wrong and focus on that. Don't focus on the product as a whole." The fastest way to find that: a one-question audit you can run on your own harness today.

Self-audit — answer the four-part question and see your verdict
Q1Can your agent lie and get away with it?
Q2Can your agent skip a step you said was important?
Q3Can your agent add skills/context that hurt your pass-rate?
Q4Do you measure pass-rate, or do you trust the run report?
A "no" on any of these is where to invest next.
Nick's three takeaways to keep: don't build agents for everything · if you do, keep it as simple as possible for as long as possible · every failure is a harness bug — go fix the harness.
What was Nick's most surprising finding from the WorkOS CLI skills experiment?

🛠️ You've got the model

Stop telling your agent what to do — encode it in a state machine · stop trusting it — demand SHA-256 hashes and screen recordings · stop dumping docs into skills — keep only the gotchas · every failure is a harness bug, not an agent bug · measure pass-rate, never vibes. Now go enforce.

Learning Reference · How I deleted 95% of my agent skills and got better results — Nick Nisi, WorkOS (AI Engineer Summit)

Get new posts in your inbox

No spam — just new posts and learning pages when they ship.

Share X LinkedIn Reddit Hacker News