Star History Monthly August 2026 | Harness

We spend a lot of time asking which AI model is best. GPT or Claude? DeepSeek or Gemini? But hand the same model to two different coding agents and it performs differently — as we'll see below, several percentage points apart on the same test.

The reason is everything around the model: what tools it can reach, what context it sees, what it remembers between steps, what rules it follows, how it checks its own work. That layer has a name now. It's the harness.

The model is the brain; the harness is the working environment. And it's the half most developers can actually change — retraining a model, altering its trained weights, is out of reach unless you work at one of a handful of labs. The harness is just code.

That asymmetry is why the term caught on. Stage five of six in Mitchell Hashimoto's February essay is Engineer the Harness: when an agent errs, you build the check or script "such that the agent never makes that mistake again." He wasn't sure the industry had a name for it. By summer it did, with an awesome-harness-engineering list at 3.5K stars.

Six months ago harness engineering was a practice. This month it showed up as four different things.

The lineup

Grouped not by what they share, but by the different question each asks about the same object.

Project	The harness question	The approach	Stars
Pi	How do I make it mine?	Keep the core small, then modify it	~98K
DeepSeek Harness (dsh)	How do I assemble one?	Everything is a plugin	~200K
Meta-Harness	Can a machine improve one?	Search over harness designs	~1.47K
Terminal-Bench	How do I know one is better?	Score agent and model together	~552

Their popularity could hardly be more different: from about 550 stars to about 200,000. Hold on to that gap; it turns out to be backwards.

Own it - Pi

🔗 https://github.com/earendil-works/pi

Pi is harness engineering as a personal practice, shipped: not a large configurable framework, but a small agent you are expected to rewrite yourself. Its homepage puts it as a slogan: "There are many agent harnesses but this one is yours."

In practice it's a coding agent you install with one command, its parts published separately: one API across 15+ model providers, the agent runtime, a terminal-UI library. Mario Zechner of libGDX started it in August 2025; Armin Ronacher of Flask is now its second-largest committer.

What explains its growth is what Pi leaves out. No built-in sub-agents, no built-in plan mode; the core stays small and the extension points sharp, so you write those or install someone else's — hence an ecosystem in the same org, including gondolin, a Linux micro-VM sandbox. The curve rewards it: 58K stars at the end of May to 98K now, roughly 600 a day across 256 releases since December.

• Tradeoffs

  • No permission system: Pi runs with its launching user's access, and sandboxing is your job
  • New contributors' issues and PRs are auto-closed by default, triaged later

• Best for: developers who would rather rewrite their agent than configure it.

Build it - DeepSeek Harness

🔗 https://github.com/deepseek-ai/deepseek-harness

Then a frontier lab industrialized the idea. Where Pi says own the code, DeepSeek says configure it — and treats an agent like Lego, where almost every piece swaps out without rebuilding the whole thing.

Taken literally: the model is a plugin. So are the tools, memory, storage, the sandbox (where the agent runs commands), the interface, and the agent loop itself — the cycle deciding what the model does next after each tool call. All of it mounts onto the Cordis kernel, a community framework from 2022 that DeepSeek adopted rather than wrote, so you reassemble an agent in configuration, never in source.

What makes it remarkable is the speed. First commit August 13; two weeks later, 200K stars — 111,089 of them in the first fourteen days, the largest two-week gain in our data, still adding 3,000 a day. For scale: Pi turned one year old on August 9 with 85,799 stars. DeepSeek matched that in a day — and Pi never slowed, holding its 600 a day straight through the frenzy. The two aren't fighting over the same developer.

• Tradeoffs

  • A self-described developer preview at v0.1.x, with breaking changes promised in capital letters
  • Issues are disabled. 200K stars, 28 contributors, and the only channels are Discussions and Discord

• Best for: anyone who wants every layer swappable, or curious how a frontier lab assembles an agent.

Optimize it - Meta-Harness

🔗 https://github.com/stanford-iris-lab/meta-harness

Now the stranger question: what if humans stop designing the harness at all?

Take one model and one hand-built harness. Run it against a hard benchmark. It scores 28.5%. Now hand a second AI agent the failed runs — the full logs, not a summary — and ask it to rewrite the harness. Test again. Repeat.

Seven rounds later: 46.5%. The model never changed. The harness did.

That's what Meta-Harness turns harness engineering into: an optimization problem. From Stanford's IRIS Lab, with Omar Khattab and Chelsea Finn among the authors, its key decision is refusing to compress the feedback — the rewriter gets every prior attempt's raw traces, up to 10M tokens a round, because harness bugs aren't diagnosable secondhand.

It spread because the harnesses generalize: one added 4.7 points across five models it was never tuned against, so the search found something real rather than memorizing a test. A March paper, 1.47K stars and a crowd of reimplementations followed, plus another group's Agentic Harness Engineering reaching 84.7% the same way. The benchmark both of them optimize against is the subject of the last section.

• Tradeoffs

  • Paper code; the maintainers say plainly it hasn't been tested beyond verifying that it runs
  • Every round is a full benchmark run, so searching is expensive and invites overfitting to that one benchmark

• Best for: teams with a fixed model, a real evaluation, and a harness already hand-tuned past the obvious wins.

Measure it - Terminal-Bench

🔗 https://github.com/harbor-framework/terminal-bench

If everyone claims their agent is better, how would you know?

Terminal-Bench is the exam. It hands agents hard, real terminal work — software engineering, machine learning, security, data science — and reports how many tasks they finish. It's a Stanford × Laude collaboration, and every frontier agent builder runs it.

Here's the part that matters. Every row on the leaderboard is not a model. It's an agent × model pair.

So you can read the same model twice. Fable 5 scores 83.8% inside Claude Code and 80.4% inside Terminus 2; GPT-5.5 scores 83.1% inside Codex and 78.0% inside Terminus 2. Identical weights both times. Only the harness changed, and it moved the result by about as much as a whole model generation would.

That is the whole argument of this post, published as a bar chart by the people whose job is to keep the scoring honest — and why "which model is best" is now half a question. The benchmark itself has grown from 80 tasks in January 2025 into a continuous one, merging new tasks year-round. It runs on Harbor, which has 4.7K stars to Terminal-Bench's 552.

• Tradeoffs

  • Running it seriously needs real sandbox infrastructure: Modal or Docker, hundreds of concurrent containers
  • Its own influence is a risk: once everyone optimizes against it, high scores measure the benchmark as much as the work

• Best for: anyone who needs an agent score they didn't report themselves, and anyone checking such claims.

Closing Thoughts

Now back to that star gap. DeepSeek Harness has more than 360 times Terminal-Bench's stars, yet Terminal-Bench is the one every lab in the top row must run before it can claim anything. Star counts are honest about attention and silent about influence, including ours.

The model race isn't over. But model choice is no longer the whole story, and this month the other half became a codebase you fork, a product a lab ships, a search space a machine explores, and the number behind every leaderboard.

Which is the small irony of the year. Harness engineering got its name because almost nobody can change the weights, making the code around them the only place left to do real work. That place turned out valuable enough that the people who can change the weights showed up to build there too.