Blog

Your agent reports “done: all tests pass.” Do you believe it?

Nothing in that sentence is evidence, and the cost of finding out lands on you, later. I’ve been building with AI coding assistants for years, and the failure that kept costing me time was never that the model wrote bad code. It was that the model told me it was done, and I believed it. This post is the mechanism I built so I don’t have to, written out in enough detail that you can judge whether it would hold up against your own agent.

Ranex is a kernel (ordinary, inspectable code) that stays outside the AI’s loop and judges every step of its work. It never asks a model what to do next. Rules an agent can read are suggestions; rules compiled into code are constraints.

The problem is not that AI writes bad code

An AI writing software is a blindfolded dart thrower with a guide shouting coordinates. Two things go wrong, and they’re separate problems:

1. The thrower is blind. It cannot perceive whether its own dart landed, so it reports success either way.
2. The guide is bad. The coordinates were wrong or vague before the throw.

There’s a third failure, and it’s the most common one:

Most tools let the thrower paint the bullseye around the dart after it lands.

One actor writes the code, writes the test, and declares success. That’s why “all tests pass” from an AI means so little: the target moved to wherever the dart went.

Notice that none of this gets fixed by a better model. A more capable agent paints a more convincing bullseye, so upgrading the model you point at your repo does not touch this. That’s why I stopped trying to improve the throw and started working on the scoring.

Three ports, and only one produces a verdict

The architecture is deliberately boring:

• Model port: one completion, forced structured output. Intake, review, translating machine state into plain language. Stateless.
• Worker port: an agent with its own loop and tools, running in an isolated git worktree. Returns a diff. Replaceable by design.
• Check port: the only thing whose output counts.

Models appear in exactly three roles (proposer, critic, translator), and none of them can pass a gate. A proposer produces a proposal. A critic produces a finding. A translator produces text. None of them decides.

It is make for a nondeterministic compiler. make invokes gcc; nobody asks gcc what to build next.

What a verdict actually is

A verdict is a pure function of (gate, evidence, subject, approver). Same inputs, same verdict, always. That’s not a design goal, it’s the thing that makes the rest possible. I wrote more about why a verdict has to be a pure function separately.

Four properties hold on every evaluation:

• Absence blocks. A required claim with no satisfying evidence is FAIL, never a default, never a skip.
• Evidence is bound to a subject digest. The same command run against a different commit proves nothing about this one.
• No self-approval. Whoever produced the evidence cannot approve it, though approver identity is unauthenticated today, so that check compares unverified strings.
• A gate that cannot block is refused at construction. A non-blocking gate is decoration, so the kernel won’t build one.

And the invariant that keeps me honest about all of it: removing every model credential from the machine must not change a single verdict. If it would, something in the verdict path is asking a model for its opinion, and that’s a bug. That one is portable, by the way: pull the credentials out of whatever grades your agent today and see whether its answers move.

The loop, end to end

take the next ready task
   → create an isolated git worktree
   → spawn a worker with the task envelope
   → wait for it to exit
   → read the DIFF ON DISK  (the worker's own summary is discarded)
   → run the checks         (code, not a model)
   ├─ pass → THE KERNEL merges          (workers never merge)
   └─ fail → retry ×3 with the failure output
               → still failing → escalate to a human in plain language

Ranex never trusts the worker, including its own loop’s. It doesn’t need to control what happens inside the loop, only what’s allowed out of it. Containment is a smaller problem than control: the exits are enumerable, the interior is not.

What stops an agent editing its own tests

An agent that can edit its own tests will always pass. If the same actor writes the code and the test in your setup, you already know how that ends. Four rules prevent it:

1. Tests are frozen before building starts. Generated, digested, read-only. Any diff touching a test file fails the gate instantly.
2. Red-then-green, enforced. Every generated test must fail against the pre-implementation tree. A test that passes before the code exists is not a target. It’s a circle painted around a dart.
3. Edge coverage as a gate, not a metric. Not “80% of lines” but “every edge in the approved graph has at least one passing test.”
4. No self-approval. The task that implements a scenario never authors or judges its test, and approver identity is unauthenticated today, so that check compares unverified strings.

This project applies those rules to itself. The SLICE-001 tests were committed red at b495e3635, before any implementation existed, red-then-green as a fact in the git history rather than a claim in a document.

What a passing build actually proves

Precisely this:

Every behavior on the graph the owner approved has at least one executable test. Every test ran. Every test passed. Here is the evidence, pinned to this exact code digest.

And these are not claims Ranex makes:

• That the plan was right. Only the person who owns the target can judge that, and only by using the thing.
• Anything off the plan. Unspecified behavior is unconstrained. Absence of a requirement is absence of a guarantee.
• Non-functional properties (performance, accessibility, security) unless you add gates for them.

I go into this boundary in more detail in what a gate verdict asserts, and what it refuses. “Conformant to an approved specification” is real, defensible, and deliverable. “Correct” is not a claim anybody can make.

Ranex does not improve aim

Not by one degree. It makes misses visible and cheap, and hits provable. I don’t claim more than that anywhere: not in the docs, not in the program output, not here.

Where this actually stands

Ranex is pre-release. It is not a usable product yet. It’s a kernel with a working verdict path and very little else, and the README says exactly that before it says anything else.

What works today: evaluate(), subject-bound evidence, absence-blocks, no-self-approval, an append-only hash-chained journal, Ed25519-signed evidence, ranex run, and worker dispatch. The kernel already gates this repository’s own test suite: 943 frozen test IDs, run provisioned, sealed, and offline against the real current commit.

What doesn’t: flow graphs, scenario compilation, budget, escalation. Designed, not built.

Roughly speaking: the hardest part to get conceptually right exists, and almost none of the surface around it does. The kernel is MIT-licensed. Read the code that decides pass or fail, and try to break it.

Questions people actually ask

What does Ranex actually do?

It judges AI-written work by evidence and executable checks instead of by the model’s own report. A kernel of ordinary code sits outside the AI’s loop, reads the diff on disk, runs the checks, and produces the verdict. The agent’s summary is discarded.

Does Ranex make my AI write better code?

No. Not by one degree. It makes misses visible and cheap, and hits provable. Ranex optimizes the scoring, not the throw.

Can I use Ranex today?

Not yet. Ranex is pre-release and the README says so before it says anything else. The verdict path works and already gates Ranex’s own 943-test suite, but flow graphs and scenario compilation are designed rather than built.

Does it need a model or cloud service to run?

No. A stated invariant is that removing every model credential from the machine must not change a single verdict. Checks run locally against your code.

Three questions you can put to whatever grades your agent right now, whether or not it is this one. Pull the model credentials: does the verdict change? Remove a required piece of evidence: does it fail, or shrug? Ask who signed off: is it the same actor that wrote the code? The answers tell you what your green light is worth.

Try it. Break it. Tell me what broke. The kernel is MIT-licensed. If you find a hole in the code that decides pass or fail, that is a contribution.

Disclosure: this post was drafted with AI assistance. Every factual claim traces to the repository’s README or slice records, the same fact gate the product enforces on code. It ships only after Anthony’s own review.