A CodeRabbit alternative that runs your code, not just reads it

CodeRabbit tells you what a diff looks like it might do. Vinv runs the build and tells you what it did — then refuses to promote it until that evidence clears a policy you wrote down. Opinions need a human to adjudicate them; verdicts are reproducible. Every row below is a difference in mechanism, and the sections further down take them one at a time. The last row is conceded outright.

⚠️  Potential issue · checkout/pricing.py:88

Consider validating that qty is positive
before computing the discount — a negative
value could produce an unexpected total.

CRITICAL · functional · checkout/pricing.py:88
run 4c1e · reproduced 12/12

  quote(qty=-3, sku="AX-9") -> total -41.97
  oracle: invariant total >= 0 violated

  -> tests/test_pricing_regression.py written
  -> certification: NO-SHIP

One is a question for a human. The other is a closed loop: found by execution, rendered to a durable test, attached to a verdict that stops the release.

A CI event triggers it; it provisions an isolated copy of the service and its dependencies, runs the testing types against it, merges the evidence, and signs a certification. What "ready to ship" means lives in your repo:

dag: full-gauntlet
stages:
  functional: { budget: 20m }
  load:       { slo_p99_ms: 250, target_rps: 800 }
policy:
  coverage_floor: 70
  block_on: [critical]

It runs in your CI, inside your perimeter. Only findings, coverage and verdicts ever cross back — your source never does.

A finding is a constructive existence proof. It is the input, the trace, the violated oracle and the recipe to replay it — so checking whether it is real costs one run. A review comment is an assertion with no witness attached, and checking it costs an engineer an argument.

Three limits bound anything that only reads code, whether the reader is a static analyser or a language model. Undecidability — non-trivial semantic properties are undecidable in general, so every static tool over- or under-approximates. The oracle problem — calling a behaviour wrong needs a specification, and a reader infers one from names and convention with nothing running to contradict it. No calibration — comments arrive without labels, so a reviewer cannot estimate its own precision; ground truth is recovered downstream, by humans, and never fed back.

Which is why counting a gate's findings and a reviewer's comments in one "issues found" number compares a proof to a suggestion. They are not substitutable inputs to a promotion decision.

The input is a running system, not a text. You cannot observe a tail latency, a leak or a race in a diff — they are properties of an execution. Provisioning stands up a faithful, disposable world per testing type: the service plus its own isolated dependencies, its database rather than a shared one, behind a locked perimeter with a TTL.

• What "faithful" claims — topology and dependency presence: real datastore, real broker, real network hops, real configuration surface.
• What it does not claim — production traffic mix or data distribution. Defects that need the real workload are not reached by provisioning alone.
• The open threat — masking changes data shape, and data shape changes which branches are reachable. We have not quantified what that costs in reach, and we would rather say so here than have you find out in a trial.

Each is a pluggable (technique, oracle, verdict): adding one registers a new component and never modifies an existing one, so it competes for budget without touching the others.

• Functional — coverage-guided search plus concolic/SMT. Passes when there is no new crash, coverage clears the floor, and no contract is violated.
• Load — open-model sustained traffic with a Universal Scalability Law fit. Passes when p99 ≤ your SLO at the target arrival rate and throughput does not go retrograde.
• Chaos — hypothesis-driven fault injection. Passes when steady state holds under fault and recovery lands inside target.
• Penetration — class-targeted adversarial inputs with taint tracking. Passes when no security policy is violated.

Three layers, ordered by cost per unit of new behaviour — each hands off exactly where it stalls, and nothing reports a defect it cannot witness.

• Static, the prior — call graphs, reachability, taint paths, contract extraction, dead code. Mostly a budget allocation: dead regions leave the coverage denominator and taint sinks nominate injection classes. It also emits lint-class findings outright, where no execution is needed to be sure.
• Algorithmic, the workhorse — coverage-guided greybox fuzzing with power scheduling, plus concolic execution: collect the path condition, negate a branch predicate, ask an SMT solver for a model that drives the sibling branch. Cheap, deterministic, and it does the overwhelming majority of the work.
• The drain — where the solver cannot construct the witness at all. Next section.

The known limits are the standard ones — path explosion, solver timeouts on non-linear and string-heavy constraints, opaque native calls that leave the symbolic model. And coverage is reported, not celebrated: it appears in the policy floor because a floor needs a computable quantity, not as a proxy for defect-finding ability.

A proposal, never a decision procedure. Solvers stall on construction, not on logic: a valid session, a signed token, a tenant-consistent order graph, a fixture that satisfies five constraints at once. That subproblem is the one place a language model is the better instrument, so it proposes a candidate and execution decides.

propose  a candidate object, from the model
execute  run the service with it
accept   only if it opens new coverage, reaches the
         target branch, or trips an oracle
record   the input + trace as a witness; else discard

A wrong guess costs budget and nothing else — it fails to run, fails to reach the branch, or trips no oracle, and it is thrown away. No verdict rests on what a model claimed: every finding we keep is backed by an execution we recorded. And once a witness is found it is just an input, so replaying it never calls the model again — the test we leave in your repo has no model in it at all.

A finding requires a violated oracle, declared in advance. Without a specification there is no wrong answer, only a surprising one. Each testing type may fire only the oracles it declares, which is what makes a type a registrable component rather than a bag of scripts.

• Crash — abnormal termination, assertion failure, sanitizer trip.
• Contract — declared pre/post-conditions, type and schema invariants.
• Differential — disagreement against a reference implementation or the prior release.
• Metamorphic — relations that must hold between related inputs where no reference exists.
• Resource — leak, unbounded growth, descriptor exhaustion under sustained run.
• Policy — the security classes, taint-directed.

The limit we do not escape: a silent wrong answer with no contract, no reference and no metamorphic relation stays invisible. Oracle incompleteness bounds all dynamic testing, and a gate claiming otherwise would be lying to you.

A ceiling, not a number. There are three standard ways to measure latency wrongly, and each flatters the system under test.

• Closed-loop harnesses — a fixed pool of looping virtual users self-throttles under queueing: the service slows, the harness offers less work, and the tail you came to measure disappears. We generate an open model with a real arrival rate.
• Coordinated omission — timing from actual send time silently deletes the worst samples when the generator stalls. We record against intended send time.
• A single RPS figure — at an arbitrary concurrency it says nothing about where the system turns over.

So the gate fits a Universal Scalability Law curve and reports contention and crosstalk coefficients: a positive crosstalk term means a peak beyond which added load reduces work done. The verdict is p99 ≤ your SLO at the target rate, and no retrograde region below the declared operating point.

Union is the join. Averaging is not a merge. Types run in isolation and their evidence has to compose without one diluting another, so coverage is unioned — idempotent, commutative, monotone. Registering a new testing type can therefore only raise merged coverage, and adding a type never flips a passing gate on coverage grounds.

Deduplication, honestly: findings are grouped across types by oracle class plus a normalised stack signature, and stack hashing both over-merges and under-merges. We have no fix for that, so the grouping ships with the evidence rather than as a deduplicated count presented as fact. Treat any "N unique bugs" number — ours included — as a claim about a hash function.

A pure function over recorded evidence. The gate is decide(policy, evidence, waivers, now) — no clock, no store, no I/O inside it. Same policy, same evidence, same answer, which is a unit-testable property rather than a promise.

• Audit — "why was build X blocked?" is answered from the record, not from someone's memory of a review thread.
• Counterfactual — re-run today's policy against last quarter's evidence and see what would have changed. Tightening a standard becomes measurable before it is enforced.
• Waivers that do not rot — an approver can waive a finding and the waiver is signed into the artifact, but the coverage floor survives even an approver. That rule is asserted by an end-to-end test, not promised by a document.

Provenance says how it was built. This says what was demonstrated. The output is a signed certification over the policy hash, the evidence digest, the verdict, the waivers, the environment and the budget spent — verifiable offline, with no call back to us, and explicitly revocable.

What a ship asserts is deliberately narrow. Not "this build is correct", but: under this policy, with this budget spent as recorded, no admissible oracle fired and coverage cleared the floor. That is weaker than correctness and much stronger than a green check — and it is the strongest claim any dynamic method is entitled to make. It composes with build-provenance frameworks rather than replacing them.

Redaction enforced by the type, not by a code review. To review a diff in the cloud, the diff has to go to the cloud — that is structural to the competing design. Vinv splits the system instead: the control plane holds engine images, policy and verdicts; the data plane, inside your perimeter, is the only place your code and data ever exist.

The outbound port accepts only a redacted batch, and a batch cannot be constructed until redaction has run — so no code path can send unredacted material, because the value does not exist. The masking pipeline is the one component that fails closed: with nothing wired, the environment starts empty, because unmasked data must never be a fallback. Everything else degrades the other way, reporting what it could not do rather than pretending it did it.

A gate is judged by its false blocks. What kills one is not a missed bug — it is three wrong stops, after which someone routes around it. So the number we hold ourselves to is the false-block rate: releases stopped by findings later judged non-issues, alongside how long a verdict takes on a slow build. A reviewer's dominant failure is the ignored comment, which is structurally unmeasurable — nothing records which suggestions were right and skipped.

What our static layer already decides: dead and unreferenced code, and separately what no captured run executed — two opposite verdicts a reader cannot tell apart. Unused parameters, imports and unreachable branches. Docstrings that document parameters which do not exist.

What we concede, and why: whether an abstraction earns its name is not a computable property, and reading is the correct instrument for it. Architectural intent and API taste are judgements no oracle encodes. We catch a docstring that drifts from the signature, not one that lies fluently about behaviour. And a service that cannot be run leaves a gate with nothing to execute, while a reviewer still has something to say. The concession is narrower than it first looks, and it is still real: the two tools partition the defect space more than they overlap it.

Test-generation results are famously easy to overclaim — single runs, unmatched budgets, uncontrolled seeds, coverage substituted for bugs, best-of-N reporting. So the methodology is committed to before the numbers:

• Ground truth — injected-bug suites with known ground truth, plus real historical defects replayed at the commit that introduced them. Coverage alone is not evidence of bug-finding ability.
• Distributions, not maxima — many trials per configuration, reported seed sets, matched budgets, survival curves of time-to-first-witness with a non-parametric test and an effect size. Never a single best run.
• Verdict reproducibility — re-running the decision over archived evidence must reproduce the historical verdict exactly.
• Cost accounting — findings per node-hour, including the model's proposals, so the expensive layer justifies itself separately from the cheap ones.

CodeRabbit is not the only comparison worth making. Each of these owns one piece of what a release gate has to do, and several are very good at their piece:

• AI pull-request review — CodeRabbit, Greptile, Graphite, Qodo, Copilot review. Reads the change; never runs it.
• Continuous fuzzing — OSS-Fuzz, Mayhem, Code Intelligence. Finds bugs in targets you write a harness for; does not decide releases.
• Deterministic simulation — Antithesis. The closest to us technically, and genuinely impressive: it explores a system far more deeply than a test suite. It does not sign a promotion verdict.
• Fault injection — Gremlin, Chaos Mesh, LitmusChaos, AWS FIS. One attack type, usually run by hand.
• API and web security testing — Burp, ZAP, StackHawk, Escape, Akto. One attack type, its own separate report.
• Release policy and approvals — Open Policy Agent, Harness, Argo Rollouts, Azure release gates. Enforces a checklist over evidence you produced somewhere else.
• Ephemeral environments — Testcontainers, Signadot, Uffizzi, Okteto. Stands the world up; nothing drives it adversarially once it is up.

Buy four of these and you still need someone to decide what they collectively mean. That decision — one run, every angle, one signed answer — is the part we sell.

• What does it cost? It meters the node-hours a run consumes — you pay for testing that happened, not for seats that might use it. Self-hosted and air-gapped are licensed.
• Does my code go to your servers? No. Code and data live only in the data plane, inside your boundary; findings, coverage and verdicts are the only things that cross back, and only after redaction.
• Is there an LLM in the loop? Only as the drain: where the solver stalls on constructing an object, a model synthesizes one. The search, the oracles and the verdict are algorithms, not prompts — no model decides whether you ship.
• Does it replace my test suite? No — it grows it. Every finding renders to a durable regression test in your repo, so the gate hands work back to the suite you already run.
• Can I keep CodeRabbit? Yes, and many teams should. Different moments, different evidence, nothing to reconcile between them.

MCP server for Python · Dead code detection · Python profiler · CodeRabbit alternative · Home