Stochastic Macro
Request access
Engineering notes · Protocol design

Detecting empty-but-blocking AI review output.

Stochastic Macro ·

If your AI reviewer emits a structured object that looks like this: 

{
  "reviewer": "security",
  "has_blocking_findings": true,
  "findings": []
}

you have a silent merge-blocker. The boolean says the PR cannot merge; the findings array says there's nothing for a human to act on. No one can resolve a finding that isn't there. The PR is stuck.

This is a failure mode every AI review pipeline eventually hits. It happens when the model hallucinates its own output schema, when a refusal path fires partway through generation, when a tool call fails silently, or when a reviewer prompt is ambiguous enough that the model flags "something" without locating it. Here's how to catch it at two layers so a silent block cannot land in production.

Layer 1: validate inside each reviewer wrapper

Every reviewer in a multi-domain pipeline has some wrapper code that calls the model, parses its response, and writes a JSON artifact. That wrapper is the first line of defense. Before you write the artifact, assert the consistency rule: 

# Pseudocode — runs inside each reviewer job, before writing the artifact.
if output.has_blocking_findings and not output.findings:
    raise ReviewerProtocolError(
        reviewer=output.reviewer,
        message="has_blocking_findings=true with empty findings array"
    )

If this fires, fail the reviewer job loudly. The CI step goes red, the wrapper surfaces the violating JSON in its logs, and the aggregator runs with a missing artifact for that reviewer rather than a corrupted one. A loud failure in one reviewer is far better than a silent merge-block across the whole pipeline.

Layer 2: validate again inside the aggregator

Defense in depth. The aggregator reads every reviewer's JSON artifact and must re-check the same invariant before using the artifact to decide the top-level merge gate. This catches two cases the reviewer wrapper misses:

  1. A malicious or buggy reviewer-wrapper emits a corrupted artifact (the invariant check is also broken).
  2. An artifact got edited after the reviewer exited — unlikely, but defense-in-depth is cheap here.
# Pseudocode — runs inside the aggregator before computing the top-level gate.
for artifact in artifacts:
    if artifact.has_blocking_findings and not artifact.findings:
        fail_aggregator(
            f"Corrupt reviewer artifact from {artifact.reviewer}: "
            f"has_blocking_findings=true with empty findings array"
        )

top_level_blocking = any(a.has_blocking_findings for a in artifacts)

A failed aggregator is the correct outcome here: you'd rather a human see "the review pipeline failed with a protocol error" and rerun it than have the PR sit blocked forever with no visible finding.

You can encode this in a JSON Schema, too

If your artifacts are validated against a schema, the invariant can be expressed as a schema constraint using allOf and not: 

{
  "allOf": [
    {
      "not": {
        "properties": {
          "has_blocking_findings": { "const": true },
          "findings": { "maxItems": 0 }
        },
        "required": ["has_blocking_findings", "findings"]
      }
    }
  ]
}

Run this at both layers. Schema enforcement is mechanical and cheap; you do not want this particular invariant to live only in model prose.

Why it happens more than you'd think

  • Refusal-style truncation. Some models will emit a shell of the expected object after partial refusal — the boolean gets filled, the array stays empty.
  • Ambiguous reviewer prompts. "Flag anything that looks risky" without a "what counts as a finding" clause encourages the model to assert risk without locating it.
  • Silent tool failures. If the reviewer uses a tool call to gather findings and the tool call fails, some wrappers default the findings array to empty while keeping whatever default the model already produced for the boolean.
  • Schema-constrained decoding gone wrong. When a model is forced to emit schema-valid JSON under pressure, it will often pick the cheapest valid combination — which here includes the blocking-with-empty-findings combination unless the schema explicitly forbids it.

Variations

  • Same invariant, inverted: has_blocking_findings=false with at least one severity=blocking finding. Catch that one too — either the boolean is wrong or the severity label is.
  • If your schema has multiple severity tiers, extend the rule: "at least one finding of severity >= N" implies has_blocking_findings=true, and vice versa.
  • Log the raw model response (redacted if needed) when a protocol violation fires. It's the fastest way to see which reviewer's prompt is misbehaving.

Why we're publishing this

Every pipeline we've seen that feeds structured LLM output into a merge gate has hit this failure mode at least once. The fix is small and the cost of not having it is an unresolvable PR block that burns your team's morning. Free to take.

← All engineering notes