Stochastic Macro
Request access
Engineering notes · Merge-gate design

Scope-based merge gating (and closing the severity-downgrade loophole).

Stochastic Macro ·

Most AI review pipelines gate the merge on severity: findings at or above some threshold block the PR; everything else is advisory. This is a natural design and it has a quiet failure mode — the reviewer, being a language model, can downgrade its own severity label and the gate stops firing. You don't even need malice to see it happen; models are often just excessively charitable to themselves about how urgent their findings are.

A more robust design gates on scope: the question is not "how bad is this?" but "does this issue belong to the code the PR actually changed?" Severity becomes a descriptive label for humans; scope does the merge-gate work. This post describes the three-way scope taxonomy, the JSON-Schema constraint that closes the severity-downgrade loophole, and how to route out-of-scope findings so nothing falls on the floor.

The three-way scope taxonomy

Every finding gets classified into exactly one of three scopes:

  • current_diff — the issue is in lines the PR directly added or modified. The PR introduced (or could have fixed) this problem. Blocking.
  • required_by_current_diff — the issue is in code the PR did not directly touch, but the PR's semantics force a change there. The canonical example: a pre-existing caller that now violates a new contract the PR established. The PR must address this, either in this PR or by demonstrably tracking it as follow-up before merge. Blocking.
  • outside_diff — the issue is pre-existing and unrelated to the PR's changes. It's a real finding but it is not the PR's responsibility to fix. Not blocking — routed to a tracker issue instead.

The required_by_current_diff category is the one that most teams get wrong. It is the "induced requirements" category: a PR that establishes a new invariant or contract typically creates work for existing callers. That work has to be either done here or explicitly scheduled; letting it slide under "outside_diff" lets the PR land a new contract with known violations in production. Making it a first-class blocking scope forces the conversation.

The merge gate

Instead of "any finding at severity >= major blocks," the gate is: 

has_blocking = any(
    f.scope in {"current_diff", "required_by_current_diff"}
    for f in findings
)

Severity is still useful — humans want to see major vs minor in the consolidated review — but severity does not drive the gate. The reviewer cannot unblock a PR by lowering severity; it can only unblock by narrowing scope, and narrowing scope is visible and reviewable.

The severity-downgrade loophole

Even with scope-based gating, there is still one evasion path if you're not careful: the reviewer can raise a finding inside current_diff and mark its severity as informational. If your pipeline uses severity as a tiebreaker or display filter, the finding may be quietly suppressed and never surfaced to a human. A real defect has just been re-labeled out of existence.

The fix is a schema-level constraint: reject any finding where scope is in-diff and severity is informational. The invariant is "if this finding belongs to diff lines, it is at least minor." Enforce it in the structured-output schema: 

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "object",
  "properties": {
    "scope":    { "enum": ["current_diff", "required_by_current_diff", "outside_diff"] },
    "severity": { "enum": ["blocking", "major", "minor", "informational"] }
  },
  "required": ["scope", "severity"],
  "allOf": [
    {
      "not": {
        "properties": {
          "scope":    { "enum": ["current_diff", "required_by_current_diff"] },
          "severity": { "const": "informational" }
        },
        "required": ["scope", "severity"]
      }
    }
  ]
}

Validation runs before the aggregator consumes the reviewer's output. A reviewer that emits a forbidden combination fails its own job loudly — you would rather a red CI step than a silently-swallowed diff-scoped defect.

Routing outside-diff findings

Once outside_diff findings are non-blocking, you have to do something with them or they fall on the floor. The most practical pattern is to open them as issues on your tracker, grouped by PR and by reviewer. A minimal routing step:

  1. After the aggregator computes the merge gate, collect all findings where scope == "outside_diff".
  2. Search the tracker for an existing open issue whose title or body substantially overlaps. Comment on the existing issue rather than opening a duplicate.
  3. If no match, open one issue per reviewer per PR grouping the new findings — not one issue per finding, unless the findings are substantively unrelated.
  4. Post a short summary on the PR linking the opened/updated issues, so a human can see what was routed.

Record the duplicate-check searches in the new issue body itself, not just the commit message. That way a reviewer can audit that the duplicate-check happened even after CI logs expire.

Why scope and severity are both required

Some teams react to this post with "can we just delete severity?" You can, but you'll regret it. Severity is how humans prioritize across a long list of findings that are all in-scope. A 20-finding PR with all major severity is a different conversation from a 20-finding PR where two are blocking and the rest are minor. The split between the two fields is:

  • Scope — answers "is this the PR's problem?" and drives the merge gate.
  • Severity — answers "how bad is this?" and drives human prioritization.

Both are necessary; neither is sufficient alone.

Checklist

  1. Add a scope field to your reviewer output schema with the three enumerated values.
  2. Make your merge gate scope-driven, not severity-driven.
  3. Add the allOf/not constraint to reject in-diff findings at informational severity.
  4. Build a routing step for outside_diff findings with duplicate-check logic.
  5. Keep severity as a display/sort field for humans; do not let it drive the gate.

Why we're publishing this

Scope-based gating is the cleanest way we've found to prevent AI reviewers from silently evading the merge gate via severity labels, and we have not seen the three-way taxonomy plus schema-level no-shrug-off rule specified together in public writing. Free to take and adapt; better for everyone if this is the default pattern instead of severity-based gates with an invisible evasion path.

← All engineering notes