The Wrong Answer Looks Exactly Like the Right One

You ship a prompt to GPT-4o. It returns a JSON blob with a confidence of 0.97. You ship the same prompt to Claude 3.5 Sonnet. It returns a different JSON blob, also with a confidence of 0.97. Both models sound certain. Both are wrong in different ways.

This is not a hypothetical. If you run any nontrivial LLM workload for more than a few thousand requests, you will hit cases where two capable models disagree on facts, classifications, or structured output. The disagreement itself is a signal. Most teams ignore it.

N-version programming, the old idea of running multiple implementations and voting on the result, was considered too expensive for most software. With LLMs, the “implementations” are just different API calls. The cost is real, but the cost of shipping the wrong answer is often worse.

What N-Version Programming Looks Like for LLMs

The classic version comes from safety-critical systems. Three independently written programs process the same input. A voter compares outputs. If two agree and one does not, the majority wins and the minority is flagged for inspection.

With LLMs, you do not need three engineering teams. You need three API keys. The independence is weaker, models share training data, but the divergence is still measurable and useful. Two models trained on overlapping corpora can still hallucinate different facts, interpret ambiguity differently, or fail on opposite edge cases.

The question is not whether they disagree. They will. The question is what you do when they do.

A Simple Disagreement Detector That Actually Works

Here is a concrete system in Python. It sends the same prompt to multiple models, extracts structured output, and compares them using domain-aware rules rather than naive string equality.

import asyncio
from dataclasses import dataclass
from typing import Callable, List, Any
import json

@dataclass(frozen=True)
class VariantResult:
    model: str
    output: dict
    raw: str
    latency_ms: float

class DisagreementResolver:
    def __init__(
        self,
        comparators: List[Callable[[Any, Any], bool]],
        tiebreaker: Callable[[List[VariantResult]], VariantResult],
    ):
        # Each comparator returns True if two outputs are "the same"
        self.comparators = comparators
        # Tiebreaker picks a winner when no clear majority exists
        self.tiebreaker = tiebreaker

    def cluster(self, results: List[VariantResult]) -> List[List[VariantResult]]:
        """Group results into equivalence classes."""
        clusters: List[List[VariantResult]] = []
        for r in results:
            placed = False
            for cluster in clusters:
                # A result joins a cluster if it matches the cluster's first member
                if all(c(r.output, cluster[0].output) for c in self.comparators):
                    cluster.append(r)
                    placed = True
                    break
            if not placed:
                clusters.append([r])
        return clusters

    def resolve(self, results: List[VariantResult]) -> tuple[VariantResult, List[VariantResult]]:
        """Returns (winner, dissenters)."""
        clusters = self.cluster(results)
        clusters.sort(key=len, reverse=True)

        if len(clusters) == 1:
            # Unanimous agreement. Pick the fastest as winner.
            winner = min(clusters[0], key=lambda r: r.latency_ms)
            return winner, []

        if len(clusters[0]) > len(clusters[1]):
            # Clear majority.
            winner = min(clusters[0], key=lambda r: r.latency_ms)
            dissenters = [r for c in clusters[1:] for r in c]
            return winner, dissenters

        # No clear majority. Invoke tiebreaker.
        winner = self.tiebreaker([c[0] for c in clusters])
        dissenters = [r for c in clusters for r in c if r.model != winner.model]
        return winner, dissenters

The key design choice is the comparator. String equality is useless for LLM output. Two models might return {"category": "refund"} and {"category": "Refund"} and disagree on nothing that matters.

def category_comparator(a: dict, b: dict) -> bool:
    return a.get("category", "").lower() == b.get("category", "").lower()

def amount_comparator(a: dict, b: dict) -> bool:
    # Floats from different models can differ slightly
    return abs(a.get("amount", 0) - b.get("amount", 0)) < 0.01

def strict_json_comparator(a: dict, b: dict) -> bool:
    return json.dumps(a, sort_keys=True) == json.dumps(b, sort_keys=True)

# Use them in order of domain importance
resolver = DisagreementResolver(
    comparators=[category_comparator, amount_comparator],
    tiebreaker=lambda candidates: min(candidates, key=lambda r: r.latency_ms),
)

Where This Actually Catches Bugs

Disagreement detection is not about finding obvious errors. Obvious errors are already caught by simpler means.

It is about catching subtle errors that look correct to any single model. Here are the three patterns we see most often:

Confident hallucination divergence. One model invents a product SKU. Another invents a different SKU. Both outputs validate against the schema. The disagreement is the only signal that something is fabricated.

Boundary condition splits. One model classifies a $999.99 transaction as “high value.” Another classifies it as “standard.” The threshold is ambiguous in the prompt. The disagreement tells you the prompt is under-specified.

Temporal drift. One model returns a date in ISO 8601. Another returns the same date in a different timezone because the prompt said “today” and the models have different inference-time clock assumptions.

In each case, neither output is obviously broken. The disagreement is the bug report.

The Cost Model Is Surprisingly Tolerable

Three API calls cost three times as much as one. That is the superficial math. The real math depends on what you do with the results.

If you run three models on every request and take the majority output, you are burning money. Most teams should not do this.

The smarter pattern is tiered:

  1. Fast path: One cheap model handles the request.
  2. Audit path: A sample of requests, or all requests above a confidence threshold, gets sent to a second model asynchronously.
  3. Escalation path: When the audit path detects disagreement, a third model breaks the tie. The disagreement is logged for prompt engineering review.

This means you pay triple only for the subset of requests that actually need scrutiny. For many workloads, that is under 5% of traffic.

async def tiered_resolve(prompt: str, primary_model: str, audit_model: str) -> dict:
    primary = await call_model(primary_model, prompt)

    # Async audit: do not block the fast path
    audit_task = asyncio.create_task(call_model(audit_model, prompt))

    try:
        audit = await asyncio.wait_for(audit_task, timeout=2.0)
    except asyncio.TimeoutError:
        # Audit lagged. Ship the primary result.
        return primary.output

    if outputs_agree(primary.output, audit.output):
        return primary.output

    # Disagreement detected. Escalate.
    tiebreaker = await call_model("gpt-4o", prompt)
    log_disagreement(primary, audit, tiebreaker)
    return tiebreaker.output

The log_disagreement call is where the long-term value lives. Every disagreement is a data point telling you where your prompt is ambiguous, where your schema is under-constrained, or where your training examples conflict.

What This Does Not Fix

N-version programming with LLMs has real limits, and pretending otherwise makes it dangerous.

Shared failure modes. If both models fail because the prompt contains a typo, both will fail the same way. The models are not independent the way three hand-written programs are. They share architecture, training objectives, and large chunks of the internet.

Systematic bias. If your prompt implicitly encodes a bias, multiple models may reproduce it consistently. Disagreement detection only catches divergence, not shared wrongness.

Cost at scale. Even a 5% audit rate becomes expensive if your volume is billions of requests per month. The economics work for high-stakes decisions, not for every autocomplete.

Latency on the escalation path. When two models disagree and you need a third, you have added two round trips to the critical path. For real-time use cases, you may need to ship the primary result and resolve the disagreement asynchronously, accepting a brief window of potential wrongness.

How to Start Without Over-Engineering

You do not need a full voting framework on day one. Start with pairwise comparison on your highest-stakes prompt.

Pick one structured output call where a wrong answer actually costs money or trust. Add a second model call in a background job. Compare the outputs with a domain-specific comparator. Log disagreements. Review the logs weekly.

After two weeks, you will know whether your prompt is solid or a disaster. Most prompts are more fragile than their authors think. The disagreement log is an honesty check.

Once you have data, you can decide whether to build a resolver, tune your prompt, or accept that the task is too ambiguous for fully automated handling.

The Real Win Is the Feedback Loop

The goal of LLM n-version programming is not to build a perfect voter. The goal is to close the loop between production disagreement and prompt improvement.

Every time two models disagree, you have found a place where your specification is incomplete. Fix the spec, not just the output. Over time, the disagreement rate drops. When it drops far enough, you can reduce the audit sample or downgrade the tiebreaker model.

The system gets cheaper as it gets better. That is the opposite of most safety mechanisms, which get more expensive as scale increases.

Start with one prompt, two models, and a log line. The disagreements will tell you exactly where to look.