You have five implementations of the same function. Three return the same result. One is slightly different. One throws an exception. Which one is right?
Most teams default to majority vote. That works fine when the outputs are identical and the errors are obvious. It falls apart the moment your implementations disagree in subtle ways, or when every variant returns a different answer. N-version programming only solves the first half of the problem: running multiple versions. The harder half is deciding which output to trust.
What is N-version programming?
N-version programming is a fault-tolerance technique where you run multiple independent implementations of the same specification and combine their results. The classic form is triple modular redundancy: three systems vote, and the majority wins. It dates back to safety-critical hardware, think avionics, where a single bug could kill people.
The same idea has resurfaced in AI engineering. When you ask an LLM to generate code, you might sample five different completions. When you have a legacy parser and a rewrite, you might run both in parallel and compare. The hardware roots show. We borrowed the architecture without always borrowing the selection logic that makes it work.
In hardware, the outputs are bits. In software, the outputs are structured data, strings, rankings, or side effects. Majority vote assumes equality is cheap to compute and common to find. For most software problems, neither assumption holds.
The voting trap: why “most common” isn’t “most correct”
I ran into this last year with a search ranking refactor. We had five ranking algorithms: the legacy system, two model-based approaches, and two heuristic baselines. On a sample query, four of them returned different top-10 lists. None matched exactly. There was no majority to vote for.
This is the normal case, not the edge case. Different implementations optimize for different things. One might favor recency. Another might weight popularity. A third might have a bug that only shows up on Tuesdays. If you vote by exact string match, you will tie yourself to the most mediocre implementation, the one that returns the blandest, least surprising output.
Exact-match voting also fails silently. Two implementations can return the same wrong answer because they share a bad assumption or a copied bug. Correlated failures break N-version redundancy completely. If three of your five parsers were trained on the same biased dataset, their agreement means nothing.
How differential testing actually works
The better approach is differential testing with structured comparison. Instead of asking “which outputs are identical,” you ask “which output is best according to criteria I can define and measure.”
Start by defining an equivalence function for your domain. For search rankings, you might compare using normalized discounted cumulative gain (NDCG). For JSON parsers, you might compare the resulting object graphs. For string outputs, you might use semantic similarity or a downstream task score. The key is that equality becomes a spectrum, not a binary switch.
Next, define a scoring function that maps each output to a scalar. This is where domain knowledge enters. A scoring function for a code-generation task might combine compilation success, test pass rate, runtime performance, and output length. The exact weights matter less than the fact that they are explicit.
With scores in hand, selection becomes a simple optimization. Pick the output with the highest score. If there is a tie, use a fallback heuristic, like preferring the implementation with the lowest historical error rate.
Here is a selector you can adapt. It runs N variants, scores each output, and returns the best one along with diagnostic metadata:
from dataclasses import dataclass
from typing import Callable, List, Optional, TypeVar
import statistics
T = TypeVar("T")
@dataclass
class VariantResult:
variant_id: str
output: Optional[T]
error: Optional[Exception]
score: float = 0.0
class NVersionSelector:
def __init__(
self,
score_fn: Callable[[T], float],
equivalence_fn: Optional[Callable[[T, T], float]] = None,
tie_breaker: Optional[Callable[[List[VariantResult]], VariantResult]] = None,
):
self.score_fn = score_fn
self.equivalence_fn = equivalence_fn or (lambda a, b: 1.0 if a == b else 0.0)
self.tie_breaker = tie_breaker
def select(self, variants: List[Callable[..., T]], *args, **kwargs) -> VariantResult:
results: List[VariantResult] = []
for variant in variants:
try:
output = variant(*args, **kwargs)
results.append(VariantResult(
variant_id=variant.__name__,
output=output,
error=None,
))
except Exception as exc:
results.append(VariantResult(
variant_id=variant.__name__,
output=None,
error=exc,
))
# Score only successful runs
for r in results:
if r.output is not None:
r.score = self.score_fn(r.output)
# Filter to successful, scored results
valid = [r for r in results if r.error is None and r.output is not None]
if not valid:
raise RuntimeError("All variants failed")
best_score = max(r.score for r in valid)
candidates = [r for r in valid if r.score == best_score]
if len(candidates) == 1:
return candidates[0]
if self.tie_breaker:
return self.tie_breaker(candidates)
return self._default_tie_break(candidates, valid)
def _default_tie_break(
self, candidates: List[VariantResult], all_valid: List[VariantResult]
) -> VariantResult:
# Prefer the candidate whose output is most similar to other outputs
def consensus_score(candidate: VariantResult) -> float:
similarities = [
self.equivalence_fn(candidate.output, other.output)
for other in all_valid
if other.variant_id != candidate.variant_id
]
return statistics.mean(similarities) if similarities else 0.0
return max(candidates, key=consensus_score)The score_fn is where you encode what “best” means for your problem. The equivalence_fn handles the case where scores tie by measuring how similar an output is to the rest of the pack. This generalizes majority vote into something that works when no two outputs match exactly.
Scoring implementations with multi-criteria evaluation
A single scalar score is clean, but dangerous if you collapse too many dimensions into one number. I prefer a two-tier approach.
Tier one is a hard filter. Outputs that fail compilation, violate invariants, or crash get discarded immediately. No partial credit.
Tier two is a soft score for the survivors. This can be a weighted sum, but keep the weights explicit and adjustable. If you find your selector consistently preferring fast but wrong answers, you can bump the correctness weight without rewriting the architecture.
def score_generated_code(output: str) -> float:
if not compiles(output):
return -1.0 # Hard filter
tests_passed = run_test_suite(output)
execution_time = benchmark(output)
line_count = len(output.splitlines())
# Weighted sum on survivors only
return (
0.6 * tests_passed
+ 0.3 * (1.0 / (1.0 + execution_time))
+ 0.1 * (1.0 / (1.0 + line_count))
)The weights above are arbitrary. Tune them against a held-out validation set, and update them when your requirements change. Do not bury them inside a class where they become invisible.
When this breaks down: correlated failures and runtime cost
N-version selection is not free. Running five variants means five times the compute, five times the latency, and five times the maintenance burden. If your implementations are LLM calls, that cost is measured in dollars and seconds. If they are microservices, it is measured in queue depth and thread count.
The bigger risk is correlated failure. Five implementations do not give you five independent draws from a bug distribution. They share languages, libraries, training data, and human authors. If all five use the same regular expression to parse dates, they will all break on the same malformed input. Diversity is hard to enforce. It requires actively auditing for shared dependencies and copied logic.
There is also the question of what to do when the selector itself is wrong. If your scoring function has a blind spot, you will consistently select bad outputs and never notice. Monitor the selection distribution. If one variant is never chosen, it is either useless or your score function is biased. Either way, you should investigate.
Putting it together: a selector you can ship today
You do not need a distributed system to start. Wrap your existing function in a selector, add one alternative implementation, and define a single scoring criterion that matters to you. Run them in parallel. Compare the outputs. Log the scores.
After a week of logging, review the cases where the variants disagreed. That disagreement is a gift. It tells you where your specification is ambiguous, where your score function is wrong, or where one implementation has a bug the others do not.
Scale up gradually. Two variants with a good selector beat five variants with majority vote.
FAQ: correlated failures, stateful systems, and runtime overhead
What if all five implementations return different results?
This is the expected case for non-trivial problems. Use a scoring function to rank the outputs instead of looking for exact matches. If you cannot define a good score function, the problem is not the selector. It is that you do not yet know what “correct” means for your domain.
How do I prevent correlated failures across variants?
Audit for shared dependencies, copied code, and common training data. Force at least one variant to use a different language or framework. The goal is uncorrelated error distributions, which is harder than it sounds.
Does N-version programming work for stateful systems?
It works poorly for systems with side effects. If your function writes to a database or charges a credit card, running it five times is destructive. N-version selection works best for pure functions, read-only queries, or operations where you can stage the side effect after selection.
How much overhead does running five variants add?
Latency is bounded by the slowest variant unless you run them in parallel. CPU and memory scale linearly. Start with two variants and measure before committing to five. The operational cost often matters more than the selection logic.