6 posts
Differential testing lets you find bugs without knowing the correct answer. The problem is that correlated mistakes look like agreement.
You can trust differential testing without a formal proof, but only if you understand exactly where it falls apart. The weakness is called common-mode failure.…
N-version programming assumes your implementations agree.
You built an n-version system. Three independent implementations of the same critical function, a voter that picks the majority result, and a warm feeling that…
Running multiple LLMs in parallel catches errors that any single model will confidently ship.
You ship a prompt to GPT-4o. It returns a JSON blob with a confidence of 0.97. You ship the same prompt to Claude 3.5 Sonnet. It returns a different JSON blob,…
N-version programming sounds simple: run multiple implementations and pick the best answer. In practice, 'best' is a lot harder to define than 'most common.'
You have five implementations of the same function. Three return the same result. One is slightly different. One throws an exception. Which one is right? Most…
N-version programming with LLMs doesn't require multiple models.
N-version programming assumes diversity comes from different authors. With LLMs, that means different models, different providers, maybe different training…
N-version programming promised that independent teams would make independent mistakes.
In the early 1980s, NASA faced a question that still haunts safety-critical engineering today: how do you tolerate bugs you haven't found yet? Their answer was…