You need to test a function where the correct output is impossible to know in advance. A route optimizer. A sentiment classifier. A physics simulation. You’ve read about metamorphic testing: find relations between inputs and outputs that must hold, then test those relations instead of exact values.

The problem is coming up with the relations. You stare at your function signature and draw a blank.

So you ask an LLM. It fires back ten metamorphic relations in seconds. Most look plausible. A few are obviously wrong. One or two are genuinely clever. The LLM is a useful brainstorming partner, but it is not a substitute for domain knowledge. Paste those relations into your test suite unchecked, and you will ship tests that pass on broken code or fail on correct code.

What the LLM Actually Gives You

When you ask an LLM to propose metamorphic relations for a function, it is doing advanced pattern matching across training data. It has seen thousands of implementations of sorting algorithms, geometric libraries, and machine learning pipelines. It recognizes that “adding a constant should shift the mean by the same constant” is a common property, and it suggests it.

This is not understanding. It is statistical association. The LLM does not know whether your function handles empty lists, NaN values, or integer overflow. It proposes the relation because it has seen similar ones before.

Here’s what a typical prompt and response looks like. You feed the LLM a function and ask for relations:

# Your function under test
def percentile(values: list[float], p: float) -> float:
    """Return the p-th percentile using linear interpolation."""
    if not values:
        raise ValueError("empty list")
    if not 0.0 <= p <= 1.0:
        raise ValueError("p must be in [0, 1]")
    sorted_vals = sorted(values)
    n = len(sorted_vals)
    idx = p * (n - 1)
    lower = int(idx)
    upper = lower + 1
    if upper >= n:
        return sorted_vals[-1]
    frac = idx - lower
    return sorted_vals[lower] * (1 - frac) + sorted_vals[upper] * frac
Prompt: Propose 5 metamorphic relations for this percentile function.

Response:
1. Monotonicity: if p1 < p2, then percentile(values, p1) <= percentile(values, p2)
2. Scaling: percentile([v * k for v in values], p) == percentile(values, p) * k
3. Translation: percentile([v + c for v in values], p) == percentile(values, p) + c
4. Permutation invariance: percentile(values, p) == percentile(shuffled(values), p)
5. Boundary: percentile(values, 0.0) == min(values), percentile(values, 1.0) == max(values)

Three of these are correct and useful. One is subtly wrong. One is trivially true but so loose it catches almost no bugs.

The monotonicity relation is correct, though you need to handle duplicate values. The scaling relation fails for k <= 0 because sorting order inverts. The translation relation is solid. Permutation invariance mostly tests whether you remembered to sort. The boundary relation is correct only if your interpolation treats the 0th and 100th percentiles as min and max.

The LLM does not warn you about any of this. It presents all five with equal confidence.

How to Filter LLM-Generated Relations

The useful workflow is not “ask the LLM, copy the output, go to lunch.” It is “ask the LLM, treat the output as a candidate list, then verify each candidate with reasoning and testing.”

Step one is to classify each proposed relation by type. Structural relations, like permutation invariance or idempotence, tend to be safer because they depend less on domain semantics. Arithmetic relations, like scaling or translation, are powerful when they hold, but they often fail on edge cases the LLM didn’t consider: negative multipliers, empty collections, floating-point rounding.

Step two is counterexample hunting. For each proposed relation, try to find an input where it fails. This is the fastest way to spot LLM hallucinations.

def test_percentile_scaling_counterexample():
    """The LLM proposed scaling. It fails for negative k."""
    values = [1.0, 2.0, 3.0, 4.0]
    original = percentile(values, 0.5)  # 2.5
    
    k = -1.0
    scaled_values = [v * k for v in values]
    scaled_result = percentile(scaled_values, 0.5)  # -2.5
    
    # The relation holds here by accident. For nearest-rank,
    # multiplying by a negative flips sort order and breaks it.
    assert abs(scaled_result - original * k) < 1e-9

The scaling relation happens to hold for linear interpolation, but you only know that because you tested it. The LLM did not know. It guessed based on pattern matching. A different percentile algorithm would break scaling in obvious ways.

Step three is mutation testing. Once you have a relation as a test, run a mutation testing tool against it. If mutants survive, your relation is too weak. If correct code gets killed, your relation is wrong.

Where LLMs Shine and Where They Flop

LLMs are genuinely useful for discovering relations in well-trodden domains. They know that image classifiers should be invariant to horizontal flips, that sorting algorithms should be idempotent, and that matrix multiplication should distribute over addition. The LLM recalls these canonical relations instantly.

They are less useful in domains with implicit constraints that don’t appear in training data. If you’re testing a custom pricing engine with business rules about regional discounts and promotional code interactions, the LLM has no idea. It will propose generic arithmetic relations that ignore the business logic, or worse, suggest relations that contradict it.

The failure modes are predictable:

Overly general relations. The LLM suggests “output should be positive” for a function that returns a probability. That’s a weak sanity check, not a metamorphic relation. It catches crashes but not logic bugs.

Relations that assume continuity. The LLM proposes that small input changes produce small output changes. That fails for threshold functions and discrete classifiers.

Relations that ignore type constraints. The LLM suggests sorting a list of dataclasses by a field, then checking that the first element’s field is the minimum. It forgets that some fields might be optional, or that the comparison operator might not be defined for that type.

Relations that are mathematically false. The LLM once suggested that the median of a concatenated list equals the average of the medians of the sublists. That’s not true. The LLM presented it with the same confidence as permutation invariance.

A Practical Workflow

Don’t ask the LLM to replace your brain. Ask it to accelerate the part where you stare at a blank page.

Start by writing a one-paragraph description of your function, including types, constraints, and known edge cases. The more context you give, the less the LLM hallucinates.

Ask for relations categorized by type: invariance relations, monotonicity relations, additive relations, and structural relations. This framing helps the LLM organize its pattern matching and produces more consistent output.

For each proposed relation, run through this checklist:

  1. Does it hold for empty inputs?
  2. Does it hold for single-element inputs?
  3. Does it hold for negative values, zero, NaN, or infinity?
  4. Does it hold when the input is already in sorted order? Reverse order?
  5. Can I write a mutation test that kills mutants only if this relation holds?

Keep the relations that survive all five. Discard the rest, and document why.

Here’s a template for the prompt we use internally:

SYSTEM_PROMPT = """
You are a testing assistant. Given a Python function, propose metamorphic relations.
For each relation:
1. State the relation clearly
2. Identify the type: invariance, monotonicity, additive, or structural
3. List edge cases where it might fail
4. Rate confidence as HIGH, MEDIUM, or LOW
"""

def generate_relations(source_code: str) -> list[dict]:
    response = openai.chat.completions.create(
        model="gpt-4o",
        messages=[
            {"role": "system", "content": SYSTEM_PROMPT},
            {"role": "user", "content": f"Propose relations:\n\n{source_code}"}
        ],
        temperature=0.3,
    )
    return parse_relation_candidates(response.choices[0].message.content)

At 0.3, the LLM is less creative but more consistent. For metamorphic relations, consistency beats creativity. You want the boring, correct relations, not the clever, wrong ones.

The Real Bottleneck Is Still You

LLMs can accelerate discovery, but they cannot replace verification. A metamorphic relation you have not personally validated is not a test. It is a guess dressed up in an assertion.

The honest answer to “can an LLM discover test oracles for me?” is partial. It can discover candidates, jog your memory, and suggest edge cases. It cannot tell you which relations are correct for your specific implementation, with your specific constraints, in your specific domain.

That part still requires a human who understands the code. The LLM is a brainstorming partner, not an oracle for the oracles.

If you’re starting from scratch, pick one function with a weak oracle, ask an LLM for five relations, then spend twenty minutes trying to break each one. The relations that survive are your seed set. The ones that break teach you more about your function than the LLM ever could.