You know exactly where production crashed. The stack trace points to line 147 of invoice_service.py. The exception is a KeyError on "customer_id". You pull the code, run the tests, everything passes. You hit the endpoint manually with a sample payload, it works fine.

The bug is real. Customers are hitting it. You cannot make it happen on your machine.

This is the standard experience of debugging from error reports. Stack traces tell you where. They do not tell you what arrived there. Without the exact inputs that triggered the failure, you are reconstructing a crime scene from a photograph of the chalk outline.

What crash replay actually means

Crash replay is the practice of capturing the complete input state that triggered a production failure and re-executing the code path with that exact state in a local environment. The goal is to turn “it crashed on line 147” into “here is a test case that makes line 147 crash every single time.”

Most developers already do a manual version of this. You read the stack trace, guess which request caused it, try to reconstruct the payload from logs, and hope your local database has similar data. This fails for the same reason astrology fails: you are matching patterns without enough information.

The difference between guessing and replaying is serialization. You must capture the exact function inputs, the exact database responses, and the exact external API return values at the moment of failure. Then you feed them back in.

How to capture production inputs for replay

The simplest effective pattern is a decorator that intercepts a function’s arguments, serializes them to disk, and ships them somewhere you can access later. When the function crashes, you have a frozen snapshot of the world that produced it.

Here is a working Python implementation:

import json
import functools
import traceback
from pathlib import Path
from datetime import datetime, timezone

CAPTURE_DIR = Path("/var/crash-captures")


def capture_for_replay(func):
    """Decorator that captures inputs and outputs for replay debugging."""
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        capture = {
            "timestamp": datetime.now(timezone.utc).isoformat(),
            "function": func.__qualname__,
            "module": func.__module__,
            "args": args,
            "kwargs": kwargs,
            "exception": None,
            "traceback": None,
        }

        try:
            result = func(*args, **kwargs)
            capture["result"] = result
            return result
        except Exception as exc:
            capture["exception"] = {
                "type": type(exc).__name__,
                "message": str(exc),
            }
            capture["traceback"] = traceback.format_exc()

            # Write crash capture to disk
            CAPTURE_DIR.mkdir(parents=True, exist_ok=True)
            filename = (
                f"{func.__name__}_"
                f"{datetime.now(timezone.utc).strftime('%Y%m%d_%H%M%S')}.json"
            )
            capture_path = CAPTURE_DIR / filename

            with open(capture_path, "w") as f:
                json.dump(capture, f, default=str, indent=2)

            raise  # Re-raise so normal error handling continues

    return wrapper

Apply it to the function that is crashing:

@capture_for_replay
def generate_invoice(customer_data: dict, line_items: list) -> dict:
    customer_id = customer_data["customer_id"]  # This is line 147
    # ... rest of invoice logic
    return {"invoice_id": "INV-123", "total": 0}

When KeyError: "customer_id" fires in production, you get a JSON file that looks like this:

{
  "timestamp": "2026-08-15T14:32:11+00:00",
  "function": "generate_invoice",
  "module": "billing.invoice_service",
  "args": [
    {},
    [{"sku": "PRO-1", "price": 99.0}]
  ],
  "kwargs": {},
  "exception": {
    "type": "KeyError",
    "message": "'customer_id'"
  },
  "traceback": "..."
}

The first argument was an empty dictionary. That is the entire bug. The upstream caller passed {} instead of a customer record.

You now have a test case:

def test_generate_invoice_with_empty_customer():
    with pytest.raises(KeyError, match="customer_id"):
        generate_invoice({}, [{"sku": "PRO-1", "price": 99.0}])

This test fails before the fix and passes after you add input validation. More importantly, you did not have to guess. The crash told you exactly what to test.

Why capture misses dependencies it cannot see

This pattern captures function arguments, not global state. If generate_invoice reads from a database, calls an external API, or checks an environment variable, those values are not in args and kwargs. The replay will only work if your local environment happens to match.

You can extend the decorator to capture external dependencies explicitly:

@capture_for_replay
def generate_invoice(
    customer_data: dict,
    line_items: list,
    db_conn
) -> dict:
    tax_rate = db_conn.execute(
        "SELECT rate FROM tax_rates WHERE region = ?",
        (customer_data["region"],)
    ).fetchone()[0]
    # ...

But db_conn is a connection object. You cannot serialize it to JSON. What you can serialize is the query and the result. The more principled approach is to separate pure logic from side effects. Pass the query result as an argument, not the connection:

@capture_for_replay
def generate_invoice(
    customer_data: dict,
    line_items: list,
    tax_rate: float
) -> dict:
    # Pure function. All inputs are serializable.
    total = sum(item["price"] for item in line_items)
    total_with_tax = total * (1 + tax_rate)
    return {
        "invoice_id": "INV-123",
        "total": round(total_with_tax, 2),
    }

This is functional core, imperative shell. It makes capture trivial because there is no hidden state. Every input that matters is in the argument list.

The non-determinism problem you cannot capture away

Even with perfect input capture, some crashes are not reproducible. Race conditions depend on thread timing. Memory corruption depends on allocator state. External APIs return different data on every call. Random number generators produce different sequences unless you seed them.

If your crash is a race condition, replaying the same inputs on a single-threaded local test will not trigger it. You need the actual concurrency pattern, which means running the original threads, which means you have moved from “replay” to “distributed tracing plus load testing.” That is a different tool.

For most application-level bugs, input replay is enough. For heisenbugs, it is not. Know which one you are dealing with before you spend three hours trying to replay a timing issue.

Making capture operational in production

The decorator above writes to local disk. In production, you want these captures shipped to object storage or your error tracker. The integration is straightforward: replace the open(capture_path, "w") call with an S3 upload or an attachment to your Sentry issue.

Most teams should start with one function. Pick the service that crashes most often. Add the decorator. Wait for the next crash. When it arrives, you will have a JSON payload that turns a thirty-minute guessing session into a five-minute test write.

If you are already using Sentry, the Breadcrumbs feature captures some of this context automatically. For a custom approach, the pattern fits in thirty lines of Python and works in any runtime that supports decorators or middleware.

Capture one endpoint this week

Add capture to your highest-error endpoint this week. Not every endpoint. Not every function. One. When it crashes, write the test case from the capture before you fix the bug. Run the test, watch it fail, apply the fix, watch it pass.

That loop, from production crash to reproducible test case, is what separates debugging from archaeology.