Most Review Checklists Are Placebos

If your team has a code review checklist, there is a decent chance it lives in a wiki page nobody opens. It probably says things like “check for off-by-one errors” and “verify error handling.” Those are true statements. They are also too vague to change behavior.

A checklist that tells you to “check for bugs” is not a checklist. It is a reminder that bugs exist.

The checklist in a Fagan inspection serves a different purpose. It is not a list of things to worry about. It is a structured tool that directs the reviewer’s attention toward specific defect categories that have been observed in the actual codebase being reviewed. It is built from data, tailored to the artifact type, and used during mandatory individual preparation. When used correctly, it is one of the main reasons structured inspection catches three to four times as many defects as informal review.

What Makes a Checklist “Structured”

The word “structured” matters here. A structured checklist is not a longer checklist. It is a checklist with specific design properties.

First, it is derived from escaped defect data. The items come from bugs that actually made it to production, not from a generic best-practices document. If your last three incidents all involved race conditions in async cleanup, that category gets its own checklist item. If null dereferences have not been a problem in two years, that item gets removed.

Second, it is scoped to the artifact type being reviewed. Fagan inspections originally used different checklists for requirements documents, design documents, source code, and test plans. Each artifact has different defect categories. A design document checklist asks about interface consistency and coupling. A source code checklist asks about boundary conditions and resource cleanup. Mixing them together dilutes both.

Third, it is used during individual preparation, not during the meeting. Each inspector reviews the material alone, with the checklist, before the group ever convenes. The checklist shapes what each person sees when they read the code in isolation. It is not a shared reference document. It is a personal lens.

Fourth, it is short enough to be usable. A checklist with forty items is a catalog, not a tool. Fagan recommended roughly ten to fifteen items per artifact type. The constraint forces prioritization. You keep the categories that matter and discard the noise.

How to Build One From Your Own Data

The best way to build a structured checklist is to look at what has already gone wrong. Here is a practical process.

Start with your incident tracker, bug database, or postmortem notes. Pull the last twenty defects that escaped review and made it to production or testing. Categorize each one by root cause, not symptom. “Page crashed” is a symptom. “Missing null check after external API response” is a root cause.

Group the root causes into categories. You will probably find that 80 percent of your escaped defects fall into three to five categories. Those are your checklist items.

For each category, write a specific trigger question, not a vague reminder. “Check for nulls” is vague. “For every function that calls an external API, verify the response is validated before use” is a trigger question. It tells the reviewer exactly what to look for and where to look.

Here is a concrete example. Suppose your team ships Python services and you analyzed the last twenty production issues. You find this distribution:

  • 6 issues: missing error handling on external calls
  • 5 issues: database transaction boundaries wrong
  • 4 issues: off-by-one in pagination logic
  • 3 issues: race conditions in cached data
  • 2 issues: logging sensitive data

Your checklist should have five items, one for each category. Each item should be a trigger question tied to your specific patterns.

#!/usr/bin/env python3
"""Build a structured review checklist from escaped defect data."""

from collections import Counter
from dataclasses import dataclass
from typing import List


@dataclass
class EscapedDefect:
    id: str
    root_cause: str
    trigger_question: str


def build_checklist(defects: List[EscapedDefect], max_items: int = 10) -> List[str]:
    """Build a Fagan-style checklist from escaped defect data.

    Groups by root cause, sorts by frequency, and returns trigger
    questions for the top categories.
    """
    counts = Counter(d.root_cause for d in defects)
    top_causes = counts.most_common(max_items)

    # Map root causes back to their most representative trigger question
    cause_to_question = {}
    for d in defects:
        if d.root_cause not in cause_to_question:
            cause_to_question[d.root_cause] = d.trigger_question

    checklist = []
    for cause, count in top_causes:
        question = cause_to_question[cause]
        checklist.append(f"[{count}x] {question}")

    return checklist


# Example: escaped defects from a Python web service
DEFECTS = [
    EscapedDefect("BUG-101", "missing-external-error-handling",
                  "For every external API call, is the response validated before use?"),
    EscapedDefect("BUG-102", "missing-external-error-handling",
                  "For every external API call, is the response validated before use?"),
    EscapedDefect("BUG-103", "missing-external-error-handling",
                  "For every external API call, is the response validated before use?"),
    EscapedDefect("BUG-104", "transaction-boundary-error",
                  "Does every database write have the correct transaction scope?"),
    EscapedDefect("BUG-105", "transaction-boundary-error",
                  "Does every database write have the correct transaction scope?"),
    EscapedDefect("BUG-106", "pagination-off-by-one",
                  "For every pagination query, are the limit and offset tested at boundaries?"),
    EscapedDefect("BUG-107", "cache-race-condition",
                  "For every cached value, is there a clear invalidation or TTL strategy?"),
    EscapedDefect("BUG-108", "sensitive-data-in-logs",
                  "Does any log statement include user data, tokens, or PII?"),
]

if __name__ == "__main__":
    checklist = build_checklist(DEFECTS, max_items=5)
    print("Structured Review Checklist")
    print("=" * 40)
    for item in checklist:
        print(f"  [ ] {item}")

Run this script against your own bug data and you have a checklist that is tied to your actual failure modes, not someone else’s.

What the Review Session Actually Looks Like

With the checklist in hand, the review session has a specific rhythm.

During individual preparation, each inspector reads the material alone at roughly 100 to 150 lines per hour. They use the checklist as a lens, not a script. They do not work through the checklist item by item in order. They read the code naturally, and when they encounter a pattern that matches a checklist category, they pause and examine it carefully.

The checklist is not a replacement for reading. It is a pattern matcher for things your brain is likely to skip.

In the inspection meeting, the reader paraphrases the code aloud. When an inspector spots a potential defect, they raise it immediately. The checklist categories provide a shared vocabulary. Instead of saying “this looks wrong,” an inspector can say “this looks like a transaction boundary error, category two.” The moderator logs it. The author listens. Nobody proposes a fix.

The checklist is not referenced during the meeting. By that point, the inspectors have already internalized it. The meeting is for cross-checking what each person found in isolation.

Why Generic Checklists Fail

Most teams that try checklists give up because they use the wrong kind.

A generic checklist copied from a blog post has no emotional weight. It asks the reviewer to look for things that may never have happened in their codebase. The reviewer skims it, sees nothing familiar, and goes back to reading the diff the way they always have.

A structured checklist built from escaped defects is different. Every item represents a real incident that cost real time. The reviewer knows these bugs because they have seen the postmortems. The checklist connects review behavior to specific, memorable failures.

The other common failure is treating the checklist as a meeting tool instead of a preparation tool. If you pull out the checklist during the review meeting, it becomes a script for group skimming. Everyone reads the same item, looks at the same code, and converges on the same obvious observations. The power of the checklist is in individual preparation, where six people apply the same categories independently and find different things.

The Maintenance Burden Most Teams Ignore

A checklist is not a monument. It is a living document that rots faster than code.

If you add a checklist item every time something goes wrong but never remove one, you will have forty items in six months. At that point, reviewers start treating it as wallpaper.

Fagan recommended reviewing the checklist itself after every few inspections. Remove items that have not triggered a finding in the last five reviews. Add items only for new defect categories that have escaped to production. Keep the total under fifteen. If you cannot keep it short, you are not prioritizing.

Here is a lightweight script to prune a checklist based on historical finding data:

#!/usr/bin/env python3
"""Prune a review checklist: remove items that no longer trigger findings."""

from dataclasses import dataclass
from typing import List, Dict


@dataclass
class ChecklistItem:
    category: str
    trigger_question: str
    findings_last_5_reviews: int


def prune_checklist(items: List[ChecklistItem], min_findings: int = 1) -> List[ChecklistItem]:
    """Remove checklist items that have not produced findings recently.

    A Fagan-style checklist should be short enough to be usable.
    Items that sit idle waste attention.
    """
    kept = [item for item in items if item.findings_last_5_reviews >= min_findings]
    removed = [item for item in items if item.findings_last_5_reviews < min_findings]

    print(f"Kept {len(kept)} items, removed {len(removed)} items")
    for item in removed:
        print(f"  REMOVED: [{item.category}] {item.trigger_question}")

    return kept


# Example: current checklist with finding counts from the last 5 reviews
CURRENT_CHECKLIST = [
    ChecklistItem("missing-external-error-handling",
                  "For every external API call, is the response validated before use?", 4),
    ChecklistItem("transaction-boundary-error",
                  "Does every database write have the correct transaction scope?", 3),
    ChecklistItem("pagination-off-by-one",
                  "For every pagination query, are the limit and offset tested at boundaries?", 0),
    ChecklistItem("cache-race-condition",
                  "For every cached value, is there a clear invalidation or TTL strategy?", 1),
    ChecklistItem("sensitive-data-in-logs",
                  "Does any log statement include user data, tokens, or PII?", 0),
]

if __name__ == "__main__":
    pruned = prune_checklist(CURRENT_CHECKLIST, min_findings=1)
    print("\nActive checklist:")
    for item in pruned:
        print(f"  [ ] [{item.findings_last_5_reviews}x] {item.trigger_question}")

Schedule this review quarterly. A stale checklist is worse than no checklist because it trains reviewers to ignore the tool.

The Trade-Off: Data vs. Speed

Building a structured checklist from defect data takes time. You need accurate incident categorization. You need discipline to keep the list short. You need a process to keep it current.

A generic checklist takes five minutes to write and five weeks to become irrelevant.

The structured version is slower to create but faster to use. A reviewer with fifteen targeted trigger questions scans code more efficiently than a reviewer with forty generic reminders. The specificity saves time.

The real cost is organizational. You need a record of escaped defects that is detailed enough to extract root causes. Many teams do not have that. Their bug tracker has titles like “user report: page broken” and no follow-up analysis. Without root cause data, you cannot build a checklist from evidence. You can only copy someone else’s.

Frequently Asked Questions

What is a structured checklist-based review?

A review process where each inspector uses a checklist, built from actual escaped defect data, during mandatory individual preparation before a group inspection meeting. The checklist contains targeted trigger questions rather than generic reminders.

How is this different from a normal code review checklist?

A normal checklist is often generic, copied from a template, and referenced casually during review. A structured checklist is derived from your specific defect history, scoped to the artifact type, and used during focused individual preparation at a defined reading rate.

How many items should a structured checklist have?

Ten to fifteen items per artifact type. More than that and reviewers start skimming. Fewer than five and you are probably missing categories.

How often should the checklist be updated?

After every three to five inspections, or quarterly. Remove items that have not produced findings. Add items only for new escaped defect categories.

Can this work without a full Fagan inspection process?

Yes. The checklist itself is the most portable part of the Fagan method. Require individual preparation, give reviewers a structured checklist built from your data, and run a time-boxed meeting with a reader. You will catch more defects than with informal review even without the full six-phase ceremony.

Start With One Module and One Quarter

You do not need to overhaul your entire review process. Pick one module that has had production defects in the last six months. Gather the root causes. Build a five-item checklist. Require two reviewers to spend thirty minutes with the code and the checklist before any group discussion.

Log what they find. Compare it to what your normal pull request review caught on the same code. That single comparison will tell you whether a structured checklist is worth the effort for your team.

If the data says yes, expand to one more module. If the data says no, your informal review is already good enough, or your defect data is not detailed enough to build a useful checklist yet. Either way, you have a real measurement instead of a borrowed opinion.