The Number Nobody Wants to Talk About
Informal code review catches between 15 and 30 percent of the defects present in the code being reviewed. That is not an opinion. It is a finding that has been replicated across four decades, multiple companies, and dozens of studies.
Michael Fagan documented it at IBM in 1976. A 1987 study at AT&T Bell Labs found 20 percent. A 1996 HP study found 25 percent. A 2013 Microsoft Research paper on modern code review found roughly the same range. The tools changed from punch cards to GitHub, but the human performance curve did not move.
If your team thinks pull request review is your quality backstop, the data says you are catching roughly one bug in four. The other three are shipping.
Where the Numbers Come From
Fagan’s original methodology was simple and brutal. He injected known defects into code, ran the review process, and counted how many the reviewers found. Then he compared that to the total defects the team eventually found through testing, production incidents, and customer reports. The ratio of found-in-review to total-defects became the removal rate.
The key insight is that the denominator matters. A review that finds ten defects sounds good until you learn there were fifty in the file. Fagan measured the full denominator. Most teams today do not.
Subsequent studies used similar designs. Researchers seeded defects, compared review types, or traced defects backward from production to see where they could have been caught. The results cluster tightly:
| Review type | Defect removal rate | Key studies |
|---|---|---|
| No review | 0% | Baseline |
| Informal / PR review | 15-30% | Fagan 1976, Porter 1995, Microsoft 2013 |
| Structured walkthrough | 30-50% | Yourdon 1979, Weller 1993 |
| Fagan inspection | 60-90% | Fagan 1976, Russell 1991, NASA 2002 |
The gap between informal and structured review is not small. It is a 3x difference in defect escape rate.
Why PR Review Performs So Poorly
The problem is not that reviewers are bad at their jobs. The problem is that pull request review is not designed to find defects. It is designed to let two people agree that code can merge.
Here is what actually happens in a typical PR review. The reviewer opens the diff. They read the summary, scan the additions, check that the tests pass, and look for anything obviously wrong. This takes five to fifteen minutes. Then they approve.
The process is optimized for speed, not thoroughness. There is no preparation time. The reviewer has not read the surrounding code, traced the data flow, or built a mental model of the change. They are reacting to a diff on a screen, and human brains are terrible at finding bugs in that format.
A 2015 study by Bacchelli and Bird at Microsoft found that the most common review comment categories were not defect-related at all. The top categories were questions about intent, requests for clarification, and suggestions for improvement. Actual defect findings were a minority of comments. The tool was functioning as a communication channel, not a quality gate.
This is fine if you know what the tool does. It is dangerous if you think it does something else.
What Structured Review Does Differently
Fagan inspections, and other structured review methods, achieve higher removal rates by changing the process design, not the people.
The biggest lever is individual preparation. In a Fagan inspection, every reviewer spends focused time with the material before any group discussion. Fagan found that prepared inspectors found roughly twice as many defects as those who walked in cold. The PR model is the cold-walk-in model by default.
The second lever is pace. Fagan recommended 100 to 125 lines of code per hour of review. Most PR reviewers process ten times that rate. Speed kills detection. Your brain fills in expected patterns instead of reading what is actually there.
The third lever is focus. Fagan meetings have a single purpose: log defects. No design debates. No solution proposals. No style discussions. PR threads routinely drift into architecture opinions, which consumes the same cognitive budget that could have found a null dereference.
The fourth lever is the reader role. Having someone who did not write the code paraphrase it aloud forces the group to process at comprehension speed instead of skimming speed. A diff on a screen lets your eyes skip the boring parts. A person speaking does not skip.
The Cost Argument Is Backwards
The usual objection to structured review is cost. A Fagan inspection consumes four to six person-hours for a few hundred lines of code. A PR review consumes fifteen minutes of one person’s time. The math looks obvious.
It is obvious, and it is wrong.
Fagan also measured the cost of finding defects at different stages. A defect found during inspection cost about one-tenth what it cost to find during testing. When it escaped to production, the ratio grew to twenty or thirty to one. The 2002 NASA study found that every hour spent in inspection prevented an average of 33 hours of maintenance work later.
The savings are invisible. You cannot measure the bug you prevented. The cost of the meeting is immediate and obvious. That is why organizations optimize for visible speed over invisible quality, even when the data says it costs them more in the end.
A Data-Driven Middle Ground
You do not need IBM’s meeting culture to get most of the benefit. You need to borrow the process features that actually move the number, and drop the ones that do not.
Here is what the data says matters:
-
Preparation time. Require reviewers to spend time with the code before commenting. Even ten minutes of focused reading beats a skim.
-
Pace limits. For critical files, enforce a maximum review speed. If a reviewer approves a 500-line change in five minutes, that is data, not diligence.
-
Defect-only focus. Separate style and architecture feedback from defect hunting. Use automated formatters for the former. Reserve human attention for the latter.
-
Checklists. Fagan found that checklist-driven reviewers found defects that intuition-driven reviewers walked past. The checklist exists because expertise creates blind spots.
Here is a lightweight script that measures review depth from Git history. It estimates whether a review had enough time to be thorough:
#!/usr/bin/env python3
"""Estimate review depth from git history."""
import subprocess
import sys
from datetime import datetime, timezone
def get_commit_info(commit_hash: str) -> dict:
"""Return author, committer, and timestamps for a commit."""
fmt = "%H|%an|%cn|%ad|%cd"
result = subprocess.run(
["git", "log", "-1", f"--format={fmt}", commit_hash],
capture_output=True,
text=True,
check=True,
)
parts = result.stdout.strip().split("|")
return {
"hash": parts[0],
"author": parts[1],
"committer": parts[2],
"author_date": datetime.strptime(parts[3], "%a %b %d %H:%M:%S %Y %z"),
"commit_date": datetime.strptime(parts[4], "%a %b %d %H:%M:%S %Y %z"),
}
def get_lines_changed(commit_hash: str) -> int:
"""Count total lines added + deleted in a commit."""
result = subprocess.run(
["git", "diff", f"{commit_hash}^", commit_hash, "--stat"],
capture_output=True,
text=True,
check=True,
)
# Last line of --stat contains totals like "3 files changed, 42 insertions(+), 7 deletions(-)"
for line in reversed(result.stdout.strip().splitlines()):
line = line.strip()
if "insertions" in line or "deletions" in line:
# Extract numbers roughly
parts = line.split(",")
total = 0
for part in parts:
digits = "".join(ch for ch in part if ch.isdigit())
if digits:
total += int(digits)
return total
return 0
def estimate_review_depth(commit_hash: str) -> dict:
"""Estimate whether a commit had time for thorough review.
Returns lines changed, time between author and commit dates
(a rough proxy for review duration), and a verdict.
"""
info = get_commit_info(commit_hash)
lines = get_lines_changed(commit_hash)
# Time between author date and commit date is a proxy for review time
# In many workflows, commit date reflects when the merge happened
review_seconds = (info["commit_date"] - info["author_date"]).total_seconds()
review_hours = review_seconds / 3600
# Fagan recommended 100-125 lines/hour for thorough review
fagan_rate = 125
needed_hours = lines / fagan_rate if lines else 0
verdict = "insufficient"
if review_hours >= needed_hours:
verdict = "adequate"
if review_hours >= needed_hours * 2:
verdict = "thorough"
return {
"hash": commit_hash[:8],
"lines": lines,
"review_hours": round(review_hours, 2),
"needed_hours": round(needed_hours, 2),
"verdict": verdict,
}
if __name__ == "__main__":
commit = sys.argv[1] if len(sys.argv) > 1 else "HEAD"
result = estimate_review_depth(commit)
print(f"Commit: {result['hash']}")
print(f"Lines: {result['lines']}")
print(f"Review time: {result['review_hours']} hours")
print(f"Fagan time: {result['needed_hours']} hours")
print(f"Verdict: {result['verdict']}")Save it as review_depth.py and run:
python review_depth.py abc1234The output will tell you whether a commit had enough review time to be thorough by Fagan’s standard. Most commits will say insufficient. That is the point. The data has been telling us this for decades, and we keep building faster pipelines instead of better ones.
What This Means for Your Team
Pull request review is not useless. It builds shared context, spreads knowledge, and catches obvious mistakes. But the data is clear about what it does not do. It does not catch most defects.
If your quality strategy depends on PR review as a primary filter, you are filtering with a sieve. The 15-30% removal rate is not a failing of your reviewers. It is a property of the process.
The teams that beat this number do not hire smarter people. They change the process. They add preparation time, enforce pace limits, separate defect finding from design discussion, and use checklists to direct attention where intuition misses.
You do not need a full Fagan inspection for every diff. You do need to know what your current process actually achieves, and stop pretending it achieves more.
FAQ
What does the data say about pull request review effectiveness?
Multiple studies across IBM, AT&T, HP, and Microsoft consistently find that informal code review catches 15-30% of defects present in the code. This range has remained stable from the 1970s through modern research on GitHub-based workflows.
Why do pull request reviews catch so few defects?
PR review is optimized for speed and merge approval, not systematic defect detection. Reviewers typically spend 5-15 minutes per review, have no preparation time, process code at 10x the rate that research recommends, and the format encourages skimming rather than deep analysis.
How much better are structured reviews like Fagan inspections?
Fagan inspections consistently report 60-90% defect removal rates, roughly 3-4x better than informal review. The difference comes from individual preparation, enforced pace limits, role separation, checklist-driven focus, and meetings that log defects instead of debating solutions.
What is the cheapest way to improve PR review effectiveness?
Require individual preparation before commenting, use checklists tailored to your team’s common defect types, separate style feedback from defect hunting, and limit review speed for critical files. Even small changes to preparation and focus can move the needle significantly without adding meeting overhead.
How do I measure my team’s actual defect removal rate?
Track defects found in review versus defects found later in testing or production. The ratio of caught-early to total-found is your removal rate. Most teams do not track this, which is why they overestimate review effectiveness. Start logging where each defect was found, then calculate the ratio monthly.