Mutation Testing: Measuring Test Suite Quality

The Coverage Lie

You have 95% code coverage. Your CI badge is green. You feel safe. But here is a disturbing experiment: go into your codebase and change a > to >=. Does any test fail? Change && to ||. Does any test notice? If the answer is no, your tests are executing code without actually checking it.

Code coverage tells you which lines were run. It says nothing about whether your assertions would catch a bug. Mutation testing fills this gap by systematically introducing bugs and checking whether your tests detect them.

How Mutation Testing Works

The process has four steps:

1. Generate mutants. The tool modifies your source code in small ways. Each modification is called a mutant. Examples: replacing + with -, changing true to false, removing a function call, swapping === for !==.

2. Run tests against each mutant. For each mutant, the tool runs your test suite.

3. Classify results. If a test fails, the mutant is killed (good — your tests caught the fake bug). If all tests pass, the mutant survived (bad — your tests missed a real code change).

4. Calculate the mutation score. The percentage of killed mutants: killed / total * 100. A mutation score of 85% means 85% of introduced bugs were caught by your tests.

Common Mutation Operators

Mutation tools use a set of operators to generate mutants:

Operator	Original	Mutant
Arithmetic	a + b	a - b
Relational	a > b	a >= b
Logical	a && b	a || b
Negation	if (ready)	if (!ready)
Return value	return x	return 0
Removal	doSomething()	(removed)
String	"hello"	""

Each operator targets a specific class of bugs. Together, they simulate the kinds of mistakes developers actually make.

Stryker: Mutation Testing for JavaScript/TypeScript

Stryker is the leading mutation testing tool for JavaScript and TypeScript projects.

Installation

npm install --save-dev @stryker-mutator/core
npx stryker init

The init command creates a stryker.config.mjs file. Here is a typical configuration:

// stryker.config.mjs
export default {
  mutate: ["src/**/*.ts", "!src/**/*.test.ts"],
  testRunner: "jest",
  reporters: ["html", "clear-text", "progress"],
  coverageAnalysis: "perTest",
  thresholds: {
    high: 80,
    low: 60,
    break: 50,
  },
};

Running Stryker

npx stryker run

Stryker generates mutants, runs your tests against each one, and produces a report. The HTML report is especially useful — it highlights surviving mutants inline in your source code.

Reading the Report

The clear-text output looks like this:

Mutant survived:
  src/discount.ts:5:23
  Replaced > with >= in: if (price > 100)
  No test failed for this mutant.

This tells you that changing price > 100 to price >= 100 did not cause any test to fail. You need a test that specifically checks the behavior at the boundary (price === 100).

Fixing a Surviving Mutant

Add the missing boundary test:

test("price exactly 100 gets no discount", () => {
  expect(calculateDiscount(100, true)).toBe(100);
});

test("price above 100 gets 10% discount", () => {
  expect(calculateDiscount(101, true)).toBe(90.9);
});

Now the > to >= mutant will be killed because the boundary test distinguishes between the two operators.

mutmut: Mutation Testing for Python

For Python projects, mutmut is a popular choice:

pip install mutmut

Running mutmut

mutmut run --paths-to-mutate=src/

View results:

mutmut results

Inspect a specific surviving mutant:

mutmut show 42

This displays the exact code change that survived, so you can write a test to kill it.

Example Output

--- src/cart.py
+++ src/cart.py (mutant 42)
@@ -12,7 +12,7 @@
 def apply_discount(total, discount_pct):
-    if discount_pct > 0 and discount_pct <= 100:
+    if discount_pct > 0 and discount_pct < 100:
         return total * (1 - discount_pct / 100)
     return total

The mutant changed <= to <. If no test uses discount_pct=100, this mutant survives.

Dealing with Surviving Mutants

Not every surviving mutant indicates a real problem. Here are strategies for handling them:

Write the missing test

Most surviving mutants point to genuine gaps in your test suite. The fix is straightforward: write a test that exercises the boundary or condition the mutant modified.

Equivalent mutants

Some mutants produce code that behaves identically to the original. For example, changing the order of an addition (a + b becomes b + a) produces an equivalent mutant that cannot be killed. These are false positives. Mark them as ignored in your configuration.

Infinite loop mutants

Some mutants create infinite loops (for example, removing a loop increment). Stryker handles this with a timeout. If a mutant causes the test to hang, it is classified as timed out and counted as killed.

Trivial mutants

Sometimes a mutant changes dead code or logging. You can exclude these files from mutation:

// stryker.config.mjs
export default {
  mutate: [
    "src/**/*.ts",
    "!src/**/*.test.ts",
    "!src/logger.ts",        // exclude logging
    "!src/config.ts",        // exclude config
  ],
};

Performance Considerations

Mutation testing is inherently slow because it runs your entire test suite once per mutant. A project with 500 mutants and a 30-second test suite would take over 4 hours.

Strategies to speed it up:

• Incremental mutation testing. Only mutate files that changed since the last run. Stryker supports this with --incremental.
• Coverage-based filtering. Only run the tests that cover the mutated code, not the entire suite. Stryker does this with coverageAnalysis: "perTest".
• Focus on critical code. Do not mutate everything. Focus on business logic, not boilerplate.
• Run overnight in CI. Mutation testing does not need to be in your fast feedback loop. Run it nightly or weekly.

# Incremental run in CI
npx stryker run --incremental

Setting Thresholds

You can configure Stryker to fail the build if the mutation score drops below a threshold:

thresholds: {
  high: 80,   // green badge
  low: 60,    // yellow badge
  break: 50,  // fail the build below this
}

Start with a low break threshold and increase it gradually as you improve your tests.

Mutation Testing in CI/CD

Add mutation testing to your CI pipeline, but keep it separate from your fast test stage:

# .github/workflows/mutation.yml
name: Mutation Testing
on:
  schedule:
    - cron: "0 2 * * 1"  # Every Monday at 2 AM
  workflow_dispatch:

jobs:
  mutate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npx stryker run --incremental
      - uses: actions/upload-artifact@v4
        with:
          name: mutation-report
          path: reports/mutation/

Key Takeaways

Code coverage measures execution. Mutation testing measures detection. A high mutation score means your tests actually verify behavior, not just run code. Start with Stryker for JavaScript/TypeScript or mutmut for Python. Focus on business-critical code, use incremental runs to manage performance, and treat surviving mutants as a guide for where to add more precise assertions. Aim for progress over perfection — even going from 0% to 60% mutation score reveals the most dangerous gaps in your test suite.