Property-Based Testing skill for Claude Code

A property test states something that must hold for every valid input and lets the library hunt for a counterexample. Done well, properties are true statements about the domain (not a re-implementation of the code), generators produce valid inputs without heavy filtering, and every failure is replayable from its seed.

Ground rules

• Keep example-based tests for known cases and documentation. Add properties to find the cases nobody thought of.
• A property that re-implements the function under test proves nothing. Prefer relationships: round-trips, invariants, comparisons to a simple oracle.
• Build generators that produce valid inputs by construction. If more than ~10% of inputs are discarded by filter/assume/fc.pre, rewrite the generator.
• When a property fails, read the shrunk counterexample first, then add it as a permanent example test before fixing.
• Fix seeds only for replay. Normal runs use random seeds so coverage grows over time.

Workflow

- [ ] 1. Pick the function/module and write down candidate properties
- [ ] 2. Build generators for the input domain
- [ ] 3. Write properties; start with numRuns default (100)
- [ ] 4. Run; triage each counterexample (bug in code, property, or generator)
- [ ] 5. Pin counterexamples as example tests
- [ ] 6. Tune runs for CI and add stateful tests for stateful components

1. Choosing properties

Pattern	Statement	Good for
Round-trip	decode(encode(x)) == x	Serializers, parsers, codecs, URL builders
Oracle	fast(x) == slowButObvious(x)	Optimized algorithms, rewrites, caches
Invariant	Output always satisfies P (sorted, within bounds, sums preserved)	Transformations, pricing, allocation
Idempotence	f(f(x)) == f(x)	Normalizers, formatters, upserts, migrations
Metamorphic	Changing input in a known way changes output predictably	Search ranking, ML, geometry, filtering
Commutativity / order independence	Result independent of input order	Aggregations, merges, CRDTs
No crash	Any input yields a result or a typed error, never an exception	Parsers and public APIs fed untrusted data

More patterns with worked examples are in references/property-patterns.md; read it when the obvious properties above do not fit.

2–3. fast-check

import fc from 'fast-check';
import { describe, it, expect } from 'vitest';

const money = fc.integer({ min: 0, max: 10_000_000 });          // cents
const lineItem = fc.record({ sku: fc.stringMatching(/^[A-Z]{3}-\d{3}$/), qty: fc.integer({ min: 1, max: 99 }), price: money });
const cart = fc.array(lineItem, { minLength: 0, maxLength: 20 });

describe('splitPayment', () => {
  it('parts always sum to the total and differ by at most one cent', () => {
    fc.assert(
      fc.property(money, fc.integer({ min: 1, max: 12 }), (total, parts) => {
        const shares = splitPayment(total, parts);
        expect(shares.reduce((a, b) => a + b, 0)).toBe(total);
        expect(Math.max(...shares) - Math.min(...shares)).toBeLessThanOrEqual(1);
      }),
    );
  });

  it('serializes carts losslessly', () => {
    fc.assert(fc.property(cart, (c) => { expect(parseCart(serializeCart(c))).toEqual(c); }));
  });
});

• Integration helper: @fast-check/vitest (and @fast-check/jest) provide test.prop([fc.integer()])('name', (n) => ...).
• Derive generators with .map (transform) and .chain (dependent values: generate a length, then an index within it). Use fc.constantFrom, fc.oneof, fc.option, fc.uniqueArray, fc.date({ noInvalidDate: true }).
• Failure output includes seed and path; replay exactly with fc.assert(prop, { seed: <seed>, path: '<path>', endOnFailure: true }).

2–3. Hypothesis

from hypothesis import given, example, settings, strategies as st

money = st.integers(min_value=0, max_value=10_000_000)

@given(total=money, parts=st.integers(min_value=1, max_value=12))
def test_split_payment_sums_to_total(total, parts):
    shares = split_payment(total, parts)
    assert sum(shares) == total
    assert max(shares) - min(shares) <= 1

@st.composite
def date_ranges(draw):
    start = draw(st.dates())
    length = draw(st.integers(min_value=0, max_value=400))
    return start, start + timedelta(days=length)

@given(date_ranges())
@example((date(2024, 2, 28), date(2024, 3, 1)))   # pinned leap-year counterexample
def test_nights_never_negative(rng):
    assert nights(*rng) >= 0

• st.builds(Model, field=strategy) and st.from_type(SomeDataclass) generate typed objects; override only the fields that need constraints.
• Settings profiles for CI vs. local:

# conftest.py
import os
from hypothesis import settings
settings.register_profile("ci", max_examples=1000, deadline=None)
settings.register_profile("dev", max_examples=100)
settings.load_profile(os.getenv("HYPOTHESIS_PROFILE", "dev"))

• Hypothesis stores failing examples in .hypothesis/ and replays them first on the next run. Commit nothing from it; add @example(...) for permanent pins. A failure prints a @reproduce_failure decorator for exact replay.
• pytest --hypothesis-show-statistics reports generation and discard rates.

4. Triage a counterexample

For each failure, decide which is wrong:

1. The code: a real bug. Pin the counterexample as an example test, then fix.
2. The property: it states something the domain does not promise (e.g. round-trip of floats through a lossy format). Narrow the property or document the exception.
3. The generator: it produced inputs outside the valid domain (NaN where the API forbids it). Constrain the generator, not the code.

Shrinking gives the minimal failing input; do not “fix” the specific value, fix the class of inputs it represents.

6. Stateful (model-based) testing

For components with operations and state (caches, queues, carts, connection pools), generate sequences of commands and compare the real system against a simple model.

from hypothesis.stateful import RuleBasedStateMachine, rule, invariant
import hypothesis.strategies as st

class LRUCacheMachine(RuleBasedStateMachine):
    def __init__(self):
        super().__init__()
        self.real = LRUCache(capacity=3)
        self.model = {}            # plus an order list for recency
        self.order = []

    @rule(k=st.integers(0, 5), v=st.integers())
    def put(self, k, v):
        self.real.put(k, v)
        self.model[k] = v
        self.order = [x for x in self.order if x != k] + [k]
        if len(self.order) > 3:
            evicted = self.order.pop(0)
            del self.model[evicted]

    @rule(k=st.integers(0, 5))
    def get(self, k):
        assert self.real.get(k) == self.model.get(k)
        if k in self.model:
            self.order = [x for x in self.order if x != k] + [k]

    @invariant()
    def size_bounded(self):
        assert len(self.real) <= 3

TestLRUCache = LRUCacheMachine.TestCase

fast-check equivalent: define command classes with check(model), run(model, real), and toString(), then fc.assert(fc.property(fc.commands([...]), (cmds) => fc.modelRun(setup, cmds))).

Run budgets

Context	Runs per property
Local / PR	100 (default)
Nightly or CI profile	1,000–10,000
Expensive property (I/O, >10 ms per run)	20–50 and move to a nightly job

Disable Hypothesis deadlines (deadline=None) for tests with variable runtime instead of letting them flake.

Gotchas

• fc.float() and st.floats() include NaN, infinities, and -0.0 by default. Exclude them explicitly if the domain does, or test that they are rejected.
• Generated strings include Unicode edge cases (combining marks, surrogates in JS); great for parsers, noisy for unrelated fields. Use constrained generators there.
• Tests that touch a shared database across runs must reset state per example, or shrinking produces nonsense.
• Properties with randomness inside the code under test are not reproducible; inject the RNG.
• Large generated collections slow everything; bound maxLength/max_size.

Verification

1. Introduce a known bug (off-by-one in the split, drop the last item in serialization) and confirm a property fails with a small shrunk counterexample.
2. Check discard rate is low (--hypothesis-show-statistics, or fast-check verbose).
3. Each historical counterexample exists as a pinned example test.