Pin the seam with a regression test

Level 5: Judging Code You Did Not Writehard40 mintest designregression testsminimal reproductiondebugging

Turn a vague bug report into a regression test that fails on the old build of the right module, then make the smallest fix that turns it green.

A fix without a red test is a rumor

You have already written probe cases against a snippet: pick an input, predict the output, run it, compare. In a repo that instinct needs one extra decision, because now you also choose where the test lives. Put it in the wrong layer and it either needs a page of setup or stays green while the bug is still there.

The claim here is narrow. A fix with no red test behind it is a rumor. You believe the bug is gone, and nothing in the repo will tell you when it comes back.

From vague report to seam

The seam is the narrowest function where the reported behavior still reproduces. Finding it is a descent, not a search:

Reproduce at the layer the report describes, usually the endpoint or the screen.
Step down one call, feed it the same shaped input, and check whether the wrong behavior is still there.
Keep going until the behavior disappears. The last layer that still shows it is your seam.

This is the shrink habit pointed at the call chain instead of the input. Once you have the seam, compare what the two candidate tests cost. Say a report claims a new record sometimes overwrites an old one, and the id is chosen by a small helper:

# At the endpoint: the report's own words, plus three things that can break
payload = create_record(db_rows, {"name": "Ada", "team": "core"})
assert payload["record"]["id"] == 6

# At the seam: one call, one behavior
assert ids.next_id([1, 2, 5]) == 6

The endpoint test drags request parsing, defaults and response shape into an id bug. Change the response format next quarter and it goes red for a reason unrelated to the bug it was written for, so the next reader deletes it. The seam test survives that change, because it only ever claimed one thing.

Check yourself

A duplicate id bug reproduces through the create-record endpoint, and it still reproduces when you call the small next_id(existing_ids) helper directly. Where does the regression test belong?

At the endpoint, because that is what the report describesOn next_id, the smallest function that still shows the bugAt both levels, so nothing can slip throughIn a manual QA checklist, since the fix is small

Red first, then green

Now the ritual. Write the test, run it against the code exactly as it is, and watch it fail. That red run is the only evidence the test detects anything at all. Reverse the order and you get a test that agrees with whatever the code currently does, bug included.

This is the failure mode that AI workflows industrialize. Ask for a fix and you usually get the patch and a matching test in one response. The test was written from the patched code, so it has never been red, and it often asserts the shape of the new implementation rather than the contract. It looks like coverage. It proves nothing.

Both workspaces in this lesson enforce the order for you. An audit suite runs your regression test twice and refuses to pass until your suite fails on the old build.

Check yourself

Why must a regression test be run against the unfixed code and seen to fail before you make the fix?

Because a test that has never failed proves nothing about what it detectsBecause red before green is the convention in most style guidesBecause a test written first lifts the coverage number moreBecause only subtle bugs need this, and this one is subtle

Why the audit swaps builds

The audit keeps a frozen copy of the buggy function and installs it with a module attribute assignment, then restores the original in a finally block. That is the same mechanism pytest's monkeypatch uses on real jobs, and it has one consequence you have to respect while writing the test:

from search import pagination

pagination.page(items, 2, 3)   # looked up on the module at call time, sees the swap

from search.pagination import page

page(items, 2, 3)              # a name bound once at import, never sees the swap

A from-import copies the function object into your module's namespace. Rebinding the attribute afterwards does not reach back and change your copy, so your test would grade the same build twice. Call through the module and your test runs whichever build the audit installed.

Check yourself

The audit swaps pagination.page between the old build and the current one by assigning to the module attribute. Which test style sees the swap?

from search import pagination, then call pagination.page(...)from search.pagination import page, then call page(...)Copy the function body into the test file and call the copyEither style, because Python imports are live references

The smallest fix

With the pin in place you can repair with more confidence, because the test you just watched go red is now watching you. Still make the smallest change that turns it green, and let the rest of the suite tell you whether you reached too far. A pin plus a small diff is a fix you can defend in review.

Pitfalls

Testing through the endpoint because that is where the report lives. Heavy setup, and it goes red later for unrelated reasons.
Binding the function with a from-import, so a build swap or a monkeypatch never reaches your test.
Writing the test after the fix. It passes on the first run, which feels good and demonstrates nothing.
Deleting the worked example test to make room for yours. Add alongside it; a suite that knows one bug is thinner than it looks.

Interview nuance: in a debugging round the follow-up is almost always "how do you know your test would have caught this?", and the only honest answer is "I watched it fail on the unfixed code". Say that and you have shown your process rather than your patch. Import binding is its own common screen question, because a candidate who has monkeypatched something for real knows why the from-import version quietly does nothing.

Check yourself

A teammate ships the fix first, then adds a passing test for it in the same pull request. The fix itself is correct. What risk remains?

The test may describe the new implementation rather than the contract, and would have passed before the fix tooNo risk, because a correct fix makes the test redundantThe extra test slows the build downThe test might turn out to be flaky

Worked example (Python)

# The red-green ritual in miniature: one pin, two builds of the same helper.
# The report: a new record sometimes overwrites an old one.
EXISTING = [1, 2, 5]


def old_build_next_id(existing_ids):
    return len(existing_ids) + 1


def current_build_next_id(existing_ids):
    return max(existing_ids) + 1


def pin(next_id):
    """The regression test: a gap in the ids must not hand back a used id."""
    got = next_id(EXISTING)
    assert got == 6, f"expected 6 after {EXISTING}, got {got}"


for label, build in [("old build", old_build_next_id), ("current build", current_build_next_id)]:
    try:
        pin(build)
        print(f"{label}: PASS")
    except AssertionError as exc:
        print(f"{label}: FAIL (expected) {exc}")

print()
print("A pin that passes on both builds is not a regression test, it is a coincidence.")

Apply

Your turn

The task this lesson builds to.

Write the regression test in tests/test_regression.py that fails on the old build of search/pagination.py, then fix page so every suite passes. Call the function as pagination.page(...) so the audit suite can swap builds under it. Some tests are hidden.

3 hints and 4 automated checks are waiting in the workspace.

Practice

Make it stick

A second problem on the same idea, so it survives past today.

The nightly export job sometimes drops the last few rows and nobody can say when it started. Write the regression test in tests/test_regression.py that fails on the old build of export/batching.py, then fix chunks so all suites pass. Call it as batching.chunks(...) so the audit can swap builds. Some tests are hidden.

3 hints and 2 automated checks are waiting in the workspace.