Why Property Testing Finds Bugs Unit Testing Does Not (2021)

Summary

April 1, 2021 Why Property Testing Finds Bugs Unit Testing Does Not I intended this newsletter to be my thoughts without editing, and I have a new thought, so here goes. I want to respond to this discussion: (Standard disclaimer, don't brigade. It's a civil discussion and Brian is a friend, but even if both weren't the case, brigading is chillul Hashem) For those of you just joining us, Property-Based Testing is when, instead of giving specific input-outputs in your test, you describe the general properties the execution should have. Then you try it on randomized values. A simple example is testing that addition is commutative: def add(a, b): return a + b @given(integers(), integers()) def test_add_is_commutative(a, b): assert add(a,b) == add(b,a) (I'm using pytest and hypothesis for everything.) You can read a deeper treatment of PBT here. PBT is powerful because it lets you test a wider range of inputs, but you have to learn how to come up with properties and how to do complex strategies (input generators), which are both skills. Anyway, Brian's argument in the tweets is roughly this: The majority of errors you find with testing are either issues with an entire "partition" of inputs or "boundary" inputs, like INT_MIN. For partition errors, you can pick any random issue and find it, in which case you might as well just write a unit test for that input. For boundary errors, those arise from the specification of the problem, and to handle those, it's necessary to think hard about the problem and what the unexpected cases will be, regardless of how you plan to test. In which case PBT doesn't provide sufficient benefit over manual unit tests. I don't think that's exactly his line of reasoning, but I think it's close enough that my response to this version should be compatible with the response to his intended argument. The value PBT has over manual unit testing is in the combinatorial explosion of boundary conditions and edge cases in even a moderately-complex problem....