Elm Test Distributions

Summary

…in which I’ll tell you how you can make sure your property based tests are testing the interesting cases. Recently I was discussing a TigerBeetle article on swarm testing with Jeroen Engels on the Elm Slack, and at one point, reading the paragraph: For example, one weakness of our test above is that we chose to pop and push with equal probability. As a result, our queue is very short on average. We never exercise large queues! He asked: How does one detect which situations are or aren’t covered in practice by property-based tests? Like, when would you say “the distribution we have doesn’t cover this case”? How do you indeed! You could use Fuzz.examples to visually check whether the generated values make sense to you: -- inside Elm REPL > import Fuzz > Fuzz.examples 10 (Fuzz.intRange 0 10) [4,6,3,6,9,9,9,3,3,6] : List Int but did you just get unlucky and saw no 0 and 10, or do they never get generated? To build the motivation a little bit, let’s try and see the issue from the TigerBeetle blogpost. Assume we have a Queue implementation (the details don’t matter): type Queue a empty : Queue a push : a -> Queue a -> Queue a pop : Queue a -> (Maybe a, Queue a) length : Queue a -> Int Now let’s try to test it! type QueueOp = Push Int | Pop queueOpFuzzer : Fuzzer QueueOp queueOpFuzzer = Fuzz.oneOf [ Fuzz.map Push Fuzz.int , Fuzz.constant Pop ] applyOp : QueueOp -> Queue Int -> Queue Int applyOp op queue = case op of Push n -> Queue.push n queue Pop -> Queue.pop queue |> Tuple.second queueFuzzer : Fuzzer (Queue Int) queueFuzzer = Fuzz.list queueOpFuzzer -- would generate [ Push 10, Pop, Pop, Push 5 ] etc. |> Fuzz.map (\ops -> List.foldl applyOp Queue.empty ops) -- instead generates a queue with the ops applied The queueFuzzer makes a sort of random walk through the ops to arrive at a random Queue. Now if we were worried we’re not testing very interesting cases, we could debug-print their lengths and look at the logs real hard and make a gut decision about whether it’s fine...