Wiggling into Correlation

Summary

Jeff Kaufman shared some data around contra dance attendance as a function of requirements on wearing surgical masks. He compares this data to survey data, which is a useful way to validate in both directions. I found the plot compelling for a different reason – depending on how we look at it, we can draw wildly different conclusions from it. On the one hand, if we draw boxes around consecutive pairs of dances, it’s fairly obvious that mask-optional dances are more popular. Tickmarks at the top indicate pairs that support the hypothesis.1 Some dances had to be excluded from pairs because there are not equal numbers of both kinds. I decided to mechanically pick “the next pair” whenever there were runs of the same type of dance, which means no bias was introduced by cleverly selecting pairs, even if that particular mechanic may result in more extreme results than some other mechanic. It doesn’t require much statistical literacy to recognise that 18/20 successes is statistically significant.2 For completeness, the Agresti–Coull (“add two successes and two failures”) confidence interval is 68.1–98.5 %, well clear of the null hypothesis of 50 %. I would entertain arguments that this is a sensible procedure, because it’s likely dances close in time share other common factors that affect popularity, so this could be said to give an apples-for-apples comparison. But we don’t know these pairs are of the all-else-equal kind. They rarely aren’t when the experiment is not intentionally designed to be paired. For example, maybe the first dance of a pair tends to steal participants from the second? We could test this by lagging the pairs by one, such that pairs tend to have the mask-required dance first. These pairs aren’t as neat, but when we do this, we discover that only 13 out of the 19 new pairs support the hypothesis. That’s much less obviously meaningful.3 By way of confirmation, the Agresti–Coull interval now comfortably straddles the null hypothesis, at 40.6–89.8 %. Give...