A short statistical reasoning test

Summary

1. Sorting fractions under uncertainty Each fraction has to be modelled as a distribution in order take its uncertainty into account. The binomial distribution is a good choice, such that \(k \sim Binom(n,p)\) where \(k\) is the number of successes, \(n\) the number of trials and \(p\) the fraction of successful trials. We are given \(k\) and \(n\), but we are interested in the distribution of \(p\). There are many specific ways to produce a confidence interval for \(p\), the lower bound of which can be used to order the fractions, and so control the risk of over-estimation. There are at least two general – from first principles – approaches to calculate a lower bound fraction without knowing specific formulae. The first – Bayesian – is to assign a prior to \(p\) and then take the \(q\) quantile of the posterior distribution as a lower bound estimate. One example of this is to use the beta distribution prior (closed form and conjugate of the binomial distribution) and then the posterior is given by \(p \sim beta(k+\alpha, n-k+\beta)\), where \(beta(\alpha, \beta)\) was your prior. In R: # flat prior beta(0.5,0.5) # 5th quantile for 3 successes and 4 trials. qbeta(0.05, 3 + 0.5, 4-3+0.5) ## [1] 0.3492928 The second – likelihoodist – is to create a profile likelihood and take the \(q\) quantile. I personally find this approach more intuitive in general because it is contextually picking model parameters, rather than to directly making claims about degrees of belief: we are just trying to pick \(p\) such that it captures the first 5% of the likelihood sum of our binomial model. # 5th quantile profile likelihood for # 3 successes and 4 trials. fs <- seq(0,1,0.001) ps <- sapply(fs, \(p) dbinom(3, 4, p)) ps <- cumsum(ps / sum(ps)) max(fs[ps <= 0.05]) ## [1] 0.342 3. Estimating the number of buses It often helps to consider what the data generating distribution for a problem could be. In this case, some hidden \(k\) buses with the same chance of geting picked with a sample...