6.4 Conclusion
We have discussed sampling - what probability samples and non-probability samples are. It is true that sample size matters, but we can quantify the uncertainty in smaller samples by calculating bounds/intervals around our estimates. This is quite simple to do if we have a probability sample, and in such cases, estimates are quite accurate even with a relatively small sample size (e.g., 1000 observations).
In many studies reported in the media, however, non-probability samples are used. Non-probability samples can be a good alternative to probability samples especially given that non-response rates are increasing. However, we cannot use formulas meant for probability samples and apply them to non-probability samples as if they are the same. Non-probability samples require different (and slighty more complex) methods to produce equally accurate estimates.
What does this case study highlight? First, we need to be transparent53 about how studies recruit members into their sample. How the sample is recruited is usually much more important than how large the sample is, but we are seldom given enough detail about it. Second, we need to be aware if appropriate methods are being used. Using a non-probability sample can seem more practical, but we become overconfident and may end up with the wrong conclusions if we simply act as if it is a probability sample.
And we need to demand transparency.↩︎