What does a p-value measure in a hypothetical test of a club’s claim?
Quick answer
A p-value measures how likely results at least as extreme as the observed results would be if the null hypothesis and test model were true. For a hypothetical club claiming average attendance is 20, a p-value of 0.04 means the test assigns a 4% probability to results at least as extreme as the sample result under that claim and model. It is not the probability that the claim is true.
The question, the details, the next step.
Our club says average attendance is 20 people per meeting. We counted a sample of meetings and got an average of 25. What would the p-value tell us? I’m trying to explain it without making it sound more certain than it is.
Start by treating the club’s stated average as the null hypothesis: the model assumes the true average attendance is 20. A p-value asks: if that assumption and the test’s other assumptions were right, how likely would we be to get sample results at least as incompatible with the claim as the results we observed? That is a probability about possible sample results under a model, not a probability that the claim itself is true. [S1]
For illustration only, suppose a suitable test produced a two-sided p-value of 0.04. You could say: “If the true average were 20 and the test model were appropriate, results at least as far from 20 as ours would occur with a probability of 4%.” This example assumes the p-value has already been calculated correctly; the sample average alone is not enough to calculate it.
Why do you say “at least as far”? Does the calculation depend on which direction we care about?
Yes. “At least as far” refers to the test’s definition of results that count as extreme. In a two-sided test, departures above or below 20 can count; in a one-sided test, the test focuses on a particular direction. The tail direction and the test model affect the p-value, so the number cannot be interpreted fully without knowing what question the test was set up to answer. [S1]
In our illustration, a two-sided test treats sufficiently low and sufficiently high sample averages as evidence against an average of 20. If the club had instead asked only whether attendance tends to be higher than 20, that would be a different test question.
So can I say there’s a 4% chance the club’s claim is true, or a 96% chance the average is different?
No. Those statements reverse the conditional probability. The 4% is calculated assuming the null claim and test model, and describes how often results at least this extreme would arise under them. It does not give the probability that the null claim is true, nor the probability that an alternative claim is true. [S1]
Other common misreadings to avoid:
- “There’s a 4% chance the null hypothesis is true.” Not what the p-value measures.
- “There’s a 96% chance attendance is different from 20.” Also not what it measures.
- “The result is 4% large” or “the average differs by 4%.” A p-value is not the size of the difference; here the observed difference in sample averages is 5 people.
- “A small p-value proves the club’s claim is false.” The p-value describes how surprising the results would be under the stated model, not proof by itself.
What should I check before sharing the result, and what’s a careful sentence we can reuse?
Check that you can state the null claim, identify whether the test was one- or two-sided, and explain what the p-value conditions on. Also keep the observed difference separate from the p-value: the sample average of 25 is five above the claimed average of 20, while the hypothetical p-value of 0.04 describes the model-based probability of results at least as extreme. A p-value alone does not establish the size of the difference between the sample average and the claimed average.
Copyable wording for this illustration: “Assuming the true average attendance is 20 and the chosen test model is appropriate, the two-sided p-value is 0.04: results at least as far from 20 as our sample average of 25 would have a 4% probability under that model. This is not a 4% probability that the club’s claim is true.” Replace the numbers and test details with your own verified values; don’t present this hypothetical example as a result from your club.
A practical checklist
Write down the null claim being tested, in context.
Confirm whether the test is one-sided or two-sided and what counts as an extreme result.
Check the test model before interpreting the reported p-value.
Report the observed sample result separately from the p-value.
Avoid describing the p-value as the probability that either hypothesis is true.
A few more useful answers
Does a smaller p-value mean the difference is larger?
No. A p-value measures how unusual results are under the null model; it is not a measure of the size of the observed difference. Report the sample difference separately.
Can the p-value tell us whether the club’s attendance claim is practically important?
Not by itself. A p-value describes the probability of results at least as extreme as those observed under the null model; it does not measure the size of the difference.
Can I interpret a p-value without knowing the test direction?
Not completely. Whether the test counts departures in one direction or both changes what results are considered at least as extreme, and therefore affects interpretation.
Sources
References used for this guide.
- [S1] OpenStax Introductory Statistics 2e: Rare Events, the Sample, Decision and Conclusion openstax.org · Checked