Reading Statistics Critically
Everything in this area now turns outward. You have designed inquiries, chosen samples and built displays. That is exactly the equipment needed to read someone else's claim and work out what it is worth.
Find or recall a statistic you have seen recently, in an advertisement, a news headline or a social media post. Write it down exactly. Now write down three things you would need to know before you could decide whether to believe it. You will find you already know what to ask.
Reading a statistic critically is not scepticism, it is checking the same things you had to decide yourself. Every question in the checklist below corresponds to a decision you made in Lessons 1 to 4, which is why this lesson comes last and why it is short.
Who was asked? · How were they chosen? · What exactly was asked? · Compared with what? · Who paid?
Ask them in that order, because the earliest failure makes the rest irrelevant. If the sample was self-selected there is no point scrutinising the percentage, and if the comparison is missing then even a perfect sample tells you nothing. The most common problem is not a wrong number; it is a number with nothing to compare it to.
Know
- The checklist of questions to ask of any reported statistic
- The common ways a true number can support a false impression
- That association between two variables does not establish that one causes the other
Understand
- Why a missing denominator or missing comparison is the most common flaw in a media statistic
- Why funding is relevant evidence without being proof of dishonesty
Can Do
- Apply the checklist to a media report and identify what is missing
- Explain how a stated figure could be true while the impression it creates is false
- Discuss how governments and companies use statistics, and probabilities, to justify decisions
Every question maps onto a decision you made when designing your own inquiry.
| Ask | Which corresponds to |
|---|---|
| Who was asked, and how many? | defining the population and the sample size |
| How were they chosen? | the sampling method, and who it excluded |
| What exactly was asked? | question wording, and whether it was leading or loaded |
| Compared with what? | the aim, and whether the comparison was fixed in advance |
| Who paid, and who reported it? | the limitations you were expected to declare |
Ask them in order. The earliest failure makes the later questions irrelevant: there is no point examining a percentage from a self-selected online poll, because the figure describes only the people who clicked.
The commonest flaw in a reported statistic is not a false number but a number with nothing attached to it.
"Cases doubled this year." From two to four, or from two thousand to four thousand? The word "doubled" is identical in both, and the two situations call for entirely different responses.
"Taking this supplement raises your risk by 50%." If the risk was $2$ in $10\,000$, a 50% rise takes it to $3$ in $10\,000$: one extra case per ten thousand people. The relative figure sounds alarming and the absolute figure is small, and both are true.
Relative figures are not dishonest, but they are chosen. A result reported only in relative terms, with the base rate absent, is usually reported that way because the absolute figure is unimpressive.
Two variables can move together for three quite different reasons, and a scatter plot cannot tell them apart.
One causes the other. More study time produces higher marks.
The causation runs the other way. Higher marks produce more study time, because success is motivating.
A third factor drives both. Ice cream sales and drowning deaths rise together; neither causes the other, and hot weather causes both. That third factor is a confounding variable.
This is why the design questions from Lesson 1 mattered. An experiment that assigns people to conditions at random breaks the confounding, because the groups then differ only by the assignment. An observational study cannot do that, and its conclusions must be worded accordingly: "students who slept more tended to score higher", not "sleep improves marks".
Statistics are used to justify decisions, and the same figure can support opposite decisions depending on what is placed beside it.
Choosing the comparison. A government reporting that unemployment fell this quarter may be comparing with the previous quarter, the same quarter last year, or the peak. Each is true and each tells a different story, and selecting the flattering one is cherry-picking without any number being false.
Choosing the measure. "Waiting times improved" may mean the mean fell while the longest waits grew, if a few very long waits are offset by many short ones. The mean and the median can move in opposite directions.
Using probabilities to price risk. An insurer setting premiums, or a company deciding whether to recall a product, is combining a probability with a cost. That is legitimate, and it is also where a small probability attached to a serious harm can be made to look negligible.
The critical reader's move is not to disbelieve, but to ask what other comparison, or other measure, would have been available.
Watch Me Solve It · 3 examples
-
1Who was asked, and how many?1,200 teachers, which is a reasonable size. But teachers of which subjects, in which schools, at what stage of implementation? The report does not say, so the population the figure describes is unknown.
-
2How were they chosen?Not stated, which is itself informative. If the survey was distributed through official channels and completion was voluntary, respondents are likely those most engaged with the department, and disengaged teachers are the ones most likely to disagree.
-
3What exactly was asked?"Is it working" is vague enough that a respondent thinking of one successful lesson and one thinking of overall outcomes would both answer yes. The published figure depends heavily on the wording, which is not given.
-
4Compared with what, and who paid?There is no comparison with the previous curriculum, so 90% approval has no baseline. And the survey was run by the body that introduced the curriculum, which does not make it wrong but means an independent check is worth more than usual.
-
1Find the base rate$5 \text{ per } 100\,000 = 0.005\%$This is the absolute risk before any additive, and the report gives it only because the question supplies it.
-
2Apply the relative increase$5 \times 1.6 = 8 \text{ per } 100\,000$
-
3State the absolute change$8 - 5 = 3 \text{ extra cases per } 100\,000$Three additional cases per hundred thousand people per year. Both "a 60% increase" and "three extra cases per hundred thousand" describe exactly the same finding.
-
4Say what the reporting choice doesReporting only the relative figure makes a small absolute change sound large. It is not false, but the choice to omit the base rate is what creates the impression, so the omission is the thing to notice.
-
1Identify what was foundAn association between two variables in an observational study: students who ate breakfast tended to score higher. Nothing was assigned or controlled.
-
2Consider the reverse directionLess plausible here, since a test score cannot cause an earlier breakfast, so this direction can be largely ruled out on grounds of timing.
-
3Look for a confounding variableHousehold organisation and income plausibly drive both: a settled home routine produces both regular breakfasts and better study conditions. That single factor could create the whole association with no direct effect of food at all.
-
4Judge the headline"Boosts" asserts causation the design cannot support. A defensible headline is "Students who eat breakfast score higher, study finds", which reports the association and leaves the cause open.
Brain Trainer · 4 problems
Four quick problems. Work each one, then reveal the answer.
-
1 "Complaints have doubled." What is missing?
From two to four, or two thousand to four thousand?The base number -
2 "Risk rises 50%." What must you ask?
A 50% rise on a tiny base rate is still tiny.50% of what base rate? -
3 Ice cream sales and drowning rise together. What is the likely explanation?
Neither causes the other; both rise in hot weather.A confounding variable -
4 A study is funded by the industry it examines. Does that make it false?
It is a reason to check the method, not a refutation.No
Multiple Choice · 5 questions
When evaluating a reported statistic, the method of sampling should be checked before the percentage because:
A treatment reduces the risk of a condition by 40%. The original risk was $10$ in $100\,000$. The number of cases prevented per $100\,000$ people is:
Ice cream sales and drowning deaths both rise in the same months. The best explanation is that:
A government reports that unemployment has fallen, comparing this quarter with the peak three years ago rather than with last quarter. This is an example of:
A study funded by a company with an interest in the result should be treated as:
Short Answer · 3 questions
(a) List four questions you would need answered before accepting this.
(b) For each, explain what a bad answer would look like and why it would matter.
(c) State what the phrase "clinically proven" does and does not guarantee.
(a) Calculate the risk with daily processed meat, and the absolute increase.
(b) Explain why the 18% figure and your absolute figure can both be correct.
(c) Explain why a news outlet might choose the relative figure, and what a reader should do about it.
(a) Explain why the finding does not establish that lights reduce crime.
(b) Suggest one confounding variable and explain how it could produce the association.
(c) The council must decide before any better study could be completed. Describe how they could act reasonably on imperfect evidence, and what they should record so the decision can be evaluated later.
(a) Write the press release a company selling the programme might issue, using only true statements.
(b) Write the press release a critic might issue, again using only true statements.
(c) Explain what is wrong with the company's use of the four-school figure, and describe what evidence would settle whether the programme helps low-scoring schools.
Ask in order
Who, how chosen, what asked, compared with what, who paid
Denominator
Out of how many? A missing base rate is the finding
Association
Not causation; look for the third factor
Funding
Raises scrutiny, does not settle the answer
Your Badges
0 of 6Mark lesson as complete
Tick when you've finished Learn, Practice and the Stretch. Earns +85 XP and +25 coins.