Evaluating Epidemiological Study Methods

Cross-sectional studies take a snapshot at one time. They are useful for prevalence, but weaker for proving which factor came first.

Not all evidence is equal. A single patient case report can flag a new phenomenon, but it cannot establish a general truth. A well-conducted systematic review that pools dozens of randomised trials gives the most reliable basis for a clinical decision. Epidemiologists rank study designs in a hierarchy, and a design's position depends on how well it controls confounding and bias.

At the top sits the systematic review with meta-analysis of randomised controlled trials, because pooling many trials averages out chance variation and single-study quirks; a Cochrane review of statin trials is the classic example. One level down, a single well-designed RCT still establishes causation through random allocation, though it may not generalise. The UKPDS trial of metformin for type 2 diabetes shaped diabetes care for decades.

Cohort studies follow exposed and unexposed groups forward in time, so they establish that the exposure came before the disease; the Nurses' Health Study tracking diet and cancer is a landmark. Case-control studies work backwards from people with and without a disease, efficient for rare conditions such as HPV and cervical cancer, but vulnerable to recall bias. Cross-sectional surveys give cheap snapshots, and case reports sit at the base.

The hierarchy explains why media claims so often mislead. A headline built on one cross-sectional survey sits near the bottom of the pyramid, yet it is reported with the confidence of a meta-analysis. In an exam, naming where a study sits in the hierarchy, and justifying that position, is an instant evaluation mark.

In 1998 the NSABP P-1 trial randomised 13,388 women at high risk of breast cancer to tamoxifen or placebo. The tamoxifen group developed 89 cases of invasive breast cancer; the placebo group developed 175. Press releases announced a 49% reduction, yet the same data (about 6,700 women per arm, so roughly 2.6% versus 1.3%) gives an absolute risk reduction of about 1.3 percentage points and a number needed to treat of about 78. All three figures are arithmetically correct.

Worked example: a statin trial

Take a statin trial that follows 10,000 patients for five years: 100 heart attacks among 5,000 on the statin (risk 0.02) and 150 among 5,000 on placebo (risk 0.03). Relative risk is 0.02 divided by 0.03, which is 0.67: the statin group carries two thirds of the placebo risk.

Absolute risk reduction is 0.03 minus 0.02, one percentage point. Relative risk reduction is that 0.01 divided by the control risk of 0.03, giving 33%. The number needed to treat is 1 divided by 0.01, so 100 patients must take the statin for five years to prevent one additional heart attack.

Each number answers a different question. Relative risk asks how the groups compare proportionally. Absolute risk reduction asks how much an individual's probability actually changes. The NNT translates that into clinical workload: how many people must take the drug, with its costs and side effects, for one person to benefit. An evaluation that quotes only one of these numbers is incomplete.

How good is a given NNT? Values below 10 are considered highly effective: one person benefits for every ten treated. Values from 10 to 100 are moderately effective, worthwhile when the outcome prevented is serious. Above 100 the benefit is marginal for most patients, and cost and side effects dominate the decision. Whether an NNT is acceptable always depends on the severity of the outcome prevented.

Relative figures amplify small effects in low-risk populations. A supplement advertised as halving cancer risk sounds dramatic, but if the baseline risk is 2 in 100,000, halving it changes the risk to 1 in 100,000: an absolute reduction of one thousandth of a percentage point and an NNT of 100,000. The relative figure is truthful, yet stripped of the context that gives it meaning.

The related measure, attributable risk, is the difference in risk between exposed and unexposed groups: the excess risk that can be attributed to the exposure itself. If 20% of smokers and 2% of non-smokers develop a disease, the attributable risk is 18 percentage points. Public health agencies use it to estimate how much disease would disappear if the exposure were removed.

The 2002 Heart Protection Study followed 20,536 high-risk patients for five years. Simvastatin cut major vascular events from 25.2% to 19.8%, an absolute reduction of 5.4 percentage points and an NNT of about 19, so statins became standard care after a heart attack. In low-risk people the same relative reduction gives an NNT near 83, so prevention prescribing is more contested.

Do not confuse relative risk with relative risk reduction. RR is the ratio of the two risks; RRR is how far the risk falls, proportionally. A drug with an RR of 0.8 gives a 20% relative reduction, calculated as 1 minus 0.8, not an 80% reduction. Markers deduct for this slip every year.

A p-value below 0.05 means there is less than a 5% probability of seeing a result this extreme if the treatment truly did nothing. It says nothing about whether the effect matters. With a large enough sample, even a trivially small difference becomes statistically significant, because random variation no longer hides it.

One trial of 500,000 patients found a new drug lowered average blood pressure by 0.3 mmHg more than placebo, with p = 0.001. The result is highly significant statistically and clinically meaningless: no patient feels or benefits from a 0.3 mmHg change. Statistical significance tells you an effect probably exists; clinical significance, judged from effect size, ARR and NNT, tells you whether it is worth acting on.

A strong answer separates what the study found from what the study proves. "Associated with lower risk" is safer than "causes lower risk" unless the method controls competing explanations.

A Kaplan-Meier survival curve shows the proportion of a study group that has not yet experienced the outcome, whether death, recurrence or hospitalisation, plotted against time. Every curve starts at 100% and steps downward as events occur. A line that falls more steeply means events are happening faster in that group, the worse outcome.

Three features carry the marks. The vertical gap between two lines at a time point is the difference in survival probability between the groups. A plateau means no observed events during that interval, often because follow-up ended, not because patients are cured. Small tick marks are censored patients who left the study or reached its end event-free; their later fate is unknown, not assumed to be survival.

In one melanoma trial, five-year survival was 52% on immunotherapy and 28% on chemotherapy, with the curves separating from six months onward. The honest conclusion is a 24 percentage point absolute difference in five-year survival, and the widening gap fits immunotherapy's mechanism of building a durable immune response. It does not show a cure: 48% of the immunotherapy group still died within five years.

Interrogate: At year five, state the absolute survival difference. Then point to one event step, one censor mark and one interval where a flat line would need the numbers at risk before interpretation.

Screening creates a subtler trap. A screening program that detects tumours earlier can lift five-year survival from 50% to 80% while mortality stays unchanged. Earlier diagnosis moves the diagnosis date forward without moving the date of death (lead-time bias), and some detected tumours would never have caused harm (overdiagnosis). Survival statistics improve; deaths do not.

Association is not causation, so epidemiologists judge causal claims against Bradford Hill's criteria. The examinable ones include temporality (the exposure must precede the disease), strength of the association, a dose-response relationship, biological plausibility, coherence with existing knowledge, and consistency across independent studies. The more criteria an association satisfies, the stronger the case for causation.

Temporality and plausibility fail in a famous spoof: countries with higher chocolate consumption produce more Nobel laureates per capita. There is no plausible mechanism linking national chocolate intake to individual genius, and the correlation is confounded by wealth and education spending. The association is real in the data; the causal story is not.

Australia's AIHW reports higher rates of cardiovascular disease, diabetes and renal failure among Aboriginal and Torres Strait Islander peoples. Genetics contributes to individual risk, but the population-level disparity is driven mainly by social determinants: income, housing, education, racism and access to healthcare interacting with biology. Attributing the whole gap to genetics fails the plausibility and coherence tests and misdirects prevention.

Evaluating a study is not finding flaws for the sake of it; it is identifying what the study can and cannot establish. Work through a fixed checklist: is the design right for the question, is the sample large enough, is it representative, were participants and assessors blinded, is the control group appropriate, was follow-up long enough, and were confounders controlled?

Apply it to a real stimulus. A six-week RCT of 200 patients found a new anti-inflammatory drug cut self-reported knee pain 35% more than placebo (p = 0.03). The trial was single-blind and excluded patients with severe kidney disease. The strength earns the first mark: randomisation controls confounders, and an RCT is the right design for testing a treatment.

The limitations decide the band. Single-blind means the researchers knew the allocations and could rate a subjective outcome like pain more favourably, an assessment bias. Six weeks is short for a condition that often improves spontaneously. Excluding kidney patients limits generalisability. And p = 0.03 with 200 patients sits close enough to the threshold that sampling variation is a live concern.