Strata Academy
Cochrane heterogeneity explained – I², τ², and prediction intervals
Interpreting statistical heterogeneity in meta-analysis, when pooling is misleading, and GRADE inconsistency downgrades
Quick answer
Heterogeneity means study results differ more than chance alone would predict. Use I², τ², and prediction intervals together — never I² alone — and investigate clinical differences before trusting a pooled effect. Unexplained inconsistency can downgrade GRADE certainty.
- Statistical heterogeneity ≠ clinical heterogeneity — return to the study table to explain I².
- I² cut-offs (25/50/75%) are rough guides only; context and study count matter more.
- Non-significant Cochran's Q does not prove homogeneity — low power with few studies.
- Prediction intervals show where a future study might land — wider than the pooled CI.
- When heterogeneity is unexplained, narrative synthesis or ranges may beat a single pooled number.
1. What is heterogeneity?
In meta-analysis, heterogeneity is variability among study results beyond what would be expected from sampling error alone. It asks a simple question: are these studies estimating the same underlying effect?
The Cochrane Handbook treats heterogeneity investigation as mandatory before interpreting a pooled estimate or claiming subgroup effects. Ignoring heterogeneity is one of the most common errors in student meta-analysis coursework.
Heterogeneity can arise from true clinical differences (populations, doses, settings), methodological differences (design, bias), or chance in small reviews. Your job as appraiser is to decide which explanation fits.
A statistically significant pooled effect can still be misleading when heterogeneity is high and unexplained – the mean may not represent any single patient population you see in NHS practice.
When reading a forest plot in journal club, look at study confidence intervals before the diamond. If they barely overlap, heterogeneity is likely — even before you read I² in the text.
2. Clinical vs statistical heterogeneity
Clinical heterogeneity describes differences in PICO elements: populations, interventions, comparators, outcomes, timing, and setting. Statistical heterogeneity is the mathematical expression of variation in effect estimates after accounting for chance.
High statistical heterogeneity often signals clinical heterogeneity, measurement differences, or differential bias – but I² alone does not tell you which. You must return to the study characteristics table.
Two reviews can show similar I² with opposite clinical implications: one pools different doses of the same drug (may be fixable with dose subgroup analysis), another pools different interventions entirely (often not poolable).
When appraising a published review, check whether authors defined a priori which clinical differences would be acceptable for pooling. Post-hoc rationalisation of pooling decisions is a red flag in dissertation marking.
| Type | Source | What to do |
|---|---|---|
| Clinical | Different PICO elements across studies | Subgroup, separate syntheses, or don't pool |
| Methodological | Different ROB profiles or outcomes | Sensitivity analysis; ROB subgroup |
| Statistical | Mathematical variation beyond chance | Report I², τ², Q, prediction interval |
| Spurious | Few studies; one outlier | Leave-one-out; check data extraction |
Tip: Read our meta-analysis guide and heterogeneity statistics article for worked interpretation.
3. I², τ², and Q statistic
Cochran's Q tests whether heterogeneity differs from zero. With few studies, Q has low power – a non-significant Q is not proof of homogeneity.
I² estimates the proportion of total variability due to heterogeneity rather than chance, expressed as 0–100%. It is widely reported but frequently misinterpreted in isolation.
τ² (tau-squared) estimates the between-study variance on the effect scale. It feeds random-effects weights and prediction intervals. Report τ² alongside I² when possible.
Historical cut-offs (25%, 50%, 75% for low, moderate, high I²) are rough guides only. Context – number of studies, direction of effects, clinical similarity – matters more than a threshold.
I² depends on study precision: imprecise studies with similar point estimates can yield high I² even when clinical effects look consistent in direction. Always read the forest plot.
Leave-one-out sensitivity: removing one large study and watching I² collapse suggests a single outlier drove apparent heterogeneity — mention this in coursework when RevMan or metafor output is available.
- I² can be high with imprecise studies even when effects are similar in direction
- I² = 0% does not guarantee clinical homogeneity
- τ² is used in random-effects models and 95% prediction intervals
- Q p-value alone is insufficient — report effect sizes of heterogeneity
Note: Do not abandon meta-analysis solely because I² > 50%. Investigate sources first.
4. Prediction intervals
A 95% prediction interval estimates where a future study's true effect might lie – it is wider than the confidence interval for the pooled mean because it includes between-study variance.
Cochrane recommends reporting prediction intervals in random-effects meta-analyses when clinically relevant. They answer: if we run another trial tomorrow in an NHS trust, what effect might we see?
If the prediction interval crosses the null (or a clinically important threshold), the pooled mean effect may be unhelpful for decision-making even when statistically significant.
Students should quote both CI and prediction interval when critiquing reviews in coursework – examiners increasingly expect this distinction.
Example: a pooled RR of 0.80 (95% CI 0.70–0.92) may look favourable, but a 95% prediction interval of 0.55–1.15 suggests the next study could show harm or null — GRADE inconsistency or imprecision may apply.
Software note: RevMan, meta, and metafor can compute prediction intervals but authors often omit them. If a review reports only I² and pooled CI, note this as a reporting gap in your appraisal — not necessarily a reason to distrust the mean, but a limitation for clinical generalisation.
When studies estimate effects on different scales (OR vs RR vs mean difference), heterogeneity metrics are computed after transformation. Check that the forest plot uses a consistent effect measure before interpreting I².
- CI → uncertainty about the mean effect across studies
- Prediction interval → uncertainty about a future single study
- Crossing null on prediction interval → pooled mean may mislead practice
- Requires random-effects framework and τ² estimate
5. Investigating heterogeneity
Pre-specified subgroup analyses (age, dose, risk of bias, geography), meta-regression with extreme caution, and separate syntheses when interventions are clinically distinct are standard approaches.
Post-hoc subgroups discovered after seeing the forest plot are exploratory. Label them as hypothesis-generating, not confirmatory.
Sensitivity analyses – excluding high risk-of-bias studies, leave-one-out analyses, fixed vs random effects – test robustness of conclusions.
When heterogeneity remains unexplained and clinically important, narrative synthesis or presenting ranges may be more honest than a single pooled number.
Meta-regression with fewer than ten studies per covariate is usually underpowered and overfits — Cochrane warns against casual use in small reviews.
Visual inspection tools — L'Abbé plots, Baujat plots, Galbraith radial plots — help identify outliers driving Q and I². Mention them in coursework when the forest plot shows one dominant study.
In network meta-analysis, heterogeneity has additional dimensions (inconsistency between direct and indirect evidence). Standard I² from pairwise meta-analysis does not capture this — use NMA-specific diagnostics when appraising network reviews.
Ordinal and rate outcomes: heterogeneity on transformed scales may look different from raw scales. When appraising reviews of ORs vs RRs, confirm the effect measure is consistent across studies before pooling interpretation.
Student meta-analyses with fewer than five studies: Cochrane Q is especially underpowered — report point estimates and CIs per study prominently even if you pool with random effects.
RevMan displays I² in the forest plot output — copy the statistic verbatim into coursework and interpret it; do not recalculate from memory in exam conditions.
- Compare study PICO in the characteristics table.
- Inspect forest plot direction and overlap of CIs.
- Report Q, I², τ², and prediction interval where appropriate.
- Run pre-specified subgroups or sensitivity analyses.
- Decide whether pooling remains clinically meaningful.
6. Worked example – interpreting a forest plot
When appraising any meta-analysis, reconstruct the heterogeneity story from the forest plot before accepting the pooled diamond. The Cochrane Handbook worked examples illustrate how one outlier study can drive I².
Pick a review in your specialty with 8–15 studies. Note whether all effects favour the same direction, whether one study's CI excludes all others, and whether ROB differs between high- and low-weight studies.
7. Heterogeneity and GRADE
Unexplained or clinically important inconsistency can downgrade GRADE certainty for the outcome under the inconsistency domain.
Document whether review authors investigated heterogeneity, whether the pooled estimate remains clinically meaningful, and whether prediction intervals support the conclusion.
A review can report a statistically significant meta-analysis yet receive low GRADE certainty if inconsistency is serious and unexplained.
Pair GRADE inconsistency judgements with the forest plot and study characteristics table – not with I² alone. Serious inconsistency includes opposing directions of effect across studies.
| GRADE inconsistency | Typical pattern | Action |
|---|---|---|
| Not serious | Effects similar direction and magnitude | May retain certainty |
| Serious | Unexplained wide spread or subgroups overlap null | Downgrade one level |
| Very serious | Opposite directions of effect | Downgrade two levels or don't pool |
8. Fixed-effect vs random-effects models
The choice of meta-analysis model affects how heterogeneity is handled and how much weight each study receives. Fixed-effect models assume one true effect underlies all studies; random-effects models assume effects vary around a distribution with mean μ and variance τ².
When I² is low and studies are clinically similar, fixed and random effects often yield similar pooled estimates. When heterogeneity is moderate or high, random-effects models shrink extreme study weights and produce wider confidence intervals — which is usually more honest.
Do not switch from fixed to random effects only because I² exceeded 50% after seeing the forest plot. Pre-specify the model in your protocol (AMSTAR 2 Item 11). Post-hoc switching undermines review credibility and GRADE imprecision judgements.
The Hartung–Knapp adjustment and other methods modify random-effects confidence intervals when study count is small. Cochrane RevMan applies standard DerSimonian–Laird τ² by default; check whether the review reports sensitivity analyses with alternative estimators.
In coursework, state explicitly: 'We used a random-effects model because clinical populations varied across NHS trusts' — linking model choice to clinical heterogeneity, not software defaults alone.
- Random effects does not 'fix' heterogeneity — it acknowledges it
- Prediction intervals require random-effects framework with τ²
- Report which model was pre-specified in PROSPERO
- Compare fixed vs random in sensitivity analysis when borderline
| Model | Assumption | When appropriate |
|---|---|---|
| Fixed effect | One true effect; differences = chance | Clinically homogeneous studies; low I²; sensitivity check only |
| Random effects | Effects vary around a distribution | Clinical or statistical heterogeneity expected; default in many fields |
| No pooling | Studies too different | Opposite directions; incompatible interventions |
9. Common mistakes
Heterogeneity misinterpretation undermines many student meta-analysis critiques. These errors appear frequently in intercalated project marking.
- Reporting I² without τ², Q, or prediction interval.
- Pooling clearly different interventions because software allowed it.
- Treating non-significant Q as proof of homogeneity.
- Ignoring opposing study directions because the diamond sits favourably.
- Using subgroup analyses without multiplicity caution or protocol pre-specification.
- Citing I² thresholds as definitive rules rather than rough guides.
11. Journal club one-pager (heterogeneity)
Use this sequence when presenting a meta-analysis forest plot in journal club. First, read the clinical question and inclusion criteria — are pooled studies actually comparable? Second, inspect individual study CIs for overlap and direction before mentioning I².
Third, quote Q (with df), I², τ², and prediction interval if reported. Fourth, describe what authors did to investigate heterogeneity — pre-specified subgroups, sensitivity analyses, or nothing. Fifth, link to GRADE inconsistency if the review includes SoF tables.
If the review pooled despite opposing directions, say so explicitly — do not let a favourable diamond anchor the discussion. Examiners and consultants notice when students skip straight to 'the meta-analysis showed significance'.
Bring one clinical scenario: 'Would this pooled effect apply to our ward population?' Prediction intervals and subgroup data help answer that question; I² alone does not.
Cochrane reviews since 2019 increasingly report prediction intervals in abstract results — if absent, note it as a reporting limitation when critiquing reviews in intercalated projects.
- Clinical homogeneity before statistical metrics
- Forest plot direction and overlap first
- Report full heterogeneity statistics package
- Pre-specified vs post-hoc subgroups
- Link unexplained spread to GRADE downgrade
11. StrataResearch and meta-analysis statistics
Meta-analysis manuscripts receive heterogeneity and robustness feedback aligned to Cochrane concepts, alongside AMSTAR 2, PRISMA, ROBIS, and GRADE pathways.
Compare automated heterogeneity commentary to your manual forest plot reading – discrepancies often reveal outcomes or subgroups you had not prioritised.
Use StrataResearch on systematic review PDFs to see whether authors reported inconsistency investigation before you assign GRADE downgrades in coursework.
Heterogeneity commentary on meta-analysis uploads includes I²-related prompts — use these as a checklist when writing the heterogeneity section of your own intercalated systematic review discussion chapter.
- I² and heterogeneity commentary on meta-analysis uploads
- Links to AMSTAR 2 Item 10 (heterogeneity investigation)
- GRADE inconsistency prompts when forest plots suggest spread
Frequently asked questions
What is heterogeneity in meta-analysis?
Heterogeneity is variability in study results beyond sampling error. It suggests studies may not estimate the same true effect — due to clinical differences, methodology, or chance.
What is a good I² value?
There is no universal 'good' I². Rough guides (25%, 50%, 75%) are context-dependent. Always interpret I² alongside the forest plot, τ², clinical similarity, and investigation of sources.
What is the difference between I² and τ²?
I² is the percentage of variability due to heterogeneity rather than chance. τ² is the absolute between-study variance on the effect scale — used for random-effects weights and prediction intervals.
What is a prediction interval?
A 95% prediction interval estimates where a future study's true effect might lie. It is wider than the pooled confidence interval because it includes between-study variance.
When should I not pool studies in meta-analysis?
When interventions or populations are clinically incompatible, effects go in opposite directions, or heterogeneity remains unexplained after pre-specified investigation. Narrative synthesis or separate analyses may be more appropriate.
Interactive walkthroughs and quizzes load when JavaScript is enabled — the checklist and tables above are fully readable without it.