Strata Academy

ROBIS explained – risk of bias in systematic reviews

Assessing the review process itself: protocol, study selection, data collection, synthesis, and reporting

1. What is ROBIS?

ROBIS (Risk Of Bias In Systematic reviews) evaluates whether the systematic review process is likely to introduce bias into the conclusions – independent of risk of bias in the primary studies included.

Developed at the University of Bristol, ROBIS is used when you need a dedicated lens on review conduct bias rather than only study-level ROB 2 ratings.

A review can include only low-risk randomised trials yet still produce a biased conclusion if the search missed entire databases, screening was single-reviewer, or meta-analysis combined incompatible studies.

ROBIS judgements are framed as low risk, high risk, or unclear risk of bias for the review result – not as a numeric quality score.

2. Three ROBIS phases

ROBIS structures assessment across three sequential phases of review conduct, plus reporting and conflicts of interest. Work through phases in order because flaws early (eligibility, search) propagate to synthesis.

Phase 1 covers study eligibility criteria – whether restrictions on populations, interventions, or designs could introduce bias in the evidence assembled.

Phase 2 covers identification and selection – comprehensiveness and reproducibility of search, duplicate screening, and handling of full-text exclusions.

Phase 3 covers data collection and synthesis – extraction quality, integration of primary-study risk of bias, appropriateness of meta-analysis, and how findings from disparate studies were combined.

• Phase 1 – Eligibility criteria appropriate and pre-specified?
• Phase 2 – Search comprehensive; selection duplicated; exclusions documented?
• Phase 3 – Extraction reliable; ROB in primary studies used in synthesis?
• Reporting – Conflicts of interest and funding transparent?

3. ROBIS vs AMSTAR 2

Overlap exists – both ask about comprehensive search, duplicate screening, and risk of bias in included studies. AMSTAR 2 produces an overall confidence rating (critically low to high); ROBIS produces phase-specific risk-of-bias judgements.

AMSTAR 2 is faster for coursework checklists with explicit critical items. ROBIS helps you narrate how process bias might have shifted the conclusion directionally.

For dissertation appraisal, completing AMSTAR 2 plus ROBIS Phase 2 on search and selection is often sufficient depth without duplicating every item twice.

Neither tool replaces reading the forest plot and GRADE certainty ratings – they evaluate process, not clinical importance.

4. Key questions to work through

Focus on signalling questions that change whether you trust the review conclusion. Was the protocol published before screening? If not, selective inclusion becomes more plausible.

Compare the published search strategy to the protocol. Post-hoc database additions after seeing preliminary results are a Phase 2 concern.

Check whether risk-of-bias assessments influenced synthesis – were high-risk trials excluded in sensitivity analysis, or ignored in the primary pool?

For narrative synthesis, did authors explain how contradictory studies were weighed, or only cite supportive trials?

• Eligibility – restrictive inclusion that excludes null trials?
• Search – MEDLINE only; no trial registries; language restrictions unexplained?
• Selection – single reviewer; high conflict rate unresolved?
• Synthesis – inappropriate fixed-effect pooling with obvious heterogeneity?
• Reporting – industry-funded review team undisclosed?

5. Review bias vs primary-study bias

ROB 2 on included trials answers primary-study bias – was randomisation credible, were outcomes selectively reported? ROBIS answers whether the review team introduced additional bias through how studies were found, chosen, and combined.

A review can include only low-risk trials but still be ROBIS-high if the search missed half the evidence or if duplicate publications inflated apparent precision.

When writing coursework, separate these layers explicitly: 'Trial-level ROB was mostly low, but review-level search limitations likely missed relevant cohort studies.'

GRADE certainty ratings should reflect both bodies of evidence quality and review conduct – inconsistency and publication bias downgrades interact with ROBIS concerns.

6. Practical ROBIS appraisal workflow

Step 1: Locate protocol registration and compare to published methods. Step 2: Reconcile PRISMA flow counts. Step 3: Read search appendix and check database coverage.

Step 4: Verify duplicate screening and extraction. Step 5: Map primary-study ROB tools to designs. Step 6: Assess whether meta-analysis model matches protocol.

Document unclear items honestly – 'unclear' is preferable to guessing yes when supplements are missing.

Use ROBIS findings to frame your trust in the abstract's conclusion sentence – not only the p-value of the pooled effect.

7. StrataResearch and ROBIS

Uploaded systematic reviews are evaluated on AMSTAR 2, PRISMA, ROBIS, and GRADE-relevant dimensions – giving both reporting and process bias structure in one report.

Compare StrataResearch ROBIS-aligned feedback to your manual phase notes on the same PDF. Gaps often appear in excluded-studies documentation and search reproducibility.

Pair with our PRISMA 2020 and AMSTAR 2 guides for a complete review appraisal toolkit.

Interactive version (quizzes, walkthroughs) loads when JavaScript is enabled.