Strata Academy

ROBIS explained – risk of bias in systematic reviews

Assessing the review process itself: protocol, study selection, data collection, synthesis, and reporting

• ROBIS judges the review process — not trial-level ROB 2 scores alone.
• Three phases: eligibility, identification/selection, data collection/synthesis.
• AMSTAR 2 gives overall confidence; ROBIS narrates how process bias could shift conclusions.
• Compare protocol to published search before trusting pooled effects.
• StrataResearch maps review PDFs to AMSTAR 2, PRISMA, and ROBIS-aligned feedback.

1. What is ROBIS?

ROBIS (Risk Of Bias In Systematic reviews) evaluates whether the systematic review process is likely to introduce bias into the conclusions – independent of risk of bias in the primary studies included.

Developed at the University of Bristol, ROBIS is used when you need a dedicated lens on review conduct bias rather than only study-level ROB 2 ratings.

A review can include only low-risk randomised trials yet still produce a biased conclusion if the search missed entire databases, screening was single-reviewer, or meta-analysis combined incompatible studies.

ROBIS judgements are framed as low risk, high risk, or unclear risk of bias for the review result – not as a numeric quality score.

ROBIS is distinct from ROB 2 spelling — ROBIS ends in 'IS' for systematic reviews. Students often confuse the acronyms in coursework; state the full name once in your introduction.

2. Three ROBIS phases

ROBIS structures assessment across three sequential phases of review conduct, plus reporting and conflicts of interest. Work through phases in order because flaws early (eligibility, search) propagate to synthesis.

Phase 1 covers study eligibility criteria – whether restrictions on populations, interventions, or designs could introduce bias in the evidence assembled.

Phase 2 covers identification and selection – comprehensiveness and reproducibility of search, duplicate screening, and handling of full-text exclusions.

Phase 3 covers data collection and synthesis – extraction quality, integration of primary-study risk of bias, appropriateness of meta-analysis, and how findings from disparate studies were combined.

After phases, ROBIS asks about conflicts of interest and whether the review team declared funding that could influence conclusions.

• Phase 1 – Eligibility criteria appropriate and pre-specified?
• Phase 2 – Search comprehensive; selection duplicated; exclusions documented?
• Phase 3 – Extraction reliable; ROB in primary studies used in synthesis?
• Reporting – Conflicts of interest and funding transparent?

3. ROBIS vs AMSTAR 2

Overlap exists – both ask about comprehensive search, duplicate screening, and risk of bias in included studies. AMSTAR 2 produces an overall confidence rating (critically low to high); ROBIS produces phase-specific risk-of-bias judgements.

AMSTAR 2 is faster for coursework checklists with explicit critical items. ROBIS helps you narrate how process bias might have shifted the conclusion directionally.

For dissertation appraisal, completing AMSTAR 2 plus ROBIS Phase 2 on search and selection is often sufficient depth without duplicating every item twice.

Neither tool replaces reading the forest plot and GRADE certainty ratings – they evaluate process, not clinical importance.

A review rated AMSTAR 2 'high confidence' can still have ROBIS concerns if the search strategy changed after preliminary results — always read the protocol amendment trail.

4. Key signalling questions to work through

Focus on signalling questions that change whether you trust the review conclusion. Was the protocol published before screening? If not, selective inclusion becomes more plausible.

Compare the published search strategy to the protocol. Post-hoc database additions after seeing preliminary results are a Phase 2 concern.

Check whether risk-of-bias assessments influenced synthesis – were high-risk trials excluded in sensitivity analysis, or ignored in the primary pool?

For narrative synthesis, did authors explain how contradictory studies were weighed, or only cite supportive trials?

Grey literature: if the review claims comprehensiveness but searched only published journal articles, Phase 2 is likely high or unclear risk.

• Eligibility – restrictive inclusion that excludes null trials?
• Search – MEDLINE only; no trial registries; language restrictions unexplained?
• Selection – single reviewer; high conflict rate unresolved?
• Synthesis – inappropriate fixed-effect pooling with obvious heterogeneity?
• Reporting – industry-funded review team undisclosed?

5. Review bias vs primary-study bias

ROB 2 on included trials answers primary-study bias – was randomisation credible, were outcomes selectively reported? ROBIS answers whether the review team introduced additional bias through how studies were found, chosen, and combined.

A review can include only low-risk trials but still be ROBIS-high if the search missed half the evidence or if duplicate publications inflated apparent precision.

When writing coursework, separate these layers explicitly: 'Trial-level ROB was mostly low, but review-level search limitations likely missed relevant cohort studies.'

GRADE certainty ratings should reflect both bodies of evidence quality and review conduct – inconsistency and publication bias downgrades interact with ROBIS concerns.

Network meta-analysis reviews add another layer: inappropriate transitivity assumptions are Phase 3 synthesis concerns, not ROB 2 issues.

6. Practical ROBIS appraisal workflow

1. Locate PROSPERO or protocol registration; compare to published methods
2. Reconcile PRISMA flow counts (screened, excluded, included)
3. Read search appendix — databases, dates, grey literature, registries
4. Verify duplicate screening and extraction; note single-reviewer steps
5. Map primary-study ROB tools to designs (ROB 2, ROBINS-I, etc.)
6. Assess meta-analysis model vs protocol; read sensitivity analyses
7. Judge phase-level risk; document unclear items honestly
8. State trust in the abstract conclusion sentence — not only the pooled p-value

Tip: Use our PRISMA checklist tool at /tools/prisma-checklist before ROBIS Phase 2.

7. Worked example – teaching review

Imagine appraising a Cochrane-style review of antihypertensive drugs in pregnancy. Included trials are mostly low ROB 2 risk. Phase 1: eligibility restricted to RCTs only — reasonable if question is comparative effectiveness, but may miss important cohort safety data (some concerns).

Phase 2: search included CENTRAL, Embase, and trial registries; dual screening reported — likely low risk if PRISMA flow is complete. Phase 3: random-effects meta-analysis pre-specified; high-ROB trials included with sensitivity analysis — low risk if documented.

If the same review searched PubMed only and one author screened titles, Phase 2 becomes high risk regardless of trial quality.

8. Common ROBIS mistakes

Students repeat predictable errors when first learning review-level appraisal. Avoiding them strengthens dissertation and journal club presentations.

• Conflating AMSTAR 2 confidence with ROBIS phase judgements without reading phases
• Skipping protocol comparison — only reading the abstract conclusion
• Ignoring duplicate publication inflation in Phase 2
• Treating PRISMA completeness as proof of low process bias
• Applying ROBIS to a primary RCT manuscript — wrong tool entirely
• Giving 'low risk' because the journal is Cochrane or high impact factor

9. StrataResearch and ROBIS

Uploaded systematic reviews are evaluated on AMSTAR 2, PRISMA, ROBIS, and GRADE-relevant dimensions – giving both reporting and process bias structure in one report.

Compare StrataResearch ROBIS-aligned feedback to your manual phase notes on the same PDF. Gaps often appear in excluded-studies documentation and search reproducibility.

Pair with our PRISMA 2020, AMSTAR 2, and systematic review methodology guides for a complete review appraisal toolkit.

Use quick analysis when you need structured output faster than manual ROBIS worksheets — then verify critical items against the protocol yourself.

Frequently asked questions

What is the difference between ROBIS and ROB 2?

ROB 2 assesses bias in individual randomised trials. ROBIS assesses whether the systematic review process (search, selection, synthesis) biased the review conclusion.

Do I need ROBIS if I already used AMSTAR 2?

AMSTAR 2 and ROBIS overlap but serve different narratives. AMSTAR 2 gives overall confidence; ROBIS explains phase-specific process bias. Many dissertations use both at selective depth.

Can a review be ROBIS low risk with high-risk included trials?

Yes. ROBIS judges the review process. Primary studies can be high ROB 2 while the review team conducted search and synthesis rigorously — though GRADE certainty may still be low.

Where is the official ROBIS tool?

The ROBIS tool and guidance are published by the Bristol team. Use the official signalling questions for formal work; pair with our AMSTAR 2 and PRISMA guides for teaching.

Does StrataResearch score ROBIS?

Systematic review PDFs receive structured feedback aligned with AMSTAR 2, PRISMA, and review-process bias concepts including ROBIS-relevant domains.

Interactive walkthroughs and quizzes load when JavaScript is enabled — the checklist and tables above are fully readable without it.