Strata Academy

ROBINS-I Explained: Bias in Non-Randomised Studies

Seven bias domains, confounding, when to use ROBINS-I instead of ROB 2, and appraisal tips for cohort and before–after studies

• ROBINS-I is for non-randomised intervention comparisons — not RCTs, diagnostics, or prevalence surveys.
• Confounding is usually the dominant threat; check adjustment and unmeasured confounder plausibility.
• Seven domains each judged before an overall rating: low, moderate, serious, or critical risk.
• Propensity scores and matching reduce but do not eliminate confounding bias.
• Pair ROBINS-I with STROBE reporting checks and compare registry to published outcomes.

1. What is ROBINS-I?

ROBINS-I (Risk Of Bias In Non-randomised Studies of Interventions) is the standard framework for assessing risk of bias when participants are not randomly assigned to intervention and comparator groups.

Version 2 refines signalling questions and overall judgement algorithms. It is published via riskofbias.info and used in Cochrane reviews when RCT evidence is insufficient or complementary.

ROBINS-I addresses a different threat model than ROB 2: confounding and selection into treatment are usually more important than allocation concealment. A well-analysed observational study can still be at serious risk of bias if treated patients differ systematically from controls at baseline.

For UK medical students, ROBINS-I appears increasingly in evidence-based medicine modules, health services research SSCs, and when appraising comparative effectiveness papers using NHS Hospital Episode Statistics or GP databases. Learning ROBINS-I alongside ROB 2 prevents the common error of applying RCT tools to observational treatment comparisons.

Target of analysis matters in ROBINS-I v2: assignment to intervention vs starting intervention can yield different confounding structures. Read the estimand paragraph in the methods before scoring — especially in per-protocol or as-treated analyses.

Crossover and time-varying treatment: when patients switch treatments during follow-up, ROBINS-I Domains 3–5 need extra attention. ITT-style observational analyses may still be biased if censoring relates to outcome.

2. When to use ROBINS-I

Use ROBINS-I for comparative studies of interventions where groups were formed by clinician or patient choice, policy roll-out, or other non-random mechanisms.

Typical settings include comparative effectiveness research in electronic health records, before–after policy changes, and cohort studies where treated and untreated patients differ systematically at baseline.

Do not use ROBINS-I for single-arm case series, cross-sectional prevalence surveys, or diagnostic accuracy studies – each has design-specific tools (JBI, QUADAS-2).

If authors claim randomisation, verify allocation concealment and sequence generation before defaulting to ROBINS-I. Quasi-random allocation (alternate days, odd/even chart numbers) is not true randomisation — document this when choosing ROB 2 vs ROBINS-I.

• Prospective or retrospective cohort studies comparing treatments
• Before–after studies with a concurrent or historical comparator
• Interrupted time series with intervention introduction
• Studies using propensity scores or instrumental variables – still ROBINS-I, not ROB 2

Note: If true randomisation with concealed allocation occurred, use ROB 2 – even if the analysis looks observational.

3. ROBINS-I bias domains (overview)

ROBINS-I organises bias into domains spanning confounding, selection, measurement, and reporting. Work through official signalling questions for the target effect (assignment or starting intervention).

Version 2 distinguishes judgements for different target populations and effect types – read the paper's estimand before scoring. A per-protocol effect and an intention-to-treat effect can have different bias profiles.

Each domain uses signalling questions mapped to algorithms producing low, moderate, serious, or critical risk of bias at domain level. Do not skip to overall judgement without domain worksheets.

Supplementary appendices and protocol registrations often contain detail on exposure definitions and follow-up windows essential for Domains 3–5. Database studies may define exposure in the supplement while the abstract uses simplified language.

Time-fixed vs time-varying confounding: when treatment changes over follow-up, standard baseline adjustment may be insufficient. Look for time-varying covariate methods or justify why static adjustment is adequate.

Tip: Confounding is often the first domain to resolve. Ask what factors predict both treatment choice and outcome.

4. Confounding – the dominant concern

In RCTs, randomisation balances confounders in expectation. In observational studies, treated patients may be sicker, more motivated, or cared for in different centres – all of which can mimic or mask treatment effects.

Look for statistical adjustment (regression, propensity scores, matching), negative control outcomes, or designs that reduce confounding (e.g. regression discontinuity). Absence of adjustment is not automatically high risk if confounding is implausible – but justify that judgement explicitly.

Immortal time bias occurs when follow-up includes a period before treatment starts during which the outcome cannot occur in the treated group. Time-varying exposure definitions in database studies need careful reading — this is a frequent ROBINS-I Domain 2 and 3 issue.

Unmeasured confounding sensitivity analyses (E-value, negative controls) strengthen papers but do not automatically downgrade ROBINS-I confounding domain — assess whether the analyses are credible for the clinical context.

• Were key prognostic variables measured and adjusted?
• Is there a plausible unmeasured confounder that could explain the finding?
• Did authors perform sensitivity analyses for unmeasured confounding?
• Does indication bias explain why sicker patients received the intervention?
• In database studies, was the new-user or active-comparator design used appropriately?

Note: Propensity matching balances measured confounders only — unmeasured confounding can remain at serious risk.

5. ROBINS-I vs ROB 2 vs NOS

ROB 2 is for randomised designs. ROBINS-I is for non-randomised intervention comparisons. Newcastle–Ottawa Scale (NOS) appraises cohort and case–control studies but is not intervention-specific – Cochrane prefers ROBINS-I for causal intervention questions in observational data.

STROBE is the reporting counterpart to ROBINS-I — like CONSORT for ROB 2. Use STROBE to check transparency; use ROBINS-I to judge bias in the effect estimate.

• Policy evaluation without randomisation → ROBINS-I
• Cohort describing prognosis without intervention comparison → may use NOS or design-specific tools
• RCT mislabelled as cohort → reclassify and use ROB 2
• Systematic review of observational interventions → ROBINS-I per study + GRADE

6. Worked example – observational intervention study

Apply ROBINS-I domain by domain to a published comparative effectiveness study. The CTT collaborators' statin observational work is often contrasted with RCT evidence — useful for discussing confounding and magnitude of effect.

For journal club, pick a database cohort comparing two treatments in NHS data. Trace exposure definition, covariate adjustment, and loss to follow-up before assigning domain judgements.

7. Overall risk of bias judgement

As with ROB 2, domains feed an overall judgement of low risk, moderate risk, serious risk, or critical risk of bias (terminology per ROBINS-I v2 materials).

Document which contrast you judged (e.g. drug A vs standard care at 12 months). Different time points can have different bias profiles.

A single critical domain — often confounding — can drive the overall judgement to critical even when other domains appear low. Do not average domains mentally.

In systematic reviews, present ROBINS-I traffic-light plots separately from ROB 2 plots for RCTs. Mixed evidence bodies require GRADE certainty judgements that reflect both study types.

1. Define the target effect (assignment vs starting treatment).
2. Complete all seven domain worksheets with signalling questions.
3. Document 'no information' items and whether they raise concern.
4. Assign overall risk for the primary outcome and time point.
5. Link domain judgements to GRADE risk-of-bias downgrade if synthesising.

8. ROBINS-I for database and registry studies

Comparative effectiveness research using CPRD, HES, OpenSAFELY, or similar NHS-linked databases is increasingly common in UK journals. ROBINS-I applies even when sample sizes exceed typical RCTs — big data does not remove confounding.

New-user or active-comparator designs reduce some immortal time bias compared with naive cohort entry at first prescription. Check whether exposure was defined from treatment initiation with appropriate lag periods.

Propensity scores, inverse probability weighting, and instrumental variables address measured confounding but leave unmeasured confounding as a ROBINS-I Domain 1 threat. Negative control outcomes and E-values strengthen papers but require critical appraisal, not automatic low risk.

Coding algorithms for exposure and outcome (Read codes, ICD-10) introduce misclassification — Domain 3 (classification of interventions) and Domain 6 (outcome measurement). Validation substudies in the same paper should be read before scoring.

High-dimensional propensity scores and machine-learning adjustment do not replace ROBINS-I signalling questions. Complete the official worksheet regardless of statistical sophistication.

Before–after studies without concurrent control: ROBINS-I still applies but confounding from temporal trends is often critical — seasonality, policy changes, and coding drift in NHS data can mimic treatment effects.

10. Journal club checklist (ROBINS-I)

Start by confirming the study is non-randomised and compares interventions — if not, switch to ROB 2 or JBI before scoring. State the contrast you will judge (drug A vs standard care at 12 months).

Spend most of journal club on Domain 1 (confounding): list prognostic factors, check adjustment, discuss unmeasured confounding plausibly. Consultants expect this depth for observational treatment papers.

Flag database-specific biases (immortal time, coding error) when applicable — they are high-yield teaching points in UK health services journal clubs.

Close with overall risk and whether you would trust the effect direction for guideline practice, not whether the p-value was significant.

Observational papers in high-impact journals still require ROBINS-I depth — do not shorten appraisal because the journal is prestigious. Domain worksheets should take longer than the results section did to read.

• Confirm non-randomised intervention design first
• Define treatment contrast and time point
• Domain 1 confounding dominates discussion time
• Name database biases when relevant
• Link overall risk to clinical trust not p-values

11. Common mistakes

These errors appear repeatedly in student appraisals of observational treatment papers. Catching them early improves dissertation and journal club marks.

• Applying ROB 2 signalling questions to observational papers.
• Ignoring immortal time bias in database studies of treatment initiation.
• Treating propensity matching as automatic protection against all confounding.
• Scoring NOS and ROBINS-I interchangeably on the same paper without clarity.
• Assuming objective outcomes (e.g. mortality) eliminate all measurement bias domains.
• Not specifying which treatment contrast was judged when multiple comparisons exist.

12. How StrataResearch applies ROBINS-I

When study-type detection identifies a non-randomised intervention design, StrataResearch routes to ROBINS-I-aligned domains rather than ROB 2.

Upload the manuscript via quick analysis to see structured domain output you can compare against the official riskofbias.info workbook.

Pair ROBINS-I appraisal with STROBE reporting checks and our regression essentials guide when authors use adjusted models for confounding.

In systematic reviews, expect reviewers to present ROBINS-I traffic-light plots alongside RCT ROB 2 plots – not a single merged score.

Database study uploads often trigger confounding-focused commentary — compare StrataResearch domain highlights to your ROBINS-I v2 workbook, especially for CPRD and OpenSAFELY-style comparative effectiveness papers common in UK journal clubs.

Teaching tip: draw a directed acyclic graph (DAG) on paper for one observational treatment paper before scoring Domain 1 — supervisors notice when confounding appraisal is structurally reasoned.

ROBINS-I v2 overall algorithms differ from v1 — confirm which version your module teaches before comparing coursework to published reviews that cite older terminology.

Export your domain worksheet to PDF for appendix submission — examiners rarely accept overall risk judgements without visible signalling-question evidence.

Show your workbook, not only the headline overall rating.

Riskofbias.info worksheets are free — download the current v2 PDF.

• Automatic routing when randomisation is absent or not credible
• Domain-level output aligned to ROBINS-I concepts
• STROBE and confounding commentary on adjusted observational papers

Frequently asked questions

What is ROBINS-I?

ROBINS-I (Risk Of Bias In Non-randomised Studies of Interventions) is a Cochrane-aligned tool for assessing bias in observational intervention studies. It covers seven domains from confounding to selective reporting.

When should I use ROBINS-I instead of ROB 2?

Use ROBINS-I when participants were not randomly assigned to intervention and comparator groups — for example cohort comparisons, before–after studies, or database comparative effectiveness research. Use ROB 2 for true RCTs.

What are the ROBINS-I domains?

Confounding; selection of participants into the study; classification of interventions; deviations from intended interventions; missing data; measurement of outcomes; selection of the reported result. Each is judged before an overall rating.

Does propensity score matching mean low risk of bias?

No. Matching addresses measured confounders but not unmeasured confounding, selection bias, or other ROBINS-I domains. Complete the full ROBINS-I worksheet regardless of statistical matching.

How does ROBINS-I relate to NOS?

NOS assigns quality stars for cohort and case–control studies but is not intervention-specific. For causal intervention questions in observational data, Cochrane prefers ROBINS-I. NOS may still appear in older meta-analyses.

Interactive walkthroughs and quizzes load when JavaScript is enabled — the checklist and tables above are fully readable without it.