Our projects

Strata Collective Ltd develops StrataScore, an AI appraisal engine for published research, and studies how well it works. This page lists our studies, how each is designed and where each protocol is registered. Results are shared through peer-reviewed publication and are not summarised here.

Last updated: 8 November 2026

Agreement study

The StrataScore agreement study

Status: Registered; manuscript in peer review

A prospective, pre-registered study of how closely StrataScore’s scores agree with those of independent expert human reviewers.

At a glance

Design
Prospective, pre-registered, multi-specialty cross-sectional study
Papers appraised
68 published clinical research papers
Specialties
Stroke medicine, cardiology, renal medicine, oncology, anaesthetics and respiratory medicine
Expert reviewers
33, from six countries
Reviews completed
198
Allocation
Balanced incomplete block design. Planned allocation was six papers per reviewer and three reviewers per paper.
Reviewer eligibility
A substantive clinical or academic post in the relevant specialty, and documented experience of research appraisal (peer review or a journal editorial role)
Reviewer task
For each assigned paper, a score from 0 to 100 and a confidence rating for each of six appraisal domains, plus free-text comments
StrataScore
Scores each paper on six pre-specified, weighted domains, each divided into three to six sub-domains scored from 0 to 100 against explicit criteria, using a design-specific appraisal pathway
Primary outcome
Agreement between StrataScore’s overall score and the mean human reviewer score, measured with a two-way mixed-effects, absolute-agreement intraclass correlation coefficient
Secondary outcomes
Domain-level agreement; agreement by specialty and by study design; mean difference between StrataScore and human scores; agreement among the human reviewers
Registration note
Respiratory medicine was added after registration.

Qualitative analysis

Qualitative analysis of reviewer comments

Status: In preparation

A directed content analysis of what expert reviewers wrote about the same papers, set against StrataScore’s written appraisals.

At a glance

Design
Directed content analysis with descriptive counts, nested in the agreement study; reported against the Standards for Reporting Qualitative Research (SRQR)
Data
Free-text comments (a clinical impression, and notable strengths and weaknesses) from 198 expert reviews of 68 papers
Unit of analysis
An evaluative point, meaning a separate strength, weakness or judgement about a paper
Points coded
1,031
Coding frame
Nine categories: statistical methods, missing data, bias and confounding, study design, generalisability, outcome definition, reporting quality, clinical relevance and impact, and other
Coders
Two. Points were identified blind to StrataScore’s output.
Question
For each point, whether StrataScore’s written appraisal of the same paper captured it, using a written decision rule. Coders’ notes on points not captured were then grouped into themes.
Registration note
The coding frame was developed during the analysis and was not pre-specified.

Scoping review

Scoping review: Large language models and generative AI for critical appraisal, peer review and research quality assessment in biomedical research

Status: Under peer review

A scoping review of studies that have evaluated large language models and other generative AI for critical appraisal, risk-of-bias assessment and peer review of biomedical research, mapping what has been tested, how, and against what comparator.

At a glance

Design
Scoping review following JBI methodology, reported against PRISMA-ScR; protocol registered on the Open Science Framework
Eligibility
Empirical evaluations of large language models or other generative AI performing research appraisal tasks (risk-of-bias assessment, methodological quality assessment, reporting guideline assessment or peer review) on biomedical or clinical research manuscripts
Databases
MEDLINE (via PubMed), Embase, Scopus, Web of Science and IEEE Xplore, plus the reference lists of included studies
Search
Run on 11 June 2026 for records published from 1 January 2020
Screening
Two reviewers independently screened all titles and abstracts and independently assessed full texts; a third reviewer adjudicated unresolved disagreements at full-text review
Records
3,521 identified; 1,782 duplicates removed; 1,739 screened at title and abstract; 123 full texts assessed; 43 studies included
Included studies
Published between 2024 and 2026, from 23 countries
Data extraction
Standardised form piloted on five studies; two reviewers independently extracted a random 20% sample and the rest were verified by a second reviewer
Critical appraisal of included studies
Not performed, in line with JBI scoping review guidance

Figures are from the manuscript as submitted on 20 October 2026 and may change during peer review.