Library Guides: Systematic review guide: 11. Quality assessment

Quality assessment: critical appraisal

Critical appraisal is an essential step of evidence synthesis. You may see it also being referred to as quality assessment or risk of bias assessment. It is focused on the methods of evidence rather than simply the results. Critical appraisal seeks to understand the internal and external validity of a study, as explained below in Gosall and Gosall, The Doctor’s Guide to Critical Appraisal (2020).

"Critical appraisal assesses the validity of the research and statistical techniques employed in studies, and generates clinically useful information from them. It seeks to answer two major questions:

Does the research have internal validity - to what extent does the study measure what it sets out to measure?
Does the research have external validity - to what extent can the results from the study be generalised to a wider population?" (pp3-4)

A systematic approach

Systematic reviews require a formal, systematic and uniform approach to appraisal of the quality (or risk of bias) of all studies included in the data synthesis. If your review only includes Randomised Controlled Trials, then all the RCTs should undergo the same appraisal process and the decisions on quality recorded in the same way e.g. by using an appropriate tool such as the Cochrane Risk of Bias Tool, which has been specifically designed for evaluating RCTs.

Not the same as certainty of evidence

While similar, the assessment of certainty of evidence is not the same as critical appraisal. This independent stage of the systematic review process is explained in this guide's section on summarising evidence certainty with GRADE. You will need to complete both the quality assessment and data extraction activities to conduct an assessment on evidence certainty.

Risk of Bias 2 (RoB 2)
Version 2 of the Cochrane risk-of-bias tool is the recomended tool to assess risk of bias in randomized trials.
ROBINS-I V2(Risk of Bias in Non-Randomized Studies)
This tool was developed to assess risk of bias in the results of non-randomized studies of interventions such as observational, case-control, interrupted-time-series and quasi-randomized trials.
ROBINS-E tool
Provides a structured approach to assessing RoB in non-randomized studies of exposure and other observational epidemiological studies.
Robvis visualization tool
Robvis is a web app for vizualizing RoB assessments in Systematic Reviews. It creates "traffic light" plots and weighted bar plots.
NOS (Newcastle-Ottawa Scale)
For quality assessment of non-randomised studies, including case control and cohort studies.
MetaQAT
A meta-tool for appraising public health evidence.
QUADAS-2
The recommended tool for evaluating the risk of bias and applicability of primary diagnostic accuracy studies.
QUADAS-C
An extension to QUADAS-2 for assessing risk of bias in comparative accuracy studies.
CATmaker
CATmaker is downloadable software that is designed to aid critical appraisal. Users can create Critically Appraised Topics (CATs) for diagnosis, prognosis and aetiology / harm studies as well as systematic reviews and randomized controlled trials. CATmaker calculates useful clinical measures, such as relative risk reduction and number needed to treat, from information entered by the user and helps formulate clinical 'bottom lines'.

In the below list you can find a range of other critical appraisal tools to aid in critically appraising and evaluating articles and other sources of medical information. They have been created by a variety of organisations. The checklists are usually tailored to specific types of article or studies such as RCTs, systematic reviews or qualitative studies. They have not been necessarily designed for use in Systematic Reviews/ formal evidence synthesis but will allow you to practice, deepen your evaluative skills and fill any gaps in the tools in the other tab.

Critical appraisal skills programme (CASP)
offers a number of free downloadable checklists to help you perform critical appraisal across a number of different study types more easily and accurately.
Centre for Evidence Based Medicine (CEBM) critical appraisal tools
worksheets to help you appraise the reliability, importance and applicability of clinical evidence. The worksheets are available in a range of languages.
EQUATOR Network
gives access to main types reporting guidelines for research such as CONSORT for RCTs, as well as those for observational, clinical practice, qualitative, quality improvement, economic evaluation studies as well as systematic reviews and more. Knowing reporting guidelines could help you to judge the transparency and accuracy of reporting of study methodologies to decide on reliability and value of material you are appraising.
ACCODS Checklist (pdf)
This checklist was designed to enable evaluation and critical appraisal of grey literature.
Naicker's Critically Appriasing for Antiracism Quality Appraisal Tool
Racial bias in research impacts a study’s relevancy, validity and reliability. However, presently this aspect is not addressed in critical appraisal tools, and consequently appraisers may often not take racial bias into account when assessing a paper’s quality. In response to this, a supplementary tool has been developed to support appraisers in explicitly addressing racial bias.

Critical Appraisal in Covidence

Covidence can help manage critical appraisal as part of it's Data Extraction workflow. You can choose to use the inbuilt template for the Cochrane Risk of BIas tool (although note this is version 1 not RoB2) or a customizable template where you can build a form to suit the tool you are using.

When your critical appraisal of all included studies has been completed you can export the Quality Assessment consenus data (or the consensus with individual data as well) via a CSV file using the Export option on the main review screen.