Subject guides: Audiology Guide: Critical Appraisal

What is a critical appraisal and why should you us it?

The critical appraisal of the quality of clinical research is one of the keys to informed decision-making in healthcare. Critical appraisal is the process of carefully and systematically examining research evidence to judge its trustworthiness, its value and relevance in a particular context.

Critical appraisal skills promote understanding of:

which treatments or interventions may really work;
whether research has been conducted properly and has been reported reliably;
which papers are clinically relevant;
which services or treatments are potentially worth funding;
whether the benefits of an intervention are likely to outweigh the harms or costs;
what to believe when making decisions when there is conflicting research.

Critical appraisals are done using checklists, depending on the type of study being appraised. Common questions asked are:

What is the research question?
What is the study type (design)?
What selection considerations were applied?
What are the outcome factors and how are they measured?
What are the study factors and how are they measured?
What important potential confounders are considered?
What is the statistical method used in the study?
How were statistical results used and applied?
What conclusions did the authors reach about the research question?
Are ethical issues considered?

Information from Al-Jundi & Sakka, 2017; CASP, 2018; Mhaskar et al., 2009

Study Types

Different types of clinical questions are answered by different types of study design.

Randomised Controlled Trial (RCT)

Used to answer questions about effects. Participants are randomised into two (or more) different groups and each group receives a different intervention. At the end of the trial, the effects of the different interventions are measured. Blinding (patients and investigators should not know which group the patient belongs to) is used to minimise bias.

Strengths	Weaknesses
Best type of study to determine causality, high level of evidence	Expensive and time consuming
High level of blinding	High internal validity, but potentially only low external validity
Eliminates confounding bias (known and unknown)	Ethical constraints
Strict guidelines for study protocols	Confounding variables may still appear, regardless of randomisation

Non-Randomised Controlled Trial

This type of study does not apply randomisation or uses a method that does not meet randomisation standards, like eg. alternate assignment to groups, age-based groupings, etc. After the allocation of participants to groups, the Non-Randomised Controlled Trial resembles a Cohort Study.

Strengths	Weaknesses
Inexpensive and can be applied to a sample of patients in a clinical practice	Methodology not as strict as for a RCT
Use of a concurrent control group	Selection bias can occur
More scientific rigour than analytical studies	Potential lack of generalisabilty
Uniform determination of outcomes for both participant groups	Limited method of randomisation may lead to issues with confounding factors

(Grimes & Schulz, 2002; Public Health Action Support Team, 2017; Sut, 2014)

Cohort study

Participants or subjects (not patients) with specific characteristics are identified as a 'cohort' (cohort=group) and followed over a long time (years or decades). Differences between them, such as exposure of possible risk factor(s), are measured. Used to answer questions about aetiology or prognosis. Cohort studies are a form of longitudinal study design that flows from the exposure to outcome. Prognostic cohort studies start with a group of patients with a specific condition and follow them up over time to see how the condition develops.

Strengths	Weaknesses
Multiple outcomes can be measured for any one exposure	Time-consuming and costly
Demonstrate direction of causality	Prone to bias due to lack of follow up
Multiple exposures can be considered	Study participation may alter participant behaviour
Measurement of incidence and prevalence	Prone to confounding

(Grimes & Schulz, 2002; Public Health Action Support Team, 2017; Sut, 2014)

Case-control study

Looks at patients (cases) who already have a specific condition and match them with a control group who are very similar except they don't have the condition. Medical records and interviews are used to identify differences in exposure to risk factors in the two groups. Used to answer questions about aetiology, especially for rare conditions where a cohort study would not be feasible.

Strengths	Weaknesses
Cost effective	Prone to bias (esp. selection, recall and observer bias)
Does not require a long follow up period, due to being retrospective	Limited to examining one outcome
Good to examine multiple exposures	Choice of controls can be an issue for proper comparison
Efficient for diseases with long latency periods or rare diseases	Limited in estimating incident rates

(Grimes & Schulz, 2002; Public Health Action Support Team, 2017; Sut, 2014)

Cross-sectional study/survey

A representative sample of a population is identified and examined or interviewed to establish whether or not a specific outcome is present. Used to answer questions about prevalence and diagnosis. For diagnostic studies, the sensitivity and specificity of a new diagnostic test is measured against a 'gold standard' or reference test. Cross-sectional studies can be descriptive or analytical.

Strengths	Weaknesses
Quick and easy to conduct	Does not measure incidence, only prevalence
Multiple exposure and outcomes can be studied	Susceptible to bias
Can measure prevalence for all factors being considered	Not suitable for the study of rare diseases or diseases with a short duration
Useful for descriptive analyses and for generating hypotheses	Factors are measured together, resulting in difficulties to determine whether the exposure preceded or followed the outcome

(Grimes & Schulz, 2002; Public Health Action Support Team, 2017; Sut, 2014)

Critical Appraisal Tools

"Critical appraisal is the process of carefully and systematically examining research to judge its trustworthiness, and its value and relevance in a particular context." (Burls, 2009)

Choosing an appraisal tool

Critical appraisal tools are designed to be used when reading and evaluating published research.

For best results, match the tool against the type of study you want to appraise. Some of the common critical appraisal tools are included here.

Critical Appraisal Skills Programme (CASP) checklists

CASP provides a number of checklists covering RCTs, cohort studies, systematic reviews and more.

JBI Checklists

Joanna Briggs Institute Critical Appraisal Tools have been developed by the JBI and collaborators and approved by the JBI Scientific Committee, following extensive peer review. JBI offers a large number of appraisal checklists for both experimental and observational studies. Word and PDF versions of each are available.

STROBE Statement checklists

The STROBE checklists are designed for the reporting of observational (cohort, case-control, and cross-sectional) studies and can be applied to the critical appraisal of these types of study. Includes individual and mixed study checklists.