EBP term 3 module 3

Question 1

Appraising SRs and meta-analysis

Answer 1

• SRs & meta-analysis are ranked at the top of “evidence
  hierarchy” because their conclusions are based on a
  synthesis of multiple studies rather than a single trial.
• This ranking, assuming the review in question to be of high
  quality in its own right, must be determined by the clinician
  (You) rather than taken for granted.
• Clinicians have to assure that the authors of the review
  have done an adequate job of locating, summarizing,
  evaluating, and synthesizing the information that guides
  clinical decisions.
• Potentials for low quality reviews exist due to
• How the review was conducted, and
• How the raw materials (e.g., individual studies) were synthesized.

Question 2

Evidence on SRs that physical therapists use should FIRST be evaluated with the
following questions that are adapted from “Critical Appraisal Worksheets” developed
by the Oxford Center for Evidence-based Medicine. Diane Jewell (2018) p.342

Answer 2

1. did investigators limit the review to high quality studies?
2. did implement a comprehensive search and study selection process?
3. did the investigators assess the quality of individual studies with standardized processes and or tools?
4. did the investigators provide details about the research validity or quality of studies included in the review?
5. did the investigators address publication bias?
6. if this is a meta analysis, did the investigators use individual subject data in the analysis?

Question 3

Appraisal of validity of Systematic Reviews (Interventions)
1. Did the investigators limit the review to high-quality studies? (e.g.,
study design being true experimental or quasi-experimental for
intervention studies)

Answer 3

• The above question addresses potential for bias if individual studies
  have weak controls in their designs.

Question 4

Appraisal of validity of Systematic Reviews (Interventions)
2. Did the investigators implement a comprehensive search and study
selection process?

Answer 4

• Addresses potential for selection bias (a non-representative sample)
  ✓A thorough search requires both electronic and manual efforts to be
  successful.

Question 5

Assessment of Study Credibility – Systematic Reviews
3. Did the investigators assess the quality of individual studies with
standardized processes and/or tools? (e.g., PEDro scale; Cochrane RoB)

Answer 5

• Administered by two independent reviewers; a third reviewer is
  needed to resolve disagreement
• Addresses potential for various forms of biases in individual studies,
  especially related to biases such as
  ✓“subjects allocation to groups”
  ✓“outcome assessors blinded to group allocation or not”
  ✓“loss of subjects leading to imbalance in group characteristics and/or
  methods used to deal with the loss”
  ✓and others to name a few here

Question 6

Example of Cochrane Risk of Bias

Answer 6

• Random assignment
• Allocation concealment
• Blinding of patients and investigators
• Blinding of assessors
• Incomplete outcome data
• Reporting bias
• Other bias

Question 7

Assessment of Study Credibility – Systematic Reviews
4. Did the investigators provide details about the research validity
(or quality) of studies included in the review?

Answer 7

• Addresses the need to interpret the review’s results in the
  context of quality ratings
• Appraising the quality of individual studies is NOT sufficient by
  itself; At a minimum, authors should ALSO report level of quality
  for each article reviewed. For example:
• Cut-off scores for PEDro scale: between 0-3 are considered ‘poor’, 4-5
  ‘fair’, 6-8 ‘good’, and 9-10 ‘excellent’ study designs.
• A SR is likely to be more useful if
  ✓level of quality of individual studies is reported and
  ✓results are interpreted in light of these appraisals.
  Grading of Recommendations Assessment, Development and Evaluation (GRADE):
  rating quality of evidence and strength of recommendations

Question 8

Grading of Recommendations Assessment, Development and Evaluation
(GRADE)

Answer 8

• Reviews based on RCTs are rated high. This rating can be downgraded
  one or more levels, however, by the presence of one or more of the
  following factors:
  1. High likelihood of bias from study design and implementation
  2. Indirectness of population, intervention, control, or outcomes (differences
  in PICO across trials)
  3. Unexplained heterogeneity or inconsistency of the findings
  4. Imprecision or wide confidence intervals
  5. Publication bias

Question 9

Assessment of Study Credibility – Systematic Reviews
5. Did the authors address publication bias?

Answer 9

Publication bias:
i. only studies in English are included in the review
✓Authors are expected to explicitly discuss about it (e.g., no author reads non-
English-language publication)
ii. preference for publication is given to studies showing statistically
significant beneficial treatment effects;
✓the tendency of journal editors to publish studies based on direction and
statistical significance of the outcomes;
✓out of authors’ control;
Raising concerns about an accessibility of ‘unpublished findings’ grey literature:
“composed of nonpublished information, often on websites, in reports,
conference proceedings, theses/dissertations. These sources can be valuable to a
full understanding of the scope of knowledge on the topic”
* Addresses the need to interpret the review’s results in the context of
this potential problem; EX: PRISMA (Eligibility criteria)
✓Specify study characteristics (e.g., PICOS, length of follow-up) and report
characteristics (e.g., years considered, language, publication status) used

Question 10

Assessment of Study Credibility – Systematic Reviews
6. If this is a meta-analysis, did the investigators use individual patient data in the analysis?

Answer 10

• Meta-analysis may be conducted
• using “summary score” of individual studies or
• using individual patients data (preferred because individual patients data provide more information)
• To obtain them, review authors need to request access from authors of individual studies; Unfortunately, it may not always be accomplished due to logistical challenges.

Question 11

Study Credibility – Systematic Reviews
Should You Use this Evidence?
7. How confident are you in the research validity of this review?

Answer 11

This checklist refers to adequacy of reporting in each component of the
review, including
* a clear research question,
* criteria for selection of studies,
* a comprehensive search,
* reliability of reviewers, and
* assessing risk of bias.
* For a meta-analysis, use of appropriate
statistical estimates,
* evaluation of heterogeneity and
* publication bias.
* The appraiser is expected to make a
judgment regarding the overall confidence in
the results of the SRs (high, moderate, low,
and critically low) based on the number of
noncritical weaknesses and critical flaws in
the SRs identified by the checklist

Question 12

Evidence and the Patient/Client’s Perspective
Consider the following questions:

Answer 12

▪ Is the review high quality (e.g., does the design minimize bias)? (Quality)
▪ Are the results important enough to use? (Importance) clinical vs.
statistical
▪ Was your patient/client represented in the review? (Similarity)
▪ Is the element of interest available, practical and safe for application in
the clinical setting? (Practicality)
AND
▪ Patient’s values and preferences
➢ Cost – financial, time, personal
➢ Belief in the value of scientific evidence
➢ Previous experiences

Question 13

Homogeneity or heterogeneity of findings among
studies leads to

Answer 13

• Either a Quantitative or a Qualitative review
• Meta-analysis (MA)
  ✓An extension of a systematic review that incorporates statistical
  pooling from clinical homogeneous studies
  ✓The major advantage of MA is increased power obtained by pooling
  data from several samples, thereby improving estimates of effect
  size and generalization of findings.

Question 14

Heterogeneity (or homogeneity) can be assessed
using either

Answer 14

✓the ‘eyeball’ test: one looks for overlap of the confidence
intervals of the trials with the summary estimate; or
✓more formally with statistical tests such as I2 statistic
(based on the Cochran Q test). A value of 0% indicates no
heterogeneity, and higher percentages suggest greater
heterogeneity.

Question 15

Statistics commonly reported for meta-analysis according to
type of research design (or research question)

Answer 15

✓Effect size (ES) or Standardized effect size (intervention studies)
✓Number needed to treat (NNT)
✓Likelihood ratio (LR; more discussion in Diagnostic studies)
✓Odds ratio (OR)
✓Relative risk (RR; aka risk ratio)
Odds ratio and relative risk ratio helps with case control or cohort study!

Question 16

Meaning of meta-analysis results
Displayed as a forest plot illustrating weighted ES
by individual studies and a cumulative summary

Answer 16

• Result for each study is represented by a tree
  plot:
  a) Square icon represents effect (e.g., Standardized
  Mean Difference – SMD between groups for
  intervention study)
  b) Whisker (horizontal line) represents confidence
  interval (CI) for the SMD
  c) Exact numbers for SMD (95%CI) are to the right.
  d) Center vertical line is null value (null hypothesis;
  no difference between group means or SMD = 0)
• Bottom diamond is pooled effect size and the
  width is CI for the pooled value.
• The study finding is in favor of intervention

Question 17

Effect size:

Answer 17

the magnitude of the difference
between two mean values; may be standardized

Question 18

95% Confidence Interval (CI)

Answer 18

a range of scores within which the true score is estimated to lie within a 95% probability 10

Question 19

effect size notes….

Answer 19

• “a statistical expression of the size of the difference between sample means”
• Calculated as actual differences or standardized differences

Mean Score (Exp) – Mean Score (Control)/Pooled Standard Deviation

Question 20

Effect size (=a/b)

Answer 20

the magnitude of the difference
between two mean values
➢ Between-group difference (a)
➢ Within-group variability (b)
* strength of Effect size
➢ 0.2 small
➢ 0.5 medium
➢ 0.8 large

Question 21

Effect Size (standardized) scores

Answer 21

• > 0.8 big treatment effect (no doubt about it!!)
• 0.5 - 0.8 big enough treatment effect that we can
  “see it with the naked eye”
• 0.2 - 0.49 treatment effect is small enough that we
  cannot “see it with the naked eye”
• < 0.2 what treatment effect?

Question 22

Number Needed to Treat (NNT)

Answer 22

✓“the number of subjects treated with an experimental
intervention over the course of a study required to achieve one good outcome or prevent one bad outcome.”

Question 23

Positive Likelihood Ratio
(LR+)

Answer 23

✓The likelihood that a positive (+) test result
was observed in a person with the disorder v. in a
person without the disorder of interest

Question 24

• Negative Likelihood Ratio
  (LR-)

Answer 24

✓The likelihood that a
negative (-) test result is
observed in a person with
the disorder v. in a person
without the disorder of
interest

Question 25

Odds Ratio (OR)

Answer 25

a measure of association between exposure and an outcome ✓The “odds” that an individual with the prognostic or risk factor will develop the outcome of interest ✓Calculated via logistic regression (can also be done via a 2 x 2 table)

Question 26

Study Results * Relative Risk or Risk Ratio (RR)

Answer 26

✓Ratio of risk for developing the adverse outcome in patients with the risk factor versus patients without the risk factor ✓Comparable to odds ratios; the null value is 1 for both OR and RR.

Question 27

Funnel Plot is for what type of bias

Answer 27

publication bias

Question 28

Funnel Plot - publication bias

Answer 28

* A funnel plot is used to examine results of MA for the problem of publication bias * Scatterplot of effect sizes plotted against measure of variability * Symmetrical distribution indicates both high and low effect sizes with large and small variability * Asymmetrical distribution implies publication bias (high or low outcomes may be missing)