1
Q
Post hoc fallacy
A
A occurred, then B occurred
Therefore A caused B to happen
2
Q
Appeal to tradition
A
Believing that something is right or true because it is part of a tradition that is revered or respected, especially when there is a more important principle or issue at risk
3
Q
Appeal to popular opinion
A
Lots of people believe it, so it must be true
4
Q
Faulty analogy
A
Assuming that because two things are alike in one (usually trivial) respect, they must be alike in some other more important respect
5
Q
Abusive ad hominem
A
Attacking a person in an abusive way as a means of discrediting their argument or distracting attention from it (eg. don’t listen to that loser)
6
Q
Ad hominem: poisoning the well
A
Preemptively presenting irrelevant information about an individual in an attempt to cause their ideas to be ignored, before they are even stated
7
Q
False alternatives
A
Promoting your (usually weak) point of view by presenting it together with an even weaker viewpoint, as if they are the only possibilities, when in reality there are other, better possibilities that have not been mentioned
8
Q
Wishful thinking
A
I want it to be true, so it must be true
9
Q
Straw man fallacy
A
Misrepresenting an opponent’s position or argument in order to make it sound weak or foolish, and therefore easier to attack
10
Q
Arguing from ignorance
A
You can’t prove me wrong, so I’m right
11
Q
Appeal to irrelevant authority
A
Attempting to support a claim by appealing to the judgment of:
* Someone who doesn’t have appropriate expertise
* An unidentified authority (e.g., ‘scientists’; the internet)
* An authority who is likely to be biased
12
Q
Fallacy of the mean
A
Assuming that a moderate or middle view between two extremes must be the best or right one, simply because it is the middle view
13
Q
CONSORT - participants & generalizability
A
What to check:
- Eligibility
There should be a clear description of who was eligible to participate
Often given in the form of ‘inclusion criteria’ and ‘exclusion criteria’
- Baseline data
A table should be provided giving a detailed description of the participants
- Generalizability
Look to see if the conclusions are aimed at a population similar to the participants in the study
14
Q
CONSORT - outcomes 1 (primary & secondary outcomes)
A
- Has a primary outcome been identified?
If more than one parameter is measured, a primary outcome should be clearly identified - Are conclusions based on the primary outcome?
Conclusions about effectiveness that are based on another measurement (i.e., a secondary outcome) are not valid - Red flags
Studies with no specific primary outcome, in which many parameters are measured, are likely to mistake false positives for treatment effects:
“We compared treatments x and y”
“Our aim was to determine the effects of treatment x”
15
Q
CONSORT - outcomes 2 (reporting)
A
- Are results for any of the stated outcomes missing?
If the authors said they were going to measure five outcomes, they should report results for all five outcomes (no cherry-picking) - Look for confidence intervals and absolute values
Look for confidence intervals (not just p-values) and absolute values (not just percentages) - Within-group comparisons to baseline are invalid
Look to see that means were compared to each other, not to baseline within the same group
16
Q
CONSORT - sample size calculation
A
- Do the authors say they did a calculation?
- Do they mention the four variables needed for a sample size calculation?
Look for alpha, power, effect size, and variance as evidence that a calculation was actually performed
17
Q
CONSORT - allocation concealment
A
- Indicators that they did it well
Try Ctrl-F for words like ‘allocation’ ‘centralized’, etc.
Look for mention of ‘third-party’ allocation (e.g., a pharmacist, on-line allocation tool) - Did they mention Sequentially Numbered Opaque Sealed Envelopes (SNOSE)?
This is not as secure as third-party allocation, but can be effective if employed properly
18
Q
CONSORT - randomization
A
- Was the method truly random?
E.g., coin flip, random number generator - If sample size was <100, was a restricted method used?
E.g., blocking or stratification (Simple methods are OK if sample size is 100+)
19
Q
CONSORT - blinding
A
- Similarity of the treatments
The treatments should be described in detail. You should be able to tell if they are sufficiently similar to prevent people from figuring out who is receiving what - Who was blinded – are specific groups named, or is the description vague (e.g., ‘double-blind’)?
Look for mention of participants, treatment providers, etc. - If blinding was not possible or not done, check the outcome:
OK if the primary outcome is objective (e.g., a number, like mortality)
A fatal flaw if it is subjective (e.g., a pain score, a survey), unblinded study is useless
20
Q
CONSORT - participant losses
A
- Look for a flow chart showing losses, with reasons
If some losses are related to the intervention (adverse effects; deaths) or are ‘unknown’, analysis should be ITT - Check the type of analysis
Check # in each group at beginning & end of trial, which number was used in the analysis?
Has everyone been accounted for? (I.e., has an ITT analysis been performed?)
(Caution: Authors may wrongly call a per protocol analysis “modified ITT” or simply “ITT”) - Losses should be modest
Arbitrary rule of thumb: Concern if losses >10%
21
Q
CONSORT - interpretation
A
- Do the data in the tables and figures support the conclusions?
If the conclusions are not supported by the data, don’t accept them - Have the authors reviewed the existing evidence?
If their new findings are contradictory, the onus is on them to explain why they are correct and previous researchers are wrong - Have the authors ignored serious limitations?
If the authors tell you of a serious flaw in their study that may make it impossible to draw any firm conclusions, they should take it into account when drawing their conclusions
22
Q
CONSORT - harms
A
- Is there a heading called “Harms”?
Reader should not have to hunt for this information
(Entitled “Harms” rather than “Safety”) - Do they provide a table with detailed information?
Should report all observed harms so reader can judge likelihood of a link to the intervention - Do they balance harms and benefits when drawing conclusions?
Or do they make vague or dismissive statements about harms (a flag)
E.g., do authors list serious adverse effects but then ignore them when drawing conclusions?
23
Q
CONSORT - funding
A
- Look for source of funding
Lowest risk of funding-associated bias with non-profit sources - Look for involvement of any for-profit funder in the research
Check author affiliations & ‘Conflict of Interest’ statement
Risk of bias is high if authors are employees or paid consultants of the funder, or if company helped with the research (performed some of it, analyzed data, wrote or edited manuscript, etc.) - There should always be a conflict-of-interest statement (may be called ‘Disclosure’, etc.)
Even if no conflict exists; can’t just assume, risk of bias is high if no statement is made
24
Q
CONSORT vs GRADE
A
- CONSORT is designed to ensure that whatever was done has been described thoroughly – whether it was done well or poorly
- GRADE is a process used to assess the quality of the science of one or more RCTs
25
What does GRADE examine?
1) Risk of bias related to study design
(Usually ~6 to 8 CONSORT checklist items)
2) Inconsistency of results across studies
3) Indirectness of evidence
4) Imprecision of the measurements
5) Publication bias
26
Rating the quality of medical evidence
- High = we are very confident that the true effect is similar to the estimate of the effect
- Moderate = the true effect is probably close to the estimate of the effect, but there is a possibility that it is substantially different
- Low = the true effect might be substantially different from the estimate of the effect
- Very low = the true effect is probably substantially different from the estimate of effect
27
GRADE - risk of bias
Rated low, unclear, or high risk
28
GRADE - inconsistancy
- Refers to unexplained heterogeneity of results
(i.e. the individual RCTs within a body of evidence yield very different estimates of effect)
- Large difference between means (point
estimates) across studies
- Minimal or no overlap of confidence
intervals, which suggests variation is more than what one would expect by chance alone
29
GRADE - indirectness
PICO:
- Population = research is conducted in the population that we hope to provide answers for
- Intervention = the RCTs include the interventions we are interested in
- Controls = compare these interventions with the appropriate controls or alternatives
- Outcomes = outcomes of interest have actually been measured, rather than something else that may or may not be informative (poor surrogates)
30
GRADE - imprecision
1) Confidence intervals (small CI = precise estimate of treatment effect)
2) Sample size (OIS = optimal information size)
3) Number of events (may need to base sample size on number of events instead of number of participants)
31
GRADE - publication bias
- Funnel plots and other methods can be used to assess the likelihood that some of the evidence has been withheld/not published/ignored
32
EBM
Best research evidence
- Clinically relevant (especially patient-centered) research
Clinical expertise
- Ability to rapidly identify each patient's unique health state or risk & benefit of intervention
Patient values
- Unique preferences, concerns, and expectations
33
Exaggeration
Data shows no difference between groups, but statements in these papers indicate that the intervention is effective
- most common to exaggerate in abstract
34
Avoiding exaggeration
Routinely check the fairness of interpretation in clinical
trials
35
Random error vs bias
Bias generally causes us to see a greater effect than what is due to our intervention (exaggerated effect)
- aka systematic error
- has a direction = more effective (can’t be less)
- this is what CONSORT aims to minimize
Random error has no direction (can make the intervention look more OR less effective)
- smaller sample = more likely that you have an unrepresentative group (less confident that groups really were similar at the start of study)
36
Systematic review
Calculating the mean of the results of all clinical trials (to balance out impact random error)
37
RCT not needed:
1) Drug doesn't heal (eg. anaesthetic has no reasonable other cause as to why patient is unconscious)
2) Effect is immediate, aka seconds to minutes (eg. lancing an abscess has no question to the cause of improvement)
3) Body is incapable (eg. insulin is clearly the cause for carbohydrate metabolism since their body can't do it naturally)
38
Conspiracy theories
Provide simple answers for unanswered questions about a crisis, and they allow people to retain a sense of predictability and control (inherent tendency to accept a false explanation over no explanation is amplified in times of uncertainty)
39
Peer review decisions
- Accept without revision (rare)
- Accept with minor revisions
- Return to authors for major revisions & resubmission
- Reject
40
Peer reviews
Reduce error, but do not completely eliminate it
41
Rational thinking
1) Rational decisions are based on evidence
2) Evidence is part of the observable world around us
3) Logic is therefore about reality
4) Facts can be objective or subjective
5) We can directly experience only a limited number of events, so in most cases we must rely on indirect evidence (important to consider the reliability of the source)
6) It is relatively easy for people to delude themselves, and to believe things that can readily be shown to be false
42
Subjective thoughts
Reveal the state of a person’s mind, not the state of the world
43
Argument
An argument is a claim supported by other claims
- NOT merely an opinion or an announcement of a belief
- A group of statements (premises) that support or provide evidence for another (the conclusion)
44
Criteria of arguments
1. Structure
- Premises (i.e., reasons/evidence) are given in support of a conclusion
- The conclusion does not also appear as a premise
- The conclusion is not simply an unsupported announcement
2. Relevance
- Reasons should be directly related to the issue
- True but irrelevant facts can leave the illusion of support for the claim
3. Acceptability
- Reasons should be believable to a rationally mature person
4. Sufficiency
- Reasons should be sufficient in number, kind, and weight to support the acceptance of the conclusion
5. Rebuttal
- Should effectively address all serious challenges to the argument
BIOM 3210
flashcards
Decks in class (2)
# Cards
