Methodology Flashcards

(95 cards)

1
Q

Intention to treat (ITT)

A

Analyzes patients based on original assignment regardless if they completed or adhered

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
2
Q

Benefits of ITT

A

Preserves randomization
Minimizes bias
Reflects real-world effectiveness
Avoids overestimating efficacy- gives robust and conservative estimate
Ensures comparability of groups over time- essential for valid causal inference
protects the internal validity of the trial and ensures the results are both scientifically rigorous and clinically relevant

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
3
Q

How does ITT preserve randomization

A

keeps participants in their originally assigned groups, regardless of adherence or dropout.
This maintains the balance of confounding variables achieved through randomization. preventing unknown systematic bias in dropouts
allowing a fair and unbiased comparison of treatment effects across groups regardless of deviations in participant behavior or protocol adherence

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
4
Q

How does ITT minimize bias

A

Prevents attrition bias (i.e., bias introduced when analyzing only those who complete the study).
Avoids cherry-picking favorable results by including all participants

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

How does randomization reduce bias

A

by assigning treatment independent of potential confounders
Minimized selection bias

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
5
Q

How does PP analysis overestimate efficacy

A

Per-protocol or as-treated analyses can inflate treatment effect estimates by excluding non-adherent or drop-out participants.
ITT gives a more conservative and robust estimate of the intervention’s impact.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
6
Q

How does ITT preserve statistical power

A

more conservative estimate of treatment effects, assumes dropouts are random (Could still be biased if really not random e.g. due to treatment AEs)
by maintaining the original, planned sample size throughout the analysis

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
7
Q

Why do you need to do sensitivity analysis in ITT

A

Would need to conduct sensitivity analyses to test how robust results are under assumptions about missing data, dropout etc.
Use LOCF or multiple imputation to account for missing data

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
8
Q

Why choose RCT

A

Random assignment reduces selection bias
Gold standard for cause-effect determination
Control group comparison
justifies use of parametric tests (since any skewedness should be same both groups)
supports generalizability
Enables calculation of risk ratios, NNT, confidence intervals
Minimizes confounding
Standardization/reproducibility of intervention
I.e. most reliable method for evaluating intervention efficacy due to their methodological rigor and ability to control bias

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
9
Q

How does RCT reduce selection bias

A

Balances known and unknown confounding variables between groups

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
10
Q

Why choose RCT

A

Random assignment reduces selection bias
Gold standard for cause-effect determination
Control group comparison
Statistical validity
Minimizes confounding
Standardization/reproducibility of intervention
I.e. most reliable method for evaluating intervention efficacy due to their methodological rigor and ability to control bias

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
11
Q

How does RCT assess cause and effect relationships

A

Controls for temporal ambiguity—intervention clearly precedes the outcome

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
12
Q

How does Blinding reduces observer bias

A

Single- or double-blinding prevents knowledge of group assignment from influencing outcomes or assessment.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
13
Q

How does RCT minimize confounding

A

Unlike observational studies (e.g., cohort or case-control), RCTs actively control for confounding at the design level, not just in analysis.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
14
Q
A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
15
Q

How does RCT ensure statistical validity

A

randomization and concealment to ensure that treatment and control groups are comparable at the start of the study, thereby eliminating selection bias and balancing both known and unknown confounding factors. This methodological rigor ensures that any observed differences in outcomes can be attributed to the intervention rather than pre-existing differences, maximizing internal validity
Randomization justifies use of parametric tests and supports generalizability when well-powered.
Enables calculation of risk ratios, NNT (number needed to treat), and confidence intervals.

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
16
Q

Stratified vs. Enriched

A

Priimary difference lies in patient selection: stratified trials enroll a broad population but balance subgroups, while enriched trials exclude patients unlikely to respond
randomizing within biomarker defined groups (each treatment group has people pos/neg for the biomarker) where enriched is just selecting for it
I.e. different biomarker same treatment all groups vs. same biomarker all groups different treatment

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
17
Q

Define stratified

A

Stratified: ensures characteristic/strata is balanced across treatment groups
Definition: assigning participants to groups to ensure characteristics balanced across
Divide into strata based on characteristic then randomize within stratum
Participants subgrouped into strata based on characteristics, then randomization done AFTER, within each subgroup

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
18
Q

Define enriched design

A

Enriched: selecting participants based on characteristic expected to increase likelihood of success (but not always, idea is to select best chance of observing drug effect-may depend on the biomarker goal)
Definition: intentionally selecting group with a characteristic likely to benefit
Enriched design focuses on one biomarker groups: biomarker positive patients who do/don’t receive the treatment
I.e. same biomarker different treatment

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
19
Q

Why choose enriched design

A

Ensures trial focuses on those most likely to respond, increasing chance of signal detection
Can be biomarker-driven, Clinical characteristics driven, or using predictive modeling
focuses on one biomarker groups: biomarker positive patients who do/don’t receive the treatmentures groups are comparable pre-treatment to confidently interpret post-treatment differences as attributed to intervention not other factors

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
20
Q
A
How well did you know this?
1
Not at all
2
3
4
5
Perfectly
21
Q

Define confound

A

Confounding = third variable associated with both the exposure and outcome, thus potentially distorting the relationship between them
Often demographic, clinical, environmental

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
22
Q

How to address confounds

A

Start by reviewing existing literature to identify possible known confounders
Then control for in statistical tests by adding as covariates

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
23
Q

How to account for dropout rates

A

ITT
Multiple imputation/FIML
Document reasons
Compare groups to ID systemic patterns

How well did you know this?
1
Not at all
2
3
4
5
Perfectly
24
How to address heterogeneity in meta analysis
Investigate sources via subgroup analyses or meta-regression. Consider excluding outliers or low-quality studies in sensitivity analysis.
25
Define heterogeneity in meta analysis
Defined: degree of variability in effect estimate between trials reported in forest plot Quantitative: e.g. I2 compares overlap between study Cis A Relative measure- shows how much of pooled effect is NOT heterogeneity Outcome of 0 doesn’t mean NO heterogeneity Qualitative: balance of studies on each side of line of no effect
26
Define sensitivity analyses
Test robustness of findings by varying inclusion criteria. removing Subsecting by pre-specified factors to compare meta-analysis results Lets you analyze more specific question within portion of cohort MUST pre-specify in protocol otherwise data-driven Fine to add but explain why post-hoc
27
Define subgroup analyses
Explore effects in subgroups (e.g., by age, sex, dosage, study quality). separating and comparing MUST pre-specific – good to have a reference for why selecting every single variable Split/dichotomize studies by a variable to compare ES and heterogeneity
28
Fixed vs random effects meta analysis
Fixed-effect model: assumes one true effect size. Random-effects model: assumes effect size may vary across studies (more realistic with heterogeneity).
29
30
Why test heterogeneity in meta analysis
Determines whether studies are consistent enough to be combined into one estimate
31
How to measure study heterogeneity
Assessed using statistical tests: I² statistic and Cochran’s Q to assess variability.
32
What is the I2 measure of heterogeneity
a percentage describing proportion of total variability due to study heterogeneity (in relation to total variability, including random error) More common because more easily interpretable >50% = high, suggests substantial heterogeneity between studies EG: I2=60% means 60% of variability in antidepressant effect is due to differences in true effects of the drug, and 40% is due to random sampling error compares overlap between study Cis Relative measure- shows how much of pooled effect is NOT heterogeneity Outcome of 0 doesn’t mean NO heterogeneity
33
What is tau2
a continuous measure showing actual/absolute variance in true effects between studies More direct measure of magnitude of difference in true effect EG: Tau2 = 0.25 means true variance of 0.25 in effect size, so the antidepressant effect differs by this amount across studies
34
What is meta regression
Meta-regression: looks at study-level potential moderators of ES and the actual ES across studies See if study factors like design, sample size, measurement tool, population influence overall meta-analysis preferred alternative to subgroup analyses as keeps continuous so more informative (though splitting is a form of)
35
How to do meta regression
Effect sizes regressed onto study-level characteristics: like linear regression, coefficients tell you how much each moderator associated with the ES ES for each study becomes the dependent Plot study-level X variable against SMD, fit regression line
36
When to do meta regression
Use when: high ES heterogeneity across studies to ID where coming from, or when want to know effects of moderators
37
How to do subgroup analysis in meta analysis
Divide studies based on moderator variable, calculate pooled ES and heterogeneity each group (>50% suggests too diverse) Compare ES and heterogeneity between subgroups: Q-test for statistical differences in ES
38
What is Cochrans Q
I2 alternative; measure of effect heterogeneity
39
What does Cochran’s Q tell you
significant p-value (typically less than 0.05) rejects the null hypothesis, indicating that the true treatment effects are not consistent - can be underpowered with small numbers of studies
40
Considerations before subgroup analysis
Can still do with 3 studies, just low power and risk of overfitting Check heterogeneity – no point doing if low Alternatives: sensitivity analysis excluding one at a time, qualitative trend descriptions
41
Define meta analysis
weighted pooled effect size estimate, with study weight determined by: 1) Study size (larger = more impact) 2) Frequency of events (more = more statistically informative even if sample smaller)
42
Define fixed effect MA
Fixed/common effect (singular): assumes there is a single true value (effect of an intervention) that all studies are estimating Study outcomes distributed around this, shows how big chance error is around true value Only thing preventing getting true value is chance error which reduces with bigger sample size – why bigger studies given more weight Similar idea to CI where giving range of uncertainty around a true effect
43
Define random effects MA
Random effects (plural): assumes there is no measurable true value - instead each study estimating the distribution of effects around true value Measuring both chance error and random effect – why weight more distributed across studies though sample size still matters Larger CIs Even with infinite sample would never get true effect - best case scenario is a distribution around Always random with humans No harm assuming random even if truly fixed, as you would still get the outcome without distribution around
44
What is a forest plot
visual of pooling estimate Summarize between-group effects at study level around middle line of no difference culminating in weighted pooled estimate For each study: N, mean, SD for TRT and CTL group Plot: mean difference between groups with CI bar Size of plot shows study weight Middle line: no effect Bottom: summed totals of MD, CI, weight all studies, adds to 100% Final estimate: diamond of weighted mean difference with own prediction interval (red)
45
Define funnel plot
Funnel plot: accounts for issues with small study effects NOT publication bias Plot each study ES against SE(size of study) Middle line of true effect Assumption: larger studies closer to real effect (high y-axis SE, central ES at line; triangle point), smaller studies more spread around estimate (low y-axis SE, deviated ES from middle; triangle base) Visual evidence impacting pooled estimate: dots outside funnel, gaps on one side Contour enhanced: shows likelihood of outliers with p-value
46
47
Limitation to meta regression
Issues with X choice- can’t combine properties of study with properties at individual level (e.g. age of all subjects not mean age)
48
Define ROB
estimate of strength/certainty of results (not really quality) RoB2 – updated to look at specific outcomes; only do for the primary outcome of interest
49
Define risk ratio
Ratio of the risk (probability) of event out of all event possibilities in TRT relative to CTL
50
Define odds ratio
Ratio of the odds of event versus non-event outcomes in TRT relative to. CTL
51
Define Relative risk ratio: RRR (relative risk reduction)
Percentage risk of event is reduced in TRT relative to CTL
52
Risk difference: RD (absolute risk ratio: ARR)
Absolute difference in risk of event in TRT compared to CTL
53
Define Number needed to treat: NNT
Number of people needed to reach one occurrence more/less in TRT vs. CTL
54
GRADE assessment
GRADE: additional domains of evidence beyond ROB – for confidence in overall MA estimate Upgrading: large effect, dose-response, direction of residual confounding and biases Downgrading: bias, inconsistency (no overalp between Cis; NOT hetero), indirect generalizability, imprecision/CI boundary around actual no effect, pub bias
55
Forest plot for Dichotomous studies
same but mean difference is the OR ratios For each study: n/N for TRT and CTL group (number of events or deaths/total group sample) Middle line of no effect= 1 (+ favours CTL, - favours TRT) Bottom: summed totaled to 100% Estimate: test for overall effect, with p-value
56
Define Full Information Maximum Likelihood (FIML)
Fr missing data Works by maximizing the likelihood of the observed data for each individual, allowing for valid parameter estimation without imputing missing values—making it a statistically robust method under the MAR assumption
57
Positive of FIML
Uses all available data even when some values are missing without imputing missing values explicitly - works directly with the likelihood function based on the observed data. Provides unbiased and efficient parameter estimates under MAR. Integrates missing data handling into the model estimation process. Uses the non-missing values to inform the estimation. Preserves sample size and statistical power by including incomplete cases Unlike listwise deletion (which drops incomplete cases) Cases with partial data still contribute to the analysis Avoids problems of imputation, like underestimating standard errors. Widely supported in structural equation modeling and mixed models.
58
Assumptions of FIML
Assumes data are missing at random (MAR) — missingness related to observed data but not unobserved -depends only on observed data, not the missing values themselves Requires appropriate model specification.
59
FIML vs other
Better than traditional methods (e.g., complete case) which assume MCAR or ignore missingness mechanisms “FIML is preferred because it efficiently uses all available data, reduces bias, and maintains statistical power, making it a robust approach for handling missing data in RCTs.”
60
How FIML works
Model-based estimation: assumes a statistical model (e.g., linear regression, SEM). Calculates the likelihood (i.e., probability) of the observed data given the model parameters, for each case. It maximizes the total likelihood across all individuals to estimate the best-fitting model parameters.
61
Define Multiple Imputation (MI)
Creates multiple complete datasets by imputing missing values based on observed data patterns. Accounts for uncertainty by combining results. Assumes data are missing at random (MAR). creating multiple "filled-in" (imputed) datasets, analyzing them separately, and pooling the results to account for uncertainty.
62
What is Maximum Likelihood Estimation
Estimates parameters directly from incomplete data under MAR assumptions.
63
Single imputation methods
Single Imputation Methods: Mean imputation: replaces missing values with the mean. Last Observation Carried Forward (LOCF): uses last available data point.
64
Issues with single imputation
These underestimate variability and can bias results.
65
Principal component analysis
PCA: unsupervised, simply reduces number of predictors while capturing their maximum variance, irrespective of relationship to outcomes i.e. just reducing dimensionality Unguided
66
Factor analysis
Vs. FA: unsupervised identifying the unobserved latent factors explaining predictor relationships i.e. just uncovering latent structures Guided but only one predictor
67
PLS regression
multivariate statistical method that models relationships between two data matrices: predictors (X) and responses (Y) Extracts latent variables (components) from predictor variables that explain maximum covariance with the response variables Combines features of principal component analysis (PCA) and multiple regression. Useful when predictors are highly collinear or when number of predictors exceeds number of observation
68
Assumptions in statistics
Normality, Homogeneity of variance, Independence of observations
69
Define linearity assumption
since models assume linear relationship between predictors and outcome
70
Define Homogeneity of variance assumption
ensure variance is similar across groups or predictors
71
Define independence of observations assumption
Absence of multicollinearity, inter-correlation between predictors distorts estimates
72
Define normality assumption
ensuring the data/residuals are normally distributed
73
How to test normality assumption
Via: histogram, quantile-quantile plot, box plot Shapiro-wilk (better for small n), Kolmogorov-smirnov, Anderson-darling
74
How to test homogeneity of variance assumption
Levene’s test (more robust to non-normality) Bartlett’s test Residual plots In regression pot residuals vs. fitted and look for random scatter
75
How to test linearity assumption
Scatter plots between predictor/outcome Residuals vs. predicted values: looking for random scatter
76
Test independence of observations assumption
Test for autocorrelation in regression residuals Absence of multicollinearity (regression): inter-correlation between predictors distorts estimates Correlation matrix of predictors Variance inflation factor Condition index and eigenvalue
77
Test for outliers
Box plots, z-scores Cook’s distance, leverage values
78
Define mixed effects models
Account for random effects of patients Especially important in repeated measures where measurements inter-correlated over time Accounts for individual variability between subjects when repeated measures over time
79
Mixed effects models vs. Repeated measures ANOVA
RM-ANOVA: no random effects, limited to simple balanced designs, assumes sphericity
80
Define power analysis
Estimated N based on fixed ES, sig threshold, power All interconnected
81
Define power
probability of correctly rejecting null when false /true effect Type 2 error 80 % power = 80% chance detecting if exists
82
Define P value
probability of rejecting null when true/type 1 error
83
Define linear contrast
Linear combination of group means Test specific comparison between levels of fixed effects More specific than just ANOVA overall group differenc
84
Define linear contrast
Linear combination of group means Test specific comparison between levels of fixed effects More specific than just ANOVA overall group differenc
85
Parametric vs non parametric test
Parametric tests are those that make assumptions about the parameters of the population distribution from which the sample is drawn. This is often the assumption that the population data are normally distributed. Non-parametric tests are “distribution-free” and, as such, can be used for non-Normal variables when comparing groups using tests. Eg/ Pearson vs spearman’s correlation
86
Why not always use non parametric tests if they work for both normal and non normal?
More statistical power, hard to do flexible modelling, more relevant to the population itself
87
Difference between Cochrane W, i2 and tau2 in meta analysis
All assess heterogeneity (variance) between studies, but differ in purpose: Q detects presence, I2 quantifies the percentage of total variance due to heterogeneity, and tau2 estimates the absolute variance of true effect sizes across studies (Presence, Percent, absolute)
88
How do you interpret Cochran’s Q
Cochran’s Q is a weighted sum of squared differences between individual study effects and the overall meta-analytic effect. Significance test-It tests the null hypothesis that all studies share a common effect size (homogeneity). Interpretation: A significant p-value (typically <0.10 due to low power) indicates that the variation among studies is greater than what would be expected by chance alone, suggesting heterogeneity exists.
89
How to interpret I2
estimates the percentage of total variability across studies that is due to heterogeneity rather than chance. Interpretation: 0%–25%: Low. 25%–50%: Moderate. 50%–75%: Substantial/High heterogeneity. > 75%: High/Significant
90
Interpret tau2
𝜏2 (Tau-squared) represents the absolute between-study variance in a random-effects model. It measures the variance of the underlying true effect sizes. Interpretation: A tau2 (r2) of 0 indicates no between-study variance (homogeneous), while a higher value indicates greater variance. More robust than I2 bc #studies and precision don’t matter, but need context to interpret outcome
91
Why is high i2 interpreted as bad if it’s a reverse estimate
indicates a large percentage of the total variance across studies is due to heterogeneity in true effect rather than sampling error (random chance). high value suggests that the studies are not estimating the same underlying effect, making a single, pooled average ("mean effect") potentially meaningless.
92
Interpret meta regression
As any regression, looking at significance Intercept: predicted effect size when all predictors are zero (or at their reference level). Coefficient (B): Indicates change in the average effect size for a one-unit increase in moderator. P-value: If the p-value is sig and the 95% CI does not include zero, moderator has a significant impact on the effect size. Residual Heterogeneity (I or tau2): Assesses if the model explains the differences between studies. A significant reduction in compared to a model without predictors suggests the covariates explain the variation. r2 Analog (Amount of Variance Explained): Indicates the percentage of true between-study variance accounted for by the covariates
93
Difference risk and odds ratio
Risk Ratios (RR) compare the probability of an outcome between groups, while Odds Ratios (OR) compare the odds of an outcome occurring vs. not occurring. RR is used in prospective studies (cohort/RCTs) for intuitive interpretation, whereas OR is used in retrospective studies (case-control) and logistic regression basic difference is that the odds ratio is a ratio of two odds whereas the relative risk is a ratio of two probabilities
94
Define homogeneity of variance
Assumltion that that different samples or groups being compared have approximately equal variances (spread of scores around their respective means.) a key requirement for ANOVA and t-tests, which compare mean differences. Violating this assumption can lead to incorrect conclusion