What are the 4 types of validity?
- Internal validity = can’t determine causation
- External validity = can’t generalize
- Statistical validity = can’t trust the statistical relationships (e.g., low power, reliability of measures, heterogeneity of respondents)
- Construct/measurement validity = can’t generalize measures or operationalize to theoretical ones
How might you resolve crossover or diffusion (threats to internal or construct validity)?
- Minimizing common influences over conditions (e.g., different therapist for each condition)
- Isolating participants geographically
- Measurement of txt implementation
- Propensity score analysis
How might you resolve implementation threats to validity (external, construct, or statistical) (e.g., delivery, receipt, nonadherance)
- Using txt manuals
- Training providers
- Verbal reminders
- On-the-spot instruction
- Observational measures
- Verbal or SMS reminders
- Treatment logs
- Measure delivery or receipt and use in analysis
How might you resolve extraneous factors that threaten external validity?
- Control as many as possible
- Measure unavoidable ones (and control statistically)
How might you control for post-assignment attrition?
- Measure attrition
- Minimize time and obstacles between randomization and txt
- Minimize treatment-correlated attrition
- Change your posttest estimator (intent-to-treat—don’t remove non-compliers—or average treatment effect or average treatment effect of the treated)
- Replace dropouts if attrition and replacement are random and both former and replacement participants have the same latent characteristics (UNLIKELY)
How can one improve rigor in the absence of a randomized trial?
- Identify and study possible threats to internal validity
- Add design elements and use stats to prevent, control, or explain confounds (e.g., propensity score matching, instrumental variable approaches)
- Pattern matching
What are the strengths and limitations of experimental, quasi-experimental, and other designs?
How can you improve research rigor in the absence of randomized trials?
Random assignment is only possible in experimental design and improves internal validity
- Enhances internal validity by guaranteeing probabilistic equivalence—can infer that manipulated variable caused differences between txt and control
- Methods
- Random number generator
- Lottery method
- Flip a coin
- More complex: randomized block designs
Random sampling improves external validity
- Takes place before assignment, acquiring total sample that can be assigned to conditions
- Enhances external validity bc sample matches population of interest so can infer relationships on pop level
- Known chance of being selected
- Independence of selection
- Unbiased (no human judgement or physical mixing)
- Sampling error = estimate for sample and population differ, which limits external validity and generalizability
- Random
- Coverage error
- Nonresponse error
What are the types of random sampling?
- Simple random sampling (AKA, SRS) (random selection from pop list)
- Systematic sample (Every kth element from list)
- Stratified random sampling—SRS drawn from each category (Helpful for small subpops; can sample proportionately and disproportionately & correct in analysis using weights)
- Cluster sampling—SRS from each cluster or grouping
What is sampling error?
When the estimate for sample and population differ, which limits external validity and generalizability
Can be:
- random
- coverage error
- nonresponse error
What is the difference between practical and statistical significance in interpreting effect sizes?
- Practical significance requires statistical significance but also an examination of effect size AND the context for that effect size
- Statistical significance only means that the difference observed is reliably non-zero and likely not due to chance alone but does not on its own (1) explain causal impact or (2) whether the difference observed is large or meaningful
What is effect size and what are some examples of commonly used effect sizes?
- Effect size = magnitude of the effect of treatment/ the amount of variation in outcome accounted for by a single parameter
- Effect sizes are independent of sample size
- Examples: Cohen’s d (mean diff), Cramer’s V (chi-square), R^2 (correlation), standardized betas (regression)
What are external and internal validity, and how do they affect fidelity/precision within samples/groups studied and the generalizability of study results?
- Internal validity: ability to make causal claims
- Evidence shown by how one addresses threats through good design and implementation
- External validity: ability to make claims about the relationship identified in relation to the overall target population
- Evidence shown by demonstrating systematic sampling methods and obtaining a representative sample that is not biased in relation to the overall pop of interest
- Internal validity can affect generalizability through fidelity and precision because inferences that are made without strong evidence of validity may not generalize outside of the study regardless of the representativeness of the sample
- e.g., if you identify a relationship among a representative sample but the finding is confounded by some historical event that is not factored into one’s design and analysis, then it would be incorrect to generalize that inference to the rest of the pop under different circumstances
- Generalizability is not possible without evidence of external validity (i.e., a study using a biased sample identified through non-probability sampling may find relationships between variables that are not applicable to the population of interest at large)
