Quantitative Study Designs, Internal and External Validity - Week 6

Question 1

Validity

Answer 1

Older definition - Does the test (study) measure what it is supposed to measure? e.g. bench press to test upper body strength
Newer definition - how confident are we about the inferences made as a result of the study

Question 2

Two types of validity

Answer 2

Internal and external validity

Question 3

Internal validity (Face and content)

Answer 3

• Is it correct to infer a cause and effect relationship? Has researcher controlled all variables

a) “Face” (logical) validity - Test appears correct (e.g., quiz 1 - M/C questions from 2049 vs CHEM/BIO 1010;VO2 in dart throwers, etc.)

b) “Content” validity - rationale for each test item (to get valid results, content of a survey must cover relevant and necessary areas of subject it intends to cover. e,g, phone preference, quiz 1 knowledge, etc.)

Question 4

Internal validity (Construct, concurrent, predictive)

Answer 4

c) “construct” validity - does the “test” measure the (unobservable idea/concept? (e.g. indicators of emotions, symptoms of anxiety, etc.)

d) “concurrent” validity - compare “established” test to new one (e.g. step test vs treadmill test, “gold standard” vs field test, etc.)

e) “Predictive” validity - test score predicts future performance (e.g. MCAT, LSAT, high school grades vs University GPA, etc.)

Question 5

Threats to Internal Validity

Answer 5

Local history - all things that occur during, but outside the study (a hidden effect) e.g. dietary intake, PA behavior, car accident, etc.

Pre-testing - learning occurs each time a participant is tested (i.e. practice effects)

Maturation - (participants change over time) e.g. mental or physical changes, etc.

Instrumentation - (changes / differences in equipment)

Question 6

threats to internal validity - Differential participant selection/ Selection Bias Effect + Statistical Regression

Answer 6

• Any time groups are formed in non random fashion (pre-existing differences between groups) e.g. multi-stage surveys
  Statistical Regression - (scores move towards the mean)
• Participants who scored poorly will improve
• Participants who scored high will be more “average”
• e.g. MSF, race results, etc.

Question 7

Threats to Internal Validity (Mortality, John henry effect, and rating halo effect

Answer 7

Mortality: (non-random mortality is a type of differential participant selection) e.g. healthcare utilization across different occupations

“John Henry” Effect: The control group tries harder
- Need for blinding to condition (e.g. classroom resources, visualization / coaching for athletes

Rating / “Halo” effect: initial rating may influence the future ratings
- tendency to rate participants in middle of the scale
- can be influenced by marketing / perceptions / prior knowledge (e.g. “health halo”

Question 8

Error Producing variables - Extraneous variables

Answer 8

• Potential to affect the dependent variable; a source of unwanted variance
• Could come from within the participant; (fear, anxiety, motivation, learning)
• Could come from outside the research situation (sex, gender, socio-economic level, health, academic achievement, age).

Question 9

Two broad classes of External validity

Answer 9

• Population validity: generalize from sample subjects to similar population
• Ecological validity: generalize from experimental conditions to other conditions

Question 10

Threats to external validity (pretesting sensitization + Hawthorne effect)

Answer 10

Pretesting sensitization (“that’s interesting”) e.g. effect of pre-treatment with a drug, accelerometer/pedometer, or “message” (stereotype threat), etc.

Hawthorne effect - novelty of the treatment and participation in an experiment motivates the participant (“I’m a guinea pig.”)

Question 11

Threats to external validity (Expectancy)

Answer 11

(Placebo effect) - participant believes that treatment is supposed to change them, so they respond with a change. e.g., weight loss trials (supplements for satiety, etc.), weight training programs (reports of perception of muscle hypertrophy, etc.)

Question 12

reliability

Answer 12

• Refers to the consistency or repeatability of the data. e.g., measure CRF today and then one week later

Question 13

Sampling for Quantitative Studies

Answer 13

Probability Sampling
- Any method that ensures that the different units in the population have equal probabilities of being chosen
- Includes random selection, stratified random sampling and systematic sampling

Non-probability sampling
- Any method that does not use random selection
- Includes convenience or purposive sampling methods

Question 14

Different Types of Probablity Sampling

Answer 14

a. Simple random sample
b. Systematic sample
c. Stratified sample
d. Cluster sample

Question 15

Random Selection

Answer 15

• The sample can be randomly selected from the larger sampling frame or population using procedures such as:
• Random numbers table
• Computer programs
• Pulling names/numbers “out of a hat”
• A key to random selection is that the sample is representative of the population to which it is being generalized

Question 16

Stratified Random sampling

Answer 16

• The population or sampling frame is divided and grouped on a characteristic before random selection takes place
• This approach can be particularly important if there is a certain characteristic that needs to be represented in the sample

Question 17

Systematic sample

Answer 17

• A process whereby researchers use lists or inventories of units in a population to select every Nth entry for the sample. For ex. very 10th or 100th person
• There are some concerns with the use of lists since they have a characteristic in common and might have inherent biases

Question 18

Convenience Sampling

Answer 18

• A process of drawing a sample from groups of people that are familiar or convenient
• Clinicians might ask patients to participate in their studies
• Kin profs might ask students, coaches, and teams to participate

Question 19

Purposive Sampling

Answer 19

• Involves identifying units (e.g., organizations, people, teams) that represent a characteristic of interest
• the sample is identified with that purpose in mind
• several methods of purposive sampling
• Snowball sampling
• quota sampling
• expert sampling