lecture 10 - experiments

Question 1

example: prospect theory

Answer 1

= famous experiment (random assignment) with participants having to respond to a scenario

scenario = outbreak of ‘‘Asian disease’’, 600 deaths expected

two programs/options to contain the disease:
two groups got different options, different framing (positive vs negative)

A. 200 will be saved -> 72%
B. 1/3 probability that 600 will be saved and 2/3 probability that no one will be saved -> 28%

C. 400 will die ->22%
D. 1/3 probability that no one will die and 2/3 probability that 600 will die -> 78%

conclusions: gains -> risk aversion (people want to keep what they have, rather than risk loosing everything), if you start from a position of loss than people are more willing to take risks

*Kahneman & Tversky

Question 2

(experiments = good or bad?)

Answer 2

in principle seen as ideal: random assignment -> conclusively establishing causal tests

• Lijphart: most nearly ideal method for scientific explanation, but unfortunately it can only rarely be used in polsci because of practical and ethical impediments
• in general more and more experiments used (also because of survey experiments (found in most surveys)) in recent years

Question 3

experiments: key components and types

Answer 3

• randomized intervention/treatment/manipulation (different conditions are assigned), there are at least two groups
  control/treatment groups
• assignment: random vs matching vs as-if/quasi
  sometimes shortcuts in assignment to make sure that diff groups are equivalent (that they e.g. have same age variations)
  quasi-randomization = when randomization is not possible (e.g. article by Mintz, Redd and Vedlitz, they can’t randomly assign as they have two distinct groups) = the exception

design:

• between-participants: analyse/compare treatment and control group
• within-participants: analysis/comparison between participants in the same group
  *participant is both control and treatment in the same time (reading)

Question 4

different types/designs

Answer 4

• laboratory: artificial, standardized, full control
• field: embedded in the real world, less control
  e.g. one area with marketing/election campaign, one without
  *people would not know they were part of an experiment
• natural ‘‘experiment’’: no random assignment => strictly speaking not an experiment
• survey: embedded in population survey

Question 5

example field experiment + natural experiment

Answer 5

field experiment = real world with manipulation/random assignment
Leonard Wantchekon: party led him randomly assign different types of campaign messages
success was measured with electoral outcomes

• public policy message = general policy goals (negative effect in most cases)
• patronage based clientelist message = promise to create jobs (positive effect)
• control group

natural ‘‘experiment’‘
COVID: ‘treatment’ effect of face masks in German city of Jena (most other cities did not oblige people to wear face masks)

• masks work
• cities without mask regulation functioned as control

= less conclusive: observed effect may be due to other cause/elements

Question 6

randomization - sampling vs assignment

+ how do you do random assignment?

Answer 6

random sampling = for external validity

random assignment = for internal validity (ruling out alternative explanations)

how to random assign?

• simple tools: coin, die
• random number tables (choose random starting points and then start reading/taking)
• software: random number generator
  ! keep in mind: computers produce random nrs, but algorithm, need to change the starting point each time to get truly random nrs

Question 7

2 basic experimental designs

Answer 7

posttest only = practical (e.g. if you want to see how people react to X)
RA treatment and control group
(RA -> M(x) Y1 and Y2)

• outcome Y is measured at the end, if the manipulation has an effect then Y1 and 2 are different

pretest-posttest = e.g. when you want to see how attitudes change

• outcome is measured at the beginning (before M(x)) and at the end (after M(x))
• this way you can see if Y changes within the group(s) itself

Question 8

solomon four-group design

Answer 8

RA:

• treatment group 1
• control group 1
• treatment group 2
• control group 2

treatment and control group 1 get pretest-posttest

treatment and control group 2 get posttest only

why?

• pretest can point participants to the elements to what the researchers are looking for
• solomon four-group design can rule this out (by comparing group 1 and 2)

!not used often: more participants + time necessary

Question 9

delayed effects design

Answer 9

pretest-posttest in four groups (2 control, 2 treatment)

measure the posttest for 1 control group and treatment group 1 earlier than that you do for control group 2 and treatment group 2

effects can change over time: this way you can see the difference between delayed effects and immediate effects (you can see if these occur)

Question 10

factorial design

Answer 10

2x2 factorial design: more than one causal factor

• want to look at interaction effects between multiple factors

X1, X2 pretest and posttest

• control: Y1 and Y2
• main effect 1: Y3 -> M(X1) -> Y4
• main effect 2: Y5 -> M(X2) -> Y6
• interaction effect: Y7 -> M(X1) and M(X2) -> Y8

aka: you do one control for both
you do 2 separate treatments
and one treatment where both factors are manipulated

Question 11

validity and ethical issues

Answer 11

validity

• ecological validity: lab experiments are really artificial -> hard to establish theories about real world (how much it matter depends)
• reactivity : people know they are in an experience and behave differently
  e.g. placebo effects
  e.g. demand characteristics
• Single Shot vs Repeated Exposure experiments (e.g frames and counter-frames (positive messages + negative messages)
• WEIRD samples: white educated industrialized rich democratic (most experiments done in populations with WEIRD)

lab experiments: strong in interal validity, not in external validity
survey + field: both

ethics:

• treatment usually requires concealment and/or deception
• field studies: real world effects, debriefing often not possible

!threats to internal and external validity in the article

Question 12

when are experiments useful

Answer 12

• communication, language and public opinion
  priming, framing, persuasion
• campaigning: field experiments
• policy making: effectiveness of different policies
• decision making
  quantitative process tracing: direct observation of behavior, e.g. Mintz, Redd and Vedlitz
  (qualitative process tracing: post hoc reconstruction (history) or participant observation)

Question 13

example: Mintz, Redd and Vedlitz

Answer 13

= computer-based laboratory experiment

RQ = gnerealizability of student samples in decision-making experiments (counterterrorism): decision-making strategies and decisions

scientific relevance = external validity of research

participants:

• elites = military officers
• public = students

manipulation (at the end see if the manipulations had impact: ask how big was the certainty e.g.)

• certainty that technologies function: high vs low
• framing: whether funding is certain 10% vs 90%

experimental design?
scenario manipulations (certainty and framing) are randomized, but participant pool is quasi-experimental

2x2x2 (between participants) -> 8 groups
factorial design (they look at interaction of factors)

*number of cells accessed maybe ceiling effect?