Why were experiments introduced?
Experiments: the primary mean of analysis - the core of all sciences
Karl Popper: good science has to be falsifiable - have to check whether it is true - via experiments, as theories are less useful if you cannot show where they are right or wrong
History
Chamberlain 1984 - introduced demand and cost structure
Ken Binmore: mobile telephony, people coordinated, cheated signalling simultaneously colluded without intending to through coding
mobile telephony auctions designed well, allocatively efficient but did not foresee the scope for collusion - experimental economics - experiment to mitigate the problem = companies, after rigorous testing were found not to be colluding
Standard Economic Theory - strictly dominant for people not to coordinate but in reality - we see people coordinating - why? social preferences!
Individual choice under uncertainty - Allais Paradox 1953 - people contradict each other!!
When to use experiments?
- Test theory
- most obvious motivation for running an experiment is to test whether theories are actually true.
- The Scientific Method- build a theory - solve via a lab experiment
- the key thing that experiments provide is control of various forms - institutions e.g. voting rules, communication
- incentives e.g. payoffs (not perfect- social preferences)
- measure and check confounding or unobservable variable e/g/ check in the field if it confirms your explanation
- randomisation (avoids self-selection problems)
=> problems with survey data: 1. noisy and 2. messy
=> lab: controls can make people behave in the way they are told - but external validity??
- Understand empirical regularities
e. g. things that we generally assume to be true e.g. Bayesian rule - and through experiments we can verify or challenge that - Inform Theory
To explain newly observed regularities, and devising new experiments to help distinguish among theories
e.g. behavioural economics, generalised and alternative models of expected utility theory, learning in games, etc.
every time an experiment contradicts standard theory - reveal something new - Evaluate assumptions
e. g. assumptions like as N increases, such and such happens for infinitely repeated prisoner’s dilemma games, at what point is N approaching infinity, informed through experiments
- Stress testing: a theory may fail for certain parameters, will it do better with others?
Experimental vs. Behavioural
Overlap heavily but there are significant differences, as experimental economics is a methodological field e.g. econometrics, that can be widely applied and not a subset of behavioural economics
A good experiment
Should it replicate reality?
- Trade-off between whether it should replicate reality and have the most external validity or to simplify it
- it should not replicate reality, reality is extremely complex and an experiment should try to simplify with levels of control, to learn something useful. for it to be useful, it has to be clear
A good experiment:
- is simple compared to reality and even simpler than relevant models - fewest variables
- is designed to test specific hypothesis
=> the more realistic, the more applicable
but also
=> the simpler you make it, the easier it is to show results, and avoid more endogeneity and causality issues
A GOOD EXPERIMENT MUST HELP AVOID CONFOUNDING:
A well-designed experiment might be the only way to disentangle explanations which would help in avoiding confounding theories that are equally plausible
Have to avoid:
- self-serving bias: favour information that increases your utility
- confirmation bias=ignore information that contradicts your beliefs, and accept info that confirms them.
Why bother disentangling them? => can produce different policy predictions!
Testing Alternatives
- Design by subtraction
e. g. twin studies, genetically identical but one difference i.e. brought up in a different environment - can test nature v nature - can isolate the effect of genes - Design may manipulation
change one parameter to make x more realistic, more feasible. a lab experiment should be simple and precise and combined with a realistic field experiment
Niederle and Vesterlund (2007): asked how people. wanted to be paid
e.g. - piece rate (safest way) - according to their position in the distribution of results
=> women were self-selecting into the piece rates, men were competing too much, like to take risks – testosterone correlated with risk-seeking preferences
Compare exercise control and treatment control
exercise control: how tightly you can define the knowledge, information, the environment of the experiment as a whole
treatment control: one group kept neutral as opposed to a treated group with a commitment device
Should only have one factor that is different between the two, in order to derive from that that it must be the thing that is different that generates the results - avoid confounds (don’t change more than one thing at a time)
How do we deal with uncontrolled factors?
We can deal with uncontrolled factors via randomisation
A good experiment must be randomised. Sometimes cannot be randomised e.g. ethical reasons, or technical issues
or we can measure variables which may affect fairness directly e.g. gender or age
to control for happiness, ask early on => priming, but may affect the behaviour as they would know they are being manipulated into being happy => behave unnaturally
Within vs. Between
Within-subject design: participants make decisions in all treatments e.g. one group and look at before and after treatment - do not have to worry about other changes in characteristics and individual effects
Under a within-subject design, each subject is its own control - do not have to worry about different characteristics => but: fatigue
Between-subject design: different participants make decisions in each treatment.
Should we have multiple rounds or one round?
- Multiple rounds may be an important part for robustness checks, but we there will be implications for learning e.g. about aptitude, risk preferences, etc.
- Overcome this by test of understanding before the main experiment begins - issue: fatigue: the individual may not exert the same effort in round 30 as they did in round 1.
Payment
Psychologists: paying people distracts them from the task at hand e.g. patronising => incentivisation crowds out effort.
Experiments in economics are always incentivised based on performance
How much?
- Enough, not trivial sums to match the real world
- sample selection problem: who is willing to join - risk profile, maybe risk averse less likely to join
The Small Stakes Problem
In practice, people do not walk in with 0 net worth
Rational standard economic theory : Permanent income hypothesis and lifecycle income - hard to get excited by the incentives brought up by the lab experiment
- individuals however show concern for risk in simple experiments where they lose or gain just £1 - maybe concerns for being right, or isolation effect?
Use language that is neutral, or frame/prime?
Now: formalised, and computerised, and they read themselves => neutral, no worries about priming or conditioning individuals
but if part of the experiment - can prime with mood-induction procedures such as make them watch a video, or listen to happy music, etc.
The right source
most common source: students - at Warwick we have SONA
in general: a good experiment has to identify an interesting question, determine a precise set of hypothesis and deal with confounding alternatives - in order to be able to draw inferences.
Problems with experiments
- Realism
- External validity - is it going to be valuable outside the lab? => linked to replicating reality
Solution:
- Experiments involve real subjects making real money
- Realism can be added in controlled steps - Representativeness
- Are the subjects representative of the real world?
Solution:
- can be tested and may not be important in some situations
- we can go into field experiments for greater realism and representativeness (will come back to this) - Incentives
- Less of an issue for experimental economics as the norm is to provide performance-related pay which provides extrinsic motivation.
- Subjects are likely to make more effort and there is likely more control over incentives
but disadvantages:
i) costly
ii) may limit the stakes and make things seem trivial
iii) may crowd out intrinsic motivation
- Clarity
- have to make sure people understand
- however if a lab is simulation of the real world, sometimes in the real world you don’t have to understand something unless you experience it
Solution: Clear wording, neutral language - help with understanding but: loss of control as you don’t know how subjects perceive their role - Priming and Framing
- Framing= how you describe a situation something may change behaviour
- Priming= the order in which you explain things may have an effect
Solution: clear statement, use neutral language, check the robustness of a result under different frames
Timeline of the Experiment
- Formulate a research question which needs to be falsifiable
- Choose design to address the research question: within vs. between, required number of observations, etc
- Prepare an experimental outline
- Seek funding
- Ethical approval
Design a pilot => prepare a questionnaire, useful for controls=>recruit for the pilot=>run pilot experiment=>run actual experiment=>analyse the data and write the paper
Good practice involves…
- No distractions
Good practice involves eliminated distractions and enabling privacy
- no talking and no distractions
- use appropriate screens if privacy is important - Pre-registration and data mining
- where you can list the main objectives, measures and hypothesis in advance
- by pre-registering you are committing to limiting yourself to specific ideas which strengthens your results if they correspond to pre-registered ideas - No deception
Experimental economics do not use deception, because with deception there comes a loss of control, especially when subjects know your are trying to deceive them, and then results are devoid of validity - Anonymity
identity: stigma attached, embarrassed to choose a profit maximising outcome
Three types of anonymity:
i) Single blind= no other subjects other than the researchers will be able to identify their actions
ii) Double blind** = subjects are guaranteed that no one can link their decisions to their identity, not even researchers
* *Gold standard
iii) Abandoning anonymity= if it is not important, but hard to get ethical approval
- Ethical issues
What is ethical depends on the discipline. Ethical approval normally requires high levels of anonymity and the ability for subject to withdraw when they wish
Sample Selection Bias
We want a random trial that starts with the recruitment process and means that people are randomly allocated
e.g. Warwick - SONA, allows for randomisation, but if experiments happen on a Wed, rules out sporty people;.
Filtering
Filter by age, subject, background, gender, nationality, lafuwafe and exclude subjects who have taken similar experiments before
Pre-conceptions
i.e. strong beliefs
=> important that individuals do not enter into the lab with pre-conceptions, not even about payment until they receive their first instructions (but give them some info for ethical reasons)
Field Experiments
In contrast with labs, they directly deal with external validity as they are tested in the real world
they are more realistic
other times, experimenters can run ‘hybrid’ experiments where they combine both lab and field.
Other benefits:
- the subject pool is spot on -e.g. use politicians for legislative bargaining
- you may want large samples, which are easier to attain in the real world
but if control is more important==> lab!
Why prefer labs?
- The lab provides more control
- it is easier to get strict instructions in the lab and are followed more easily
- more transparency whereas in the field random thing may happen due to millions of variables out of our control - prone to some bias
Kahneman and Tversky
Work on heuristics
Q: are people drawing Bayesian inference to correctly estimate probabilities, are they updating correctly?
Urns and balls:
2 urns filled with red and white balls:
Urn 1 has 75% red and Urn 2 has 75% white
Draw one ball with replacement. If it is red, there is a 3:1 - 75% chance it came from urn 1
Replace the ball, if it is red again, the chance it came from urn 1 rises 9:1
What people didn’t realise is that the more times they drew a red ball, the probability increased
Applications:
- Worker in a firm where there is a 75% chance good, and 25% chance bad. the more the tasks, the higher the probability the worker is good.
Conservative Bayesians?
Early work suggested that people update according to Bayes rules, but not always.
but: Kahneman and Tversky - there was a systematic failure to update enough - why? Heuristics: a way of thinking that is not rational/optimal
Heuristics are exacerbated by three things:
- anchoring - relying on the first piece of information
- representativeness - confirmation bias
- availability of information - relying on the most recent and the most pronounced memories e.g. peak end bias
