Chapter 10: More on Experiments- Confounding and Obscuring Variables

Question 1

one group, pretest/posttest design?

Answer 1

• one group of participants, measured on a pretest, exposure to treatment, intervention or change and than measured on a posttest. (bad experiment)

Question 2

What are six threats to internal validity that the one group, pretest/posttest design introduces?

Answer 2

1. maturation threats
2. history threats
3. regression threats
4. attrition threats
5. testing threats
6. instrumentation threats

Question 3

Maturation threats ?

Answer 3

change in behaviour merges more or press spontaneously over time…people slowly adapt to strange environments.
L> spontaneous remission…symptoms get better for an unknown reason.

Question 4

Prevention of maturation threats?

Answer 4

comparison group

Question 5

History threats?

Answer 5

threats to internal validity that occur when a historical or external event occurs to everyone in the treatment group at the same time as the treatment so it is unrelieved ….to be one it must effect everyone or almost everyone

Question 6

Prevention of History Threats?

Answer 6

a comparison group

Question 7

Regression threat??

Answer 7

aka regression toward the mean
L> when a performance is extreme at time 1 the next time that performance is measured it is likely to be less extreme than that is closer to a typical of avg performance.

Question 8

Regress works at??

Answer 8

both extremeness

Question 9

Regression and internal validity?

Answer 9

when measured in a pretest condition are extreme on DV….most likely to be a threat
L> if a group is unusually high pretest we can expect their scores to go towards the number

Question 10

Preventing regression?

Answer 10

comparison

Question 11

Regression is a big threat when?

Answer 11

one group in exactly this situation

Question 12

Attrition??

Answer 12

occurs when people drop out of the study before it ends.

Question 13

How does Attrition usually happen in pret-test and post-test designs and when does it become a threat to internal validity?

Answer 13

• when the two tests are administered on separate days…..and its a threat to internal validity when it is systematic…aka when only a certain kind of participant drops out

Question 14

Is a comparison group a cure all for attrition?

L> if not why?

Answer 14

NO
L> in a two group experiment like a prettest or post-test design if both groups experience the same pattern of dropouts then attrition is not an internal validity threat. But it can be a threat when only one group experiences it.

Question 15

How do we prevent attrition threats?(2)

Answer 15

• remove the original scores of the participants that drop out so only the scores of those that fully complete the study are included.
• another way is to check the pretest scores of dropouts if they have extreme scores on the pretest their attrition is more threat to internal validity than if it was closer to the group avg.

Question 16

Testing threat?
L> what type of effect are they?
L> they lead to what five things?

Answer 16

• kind of order effect
• scores have changed over time just because participants have taken the test more than once……they become practiced….bored….jitters for the first testing, sensitization (pretest may sensitize the participant to certain things that influence their post-test answers)

Question 17

How do we prevent testing threats?

Answer 17

-abandon a pretest all together and use only post test design
- if a pretest is used.they might opt to use alt forms of the test for the two measurements.
- comparison group
L> it takes both the pre test and post test …if a larger effect is seen in the treatment group = testing threats are not harming internal validity.

Question 18

Instrumentation threats?

Answer 18

• aka instrumental decay

- occur when a measuring instrument changes over time from having been used before.

Question 19

How do testing threats differ from instrumentation threats?

Answer 19

they occur when a participant changes over time from having been tested before.
L> instrumentation = instruments used to measure

Question 20

How do we prevent instrumentation threats?

Answer 20

-use only a post-test only design
- if a pre-test, post-test is used….one should make sure the pretest and post-test measurements are equivalent
L> collect data from each instrument and be sure they are calibrated the same
- avoid shifting standards of behavioural coders….via training the coders.. multiple times throughout the experiment establishing reliability and validity at both pretest and post test.
- use clear coding manuals
- counterbalance the versions of the test giving some version A at pretest and version B at post test and giving other participants version B and than version A .

Question 21

Many threats to one group pretest/post-test design are corrected by adding a comparison group When one has a two group pretest/post-test design what are three more threats to internal validity ?

Answer 21

• observer bias, demand characteristics and placebo effects

Question 22

Observer bias?

Answer 22

when researchers expectations influence their interpretation of the results or even influence the outcome of the study.

Question 23

Observer biases can threaten what two validities?

Answer 23

• internal validity because an alt explanation exists for the results….and construct validity of the DV because it means that the results do not represent the true levels

Question 24

Comparison groups always or do not always control for observer bias?

Answer 24

• not
• if the experimenter knows what participants are in each group their expectations could lead them to see more improvement in the therapy groups vs the comparison group

Question 25

Demand characteristics?

Answer 25

when participants guess the studies purpose and change their behaviours in the expected direction.

Question 26

What can control for both demand characteristics and observer bias?

Answer 26

- double blind study in which neither the participants nor the researchers who evaluate them know who is in the treatment group and who is in the comparison group

Question 27

When trying to control for both demand characteristics and observer bias what can be used when double blind studies are not possible?

Answer 27

- a variation of it can be done | L> ex: in some studies participants know which group they are in but the observers do not...single blind study?

Question 28

How does only participating in one condition of an experiment control for demand characteristics and observer biases?

Answer 28

participants were blind to the reason

Question 29

Keeping observers blind to the conditions is especially important when?

Answer 29

they re rating behaviours that are more difficult to code

Question 30

Placebo effect?

Answer 30

when people receive a treatment and really improve but only because they believe they are receiving a valid treatment. L> inert pill, injection or therapy = placebo

Question 31

Are placebos imaginary?

Answer 31

no

Question 32

How do researchers rule out the placebo effects in a study?

Answer 32

- using a double blind placebo control study L> one group receives the real drug or therapy, the second group receives the placebo drug or therapy. Neither the person treating the patients nor the patients themselves know whether they are in the real group or the placebo group.

Question 33

If the treatment group shows a larger increase/improvement in comparison to the placebo group what does this mean?

Answer 33

- placebo effects plus the effects of the real drug

Question 34

If someone wants to show a placebo effect specifically you would need to include what?

Answer 34

- a no treatment comparison group one that receives neither drug nor placebo.

Question 35

Null effect??

Answer 35

when researchers find that the IV did not make a difference in the DV

Question 36

Why do we not hear much about null effects?

Answer 36

- journals, newspapers and webistes are more likely to report the results of a study in which the IV does have an effect

Question 37

Do null effects occur only in post-test only designs?

Answer 37

- no | - they also occur in within groups design or a pretest/posttest design and even correlational studies

Question 38

What is a very simple explanation that a null effect is occurring?

Answer 38

- that the IV really does not affect the DV

Question 39

What is a second possible explanation for a null effect?

Answer 39

- the study was not designed well enough ...aka the IV does effect the DV but some obscuring factor in the study prevented the researchers from detecting the true difference.

Question 40

When a study returns a null result sometimes the culprit is the design of the study. What are three explanations for this?

Answer 40

- weak manipulation, insensitive measures and reverse confounds

Question 41

Weak manipulations?

Answer 41

- how the researchers operationalized the IV...aka ask about construct validity ex: someone gave each group money 0, 0.25 and 1.00. vs giving each group 0, 5 and 150 dollars L> each group will yield very different patterns results

Question 42

Insensitive measures?

Answer 42

- researchers have not used an operationalization of the DV with enough sensitivity - when it comes to dependent measures it is smart to use dependent measures that have detailed, quantitative increments not just two or three levels.

Question 43

What are ceiling and floor effects?

Answer 43

- special cases of weak manipulations and insensitive measures. - cause IV groups to score almost the same on the dependent variable. - all scores are squeezed together either at the high end (ceiling) or the low end ( floor)

Question 44

Ceiling and floor effects can be the result of what?

Answer 44

- a problematic IV | L> as a result the various levels of the IV would appear to make no difference

Question 45

Other than a problematic IV what can also lead to ceiling and floor effects?

Answer 45

- poorly designed DV L> the measure for the DV results in low scores in all groups L> no room for between group variability

Question 46

Manipulation checks can hep with what?

Answer 46

- detect weak manipulations, ceilings and floors | L> help identify potential problems with the IV....a separate DV is included to make sure the manipulation works

Question 47

Explain confounds acting in reverse?

Answer 47

- when a study is designed in such a way that a confound actually counteracts some true effect of an IV

Question 48

What is noise?/noisy data?

Answer 48

- too much unsystematic variability within each group can rust in a null effect L> it gets in the way of detecting a true difference between groups. aka error variance

Question 49

Noisy data can come from three sources what are they?

Answer 49

- measurement error - individual differences - situation noise

Question 50

How does more variability within groups obscure the difference between the groups?

Answer 50

there is more overlap between the members of the two groups...the greater the overlap the smaller the effect size and the less likely it is that the two group means will be stat sig

Question 51

Why do most researchers prefer to keep within group variability to a minimum?

Answer 51

so that they can more easily detect between group differences

Question 52

Measurement error?

Answer 52

factors that can inflate or deflate a person's true score on a DV - this can be a reason for high within group variability

Question 53

Do all measurements involve a certain amount of measurement error?

Answer 53

yes but researchers need to keep those errors as small as possible

Question 54

When measurement errors are random what happens?

Answer 54

they cancel each other out across a sample of people and will not effect the group's average or mean. BUT an operationalization with a lot of measurement error will result in a set of scores that are more spread out around the group mean.

Question 55

With measurement erros the more sources of random error there are in a DV's measurement that more what exists?

Answer 55

more variability within each group in the experiment

Question 56

The more precisely and carefully a DV is measured the less what exists?

Answer 56

variably within each group which makes it easier to detect a difference between IV groups

Question 57

What are two solutions to measurement error?

Answer 57

1. reliable, precise measurements L> use measurement tools that have excellent reliability (internal, interrupter and test retest)...to reduce errors.....when a tool has good construct validity there will be less error in a measurement L> the more precise an experimenter measures the DV the less chance there is that that measurement error will obscure results. 2. Measure more instances L> measure larger sample....aka take more measurements L> the more participants there are the better the chances of having a full representation of all possible measurement errors so they cancel each other out and we get a better estimate of the true average for the group * also when a measurement tool has little measurement error...researchers will get away with having a small sample

Question 58

Individual differences???

Answer 58

- problem in independent group designs. - effect of spreading out the scores of the people within each group - data is mixed and far from consistent ...a lot of overlap

Question 59

Two solutions to individual differences?

Answer 59

1. use a within groups design instead of a independent groups design L> it compares each participant with themselves and controls for individual differences L> requires less participants and controls for individual differences in comparison to independent variables. L> similar effects are seen in matched groups design 2. Add more participants L> when a great deal of variability exists because of the individual differences, one simple thing to do is to measure more people...the more you measure the less impact any one extreme person will have on the avg. L> reduces impact of individual differences within groups and will increase a study's ability to detect differences between groups. ( think of t-test formula )

Question 60

Situation noise??

Answer 60

third factor that could cause variability within groups and obscure true group differences L> variability in the external situation can great variability within each group in an experiment

Question 61

How do researchers control for situation noise?

Answer 61

controlling the external situations L> sometimes these controls need to be extreme - they typically try to control the potential distractions that might affect the DV

Question 62

If measurement error, individual differences , and situation noise were all investigated and the study was found to be conducted appropriately one may conclude what??

Answer 62

that the IV truly does not affect the DV

Question 63

Publication bias??

Answer 63

- differences seem more interesting than null effects so scientific journals magazines and newspapers that pick up story ideas from them are more interested in IVs that matter than those that do not.