1
Q
replicability
A
whether a published study’s findings can be repeated in a different lab, using the same or similar research methods
2
Q
the replication crisis in psychology
A
a failure for published research findings to be reproduced in other labs when they followed the same or similar research methods as the published findings
3
Q
the gold standard
A
a scientific finding’s replicability is often considered the best possible evidence for the accuracy of a finding
4
Q
reproducibility
A
the ability of a different researcher to reproduce another researcher’s published analyses, given the original dataset and computer code for the statistics used
5
Q
U.S. National Science Foundation (NSF) subcommittee’s definition of reproducibility
A
- Reproducibility refers to the ability of a researcher to duplicate the results of a prior study using the same materials as were used by the original investigator
- A second researcher might use the same raw data to build the same analysis files and implement the same statistical analysis in an attempt to yield the same results
6
Q
replication in social and cognitive psychology studies
A
Only 36% of social psychology and cognitive psychology studies published in scientific journals were replicated at the p < .05 level
7
Q
replication rates of cognitive vs. social psychology
A
- Cognitive psychology (50%)
- Social psychology (26%)
8
Q
why do researchers think there are lower rates of replication in social psychology?
A
Many researchers think that the lower rates in social psychology are due to the frequent use of surveys and the lack of control over who answers surveys
9
Q
replication in other science domains
A
- 70% reported a failure to replicate the results of other studies
- 87% of chemists
- 77% of biologists
- 69% of physicists and engineers
- 67% of medical researchers
- 64% of environmental and Earth scientists
- 62% of all other respondents
10
Q
replicating one’s own experiment
A
50% had failed to replicate their experiment
11
Q
do researchers agree that there is a replication crisis?
A
The majority of researchers surveyed agree that there is a replication crisis
12
Q
is the replication crisis specific to psychology?
A
no
13
Q
are most labs working to enhance replicability
A
yes
14
Q
common methods to enhance replicability
A
- Better document the research methods used
- Run the study again
- Ask the lab member to replicate the study
15
Q
why are replication rates lower for psychology?
A
the use of human participants can introduce significant variability
16
Q
failures to replicate in social psychology
A
- Stanford Prison Experiment: was conducted by Zimbardo and was widely criticized for ethical concerns and has faced skepticism regarding the generalizability of its findings. Subsequent replications and reevaluations have raised doubts about the validity of the original study
- The Bystander Effect: suggests that individuals are less likely to help in emergencies when others are present. This has faced mixed replication results. Situational and contextual factors may play a significant role in whether or not bystanders intervene
- Stereotype Threat: individuals from stigmatized groups underperform due to the fear of confirming stereotypes. This has faced mixed replication results, particularly in real-world settings
17
Q
failures to replicate in cognitive psychology
A
- Spotlight Attention: individuals focus their attention on a specific area of the visual field. This has faced challenges in replication. Some studies have found that attention is more distributed and flexible than originally theorized
- The Dual-Process Model of Memory: posits that memory consists of two separate systems, one for implicit and another for explicit memory. This model has faced scrutiny, and some researchers have proposed alternative models of memory
- The Mirror Neuron System: a system believed to play a role in understanding and imitating the actions of others and has faced some challenges in replication. Some studies have reported inconsistencies in the neural responses associated with mirror neurons
18
Q
failures to replicate in developmental psychology
A
- The Critical Period Hypothesis for Language Acquisition: implies that there is a specific window during which language learning is optimal. Some studies have reported individual variations and exceptions to this hypothesis
- Attachment Theory: describes different attachment styles in children. Some replication studies have suggested that attachment styles are more fluid and context-dependent than originally proposed
- The Mozart Effect on Infant Intelligence: suggests that listening to classical music, particularly Mozart, can boost children’s cognitive abilities, has faced some replication challenges. Subsequent studies have not consistently replicated the idea that listening to Mozart leads to long-term cognitive enhancements
19
Q
failures to replicate in clinical psychology
A
- The Dodo Bird Verdict: refers to the idea that all psychotherapies are equally effective and that it is the therapeutic relationship that accounts for positive outcomes. Specific therapeutic approaches, such as CBT, can have different effectiveness for certain conditions
- The Power of Positive Thinking: the idea that a positive attitude can lead to improved physical health outcomes has faced challenges. Replication studies have not produced consistent results
- The Clinical Efficacy of Memory Recovery Techniques: some memory recovery techniques, such as hypnosis and guided imagery, have been used in therapy to help individuals recall repressed memories. However, replication studies have shown that the accuracy of such recovered memories can be unreliable
20
Q
failures to replicate in neuroscience
A
- The Amygdala’s Role in Fear Processing: while the amygdala is traditionally associated with fear processing and emotional responses, some replication studies have reported variations in the amygdala’s involvement in fear- and non-fear-related tasks
- The Left Brain-Right Brain Distinction: the notion that the left hemisphere of the brain is primarily associated with logical thinking and the right is associated with creativity and emotion has faced challenges. Neuroscience research has shown that both hemispheres are involved in a wide range of cognitive functions
- The Brain-Training Effect: some studies have suggested that engaging in cognitive training exercises can improve cognitive function and memory in older adults. However, replication efforts have produced mixed results, with some studies failing to consistently replicate the benefits of brain-training programs
21
Q
causes of the replication crisis
A
- ignoring or misunderstanding statistics
- publication bias
- falsifying data
- quality of replication
22
Q
ignoring or misunderstanding statistics
A
- Misunderstanding the null hypothesis and the meaning of p-values
- Small sample sizes
- Effect size and power
23
Q
publication bias
A
the way we conduct, publish, distribute, and fund our science
24
Q
examples of falsifying data
A
- D. Stapel, 2011: psychology professor at Tilburg University who had 55 cases of fabricated data in social psychology
- Marc Hauser, 2012: psychology professor at Harvard who was accused of faking results on morality and cognition
- Karen Ruggiero, 2001: psychology professor at the University of Texas who was accused of faking results on discrimination research
25
quality of replication
failure to follow the original procedures
26
causes of failures in quality of replication
- This can be due to the decision-making of the replicator
- This can be due to incomplete method descriptions by the first researcher
27
HARKing (Hypothesizing After the Results are Known)
involves formulating or changing hypotheses after analyzing the data. In this practice, researchers may first explore their data without a specific hypothesis and then generate a hypothesis based on what they find. This can lead to confirmation bias, as it gives the appearance that the results were predicted when they were discovered post hoc
28
why does HARKing go against the principle of research strategy?
it shows that the IV causes (precedes) the change in the DV
29
SHARKing
- Secretly HARKing in the introduction section
- “Publicly presenting in the Introduction section of an article hypothesis that emerged from post hoc analyses and treating them as if they were a priori”
- Instead of making hypotheses based on results from all existing studies (standard practice), the researcher hypothesizes after knowing the results from the data at hand
30
is SHARKing justified?
no, it's never justified in science
31
THARKing
- Transparently (openly) HARKing in the discussion section
- “Clearly and transparently presenting new hypotheses that were derived from post hoc results in the Discussion section of an article”
32
is THARKing justified?
- Can promote effectiveness and efficiency of science
- Ethically required in some cases
- THARKing is justified in science, according to Hollenbeck
33
Hollenbeck's two case studies
1. A researcher desperate to get a job takes 30 of the shortest and most easily obtained survey measures and creates a pair of long questionnaires… they run a new survey find some significant correlations and publish them as a priori hypotheses. No one can replicate the findings. (example of HARKing)
2. Epidemiologists test 100 patients on new drug to protect against virus. Correlation between treatment (drug) and survival rate = r of .1 (small). Some researchers notice that females react differently to the drug than males. The researchers re-evaluate the findings by peak estrogen age of participants and publish a short report as a post-hoc analysis. Others replicate the findings. (example of THARKing)
34
Null Hypothesis Significance Testing (NHST)
an assumption that the null hypothesis means that there is no significant difference between the groups or conditions being compared
35
example of Null Hypothesis Significance Testing (NHST)
condition A yields the same outcome as condition B
36
issue with NHST
With a large enough N, virtually every study would yield significant results
37
p-hacking
the unethical and questionable practice of manipulating or "hacking" statistical analyses to achieve statistically significant results. Researchers may engage in various data techniques (trying multiple outcome measures, analyses, or excluding data points selectively) until a statistically significant result is obtained. This can lead to false-positive findings and misrepresentation of the true state of affairs.
38
methods of p-hacking
- Stop collecting data once p < .05
- Analyze many measures but report only those with p < .05
- Collect and analyze many conditions, but only report those with p < .05
- Use covaraites to get p < .05
- Exclude participants to get p < .05
- Transform the data to get p < .05
39
cherry-picking data
Researchers may selectively report only the data or results that support their hypotheses while disregarding data that contradicts their expected outcomes. This practice can lead to a biased representation of the research findings.
40
data fabrication and falsification
Intentionally creating or altering research data to support desired outcomes. It is a clear violation of research ethics and can have severe consequences, including academic and professional repercussions.
41
the file drawer problem
Studies with non-significant or null results are less likely to be published or reported compared to studies with positive or significant findings.
42
solution to the file-drawer problem
publish a meta-analysis that analyzes data from all articles on a particular topic (including outcomes that show no effect – whether published or not)
43
selective reporting
Researchers and academic institutions tend to favour the publication of studies that reveal a significant effect or a novel discovery, leading to an imbalance in the scientific literature.
44
solution to selective reporting
shift the balance toward journals that agree to publish null results when the research methods are judged in advance to be strong
45
incomplete knowledge
When non-significant results are not published or made
publicly available, the scientific community may have an incomplete or biased view of a particular research question, as only a portion of the available data is accessible.
46
solution to incomplete knowledge
larger sample sizes, reduce problems of internal validity and confounding variables
47
replication challenges
When other researchers attempt to replicate findings based on published studies, they may encounter difficulties if non-published studies with null results exist in the "file drawer" and have not been considered in the replication effort.
48
solutions to replication challenges
Open science practices (registration of research studies, publication of null findings) have been advocated to improve transparency and the reliability of scientific research
49
types of publication biases
1. the file drawer problem
2. selective reporting
3. incomplete knowledge
4. replication challenges
50
pre-registration of research methods
A detailed plan for research methods that are filed online (open) ahead of data collection
51
what does pre-registration include?
includes writing what your hypotheses are, which specific methodologies you’ll use, and which analyses you’ll conduct. These are set in stone
52
is there review prior to data collection with pre-registation?
no
53
what do you get when you complete pre-registration?
a DOI
54
DOI
a unique and permanent string of letters and numbers that identifies each article filled out online
55
can you pre-register studies that use secondary data?
yes
56
where can you find templates for pre-registration?
- It depends on the scientific field, but usually online
- For psychology, you use PsychArchives online
57
sections of pre-registration
- Title page
- Introduction
- Methods: sampling participants, IV, DV, materials, procedure
- Planned analyses
58
registered reports
peer review of the research methods prior to data collection
59
when are registered reports accepted?
if the authors follow through with the registered methodology
60
traditional review method for scientific articles
is only Stage 2 peer review when it is too late for the authors to adjust their research methods
61
stage 1 peer review
registered report method (review the research methods)
62
stage 2 peer review
traditional review of contents (did the authors use the research methods they promised)
63
pre-registered vs. registered reports
registered reports have stage 1 review and in-principle acceptance but pre-registered reports don't
64
prevalence of pre-registration vs. registration
There are many more pre-registered than registered studies
65
benefits of registered reports
- Encourages the creation of sound research questions and a priori methods and analysis plans
- Encourages open data and transparency
- Reduces publication bias
- Reduces p-hacking and HARKING
- Gives you guaranteed publication
- Gives you peer review when it’s most helpful
- Reduces selection bias
66
which of the benefits of registration reports DO NOT apply do pre-registration reports
- Reduces publication bias
- Gives you guaranteed publication
67
does pre-registration improve replication rates?
- Evidence suggests that pre-registration alone does not improve replication rates
- But, it seems to improve perceptions of higher-quality research
68
does registration improve replication rates?
- Evidence suggests that registered reports improve replication rates
- This doesn’t solve publication bias
69
why tharking may have value
- If we were just left to our powers of deductive, researchers may miss almost everything
- The history of science is replete with accidental findings (33-50% of findings)
- Historically low power in primary studies
- Most approaches to grounded theory rely on the richness of qualitative observations that cannot be replicated in a strictly technical sense
- Improves clarity in writing and exposition
- There are ethical costs of failing to leverage costly data
PSYC 306
flashcards
Decks in class (17)
# Cards
-
Lecture 13: The Correlational Research Design61
-
Lecture 14: The Experimental Research Design l52
-
Lecture 15: The Experimental Research Strategy ll70
-
Lecture 16: Between-Subjects Designs60
-
Lecture 17: Within-Subjects Designs67
-
Lecture 18: Single-Subject Designs44
-
Lecture 19: Factorial Designs l33
-
Lecture 20: Factorial Designs ll27
-
Lecture 21: Non-Experimental and Quasi-Experimental Designs41
-
Lecture 22: Replication Study Designs69
-
Ch. 12: The Correlational Research Strategy42
-
Ch. 7: The Experimental Research Strategy55
-
Ch. 8: Experimental Designs: Between-Subjects Design40
-
Ch. 9: Experimental Designs: Within-Subjects Design56
-
Ch. 14: Single-Subject Designs54
-
Ch. 11: Factorial Designs39
-
Ch. 10: The Nonexperimental and Quasi-Experimental Strategies66
