Replication Crisis

Question 1

Replicability VS Reproducibility

Answer 1

Replicability→ whether a published study’s findings can be repeated in a different lab, using the same or similar research methods

Reproducibility→ ability of a different researcher to reproduce another researcher’s published analyses, given the original dataset and computer code for the statistics used

Question 2

Why can reproducibility fail?

Answer 2

• process reproducibility failure→ unavailability of data or software
• outcome reproduced failure→ not the same result/ error either in original or reproduction study
  —>authors not given enough information about methods of data collection, analysis steps

Question 3

How to enhance replicability?

Answer 3

• better document the research methods used
• run the study again
• ask lab member to replicate the study

Question 4

What are the causes of the replication crisis?

Answer 4

Ignoring or Misunderstanding Statistics
Null hypotheses
-HARKing (Hypothesizing After the Results are Known)
Meaning of P-value
-Null hypothesis significance testing→ no significant difference between the groups
-P-hacking
-Cherry-Picking Data

Publication biases
-File drawer problem-> studies with non-significant or null results are less likely to be published
-selective reporting
-Incomplete knowledge

Falsifying data

Question 5

HARKing, THARKing and SHARKing?

Answer 5

HARKing (Hypothesizing After the Results are Known)→ formulate hypotheses after analysing data
Sharking (Secretly HARKing)→ hypotheses in Intro based on results from data at hand
Tharking (Transparently HARKing)→ clearly present new hypotheses in Discussion section of an article

Question 6

What are different P-hacking techniques?

Answer 6

P hacking→ unethical and questionable practice of manipulating statistical analyses in order to achieve statistically significant results
- Stop collecting data when p < .05
- Analyze many measures, but report only those with p < .05
- Collect and analyze many conditions, but only report those with p < .05
- Add covariates to reach p < .05
- Exclude participants to reach p < .05
- Transform the data to reach p < .05

Question 7

What is Cherry-picking data?

Answer 7

only report results supporting your hypotheses

Question 8

What are some solutions to replication crisis?

Answer 8

Pre-registration of Research Methods for a given study = A detailed plan for research methods that are filed online (open) ahead of data collection
- unchangeable
- no review prior to data collection
- get a DOI (Digital Object Identifier) to put in final paper

Registered report→ detailed plan for research methods filed online that undergoes peer review prior to data collection
- provisionally accepted for publication if the authors follow through with the registered methodology

Question 9

Meta Analysis VS Systematic Reviews?

Answer 9

Meta analysis->statistical combination of the results of multiple studies addressing a similar research question
- multiple replications from several studies

Systematic reviews-> No statistical analyses but search to identify all studies that meet those eligibility criteria
- assessment of the validity of the findings