1
Q
replication - the crisis
A
- Open Science Collaboration (2015)
- looked at hundred studies
–> taken from quite esteemed journal
–> arguably the best studies taken from the best journals at the time - only 36% of the studies
replicated when the collaboration replicated them again
–> only 23% from social psychology were replicable - scared psychologists
–> crisis of confidence
2
Q
cristea et al - emotion studies
A
- top 65 studies is emotion psychology
–> 40 cited observations
–> 25 experimental - showed greater effects than meta-analyses and large studies using the same questions
3
Q
what is meant by replication?
A
- doing the study again
- aim to see if the same findings are found
- more evidence we have that shows the same thing again and again, more likely we are to believe it
–> e.g. if one study finds ‘81% of people believe this’ it may be hard to believe, but if a second study finds ‘78% of believe this’ then they both become more believable
4
Q
define replication
A
repeatedly findings the same results
5
Q
benefits of replication
A
- Protects against false positives
–> e.g. sampling error - Controls for artifacts
–> maybe you had leading questions
–> maybe the look of the researcher impacted results - Addresses researcher fraud
- Test whether findings generalise to different populations
- Test the same hypothesis using a different procedure
6
Q
direct replication
A
- scientific attempt to recreate the critical elements of the original study
–> samples, questions, procedures, measures - The same or similar results are an indication that the findings are accurate and reproducible
- way of replicating the elements of the question you think are impacting results
- NOT EXACT replication
–> practically impossible in psychology
7
Q
conceptual replication
A
- test the same hypothesis using a different procedure
- same or similar results are an indication that the findings are robust to alternative research designs, operational definitions and samples
8
Q
registered replication reports - APA
A
- collection of independently conducted, direct replications of an original study, all of which follow a shared, predetermined protocol
- results of the replication attempts are published regardless of the outcome
9
Q
reasons for replication - faking
A
- Diederik Stapel
- found out he was faking and fabricated results
- hugely influential psychologist
- started off properly and fairly
–> found complicated and messy results across many variables
–> couldn’t bet his paper published
–> journal editors suggested cutting out the messy bits and added details - started to write more neat articles
–> made the data support the argument and create a narrative
10
Q
reasons for replication - sloppy science
A
- nine circles of scientific hell
1. limbo
–> seeing bad practice and not saying anything
2. overselling
–> focus on the bits of the study that worked
3. post-hoc storytelling
4. p-value fishing
–> outcome switching
5. creative outliers
–> deciding who to remove to make the data look better
6. plagiarism
7. non-publication
–> not publishing your papers
8. particle publication
9. inventing data - further down you are the worse it is
11
Q
outcome switching - sloppy science
A
- part of p-value fishing
- changing the outcomes of interest in the study depending on the observed results
- an example ‘p-hacking’
–> taking decisions to maximise likelihood of a statistically significant effect
–> rather than an objective or scientific grounds - if you do two tests, p value is now not 0.05 (5%) its actually 10%
–> if you found to be non-significant and one to be significant but you IGNORE the non-significant, you have changed you interested outcome due the results
12
Q
need for replication - small samples
A
- small samples and lack of statistical power can be a problem
- can say you have found an effect but if you find this in only a few number of people, that might not be the same in a larger sample
–> needs to be replicated in a larger group to be accepted
13
Q
need for replication - publication bias
A
- part of ‘non-publication’ and ‘partial publication’
- findings that are statistically significant are more likely to be published than those that are not
–> in general, there are good reasons for this - But could published studies represent the 5% of findings that occur by chance alone?
–> known as “the file drawer problem’ - quite scary to only see significant results
14
Q
how common is sloppy science?
A
- John et al (2012)
- Surveyed over 2,000 psychologists in the US about their involvement in questionable research practices
–> failing to report all the measures or conditions
–> deciding whether to collect more data after looking to see whether the results were significant
–> selectively reporting studies that “worked” - Concluded that the percentage of respondents who have engaged in questionable practices was surprisingly high
15
Q
peaking
A
- can’t stop study when you reach a peak
- have to decide on a set sample
- complete the entire study and the whole sample
- then calculate results
- data changes all the time, stopping at a point of your choosing doesn’t highlight the entire pattern
- might have just caught the data at a particular peak or stoop
16
Q
is sloppy science really a problem?
A
- Simmons, Nelson and Simonsohn (2011)
- Flexibility in data collection, analysis, and reporting dramatically increases actual false-positive rates
- Tested if listening to certain music makes you younger (2 songs)
- did find a significant effect that listening to a certain song made you 1.5 years younger
–> this is impossible
–> how? - they only reported the parts that gave them a significant effect
–> e.g. only 2 songs
–> controlling for father’s age - therefore appears impossible
17
Q
reporting using 0.05
A
- 0.05 means that there is a 1/20 chance you’ll find something
- if you test 20 things you will find an effect
- if you only report this, this looks like a significant effect
- if you reported all other 19 things and said they found no effect, people would question the significance of the 20th variable
–> but people didn’t use to do this
18
Q
moderators
A
- variables that influence the nature
–> e.g. direction and/or size of an effect - For example, country or culture
–> e.g. ”Reverse” ego-depletion - Identifying moderators is good because it improves our understanding.
–> “Second generation research”
–> moderators test when there is an effect –> 1st gen tests that there IS an effect, 2nd is WHEN
19
Q
poor replication - scientist error
A
- replication of a study done into elderly, priming and walking speeds (Bargh, 2012)
- Doyen replicated but didn’t find the same results
–> responded with ‘Doyen and her colleagues are “incompetent and ill informed” making “gross” methodological changes’ - replication needs to be reported whether it replicates results or not
20
Q
summarise replication
A
- replication is a cornerstone of science
- however, there are concerns that may findings in psychology may not be replicable
- there is, therefore, “substantial room for improvement with regard to research practices”
21
Q
one solution to poor science
A
open science
22
Q
open science
A
- the process of making the content and process of producing evidence and claims transparent and accessible to others
- without transparency, claims only achieve credibility based on trust in the confidence or authority of the originator
–> transparency is superior to trust
23
Q
open methodology
A
- documenting the methods and process by which those methods were developed / decided upon
24
Q
pre registration
A
- define research questions, methods and approach to analysis BEFORE observing the research outcomes
–> definitely before analysis - prevents HARKing
–> Hypothesizing After Results are Known
–> Hindsight bias
25
chris chambers paper
- we must register results
- moves to uphold transparency not only make psychology more scientific but they harness our knowledge of the mind to strengthen science
26
open science framework
- can upload all registrations
- they get time stamped and date checked
- can pre register what you want to do before you do it
- all get saved
- anyone can do it
- not peer reviewed
27
registered reports
- not the same as a registered replication report
- you can split the peer review process into two stages
28
two stages of the peer review process - registered reports
1. Reviewers and editors assess a detailed protocol
--> study rationale, procedure and a detailed analysis plan
2. Following favourable reviews (and probably revision to meet methodological standards), the journal offers acceptance in-principle
--> publication of the findings is guaranteed provided that the authors adhere to the approved protocol, the study meets pre-specified quality checks, and conclusions are appropriately evidence-bound
29
debate about the value of preregistration
- can slow things down
- can be constraining
- but they make it clear what is exploratory and what is not
--> if you do this it is, pre registering is fine
- avoids chance discoveries
- when comparing standard reports with pre-registered reports
--> standard reports publish supported hypotheses 95% of the time
--> pre registration only publish supported hypotheses 50% of the time, they ALSO report 50% of the time unsupported hypotheses
30
open source materials and code
- Use open source technology (software and hardware) and open your own technologies
--> e.g. the code used to programme the questionnaire or experiment
- this allows for better replication
--> people can use the exact same code and programme
--> or they can adapt it
31
open data
- make data set freely available:
--> allows other scientists to verify the (original) analyses
--> facilitates research beyond the scope of the original research
--> avoids duplication of data collection
- there some issues with open data
32
issues with open data
- where do we put it?
--> how will people find it
- how do you prepare the data for submission?
--> data needs to be FAIR (findable, accessible, interoperable, reusable)
--> anonymity / confidentiality
33
open access (publication of findings)
- traditional model of publication
--> researchers submit a paper to a scientific journal who decide whether or not to publish
--> researcher then signs copyright over to the journal who then charge universities / libraries for access
- problems:
--> unfair
--> limits access to those who have funds to pay for subscriptions
34
types of open access publishing
- gold open access
- green open access
35
gold open acces
- researchers (or more likely the funders or host institution) pay the journal to publish the article
- the final (formatted) version is freely accessible and permanently accessible for everyone
36
green open access
- Also referred to as self-archiving
- Put an (unformatted) version of a manuscript into a repository
--> e.g. psyarchive
37
effects of open access publication
- Open access works are used more:
--> within academia, open access works are cited between 36% and 600% more than works that are not open access
--> outside of academia, open access works are given more coverage by journalists and discussed more in non-scientific settings (e.g., on social media)
- Open access works facilitate meta-research
--> enable the use of automated text- and data-mining tools
38
issues with open access
- very expensive
- can cost 1000s to publish a paper and grant access
--> penalizes more junior researchers with less grants and little money to publish
39
Lisa Barrett - APS president
making open access complete an immediate is a great goal and a necessary element of any plan to democratize science
40
amount of open science
- very little between 2014 and 2017
41
ways to promote open access
- legislation
--> if funded by certain organisations, you have to do open science
- digital badges to acknowledge open science practices
--> there is evidence that badges promote open science (or at least open data)
--> seeing the badges made student teachers and social scientists trust the paper more but not the public
- societies
- groups
- guidelines
42
Transparency and Openness promotion - TOP guidelines
1. Citation standards
2. Data transparency
3. Analytic methods (code) transparency
4. Research materials transparency
5. Design and analysis transparency
6. Preregistration of studies
7. Preregistration of analysis plans
8. Replication
43
summarise open science
- open science and other practices that improve rigour increases the replicability of research
--> can lead to replication of up to 86% of studies (better than the previous 36%)
- makes research more credible
--> better to be credible than trying to be incredible
- its science done the right way
