- papers are only peer-reviewed, not reproduced
- many papers cannot be reproduced
- only 36% of papers were reproducible (with psychology)
- 2/3 of all papers are just wrong
- astrophysicists are statistically proven wrong the most times → therefore best papers
- https://en.wikipedia.org/wiki/Replication_crisis
Why did it happen?
- publishing is a wrongly incentivised field
- incentive for answer to be…
- always correct
- universally applicable
- easy to believe
Remedies
- the pressure is not as high to create positives, nulls are ok and are published
- data must be public, so at least the statistics are reproducible
- pre-registration: defining the experiment before actually doing it and analyzing
- meta-analysis: taking a bunch of papers (40-100) and finding out about differences
- look for a meta analysis when diving into new subjects!!!
Methods of Deception
p-hacking
Effect Size
- huge effect: women vs men weight difference
- minimum 80 samples needed to be significant
- should be 100 or 200 for most studies
- small effect: milliseconds spent when looking at numbers
- much larger sample sizes needed
- using a too small sample size proves insignificant and is unlikely to be reproducible
Harking
Funnel Plot:
- plot result vs population size
- easy to spot publication bias
- if some parts/results are left out one will find it here
- mostly “too boring” to be published
