Challenges in irreproducible research : Nature News & Comment

Special

Challenges in irreproducible research

Science moves forward by corroboration – when researchers verify others’ results. Science advances faster when people waste less time pursuing false leads. No research paper can ever be considered to be the final word, but there are too many that do not stand up to further study.

There is growing alarm about results that cannot be reproduced. Explanations include increased levels of scrutiny, complexity of experiments and statistics, and pressures on researchers. Journals, scientists, institutions and funders all have a part in tackling reproducibility. Nature has taken substantive steps to improve the transparency and robustness in what we publish, and to promote awareness within the scientific community. We hope that the articles contained in this collection will help.

Editorial
Features
News and analysis
Comment
Perspectives and reviews

Editorial

Repetitive flaws

Strict guidelines to improve the reproducibility of experiments are a welcome move.

Nature (20 January 2016)
Let’s think about cognitive bias

The human brain’s habit of finding what it wants to find is a key problem for research. Establishing robust methods to avoid such bias will make results more reproducible.

Nature (07 October 2015)
Announcement: Time to tackle cells’ mistaken identity

Nature asks authors to check that they are not working on cells known to have been misidentified or cross-contaminated, and for more details about their cell lines.

Nature (15 April 2015)
Data-access practices strengthened

A new Initiative from Nature and the Nature research journals that strengthens our editorial links with the journal Scientific Data to enhance data-availability practices.

Nature (19 November 2014)
Journals unite for reproducibility

Consensus on reporting principles aims to improve quality control in biomedical research and encourage public trust in science.

Nature (05 November 2014)
Code share

Papers in Nature journals should make computer code accessible where possible.

Nature (29 October 2014)
Reducing our irreproducibility

A bid to improve the reliability and reproducibility of our published papers.

Nature (25 April 2013)
Further confirmation needed

A new mechanism for independently replicating research findings is one of several changes required to improve the quality of the biomedical literature.

Nature Biotechnology (10 September 2012)
Error prone

Biologists must realize the pitfalls of work on massive amounts of data.

Nature (25 July 2012)
Must try harder

Too many sloppy mistakes are creeping into scientific papers. Lab heads must look more rigorously at the data — and at themselves.

Nature (28 March 2012)

Features

How quality control could save your science

It may not be sexy, but quality assurance is becoming a crucial part of lab life.

Nature (27 January 2016)
The biggest mystery in mathematics: Shinichi Mochizuki and the impenetrable proof

A Japanese mathematician claims to have solved one of the most important problems in his field. The trouble is, hardly anyone can work out whether he's right.

Nature (07 October 2015)
How scientists fool themselves – and how they can stop

Humans are remarkably good at self-deception. But growing concern about reproducibility is driving many researchers to seek ways to fight their own worst instincts.

Nature (07 October 2015)
Reproducibility crisis: Blame it on the antibodies

Antibodies are the workhorses of biological experiments, but they are littering the field with false findings. A few evangelists are pushing for change.

Nature (19 May 2015)
Statistical errors

P values, the 'gold standard' of statistical validity, are not as reliable as many scientists assume.

Nature (12 February 2014)
Replication studies: Bad copy

In the wake of high-profile controversies, psychologists are facing up to problems with replication.

Nature (16 May 2012)

News and analysis

A mouse’s house may ruin experiments

Environmental factors lie behind many irreproducible rodent experiments.

Nature (12 February 2016)
Biotech giant publishes failures to confirm high-profile science

Amgen posts three studies at new online channel for discussing reproducibility.

Nature (04 February 2016)
Make journals report clinical trials properly

There is no excuse for the shoddy practice of allowing researchers to change outcomes and goals without saying so, says Ben Goldacre.

Nature (02 February 2016)
Surge in support for animal-research guidelines

Journals throw their weight behind checklist for rigorous animal experiments.

Nature (01 February 2016)
Missing mice: gaps in data plague animal research

Reports of hundreds of biomedical experiments lack essential information.

Nature (05 January 2016)
Brain-manipulation studies may produce spurious links to behaviour

Study shows that controlling neurons with light or drugs may affect the brain in more ways than expected.

Nature (09 December 2015)
Cancer reproducibility project scales back ambitions

Budget problems force replication project to drop one-quarter of its workload.

Nature (02 December 2015)
How to make biomedical research more reproducible

Neuropsychologist Dorothy Bishop discusses a UK report on irreproducibility in science.

Nature (29 October 2015)
Biomedical researchers lax in checking for imposter cell lines

Many scientists still not bothering to address an issue that undermines the reproducibility of research findings.

Nature (12 October 2015)
Reproducibility will not cure what ails science

A bill to make data for environmental regulation more transparent reveals the fuzzy boundary between science and ideology, argues Daniel Sarewitz.

Nature (09 September 2015)
Over half of psychology studies fail reproducibility test

Largest replication study to date casts doubt on many published positive results.

Nature (27 August 2015)
Irreproducible biology research costs put at $28 billion per year

Study calculates cost of flawed biomedical research in the United States.

Nature (09 June 2015)
Sluggish data sharing hampers the reproducibility effort

Initiative trying to validate 50 cancer papers finds difficulty in accessing original study data.

Nature (03 June 2015)
Researchers argue for standard format to cite lab resources

Research Resource Identifier (RRID) aims to clean up poorly referenced data.

Nature (29 May 2015)
First results from psychology’s largest reproducibility test

Crowd-sourced effort raises nuanced questions about what counts as replication.

Nature (30 April 2015)
US societies push back against NIH reproducibility guidelines

'Premature' rules for preclinical research need more flexibility and greater community involvement, say scientific society leaders.

Nature (17 April 2015)
Unconscious thought not so smart after all

Study on decision-making stokes controversy over power of distracted mind.

Nature (27 January 2015)
Parasite test shows where validation studies can go wrong

First report from Reproducibility Initiative highlights importance of publishing details of experimental methods.

Nature (17 December 2014)
The automated lab

Start-up firms say robotics and software that autonomously record every detail of an experiment can transform the efficiency and reliability of research.

Nature (03 December 2014)
Independent labs to verify high-profile papers

Reproducibility Initiative aims to speed up preclinical research.

Nature (12 August 2012)
Reliability of 'new drug target' claims called into question

Bayer halts nearly two-thirds of its target-validation projects because in-house experimental findings fail to match up with published literature claims, finds a first-of-a-kind analysis on data irreproducibility.

Nature Reviews Drug Discovery (01 September 2011)

Comment

Blind analysis: Hide results to seek the truth

More fields should, like particle physics, adopt blind analysis to thwart bias, urge Robert MacCoun and Saul Perlmutter.

Nature (07 October 2015)
Crowdsourced research: Many hands make tight work

Crowdsourcing research can balance discussions, validate findings and better inform policy, say Raphael Silberzahn and Eric L. Uhlmann.

Nature (07 October 2015)
Robust research: Institutions must do their part for reproducibility

Tie funding to verified good institutional practice, and robust science will shoot up the agenda, say C. Glenn Begley, Alastair M. Buchan and Ulrich Dirnagl.

Nature (01 September 2015)
Standardize antibodies used in research

To save millions of dollars and dramatically improve reproducibility, protein-binding reagents must be defined by their sequences and produced as recombinant proteins, say Andrew Bradbury, Andreas Plückthun and 110 co-signatories.

Nature (04 February 2015)
Chemical con artists foil drug discovery

Naivety about promiscuous, assay-duping molecules is polluting the literature and wasting resources, warn Jonathan Baell and Michael A. Walters.

Nature (24 September 2014)
Preclinical research: Make mouse studies work

More investment to characterize animal models can boost the ability of preclinical work to predict drug effects in humans, says Steve Perrin.

Nature (26 March 2014)
NIH plans to enhance reproducibility

Francis S. Collins and Lawrence A. Tabak discuss initiatives that the US National Institutes of Health is exploring to restore the self-correcting nature of preclinical research.

Nature (27 January 2014)
The risks of the replication drive

The push to replicate findings could shelve promising research and unfairly damage the reputations of careful, meticulous scientists, says Mina Bissell.

Nature (20 November 2013)
If a job is worth doing, it is worth doing twice

Researchers and funding agencies need to put a premium on ensuring that results are reproducible, argues Jonathan F. Russell.

Nature (03 April 2013)
Research methods: Know when your numbers are significant

Experimental biologists, their reviewers and their publishers must grasp basic statistics, urges David L. Vaux, or sloppy science will continue to grow.

Nature (12 December 2012)
Face up to false positives

Scientists and journals must work together to ensure that eye-catching artefacts are not trumpeted as genomic insights, says Daniel MacArthur.

Nature (25 July 2012)
Drug development: Raise standards for preclinical cancer research

C. Glenn Begley and Lee M. Ellis propose how methods, publications and incentives must change if patients are to benefit.

Nature (28 March 2012)
Why animal research needs to improve

Many of the studies that use animals to model human diseases are too small and too prone to bias to be trusted, says Malcolm Macleod.

Nature (28 September 2011)

Perspectives and reviews

The fickle P value generates irreproducible results

Lewis G. Halsey, Douglas Curran-Everett, Sarah L. Vowler & Gordon B. Drummond.

Nature (26 February 2016)
Reproducibility: Experimental mismatch in neural circuits

The finding that acute and chronic manipulations of the same neural circuit can produce different behavioural outcomes poses new questions about how best to analyse these circuits.

Nature (17 December 2015)
Clarifying the terminology that describes scientific reproducibility

Ron S Kenett & Galit Shmueli

Nature (30 July 2015)
Reproducibility: changing the policies and culture of cell line authentication

Leonard P Freedman, Mark C Gibson, Stephen P Ethier, Howard R Soule,Richard M Neve & Yvonne A Reid

Nature Methods (28 May 2015)
Improved reproducibility by assuring confidence in measurements in biomedical research

Anne L Plant, Laurie E Locascio, Willie E May et al.

Nature Methods (28 August 2014)
Power failure: why small sample size undermines the reliability of neuroscience

Katherine S. Button, John P. A. Ioannidis et al.

Nature Reviews Neuroscience (10 March 2013)
Reuse of public genome-wide gene expression data

Johan Rung & Alvis Brazma

Nature Reviews Genetics (27 December 2012)
A call for transparent reporting to optimize the predictive value of preclinical research

Story C. Landis, Susan G. Amara et al.

Nature (10 October 2012)
Next-generation sequencing data interpretation: enhancing reproducibility and accessibility

Anton Nekrutenko & James Taylor

Nature Reviews Genetics (17 August 2012)
The case for open computer programs

Darrel C. Ince, Leslie Hatton & John Graham-Cumming

Nature (22 February 2012)
Believe it or not: how much can we rely on published data on potential drug targets?

Florian Prinz, Thomas Schlange & Khusru Asadullah

Nature Reviews Drug Discovery (31 August 2011)
Tackling the widespread and critical impact of batch effects in high-throughput data

Jeffrey T. Leek, Robert B. Scharpf et al.

Nature Reviews Genetics (14 September 2010)

See other News & Comment articles from Nature

Biological boom

What sparked the Cambrian explosion?

An evolutionary burst 540 million years ago filled the seas with an astonishing diversity of animals. The trigger behind that revolution is finally coming into focus.