Huge reproducibility project fails to validate biomedical studies

Summary

A replication drive focused on results that lean on three methods commonly used in biomedical research in Brazil. Credit: Mauro Pimentel/AFP/Getty In an unprecedented effort, a coalition of more than 50 research teams has surveyed a swathe of Brazilian biomedical studies to double-check their findings — with dismaying results.The teams were able to replicate the results of less than half of the tested experiments1. That rate is in keeping with that found by other large-scale attempts to reproduce scientific findings. But the latest work is unique in focusing on papers that use specific methods and in examining the research output of a specific country, according to the research teams.The results provide an impetus to strengthen the country’s science, the study’s authors say. “We now have the material to start making changes from within — whether through public policies or within universities,” says Mariana Boechat de Abreu, a metascience researcher at the Federal University of Rio de Janeiro (UFRJ) in Brazil and one of the coordinators of the project.The work was posted on 8 April to the bioRxiv preprint server and has not yet been peer reviewed.Ambitious undertakingThe massive experiment was coordinated by the Brazilian Reproducibility Initiative, a collaborative effort launched in 2019 by researchers at the UFRJ. The scientists wanted to assess publications “based on methods, rather than research area, perceived importance or citation counts”, de Abreu says. And they wanted to do so on a large scale. Ultimately, 213 scientists at 56 laboratories in Brazil were involved in the work.The project unfolded during the COVID-19 pandemic, which brought numerous logistical challenges. And teams disagreed about how closely to follow the tested protocols. “It was like trying to turn dozens of garage bands, each with its own way of playing, into an orchestra,” says project coordinator Olavo Bohrer Amaral, a physician at the UFRJ.Reproducibility trial: 246 biologists get diffe...