Huge reproducibility project fails to validate biomedical studies

All it takes is 14 grad students studying the same thing targeting a 95% confidence interval for, on average, one to stumble upon a 5% case. Factor in publication bias and you get a bunch of junk data.

I think I heard this idea from Freakonomics, but a fix is to propose research to a journal before conducting it and being committed to publication regardless of outcome.

The state of CS papers is truly awful, as they're uniquely poised to be 100% reproducible. And yet my experience aligns with yours in that they very rarely are.

It is an ongoing crisis how much Alzheimer’s research was built on faked amyloid beta data. Potentially billions of dollars from public and private research which might have been spent elsewhere had a competing theory not been overshadowed by the initial fictitious results.

Pre-registration is a pretty big one: essential you outline your research plan (what you’re looking for, how you will analyze the data, what bars you are setting for significance, etc.) before you do any research. You plan is reviewed and accepted (or denied), often by both funding agency and journal you want to submit to, before they know the results.

Then you perform the experiment exactly* how you said you would based on the pre-registration, and you get to publish your results whether they are positive or negative.

* Changes are allowed, but must be explicitly called out and a valid reason given.

https://en.wikipedia.org/wiki/Preregistration_(science)

Disclosure: I'm a scientist, specializing in scientific measurement equipment, so of course reproducibility is my livelihood.

But at the same time, I doubt that fields like physics and chemistry had better practices in, say, the 19th century. It would be interesting to conduct a reproducibility project on the empirical studies supporting electromagnetism or thermodynamics. There were probably a lot of crap papers!

Those fields had a backup, which was that studies and theories were interconnected, so that they tended to cross-validate one another. This also meant that individual studies were hot-pluggable. One of them could fail replication and the whole edifice wouldn't suddenly collapse.

My graduate thesis project was never replicated. For one thing, the equipment that I used had been discontinued before I finished, and cost about a million bucks in today's dollars. On the other hand, two labs built similar experiments that were considerably better, made my results obsolete, and enabled further progress. That was a much better use of resources.

I think fixing replication will have to involve fixing more than replication, but thinking about how science progresses as a whole.

A dream of mine was that in order to get a PhD, you would not have to publish original research, but instead you would have to _reproduce existing research_. This would bring the PhD student to the state of the art in a different way, and it would create a natural replication process for current research. Your thesis would be about your replication efforts, what was reproducible and what was not, etc.

And then, once you got your PhD, only then you would be expected to publish new, original research.

Reproducibility studies are costly in time, reagents, and possibly irreplaceable primary samples. I usually would prefer a different study looking at similar mechanisms using different methods than a reproduction of the original methods, although there’s an important place for direct replication studies like this as well. We can also benefit from data sleuths uncovering fraud, better whistleblower systems, and more ability for graduate students to transfer out of toxic labs and into better ones with their funding, reputation and research progress intact.

Scientists have informal trust networks that I’d like to see made explicit. For example, I’d like to see a social media network for scientists where they can PRIVATELY specify trust levels in each other and in specific papers, and subscribe to each others’ trust networks, to get an aggregated private view of how their personal trusted community views specific labs and papers.

Yes, but nobody wants to acknowledge the elephant in the room. Once again, this is why defunding research has gained merit. If more than half of new research is fake, don't protest when plugs are being pulled; You're protesting empirical results.

I had always envisioned an institute for reproducibility & Peer review. It would be a federally funded institute that would require Phd candidate participation as an additional requirement to receive your degree. Really it wouldn't be a single place but office or team at each university where proper equipment was available and perhaps similar conditions for reproducing specific research. Of course the feasibility of this is pretty low.

IMO, stopping the race toward better h index.

There is an huge amount of pressure to publish publish publish.

So, many researchers prefeer to write very simple things that are probably true or applicative work, which is kind of useful, or publish false/fake results.

On the data analysis side, I think making version control both mandatory and automatic would go a long way.

One issue is that internal science within a company/lab can move incredibly fast -- assays, protocols, datasets and algorithms change often. People tend to lose track of what data, what parameters, and what code they used to arrive at a particular figure or conclusion. Inevitably, some of those end up being published.

Journals requiring data and code for publication helps, but it's usually just one step at the end of a LONG research process. And as far as I'm aware, no one actually verifies that the code you submitted produces the figures in your paper.

It's a big reason why we started https://GoFigr.io. I think making reproducibility both real-time and automatic is key to make this situation better.

Are you one of those people who somehow believe that, because the pop culture "chemical imbalance" ideology was never factual, SSRIs don't work.

They are continually prescribed because their actual mechanism doesn't matter, they demonstrably work. That is a matter of statistics, not science.

Anti-science types always point to the same EXTREMELY FEW examples of how science "fails", like Galileo (which had nothing to do with science) and ulcers.

They never seem to point to the much more common examples where people became convinced of something scientifically untrue for decades despite plenty of evidence otherwise. The British recognized a link between citrus and scurvy well before they were even called "Limeys"! They then screwed themselves over by changing some variables (cooking lime juice) and instead turned to a quack ("respected doctor" from a time when most people recognized doctors were worse than the sickness they treated) who insisted on alternative treatment. For about a hundred years, British sailors suffered and died due to one quacks ego.

Phrenology was always, from day one, unscientific. You STILL find morons pushing it's claims, using it to justify their godawful, hateful, and murderous world views.

Ivermectin is a great example, since you can create a "study" in Africa to show Ivermectin cures anything you want, because it is a parasite killer and most people in impoverished areas suffer from parasites, so will improve if they take it. It's entirely unrelated to the illness you claim to treat, but nobody on Facebook will ever understand that, because they tuned out science education decades ago.

How many people have died from alternative medicine quacks pushing outright disproven pseudoscience on people who have been told not to trust scientists by people pushing an agenda?

How much money is made selling sugarpills to idiots who have been told to distrust science, not just "be skeptical of any paper" but outright, scientists are in a conspiracy to lie to you!

The answer is straightforward. They are a coalition of Brazilian labs (click on the link in the first sentence to get more information) so it seems normal that they would be focusing on the research conducted in their country. Also it is not the first research of its kind as the nature article provides context

> The teams were able to replicate the results of less than half of the tested experiments1. That rate is in keeping with that found by other large-scale attempts to reproduce scientific findings. But the latest work is unique in focusing on papers that use specific methods and in examining the research output of a specific country, according to the research teams.

"This goes 10X for such difficult sciences as psychology (which is literally still in infancy) and biology. In these fields, designing a proper experiment (controlling as much as you can) is basically impossible, so we have to tease signal out of noise and it's failure prone."

For psychology replace "Difficult" with "Pseudo".

To lose that tag, Psychology has to take a step back, do basic research, replicate that research multiple times, think about how to do replicatable new research, and only then start actually letting psychologists do new research to advance science.

Instead of that, unreplicated pseudo-scientific nonsense psychology papers are being used to tell governments how to force us to live our lives.

> Hell, go watch Youtube Chemists who have Phds fail to reproduce old papers. Were those papers fraudulent? No, science is just difficult and failure prone.

Agreed. Lab technique is a thing. There is a reason for the dark joke that in Physics, theorists are washed up by age 30, but experimentalists aren't even competent until age 40.