Or this[2] 2007 Science paper on ultra high energy cosmic ray source candidates ("anisotropy") that we had to retract because significance started dropping almost the day the paper was approved.
It was a fascinating experience as a junior member to follow the collaboration internal conversation and investigation on this, because a lot of extremely principled scientists were clearly deeply worried about losing their hard earned reputation. In the end, I am convinced that we were simply unlucky.
Reminds me of the FTL neutrinos too, where the scientist where pretty much "hey, something is wrong, can you help us figure it out?" and the general public were the ones screaming "OMG! Physics is dead!"
Then when it comes out as measurement error, the public is all "Damn these scientists are all hype machine clowns..."
Unfortunately it wasn't just the public: it caused so much uproar within the experiment that two of the highest ranking members resigned their posts [1].
I was a bit dismayed at the reaction within the physics community. Experiments absolutely do need to follow procedures like blinding and careful internal review (especially before the data unblinding), but you can only spend so long designing the analysis before you unblind, and there are opportunity costs to cross checking everything. In an optimized community experiments will inevitably make mistakes. And once you unblind, it does no one any good to sit on an anomalous result forever.
I’m fine with that. Put it at the feet of pop science blogging.
I’m less fine with the time and resources spent on mouse models. They already know you’d get the same utility from a magic 8 ball, but they do it anyway.
Mouse models are useful to discard very bad ideas. There was a recent experiment to use bacteria to kill cancer https://news.ycombinator.com/item?id=46306894 They tried like 40 bacterias in vitro, then like 9 in mice, and only 1 was useful in mice and they will continue only with that, perhaps in humans. Anyway, as you suggest, there is a high chance it will fail.
Also, you can do nasty stuff to mice that would never be allowed with humans. In that experiment they injected cancer cells in mice with a bad inmune system, so they could get like 90 mice with cancer and run the experiment in a short time. No ethical committee would approve that in humans.
As another comment points out: that's nothing compared to high energy/particle physics!
The Pierre Auger Observatory certainly is a large collaboration for the astroparticle physics domain though. It's a big international collaboration.
Quick anecdote: my name (S Mueller) is not on that paper's author list because we had a rule that you had to be in the collaboration for a year before getting authorship. You stayed on for a year after leaving. Very reasonable! At the time I was nonetheless a bit bummed about missing out on the big Science paper. I guess I'm on the retraction though ;)
It was a fascinating experience as a junior member to follow the collaboration internal conversation and investigation on this, because a lot of extremely principled scientists were clearly deeply worried about losing their hard earned reputation. In the end, I am convinced that we were simply unlucky.
[2] https://arxiv.org/pdf/0712.2843