Tuesday, July 8, 2014

Are replication efforts useless?

A nice little dust-up is happening in neuroscience right now:  An experimental neuroscientist claims that we should not waste our time replicating published results. Why? because:

"unsuccessful experiments have no meaningful scientific value."

Richard Tomsett goes through the piece here: Are replication efforts pointless? And Neurosceptic has a good take-down too: On "On the emptiness of failed replications"

The gist of the argument is that experiments can fail for any number of reasons, and so they can't falsify the published result. Null findings should not, in his view, even be published at all. He only gives a cursory nod to the possibility that the initial positive result may be false, then proceeds to ignore it.

This sounds almost bizarre. But here is the unstated assumption that his entire argument rests on: "I already know my idea is right, and the experiment is only there to confirm what I already know." His whole chain of arguments depends on this, and would make no sense without it.

In his view, an experiment is simply there to give evidence for something we already know (or wish) to be true. If it works, that confirms what we already know. A failed replication must thus fail because of experimental error of some kind; since we already know our hypothesis must be true, that's the only inescapable conclusion. 

This attitude is the real danger here. If your base assumption is that your failures happen because of experimental error, not because your idea is wrong, then it can become ever so tempting to help an uncooperative experiment along just a bit. Add a few subjects — or remove a couple of "obviously" aberrant data points — to reach statistical significance. Clean up that blurry, messy picture a little. Don't include the failures in your analysis. Make the story clearer and neater. No need to actually run that time-consuming, expensive confirmatory experiment the reviewers wanted. We already know we're right after all.

I bet most cases of falsification and fraud in science started out from this assumption. People came in to the lab knowing they had the right idea, and simply wanted to get the confirmation that will convince everybody else. Telling people — young people just starting out — that this is the right attitude for doing science is dangerous.


Richard said...

You can reductio ad absurdam the entire essay by imagining he's writing about studies into the efficacy of homeopathy.

Jan Moren said...

Not sure what you mean. Yes, if your entire field is imaginary and you refuse to recognize that, then neither successful nor failures matter at all. You can always force any result to support your views.

I do imagine (I haven't checked his publication record) that he's one of those groups that make a good living by incremental dotting of i's and crossing of t's. In most cases he probably already does know what result he'll get since it's so similar to earlier, confirmed work. Many labs work like that by necessity, as that's what gives you dependable funding.

of course it's tempting for him to conclude that replication is pointless. But as I write, that attitude is insidious - and especially so when your work is incremental and not that likely to be falsified. You get lulled into a belief that real falsification simply won't happen, and that your approach is infallible.

Richard said...

I mean if you imagine he is talking about homeopathy research throughout the essay, it amplifies the wrongness of many of the arguments. Of course you hope that most people's hypotheses are predicated on better reasoning than homeopathy...

Jan Moren said...

Ah, OK; now I get what you mean.

Richard Carlsson said...

I read the title as referring to efforts to build replicants. Was slightly disappointed by the actual topic.

Jan Moren said...

That would be a much cooler topic, I agree. Maybe we could ask Prof. Mitchell to pen a rant against them for his triumphant return performance?