Den sårbare videnskab - bevisernes potentielle flygtighed

Diverse — Drokles on March 2, 2011 at 7:50 pm

The New Yorker havde for et par måneder siden en rasende interessant artikel om videnskabelige bevisers skrøbelighed. Man har gennem længere tid registreret det foruroligende fænomen, at videnskabelige beviser kan forsvinde over tid efterhånden, som man udfører flere forsøg. Artiklen fokuserer især på medicinske forsøg, hvor en ny generation af psykofarmaka tegnede en revolution inden for behandlingen af sindslidelser, men efterhånden, som flere forsøg blev lavet dalede de ellers lovende resultater, forsøg for forsøg indtil de nye midler ikke var stort bedre end de gamle.

This phenomenon doesn’t yet have an official name, but it’s occurring across a wide range of fields, from psychology to ecology. In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.


According to Ioannidis, the main problem is that too many researchers engage in what he calls “significance chasing,” or finding ways to interpret the data so that it passes the statistical test of significance—the ninety-five-per-cent boundary invented by Ronald Fisher. “The scientists are so eager to pass this magical test that they start playing around with the numbers, trying to find anything that seems worthy,” Ioannidis says. In recent years, Ioannidis has become increasingly blunt about the pervasiveness of the problem. One of his most cited papers has a deliberately provocative title: “Why Most Published Research Findings Are False.”

The problem of selective reporting is rooted in a fundamental cognitive flaw, which is that we like proving ourselves right and hate being wrong. “It feels good to validate a hypothesis,” Ioannidis said. “It feels even better when you’ve got a financial interest in the idea or your career depends upon it. And that’s why, even after a claim has been systematically disproven”—he cites, for instance, the early work on hormone replacement therapy, or claims involving various vitamins—“you still see some stubborn researchers citing the first few studies that show a strong effect. They really want to believe that it’s true.”

That’s why Schooler argues that scientists need to become more rigorous about data collection before they publish. “We’re wasting too much time chasing after bad studies and underpowered experiments,” he says. The current “obsession” with replicability distracts from the real problem, which is faulty design. He notes that nobody even tries to replicate most science papers—there are simply too many. (According to Nature, a third of all studies never even get cited, let alone repeated.) “I’ve learned the hard way to be exceedingly careful,” Schooler says. “Every researcher should have to spell out, in advance, how many subjects they’re going to use, and what exactly they’re testing, and what constitutes a sufficient level of proof. We have the tools to be much more transparent about our experiments.”

In a forthcoming paper, Schooler recommends the establishment of an open-source database, in which researchers are required to outline their planned investigations and document all their results. “I think this would provide a huge increase in access to scientific work and give us a much better way to judge the quality of an experiment,” Schooler says. “It would help us finally deal with all these issues that the decline effect is exposing.”


While Karl Popper imagined falsification occurring with a single, definitive experiment—Galileo refuted Aristotelian mechanics in an afternoon—the process turns out to be much messier than that. Many scientific theories continue to be considered true even after failing numerous experimental tests. Verbal overshadowing might exhibit the decline effect, but it remains extensively relied upon within the field. The same holds for any number of phenomena, from the disappearing benefits of second-generation antipsychotics to the weak coupling ratio exhibited by decaying neutrons, which appears to have fallen by more than ten standard deviations between 1969 and 2001. Even the law of gravity hasn’t always been perfect at predicting real-world phenomena. (In one test, physicists measuring gravity by means of deep boreholes in the Nevada desert found a two-and-a-half-per-cent discrepancy between the theoretical predictions and the actual data.) Despite these findings, second-generation antipsychotics are still widely prescribed, and our model of the neutron hasn’t changed. The law of gravity remains the same.

Jeg kan ikke lade være med at tænke på klimadebatten i denne sammenhæng. Vores lidt uheldige evne til ubevidst at konstruere den virkelighed vi gerne vil bekræfte forklarer måske hvorfor nogle tror på en klimakatastrofe. Eller måske forklarer det, hvorfor vi er nogle der afviser det. Men det forklarer i hvert fald, hvorfor debatten er så ophedet og fronterne så stejle.

Mere vigtigt bærer det først og fremmest en advarsel til politikerne om ikke at lade sig rive med af, hvad der meget vel kan være en bogstavtro tro på videnskabens evne til at lede os. Som artiklen slutter “When the experiments are done, we still have to choose what to believe”. Og for de, der går med hovedet oppe i røven er enden altid nær. Videnskab er en søgen og ikke en sandhed og jeg vil op til og hen over weekenden komme med et par opfølgende posteringer dels for at uddybe denne pointe og dels fordi jeg har en barnlig lyst til at lave “en serie artikler”. Hele New Yorker-artikelen kan selvfølgelig varmt anbefales.

0 Kommentarer »

Ingen kommentarer endnu.

RSS feed for comments on this post. TrackBack URI

Kommentér indlægget...

Monokultur kører på WordPress