Saturday, September 15, 2007

Meta-science

From the Wall Street Journal today:

Dr. Ioannidis is an epidemiologist who studies research methods at the University of Ioannina School of Medicine in Greece and Tufts University in Medford, Mass. In a series of influential analytical reports, he has documented how, in thousands of peer-reviewed research papers published every year, there may be so much less than meets the eye.

These flawed findings, for the most part, stem not from fraud or formal misconduct, but from more mundane misbehavior: miscalculation, poor study design or self-serving data analysis. "There is an increasing concern that in modern research, false findings may be the majority or even the vast majority of published research claims," Dr. Ioannidis said. "A new claim about a research finding is more likely to be false than true."

The hotter the field of research the more likely its published findings should be viewed skeptically, he determined.
Unsurprisingly, the folks to seize a hold of this are no strangers to self-serving data analysis:

Exhibit #1: Blue Crab Blvd: "And what is one of the hottest fields right now? Why, it's global warming. What a surprise. "

Does Dr. Ioannides say that global warming research is one of the problem areas? No. This is entirely Mr. Crabs' interpolation.

Exhibit #2: Vox Popoli: "This doesn't surprise me in the slightest, especially in light of what we've learned in the course of the ongoing evolution debate."

Nothing that seems to confirm one's already set in stone opinions is likely to surprise one.

Of course, on the one hand we have a scientist critiquing scientists. Thus, science is corrected by more science. On the other hand, we have folks who will not believe anything science says unless it corresponds to their preconceptions and prejudices. Can you tell the difference?