Home > AGW, Climate Change, Global Warming, Omniclimate, Science, Skepticism > Peer-Review Flaws, circa 2002

Peer-Review Flaws, circa 2002

People familiar with what happened in climate science during the last year might find Lawrence K. Altman’s NYT article “THE DOCTOR’S WORLD; When Peer Review Produces Unsound Science” of June 11, 2002 more than prescient (emphasis all mine, of course):

[…] Yet for all its acclaim, the system [of peer-review] has long been controversial. Despite its system of checks and balances, a number of errors, plagiarism and even outright fraud have slipped through it.

[…] A particular concern is that because editors and reviewers examine only what authors summarize, not raw data, the system can provide false reassurances that what is published is scientifically sound.

[…] Researchers reported [in the “The Journal of the American Medical Association” in June 2002] considerable evidence that many statistical and methodological errors were common in published papers and that authors often failed to discuss the limitations of their findings. Even the press releases that journals issue to steer journalists to report peer reviewed papers often exaggerate the perceived importance of findings and fail to highlight important caveats and conflicts of interest.

[…] Because the anonymous peers chosen to review manuscripts are often the authors’ scientific competitors, jealousies and competitive advantage can become factors in the reviews.

[…] The peer review system also tends to set a very high barrier for authors to publish truly novel findings.

[…] Yet research on peer review has found that many studies are conducted without the benefit of adequate consultation with statisticians, sometimes because none were available.

[…] Once statistical errors are published, it is hard to stop them from spreading and being cited uncritically by others. […]

Advertisements
  1. J.Stern
    2010/07/20 at 16:56

    One of the problematic gray zones in these matters is that one needn’t state something manifestly wrong to create a misleading bias. In my view, your critical innuendo is both factually correct in regard to profound problems with our productivity in the academic-industrial complex in general (*), but deeply misleading in so far as you seem to try to suggest that this is a problem specific to climate science. You neither mention that Altman’s article is not about climate science at all, nor do you provide any constructive criteria for how to systematically, structurally handle the issues any better. Yes, the IPCC and climate science are far from perfect and their efforts at transparency could be greater(**). But your blog is far more biased and devoid of explicit methodological criteria for sound criticism, say, than they are. You seem to make a career on that bias and remind me of tobacco-company employees who were indulging in “skepticism” and expressing their “sober, anti-ideological rational prudence” in regard of the evidence that smoking causes diseases, until they were just overwhelmed by the facts.

    Here are some data on temperature, out of zillions, and it really takes remarkable intellectual powers, such as yours, to ignore the obvious :
    http://data.giss.nasa.gov/gistemp/tabledata/GLB.Ts.txt

    So what? you might ask. well, there is much to do. First, digest the sad point that we are indeed on a thoroughly destructive path and then, pick your favorite area of involvement.

    (*) see, amongst many examples, eg.
    – Lewontin on Dishonesty in Science:
    http://www.nybooks.com/articles/archives/2004/nov/18/dishonesty-in-science
    (see also: http://www.nybooks.com/articles/archives/2005/feb/10/on-fraud-in-science-an-exchange/)
    -D. Goodstein, On Fact and Fraud: Cautionary Tales from the Front Lines of Science, Princeton UPress, http://press.princeton.edu/titles/9148.html
    -S.Shapin: The Scientific Life: A Moral History of a Late Modern Vocation. U Chicago Press, http://www.press.uchicago.edu/presssite/metadata.epl?mode=synopsis&bookkey=263302
    – L.Daston & P.Galison Objectivity, Zone Books
    etc

    (**)http://dotearth.blogs.nytimes.com/2010/07/17/climate-panel-clarifies-its-media-plan/

    • 2010/07/20 at 17:21

      J – one’d need to be an Olympian to follow the jumps in your logic. First how can I “seem to try to suggest that this is a problem specific to climate science” in a blog post that lists the problems of peer-review as identified eight years ago in a completely different scientific area?

      And why should I “mention that Altman’s article is not about climate science at all” when the title of the article starts with “THE DOCTOR’S WORLD” and it mentions “The Journal of the American Medical Association“?

      The point about providing “constructive criteria for how to systematically, structurally handle the issues any better” is moot. This is my blog and I will provide whatever I fancy providing, thank you very much. Furthermore, I have never claimed that any blog post or comment will be the definite word in any subject. Actually, this being a blog there will be plenty of ideas, suggestions and thoughts, and very little of complete treatises on a particular topic.

      The comparison to “tobacco-company employees” is meaningless. I might as well declare that your alarmism is unfounded and that it reminds me of the “pharmaceutical-company employees” that were oh so helpful in making us all worry in regard of the evidence that swine flu will kill millions, until they were just overwhelmed by the facts. And then…in what specific way would those comparisons help any of us understand a thing about the subject at hand?

      ps if my career were built on climate blogging, I would have had by now very little need to start a diet 😎

  2. Alan McIntire
    2010/07/19 at 15:35

    Here’s an example of a “peer reviewed paper” where even someone with only a knowledge of elementary algebra can spot the error:

    http://pubs.giss.nasa.gov/abstracts/1984/Hansen_etal_1.html

    Click on the Download PDF at the bottom.
    He estimates a feedback of 1.6 for water vapor, 1.3 for clouds, and 1.1 for ice/albedo effects.

    I think the feedback factors are overestimated, but take a look at equations
    #10 and #12. Anyone should be able to figure out that they’re obviously wrong.

    For a multiplier of 1.6, lambda must be 0.375 for water vapor, since 1/(1-0.375) = 1.6.
    likewise, lambda must be 0.231 for clouds and 0.091 for ice/albedo feedback, using Hansen’s figures.

    Hansen plugged in 1/(1 -.375-.231-.091) and got a multiplier effect of 3.3 total times the original increase of around 1.2 C.

    Using Hansen’s equation, with 3 multiplier effects of 0.333… each, you get a multiplier effect of
    1/(1 – .333… -.333…-.333…)= infinity.

    Using 4 feedbacks with lambda of 0.333, each, you get a multiplier of 1/(-.333) = MINUS 3, so instead of an increase of 1.2 C, you plug in that -3 multiplier and get a DROP of 3.6 C – obviously the equation is flawed badly.

  3. Will Delson
    2010/07/16 at 19:04

    That is indeed very interesting. Had you not prefaced this by telling me it came from outside climate science, I would have thought it was just more fallout from climategate. Thanks for sharing this.

  1. No trackbacks yet.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: