HadCRUT4…not what it pretends to be!

Ungracious loser James Annan couldn’t wait posting about the new all-singing all-dancing HadCRUT4 dataset, showing “that reports of the death of global warming have been somewhat exaggerated“.

It’s really really hard to avoid laughing when adjustments come out of thin air but anyway…let’s consider HadCRUT4 less of a joke for a moment.

Since temperatures go up by including the upper Arctic, it is obvious that the rest of the world, and especially the inhabited regions, have not warmed as expected.

And a 5C increase in an area where the average is -20C is _not_ the same thing as a 5C increase where the average is +10C. The former is inconsequential, the latter a change in all seasons.

So HadCRUT4 simply confirms things are going as predicted only in faraway places where there are few measurements and nothing is really changing anyway.

If this doesn’t kill global warming, it certainly helps putting it in the right place.

Data Sharing And Proprietary Periods, Away From The Climate Madness

The policy for the future mission will require the immediate dissemination of data.
“You get a larger community and you get a bigger workforce for free,” he says.
“It’s clear that the more people you get involved, the more support you get.”

How often have climate scientists lamented their inability to share their data, either because evil people might use them “improperly”, or because it is unfair that somebody take advantage of all their hard work.

Is this a problem of climate science alone? Perhaps, it is. How strange.

From the UK Space Agency, about the Planck mission:

the proprietary period, which nominally ends with the release of the scientific products to the community 3.5 yr after launch, i.e. in January 2013.

From the Planck mission organization itself: “Planck Science Management Plan”

The nature of the mission is such that the data products can only be generated from an analysis of the full set of data; i.e. distribution of partial sets of data before the complete set is processed is not useful and will be avoided. In particular, all of the data acquired over the duration of the mission will be simultaneously used to calibrate and remove systematic effects. Thus, the time at which the 1-year proprietary period begins can be appropriately defined as the time at which the 9 all-sky channel maps (the main product described above) have reached a level of maturity such that they could be delivered to the community. The exact time when this level is reached will be determined by the PST, but will not be later than 1 year after the spacecraft power has been switched off.

It is worth noting here that the 1-year proprietary period will also be used to prepare the physical means of distribution of the scientific data products to the astronomical community, and the associated documentation (the ?Explanatory Supplement”); given the large amounts of data involved (in particular in the time series) this operation will require a non-negligible amount of time.

The Planck Mission Science Case, from Helsinki:

The observing program lasts 14 months. The Planck data will be made public two years after completion of the mission, in 2012. Prior to this, during the proprietary period, the data is analyzed and used by Planck Scientists and their collaborators

And now about the large European astronomical observatories (note how much emphasis there is on making the data easy to share):

Larger telescopes, more sophisticated instrumentation and solid state detectors have increased the efficiency of the astronomical observations by several orders of magnitude. All the data collected by these new instruments are produced directly in digital form and stored on computer accessible media. Controlled calibration procedures are applied to the raw data that then fill Science Archives that are readily accessible via the WWW for scientific analysis and research. For most of these Archives, the data they contain are becoming publicly available after a short period of proprietary time.

Most observatories are currently distributing Principal Investigator data to their user communities after the proprietary period has expired. This enables the scientific utilisation of these data for completely different purposes than the ones of the original proposer, and thus optimises the scientific return of observation infrastructure. But the distribution method is quite variable, from distribution on demand to full on-line archives with sophisticated data retrieval tools.

Observatories and organizations such as ESO, ESA, PPARC, CNRS and Jodrell Bank have invested telescope time and financial resources in producing archives with the potential to be reutilized for a variety of scientific purposes. The AVO offers the opportunity to federate these resources and provide astronomers with access to an integrated multi-wavelength repository of data for data mining and enabling new science.

Astronomical archives are potential a valuable teaching tool for school and university students. ESO and ESA have received funding for education purposes through the European Week for Scientific and Technological Culture Program. Through these efforts ESO has realized the magnitude of the potential utilization of archives by teachers and students. By enabling uniform and interactive access to distributed data archives, the AVO will provide an environment in which the public can participate in exploring the digital cosmos and in which teachers can build new educational resources.

A basic issue to be addressed during the initial phases of the AVO design is the uniformity in the quality of the data. Different Archives and, within them, data from different telescopes and instruments, have adopted different quality control procedures and indicators. The issue is particularly acute for the data collected by individual PI (Principal Investigators) that become public after the expiration of the proprietary period. The first step in addressing this issue is a review of the scientific quality of the existing Archives and the drafting of the scientific requirements for the standard quality control procedures to be adopted by Archives that want to be part of the AVO. The study will proceed by defining a set of quality indicators that are suitable for use by automatic querying and browsing systems. In some cases, the production of higher level Archive products (metadata or specific survey products) will be suggested and implemented.

Blast from the past, when oceans could be worked upon without having to worry about politics: the World Ocean Circulation Experiment (WOCE) 1990 – 2002 (including an appropriate data quality control process)

The WOCE Hydrographic Programme Office in Woods Hole was the first WOCE Data Assembly Centre (DAC) to be established (in 1990). Three years on we can look at how well we are doing. WHPO ‘data’ is digital data of either station locations and times or CTD and bottle data. These data with their attendant documentation come from PIs and at the end of the WHPO involvement are transferred to the WHP Special Analysis Centre (SAC) in Hamburg. After the proprietary period is over, data are made available to the WOCE community on an ftp server. The start of the path is the submission to WHPO of data sets and documentation by individual Chief Scientists. The WHPO usually re-formats the data and, in some cases, generates information files that are not otherwise available. Data are then sent out to a Data Quality Evaluator (DQE) for independent assessment of the quality. In the case of some repeat hydrography cruises, we hold a data set until we receive another one from the same group since it is as easy for a DQE to evaluate two data sets in the same region as one. Once the DQE reviews are complete, they are sent to the Chief Scientist, who is asked to revise his data set, (usually only by changing data flags or submitting additional information). The revised data set is re-submitted and, after further checking at the WHPO, is sent on to SAC and made publicly available once the proprietary period of 2 years elapses.

In fact this is what the WOCE home page shows:

The WOCE data set is the most comprehensive data set ever collected from the global ocean.This site provides access to the final (3rd) Version of the data set collected during the World Ocean Circulation Experiment (1990-1997).

This page provides access to:

Browse the WOCE Data DVD Disk One

Browse the WOCE Data DVD Disk Two

Browse updates and amendments to the WOCE Data DVDs

WOCE 2002 Conference Presentations

WOCE Global Data Resource (PDF, 19.8MB)

How about the Hubble Space Telescope (where we are told what NASA does when it’s not climate in the picture)

In accordance with NASA policy, all science data from the Hubble Space Telescope is archived with a one-year proprietary period by default. This period may be extended or shortened at the request of the principle investigator (PI) and on approval by the STScI Director’s Office. Calibration data (i.e., data obtained under calibration proposals), by default, carries no proprietary period; neither does engineering data, calibration files (derived from calibration observations), and observatory monitoring data.

In addition to the regular proprietary period, observations by General Observers (GOs) which are found to be duplicates of concurrent observations by a Guaranteed Time Observer (GTO) may be placed under restriction. Data under restriction (or “embargoed”) cannot be distributed to the GO until the restriction expires (usually, when the GTO data goes public).

How easy is it, to get a Proprietary Rights Modification? Not much:

Requests for proprietary rights modifications (extensions, reductions, and/or restrictions) should be sent to the archive hotseat (archive@stsci.edu). The hotseat staff will forward all requests to the appropriate authorizing officials. All requests must be approved by the Director’s Office before being enacted.

More at the Space Telescope Science Institute, details about the “Extention of Proprietary Data Period”:

While individual images of my target will have some value, the scientific goals of this program are possible only when the entire dataset has been collected. I am now requesting that the proprietary data period be set to run from the completion of the last visit in the set. The timing on this request is important since your answer will greatly impact the upcoming budget submission.

Requests for non-standard proprietary data periods are unusual and are granted only in exceptional circumstances. Such a request should be addressed to Duccio Macchetto as the Associate Director for Science Policy.

And finally…on 14 April 2010, Nature magazine hosts a protest against “Telescope team may be allowed to sit on exoplanet data“:

on Monday a NASA advisory panel recommended that Kepler be allowed to censor 400 “objects of interest” — presumably good planet candidates — until February 2011, giving the mission team more time to firm up discoveries, rule out false positives and publish. If enacted, the new policy would represent a selective editing of data on the basis of its science content, rather than its quality — unprecedented for such NASA missions.

Many astrophysics programmes allow researchers a proprietary period with the data. For instance, guest observers on the Hubble Space Telescope get exclusive use of their data for a year before public release. But the tradition for NASA Discovery missions — small, principal-investigator-led missions like Kepler — is to make calibrated data available immediately. That policy has already been changed once for Kepler, last year, when the team was given more than a year to pursue confirmations and work out the kinks in its data processing.

Exoplanet astronomers outside the mission, however, are critical of what they say is an overly cautious approach. Scott Gaudi, an astronomer at Ohio State University in Columbus, says external astronomers might help the Kepler team, as it will be unable on its own to follow up and confirm all its candidate planets. “I think Kepler is being far too conservative, and far too closed about what’s going on,” he says, “and I think it’s to the detriment of science”.

[Malcolm] Fridlund, who is ESA’s study scientist for a planned follow-up planet-hunting mission called PLATO (Planetary Transits and Oscillations of Stars), wants to do things differently next time. The policy for the future mission will require the immediate dissemination of data. “You get a larger community and you get a bigger workforce for free,” he says. “It’s clear that the more people you get involved, the more support you get.”

 

UEA News: Help Rebrand The University Of East Anglian Silly Walks!

London and Norwich, 26 Nov 2010 (MNN) – In an effort to reposition itself after the Climategate debacle, the University of East Anglian Silly Walks (aka UEA) is launching today an international competition aimed at exposing its true nature in a new spirit of transparency.

However, due to budgetary concerns and in order to lower CO2 emissions, the acronym will have to remain the same. This means the additional challenge for all budding brand experts is to come up with a more meaningful long-form for “UEA”.

Among the proposals received so far:

• University of Eventful Archives
• University of Expanded Animosity
• University of Excised Adjustments
• University of Extreme Airs
• University of Enough Amusement
• University of Effectively-measured-data–changing Attempts
• University of Expert Attacks
• University of Erased Authority
• University of Expanded Alterations
• University of Easy Annoyance

ps Sadly, all FOI requests to UEA for details about the above will never be fulfilled, as there was a North Korean student once at the University and any openness would be contrary to Juche.

Climategate: Mr Bean At The UEA

If I had to bet money on Climategate, most of it would go to back up Fred Pearce’s interpretation, as described in Damian Carrington’s blog about the Jul 14 Guardian debate:

Pearce was passionate in arguing that ‘Climategate’ was a very human tragedy, in respect of scientists feeling under siege and becoming fiercely defensive – which only spurred on the sceptics, who thought there must be something to hide. But he thought many CRU critics were not sceptics at all: “They are actually data libertarians, rather than climate sceptics, still less climate deniers. It turned into data wars.” Pearce’s conclusion was that at this turning point for climate science, more “candour” was needed from all.

Count me in as Data Libertarian!

“Scientists feeling under siege and becoming fiercely defensive – which only spurred on the sceptics, who thought there must be something to hide“? Just like Mr Bean at the airport then…

And yes, I would recommend medication for anybody still trying to smother FOI and/or in the business of hiding any data directly related to published scientific papers…

Report From Climategate Guardian Debate with Monbiot, McIntyre, Pearce, Watson, Keenan and some uea guy

As posted by Latimer Alder in my previous post:

Just back from the Climategate debate run by the Guardian tonight. We’re assured that the Guardian website will have a full video of the whole proceeding sometime tomorrow. So just some very sketchy impressions.

Steve obviously read the remarks from last night’s meeting and insisted on speaking from a lectern. This was a good move as it gave him more ‘authority’. And he was (mostly) crisper…making his points more directly. The others spoke while seated.

George Monbiot chaired the meeting and I think he did a fair job of it. He tried hard to be unbiased, and only once or twice strayed into partisan territory. And he managed to keep the speeches and questions mostly to time and to the point

Fred Pearce took a longer perspective than the others. He spoke well and described Climategate as a tragedy rather than a conspiracy…the tragedy being that the CRU guys had adopted siege mentality. Climategate has certainly widened his perspective.

Trevor Davies representing UEA/CRU was appallingly bad. He mouthed platitudes by the shedload, but was unfamiliar with the details of any of the subjects likely to be raised. And was several times embarrassed by doing so. Apart from the fact that he had a sharp suit. I can find nothing positive to say about him. Struck me as a devious smooth cove.

Bob Watson opening remark was that he hadn’t read the e-mails in question. This was a bad mistake – many in the audience were very familiar with them, and not happy to be lectured by somebody who wasn’t. IPCC was imperfect but the best that could be devised 95% of scientists agree…it is now just a risk management exercise. Errors corrected quickly…As good as having Ravendra, but no need for the extra slot at Heathrow for him to land his jet. Very much the Scientific Establishment figure.

Keenan was interested in research fraud and the lack of accountability in science as a whole. He accused Jones of committing fraud, even after being given a chance to withdraw the remark. Davies tried to defend Jones but had no details. Keenan showed a more street-savvy business approach than any of the other participants. I’d like to have heard him at greater length.

Overall conclusion: there was no conclusion. Everybody agreed that openness and transparency were good, that debate should be with all parties and that uncertainties should be made more clear.

But my own view is that the proof of the pudding is in the eating. This one still has legs and will run and run.

CRU Who? Or…Can The IPCC Survive The Oxburgh Review?

Forget Phil Jones and the CRU…for all intents and purposes, Lord Oxburgh’s “International Panel” has hit the IPCC itself with quite a broadside.

This is what the friendly Panel has just deemed necessary to write about the IPCC (my emphasis):

Recent public discussion of climate change and summaries and popularizations of the work of CRU and others often contain oversimplifications that omit serious discussion of uncertainties emphasized by the original authors. For example, CRU publications repeatedly emphasize the discrepancy between instrumental and tree-based proxy reconstructions of temperature during the late 20th century, but presentations of this work by the IPCC and others have sometimes neglected to highlight this issue. While we find this regrettable, we could find no such fault with the peer-reviewed papers we examined

And here’s what the IPCC says about the IPCC (my emphasis again):

The Intergovernmental Panel on Climate Change is the leading body for the assessment of climate change, established by the United Nations Environment Programme (UNEP) and the World Meteorological Organization (WMO) to provide the world with a clear scientific view on the current state of climate change and its potential environmental and socio-economic consequences.

Clearly, it has now been established beyond all doubt that the IPCC has been a failure regarding the provision of “a clear scientific view” on the “peer-reviewed papers” by CRU researchers. Those papers said one thing, the IPCC another.

With Climate Change too serious an issue to be left in oversimplifying hands,  the Fifth Assessment Report is unlikely to be any good unless substantial organizational changes are implemented in the IPCC.

ClimateGate: Had It Been For AGW Believers, Enron Would Still Be In Business

Professor Trevor Davies, Pro-Vice Chancellor for Research and Knowledge Transfer of the UEA, quoted yesterday by Willis Eschenbach in a comment to his “Freedom of information, my okole…“:

The University [of East Anglia, home of the CRU] takes its responsibilities under the Freedom of Information Act 2000, Environmental Information Regulations 2004, and the Data Protection Act 1998 very seriously and has, in all cases, handled and responded to requests in accordance with its obligations under each particular piece of legislation.

Kenneth Lay answering an analyst’s question on August 14, 2001, as quoted in Wikipedia:

There are no accounting issues, no trading issues, no reserve issues, no previously unknown problem issues. I think I can honestly say that the company is probably in the strongest and best shape that it has probably ever been in.