Back in March, I put up a post, Phil Jones and the ‘expert judgement’ of the IPCC. This raised questions about one of the most important tables in the Summary for Policymakers (SPM) of Working Group I (WGI) in the  IPCC’s Fourth Assessment Report (AR4). This table assigns levels of ‘likelihood’ to evidence of observed trends in extreme weather events, the possibility that there is a human contribution to these trends, and predictions that the trends will continue during the 21st century.

According to this table, the authors of the IPCC report have greater confidence in the predictions than either the observations they are derived from or the hypotheses that they are based on, which seems to turn logic on its head. So when I started ploughing through the InterAcademy Council’s Review of the Processes and Procedures of the IPCC I was interested to see that the same table turns up on page 31 in a very critical chapter headed IPCC’s Evaluation of Evidence and Treatment of Uncertainty.

table-3_4-ar4_wgi_spm_edited.jpg

So far as I can see, the IAC have not addressed the precise point that I was making, but it is quite clear that they are very concerned about the way in which the last assessment report represented confidence and uncertainty, and that they have chosen to highlight this table as an example of the impact that expert judgement of confidence in research findings has.

Before going any further, lets bear a couple of points in mind. The phenomena listed in the table droughts, storms, heatwaves, sea level rise, tropical storms (hurricanes and cyclones) and heavy precipitation (floods) are the stuff of which anthropogenic climate change nightmares are made. The media, politicians, and environmental activists have used evidence that these are probably increasing in frequency and severity, and are likely to continue to do so, as the main plank in their justification of action on climate change. The credibility of their assertions depends on one authority only: the IPCC reports.

Here is what the IAC report has to say in the introduction to the chapter on The IPCC’s Evaluation of Evidence and the Treatment of Uncertainty.

The evolving nature of climate science, the long timescales involved, and the difficulties of predicting human impacts on and responses to climate change mean that many of the results presented in IPCC reports have inherently uncertain components. To inform policy decisions properly, it is important for uncertainties to be characterized and communicated clearly and coherently.

This chapter … explores whether uncertainty is characterized appropriately, given the nature of IPCC assessments, and whether the scales used to characterize confidence in results are appropriate, given the nature of the conclusions. At the end of the chapter, the Committee summarizes its conclusions and recommendations for improving the presentation of evidence and treatment of uncertainty in IPCC assessment reports.

My emphasis
Page 27

The IAC, therefore, recognise the very significant role that uncertainty must to play in view of our incomplete understanding of the climate system, and also that improvements are required in the way that this is represented in IPCC reports. As they accept that improvements are necessary, that implies that there are problems with present practice.

Under the heading Uncertainty Guidance in the Fourth Assessment Report, the IAC have this to say:

IPCC authors are tasked to review and synthesize available literature rather than to conduct original research. This limits the authors’ abilities to formally characterize uncertainty in the assessment reports. As a result, IPCC authors must rely on their subjective assessments of the available literature to construct a best estimate and associated confidence intervals.
My emphasis
Page 27

It would be difficult to overestimate the influence that the expert judgement of authors has on the credibility of climate science as it is presented to policymakers by the IPCC. Most will not be able to make their own decisions on whether scientific findings are well supported by evidence. They must rely entirely on the levels of confidence assigned by the authors.

Later, the report makes clear just how ubiquitous these expert judgements are:

In addition to characterizing uncertainty using confidence intervals and probability distributions, Working Group I used a combination of the confidence and likelihood scales to characterize the certainty of their conclusions. Virtually every statement in the Summary for Policy Makers is characterized using the terms employed by one of these scales. Table 3.4 illustrates the use of the likelihood scale, including the likelihood of a trend in extreme weather events in the late 20th century, the likelihood of a human contribution to that trend, and the likelihood of future trends in the 21st century, based on the SRES scenarios.
My emphasis
Page 30

Among the conclusions and recommendations at the end of the chapter is this:

The IPCC uncertainty guidance urges authors to provide a traceable account of how authors determined what ratings to use to describe the level of scientific understanding (Table 3.1) and the likelihood that a particular outcome will occur (Table 3.3). However, it is unclear exactly whose judgments are reflected in the ratings that appear in the Fourth Assessment Report or how the judgments were determined. How, exactly, a consensus was reached regarding subjective probability distributions needs to be documented. The uncertainty guidance for the Third Assessment Report required authors to indicate the basis for assigning a probability to an outcome or event (Moss and Schneider, 2000), and this requirement is consistent with the guidance for the Fourth Assessment

Report. Recommendation: Chapter Lead Authors should provide a traceable account of how they arrived at their ratings for level of scientific understanding and likelihood that an outcome will occur.
Emphasis in the original
Page 37

Back in March, when I posted about the apparent illogicality of the confidence assessment in the extreme weather events table used in WGI SPM, it would have been very interesting to know whose expert judgement had been relied on and how they had reached their conclusions. It is seems likely that scientists who were implicated in the Climategate scandal were involved, as I said in that post, but there was no way of knowing.

It would seem that anyone reading the immensely influential Fourth Assesment Report must apply blind faith to what they are told about the reliability of the science that underpins its findings and recommendations. That faith must also extend to assuming that those whose expert judgement they are relying on are not only competent, but that they are free of any preconceptions and prejudices relating the science they are assessing and they are undertaking this quite objectively. Anyone who has read the Climategate email has no reason to be confident that these conditions have been met. Apparently there are no records of who made these judgements, or how they were arrived at, so there is no way that they can either be validated or challenged other than repeating the whole process. This obviously is not going to happen.

Finally:

The IPCC uncertainty guidance provides a good starting point for characterizing uncertainty in the assessment reports. However, the guidance was not consistently followed in the fourth assessment, leading to unnecessary errors. For example, authors reported high confidence in statements for which there is little evidence, such as the widely-quoted statement that agricultural yields in Africa might decline by up to 50 percent by 2020. Moreover, the guidance was often applied to statements that are so vague they cannot be falsified. In these cases the impression was often left, quite incorrectly, that a substantive finding was being presented.
My emphasis
Page 36

The report refers specifically to a failing in the WGII report, but may apply equally to the extreme weather table in the WGI SPM. This is tantalising, because throughout the various revelations of Climategate, and the subsequent furore about the various errors found in AR4, those who seek to defend the IPCC have consistently claimed that the mistakes made by the IPPC have been peripheral and irrelevant to the core findings on the physical basis for concern about climate change.

I wish that I could find someone who is better qualified than I am, and has more authority, who would be prepared to examine the claims made in the extreme weather table. The science that it summarises is at the core of so many claims about the perils of AGW, and has been so influential in the formulation of public policy over the last few years. It is very important that the levels of confidence that it assigns are well founded and correct, and that they can be shown to be so.

A golden opportunity to make a detailed examination of the very strange logic that this table seems to apply has been missed, or ignored, in the course of the IAC’s review.

10 Responses to “The IAC report and IPCC ‘expert judgement’ of confidence”

  1. TonyN

    Obviously great minds think alike as Judith Curry has just started a similar thread here

    http://judithcurry.com/2010/09/22/the-uncertainty-monster/#more-146

    Perhaps you can pose your question over there as she knows more than most about uncertainty having insisted on using virtually non existent SST’s in order to put together a paper on warming oceans.

    I think virtually all the subjects in your list have been rebutted at some time or other but failed to achieve as much lasting prominence as the original IPCC reports did.

    So perhaps we sceptics need to market our knowledge better rather than necessarily write yet more rebuttals which never get much traction in the places that matter. Even then if the brass monkeys have their paws over their ears they aren’t going to hear much anyway.
    Tonyb

  2. TonyB
    You say:

    we sceptics need to market our knowledge better, rather than necessarily write yet more rebuttals which never get much traction in the places that matter.

    Agreed, but what we know is precisely the fact that we don’t know very much, especially about the future. Trying to get that message across is not easy, as Donald Rumsfeld found to his cost when he spoke of “known unknowns”.

    if the brass monkeys have their paws over their ears they aren’t going to hear much anyway

    We probably won’t get a hearing until it gets so cold that the brass monkeys have to take their hands from their ears in order to protect more sensitive parts of their anatomy.

  3. Looking at the first cell in the table it looks like at least 4 different phenomena rolled into one cell:

    “Warmer and fewer cold days and nights over most land areas”

    Warmer cold days (?)
    Fewer cold days
    Warmer cold nights (?)
    Warmer nights following a cold day
    Fewer cold nights

    It’s so vague that it means nothing. Then we get a discussion of “experts” saying it is virtually certain for the future based on projections based on scenarios. This is like a bad episode of Star Trek – its divorced from reality.

    It got worse in one of the other IPCC iterations – they invented some numbers instead of “likely”, “very likely” etc. There was a table showing how to turn the adverbs into numbers. Numbers look more scientific but they were just based on the words – not the results of any calculations.

  4. And real science doesn’t involve this tosh anyway. Real science makes specific and testable predictions:

    * There will be a lunar eclipse on a date next week.
    * The current will be 20A when we switch it on.
    * The bridge will support a 40 tonne lorry today.

    Real science does not do vague scenarios about the end of the century.

  5. Jack Hughes,

    Yes like the Big Bang theory which explains the origin of the universe. Every child can duplicate that in their bedrooms now! Just take the singularity out of the box let it go and watch a universe be created before your very eyes!

    Or the theory of Evolution. Just take a couple of apes -one male one female- put them in the right environment, wait a couple of million years and anyone can observe the process of their change into humans.

    I just can’t understand why these climate scientists can’t do the same.

  6. Jack Hughes and PeterM

    Jack Hughes is right, of course.

    It’s really “voodoo science” (to use a term coined by Pachauri, when they caught IPCC lying about disappearing Himalayan glaciers).

    Peter, you ask (comparing “Evolution” and “Big Bang” theories to the “dangerous AGW” postulation of IPCC):

    I just can’t understand why these climate scientists can’t do the same.

    The main reason is because they (IPCC) do not want to; they just want to sell their preconceived notion that AGW has been the principal cause of 20th century warming and represents a serious potential threat for the future.

    “The Keepers of All Climate Knowledge” (as Roy Spencer calls the mainstream “in” group of climatologists favored by IPCC) are not searching for scientific truth, they are simply trying to find scientific proof for their preconceived “dangerous AGW” premise. (Even if it means bending the truth a bit, as the Himalayan glacier case as well as others have demonstrated.)

    Pretty straightforward to me.

    Max

  7. Jack Hughes

    You commented on the first cell of IPCC’s Table SPM.2 (cited by TonyN).

    As you point out, it gets even “curiouser” when you look at the other two cells. But let’s look at this in more detail.

    IPCC tells us that the frequency of “warm spells/heat waves” and “heavy precipitation events” is “likely (>66%) to have increased in the late 20th century”, with a “more likely than not (>50%) likelihood” of an (undefined) “human contribution”, “based on expert judgment rather than formal attribution studies”.

    Let’s do a bit of arithmetic on this. Past events over “late 20th century” have a “likelihood” of:

    66% * 50% * 50% (assumed human contribution) * 50% (assumed expert judgment hit rate) = >8.25% probability

    From this IPCC model scenarios figure out that the “likelihood of future trends based on projections for the 21st century” is “very likely (>90%)“!

    “Tosh” indeed (as you put it).

    Max

  8. Jack Hughes #3
    I think the absurd complexity of the first cell definition may be because proper scientists were unwilling to sign a statement asserting “warmer nights” or “fewer cold days”, given that figures are based on an average of continuous measurement, measurement at a fixed hour, and 24 hour maximum and minimum temperatures. Another example of your excellent formulation “figures based on words”. In fact, the whole IPCC process might be defined as “fear based on speculation based on figures based on words”.
    Read down the left hand column; three examples of global warming, followed by four weather events thought to be results of global warming; apples and oranges, in other words. Read across the top; three totally different types of observation / speculation, presented as if on some kind of continuum. Read the likelihood cells; there’s more certainty about the results of computer models a hundred years hence than there is about observations made yesterday.
    It’s nonsense parading as science, a marketing man’s presentation aid. Is there a scientist anywhere in the world willing to defend it?
    No wonder Peter M wants to change the subject and talk about evolution.

  9. “More likely than not”.

    What kind of science is that?

  10. tonyb #1:

    Thanks!

    I actually drafted this post before I went away, so it is indeed a strange coincidence. As soon as I get time I’ll ask Judith Curry for an opinion as I find this a particularly aggravating conundrum, and I’d like to see it settled one way or the other.

Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

(required)

(required)


4 − two =

© 2011 Harmless Sky Suffusion theme by Sayontan Sinha