When 97% is not enough

This study is a teachable moment, a future textbook example of scientific scams…
The people who conducted the Cook study don’t understand rudimentary epistemology, or what counts as evidence of anthropogenic climate change….
They’re willing to do absolutely stunning, unbelievable things to score political points.

-Jose Duarte

Cook did achieve something. Anyone who, for whatever reason, wants to argue that climate researchers are incompetent, secretive and dishonest has found a prime example in John Cook.

Professor Richard Tol

We all know that the science on climate change is settled, right? Anyone with any doubts can simply refer to the 97% consensus “study” by John Cook and Nuccitelli of the SkepticalScience website.

That study has come under a lot of criticism- inappropriate sampling techniques, failure to release data, and other methodological issues.

Professor Richard Tol, whose own papers were rated as part of the study, writes:

Cook and co selected some 12,000 papers from the scientific literature to test whether these papers support the hypothesis that humans played a substantial role in the observed warming of the Earth. 12,000 is a strange number. The climate literature is much larger. The number of papers on the detection and attribution of climate change is much, much smaller.

Cook’s sample is not representative. Any conclusion they draw is not about “the literature” but rather about the papers they happened to find.

Most of the papers they studied are not about climate change and its causes, but many were taken as evidence nonetheless. Papers on carbon taxes naturally assume that carbon dioxide emissions cause global warming – but assumptions are not conclusions….

The data is also ridden with error. By Cook’s own calculations, 7% of the ratings are wrong. Spot checks suggest a much larger number of errors, up to one-third.

Cook tried to validate the results by having authors rate their own papers. In almost two out of three cases, the author disagreed with Cook’s team about the message of the paper in question.

These are serious basic flaws that anyone with a rudimentary understanding of methodology- especially for the social sciences- should readily be able to comprehend. Yet not only did this study pass peer review, it has been cited thousands of times and is the go-to reference to “prove” the consensus on climate change. More than anything else, it provides the rational behind the constant refrain we hear of “deniers” leveled at anyone daring to even so much as ask questions about “the consensus.”

Cook tells us

Once our paper (Cook et al. 2013) was published, our results generated a great deal of interest. Distributing press releases from the universities of myself and my coauthors led to media coverage in over 28 countries, including a number of non-English speaking countries (thank goodness for Google Translate). Interest peaked when President Obama’s Twitter account tweeted about our research to 31 million followers (Obama 2013b). Several weeks later, President Obama mentioned the 97% consensus in a landmark speech on climate change (Obama 2013a).

Our paper was ranked as the 11th most talked about scholarly paper in 2013 (Altmetric 2014). Sadly, we were pipped out of the top ten by a paper about Sudoku. Our research was also listed in the top 5% of all scholarly papers…

The influence of this paper has been huge, and in online debates on climate change I can pretty much guarantee it being leveled at me every time, often along with the redundant comment, “I can’t believe we are even discussing climate change! The science is settled!”

Sure, the science is settled- but only in so far as we know that CO2 is a human gas, and as we burn more fossil fuels and put more CO2 into the atmosphere, we can expect the earth to warm. But continue much beyond that, and all bets are off.
There is no consensus on how much CO2 will cause how much warming;
no consensus on how many other natural and man-made effects will either add to or reduce the warming;
no consensus on specific impacts;
and especially, there is no consensus on policy– what, if anything, we can or should do about it, which is not, of course, a question for science alone.

Recently another critique appeared of the Cook consensus paper- not so much a rebuttal or refutation as an evisceration.
Jose Duarte is a social psychology doctoral candidate at Arizona State University specializing in methodological validity in social science. He seems to have been surprised at what he found when he came to examine the Cook paper. His findings have lead him to call for the paper’s withdrawal. Duarte’s powerful and uncompromising use of language makes his views worth quoting at length:

This study is a teachable moment, a future textbook example of scientific scams.

This paper is vacated, as a scientific product, given that it included psychology papers, and also given that it twice lied about its method (claiming not to count social science papers, and claiming to use independent raters), and the professed cheating by the raters…

The people who conducted the Cook study don’t understand rudimentary epistemology, or what counts as evidence of anthropogenic climate change….

Those of you who have shaped yourself into pretzels defending this study should be ashamed….

Duarte particularly points to the fact that the raters themselves- all “volunteers from the SkepticalScience team” in Cook’s own words- were themselves political activists, rating papers in their own area of activism- something he can scarcely believe could actually happen at all, never mind pass peer-review;
and that many of the papers examined were not climate science papers at all, but included one paper on cooking stove design in Bangladesh, as well as psychology papers, marketing papers and general endorsements of climate change from the general public.

“Nuccitelli” [one of the co-authors] Duarte tells us, “thinks that if a psychology paper uses the phrase “climate change denial”, it could count as scientific endorsement of anthropogenic climate change. We should linger on that. This is a staggering level of stupidity with respect to what would count as scientific evidence of AGW. ” (emphases added.)

he continues:

I get the impression that Cook and company don’t think they’re militant political activists, as though being a staunch leftist is the default rational position, not partisan (even though they talk about politics more than science on some of their pages, savage skeptical scientists, Republicans, and oil companies, ignore scientific evidence and papers that conflict with their views, ignore a large swath of economics, and are just off the charts in their “denier! denier!” hostility) — if you seriously think that the only “partisans” are people who disagree with you, then you’ve not yet achieved mature adulthood….

I think some of you who’ve defended this “study” got on the wrong train. I don’t think you meant to end up here. I think it was an accident. You thought you were getting on the Science Train. You thought these people — Cook, Nuccitelli, Lewandowsky — were the science crowd, and that the opposition was anti-science, “deniers” and so forth. I hope it’s clear at this point that this was not the Science Train. This is a different train. These people care much less about science than they do about politics. They’re willing to do absolutely stunning, unbelievable things to score political points. What they did still stuns me, that they did this on purpose, that it was published, that we live in a world where people can publish these sorts of obvious scams in normally scientific journals….

If our science category or camp includes people like Cook and Nuccitelli, it’s no longer a science category. We won’t have credibility as pro-science people if those people are the standard bearers. Those people are in a different category, a different camp, and it won’t be called “science”. Those climate scientists who have touted, endorsed, and defended the Cook et al. study – I suggest you reconsider. I also suggest that you run some basic correction for the known bias, and cognitive dissonance, humans have against changing their position, admitting they were wrong, etc. Do you really want to be on the historical record as a defender of this absurd malpractice? It’s not going to age well, and as a scientist, certain values and principles should matter more to you than politics.

There have been several previous studies on the scientific consensus on climate change before, but many of them suffer from much the same methodological flaws, and most ask much the same binary questions- is human-caused CO2 warming the planet or not?

Duarte comments:

The American Association for the Advancement of Science (2014) broke my heart, by releasing a wildly unscientific report that cherry-picked only the studies that gave it the inflated consensus figures it wanted — many of which are so bad as to be inadmissable. When scientists want to review a body of research, they conduct a meta-analysis that includes all the research that meets certain criteria of rigor and validity. The AAAS strangely chose not to perform a meta-analysis — they simply ignored most studies, and cherry-picked four studies that gave them the inflated, shock-value numbers they wanted. Among the four was an obsolete one-page study from 2004 that doesn’t clearly describe its methods (Oreskes, 2004, yes, really, one page long). That is, they skipped past all the more recent and credible studies from the intervening decade (e.g. Harris (2007), Bray and van Storch (2008), and others) to reach all the way back to a junk study from 2004. I’ve never seen such behavior – we clearly can’t do anything with mysterious one-pagers from 2004. This isn’t what I expected.

NASA come to the same conclusion, apparently based on the same three cherry-picked studies:

W. R. L. Anderegg, “Expert Credibility in Climate Change,” Proceedings of the National Academy of Sciences Vol. 107 No. 27, 12107-12109 (21 June 2010); DOI: 10.1073/pnas.1003187107.

P. T. Doran & M. K. Zimmerman, “Examining the Scientific Consensus on Climate Change,” Eos Transactions American Geophysical Union Vol. 90 Issue 3 (2009), 22; DOI: 10.1029/2009EO030002.

N. Oreskes, “Beyond the Ivory Tower: The Scientific Consensus on Climate Change,” Science Vol. 306 no. 5702, p. 1686 (3 December 2004); DOI: 10.1126/science.1103618.

The Doran paper was not the actual study, but merely cites a survey by an MSc student, Zimmerman. Barry Woods explains:

a survey of 10,256 with 3146 respondents was whittled down to 75 out of 77 “expert” ‘active climate researchers’ (ACR) to give the 97% figure, based on just two very simplistic (shallow) questions that even the majority of sceptics might agree with.

Lawrence Soloman examined the Andregg paper:

To encourage a high participation among these remaining disciplines, the two researchers decided on a quickie survey that would take less than two minutes to complete, and would be done online, saving the respondents the hassle of mailing a reply. Nevertheless, most didn’t consider the quickie survey worthy of response – just 3146, or 30.7%, answered the two questions on the survey:

1. When compared with pre-1800s levels, do you think that mean global temperatures have generally risen, fallen, or remained relatively constant?

2. Do you think human activity is a significant contributing factor in changing mean global temperatures?

The questions were actually non-questions. From my discussions with literally hundreds of skeptical scientists over the past few years, I know of none who claims that the planet hasn’t warmed since the 1700s, and almost none who think that humans haven’t contributed in some way to the recent warming – quite apart from carbon dioxide emissions, few would doubt that the creation of cities and the clearing of forests for agricultural lands have affected the climate. When pressed for a figure, global warming skeptics might say that human are responsible for 10% or 15% of the warming; some skeptics place the upper bound of man’s contribution at 35%. The skeptics only deny that humans played a dominant role in Earth’s warming. …

As for the second question, 82% of the earth scientists replied that that human activity had significantly contributed to the warming. Here the vagueness of the question comes into play. Since skeptics believe that human activity been a contributing factor, their answer would have turned on whether they consider a 10% or 15% or 35% increase to be a significant contributing factor. Some would, some wouldn’t.

In any case, the two researchers must have feared that an 82% figure would fall short of a convincing consensus – almost one in five wasn’t blaming humans for global warming – so they looked for subsets that would yield a higher percentage. They found it – almost – in those whose recent published peer-reviewed research fell primarily in the climate change field. But the percentage still fell short of the researchers’ ideal. So they made another cut, allowing only the research conducted by those earth scientists who identified themselves as climate scientists.

Once all these cuts were made, 75 out of 77 scientists of unknown qualifications were left endorsing the global warming orthodoxy. The two researchers were then satisfied with their findings. Are you?

This paper by Bodenstein in PNAS concluded:

The study by Anderegg et al. (1) employed suspect methodology that treated publication metrics as a surrogate for expertise. Credentialed scientists, having devoted much of their careers to a certain area, with multiple relevant peer-reviewed publications, should be deemed core experts, notwithstanding that others are more or less prolific in print or that their views stand in the minority. In the climate change (CC) controversy, a priori, one expects that the much larger and more “politically correct” side would excel in certain publication metrics. They continue to cite each other’s work in an upward spiral of self-affirmation. The authors’ treatment of these deficiencies in Materials and Methods was unconvincing in the skewed and politically charged environment of the CC hubbub and where one group is in the vast majority

Thomas Fuller comments:

the worst part of this is the violation of the rights of those they studied. Because Prall keeps lists of skeptical scientists on his weblog, obsessively trawling through online petitions and published lists of letters, and because those lists were used as part of the research, anyone now or in the future can have at their fingertips the names of those who now or in the past dared to disagree.

The Joe Romm’s of this world have already called for this list to be used to deny funding, tenure and grants to scientists. And it will be. It doesn’t matter that the nature of the letters and petitions they signed varied widely, from outright skepticism to really innocuous questioning of the state of the science.

The paper is tagged ‘Climate Deniers.’ Now, so are they.

The question is, why do these “consensus” studies? Why do such appallingly awful studies when it is quite possible to do informative and useful studies (see van Storch 2008) ?

Note that Solomon shows how the sample size in the Andregg survey was whittled down, and the questions simplified, until the magic number of 97% consensus was reached. Basically, you can get any number you want depending on what questions you ask. The reason the questions asked in each of  these studies are set at such a low bar should be obvious: the further away you get from the “Are humans causing Global Warming?” sort of question- which nearly all so-called “climate skeptics” would readily agree to anyway- the lower the figure would be. Add in “dangerous” it will drop; add in “impacts” it will drop; add in “Kyoto” and you will get a very different answer.
Somehow while 83% or 86% or even 91% are still very large majorities, they are not enough.
  These studies are made-to-order, to get to the magic 97% which means “done deal, no wiggle-room”. They have to be able to claim game, set, match- any nuance must be removed, there has to be a straight-forward unimpeachable black-and-white message for policy makers, so they can say to the public “We have to Act. {And we are not going to ask you how you think we Should Act.]  You can’t argue with The Science.”

That is why the methodology doesn’t matter. It is getting to the end result that matters. If Cook had had no other choice, he would have just invented the entire study from his desk. In light of the scrutiny that is now being drawn to his methods, this would probably have been a better option.

The strategy of fabricating a near-100% “consensus” was chosen for specific reasons- the thinking goes like this: if people know that the Science says climate change is Dangerous We Have to Act (which is actually NOT the questions asked in the surveys) then people won’t ask questions about HOW we act, ie what policies to follow- because the policy in question has already been decided: Kyoto. According to this line of thinking, there is a straight line from science to policy. Is AGW happening? yes- now we go straight to Kyoto without passing Go. The politics and science are merged seamlessly with the sledge-hammer of “97%”.

Cook himself says explains here:

Law and psychology professor Dan Kahan questioned whether communicating the scientific consensus was an optimal approach, given that it may provoke a negative reaction from those dismissive of climate change (Kahan 2013). In response, I outlined the evidence for both the efficacy and importance of consensus messaging (Cook 2013b). Consensus information increases both acceptance of human-caused global warming (Lewandowsky, Gignac, and Vaughan 2013) and support for climate policy (Bolsen, Leeper, and Shapiro 2013). But crucially, consensus messaging was shown in an Australian experiment to partially neutralize the biasing influence of ideology with conservatives showing a greater increase in belief in human-caused global warming than liberals (Lewandowsky, Gignac, and Vaughan 2013). A study with U.S. participants found that the increase in perceived consensus in response to consensus information was greatest among conservatives (Kotcher et al. 2014)…

Consensus information increases both acceptance of human-caused global warming…and support for climate policy…

This clearly shows that Cook is motivated to use the consensus message to drive policy- he doesn’t say which policy, so how can he say understanding the consensus would lead to greater acceptance of it? Acceptance of policy depends on what it is, and how much it will cost! This is why the hand-flapping “we have to Act!” is so disingenuous- the policy has already been decided, when Cook says “support for climate policy” he means Kyoto, as if there is only one approach and it follows directly from the 97% consensus.

An article in Scientific American earlier this summer reinforces the obvious with respect to Cook and his agenda:

There’s no doubt that Cook regards climate change as a moral issue.

“As a father, I realized that we are handing over a world to our children that is worse than the world we were given,” he said over the phone from Brisbane, Australia. “And as a Christian, I saw climate change as a social justice issue.”

Curry has a discussion about the history of the use of consensus in this way here:

The role of consensus in this decision making is described by Oreskes (2004): “If we feel that a policy question deserves to be informed by scientific knowledge, then we have no choice but to ask, what is the consensus of experts on this matter? If there is no consensus of experts—as was the case among earth scientists about moving continents before the late 1960s—then we have a case for more research. If there is a consensus of experts—as there is today over the reality of anthropogenic climate change —then we have a case for moving forward with relevant action.

Again, what does “relevant action” mean? Kyoto.

So the wider questions raised are whether policy follows directly from science; what role can or should science have in directing policy; and so on. Cook, Oreskes, and the rest of Climate Orthodoxy have already decided the answer to this, but in following this agenda they have seriously compromised the scientific process and the integrity of science itself.

It is no surprise if people are losing faith in public science if this kind of thing is going on, all the more so since apparently much of the rest of the scientific community- with the majority of the “skeptics” community acting as bullyish gate-keepers- has chosen to circle the wagons to support Cook and his methods -not the *scientific consensus* but the political consensus.

Duarte again:

Cook’s initial response was to say that they weren’t political. I was dumbfounded that he thought he could get away with that, or even more, that he might actually believe it. People who talk about Republicans more than science on some of their issue pages are definitely political. People who use the word denier, or denialist, or denialism, are definitely political, and they’ve made a very strong commitment to their worldview….

A big problem with the cheap thrill of “denier! denier!” is that you’ve made it much harder for your future self to cleanly process evidence that doesn’t fit that narrative. From the perspective of that future self, you didn’t just disagree with other people on this issue — you called them deniers. That’s a bigger thing to be wrong about, from your future self’s perspective — you’ve got more skin in the game. You wouldn’t just be wrong, you’d be someone who was wrong and who smeared your opponents, which will be harder to cop to. The dissonance will likely be greater, and if so, you’ll be more biased. This is ancient and stable social psychology findings. See my collaborator Jon Haidt’s book “Righteous Minds” for a complete take.

Whatever about the cooked-up Cook study, the idea of “communicating science” by forcing the idea of a “consensus” down people’s throats has also received criticism from many quarters. Cook’s response to Kahan is above, but Kahan’s real point is that “the implicit message is that the people who disagree with 97 percent of scientists must be very stupid.”

Tol points out

Consensus is irrelevant in science. There are plenty of examples in history where everyone agreed and everyone was wrong. Cook’s consensus is also irrelevant in policy. They try to show that climate change is real and human-made. It is does not follow whether and by how much greenhouse gas emissions should be reduced.

Political scientist David Victor of the University of California said in a recent talk:

First, we in the scientific community need to acknowledge that the science is softer than we like to portray. The science is not “in” on climate change because we are dealing with a complex system whose full properties are, with current methods, unknowable. The science is “in” on the first steps in the analysis—historical emissions, concentrations, and brute force radiative balance—but not for the steps that actually matter for policy….

Second, under pressure from denialists we in the scientific community have spent too much time talking about consensus. That approach leads us down a path that, at the end, is fundamentally unscientific and might even make us more vulnerable to attack, including attack from our own. The most interesting advances in climate science concern areas where there is no consensus but the consequences for humanity are grave, such as the possibility of extreme catastrophic impacts. We should talk less about consensus and more about the consequences of being wrong—about the lower probability (or low consensus) but high consequence outcomes.

Ultimately, banging on about a consensus in climate change is misleading and uninformative, concealing far more than it reveals.

Of course, I reject Victor’s reference to “denialists” in this sense, and it is ironic since he fails to draw the conclusion his own analysis should lead him to:

If scientists, “science communicators” and policy makers are wondering where the “denialists” come from, or why trust in public science may be waning, yet still rally round, cite or defend in any way “studies” like that of Cook, they need only look at themselves.

2 thoughts on “When 97% is not enough

  1. The “consensus” does seem to be crumbling: Tol, Curry, Pielke Sr, Lindzen (of course). As Lomborg and Ridley have pointed out there are more important things to worry about. And Ridley continues to point out that the *best* course is to have the world (in general) and the developing nations (in particular) get wealthier to achieve the IPCC’s goal, not Kyoto.

  2. Jose Duarte’s post stunningly damns the Cook, et. al. paper.

    ‘Let’s quote the methods section once more:’, Duarte writes. ‘”Abstracts were randomly distributed via a web-based system to raters with only the title and abstract visible. All other information such as author names and affiliations, journal and publishing date were hidden. Each abstract was categorized by two independent, anonymized raters.”‘

    Yet Cook writes in an online discussion forum, “FYI, here are all papers in our database by the author Wayne Evans:”


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s