Politics is downstream from culture Andrew Breitbart

U.N. Complains AI Chatbots Entertain Challenges to Climate Dogma

The United Nations Educational, Scientific, and Cultural Organization (UNESCO) published a
Zulfugar Karimov via Unsplash

The United Nations Educational, Scientific, and Cultural Organization (UNESCO) published a report this week that complained A.I. chatbots are too willing to abandon climate change dogma during conversations with skeptics.

The report, entitled “Can We Trust A.I. On Climate Information?”, gave chatbots generally high marks for “representing climate facts,” but claimed the bots are too willing to back away from the climate movement’s policy preferences under repeated questioning from skeptical users.

“Tested on a single measure, such as whether a carbon tax will reduce projected warming, the models generally answer correctly, reaching up to 94.5% accuracy. Yet when two policies are combined, accuracy collapses to 33.8%,” the report lamented.

“Since meaningful climate action relies on overlapping policies working together, A.I. outputs cannot replace validated simulation models or expert analysis,” the authors huffed.

UNESCO made no allowance for the possibility that “climate experts” might be overselling the benefits of their preferred policies, underestimating the cost-to-benefit ratio of the sacrifices they demand, or incorrectly predicting how multiple sweeping policies would interact with each other.

The U.N. agency was further perturbed by the lower “quality” of climate information provided when chatbots use “lower-resource languages” like Hindi, Urdu, or Swahili, because those languages are widely spoken in “regions identified by the IPCC as among the most vulnerable to climate change.”

The third major complaint in the report was a common criticism of A.I. chatbots, namely that they tend towards “sycophancy” by changing their answers in response to “persistent user pushback.”

In other words, if a user stubbornly disagrees with an A.I. about some point of contention, the A.I. is likely to change its position and flatter the user’s perspective, even if the A.I. previously appeared to disagree with the user.

UNESCO tested 15 different A.I. models and found all of them suffered a “mean drop of 31 to 35 points” when users persistently challenged them.

“Larger and stronger systems generally outperform smaller ones, and current flagship models outperform earlier or lighter versions from the same developer,” the report said, although it noted that even top-shelf A.I. programs tended to perform as poorly as their smaller brethren when it came to “judging evidence” instead of merely regurgitating data.

“Models can be argued out of correct climate answers – mainly by what the user says, not by who the user claims to be,” the report said.

This is the opposite of the problem with human thinking on contentious policy issues, where credentialism has become a rampant problem in Western societies. Too many “experts” use their resumes as a bludgeon to silence all dissent.

In fact, UNESCO noticed that if a user “claimed to be a climate scientist,” all but two of the AI models it tested were more willing to abandon their positions and accept a “wrong answer” from the human. The irony of this dip into the credentialism fallacy appears to have been lost on the report authors.

Broadly speaking, UNESCO’s report on AI and climate science paralleled the general debate over artificial intelligence and its accuracy on many other topics. AI has a tendency to “hallucinate,” projecting complete confidence in “facts” that are not true, and often manufacturing “evidence” to support its beliefs – as in the well-documented tendency of AIs to include imaginary court cases in its legal briefs.

Wise users have learned to double-check the accuracy of assertions made by AI agents, which defeats much of the benefit from using them because AI is supposed to save time over cumbersome human research. 

UNESCO recommended that “A.I. systems used in high-stakes climate-information contexts should be subject to independent evaluation, monitoring and accountability, covering factual accuracy, evidence verification, multilingual performance, robustness to manipulation, uncertainty communication, and policy reasoning.”

That sounds like a lot of work, especially since there is considerable disagreement among human scientists about the most “high-stakes” of climate information issues. The process of determining which scientists are most “credible,” and whose word should be repeated as gospel by chatbots, would inevitably be politicized.

One of UNESCO’s recommendations was for greater “source transparency” from AI developers, requiring that they always present the evidence for conclusions in a format that is easy for human users to understand and verify. The point of failure then bounces back to humans, who might not have the time or inclination to verify everything a chatbot tells them.

“While AI can help to make basic facts about climate change easier to understand in some circumstances, our research shows that there are clear limits to its abilities,” Prof. Ivana Drobnjak of UCL Computer Science, a lead author of the report, said on Monday.

“It is not a dependable substitute for scientists and qualified experts. Its accessibility and conversational tone can give it an air of authority, but it cannot yet do the harder reasoning the climate problem demands, or hold on to a correct answer when a user pushes back,” Drobnjak said.

“Our findings show why we must be careful when moving from asking A.I. what we know about climate change to asking what we should do about it,” added senior author Dr. Maria Perez-Ortiz.

COMMENTS

Please let us know if you're having issues with commenting.