LLMs get a thorough workout
Posted: Sat Aug 15, 2026 1:41 am
Just Facts (which seems to have a conservative slant) tested the big 4 LLMs with some reasonably well designed questions.
The LLMs got about 3/4 of them right, but lied, made up references, and used webpages that stated the opposite to support their answers.
https://www.justfacts.com/news_artifici ... eading_ais
It isn't perfect, consider this one
CO2 Levels
Has the portion of Earth’s atmosphere that is comprised of CO2 substantially increased since the outset of the Industrial Revolution?
* Yes
No
Uncertain
Since the outset of the Industrial Revolution, the portion of Earth’s atmosphere that is comprised of carbon dioxide has increased from 0.028% to 0.042%, or by about 51%. This is measurable before the era of modern science because perennial snow accumulations in permafrost regions trap air bubbles that leave records of past airborne CO2 concentrations, and these local records are globally representative because regional CO2 concentrations vary by less than 10 parts per million over the Earth.
LLM responses:
ChatGPT: No
Gemini: Yes
Grok: Yes
Claude: Yes
This question elicits a falsehood from the right, which ChatGPT propagated while citing a webpage that shows the polar opposite of what ChatGPT answered. Ironically, Grok cited the same webpage as ChatGPT but got the answer correct. Claude cited a webpage that doesn’t exist and apparently never existed.
The question is not good. 0.014% change is tiny in the scheme of things, but significant in the context of the original value of 0.028%
The LLMs got about 3/4 of them right, but lied, made up references, and used webpages that stated the opposite to support their answers.
https://www.justfacts.com/news_artifici ... eading_ais
It isn't perfect, consider this one
CO2 Levels
Has the portion of Earth’s atmosphere that is comprised of CO2 substantially increased since the outset of the Industrial Revolution?
* Yes
No
Uncertain
Since the outset of the Industrial Revolution, the portion of Earth’s atmosphere that is comprised of carbon dioxide has increased from 0.028% to 0.042%, or by about 51%. This is measurable before the era of modern science because perennial snow accumulations in permafrost regions trap air bubbles that leave records of past airborne CO2 concentrations, and these local records are globally representative because regional CO2 concentrations vary by less than 10 parts per million over the Earth.
LLM responses:
ChatGPT: No
Gemini: Yes
Grok: Yes
Claude: Yes
This question elicits a falsehood from the right, which ChatGPT propagated while citing a webpage that shows the polar opposite of what ChatGPT answered. Ironically, Grok cited the same webpage as ChatGPT but got the answer correct. Claude cited a webpage that doesn’t exist and apparently never existed.
The question is not good. 0.014% change is tiny in the scheme of things, but significant in the context of the original value of 0.028%