AI stumbles on data analysis but eventually gets it right. Ready for another AI truth..
AI stumbles on data analysis but eventually gets it right.
Ready for another AI truth engine experiment? Last time, we tackled AI logic, and the Pi (the LLM from Inflection) demonstrated how AI could apply logic in instances where even trusted experts fall short.
In Skeptiko 634, I tasked AI with some mildly complex data analysis of a published scientific paper. AI stumbled mightily but eventually show the potential we’re all hoping for.
The Experiment
My methodology was straightforward:
* I selected a published scientific paper: the Bangladesh mask study from the journal Science.
* I copied and pasted the results section into various AI language models (LLMs).
* I asked the AIs to interpret the results and, through inductive reasoning, generate numbers not immediately available in the abstract but calculable with a scientific background.
The Results
The performance of the AI models varied significantly:
* Gemini: Not surprisingly, was the worst performer. Google even accused me of being dishonest. A claim it later withdrew after its “mistake” was revealed.
* Perplexity: The star of the show, generating correct answers with minimal prompting.
* Other LLMs: Most struggled initially but showed potential for improvement.
Interestingly, while the numerical analysis proved challenging for many models, all AIs demonstrated an ability to identify the “Junk Science” issues in the study’s methodology. They correctly pointed out that the study’s conclusions rested on a difference of just 16 individuals in a population of 340,000 – a detail that raises questions about the strength of the findings.
What This Means
This experiment highlights several key points:
* AI’s potential in scientific data analysis is promising but not yet fully realized.
* Different AI models have vastly different capabilities, even when tackling the same task.
* AI can be a valuable tool in identifying potential weaknesses in scientific methodologies.
As we continue to develop and refine AI technologies, experiments like this one offer valuable insights into their current capabilities. It’s clear that AI has the potential to revolutionize how we analyze and interpret scientific data.
Transcript: https://docs.google.com/document/d/e/2PACX-1vS7KghCmyv6REaxKvZHakSeax56RSoX2bhppRuLtl8V8NqQ6PT9K6YA2qlhYTRZ1LDj7k596cyKcJkQ/pub
Youtube: https://youtu.be/fO9Idq5gLLY
Rumble: https://rumble.com/v59crxe-ai-tackles-yalestanford-junk-science-634.html
[box][/box]