Researchers at the University of California – Irvine have created a hybrid human-machine framework that they say is key to building smarter artificial intelligence (AI) systems. The study involved a new mathematical model that can improve performance by combining human and algorithmic predictions and confidence scores.
The study was published in Proceedings of the National Academy of Sciences.
Humans vs. Machine Algorithms
Mark Steyvers is UCI professor of cognitive sciences and co-author of the paper.
“Humans and machine algorithms have complementary strengths and weaknesses. Each uses different sources of information and strategies to make predictions and decisions,” Steyvers said. “We show through empirical demonstrations as well as theoretical analyses that humans can improve the predictions of AI even when human accuracy is somewhat below that of the AI — and vice versa. And this accuracy is higher than combining predictions from two individuals or two AI algorithms.”
The researchers tested the framework by conducting an image classification experiment where human participants and computer algorithms worked separately to correctly identify distorted pictures of animals and everyday items. These were then ranked by the human participants by their confidence in the accuracy of each image identification as low, medium, or high. On the other hand, the machine classifier generated a continuous score.
Carrying Out Tests
The results of the experiments demonstrated significant differences in confidence levels between humans and AI.
Padhraic Smyth is an UCI Chancellor Professor of computer science and co-author of the paper.
“In some cases, human participants were quite confident that a particular picture contained a chair, for example, while the AI algorithm was confused about the image,” Smyth said. “Similarly, for other images, the AI algorithm was able to confidently provide a label for the object shown, while human participants were unsure if the distorted picture contained any recognizable object.”
The researchers used their new framework to combine the predictions and confidence scores from both the humans and AI, and the hybrid model achieved better performance than either human or machine predictions alone.
“While past research has demonstrated the benefits of combining machine predictions or combining human predictions — the so-called ‘wisdom of the crowds’ — this work forges a new direction in demonstrating the potential of combining human and machine predictions, pointing to new and improved approaches to human-AI collaboration,” Smyth continued.
The new project responsible for developing this framework was organized by the Irvine Initiative in AI, Law, and Society, which is looking to provide deeper insight into how humans and machines collaborate to create more accurate AI systems.
The research also included co-authors Heliodoro Tejada and Gavin Kerrigan. Heliodoro is a UCI graduate student in cognitive sciences, and Kerrigan is a UCI Ph.D. student in computer science.
Credit: Source link
