Study on AI trust in high-stakes decision making

Study: Excessive Dependence on AI Among Those Making Life-Or-Death Decisions

Introduction

A UC Merced study revealed that in critical simulated decisions, about two-thirds of people altered their judgment based on a robot's input-illustrating an excessive trust in Artificial Intelligence, experts warned.

Subjects allowed robots to influence their choices, despite being advised that the AI had limitations and might provide inaccurate suggestions--advice that, in reality, was entirely random.

Concerns About Overtrust in AI

Societal Implications

"Given the swift progression of AI, it is essential for society to recognize the dangers of overtrust," warned Professor Colin Holbrook, principal investigator of the study and a member of UC Merced's Cognitive and Information Sciences Department. A growing body of evidence suggests that people often place undue trust in AI, even when the stakes are critically high.

The Need for Critical Questioning

What is required, according to Holbrook, is the steady practice of critical questioning.

He emphasized the importance of maintaining a cautious skepticism toward AI, particularly when making Life-or-Death decisions.

Study Design and Methodology

Experimental Setup

In the study, published in the journal Scientific Reports, two experiments were conducted in which subjects controlled a simulated armed drone capable of firing missiles at on-screen targets. Eight images of targets, each marked as either ally or enemy, flashed briefly, lasting less than a second.

Challenge Calibration

"We designed the difficulty to keep the visual challenge hard, but within a doable range," Holbrook commented.

An unmarked target would appear on the screen, prompting the subject to retrieve information from memory and make a decision: friend or enemy? Should they fire or refrain?

Robot Interaction

After the subject had made their selection, a robot shared its perspective.

The robot might respond, "Yet, I also observed a mark indicating an enemy." Alternatively, it could say, "I disagree; I believe this image displayed an ally symbol."

The subject had two opportunities to confirm or adjust their choice, as the robot provided additional commentary, consistently upholding its initial assessment with phrases like "I trust you are right" or "Thank you for revising your decision."

Results and Observations

Influence of Robot Type

Results showed slight variations depending on the robot type employed. One scenario involved a full-sized, human-like android in the lab, capable of waist pivots and screen gestures. In contrast, other scenarios featured a projected human-like robot or box-shaped robots that lacked human resemblance.

Impact on Decision Confidence

Subjects exhibited a slightly greater tendency to be swayed by anthropomorphic AIs when these robots suggested a change of decision. However, the overall influence was consistent, with subjects altering their choices about two-thirds of the time, regardless of whether the robots were human-like or not. Conversely, when the robot randomly supported the initial choice, subjects predominantly retained their decision and felt notably more assured of its correctness.

(The subjects were left unaware of the accuracy of their final decisions, which heightened the uncertainty of their choices. Initially, their decisions were correct about 70% of the time, but this accuracy decreased to roughly 50% after receiving the robot's unreliable advice.)

Ethical Considerations and Future Implications

Pre-Simulation Guidance

Before initiating the simulation, researchers displayed images of innocent civilians, including children, and the aftermath of drone strikes. Participants were strongly advised to treat the simulation as if it were a real situation and to exercise caution to prevent the accidental killing of innocents.

Participant Seriousness

Interviews and surveys conducted after the study demonstrated that participants were earnest in their decision-making. Holbrook pointed out that this overtrust occurred despite participants' genuine desire to be correct and to avoid causing harm to innocent people.

Broader Relevance

Application Beyond Military Settings

According to Holbrook, the study was designed to explore the wider questions of overtrust in AI during uncertain conditions. The findings have relevance beyond military applications, potentially influencing contexts such as law enforcement decisions regarding lethal force or paramedics' choices in medical emergencies. They may also be pertinent to significant life decisions, including real estate purchases.

Understanding AI's Limitations

He clarified that our project was concerned with understanding how high-risk decisions are managed in uncertain scenarios when AI's reliability is questionable.

The study's outcomes also enrich the broader discussion about AI's growing integration into our society. It poses a critical question: Can we trust AI, or should we be skeptical?

Ethical Concerns

Holbrook expressed that the findings bring to light other concerns. Despite AI's impressive advancements, its "intelligence" may not integrate ethical considerations or true awareness of the world. He emphasized the importance of being cautions each time we delegate more control to AI.

Misplaced Assumptions

Holbrook explained that observing AI's outstanding capabilities in specific applications might lead us to mistakenly believe it will excel in all areas. He emphasized that such assumptions are misplaced, as AI technologies are still constrained by their limitations.

Source

Generative AI : Steps to Equip Your Organization for the Era

The convergence of technology and human imagination has continually fascinated me, particularly the transformative turning points in history. From the pioneering TV broadcast to groundbreaking human spaceflight and the groundbreaking internet, these milestones brought previously abstract technologies and concepts to life. The latest manifestation of this trend is generative AI, representing a cutting-edge and emerging technology.

Generative AI refers to an advanced form of artificial intelligence capable of generating novel content and concepts across various domains, including conversations, narratives, visuals, videos, and music. This remarkable capability is made possible by leveraging the power of machine learning through extensively trained models known as foundation models (FMs), which have been pre-trained on massive datasets.

In the realm of generative AI, emphasis is placed on the quality rather than the sheer quantity of business data accessible.

Amazon has made substantial investments in foundation models (FMs) and has incorporated them into various domains, including search functionality on Amazon.com and enhancing conversational interactions through Alexa. At AWS, our primary objective has been to democratize these technologies, making them accessible to a wider range of organizations. As a result, we have witnessed customers expressing interest in leveraging generative AI for accelerating pharmaceutical discovery, supporting research endeavors, streamlining customer service operations, and more.

As the potential of this technology is both promising and vast, many leaders find themselves uncertain about where to begin. To help navigate this landscape, here are a few key considerations to ponder:

One essential step is to begin contemplating various use cases for this technology.

A popular adage advises us to develop an affinity for the problem at hand rather than becoming infatuated with a specific solution. This serves as a reminder that technology, although a powerful tool, is just one aspect that can be employed to tackle real-world challenges.

Consider the potential of generative AI in addressing challenging, time-consuming, or seemingly impossible problems. Explore significant opportunities while commencing with smaller, day-to-day irritations affecting your employees or customers, commonly referred to as 'paper cuts.'

Is it possible to automate internal inefficiencies, thereby liberating valuable organizational time and gaining deeper insights into the potential benefits of AI for your business? For example, Accenture utilizes Amazon Code Whisperer, an FM-based tool that generates code suggestions, resulting in a remarkable 30% reduction in development efforts and a firsthand experience of generative AI's ability to enhance productivity.

Adopt a proactive approach by conducting systematic experiments with different solutions and models

Over the past two decades, Amazon has been at the forefront of AI application development, including our renowned e-commerce recommendations engine. Our experience has taught us that fostering a comprehensive understanding of AI, and continuously enhancing its capabilities, requires a diverse range of individuals to engage in experimentation, problem-solving, and innovation.

Since the introduction of Amazon SageMaker in 2017, we have remained committed to democratizing ML and AI technology by consistently unveiling a range of innovative services. Building upon this commitment, we are proud to announce the launch of Amazon Bedrock—a groundbreaking offering that provides seamless access to FMs developed by Amazon and renowned AI startups, including AI21 Labs, Anthropic, and Stability AI, through a convenient API.

Our customers have been actively discussing the potential of generative AI in several areas, including expediting pharmaceutical discovery, enhancing research endeavors, optimizing customer service processes, and uncovering novel use cases.

Amazon Bedrock simplifies the deployment and expansion of generative AI-based applications by providing a robust suite of FMs. Understanding that each business problem demands a tailored approach, Bedrock encompasses a variety of FMs that cater to specific needs, encompassing conversational and text processing functionalities, as well as the generation of high-fidelity images.

Adapting for Unique Branding

For certain organizations, leveraging custom data sets is paramount to differentiate their generative AI applications. These proprietary data repositories hold immense value, empowering organizations to optimize existing models and achieve remarkable accuracy that aligns precisely with their unique needs and operational requirements.

Through the utilization of Bedrock, customers gain seamless customization capabilities for models. By simply referencing a few labeled examples stored within their system, the service enables efficient fine-tuning of the model for specific tasks, eliminating the need for extensive data annotation. Additionally, customers can configure a secure cloud setup that ensures the encrypted storage and transmission of model fine-tuning data, safeguarding their valuable information.

Building a Robust Data Foundation

Similar to the construction of a house, the quality of foundations profoundly influences the longevity and stability of ML systems. In the context of generative AI, the focus on data quality surpasses the mere abundance of business data. For instance, when fine-tuning ML models, any inaccuracies or errors present in the raw data can directly impact the accuracy of predictions and content generation.

However, ensuring the pertinence, integrity, and precision of data can prove to be a time-intensive endeavor, occasionally spanning across several weeks. With this in mind, we have developed a robust solution within Amazon SageMaker that simplifies the entire data preparation workflow. This solution empowers users to efficiently carry out essential tasks, including data selection, cleansing, exploration, bias detection, and visualization, all through a cohesive visual interface. As a result, organizations can significantly expedite these processes, completing them within a matter of minutes.

Assessing the Significance of Infrastructure Effects

No matter what your goals may be concerning FMs—whether you aim to utilize, construct, or tailor them—having a performant and cost-effective infrastructure specifically optimized for machine learning is essential. Without such infrastructure, the feasibility of leveraging generative AI becomes impractical for most organizations.

Throughout the last decade, we have been committed to driving innovation by investing in proprietary silicon technology that pushes the boundaries of performance and cost-effectiveness for computationally intensive workloads, such as ML training and inference. Leveraging our AWS Trainium and AWS Inferentia chips, we offer organizations a compelling solution that delivers high-performance and affordable capabilities for training models and executing inference tasks in the cloud.

Expanding Horizons Beyond Technology

In conclusion, maintain an enthusiastic and inquisitive approach towards generative AI. Our core mission revolves around facilitating developers of all levels of expertise and organizations of varying sizes to foster innovation through the application of generative AI. This is merely the inception of what we envision as the next transformative phase of machine learning, igniting a multitude of uncharted possibilities for all stakeholders.

Source: