AHA Recommends Safegaurds for AI-Enabled Medical Tools

The American Hospital Association (AHA) is calling on the Food and Drug Administration (FDA) to develop evaluation standards and measurements to assess the impacts of medical tools that use artificial intelligence.

On Monday, AHA Senior Vice President for Public Policy Analysis and Development Ashley Thompson responded to a request from the FDA for an evaluation of AI-enabled medical devices.

“AI-enabled medical devices offer tremendous promise for improved patient outcomes and quality of life,” the AHA wrote. “At the same time, they also pose novel challenges — including model bias, hallucinations and model drift — that are not yet fully accounted for in existing medical device frameworks.”

The AHA said it supports AI policy frameworks that “balance flexibility to drive market-based innovations with appropriate safeguards to protect privacy and patient safety.”

Why It Matters

The AHA noted in its recommendation that there has been significant growth in the application and use cases for AI in health care over the last three years. Hospitals and health systems have also significantly expanded their use of AI products and are continuing to identify new AI use cases to improve operational efficiency and augment care delivery.

AI-enabled devices are becoming increasingly prevalent in diagnostic imaging and radiology, AHA said, acknowledging how algorithms and machine learning techniques can enable AI to quickly identify patterns and anomalies in medical images that the human eye might miss.

However, AI integration into hospitals has led to some pushback from nurses and other staff who say health systems are rushing to deploy new technologies without including staff in the decision-making process. Some New York City-based nurses recently held a rally to highlight their concerns with AI integration, noting the additional time it takes to review AI tools for mistakes, the depersonalized nature of augmented care and how AI tools may exacerbate issues of short staffing in hospitals.

This comes as the FDA announced yesterday that it will deploy agentic AI capabilities for all agency employees to create more complex AI workflows to assist with multi-step tasks. This tool is optional for staff and is to be used voluntarily.

“We are diligently expanding our use of AI to put the best possible tools in the hands of our reviewers, scientists and investigators,” FDA Commissioner Marty Makary said in a statement. “There has never been a better moment in agency history to modernize with tools that can radically improve our ability to accelerate more cures and meaningful treatments.”

What To Know

The AHA said it encourages the FDA to develop standards for health systems and AI vendors and developers to follow that align with existing FDA frameworks to ensure AI tools are used safely and effectively.

AI-enabled devices are already held to the same FDA guidelines as other medical devices, which are dictated by the level of risk, intended use and indications. But the AHA said there are gaps for post-deployment evaluation.

Because the potential for bias, hallucination and model drift, the AHA said there is a need for a more nuanced approach to measurement and evaluation after deployment that addresses the unique factors that impact the integrity of AI devices.

“We also encourage the FDA to pursue a risk-based approach to monitoring and evaluation activities, whereby factors for potential risk to quality and patient safety are accounted for in the measures and scope of monitoring,” the AHA said, adding that the FDA should seek feedback from device makers, hospitals and other providers, as well as standards development groups.”

The AHA notes that evaluation and monitoring “should not be overly burdensome and resource-intensive.”

The AHA also recommends that the measurements and evaluation should be aligned with existing FDA frameworks, including the total product lifecycle approach established for medical devices.

This may also lead to a streamlining of the clearance process, AHA said. It encourages the FDA to explore clearance pathways for vendors and developers to provide detailed post-market evaluation and monitoring plans, including additional special controls, to streamline 510k clearances under fewer applications. These standards would not apply to clinical decision support tools or administrative AI tools.

“We believe such an approach could shorten the time it takes to get evidence-based, safe uses of AI approved for use, benefiting patients and providers,” the AHA said. “It could also help reduce regulatory burden and cost.”

Finally, the AHA said vendors and developers should be responsible for the ongoing integrity of the tools they sell. While hospitals and health systems do assess the strengths and limitations of AI tools, the AHA said the “black box” nature of these systems makes it difficult to identify flaws in the models that may impact the accuracy or validity of the AI’s analysis.

Post-development standards, therefore, should be established for vendors. These standards may include metrics for performance, thresholds for further evaluation and communication requirements to end users related to ongoing performance.

To address the disparity in resources and staffing among hospitals due to funding, infrastructure or geography, the AHA encourages the FDA to collaborate with other federal health agencies to develop training, technical assistance and potential grant funding to support educational efforts and governance for AI health tools.

What People Are Saying

FDA Chief AI Officer Jeremy Walsh said in a statement: “FDA's talented reviewers have been creative and proactive in deploying AI capabilities — agentic AI will give them a powerful tool to streamline their work and help them ensure the safety and efficacy of regulated products.”

Newsweek has reached out to both the FDA and the AHA for additional comment.

Have an announcement or news to share? Contact the Newsweek Health Care team at [email protected].

Request Reprint & LicensingSubmit CorrectionView Editorial & AI Guidelines