From building consensus on the definition of responsible artificial intelligence to waxing on about semi-autonomous health AI use cases, industry and government appear to be heading toward shared values, even in these polarized times, as we head into 2025, says Coalition for Healthcare AI CEO Brian Anderson.
“We need our policymakers and regulatory officials understanding these kinds of frameworks that are being developed in the private sector about what responsible AI looks like in health, and then developing the regulatory frameworks around that,” Anderson explained to Healthcare IT News.
Congruence to date
There’s been a lot of thought by the technology industry and federal healthcare regulatory agencies into AI model cards, or ‘AI nutrition labels’ – a digestible form of communication used to identify key aspects of AI model development for users.
On Thursday, when the CHAI released its open-source version of its draft AI model card, we spoke with Anderson about the coalition’s recent experience and his insights on what the near future may hold in developing public-private framework for safely disseminating healthcare AI.
“It’s great to see alignment between where the private sector innovation community is going and where the public sector regulatory community is going,” he said.
While CHAI is looking for feedback on its open-source draft model card this month – which “flows from” the Office of the National Coordinator for Healthcare Technology’s Health Data, Technology, and Interoperability: Certification Program Updates, Algorithm Transparency and Information rule – with a plan to roll out an update sometime in the next six months, Anderson said that he is hopeful that various regulators are and will continue to move in the same direction.
Specifically, he cited a degree of alignment on AI requirements with medical device regulators.
Earlier this week, the U.S. Food and Drug Administration included an example of a voluntary AI model card in its draft total product life cycle recommendations – design, development, maintenance and documentation – for AI-enabled devices.
“One of the exciting things in looking at the FDA example model card, and then certainly looking at ONC’s HTI-1 rule – is that CHAI’s model card is very much in strong alignment to both,” Anderson said.
According to the FDA, a model card for AI-enabled medical devices can address communication challenges in presenting important information to healthcare users – patients, clinicians, regulators and researchers – and the public.
“Research has demonstrated that the use of a model card can increase user trust and understanding,” FDA said in its draft guidance. “They are a means to consistently summarize the key aspects of AI-enabled devices and can be used to concisely describe their characteristics, performance and limitations.”
As one healthcare AI regulatory example, it’s an indication of where FDA regulators are heading in their work to frame trust around the use of AI.
“However, private and public sector stakeholder groups need to work collaboratively together informing one another,” Anderson said.
When asked, he noted the incoming administration, “and every leader that I’ve spoken to in the Senate and in the House, is very much interested in understanding how they can partner in a public-private partnership with organizations like CHAI.”
With the door open to continuing on the more rigorous healthcare AI tasks – such as government and industry alignment on AI assurance labs and how they will function – “work still needs to be done,” said Anderson.
“We need time to do that, and appreciating from the incoming administration that they are willing to work with us and work alongside us – and hopefully provide that time – I think is very refreshing and very exciting.”
Annual label updating and use IRL
Anderson said CHAI’s model card is intended to be “a living, breathing document as new capabilities emerge, particularly in generative AI space.”
It’s “very likely that the metrics and the methodologies we use to evaluate those emerging capabilities will need to change or need to be created,” he said.
Even before the FDA issued its draft total lifecycle guidance for medical devices, it finalized the Predetermined Change Control Plan review for AI and machine learning submissions – without triggering the need for new marketing submissions.
“As we think about the various sections of the model card, different data will need to be included – different evaluation results, different metrics…different kinds of use cases,” etc, said Anderson.
“That kind of flexibility is going to be really important,” he added, noting that an AI model or system’s nutrition label will require regular updates, “certainly, at least at an annual level.”
For providers, there is a great amount of complexity to consider when using AI-enabled clinical decision-support tools to minimize mistakes or oversights.
“Imperfect transparency is going to be something that we’re going to struggle with and need to work through again,” he stressed.
Whether or not a model was trained on a specific set of attributes that may relate to a particular patient may not be included on user-friendly model cards.
“You can include all the information under the sun in these model cards, but the vendor community would be at risk of disclosing [intellectual property,” he said. “So, it’s a balance of how do you protect the IP of the vendor, but give the customer – the doctor in this case – the necessary information to make the right decision about whether or not they should use that model with the patient they have in front of them,” Anderson said.
“The causal relationship is really profound on how that might affect a certain outcome for the patient in front of you,” he acknowledged.
Bringing others to the AI eval table
While HTI-1’s 31 categorical areas – aimed at electronic health records and other certified health IT – “is a really great starting place,” it’s not enough for the different use cases of AI – particularly in the direct-to-consumer space, said Anderson.
“The model cards that we’re developing are intended to be used quite broadly across those different use cases, and in the consumer space, particularly with generative AI, there’s going to be a whole bunch of new use cases coming up over the next year,” he explained.
Over the next two to five years, however, evaluating healthcare AI models is going to get even more complicated, raising questions about how they define “human flourishing.”
Anderson said that he believes use cases are going to be intimately tied to health AI agents, and developing trust frameworks around those is going to require the support of “ethicists, philosophers, sociologists, and spiritual leaders” to help advise technologists and AI experts in thinking through an evaluation framework for those tools.
“It’s going to be a real challenge developing the kind of framework for evaluation in that agentic AI future,” he said. “It’s a very intimate personal space, and how do we build that trust with those models? How do we evaluate those models?”
Beginning over the next year, Anderson said CHAI will spearhead a “very intentional effort to bring community members together and stakeholders that you wouldn’t necessarily think first of the kinds of stakeholders that you would include in an effort like this.”
“We really need to ensure that these models are aligned to our values, and we don’t have a rubric on how you evaluate a model. I don’t know how to do that yet. I don’t think anybody does, yet.”
Andrea Fox is senior editor of Healthcare IT News.
Email: afox@himss.org
Healthcare IT News is a HIMSS Media publication.