Implementing AI at a national scale, particularly in such critical industries as healthcare, is no easy feat.
In the HIMSS24 APAC session, “Implementing AI-Powered Triage for a National Virtual and Digital Front Door,” Dr Nirvana Luckraj, chief medical officer of Healthdirect, described how they chose, tested, and deployed an AI-powered clinical decision support system to augment its national helpline services.
Healthdirect offers a range of free 24/7 helpline services, including nurse triage and GP telehealth. It has dedicated lines for pregnant persons, the elderly, and most recently, people dealing with mental health problems. Additionally, Healthdirect provides information on medicines and curated health content on their website and features online booking and referrals, which directly integrate with national care pathways including My Health Record.
THE PROBLEM
Acquiring a CDSS is part of the actions Healthdirect took when it recently migrated its technology stack to its own operating platform.
It required the tool to consider multiple symptoms at the outset. “We needed a CDSS that could tolerate and analyse multiple symptoms. We also wanted it to reflect real consultation that considered risk factors such as predisposing factors, chronic history, and travel history.”
“We also wanted something that was next generation that could provide a more comprehensive and accurate triage,” Dr Luckraj added.
Previously, the service used a CDSS based on a decision tree-type algorithm which required “extensive maintenance” and “could only tolerate a single symptom input.”
PROPOSAL
The organisation then went to the market to seek a provider of an AI-based CDSS. It initially shortlisted five global vendors whose offerings were subjected to clinical testing.
To test the AI system’s applicability in the Australian public health system, Healthdirect worked with AI experts at Macquarie University’s Australian Institute of Health Innovation. They adopted a test methodology from a recent study that used case vignettes to check a CDSS’s accuracy.
“We ran these vignettes through the CDSS and found the performance was consistent with the recently published study. We also ran the same vignette through our old CDSS to compare and found that the old and new CDSS had similar results but that the new one was more accurate,” Dr Luckraj noted.
Eventually, Healthdirect went with the Polish firm Infermedica’s AI CDSS, which runs on a probabilistic Bayesian inference engine that considers symptoms, risk factors, and diseases, an updated medical knowledge base, and an API that connects to its CRM.
MEETING THE CHALLENGE
Immediately after launching the new CDSS on its helpline in March 2023, the implementing team encountered resistance from staff. “We had underestimated the change management required when getting healthcare professionals to use new technology to perform old tasks,” Dr Luckraj admitted.
From inputting one symptom to the CDSS, their nurses now have to enter multiple symptoms of patients. “The nurses initially were frustrated with the tool because it seemed to ask seemingly unrelated questions to the presentation of the patient. They also had to determine their past history to determine individual risk factors and the questions also seemed more advanced than the previous.”
In dealing with nurses’ initial frustration, the team took on their feedback and assisted them in adapting to the new tool. One of the key areas for improvement here, according to Dr Luckraj, was working with their vendor on a feedback loop mechanism where nurses do not have to agree with every CDSS output.
“It’s a support system, not a diagnostic system. So the nurses can either upgrade or downgrade the triage outcome giving a reason for their change of disposition. We also looked at those flows where the nurses disagree with the CDSS output to continuously improve the tool.”
RESULTS
A year after improving the CDSS, the implementing team observed that nurses are changing triage output less and agreeing more with the CDSS outcomes and that they are more satisfied with its new user interface. The tool’s accuracy is constantly improving, Dr Luckraj noted, as nurses also continue to provide feedback. On the consumer side, no serious clinical incidents attributable to the CDSS have been reported; they are now more satisfied with the service, with satisfaction scores reaching the highest in three years.
In December last year, the same AI-powered CDSS was introduced in Healthdirect’s consumer-facing online symptom checker, making it a “digital front door to numerous alternative care options, including direct access to urgent care services and virtual emergency departments.”
“In the future, we’re also looking to provide consumers with tailored information specific to their symptom input to provide an even better consumer experience,” Dr Luckraj shared.
Since incorporating AI into their helpline, Healthdirect claims to divert half of emergency calls to less acute services. From July 2023 to June 2024, it connected 44,000 calls to a virtual emergency department. It also booked nearly 5,500 people online for GP-led urgent care services. Meanwhile, almost 350,000 people were advised on safely managing their symptoms at home. Overall, Healthdirect nurses helped book over 370,000 GP service appointments.