AI/ML significantly amplify society’s desire at a rapid rate to apply such technologies in day-to-day workings without being proportionately rational about their adverse consequences
Image: Getty Images
Big data is central to medicine and health. Consequently, the medical field is becoming increasingly accepting of AI and machine learning (ML) applications in tasks such as (but not limited to) medical image analysis, disease screening, drug discovery, patient experience, healthcare data management, prognostics and diagnostics, decision support, surgical robotics, virtual health assistants, and screening drug targets, and personal healthcare. The increasing reliance of such applications on AI/ML is continuously driving software innovation in the medical AI business, which is currently valued globally at around $5 billion and projected to reach approximately $50 billion by 2026. One of the reasons is that the biggest tech giants (including Google, Apple, IBM, Samsung, Microsoft, NVIDIA, and Amazon) are investing in health/medical AI.
It is well known through the Collingridge dilemma in the social sciences that technological breakthroughs such as (generative) AI/ML significantly amplify society’s desire at a rapid rate to apply such technologies in day-to-day workings without being proportionately rational about their adverse consequences. Alternatively, prejudging the negative societal consequences of disruptive path-breaking technology positively affecting (or seducing) every sphere of business, engineering, and governance for the social good is extremely difficult. One such negative consequence is with respect to cyber assurance wherein medical AI companies try to convince the healthy industry stakeholders (that include consumers) that their products and services are relatively far more cyber ethical and trustworthy when compared to the risks they pose to society – in reality, nothing could be farther from the truth.
In this article, we lay down five important challenges to ensuring cyber assurance in the medical AI business from the contexts of both the supply side (i.e., the designers of medical AI technology) and the demand side (i.e., the clients that use medical AI applications sold by the supply side) viewpoints driving arguments solely on factors affecting society’s trust in medical AI.
Challenge #1 – Data Acquisition
The performance of medical AI relies greatly on the quality of the data it gets to work with. The most common sources of medical data include (but are not limited to) research literature data, electronic health records (EHRs), data from clinical trials, and data obtained from modern mobile and intelligent wearables (including fitness applications). If the data from such sources are not from human entities, it could
1. Generate wrong diagnoses, resulting in poor medicare if used to train medical AI models (e.g., as in the case of IBM Watson’s oncology prediction system that often uses synthetic oncology data to train the latter and is known to generate erroneous and unsafe recommendations), and
2. Reduce the trust for lot-at-stake and high-risk-averse patients/stakeholders in such non-cyber-assured medical AI services.
One could argue the need for synthetic data in the medical AI business when there is usually enough non-synthetic data available to train AI models. Synthetic data for medical AI training is in fashion (among other things) because it addresses stakeholder privacy concerns, complies with data utility agreements and protocols, is relatively much less costly to acquire compared to non-synthetic data, and can support digital twins of patients and their trajectories to optimise treatment plans.
Challenge #2 – Data Processing
The data template for medical AI is very nuanced in practice. The typical challenges to process (annotate) data include faulty or error-prone (manual) data entry, non-standardised standards in metadata entry, lack of proper data management, inefficient database management caused by unstructured medical data in the form of multimedia (e.g., text and images) that is difficult to pre-process for an AI algorithm, and poor data cleaning processes in place. Poor and inconsistent data annotation implies poor data quality even if the collected raw data is accurate and non ‘noisy’.
As an example, pulmonary nodules should ideally be diagnosed using pathological biopsies. However, not every patient with pulmonary nodules undergoes a biopsy and alternatively resorts to clinical CT scans. The images post CT scans are used as input to medical AI that necessarily needs to be annotated by specialist doctors. In this example of pulmonary nodules, the data attributes include (but are not limited to) the nodule size, CT values, type of nodule (e.g., solid, glassy), and signs (lobar, burr, concave umbilical). These input images are not uniform across hospitals because the hardware and software used to output these images differ across hospitals. Add to this the heterogeneity in
- The judgement biases of specialists in these hospitals and
- The quality and standards of labelled image data. All these factors taken together might result in a lack of standardised data clarity for the specialists who will rely on AI/ML inference algorithms for patient diagnostics.
Challenge #3 – Bias and Opacity
Algorithmic bias is inherent and undetectable in black-box AI/ML algorithms and is sourced from human-data convergence. Humans use their subjective and often imperfect perceptions to handle sparse/noisy medical data and weigh ML algorithmic attributes that work for/against certain sections of demography. These human-driven data handling/processing biases then amplify the AI/ML algorithmic output bias that scales with (a) data points fitting a certain demography and (b) iterations of a machine learning inference algorithm. As an example of algorithmic bias in medical AI, the database of certain skin diseases, such as melanoma, is mostly populated with whites. Hence, an AI inference algorithm will be difficult to correctly apply to the black population due to the lack of sufficient melanoma samples for such a population – resulting in biased race discrimination. The bias from AI/ML black boxes in the medical AI business will likely over/underestimate patient risks and consolidate/exacerbate health care needs based on demography-driven skewed human sample data.
Also read: Considering insurance to manage IoT-driven catastrophic cyber-risk
Algorithmic opacity of AI/ML products arises because:
- The developers want to protect trade secrets and IP information in a competitive environment.
- The general public does not understand domain-specific terminologies such as AI/ML, algorithms, and software coding.
- Intricate complexities of AI, ML, and specifically multi-layer neural-networked deep ML algorithms that are often too hard for software designers to detangle – especially for data-based AI/ML where the complex form of the function that relates input data to inference outputs (i.e., judgement) is often beyond human cognition.
This opacity can lead to
- Patients and doctors losing trust in medical AI for diagnostics and
- Doctors facing AI explainability challenges on diagnostics and AI-outputted treatment plans.
Challenge #4 – Cyber Safety/Security in Medical AI
In principle, safety and security should not be compromised in medical AI. While humans are never perfect, the errors due to humans do not scale as much when compared to AI errors. As an example, AI-driven surgical robots caused approximately 1500 safety incidents and 150 deaths in the US between 2000 and 2015. Such statistics usually emerge due to
- Flawed AI algorithms and
- Insufficiently trained AI black boxes that improperly handle ‘blind spots’ that interfere at scale with the correct operational practices of human doctors.
On another angle related to scale, medical chatbots and care robots are posed with the challenge of updating their AI logic to handle the dynamics of diagnostic/treatment/care/preferences of a patient over time. An inability to properly internalise these dynamics in the AI increases healthcare risks over a population scale. It gives rise to ethical dilemmas such as whether to propose the best computerised medicare with increased pain or a preference-based and mutually agreed (with the doctor) sub-optimal medicare with less/no pain.
Adversarial AI is another challenge to the safety of modern AI/ML-based medical AI services, especially in medical imaging. A 5 percent or more adversarial statistical noise addition to a biomedical image is often not detectable to the human eye. Still, it is sufficient to change the decision of a tumour from non-malignant to malignant. Such attacks pose serious threats to the safety of medical AI. Imagine a scary world with AI-integrated pacemakers under the control of a rogue adversary!
Finally, the challenges due to unwanted medical data breaches (approximately 15 percent of global data breaches) is a significant point to consider in medical AI.
While patients’ personal medical information is private between the doctor and the patient, adversarial AI can lead to dignity-affecting privacy breaches, resulting in the patient’s family knowing the information they are not supposed to know. In addition, such breaches might leak information to insurance companies, unfairly increasing client premiums without a thorough and holistic analysis of client medical conditions. Moreover, medical databases stored on the cloud and third-party servers are always under threat of a privacy cyber-attack with enough incentives for adversaries to get access to data, code, and AI training data.
Challenge #5 – The Liability of Medical AI
In the inevitable event of an AI/ML-driven medical AI service failing or becoming dysfunctional, who should be held responsible? Is the AI alone or the team building the AI behind the service? This is a fundamental question to which there are no clear answers, but it is important enough for effective risk management and regulation of medical AI services. Though there have been Turing tests in computer science research that have verified certain degrees of consciousness of advanced AI, it is difficult for AI to be solely liable for mishaps when they do not have free will. In a medical context, AI is, at most, an auxiliary tool used by doctors and should not be held as a responsible subject simply because there is a wide gap between rule/probability-based diagnosis and emotion and empathy-induced human/doctor judgment. This argument leaves us with doctors, medical institutions endorsing AI in services, and AI software developers taking liability for AI-led service mishaps. However, this is a multi-stakeholder liability problem parallel to cyber risk allocation among stakeholders that has been unsolved for decades.
Ranjan Pal (MIT Sloan School of Management), Peihan Liu (Harvard University), Bodhibrata Nag (Indian Institute of Management Calcutta), Mahesh Devnani (PGIMER Chandigarh), Surekha Kashyap (AFMC Pune)
[This article has been published with permission from IIM Calcutta. www.iimcal.ac.in Views expressed are personal.]