Concerns on medical devices

 An astounding paper appeared in Journal of Medical Internet Research (JMIR) on Jun 20, 2022. Sean Day et al in a work titled “Assessing the Clinical Robustness of Digital Health Startups: Cross-sectional Observational Analysis” find out that despite making claims on diagnosis, treatment and prevention (making them medical devices): - 44% of digital health start-ups have zero published research or regulatory approvals.

The digital health sector has experienced rapid growth over the past decade. However, health care technology stakeholders lack a comprehensive understanding of clinical robustness and claims across the industry.

The analysis aimed to examine the clinical robustness and public claims made by digital health companies.

A cross-sectional observational analysis was conducted using company data from the Rock Health Digital Health Venture Funding Database, the US Food and Drug Administration, and the US National Library of Medicine. Companies were included if they sell products targeting the prevention, diagnosis, or treatment phases of the care continuum. Clinical robustness was defined using regulatory filings and clinical trials completed by each company. Public claims data included clinical, economic, and engagement claims regarding product outcomes made by each company on its website.

A total of 224 digital health companies with an average age of 7.7 years were included in our cohort. Average clinical robustness was 2.5 (1.8 clinical trials and 0.8 regulatory filings) with a median score of 1. Ninety-eight (44%) companies had a clinical robustness score of 0, while 45 (20%) companies had a clinical robustness score of 5 or more. The average number of public claims was 1.3 (0.5 clinical, 0.4 economic, and 0.4 engagement); the median number of claims was 1. No correlation was observed between clinical robustness and number of clinical claims (r2=0.02), clinical robustness and total funding (r2=0.08), or clinical robustness and company age (r2=0.18).

Many digital health companies have a low level of clinical robustness and do not make many claims as measured by regulatory filings, clinical trials, and public data shared online. Companies and customers may benefit from investing in greater clinical validation efforts.

As Hugh Harvey commented, “put it another way, imagine if 44% of drugs had no clinical evidence, but were being sold to doctors and patients anyway.”

UK doctors warning on skin cancer apps

 Doctors issue warning about dangerous AI-based diagnostic skin cancer apps. 41% of Brits would trust an AI-based app to spot skin cancers.

41% of Brits would trust an AI-based app to spot skin cancers

Smartphone apps that use artificial intelligence (AI) to spot skin cancer are endangering the public, many of whom trust that these apps are safe to use, experts at the British Association of Dermatologists (BAD) have warned. This trust, along with a failure of many such apps to meet the appropriate regulatory standards, is putting users at risk.

This warning, from the BAD AI Working Party Group, follows a recent YouGov survey that has found that 41% of people in the UK would trust a smartphone application that employs AI to spot potential skin cancers.
While many people would trust apps which use AI to diagnose skin cancer, only 4% of respondents said they would be “very confident” in their ability to judge whether apps can do what they claim. Over half (52%) said they would not be confident in their ability to judge this.

The BAD AI Working Party Group position is that the published evidence to support that AI can be used safely and effectively to diagnose skin cancer is weak, putting people at risk of misdiagnosis and missed cancer diagnoses.

All apps that use AI for medical diagnosis or treatment are classified as ‘medical devices’ by regulatory bodies such as the Medicines and Healthcare products Regulatory Agency (MHRA). This means that they must undergo an authorisation process to make sure that they are safe and perform with the accuracy that they claim to achieve.

All medical devices sold in the UK must be certified by one of three marks: the CE (Conformité Européene), UKCA (UK Conformity Assessed) or UKNI (UK Northern Ireland). Class IIa and above marks confirm that they have undergone an official authorisation process overseen by national or international health authorities. The CE or UKCA mark may be applied by the manufacturer for Class I self-certified products but these products will not have undergone independent scrutiny.

The BAD has produced a guide to help the public spot potential warning signs when using diagnostic skin cancer apps.

Dr Rubeta Matin, Chair of the BAD Working Party Group, said:

“The results of this survey are concerning. The public has an understandable belief that these AI-based skin cancer detection apps are safe to use. The reality is that many don’t meet the standards required by regulators to diagnose medical conditions.”

“Part of the problem is how easy it is for AI diagnostic apps which don’t meet the regulatory requirements, produced by developers around the world, with wildly varying levels of expertise and experience, to be promoted alongside perfectly legitimate apps. These apps often have slick branding but if you look closer there is little evidence for their effectiveness.”

“It is all too common to see apps which claim to be able to check moles, which then follow these assertions with a disclaimer that the app is not a diagnostic device. These sort of small print U-turns on advertised features are not only dishonest but are clearly not permitted by the regulators.

While AI has immense potential to improve healthcare, it is important that AI is rolled out in a manner that is safe for the public.

AI assisted radiologists do better

 AI-assisted radiologists work better and faster compared to unassisted radiologists. AI is of great aid to radiologists in daily trauma emergencies, and could reduce the cost of missed fractures.

Lisa Canoni-Moyet et al published on Jun 29, 2022 (here) a work titled “Added value of an artificial intelligence solution for fracture detection in the radiologist's daily trauma emergencies workflow”.

The main objective of this study was to compare radiologists' performance without and with artificial intelligence (AI) assistance for the detection of bone fractures from trauma emergencies.

Five hundred consecutive patients (232 women, 268 men) with a mean age of 37 ± 28 (SD) years (age range: 0.25–99 years) were retrospectively included. Three radiologists independently interpreted radiographs without then with AI assistance after a 1-month minimum washout period. The ground truth was determined by consensus reading between musculoskeletal radiologists and AI results. Patient-wise sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for fracture detection and reading time were compared between unassisted and AI-assisted readings of radiologists. Their performances were also assessed by receiver operating characteristic (ROC) curves.

AI improved the patient-wise sensitivity of radiologists for fracture detection by 20% (95% confidence interval [CI]: 14–26), P< 0.001) and their specificity by 0.6% (95% CI: -0.9–1.5; P = 0.47). It increased the PPV by 2.9% (95% CI: 0.4–5.4; P = 0.08) and the NPV by 10% (95% CI: 6.8–13.3; P < 0.001). Thanks to AI, the area under the ROC curve for fracture detection of readers increased respectively by 10.6%, 10.2% and 9.9%. Their mean reading time per patient decreased by respectively 10, 16 and 12 s (P < 0.001).

AI-assisted radiologists work better and faster compared to unassisted radiologists. AI is of great aid to radiologists in daily trauma emergencies, and could reduce the cost of missed fractures.

Systemic racism

We often wonder if there is indeed such a thing called ‘systemic racism’ or if it can even be proven. We often reconcile to the fact that it may be ingrained in people from early childhood and hence becomes ‘systemic’.

Scientifically speaking, there were two papers published in Journal of American Medical Association (one in Internal Medicine and other in Oncology), which provide a stunning definition of how systemic racism works.

JAMA Internal medicine – On Jul 11, 2022, Eric Gottlieb et al published a research paper in JAMA Internal medicine titled “Assessment of racial and ethnic differences in oxygen supplementation among patients in the intensive care unit”. This work used the data between 2008 and 2019 in a Boston hospital available publically as MIMIC-IV dataset. In this cohort study of 3069 patients in the ICU, asian, black and hispanic patients had a higher adjusted time-weighted average pulse oxymetry reading and were administered significantly less supplemental oxygen for a given average hemoglobin oxygen saturation compared to white patients.

JAMA Oncology – On Jul 14, 2022, Manali Patel et al published a paper in JAMA oncology titled “Racial and Ethnic disparities in cancer care during the covid-19 pandemic”. In the survey study of 1240 US adults with cancer, black and latinx adults reported experiencing higher rates of delayed cancer care and more adverse social and economic effects than white adults. This study suggests that the covid-19 pandemic is associated with disparities in the receipt of timely cancer care. Could be due to limited or overburdened resources, but the systemic nature of racism manifests.

Both these studies nail the systemic racism that exists, even in healthcare. These are not one-off incidences. These are systematically administered biases over a long period of time over many individuals. Human race, have a long way to go to be equitable!

Last year, Zinzi Bailey et al published in New England Journal of Medicine (here) an article on how structural racism works, with racist policies as a root cause of US racial health inequities.

It concludes by saying structural racism reaches back to the beginnings of U.S. history, stretches across its institutions and economy, and dwells within our culture. Its durability contributes to the perception that Black disadvantage is intrinsic, permanent, and therefore normal. But considering structural racism as a root cause is not a modern analogue of the theory that disease is caused by “miasmas” — something that’s “in the air,” amorphous and undifferentiated. The article suggests four ways of moving forward by:

1. embracing the intellectual project of documenting the health impact of racism.
2. making available data that include race and ethnicity must improve, and efforts to develop and improve measurement of structural racism need to be supported, particularly those using available administrative databases.
3. self-introspecting - the medical and public health communities need to turn a lens on themselves, both as individuals and as institutions. Faculty and students need a more complete view both of U.S. history and of the ways in which medicine and public health have participated and continue to participate in racist practices.
4. acknowledging that structural racism has been challenged, perhaps most successfully, by mass social movements

Role of race and ancestry in estimating kidney function in CKD

Consideration of race in clinical decision making has recently come under much scrutiny and criticism. In particular, the use of indicators for Black race in equations that are widely used to estimate the glomerular filtration rate (GFR) from the serum creatinine level has been questioned.

Adults who identify as Black have higher serum creatinine levels on average, independent of age, sex, and GFR, than those who do not identify as Black. Thus, equations that have been developed to estimate the GFR from the serum creatinine level have generally incorporated information on race.9-11 It has been argued that the race coefficient should be removed from these equations, in part because its inclusion suggests that race is a biologic rather than primarily a social construct.3,12,13 However, concerns have also been raised about possible mis-classification of the estimated GFR that would ensue after removing the race coefficient from current equations.

In an article published in NEJM in Nov 2021, the inclusion of race in equations to estimate the glomerular filtration rate (GFR) has become controversial. Alternative equations that can be used to achieve similar accuracy without the use of race are needed.

In a large US study involving adults with chronic kidney disease, we conducted cross-sectional analyses of baseline data from 1248 participants for whom data, including the following, had been collected: race as reported by the participant, genetic ancestry markers, and the serum creatinine, serum cystatin C, and 24-hour urinary creatinine levels.

Using current formulations of GFR estimating equations, we found that in participants who identified as Black, a model that omitted race resulted in more underestimation of the GFR (median difference between measured and estimated GFR, 3.99 ml per minute per 1.73 m2 of body-surface area; 95% confidence interval [CI], 2.17 to 5.62) and lower accuracy (percent of estimated GFR within 10% of measured GFR [P10], 31%; 95% CI, 24 to 39) than models that included race (median difference, 1.11 ml per minute per 1.73 m2; 95% CI, −0.29 to 2.54; P10, 42%; 95% CI, 34 to 50). The incorporation of genetic ancestry data instead of race resulted in similar estimates of the GFR (median difference, 1.33 ml per minute per 1.73 m2; 95% CI, −0.12 to 2.33; P10, 42%; 95% CI, 34 to 50). The inclusion of non-GFR determinants of the serum creatinine level (e.g., body-composition metrics and urinary excretion of creatinine) that differed according to race reported by the participants and genetic ancestry did not eliminate the misclassification introduced by removing race (or ancestry) from serum creatinine–based GFR estimating equations. In contrast, the incorporation of race or ancestry was not necessary to achieve similarly statistically unbiased (median difference, 0.33 ml per minute per 1.73 m2; 95% CI, −1.43 to 1.92) and accurate (P10, 41%; 95% CI, 34 to 49) estimates in Black participants when GFR was estimated with the use of cystatin C.

The use of the serum creatinine level to estimate the GFR without race (or genetic ancestry) introduced systematic misclassification that could not be eliminated even when numerous non-GFR determinants of the serum creatinine level were accounted for. The estimation of GFR with the use of cystatin C generated similar results while eliminating the negative consequences of the current race-based approaches. (Funded by the National Institute of Diabetes and Digestive and Kidney Diseases and others.)

Back to the future 3 – the James Webb Space Telescope

At 5:49pm (IST) on Christmas day of 2021, NASA successfully launched the largest and the most powerful telescope ever produced – the James Webb Space Telescope (JWST)! Carl Sagan was once said to have been proud of the “Back to the Future” movie and had said that it depicted the science in it pretty well. Time has progressed and so has technology. So, the proverbial movie “Back to the Future 3” happened on Dec 25, 2021, just a week ago - in real life! The JWST was launched with the Ariane5 launcher from French Guiana. It was not just another launch. It was special and first in many ways. Carl Sagan also once said, “We can judge our progress by the courage of our questions and the depth of our answers, our willingness to embrace what is true rather than what feels good.” I am not privy to the context in which he was speaking, but we can relate the quote to the launch of JWST – because it is going to do precisely that – ask better questions and maybe redefine our understanding of the field if the answers are inconvenient. In science, we do that all the time!

You can Google about the JWST and find all the fun facts and trivia and indeed the serious science surrounding it. The point of this article is not to repeat it yet again, but instead to offer a different perspective.

Before we begin, let’s first understand what we are talking about. Why is James Webb space telescope named after James Webb? Who is he? Well, he wasn’t an astronomer like Hubble (Hubble space telescope has been servicing us with spectacular imagery past 30 years or so). James Webb was a government officer! He held together the fledgling NASA space program between 1961 and 1968 and worked towards ensuring the Apollo moon mission went ahead. This space telescope has been named after him, to honour his singular contribution to asking difficult questions and accepting uncomfortable truth, in pursuit of the unknown. So what is JWST? And what is the big deal?

In layman’s terms, JWST is as long and wide as a tennis court and is as high as a 3-storied building! Even the mighty Ariane5 could not have taken off with the telescope in this form! So the telescope was designed to be folded and will then be unfurled in space once it reaches its designated position. The idea of having a better telescope than Hubble had begun even as Hubble telescope was being launched. JWST is a multinational effort, spanning over 25 years of push and pull. 10 billion dollars later, we had the moment we witnessed on Dec 25^th!

Successful launch was only the beginning of the complicated mission. It has 341 points of failure in its next journey of around six months and anything can still go wrong! The telescope is now hurtling through the space to its designated point, the Lagrange point (L2), about 1.5 million kms away from earth. At L2, the gravity of earth and the Sun is balanced out and will eventually dock in its position about 28 days from the launch. It wont be until summer though, that the first pictures from JWST will be received.

So whats different in James Webb compared to Hubble?

1. Size. Hubble’s mirror size is approx 8 feet, whereas JWST mirror is about 21 feet in diameter. Hubble is the size of a bus, JWST is the size of a tennis court!.
2. The light itself that JWST will see will be different from Hubble. Hubble uses visible light – so it will see what we see, if we were in space at that location! JWST is only going to see the orange/red that we see and then infrared light beyond the red. The idea is to peep deeper into the space and it turns out that the deeper you peep, the more ‘red-shifted’ the light becomes. Stated alternately, what we see as normal visible light emitted billions of years ago now appears in infrared.
3. Hubble is orbiting the Earth. JWST will be orbiting the Sun. The infrared instruments on JWST need to be maintained at a very cold temperature (-266 deg C). The Lagrange point L2 offers conditions to achieve this. In fact the sun shields of JWST are so powerful that it can hide all of solar power and only let 1W be generated

As a fun fact, it is said that the JWST is so powerful that it can detect heat changes in fluttering of bees from a distance of 500,000 kms! That’s almost the distance between the Earth and the Moon!

How JWST moved out of sight of humanity?

I will summarize a rather elaborate process of how JWST detached from the Ariane5 upper stage and continued its journey – alone in the vast expanse of the universe – only to bring us more data to ascertain our theories or hypothesize new ones.

First, the moment the JWST detached from the upper stage of the Ariane5 launcher. This was the moment, 27 minutes after the launch that set JWST free!

Then, the rather spectacular views of the JWST from atop the upper stage as JWST drifts away into the space. Fig (b) shows the JWST back side view after detachment. Fig (c) shows the solar panels being unfurled through a carefully maneuvered time critical operation. This was critical for the JWST to start receiving the power and not become a piece of debris!

The solar panels were fully lit up by the solar rays – thereby signaling that all went well so far and the solar power was being used to charge the electronics on board!

The final view by humanity of the litup JWST as it hurtles away into the space towards its designated Lagrange point (L2). After this, we will never see the JWST again! We will see the images captured by it over next 5-10 years, but never the telescope itself!

As the final images like the one in fig (e) were being flashed, I was comprehending the magnitude of the moment. This was a picture for posterity. So many philosophical essays or science fictions can be written just around this moment of time! Many artists will come forth with their art forms to capture this moment. This moment has changed human quest for understanding of universe. To those who stare at night sky (Bangalore clouds willing :), do have these thoughts that eventually cross their minds at some time – just where does this infinite begin and where does it end? And the methodical answers to such questions so far agree with the big bang. We will know soon from the pictures of JWST, what were the earliest galaxies like? What were they composed of etc. I said its proverbial “Back to the Future 3” because it will really start detecting the faintest of signals that may have originated close to the time Big Bang occurred, or when the universe just came into being. These are fascinating times for the scientific pursuit and we all look forward to the treasure trove that JWST will share with us!

Finally, what new science can we expect? 

NASA, ESA and Canada spent around 10 billion dollars for a few top-level goals

1. To study light from the first stars and galaxies after the Big Bang.

2. To study the formation and evolution of these galaxies.

3. To understand the formation of stars and planetary systems.

4. To study planetary systems and origin of life!

As Ken Sembach, Director of the space telescope at Science Institute in Baltimore said, “Science wont be the same after today. Webb is more than a telescope – it is a gift to everyone who contemplates the vastness of the universe”. And gift it is. Hopefully, by the time of refleXion’s next issue, the JWST will be in L2 and an issue further later, we would have the first pictures.

AI enabled medical devices by US FDA

 

In India, it is difficult to track regulatory approvals for many products. Much worse are the AI/ML enabled algorithms. Around the world, things are not great either. However, recently, US FDA decided to publish a list of the AI/ML enabled medical devices (or algorithms) that it has approved by category. It is interesting to browse the list as it shows some interesting patterns.

Interest in medical devices incorporating ML functionality has increased in recent years. Over the past decade, the FDA has reviewed and authorized a growing number of devices legally marketed (via 510(k) clearance, granted De Novo request, or approved PMA) with ML across many different fields of medicine—and expects this trend to continue.

The FDA is providing this initial list of AI/ML-enabled medical devices marketed in the United States as a resource to the public about these devices and the FDA’s work in this area.

On October 14, 2021, FDA’s Digital Health Center of Excellence (DHCoE) held a public workshop on the transparency of artificial intelligence/machine learning-enabled medical devices. The workshop followed the recently published list of nearly 350 AI/ML-enabled medical devices that have received regulatory approval since 1997. The workshop was aimed at moving forward the objectives of FDA’s DHCoE to “empower stakeholders to advance healthcare by fostering responsible and high-quality digital health innovation.” The DHCoE was established in 2020 within FDA’s Center for Devices and Radiological Health (CDRH) under Bakul Patel.

This initial list contains publicly available information on AI/ML-enabled devices. The FDA assembled this list by searching FDA’s publicly-facing information, as well as by reviewing information in the publicly available resources cited below and in other publicly available materials published by the specific manufacturers.

This list is not meant to be an exhaustive or comprehensive resource of AI/ML-enabled medical devices. Rather, it is a list of AI/ML-enabled devices across medical disciplines, based on publicly available information.

If grouped by category, this is what we see.

Radiology 241

Cardiovascular 41

Hematology 13

Neurology 12

Ophthalmic 6

Chemistry 5

Surgery 5

Microbiology 5

Anesthesia 4

GI-Urology 4

Hospital 3

Dental 1

Ob/Gyn 1

Orthopedic 1

Pathology 1

Radiology is no surprise with almost 70% share of listed devices in that area as most of the AI work in healthcare and indeed in medical imaging has been primarily around chest X-rays and there are many algorithms and solutions available. What is surprising is the last in the list, pathology! Considering that too in some ways is also imaging based (whole slide scans for example), it is intriguing that it does not list as many as it should.

What is also visible from the list is that other than radiology really, there are not many solutions in other areas. Radiology is the so to speak, low hanging fruit in healthcare and imaging.

There is so much scope to do in healthcare. The need of the hour is for computer science community to engage with medical fraternity and help deploy some of the algorithms, not as to replace those in there, but to aid them in making decision, the proverbial second opinion. It does not harm. Can it bias the practitioner to just go with the AI prediction? It may, but if there is uncertainty, there is anyways a dilemma the practitioners face.

It is time, given the scale and scarcity of resources we have in India and population so widely spread geographically, that such solutions will only help provide better healthcare. How to achieve that is a different question though.