Concerns on medical devices

 An astounding paper appeared in Journal of Medical Internet Research (JMIR) on Jun 20, 2022. Sean Day et al in a work titled “Assessing the Clinical Robustness of Digital Health Startups: Cross-sectional Observational Analysis” find out that despite making claims on diagnosis, treatment and prevention (making them medical devices): - 44% of digital health start-ups have zero published research or regulatory approvals.

The digital health sector has experienced rapid growth over the past decade. However, health care technology stakeholders lack a comprehensive understanding of clinical robustness and claims across the industry.

The analysis aimed to examine the clinical robustness and public claims made by digital health companies.

A cross-sectional observational analysis was conducted using company data from the Rock Health Digital Health Venture Funding Database, the US Food and Drug Administration, and the US National Library of Medicine. Companies were included if they sell products targeting the prevention, diagnosis, or treatment phases of the care continuum. Clinical robustness was defined using regulatory filings and clinical trials completed by each company. Public claims data included clinical, economic, and engagement claims regarding product outcomes made by each company on its website.

A total of 224 digital health companies with an average age of 7.7 years were included in our cohort. Average clinical robustness was 2.5 (1.8 clinical trials and 0.8 regulatory filings) with a median score of 1. Ninety-eight (44%) companies had a clinical robustness score of 0, while 45 (20%) companies had a clinical robustness score of 5 or more. The average number of public claims was 1.3 (0.5 clinical, 0.4 economic, and 0.4 engagement); the median number of claims was 1. No correlation was observed between clinical robustness and number of clinical claims (r2=0.02), clinical robustness and total funding (r2=0.08), or clinical robustness and company age (r2=0.18).

Many digital health companies have a low level of clinical robustness and do not make many claims as measured by regulatory filings, clinical trials, and public data shared online. Companies and customers may benefit from investing in greater clinical validation efforts.

As Hugh Harvey commented, “put it another way, imagine if 44% of drugs had no clinical evidence, but were being sold to doctors and patients anyway.”

UK doctors warning on skin cancer apps

 Doctors issue warning about dangerous AI-based diagnostic skin cancer apps. 41% of Brits would trust an AI-based app to spot skin cancers.

41% of Brits would trust an AI-based app to spot skin cancers

Smartphone apps that use artificial intelligence (AI) to spot skin cancer are endangering the public, many of whom trust that these apps are safe to use, experts at the British Association of Dermatologists (BAD) have warned. This trust, along with a failure of many such apps to meet the appropriate regulatory standards, is putting users at risk.

This warning, from the BAD AI Working Party Group, follows a recent YouGov survey that has found that 41% of people in the UK would trust a smartphone application that employs AI to spot potential skin cancers.
While many people would trust apps which use AI to diagnose skin cancer, only 4% of respondents said they would be “very confident” in their ability to judge whether apps can do what they claim. Over half (52%) said they would not be confident in their ability to judge this.

The BAD AI Working Party Group position is that the published evidence to support that AI can be used safely and effectively to diagnose skin cancer is weak, putting people at risk of misdiagnosis and missed cancer diagnoses.

All apps that use AI for medical diagnosis or treatment are classified as ‘medical devices’ by regulatory bodies such as the Medicines and Healthcare products Regulatory Agency (MHRA). This means that they must undergo an authorisation process to make sure that they are safe and perform with the accuracy that they claim to achieve.

All medical devices sold in the UK must be certified by one of three marks: the CE (Conformité Européene), UKCA (UK Conformity Assessed) or UKNI (UK Northern Ireland). Class IIa and above marks confirm that they have undergone an official authorisation process overseen by national or international health authorities. The CE or UKCA mark may be applied by the manufacturer for Class I self-certified products but these products will not have undergone independent scrutiny.

The BAD has produced a guide to help the public spot potential warning signs when using diagnostic skin cancer apps.

Dr Rubeta Matin, Chair of the BAD Working Party Group, said:

“The results of this survey are concerning. The public has an understandable belief that these AI-based skin cancer detection apps are safe to use. The reality is that many don’t meet the standards required by regulators to diagnose medical conditions.”

“Part of the problem is how easy it is for AI diagnostic apps which don’t meet the regulatory requirements, produced by developers around the world, with wildly varying levels of expertise and experience, to be promoted alongside perfectly legitimate apps. These apps often have slick branding but if you look closer there is little evidence for their effectiveness.”

“It is all too common to see apps which claim to be able to check moles, which then follow these assertions with a disclaimer that the app is not a diagnostic device. These sort of small print U-turns on advertised features are not only dishonest but are clearly not permitted by the regulators.

While AI has immense potential to improve healthcare, it is important that AI is rolled out in a manner that is safe for the public.

AI assisted radiologists do better

 AI-assisted radiologists work better and faster compared to unassisted radiologists. AI is of great aid to radiologists in daily trauma emergencies, and could reduce the cost of missed fractures.

Lisa Canoni-Moyet et al published on Jun 29, 2022 (here) a work titled “Added value of an artificial intelligence solution for fracture detection in the radiologist's daily trauma emergencies workflow”.

The main objective of this study was to compare radiologists' performance without and with artificial intelligence (AI) assistance for the detection of bone fractures from trauma emergencies.

Five hundred consecutive patients (232 women, 268 men) with a mean age of 37 ± 28 (SD) years (age range: 0.25–99 years) were retrospectively included. Three radiologists independently interpreted radiographs without then with AI assistance after a 1-month minimum washout period. The ground truth was determined by consensus reading between musculoskeletal radiologists and AI results. Patient-wise sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) for fracture detection and reading time were compared between unassisted and AI-assisted readings of radiologists. Their performances were also assessed by receiver operating characteristic (ROC) curves.

AI improved the patient-wise sensitivity of radiologists for fracture detection by 20% (95% confidence interval [CI]: 14–26), P< 0.001) and their specificity by 0.6% (95% CI: -0.9–1.5; P = 0.47). It increased the PPV by 2.9% (95% CI: 0.4–5.4; P = 0.08) and the NPV by 10% (95% CI: 6.8–13.3; P < 0.001). Thanks to AI, the area under the ROC curve for fracture detection of readers increased respectively by 10.6%, 10.2% and 9.9%. Their mean reading time per patient decreased by respectively 10, 16 and 12 s (P < 0.001).

AI-assisted radiologists work better and faster compared to unassisted radiologists. AI is of great aid to radiologists in daily trauma emergencies, and could reduce the cost of missed fractures.

Systemic racism

We often wonder if there is indeed such a thing called ‘systemic racism’ or if it can even be proven. We often reconcile to the fact that it may be ingrained in people from early childhood and hence becomes ‘systemic’.

Scientifically speaking, there were two papers published in Journal of American Medical Association (one in Internal Medicine and other in Oncology), which provide a stunning definition of how systemic racism works.

JAMA Internal medicine – On Jul 11, 2022, Eric Gottlieb et al published a research paper in JAMA Internal medicine titled “Assessment of racial and ethnic differences in oxygen supplementation among patients in the intensive care unit”. This work used the data between 2008 and 2019 in a Boston hospital available publically as MIMIC-IV dataset. In this cohort study of 3069 patients in the ICU, asian, black and hispanic patients had a higher adjusted time-weighted average pulse oxymetry reading and were administered significantly less supplemental oxygen for a given average hemoglobin oxygen saturation compared to white patients.

JAMA Oncology – On Jul 14, 2022, Manali Patel et al published a paper in JAMA oncology titled “Racial and Ethnic disparities in cancer care during the covid-19 pandemic”. In the survey study of 1240 US adults with cancer, black and latinx adults reported experiencing higher rates of delayed cancer care and more adverse social and economic effects than white adults. This study suggests that the covid-19 pandemic is associated with disparities in the receipt of timely cancer care. Could be due to limited or overburdened resources, but the systemic nature of racism manifests.

Both these studies nail the systemic racism that exists, even in healthcare. These are not one-off incidences. These are systematically administered biases over a long period of time over many individuals. Human race, have a long way to go to be equitable!

Last year, Zinzi Bailey et al published in New England Journal of Medicine (here) an article on how structural racism works, with racist policies as a root cause of US racial health inequities.

It concludes by saying structural racism reaches back to the beginnings of U.S. history, stretches across its institutions and economy, and dwells within our culture. Its durability contributes to the perception that Black disadvantage is intrinsic, permanent, and therefore normal. But considering structural racism as a root cause is not a modern analogue of the theory that disease is caused by “miasmas” — something that’s “in the air,” amorphous and undifferentiated. The article suggests four ways of moving forward by:

1. embracing the intellectual project of documenting the health impact of racism.
2. making available data that include race and ethnicity must improve, and efforts to develop and improve measurement of structural racism need to be supported, particularly those using available administrative databases.
3. self-introspecting - the medical and public health communities need to turn a lens on themselves, both as individuals and as institutions. Faculty and students need a more complete view both of U.S. history and of the ways in which medicine and public health have participated and continue to participate in racist practices.
4. acknowledging that structural racism has been challenged, perhaps most successfully, by mass social movements

Role of race and ancestry in estimating kidney function in CKD

Consideration of race in clinical decision making has recently come under much scrutiny and criticism. In particular, the use of indicators for Black race in equations that are widely used to estimate the glomerular filtration rate (GFR) from the serum creatinine level has been questioned.

Adults who identify as Black have higher serum creatinine levels on average, independent of age, sex, and GFR, than those who do not identify as Black. Thus, equations that have been developed to estimate the GFR from the serum creatinine level have generally incorporated information on race.9-11 It has been argued that the race coefficient should be removed from these equations, in part because its inclusion suggests that race is a biologic rather than primarily a social construct.3,12,13 However, concerns have also been raised about possible mis-classification of the estimated GFR that would ensue after removing the race coefficient from current equations.

In an article published in NEJM in Nov 2021, the inclusion of race in equations to estimate the glomerular filtration rate (GFR) has become controversial. Alternative equations that can be used to achieve similar accuracy without the use of race are needed.

In a large US study involving adults with chronic kidney disease, we conducted cross-sectional analyses of baseline data from 1248 participants for whom data, including the following, had been collected: race as reported by the participant, genetic ancestry markers, and the serum creatinine, serum cystatin C, and 24-hour urinary creatinine levels.

Using current formulations of GFR estimating equations, we found that in participants who identified as Black, a model that omitted race resulted in more underestimation of the GFR (median difference between measured and estimated GFR, 3.99 ml per minute per 1.73 m2 of body-surface area; 95% confidence interval [CI], 2.17 to 5.62) and lower accuracy (percent of estimated GFR within 10% of measured GFR [P10], 31%; 95% CI, 24 to 39) than models that included race (median difference, 1.11 ml per minute per 1.73 m2; 95% CI, −0.29 to 2.54; P10, 42%; 95% CI, 34 to 50). The incorporation of genetic ancestry data instead of race resulted in similar estimates of the GFR (median difference, 1.33 ml per minute per 1.73 m2; 95% CI, −0.12 to 2.33; P10, 42%; 95% CI, 34 to 50). The inclusion of non-GFR determinants of the serum creatinine level (e.g., body-composition metrics and urinary excretion of creatinine) that differed according to race reported by the participants and genetic ancestry did not eliminate the misclassification introduced by removing race (or ancestry) from serum creatinine–based GFR estimating equations. In contrast, the incorporation of race or ancestry was not necessary to achieve similarly statistically unbiased (median difference, 0.33 ml per minute per 1.73 m2; 95% CI, −1.43 to 1.92) and accurate (P10, 41%; 95% CI, 34 to 49) estimates in Black participants when GFR was estimated with the use of cystatin C.

The use of the serum creatinine level to estimate the GFR without race (or genetic ancestry) introduced systematic misclassification that could not be eliminated even when numerous non-GFR determinants of the serum creatinine level were accounted for. The estimation of GFR with the use of cystatin C generated similar results while eliminating the negative consequences of the current race-based approaches. (Funded by the National Institute of Diabetes and Digestive and Kidney Diseases and others.)