ƬƵ

Please note: The Research + Innovation office will be closed for the holidays from noon on December 24 through January 2nd, inclusive. ❄️ Veuillez noter que le bureau de la Recherche et de l'Innovation sera fermé pour les vacances du 24 décembre à midi jusqu’au 2 janvier inclus.

News

Researchers find design flaws and oversight issues in certain health apps, offer solutions for more effective tools

‘Potential misdiagnosis is just a click away’
Published: 6 November 2024

AI-powered apps offering medical diagnoses at the click of a button are often limited by biased data and a lack of regulation, leading to inaccurate and unsafe health advice, a new study found.

ƬƵ University researchers presented symptom data from known medical cases to two popular, representative apps to see how well they diagnosed the conditions. While the apps sometimes gave correct diagnoses, they often failed to detect serious conditions, according to published in the Journal of Medical Internet Research. This potentially resulted in delayed treatment.

The researchers identified two main issues with the health apps they studied: biased data and a lack of regulation.

Bias and the ‘black box’ phenomenon

The bias issue is known as the "garbage in, garbage out" problem.

"These apps often learn from skewed datasets that don’t accurately reflect diverse populations," said Ma’n H. Zawati, lead author and Associate Professor in ƬƵ’s Department of Medicine.

Because the apps rely on data from smartphone users, they tend to exclude lower-income individuals. Race and ethnicity are also underrepresented in the data, said the authors. This creates a cycle where an app's assessments are based on a narrower group of users, leading to more biased results and potentially inaccurate medical advice.

While apps often include disclaimers stating they do not provide medical advice, the scholar argues that users' interpretations of these disclaimers – if read – do not always align.

The second issue is the "black box" nature of AI systems, where the technology evolves with minimal human oversight. Zawati said lack of transparency means even an app’s developers may not fully understand how it reaches conclusions.

"Without clear regulations, developers aren’t held accountable, making doctors reluctant to recommend these tools. For users, this means a potential misdiagnosis is just a click away,” said Zawati, who is also an Associate Member in ƬƵ’s Department of Equity, Ethics and Policy and the Faculty of Law and Research Director of the Centre of Genomics and Policy in ƬƵ’s Department of Human Genetics.

Call for AI oversight

To overcome limitations, developers can train apps on more diverse data sets, conduct regular audits to catch biases, enhance transparency to improve understanding of how algorithms work and include more human oversight in the decision-making process, he suggested.

“By prioritizing thoughtful design and rigorous oversight, AI-powered health apps have the potential to make health care more accessible to the public and become a valuable tool in clinical settings,” Zawati said.

This research was funded by the Fonds de recherche du Québec.

About the study

by Ma’n H. Zawati and Michael Lang was published in the Journal of Medical Internet Research.

Back to top