Credit score: CC0 Public Area
Institute for Techniques Biology (ISB) researchers have gained new insights into the strengths and limitations of utilizing synthetic intelligence (AI) to establish social determinants of well being from digital well being data. Their outcomes are printed within the Journal of Medical Web Analysis.
The ISB crew, collaborating with Windfall, leveraged massive language fashions (LLM) developed from generative pre-trained transformers (GPT). Their analysis was performed fully throughout the safe Windfall inside surroundings.
The examine—geared toward detecting housing instability—was performed on over 25,000 medical notes from 795 pregnant ladies and evaluated two massive language fashions (GPT-4 and GPT-3.5), a named entity recognition mannequin, common expressions, and human evaluate.
This analysis goes past earlier research in two essential methods. First, researchers measured how properly AI can discover housing challenges, distinguish between present and previous housing instability, and supply direct proof from medical notes. Second, they measured whether or not AI carried out otherwise if the notes had been de-identified.
GPT-4 was the best of the 4 applied sciences examined, and was higher than people at discovering circumstances of housing instability (recall). People, nonetheless, have been higher at understanding when individuals didn’t have housing instability (precision). People have been additionally higher at offering right proof from a medical notice.
“These results show that LLMs present a scalable, cost-effective solution for an initial search for patients who may benefit from outreach,” stated ISB Affiliate Professor Jennifer Hadlock, MD, corresponding writer of the paper.
GPT-4 typically supplied the identical textual content that people had chosen to justify solutions. Notably, no hallucinated feedback appeared within the GPT-4 responses that have been reviewed, most probably as a result of the researchers designed the LLM directions to request verbatim proof from notes.
Nevertheless, there have been circumstances the place the AI interpretation of notice textual content was incorrect in ways in which may very well be deceptive. That is particularly essential as a result of housing standing can intersect with many different difficult or dangerous conditions, similar to home abuse.
“When a health care professional decides whether and how to reach out to offer help, they take great care to consider patient safety. Our results illustrate that it would still be essential to have a human read the actual text in the chart, not just the LLM summary,” Hadlock added.
Additional, in a novel experiment, researchers confirmed that recall was worse when run on de-identified variations of the identical medical notes. These notes had been de-identified with an automatic method known as “hide in plain sight,” which replaces probably delicate info (similar to names, areas and dates) with lifelike however fictitious options. The de-identification generally reclassified important info sufficient to skew the power to precisely decide housing instability.
“This highlights the need to refine de-identification methods to preserve privacy without losing important details about social determinants of health,” stated Alexandra Ralevski, Ph.D., lead writer of the examine.
Extra info:
Alexandra Ralevski et al, Utilizing Massive Language Fashions to Summary Complicated Social Determinants of Well being From Authentic and Deidentified Medical Notes: Growth and Validation Research, Journal of Medical Web Analysis (2024). DOI: 10.2196/63445
Offered by
Institute for Techniques Biology
Quotation:
New analysis highlights AI’s potential and pitfalls in scouring well being information (2024, November 21)
retrieved 22 November 2024
from https://medicalxpress.com/information/2024-11-highlights-ai-potential-pitfalls-scouring.html
This doc is topic to copyright. Other than any honest dealing for the aim of personal examine or analysis, no
half could also be reproduced with out the written permission. The content material is supplied for info functions solely.
