Credit score: Nature Medication (2025). DOI: 10.1038/s41591-024-03445-1
By conducting checks beneath an experimental state of affairs, a crew of medical researchers and AI specialists at NYU Langone Well being has demonstrated how simple it’s to taint the information pool used to coach LLMs.
For his or her examine revealed within the journal Nature Medication, the group generated 1000’s of articles containing misinformation and inserted them into an AI coaching dataset and performed normal LLM queries to see how usually the misinformation appeared.
Prior analysis and anecdotal proof have proven that the solutions given by LLMs comparable to ChatGPT are usually not all the time right and, actually, are generally wildly off-base. Prior analysis has additionally proven that misinformation planted deliberately on well-known web websites can present up in generalized chatbot queries. On this new examine, the analysis crew wished to know the way simple or tough it is likely to be for malignant actors to poison LLM responses.
To search out out, the researchers used ChatGPT to generate 150,000 medical paperwork containing incorrect, outdated and unfaithful information. They then added these generated paperwork to a check model of an AI medical coaching dataset. They then educated a number of LLMs utilizing the check model of the coaching dataset. Lastly, they requested the LLMs to generate solutions to five,400 medical queries, which have been then reviewed by human consultants trying to spot examples of tainted information.
The analysis crew discovered that after changing simply 0.5% of the information within the coaching dataset with tainted paperwork, all of the check fashions generated extra medically inaccurate solutions than they’d previous to coaching on the compromised dataset. As one instance, they discovered that every one the LLMs reported that the effectiveness of COVID-19 vaccines has not been confirmed. Most of them additionally misidentified the aim of a number of frequent drugs.
The crew additionally discovered that lowering the variety of tainted paperwork within the check dataset to only 0.01% nonetheless resulted in 10% of the solutions given by the LLMs containing incorrect information (and dropping it to 0.001% nonetheless led to 7% % of the solutions being incorrect), suggesting that it requires just a few such paperwork posted on web sites in the actual world to skew the solutions given by LLMs.
The crew adopted up by writing an algorithm in a position to establish medical information in LLMs after which used cross-referencing to validate the information, however they observe that there isn’t a sensible technique to detect and take away misinformation from public datasets.
Extra info:
Daniel Alexander Alber et al, Medical massive language fashions are susceptible to data-poisoning assaults, Nature Medication (2025). DOI: 10.1038/s41591-024-03445-1
© 2025 Science X Community
Quotation:
Check of ‘poisoned dataset’ reveals vulnerability of LLMs to medical misinformation (2025, January 11)
retrieved 11 January 2025
from https://medicalxpress.com/information/2025-01-poisoned-dataset-vulnerability-llms-medical.html
This doc is topic to copyright. Other than any truthful dealing for the aim of personal examine or analysis, no
half could also be reproduced with out the written permission. The content material is supplied for info functions solely.
