What happens if an employee unknowingly enters sensitive information into a public large language model (LLM)? Could that information then be leaked to other users of the same LLM?
For example, if you ask ChatGPT or Claude to read and summarize a confidential contract, a patient record or a customer complaint transcript, you are inadvertently feeding that information to a public source that could then share your input with other users without your permission.
Potential data leakage from LLMs is a serious issue we don’t speak enough about. Currently, we see a lot of focus on hallucinations as a risk when using LLMs. However, preventing hallucinations is mostly a quality measure. Data leakage, on the other hand, is more of a safety and data privacy issue, and a key concern to address when building safety and trust in your AI systems.
Every organization should monitor three areas closely to prevent data leakage in LLMs: prompt data leakage, model data leakage and test data leakage in training data.
I’ll describe each in more detail below, along with tips for preventing data leakage.
1. Leakage in prompts
Prompt leakage occurs innocently enough, but it’s a critical security risk because few realize that LLMs retain everything they’re told and can be exploited by those with ill intentions.
Writing prompts is more art than science, so while employees may be experimenting with generative AI (GenAI), they may use personally identifiable information (PII) or intellectual property from the organization to create prompts.
Before making GenAI available to employees, make sure they have good training in writing prompts and preventing PII leakage.
Also, be aware that some employees may be using GenAI without permission or your knowledge, and they may be using free services that do not protect the data entered into a prompt. Once again, organization-wide training in GenAI and LLMs is essential in preventing the leakage of important data.
Report: GenAI burst onto the scene a couple of years ago, but where are we now?
2. Model data leakage
Model data leakage occurs when your model returns PII in its response.
Let’s say someone asks the model about general investment strategies for high-net-worth individuals. Perhaps the model was trained on a data set including specific details such as the names of these individuals or their unique investment strategies.
The model may inadvertently reveal a person’s name even when you’re just asking about general investment strategies.
This is really concerning because we know the model has memorized this information and it may be spread widely to any LLM prompt.
To prevent these risks, teams building AI models should train the models not to release private information, and users should report any leakage of private information.
3. Leakage of test data in training data
When test data is included in training data, it’s hard to validate the model’s accuracy and generalizations because we can’t be sure what the model has already seen.
It’s essential that LLM training data be kept segregated from model data to prevent inaccurate results that lead to poor decision making. You’ll also need to perform regular data audits to make sure data sets are free from contaminated data.
What does it all mean?
While many of us no longer consider LLMs and GenAI new technologies, there’s still much to learn before blindly adopting the latest tools of the trade.
Organizations must understand the dangers of data leakage before they can start rooting it out. What you don’t know right now may be far more important than what you do.
While it might seem that keeping up with the latest technologies is important enough to overrule worries about the possibility of inadvertently leaking small bits of PII or intellectual property in LLMs, you need to consider how connected the world is today. What’s leaked in one piece of data may end up in the hands of data-peddling organizations around the world.
While it’s important to keep looking forward, it’s even more important to guard against those who may steal and sell your data without your knowledge.
