The corporate AI bans tell a story
January 2023. Amazon warns employees: “Do not share confidential Amazon information with ChatGPT.” The internal memo describes engineers using the tool to debug code and notes that Amazon’s security team has identified “several instances” of proprietary source code appearing in ChatGPT output. The output matched Amazon’s internal code so closely that Amazon’s legal team concluded the tool had ingested it.
May 2023. Apple restricts ChatGPT, Copilot, and several other generative AI tools across internal networks. The restriction covers the company’s development environment, email systems, and messaging platforms. Apple’s memo cites “concerns about the protection of confidential and proprietary data.”
June 2023. Verizon blocks ChatGPT on corporate devices. The telecom giant joins a growing list of enterprises actively preventing employees from accessing external AI services on company infrastructure.
March 2024. Samsung lifts its ChatGPT ban and announces plans for an internal AI tool. The company’s CTO tells investors: “We cannot ban our engineers from using tools that double their productivity. We need to give them tools that do not leak.”
The pattern is consistent. Large enterprise discovers employees feeding sensitive data to public AI. Large enterprise bans public AI. Large enterprise discovers the ban does not work and the productivity cost is enormous. Large enterprise builds or buys a private alternative.
Small and mid-market firms skip straight from step one to the data breach notification they write months later.
What your support team pastes into ChatGPT right now
A customer emails your support desk. They include their full name, their policy number, their ID number, and a description of the claim they want to dispute. The description mentions a pre-existing condition they did not disclose on their application.
Your support agent opens the email. They copy the entire thread. They paste it into ChatGPT with the prompt: “Draft a professional response. Deny the claim. Cite section 14.2 of the policy. Do not mention the pre-existing condition explicitly but reference the non-disclosure clause.”
ChatGPT writes the response. The customer receives a professionally worded denial letter. OpenAI receives the customer’s full name, policy number, ID number, medical history, and the agent’s instruction to deny the claim without mentioning the reason.
The agent did not violate company policy. Your company has no policy covering this scenario. You were focused on the data warehouse migration and the CRM rollout. The compliance review for AI tools was scheduled for Q3.
Where the data actually goes
When you paste text into ChatGPT, the data travels through an encrypted connection to OpenAI’s servers. OpenAI’s privacy policy grants them a license to use your content “to provide, maintain, and improve our services.” The term “improve” does not carry a statutory definition.
OpenAI states they do not train on data from API customers, ChatGPT Enterprise, or ChatGPT Teams accounts where the admin has opted out of training. They do train on conversations from free ChatGPT users and Plus subscribers who have not opted out.
Here is the path for a typical support agent. They create a personal ChatGPT account on their personal email. They use the free tier because it works and they have not been told to do otherwise. They paste customer data into the chat window. The data enters the training pipeline. It becomes part of the next model.
The agent does not know they opted in to training. They did not read the terms of service. They clicked “sign up with Google” and started typing. The “opt out of training” toggle lives in the settings panel beside the memory toggle. Neither toggle has been opened.
The deletion that never happens
Delete your ChatGPT account. OpenAI’s data retention policy states that account data is deleted within 30 days. The conversations you had are removed from OpenAI’s active systems.
The model that trained on your conversations cannot be un-trained. The weights are fixed. Your conversations contributed to the probability distribution that determines how the model responds to every prompt from every user globally. The deletion applies to the chat logs. It does not and cannot apply to the trained artefact.
This is not a flaw in OpenAI’s deletion process. It is a property of how large language models work. Once a dataset enters the training pipeline and produces a model, the information in that dataset is distributed across billions of parameters. Extracting it requires a technique researchers call “machine unlearning,” which remains a research problem, not a product feature.
Google published a paper on machine unlearning in 2023. The paper demonstrates that you can approximate the removal of a single training example. It works on small models trained on small datasets. On GPT-4 scale models with internet-scale training corpora, the technique does not scale. Nobody has demonstrated it. Nobody has shipped it.
The practical outcome: your customer’s ID number, policy details, and medical history contributed to a model that will serve responses for the remainder of its deployment life. You cannot recall the data. The system that holds it does not belong to you.
The global regulatory picture
South Africa’s Information Regulator issued guidance on AI and automated decision-making in mid-2025. The guidance emphasises transparency, accountability, and the need for a data protection impact assessment before deploying AI tools that process personal information. It does not address the scenario where an employee voluntarily uploads customer data to a third-party AI service without the employer’s knowledge.
The EU AI Act classifies general-purpose AI systems as a regulated category. Providers must document training data sources, implement watermarking, and publish summaries of copyrighted material used in training. The obligations fall on the provider, not on the user who pastes data into the provider’s product.
This creates a liability gap. The provider is responsible for their training practices. The user is responsible for the data they upload. The upload happens without the user’s employer knowing it happened. The employer bears the liability for the data breach under POPIA, GDPR, CCPA, or whichever privacy regime covers the affected data subjects. The employer did not authorize the upload, did not benefit from it, and cannot prove it occurred because it happened on an employee’s personal device on a personal account on a cellular connection.
How the Information Regulator investigates a complaint
A POPIA complaint arrives. A data subject alleges their personal information was processed without consent. The Regulator issues a notice under Section 91, requesting information from the responsible party.
Your response requires you to identify all processing activities involving the data subject’s information. You query your HR system, your CRM, your finance platform, your marketing automation, your cloud infrastructure. You report the systems you control.
The Regulator’s investigator asks: “Do your employees use any AI tools in the course of their duties?”
You say no because you banned ChatGPT on corporate devices six months ago. The investigator does not ask about personal devices. The prohibition on corporate devices was never tested against the reality of personal phones and cellular data. Your risk register does not reflect the gap. Your compliance certificate was signed while the gap existed.
This is not a hypothetical investigation scenario. The Regulator is resourced, active, and increasingly interested in automated processing. The first enforcement action will establish precedent. The firm it targets will be the one that could not answer the simple question: “Where does the data go?”
What stops the leak
You need three things. Not policies. Policies do not stop a support agent with a deadline and a customer on hold.
You need models hosted inside your network. The support agent opens the same interface they know, connected to a language model running on your infrastructure. The data never leaves your custody. The network path does not cross the public internet.
You need automatic filtering. Before a prompt reaches the model, a filter scans for PII: ID numbers, policy numbers, medical terms, financial account details. The filter redacts what it finds. The redaction is logged. The log generates the evidence you hand to the regulator.
You need an audit trail. Every prompt, every response, every redaction, every approval: hashed, timestamped, and stored in a format you can query within sixty seconds of receiving a Section 91 notice. “Show us what the AI processed” becomes a database query. Not a panic.
Private AI puts the models inside your network. The Airlock strips PII before prompts reach the model. Every interaction is logged. When the regulator asks, you answer with a report. Not a prayer.