The prompt that builds a manipulation engine
A marketer opens ChatGPT. They paste a year of chat transcripts between a customer and their support team. Then they type:
“Based on this customer’s communication style, personality, objections, values, and decision patterns, write a detailed persuasion profile. Include: what motivates them, what scares them, the language patterns they respond to, the objections they raise and how to overcome each one, the exact phrasing of a follow-up message most likely to convert them, and the time of day and channel on which they are most receptive.”
ChatGPT returns a 900-word psychological profile. It identifies the customer’s linguistic markers: deference to authority figures, a pattern of deferring decisions to a spouse who is not in the conversation, and a specific phrase that appears whenever the customer is about to buy. It recommends wording that invokes urgency without triggering the spouse-deferral pattern. It suggests Tuesday afternoons because the customer’s response latency drops by 40 percent at that time.
This is not hypothetical. I ran this prompt. The output was granular, specific, and actionable. The customer in question had no idea their support conversations were being used to build a persuasion model. They consented to receiving customer support. They did not consent to being profiled by an AI system their bank does not control.
The prompt that makes you the target
The manipulation engine cuts both ways. An AI vendor’s customer success team runs the same prompt on you. They paste your company’s RFQ responses, your support tickets with their platform, your negotiation emails, and the transcript of a demo call where your CTO asked about a specific integration. Then they ask the model:
“How do I sell to this person? Build me a sales strategy.”
The model returns the avatar of your purchasing psychology. It knows your CTO cares about uptime SLAs more than price. It knows your procurement team stalls when the contract exceeds one year. It knows you respond to a deadline (“pricing goes up next quarter”) and resist feature comparisons because your head of engineering distrusts vendor benchmarks.
The vendor did not ask you for permission to profile you. They used the tool. The tool profiled you. The profile informed their negotiation. You sat across the table from someone who had read a machine-generated analysis of your decision-making patterns and you did not know.
The real-world examples that are not making headlines
Several marketing agencies offer “AI-powered persona development” as a service. The workflow is consistent. The agency collects a client’s customer data: support transcripts, NPS survey responses, abandoned cart logs, email engagement histories, call recordings. They run the data through a language model. The model produces a persona document that would have taken a research team six weeks to compile.
The agency calls it “accelerated research.” The regulator calls it an unauthorised processing activity if the customer did not consent to their data being used for automated profiling.
POPIA Section 26 prohibits automated decision-making that produces legal effects or significantly affects the data subject without human intervention. GDPR Article 22 does the same. The customer whose support transcripts fed a persuasion profile never received a notice that their data was being used this way. They never provided consent beyond the generic “we use your data to improve our services” clause in a terms of service they scrolled past in 2022.
A South African insurance company ran this workflow in early 2025. A claims handler pasted a customer’s claim history, medical questionnaire responses, and policy adjustment requests into a public AI tool. The prompt asked the model to “assess the claimant’s likelihood of pursuing legal action if we reject the claim.” The model returned a risk assessment with supporting evidence drawn from the customer’s communication patterns. The claims handler used the assessment to calibrate the rejection letter.
The Information Regulator has not received a complaint about this case because the customer does not know it happened. The claims handler’s manager does not know it happened. The compliance officer who signed off on the company’s AI usage policy does not know the policy has no controls preventing a claims adjuster from pasting a customer’s medical history into a chatbot.
The psychology of the prompt author
The marketer who builds a persuasion profile of a customer is not evil. They are doing their job with a new tool that makes their job faster. The prompt felt like a shortcut, not a violation. The customer’s data was already inside the company’s systems. The marketer accessed it through approved channels. Moving it from a Zendesk export to a ChatGPT window felt like a drag-and-drop operation, not a data transfer.
This is the problem. Privacy controls built around system access do not account for the tool that sits outside the systems. The marketer has permission to view customer chat transcripts. They use that permission. Then they feed the transcript to a third-party AI service that their company’s IT department has not approved, has not configured, and cannot audit.
The violation occurs in the gap between what the employee can access and what the organization can control. The gap is widening every quarter as AI tools multiply.
What a compliance audit actually asks you
A POPIA subject access request arrives. A customer asks: “What personal information do you hold about me, and how have you used it?”
You query your CRM. You query your support platform. You query your marketing automation tool. You do not query ChatGPT because it is not your system.
The customer’s data sits in a conversation history on OpenAI’s servers. It generated a persuasion profile. It informed a pricing decision. It influenced the wording of a collection letter. None of this appears in your response to the access request because you did not know it existed and the tool that holds it does not belong to you.
The customer escalates to the Information Regulator. The Regulator asks you to demonstrate that you have identified all processing activities involving the customer’s data. You produce your data map. The Regulator asks whether your data map includes third-party AI services accessed by employees through personal accounts on personal devices. You ask for time to consult legal counsel. The silence is the answer.
The only defence
You cannot stop employees from using AI. You tried banning it. They brought their phones. You tried monitoring network traffic. They used mobile data. You tried an acceptable use policy. They signed it and continued pasting.
The defence is structural. Run the models yourself. Put them inside your network, behind the same access controls your CRM uses. Strip PII before prompts reach the model. Log every interaction. Build the audit trail that answers the subject access request without a phone call to your outside counsel.
The question a regulator asks is not “did you have a policy?” The question is “can you prove what happened?” Most firms answer the first question with a PDF. They fail the second question in silence.
We build Private AI for this reason. Models hosted inside your network. The Airlock strips PII from prompts, uploads, and database queries before anything reaches the language model. Every interaction is logged, hashed, and auditable. When a regulator asks what the AI did, you answer with a report instead of a phone call.