AI Privacy: Controlling Identifiability, Memorization, and Unauthorized Use Along Data Flows

To determine whether a piece of data involves privacy, the first step is not to ask “does it contain names and phone numbers,” but to ask “can this data be linked to a specific person.” Deleting direct identifiers—name, ID number, phone number—does not automatically make the data anonymous. Direct identifiers are only the most visible layer: after they are removed, the data may still point back to a person through other fields.

Age, postal code, rare disease, timestamp, device fingerprint, and conversation content—each alone is not enough to locate a person, but combined they form quasi-identifiers and, when linked with external data, are enough to re-identify an individual. For example, a hospital's treatment record has the name removed but still retains “82 years old, lives in a postal code area, has a certain rare disease”; as long as this record is linked with external public information (such as population registers, social media profiles), there may be only one individual who satisfies all these conditions, and re-identification is completed at that step. The key is not whether a field itself is “sensitive,” but whether it can be combined with other known information to narrow the candidate set.

Beyond re-identification, there are two other types of privacy risk. One is sensitive inference: the model does not need to obtain the sensitive field itself; it can infer sensitive attributes for the user from seemingly neutral inputs, such as inferring health status from purchase records or economic status from browsing behavior. The other is use beyond the stated purpose: data is used for purposes outside the scope agreed at collection, or accessed by people who have no business need. Neither of these two types of risk depends on “finding the name,” yet both directly harm the data subject.

Therefore, what privacy identification really needs to do is treat “can it be linked to a person” as a question that must be checked item by item, rather than ending after checking identifiers once. Its input has four categories: the data fields themselves, external information that can be linked to them, the purpose of processing the data, and who can access the data. The corresponding output is four determinations: direct identifiers, quasi-identifiers, sensitive attribute inference risk, and use-beyond-purpose risk. Deleting the name only removes the most obvious clue; age, postal code, rare disease, and time can still combine to achieve re-identification, so the output must cover all risk surfaces beyond identifiers.

To support this check, the asset inventory cannot only record “what data exists”; it must also record for each asset the data subject, sensitivity, purpose, legal or contractual basis, retention period, and acceptable recipients. Among these, the “de-identified” label requires particular caution: de-identification is not a permanent property of the data; it only holds relative to “what the assumed attacker knows” and “what method was used to verify it.” Claiming that a dataset has been de-identified requires also stating the attacker's knowledge boundary and the verification method actually performed; with an attacker who possesses more external information, the same dataset may no longer be safe.

Privacy threats do not appear only at the model's output. Even if the model never directly prints any personal information, the system may still harm privacy elsewhere. To systematically identify these risks, examine each path along which data actually flows, rather than focusing only on the model itself.

Threats are distributed across three paths, and each path has its own typical risks and requires corresponding evidence for verification.

Risks on the training path center on the model's memorization of data and dataset provenance. A model may memorize rare sequences and, after training, be induced to reproduce them through extraction tests; membership inference attacks can determine whether a particular record was in the training set, thereby leaking the very fact that 'someone is in the dataset'; dataset provenance overreach means data is used for training purposes beyond the authorization under which it was collected. Verifying these risks requires extraction tests, membership attacks, and data lineage records as evidence.

Risks on the inference path center on data crossing boundaries and tenant isolation. Prompts and documents retrieved by Retrieval-Augmented Generation (RAG) may be sent to model providers outside the authorized scope; in multi-tenant scenarios, retrieval results may leak across tenants. Evidence comes from network flow logs, tenant-level ACL configurations, and supplier processing records.

Risks on the operations path come from new copies of data created in logs, tracing, human annotation, and backups. Each copy is a new exposure surface and requires retention policies, access logs, and deletion receipts to constrain and verify.

Finally, the output path: the model may output real sensitive information, or infer sensitive attributes from non-sensitive inputs. Verification methods are canary tests and field-by-field authorization: the former plants recognizable markers in the data and monitors whether they appear in model output; the latter confirms field by field whether the information is authorized to appear in the output.

The four paths together make one point: the inputs to threat modeling are the data copies and data subjects at each stage—training, inference, operations, and output—and the outputs are specific attack paths such as memorization extraction, membership inference, cross-tenant retrieval, log leakage, or sensitive attribute inference. Inspection must follow the actual flow of data, covering suppliers, caches, annotation, and backups, and record attack preconditions and verification evidence for each attack path separately, rather than generically claiming 'secure.'

This framework also identifies two common protection blind spots. A conclusion of 'model-layer security' cannot cover copying that occurs in logs and the supply chain; likewise, output filtering cannot retract prompts that have already been sent to third parties. The fact that the model did not directly print a name does not mean upstream data copying and use beyond the original purpose did not occur.

Consider a real customer-service scenario: a user submits a ticket containing an ID number and medical history, the system sends it to an external model to generate a summary, and then writes the summary back to the ticket. The question is, which data copies on this pipeline are most easily missed by an asset inventory? This example compresses the three threat paths into a concrete request chain, and each step corresponds to a privacy decision that needs to be actively handled.

First, the browser submits the original ticket. The access layer first separates the account identifier from the body by purpose: from this point on, the identifiers needed for authentication and the body needed for summarization follow different processing channels. Second, direct identifiers are identified within the local domain and replaced with short-term tokens; the identifier-to-token mapping table remains in the controlled domain and is never sent outside. In this way, any subsequent step receives tokens rather than the ID number itself. Third, the retrieval layer takes only the minimum passage that the tenant and the agent are authorized to view, rather than pushing the entire ticket to the model—the retrieval scope itself becomes an authorization check.

Fourth, the payload actually sent to the model must record the field list, data region, retention period, and training-use agreement; this metadata goes into the audit record, while the body is not written to general-purpose telemetry. Fifth, after the model returns the summary, before backfilling tokens, perform another authorization and sensitive-attribute check: confirm that the agent still has permission to view the corresponding content, and confirm that the output does not reintroduce sensitive attributes that should have been replaced. Sixth, the tracking logs save only the hashed request ID, model version, and policy decision result; body samples are saved only on demand in a restricted fault library, isolated from general-purpose telemetry. Seventh, when a user initiates a deletion request, the deletion action propagates station by station along the ticket, cache, vector index, logs, annotation set, and backup inventory, and each station generates a deletion receipt.

Only when these seven steps are connected do they form a complete causal chain. If the system only filters the output at the sixth step, then the outbound data transfer, cross-tenant leakage, and use beyond the stated purpose that occurred in the first five steps have not been addressed: the identifiers have already left the controlled domain along with the body in earlier steps, and output filtering can neither retract prompts that have already been sent to third parties nor make passages that were retrieved without authorization become invisible again.

From an input-output perspective, the inputs to this case are the ticket containing an ID number and medical history, tenant permissions, the external model, and the deletion request; the outputs are the minimized payload, controlled token mapping, authorization decision result, audit metadata, and station-by-station deletion receipts. The overall processing sequence can be summarized as follows: first replace identifiers in the local domain, then retrieve only the passages that are authorized for viewing, record fields and retention agreements before sending outside, authorize again before backfilling, and propagate deletion along the cache, vector index, logs, annotation set, and backups. Filtering only at the output cannot recover data that has already crossed the boundary—this is why privacy is treated as a full-chain property rather than an end-of-pipe filter.

Adding “a little noise” to statistical results sounds simple, but the noise cannot be added arbitrarily, because privacy leakage accumulates with the number of queries. Differential privacy formalizes this intuition: it compares two neighboring datasets—differing in only one person—and requires that for any output event, the ratio of their probabilities does not exceed e^ε, with an additional small-probability exception δ to handle extreme cases. Intuitively, whether a person's record is in the dataset or not, the output distribution observed by an outside observer is nearly identical; the smaller ε is, the harder it is for an attacker to determine whether someone participated, but statistical utility is usually also lower, because stronger protection means more noise.

The core problem that the concept of “budget” solves is the cumulative leakage caused by repeated releases. Suppose three queries are executed sequentially on the same dataset, consuming ε=0.4, 0.3, and 0.2 respectively. According to the basic composition theorem, the total privacy budget is at most 0.9, rather than still being equal to 0.4 of some single query; rerunning the same query every day will continue to accumulate the budget. Therefore the budget must be bound to a specific dataset, subject scope, and time window, with a budget accountant keeping a continuous ledger and rejecting subsequent queries when the limit is exceeded.

From an input-output perspective, the differential privacy budget mechanism receives neighboring datasets, the budget εi for each query, the allowed small-probability exception δ, and a time range; it outputs the combined total budget εtotal and whether further queries are permitted. Under basic composition, the εi values on the same dataset are directly summed: 0.4 + 0.3 + 0.2 = 0.9.

Finally, it is necessary to clarify its boundaries. Differential privacy protects “participation”—limiting the marginal impact of a single record on the output distribution, so that an attacker cannot reliably determine whether someone is in the dataset. It does not automatically guarantee access authorization, does not ensure data accuracy, and does not ensure that the output is non-discriminatory; these are independent requirements that must be addressed separately by other mechanisms.

How can you spot at a glance a gap like “deletion only covered the primary database, while the logs and vector database were not deleted”? The answer is to draw a trust boundary diagram along the data flow and check station by station.

The diagram shows the complete flow of personal data from collection, through minimization, authorized retrieval, model provider, logs, and output stations, and marks the return path of deletion receipts. Its input is all nodes and copies of the data from the original ticket to the controlled domain, model side, output tracing, and deletion chain; its output is the controls each station must have whenever it crosses a boundary: minimization, authorization, retention agreement, and receipt.

The way to read the diagram is to follow the arrows and check three things station by station: what fields this station holds, who is responsible for it, and whether deletion status can be confirmed. Each time you cross a trust boundary, a set of “reduce data + leave evidence” controls should appear: minimization before sending to the model provider, authorization for retrieval results, retention agreement for external transmission, and receipt for deletion. Any node with no owner or no receipt is where the gap lies. Endpoint filtering can only affect the output station; it cannot repair copies that have already crossed the boundary upstream, so verification must go from collection all the way to the deletion chain, not just looking at the last station.

The boundaries of this diagram's usefulness are equally clear: it helps discover breakpoints in the data flow, but cannot replace actual network logs and deletion replay verification. The controls drawn on the diagram are not equal to controls actually executed in reality; gaps must ultimately be confirmed with replayable evidence.

Business stakeholders often say a field “might be useful later” and hope to extend retention on that basis. But this cannot justify unlimited retention, because the core of minimization is not “what to delete now,” but “prove item by item which data is necessary for the current task.” It simultaneously constrains four dimensions: fields, precision, number of individuals, and retention time.

Minimization on fields and precision follows a set of substitution rules: use age ranges instead of birthdays when possible, use aggregate statistics instead of transmitting person-level records when possible, compute locally instead of uploading when possible, and use short-term caching instead of writing permanent logs when possible. Each substitution reduces the identifiability or exposure surface of the data.

Minimization on the number of individuals and time is bound to purpose. Training, debugging, product analytics, and security investigations are different purposes and should be separately authorized and separately retained, rather than sharing a generic “business need.” A field retained under authorization for training cannot automatically be used for security investigations, and vice versa.

Verifying necessity cannot rely on verbal claims; before launch, use field ablation experiments to check: delete a field and see how much task quality actually drops. If the benefit is small and the identifiability risk is large, it should not be collected. This experiment is reproducible evidence, turning “necessary” from an adjective into a measurable conclusion.

From an input-output perspective, data minimization takes as input the task purpose, candidate fields, precision, number of data subjects, retention duration, and utility experiments, and outputs decisions to retain, generalize, compute locally, aggregate, or delete. For future uses, the default stance is denial: any newly added downstream use must reassess necessity and obtain new authorization, rather than inheriting old consent. “Might be useful later” therefore can never constitute a current collection justification.

The fact that a user can ask a model questions does not mean the model can read all documents on the user's behalf. This principle defines where access control applies in Retrieval-Augmented Generation (RAG) and AI Agent scenarios: authorization decisions must happen before the moment data is retrieved, not after results are generated.

RAG permission checks must be completed before retrieval, covering four levels: subject, tenant, document, and field. The correct order is to first determine which documents and fields the subject can access within that tenant, then retrieve within the allowed scope; if instead you retrieve from the entire database first and then mask after generation, unauthorized content has already been read and placed into context, and masking merely conceals it. The same constraint extends to embedding vectors and caches: they also hold document content and must be isolated by tenant, otherwise one tenant's query may hit another tenant's vectors.

When an Agent calls a tool, the agent holds short-term, narrowly scoped identity credentials, with permissions no greater than what the current session requires. Every write operation must be revalidated—not whether “the session is legitimate,” but whether “the authorization for this specific action under the current authoritative state still holds.” In this way, even if prompt injection steers a legitimate session toward malicious instructions, unauthorized write actions will be blocked at the authorization layer, and prompt injection cannot expand a legitimate session into an unauthorized channel.

For people, permissions must also be subdivided by role. Administrators, developers, and human reviewers should each have defined purposes and visible fields; sensitive access requires justification, approval, and immutable logs. When assessing risk, sampling “who looked at what, when, and for what purpose” comes closer to real risk than merely checking role names—role names only indicate preset capabilities, while access records show actual behavior.

From an input-output perspective, access control receives the current subject, tenant, resource, field, action, and short-term credentials, and outputs allowed retrieval candidates, tool calls, and immutable audit records. The execution order is: RAG filters the allowed scope before retrieval, reauthorizes returns and write operations according to the authoritative state, and caches and embeddings are also isolated by tenant.

Deleting a row from a database does not mean you can immediately claim the data has been “forgotten.” A row of data has usually already produced many copies in the system, and the difficulty and verifiability of deletion depend on which station the data has reached. Therefore, the first thing in deletion engineering is to establish a traceable lineage: record all derived relationships from the original object to the cleaning set, training slices, embeddings, caches, annotation tasks, checkpoints, and backups.

Along this lineage, each station is handled differently. For storage that can be deleted directly, a completion receipt must be returned after deletion as verifiable evidence. Backups usually cannot be rewritten immediately; the correct approach is to freeze them: specify that once a backup is restored, deletion must be performed again, and state the duration of this window, rather than assuming by default that backups will never be restored. The hardest station is model weights: if the data has already entered training, proving that the influence of deleting a single sample on the weights has been completely eliminated is technically very difficult. Feasible paths include retraining, machine unlearning, or risk isolation, but each one requires independent verification.

For users, the description of deletion status must be truthfully distinguished by station: deleted, queued for deletion, only stopped from further use, and the parts that cannot yet be technically verified. Describing “disappearance from the product interface” as complete removal across the entire chain conceals the fact that other stations still hold copies.

From an input/output perspective, the deletion lineage receives the original object and its derived relationships in the cleaning set, training slices, embeddings, caches, annotations, checkpoints, and backups, and outputs the deletion status of each station: deleted, queued for deletion, stopped from use, or temporarily unverifiable. Disappearance from the interface does not equal forgetting across the entire chain; the gap between the two is precisely the part that the lineage must fill in station by station.

Randomly checking outputs and not seeing phone numbers cannot prove the system is secure. This kind of check assumes an attacker who “only does random scanning,” while real attackers use auxiliary information: they may know part of a name, message templates, or document structure, and use that to steer the model in a targeted way. Therefore the first principle of privacy evaluation is to simulate an attacker with auxiliary information.

Evaluation should establish a set of targeted tests: rare canaries—inserting markers that almost never appear naturally and observing whether they are output; training data extraction; membership inference; attribute inference; cross-tenant retrieval; log access checks; and deletion replay, verifying whether deletion requests actually take effect at each stage. Each type of test corresponds to a specific attack path, not a generic “security scan.”

Result reports should be broken down by data sensitivity: attack success rate, number of people exposed, number of fields exposed per person, access duration, and deletion completion rate. These metrics answer different questions; mixing them into one aggregate only dilutes each other. Precision and recall for PII in output are only one layer: they measure the filter's performance on direct identifiers, but overly aggressive filtering can wrongly block normal addresses or medical content, harming business utility and increasing manual recovery costs, so false-blocking rate and utility must also be measured simultaneously.

From an input-output perspective, privacy evaluation takes sensitive slices, attacker auxiliary knowledge, and tests for training extraction, membership and attribute inference, cross-tenant access, and deletion replay, and outputs attack success rate, number of people and fields exposed, access duration, false blocking cases, and deletion completion rate. No single “privacy score” should obscure high-impact tail events: rare but severe leaks must be reported separately, because they are what really constitutes privacy risk.

Saying "not used for training" in a contract is one thing; whether it is actually honored at runtime is another. The goal of supplier verification is to make the two consistent, and to prove that consistency with checkable evidence.

First, confirm that the contract itself covers the key points: who the sub-processors are, in which regions processing takes place, whether data in transit and at rest is encrypted, who controls the keys, how long data is retained by default, what abuse monitoring retains, how incidents are notified, whether a deletion interface exists, and whether data is portable upon exit. Second, perform runtime checks: use network proxies, audit logs, and sampled requests to check whether the fields actually sent match what is declared. This step cannot only look at interfaces you explicitly call; SDK telemetry may quietly send extra fields, and must be included in the scope of observation.

When the supplier's model or terms change, the previous verification conclusions no longer hold, and reassessment must be triggered. If the supplier cannot provide the required evidence, there are only two options: restrict the categories of data sent, or switch to a local solution.

Self-hosting is often treated as a "data doesn't leave the building, so it's compliant" solution, but it only reduces external transmission, while shifting patch management, backups, internal abuse prevention, and key management entirely onto your own responsibility. Self-hosting does not equal automatic compliance; it merely shifts risk from a third party to internal.

From an input-output perspective, supplier verification takes as input contract terms, sub-processors, regions, fields, retention periods, training-use agreements, encryption keys, and deletion interfaces, and outputs evidence of consistency between declarations and actual network flows. The core action is to use proxies and audit logs to sample-check the fields the SDK actually sends; model or terms changes trigger reassessment; when evidence cannot be provided, restrict the data or replace the solution.

If requests are not logged at all, how can the scope of impact be determined after an incident? Privacy and observability are not an either/or choice: observability does not need to copy all content; auditing needs evidence of “who did what,” not storing all the body text. The approach is layered evidence retention.

Regular telemetry stores request ID, policy version, model, latency, token count, and risk labels, and does not store body text. Only failure samples that meet explicit trigger conditions enter an isolated fault repository; before entry, field minimization is performed, access requires approval, and a short retention period is set. In highly sensitive scenarios, local statistics, aggregated metrics, and controlled replay can replace raw-text logs—all three can support problem localization without creating new full-text copies. If full text must be preserved, the debugging purpose, number of people involved, retention period, environment, and deletion mechanism must be specified as independent controls, rather than as the default behavior of logging.

The opposite of layered evidence retention is treating common control statements as complete privacy guarantees. Only by placing statements such as “name removal, encryption, no training, local deployment, output filtering, deletion of the primary database” back onto the data flow one by one can you see the risks each statement actually covers and the parts still uncovered. When checking, ask in order: purpose, linkability, access, derived copies, and deletion evidence. Take encryption as an example: it protects only content in transit and at rest, and does not restrict misuse of data by legitimate accounts after decryption. The gap between statements and their scope of coverage is where residual risk lies.

The input to this kind of analysis is various privacy statements, and the output is the risks the statement actually covers and the data flows still uncovered. Its conclusions are used to correct the scope of controls, but do not replace specific legal judgment, nor do they replace attack verification of the complete system.

Connecting the previous links into a causal chain from problem to practice can be summarized as eight steps. Step one: define data subjects, processing purposes, and unacceptable impacts—without this step, none of the later controls has a criterion. Step two: enumerate direct identifiers, quasi-identifiers, and the knowledge of hypothetical attackers to make “linkability” concrete. Step three: draw all data copies and trust boundaries from collection to deletion; this step produces the diagram for station-by-station verification. Step four: perform minimization station by station and bind permissions to tenants. Step five: select controls for data that still must be retained: de-identification, aggregation, or differential privacy. Step six: verify that vendor contracts and actual network flows are consistent. Step seven: accept the entire system by attack simulation and deletion replay. Step eight: after go-live, continuously monitor purpose drift and respond to data subject rights at any time.

Every step in the chain requires observable evidence; otherwise it is only a process description. The verification layer specifies for each link what to fix and what to observe: for input, fix the same batch of samples, preprocessing, and permission boundaries; observe input hashes, slice labels, and rejection reasons; for mechanism, change only one core variable and lock all other configurations; observe key intermediate states and the position where the first deviation from expectation occurs; for output, fix the same acceptance rules and resource budget; observe stratified differences in quality, cost, latency, and failure rate; for falsification, retain a control group with the target mechanism disabled; observe whether benefits reproduce stably across samples and random seeds. Fixed items ensure the difference comes from the verified mechanism itself, and observed items make conclusions reviewable.