AI Content Detection and Provenance: Distinguishing Statistical Guesses, Watermark Signals, and Signature Origins

The first problem AI content detection must solve is not “how to judge”, but “what exactly is being judged”. A photo taken by a real camera, after being retouched by a generative tool, should it be labeled AI or non-AI? This seemingly simple question exposes a fundamental flaw in binary classification labels: it forcibly compresses multiple non-equivalent propositions into a single “yes/no”.

There are multiple propositions that can actually be tested, and they are independent of each other. One can ask “whether this image was fully synthesized by a certain generator”, which examines the generation source; one can ask “whether the file embeds a watermark from a certain vendor”, which examines a verifiable signal; one can ask “whether the file carries a trusted signature”, which examines a provenance declaration; one can ask “which regions have been edited”, which examines local edit history; and one can ask “whether the event depicted in the image actually happened”, which examines factual truth. Generation source, edit history, and factual truth are three different things: a piece of text can be entirely written by a human yet state false facts, or be generated by a model yet happen to describe a true event; a photo can be genuinely captured but partially completed by generative inpainting. No single true/false conclusion can answer all these propositions simultaneously.

Therefore, a detection product must first declare what its detection target is, which models and modalities it supports, which point in time the result corresponds to, and what it explicitly does not cover. The detector outputs a probabilistic judgment about a precisely defined proposition, not a factual verdict on the “authenticity” of the entire content. Treating a vague binary classification as an authoritative conclusion causes damage in two directions: on one hand, it misclassifies works by real authors that have undergone normal editing as machine-generated, harming their credibility; on the other hand, it gives malicious content a laundering channel of “passed detection, therefore trustworthy”. Defining the proposition first is the prerequisite for all subsequent mechanisms to be correctly interpreted.

A statistical detector's core assumption is that generator outputs and human outputs leave different distributional traces in some observable space, and the classifier fits these differences to determine which source a sample resembles more. Its input is the raw bytes, pixels, or token sequence of the content under inspection, and its output is usually a confidence score between 0 and 1, representing a probability estimate that it is "machine-generated." Only after being compared with a threshold does this score become a binary classification label.

The problem is that the differences a classifier actually learns do not necessarily correspond to "the stable nature of AI." It may exploit high-frequency distributions in the frequency spectrum, texture statistics, skew in token probabilities, compression traces, or correlations between metadata—any one of these features may be merely an incidental byproduct of a specific generator, a particular sampling parameter, or a particular post-processing pipeline in the training set. As soon as the generator's architecture and weights change, these traces change accordingly. A single ordinary JPEG re-encoding is enough to erase the spectral and compression features that the classifier relies on, and a newly released generator may fall outside the training distribution, causing the detector's high confidence to fail instantly.

This gives rise to a causal chain: any change in the models, samplers, languages, devices, or post-processing methods in the training data causes distribution shift—that is, the distribution the detector saw during training no longer overlaps with the distribution it faces at deployment. The greater the distribution shift, the less reliable the superficial features that the classifier relies on, and the lower the detection rate. Attackers can also treat the detector's score as an optimization target, specifically searching for perturbations that lower the score to create adversarial examples and achieve evasion. This further reveals that the classifier has learned only fragile surface correlations, not deep causal signals.

Therefore, evaluating a statistical detector cannot only look at how accurate it is on data from the same distribution as its training data. It must be tested with unseen models, real editing chains, and data generated after the training time; otherwise the detector is likely only recognizing "which dataset the sample comes from" rather than recognizing "that the sample was machine-generated." Any detection rate that cannot remain stable under such out-of-time and out-of-model conditions should not be interpreted as reliable proof of the generation source.

Even if a detector’s recall and precision appear good, after deployment its positive judgments may still be mostly wrong. The reason is that the “positive” returned by the detector is only a conditional probability, while what users actually care about is “how likely is this positive to really be AI content”; the latter must be calculated by incorporating the base rate—the proportion of AI content in the scenario.

Consider a platform in which only 1% of content is generated by AI. The detector’s true positive rate TPR = 90%, meaning that out of every 100 true AI items it detects 90; false positive rate FPR = 5%, meaning that out of every 100 non-AI items it falsely flags 5. Among 10,000 items, 100 are AI, of which 90 are correctly detected; among the remaining 9,900 non-AI items, 495 are falsely flagged. Thus the total positives are 90 + 495 = 585, of which only 90 are truly AI. Positive predictive value PPV = 90 / 585 ≈ 15.4%, meaning that only about one-sixth of items labeled positive are actually AI.

This can be written in Bayesian form: P(AI | +) = (TPR × base rate) / (TPR × base rate + FPR × (1 − base rate)); substituting gives 0.9 × 0.01 / (0.9 × 0.01 + 0.05 × 0.99) ≈ 15.4%. The second term in the denominator, 0.05 × 0.99, is the total number of false positives, which comes from multiplying a large amount of non-AI content by a not especially high false positive rate. As long as the base rate of AI content is low enough, the size of the non-AI group will amplify the number of false positives to overwhelm the true positives.

A direct implication of this result is that overall accuracy is a misleading metric. A detector with “99% accuracy” on a platform with a 1% base rate will likewise produce many false positives, because accuracy mainly reflects correct classification of the vast majority of negative examples, not the credibility of positives. Choosing the threshold must return to the specific scenario: what is the base rate, and what are the respective costs of misjudging a real author and letting a piece of malicious content through. Only when the action consequences brought by a positive match the true credibility represented by PPV is the detector’s output meaningful.

When a detector gives a social media image a score of “82% AI,” what the editor should do is not to accept that number directly, but to treat it as a technical clue to be verified and place it, along with other evidence, in a complete verification chain.

The first step is to save the original file and its hash, and confirm whether the image underwent platform transcoding before reaching the detector. Compression and transcoding alter spectral and compression features, causing the statistical detector's input to deviate from its training conditions, so you must first record the file's actual state before interpreting any score.

The second step is to check source credentials. If the file claims to carry a signature, verify that the signature itself, the issuer identity, the issuance time, and the stated edit chain are consistent. Missing credentials should be recorded only as “no credentials”; it cannot be used as evidence that the source is suspicious, nor as evidence that the content is true.

The third step is to check for invisible robust watermarks. If the detection tool supports a particular watermarking scheme, record the detection version used and the detected signal strength; the watermark's strength directly affects whether the signal can be destroyed by post-processing, and also whether “not detected” means “no watermark” or “insufficient detection capability.”

The fourth step is to run multiple applicable forensic classifiers or generative classifiers, and confirm that each classifier's training covered that modality and that compression level. A single classifier may be effective only for specific generators; multiple independent classifiers can reduce the risk of single-model failure.

The fifth step is to conduct factual verification: use reverse image search to find earlier appearances of the image, contact the publisher directly to verify, compare the weather and geographic features at the shooting location, and check the coherence of the original photo sequence. These steps answer the proposition “did the event depicted in the image actually happen,” which is different from the proposition “was the content machine-generated” that technical signals address.

The whole process is organized by recording technical signals, source evidence, and content facts in three columns, rather than voting various scores into a single conclusion. An 82% statistical score, a valid signature, and a successful reverse image search cannot cancel each other out, nor can they compensate for one another. High-impact conclusions must be reviewed by a human, and the publisher must be provided with the original file and an appeal channel. Source credentials help answer “where did this image come from,” but “did it happen” must still be answered by fact-checking.

After a piece of content to be checked enters the verification process, it simultaneously goes through three mutually independent evidence chains; each chain answers a different question, and no single chain can replace the other two.

The first is the statistical classification chain. The content is fed into a statistical classifier, which outputs a probability estimate about whether the generation source 'looks machine-like.' It answers 'how close the distribution of these pixels or tokens is to a particular generative model,' but it knows nothing about whether a watermark is embedded or whether there is a signature, nor can it answer whether the facts are true or false.

The second is the watermark verification chain. The content is fed into a watermark verifier, which detects whether it carries a signal embedded by a specific scheme. It answers 'whether a verifiable mark was left in the generation process or output,' along with the strength and version of the signal. The result of watermark verification is only related to the embedded signal; it has nothing to do with statistical distribution, nor with signatures.

The third is the signature provenance verification chain. The content is fed into a signature provenance verifier, which checks whether the file carries the issuer's digital signature, issuance time, and the claimed edit chain. It answers 'who declared what this content went through and when,' proving the integrity of the declaration, not the honesty of the declarant, nor the truthfulness of the content.

The raw outputs of these three chains have different semantics, and are ultimately summarized by human fact-checking. The summarization action is not to weighted-sum the three types of scores into a single number, but to judge what each piece of evidence can support or rule out while preserving their respective semantics. The core relationship in the figure is: classification, embedded signal, and signature declaration correspond respectively to three different propositions—statistical features of the generation source, verifiable marks in the generation process, and integrity of the source declaration.

From this we get a direct inference: 'no watermark detected' cannot be interpreted as 'human-created.' The watermark chain only says 'the embedded signal of this scheme was not found within the detection capability range'; it neither excludes statistically machine-generated nor excludes the absence of a signature; and a negative from the statistical classifier likewise cannot prove 'no watermark' or 'has a signature.' The absence and presence of the three chains are independent of each other; only by recording them separately in the corresponding question columns can verification conclusions avoid being incorrectly merged.

Watermarking differs from the statistical detectors in the previous two chapters: it does not passively look for distribution traces left incidentally by generators, but actively embeds a key-related detectable signal during generation. Image watermarks can be encoded at the pixel level, in frequency-domain coefficients, or in latent variable space; text watermarks leave detectable statistical patterns by biasing token selection; audio and video embed markers in the time-frequency signal. The verifier uses the corresponding key or statistical detection method to determine whether the signal is present.

Watermark design must simultaneously satisfy two competing requirements. Robustness requires the signal to survive cropping, scaling, compression, and light editing; otherwise a single screenshot or platform transcode can invalidate the watermark. Imperceptibility requires the signal not to be noticed by human perception or common quality metrics; otherwise it will damage the value of the content itself. There is tension between the two: the stronger and more robust the signal is made, the more likely it is to leave perceptible artifacts in vision or hearing; the more imperceptible it is made, the more easily it is erased by post-processing.

Attackers' methods against watermarks are not complicated: regenerate the content, make heavy edits, stitch different segments, or directly estimate the watermark pattern and then try to remove it. More importantly, a watermark can only exist in the output of a cooperative generator; any content not integrated with the scheme has no signal from the outset. Therefore, “watermark detected” is strong evidence—it indicates that the content went through a generation process that embeds a watermark. But “watermark not detected” is weak evidence: it cannot distinguish between “not generated by a cooperative generator” and “the watermark has been destroyed or detection capability is insufficient.”

Precisely because of these boundaries, a watermark verification report must state the supported scheme version, false positive rate, targeted attack budget, and detected location information. Only when these conditions are given together can a watermark result be interpreted as clearly bounded evidence, rather than a general proof of authenticity.

The question that signature provenance mechanisms answer is narrower and more specific than watermarking: it does not determine who generated the content, but verifies whether the provenance claim attached to a piece of content is intact and has not been tampered with. Taking content provenance standards such as C2PA as an example, the signature manifest can record the tool used to capture or generate the content, the timestamp, every subsequent edit action, and the corresponding asset hash for each step. The verifier's job is to confirm that the manifest itself has not been modified and that the certificate chain that signed the manifest is trustworthy.

Passing verification means that "the file and its claimed edit history have not been tampered with, and the claim is signed by a certificate that can be traced back to a trusted root." It does not mean "the content of the claim is true." A trustworthy device can perfectly well capture a staged scene; an account holding a legitimate key may be abused; a technically trustworthy signer may itself be unreliable. The signature ensures the integrity of the claim, but it does not guarantee the honesty of the claimant, nor does it guarantee that the claimed event actually occurred.

The trustworthiness of a signature system therefore depends on several infrastructure components: how the trust root is established, how private keys are protected and revoked, how clocks ensure trustworthy time, and how the edit chain is displayed to users in a complete and understandable way. If any one of these components fails, the meaning of the signature is diminished.

Furthermore, signatures depend on metadata traveling with the file. Screenshots, transcoding, and re-encapsulation by most social platforms will discard this metadata, causing the signature chain to break during verification. The correct handling is to display this as "chain broken" rather than judging the content as "false." A broken chain means that the source cannot be verified through the signature, which is different from the content being false. Confusing the two will cause all genuine content that has been transcoded by platforms to be wrongly harmed.

In reality, the vast majority of content is neither purely generated nor purely real, but a mixture of the two. For a photo taken by a real person, where only the sky has been replaced with a generative tool, what should the entire file be labeled? If the system can only output two labels, “fully AI” or “fully real”, the answer to this question is wrong either way: labeling it “fully AI” denies that the vast majority of pixels come from a real camera, while labeling it “fully real” conceals the existence of generative editing.

A more accurate description is “contains generative editing”, and to record the location and process of the editing. This means provenance information cannot save only a final file, but must save the complete transformation chain: the original capture, a local mask indicating which regions were replaced, the identifier of the model used, and the export step. The user interface then displays which regions were generated or edited, rather than giving the entire file a blanket classification.

In this scenario, statistical detectors can scan patch by patch, attempting to give a region-level generation probability. However, multiple compression and the mixing of real and generated pixels dilute the signal, and the boundaries between regions may also be uncertain—the traces left by the generative model in transition zones are not always clearly distinguishable. Therefore, region-level detection provides a localization result with uncertainty, rather than a precise pixel mask.

The correct data structure must preserve references to parent resources and a record of each transformation step, so that it can later be traced back to “at which step and with which model was this sky replaced”. Whole-file binary classification cannot support this tracing requirement, because it discards the local information most critical to mixed content.

A 99% detection rate measured in the lab on original generated images hardly represents a detector's real performance on social platforms. The reason is that content, from generation to being seen, passes through a real propagation chain, where each post-processing step changes the signal the detector relies on for its judgment.

Evaluation must actively reproduce these transformations: JPEG compression, repeated resizing, screenshots, cropping, filters, compositing different content, speech transcription, translation, and resampling. Each of these may erase statistical traces, break watermarks, or discard signature metadata. In addition, evaluation data must include unseen generators, camera models, languages, and content produced after the training time; otherwise, high scores may merely reflect memorization of training data sources.

More challenging is the adaptive attacker. When attackers know of the detector's existence and its general mechanism, they can directly optimize the generation process against the detector to make output actively evade; they can also turn around and plant watermarks on real content, weaponizing the detector's positives. If evaluation only targets passive scenarios, it cannot measure the system's reliability when someone is deliberately adversarial.

Therefore, an evaluation report cannot give only a single overall number. It should separately report the ROC curve, calibration error, rejection rate, and unknown rate after each transformation. Calibration error measures whether confidence scores actually correspond to real probabilities; rejection rate measures how much content is refused a decision because it falls outside the detection range; unknown rate measures whether the system honestly says “I don’t know” when facing unfamiliar distributions. At the same time, the detector itself will be upgraded, and the same file may be judged differently under different versions, so the version number and the original score at the time must be retained so that historical judgments can be reproduced and traced.

The same detector should not use the same threshold when applied to different decision scenarios. The essence of a threshold is to map a continuous confidence score into a demarcation point for “taking a certain action,” and the error cost of an action varies enormously across different use cases.

Automated content removal by a platform is a high-cost operation: once content is mistakenly removed, genuine authors suffer irreversible harm with almost no buffer. Therefore, automatic penalties require a higher evidence threshold, requiring support from multiple independent signals at the same time, plus human review, and are allowed to trigger only when the confidence is far higher than for ordinary alerts. In contrast, an editorial desk's review prompt is a low-impact operation; the cost of a false positive is merely one extra look, so a broader threshold can be used, but the uncertainty must be clearly expressed so that the person seeing the prompt knows it is only a clue to be verified, not an established conclusion.

The appeal channel must be genuinely usable, especially for genuine authors. Authors should be able to submit original materials, signatures, and complete production records to rebut the machine's judgment. At the same time, differences across different groups and languages must be continuously monitored to avoid the detector producing systematic misjudgments for certain writing styles or image types and pushing specific groups disproportionately toward appeals.

Under no circumstances should detection output alone serve as the basis for disciplinary action, academic adjudication, or legal conclusions. Such consequences require independent standards and evidence systems; the detection score is only one input among them. Who makes the final decision and under what evidence must be clearly recorded, so that the entire decision chain can be audited and held accountable.

AUC is the most common metric in detector evaluation. It measures the classifier's ability to rank positive and negative samples—that is, overall average performance across all thresholds. But a high AUC does not mean positive results are trustworthy in production. AUC is insensitive to the threshold and does not contain any information about the actual base rate or the cost of misclassification. A detector with a high AUC can still misclassify the vast majority of positives in a low-base-rate scenario, because AUC does not answer "at the threshold I want to use, how many of the reported positives are real?"

Evaluation must therefore report concrete metrics at the target threshold: TPR and FPR, PPV and NPV, as well as a reliability diagram—the latter aligns predicted confidence with the true positive rate segment by segment, exposing calibration bias. It should also report the unsupported or rejection rate, that is, how much content the system refuses to judge because it falls outside the detection scope; performance on unknown generators; score changes under different post-processing slices; and confidence intervals for these metrics. Without this information, a pretty AUC number is only a vague description of the data distribution.

For watermarking and signature mechanisms, the evaluation dimensions are different. Watermarking should separately measure verification success rate and false positive rate, plus robustness under transformations such as cropping, compression, and scaling; signatures should measure verification success rate and chain integrity rate—that is, how much content still maintains a complete signed metadata chain after propagation.

Evaluation data itself also has pitfalls. Public datasets may leave provenance shortcuts—for example, samples generated by the same batch of generators share some metadata or compression fingerprint, and classifiers can use these shortcuts to get high scores but fail on the real distribution. The correct approach is to construct paired data from the same device and same compression level, so that generated samples and real samples differ only in "whether they were generated by a model," while actively checking for and preventing metadata leakage to ensure the classifier learns the generation traces themselves rather than the data's provenance.

Stringing together the previous chapters, the verifiable practice of AI content detection and provenance follows a complete causal chain from problem definition to continuous re-evaluation.

The starting point of the chain is to clarify the object of judgment: whether what is being judged is generation, editing, source, or a real event. This choice determines what evidence to use and how to interpret the results at each subsequent step. Then save the original file and the propagation context it has gone through, because transcoding, compression, and screenshots change all technical signals, and the actual state of the file is the precondition for interpreting scores.

With a clear proposition and original materials, run three independent verifications separately: statistical classification, watermark verification, and signature provenance verification. The output of each verification must be annotated with its coverage scope, the version used, and uncertainty; you cannot compress an 82% statistical score, a valid signature, and a single watermark hit into one number. Then use scenario base rates and misclassification costs to select thresholds, mapping continuous confidence scores into actions commensurate with risk. Beyond technical evidence, also incorporate external sources such as reverse image search and publisher verification to complete fact-checking, because source and generation evidence cannot answer "whether the event actually happened". High-impact decisions must go through human review and provide an appeal channel for those who are misclassified. Finally, the whole process must be continuously re-evaluated with unseen models and real post-processing chains to prevent conclusions from remaining in the memory of the training distribution.

Each step in this chain can also be independently verified, and the verification method follows a common principle: change only one core variable at a time while locking the remaining configuration, then observe the key intermediate states and the location where it first deviates from expectation. At the input layer, use the same batch of samples, the same preprocessing, and the same permission boundaries for comparison, while recording input hashes, slice labels, and rejection reasons to ensure that comparison starts from an identical baseline; at the mechanism layer, after changing the target variable, observe at which step the intermediate state first deviates from expectation; at the output layer, use the same acceptance rules and resource budget to measure stratified differences in quality, cost, latency, and failure rate; at the falsification layer, retain a control group that does not enable the target mechanism, and check whether benefits replicate stably across different samples and random seeds, rather than being an accidental result under a specific configuration.

Only when each step defines a clear proposition, records the scope and uncertainty of the evidence, and can be stably reproduced through controlled comparisons, can the conclusions given by the detection and provenance system be worthy of the trust it carries.