System Prompts: Declare Behavioral Contracts in Probabilistic Models, Not Security Boundaries

After labeling a piece of text with the system role, does it become a program that the model cannot violate? No. Like other messages, system prompts ultimately are just a string of tokens the model sees; what actually changes model behavior is probability, not enforced execution.

To see this clearly, first look at the input and output of system prompts. Its input is a piece of text injected by the application, usually containing role settings, task descriptions, behavior boundaries, and output contracts; its output is the distribution of answers the model gives conditioned on this string of tokens. The chat runtime does not pick out the system message to "execute" separately; instead, it encodes messages of different roles into the input prefix according to the chat template, and then lets the model compute the conditional probability of the next token over the entire prefix. The special status of the system role is mainly reflected in template encoding: it is usually placed at the very beginning of the conversation and separated from other messages with special markers.

This separation brings about a tendency at the training level, not enforcement at the runtime level. During training, the model learns a statistical regularity: instructions from higher-priority roles are more worth following, so content marked as system gains greater influence statistically. But the generation process itself has no formal interpreter that reads prompts one by one and executes rules one by one. The model always does the same thing: estimate the probability distribution of the next token based on the entire prefix, then sample. The so-called "following system prompts" is essentially just that the model is more likely to produce text that meets the prompt's expectations under this conditional probability.

Since following is a probabilistic behavior, its stability necessarily fluctuates with input characteristics. The longer the prompt, the more hidden internal conflicts are, and the more unfamiliar the task, the lower the model's confidence in correctly separating different requirements and implementing them one by one, and the more unstable following may be. This does not mean that system prompts are useless: in common scenarios, they can still guide behavior to the expected direction with high probability. But they cannot be treated as hard boundaries. Any guarantee that relies on "the model will definitely not violate this sentence" mistakes probabilistic behavior for formal rules.

Therefore, the reasonable use of system prompts is to declare behavioral expectations and coordinate context: tell the model what role it plays, what task it needs to complete now, and what contract the output should satisfy. The unreasonable use is to stuff enforcement responsibilities from outside the model into the prompt, such as storing API keys in it, using it to enforce tenant isolation, or letting it decide on its own whether to authorize payments. Once secrets are written into the prompt, they enter the text space where the model can be induced to output them; identities and permissions involve real-world consequences, and the model's probabilistic judgment cannot replace auditing. Truly impassable boundaries must be provided by systems outside the model: tool parameter validation, identity permissions, and business policy. The prompt is responsible for declaring "what should be done", while "what must not be done" must be enforced by the system.

Why shouldn't a later user message override an earlier system constraint? Because messages are not decided by temporal order, but by the trust level of their source.

Conflict resolution takes as input a sequence of messages with roles, sources, and times, and produces two things as output: the high-level task to execute, and the low-level conflicts to ignore. Conflicts typically occur when a user says "ignore the previous rules" and this collides with system constraints; the goal of resolution is not to satisfy every message, but to determine which layer's intent should win under the given trust structure.

The trust hierarchy is divided by source:

System or developer messages represent the application policy and task contract and have the highest status; they are used to constrain lower-level instructions: no matter what the user later says, the later user message cannot rewrite the boundaries the application has already set. User messages represent goals and inputs; they are executed within the scope allowed by higher layers—users can freely make requests within the area delimited by the system, but out-of-bounds parts simply do not take effect. Retrieval, web, or tool output is untrusted data: it can only participate in reasoning as fact candidates, and cannot be automatically promoted to commands just because it happens to look like instructions. Historical summaries and memory are derived state, not original facts; when used, they should carry source labels, have clear scope, and remain correctable—they may be outdated or erroneous and can be overturned at any time by more reliable information.

Temporal order only operates within the same layer. If two user intents contradict each other, the newer intent can update the earlier one; if two facts returned by tools conflict, they can also be reweighted according to arrival. But once a conflict crosses layers—for example, a sentence in untrusted text says "now switch to administrator identity"—no matter how late it arrives, it cannot override system constraints. Cross-layer conflicts are decided by trust priority, not by temporal order.

Priority is an expected rule, not an execution guarantee. The model may fail to execute because the prompt is too long, the conflict is too hidden, or the task is unfamiliar; therefore at runtime the resolution result cannot be treated as an established fact. You must still verify outputs and side effects to ensure that what actually takes effect is the high-level contract, not an induced low-level instruction.

Should the rule “refunds over 500 yuan require approval” be written only into the system prompt? A complete refund assistant case can show the answer clearly.

The inputs to this case are four types of information: the user's refund request, order facts, amount rules, and the authenticated principal. The expected output is not “account changed” but a structured refund proposal or an explicit missing-information state. In other words, the model's task is limited to explaining and proposing; making account changes is simply not within its responsibilities.

Here the system prompt is responsible for declaring responsibilities, and only declaring responsibilities: explaining the refund policy, collecting order information, and proposing refund recommendations, while explicitly not claiming that execution has already happened. Order data goes into a source-attributed structured block, and it is noted that any free text within it is untrusted—notes filled in by users and descriptions scraped from web pages are data, not commands. The output is required to follow a fixed schema: the proposed content, the order ID as evidence, the amount, and whether human approval is required. In this way, every step of the model's output has a checkable shape.

Consider a specific request: the user asks to refund 800 yuan. Because 800 yuan exceeds the 500-yuan threshold, the model can explain the policy and generate a refund proposal pending approval. Note that nothing has happened at this point—if the prompt passes, it does not mean the refund is approved. What actually performs the action is the refund tool, which independently verifies the authenticated principal, order ownership, amount, and approval token; when the amount exceeds 500 yuan and there is no approval, the executor must reject the call. The intent proposed by the model can become a real operation only after each check in the deterministic policy gate has passed.

After the tool returns a status, the model can only report the actual result. If the tool says “submitted,” the model must not write “credited”; if the tool rejects, the model must not claim success. The prompt determines how the model explains and proposes, while the executor has final authority.

This division of responsibilities directly affects test design. Scenarios that need to be covered include: the normal flow, an amount exactly at the boundary, missing fields, the user attempting to override the rules in their own words, injection instructions embedded in order notes, and errors from the tool itself. Each type of scenario verifies the same thing: whether the model always stays within the scope of proposing and explaining, and whether every real side effect is independently decided by the executor.

After adding one page of rules, overall effectiveness drops; how can you tell whether the rules are conflicting with each other or the longer context is squeezing out attention? The answer is not to keep adding rules, but to perform prompt ablation.

The input to prompt ablation is the same contract test set plus several prompt versions: a short version, a long version, and a version where the new rules are rewritten as structured expressions. The output is a set of comparable metrics: pass rate, number of fixes, number of regressions, token cost, and first-token latency. One discipline applies: change only how the rules are expressed, one factor at a time, then compare slice by slice. If a version's net benefit is negative, meaning it causes more regressions than fixes, that version is not eligible for release.

Two basic metrics are worth writing down. Pass rate r equals the number passed m divided by the total number of tests n, that is, r = m ÷ n. Net benefit Δ equals the number of old failures fixed by the new version f minus the number of new regressions caused by the new version g, that is, Δ = f − g. A negative Δ means that the new regressions outnumber the fixes; then the version cannot be released even if some new rules actually took effect.

Look at a specific ablation run. The test set has 200 contract tests in total. The short prompt passes 176, an adherence rate of 88%. After adding the new rules, the long version passes 170, an adherence rate of 85%; examining the composition of those 170, old capabilities regressed in 12 tests, and the new rules fixed only 6, so the net change Δ = 6 − 12 = −6. The surface difference is 6 tests, but the nature is completely different: the long version broke 12 tests that used to pass, in exchange for only 6 fixes.

This still does not justify concluding that “the model cannot remember all the rules.” Break the new rules into a structured conflict table and test again; if it passes 184 tests, i.e., 92%, that shows the problem lies in how the rules are expressed and how they conflict, not simply in capacity limits. The same mechanism, expressed differently, yields a difference from 85% to 92%, and that is exactly the value of ablation: it decomposes the vague phenomenon of “adding rules makes it worse” into causes that can be located.

A single overall pass rate can still hide a critical safety regression—for example, in the overall 85%, it is precisely the “excess refund is rejected” test that failed. Therefore release decisions cannot look at only a single total; they must consider fixes, regressions, and cost together. In addition to pass rate and net benefit, you must also measure token cost and first-token latency, and validate separately on different task slices; do not let a single example decide whether a version stays or goes.

When the model is injected, which layer can still prevent real side effects? This layered diagram gives the answer: the deterministic layer after the model.

Figure 1 depicts the control flow. On the left side of the diagram, three inputs enter the model: the system prompt, the user goal, and untrusted data. The model internally fuses them into a structured intent—not free-form text, but a proposal with fields that can be checked by downstream programs. After this intent flows out of the model, it passes through two layers in sequence: first the policy permission layer, whose inputs are the structured intent proposed by the model plus authoritative state such as identity, amount, and target domain; its outputs are tool calls that are allowed, denied, or pending approval. Then the business tool layer is responsible for executing idempotent state changes and returning the real outcome.

The division of roles among the three components is fixed in the diagram: the probabilistic model only proposes intent, the deterministic policy gate performs validation, and the business tool is responsible for idempotent state changes and returning facts. The model layer has no real permissions—its output intent can become a tool call only if it passes policy validation, and the identity, amount, and target domain used for validation all come from authoritative state outside the model, not from the model's own judgment.

One easy misreading of this diagram: the arrows in the diagram represent control flow, not that the system prompt itself has authorization capability. The system prompt appears on the input side of the diagram; its role is to manage model behavior, prompting the model to propose correctly formatted, clearly bounded intents. Real permissions are managed by the deterministic policy. The responsibilities of the two layers do not overlap and cannot replace each other—this is exactly the meaning of the title of Figure 1: the system prompt manages model behavior, and the deterministic policy manages real permissions.

What happens when role voice, policies, data, and examples are mixed in one paragraph? The model has difficulty distinguishing which are rules that must be followed, which are merely tone examples, and which are changeable facts, and conflicts follow. Prompt design is meant to solve exactly this problem.

The inputs for designing a prompt include: stable goals, dynamic data, trust boundaries, failure handling, and an output schema. The output after design should be a minimal, partitioned, and contradiction-free behavioral contract.

The partitioning approach is to split content by nature: stable goals, inviolable constraints, input trust boundaries, decision process, output schema, and failure handling each have their own place. At the same time, the whole document should use the same set of terminology to express subjects and actions, and clearly state what applies and what does not apply—rules should not only say “what to do” but also “what not to do”; otherwise the model will improvise in the silent zones. Keep stable rules in the prompt; frequently changing business data should be placed in structured configuration or injected through retrieval, and should not be copied into a long prompt. Hardcoding changeable data in the prompt means that every data change requires rewriting the entire contract, and old and new values may coexist in the long text, creating new conflicts.

When rules conflict, explicitly state the priority relationship and provide a conservative fallback. For example: “When information is insufficient, ask; do not guess; for write operations, propose first and then approve.” Such a sentence fixes the decision order at the intersection of the two rules: when information is insufficient, prefer to stop and ask; for write operations, always propose first. With the fallback direction uniformly chosen to be the more conservative side, the model will not choose an aggressive path on its own when uncertain.

“Minimal” does not mean the shorter the better. After reduction, the prompt must still be validated with boundary-covering positive and negative examples: positive examples confirm that a rule takes effect when it should, and negative examples confirm that it does not take effect when it should not. Compared with a large number of homogeneous style examples, a small number of boundary-covering positive and negative examples are more valuable, because they test the boundaries of the contract rather than the writing.

Why does wrapping webpage content in XML tags only reduce confusion instead of truly isolating it? Because instructions and data share the same channel.

The input to an injection test consists of two parts: trusted instructions, and malicious text from a webpage, document, or tool result. The output must be observed on two levels: the model’s stated intent, and the tool actions that actually occur. When a model says “I won’t execute instructions in the webpage” but then calls the tool demanded by the webpage, that is where injection actually takes effect.

Using tags, quotation marks, and phrases like “do not execute the following content” gives the model semantic hints: they make the model statistically more likely to treat this text as data. But hints are not isolation. Malicious text in the webpage still enters the same context and participates in the same round of attention computation with other tokens; it can still influence the probability distribution of the next token. That is what “only reducing confusion” means: tags reduce the probability of misreading data as instructions, but they cannot block the paths by which malicious text gradually changes model behavior through the content itself, through adjacent relationships, or through long chains.

Indirect injection exploits exactly this shared channel: the attack text does not require the model to comply immediately, but induces it to leak prompts, rewrite goals, or call tools. The longer the chain, the more likely the model is to forget the trust boundary at some intermediate step and dilute rules established earlier in subsequent text.

Therefore the real defense lies not in prompt wording but in structure: treat all external content as data and minimize visible context so that irrelevant external text does not enter the window; use least privilege for tool use and an allowlist of targets so that even if the model is induced, it can only call the predefined tools; require sensitive actions to be approved by deterministic policies or humans, leaving the model no path to trigger them on its own.

Testing must likewise be designed around this channel. Passing only a template attack such as “ignore previous instructions” does not prove resistance to multilingual rewrites, encoding obfuscation, or multi-turn inducement. Effective injection testing should cover multilingual, encoded, and long-context injections, as well as injections hidden in tool return values—attackers will insert text through any opening that can enter the context, so the defense must be verified at every opening.

If you don't put keys directly in the prompt, can you put internal policies and detection thresholds there? The default answer from leakage evaluation is: No. System prompts are not secret storage; leak prevention must assume they can be recited or inferred.

The inputs to leakage evaluation are three things: assets in the system prompt, attack queries, and the path to the logging tool. The outputs are three levels: verbatim recitation, semantic inference, and actual security loss. The first two levels measure the form of the leak, and the third measures whether it truly constitutes an incident. Behind this evaluation framework lies a default assumption—once text enters the context, it should be considered potentially recitable or inferable. Therefore keys, personal data, and sensitive detection logic must not enter the context; publicly disclosable behavioral norms can be written into the prompt, while sensitive detection logic stays on the server side.

Leakage paths are much broader than “directly asking for it.” Users can obtain prompt content through direct requests, role-playing, piece-by-piece inference, error echo, or tool logs, and the model may also actively recite its meaning. Error echo is especially insidious: a single error that spits back a prompt fragment verbatim is equivalent to completing an unintentional leak. This means leak prevention cannot rely on wording tricks such as “we didn't write keys into it”; it must rely on asset classification based on “whatever enters the context may leak.”

During evaluation, keep two metrics separate: prompt leakage and instruction violation. The fact that the model does not recite the original text does not mean it has not been controlled by low-level malicious instructions—an attacker can fully alter model behavior without triggering recitation. Conversely, if a public behavioral norm is recited, that does not necessarily cause security loss, because it was never secret. To decide whether a leak counts as a failure, first define by asset value: what is lost if this text leaks? Define assets first, then test; otherwise you will either treat inconsequential recitation as an incident or miss genuinely valuable information.

The model says “needs approval”—why can't downstream just read this sentence? Because natural language has no structure that a program can verify, and no rejection path that a program can process. The output contract exists to solve exactly this.

The input to the output contract is the model's proposal; the output is versioned state, fields, and rejection reasons for downstream consumers to parse. By versioned, we mean the schema produced by the model carries an explicit version identifier; the state must also take values from a finite enumeration, such as propose_refund, needs_approval, insufficient_data. Rejection or missing information is not an unexpected condition but a first-class state—like successful proposals, it has a fixed shape, and consumers do not need to guess what happened from an explanatory text.

After receiving the output, consumers must perform five kinds of validation in order: type validation, enumeration validation, cross-field validation, identity validation, and business state validation. Type and enumeration validation ensure that the fields themselves are valid; cross-field validation ensures that fields do not contradict each other; identity validation confirms that the initiator actually has permission to act; business state validation confirms with the authoritative system whether the current state allows this operation. Free-text explanations can never trigger side effects—they may explain why, but no piece of explanation is authorized to drive real operations.

There is a boundary that must be made clear: structural validity only means the format passes; it does not prove semantic correctness. A perfectly formatted propose_refund may still belong to someone else's order, or exceed the allowed amount. Whether the order belongs to the user and whether the amount is allowed can only be determined by querying the authoritative system, not by the model's wording. Structured output guarantees format, not semantics.

Finally, the prompt and schema versions must be regression-tested together. Once the prompt version the model sees and the schema version the consumer parses become misaligned, new state names will appear, and old consumers will reject or misjudge due to enumeration mismatch. Contract drift occurs when these two versions are upgraded independently without each other's knowledge, so their regression tests should be performed as part of the same change.

The system prompt has never changed, so why can behavior still change after a long conversation? Because the context the model sees at each turn is not the same. The prompt is only one part of the context; the rest continually drifts during the session.

The assembly process for long-term state can be viewed as follows: its inputs include system invariants, historical messages, compressed summaries, tool observations, and scoped memory, and its output is the context actually visible to the model this turn. Historical messages accumulate continuously, compressed summaries rewrite and condense earlier content, tool observations inject new facts, and memory is read in by task or user scope. These components continually change the conditional context—in the "current situation" the model sees, old user goals may already have been summarized into high-confidence facts, or they may persist across tasks. Initial rules appear only once at the beginning of the conversation; they cannot guarantee that they still exist after summarization, nor can they prevent old goals from continuing to take effect in the next task.

The countermeasure is to turn invariants into a per-turn routine: restate the necessary boundaries every turn, rather than relying on "having said it once at the beginning". Memory is read by task/user scope, bringing in only the parts relevant to the current task; derived state passed to the model should carry its source and be marked as correctable—the model should know where a summary comes from, under what scope it holds, and when it should be overridden by authoritative information. When switching tasks, clear local memory so that the goals of the previous task do not contaminate the next task.

Validation should also target this drift mechanism: after compression, rerun conflict tests and safety sentinel tests to check whether summaries swallowed key boundaries and whether old goals still remain after switching. If the safety of a long-term AI Agent is built on "saying it once in the opening", it has already placed the insurance where drift can wash it away.

Why can changing just one phrase also change completely unrelated tasks? Because every token in the prompt participates in computing generation probabilities; any text change redistributes the generation distribution.

Prompt release is not as simple as pasting a new piece of text. The complete input for a release is six versions: prompt, template, model, sampling parameters, tool schema, and evaluation set. They must be recorded together, because behavior is their joint product. The output is a candidate release with regression evidence and a rollback package—the latter means that when an anomaly occurs, the prompt and the associated schema can be rolled back together to the previous consistent state.

Why does a wording change have such a wide surface area? Prompt changes alter the model's internal attention and generation distribution, and this change is not local: it can affect outputs in different languages, behavior in long contexts, tool selection preferences, and the timing of rejections. A seemingly harmless wording adjustment may cause some tool to be selected one extra time in an edge case, or cause a certain type of request to shift from "correct rejection" to "false rejection." Therefore, before release, use canary comparison to run the new and old versions in parallel on a small portion of real traffic, comparing task success, contract adherence, conflict resolution, false rejection, token cost, and manual corrections.

Text diff shows what changed, not how behavior changed. A diff can tell you which line changed, but it cannot prove that unrelated tasks did not suffer regressions through side paths. Therefore, release requires three sets of data: a frozen regression set, to ensure known behavior does not degrade; a hidden attack set, to ensure the safety boundary is not silently weakened by this change; and production shadow samples, to observe changes on the real distribution. When an anomaly occurs, rollback is based on this evidence, not on the text itself—rollback means rolling back a whole set of mutually locked versions, not changing a sentence back.

Why can't prompt testing just save a few ideal conversations? Because ideal conversations cover only one quadrant, and the other three quadrants are exactly where the model is most likely to make mistakes.

The inputs to contract testing are four categories of samples: normal, conflict, malicious, and failure. The outputs are assertion results about structure, decisions, side effects, and user-visible facts. The object of assertions is not verbatim wording—it is acceptable for the model to use different phrasing to accomplish the same thing—but the things that must hold: whether the output structure is correct, whether the decision took the right path, whether tool side effects occurred, and whether the facts the user sees are consistent with the true state. Verbatim wording need not be identical, but high-risk actions and fact boundaries must pass deterministically.

Each of the four quadrants has its own task. The normal set validates the main task itself: given standard input, does the model produce correct proposals and explanations? The conflict set makes the user, history messages, and tool content conflict with high-level constraints: the user requests unauthorized actions, history retains old goals, the tool returns text that contradicts the rules—does the model still maintain high-level constraints? The malicious set covers injection, extraction, encoding obfuscation, and multi-turn inducement, checking whether attack text changes behavior. The failure set covers missing fields, timeouts, refusals, and inconsistent tool returns, checking whether the model stops to ask when information is incomplete rather than guessing.

Test results cannot be reduced to a single total. They must be broken down by language, length, model version, and risk slice, report confidence intervals, and retain failing examples—failing examples are direct material for the next round of fixes and expanded testing. Model review can help expand the test set and identify missed scenarios, but high-risk determinations do not use model review as the final basis; they use deterministic checks or human review: if a test involves whether a refund actually occurred, deterministic code must assert the actual result of the tool call, rather than having another model grade it.

The entire causal chain can go from a single question to verifiable practice in eight steps.

Step one, split business requirements into stable contracts and dynamic data: rules stay in the prompt, while data that changes goes through structured configuration or retrieval. Step two, label each type of context with role, source, and trustworthiness: system constraints, user goals, untrusted data, and derived memory each go to their own layer from then on. Step three, use a minimal prompt to declare goals, conflicts, and failure paths: state only behavioral expectations in the prompt, and make explicit the priority relationships and conservative fallbacks when conflicts occur. Step four, have the model propose only structured intent: proposals with a schema, not free-text commitments. Step five, use deterministic policy to verify identity, permissions, and business state: what truly blocks side effects is not wording but a policy gate. Step six, pass tool facts back without fabricating success: the model reports the real state returned by the tool, adding nothing and removing nothing. Step seven, evaluate with a four-quadrant contract set and release via canary: freeze normal, conflict, malicious, and failure samples into a regression set, and compare them with low-traffic canary releases. Step eight, monitor long-term state drift and versioned rollback: restate invariants each turn, read memory by scope, and on anomaly roll back together with the schema.

The chain itself must withstand verification, and verification proceeds at four levels:

The input layer fixes the same batch of samples, preprocessing, and permission boundaries, and observes input hashes, slice labels, and rejection reasons, ensuring that every experiment faces the same input distribution. The mechanism layer changes only one core variable at a time while locking all other configurations, and observes key intermediate states and the first position where expectations diverge—whichever step the divergence occurs at indicates which link in the causal chain has broken. The output layer uses the same acceptance rules and resource budget, and observes stratified differences in quality, cost, latency, and failure rate to prevent a single aggregate from hiding local differentiation. The falsification layer keeps a control group that does not enable the target mechanism, and tests whether gains reproduce stably across samples and random seeds—only if the effect becomes measurably worse after removing the mechanism can we say the effect comes from that mechanism rather than from chance.

The end of this chain returns to the central theme: a system prompt declares a behavioral contract in a probabilistic model, rather than establishing a safety boundary. The contract is honored through structure, policy, and evidence; the boundary is always guarded by deterministic systems outside the model.