Reflection and Self-Correction: Making Failure Evidence Change the Next Action
Distinguish vague self-assessments, external feedback, and verifiable corrections; use minimal changes, stopping conditions, and versioned experience to avoid “thinking it over again” producing more confident errors.
1Why “Thinking Again” Is Not the Same as Error CorrectionIntuition
The most common misunderstanding caused by the word “reflection” is equating self-correction with “having the model think again.” The same model reading the same context—why should the second pass necessarily be more correct than the first? No mechanism guarantees this. During the second pass, the weights and context are exactly the same as in the first pass; the model does not gain any new information just because of the instruction to “think again.” It may discover some surface contradictions, or it may follow the original hypothesis from the first pass once more, and it may even, because it is asked to “please find errors,” comply with the instruction and change an originally correct answer for the worse. The latter two cases are exactly the opposite of “error correction”: one is verbatim repetition, and the other introduces new damage.
To understand why, we need to distinguish how two kinds of input change the information state. Model self-criticism occurs only inside the model; it does not touch the world, only its own existing distribution. Therefore it can only resample within the same set of hypotheses. In contrast, test failures, compiler diagnostics, retrieved sources, environmental observations, or human rule judgments come from outside the model and carry information that the model did not previously possess. Only the latter can truly change the information state: when test execution returns a failure, the model learns for the first time that “the code written according to this candidate cannot pass”; when the compiler gives a diagnostic, the model learns for the first time exactly which line violates which constraint. The task of the reflector is to explain how these external pieces of evidence negate the current candidate, not to generate a longer inner monologue. No matter how long the monologue is, as long as there is no new evidence, it remains within the original blind spot.
From this we can derive an operational definition: a correction is successful only when “the candidate undergoes a localizable change, passes the acceptance that previously failed, and does not break any protected tests.” This definition turns “reflection” from an attitude into a testable causal chain—what the input is, what the output is, and how success is judged. The inputs to reflection-based correction are four: the current candidate, external failure evidence, protected acceptance, and modification budget; the outputs are three: a falsifiable causal hypothesis, a minimal patch, and a retest decision. The causal hypothesis must be falsifiable, meaning it must be capable of being refuted by subsequent testing; otherwise it is just another explanatory monologue. The patch must be minimal, because the smaller the change, the easier it is to confirm exactly which change eliminated the failure. The retest decision sends the new candidate back to the same acceptance that produced the failure evidence.
The causal chain can be fully stated as follows: external acceptance (tests, compilation, retrieval, environment, or human rules) executes against the candidate and produces failure evidence; the failure evidence is input to the reflector, changing the model’s information state; based on this, the model proposes a falsifiable causal hypothesis and produces a minimal patch; the modified candidate is again checked by the original acceptance and protected tests; if the original failure disappears and the protected tests are not broken, the correction is successful. If any link is missing, the chain breaks: without failure evidence, “reflection” is just resampling within the same blind spot; without retesting, there is no way to know whether the correction is truly effective.
The applicability conditions of this boundary are strict: reflection-based correction is meaningful only when there is external failure evidence. Model self-criticism by itself does not change the information state; tests, compilation, retrieval, environment, or human rules do. Therefore, any scenario in which one merely “thinks again” without the introduction of new evidence should not be regarded as error correction, nor should it be counted in the correction success rate. Fixing this point provides a common yardstick for subsequent discussions of the reflection loop, reflection records, and stopping conditions.
2What roles does a reflection loop have?mechanism
A loop that can truly accomplish “reflection–correction” does not rely on a single output from one model, but on the cooperation of several roles with clear responsibilities and separate artifacts. The roles are separated to prevent a very specific failure mode: if the same person both sets the questions and grades the papers, they have an incentive to quietly lower the standard; if the same person both explains the failure and makes the changes, they may put an unverified explanation directly into the candidate as fact. Therefore, the states and interfaces of different roles in the loop must be separated, even if they are handled by the same model at different times.
The loop starts with the executor. The executor has only one responsibility: produce a candidate. A candidate can be a piece of code, a document, a plan—it is only the “current version” and does not carry any judgment about whether it is right or wrong. The candidate is then handed to the validator. The validator’s responsibility is likewise limited to one thing: perform external acceptance and return only reproducible pass/fail evidence. Note the qualifier “reproducible”—the validator’s output must be evidence that can be rerun and yields the same result for anyone who runs it, such as a test’s failure output, a compiler’s diagnostic, a retrieved source, an environment observation, or a human rule’s judgment; it is not responsible for explaining the cause of failure, nor for proposing modifications. This restriction is deliberate: if the validator also supplied a fix, its judging position would be contaminated.
The reflector appears only when the validator returns failure evidence. The reflector takes as input the failure evidence and the current candidate, proposes a falsifiable causal hypothesis, and based on that designs a minimal change (a single patch). The reflector does not directly modify the candidate; what it outputs is a hypothesis about “why it failed” and a plan for “where to change”. The executor then applies the patch to the candidate, forming a new candidate, and the same validator retests the new candidate. Here “the same validator” is a hard constraint: whether the correction succeeds must be confirmed by the same acceptance suite that originally judged it a failure; retesting with a different, easier-to-pass acceptance suite is equivalent to quietly lowering the standard. The retest runs two kinds of checks at the same time—the original tests that failed before, and protective tests. The former proves the failure has been eliminated, and the latter proves this change did not break the parts that were originally working. Finally, based on the retest results, a decision is made: keep the new candidate, or roll back to the version before the change; if the failure evidence is insufficient to locate the cause, continue investigating rather than making blind changes.
Writing out this causal chain in full: candidate → external validation → failure evidence → causal hypothesis → single patch → original tests + protective tests → keep or roll back. It also defines the inputs and outputs of the reflection loop: the inputs include the candidate, the validation result, the causal hypothesis, the patch, and the same acceptance suite; the outputs are one of four decisions: keep, roll back, continue investigating, or stop. When the external evidence is sufficient and the patch passes both the original tests and the protective tests, the output is keep; when the patch does not pass acceptance, the output is roll back; when the failure evidence is insufficient to support a reliable causal hypothesis, the output is continue investigating; when further investment no longer yields benefit, the output is stop.
Here we need to emphasize again the significance of separating interfaces. The validator’s “only reproducible evidence” is not technical purity, but the source of the loop’s credibility: if the judge can quietly lower the acceptance standard after a failure, then any candidate will eventually pass, and the loop will degenerate into a ritual. Likewise, the reflector’s hypothesis is always only a hypothesis until it is validated; the modifier must not treat an unverified explanation as fact; otherwise, a wrong attribution will directly become damage to the candidate. The separation of roles allows the artifacts of each step to be individually inspected and individually held accountable: who generated the candidate, who provided the evidence, who proposed the hypothesis, what the patch changed, and who completed the retest. This traceability is the basis on which the loop can continue to operate after failure instead of spinning in place.
3A reflection record must be able to drive actionrecord
The phrase “check the code logic” sounds like a reflection, but it is not. It does not point out any locatable defect, does not specify any executable action, and does not define any verifiable expectation; therefore it cannot drive any acceptance, nor can it move the loop forward one step. The entire purpose of a reflection record is to drive action; the criterion is not how sincere the tone is, but whether every item in the record maps to the next link in the loop.
We can use a cache isolation case to see the structure of a valid record in full. In a multi-user system, the cache key is composed only of product_id. The failure evidence is: when user B reads the same product_id as user A, it hits user A's price. This evidence is reproducible—as long as we construct two users accessing the same product, we can observe the cross-user value again; it does not rely on any judgment about tone or attitude. Once the failure phenomenon is reproducible, it meets the verifier's requirement of “only giving reproducible evidence.”
Based on this evidence, the reflector proposes a causal hypothesis: the cache key has only product_id and lacks user_id. This hypothesis is valid because it points out a falsifiable mechanism—if the hypothesis holds, then as long as user_id is included in the key, the cross-user value should disappear; if the cross-user value remains after adding user_id, the hypothesis is negated. Next is a single change: change the cache key to (user_id, product_id). Changing only this one place is to limit the attribution scope—the retest result can be clearly attributed to the change in key structure, rather than the mixed effect of several modifications. Finally, expected verification must protect two goals simultaneously: cross-user access to the same product no longer hits each other (isolation), and repeated requests from the same user still hit the cache (cache benefit is not lost). If only isolation is verified and hits are ignored, the fix might render the cache completely useless.
A high-quality record can neither merely restate the error—“here's a bug” adds no information; nor can it list ten possible causes at once—a ten-way guess is not a causal hypothesis, but an unconverged enumeration that cannot guide a minimal change. When the cause is still uncertain, the correct action is not to change the code first, but to design a discriminating experiment: for example, record the actual generated key and directly observe what is actually stored in the cache; or construct a paired test with two users and the same product to confirm whether the cross-user value reproduces stably. The result of the experiment will converge “possible causes” into a falsifiable hypothesis, and only then decide where to change.
From this, we can give the complete input and output of a reflection record. The input has five items: failure phenomenon, environment version, causal hypothesis, single action, and expected verification. The environment version is listed separately because the same failure may not reproduce on different versions, and a record lacking version information cannot be audited and retested. The output is an executable and auditable correction plan: evidence points out the cache cross-user value phenomenon, hypothesis gives the falsifiable mechanism of missing user_id, action is to change the key to a composite key, expected is that both isolation and hit tests pass simultaneously. Conversely, any record missing one of these items can be directly judged as unqualified—missing evidence means the record is only a guess; missing hypothesis means the record is only a restatement; missing action means the record is only a complaint; missing expected verification means the record cannot be accepted. “Check the code logic” is invalid precisely because it lacks all three: hypothesis, action, and expected.
| Field | Cache isolation case | Acceptance role |
|---|---|---|
| Failure evidence | When user B reads the same product_id, it hits user A's price | Reproducible, does not rely on tone judgment |
| Causal hypothesis | Cache key has only product_id, lacks user_id | Points out falsifiable mechanism |
| Single change | Change key to (user_id, product_id) | Limits attribution scope |
| Expected verification | No cross-user hits; repeated requests from same user still hit | Protects both isolation and cache benefit |
4How a Single Failure Becomes a Minimal FixWorked Example
For a failure to become a credible correction, it must answer one question: How do you prove that the fix came from feedback, rather than the second generation just getting lucky? The answer lies not in how well-written the fix is, but in whether the fix is constrained by an external counterexample. The cache isolation case walks through this process, fully demonstrating a reflection loop from test failure to minimal patch to regression verification.
The case's inputs are four things: a v1 single-key cache (key only product_id), a cross-user counterexample (user B hits user A's price), an isolation test, and a same-user hit protection test. v1's retest results are: isolation test 0/2 passed, same-user hit test 2/2 passed. The combined conclusion from the two numbers is "there is caching, but unauthorized cross-user value mixing" — the caching mechanism itself works; the problem is only in the key granularity. This initial state is important: it shows the failure is local, not that the entire cache is broken.
Based on this, the reflector proposes a minimal fix: change only the key to (user_id, product_id), leaving everything else unchanged. The retest results for the v2 composite-key version are: isolation test 2/2 passed, same-user hit test 2/2 passed, so the conclusion is to keep. The key here is that the retest includes both the failing test and the protection test; both are indispensable — if only the isolation test is rerun, you cannot discover whether the fix broke the originally normal cache benefit.
Comparing an alternative makes this clear. The alternative is "disable caching": isolation test 2/2 passed, but same-user hit test 0/2 passed. If you look only at the failing cases, both composite key and disable caching get 100% pass, and the two options are indistinguishable in front of the failing test; once the protection test is added, disable caching is immediately exposed — it fixes the isolation symptom but breaks the performance contract. Only the composite key simultaneously satisfies the full objective: no cross-user value mixing, while same-user hits still occur. The fix is thus proven to be feedback-constrained: the external counterexample pinpointed the defect location, the protection test ruled out shortcuts that would break performance, rather than the model happening to draw a correct implementation on the second try.
This comparison also establishes a more general boundary: reflection quality is bounded by the upper limit of acceptance coverage. Reflection can only be responsible for whatever tests exist in acceptance; missing tests cannot be compensated for by critical wording. If the "same-user hit" protection test had not been written initially, then both "disable caching" and "composite key" would be judged as successful fixes, and the loop would quietly accept a version that breaks the performance contract. No matter how articulate the reflector is, it cannot discover a defect that was never tested. Therefore, the rule for this case can be summarized as: change only the key to (user_id, product_id), rerun the failing test and the protection test; if both groups are 2/2 passed, keep; if either group fails, roll back. The output candidate is the v2 composite key, or rollback to v1; and the seemingly reasonable path of "disable caching" is precisely blocked by the protection test.
Scroll horizontally to view the full diagram on small screens.
| Round | Isolation test | Same-user hit test | Conclusion |
|---|---|---|---|
| v1 | 0/2 passed | 2/2 passed | There is caching but unauthorized cross-user value mixing |
| v2 composite key | 2/2 passed | 2/2 passed | Keep |
| Alternative "disable caching" | 2/2 passed | 0/2 passed | Fixes symptom but breaks performance contract |
5How to Control Change Scope and AttributionMinimal Change
Changing five places at once with all tests green sounds like a clean fix, but such a result is almost useless for learning. Imagine simultaneously changing the cache key, TTL, serialization format, and database query: after tests pass, no information tells you which of the five changes actually eliminated the defect; and if a new regression appears, you likewise cannot locate which change introduced it. One large change stirs all causes and effects together; the loop yields only a “passed” conclusion while losing the ability to attribute. Therefore, the principle for controlling change scope is: first apply the smallest patch that can distinguish the current causal hypothesis, record the diff and verification result; if it fails, revert and preserve the evidence, then consider the next hypothesis. Each step should be small enough to attribute the result to a single change.
“Minimal” does not mean an absolute one-line fix. Some changes are naturally interdependent—changing the key structure may simultaneously require adjusting the serialization format, otherwise the new key cannot be written. For such cases, the correct approach is to break the large change into a sequence with checkpoints: after completing each step, run a verification and record the result, rather than requiring a one-shot completion. The role of checkpoints is to cut a set of interdependent changes into attributable fragments: if tests already pass after the first checkpoint, then subsequent hypotheses are no longer necessary; if a regression appears after a certain checkpoint, then the problem is in the step just completed. Although serialized changes are not as sharp as a one-line fix, they preserve the evidence chain of “which step caused what change” and still meet the attribution requirements of minimal modification.
Equally important as the scope of modification is the position of the verification standard. The verification standard must be protected outside the loop. The model must not create green results by deleting failing tests, relaxing assertions, or swallowing exceptions—these tactics can all make tests pass, but what they change is not the candidate, but the grading standard, which exactly violates the constraint that the verifier must be independent. Tests are not forever immutable, but when a test needs to be modified, independent requirement evidence must prove that the original acceptance criterion itself was wrong, not that the standard is changed because the candidate cannot pass. In other words, the threshold for changing tests is different from the threshold for changing candidates: candidates can be repeatedly modified because hypotheses are falsified, but acceptance criteria can only be modified when external evidence shows they were written incorrectly, and such modification must itself be recorded.
Putting these constraints together yields the inputs and outputs of minimal modification. Inputs include: the current causal hypothesis, code diff, interdependencies among changes, failing tests, and protective tests. Outputs include: a revertible sequence of patches, and the attribution evidence for each step. The execution rules are: first change the smallest scope that can distinguish hypotheses; if it fails, revert and retain the result; if there are multiple dependencies, advance step-by-step with checkpoints; after tests pass, you can only attribute the benefit to this controlled set of modifications, and must not exaggerate to “the whole system has thus become better.” The boundary of attribution is determined by the boundary of modification—whatever was changed is all you can claim; this both prevents mistaking luck for skill and prevents describing a local fix as an overall improvement.
6What Does Reflection Memory Store?Memory
The product of a reflection loop is not only the repaired candidates, but also the experience learned from this failure. Experience should be stored in long-term memory, but what to store and how to store it directly determine whether memory is an asset or pollution. Why is storing the complete failure conversation verbatim in long-term memory likely to contaminate the next task? Because the conversation contains a large amount of content that belongs only to that particular situation: specific code snippets, the tone at the time, temporary guesses, and rejected hypotheses. When a new task retrieves this content, the model is likely to treat the previous task's contextual details as this task's facts, or to copy a specific repair as a universal rule. The goal of memory storage is therefore not to “remember what happened,” but to distill “under what conditions, what signals appear, what validated strategies can be taken, and under what circumstances they fail.”
Reusable experience should be written as a five-part structure: applicability conditions, failure patterns, diagnostic signals, validated strategies, and counterexamples and failure conditions, along with source, version, and expiration rules. Use the caching case as an illustration: the transferable experience is “multi-tenant cache keys must cover the tenant boundary”—it preserves the condition (multi-tenant), the failure pattern (cross-tenant value mixing), the diagnostic signal (two users hitting the same price for the same product), and the validated strategy (keys covering the tenant dimension). In contrast, “add user_id to all caches” is an erroneous generalization: it loses the “multi-tenant” applicability condition, isolates public data by user, and causes unnecessary cache bloat or even logic errors. The difference between the two sentences lies in the conditional boundary—a good experience preserves the premises under which it holds, while a bad experience quietly turns a one-time conclusion into a universal rule.
Even if the experience is well written, its premises must still be re-verified at retrieval time. After retrieving “multi-tenant cache keys must cover the tenant boundary” in a new task, the first thing is to confirm whether the current task really is a multi-tenant scenario, whether the cache is truly shared across tenants, and whether the cross-tenant value mixing signal actually appears. Past conclusions are only candidate explanations, not current facts; when the applicability conditions are not met, the experience should be discarded rather than forcibly applied. This also corresponds to the memory expiration rule: experience should record its version and source; when the environment version changes or the source becomes invalid, the experience is automatically downgraded or even invalidated, rather than remaining effective indefinitely.
A deeper issue than memory structure is same-source bias. Even if the generator and the reflection model are two independent calls with two sets of role prompts, they still come from the same training distribution and may share the same biases—the blind spots that the reflection model cannot see are often exactly the blind spots caused by the generator itself. Therefore, “switching to a reviewer role” does not truly add independent information: two models with shared biases checking each other yields two samples from within the same blind spot. The correct approach is to introduce genuinely independent information sources according to the nature of the task: for factual tasks, prioritize external sources; for coding tasks, prioritize running tests; for high-risk judgments, introduce independent rules or qualified humans. Mutual criticism between model roles cannot replace these external signals—this is exactly the extension of the principle that “thinking again is not the same as error correction” to the level of memory and bias. The input of reflection memory is validated failure patterns, applicability conditions, diagnostic signals, strategies, counterexamples, source, version, and expiration rules; the output is conditionalized experience that is retrievable but must have its premises re-verified, not conclusions that can be directly copied.
7When to Stop Instead of Continuing ReflectionBudget
The reflection loop does not automatically get better just because of "another round of fixes." As the number of correction rounds increases, marginal benefit declines and the risk of breaking things rises; the two together determine whether to continue. Consider an order-of-magnitude example: suppose there are 100 first-round failure cases. In round 1, 35 are fixed while 4 originally correct results are broken, for a net benefit of 31; in round 2, another 10 are fixed and 6 are broken, for a net benefit of only 4; in round 3, 3 are fixed and 7 are broken, for a net benefit of −4. Note a phenomenon that is easy to overlook: the overall success rate may still be rising, because each round still fixes some cases; but the marginal net benefit in the third round is already negative—running one more round creates more new damage on average than the failures it removes. The rising overall success rate conceals the fact that a third round is no longer worth executing by default. Therefore, the stop decision cannot look only at cumulative successes; it must report, per task slice, each round's marginal fix rate, break rate, cost, and latency—if any of the four dimensions deteriorates, it should trigger questioning whether to continue correcting.
In addition to marginal benefit turning negative, there are several clear stop signals. First category: two consecutive rounds produce no new failure evidence. At this point the reflector has no new information to process; continuing to correct is just resampling the same information, and it should stop according to the principle that "thinking again is not the same as error correction." Second category: the same test oscillates between two patches—patch A makes the test pass, patch B makes it fail again, and reverting to A makes it pass. This back-and-forth indicates that the root cause is not at the patch layer but in the underlying requirements or state modeling; the correct action is to go back to requirements and state modeling and find the common constraint behind the two patches, rather than continuing to switch between A and B. Third category: the validators are unstable or conflict with each other—the same candidate gives different conclusions at different times or under different validators. The problem is in the acceptance stage; continuing to modify the candidate is meaningless. Fix the acceptance first, then discuss the candidate. Fourth category: cost, time, or risk limits are reached. In this case, the output of the loop is not to keep forcing fixes, but to report the known and unknown parts truthfully and escalate to humans.
Combining these signals, the inputs to the stop strategy are: the number fixed per round, the number broken per round, cost and latency, whether new evidence appears, and validator stability; the output is one of four decisions: continue, roll back, report unknown, or escalate to a human. Judging by that order-of-magnitude example: the net benefits are 31, 4, −4 in sequence, so the third round should not be executed by default; even if the number fixed alone is growing, the number broken already exceeds the number fixed. This leads to a counterintuitive but important conclusion: more rounds do not mean more reliable correction. Each round of correction is simultaneously doing two things—removing known failures and introducing unknown damage; when the latter's speed exceeds the former's, the most responsible reflection is precisely to stop and hand over the uncertainty truthfully, rather than using more rounds to create an illusion of "effort."
| Stop signal | Action |
|---|---|
| Two consecutive rounds without new failure evidence | Stop, avoid resampling the same information |
| Same test goes back and forth between two patches | Go back to requirements/state modeling and find common constraints |
| Validators unstable or conflicting with each other | Fix acceptance first, do not continue modifying the candidate |
| Reached cost, time, or risk limit | Return known/unknown and escalate to humans |
8How to prove reflection outperforms multi-samplingEvaluation
Claiming that “reflection works” is not enough; you must prove that reflection outperforms cheaper alternatives. There is a key confounding factor here: simply calling the model one more time may improve the hit rate—sample a few more times, and pure luck can also produce a better answer. Therefore, to measure the contribution of reflection, you must isolate the gain from “feedback” from the gain from “multi-sampling.”
The way to isolate it is to run controlled comparisons under the same budget. Align all three budget dimensions at once: token consumption, number of model calls, and wall-clock time. Within the same three-dimensional budget, compare four schemes: direct single generation; feedback-free resampling—that is, without introducing any failure evidence, just repeatedly generating and selecting the best; reflection with external feedback—that is, the complete closed loop; and single generation from a stronger model. If feedback-free resampling under the same budget is comparable to the reflection loop, then the complex reflection framework has not been proven necessary—the extra mechanism cost bought no incremental benefit. Conversely, only when reflection with feedback significantly exceeds feedback-free resampling under the same budget is “feedback” itself proven to be the effective ingredient. Likewise, single generation from a stronger model is the other end of the comparison: if switching to a stronger model can achieve the same effect as multiple rounds of the reflection loop in one go, then the benefit of reflection must be re-compared with the cost of “switching models.”
Looking only at the final success rate misses an important damage surface, so evaluation must also report several other metrics: first-round correct-to-wrong rate—how many originally correct answers were changed to wrong during reflection; marginal gain per round—how much each round of correction net-improved; test coverage—what proportion of the behavior surface acceptance covered; average number of rounds, fallback rate, and human escalation rate—how many rounds the loop actually consumed, how many times it had to revert to the original or hand off to a human. Together these metrics can answer whether reflection is net repair or net churn. A reflector whose final success rate rises but whose first-round correct-to-wrong rate also rises may simply be moving errors around.
The construction of the evaluation set itself also has two hard requirements. First, retain hidden tests: the reflector can see the visible tests during correction; if evaluation uses only visible tests, the reflector will overfit to those tests—it learns to make these particular cases pass rather than truly fixing defects; hidden tests are specifically used to check whether the fix generalizes. Second, include control cases where “the original answer is correct”: if the evaluation set contains only wrong answers, the wrong-change rate cannot be observed at all; only by including originally correct samples can excessive modification be measured—a reflector that cannot resist modifying an answer even when the original answer is correct will be exposed on such samples. Thus the input to reflection evaluation is four same-budget schemes, and the output is a set of metrics: final success, first-round correct-to-wrong rate, marginal gain, number of rounds, fallback, and escalation; hidden tests guard against overfitting to visible acceptance, and control samples with correct original answers measure excessive modification. Only if it still wins under this set of comparisons and metrics is reflection not just multi-sampling under a different name.
9Connecting the Causal ChainSynthesis
Stringing the preceding stages together, reflection and self-correction form a six-step causal chain, each step providing the next step with the one thing it lacks. The chain starts from a problem—a candidate fails under external acceptance—and leads to verifiable practice: either leaving behind an evidence-constrained repair, or leaving behind an honest rollback or stop decision.
Step one, define protected acceptance. Before the candidate runs, clarify what counts as pass, what counts as regression, and what evidence counts. Acceptance includes both failure tests and protection tests; the former expose defects and the latter guard existing correct behavior. Without this step, none of the later “pass” outcomes can be discussed, because no grading standard exists. Step two, run the candidate and collect external failures. The candidate is handed to the validator for execution, yielding reproducible failure evidence—test output, compiler diagnostics, retrieved sources, environmental observations, or human rule judgments. This evidence is the first input in the entire chain that truly changes the information state; without it, everything that follows is just resampling within the same blind spot. Step three, compress the evidence into a causal hypothesis. The reflector does not restate the failure but distills a falsifiable mechanism from the evidence, such as a cache key lacking user_id causing cross-user value mixing; if the cause is not yet clear, perform discriminative experiments first rather than acting directly. Step four, apply a single reversible change. The patch should be small enough to attribute the retest result to the single change, and can be rolled back at any time; interdependent changes are split into a sequence with checkpoints rather than changing five places at once. Step five, rerun the failure and protection tests. Use the same validator to rerun the original failing test and confirm the failure disappears; then run protection tests to confirm that no previously correct parts are broken—the alternative of disabling caching is exactly what gets blocked at this step. Step six, stop based on marginal benefit and no progress. Each round compares the number fixed and the number broken, and stop is triggered by negative marginal net benefit, two consecutive rounds without new failure evidence, a patch oscillating between tests, validator self-conflict, or exhausted budget, with output of continue, rollback, report unknown, or human escalation.
Between the six steps there is a strict one-way dependency: acceptance is defined before running, evidence is collected before hypotheses, hypotheses are proposed before changes, retesting is performed after changes, and stop decisions are based on each round's marginal results. If any step is missing or reversed, the chain breaks: without protected acceptance, there is no trustworthy pass; without external failure, there is no true reflection; without a single change, there is no attribution; without retesting, there is no verification; without a stop condition, reflection continues to spin in place when benefits are negative. Finally, there is always a test hanging above the chain: that reflection outperforms multiple sampling must itself be proved, not assumed. Only when, under the same token, call count, and wall-clock budget, feedback-driven reflection significantly outperforms non-feedback resampling, and the first-round correct-to-broken rate, rollback rate, and hidden tests all pass, is the cost of this six-step mechanism proven worthwhile. The correctness of the causal chain is guaranteed by the evidence at each step, while its necessity is guaranteed by this layer of comparative testing—together, these two are the complete basis for reflection changing from “think again” to verifiable practice.
- Reflexion: Language Agents with Verbal Reinforcement Learning: Verbal feedback-based correction and experience memory
- Self-Refine: Iterative Refinement with Self-Feedback: Generation–feedback–revision loop
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing: Using external tool feedback to constrain criticism
- Large Language Models Cannot Self-Correct Reasoning Yet: The limits of self-correction without external feedback