Reranking: Making Fine-Grained Relevance Judgments Among High-Recall Candidates

The fundamental contradiction of retrieval systems is that the strongest model can make the most fine-grained relevance judgments, but having it interact one by one with all documents in the database is unacceptably costly. Two-stage retrieval is a division-of-labor scheme designed for this contradiction—the first stage uses cheap vector retrieval or keyword matching to quickly filter over the entire collection, guaranteeing only high recall: better to retrieve too many than to miss documents that may contain the answer; the second stage applies a stronger model (such as a cross-encoder or a Large Language Model (LLM)) only to this greatly reduced candidate set, performing fine-grained pairwise ordering. Speed comes from the first stage, precision comes from the second stage, and both share the same query, the same full collection, and a latency budget.

The inputs to the whole pipeline are the query, the complete document collection, and the latency budget; the outputs are the high-recall candidate list from the first stage and the top-ranked passages sorted by the reranker. The first stage uses cheap representations to shrink the set to a size that can withstand fine-grained comparison; the reranker works only within this set and never goes outside the candidate set to re-compare against the full collection. Therefore, if the correct evidence never entered the candidates, the problem lies in the recall stage, not the reranking stage—the reranker by definition cannot see documents outside the candidate set, so blaming it for missing them is meaningless. Only when the full collection is small enough to bear one-by-one fine-grained comparison against all documents can the first stage be omitted, letting the strong model directly face the entire collection.

Another consequence of this division of labor is that when a problem occurs online, the failure type must be distinguished first; otherwise repair directions will cancel each other out. Diagnosis requires saving the candidate IDs before and after reranking, first-stage scores, reranking scores, document versions, and the evidence ultimately cited. There are three failure points of different natures along this chain: the correct evidence is not among the candidates—that is a recall failure, and what needs fixing is the first-stage retrieval representation or index; the evidence is among the candidates but still gets ranked lower—that is a ranking failure, and what needs fixing is the reranker itself; the evidence is ranked into the context sent to generation but ultimately not cited—that belongs to an assembly or generation failure and is unrelated to the reranker. Conflating the three and blaming the reranking model entirely will lead to ineffective changes to unrelated components.

When used across domains, one must also guard against overall drift in absolute scores: on data distributions from different domains, the relevance scores output by the same model may be systematically inflated or deflated. In this case, a fixed threshold calibrated for one domain cannot be directly transferred to another domain; it must be recalibrated according to query type, otherwise systematically too many or too few items will be retrieved when truncating the top results by threshold.

Rerankers are generally more accurate but slower than first-stage representations, because the two encoding methods differ in interaction granularity. A two-tower model feeds the query and the document into separate encoders, each producing a vector before similarity is computed: because document vectors can be precomputed offline and stored in the index, only the query needs to be encoded online, so it is cheap enough for corpus-scale recall; the cost is that the query and document never meet during encoding, their representations do not influence each other, and interaction happens at the compressed vector level, where fine-grained semantics can be lost. A cross-encoder, by contrast, concatenates the query and document and feeds them in together, allowing every token of the query to interact fully with every token of the document in the attention mechanism, so relevance judgments are naturally finer-grained; the cost is that every candidate document must go through a complete forward pass, and the cache cannot be reused once the candidates change, so it can only be used in the reranking stage after the candidate count has been greatly reduced.

The value of this token-level interaction is most evident in scenarios that require precise logical judgment: conditions, numbers, negation, and entity relations. For example, "quality issues are not subject to the 30-day limit" and "all goods are subject to the 30-day limit" share a large amount of vocabulary; a two-tower vector may judge them as highly similar, whereas a cross-encoder can see the combination of negation structure and numerical constraint, thereby distinguishing the two opposite rights states. This is exactly why the reranker is placed in the second stage: the first stage is responsible for pulling in relevant documents, and the reranker is responsible for ranking, within this small set, the ones that truly satisfy the query constraints.

Architecture selection is a constrained trade-off problem. The input is the query, candidate documents, the required interaction granularity, and the computational budget; the output is either a two-tower similarity score or a cross-encoder relevance score. When the budget is sufficient and candidates are few, choose a cross-encoder in exchange for precision; when the budget is tight or candidates are very numerous, fall back to a two-tower model. Whichever is chosen, there is a common boundary: model input length is finite, and a long document may, at the position where it is truncated, happen to lose the key sentence that carries the answer. In this case, a low score reflects "the portion that was seen is not relevant to the query" rather than "the full document is not relevant". When processing long documents, you should choose summarization, a sliding window, or extraction of key passages to construct the input according to the document structure, rather than truncating the whole document by default. Another boundary is that a high score only means that the document is judged relevant under the model's training objective; it does not prove that the document's source is authentic or that the content is correct—relevance judgment and fact-checking are two different things.

The reranking stage must answer a seemingly simple question: exactly how many of the first-stage top results should be passed to the reranker. Taking top-20 versus top-200 represents two opposing tendencies. Increasing k does improve the probability that the candidate set contains the correct evidence—although first-stage ranking is coarse, the correct evidence usually falls within some depth range, and the larger k is, the greater the chance of bringing this evidence in; at the same time, reranking cost grows approximately linearly with k because the cross-encoder must perform a forward pass for each candidate, and deeper candidates are themselves less relevant to the query, bringing in more noise than gain. The choice of k is about striking a balance between "the probability of capturing the correct evidence" and "computation cost plus noise".

Decision-making cannot rely on intuition; it must be driven by data. There are three inputs: the first-stage Recall curve, i.e., the proportion of correct evidence included at different k; the cost of reranking a single candidate; and the context budget that the downstream generation stage can accept. The output is the actual reranking depth k. The way to judge is to observe the Recall@k curve separately for each query bucket—different query types have correct evidence distributed at different depths, and mixing them together obscures the differences. When the curve has not yet saturated, increasing k yields real recall gains; when the curve has flattened, continuing to increase k only linearly increases latency and floods the reranker with more query-irrelevant candidates, and the top rankings may also be perturbed by noise.

Here is a prerequisite-style causal chain: the reranker can only reorder the set handed to it by the first stage and can never conjure evidence beyond the candidates out of thin air. Therefore, when Recall@k itself is low, using a stronger reranker is meaningless—the correct evidence is not even in the set, and no matter how precise the ranking, it cannot be ranked into existence. The correct action at this point is to fix first-stage recall: improve vector representations, supplement keyword indexes, or adjust the retrieval strategy, and only after raising Recall@k to a reasonable level should you then discuss reranking depth and reranker selection. Conversely, if Recall@k is already high but final results are still poor, the problem truly lies in the reranking or downstream stages.

Both two-tower and cross-encoder models sacrifice something: two-tower models sacrifice token-level interaction for speed, and cross-encoders sacrifice precomputation for interaction. Late interaction attempts to preserve both, and its representative implementation is ColBERT. It encodes the query and document each into a sequence of token-level vectors—each document token has its own vector, rather than the entire document being compressed into one—and these document token vectors can be precomputed offline and stored; when a query arrives online, each query token vector is matched locally against each document token vector and aggregated into a relevance score through MaxSim: for each query token, take the maximum similarity between it and all document tokens (that is, the strongest matching position of that query term in the document), then sum the maximum similarities across all query tokens. Thus, between query tokens and document tokens, it preserves the information of "which term matched which part of the document" that two-tower models completely lack, while the document side can still be precomputed, avoiding the cost of full forward passes per candidate in cross-encoders.

Thus, the input to Late interaction is the token-level vectors of the query and the document, and the output is a relevance score aggregated via MaxSim, positioning it exactly between two-tower and cross-encoder models: higher accuracy than two-tower models and faster than cross-encoders. The cost is also direct—the storage unit expands from one vector per document to one vector per token per document, significantly increasing index size, and the computation and memory pressure during retrieval rises accordingly. This is a cost that must be budgeted for in scenarios with many documents or fine-grained tokenization.

Like the previous two architectures, the relevance score from Late interaction measures only "the degree of semantic match between query and document". It does not encode whether the document is still valid, whether the visitor has permission to view it, or whether the content has undergone fact-checking. Timeliness, permissions, and truthfulness must be guaranteed by independent mechanisms, and no relevance score should be treated as a substitute for these three.

When handing candidate ranking to an LLM, you must first choose how it outputs judgments, and then face the biases that come with that method. Inputs include the candidate documents, the order in which candidates are placed in the prompt, the scoring prompt itself, and the comparison method; outputs can be pointwise scoring—giving each candidate a score one by one—or pairwise comparison—deciding which of two is more relevant—or listwise ranking—having the model directly output a ranking of the entire candidate set. The three methods have different costs: pointwise scoring requires one call per candidate, the number of pairwise comparisons grows quadratically with the number of candidates, and listwise ranking handles all candidates in one call but judgment quality is affected by list structure. The real risk is concentrated in listwise ranking.

Listwise outputs exhibit several measurable biases. Primacy bias makes the model favor candidates placed near the top of the list, and even a different set of candidates filling the same positions may receive a boost; length bias makes longer documents appear more authoritative or more relevant; order bias means that with the same candidate set, merely changing the order in which they are input can change the model's ranking. In addition, candidates with prominent wording or neat formatting may also gain advantages unrelated to content relevance. To distinguish "the model really thinks A is more relevant than B" from "A just happens to be in a certain position or is longer", controlled diagnostics are needed: randomly permute the input order of the same candidates and run multiple times to see whether the ranking is stable; use sliding windows to compare in batches, or introduce pairwise comparison results for calibration. Only when the ranking remains stable under permutation can the change between two rankings serve as credible evidence; if the ranking drifts significantly after permutation, it indicates the output is contaminated by position factors and conclusions should not be drawn from it.

The cost of these diagnostics is a multiplied increase in model calls, and LLM inference itself is not cheap. Therefore, the premise for making LLM reranking the default approach is that the call cost is affordable and position bias has been calibrated or circumvented by design. In high-throughput scenarios where position bias is uncalibrated, per-candidate LLM reranking is both expensive and unreliable, and is usually not suitable to replace two-tower or cross-encoder as the first choice.

Judging whether reranking succeeds solely by ranking metrics such as NDCG misses most of the failures later in the pipeline. The inputs to end-to-end evaluation are not simply the candidate lists before and after reranking; they also include the evidence necessary to correctly answer the question, the citations finally provided by the generator, and whether the final task is completed. The outputs are a set of numbers reported side by side: ranking metrics such as NDCG and MRR, the coverage of whether critical evidence enters the context, whether citations are faithful to the cited documents, plus latency, cost, and task success rate. An improvement in ranking metrics only shows that the candidate order has become better under relevance annotations; it does not tell us whether the generator actually used the evidence that was moved to the front—evidence coverage and citation faithfulness must be measured separately, and task success must be judged independently.

"Relevant" and "credible" are not the same thing. The reranker optimizes the degree of match between query and document; a document that is highly on-topic for the question but factually wrong can perfectly well be ranked first. In this case ranking metrics may actually rise, because it happens to put the most "answer-like" item at the very front, and if the generator then answers based on it, the output is wrong. Therefore evaluation slices cannot be only about relevance: metrics must be observed separately by query type, document length, and freshness requirements, while also reporting latency and cost, so that a judgment can be made between precision gains and cost increases rather than seeing only an isolated NDCG number.

Launch gates must also cover content beyond ranking metrics: whether candidate document versions are correct, whether access permissions are satisfied, and whether answer slices are consistent with the evaluation set. Without these constraints, a reranking configuration with higher offline NDCG may still produce outdated, unauthorized, or irrelevant results online.

The first stage may already retrieve the correct material, but the answer can still be led astray by similar material ranked ahead—this is precisely the purpose of the reranking stage, which is clearest with a concrete example. The user asks whether a product's quality issue is subject to a 30-day deadline; the first stage retrieves four candidates. The two-tower model compresses the query and each document into independent vector summaries and then computes similarity, which cannot distinguish fine-grained conditional differences such as "quality issue exception" versus "all products are subject to the 30-day limit", so it produces this order according to recall rank:

The cross-encoder allows tokens such as "30 days""quality issue""not subject to deadline" to interact directly, and can raise the exception clause that actually answers the question from rank 3 to rank 1; while the "refund arrival time" originally ranked 1st shares only the "refund" theme with the query and has a completely different task intent, so it is demoted to rank 4. Measured by the reciprocal of the rank of the first relevant result, i.e., MRR = 1 ÷ rank: if relevance level ≥ 2 is considered relevant, before reranking the first relevant result is at rank 2, so MRR is 1/2 = 0.5; after reranking it moves to rank 1, so MRR rises to 1/1 = 1.0.

This 0.5 → 1.0 improvement only shows that "the first relevant result moved up"; all other conclusions require separate testing: although the outdated quality clause is textually relevant, it still needs independent recency filtering to be excluded; whether the generator uses both the general rule and the exception to compose a complete answer also depends on top-2 evidence coverage, version filtering, citation faithfulness, and final answer checks—MRR itself does not answer these questions.

The example also shows the constraint of the candidate set upper bound: if the first stage takes only top-4 and the quality issue exception is not among them, candidate Recall@4 is 0, and the best result of any reranking model can only be 0—reranking cannot create candidates out of thin air. Therefore, one should first draw the Recall@k curve, determine at what depth the correct evidence usually appears, and then decide the k for reranking. The input to this ranking example is the recall rank, human relevance level, and reranked rank of each of the four candidates; the output is the MRR before and after reranking and the top-2 evidence coverage, as a quantitative record of one local ranking change.

More candidates is safer, but online systems cannot always hand over all top-200 to the cross-encoder, because the growth in latency is real and predictable. Use a concrete assumption to calculate: a cross-encoder takes 35 ms to process a batch of 20 query–document pairs, and each reranking request also has 25 ms of fixed network and orchestration overhead. Then reranking 20 items requires only 1 batch, taking about 25 + 35 = 60 ms; reranking 100 items requires 5 batches, taking about 25 + 5 × 35 = 200 ms. Abstract this step into a formula: input fixed orchestration overhead T_fixed, per-batch time T_batch, batch candidate count b, and candidate count k, output reranking latency T_rerank, the number of batches is k divided by b, ceiling, so T_rerank = T_fixed + ⌈k/b⌉ × T_batch. Substitute the numbers: 20 items about 60 ms, 100 items about 200 ms. Batching reduces the average cost per candidate, but it cannot eliminate the total computation brought by increasing the candidate count—batching only pushes ⌈k/b⌉ closer to k/b, total time still grows with k.

Whether the latency is worth it should be compared against the coverage gained. If Recall@20 = 92% and Recall@100 = 96%, increasing candidate depth from 20 to 100 costs about 140 ms more, and the gain is only 4 percentage points of candidate coverage, and this part of coverage may not necessarily convert into improvement in the final answer. Following this line of thought yields a production decision table:

The correct order of production tuning is to first set the end-to-end latency SLO, then allocate the budget to recall, reranking, and generation stages, rather than letting each stage use up the latency on its own. Depth can also be adaptive based on query difficulty: precisely numbered queries can be satisfied by keyword retrieval, so reranking a small number of candidates is enough; highly ambiguous natural language queries deserve expanding candidate depth; high-risk questions should undergo fine ranking after version and authority filtering. The latency estimates in the table only apply to the given hardware and batch settings; changing devices or batch sizes requires recalculation.

The cross-encoder has an even more hidden failure boundary: it learns annotated relevance. If the training set mislabels "wording more like the question" as "can support the answer", the model will consistently rank textually fitting but missing key conditions passages to the front, and this bias is not easily exposed under a single metric. For new domains, long documents, negated sentences, and time-sensitive queries, one must separately slice and evaluate, and cannot reuse conclusions from generic collections. Whether extra Recall is worth it ultimately must be proven by whether it improves the final task.

Put reranking back into the entire retrieval chain: it is only one of six stages, where the output of each stage is the input to the next; if any stage breaks, the final answer will be wrong. The chain starts from the question: the first stage uses cheap representations to quickly recall candidates across the entire corpus, and this step only guarantees high recall without guaranteeing order; then you must check the Recall@k of the correct evidence—if the evidence did not enter the candidates at all, the best performance of all subsequent stages is capped at 0, and at this point you should go back and fix retrieval rather than adjust reranking. After Recall meets the target, the reranker performs fine-grained interaction within the candidate set, using token-level condition, numeric, and negation matching to rank the passages that truly answer the question ahead; then, according to latency and context budget, it truncates the top few items into the generation context; the generator cites these passages and gives an answer; finally, use retrieval metrics and answer metrics to jointly regress and verify whether the ranking improvement truly translates into evidence coverage, citation faithfulness, and task success.

Looking back at the decisions in each stage across the chapter along this chain, we can see that they constrain each other. The architecture choice (two-tower, cross-encoder, or Late interaction) determines the degree of fine-grained interaction possible and also determines the per-candidate computation and storage cost; the candidate depth k is jointly determined by the saturation point of the Recall@k curve and the reranking latency formula T_rerank = T_fixed + ⌈k/b⌉ × T_batch; evaluation must simultaneously report ranking metrics, evidence coverage, citation faithfulness, latency, cost, and task success rate, and enforce the slice gates for version, permission, and timeliness. An improvement in any single stage's metrics—whether MRR from 0.5 to 1.0, or an overall rise in NDCG—only indicates that stage became locally better; only by verifying along the causal chain stage by stage to the final answer can we confirm that a reranking change is truly effective.