Embedding

To determine whether two pieces of text are “similar in meaning,” a machine must first convert text into a numerical representation that can be computed and compared. Embedding is the method that completes this conversion: the input is a piece of text, the model first encodes the text, and outputs a continuous vector composed of hundreds to thousands of numbers. This vector is not arbitrary numbering, but the coordinates of the text in the semantic space learned by the model.

The key constraint of embedding is to make semantic relationships manifest as spatial relationships: texts with more similar meanings usually have closer vector distances; texts with greater meaning differences usually have vectors farther apart. Thus, the originally non-computable question of “whether meanings are similar” can be transformed into the computable question of “whether two vectors are close.” In the semantic map shown in Figure 1, similar content naturally clusters into groups. Although “how to return an item” and “product return process” have no common keywords, they fall in nearby positions because they express the same need—this is exactly the value of embedding relative to literal matching.

This semantic map is not a unified coordinate system shared by all models. Each embedding model forms its own representation space; the same text will get different coordinates when using another model. Therefore, only vectors in the same model's coordinate system are suitable for direct comparison; you cannot mix vectors generated by different embedding models and interpret distances.

That vector distances reflect semantic distance is not a coincidence; it is a geometric property continuously shaped by the training objective. The training data contains a large number of text sample pairs, some with similar meanings and some unrelated to each other. The model first encodes each piece of text into a vector, and then adjusts its parameters based on the known relationships between samples: vectors of similar samples are pulled closer together, while vectors of unrelated samples are pushed farther apart. As this process is repeated, semantic relationships are encoded into the structure of the entire vector space.

Therefore, “being close” is not a fixed coordinate artificially assigned to a particular word; it is the result of associations the model has learned under a large number of pairwise constraints. Two similar samples are repeatedly required to shorten their distance and eventually fall into neighboring regions; unrelated samples are separated from each other under training pressure. After the vector space is formed, distance can serve as a numerical signal for the degree of semantic relatedness.

Earlier word vectors embodied a simpler version of the same idea: words with similar contexts tend to have similar meanings. For example, around “cat” and “dog” one may see words such as “feed, raise, cute, pet store”; their contexts overlap heavily, so training causes the vectors of the two words to gradually move closer. Whether using sample pairs to directly pull together or push apart, or learning associations from shared contexts, “distance corresponds to semantics” is not an extra hard-coded rule but a result produced by the training process. This correspondence is also constrained by the training material and training objective: what the model can express is the semantic structure it has learned under these constraints.

Once text is represented as embedding vectors, the question “whether two passages are about the same thing” can be turned into a numerical comparison. A common metric is cosine similarity: it does not require the two texts to share the same vocabulary but instead compares the directions of two vectors. The smaller the angle, the more aligned the two vectors are, and the higher the similarity; the larger the angle, the more pronounced the difference in direction, and the lower the similarity.

Similarity = cos(θ) = (A·B) / (‖A‖ ‖B‖)

Here, A and B are the two embedding vectors to compare, θ is the angle between them, A·B denotes the dot product, and ‖A‖ and ‖B‖ are the lengths of the two vectors respectively. In computation, first the dot product measures the degree to which they agree in direction, then divide by the product of their lengths. The denominator removes the effect of vector length, so cosine similarity focuses on direction rather than how long the vectors themselves are.

The result lies between −1 and 1: the closer to 1, the more aligned the directions; near 0, the two vectors are nearly perpendicular; near −1, the directions are opposite. This number only describes similarity within the same embedding model space; vectors generated by different models cannot be compared across spaces, nor can the similarity be directly interpreted as the probability that an answer is correct.

“king − man + woman ≈ queen” is the classic demonstration of vector semantics. Here it is not just the position of a single point that matters; directions in the vector space also carry relationships: the direction of change “from man to woman” can be applied to “king,” bringing the result close to “queen.” This shows that semantics is not only placed into coordinates but also encoded into the geometric structure of the space.

Once semantics can be computed with vectors and similarity, retrieval and semantic search, vector databases, Retrieval-Augmented Generation (RAG), and clustering share a common foundation: either find the nearest neighbors to the current vector, or group vectors that are close to each other. They face different specific tasks, but all are built on the ability to “turn semantics into computable vectors.”

Using two-dimensional toy vectors, we can fully recalculate how cosine similarity turns "more aligned" into sortable scores. Let the query vector q = [1, 0]. The vector for the candidate "return goods" is a = [0.8, 0.6], and the vector for the candidate "train schedule" is b = [−0.6, 0.8]. These three vectors all have length 1, so the length products in the cosine similarity formula are all 1.

For candidate a, first compute the dot product:

q·a = 1×0.8 + 0×0.6 = 0.8

Then divide by the length product 1 to obtain cos(q, a) = 0.8. This relatively high positive value indicates that a is more aligned with the query direction, so "return goods" is more relevant among these two candidates.

For candidate b, compute similarly:

q·b = 1×(−0.6) + 0×0.8 = −0.6

After length normalization, we obtain cos(q, b) = −0.6. The negative value indicates that b is opposite to the query direction, so "train schedule" is less relevant in the model space. When sorting scores from high to low, a will appear before b. The angular relationships in the figure express the same thing as the calculation results: a has a smaller angle with the query, and b has a larger angle with the query.

Real embeddings usually have hundreds of dimensions, but the computation process remains the same: input one query vector and several candidate vectors, compute dot products separately, then normalize by their respective lengths, output each candidate's similarity and rank them accordingly. Cosine only compares direction, not length; in practical retrieval, vectors are often normalized to unit length first, in which case the dot product directly equals the cosine similarity.

High similarity only indicates that two objects are close in that model's vector space; it does not guarantee that a candidate is necessarily correct for a specific business task. The threshold used to filter results needs to be calibrated based on real queries, hard negatives, and labels; different models shape different spaces, so a 0.8 in one model and a 0.8 in another model do not have a unified meaning, and thresholds cannot be directly copied across models.

“Embedding” often refers to two different things: the input embedding layer inside a Transformer, and the embedding model used in retrieval scenarios such as RAG. They both take text and produce vectors, but they differ in where they sit, output granularity, and purpose; they cannot be substituted for one another just because they share the name.

The input embedding layer of a Transformer is part of the model architecture. After text is split into tokens, this layer generates a vector for each token, and then passes this set of vectors to the subsequent network for processing. Its output is not a single representation of the whole text, but a series of vectors corresponding to each token; its responsibility is to convert discrete tokens into input that the model can process internally.

By contrast, the embedding model in a retrieval context is an independently trained model. It takes an entire text, summarizes the information within it into one vector, and makes the overall semantics of different texts comparable and retrievable. What is needed here is a representation for each text that can be used for similarity computation, not just the internal vector of each token as it first enters the Transformer.

The difference between the two can be understood along the processing chain: the input embedding layer performs “token → vector for each token → subsequent network,” while the embedding model performs “entire text → one whole-text representation vector.” The former serves computation inside the Transformer; the latter serves comparison of whole-text semantics. When selecting a model, first confirm whether the task needs a per-token model input representation or a whole-text representation for retrieval; confusing the two concepts will directly lead to errors in model selection and implementation.

Keyword search relies on literal matching: the words in the query need to match the words in the document. When a user searches for “how to return an item,” a document that only says “product return process” is semantically related but may not be retrieved because there are no common keywords. Semantic search aims to solve exactly this kind of missed retrieval; it finds nearest neighbors by meaning rather than by literal form.

The input to semantic search includes a query and a document collection. The system first uses the same embedding model to convert the query and each document into vectors, then compares the distances between the query vector and document vectors, and outputs the document closest to the query. In this way, “return goods” and “return process” may be retrieved because their vectors are close in the semantic space even though the wording differs. Retrieving a document means the model judges that it is close in meaning to the query, not that it necessarily contains the original query terms.

This mechanism is also a key part of Retrieval-Augmented Generation (RAG). A common vector-retrieval RAG, before answering a question, uses embeddings to perform semantic search to find materials related to the question from a knowledge base, then passes these materials to a large language model to generate the answer; RAG can also adopt keyword, hybrid, or structured retrieval mechanisms and does not always rely on embeddings. The causal chain is: embeddings represent query and document → similarity retrieval selects materials → large language model answers based on materials. Embedding quality directly affects whether retrieval picks the right content, and retrieval results in turn determine what basis the answering stage can obtain, so retrieval quality often becomes the bottleneck of RAG effectiveness.

Semantic similarity is only a judgment in model space and does not necessarily mean correctness in a business sense. Retrieval quality depends on whether the embedding can accurately represent the semantic relationships in the current domain; even if the overall average retrieval performance is good, it cannot guarantee that every result is correct. Retrieval, vector databases, and RAG will continue to handle storage, nearest-neighbor search, and generation issues on this basis.

When embeddings are deployed, two types of problems are especially likely to distort results: mixing vectors from different models, and mistaking surface similarity for semantic consistency.

After changing the embedding model, you must rebuild the entire vector index. The same batch of texts processed by different models gets mapped into different vector spaces; even if the "cat" produced by model A and the "cat" produced by model B come from the same text, their coordinates are not directly comparable. If the query uses the new model while the document index still retains vectors from the old model, similarity computation faces two different coordinate systems, and the results become nothing but noise. Therefore, changing models means all documents must be re-embedded, the index rebuilt with the new vectors, and validation performed again on the same evaluation data.

Negative expressions expose the limits of similarity itself. "Suitable for children" and "not suitable for children" differ only by one negative word, and most of the literal content is the same, so their vectors may be very close, but the actual meanings are opposite. In such cases, high similarity reflects proximity in surface and topic, but cannot guarantee that the key judgment direction is consistent. For these hard negatives, where a small amount of text can reverse the entire meaning, you cannot rely only on vector similarity; you need to add rules, structured filtering, or reranking as a fallback.

Embedding vectors are usually high-dimensional. "High-dimensional" only means that a vector contains many coordinates; for example, a 768-dimensional vector describes a piece of text with 768 numbers together. As dimensionality increases, distances between samples may become closer and closer, making them harder to distinguish; this phenomenon is often summarized as the "curse of dimensionality." Trained embeddings are not random scattered points; they form structures related to the training task, so high-dimensional distances may still be meaningful. However, this effectiveness is not naturally guaranteed; it must be tested with real retrieval samples.

To let people observe the distribution of vectors with hundreds of dimensions, you can temporarily map them to two or three dimensions through dimensionality reduction. A dimensionality-reduction plot is a lossy projection; it is suitable for helping discover suspicious clusters and outliers, but it cannot prove the retrieval quality in the original high-dimensional space. "Clustering together" on a two-dimensional plot does not mean retrieval is necessarily reliable, nor can it replace task-oriented metrics.

Reliable validation should first build a small golden set of query–document pairs with relevance grades, then report Recall@k, nDCG, or MRR, and slice by negative expressions, numbers, proper nouns, language, and text length separately. If average recall is good but "not suitable for children" still retrieves "suitable for children", the problem should be located in hard negatives and ranking boundaries rather than being masked by the overall average. At this point, you should add structured filtering or reranking; after changing models, you should retest on the same golden set while regenerating all vectors and rebuilding the index.

Embedding ties together "how objects become vectors" and "how vectors produce retrieval results" into one complete causal chain. The starting point is that machines cannot directly compute text meaning, so text must be encoded into numeric vectors and semantic closeness made to manifest as spatial closeness. This mapping is not arbitrary numbering: training continually pulls samples with similar meanings closer together and pushes unrelated samples apart, ultimately making semantic relationships become the geometric structure of the vector space.

With this structure, the semantic relationship between two pieces of text can be turned into a numeric result through vector comparison. Cosine similarity converts the directional relationship between two vectors into a sortable score: the more aligned the directions, the higher the score, which usually indicates the model considers the semantics more similar. This score is only meaningful within the same model space; the threshold used for business filtering is also not a fixed constant, but must be calibrated against real task data.

From here to semantic search is only one step away. After queries and documents are converted into vectors by the same embedding model, the system can find the query's nearest neighbors by similarity. Matching is based on closeness in semantic space, not on whether they share common keywords, so "how to return an item" can retrieve "product return process." RAG first uses this step to retrieve relevant material from a knowledge base, then hands that material to a large language model to answer.

The choices in this chain together define "similarity": the training objective determines what relationships the vector space learns, the embedding model determines which coordinate system text lands in, the similarity calculation determines how directions are compared, and the task data determines how thresholds are calibrated. In implementation, you must also distinguish between the per-token input embedding layer inside a Transformer and the standalone embedding model that produces a single vector for a whole piece of text; semantic retrieval requires the latter.

A mismatch in any link of the chain will ruin the final result. Replacing the model while continuing to use the old index is equivalent to mixing two incomparable coordinate systems, so you must re-embed all documents and rebuild the index. When encountering negative expressions such as "suitable for children" and "not suitable for children," apparently high similarity may hide a meaning reversal, requiring rules, filtering, or reranking to cover the limits of raw vector similarity. Ultimately, the reason vector distance can carry semantics is that training shapes this geometric relationship; the reason documents without common keywords can be retrieved is that retrieval compares this relationship, not literal overlap.

Understanding embeddings requires a few foundational concepts. Vectors provide the numerical representation of text; dot product and angle explain how the directions of two vectors are compared; tokens and tokenization explain how text is split into units that a model can accept; neural networks are the basis for the model learning mapping relationships. With these prerequisites, semantic coordinates, cosine similarity, and the process by which training shapes the vector space can be connected into a coherent whole.

The core knowledge of embeddings centers on four relationships: how text is placed into semantic coordinates, how cosine similarity compares vector directions, why the Transformer's input embedding layer and standalone embedding models should not be conflated, and how semantic search uses vector nearest neighbors to recall content by meaning. These relationships together explain the complete path from text input to similarity scores and then to retrieval results.

Extension topics closely adjacent to embeddings include retrieval and semantic search, vector databases, RAG, clustering, the curse of dimensionality, and dimensionality reduction. Retrieval and semantic search focus on how to find similar content; vector databases handle the storage and lookup of vectors; RAG connects recalled material into the generation process; clustering groups similar vectors. The curse of dimensionality and dimensionality reduction help understand the difficulties of high-dimensional spaces and how to project high-dimensional structure in a lossy manner for observation.

Further outward, multimodal research studies how to bring different modalities into a shared embedding space, CLIP is an adjacent specific learning topic, and knowledge graphs provide another path for organizing and connecting knowledge. Learning along the hierarchy “prerequisite foundations → embedding core → nearest-neighbor applications and high-dimensional issues → cross-modality and knowledge organization” can keep the dependency relationships between concepts clear.