Early natural-language processing treated every word as its own isolated token, with no relationship to any other. “Cat” and “kitten” were as foreign to each other as “cat” and “parliament.” Train a model to predict which words tend to appear near which other words across a large corpus, and the network learns a different arrangement: words used in similar contexts end up near each other in the space the network uses internally. Those positions in the learned space are what the field calls embeddings.
How embeddings are produced
Mikolov et al. introduced the method that made this practical at scale with their 2013 Word2Vec paper. They trained a shallow neural network to predict surrounding words from a given word, or vice versa, forcing the network to compress meaning into dense vectors (Mikolov et al., 2013). A vocabulary of 100,000 words, which would otherwise occupy a 100,000-dimensional space with one slot per word, could be represented in 300 dimensions, with meaning encoded geometrically in the positions.
The model never directly learns “cat and kitten are similar.” It discovers the relationship because they appear in similar contexts across millions of sentences. The vectors end up close not because anyone specified it, but because the training signal (predict the neighbours) pushed them there.
Modern embedding models follow the same intuition at larger scale, encoding not just individual words but full sentences or entire documents into single vectors.
What the numbers mean
Each dimension in an embedding vector does not correspond to a single human-legible concept. Meaning is distributed across all dimensions simultaneously. It is not that dimension 47 means “animal” and dimension 112 means “domestic.” The representation is holistic. What matters is the geometry: the direction a vector points relative to other vectors in the space.
Similarity between two embeddings is typically measured by cosine similarity, specifically the cosine of the angle between the two vectors. A score near 1 means the vectors point in the same direction (similar meaning); near 0 means unrelated. Cosine is preferred over raw distance because it is scale-invariant: a one-sentence document and a ten-page document can represent the same idea and should compare as similar even if their raw magnitudes differ.
The geometry has useful algebraic structure. In a Word2Vec space, the vector arithmetic “king - man + woman” lands close to “queen”, not because anyone hard-coded that relationship, but because the training data reflected it (Mikolov et al., 2013). The model learned a consistent axis for the concept of gender without being told one existed.
Embeddings are not only for text
The training intuition (put similar items near each other) applies beyond text. Images can be embedded: a neural network trained on a large image dataset encodes each image as a vector where visually or semantically similar images cluster. Audio and code follow the same logic.
Models like CLIP, trained on image-text pairs, learn a shared embedding space across data types: an image of a beach will land close to the text phrase “sunny beach,” even though the two are different modalities entirely. This is what makes multimodal search possible: finding the image that matches a text description, or the code that implements what a plain-English query describes.
How embeddings connect to search and RAG
Embeddings make semantic search possible. A keyword search looks for exact term overlap; a semantic search converts both the query and every candidate document to embeddings and retrieves the candidates with the closest vectors. “How do I reduce my tax bill?” will surface documents about tax deductions even if none of them contain the phrase “tax bill.”
This retrieval mechanism is the foundation of retrieval-augmented generation. RAG builds a library of pre-embedded document chunks, stores them in a vector database, and at query time converts the user’s question to an embedding and fetches the nearest matches. Those matches are handed to the language model as context. The embedding comparison handles retrieval; the model handles reasoning.
The part most explainers leave out
Embeddings are not an intrinsic property of a piece of text. They are produced by a specific model, and a different model will produce different vectors for the same input. Two embedding models are not interchangeable. Embed your document library with one model and your queries with another, and the distance comparisons are meaningless; the space belongs to the model that generated it.
This is why retrieval systems commit to one embedding model for both indexing and querying, and why switching models later means re-indexing the entire library.
Sources: Mikolov et al., Efficient Estimation of Word Representations in Vector Space (arXiv:1301.3781, 2013).
Questions, answered
Are embeddings the same thing as vectors?
Yes. An embedding is a vector (a list of numbers) where position in a high-dimensional space encodes meaning. The two words are used interchangeably, though 'embedding' emphasises that the representation was learned from data rather than assigned by hand.
How is similarity measured between two embeddings?
The most common measure is cosine similarity, specifically the cosine of the angle between two vectors. A score near 1 means the vectors point in the same direction (similar meaning); near 0 means unrelated. Cosine is preferred because it is scale-invariant: a one-sentence document and a ten-page document can represent the same idea and still compare as similar.
Do embeddings work for things other than text?
Yes. The same training intuition applies to images, audio, and code. Models like CLIP, trained on image-text pairs, learn a shared embedding space across data types: an image of a cat will land close to the phrase 'a cat sitting on a mat,' even though the two are different modalities.
What is the connection between embeddings and RAG?
Retrieval-augmented generation uses embeddings as its retrieval step. The user's question is converted to an embedding and compared against a library of pre-embedded document chunks; the nearest matches are handed to the language model as context. The embedding comparison does the retrieval; the language model does the answering.
Brief is a team of AI associates you direct in plain language. Opening to a small group at a time.
Request access