“We just call the embedding API and drop the vectors in a store.” That sentence explains most of the agent-retrieval failures we get called in to look at. Vectorization is not a one-line call — it is a pipeline of decisions, and each one silently sets a ceiling on how relevant your agent’s retrieved context can ever be. The problem rarely shows up in a demo. On 100 documents with default settings, almost any embedding model and any chunking scheme returns something plausible, because the corpus is small enough that even a mediocre match is close enough. The gap opens at production scale. At millions of vectors, the wrong chunk boundaries and the wrong embedding model degrade recall in ways that no orchestration layer can repair — because the relevant context was never retrievable in the first place. Teams then blame the agent framework for “flakiness” when the real defect sits three steps upstream. What does data vectorization actually mean in practice? Vectorization is the process of turning a piece of content — a paragraph, an image, a row of structured data — into a fixed-length array of floating-point numbers that encodes its meaning in a way a machine can compare. An embedding is that array. Two pieces of content with similar meaning end up close together in the vector space; unrelated content ends up far apart. Retrieval then becomes a geometry problem: given a query vector, find the nearest stored vectors. That framing is deceptively simple, and the simplicity is where teams get hurt. “Turn text into a vector” hides at least five independent decisions, each of which shapes retrieval quality: Chunking — how you split source content before embedding it. Embedding model — which model produces the vectors, and what it was trained to represent. Dimensionality — how many numbers each vector carries. Normalisation — whether vectors are scaled to unit length and which distance metric you use. Index selection — the data structure that makes nearest-neighbour search tractable at scale. None of these has a universally correct answer. Each is a trade-off against your corpus, your latency budget, and your storage cost. The naive one-line call accepts every default silently, and the defaults were chosen for a generic benchmark, not for your documents. How do chunking and embedding model choice affect retrieval quality? Chunking decides what unit of meaning gets a vector. Embed an entire 40-page document as one vector and you get a blurry average that matches almost any query weakly and no query strongly. Embed every sentence in isolation and you shatter the context — a chunk that says “this reduced the failure rate” is useless when “this” refers to a change described two sentences earlier. The useful middle ground depends on how information is distributed in your corpus, which is why we treat chunking as a corpus-specific experiment rather than a setting to copy from a tutorial. The pattern we see repeatedly (an observed pattern across our generative-AI engagements, not a benchmarked rate) is that recall failures blamed on “the model not understanding the question” turn out to be chunking failures: the answer existed in the corpus but was split across two chunks, so no single retrieved vector carried the whole fact. Fixing chunk boundaries — respecting section headers, keeping tables intact, adding a sentence of overlap — often lifts groundedness more than swapping the embedding model does. Embedding model choice is the second lever. A model trained mostly on general web text will represent a legal contract or a genomics protocol coarsely, collapsing distinctions that matter to your users. Domain-mismatched embeddings produce vector spaces where semantically different chunks sit deceptively close, and no amount of index tuning recovers the lost separation. This is the same failure class we describe in the data-centric approach to AI and what it means for feasibility: the quality of what you feed the system bounds everything downstream. Before locking a model, measure recall@k against a labelled query set drawn from your own domain — that is a benchmark-class measurement on your data, and it is the only one that predicts production behaviour. It is worth separating vectorization from the tokenization step that precedes it inside the embedding model. Tokenization splits text into the units the model reads; vectorization is what the model emits. If you want the upstream half of that story, we cover it in how text becomes tokens for generative models. This article is about the vectors that come out and what happens to them. Why do vectorization defaults degrade at millions of vectors? At small scale, exact nearest-neighbour search over every stored vector is cheap, so retrieval quality equals embedding quality — the search itself is perfect. The demo works because the geometry problem is trivial when there are only a few thousand candidates. At millions of vectors, exact search over the full set becomes too slow to hit an interactive latency budget, so you switch to an approximate nearest-neighbour index. Approximation trades a small, tunable amount of recall for a large speed gain. That is a reasonable trade — but it introduces a second recall ceiling on top of the embedding one, and the two compound. If your chunking already loses 15% of retrievable facts and your index configuration drops another 10%, the agent sees a corpus that is quietly a quarter less useful than the raw data suggested. Scale also exposes decisions that were harmless when small. Dimensionality that was affordable at 10,000 vectors becomes the dominant storage and query-latency cost at 10 million. A distance metric mismatch — cosine similarity assumed but Euclidean distance used, or vectors never normalised — barely registers on a tiny set and produces systematic ranking errors on a large one. The through-line is that vectorization defaults are validated at demo scale and silently fail the assumptions of production scale, which is exactly why we treat retrieval quality as something to measure before the agent framework is chosen, not after. How do dimensionality and index type trade off precision against latency and cost? The two knobs that most directly set the cost-and-latency floor are vector dimensionality and index type. Higher dimensionality can capture finer semantic distinctions but multiplies storage and slows every distance computation. Index type governs how much of the corpus you actually compare against per query. Here is the trade space we walk clients through: Index type How it searches Recall behaviour Query latency at millions of vectors When it fits Flat (exact) Compares the query against every vector Perfect recall by construction Highest — scales linearly with corpus size Small corpora, or offline batch retrieval where latency is not user-facing HNSW (graph) Navigates a layered proximity graph High recall, tunable via graph parameters Low, but high memory footprint to hold the graph Interactive agents needing high recall and able to pay the RAM cost IVF (inverted file) Partitions vectors into clusters, searches a few Recall depends on how many partitions you probe Low, more memory-efficient than HNSW Very large corpora where memory is the binding constraint The table is not a ranking — the right choice is context-dependent. HNSW typically buys higher recall at a fixed latency at the cost of memory; IVF typically trades a little recall for a smaller footprint. For a deeper walk through the partitioning mechanism, we cover how the inverted file index speeds up vector search in its own right. The point for an architecture decision is that dimensionality and index type together set your retrieval precision at a fixed latency budget — and that single number, measured on your corpus, is more decision-relevant than any vendor’s headline benchmark. How does vectorization quality feed into an agent framework’s retrieval, and how do I evaluate it first? Agent frameworks lean on a vector store for state persistence and long-term memory: the agent retrieves relevant past context to ground its next step. That retrieval is only as good as the vectorization pipeline behind it. If chunking, model choice, and index configuration are wrong, the agent’s memory layer returns off-topic context, and the framework’s orchestration — however sophisticated — faithfully reasons over the wrong material. This is the mechanism behind “the framework is flaky”: the flakiness is inherited from upstream vectorization the framework has no control over. The practical consequence is a sequencing rule. Evaluate retrieval quality before committing to a framework, because it is measurable independently of any framework and it is the precondition for the framework’s memory layer being production-ready. A short evaluation checklist we use: Build a labelled query set from your own domain — real questions with known correct source chunks. Measure recall@k — of the top-k retrieved chunks, does the correct one appear? This is a benchmark-class number on your data. Measure answer groundedness — with the retrieved context supplied, does the generated answer cite it correctly, or hallucinate? Sweep chunking and model choices against the same query set before touching the index. Then sweep index type and parameters to find the recall achievable at your latency budget. Record retrieval precision at fixed latency — the single figure that lets you compare framework candidates on equal terms. This evaluation is also the input to a feasibility judgement. Deciding whether a candidate agent framework’s memory and retrieval layer is production-ready for your use case depends on knowing your achievable retrieval precision first — which is why we treat vectorization quality as a supporting precondition in a feasibility assessment rather than an implementation detail deferred to build time. If you are choosing the surrounding infrastructure at the same time, how to choose an MLOps platform for agentic and generative workloads covers the operational side of the same decision. The broader engineering context for all of this sits under our generative AI practice. FAQ How does data vectorization work? Vectorization turns content into fixed-length arrays of numbers (embeddings) that place similar meanings close together in a vector space, so retrieval becomes a nearest-neighbour search. In practice it is not a single API call but a pipeline of decisions — chunking, model, dimensionality, normalisation, and index — each of which caps how relevant retrieved context can be. What is an embedding, and how does vectorization turn text, images, or structured data into vectors? An embedding is the fixed-length numeric array a model emits to represent a piece of content’s meaning. Vectorization runs that content — a text chunk, an image, a structured record — through a model trained so that semantically similar inputs map to nearby vectors and unrelated inputs map far apart, making similarity comparable as distance. How do chunking strategy and embedding model choice affect retrieval quality in an agent’s memory layer? Chunking decides the unit of meaning that gets a vector; boundaries that split a fact across two chunks make it unretrievable regardless of the model. The embedding model decides how faithfully your domain is represented — a domain-mismatched model collapses distinctions your users care about. Both set a recall ceiling upstream of any orchestration. Why do vectorization defaults that work in a demo degrade at millions of vectors, and what changes at scale? At small scale, exact search is cheap so retrieval quality equals embedding quality. At millions of vectors you must switch to an approximate index, adding a second recall ceiling that compounds with chunking losses, while dimensionality and distance-metric choices that were harmless become dominant cost and correctness factors. How do embedding dimensionality and vector index type (flat, HNSW, IVF) trade off retrieval precision against latency and cost? Higher dimensionality captures finer distinctions but raises storage and per-query cost. Flat gives perfect recall but scales linearly in latency; HNSW gives high recall at a fixed latency at the cost of memory; IVF trades some recall for a smaller footprint. Together they set your retrieval precision at a fixed latency budget. How does vectorization quality feed into an agent framework’s state persistence and retrieval, and how do I evaluate it before committing to a framework? An agent’s memory layer retrieves from the vector store, so its quality is inherited entirely from the vectorization pipeline. Evaluate it first, framework-independently: build a labelled query set from your domain, measure recall@k and answer groundedness, sweep chunking and model, then index parameters, and record retrieval precision at your latency budget. Where this leaves the framework decision The uncomfortable implication is that a lot of agent-framework evaluation is measuring the wrong layer. If two frameworks retrieve from the same badly vectorized store, they will both look flaky, and if they retrieve from a well-vectorized one, the memory layer will look competent regardless of the orchestration around it. Fix the retrieval geometry first, measure precision at your latency budget, and only then ask whether a framework’s memory layer is production-ready — that ordering is the difference between a feasibility assessment that predicts production and one that flatters a demo.
