Retrieval-augmented generation (RAG) was conceived as an answer to a known limitation of language models: parametric knowledge is opaque, hard to update and prone to unsupported claims. The architecture proposed by Lewis and colleagues (2020) separates the two problems, entrusting a retrieval component with the selection of evidence and the model with its synthesis. The quality of the result therefore depends, decisively, on what retrieval delivers to the model.
This is where structured documents break current practice. A financial statement, a transport contract, a cooperative's bylaws are not uniform sequences of paragraphs: they are objects with a topology, made of section hierarchies, tables, notes, definitions and cross-references. Fixed-length segmentation, chunking, treats this topology as if it did not exist.
What is lost when you cut at fixed length
The simplest case is the table. In a balance sheet the cell «1,250» acquires meaning only together with its column header (the financial year), its row header (the line item), the declared unit (thousands of euros) and the explanatory note that comments on it, often pages away. A 512-token block that contains the cell but not the headers produces evidence that is formally pertinent and semantically mutilated; the model will read it with the same confidence as complete evidence, and the resulting error will be indistinguishable, on the surface, from a correct answer.
Cross-references present the same problem in a different form. «See note 12» is null information if note 12 lies in another block; a contractual clause read without the definition that precedes it can change meaning, because in legal texts definitions are binding rather than decorative; a table split in half produces rows orphaned of their columns. In all these cases the degradation does not happen in the model but before the model: it is a problem of input representation, not of generative capability.
The retrieval unit is a logical unit
The first correction is to analyse structure before indexing. Document layout and structure analysis is a mature field, with established benchmarks and tools: the point is not technical feasibility but the architectural choice of spending work at ingestion time. Concretely: sections and subsections are preserved as a hierarchy; tables remain tables, and every row travels with its own headers; notes are linked to the point where they are referenced; definitions are attached to the clauses that use them.
The second correction is to index metadata together with content: reference period, unit of measure, currency, source, document version. A general principle follows: the retrieval unit is a logical unit of the document, not a quantity of characters. Well-formed evidence must stand on its own, that is, be interpretable without the reader, human or artificial, having to reconstruct the missing context.
Hybrid retrieval, in the right order
On retrieval proper the literature offers two families of methods: lexical retrieval, of which BM25 remains the reference representative (Robertson and Zaragoza, 2009), and dense retrieval over learned representations (Karpukhin et al., 2020). Semantic search alone, however, is not enough, for a reason operational experience makes obvious: similarity between texts is not pertinence. A question about the 2023 bad-debt provision closely resembles a paragraph about the 2021 bad-debt provision, and dense retrieval, by itself, has no way to prefer the right one.
A robust system combines similarity with deterministic filters (document type, financial year, section), with section classification and with relations between entities: the line item, the note that explains it, the annex that details it. The order of operations matters: first the candidate set is narrowed with filters, then similarity search runs inside the subset. Inverting the order means letting evidence from the wrong year, very similar to the question, overtake the correct one.
Evaluation follows architecture
If evidence is the limiting factor, evaluation must measure evidence. Generation metrics (fluency, perceived coherence) are weak indicators, because they reward convincing answers regardless of their grounding. The pertinent measure is retrieval precision against a set of annotated ground-truth items: for this question, on this document, the correct evidence is this cell, this note, this clause. It is a costlier evaluation to build and a harsher one in its results, and it is the only one that predicts the system's behaviour in production.
A rule of conduct for the system follows: when evidence is missing, say so. A system that completes anyway, drawing on the model's parametric knowledge, produces exactly the class of errors the RAG architecture was born to eliminate.
The applied context
In Analytiko™ products these choices translate into two coexisting indexes, one structural and one semantic, into deterministic filters applied before similarity and into the mandatory citation of the source next to every returned figure. The cost is a more complex ingestion pipeline to maintain; the benefit is that every answer can be verified by tracing back to the source document, section and cell. When the document is a financial statement or a contract, verifiability is not an accessory property: it is the condition for the system to enter a process someone answers for.
References: Lewis P. et al., «Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks», NeurIPS 2020 · Robertson S., Zaragoza H., «The Probabilistic Relevance Framework: BM25 and Beyond», Foundations and Trends in Information Retrieval, 2009 · Karpukhin V. et al., «Dense Passage Retrieval for Open-Domain Question Answering», EMNLP 2020.
Analytiko · 2 September 2026