Last time I promised an answer. My chunking pipeline had cut Rule 5.6 in half, the heading in one chunk and the substance in the next, and I wondered out loud whether that split would cost me at retrieval time or whether it was an ugly detail with no consequences.
It cost me. The evidence came from a direction I didn't plan for.
While preparing the experiment I'm about to describe, I generated synthetic test questions from my golf rules corpus. Five of them cited rule numbers directly, phrased like "Under Rule 16.3a, what relief is available to a player?" Those questions were created straight from the source chunks, so they should have been the easiest possible test. All five scored zero. The retriever couldn't find the correct content because the rule number was in the heading chunk, whereas the answer was in the chunk after it. The chunk boundary trap from my last post showed up in my own data, scored as a plain zero in my results table.
That finding was a side effect, though. The experiment itself was about something bigger, and its results changed how I'll listen to every RAG accuracy claim from now on.
The exam the model wrote for itself
A question had been bothering me since I finished tagging ground truth for my golf rules assistant. When teams evaluate a RAG system, the standard shortcut is to generate test questions with an LLM. You feed it your documents, ask for questions those documents can answer, and run the set against your retriever. It's fast, it's cheap, and most tutorials recommend it.
But look at what actually happens. The model reads a paragraph and writes a question about it. Naturally, the question reuses the vocabulary of that paragraph. I started calling this the vocabulary echo: a question generated from a paragraph inevitably echoes the paragraph's own words. The retriever works by comparing the words in a question against those in the stored chunks. So the test and the answer key share vocabulary by construction. The system is graded on an exam it wrote for itself, open book.
I wanted to know how much that inflates the score, not from a paper, but from my own pipeline.
Before the numbers, a word about what this experiment is and what it isn't. Nineteen questions per set, a hobbyist pipeline, a corpus of golf rules I know well enough to check every answer by hand. It proves nothing about RAG in general. I built it to learn where retrieval numbers come from and how easily they bend. What follows is a report from that road, not a benchmark.
Four question sets, one variable
The design is the part I'm most satisfied with, because it isolates a single variable: phrasing.
I started at the human end. I wrote nineteen questions based on real situations from my own rounds and asked a few golfing friends what they would want to ask a rules assistant. None of them had read the rulebook PDFs. That's the point. One friend's question came in almost word-for-word: "My ball landed in a red-staked area near a pond, what now?" This became Set D, my gold standard.
Then I generated three synthetic siblings for each human question. For every question in Set D, I took the chunk containing its answer and asked an LLM to generate a matching question from that same chunk in three ways. Set A used the naive prompt most teams use: generate questions this text can answer. Set B added a persona: you're a weekend golfer, use casual language, don't reuse wording from the text. Set C went adversarial: take the Set A question and rephrase it with completely different vocabulary.
Same corpus. Same retriever. Same ground truth chunks for every question family. The only thing that changes between sets is how the question is worded.
To see why wording matters, put two questions side by side. The rulebook devotes a section to "loose impediments" and the relief available when your ball lies near one. A synthetic question asks: "What restrictions apply to loose impediments?" My human question asks: "Can I move a loose twig that is right next to my ball?" Same rule, same answer chunk. The rulebook has never heard of a twig.
The pipeline underneath is deliberately vanilla: PDFs are split into chunks of roughly 500 tokens, embedded with a small open-source model, stored in a local vector database, and retrieved via cosine similarity, returning the top 5 chunks. No reranking, no hybrid search, no query rewriting. I kept it plain on purpose, so that any differences in the results would be attributable to the questions, not to pipeline sophistication.
Two metrics matter here. Precision@5 asks: Of the five chunks retrieved, how many did I tag as actually containing the answer? MRR (mean reciprocal rank) asks: how high on the list did the first correct chunk appear? I also computed recall, but my broad ground-truth tagging suppresses it across all sets equally, so I'll leave it out.
What the numbers said
Question set | Precision@5 | MRR |
|---|---|---|
A: naive synthetic | 0.232 | 0.440 |
B: persona-prompted | 0.095 | 0.210 |
C: adversarial | 0.021 | 0.079 |
D: human-written | 0.137 | 0.247 |
Start with the comparison the experiment was designed for: Set A against Set D. Naive synthetic questions scored 0.232 precision, compared with 0.137 for human questions. You can state that gap in two ways. I'll give you both, because the framing itself is a lesson in honest measurement. In relative terms, synthetic questions scored about 70% higher. In absolute terms, the difference is 9.5 percentage points. Both numbers are true. A vendor would pick one. You should ask for both.
One methodological wrinkle actually works against the synthetic set. In plain terms, I graded the synthetic questions with a stricter answer key than they were written for. Each synthetic question was generated from a single source chunk but evaluated against all the chunks I had tagged for its human twin, sometimes ten or more. Set A was held to a broader standard than it was designed for and still came out 70% ahead. The real gap is likely wider than my numbers show.
The vocabulary echo turned out to be real and measurable. Lexical overlap between a question and its source chunk correlated with precision across all four sets, pointing to a mechanism rather than a coincidence: the more words a question borrows from the document, the better retrieval performs.
But the number that actually stopped me wasn't the gap. It was this: thirteen of my nineteen human questions scored zero. The retriever found nothing useful in the top five results for most of the questions real golfers would ask.
The failures follow one pattern. My friends and I ask about twigs; the rulebook says "loose impediments." We say "red-staked area near a pond"; the rulebook says "penalty area." We say the ball is "stuck half in the ground"; the rulebook says "embedded ball." One question used the term "water hazard," which every golfer of my generation grew up with, and scored zero because the 2023 rules retired that term. The six human questions that worked all happened to contain rulebook terms: "out of bounds," "provisional ball," and "ball marks."
Here's what hit me: I had the story backwards. I went in expecting to catch synthetic questions that would flatter a decent system. What I found was synthetic questions hiding a failing one. My retriever performs vocabulary matching, refined and fast, but vocabulary matching is all the same. When the words align, it shines. When a human shows up with human words, it fails more often than it succeeds. The synthetic score didn't exaggerate the truth so much as describe a much easier test.
The cheap fix made things worse.
Everything above has a well-known remedy, at least according to conventional advice: prompt the question generator with a persona. Tell it to write like a real user, ban it from reusing the source wording, and your synthetic questions become realistic. That was Set B, and I expected it to land somewhere between the naive set and the human set.
It landed below the human set, closer to the adversarial one. Precision of 0.095 against the human 0.137, with the deliberately hostile Set C at 0.021.
Reading the individual questions explains why. The persona prompt produced a wide spread. Some questions kept enough concrete detail to work. Others drifted into vagueness so complete that no retriever could help, long rambles about "bringing some extra gear to help me play better" that never named anything the corpus could match. The persona didn't imitate a human asking a real question. It imitated the texture of casual speech while losing the intent underneath.
So the cheap fix is unreliable, at least in the naive form I tested. That result matters to me more than the headline gap, because it breaks the comfortable narrative. If persona prompting had worked, the lesson would be a one-line prompt change. It didn't, and the lesson became harder: on this evidence, there's no shortcut around collecting questions from actual humans who haven't read your documents.
The question I'll bring into every review
I've sat in enough reviews of dashboards and automation projects over the years to recognise the shape of this problem. Systems tend to get measured in ways that are convenient for the people building them. BI reports counted what was easy to count. Test scenarios for automation covered the happy path. Synthetic RAG evaluation is the same pattern applied to a new technology, and it's now standard practice in most RAG tooling.
So here's what changes for me, and what I'd offer to anyone who reviews or funds these systems. The next time someone presents a retrieval accuracy number, I'll ask two questions before reacting to the number itself. Who wrote the test questions? And had they read the documents first? If the answer is "we generated them from the corpus," the number is real, but it describes an easier world than the one your users live in. My own data suggests the honest number may sit far below it.
I owe you the limitations, in the same spirit. My embedding model is small, and I run a pure similarity search with no reranking or query rewriting. A production-grade pipeline would close part of the vocabulary gap, and I genuinely don't know how much. That's a future experiment, and I'm looking forward to it. What survives any pipeline upgrade is the relative comparison, because the gap between question sets measures the test data, not the retriever.
One more thing waits in this dataset. There's a single failed question I keep returning to, about practice swings, where the retriever confidently answered a completely different question than the one asked. It deserves its own post because it exposes a limitation that no accuracy metric captures.
Building this evaluation taught me more about RAG than any course I've taken, and the biggest lesson wasn't technical. It was how easily I could have fooled myself, with my own pipeline, my own questions, and every incentive to believe the higher number. Next time someone shows you an accuracy score, will you know to ask who wrote the exam?
Wojciech Pozarzycki, July 2026