2WikiMultiHopQA Explained: Multi-Hop Reasoning Benchmarks in a Task-Specific LLM Eval

What 2WikiMultiHopQA tests, how far its score predicts your own RAG task, and which shortcut and contamination limits to disclose in an LLM eval.

2WikiMultiHopQA Explained: Multi-Hop Reasoning Benchmarks in a Task-Specific LLM Eval
Written by TechnoLynx Published on 11 Jul 2026

2WikiMultiHopQA is a question-answering dataset built from the overlap between Wikipedia and Wikidata. It holds 192,606 questions of four types (comparison, inference, compositional and bridge-comparison). Each question comes with evidence information, a set of triples drawn from Wikidata. The dataset tests whether a model can chain facts from two paragraphs to reach an answer. A model’s score on it predicts performance on your own multi-hop or retrieval-augmented task only in part, and less than a leaderboard headline suggests.

Multi-hop QA scores are accuracy measures, not speed measures: unlike LLM inference benchmarks, they tell you nothing about latency, throughput or cost per answer. Measuring hardware and serving speed is a separate discipline, covered where the procurement evidence pack stops and LynxBenchAI’s benchmark methodology begins. This article deals only with the accuracy side.

What kind of multi-hop reasoning does 2WikiMultiHopQA actually test?

A question counts as multi-hop when no single context is enough to answer it. In the formal definition, the answer has to follow from more than one gold supporting context, and with a single context the task falls back to ordinary QA. 2WikiMultiHopQA turns that requirement into four question templates:

  • Comparison (57,989 questions): compare a property of two entities, such as which person was born first.
  • Compositional (86,979): answer primitive sub-questions in sequence, for example “Who is the founder of Versus?” and then “Why did he or she die?”.
  • Inference (7,478): apply a logical rule, such as finding a child’s child to get a grandchild.
  • Bridge-comparison (40,160): find a bridge entity first, then make a comparison.

Inference and compositional questions are both subtypes of the bridge question, where a bridge entity links the two paragraphs. The construction rules are designed to make sure each question needs more than one hop. Each sample is built from a two-hop tuple (e, r1, e1, r2, e2). The paragraph about e has to mention e1 and must not contain the answer e2, and the paragraph about e1 must not mention e. The aim is to stop a single paragraph from giving the answer away.

The evidence format is another difference. HotpotQA explains answers with sentence-level supporting facts, which the 2WikiMultiHopQA authors argue reduces explanation to binary classification, and that cannot evaluate reasoning skill. 2WikiMultiHopQA asks for the reasoning path as a set of structured triples instead. Its official scoring reports exact match (EM) and F1 for each task, plus joint scores that combine sentence-level, entity-level and answer prediction.

The splits are worth knowing before you read a score. The authors ran a single-hop model over all the data. Every question it solved went into a training subset (train-medium), and the dev and test sets were drawn from the questions it failed. So the test set is hard by construction against that one baseline. That is a filter, not a guarantee that every question needs multi-hop reasoning. About 31.2% of answers are yes/no. That proportion matters when a model’s EM looks respectable but its F1 on entity answers does not.

Does a strong 2WikiMultiHopQA score predict our own RAG pipeline?

Only partly. The gap comes from three places: retrieval, data shape and score inflation.

Retrieval is mostly taken out of the test

2WikiMultiHopQA is often described as a benchmark for retrieval-augmented generation. It ships Wikipedia paragraphs and supporting-fact triples with the questions. When a model gets its contexts with the question, the setting is closed-domain, which guarantees the supplied contexts are enough to answer. In the open-domain setting, where the model must retrieve from the whole corpus, there is no such guarantee. The retriever then has to find the second hop itself, and an error in any hop carries into every hop after it, a problem known as semantic drift. One test makes the size of the problem clear. Plain BM25 retrieval on HotpotQA fell from 53.7% accuracy on the mostly single-hop “easy” subset to 25.9% on the “hard” multi-hop subset. In another analysis, BM25 with BERT reranking did well on the first hop and failed on the second.

The data is cleaner than yours

The authors removed ambiguous cases on purpose. They kept only relations with a single value in Wikidata, discarded bridge questions where an entity has more than one candidate, kept only films with exactly one director or producer, and dropped comparisons where the two values are equal. The benchmark therefore does not test questions with several valid candidates, multi-valued relations or tied values. The answers are also Wikipedia entities, so the dataset says nothing direct about contracts, tickets or engineering logs.

Scores can be pushed up in-distribution

On each of HotpotQA, 2WikiMultiHopQA and MuSiQue, RL fine-tuning on that dataset’s own training data beat the other data sources tested. A grokking study reports accuracy of up to 95–100% on 2WikiMultiHopQA from synthetic data augmentation. A survey of multi-hop QA reaches the general version of the point: imitating how a dataset was built can raise accuracy on it without carrying over to other datasets. Hosted models’ scores also drift over time. Between March and June 2023, GPT-4’s exact-match rate on HotpotQA rose sharply while GPT-3.5’s fell.

What does transfer, then? We read a strong one as permission to test further, not as evidence about your workload.

Shortcut and contamination limits to disclose when defending a model choice

A model choice that rests on a public multi-hop score should state the score’s known weaknesses alongside it. An approval committee is right to discount a number that arrives without them.

Reasoning shortcuts

The 2WikiMultiHopQA paper starts from the finding that many examples in earlier multi-hop datasets can be answered without multi-hop reasoning. In one test, baselines that scored each sentence independently, with no access to other sentences, came “overwhelmingly closer” to the state of the art than to a random classifier, most of all on multiple-choice datasets. In one analysis of HotpotQA, about 20% of questions had the answer span in both supporting passages. A model with access to every supporting passage scored only slightly better than one that saw just the answer passage (50.96 EM against 49.43).

A single-hop BERT model scored 55.9 F1 on 2WikiMultiHopQA against 64.6 on HotpotQA, 8.7 points lower. The authors read that gap as evidence that many questions need multi-hop reasoning, yet a single-hop model still reaches 55.9 F1 on the dataset. Later analysis found that the word-overlap shortcut common in a HotpotQA subset is not common in 2WikiMultiHopQA. 2WikiMultiHopQA shows more position bias instead, with a lean towards the first sentence. In one reported comparison, a model trained only on answer prediction dropped 13.4% EM.

Entity-level evidence triples also have no significant effect on the main QA process. On adversarial sub-questions, correctly rebuilding the entity-level reasoning path led to the correct answer in only 37.5% of cases. Baseline models in the original paper reached 16.7 F1 on evidence generation, against 78.8 for humans. The data has some noise too: in a manual check of 100 training samples, the authors found 8 where the Wikipedia article and the Wikidata triple disagreed. Some ground-truth nationality answers are an adjective rather than a country name, an inconsistency that kept one model from 100% accuracy.

Contamination from Wikipedia pretraining

Data contamination means test examples, or their answers, also appear in a model’s training data, so the model can score by memorising instead of generalising. Real-world multi-hop benchmarks are built from Wikipedia, which was probably part of each model’s pretraining corpus. Gains on those benchmarks may therefore come from composing memorised knowledge rather than from a general composition skill. The authors of the augmentation paper make a similar point about GPT-4o: a model that has seen so much Wikipedia text may not offer a clean split between in-distribution and out-of-distribution questions.

Contamination has been measured directly on six multiple-choice QA benchmarks, including MMLU and Hellaswag. One study checked those benchmarks against Common Crawl and found verbatim overlap in 1% to 45.8% of examples, rising over time. The same study found that contamination does not inflate scores evenly. The effect depends on the dataset and the model scale, and it is largest when both question and answer leaked. A buyer’s guide from BERI cites Scale’s GSM1k study, which found accuracy drops of up to 13% on fresh problems in the paper’s first version, and OpenAI’s decision in February 2026 to stop reporting SWE-bench Verified over contamination. Researchers have started building counterfactual rewrites of HotpotQA, 2WikiMultiHopQA and MuSiQue questions that remove the factual knowledge a model could recall. A result on such a set is worth asking for whenever one exists.

Disclosure checklist for a multi-hop score in a decision pack

  1. Setting: closed-domain with supplied paragraphs, or open-domain with the model retrieving.
  2. Metric: answer EM, answer F1 or joint score, named explicitly.
  3. Split and version: which split was scored, and on what date for a hosted model, given documented drift.
  4. Training exposure: whether the model or method trained on the benchmark’s own training data.
  5. Shortcut robustness: results on debiased or adversarial sets where available, or a statement that there are none.
  6. Contamination status: whether a check was run, since Wikipedia-derived benchmarks were likely in pretraining.

How 2WikiMultiHopQA, HotpotQA and MuSiQue differ as selection evidence

The three datasets share a Wikipedia origin, so all three carry the contamination caveat above. They differ in how they were built and in what evidence of reasoning they ask for. The supplied research says much less about MuSiQue’s shortcut profile than about the other two, and the table leaves that cell open rather than guess.

  HotpotQA 2WikiMultiHopQA MuSiQue
Source Wikipedia Wikipedia plus Wikidata Wikipedia-based
Size 112,779 questions 192,606 questions 19,938 training, 2,417 dev
Hops 1, 2 or 3 hops 2-hop templates in four types 2 to 4 hops
Construction Second hop is the bridge entity’s Wikipedia paragraph Templates and logical rules over Wikidata triples Composition of single-hop questions
Reasoning evidence Sentence-level supporting facts Entity-level triples Intermediate sub-questions with answers; few triples
Retrieval settings Distractor (2 gold plus 8 TF-IDF distractors) and full-wiki Supplied paragraphs Answerable split, context sufficient
Known shortcut profile About 20% single-passage answerable; word overlap common in a small subset Single-hop BERT still 55.9 F1 (64.6 on HotpotQA); position bias Not covered by the supplied research

For a selection decision, we would weigh the three as complementary probes rather than as a ranking. HotpotQA’s full-wiki setting comes closest to open retrieval. 2WikiMultiHopQA defines comparison and compositional questions as separate types. MuSiQue supplies intermediate sub-questions with ground-truth answers, which one study used to check model reasoning traces. Gains from fitting one dataset’s construction are not guaranteed to carry over to the others.

When should an approval committee weight its own task eval over public multi-hop scores?

Almost always, once the decision is real money or real risk. The public score earns a place in screening, and the task-specific eval earns the decision. The BERI buyer’s guide puts it directly: a vendor-run score on hidden tasks at settings the vendor chose belongs in the screening pile, and evidence for a business case comes from independent evaluators that publish their logs, or from a replication on your own data at the configuration you will pay for. The guide describes that replication as a small blinded test of about 50 of your own tasks, scored against answers written down before the run.

Fifty tasks is enough to catch a collapse, not to split two good models. At an 80% pass rate, the 95% interval on 50 tasks is roughly plus or minus 11 points. The working: 1.96 × √(0.8 × 0.2 / 50) ≈ 0.11. So 50 tasks will catch an 85% claim that lands at 60% on your data, but resolving a 4-point gap needs several hundred tasks. The same guide adjusts the weighting by situation:

  • In a regulated domain with no public benchmark close to the work, private evaluation plus a replication of 200 tasks or more beats public leaderboards.
  • For open-weight models you host yourself, you control the configuration, so independent public numbers transfer better and the replication can be smaller.
  • A vendor offer to run the bake-off for you should be declined, because a vendor-run replication carries every problem of a vendor-run benchmark.

For multi-hop and RAG work in particular, we would build the task set from questions where the second hop is hard to retrieve. Supplied-context benchmarks take that retrieval step out of the test, and retrieval errors accumulate with each hop. The steps for assembling that set are in our checklist on how to build a task-specific LLM evaluation for procurement. Keeping the results defensible in front of a committee is covered in running a task-specific LLM evaluation that survives a procurement review. LLM selection and evaluation are part of TechnoLynx’s work in AI for AI infrastructure and production SaaS.

What a committee should ask of a 2WikiMultiHopQA number

The useful question is not whether the score is high. It is what setting produced the score, whether the model ever trained on the dataset, and how the model did on the evidence-generation task the dataset was designed for. The authors offer that task as a new challenge for explaining multi-hop reasoning, given the large model-human gap. If the evidence-generation figure is missing, treat the answer score as a screen that the model passed, and let your own tasks make the decision.

Back See Blogs
arrow icon