Project case study
Agentic Multi-Hop RAG
A single retrieval pass fails when a question’s evidence is spread across several documents. This system retrieves iteratively instead — judging after each hop whether it has enough evidence, and asking a focused follow-up question when it doesn’t, for up to three hops.
A learned router variant, Adaptive RAG, tests whether that same quality can be approximated more cheaply — measured honestly, once, on a final holdout set it never saw during development.
Business Problem
Retrieval-augmented generation typically retrieves once and answers once. That works when a single query surfaces every fact a question needs. It breaks down for questions whose answer depends on synthesizing facts that live in different documents — comparing two sources, reconstructing a timeline, or chaining several inferences together. A single retrieval pass either misses part of the evidence, or surfaces it too far down the ranking to reach the model’s context window.
The failure is quiet. The system doesn’t know it is missing evidence, so it answers anyway — confidently, and sometimes wrongly. There is no built-in mechanism to notice the gap and go look again.
This project measures that gap directly, on MultiHop-RAG — a benchmark built specifically so single-pass retrieval cannot reliably answer every question — and builds a system that retrieves again when the first pass isn’t enough, plus a cheaper variant that learns when that extra work is actually worth doing.
Architecture
Every hop runs the same frozen retrieval pipeline. What changes hop to hop is only the query: the original question first, then a controller- written follow-up whenever the merged evidence isn’t judged sufficient yet.
- 01Questiontext only — no gold answer or evidence
- 02Dense + BM25two independent retrieval passes
- 03Reciprocal rank fusiondeterministic RRF, k = 60
- 04Cross-encoder rerankerBAAI/bge-reranker-base
- 05Evidence sufficiencyone structured GLM judgment
- 06Follow-up retrievala new focused query
- 07Final answerQwen, from all merged evidence
Step 06 repeats step 02 onward — capped at 3 hops total, then answers from whatever evidence it has
Hard limits, not just prompts
Max 3 retrieval calls, a context token budget, and a wall-clock timeout are enforced in code — the loop cannot exceed them regardless of what the controller returns.
One controller call per hop
A single structured call returns whether the evidence is sufficient and, if not, the next query — never a separate sufficiency check and sub-query call.
Same answer model, every path
Every pipeline on this page — baseline, agentic, or adaptive — answers with the identical model, prompt, and context assembly. Only the evidence reaching that call differs.
Live Agentic RAG Demo
Ask a real question against the same indexed corpus, through the same frozen pipeline measured throughout this page. This runs live — it retrieves, reranks, checks evidence sufficiency, and answers with the exact code behind every number above it.
Live demo available locally from the repository. This section calls a separate FastAPI backend (backend/) that only runs alongside a local clone — it isn’t part of this static deployment. Clone the repository and follow backend/README.md to ask a real question against the live pipeline.
Baseline Comparison
Three single-pass baselines establish how far retrieval alone can go. Agentic Multi-Hop RAG is the main system this project builds; Adaptive RAG is measured as a cost-optimized alternative to it, not a fourth independent baseline.
Retrieval quality — 265 non-null development questions
| Method | Recall@10 | Hit@10 | MRR@10 | NDCG@10 | Complete-Evidence@10 |
|---|---|---|---|---|---|
| Dense RAG | 77.4% | 98.9% | 0.681 | 0.621 | 52.5% |
| Hybrid RAG | 84.1% | 99.6% | 0.758 | 0.701 | 62.3% |
| Hybrid + Reranker | 82.6% | 99.6% | 0.795 | 0.729 | 60.8% |
Evidence coverage, all five pipelines — 50-question development sample
| Pipeline | Evidence coverage |
|---|---|
| Dense RAG | 46.2% |
| Hybrid RAG | 60.0% |
| Hybrid + Reranker | 68.4% |
| Agentic Multi-Hop RAG | 81.1% |
| Adaptive RAG | 77.5% |
Evidence coverage is the fraction of a question’s gold source documents that actually reached the final-answer prompt — a stricter, end-to-end measure than retrieval-ranking metrics alone.
Final Holdout Results
Measured once, on 50 questions from a split neither the router nor any prompt or threshold was ever tuned against. This is the number that matters more than anything on the development split.
Agentic Multi-Hop RAG achieved higher answer quality and evidence coverage. Adaptive RAG reduced cost and latency at a measurable quality trade-off. Neither result is smoothed over — both are held side by side in the sections below.
Results
Answer quality
Judge score for open-ended questions blended with abstention correctness for null questions.
Development sample (n = 50)
| Group | Series | Value |
|---|---|---|
| Hybrid + Reranker | Hybrid + Reranker | 0.54 |
| Agentic Multi-Hop RAG | Agentic Multi-Hop RAG | 0.64 |
| Adaptive RAG | Adaptive RAG | 0.60 |
Final holdout (n = 50, one-time)
| Group | Series | Value |
|---|---|---|
| Agentic Multi-Hop RAG | Agentic Multi-Hop RAG | 0.70 |
| Adaptive RAG | Adaptive RAG | 0.56 |
Evidence coverage
Fraction of a question’s gold source documents that reached the final-answer prompt.
Development sample (n = 50)
| Group | Series | Value |
|---|---|---|
| Dense RAG | Dense RAG | 46.2% |
| Hybrid RAG | Hybrid RAG | 60.0% |
| Hybrid + Reranker | Hybrid + Reranker | 68.4% |
| Agentic Multi-Hop RAG | Agentic Multi-Hop RAG | 81.1% |
| Adaptive RAG | Adaptive RAG | 77.5% |
Final holdout (n = 50, one-time)
| Group | Series | Value |
|---|---|---|
| Agentic Multi-Hop RAG | Agentic Multi-Hop RAG | 75.9% |
| Adaptive RAG | Adaptive RAG | 68.0% |
Cost per query
Mean Mantle cost per question, in USD.
Development sample (n = 50)
| Group | Series | Value |
|---|---|---|
| Dense RAG | Dense RAG | $0.00045 |
| Hybrid RAG | Hybrid RAG | $0.00046 |
| Hybrid + Reranker | Hybrid + Reranker | $0.00048 |
| Agentic Multi-Hop RAG | Agentic Multi-Hop RAG | $0.00122 |
| Adaptive RAG | Adaptive RAG | $0.00101 |
Final holdout (n = 50, one-time)
| Group | Series | Value |
|---|---|---|
| Agentic Multi-Hop RAG | Agentic Multi-Hop RAG | $0.00116 |
| Adaptive RAG | Adaptive RAG | $0.00091 |
Latency per query
Mean end-to-end latency per question, in milliseconds.
Development sample (n = 50)
| Group | Series | Value |
|---|---|---|
| Dense RAG | Dense RAG | 1481 |
| Hybrid RAG | Hybrid RAG | 1404 |
| Hybrid + Reranker | Hybrid + Reranker | 5150 |
| Agentic Multi-Hop RAG | Agentic Multi-Hop RAG | 11111 |
| Adaptive RAG | Adaptive RAG | 9501 |
Final holdout (n = 50, one-time)
| Group | Series | Value |
|---|---|---|
| Agentic Multi-Hop RAG | Agentic Multi-Hop RAG | 9660 |
| Adaptive RAG | Adaptive RAG | 7573 |
Quality by query type
Combined quality, Agentic Multi-Hop RAG vs. Adaptive RAG, by MultiHop-RAG question type.
Development sample
- Agentic Multi-Hop RAG
- Adaptive RAG
| Group | Series | Value |
|---|---|---|
| Inference (n=16) | Agentic Multi-Hop RAG | 0.94 |
| Inference (n=16) | Adaptive RAG | 1.00 |
| Comparison (n=17) | Agentic Multi-Hop RAG | 0.47 |
| Comparison (n=17) | Adaptive RAG | 0.47 |
| Temporal (n=11) | Agentic Multi-Hop RAG | 0.27 |
| Temporal (n=11) | Adaptive RAG | 0.00 |
| Null (n=6) | Agentic Multi-Hop RAG | 1.00 |
| Null (n=6) | Adaptive RAG | 1.00 |
Final holdout
- Agentic Multi-Hop RAG
- Adaptive RAG
| Group | Series | Value |
|---|---|---|
| Inference (n=16) | Agentic Multi-Hop RAG | 1.00 |
| Inference (n=16) | Adaptive RAG | 0.94 |
| Comparison (n=16) | Agentic Multi-Hop RAG | 0.56 |
| Comparison (n=16) | Adaptive RAG | 0.31 |
| Temporal (n=12) | Agentic Multi-Hop RAG | 0.33 |
| Temporal (n=12) | Adaptive RAG | 0.17 |
| Null (n=6) | Agentic Multi-Hop RAG | 1.00 |
| Null (n=6) | Adaptive RAG | 1.00 |
Quality by hop-count difficulty
Combined quality by the number of gold documents a question requires.
Development sample
- Agentic Multi-Hop RAG
- Adaptive RAG
| Group | Series | Value |
|---|---|---|
| 2-hop (n=22) | Agentic Multi-Hop RAG | 0.55 |
| 2-hop (n=22) | Adaptive RAG | 0.45 |
| 3-hop (n=17) | Agentic Multi-Hop RAG | 0.53 |
| 3-hop (n=17) | Adaptive RAG | 0.53 |
| 4-hop (n=5) | Agentic Multi-Hop RAG | 1.00 |
| 4-hop (n=5) | Adaptive RAG | 1.00 |
| Null (n=6) | Agentic Multi-Hop RAG | 1.00 |
| Null (n=6) | Adaptive RAG | 1.00 |
Final holdout
- Agentic Multi-Hop RAG
- Adaptive RAG
| Group | Series | Value |
|---|---|---|
| 2-hop (n=23) | Agentic Multi-Hop RAG | 0.61 |
| 2-hop (n=23) | Adaptive RAG | 0.43 |
| 3-hop (n=15) | Agentic Multi-Hop RAG | 0.60 |
| 3-hop (n=15) | Adaptive RAG | 0.40 |
| 4-hop (n=6) | Agentic Multi-Hop RAG | 1.00 |
| 4-hop (n=6) | Adaptive RAG | 1.00 |
| Null (n=6) | Agentic Multi-Hop RAG | 1.00 |
| Null (n=6) | Adaptive RAG | 1.00 |
Development vs. holdout
Does Adaptive RAG’s cost/latency trade-off hold up on unseen data?
- Development
- Final holdout
| Group | Series | Value |
|---|---|---|
| Quality retention | Development | 93.8% |
| Quality retention | Final holdout | 80.0% |
| Cost reduction | Development | 17.2% |
| Cost reduction | Final holdout | 21.8% |
| Latency reduction | Development | 14.5% |
| Latency reduction | Final holdout | 21.6% |
Experiment Journey
- 01
Dense retrieval
77.4% Recall@10A single embedding-similarity pass over the corpus establishes the floor.
- 02
Hybrid retrieval
84.1% Recall@10Dense and BM25 fused by deterministic reciprocal rank fusion recover evidence neither method alone surfaces.
- 03
Cross-encoder reranking
0.795 MRR@10Reordering the fused candidates by a cross-encoder trades a little Recall@10 for a markedly better top-of-ranking precision.
- 04
Agentic Multi-Hop RAG
0.700 answer qualityA bounded, evidence-checking loop replaces one retrieval pass with up to three — the main system, measured once on final holdout.
- 05
Adaptive RAG
21.8% cost reductionA learned router sends most questions through a cheaper path and reserves the full loop for the ones that need it.
Failure Analysis
Temporal and comparison questions are the clearest weakness
On final holdout, temporal questions score 0.33 for Agentic and 0.17 for Adaptive — the lowest of any category, on both systems, on both splits (development temporal quality for Adaptive was 0.00). Comparison questions score 0.56 (Agentic) and 0.31 (Adaptive) — the second-weakest category, with Adaptive trailing Agentic by a visible margin. Neither weakness is new to Adaptive RAG; the router does not compensate for either.
Adaptive under-routing
5 of 44 non-null development questions, and 10 of 44 on final holdout, were routed to a cheaper path and scored — or covered evidence — strictly worse than Agentic would have. Both sets concentrate in comparison and temporal questions: the router is paying its cheap-route bet and losing it on exactly the categories already flagged above, not a separate, unrelated failure mode.
The development-to-holdout generalization gap
Adaptive’s quality retention was 93.8% on the development sample and 80.0% on final holdout — a real drop, not sampling noise smoothed away. Cost and latency reductions moved the opposite direction, growing larger on holdout. Both numbers moving together — bigger savings, bigger quality gap — is consistent with the router escalating slightly less on this split, not with two unrelated metrics drifting independently.
Evaluation Methodology
- MultiHop-RAG
- 609 news documents, 2,556 questions labeled inference / comparison / temporal / null, each citing 0–4 supporting source documents — a benchmark built specifically so single-pass retrieval cannot reliably answer every question.
- Frozen configurations
- Every model id, retrieval config, router threshold, agent config, and prompt template used for the final holdout evaluation was fixed before that evaluation began — see the pre-access manifest below.
- An untouched final holdout, until one measurement
- A configuration manifest was SHA-1 hashed BEFORE the holdout file was read for the first time. The aggregation step re-hashes the same files and would raise on any drift — mechanically checked, not just claimed. Integrity check: "PASSED — all frozen files unchanged since pre-access manifest".
- A separate judge model
- openai.gpt-oss-120b grades every answer — a third model, distinct from both the answer-generation model (Qwen) and the agent controller (GLM), seeing only the question, reference answer, and candidate answer. It never learns which pipeline, route, or model produced what it's grading.
- Checkpointed, paid evaluation
- Every live model call is written to disk immediately after it completes. An interrupted run resumes without repeating a completed, already-paid call — verified in practice, not just in theory, across this project's development.
- 877 passing tests
- Unit tests run entirely offline against fake models and fixtures; structural guard tests grep the actual source of every live script, proving in code — not just in documentation — that gold answers can never reach a routing or generation decision, and that no script can reach the final holdout file outside its one deliberate, clearly named exception.
- Consumed
- final_holdout.json is marked "CONSUMED" (seed 2029, 50 questions) and is not evaluated against again.
Case Study: Why Iterative Retrieval Earns Its Cost
A single retrieval pass either finds everything a question needs or it doesn’t, with no way to notice which. After every hop, the agentic controller judges whether the evidence gathered so far is sufficient and, if not, issues one focused follow-up query aimed at what’s missing. On the development questions this project can trace end to end — where hop 1 alone genuinely wasn’t enough — that mechanism visibly earns its keep.
Evidence coverage: Baseline → Context-Matched → Agentic
Measured on the 86 development questions where Agentic took more than one hop and all three pipelines have a result to compare. Context-Matched is the same single-pass retrieval given as many chunks as Agentic actually used for that question — controlling for evidence volume, so any remaining Agentic advantage over it comes from which evidence iteration found, not simply how much of it there was.
| Group | Series | Value |
|---|---|---|
| Baseline | Baseline (Hybrid + Reranker) | 63.2% |
| Context-Matched | Context-Matched (control) | 75.7% |
| Agentic Multi-Hop | Agentic Multi-Hop (all hops) | 81.5% |
| Pipeline | Mean evidence coverage | vs. Baseline | vs. Context-Matched |
|---|---|---|---|
| Baseline (Hybrid + Reranker) | 63.2% | — | — |
| Context-Matched (control) | 75.7% | +12.5% | — |
| Agentic Multi-Hop (all hops) | 81.5% | +18.3% | +5.8% |
Mechanism evidence
Of the 92 development questions Agentic resolved in 2+ hops, did a later hop retrieve a required (gold) document that hop 1 had missed — and did that recovered evidence go on to flip the final answer?
7 of the 33 questions where a later hop added required evidence went on to a correctly-graded Agentic answer where both Baseline and Context-Matched were graded incorrect. One of those, 03ea05f6e99ffb38, is excluded from the worked examples below only: known Task Success evaluator quirk identified during Phase 3/4 hardening (judge grade unreliable on this qa_id's answer text) — excluded from example selection only, not from any aggregate count in this report. This leaves 6 used for examples. Both counts are reported so the reduction is explicit, not smoothed over: finding the missing evidence is necessary but not sufficient for a better final answer, and the 26 questions that added evidence without flipping the grade are the rest of that story.
Worked examples
Each trace below combines the original benchmark run (final answer, baseline answer, judge grade) with an independent live replay of that exact question, development split only, which recovered the actual verbatim hop-by-hop queries — the original benchmark script never persisted them, only the final aggregate result. Hop count and stop reason reproduced exactly in 5 of 6; the exact final evidence-document set reproduced exactly in 4 of 6 — the replayed answer text below is quoted from the live replay and is not guaranteed identical to the original graded run (see Limitations).
Did the report from Cnbc | World Business News Leader on "Nike's Latin America and Asia Pacific unit" and the article from Fortune on the "U.S. home sales price" both report an increase in their respective financial figures?
- 01Hop 1Did the report from Cnbc | World Business News Leader on "Nike's Latin…
- 02Hop 2Fortune article U.S. home sales price increase
- 03StopEvidence sufficient
Agentic answer — graded correct
Yes, the report from CNBC | World Business News Leader on Nike's Latin America and Asia Pacific unit reported an increase, with sales up 2% to $1.57 billion compared to the year-ago period. The article from Fortune on U.S. home sales price also reported an increase, noting that existing-home sales prices topped $306,000, a 5% increase since the year began.
Baseline answer — graded incorrect
The available information does not include any report from CNBC | World Business News Leader on Nike's Latin America and Asia Pacific unit, nor does it include any article from Fortune on U.S. home sales prices. Therefore, it is insufficient to answer whether both reports indicated an increase in their respective financial figures.
Final evidence documents used: 3 · Retrieval calls: 2 · Controller calls: 2
Which individual, covered by both 'The Verge' and 'TechCrunch', is implicated in using customer funds for a buyout, faced challenges managing two companies due to rapid growth, and is accused of committing fraud for personal gain?
- 01Hop 1Which individual, covered by both 'The Verge' and 'TechCrunch', is imp…
- 02Hop 2Monzo CEO Conor Walsh customer funds buyout fraud
- 03Hop 3Conor Walsh Monzo fraud customer funds buyout
- 04StopMax hops reached
Agentic answer — graded correct
The individual implicated in using customer funds for a buyout, faced challenges managing two companies due to rapid growth, and is accused of committing fraud for personal gain is Sam Bankman-Fried. He directed the use of $1 billion in FTX customer funds to buy back FTX shares from Binance, as testified by Caroline Ellison. He also oversaw FTX and Alameda Research, two companies that grew rapidly and were managed with insufficient internal controls, as evidenced by his lack of policies to prevent misuse of customer funds. The prosecution alleges he committed fraud for personal gain, including misappropriating customer funds, creating misleading balance sheets with “Sam coins” like FTT, and failing to implement safeguards despite his public claims of protecting customer money.
Baseline answer — graded incorrect
The available information is insufficient to answer.
Final evidence documents used: 7 · Retrieval calls: 3 · Controller calls: 3
Do 'The Verge' and 'Engadget' articles both suggest that 'Consumers' can find guidance or deals on tech products, while the 'TechCrunch' article proposes a different interest of 'Consumers' in the realm of social networking?
- 01Hop 1Do 'The Verge' and 'Engadget' articles both suggest that 'Consumers' c…
- 02Hop 2TechCrunch article consumers social networking interest
- 03Hop 3The Verge article consumers guidance deals tech products
- 04StopToken budget reached
Agentic answer — graded correct
Yes, both The Verge and Engadget articles suggest that consumers can find guidance or deals on tech products. Engadget provides curated lists of Black Friday and Prime Day deals on gadgets like headphones, tablets, and smartwatches, helping consumers identify worthwhile purchases based on tested reviews. The Verge offers advice on tracking deals independently using tools like CamelCamelCamel and discusses alternatives to services like Mint, indicating consumer interest in managing spending and finding savings. In contrast, the TechCrunch articles focus on consumers’ interest in new social networking platforms—such as Pebble, ID, and Mozilla’s involvement in the fediverse—that prioritize trust, safety, and decentralization over profit-driven models. These pieces highlight a shift in consumer desire toward alternatives to mainstream platforms like Twitter and Facebook, emphasizing community, moderation, and user control rather than product deals or reviews. Thus, while The Verge and Engadget center on consumer guidance for purchasing tech, TechCrunch explores a different consumer interest: reimagining social networking itself.
Baseline answer — graded incorrect
The available information does not mention The Verge or TechCrunch at all, so there is no basis to compare their content with Engadget’s. Therefore, it is insufficient to answer whether The Verge and Engadget both suggest that consumers can find guidance or deals on tech products, or whether TechCrunch proposes a different interest of consumers in social networking.
Final evidence documents used: 11 · Retrieval calls: 3 · Controller calls: 2
Did the article from The Verge defending Apple's Google Search deal suggest that there were valid alternatives to Google for search engine services, while the other article from The Verge about the public and media's understanding of search engines imply a satisfaction with the public's knowledge of how search engines work?
- 01Hop 1Did the article from The Verge defending Apple's Google Search deal su…
- 02Hop 2The Verge article public understanding search engines satisfaction
- 03StopDuplicate query
Agentic answer — graded correct
The provided context does not contain an article from The Verge defending Apple’s Google Search deal. Instead, The Verge is cited as a source reporting facts from the trial, such as Google’s $26.3 billion payment in 2021 and details of testimony from Apple and Google executives. There is no indication that The Verge defended the deal or suggested there were valid alternatives to Google. Additionally, while one context excerpt mentions that Microsoft and DuckDuckGo CEOs testified their search engines could have been competitive had they secured similar deals with Apple, this does not reflect The Verge’s stance or imply satisfaction with the public’s understanding of how search engines work. Another excerpt quotes a former Google employee suggesting the public underestimates how Google shapes search results, implying a lack of awareness — not satisfaction. Therefore, the question contains premises not supported by the context. The available information does not show that The Verge defended the deal or made claims about valid alternatives or public understanding in the way the question suggests.
Baseline answer — graded incorrect
The provided context does not contain an article from The Verge that defends Apple’s Google Search deal. It also does not include any article from The Verge about the public and media’s understanding of how search engines work. Therefore, there is insufficient information to answer the question as posed.
Final evidence documents used: 4 · Retrieval calls: 2 · Controller calls: 2
Does the TechCrunch article on Meta's advertising practices involve the European Commission in a similar capacity to how the European Commission is involved in Amazon's iRobot purchase and Elon Musk's X company's DSA probe, as reported by TechCrunch in both cases?
- 01Hop 1Does the TechCrunch article on Meta's advertising practices involve th…
- 02Hop 2TechCrunch article Amazon iRobot European Commission investigation
- 03Hop 3TechCrunch article Meta advertising practices European Commission
- 04StopToken budget reached
Agentic answer — graded correct
Yes, the European Commission is involved in Meta’s advertising practices in a similar capacity to its involvement in Amazon’s iRobot purchase and Elon Musk’s X company’s DSA probe. In all three cases, the European Commission is acting as a regulatory enforcer under EU digital and competition rules. For Meta, the Commission is overseeing compliance with the Digital Services Act (DSA) and Digital Markets Act (DMA), particularly regarding ad tracking, consent mechanisms, and potential dark patterns. The Commission has already requested information from Meta and may investigate whether its subscription-for-privacy model meets legal standards for consent and fairness. Similarly, in the case of X, the Commission opened a formal DSA investigation into potential breaches related to content moderation, dark patterns, advertising transparency, and data access — directly enforcing the DSA against a very large online platform. In the case of Amazon and iRobot, the Commission issued a formal statement of objections under competition law, raising concerns about foreclosure risks and anti-competitive behavior in the robot vacuum cleaner market, exercising its authority under EU antitrust rules. In each instance, the Commission is not merely advising or monitoring — it is actively investigating, demanding information, and preparing to enforce legal consequences if violations are found. The capacity of the Commission in all three cases is therefore comparable: it is the sole enforcer of the relevant EU regulations (DSA, DMA, or competition law) and is taking formal steps to assess and potentially penalize non-compliance.
Baseline answer — graded incorrect
The available information does not mention Amazon's acquisition of iRobot or the European Commission’s role in that matter. Therefore, it is insufficient to compare the Commission’s involvement in Meta’s advertising practices with its involvement in Amazon’s iRobot purchase. However, regarding Meta and X: the context shows that the European Commission is involved in both cases under the Digital Services Act (DSA). For Meta, the Commission is assessing compliance with DSA requirements related to advertising, consent, and dark patterns, particularly concerning its “pay or track” model and handling of children’s data. For X, the Commission has opened a formal investigation into potential DSA breaches, including issues like content moderation, dark patterns, advertising transparency, and data access. Thus, the Commission’s role in Meta’s advertising practices and X’s DSA probe is similar — both involve enforcement of the DSA. But since there is no information about the Commission’s role in Amazon’s iRobot purchase, a full comparison cannot be made.
Final evidence documents used: 5 · Retrieval calls: 3 · Controller calls: 2
Does 'The New York Times' article attribute the failure of the Buffalo Bills' defense to the contributions of Jordan Poyer, while the 'Sporting News' article suggests that the Baltimore Ravens' defense needs to improve before their game against the Cincinnati Bengals?
- 01Hop 1Does 'The New York Times' article attribute the failure of the Buffalo…
- 02Hop 2Sporting News article Baltimore Ravens defense needs improvement befor…
- 03StopEvidence sufficient
Agentic answer — graded correct
The available information does not include any article from The New York Times or Sporting News, nor does it attribute the failure of the Buffalo Bills' defense to Jordan Poyer. In fact, the context portrays Poyer as a vital, versatile, and uplifting force for the Bills’ defense, with former coordinator Les Frazier crediting him for enabling the defense to be its best through leadership, toughness, and communication. Additionally, while Source 7 mentions that the Baltimore Ravens’ defense “needs to pick up the pieces,” this is presented as a general observation, not as a direct suggestion from Sporting News, and it is not explicitly tied to their upcoming game against the Bengals in a way that matches the question’s framing. Therefore, the claim in the question cannot be confirmed using the provided context. The available information is insufficient to answer as stated.
Baseline answer — graded incorrect
The available information does not contain any article from The New York Times or Sporting News, nor does it attribute the failure of the Buffalo Bills' defense to Jordan Poyer or suggest anything about the Baltimore Ravens' defense. Therefore, the question cannot be answered based on the provided context.
Final evidence documents used: 7 · Retrieval calls: 2 · Controller calls: 2
Limitations of this analysis
- Iteration finding evidence is not the same as iteration fixing the answer. Only 6 of the 33 questions where a later hop added required evidence flipped from “both baselines wrong” to “Agentic correct” — the rest saw no such clean win. This section reports both numbers, not just the flattering one.
- Generation is not byte-deterministic. The worked-example answers above are the original graded run, not the live replay’s own regenerated text — even at fixed decoding settings, hosted LLM inference is not guaranteed to be token-identical across separate calls, so the two differ slightly in wording even when the underlying evidence is identical.
- Generation is also not fully deterministic in its stopping decision: hop count and stop reason reproduced exactly in 5 of 6replayed questions. Retrieval itself showed minor variation on replay too: the exact final evidence-document set reproduced in 4 of 6 replayed questions — most likely because approximate nearest-neighbor search isn’t guaranteed bit-identical across separate live queries, and because the controller’s own stop/continue judgment is itself a hosted LLM call. Disclosed, not hidden.
- This is a targeted, traceable subset, not the whole system. The 86–92 question population above is genuinely multi-hop-resolved development questions specifically — not the headline sample — and these findings should not be read as “Agentic always improves the answer.”
- Fact-grounding, measured elsewhere in this project, is retrieval-side only — whether a required fact reached the generation context, not whether the generated answer correctly used it — and is intentionally not used as evidence for this section’s claims.
- Development-stage analysis only. Every number in this section comes from the development split; the one-time final holdout evaluation is reported separately and is not part of this case study.
Tech Stack
Retrieval & indexing
- Qdrant
- sentence-transformers (BAAI/bge-base-en-v1.5)
- FastEmbed sparse BM25
- BAAI/bge-reranker-base
Routing & evaluation
- scikit-learn (logistic regression router)
- pytest
- NumPy
Language models
- qwen.qwen3-next-80b-a3b-instruct — generation
- zai.glm-4.7-flash — agent controller
- openai.gpt-oss-120b — judge
Model access
- Amazon Bedrock Mantle
- OpenAI Python SDK (OpenAI-compatible endpoint)
This case study
- Next.js (App Router)
- TypeScript
- Tailwind CSS
- Hand-built SVG charts — no charting library
Key Findings & Limitations
Findings
- Agentic Multi-Hop RAG outperforms every single-pass baseline and the Adaptive variant on both answer quality and evidence coverage, on both the development sample and final holdout.
- Adaptive RAG recovers most of that quality — 80% on final holdout — at meaningfully lower cost and latency, by routing roughly a quarter of questions away from the full agentic loop.
- The quality gap and the cost/latency savings both grew from development to holdout, moving together rather than independently — a real, not noisy, generalization finding.
Limitations
- 50 questions per split is enough to see a clear, consistent signal, not enough for tight confidence intervals on any single percentage.
- The router's MEDIUM class is rare and hard to hit — a direct consequence of a conservative, safety-first threshold objective, not an oversight.
- Judge fallback rate on final holdout was 3.4% — sensitivity-checked, and it does not change the headline finding, but it is a real source of noise, not zero.
- Temporal and comparison questions remain unresolved weaknesses for the main agentic system, not just the router built on top of it.
- Normalized exact match and token F1 are close to uninformative for this dataset's explanatory gold answers — the judge score and evidence coverage carry the real signal, and both depend on judge-model behavior validated on a small sample rather than cross-checked against a second judge.
- The final holdout evaluation is one-time by design — these numbers cannot be improved by further tuning without invalidating the entire held-out measurement.
Reproducibility
Every retrieval, routing, and generation step is deterministic given its frozen inputs — no sampling temperature above zero anywhere in the pipeline, a fixed cross-validation seed (42) for router training, and a fixed stratified-sampling seed (2029) for both evaluation samples.
Full commands, configuration file layout, and the final holdout evaluation’s exact sequence live in the project README. This page reads the same committed result artifacts directly — it does not call a model, and it cannot run or repeat any evaluation.
python scripts/build_index.py && python scripts/build_hybrid_index.py python scripts/run_retrieval_eval.py python scripts/select_phase9_sample.py python scripts/run_phase9_benchmark.py --pipeline adaptive python scripts/analyze_phase9_sample.py pytest