Tree navigation vs vector search on long and look-alike documents

The short version

We asked 847 questions about financial documents and compared six ways of finding the passage that answers them. Every method handed what it found to the same answering model, so only the search differed. The idea under test is PageIndex: a language model reads a document’s table of contents and steps through it the way a person would, instead of searching for passages that look similar to the question.

Where documents are near-copies of each other, 78 final terms (the legal terms of individual investment products) sharing one template and differing in their numbers, PageIndex answered 82% correctly and the best-tuned conventional search 57%. The conventional search drew most of its passages from the wrong products; PageIndex mostly read the right one. On long prospectuses, and on FinanceBench, the benchmark behind PageIndex’s 98.7% claim, we found no detectable difference. Handing the model the whole right document, with no search at all, scored highest wherever it fitted.

For people building agents: if your documents look alike, picking the right document is the problem to solve, and an agent that chooses it by reading, as PageIndex does, closed much of the gap. If they do not, a well-tuned conventional search was not detectably worse while processing less than half as much text.

Verdict

On near-identical product documents, PageIndex beat tuned hybrid search by +25 points: it read mostly the right product's pages, while similarity search drew most of its passages from look-alikes. On long prospectuses and on FinanceBench there was no detectable difference. PageIndex's own answers scored 87% on FinanceBench, below the 98.7% claimed.

Original measurement

+25 pts

PageIndex vs hybrid search on look-alike documents (95% CI +19 to +30)

−3 pts

on long prospectuses: no detectable difference (95% CI −9 to +4)

87%

PageIndex's own answers on FinanceBench (95% CI 81–93%; 98.7% claimed)

847 test questions, every method answering the same ones; 95% intervals from resampling the questions. One model family navigates, answers and judges. The term-sheet and prospectus answer keys were checked by a model, not a person. A clean rerun of 21 questions reproduced 96.4% of verdicts.

Claim testedPageIndex, retrieval by reasoning over a document's tree instead of vector search, reached 98.7% accuracy on FinanceBench. PageIndex (VectifyAI)

How each method looked for the answer

In short: six ways to find evidence, one answer step. Each search method looks in the same parsed text and passes up to 9,000 words of what it finds to the same model with the same prompt.

• Chunks + embeddings. The documents are cut into passages and each is turned into an embedding, a list of numbers that places similar texts close together. The 45 passages closest to the question are kept.
• Hybrid + reranker, the baseline every method is compared against. Keyword search (BM25) and embeddings each rank passages; the two rankings are merged, and a reranker model, which reads question and passage together, rescores the top 100 and keeps 45. Its settings were tuned on separate development questions.
• RAPTOR. Passages are grouped by similarity and each group is summarised by the model, then the summaries are grouped and summarised again. Search runs over passages and summaries together.
• PageIndex. The model builds a table-of-contents tree for each document; at question time an agent, a model that decides its next step itself, walks the tree and asks for the pages it wants to read. We ran the released code with its default settings.
• Vectors → tree. Embeddings pick the 8 most similar documents, then PageIndex navigates inside them.
• Whole document. The question’s own document goes into the model in full, with no search. Where a document has more than 150,000 words, the method does not apply.
• Oracle. The model gets the pages that hold the answer. It is the ceiling for the answer step, not a contender.

The figure follows five questions, chosen after the run to illustrate each method’s best and worst cases, through PageIndex and the two vector searches: which part of the table of contents the navigator opened, in what order, and which pages each method brought back.

Accuracy against cost

Methods that read more cost more; the question is whether the extra reading pays.

Accuracy against cost for each retrieval method. Up and to the left is better; larger dots are on the best trade-off line (no other method is both cheaper and more accurate); whiskers are 95% bootstrap intervals.

Method	Group	Accuracy	Tokens per question (incl. indexing)	Median seconds per question (incl. queueing)
Whole document	Long context	96% (94–97%)	93,187	290.38 s
PageIndex (its own answer) · qwen3.8-27b	Tree	92% (91–94%)	193,494	368.24 s
PageIndex · qwen3.8-27b	Tree	81% (78–83%)	197,603	570.80 s
RAPTOR · qwen3.8-27b	Tree	69% (66–72%)	77,141	299.38 s
Hybrid + reranker	Vector	66% (62–69%)	83,817	294.26 s
Vectors → tree · qwen3.8-27b	Hybrid	65% (61–68%)	145,944	594.05 s
Chunks + embeddings	Vector	54% (51–57%)	40,126	305.75 s

Tokens, the word pieces a model reads and writes, are the measure to compare. PageIndex used 198,000 tokens per question against 84,000 for hybrid search, and made 4.8 model and service calls in a row against 3 (an embedding, a reranking and an answer). Whole-document reading used 93,000 tokens per question on the documents it could take, fewer than PageIndex. Times are shown but not comparable: the methods ran side by side, in changing combinations, on one shared model server that was often saturated, so much of a call’s time was spent waiting in a queue.

Where each method wins

The answer depends on what makes the documents hard.

Accuracy of each method by question regime. Darker is more accurate; the regime table below gives each difference from hybrid retrieval with a reranker.

Row	Many near-identical documents	Few very long documents	FinanceBench claim check
Whole document	98%	96%	84%
PageIndex (its own answer) · qwen3.8-27b	94%	90%	87%
PageIndex · qwen3.8-27b	82%	84%	72%
RAPTOR · qwen3.8-27b	65%	83%	69%
Hybrid + reranker	57%	86%	72%
Vectors → tree · qwen3.8-27b	55%	87%	74%
Chunks + embeddings	42%	81%	67%

Look-alike documents (546 questions over 78 final terms from three issuers; 68 of them also need the issuer’s prospectus). PageIndex beat hybrid search by +25 points (95% CI +19 to +30). The retrieved evidence shows why. On questions about one product, only 12% of the passages hybrid search returned came from that product’s document, spread over about 25 documents with the same wording. Of the pages PageIndex’s agent read, 98% came from the right document; on another 85 of these questions it read no page at all and answered from its section summaries (see Method). Naming the product by its ISIN, a unique identifier, instead of describing it helped hybrid search (60% against 54%) and PageIndex (85% against 79%) alike, so it did not close the gap. Vectors → tree did no better than hybrid search (−2 points): one embedding per document is a coarse way to choose 8 candidates, and on 158 of 478 single-product questions the right document was not among them. Whole-document reading scored 98%, but it is handed the right document: it shows what reading achieves once the document is chosen, not how to choose it.

Long prospectuses (183 questions over 9 base prospectuses, securities notes and registration documents of 184 to 1,709 pages: the long legal documents under which an issuer’s products are offered). No search method differed detectably from hybrid search except naive chunk search, which was 5 points worse (fragile). Whole-document reading led by +9 points on the 138 questions whose documents fitted; that lead is also marked fragile, because it clears zero by less than the run-to-run noise, and the method is handed the right document while the others search all 9.

FinanceBench (118 questions, each asked of its own company filing, as in the vendor’s setting). PageIndex and hybrid search scored the same, 72%. PageIndex’s own answers, which come from its agent rather than our shared answer step, scored 87%: higher than our answer step given the pages PageIndex read. Its agent reads its own text extraction of the PDF, also sees the tree’s section summaries, and its answers are always graded by the judge model, whose verdicts vary more from run to run than rule-based scoring; we cannot say which of these explains the gap. It is not detectably above our answer step given the gold pages (82%). The PageIndex README credits it with 98.7% on FinanceBench; the detailed results it links are from the vendor’s Mafin 2.5 system, while we ran the open-source code with a local model.

What we predicted and what happened

The pre-registration was meant to carry the author’s own predicted ordering of the methods, written before any result. The author handed that task to Claude (Opus 5.5), which wrote the predictions after seeing the development results, with each method’s development accuracy beside them. So they are a forecast from development data, not a blind prior, and this section measures how well development predicted test.

They were revised once, after the methods were tuned on development questions and before the pre-registration was frozen. The first version, written from the untuned methods, put PageIndex about 40 points ahead of hybrid search on look-alike documents and 15 to 25 points ahead on FinanceBench; on the test they were +25 points and 0 points, so that version would have missed both. Both versions are in the harness history.

Each relation is checked as written, between neighbours in the ordering. > holds if the 95% interval of the difference excludes zero in that direction, ≥ if the first method is not detectably behind, and ≈ if the interval spans zero. The PageIndex gap holds if it falls in the predicted range and is or is not detectable, as predicted (“about +25” is read as 20 to 30).

Fourteen of the eighteen held, including all three PageIndex gaps. The four misses:

• Long prospectuses, whole document (two relations). Predicted level with hybrid search and PageIndex; it was ahead of both: +9 points (95% CI +3 to +14) over hybrid, +11 points (95% CI +6 to +17) over PageIndex. On development it had trailed hybrid search, 90% to 95%, on only 30 questions whose documents fitted.
• FinanceBench, RAPTOR. Predicted level with whole-document reading at the top; whole-document reading led it by +14 points (95% CI +8 to +22), and RAPTOR finished below hybrid search, though not detectably.
• FinanceBench, vectors → tree. Predicted ahead of chunk search; it was ahead by +7 points (95% CI −3 to +15), which is not detectable.

Three of the four misses involve whole-document reading, which development underrated outside the term sheets, where it had about 30 questions per collection to go on.

Why methods fail

Each wrong answer is put in one class, by where the method went wrong first.

“Wrong document” means none of the retrieved pages came from the right document; “wrong section or page” means some did, but not the pages holding the answer; “right evidence, wrong answer” means the answer was in front of the model and it still got it wrong. For the vector methods, mixing up look-alike products mostly shows up as a wrong page: among 45 passages, a few usually come from the right product, just not the ones that matter. “Navigator ran out of turns” means the PageIndex agent hit its built-in step limit three times on the same question.

Every question

Every question, every method’s answer and the evidence it retrieved. FinanceBench’s licence does not cover republishing its questions and answers, so for that dataset only the outcomes are shown.

Method

The design, metrics and predictions were fixed in a pre-registration before the first test question was run.

Documents and questions. Look-alike: 78 final terms (Julius Bär 31, Raiffeisen Bank International 28, Deutsche Bank 19, the last in German). Long: 9 base prospectuses, securities notes and registration documents of 184 to 1,709 pages. FinanceBench: the open 150-question set over 84 filings, each question asked of its own document. Questions were split 20% development, 80% test; the test split has 847 questions. Look-alike questions come in pairs, one naming the product by ISIN and one by description. The answer keys for the term sheets and prospectuses were drafted by a model and checked against the source pages by Claude (Opus 5.5), not by a person.

Shared pipeline. Every PDF is converted to text once, by the open-source converter Docling without OCR (no text recognition from page images), and every method reads that text, so a parser difference cannot look like a retrieval difference. The answer step is shared: one model, one prompt, up to 9,000 words of evidence from a search method. Qwen3.8-27B navigates and answers with thinking on (it reasons step by step before answering; medium effort), at temperature 0 and a fixed random seed; the same weights with thinking off build RAPTOR’s summaries and PageIndex’s trees, and judge free-text answers. Embeddings come from Qwen3-Embedding-4B and reranking from Qwen3-Reranker-4B. All models ran locally.

Tuning. Each method could try up to five settings on the development questions. Chosen: 45 passages for both vector methods, a 9,000-word budget for RAPTOR, 8 candidate documents for vectors → tree. PageIndex ran its released defaults only.

PageIndex as run. Version 0.2.21 of the open-source SDK, in local mode, pointed at our model server instead of the vendor’s hosted service. It builds its own tree from the PDF; the pages its agent asks for are taken from our shared text and passed to the shared answer step. When the agent read no page (99 of 847 questions), the answer step got what the agent had been shown instead, the documents’ descriptions and the tree’s section summaries, a rule fixed before the run. Its own answer, written from its own text extraction of the PDF, is kept and scored separately as “PageIndex (its own answer)”, reported beside the methods, not as one of them, and is always graded by the judge. When the agent ran out of turns it was retried up to three times with the same settings, then scored wrong.

RAPTOR as run. Our implementation, simplified from the paper: seeded k-means clustering instead of UMAP with Gaussian mixtures, so each passage belongs to one group, and a word budget instead of a token budget.

Scoring. Identifiers and dates by exact match (English and German forms), numbers within ±0.5% or the precision the answer key states, free text by the judge model. The judge agreed with an independent grader on 117 of 120 development answers (Cohen’s κ 0.895, a measure of agreement beyond chance); that grader was Claude (Opus 5.5), so this is agreement between two models, not with a person. An unanswerable question is scored right only when the method says the answer is not in the document.

Statistics. Each method’s difference from hybrid search is paired by question, with a 95% interval from 10,000 bootstrap resamples (resampling the questions with replacement). A difference counts only if its interval excludes zero. The model server is not deterministic even at temperature 0: in a check before the test run, the same question flipped verdict 3.6% of the time when scored by rule and 18% when scored by the judge (8.1% overall). From that, run-to-run noise on a difference is about ±2 points overall and on look-alike documents, ±3 on long prospectuses and ±8 on FinanceBench; a win that clears zero by less is marked fragile (†).

Reproduction. A clean copy of the code at the run’s commit reran 21 test questions with every method (the pre-registration said 20 random questions; we took the 21 that a fixed hash of the question id puts in one shard, chosen before looking at them); 96.4% of the 168 verdicts matched the full run (the pre-registered bar was 80%). It reused the full run’s indexes, so it reproduces search, answering and judging, not index building.

Deviations from the pre-registration.

• One FinanceBench filing is AES-encrypted, and its PageIndex tree failed to build for a missing decryption package in our setup. We added the package and rebuilt the tree with the same settings; no row had been scored under the failure.
• One whole-document question failed every time: its document is under the 150,000-word limit but needs more tokens than the model’s window. It is counted as not applicable, the limit’s intent, rather than left as an error.
• The answer keys were checked by a model and the judge was validated against a model, both by the author’s decision. The predictions are Claude’s, as above.

Run manifest sha256:cd9177958fbb4ec7aedbbd2cf50399f7d38df44feeacbecea1dce71111297358, data version 2026-10-14.1, 6,776 rows.

Limitations

• The judge under-counts PageIndex’s own refusals. On 15 unanswerable questions PageIndex’s own answer was scored as answering anyway. Reading them after the run, we count 12 that say in prose that the document has no such feature (“does not have an autocall trigger level”), which should count as right. That would lift its score by up to 1.4 points overall. Scores were not changed after the run.
• At least one answer key is wrong. One question marked unanswerable is answered in a footnote of its document, and both vector methods found it. Answer keys checked by a model will hold more such errors, and an error counts against whichever methods found the true answer.
• One model family. Every role is played by one open model. A larger or different model may navigate and answer differently, and the gaps between methods may change with it.
• Small FinanceBench slice. 118 test questions, with run-to-run noise of about ±8 points on a difference.