Original-question accuracy
236 of 503 LongBench v2 questions
Research project · Autonomous LLM infrastructure
A semantic cache built for agents that reason over long context.
This project explores a two-stage cache that reuses verified LLM work without confusing similar-looking questions. It combines local retrieval, model-based equivalence checks, source provenance, and persistent knowledge extraction.
QUERY“Report the monthly recurring revenue.”
DRAGNET5 candidates · local FAISS
SNIPERsemantic equivalent · verified
CACHE HITsource-scoped · provenance attached
Repeat-query hit rate
1,000 of 1,006 exact + semantic repeats
Semantic-hit fidelity
497 of 497 hits matched the original prediction
Fewer input tokens
vs. repeating full context for all three query forms
The problem
Vector similarity is useful. On its own, it is not safe enough.
Autonomous agents repeat work. A cache can save time and compute, but a naive vector lookup may treat “include timeout errors” and “exclude timeout errors” as the same request.
The system separates recall from judgment: a fast local Dragnet finds plausible matches, then a focused Sniper decides whether a candidate asks for exactly the same information.
How it works
One query, four disciplined stages.
Cache reuse comes first. Expensive reasoning only happens when the system cannot prove that earlier work applies.
Check
Look for exact, meaning-equivalent, or previously extracted knowledge within the active corpus.
Retrieve
Use local Qwen3 embeddings and FAISS to cast a wide net, then rerank the strongest evidence.
Reason
Route misses to an executor that reads selected context iteratively and synthesizes a grounded answer.
Verify + learn
Check provenance and model consensus, store the answer, and index atomic facts for future queries.
Two indexes, two kinds of reuse
Answers are cached. Knowledge is decomposed.
The answer cache serves exact or meaning-equivalent requests. A separate fact index turns useful answers into atomic subject–relation–object triples, so later questions can reuse facts that were buried inside a longer response.
Cached answer
“The filing reports ARR of $12.4M and identifies the CFO as Jordan Lee.”
Companyreported ARR$12.4MCompanyhas CFOJordan LeeSafety boundary
Corpus namespace isolation
Cache entries and extracted facts remain tied to the document set that produced them. A load-time mismatch is refused rather than silently mixing knowledge across domains.
Context control
Collapse guard
Oversized cached results are flagged as ephemeral or recursively summarized before they can overwhelm the calling agent’s working context.
LongBench v2 · 20260716T204202Z
What the full benchmark showed.
503 independent questions were each evaluated as an original, exact repeat, and semantic variation: 1,509 scored rows in total. Repeat rows measure reuse, not additional benchmark accuracy.
Original-question accuracy
46.9%
All 503 originals produced valid A–D predictions. The result is competitive for a deliberately difficult long-context benchmark, but official comparisons are contextual—not controlled head-to-head tests.
53.7% human experts50.1% best direct result at publication25.0% random choice
Accuracy by task family
Performance was uneven by domain.
| Query form | Rows | Accuracy | Cache hit rate | Model calls | Input tokens | Avg. latency |
|---|---|---|---|---|---|---|
| Original | 503 | 46.9% | 0% | 6,837 | 173.63M | 67.32s |
| Semantic | 503 | 46.9% | 98.8% | 601 | 2.09M | 1.08s |
| Exact | 503 | 46.9% | 100% | 0 | 0 | 0.02s |
Token economics
The baseline changes the conclusion.
The run demonstrates amortization across repeated queries. It does not establish lower dollar cost: pricing, GPU-hours, energy, and utilization were not recorded.
tokens processed by executor and evaluator calls
Observed input was 38.09M lower, a 17.8% reduction
Observed input was 146.6% higher, or 2.47× this baseline
179.3MTotal tokens
7,438API calls
9.56hCumulative row time
23 / 509Miss-path rows stopped early
What we learned
Strong reuse. Expensive cold paths. Clear next steps.
Semantic reuse can stay faithful.
All 497 semantic cache hits reproduced the paired original prediction. The six misses produced two improvements and two regressions, leaving net accuracy unchanged.
The first read remains the bottleneck.
The ten most expensive originals generated 27.1% of original input tokens. Code-repository tasks were 9.9% of questions but 40.7% of original input.
Cost needs direct measurement.
The next defensible comparison needs a measured full-context control plus GPU-hours, energy, utilization, and per-model pricing—not token totals alone.
Core stack
Local retrieval. Routed reasoning. Reproducible evaluation.
Qwen3 Embedding 0.6BQwen3 Reranker 0.6BFAISS IndexFlatIPAnthropic + OpenAI-compatible APIsPythonLongBench v2