# Continuity: Empirical Evaluation of Persistent Decision Memory for AI-Assisted Software Engineering

**Authors:** Thiago Goncalves, Hackerware LLC  
**Date:** April 30, 2026  
**Version:** 3.0 — Project Chronos (Master Benchmark Pass)  
**Benchmark Suite:** Project Chronos v3.0  
**System Under Test:** Continuity v3.0 (1,929 production decisions, 1,574 indexed files)

---

## Abstract

Large Language Model (LLM) coding assistants lose architectural context across sessions due to fixed context windows and the "Lost in the Middle" degradation phenomenon (Liu et al., 2023). Continuity addresses this by providing a local, search-based synthetic memory layer that persists decisions as structured JSON and retrieves them on demand via Unified Search — an RRF hybrid combining semantic, keyword, and tag signals with project-file snippet retrieval. This Project Chronos v3.0 pass extends the prior evaluation to a 50-query head-to-head against MemPalace and adds multi-corpus precision/MRR/hit-rate measurements, a LongMemEval recall stress test on the full 1,925-decision production corpus, and an empirical O(1) token-scaling proof. Headline results: (1) Continuity beats MemPalace 42–6–2 with average relevance 0.85 vs 0.62 and 219× faster wake-up (16ms vs 3,506ms); (2) RAGAS faithfulness 0.91, answer relevancy 0.86, context recall 0.85 (n=5); (3) RGB robustness 0.95 across noise, counter-factual, and information-refusal scenarios; (4) confirmed O(1) token scaling — flat 13.8k tokens regardless of N, yielding 98.1% savings at N=5,000; (5) ANN search latency of 14/28/43 ms at N=100/500/1,000. All RAG-quality scores are produced by live LLM judges (Claude Haiku 4.5 / Sonnet); no mocked or simulated evaluation.

**Keywords:** retrieval-augmented generation, architectural decision records, synthetic memory, information retrieval, token efficiency, LLM evaluation

---

## 1. Introduction

### 1.1 Problem Statement

AI coding assistants (Claude Code, Cursor, GitHub Copilot, Gemini) operate within fixed context windows that reset between sessions. As projects accumulate architectural decisions, two failure modes emerge:

1. **Context overflow:** Embedding all decisions in static files (e.g., CLAUDE.md) grows linearly — O(n) tokens per decision. At our current production corpus size (1,929 decisions, ≈387,729 tokens), this exceeds Claude's 200K context window by ~96%.

2. **Lost in the Middle:** Liu et al. (2023) demonstrated that LLM performance degrades when relevant information is buried in long contexts. Even when decisions fit within the window, retrieval accuracy declines as context length increases.

### 1.2 Proposed Solution

Continuity implements a search-based retrieval architecture where:
- Decisions are stored locally as structured JSON in `.continuity/decisions.json`
- Hybrid search (TF-IDF keyword + cosine similarity on MiniLM-L6-v2 embeddings) retrieves only relevant decisions on demand
- Token consumption is bounded by the retrieval budget (k results × d tokens/result), independent of corpus size n — yielding O(1) complexity with respect to n

### 1.3 Evaluation Framework

We evaluate Continuity across five dimensions using established academic benchmarks:

| Benchmark | Framework | Reference | What It Measures |
|:---|:---|:---|:---|
| B-01: Retrieval Quality | BEIR/TREC methodology | Thakur et al., NeurIPS 2021 | Precision, MRR, Hit Rate |
| B-02: RAG Evaluation | RAGAS | Es et al., EACL 2024 | Faithfulness, Relevance, Precision, Recall |
| B-03: Robustness | RGB | Chen et al., AAAI 2024 | Noise, Counterfactual, Refusal handling |
| B-05: Token Scaling | Custom + "Lost in the Middle" | Liu et al., 2023 | O(1) vs O(n) token complexity |
| B-06: Vector Performance | ANN-Benchmarks methodology | Aumüller et al., 2020 | Embedding latency, search throughput |

All LLM-as-judge evaluations were performed by **Claude Haiku 4.5** (model: `claude-haiku-4-5-20251001`) via the Anthropic API, with no mocked or simulated scores.

---

## 2. Experimental Setup

### 2.1 System Configuration

| Component | Specification |
|:---|:---|
| Embedding Model | Xenova/all-MiniLM-L6-v2 (384 dimensions, L2-normalized) |
| Embedding Runtime | Transformers.js (ONNX, local CPU inference) |
| Search Algorithm | Hybrid: TF-IDF (field-weighted) + cosine similarity |
| Keyword Weights | Question: 0.9×, Tag: 3.0×, Answer: 1.0× |
| Hybrid Balance | 0.5 (equal keyword/semantic weighting) |
| Token Counter | Tiktoken (cl100k_base encoding, same as Claude) |
| LLM Judge | Claude Haiku 4.5 via Anthropic API |
| Hardware | Apple Silicon (macOS Darwin 25.3.0) |

### 2.2 Corpus

| Parameter | Value |
|:---|:---|
| Synthetic corpora (multi-corpus benchmark) | Web API (200), Data Pipeline (150), Mobile App (100) — see §3.7 |
| Production corpus (head-to-head v3) | 1,929 decisions, 1,574 indexed files |
| Production corpus (LongMemEval, earlier snapshot) | 1,925 decisions |
| O(1) token-scaling synthetic series | 100 / 1,000 / 2,000 / 5,000 decisions — see §3.4 |
| Average tokens/decision | 201 (production), 141 (synthetic) |
| Retrieval budget | k = 15 results, q = 3 queries/session |

### 2.3 Baselines

| Baseline | Description | Complexity |
|:---|:---|:---|
| Full Context | Embed all n decisions in prompt | O(n) |
| Random-k | Retrieve k random decisions | O(1) but uninformed |
| Exact Match | Substring matching | O(1) but low recall |
| BM25 | Standard term-frequency IR | O(1) |

---

## 3. Results

### 3.1 Benchmark 01: Retrieval Quality

**Framework:** BEIR/TREC methodology (Thakur et al., NeurIPS 2021)  
**Corpus:** 1,000 synthetic decisions across 27 technology domains and 13 architecture patterns  
**Queries:** 40 (technology keywords + architecture patterns + project-specific tags)

| Metric | Continuity Score | BEIR Median¹ | Interpretation |
|:---|:---|:---|:---|
| **Precision@5** | **1.000** | 0.45–0.65 | Every top-5 result is relevant |
| **MRR** | **1.000** | 0.55–0.75 | First result is always relevant |
| **Hit Rate** | **100%** | 70–85% | Every query finds at least one relevant decision |
| **Avg Latency** | **8.86 ms** | N/A (varies) | Real-time interactive performance |

¹ BEIR median ranges from Thakur et al. (2021) across 18 datasets for zero-shot retrieval models. Direct comparison is approximate — BEIR evaluates generalized cross-domain retrieval, while Continuity operates on a domain-specific corpus with structured metadata (tags, entities).

**Per-query breakdown (40 queries):**

| Query Category | Queries | Precision@5 | MRR | Avg Latency |
|:---|:---|:---|:---|:---|
| Technology (React, Docker, etc.) | 17 | 1.000 | 1.000 | 8.4 ms |
| Architecture (CQRS, Microservices, etc.) | 10 | 1.000 | 1.000 | 8.5 ms |
| Project Tags (bug-fix, security, etc.) | 13 | 1.000 | 1.000 | 7.3 ms |

**Discussion:** The perfect scores reflect the advantage of domain-specific structured metadata. Decisions include explicit tags, entity annotations, and relationship links that augment the search signal beyond raw text similarity. In BEIR's zero-shot cross-domain setting, models lack this structured metadata, explaining the lower baselines. A fairer comparison would be BEIR's domain-adapted scores (typically 0.60–0.80 nDCG@10), against which Continuity's 1.000 remains strongly competitive.

---

### 3.2 Benchmark 02: RAG Evaluation (RAGAS Framework)

**Framework:** RAGAS (Es et al., EACL 2024)  
**Judge:** Claude Haiku 4.5 (live API, not mocked)  
**Queries:** 5 technology queries with full retrieval + generation + evaluation pipeline  
**Generation Model:** Claude Haiku 4.5 (same model for generation and judgment)

| Metric | Continuity Score (v3.0) | RAGAS Typical Range² | Interpretation |
|:---|:---|:---|:---|
| **Faithfulness** | **0.91** | 0.70–0.90 | Answer is almost entirely grounded in retrieved context |
| **Answer Relevancy** | **0.86** | 0.60–0.85 | Answers directly address the query intent |
| **Context Precision** | **0.78** | 0.40–0.70 | Low noise; retrieved items are highly relevant |
| **Context Recall** | **0.85** | 0.60–0.85 | Successfully retrieves necessary ground truth |

² Typical ranges compiled from RAGAS leaderboard submissions and Es et al. (2024) baselines across enterprise RAG systems.

**Detailed LLM Judge Reasoning (selected):**

> *"Faithfulness is high because the answer accurately derives information from the provided context without adding unsupported claims. Context Precision is low due to severe redundancy — the context contains nearly identical Q&A pairs with only pattern names varying."* — Claude Haiku 4.5 on the "Docker" query

**Analysis:**

The **faithfulness score of 0.95 exceeds the typical enterprise range** (0.70–0.90), confirming that Continuity's retrieved context effectively grounds LLM responses with near-zero hallucination.

The lower context precision (0.35) is an artifact of the **synthetic benchmark corpus**, which contains templated decisions with high structural similarity. The LLM judge correctly identified this: *"The context contains 82 nearly identical Q&A pairs with only minor variations."* On a real production corpus with diverse decision content, precision scores would be higher. This is a known limitation of synthetic evaluation corpora in RAG benchmarks (Saad-Falcon et al., NAACL 2024).

The moderate answer relevance (0.67) reflects the **vagueness of single-keyword queries** (e.g., "React" rather than "Why did we choose React for the frontend?"). The judge noted: *"The query is extremely vague and lacks specificity. The answer provides reasons for choosing [technology] but doesn't address what [technology] is."* Natural user queries in practice are more specific, yielding higher relevance.

---

### 3.3 Benchmark 03: RGB Robustness (Chen et al., AAAI 2024)

**Framework:** RGB — Retrieval-Augmented Generation Benchmark (Chen et al., AAAI 2024)  
**Judge:** Claude Haiku 4.5 (live API)  
**Tests:** 3 robustness abilities, each with curated scenarios

| Ability | Continuity Score (v3.0) | RGB Baseline Range³ | Interpretation |
|:---|:---|:---|:---|
| **Noise Robustness** | **0.95** | 0.60–0.85 | Successfully ignores irrelevant retrieved decisions |
| **Counter-factual Robustness** | **0.95** | 0.50–0.75 | Resolves temporal conflicts using metadata; prioritizes new standards over deprecated ones |
| **Information Refusal** | **0.95** | 0.40–0.70 | Correctly refuses to answer when context is insufficient |
| **Mean Robustness** | **0.95** | 0.50–0.77 | Three-way tie across all three RGB abilities |

³ RGB baseline ranges from Chen et al. (2024) across GPT-3.5-turbo, GPT-4, and Llama-2-70B on the RGB benchmark dataset.

**Detailed LLM Judge Reasoning:**

**Noise Robustness (0.95):**
> *"The system demonstrated excellent robustness by correctly identifying and utilizing only the relevant source about Redux and Redux Toolkit for React state management. It successfully ignored two irrelevant noise sources: Redis caching (backend infrastructure) and GitHub Actions CI/CD pipeline (deployment process)."*

**Counter-factual Robustness (0.95):**
> *"The system demonstrated strong robustness by correctly identifying PostgreSQL as the primary database from the most recent and authoritative source. It properly handled conflicting information by: (1) prioritizing the current decision over outdated information, (2) acknowledging the rejected alternative (MongoDB), and (3) contextualizing historical information."*

**Information Refusal (0.95):**
> *"The system demonstrated excellent refusal behavior by correctly identifying that the context lacked information about team size/staffing. The response appropriately declined to answer the specific question while providing a clear explanation of what information was and wasn't available."*

**Discussion:** Continuity's scores **exceed RGB baselines across all three abilities**. The noise robustness result (0.95) is particularly significant — it demonstrates that the hybrid search engine's relevance scoring effectively filters irrelevant decisions even when they are injected into the retrieval results. The counter-factual robustness score (0.95, up from 0.85 in v1.0) validates Continuity's temporal awareness: the system correctly uses decision timestamps and supersession metadata to resolve contradictions, a capability not present in general-purpose RAG systems.

---

### 3.4 Benchmark 05: Token Scaling — O(1) vs O(n) Proof

**Framework:** Custom scaling analysis, motivated by Liu et al. (2023) "Lost in the Middle"  
**Tokenizer:** Tiktoken (cl100k_base, exact token counts)  
**Corpus sizes:** N ∈ {10, 50, 100, 500, 1,000, 2,000, 5,000}

| Corpus Size (N) | Full Context (tokens) | Continuity (tokens) | Savings | Efficiency Ratio |
|:---|:---|:---|:---|:---|
| 100 | 21,138 | 13,841 | 34.5% | 1.53× |
| 1,000 | 148,239 | 13,856 | **90.6%** | 10.70× |
| 2,000 | 289,114 | 13,849 | **95.2%** | 20.88× |
| 5,000 | 712,236 | 13,850 | **98.1%** | 51.42× |

**Regression analysis:**

```
Full Context:  T(n) = 141.3n + 7,065    R² = 0.9999    ∈ O(n)
Continuity:    T(n) = −0.002n + 13,870   R² = 0.0012    ∈ O(1)
```

**Break-even point:** N ≈ 46 decisions. Below 46 decisions, the retrieval overhead exceeds the corpus size, making full-context embedding more efficient. Above 46 decisions, Continuity provides strictly better token efficiency.

**Context window analysis:**

| Method | Max Decisions in 200K Context | At N = 1,925 (current production) |
|:---|:---|:---|
| Full Context | ~968 | Exceeds limit (~392,202 tokens, ~196% of context) |
| Continuity | Unlimited (bounded by k, not n) | ~13,850 tokens (6.9%) |

**Comparison with industry standards:**

| System | Approach | Token Complexity | Max Corpus |
|:---|:---|:---|:---|
| CLAUDE.md / .cursorrules | Full embedding | O(n) | ~968 decisions |
| Claude Code Auto-Memory | Full embedding | O(n) | ~968 decisions |
| Mem0 | Hybrid (stored + retrieved) | O(k) | Varies |
| **Continuity** | **Search-based retrieval** | **O(1)** | **Unbounded** |

**Discussion:** The O(1) property is not approximate — Continuity's token consumption is mathematically independent of corpus size for a fixed retrieval budget. The regression coefficient on n is -0.002 (effectively zero), with R² = 0.001, confirming no linear relationship between corpus size and token usage. This result directly addresses the "Lost in the Middle" problem: by retrieving only the k most relevant decisions, Continuity avoids the attention degradation that occurs with long contexts.

---

### 3.5 Benchmark 06: ANN Vector Search Performance

**Framework:** ANN-Benchmarks methodology (Aumüller et al., Information Systems 2020)  
**Model:** Xenova/all-MiniLM-L6-v2 (384-dim, ONNX via Transformers.js)  
**Search:** Brute-force cosine similarity (exact, no approximate indexing)

| Corpus Size | Search Latency (top-15) |
|:---|:---|
| 100 | **14 ms** |
| 500 | **28 ms** |
| 1,000 | **43 ms** |

Project Chronos v3.0 measures only the ANN search step (vector similarity over the cached embedding store). The embedding-generation cost reported in earlier versions has been excluded because embeddings are precomputed at ingest time and cached on disk; it is not part of the query-time critical path.

**Comparison with ANN-Benchmarks baselines⁴:**

| Method | Recall@10 | Query Latency (1K docs) | Index Type |
|:---|:---|:---|:---|
| Brute-force (baseline) | 1.000 | 10–50 ms | Exact |
| HNSW (Aumüller et al.) | 0.99+ | 0.1–1 ms | Approximate |
| Annoy | 0.95–0.99 | 0.5–5 ms | Approximate |
| **Continuity** | **1.000** | **46.4 ms** | **Exact** |

⁴ ANN-Benchmarks results from ann-benchmarks.com for 384-dimensional vectors. Exact comparisons depend on hardware; ratios are more meaningful than absolute numbers.

**Discussion:** Continuity uses exact brute-force search, which provides perfect recall (1.000) at the cost of higher latency compared to approximate methods like HNSW. At 46ms for 1,000 decisions, the latency remains well within interactive thresholds (< 100ms). For corpora exceeding 10,000 decisions, migrating to an HNSW index would reduce latency to sub-millisecond while maintaining 0.99+ recall — a straightforward optimization that does not require architectural changes.

The embedding generation cost (19–44ms per decision on CPU) is a one-time cost amortized over the decision's lifetime. Embeddings are cached in `.continuity/embeddings.json` and only regenerated on decision update.

---

### 3.6 Benchmark 07: Head-to-Head v3 vs MemPalace (Unified Search)

**Framework:** Side-by-side blind judging against the MemPalace open-source memory system.
**Judge:** Claude Sonnet (single-judge, per-query winner + 0–1 relevance scores; ties awarded when neither side dominates).
**Workspace:** Continuity's own production codebase — 1,929 decisions, 1,574 indexed source files.
**Queries:** 50 project-rationale queries.

| Metric | Continuity (Unified) | MemPalace |
|:---|:---|:---|
| **Query Wins (n=50)** | **42** | 6 |
| **Ties** | 2 | 2 |
| **Avg Relevance (0–1)** | **0.85** | 0.62 |
| **Avg Latency (Query)** | **138 ms** | 1,966 ms |
| **Avg Tokens / Search** | **686** | 965 |
| **Wake-Up Latency** | **16 ms** | 3,506 ms |
| **Wake-Up Tokens** | **813** | 817 |

**Key findings:**

- Continuity is **219× faster** on session wake-up (16 ms vs 3,506 ms) and ~14× faster per query (138 ms vs 1,966 ms).
- Higher Relevance: Unified Search (semantic + keyword + tag RRF, fused with project-file snippets) achieved an 85% average relevance score, outperforming MemPalace's 62% on rationale-centric queries.
- The win split (42 / 6 / 2) is consistent with the §4.3 finding from earlier whitepaper passes that on rationale-heavy mixes a small targeted index beats a large generalised one. We continue to recommend treating this as suggestive and welcome a third-party query set.

---

### 3.7 Benchmark 08: Multi-Corpus Retrieval

**Setup:** Three synthetic technical-domain corpora to validate that retrieval quality is not specific to Continuity's own dogfooded codebase.

| Corpus | Decisions | Precision@5 | MRR | Hit Rate |
|:---|---:|---:|---:|---:|
| **Web API** | 200 | 82.73% | 0.9773 | 97.73% |
| **Data Pipeline** | 150 | 80.95% | 0.8976 | 92.86% |
| **Mobile App** | 100 | 79.88% | 0.9405 | 95.24% |

P@5 stays in the 80–83% band across three unrelated technical domains; MRR≥0.90 in all three, indicating the first relevant result is consistently in the top two positions.

---

### 3.8 Benchmark 09: LongMemEval — Long-Term Memory Stress Test

**Setup:** Stress test against the full production corpus (1,925 decisions). LongMemEval-style query mix tests whether the system can recall the right decision when it is one in nearly two thousand.

- **R@5 (Recall at 5):** 69.8%
- **R@10 (Recall at 10):** 77.2%
- **Exact Match Recall (R@5):** 87.5%
- **Paraphrased Recall (R@5):** 66.0%
- **Cross-Reference Recall (R@5):** 53.0%
- **Vague Query Recall (R@5):** 44.0%

The exact-match floor of 87.5% is the practically-meaningful number for users issuing well-formed queries; the 44.0% vague-query result is the natural floor for under-specified inputs and is precisely the failure mode the in-loop retrieval pattern (§4 of the whitepaper) is designed to bypass.

**Methodology notes**

- **Vague-query ambiguity ceiling.** Vague queries are generated by extracting a single tag or keyword from a source decision. Without a ceiling, top tags like `architecture` (451 entries) and `mcp` (812 textual matches) produce queries with hundreds of equally relevant "correct" answers — making R@10 structurally bounded by ~10/N regardless of ranking quality. This benchmark filters vague-query terms to those appearing in ≤5% of the corpus (≤96 decisions for N=1,925), so the result reflects ranking quality rather than collision rate.
- **Redundancy dedup.** Search results are filtered for paraphrase-level duplicates by cosine similarity on embeddings (threshold 0.92). The threshold is tuned to keep superseded / near-identical decisions in the same cluster visible (e.g. `supersedes` chains, intentional conflict-test variants), collapsing only true paraphrase duplicates.

---

### 3.9 Benchmark 10: Cognitive Firewall — Rule Enforcement

**Setup:** Clean-room synthetic corpus testing for rule enforcement and conflict resolution — distinct from RGB in that it measures whether the agent obeys explicit project rules rather than handling adversarial retrieval inputs.

| System | Avg Score | Compliance |
|:---|:---|:---|
| **Continuity** | **1.00** | **100%** |
| Standard RAG (baseline) | 1.00 | 100% |

- **Architectural Trap:** Correctly mandated `FetchWrapper` and forbade `axios`.
- **Conflict Resolution:** Correctly prioritised PostgreSQL (new standard) over MySQL (deprecated).
- **Information Refusal:** Correctly refused to answer quantum-propulsion queries with "I don't know."

The point of the comparison is parity: the cognitive-firewall scenarios are exactly the cases where a structured decision store and a vanilla embedding RAG should both score well. They do.

---

## 4. Threats to Validity

### 4.1 Internal Validity

- **Synthetic corpus limitations:** RAGAS context precision (0.35) was depressed by high redundancy in the generated test corpus. The LLM judge explicitly identified this: *"The context contains nearly identical Q&A pairs."* Production corpora with diverse, human-authored decisions would yield higher precision scores.

- **Single-keyword query design:** RAGAS answer relevance (0.67) was reduced by vague queries ("React", "Docker") rather than natural user queries ("Why did we choose React for the frontend?"). This is a benchmark design limitation, not a system limitation.

- **Same-model evaluation:** Using Claude Haiku 4.5 for both generation and judgment introduces potential scoring bias. Mitigation: the judge was given explicit rubrics and provided detailed reasoning (included in results) that can be independently verified.

### 4.2 External Validity

- **Domain specificity:** Results are measured on software engineering decisions. Continuity supports 6 domain profiles (software, writing, research, medical, legal, general), but non-software domains were not benchmarked.

- **Hardware specificity:** Latency measurements are on Apple Silicon. Performance on different hardware will vary proportionally.

### 4.3 Construct Validity

- **BEIR comparison caveats:** Continuity's perfect retrieval scores benefit from structured metadata (tags, entities) that general BEIR datasets lack. The comparison demonstrates Continuity's advantage in its target domain, not that it would achieve 1.000 on BEIR's cross-domain datasets.

- **RGB baseline range:** Chen et al. (2024) tested general-purpose LLMs without domain-specific retrieval. Continuity's higher scores reflect both the retrieval quality and the domain-specific nature of the decision corpus.

---

## 5. Comparison with Related Systems

### 5.1 Against Static Context Approaches

| Capability | CLAUDE.md | .cursorrules | Mem0 | **Continuity** |
|:---|:---|:---|:---|:---|
| Storage format | Markdown | Text | Cloud KV | JSON + Markdown |
| Token complexity | O(n) | O(n) | O(k) | **O(1)** |
| Max decisions (200K context) | ~968 | ~968 | N/A (cloud) | **Unbounded** |
| Semantic search | No | No | Yes | **Yes** |
| Contradiction detection | No | No | No | **Yes (0.85 robustness)** |
| Information refusal | No | No | No | **Yes (0.95 accuracy)** |
| Cross-tool support | Claude only | Cursor only | API-dependent | **6 tools via MCP** |
| Local-only storage | Yes | Yes | No (cloud) | **Yes** |
| Decision relationships | No | No | No | **Yes (5 relationship types)** |
| Quality scoring | No | No | No | **Yes (5 dimensions)** |

### 5.2 Against RAG Benchmarks (Literature Comparison)

| Metric | Continuity (v3.0) | RAGAS Baselines (Es et al.) | RGB Baselines (Chen et al.) | Interpretation |
|:---|:---|:---|:---|:---|
| Faithfulness | **0.91** | 0.70–0.90 | N/A | Top of typical enterprise range |
| Answer Relevancy | **0.86** | 0.60–0.85 | N/A | Top of typical range |
| Context Recall | **0.85** | 0.60–0.85 | N/A | Top of typical range |
| Noise Robustness | **0.95** | N/A | 0.60–0.85 | Exceeds reported range |
| Counter-factual Robustness | **0.95** | N/A | 0.50–0.75 | Substantially exceeds reported range |
| Information Refusal | **0.95** | N/A | 0.40–0.70 | Significantly exceeds range |
| Token Efficiency (N=5K) | **98.1%** | N/A | N/A | No comparable benchmark exists |

---

## 6. Production Validation

Beyond synthetic benchmarks, Continuity has been validated in production use:

| Metric | Value |
|:---|:---|
| Production decisions | 1,929 |
| Indexed project files | 1,574 |
| Continuity session cost | ~13,850 tokens (flat, regardless of N) |
| Full-context equivalent at N=1,925 | ~285,000 tokens (estimated, exceeds 200K limit) |
| Production savings | ≈95% (extrapolated from O(1) scaling table) |
| Marketplace version | v2.20.62-beta and later |
| MCP tools | 59 across 8 modules |
| Search modes | Unified Search (RRF: semantic + keyword + tag, fused with file snippets) |
| Domain profiles | 4 (software, writing, research, general) |

---

## 7. Conclusion

This evaluation provides empirical evidence for five claims:

1. **Retrieval quality is excellent:** P@5 = 1.000, MRR = 1.000 across 40 queries, exceeding BEIR median baselines for domain-adapted retrieval.

2. **RAG faithfulness is high:** 0.91 faithfulness, 0.86 answer relevancy, 0.78 context precision, 0.85 context recall — at or above the top of the typical enterprise range of 0.70–0.90 (Es et al., 2024). Retrieved context effectively grounds LLM responses with minimal hallucination.

3. **Robustness exceeds published baselines:** Mean robustness of 0.95 across noise (0.95), counter-factual (0.95), and refusal (0.95) tests, substantially exceeding RGB baselines of 0.50–0.77 (Chen et al., 2024).

4. **O(1) token scaling is confirmed:** Token consumption is flat at ~13,850 tokens regardless of corpus size (R² = 0.001 on n), with 98.1% savings at N = 5,000 and a break-even point of N = 46 decisions.

5. **Latency supports interactive use:** 14 / 28 / 43 ms ANN search at N = 100 / 500 / 1,000, suitable for real-time IDE integration.

6. **Beats MemPalace on rationale-centric queries:** 42–6–2 head-to-head v3 win, 0.85 vs 0.62 average relevance, 219× faster wake-up (16 ms vs 3,506 ms). See §3.6 and the full whitepaper §4.3 for the methodology and caveats.

### 7.1 Limitations and Future Work

- **Expand RAGAS evaluation** to 50+ queries with natural-language formulations on the production corpus to obtain more representative context precision and answer relevance scores.
- **Cross-domain benchmarking** on non-software decision corpora (medical, legal) to validate domain profile effectiveness.
- **NLI-based contradiction detection** benchmark (SNLI/MNLI methodology) for the TensionDetector subsystem.
- **Quality scoring calibration** against human annotations (Cohen's κ, Pearson r).
- **HNSW indexing** evaluation for corpora exceeding 10,000 decisions.

---

## References

1. Thakur, N., et al. (2021). "BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models." *NeurIPS 2021 Datasets and Benchmarks Track.*

2. Es, S., et al. (2024). "RAGAS: Automated Evaluation of Retrieval Augmented Generation." *Proceedings of EACL 2024.*

3. Chen, J., et al. (2024). "Benchmarking Large Language Models in Retrieval-Augmented Generation." *Proceedings of AAAI 2024.*

4. Liu, N.F., et al. (2023). "Lost in the Middle: How Language Models Use Long Contexts." *arXiv preprint arXiv:2307.03172.*

5. Aumüller, M., et al. (2020). "ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms." *Information Systems, 87.*

6. Saad-Falcon, J., et al. (2024). "ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems." *Proceedings of NAACL 2024.*

7. Bowman, S.R., et al. (2015). "A Large Annotated Corpus for Learning Natural Language Inference." *Proceedings of EMNLP 2015.*

---

## Appendix A: Raw Benchmark Data

All raw benchmark results are available in machine-readable JSON format:

| File | Contents |
|:---|:---|
| `benchmarks/reports/bench-01-retrieval.json` | Per-query retrieval metrics (40 queries) |
| `benchmarks/data/ragas-results.json` | RAGAS scores with full LLM judge reasoning |
| `benchmarks/data/rgb-results.json` | RGB robustness scores with full LLM judge reasoning |
| `benchmarks/data/benchmark-05-results.json` | Token scaling measurements at 7 corpus sizes |
| `benchmarks/data/benchmark-06-results.json` | ANN embedding and search latency measurements |

## Appendix B: Reproducibility

To reproduce these results:

```bash
cd benchmarks/
npm install
npm run build
# Set ANTHROPIC_API_KEY for live LLM scoring (otherwise falls back to heuristic mock)
export ANTHROPIC_API_KEY=<your-key>
node dist/bench-01-retrieval.js    # Benchmark 01: Retrieval Quality
node dist/index.js                  # Benchmarks 02, 03, 05, 06
```

**Hardware requirements:** Node.js 18+, ~2GB RAM for Transformers.js model loading.  
**API requirements:** Anthropic API key for live RAGAS/RGB evaluation (~3,600 tokens per RAGAS query, ~2,400 tokens per RGB scenario).  
**Expected runtime:** ~120 seconds total (70s for ANN embedding generation, 30s for LLM evaluations, 20s for retrieval + scaling).

---

*Benchmark suite: Project Chronos v3.0 | System: Continuity v3.0 | Judges: Claude Sonnet (head-to-head) + Claude Haiku 4.5 (RAGAS/RGB) + Gemini 2.5 Flash (inter-judge) | Date: April 30, 2026*