Brain. Memory API

Docs / Measurements

Measurements

Dated product proof and its dataset tags.

Source: docs/initiatives/memory-api/00_MEASUREMENTS.md — this page is rendered from it, not copied. Where the home page's board numbers come from, with the dataset tag and the command that reproduces each one.

Date: 2026-07-26 · Status: measured, reproducible · Dataset tag: real founder corpus, Cloud SQL brain_rebuild, user founder, live spine (events 42,081 · memories 2,365 · search_chunks 11,216 · embeddings 10,643 · kg_entities 408 · people 1,460 · fact_assertions 7,042).

Gates every number in the API design. Two of the three planned experiments ran; the third (contextual retrieval) is scoped at the end and has not run.

Reproduce:

cURL
.venv/bin/python scripts/measure_fast_context.py --out artifacts/FAST_CONTEXT_2026_07_26.json

Artifact: artifacts/FAST_CONTEXT_2026_07_26.json (per-case grades, windows, latencies).

1. The headline: generation is currently SUBTRACTIVE

The question was whether a fast Context — retrieved, cited evidence that composes nothing — still carries the answer.

CORRECTED 2026-07-29. This section said "zero generative calls" and the table below recorded 0. Both were false when written: _envelope returned a hardcoded 0 while brain/retrieve/rewrite.py:expand() called Gemini on every request. The real cost was 2 — one embedding, one generative rewrite. The rewrite has since been removed from the fast path (artifacts/_ablation_fast_rewrite/, branch DROP), so fast now spends 1 model call: the query embedding, which the ledger charges. The carry figures below are unaffected — they were graded on retrieved windows, and the rewrite did not move them (clean arm 1-1, p=0.75). Only the COST column was wrong.

AND THEN THE RECEIPT SPLIT, 2026-07-29 (later the same day). For a few hours this note said that one call was what cost.generative_calls reports. It is not, and that was the same category error one layer up: an embedding is not a generative call. CostReport now carries three fields, and a ready fast Context reports model_calls_total: 1 · embedding_calls: 1 · generative_calls: 0. Read the column below as model calls, which is what it counts and what the header now says.

Graded on the same 43 held-out founder-confirmed cases, with the ruler's

own deterministic crit-fact grader (brain.eval.heldout.grader; writer ≠ checker holds, B4).

PathModel calls AS RUN (see note)Carries / serves the answerConfidently wrong
Non-generative Context, window 12232/43 (74%)0 — structural
Non-generative Context, window 8230/43 (70%)0 — structural
Full serve path ask() (v2-40 ruler, 2026-07-24)up to 627/43 (63%)0

The evidence already contains five answers the generative path fails to serve. Route → synthesize → judge is not adding comprehension on this corpus; it is losing cases retrieval found.

And the fast tier cannot be confidently wrong, because it asserts nothing. It hands over cited evidence for the caller's model to read. It can be incomplete; it cannot fabricate. That is a structural property, not a score — which is exactly the kind of guarantee a memory API can sell.

Trap behaviour: of 8 traps, the bait text appeared in the evidence window in 1 (V2-04). That is a retrieval observation, not a B14 safety failure — conflating them would launder a retrieval fact into a safety fact, which the grader doctrine forbids.

Context size is comfortable: median 2,585 estimated tokens at window 12 (max 8,633), well inside a 4k budget.

The evidence-window budget costs 2 cases

Window 8 → 30/43; window 12 → 32/43. Two cases have their answer retrieved but crowded out of the synthesizer's evidence window. This quantifies the "serve-window crowding class" already named in status/BRAIN_STATE_OF_THE_UNION_2026_07_24.md — it is worth 2 cases, and budget is the knob that exposes the trade to the caller.

2. The latency finding: generation was never the bottleneck

The design assumed removing generation was the route to the 200 ms SLO. It is not. Retrieval alone measured p50 2,437 ms · p95 3,495 ms over the 43 cases — and note those numbers INCLUDE the generative query rewrite nobody had noticed was on the path. With it removed (2026-07-29): p50 1,433.5 ms · p95 2,163.5 ms · p99 11,073.0 ms, n=258 → see the run note. Still not 200 ms, which was the finding then and remains it now.

NAME THE RUN, 2026-07-29. Two ablations ran that day and this line quoted the first one without saying so, which reads as a contradiction of docs/status/CURRENT_TRUTH.md §1b:

artifactn (variant B, counted, error-free)p50p95p99
_ablation_fast_rewrite/20260729T024204Z_all.json (quoted above)2571,433.52,163.511,073.0
_ablation_fast_rewrite/20260729T031018Z_all.json (the quotable one, CURRENT_TRUTH.md §1b)2581,433.42,085.011,116.2

Both recomputed here from the raw artifacts with the same nearest-rank rule the harness uses (index = ceil(q·n) − 1, no interpolation, scripts/_ablation_fast_rewrite.py:448). The two agree on p50 to 0.1 ms and disagree on p95 by 79 ms — sampling, not a change in the system, and the conclusion (not 200 ms; p95 above 2 s either way) is identical. Quote §1b of CURRENT_TRUTH.md when a number leaves this repo, and always carry the run id: an unlabelled p95 is the reason this note exists.

Breakdown (12 queries, pooled connection and transport, embedding precomputed where noted):

Componentp50Note
embed_query (Vertex round trip)170 msp95 181 ms. Not the problem, and cacheable — agents repeat queries.
Lanes + fusion1,586 msThe bottleneck. 90% of a pooled request.
Per-call setup (fresh DB conn + GeminiClient)464 msPure waste; a pooled server pays zero. The measurement harness paid it.

Per-lane, and why this is good news

Lanep50max
state_lane425 ms804 ms
graph_lane352 ms539 ms
fts_lane347 ms1,043 ms
person_lane97 ms286 ms
vector_lane90 ms191 ms
temporal_lane65 ms418 ms
life_event_lane40 ms159 ms
episode_lane34 ms37 ms
calendar_lane33 ms65 ms
kinship_lane0 ms0 ms
sum of medians1,483 ms≈ the 1,586 ms measured — the lanes run sequentially

The slowest single lane is 425 ms against a sequential sum of 1,483 ms. So the lanes were made to run concurrently, each on its own pooled read connection (brain/spine/pool.py), with the embed submitted into the same fan-out so its 170 ms overlaps the eight lanes that do not need meaning.

Fan-out: built and measured, 2026-07-26

p50p95max
Sequential (pooled conn+transport)1,756 ms2,129 ms2,255 ms
Concurrent, warm1,020 ms1,078 ms1,176 ms
End-to-end over the 43 cases (harness pays fresh setup)2,437 → 1,465 ms3,495 → 1,872 ms

Measured speedup 1.72× — not the ~3.5× predicted above. The prediction assumed the lanes were independent; they are independent in code but they contend for one Postgres instance. Wall clock did not fall to the slowest lane (425 ms) because under concurrency every lane gets slower. The database is the shared resource, and that is the honest ceiling on this technique.

Correctness: proven identical. Fused candidate order compared sequential vs concurrent across all 43 cases — 43/43 byte-identical. The non-generative board is unchanged (30/43 at window 8, 32/43 at window 12, same single trap-bait case). It is a scheduling change and nothing else.

Consequence for the roadmap: the remaining path to 200 ms is not more concurrency. It is making the three slow lanes cheaper — state (425 ms), graph (352 ms), fts (347 ms) — a targeted indexing problem, plus connection reuse (pg_connect measures 320 ms, so a server must never open one per request).

A correction to the design brief

The vector lane costs 90 ms. ANN over 10,643 halfvec(3072) vectors is not a latency problem, so the embedding-dimension decision is purely a storage and index-memory question at scale — it must not be justified on serving latency. The design brief implied an ANN-latency motive; that is wrong and is corrected there.

3. What this means for the product

  1. fast is the product. It carries more (74% vs 63%) for one model call against up to six, and cannot fabricate. thorough is the upsell, and it must be shown to add something before it is worth its price — on this corpus it currently subtracts. > CORRECTED 2026-07-29 — the cost RATIO is withdrawn; the call counts replace it. This read > "costs ~1000× less (one embedding call versus six generative calls at ~$0.0075)". Two things > were wrong with it. The numerator was not one call — when this was written fast was > spending two, an embedding plus an undetected generative query rewrite, so the comparison was > 2-vs-6 dressed as 1-vs-6. And the denominator of the ratio was a zero that is not zero: the > embedding is a real Vertex round trip (p50 170 ms, §2), never priced here, so "1000× less" was > a generation-token ratio presented as a request-cost ratio. The $0.0075 is a real figure but > it is the ask() path's per-answer cost from docs/status/STRESS_REPORT_2026_07.md, not a > per-Context one. 1 model call vs up to 6 is countable, verifiable and enough — a priced > ratio needs the embedding's price, and nobody has looked it up.
  2. The 200 ms SLO is credible but not free, and the work is lane concurrency plus three SQL indexes — not model choice, not a rewrite, not a second store.
  3. Publish p50/p95 next to confidently_wrong_rate. Both are now measured.

4. Two live defects found while connecting (not part of the experiment)

Both concern BRAIN_DATABASE_URL and both are real today:

  1. db.ping() and db.table_exists() return False against a perfectly healthy database. The com.brain.sync launchd agent supplies a bare postgresql://… URL. SQLAlchemy resolves that to psycopg2, which is not installed (ModuleNotFoundError: No module named 'psycopg2'), so every SQLAlchemy-path helper fails closed. /health reports the DB down while the pipeline runs fine, and every liveness detector built on table_exists() would conclude no table exists.
  2. No single value of the variable satisfies both paths. brain/spine/db.py:70 strips +psycopg for the raw psycopg3 path, and founder_confirmed_v2._data_now() calls psycopg.connect(os.environ["BRAIN_DATABASE_URL"]) directly — which requires the bare form — while db.ping() requires the +psycopg form. The two are mutually exclusive.

Fix: normalize in one place (config.load_settings), so the SQLAlchemy URL always carries +psycopg and the raw path always strips it, and route _data_now() through db.pg_connect(). One-line-class changes, but they need the gate, so they are filed here rather than made.

5. Not run

Experiment 3 — contextual retrieval. Prepending a short statement of where a chunk sits before embedding it, published at ~35% fewer top-20 retrieval failures (~49% with a lexical lane, which we have). Requires re-embedding a bounded sample and re-running this same harness — the harness now exists, so this is a scoped, cheap follow-up. Given §1's result (evidence carries 74%, the misses are retrieval misses, not synthesis misses), this is now the highest-value remaining experiment: every one of the 11 misses at window 12 is a case where the answer was not retrieved at all.