Search and retrieval

What this is

sophia.search is the one place search-routing logic lives in this daemon: a single call that searches three corpora — structured rows, document bodies, and wiki titles — and returns one flat, source-tagged result list. An optional intent parameter (facts / documents / code / provenance, default auto) shapes the call: some intents re-route to a specialized tool’s own implementation, others stay cross-corpus and add a routing pointer.

Why it exists

An agent that doesn’t know which corpus holds the answer shouldn’t have to fan out across query_knowledge, search_documents, and list_wiki_pages itself to find out. sophia.search covers all three in one round trip, with structured-field matches (an entity name, a fact’s content) ranked ahead of the softer body-text and title-overlap signals, since an exact field hit is a stronger signal than a BM25 or dense score.

The intent parameter exists for the opposite case: once an agent already knows it wants document bodies, or code-module summaries, or ranked knowledge facts, re-typing the same query into a second, specialized tool call is wasted latency. Passing intent gets the specialized tool’s own result shape back through the same entry point, so sophia.search stays the one thing an agent has to remember, whether it’s the first guess or the deliberate final call.

Returning the three decision-relevant hits instead of the thirty-file sweep keeps the context window dense with signal — which protects reasoning quality deep into a session.

How it works

The auto (or omitted) path runs five structured-row sub-queries — entities, transactions, knowledge facts, deadlines, contacts — as LIKE-substring matches, all scoped to the connection, all ranked first in the merged list. Alongside those, a document leg runs the same FTS5 BM25 + dense hybrid sophia.search_documents uses: BM25 first (cheap, no embedding call), and only when lexical hits come back sparse (fewer than a quarter of the requested limit) does it re-run with the dense arm enabled, fusing the two ranked lists with Reciprocal Rank Fusion (RRF_K = 60) — a deliberate cost trade-off, since embedding the query and scanning chunk vectors is the expensive part of a search and most queries are well served by BM25 alone. A third leg, wiki near-title, does cheap token-overlap scoring against page titles and slugs (≥2 overlapping tokens required) to catch “the page roughly titled X” queries that body-FTS can miss; a wiki hit whose artifact already surfaced as a document hit is dropped rather than shown twice.

Dense candidates — for documents, knowledge, and code summaries alike — are ranked through a shared engine seam: an exact cosine scan by default, switching to a quantized int8 scan only when every embedding row in that query’s candidate set has int8 coverage, checked fresh per query rather than cached. The team’s own wave-gate benchmark requires the quantized path to hold a 95th-percentile latency under two seconds at five times today’s corpus size (roughly 60,000 vectors) — a measured target, not an assumption.

intent: "facts" and intent: "documents" re-route byte-for-byte to sophia.query_knowledge’s and sophia.search_documents’s own implementations — same function, same result shape, no cross-corpus merge. intent: "code" re-routes to sophia.search_code_summaries. intent: "provenance" is the one exception: there’s no single specialized tool for “trace this,” so it returns the same cross-corpus results as auto plus a routing_pointer naming five provenance-specific tools (explore, evidence_for, find_quote, trace_belief, find_contradictions) to reach for once a specific fact or entity is in hand.

flowchart LR
  A["sophia.search(query, intent)"] --> R{intent}
  R -->|"auto (default)"| X["structured rows (rank first)<br/>+ document BM25/dense hybrid<br/>+ wiki near-title"]
  R -->|facts| F["re-route: query_knowledge impl"]
  R -->|documents| D["re-route: search_documents impl"]
  R -->|code| C["re-route: search_code_summaries impl"]
  R -->|provenance| P["same as auto + routing_pointer"]
  X --> M["merged, source-tagged list"]

What your agent does with it

// Real responses from this daemon, captured 2026-07-09:
const facts = await sophia.search({ query: 'daemon', intent: 'facts', limit: 2 });
// → { intent: 'facts', routed_to: 'sophia.query_knowledge', _search_method: 'filter_rank',
//   count: 2, knowledge_facts: [ { knowledge_type: 'summary',
//     content: 'S65+ forward plan for Ouroboros post-reframe...', confidence: 'high', ... } ] }

const docs = await sophia.search({ query: 'mining economics', intent: 'documents', limit: 3 });
// → { intent: 'documents', routed_to: 'sophia.search_documents', count: 3,
//   hits: [ { filename: 'Portfolio.tsx', chunk_idx: 8, fusion_score: 0.0147,
//     signals: { dense: 8, bm25: null }, snippet: '...facts will split into three buckets...' } ] }

const prov = await sophia.search({ query: 'vector ranker', intent: 'provenance', limit: 3 });
// → { intent: 'provenance', items: [ { source: 'document',
//     data: { filename: 'vectorRanker.ts', fusion_score: 0.0156, signals: { bm25: 4, dense: null } } } ],
//   routing_pointer: 'Once you have a specific fact/entity/document in hand, use: sophia.explore ... ' }

The facts call above never touched the dense arm — sophia.search forwards the query as a plain search filter, not question_text, so intent: "facts" always takes query_knowledge’s substring-and-rank path (_search_method: "filter_rank"); reaching the hybrid-ranked version of facts search means calling sophia.query_knowledge directly with question_text set. The documents call shows the dense arm actually firing (signals.dense: 8, bm25: null — the lexical pass came back sparse, so the fallback re-ran with dense enabled). The provenance call is the routing case: three document hits, plus a pointer toward the five tools that trace an individual fact once you have one.

Boundaries

The dense arm — for documents, knowledge, and code summaries — depends on a local ONNX embedding model (BAAI/bge-m3, 1024-dim) loading successfully. When it can’t, every corpus falls back to BM25/FTS5-only, silently and per-call: results still come back, just without the semantic-similarity signal. That fallback rate is now tracked, not just probed: a rolling 15-minute window records every dense-eligible search outcome, and search_degraded on the owner-facing work feed goes true either when the embedding model fails a live load probe or when the observed fallback rate crosses 50% over at least 20 samples in that window — a real, sustained pattern, not a single blip. Calling sophia.subsystem_health live just now against this daemon returned search: { sample_count: 3, fallback_count: 0, sustained: false, by_surface: { documents: { samples: 2, fallbacks: 0 }, code_summaries: { samples: 1, fallbacks: 0 } } } — a healthy window, generated from the very calls made to write this page.

sophia.search decides where to look; it does not decide what counts as ground truth once a fact is found — that model, and how a claim’s evidence gets checked, belongs to Truth and Knowledge Graph. Composing several sophia.search calls (or search plus a follow-up read) into one round trip is what MCP Surface’s execute_code isolate is for. The extraction pipeline that populates the corpora this page searches — chunking, embedding, FTS indexing — is Mining. The daemon these corpora live in, and the encrypted-database/vault split underneath them, is The State Layer.