Mining

What this is

Mining is the extraction step that turns a document or code module into rows in the knowledge graph — a queue holding what needs doing, a run tracking who’s doing it and how far they’ve gotten, and a time-boxed lease handing one unit of work to one agent at a time. sophia.mine is the front door: one call resolves which pipeline a document needs and returns a work order naming the exact next tool calls, instead of an agent hand-driving queue creation, run claiming, and session approval itself.

Why it exists

Before this pipeline existed as a single front door, mining a document meant queue_create → add_items → run_claim → begin_import_session → check_approval → re-call → get_document_for_mining → submit_claim_graph — a roughly ten-call ceremony an agent had to get right every time, per the façade’s own source comment. sophia.mine collapses that to sophia.mine plus the one submit call the returned work order names.

The second problem was worse than ceremony: coverage didn’t refill itself. Before this quarter’s self-sustaining-queue work, newly ingested code modules and documents sat at a NULL mining intent until someone manually enqueued them — a queue that only drains and never restocks eventually empties, and stays empty. The fix is three cheap hooks: new code modules and mining-intent documents auto-enqueue at ingest with a derived priority (import fan-in, churn, recency, curated-entity status), and a bounded nightly pass tops up anything that slipped through and re-ranks what’s already queued. None of this calls an LLM — the daemon only ever flips an intent column and sets a priority; the extraction itself stays agent labor.

How it works

A mining queue (subscriber_mining_queue + its items) is a durable list of sources to mine — one per origin (a large document’s shard plan, a repo’s code-module backlog, an ad-hoc single-document handoff). A mining run (sophia.mining_run_claim → repeated mining_run_report progress events → mining_run_complete or mining_run_fail) is the accountability unit: it moves through claimed → running → (optionally blocked) → completed/failed, and every step lands in an append-only event stream (sophia.mining_run_events) rather than a mutable status field. A source- bearing submit whose run doesn’t resolve, or whose queue doesn’t hold that source, is rejected outright with mining_run_required and writes zero rows — the run isn’t a courtesy field, it’s a hard close.

Two lease shapes hand work to a specific agent. sophia.lease_next_shard claims one window of a large document’s shard plan (default 5-minute TTL). sophia.lease_next_code_module claims one queued code module for deep analysis — metadata only (path, language, symbol count), not source text, with a lease TTL of 10 minutes per the tool’s own description, matching what an earlier page on this site already verified for the same lease. Once a claim is written, it still has to clear the same substring-verification gate this site’s Truth page owns in depth: a claim’s evidence has to be a literal quote from the source or the write never happens — submit_code_module_summary runs that same for-loop check against any evidence_quotes it’s given, and a single miss rejects the whole submit.

flowchart LR
  ING["ingest (doc or code module)"] -->|auto-enqueue, derived priority| Q[("mining queue")]
  Q --> R["mining run: claimed → running"]
  R --> L["lease_next_shard /
lease_next_code_module
(time-boxed)"]
  L --> SUB{"submit: run resolves?
evidence substring-checked?"}
  SUB -->|no| REJ["rejected, zero rows written"]
  SUB -->|yes| KG[("knowledge graph")]
  KG -.->|review-gated| CAND[("mining_candidate,
review_state=pending")]
  CAND -->|promote_candidate| ACC["accepted_knowledge"]
  CAND -->|reject_candidate| REJ2["review_state=rejected
(not deleted)"]

What your agent does with it

// Tool names, params, and response shapes verified against source
// (miningFacadeTools.ts, codebaseTools.ts) — not called live, since
// lease_* and submit_* take real leases/writes and are out of scope
// for this page's read-only research pass.

// 1. One resolve call instead of the ~10-call setup ceremony:
const order = await sophia.mine({ artifact_id: 'doc-8f2c…', depth: 'full' });
// → { ok: true, mode: 'single', run_id: 'run-…', queue_id: 'queue-sys-adhoc-user_handoff',
//   submit_with: 'submit_claim_graph', import_session_required: true,
//   next_steps: [ 'sophia.get_document_for_mining({ artifact_id })',
//     "sophia.begin_import_session({ description, entity_scope, item_types: ['claim_graph'] })",
//     "sophia.submit_claim_graph({ artifact_id, run_id, import_session_id, claim_graph })" ] }

// 2. Code-module lease/submit loop (10-minute lease TTL):
const lease = await sophia.lease_next_code_module({ entity_id: repoId });
// → { module_id, rel_path, language, lease_token, expires_at, run_id }
// run_id is auto-opened server-side — the lease response carries it so the
// submit's requireActiveRun check has something to validate against.
await sophia.submit_code_module_summary({
module_id: lease.module_id,
lease_token: lease.lease_token,
run_id: lease.run_id,
summary_what_it_does: '…', summary_api_surface: '…', summary_dependencies: '…',
confidence: 'medium',
evidence_quotes: ['…'], // substring-verified; any miss rejects the whole submit
});

sophia.lease_next_code_module deliberately withholds source text — a leased module’s structure comes from sophia.get_module_skeleton, and full source from sophia.get_module_source — the same drill-down tools any agent can use outside mining. Confidence is self-graded (high / medium / low), and an honest gaps list is preferred over a fabricated section — the submit tool says so directly: “submitting but flag for re-mining” beats silently guessing.

Boundaries

Mining costs real LLM tokens — this is the “pay once” half of the pay-once-query-forever trade The State Layer describes; a mining run is where that cost is actually paid — once, at mining time, not on every later read. (For long documents extraction is windowed; The state layer covers the exact mechanics.) The daemon itself never makes that call: auto-enqueue and the nightly catch-up pass only flip an intent column and set a priority, by design — the extraction call is agent labor, dispatched through the lease tools above.

Coverage is incremental, not a one-time sweep. Checked live against this dogfood instance’s repo entity while writing this page (sophia.list_unmined_modules, one call per status): 515 code modules done, 877 queued, 15 errored, 0 sitting at NULL — roughly 37% done, with the self-sustaining queue keeping the NULL bucket empty rather than letting it silently pile up between mining sessions. A changed file’s whole mining intent re-queues on content-hash drift today, not just the touched lines — finer delta-scoped re-mining is a known, not-yet-built direction.

Not every write lands as an accepted fact. submit_claim_graph writes directly to sophia_knowledge, but a separate review lane (subscriber_mining_candidate, surfaced via sophia.mining_candidates_list) starts every row at review_state: 'pending' and only moves to accepted/rejected through sophia.promote_candidate / sophia.reject_candidate — both gated behind an active run whose queue’s intent is specifically "promotion". Rejection there is a review verdict, not archival: the underlying fact, if one exists, is never mutated by a reject. Knowledge Graph covers what a row looks like once it’s through either path.