VectorStore in src/core/store/sqlite.ts manages four SQLite tables in a single database: l1_records (L1 relational metadata), l1_vec (vec0 virtual table for L1 cosine search), l0_conversations (L0 relational metadata), and l0_vec (vec0 virtual table for L0 cosine search).[1] VectorStore requires Node.js 22+ because it uses the built-in node:sqlite (DatabaseSync API) together with the sqlite-vec extension from the root workspace.[1] All VectorStore operations are synchronous, using the DatabaseSync API; writes use manual BEGIN/COMMIT transactions to atomically update both the metadata table and the vec0 virtual table together.[1] Upserts in VectorStore are implemented as delete-then-insert because the vec0 virtual table does not support ON CONFLICT clauses.[1] vec0 is a SQLite virtual table extension provided by sqlite-vec that stores and indexes embedding vectors, enabling approximate nearest-neighbor cosine search.
The VectorSearchResult type expresses cosine similarity as score = 1.0 − cosine_distance, so higher scores indicate greater similarity.[1] bm25RankToScore() converts a BM25 rank (negative = more relevant) to a 0–1 score using relevance / (1 + relevance) where relevance = -rank for negative ranks, or 1 / (1 + rank) for non-negative ranks.[1]
buildFtsQuery() builds an FTS5 MATCH query string by segmenting text with jieba's cutForSearch mode (when @node-rs/jieba is available) or falling back to Unicode-regex splitting (/[\p{L}\p{N}_]+/gu). Tokens are OR-joined as quoted FTS5 phrase terms for maximum recall, with BM25 ranking preserving precision.[1] buildFtsQuery() filters a small set of Chinese stop-words (e.g., 的、了、在、是) from FTS5 query tokens to reduce noise; the list is intentionally limited to high-frequency function words only.[1] On the write side, tokenizeForFts() uses jieba cutForSearch to index both full words and sub-word components — for example, "人工智能" is indexed as "人工 智能 人工智能" — ensuring query-side tokens always find a match. If jieba is unavailable, the original unmodified text is stored.[1] Jieba is lazy-loaded as a singleton on the first call to buildFtsQuery; if @node-rs/jieba is unavailable, _jieba is set to null and the Unicode-regex fallback path is used permanently without retrying.[1] Example buildFtsQuery output: with jieba, "用户喜欢编程和TypeScript" produces '"用户" OR "喜欢" OR "编程" OR "TypeScript"'; without jieba, "旅行计划 API" produces '"旅行计划" OR "API"'.[1]
The L1QueryFilter interface supports narrowing L1 record queries by sessionKey (conversation channel), sessionId (single conversation instance), and updatedAfter (ISO 8601 UTC timestamp for incremental sync).[1]
src/core/store/search-utils.ts provides rrfMerge(), a shared Reciprocal Rank Fusion helper used across the SQLite hybrid-search code paths (auto-recall, memory-search, and conversation-search), eliminating duplication between them.[2] rrfMerge() uses the standard RRF constant k = 60 from the original RRF paper by default; each item's score is 1 / (k + rank + 1), summed across all lists, and results are returned sorted by descending rrfScore.[2] rrfMerge() is generic: it accepts an getId callback to extract a string key from each item, allowing it to operate on any result type, with items appearing in multiple ranked lists accumulating their scores.[2]
Canonical usage of rrfMerge() from src/core/store/search-utils.ts to merge FTS and vector search results:
const merged = rrfMerge(
[ftsResults, vecResults],
(item) => item.record_id,
);
_resetJiebaForTest() and _setJiebaForTest() are exported from src/core/store/sqlite.ts solely for testing: the former resets the jieba singleton so the next buildFtsQuery call re-initialises it; the latter injects a mock instance or forces the Unicode-regex fallback path.[1]
Sources