src/core/record/l1-extractor.ts implements the L1 extraction pipeline: it reads L0 conversation messages, calls an LLM to perform scene segmentation and memory extraction in one call, runs batch conflict detection, and writes results to L1 JSONL files.[1] The L1 extractor applies a quality gate (shouldExtractL1) to filter messages by length, symbols, and prompt-injection patterns before sending them to the LLM. L0 captures everything; strict filtering happens at the L1 stage.[1]
extractL1Memories splits qualified messages into a newMessages slice (the last maxMessagesPerExtraction messages, default 10) and a backgroundMessages slice (up to maxBackgroundMessages older messages, default 5) for LLM context.[1] The maximum memories stored per extraction call is bounded by options.maxMemoriesPerSession (default 10); excess memories, sorted by extraction order, are truncated before dedup.[1] extractL1Memories accepts an optional llmRunner (LLMRunner) that decouples it from the OpenClaw runtime; when provided, this runner is used instead of creating a CleanContextRunner directly.[1]
src/core/prompts/l1-extraction.ts implements the L1 extraction stage as a single LLM call combining scene segmentation (情境切分) and memory extraction (记忆提取), based on Kenty's validated prototype prompt.[2] The L1 system prompt restricts extraction to exactly three memory types: persona (stable user attributes/preferences), episodic (objective events/decisions/plans), and instruction (long-term behavioral rules the user sets for the AI).[2] Priority score ranges are defined per type: persona — 80–100 (health/core traits), 50–70 (general preferences); episodic — 80–100 (important events), 60–70 (general activities), below 60 discard; instruction — -1 for absolute global commands, 90–100 (core rules), 70–80 (important), below 70 discard.[2] For episodic memories, the system prompt mandates recording absolute time using the message timestamp when possible, outputting activity_start_time and activity_end_time in ISO 8601 format inside the metadata object.[2] formatExtractionPrompt builds the user-side L1 prompt from newMessages (messages to extract from), backgroundMessages (context only, not extracted), and previousSceneName (for scene continuity), with backgroundMessages defaulting to [] and previousSceneName defaulting to "无".[2]
Example: formatExtractionPrompt in src/core/prompts/l1-extraction.ts formats each message as [id] [role] [timestamp]: content and separates background from new messages with a visual rule, instructing the LLM to extract only from the new-messages section.
const bgText = backgroundMessages.length > 0
? backgroundMessages
.map((m) => `[${m.id}] [${m.role}] [${formatForLLM(m.timestamp)}]: ${m.content}`)
.join("\n\n")
: "无";
const newText = newMessages
.map((m) => `[${m.id}] [${m.role}] [${formatForLLM(m.timestamp)}]: ${m.content}`)
.join("\n\n");
Scene segmentation (情境切分) divides a conversation into distinct topical or contextual scenes. The L1 extractor uses scene boundaries to group related messages and maintains narrative continuity across calls by passing the detected previousSceneName forward to each successive extractL1Memories invocation.
Memories with an unrecognized type field from the LLM output are skipped with a warning log; only memories with a valid, normalized MemoryType are added to the extraction result.[1] Each extracted memory is assigned a temporary record_id (via generateMemoryId) before being passed to batch dedup; this ID is required for the dedup system to correlate decisions back to individual memories.[1]
Conflict/dedup detection is enabled by default in extractL1Memories (enableDedup defaults to true); it can be disabled via options.enableDedup = false, in which case all extracted memories are stored directly without conflict checking.[1] src/core/record/l1-dedup.ts implements batch conflict detection using a two-phase approach: (1) fast candidate recall per new memory using vector search or FTS5 BM25 (no LLM), then (2) a single batch LLM call to judge all new memories against their candidate pools.[3] The JSONL-based Jaccard word-overlap fallback was removed in l1-dedup.ts v4. Candidate recall now degrades through three tiers: (1) vector cosine similarity, (2) FTS5 BM25 keyword recall, (3) skip dedup entirely — never a full-file scan.[3] The default conflictRecallTopK in batchDedup is 5 — the number of candidate existing memories recalled per new memory for the LLM conflict judgment.[3] Tier 1 (vector recall) in batchDedup requests topK + memories.length results from the vector store to account for self-batch filtering, then slices to topK after excluding records from the current extraction batch.[3] FTS5 candidate recall in l1-dedup.ts fetches up to 10 FTS results per memory, then slices to 5 candidates after filtering out records in the current batch.[3] If vector recall fails during Tier 1, batchDedup automatically degrades to FTS5 keyword recall. If FTS is also unavailable, conflict detection is skipped and all memories are stored directly.[3] In batchDedup, if neither vector data nor FTS is available, conflict detection is skipped entirely and all memories receive a "store" decision — dedup is not attempted to avoid an O(N) full-file-scan cost.[3] If no memory has any candidate matches after recall (all candidate lists are empty), batchDedup skips the LLM judgment call entirely and stores all memories directly.[3]
When llmRunner is provided to batchDedup, the LLM judgment call uses it directly via llmRunner.run(); otherwise a CleanContextRunner is instantiated as the OpenClaw-specific fallback path.[3] The LLM conflict detection call in l1-dedup.ts applies a timeout of 180_000 ms (3 minutes) on both the LLMRunner path and the CleanContextRunner fallback path.[3] batchDedup accepts an optional embeddingTimeoutMs to override the embedding call timeout specifically on the capture path, passed through to embeddingService.embedBatch().[3] If the batch LLM judgment call fails in runLlmJudgment, all memories default to the "store" action rather than failing the entire extraction.[3]
L1ExtractionResult (exported from src/core/record/l1-extractor.ts) carries success, extractedCount, storedCount, records (the stored MemoryRecord[]), sceneNames, and lastSceneName (the last detected scene, for continuity in the next extraction call).[1] If the LLM extraction call itself fails in extractL1Memories, the function returns { success: false, extractedCount: 0, storedCount: 0, records: [], sceneNames: [] } — it does not fall back to partial storage.[1] When instanceId and logger are both provided, extractL1Memories emits an l1_extraction metric via report(), including inputMessageCount, memoriesExtracted, memoriesStored, memoriesByType distribution, totalDurationMs, and the content/type/scene of each stored memory.[1] CaptureResult in src/core/types.ts reports the count of L0 messages recorded (l0RecordedCount), whether the pipeline scheduler was notified (schedulerNotified), the number of L0 vectors written (l0VectorsWritten), and the filtered messages that were actually captured (filteredMessages) — see L0 capture for full capture-path context.[4]
Sources