Book

Agent Memory

Compiled from 29 nodes · est. 64 min read

1Introduction

Updated 2026-08-11

TencentDB Agent Memory is a persistent, multi-layer memory service for AI agents — its guiding idea is to let an agent's experience sediment into reusable assets so the next session can "load from save." The system is organized as a four-layer pipeline — L0 (raw conversation) → L1 (atomic facts) → L2 (scene summaries) → L3 (user persona) — replacing flat vector storage with a semantic pyramid where the Persona layer carries day-to-day preferences and the system drills down to Atoms only when details matter. It began as an embedded plugin for the OpenClaw coding-agent host and was later extracted into a standalone Gateway service; the codebase now supports OpenClaw, Hermes, and generic HTTP hosts through a host-neutral TdaiCore layer with HostAdapter and LLMRunner abstractions. The package is distributed as an ES module under the MIT license, with the core engine in TypeScript and a Python memory_tencentdb adapter for Hermes.

The Setup & distribution section — with its child pages Package & distribution, Installation & deployment, and Upgrading — covers how to install, configure, and migrate between the 0.x plugin line and the 1.x standalone service line. The Memory architecture section explains the memory model itself: Architecture & memory layers walks through the L0→L1→L2→L3 pyramid, Context Offload covers the independent short-term compression subsystem, and Seed & CLI documents the offline pipeline runner. The TdaiCore & host abstraction section documents the host-neutral core and its per-layer implementations — L0 capture, L1 extraction & dedup, L2 scene extraction, L3 persona generation, Pipeline scheduling, and Recall & context injection. The Storage abstraction section defines the IMemoryStore contract and its concrete backends in SQLite backend, TCVDB backend, and Embedding services. The Integrations & hosts section covers host-specific wiring in OpenClaw integration and Hermes & Gateway hosts, while Features & tools groups user-facing capabilities — Configuration, Search tools, and LLM thinking suppression. The Benchmarks & results and Bug fixes sections capture measured performance on long-horizon agent tasks and the architecturally-significant fixes that shaped the current codebase.

If you want to understand how the memory model works before touching anything, read Architecture & memory layers first, then Recall & context injection to see how memories are actually surfaced to the LLM. If you are installing the plugin into an OpenClaw or Hermes host, start with Installation & deployment, then read OpenClaw integration or Hermes & Gateway hosts depending on your host, and consult Configuration for the tunable fields. If you are adding a feature or a new storage backend, begin with TdaiCore & host abstraction to understand the boundary interfaces, then Storage abstraction for the IMemoryStore contract and factory. If you are debugging a failed extraction, missing recall, or an upgrade regression, jump to Pipeline scheduling, the relevant layer page (L1 extraction & dedup, L2 scene extraction, or L3 persona generation), and cross-reference Bug fixes and Upgrading for known breakages.

2.2Installation & deployment

Updated 2026-08-22

Agent Memory installs into OpenClaw via the native openclaw plugins install command and requires enabling in config and applying a one-time patch to hook after-tool-call messages for correct behavior. v2.0.0 Docker deployment splits Agent Memory across three images with multi-arch support; start-all.sh auto-initializes the admin account and prints the claude startup command once ready. The after-tool-call hook is an OpenClaw lifecycle event fired immediately after any tool executes; Agent Memory uses it to offload tool results to storage and recover context between turns.

Installing and enabling the plugin in OpenClaw takes two commands: openclaw plugins install @tencentdb-agent-memory/memory-tencentdb followed by openclaw gateway restart.[1] Use the native openclaw plugins install command (not npm directly) to avoid semantic-version-range issues that can disable the plugin; the same rule applies when upgrading via openclaw plugins update @tencentdb-agent-memory/memory-tencentdb.[1] Minimal zero-config OpenClaw setup requires adding { "memory-tencentdb": { "enabled": true } } to ~/.openclaw/openclaw.json.[1]

The openclaw-after-tool-call-messages.patch.sh script must be applied once per OpenClaw installation to hook after-tool-call messages for correct offload and recovery; the patch must be re-applied after each OpenClaw upgrade — see OpenClaw integration for behavioral details.[1]

A postinstall hook runs node scripts/postinstall.mjs automatically after npm install.[2]

v2.0.0 Docker deployment uses three images — agentmemory/memory-core, agentmemory/memory-hub, and agentmemory/memory-proxy — with multi-arch support (linux/amd64 + linux/arm64).[3] Running start-all.sh auto-initializes the admin account and generates .admin-key on first start, then prints a ready-to-use claude startup command once self-checks pass.[3] stop-all.sh --purge completely removes volumes and the admin key, making environment resets straightforward.[3]

Sources

2.3Upgrading

Updated 2026-08-22

Agent Memory maintains two parallel version lines: 0.x as an embedded OpenClaw plugin for single-machine use, and 1.x as a standalone Gateway service for multi-agent and service deployments. Each version line contains breaking changes (slot renames, tokenizer defaults, timing adjustments) and bug fixes; upgrading requires reviewing which changes affect your deployment model and configuration. In Agent Memory, L1 and L2 are distinct processing stages: L1 is a fast first-pass memory capture, and L2 is a slower, deeper extraction step that runs after L1 completes.

v1.0.0 and 0.x are maintained in parallel: the 0.x line (main branch) is the embedded OpenClaw plugin for lightweight single-machine use, while the 1.x line (feat/server branch) is the standalone Memory service for multi-agent, multi-framework, and service deployments.[1] In v1.0.0, the memory engine was split from an embedded OpenClaw plugin into an independent Gateway service process, changing both deployment and integration methods — see Hermes & Gateway hosts for the resulting architecture.[1]

v0.2.1 was deprecated because a missing undici dependency caused the plugin to fail to start; users should migrate to v0.2.2 or later, where the issue is fixed.[2]

In v0.3.6, the l3TiktokenEncoding default changed from o200k_base to cl100k_base to better match mainstream domestic and open-source model tokenizers (DeepSeek, GLM, MiniMax); only users explicitly depending on o200k_base need to override it.[3] Also in v0.3.6, the contextEngine slot ID was renamed from openclaw-context-offload to memory-tencentdb to prevent openclaw doctor --fix from resetting the slot.[3] In v0.3.5, the l2DelayAfterL1Seconds default was reduced from 90 s to 10 s so that cold-start users no longer wait roughly 90 seconds before seeing L2 scene extraction results.[4]

v0.3.6 fixed an infinite-recursion bug in the install script: when run as root, su - root re-entered the script, which saw EUID=0 again and looped indefinitely.[3]

Sources

3.1OpenClaw integration

Updated 2026-08-22

The plugin registers tools (tdai_memory_search, tdai_conversation_search) and hooks with OpenClaw via index.ts, which parses the full pluginConfig on each host startup and translates OpenClaw events into TdaiCore calls; core memory logic is confined to src/core/tdai-core.ts while OpenClaw-specific bindings live in OpenClawHostAdapter. The plugin caches user prompts and recall state by session during before_prompt_build, sweeping stale entries after 10 minutes or at 10k capacity, enabling post-agent metrics and preventing unbounded memory growth under high concurrency.

The plugin's entry point is ./index.ts, declared in the openclaw.extensions field of package.json — the file OpenClaw loads to register the plugin.[1] Activation is automatic on host startup via "onStartup": true in openclaw.plugin.json.[2] The plugin contracts two tools with the OpenClaw host: tdai_memory_search and tdai_conversation_search.[2] The openclaw.bundle.stageRuntimeDependencies flag is set to true, instructing the OpenClaw bundler to stage runtime dependencies alongside the plugin.[1]

To route context-offload requests to the plugin, the plugins.slots.contextEngine field must be set to "memory-tencentdb" in the OpenClaw configuration.[3] An OpenClaw plugin slot (plugins.slots.<name>) is a configuration mechanism that routes a named class of internal requests — such as context-offload — to a specified plugin instead of the default handler.

index.ts is a thin shell that registers tools and hooks with OpenClaw and translates OpenClaw events into TdaiCore calls; all core memory logic lives in src/core/tdai-core.ts.[4] OpenClaw calls register() multiple times (plugin scan → gateway start → per-channel bootstrap → config reload); each call receives the full pluginConfig from openclaw.json, and index.ts parses it fresh every time.[4] resetReporter() is called on every register() invocation so that reporter singleton state is flushed and config changes take effect on hot-reload.[4] index.ts supports a cli-metadata registration mode in which only CLI commands are registered and all runtime initialization is skipped; OpenClaw uses this mode to discover CLI subcommands without starting the full plugin.[4] CLI commands are exposed under the memory-tdai subcommand group (seed, query, stats), registered via registerMemoryTdaiCli.[4]

The hook-policy auto-patch (ensurePluginHookPolicy) is applied only when the OpenClaw host version is ≥ 2026.4.24; on older hosts, or when api.runtime.version is undefined, the patch is skipped to avoid silently mutating the user's openclaw.json.[4] If the embedding configuration is incomplete, index.ts logs a prominent error via api.logger.error with the prefix [EMBEDDING CONFIG ERROR] so the user can identify the misconfiguration at startup.[4] pluginStartTimestamp is set to Date.now() when the plugin registers in full mode and is passed to performAutoCapture as a fallback cursor to prevent the first agent_end from dumping the entire session history into L0 when no checkpoint exists yet.[4] sharedMemoryCleaner is a process-level singleton (LocalMemoryCleaner | undefined) used to prevent concurrent cleanup races when the plugin is started more than once in the same process.[4]

pendingOriginalPrompts caches the clean user prompt (before prependContext injection), cache creation time, and session message count at before_prompt_build time; the message count serves as a fallback slice offset when the timestamp cursor is unreliable.[4] pendingRecallCache caches L1 memories, L3 persona, recall strategy, and duration from before_prompt_build for retrieval at agent_end, enabling the agent_turn metric event; it is keyed by sessionKey.[4] pendingRecallEndTimestamps stores the recall-completion timestamp per session and is used in agent_end to estimate LLM reasoning time as agent_end_start − recall_end_ts; stale entries are swept alongside the prompt cache.[4] The prompt cache has a TTL of 10 minutes (PROMPT_CACHE_TTL_MS = 10 * 60 * 1000) and a hard size cap of 10,000 entries (PROMPT_CACHE_MAX_SIZE = 10_000) to prevent unbounded growth in high-concurrency scenarios.[4] sweepStaleCaches() sweeps both pendingOriginalPrompts and pendingRecallCache using the same TTL and hard-cap logic, evicting the oldest entries first when either map exceeds PROMPT_CACHE_MAX_SIZE.[4]

OpenClawHostAdapter in src/adapters/openclaw/host-adapter.ts translates the OpenClaw plugin API into the host-agnostic HostAdapter interface consumed by TdaiCore, confining all OpenClaw-specific dependencies to the adapter layer — see TdaiCore & host abstraction for the full host-adapter contract.[5] OpenClawHostAdapter carries readonly hostType = "openclaw" for runtime discrimination between adapter implementations.[5] OpenClawHostAdapter is constructed with { api, pluginDataDir, openclawConfig } and immediately creates an OpenClawLLMRunnerFactory from api.runtime.agent and api.logger.[5] OpenClawHostAdapter.getRuntimeContext() returns a RuntimeContext with userId: "default_user", empty sessionId/sessionKey, platform: "openclaw", workspaceDir: process.cwd(), and dataDir set to the resolved plugin data directory.[5] OpenClawHostAdapter.buildRuntimeContextForSession(sessionKey, sessionId?) merges per-hook session identifiers into the base RuntimeContext; if sessionId is omitted it defaults to an empty string.[5] OpenClawHostAdapter exposes getPluginApi(), getOpenClawConfig(), and getPluginDataDir() as escape hatches for legacy callers still accessing OpenClaw internals directly during migration.[5]

Sources

3.2Hermes & Gateway hosts

Updated 2026-08-22

Beyond OpenClaw, TencentDB-Agent-Memory also supports the Hermes agent framework; the Hermes adapter lives in hermes-plugin/ and is named memory_tencentdb.[1] The Hermes Docker image bundles hermes-agent and the memory_tencentdb provider together, exposes the gateway on port :8420, and requires plugin version ≥ 0.3.4.[1] The plugin manifest at hermes-plugin/memory/memory_tencentdb/plugin.yaml describes it as implementing the four-layer memory pipeline (L0 conversation recording, L1 episodic extraction, L2 scene blocks, L3 persona synthesis) via a local Node.js Gateway — see Architecture & memory layers.[2]

src/gateway/server.ts implements the TDAI Gateway HTTP server using Node.js's native http module — no Express or Fastify dependency — and is designed to run as a managed sidecar alongside Hermes.[3] Gateway version "0.1.0" is defined as the module-level constant VERSION in src/gateway/server.ts.[3] The TdaiGateway constructor wires together a StandaloneHostAdapter, a TdaiCore instance, and a SessionFilter; configuration is loaded via loadGatewayConfig() with optional overrides.[3] TdaiGateway.start() initializes data directories via initDataDirectories(), calls this.core.initialize(), then starts the HTTP server; TdaiGateway.stop() closes the HTTP server and calls this.core.destroy().[3]

src/gateway/server.ts exposes these HTTP routes: GET /health, POST /recall, POST /capture, POST /search/memories, POST /search/conversations, POST /session/end, and POST /seed.[3] A v2 API provides 14 standard routes covering memory CRUD, atomic updates, scene indexing, and pipeline status queries.[4] GET /health bypasses the auth gate unconditionally so that orchestrators (Kubernetes liveness probes, Docker health checks) can reach it cheaply.[3] The Gateway enforces a 1 MiB request body size limit to prevent abnormal requests.[4] CORS headers are applied based on a configured allowlist — when corsOrigins is empty, no CORS headers are added; OPTIONS preflight requests always receive a 204 response.[3] Unhandled route errors are caught at the top-level handleRequest try/catch, logged with method and pathname, and returned as HTTP 500 with the error message in the GatewayErrorResponse shape.[3]

When server.apiKey is not configured, all routes except GET /health are open to any caller that can reach the port — the documented legacy default — and a startup warning is emitted via logSecurityPosture() to make this visible.[3] Setting server.apiKey / TDAI_GATEWAY_API_KEY requires every non-/health route to carry an Authorization: Bearer <apiKey> header; requests with a missing, malformed, or incorrect token receive HTTP 401.[5][3] checkAuth() uses crypto.timingSafeEqual for API key comparison to prevent timing-based prefix-match attacks; length mismatches are rejected without comparing bytes.[3] A startup WARN is emitted when the gateway binds to a non-loopback host without apiKey set, to avoid silently exposing an unauthenticated endpoint to the network.[6] When server.corsOrigins contains "*", a startup warning is emitted: every browser origin can reach the gateway, and a concrete allowlist is required for non-local deployments.[3] At startup the gateway logs a one-shot security posture summary covering auth status, bound host, and CORS configuration — and never logs the API key value itself.[3]

src/gateway/config.ts loads gateway configuration from four sources in order: (1) TDAI_GATEWAY_CONFIG env var (explicit path), (2) tdai-gateway.yaml or tdai-gateway.json in CWD, (3) the same filenames under the default data dir, (4) pure environment-variable config with no file.[6] YAML config files are parsed with full YAML support (arbitrary nesting, anchors, lists, multi-line strings); ${VAR} environment-variable interpolation is applied to string leaves as a post-processing step for backward compatibility.[6] A malformed config file is silently swallowed and falls back to environment-variable-only configuration; the config file is treated as optional.[6] loadGatewayConfig defaults the gateway server port to 8420 (overridable via TDAI_GATEWAY_PORT env or server.port yaml) and the host to 127.0.0.1 (overridable via TDAI_GATEWAY_HOST or server.host yaml).[6] server.corsOrigins defaults to an empty list, meaning no Access-Control-Allow-* headers are sent and CORS preflight with an Origin header is rejected with 403; setting it to ["*"] restores permissive behaviour for local development.[6] TDAI_CORS_ORIGINS (comma-separated) or yaml server.corsOrigins (string array or comma-separated string) configures allowed CORS origins; returning [] means no CORS headers are emitted.[6] The default data directory is ~/.memory-tencentdb/memory-tdai/, overridable via TDAI_DATA_DIR or MEMORY_TENCENTDB_ROOT.[6] A leading ~/ in TDAI_DATA_DIR or data.baseDir is expanded to $HOME (or $USERPROFILE on Windows, falling back to /tmp).[6] Backward compatibility is preserved: if the new ~/.memory-tencentdb/memory-tdai/ data directory does not exist but the legacy ~/memory-tdai/ does, the legacy path is used and a deprecation warning is written to stderr.[6] loadGatewayConfig defaults the LLM base URL to https://api.openai.com/v1, model to gpt-4o, max tokens to 4096, and timeout to 120000 ms.[6] Memory configuration parsing is delegated to the shared parseConfig imported from ../config.js, ensuring full compatibility between gateway and plugin configurations.[6] loadGatewayConfig accepts a Partial<GatewayConfig> overrides argument merged one level deep — partial server, data, or llm objects are spread on top of defaults, so sibling fields such as corsOrigins introduced after callers were written are not accidentally dropped.[6]

src/gateway/types.ts defines the HealthResponse shape returned by GET /health: { status: "ok"|"degraded", version, uptime, stores: { vectorStore, embeddingService } }.[7] RecallRequest requires query and session_key; user_id is optional. RecallResponse returns context (string), optional strategy, and optional memory_count.[7] CaptureRequest requires user_content, assistant_content, and session_key; session_id, user_id, and messages are optional. CaptureResponse returns l0_recorded (count) and scheduler_notified (bool).[7] MemorySearchRequest accepts query (required), optional limit, type, and scene filters; MemorySearchResponse returns results (string), total (number), and strategy (string).[7] SessionEndRequest requires session_key and accepts optional user_id; SessionEndResponse returns flushed: boolean.[7] SeedResponse reports sessions_processed, rounds_processed, messages_processed, l0_recorded, duration_ms, and output_dir.[7] GatewayErrorResponse is the common error envelope with a required error string and an optional code string.[7]

StandaloneHostAdapter in src/adapters/standalone/host-adapter.ts is the HostAdapter implementation for the TDAI Gateway (Hermes sidecar); it has no dependency on OpenClaw and constructs context purely from Gateway config and per-request parameters.[8] StandaloneHostAdapter is constructed with { dataDir, llmConfig, logger, defaultUserId?, platform? } and delegates LLM execution to StandaloneLLMRunnerFactory from ./llm-runner.js.[8] StandaloneHostAdapter.getRuntimeContext() returns a RuntimeContext with both workspaceDir and dataDir set to the configured dataDir, empty sessionId/sessionKey, and the configured userId and platform.[8] StandaloneHostAdapter.buildRuntimeContextForRequest() scopes each Gateway request to the correct user/session: sessionKey falls back to sessionId when not explicitly provided, and all per-request params fall back to adapter defaults when omitted.[8]

The plugin manifest registers the legacy aliases tdai and memory-tencentdb so that existing user configs specifying memory.provider: tdai continue to resolve to this provider without changes.[2] The Hermes plugin uses a watchdog + lazy probe mechanism to automatically recover when the Gateway encounters an error.[9]

Sources

4.1Architecture & memory layers

Updated 2026-08-22

TencentDB Agent Memory structures long-term memory as a semantic pyramid — L0 raw conversations → L1 atomic facts → L2 scene summaries → L3 user persona — where the system defaults to coarse summaries but drills down to detailed facts only when needed. Short-term task context is encoded as lightweight Mermaid diagrams with full traceability back to raw tool logs, ensuring every symbol in the agent's working context maps to retrievable source data via mid-layer indices.

TencentDB Agent Memory implements a four-layer long-term memory pipeline — L0 (raw conversation) → L1 (atomic facts) → L2 (scene summaries) → L3 (user persona) — replacing flat vector storage with a semantic pyramid. The Persona layer carries day-to-day preferences; the system drills down to Atoms only when details matter.[1][2] Each layer's processing is handled locally, with zero external API dependencies: L0 records conversations to local JSONL automatically, L1 extracts structured facts via LLM with deduplication, L2 aggregates scene blocks via LLM scene extraction, and L3 synthesises a user persona via LLM.[2]

Short-term context is also structured in three sub-levels: the bottom layer archives raw tool outputs in refs/*.md files; the middle layer extracts step-level JSONL summaries; the top layer condenses state into a lightweight Mermaid canvas — the only layer injected into the agent's context.[1] Task state in the top layer is encoded in high-density Mermaid graph syntax rather than verbose prose or flat JSON, with full tool logs offloaded to external refs/*.md files and only a lightweight Mermaid task map plus node_id pointers remaining in context.[1]

A drill-down traceability guarantee spans all layers — top-layer symbol (Persona / canvas) → mid-layer index (Scenario / JSONL) → bottom-layer raw text (L0 conversation / refs) — ensuring no irreversible lossy compression.[1] Heterogeneous storage backs the two poles: bottom-layer facts, logs, and traces are persisted in databases for robust full-text retrieval, while top-layer personas, scenes, and canvases are stored as human-readable Markdown files for high information density and white-box inspection.[1] Out of the box the system defaults to a local SQLite + sqlite-vec backend, requiring no external database to get started — see SQLite backend for configuration details.[1]

With a single enabled: true flag, TencentDB Agent Memory automatically handles conversation capture, memory extraction, scene aggregation, persona generation, and recall before each new turn — no additional wiring is required.[1]

Sources

4.2Context Offload

Updated 2026-08-22

Context Offload is a separate, independently-toggled multi-layer context compression subsystem that is disabled by default (offload.enabled: false) and does not affect Memory pipeline functionality when off.[1] Enabling short-term compression requires plugin version ≥ 0.3.4 and is opt-in via config.offload.enabled: true in the plugin configuration.[2] src/offload/index.ts is the merged entry point for the Context Offload module, equivalent to the standalone context-offload-plugin's index.js, adapted to co-exist with the memory-tencentdb plugin.[3] registerOffload(api, offloadConfig) in src/offload/index.ts is the public entry point for conditional offload registration, called from the main plugin index.ts when offload.enabled: true.[3] The Context Offload pipeline is organized into four layers — L1, L1.5, L2, and L3 — each performing progressively heavier compression or graph-building work. A context window is the maximum number of tokens an LLM can process in a single call; when conversation history fills this window, older content must be discarded or compressed before new messages can be sent.

OffloadConfig.mode in src/config.ts has three values: "local" (calls LLM directly via AI SDK), "backend" (routes through a remote backend service), and "collect" (data collection only — runs L1/L1.5/L2 async but disables L3 compression and does not occupy the contextEngine slot).[4] The offload.mode: "collect" configuration executes only data collection (L0 capture + vector writes) without triggering L3 compression, useful for pure data accumulation or debugging.[5] PluginConfig.model in src/offload/types.ts selects the LLM for offload tasks using the format "provider/model-id" (e.g. "dashscope/kimi-k2.5").[6]

The offload.dataDir option overrides the default Context Offload data directory (default ~/.openclaw/context-offload); the value must be an absolute path.[1] PLUGIN_DEFAULTS in src/offload/types.ts sets compression thresholds: mild offload triggers at 50% context fill (mildOffloadRatio: 0.5), aggressive compress at 85% (aggressiveCompressRatio: 0.85), and emergency at 95% (emergencyCompressRatio: 0.95) targeting down to 60% (emergencyTargetRatio: 0.6).[6] PLUGIN_DEFAULTS in src/offload/types.ts caps injected Mermaid MMD content to at most 20% of the total context window token budget (mmdMaxTokenRatio: 0.2).[6] PLUGIN_DEFAULTS in src/offload/types.ts sets defaultSystemOverheadRatio: 0.12, assuming 12% of the context window is consumed by the system prompt and tool schemas when no cached overhead is available from the llm_input hook.[6] PLUGIN_DEFAULTS in src/offload/types.ts sets the default l3TiktokenEncoding to "cl100k_base" (matching DeepSeek/GLM/MiniMax tokenizers), changed from o200k_base in v0.3.6.[6] PLUGIN_DEFAULTS in src/offload/types.ts sets the default l3TokenCountMode to "tiktoken" (exact BPE counting); the alternative is "heuristic" (CJK/1.7 + others/4).[6] PLUGIN_DEFAULTS in src/offload/types.ts sets L2 to trigger when there are 4 or more node_id=null entries (l2NullThreshold: 4), or after 300 seconds without a run (l2TimeoutSeconds: 300); node_id="wait" entries are retried after 120 seconds (l2WaitRetrySeconds: 120).[6] PLUGIN_DEFAULTS in src/offload/types.ts sets l2TimeTriggerRequiresNewOffload: true, meaning time-based L2 only fires when at least one node_id=null entry has a timestamp after lastL2TriggerTime; set to false for legacy behavior that retries stale nulls.[6]

OffloadEntry in src/offload/types.ts is the record stored in offload.jsonl; the node_id field is null until L2 runs and assigns a Mermaid flowchart node ID.[6] OffloadEntry.score in src/offload/types.ts is a replaceability score (0–10) assigned by the L1 LLM: higher scores mean the LLM summary can better replace the original tool result.[6]

registerOffload() in src/offload/index.ts carries no idempotency guard — OpenClaw calls it multiple times during its lifecycle (plugin scan → gateway start → config reload); each call receives a different api instance, and only the last one is the live runtime API.[3] When registerContextEngine returns ok=false or throws, _contextEngineRejected is set to true and all offload functions are disabled for the lifetime of the module.[3] The L2 scheduler state (_l2Running, _l2PollHandle, _l2FirstNotifyAt) and the L1.5 retry-loop dispose flag (_l15Disposed) in src/offload/index.ts are module-level and shared across all registerOffload() invocations.[3] Internal memory-pipeline sessions matching the pattern memory-{taskId}-session-{ts} are detected by INTERNAL_SESSION_RE in src/offload/index.ts and excluded from offload processing.[3]

_isHeartbeatText() in src/offload/index.ts identifies both user heartbeat prompts and assistant HEARTBEAT_OK replies by checking for the substrings "HEARTBEAT" or "heartbeat"; heartbeat messages are skipped in history extraction and current-prompt context building.[3] _buildL1RecentContext() formats the offload state for L1 requests as ## current msg:\n{prompt}\n\n## history msg:\n{history}, falling back to (none) for missing sections.[3] _buildL15RecentContext() formats the offload state for L1.5 requests in Chinese, placing history first as reference and the latest user message last as focus: 历史消息，可作为参考：\n{history}\n\n最新user message：\n{currentLine}.[3] _msgFingerprint() in src/offload/index.ts computes a message identity hash from its role and the first 200 characters of its content using a DJB2-variant hash.[3] The /create-skill command parsed by parseCreateSkillCommand() in src/offload/index.ts accepts an optional MMD name and skill focus as space-separated arguments: /create-skill [mmdName] [skillFocus...].[3]

src/offload/index.ts imports compression utilities from ./hooks/llm-input-l3.js including compressByScoreCascade, aggressiveCompressUntilBelowThreshold, emergencyCompress, and EMERGENCY_MIN_MESSAGES_TO_KEEP, and uses isTokenOverflowError to detect LLM context-overflow errors.[3] Token-counting is sourced from ./context-token-tracker.js (buildTiktokenContextSnapshot, configureTokenTracker, tiktokenCount, jsonReplacer) and a fast estimator from ./fast-token-estimate.js (fastEstimateMessages).[3]

OffloadStateManager in src/offload/state-manager.ts is session-scoped: each instance is bound to a single session through an immutable StorageContext, and the class contains no global mutable state — all I/O goes through the frozen ctx.[7] OffloadStateManager.init() creates a StorageContext, ensures required directories exist, and loads persistent state from state.json before the manager is usable.[7] Accessing OffloadStateManager.ctx before calling init() or switchSession() throws Error: "OffloadStateManager: ctx not initialized, call init() or switchSession() first".[7] The default persistent state in src/offload/state-manager.ts initializes all fields to null or 0: activeMmdFile: null, activeMmdId: null, mmdCounter: 0, lastSessionKey: null, lastOffloadedToolCallId: null, lastL2TriggerTime: null, estimatedSystemOverhead: null.[7] OffloadStateManager exposes a readonly _instanceId (monotonically incrementing static counter) on each instance for debugging.[7] OffloadStateManager.addToolPair() is idempotent: it silently drops a ToolPair whose toolCallId is already in processedToolCallIds.[7] OffloadStateManager.takePending() removes up to max tool pairs from the front of the pending buffer and marks their toolCallIds as processed in one atomic step, preventing double-processing.[7] OffloadStateManager.nextMmdNumber() reconciles the in-memory mmdCounter with the highest numeric prefix found among MMD files on disk, preventing counter regression after restarts.[7] OffloadStateManager exposes a mutex field l1Lock (a Promise<unknown>, initialized to Promise.resolve()) for the L1 pipeline to prevent concurrent runs.[7] The l15Settled flag on OffloadStateManager gates L2 triggering: L2 must wait for l15Settled === true before it can run, ensuring L1.5 completes first.[7] The _lastAggressiveBoundary field on OffloadStateManager caches boundary info from the last aggressive deletion — originalIndex, fingerprint, keptMsgCount, and remainingTokens — to enable O(1) head-delete on replay instead of O(N×rounds) re-scanning.[7]

OffloadStateManager.switchSession() accepts a full session key (e.g. "agent:main:session-123"), rebuilds the StorageContext, and reloads persistent state from disk; the realSessionId parameter optionally overrides the parsed session ID.[7] On switchSession(), persistent state is only reloaded when the agent name changes; switching between sessions of the same agent preserves the current in-memory state object.[7] switchSession() resets all session-scoped runtime state — pendingToolPairs, injectedMmdVersions, mmdInjectionReady, l15Settled, lastMmdInjectedTokens, cachedUserPrompt, lastL15PromptHash, l15Boundaries, and P1 quick-skip counters — while preserving cachedSystemPrompt and its token count across session switches within the same agent.[7] switchSession() reconstructs processedToolCallIds from persisted offload entries, including both the raw tool_call_id and its underscore-stripped normalized form, to handle ID format variations.[7]

The before_prompt_build hook in src/offload/hooks/before-prompt-build.ts runs a three-phase context cleanup pipeline before each LLM call: (1) fast-path re-apply of confirmed mild replacements and deletion of aggressive-deleted messages, (2) a token guard that triggers full L3 (Aggressive + Mild) compression inline if still above thresholds, and (3) MMD injection of active/history memory into the message list.[8] The before_prompt_build handler skips processing entirely for internal memory-pipeline sessions whose session key matches the pattern /memory-.*-session-\d+/.[8] When there are no confirmed or deleted offload IDs, the handler skips all compression phases and goes directly to MMD injection with waitForL15: true, then returns.[8] The token guard computes aggressive and mild thresholds as floor(contextWindow * aggressiveRatio) and floor(contextWindow * mildRatio), using PLUGIN_DEFAULTS values when pluginConfig does not supply them.[8] Aggressive compression performance is monitored: if a round takes longer than 10 000 ms, a warning is logged with round count, deleted count, and remaining token estimate.[8] When aggressive compression stalls because user messages are protected and the token count is still at or above the aggressive threshold, stateManager._forceEmergencyNext is set to true to trigger an emergency fallback on the next cycle.[8] Mixed assistant messages that contain both text and tool_use blocks have deleted tool_use blocks stripped inline (by splicing from content) before the message list is sent to the LLM, preventing Anthropic 400 errors from orphaned tool_use without a matching tool_result.[8] After aggressive compression, src/offload/hooks/before-prompt-build.ts calls buildHistoryMmdInjection and splices the resulting MMD messages into the message list at the history insertion point, adding their token cost to workingTokens before the mild-compression threshold check.[8] Both aggressive and mild compression outcomes update persistent offload status via markOffloadStatus, with aggressive deletions stored as "deleted" and mild replacements stored as true; failures are logged but do not interrupt the pipeline.[8]

parseL1Response in src/offload/local-llm/parsers/l1-parser.ts converts raw LLM output into an OffloadEntry[], tolerating markdown wrapping and missing fields.[9] parseL1Response requires tool_call_id to be non-empty — any entry missing this field is silently skipped — and defaults the score field to 5 when the raw LLM output does not supply a numeric value, setting node_id to null.[9]

resolveApiKeyFromAuthProfile in src/offload/auth-profile-key.ts returns a plaintext API key only for profiles of type api_key that have a key string; profiles with only a keyRef (e.g., keychain-backed) return undefined.[10] resolveApiKeyFromAuthProfile returns undefined for providers whose profiles are of type oauth or token (non-API-key auth flows), and also returns undefined gracefully when the provider-auth SDK is unavailable (e.g., older host versions that do not supply it).[10]

Example: canonical usage of resolveApiKeyFromAuthProfile — pass the agent API object, provider name, optional override, and an SDK factory; receive the plaintext key or undefined.

const key = resolveApiKeyFromAuthProfile(api, "xiaomi", undefined, () => sdk);
expect(key).toBe("sk-xiaomi-123");

[10]

Sources

4.3Seed & CLI

Updated 2026-08-22

src/core/seed/seed-runtime.ts orchestrates the full seed pipeline in the sequence L0 → L1 → L2 → L3, using pipeline-factory for VectorStore/EmbeddingService init, L1 runner, L2 runner, L3 runner, and persister wiring.[1] executeSeed() is the core runtime called by src/cli/commands/seed.ts after all input validation and user confirmation are complete.[1]

The SeedRuntimeOptions interface exposes: outputDir (all seed output), openclawConfig, optional pluginConfig, optional inputFile (for manifest traceability), logger, and an optional onProgress callback invoked after each round.[1] In createSeedPipeline(), two separate LLMRunner instances are created from StandaloneLLMRunnerFactory: l1LlmRunner with enableTools: false and l2l3LlmRunner with enableTools: true; standalone runners are created only when cfg.llm.enabled && cfg.llm.apiKey.[1]

In executeSeed(), captureStartTimestamp is hardcoded to 0 so that the cold-start guard in captureAtomically() does NOT filter out historical messages — unlike live mode, which uses Date.now() to prevent the first agent_end from dumping full session history.[1] Round messages are mapped to objects with a timestamp field (not ts) before being passed to performAutoCapture(), because l0-recorder's extractUserAssistantMessages reads m.timestamp for incremental filtering.[1] everyNConversations (from openclawConfig) sets the batch size: executeSeed() feeds that many conversation rounds to the pipeline before pausing to wait for L1 to become idle.

After every everyNConversations rounds, executeSeed() must wait for L1 to finish before feeding more rounds; skipping this pause would cause L1 to run only once on the full batch, defeating the "every N" batching semantics.[1] The seed runtime only waits for L1 to become idle before calling pipeline.destroy(); L2 (scene extraction) and L3 (persona generation) may still be in-flight at that point, meaning seed output may not include the latest L2/L3 artifacts.[1]

waitForL1Idle() polls scheduler.getQueueSizes(), scheduler.getBufferedMessageCount(), and scheduler.getSessionState() until L1 is idle, with defaults: pollIntervalMs=1000, stableRounds=3, maxWaitMs=300_000 (5 minutes).[1] L1 is considered truly idle only when queues.l1Idle is true, totalBuffered === 0, and totalConversationCount === 0 — all held for stableRounds consecutive polls.[1]

executeSeed() handles Ctrl+C gracefully: the first SIGINT sets an interrupted flag and allows the current round to finish before shutting down; a second SIGINT calls process.exit(1) immediately.[1]

Sources

src/core/seed/seed-runtime.ts

5.1Configuration

Updated 2026-08-22

src/config.ts organizes all plugin configuration into flat functional groups: capture, extraction (L1), persona (L2/L3), pipeline, recall, and embedding. The minimal valid config is {} — every field has a default.[1] The configSchema in openclaw.plugin.json sets "additionalProperties": true, so unrecognized config keys are accepted and not rejected by schema validation.[2]

The storeBackend option selects the storage backend: "sqlite" (local SQLite + sqlite-vec, the default) or "tcvdb" (Tencent Cloud Vector Database). Backend-specific behavior is covered in SQLite backend and TCVDB backend.[2] The timezone option defaults to "system" (follows the process system timezone); accepted values include IANA timezone names (e.g., "Asia/Shanghai") and UTC offset strings (e.g., "+08:00"). Storage timestamps are always UTC — this setting only affects the presentation layer.[2]

capture.l0l1RetentionDays defaults to 0, which disables cleanup of L0/L1 local files entirely. A non-zero value must be >= 3 unless capture.allowAggressiveCleanup is explicitly set to true, which permits 1- or 2-day retention.[1][2] Daily cleanup runs at 03:00 by default and is controlled by capture.cleanTime (format HH:mm). Cleanup only activates when retentionDays is a positive number.[1][2]

L1 background memory extraction is enabled by default (extraction.enabled: true), and smart deduplication — based on vector similarity or keyword conflict detection — is also enabled by default (extraction.enableDedup: true).[1][2] Extraction is capped at extraction.maxMemoriesPerSession memories per session per run (default 20). The extraction.model field is optional and falls back to the OpenClaw default model when omitted.[2][1] StandaloneLLMOverrideConfig allows using a different — typically cheaper or faster — model for memory extraction while the main agent uses a premium model. When enabled: false (the default), the host's native LLM mechanism is used instead.[1]

BM25Config defaults language to "zh" (Chinese); set to "en" for English-language pre-trained BM25 parameters. This setting uses the local @tencentdb-agent-memory/tcvdb-text package.[1] BM25 is a keyword-based ranking algorithm used for text search, providing a fast, non-vector recall path that complements semantic (embedding) search during memory retrieval.

Metrics reporting is disabled by default (report.enabled: false). When enabled, report.type: "local" outputs metrics as structured JSON logs via the Gateway logger.[1][2]

Sources

5.2Search tools

Updated 2026-08-22

Agent Memory provides memory_search and conversation_search tools that each employ three strategies with automatic degradation: hybrid (FTS5 + vector embedding merged via Reciprocal Rank Fusion), embedding-only, and FTS5-only, gracefully falling back when indexing services are unavailable. Both tools apply filtering and over-retrieval to candidate results before merging ranked lists, then render formatted responses that report which strategy produced results and any configuration guidance needed. Reciprocal Rank Fusion (RRF) is a rank-merging algorithm that combines independently ranked result lists into a single ordering without requiring score scales to be aligned, enabling hybrid search to meaningfully merge FTS5 keyword ranks and vector similarity scores.

The memory_search tool, implemented in src/core/tools/memory-search.ts, supports three search strategies with automatic degradation: hybrid (FTS5 keyword + vector embedding in parallel, merged via Reciprocal Rank Fusion — the default), embedding (pure vector similarity, used when FTS5 is unavailable), and fts (pure FTS5 keyword search, used when embedding is unavailable).[1] Tool registration is handled via api.registerTool() in index.ts; src/core/tools/memory-search.ts contains only the search logic and response formatting.[1]

Hybrid search runs FTS5 and vector lookups in parallel via Promise.all, each over-retrieving limit × 3 candidates before merging.[1] rrfMergeL1 merges the ranked result lists using Reciprocal Rank Fusion with the standard RRF constant k = 60; items appearing in multiple lists accumulate scores of 1 / (60 + rank + 1), and the score field of each returned item is replaced with the computed RRF score.[1] FTS5 and vector search failures inside executeMemorySearch are non-fatal: each branch catches errors, logs a warning, and returns an empty list so the surviving strategy can still produce results.[1]

Scene filtering in executeMemorySearch is applied after merging using case-insensitive substring matching (r.scene_name.toLowerCase().includes(normalizedScene)) rather than exact equality.[1] formatSearchResponse renders each memory item with its type, priority, scene name, and score; items with priority < 0 are labeled (global instruction) instead of showing the numeric priority.[1] The MemorySearchResult interface exposes a strategy field reporting which path was used ("hybrid", "embedding", "fts", or "none") and an optional message field for error or advisory text.[1] When neither an embedding service nor FTS5 is available, executeMemorySearch returns { results: [], total: 0, strategy: "none" } with a message advising the caller to configure an embedding provider (e.g. openai_compatible) via the embedding.provider setting.[1]

The conversation_search tool, implemented in src/core/tools/conversation-search.ts, mirrors the same three-strategy pattern with automatic degradation: hybrid (FTS5 + vector in parallel, merged via RRF), embedding (pure vector similarity), and fts (pure FTS5 keyword search).[2] Like memory_search, the tool is registered via api.registerTool() in index.ts, with core logic confined to src/core/tools/conversation-search.ts.[2] ConversationSearchResultItem represents a single L0 message with fields: id, session_key, role ("user" or "assistant"), content, score, and recorded_at.[2]

executeConversationSearch returns { results: [], total: 0, strategy: "none" } immediately for empty or whitespace-only queries, without touching any store.[2] If vectorStore is not provided, executeConversationSearch also returns { results: [], total: 0, strategy: "none" } regardless of other parameters.[2]

Over-retrieval in conversation-search.ts uses candidateK = limit × 4 when a sessionKey filter is present, and limit × 3 otherwise, to compensate for post-merge session filtering.[2] FTS5 and vector searches are executed in parallel via Promise.all; failures in either branch are non-fatal and return an empty array so the other strategy can still produce results.[2] Reciprocal Rank Fusion in conversation-search.ts uses the standard RRF constant K = 60; items appearing in multiple ranked lists have their scores summed, and the score field of each returned item is replaced by its RRF score.[2] Session-key filtering is applied after RRF merging: the merged result set is filtered to r.session_key === sessionFilter, then trimmed to limit.[2]

The effective strategy reported in ConversationSearchResult.strategy reflects which branches actually returned results: both → "hybrid", only vector → "embedding", only FTS5 → "fts"; when neither returns results, the value falls back to "embedding" or "fts" depending on availability.[2] The ConversationSearchResult interface carries an optional message field for configuration advice; when that field is present, formatConversationSearchResponse renders it as its entire output.[2] When neither the embedding service nor FTS5 is available, executeConversationSearch populates message advising the caller to configure an embedding provider (e.g. openai_compatible) via the embedding.provider setting.[2] formatConversationSearchResponse formats each result as **[role]** Session: <session_key> [recorded_at] (score: X.XXX) followed by the message content, with results separated by --- dividers.[2]

Sources

5.3LLM thinking suppression

Updated 2026-08-22

src/utils/no-think-fetch.ts provides a multi-strategy fetch wrapper for suppressing LLM thinking/reasoning tokens, defining the DisableThinkingStrategy type as false | "vllm" | "deepseek" | "dashscope" | "openai" | "anthropic" | "kimi" | "gemini".[1] The llm.disableThinking config option (and its parallel offload.disableThinking) accepts false (default — no suppression) or one of those seven provider strategy strings; the TDAI_LLM_DISABLE_THINKING environment variable is also accepted and normalized via normalizeDisableThinking from no-think-fetch.js.[2][3] Thinking/reasoning tokens are intermediate chain-of-thought outputs generated by some LLMs before their final answer; suppressing them reduces latency and cost when only the final response is needed.

Each strategy injects provider-specific request fields via STRATEGY_TRANSFORMERS in no-think-fetch.ts: "vllm" → chat_template_kwargs.enable_thinking = false; "deepseek"/"dashscope" → top-level enable_thinking: false; "openai" → reasoning_effort: "low"; "anthropic"/"kimi" → thinking: { type: "disabled" }; "gemini" → thinking_config: { thinking_budget: 0 }.[1][4] The "openai" strategy sets reasoning_effort: "low" rather than fully disabling thinking, because the OpenAI o-series API does not support full reasoning suppression.[1] The "vllm" strategy merges enable_thinking: false into any pre-existing chat_template_kwargs object rather than replacing it, preserving other keys already present on the request body.[1] The "kimi" (Moonshot) strategy shares the same transformer as "anthropic", injecting thinking: { type: "disabled" }.[1]

normalizeDisableThinking() treats true as a shorthand for "vllm" (the most common self-hosted scenario), treats false/undefined as false, and emits a console.warn for any unrecognized string before falling back to false.[1] createNoThinkFetch() silently passes through requests with non-JSON bodies: parse errors are caught and the original request is forwarded unchanged.[1]

Sources

6TdaiCore & host abstraction

Updated 2026-08-22

TdaiCore is a host-neutral facade in src/core/tdai-core.ts that centralizes memory operations (recall, capture, search, pipeline) for both OpenClaw and Hermes/Gateway; it depends only on abstract HostAdapter and LLMRunner interfaces, never concrete host logic. HostAdapter abstracts three concerns — runtime context (user/session), LLM invocation, and logging — allowing TdaiCore to work across OpenClaw, Hermes, and Gateway without hardcoded host coupling.

TdaiCore in src/core/tdai-core.ts is the host-neutral facade for all TDAI memory capabilities — recall, capture, search, and pipeline management — that both OpenClaw and Hermes/Gateway call. It depends only on abstract HostAdapter and LLMRunner interfaces, never on a specific host.[1] src/core/types.ts is the host-neutral boundary layer that defines the abstract interfaces (HostAdapter, LLMRunner, LLMRunnerFactory, RuntimeContext, Logger) TdaiCore depends on; each host environment (OpenClaw, Hermes, Gateway) must provide its own implementations of HostAdapter and LLMRunnerFactory.[2]

HostAdapter answers three questions for TdaiCore: who is the current user/session (getRuntimeContext()), how to call an LLM (getLLMRunnerFactory()), and where to log (getLogger()). Its hostType is one of "openclaw", "hermes", or "standalone".[2] Two concrete HostAdapter implementations exist: OpenClawHostAdapter (wraps OpenClawPluginApi) and StandaloneHostAdapter (wraps Gateway HTTP request context for Hermes/Gateway hosts).[2] RuntimeContext.platform accepts the known literals "openclaw", "hermes", "cli", or "gateway", or any arbitrary string for future or custom hosts.[2] RuntimeContext.agentContext distinguishes between four execution modes: "primary", "subagent", "cron", and "flush".[2]

LLMRunner.run() returns an empty string (not null/undefined) when the LLM produces no output, and throws on timeout, network error, or unrecoverable LLM failure.[2] Two concrete LLMRunner implementations exist: OpenClawLLMRunner (wraps CleanContextRunner/runEmbeddedPiAgent for the OpenClaw host) and StandaloneLLMRunner (direct OpenAI-compatible HTTP calls for Gateway/Hermes hosts).[2] LLMRunParams.timeoutMs defaults to 120_000 ms (2 minutes) when omitted.[2] LLMRunnerCreateOptions.enableTools defaults to false (text-only output); set it to true for LLM tasks that require file tool access, such as L2 scene or L3 persona generation.[2] LLMRunnerCreateOptions.modelRef takes a "provider/model" string (e.g. "openai/gpt-4o") and takes precedence over the host's default model when specified.[2] When LLMRunParams.workspaceDir is omitted in a tool-enabled run (enableTools: true), a clean empty workspace is used instead of inheriting the session workspace.[2]

The Logger interface in src/core/types.ts requires info, warn, and error methods; debug is optional. Named variants such as StoreLogger and PluginLogger are type aliases of Logger, kept only for backward compatibility.[2]

TdaiCore.initialize() must be called once before any other methods. It initializes data directories, starts async store initialization, and — if extraction is enabled — creates and wires the pipeline manager.[1] Pipeline runners are wired after the store is ready, but initialize() also wires them in degraded mode (JSONL fallback, no embedding) if store initialization fails, so the pipeline remains functional despite storage errors.[1] TdaiCore.handleBeforeRecall(userText, sessionKey) performs memory retrieval before an LLM turn, corresponding to OpenClaw's before_prompt_build hook or Hermes's prefetch() call.[1] TdaiCore.handleTurnCommitted(turn) handles conversation capture and pipeline trigger after a turn completes, corresponding to OpenClaw's agent_end hook or Hermes's sync_turn() call.[1] TdaiCore.searchMemories(params) searches L1 structured memories and maps to the tdai_memory_search tool; the limit parameter defaults to 5.[1] TdaiCoreOptions accepts an optional sessionFilter (defaults to a SessionFilter with an empty exclusion list) and an optional instanceId for metric reporting.[1]

CompletedTurn.originalUserMessageCount in src/core/types.ts records the number of messages in the session at before_prompt_build time; the L0 recorder uses this to locate the exact user message that was modified by prependContext injection.[2]

TdaiCore.destroy() drains all in-flight background tasks (tracked in bgTasks) with a 5-second hard timeout before closing vectorStore and embeddingService, preventing late updateL0Embedding calls from landing on an already-closed database connection.[1] After closing vector store and embedding service, TdaiCore.destroy() calls resetStores(this.dataDir), ensuring per-directory store caches are cleared on shutdown.[1] Each background task registered in bgTasks removes itself in its own finally handler, keeping the set bounded to the number of currently-running background tasks.[1]

Canonical usage of TdaiCore with the OpenClaw adapter:

// OpenClaw path (in-process)
const adapter = new OpenClawHostAdapter({ api, pluginDataDir, config });
const core = new TdaiCore({ hostAdapter: adapter, config: parsedCfg });
await core.initialize();
const recall = await core.handleBeforeRecall("user query", "session-1");

[1]

Sources

6.1Recall & context injection

Updated 2026-08-22

Recall output splits into prependContext (dynamic L1 memories in the user prompt prefix, cached per-turn) and appendSystemContext (stable persona, scene, and tools guide in the system prompt, cached across turns), optimizing provider prompt caching in auto-recall.ts. L1 memories are injected as a <relevant-memories> XML block with truncation limits (maxCharsPerMemory, maxTotalRecallChars) applied in score order, while a static <memory-tools-guide> instructs the agent to call tdai_memory_search and tdai_conversation_search for deeper retrieval. Prompt caching is a provider-side optimization where an unchanged prompt segment is reused across requests without reprocessing, reducing latency and cost.

Recall output is split into two fields on RecallResult (defined in src/core/types.ts): prependContext carries dynamic, per-turn L1 memories prepended to the user prompt, while appendSystemContext carries stable content — persona, scene navigation, and tools guide — appended to the system prompt.[1] This two-field design, implemented in src/core/hooks/auto-recall.ts, optimises provider prompt caching: stable system-prompt content rarely changes across turns, so providers such as Anthropic and OpenAI can cache it, while the dynamic L1 block is kept in the user prompt prefix where per-turn changes do not bust that cache.[2] L1 memories arrived in prependContext (before the user message) starting in v0.3.3, having previously lived in appendSystemContext, where per-turn changes caused system-prompt cache busting.[3]

Recalled L1 memories are injected into the user prompt prefix as a <relevant-memories> XML block containing a Chinese-language disclaimer that the memories are reference context and do not represent the current task state.[2] Memory lines that exceed the per-memory character limit are truncated, and the suffix …（已截断；可用 tdai_memory_search 或 tdai_conversation_search 查看详情） is appended; the minimum retained length before truncation is 40 characters (MIN_TRUNCATED_RECALL_LINE_CHARS).[2] At the end of the stable system context, auto-recall.ts injects a static MEMORY_TOOLS_GUIDE XML block (<memory-tools-guide>) that instructs the agent it may call tdai_memory_search, tdai_conversation_search, and read_file for deeper retrieval, with a combined cap of 3 calls per turn for the first two tools.[2]

RecallConfig in src/config.ts ships with defaults of enabled: true, maxResults: 5, scoreThreshold: 0.3, strategy: "hybrid", and timeoutMs: 5000.[4] The strategy field accepts "embedding", "keyword", or "hybrid" (default); choosing "embedding" or "hybrid" requires a configured embedding provider (see Embedding services).[4] When the recall timeout (recall.timeoutMs, default 5000 ms) is exceeded, memory injection is skipped entirely and a warning is logged rather than failing the request.[5]

recall.maxCharsPerMemory caps the character count of each injected L1 memory, and recall.maxTotalRecallChars caps the total character budget across all recalled L1 memories in a single auto-recall pass; setting either to 0 disables that limit.[4][5] Both options were introduced in v0.3.6 and apply truncation in score order, discarding overflow to prevent long sessions from having their context crowded out by memory bloat.[6]

In src/core/hooks/auto-recall.ts, when userText is empty or undefined, L1 memory search is skipped entirely, but L3 persona and L2 scene navigation are still injected into appendSystemContext.[2] The RecalledMemory interface exported from src/core/hooks/auto-recall.ts holds content (string), score (number), and type (string) — one entry per recalled L1 memory — and is used for metric reporting in the agent_turn event.[2]

Sources

6.2L0 capture

Updated 2026-08-22

Automatic L0 conversation capture is enabled by default (capture.enabled defaults to true); agents matching patterns in capture.excludeAgents are excluded from capture, recall, and pipeline scheduling entirely.[1] At the L0 stage, auto-capture.ts extracts new messages from each agent_end event, sanitizes them, and passes them to the pipeline — no remote call happens at this stage; messages are buffered locally per-session.[2] L0 capture filters out MMD context blocks injected by the offload mechanism to prevent storing compression intermediates as memories.[3]

src/core/conversation/l0-recorder.ts implements the L0 raw conversation capture layer, writing sanitized messages to ~/.openclaw/memory-tdai/conversations/YYYY-MM-DD.jsonl — one file per day, all sessions merged into it.[4] JSONL format is used with one message per line so the output is flat and easy to grep or stream; the sessionKey is stored as a field in each JSONL line, not in the filename.[4] The L0MessageRecord interface defines the flat JSONL record shape: sessionKey, sessionId, recordedAt (ISO timestamp), id, role, content, and timestamp (epoch ms).[4]

generateMessageId() in src/core/conversation/l0-recorder.ts produces short IDs of the form msg_<epoch>_<6-hex-chars> using crypto.randomBytes(3); the id field is later used by L1 for source_message_ids tracking.[4] L0 vector record IDs generated by auto-capture.ts include the session key, current timestamp, per-message index, and 3 random bytes of hex to ensure uniqueness across multiple messages in the same capture round.[5]

auto-capture.ts accepts an originalUserText parameter — the clean user prompt captured at before_prompt_build time before prependContext was injected — alongside originalUserMessageCount to locate the exact message position, preventing memory-injected user text from being written to L0.[5] After content replacement, recordConversation sanitizes all messages with sanitizeText() and additionally strips fenced code blocks from assistant replies using stripCodeBlocks() to reduce embedding noise.[4] recordConversation returns an empty array and skips file I/O when no new user/assistant messages remain after position-slicing, timestamp-cursor filtering, and noise filtering.[4] When the position slice is unavailable and all messages pass the timestamp filter with more than 8 messages present, recordConversation emits a warning about possible timestamp drift after a gateway restart.[4]

auto-capture.ts prevents duplicate L0 records from concurrent agent_end events by performing L0 recording and checkpoint cursor advance atomically inside checkpoint.captureAtomically(), which holds a file lock across the entire read-cursor → recordConversation → advance-cursor sequence.[5] When no per-session checkpoint cursor exists, pluginStartTimestamp is used as the floor cursor to prevent the first agent_end from dumping all session history into L0.[5] A checkpoint cursor marks the timestamp of the last successfully captured message in a session, acting as a lower bound so that each capture round in auto-capture.ts processes only new messages and avoids re-recording earlier turns.

auto-capture.ts supports two L0 vector indexing paths: Path A (stores with supportsDeferredEmbedding === true, e.g. SQLite) writes metadata and FTS immediately and fires a background task for embedding to avoid blocking agent_end with 2–3 s embedding calls; Path B (VDB/remote stores) embeds synchronously then upserts with the embedding in one call.[5] When using a server-side embedding service (embeddingService.getDimensions() === 0, i.e. NoopEmbeddingService), local embedding is skipped during L0 vector indexing.[5] Embedding failure for an individual L0 message is non-fatal: a warning is logged and the record is still written to the store with metadata only (no embedding vector).[5]

The bgTaskRegistry parameter (a Set<Promise<void>>) lets TdaiCore.destroy() await all in-flight background L0 embedding tasks before closing vectorStore/embeddingService, preventing use of an already-closed DB connection; the parameter is optional for backwards compatibility.[5]

The AutoCaptureResult interface exported from auto-capture.ts exposes schedulerNotified (bool), l0RecordedCount (number of messages written to L0), l0VectorsWritten (number of L0 message vectors written), and filteredMessages (messages for L1 immediate use — see L1 extraction & dedup).[5]

Sources

6.3L1 extraction & dedup

Updated 2026-08-22

src/core/record/l1-extractor.ts implements the L1 extraction pipeline: it reads L0 conversation messages, calls an LLM to perform scene segmentation and memory extraction in one call, runs batch conflict detection, and writes results to L1 JSONL files.[1] The L1 extractor applies a quality gate (shouldExtractL1) to filter messages by length, symbols, and prompt-injection patterns before sending them to the LLM. L0 captures everything; strict filtering happens at the L1 stage.[1]

extractL1Memories splits qualified messages into a newMessages slice (the last maxMessagesPerExtraction messages, default 10) and a backgroundMessages slice (up to maxBackgroundMessages older messages, default 5) for LLM context.[1] The maximum memories stored per extraction call is bounded by options.maxMemoriesPerSession (default 10); excess memories, sorted by extraction order, are truncated before dedup.[1] extractL1Memories accepts an optional llmRunner (LLMRunner) that decouples it from the OpenClaw runtime; when provided, this runner is used instead of creating a CleanContextRunner directly.[1]

src/core/prompts/l1-extraction.ts implements the L1 extraction stage as a single LLM call combining scene segmentation (情境切分) and memory extraction (记忆提取), based on Kenty's validated prototype prompt.[2] The L1 system prompt restricts extraction to exactly three memory types: persona (stable user attributes/preferences), episodic (objective events/decisions/plans), and instruction (long-term behavioral rules the user sets for the AI).[2] Priority score ranges are defined per type: persona — 80–100 (health/core traits), 50–70 (general preferences); episodic — 80–100 (important events), 60–70 (general activities), below 60 discard; instruction — -1 for absolute global commands, 90–100 (core rules), 70–80 (important), below 70 discard.[2] For episodic memories, the system prompt mandates recording absolute time using the message timestamp when possible, outputting activity_start_time and activity_end_time in ISO 8601 format inside the metadata object.[2] formatExtractionPrompt builds the user-side L1 prompt from newMessages (messages to extract from), backgroundMessages (context only, not extracted), and previousSceneName (for scene continuity), with backgroundMessages defaulting to [] and previousSceneName defaulting to "无".[2]

Example: formatExtractionPrompt in src/core/prompts/l1-extraction.ts formats each message as [id] [role] [timestamp]: content and separates background from new messages with a visual rule, instructing the LLM to extract only from the new-messages section.

const bgText = backgroundMessages.length > 0
  ? backgroundMessages
      .map((m) => `[${m.id}] [${m.role}] [${formatForLLM(m.timestamp)}]: ${m.content}`)
      .join("\n\n")
  : "无";
const newText = newMessages
  .map((m) => `[${m.id}] [${m.role}] [${formatForLLM(m.timestamp)}]: ${m.content}`)
  .join("\n\n");

[2]

Scene segmentation (情境切分) divides a conversation into distinct topical or contextual scenes. The L1 extractor uses scene boundaries to group related messages and maintains narrative continuity across calls by passing the detected previousSceneName forward to each successive extractL1Memories invocation.

Memories with an unrecognized type field from the LLM output are skipped with a warning log; only memories with a valid, normalized MemoryType are added to the extraction result.[1] Each extracted memory is assigned a temporary record_id (via generateMemoryId) before being passed to batch dedup; this ID is required for the dedup system to correlate decisions back to individual memories.[1]

Conflict/dedup detection is enabled by default in extractL1Memories (enableDedup defaults to true); it can be disabled via options.enableDedup = false, in which case all extracted memories are stored directly without conflict checking.[1] src/core/record/l1-dedup.ts implements batch conflict detection using a two-phase approach: (1) fast candidate recall per new memory using vector search or FTS5 BM25 (no LLM), then (2) a single batch LLM call to judge all new memories against their candidate pools.[3] The JSONL-based Jaccard word-overlap fallback was removed in l1-dedup.ts v4. Candidate recall now degrades through three tiers: (1) vector cosine similarity, (2) FTS5 BM25 keyword recall, (3) skip dedup entirely — never a full-file scan.[3] The default conflictRecallTopK in batchDedup is 5 — the number of candidate existing memories recalled per new memory for the LLM conflict judgment.[3] Tier 1 (vector recall) in batchDedup requests topK + memories.length results from the vector store to account for self-batch filtering, then slices to topK after excluding records from the current extraction batch.[3] FTS5 candidate recall in l1-dedup.ts fetches up to 10 FTS results per memory, then slices to 5 candidates after filtering out records in the current batch.[3] If vector recall fails during Tier 1, batchDedup automatically degrades to FTS5 keyword recall. If FTS is also unavailable, conflict detection is skipped and all memories are stored directly.[3] In batchDedup, if neither vector data nor FTS is available, conflict detection is skipped entirely and all memories receive a "store" decision — dedup is not attempted to avoid an O(N) full-file-scan cost.[3] If no memory has any candidate matches after recall (all candidate lists are empty), batchDedup skips the LLM judgment call entirely and stores all memories directly.[3]

When llmRunner is provided to batchDedup, the LLM judgment call uses it directly via llmRunner.run(); otherwise a CleanContextRunner is instantiated as the OpenClaw-specific fallback path.[3] The LLM conflict detection call in l1-dedup.ts applies a timeout of 180_000 ms (3 minutes) on both the LLMRunner path and the CleanContextRunner fallback path.[3] batchDedup accepts an optional embeddingTimeoutMs to override the embedding call timeout specifically on the capture path, passed through to embeddingService.embedBatch().[3] If the batch LLM judgment call fails in runLlmJudgment, all memories default to the "store" action rather than failing the entire extraction.[3]

L1ExtractionResult (exported from src/core/record/l1-extractor.ts) carries success, extractedCount, storedCount, records (the stored MemoryRecord[]), sceneNames, and lastSceneName (the last detected scene, for continuity in the next extraction call).[1] If the LLM extraction call itself fails in extractL1Memories, the function returns { success: false, extractedCount: 0, storedCount: 0, records: [], sceneNames: [] } — it does not fall back to partial storage.[1] When instanceId and logger are both provided, extractL1Memories emits an l1_extraction metric via report(), including inputMessageCount, memoriesExtracted, memoriesStored, memoriesByType distribution, totalDurationMs, and the content/type/scene of each stored memory.[1] CaptureResult in src/core/types.ts reports the count of L0 messages recorded (l0RecordedCount), whether the pipeline scheduler was notified (schedulerNotified), the number of L0 vectors written (l0VectorsWritten), and the filtered messages that were actually captured (filteredMessages) — see L0 capture for full capture-path context.[4]

Sources

6.4L2 scene extraction

Updated 2026-08-22

SceneExtractor is the L2 memory pipeline layer that autonomously reads and writes scene blocks using an LLM agent with tool access, replacing the earlier keyword-based approach and operating in a sandboxed scene_blocks/ directory invisible to system files. SceneExtractor follows a five-phase flow for each extraction: snapshot state, assemble LLM prompt with memories and scene context, run the agent with tool access, clean up soft-deletes and sync the index, then parse output for persona signals — returning early with success if memories are empty.

SceneExtractor in src/core/scene/scene-extractor.ts is the L2 layer of the memory pipeline, replacing the keyword-based SceneManager.processNewMemories() with an LLM agent that autonomously reads and writes scene block files using tools.[1] The ExtractionResult interface exported from src/core/scene/scene-extractor.ts has three fields: memoriesProcessed: number, success: boolean, and optional error?: string.[1]

SceneExtractor sandboxes the LLM to scene_blocks/ by setting workspaceDir to that directory, making system files (checkpoint, scene_index, persona.md) physically invisible to the LLM.[1] Before invoking the LLM, SceneExtractor snapshots both the scene index and the content of every scene file, enabling diffing of created, updated, and deleted scenes after the run.[1] An optional SceneExtractorOptions.llmRunner injection point accepts any host-neutral LLMRunner; when provided it is used instead of creating a CleanContextRunner, decoupling SceneExtractor from the OpenClaw runtime — see OpenClaw integration. The injected runner must be configured with enableTools: true.[1]

SceneExtractor.extract() follows a five-phase flow: (1) backup + load scene index + build summaries, (2) assemble extraction prompt with memories and scene context, (3) run via CleanContextRunner sandboxed to scene_blocks/, (4) cleanup soft-deletes + sync index + update navigation, (5) parse LLM text output for out-of-band persona update signals.[1] SceneExtractor.extract() returns { memoriesProcessed: 0, success: true } immediately — skipping all LLM work — when called with an empty memories array.[1]

The LLM "deletes" scene files by writing the marker [DELETED] into the file rather than issuing shell commands; SceneExtractor post-processes these soft-deletes by detecting and removing marked files before calling syncSceneIndex, preventing stale entries from being re-indexed.[1] On LLM runner failure, SceneExtractor.extract() attempts a fail-soft restore of scene_blocks/ from the Phase 1 backup so partial LLM writes do not leak into the next recall cycle; a restore failure is logged but does not mask the original LLM error.[1]

SceneExtractor defaults: maxScenes = 15, sceneBackupCount = 10, and timeoutMs = 300,000 ms (5 minutes, to accommodate multiple tool calls).[1] parsePersonaUpdateSignal(text) is an exported function that parses LLM output for out-of-band persona update request signals, supporting a block format ([PERSONA_UPDATE_REQUEST]reason: xxx[/PERSONA_UPDATE_REQUEST]) and an inline format (PERSONA_UPDATE_REQUEST: xxx).[1]

Sources

src/core/scene/scene-extractor.ts

6.5L3 persona generation

Updated 2026-08-22

PersonaGenerator in src/core/persona/persona-generator.ts implements the L3 layer, generating or updating the user persona using a four-layer deep scan model and writing persona.md via the LLM agent's file tools.[1] L3 persona generation is triggered every persona.triggerEveryN new memories (default 50), with a maximum of persona.maxScenes scenes processed per run (default 15).[2] The four-layer deep scan model comprises four hierarchical memory levels — raw events, scenes, summaries, and persona — with L3 persona generation synthesizing from scene-level data within this hierarchy.

PersonaGenerator accepts an optional llmRunner?: LLMRunner injection; when provided, it is used instead of CleanContextRunner, decoupling the generator from the OpenClaw runtime. The injected runner must be configured with enableTools: true.[1] PersonaGenerator defaults backupCount to 3, controlling how many rolling backups of persona.md are retained.[1]

PersonaGenerator.generateLocalPersona() uses "incremental" mode when a non-empty persona.md already exists, and "first" mode on the initial run; it skips generation entirely (returning false) when no scene changes have occurred since the last persona update and a persona file already exists.[1] Changed scenes are detected by comparing each scene's updated timestamp against checkpoint.last_persona_time; if either date is unparseable (NaN), the scene is conservatively treated as changed.[1] The LLM agent runs sandboxed to dataDir (not scene_blocks/) with a 180-second timeout, writing persona.md directly via file tools; maxTokens is omitted, so the core resolves the model's limit from its catalog.[1] In "incremental" mode, the LLM agent receives only changed scenes; in "first" mode, it receives all available scene data — determining the agent's context window and prompt construction for each run.

PersonaGenerator emits an l3_persona_generation metric report (via report()) including triggerReason, mode ("incremental" or "initial"), persona content and length, total duration, and a success flag — but only when instanceId is set.[1]

Sources

6.6Pipeline scheduling

Updated 2026-08-22

MemoryPipelineManager in src/utils/pipeline-manager.ts coordinates the L0→L1→L2→L3 memory extraction pipeline, managing timers, queues, and runners for each layer.[1]

L1 batch processing fires every pipeline.everyNConversations conversation turns (default 5). With pipeline.enableWarmup: true (the default), new sessions begin triggering after just 1 conversation and double the threshold each time (1→2→4→…→everyN) to accelerate early memory extraction.[2] After a user goes quiet, L1 is also triggered after pipeline.l1IdleTimeoutSeconds seconds of idle time (default 600 s / 10 minutes).[2] On failure, MemoryPipelineManager retries L1 up to L1_MAX_RETRIES (5) times with a L1_RETRY_DELAY_MS delay of 30,000 ms (30 seconds) between attempts; the retry count resets on success or when a new conversation arrives.[1]

L2 is triggered by three distinct paths: (A) a delay after L1 completes, advancing the timer to max(now + delay, lastL2 + min); (B) a maxInterval guarantee that polls active sessions; and (C) a shutdown flush of all pending L2 timers.[1] A session is considered active for L2 polling for pipeline.sessionActiveWindowHours hours (default 24); sessions idle beyond that window stop receiving L2 polls.[2]

L3 persona generation uses a global dedup mechanism — l3Pending and l3Running flags combined with a SerialQueue at concurrency=1 — so that a second trigger while L3 is already queued or running is collapsed into the pending flag rather than enqueued twice.[1]

All three processing layers use SerialQueue instances (concurrency=1) with named labels "L1", "L2", and "L3" for diagnostics.[1] Per-session timer state — the L1 idle timer, L2 schedule timer, L1/L2 queued flags, and L1 retry count — is held in memory only and is not persisted to any checkpoint.[1] Session garbage collection runs every 50 notifyConversation() calls (SESSION_GC_EVERY_N_NOTIFICATIONS), evicting sessions inactive for more than 3× sessionActiveWindowMs (SESSION_GC_INACTIVE_MULTIPLIER).[1] The L1Runner callback signature accepts { sessionKey, msg, bg_msg }, where bg_msg is reserved for background context and is currently always empty.[1] A process restart or crash clears all in-memory L1/L2 timers and retry counts; sessions resume normal scheduling only after new conversation activity arrives to re-initialize their per-session state.

Sources

7Storage abstraction

Updated 2026-08-22

The Storage Abstraction Layer in src/core/store/types.ts defines all storage contracts that every backend must implement; upper-layer modules depend only on these interfaces (IMemoryStore), never on concrete implementations (SQLite or TCVDB), enforcing a backend-agnostic design. The Store Factory in src/core/store/factory.ts selects and constructs the active backend, embedding service, and BM25 encoder from config, bundling them into a StoreBundle that carries both runtime collaborators and a manifest snapshot of the deployment choice.

src/core/store/types.ts is the Memory Store Abstraction Layer — it defines all storage contracts (interfaces and types) that every backend implementation must satisfy, and is the single import point for upper-layer modules.[1] Upper-layer modules (hooks, tools, pipeline, record) depend only on the interfaces in src/core/store/types.ts, never on concrete store implementations — an explicit design principle the codebase calls "backend-agnostic".[1] StoreBackend in src/config.ts is a union type "sqlite" | "tcvdb" that selects the storage backend for vector and memory data.[2] Two concrete implementations satisfy IMemoryStore: SqliteMemoryStore (local SQLite + sqlite-vec + FTS5, in sqlite.ts) and TcvdbMemoryStore (Tencent Cloud VectorDB, in tcvdb.ts) — details of each live on the SQLite backend and TCVDB backend sibling pages.[1]

All IMemoryStore methods are documented as fault-tolerant: they return empty results or false on failure rather than throwing, unless explicitly documented otherwise.[1] IMemoryStore uses the MaybePromise<T> return type (T | Promise<T>) for most methods — callers must always await the result to work safely with both sync and async backends.[1] StoreCapabilities exposes four boolean flags — vectorSearch, ftsSearch, nativeHybridSearch, and sparseVectors — that callers inspect to select search strategies and degrade gracefully when a feature is absent.[1] Similarity and BM25 scores in L1SearchResult, L1FtsResult, L0SearchResult, and L0FtsResult are normalized to the range 0–1, where higher is better.[1] L1QueryFilter.updatedAfter accepts an ISO 8601 UTC timestamp and returns only records with updated_time strictly after (not equal to) that timestamp.[1]

The optional supportsDeferredEmbedding flag on IMemoryStore controls embedding write strategy: when true, auto-capture writes metadata-only via upsertL0(record, undefined) and later calls updateL0Embedding() as a background task; when false or absent, embedding is computed inline.[1] updateL0Embedding() is an optional IMemoryStore method that updates only the vector embedding for an existing L0 record — it exists specifically to support the SQLite background-embedding path.[1]

ProfileRecord.id is a stable, deterministic ID derived as profile:v1:${sha256(scope + "\0" + type + "\0" + filename)} — callers must not generate ad-hoc IDs for profile records.[1] ProfileSyncRecord extends ProfileRecord with an optional baselineVersion field that carries the optimistic-lock baseline from the last pull, used to detect concurrent modification during sync.[1] L0Record.timestamp holds the original message timestamp in epoch milliseconds, while L0SessionGroup messages carry recordedAtMs (also epoch ms) representing when the message was recorded into L0 — these two fields serve different cursor purposes.[1] IEmbeddingService exported from src/core/store/types.ts is a re-export alias of EmbeddingService from ./embedding.ts for backward compatibility — all concrete implementations (LocalEmbeddingService, OpenAIEmbeddingService, NoopEmbeddingService) implement the canonical EmbeddingService interface.[1]

src/core/store/factory.ts is the Store Factory — it selects and constructs the correct storage backend (sqlite or tcvdb), embedding service, and optional BM25 encoder from the resolved plugin config, returning them as a StoreBundle.[3] StoreBundle groups three runtime collaborators — store (IMemoryStore), embedding (IEmbeddingService), and optional bm25Encoder (BM25LocalEncoder) — plus a storeSnapshot for manifest writing.[3] "sqlite" is the default backend in createStoreBundle() — the switch statement's default branch handles both an explicit "sqlite" value and any unrecognized value.[3] The BM25 local encoder is always constructed first in createStoreBundle(), regardless of backend, and is passed into both the TCVDB store and returned in the StoreBundle.[3] The SQLite store database file is always placed at vectors.db inside the plugin dataDir, constructed as path.join(options.dataDir, "vectors.db").[3] When the sqlite backend is selected, a local embedding service is only created when config.embedding.enabled is true, config.embedding.provider is not "local", and config.embedding.apiKey is present; otherwise embeddingService is undefined.[3] When the tcvdb backend is selected, the embedding service is always NoopEmbeddingService — TCVDB performs server-side embedding, so no local embedding service is constructed.[3] createStoreBundle() throws a hard error (not a fault-tolerant return) when the tcvdb backend is selected but tcvdb.url, tcvdb.apiKey, or tcvdb.database are missing from config.[3] The storeSnapshot embedded in the returned StoreBundle differs by backend: the TCVDB snapshot records type, tcvdbUrl, tcvdbDatabase, and optionally tcvdbAlias; the SQLite snapshot records type and a relative sqlitePath.[3]

Sources

7.1SQLite backend

Updated 2026-08-22

VectorStore in src/core/store/sqlite.ts manages four SQLite tables in a single database: l1_records (L1 relational metadata), l1_vec (vec0 virtual table for L1 cosine search), l0_conversations (L0 relational metadata), and l0_vec (vec0 virtual table for L0 cosine search).[1] VectorStore requires Node.js 22+ because it uses the built-in node:sqlite (DatabaseSync API) together with the sqlite-vec extension from the root workspace.[1] All VectorStore operations are synchronous, using the DatabaseSync API; writes use manual BEGIN/COMMIT transactions to atomically update both the metadata table and the vec0 virtual table together.[1] Upserts in VectorStore are implemented as delete-then-insert because the vec0 virtual table does not support ON CONFLICT clauses.[1] vec0 is a SQLite virtual table extension provided by sqlite-vec that stores and indexes embedding vectors, enabling approximate nearest-neighbor cosine search.

The VectorSearchResult type expresses cosine similarity as score = 1.0 − cosine_distance, so higher scores indicate greater similarity.[1] bm25RankToScore() converts a BM25 rank (negative = more relevant) to a 0–1 score using relevance / (1 + relevance) where relevance = -rank for negative ranks, or 1 / (1 + rank) for non-negative ranks.[1]

buildFtsQuery() builds an FTS5 MATCH query string by segmenting text with jieba's cutForSearch mode (when @node-rs/jieba is available) or falling back to Unicode-regex splitting (/[\p{L}\p{N}_]+/gu). Tokens are OR-joined as quoted FTS5 phrase terms for maximum recall, with BM25 ranking preserving precision.[1] buildFtsQuery() filters a small set of Chinese stop-words (e.g., 的、了、在、是) from FTS5 query tokens to reduce noise; the list is intentionally limited to high-frequency function words only.[1] On the write side, tokenizeForFts() uses jieba cutForSearch to index both full words and sub-word components — for example, "人工智能" is indexed as "人工智能人工智能" — ensuring query-side tokens always find a match. If jieba is unavailable, the original unmodified text is stored.[1] Jieba is lazy-loaded as a singleton on the first call to buildFtsQuery; if @node-rs/jieba is unavailable, _jieba is set to null and the Unicode-regex fallback path is used permanently without retrying.[1] Example buildFtsQuery output: with jieba, "用户喜欢编程和TypeScript" produces '"用户" OR "喜欢" OR "编程" OR "TypeScript"'; without jieba, "旅行计划 API" produces '"旅行计划" OR "API"'.[1]

The L1QueryFilter interface supports narrowing L1 record queries by sessionKey (conversation channel), sessionId (single conversation instance), and updatedAfter (ISO 8601 UTC timestamp for incremental sync).[1]

src/core/store/search-utils.ts provides rrfMerge(), a shared Reciprocal Rank Fusion helper used across the SQLite hybrid-search code paths (auto-recall, memory-search, and conversation-search), eliminating duplication between them.[2] rrfMerge() uses the standard RRF constant k = 60 from the original RRF paper by default; each item's score is 1 / (k + rank + 1), summed across all lists, and results are returned sorted by descending rrfScore.[2] rrfMerge() is generic: it accepts an getId callback to extract a string key from each item, allowing it to operate on any result type, with items appearing in multiple ranked lists accumulating their scores.[2]

Canonical usage of rrfMerge() from src/core/store/search-utils.ts to merge FTS and vector search results:

const merged = rrfMerge(
  [ftsResults, vecResults],
  (item) => item.record_id,
);

[2]

_resetJiebaForTest() and _setJiebaForTest() are exported from src/core/store/sqlite.ts solely for testing: the former resets the jieba singleton so the next buildFtsQuery call re-initialises it; the latter injects a mock instance or forces the Unicode-regex fallback path.[1]

Sources

7.2TCVDB backend

Updated 2026-08-22

TcvdbMemoryStore wraps Tencent Cloud VectorDB as a dense+sparse hybrid search backend, offloading embeddings server-side and combining dense vectors with client-side BM25 sparse encoding for ranked retrieval. The implementation is a thin HTTP client layer (TcvdbClient) that authenticates via bearer tokens, normalizes errors, handles retries, and exposes low-level API operations that TcvdbMemoryStore adapts into the IMemoryStore interface with fault tolerance and scalar filtering. RRFRerank (Reciprocal Rank Fusion) is a score-fusion algorithm that merges ranked lists from dense and sparse retrievers into a single ranked result without requiring score normalization.

TcvdbMemoryStore in src/core/store/tcvdb.ts implements IMemoryStore using Tencent Cloud VectorDB as the storage backend, supporting server-side dense embedding, client-side BM25 sparse vectors, native hybrid search (dense + sparse + RRFRerank), scalar filter expressions, and time fields stored as uint64 epoch milliseconds.[1] All methods on TcvdbMemoryStore are fault-tolerant: they return empty values or false on error and never throw exceptions to callers.[1]

TcvdbMemoryStoreConfig requires url, username, apiKey, database, embeddingModel, and timeout; optional fields include caPemPath (path to a CA certificate PEM file for HTTPS), logger, and bm25Encoder.[1] TcvdbClientConfig.username defaults semantically to "root" and TcvdbClientConfig.timeout defaults to 10000 ms.[2] TcvdbClient strips trailing slashes from the url config field when constructing baseUrl.[2]

TcvdbClient in src/core/store/tcvdb-client.ts is a thin HTTP wrapper around the Tencent Cloud VectorDB API, handling authentication, timeouts, retries, and error normalization.[2] TcvdbClient constructs its Authorization header as Bearer account=<username>&api_key=<apiKey>, combining the username and API key in a single header.[2] TcvdbApiError exposes the raw VectorDB API error code via the apiCode readonly property, allowing callers to branch on specific API error codes.[2] TcvdbClient.createDatabase() is idempotent: it lists existing databases and skips creation if the target database already exists.[2] TcvdbClient.upsert() always sends buildIndex: true to the /document/upsert endpoint.[2] TcvdbClient.search(), hybridSearch(), and query() all use readConsistency: "strongConsistency" in their API requests, eliminating read-after-write inconsistency.[2][3]

TcvdbMemoryStore.init() always returns { needsReindex: false } because embedding is managed server-side by TCVDB; re-indexing is never required from the client.[1] The VectorDB /document/query API page size is capped at 100 documents (QUERY_PAGE_SIZE = 100).[1]

L1 output fields returned by query/search are: id, text, type, priority, scene_name, session_key, session_id, timestamp_str, timestamp_start, timestamp_end, metadata_json, created_time_ms, and updated_time_ms; vector and sparse vector fields are excluded.[1] L0 output fields returned by query/search are: id, message_text, agent_id, session_key, session_id, role, recorded_at_ms, and timestamp.[1] The extractAgentId helper in src/core/store/tcvdb.ts parses an agent ID from session keys in the format agent:<agentId>:<channel>, returning an empty string if the format does not match.[1]

BM25LocalEncoder in src/core/store/bm25-local.ts is a pure TypeScript replacement for the old Python sidecar BM25 client, using the @tencentdb-agent-memory/tcvdb-text package (jieba-wasm) for tokenization and BM25 encoding.[4][3] BM25LocalConfig has two fields: enabled: boolean (whether sparse encoding is active) and optional language?: "zh" | "en" (pre-trained BM25 params language, default "zh").[4] BM25LocalEncoder constructor defaults language to "zh" when not specified, using BM25Encoder.default(language) from the @tencentdb-agent-memory/tcvdb-text package.[4] createBM25Encoder(config, logger?) returns undefined when config.enabled is false; callers must check for undefined before using the encoder.[4] BM25 (Best Match 25) is a term-frequency/inverse-document-frequency ranking algorithm that produces sparse vectors — non-zero weights only for terms present in a given text, with most dimensions remaining zero — enabling efficient keyword-based similarity scoring.

Sources

7.3Embedding services

Updated 2026-08-22

Embedding services in Agent Memory are pluggable providers (remote OpenAI-compatible APIs, a local offline model via node-llama-cpp, or disabled entirely) configured through EmbeddingConfig; local and remote providers have different initialization, timeout, and input-length behaviors. The local embedding provider (embeddinggemma-300m) has a four-state lifecycle and outputs sanitized L2-normalized vectors, while remote providers are stateless; both must be queried before falling back to keyword-only search.

Vector search (embedding) is disabled by default — EmbeddingConfig.provider defaults to "none". Setting it to any other value (e.g. "openai", "deepseek") treats the target as an OpenAI-compatible remote provider; "zeroentropy" routes through ZeroEntropy's native /v1/models/embed protocol; "qclaw" forwards requests through a local proxy specified by proxyUrl.[1][2] src/core/store/embedding.ts defines two concrete embedding providers: "openai" (OpenAI-compatible HTTP APIs, covering OpenAI, Azure OpenAI, self-hosted endpoints, and the qclaw proxy) and "local" (fully offline, via node-llama-cpp). When no remote embedding is configured, the system automatically falls back to the local provider.[3]

The OpenAIEmbeddingConfig interface in src/core/store/embedding.ts requires baseUrl, apiKey, model, and dimensions to be explicitly provided by the caller — there are no defaults for these fields.[3] EmbeddingConfig.sendDimensions defaults to true, which includes a dimensions field in the request body for OpenAI text-embedding-3-* Matryoshka truncation. Fixed-dimension models such as BGE-M3 reject this field with HTTP 400 ('does not support matryoshka representation'); set sendDimensions: false to omit it.[2][4] When embedding.provider is "qclaw", embedding.proxyUrl is required; embedding requests are forwarded through the proxy with the original baseUrl passed as a Remote-URL header.[3][2] EmbeddingConfig.maxInputChars (default 5000) truncates input text with a warning before it is sent to the API. Single API calls time out after embedding.timeoutMs ms (default 10000 ms) and auto-retry up to 3 times.[1][2] embedding.recallTimeoutMs and embedding.captureTimeoutMs override embedding.timeoutMs per code path. Because the user is waiting during recall, a shorter timeout (e.g. 3000 ms) is recommended there; background capture can safely use a longer value (e.g. 15000 ms). Both fall back to timeoutMs when not set.[1][2] EmbeddingConfig.configError is an internal field: when set, it carries an error message about invalid remote configuration and disables embedding. The field is not exposed in the plugin schema.[1] Matryoshka representation is an embedding technique in which a model is trained so that any prefix of its output vector is itself a valid, lower-dimensional embedding; only models explicitly trained this way support the dimensions field in API requests.

The local embedding provider defaults to Google's embeddinggemma-300m model (quantized Q8_0, ~300 MB), downloaded from HuggingFace via node-llama-cpp. LocalEmbeddingService outputs 768-dimensional vectors (LOCAL_DIMENSIONS = 768).[3] Local input is capped at 512 characters (LOCAL_MAX_INPUT_CHARS = 512) — a conservative universal limit driven by embeddinggemma-300m's 256-token context window. CJK text tokenizes at 1–2 tokens per character, so 600 chars risks overflow; Latin text is safe to approximately 800 chars.[3] All vectors produced by LocalEmbeddingService are sanitized (NaN/Inf values replaced with 0) and L2-normalized via sanitizeAndNormalize() before being returned — matching the behavior of OpenClaw's own sanitizeAndNormalizeEmbedding().[3]

LocalEmbeddingService has a four-state lifecycle: "idle" (not started), "initializing" (download/load in progress), "ready" (model loaded), and "failed" (initialization failed, retryable via startWarmup()).[3] startWarmup() is idempotent: calling it when state is "initializing" or "ready" is a no-op; calling it when state is "failed" re-triggers initialization. For remote (OpenAI) providers, startWarmup() is always a no-op.[3] The EmbeddingService interface defines isReady() as always returning true for remote providers (stateless HTTP) and returning true for the local provider only after model download and load complete.[3] EmbeddingNotReadyError is thrown by embed() and embedBatch() when the local model has not finished loading. Callers are expected to catch it and fall back to keyword-only mode.[3] LocalEmbeddingService.embedBatch() processes texts sequentially in a for...of loop (not in parallel) and returns an empty array immediately when passed an empty list.[3] LocalEmbeddingService.close() calls dispose() on the embedding context (if present), nulls state, and resets to "idle". The method is idempotent and safe to call multiple times.[3]

The ImportLlamaFn type in src/core/store/embedding.ts is exported and overridable via the LocalEmbeddingService constructor, enabling injection of a mock for unit testing without loading node-llama-cpp.[3]

Sources

Agent Memory

Agent Memory

1Introduction

2Setup & distribution

2.1Package & distribution

2.2Installation & deployment

2.3Upgrading

3Integrations & hosts

3.1OpenClaw integration

3.2Hermes & Gateway hosts

4Memory architecture

4.1Architecture & memory layers

4.2Context Offload

4.3Seed & CLI

5Features & tools

5.1Configuration

5.2Search tools

5.3LLM thinking suppression

6TdaiCore & host abstraction

6.1Recall & context injection

6.2L0 capture

6.3L1 extraction & dedup

6.4L2 scene extraction

6.5L3 persona generation

6.6Pipeline scheduling

7Storage abstraction

7.1SQLite backend

7.2TCVDB backend

7.3Embedding services

8Benchmarks & results

9Bug fixes