diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..492e716 --- /dev/null +++ b/.gitignore @@ -0,0 +1,59 @@ +# TrueRecall v2 - .gitignore +# Files that should not be committed to version control + +# Backup files +*.bak +*.bak.* +*~ +*.swp +*.swo +*.orig +*.save + +# Debug and temporary files +debug_* +test_* +temp_* +*.tmp + +# Log files +*.log +logs/ +log/ + +# Python cache +__pycache__/ +*.py[cod] +*$py.class +*.so +.Python + +# Virtual environments +venv/ +env/ +ENV/ + +# IDE/editor files +.vscode/ +.idea/ +*.iml + +# OS files +.DS_Store +Thumbs.db + +# Environment/config with secrets (if they exist locally) +.env +.env.local +config.local.json + +# Local data that shouldn't be shared +data/ +datasets/ +*.db +*.sqlite + +# Build artifacts +build/ +dist/ +*.egg-info/ diff --git a/README.md.bak.2026-02-24-1518 b/README.md.bak.2026-02-24-1518 deleted file mode 100644 index a3ace43..0000000 --- a/README.md.bak.2026-02-24-1518 +++ /dev/null @@ -1,308 +0,0 @@ -# TrueRecall v2 - -**Project:** Gem extraction and memory recall system -**Status:** ✅ Active -**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/` - ---- - -## Overview - -TrueRecall extracts "gems" (key insights) from conversations and stores them for context injection. It's the memory curation system that powers Kimi's contextual awareness. - -### Current Flow - -``` -1. Conversation happens → Real-time watcher → memories_tr (Qdrant) -2. Daily (2:45 AM) → Curator reads memories_tr → extracts gems -3. Gems stored → gems_tr collection (Qdrant) -4. On each turn → memory-qdrant plugin injects gems as context -``` - -**Verified:** 2026-02-24 — Real-time watcher capturing (12,223 → 12,228 points) - ---- - -## Architecture - -### Collections (Qdrant) - -| Collection | Purpose | Content | Status | -|------------|---------|---------|--------| -| `memories_tr` | Full text storage | Every conversation turn (migrated from `kimi_memories`) | ✅ Active | -| `gems_tr` | Gems (extracted insights) | Curated key points (for injection) | ✅ Active | -| `true_recall` | Legacy gems | Archive of previously extracted gems | 📦 Preserved | -| `kimi_memories` | Original collection | Backup (12,223 points preserved) | 📦 Preserved | - -### Migration Script - -| File | Purpose | -|------|---------| -| `migrate_memories.py` | Migrate data from `kimi_memories` → `memories_tr` with cleaning | - -**Usage:** -```bash -python3 migrate_memories.py -``` - -**What it does:** -- Reads all points from `kimi_memories` -- Cleans content (removes metadata, thinking tags) -- Stores to `memories_tr` -- Preserves original `kimi_memories` - ---- - -### Components - -| Component | Location | Purpose | -|-----------|----------|---------| -| **memory-qdrant plugin** | `/root/.openclaw/extensions/memory-qdrant/` | Injects gems as context | -| **Curation script** | `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` | Extracts gems | -| **Curator prompt** | `/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md` | Instructions for gem extraction | - ---- - -## File Locations - -### Core Files - -``` -/root/.openclaw/workspace/.projects/true-recall-v2/ -├── README.md # This file -├── session.md # Development notes -├── curator-prompt.md # Gem extraction prompt -│ -├── tr-daily/ # Daily curation -│ └── curate_from_qdrant.py # Main curator script -│ -├── tr-compact/ # (Reserved for v2 expansion) -│ └── hook.py # Compaction hook (not active) -│ -├── tr-worker/ # (Reserved for v2 expansion) -│ └── worker.py # Background worker (not active) -│ -└── shared/ # Shared resources -``` - -### Plugin Files (Located in OpenClaw Extensions) - -> **Note:** These files live in `/root/.openclaw/extensions/`, not in this project folder. Documented here for reference. - -| Actual Location | Project Reference | -|----------------|-------------------| -| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code (capture + injection) | -| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema | -| `/root/.openclaw/openclaw.json` | Plugin configuration | - -### Configuration - -| File | Purpose | -|------|---------| -| `/root/.openclaw/openclaw.json` | Plugin config (collectionName: gems_tr) | -| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code | -| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema | - -### Cron Jobs - -| Time | Command | Purpose | -|------|---------|---------| -| 2:45 AM | `curate_memories.py` | Daily gem extraction (v1) → stores to gems_tr | -| 2:50 AM | `archive_to_memories_tr.py` | Archive to memories_tr | - -View cron: `crontab -l` - ---- - -## Process Flow - -### Step 1: Capture (Real-Time Watcher) - -``` -OpenClaw Session ──→ Real-Time Watcher ──→ Qdrant (memories_tr) -``` - -**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` - -**What it does:** -- Watches session JSONL files in real-time -- Parses each conversation turn (user + AI) -- Embeds with `snowflake-arctic-embed2` via Ollama -- Stores directly to `memories_tr` collection -- Cleans content (removes metadata, thinking tags) - -**Systemd Service:** `mem-qdrant-watcher.service` - -**Status:** ✅ Created, ready to deploy - -### Step 2: Curation (Daily) - -``` -2:45 AM → curate_memories.py → memories_tr → qwen3 → gems_tr -``` - -**Location:** `/root/.openclaw/workspace/.projects/true-recall-v1/tr-process/curate_memories.py` - -**What it does:** -1. Reads from Redis buffer (`mem:rob`) -2. Passes to qwen3 (curator model) -3. Extracts gems using prompt -4. Stores gems to `gems_tr` collection - -### Step 3: Injection - -``` -User message → memory-qdrant plugin → Search gems_tr → Inject as context -``` - -**Location:** `/root/.openclaw/extensions/memory-qdrant/index.ts` - -**What it does:** -1. Listens to `before_agent_start` events -2. Embeds current prompt -3. Searches `gems_tr` collection -4. Returns `{ prependContext: "..." }` with gems -5. Gems appear in my context (hidden from UI) - ---- - -## Configuration - -### memory-qdrant Plugin - -```json -{ - "autoCapture": true, - "autoRecall": true, - "collectionName": "gems_tr", - "embeddingModel": "snowflake-arctic-embed2", - "maxRecallResults": 2, - "minRecallScore": 0.7 -} -``` - -**Key setting:** `collectionName: "gems_tr"` — tells plugin to inject gems, not full text. - ---- - -## Gem Format - -```json -{ - "gem": "User prefers dark mode for interface", - "context": "Discussed UI theme options", - "snippet": "rob: I want dark mode\nKimi: Done", - "categories": ["preference"], - "importance": "high", - "confidence": 0.95, - "date": "2026-02-24", - "conversation_id": "uuid", - "turn_range": "5-7" -} -``` - ---- - -## Validation Commands - -### Check Qdrant Collections - -```bash -curl -s "http://10.0.0.40:6333/collections" | python3 -m json.tool -``` - -### Check Collection Points - -```bash -# memories_tr (full text) -curl -s -X POST "http://10.0.0.40:6333/collections/memories_tr/points/scroll" \ - -H "Content-Type: application/json" \ - -d '{"limit": 5, "with_payload": true}' | python3 -m json.tool - -# gems_tr (gems) -curl -s -X POST "http://10.0.0.40:6333/collections/gems_tr/points/scroll" \ - -H "Content-Type: application/json" \ - -d '{"limit": 5, "with_payload": true}' | python3 -m json.tool -``` - -### Check Cron Jobs - -```bash -crontab -l | grep -E "true|recall|curate" -``` - -### Check Plugin Logs - -```bash -tail -50 /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant -``` - ---- - -## Troubleshooting - -### Issue: `` showing in UI - -**Cause:** Plugin was using `memories_tr` (full text) instead of `gems_tr` (gems) - -**Fix:** Set `"collectionName": "gems_tr"` in openclaw.json - -**Reference:** KB entry `relevant-memories-ui-issue.md` - -### Issue: No gems being extracted - -**Check:** -1. Is memories_tr populated? `curl .../collections/memories_tr/points/scroll` -2. Is curator prompt valid? `cat curator-prompt.md` -3. Is qwen3 available? `curl -s http://10.0.0.10:11434/api/tags` - ---- - -## Related Projects - -| Project | Location | Purpose | -|---------|----------|---------| -| **true-recall-v1** | `/.projects/true-recall-v1/` | Original (Redis-based) | -| **memory-qdrant** | `/root/.openclaw/extensions/memory-qdrant/` | Plugin | -| **mem-redis** | `/root/.openclaw/workspace/skills/mem-redis/` | Redis utilities | - ---- - -## Backups - -| File | Backup Location | -|------|-----------------| -| openclaw.json | `/root/.openclaw/openclaw.json.bak.2026-02-24` | -| Plugin | `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24` | - ---- - -## Final Status (2026-02-24 15:08) - -| Component | Status | -|-----------|--------| -| Collection `kimi_memories` → `memories_tr` | ✅ Migrated (12,223 points) | -| Real-time watcher | ✅ **Deployed & verified** (12,228 points, +5 new) | -| Collection `gems_tr` | ✅ Active (5 gems stored) | -| Curator v2 | ✅ Working - tested with 327 turns | -| Config | ✅ Updated to `gems_tr` | -| Cron jobs | ✅ Cleaned | -| Documentation | ✅ Updated | - -### Daily Schedule (Simplified) - -| Time | Job | Flow | -|------|-----|------| -| Continuous | autoCapture | Conversation → `memories_tr` | -| 2:45 AM | Curator v2 | `memories_tr` → `gems_tr` | -| Each turn | Injection | `gems_tr` → Context | - -**Redis usage:** Disabled for memory. Used only for `delayed:notifications` queue. - -**Auto-Capture Status:** ✅ **Working** - Real-time watcher deployed and verified - ---- - -**Last Updated:** 2026-02-24 -**Project Lead:** Rob -**AI Assistant:** Kimi 🎙️ diff --git a/README.md.bak.2026-02-24-1518-final b/README.md.bak.2026-02-24-1518-final deleted file mode 100644 index c335a92..0000000 --- a/README.md.bak.2026-02-24-1518-final +++ /dev/null @@ -1,325 +0,0 @@ -# TrueRecall v2 - -**Project:** Gem extraction and memory recall system -**Status:** ✅ Active -**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/` - ---- - -## Table of Contents - -- [Overview](#overview) -- [Architecture](#architecture) -- [Collections](#collections-qdrant) -- [Components](#components) -- [Process Flow](#process-flow) -- [Configuration](#configuration) -- [Validation Commands](#validation-commands) -- [Troubleshooting](#troubleshooting) -- [Related Projects](#related-projects) -- [Backups](#backups) -- [Final Status](#final-status-2026-02-24-1508) - ---- - -## Overview - -TrueRecall extracts "gems" (key insights) from conversations and stores them for context injection. It's the memory curation system that powers Kimi's contextual awareness. - -### Current Flow - -``` -1. Conversation happens → Real-time watcher → memories_tr (Qdrant) -2. Daily (2:45 AM) → Curator reads memories_tr → extracts gems -3. Gems stored → gems_tr collection (Qdrant) -4. On each turn → memory-qdrant plugin injects gems as context -``` - -**Verified:** 2026-02-24 — Real-time watcher capturing (12,223 → 12,228 points) - ---- - -## Architecture - -### Collections (Qdrant) - -| Collection | Purpose | Content | Status | -|------------|---------|---------|--------| -| `memories_tr` | Full text storage | Every conversation turn (migrated from `kimi_memories`) | ✅ Active | -| `gems_tr` | Gems (extracted insights) | Curated key points (for injection) | ✅ Active | -| `true_recall` | Legacy gems | Archive of previously extracted gems | 📦 Preserved | -| `kimi_memories` | Original collection | Backup (12,223 points preserved) | 📦 Preserved | - -### Migration Script - -| File | Purpose | -|------|---------| -| `migrate_memories.py` | Migrate data from `kimi_memories` → `memories_tr` with cleaning | - -**Usage:** -```bash -python3 migrate_memories.py -``` - -**What it does:** -- Reads all points from `kimi_memories` -- Cleans content (removes metadata, thinking tags) -- Stores to `memories_tr` -- Preserves original `kimi_memories` - ---- - -### Components - -| Component | Location | Purpose | -|-----------|----------|---------| -| **memory-qdrant plugin** | `/root/.openclaw/extensions/memory-qdrant/` | Injects gems as context | -| **Curation script** | `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` | Extracts gems | -| **Curator prompt** | `/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md` | Instructions for gem extraction | - ---- - -## File Locations - -### Core Files - -``` -/root/.openclaw/workspace/.projects/true-recall-v2/ -├── README.md # This file -├── session.md # Development notes -├── curator-prompt.md # Gem extraction prompt -│ -├── tr-daily/ # Daily curation -│ └── curate_from_qdrant.py # Main curator script -│ -├── tr-compact/ # (Reserved for v2 expansion) -│ └── hook.py # Compaction hook (not active) -│ -├── tr-worker/ # (Reserved for v2 expansion) -│ └── worker.py # Background worker (not active) -│ -└── shared/ # Shared resources -``` - -### Plugin Files (Located in OpenClaw Extensions) - -> **Note:** These files live in `/root/.openclaw/extensions/`, not in this project folder. Documented here for reference. - -| Actual Location | Project Reference | -|----------------|-------------------| -| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code (capture + injection) | -| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema | -| `/root/.openclaw/openclaw.json` | Plugin configuration | - -### Configuration - -| File | Purpose | -|------|---------| -| `/root/.openclaw/openclaw.json` | Plugin config (collectionName: gems_tr) | -| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code | -| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema | - -### Cron Jobs - -| Time | Command | Purpose | -|------|---------|---------| -| 2:45 AM | `curate_memories.py` | Daily gem extraction (v1) → stores to gems_tr | -| 2:50 AM | `archive_to_memories_tr.py` | Archive to memories_tr | - -View cron: `crontab -l` - ---- - -## Process Flow - -### Step 1: Capture (Real-Time Watcher) - -``` -OpenClaw Session ──→ Real-Time Watcher ──→ Qdrant (memories_tr) -``` - -**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` - -**What it does:** -- Watches session JSONL files in real-time -- Parses each conversation turn (user + AI) -- Embeds with `snowflake-arctic-embed2` via Ollama -- Stores directly to `memories_tr` collection -- Cleans content (removes metadata, thinking tags) - -**Systemd Service:** `mem-qdrant-watcher.service` - -**Status:** ✅ Created, ready to deploy - -### Step 2: Curation (Daily) - -``` -2:45 AM → curate_memories.py → memories_tr → qwen3 → gems_tr -``` - -**Location:** `/root/.openclaw/workspace/.projects/true-recall-v1/tr-process/curate_memories.py` - -**What it does:** -1. Reads from Redis buffer (`mem:rob`) -2. Passes to qwen3 (curator model) -3. Extracts gems using prompt -4. Stores gems to `gems_tr` collection - -### Step 3: Injection - -``` -User message → memory-qdrant plugin → Search gems_tr → Inject as context -``` - -**Location:** `/root/.openclaw/extensions/memory-qdrant/index.ts` - -**What it does:** -1. Listens to `before_agent_start` events -2. Embeds current prompt -3. Searches `gems_tr` collection -4. Returns `{ prependContext: "..." }` with gems -5. Gems appear in my context (hidden from UI) - ---- - -## Configuration - -### memory-qdrant Plugin - -```json -{ - "autoCapture": true, - "autoRecall": true, - "collectionName": "gems_tr", - "embeddingModel": "snowflake-arctic-embed2", - "maxRecallResults": 2, - "minRecallScore": 0.7 -} -``` - -**Key setting:** `collectionName: "gems_tr"` — tells plugin to inject gems, not full text. - ---- - -## Gem Format - -```json -{ - "gem": "User prefers dark mode for interface", - "context": "Discussed UI theme options", - "snippet": "rob: I want dark mode\nKimi: Done", - "categories": ["preference"], - "importance": "high", - "confidence": 0.95, - "date": "2026-02-24", - "conversation_id": "uuid", - "turn_range": "5-7" -} -``` - ---- - -## Validation Commands - -### Check Qdrant Collections - -```bash -curl -s "http://10.0.0.40:6333/collections" | python3 -m json.tool -``` - -### Check Collection Points - -```bash -# memories_tr (full text) -curl -s -X POST "http://10.0.0.40:6333/collections/memories_tr/points/scroll" \ - -H "Content-Type: application/json" \ - -d '{"limit": 5, "with_payload": true}' | python3 -m json.tool - -# gems_tr (gems) -curl -s -X POST "http://10.0.0.40:6333/collections/gems_tr/points/scroll" \ - -H "Content-Type: application/json" \ - -d '{"limit": 5, "with_payload": true}' | python3 -m json.tool -``` - -### Check Cron Jobs - -```bash -crontab -l | grep -E "true|recall|curate" -``` - -### Check Plugin Logs - -```bash -tail -50 /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant -``` - ---- - -## Troubleshooting - -### Issue: `` showing in UI - -**Cause:** Plugin was using `memories_tr` (full text) instead of `gems_tr` (gems) - -**Fix:** Set `"collectionName": "gems_tr"` in openclaw.json - -**Reference:** KB entry `relevant-memories-ui-issue.md` - -### Issue: No gems being extracted - -**Check:** -1. Is memories_tr populated? `curl .../collections/memories_tr/points/scroll` -2. Is curator prompt valid? `cat curator-prompt.md` -3. Is qwen3 available? `curl -s http://10.0.0.10:11434/api/tags` - ---- - -## Related Projects - -| Project | Location | Purpose | -|---------|----------|---------| -| **true-recall-v1** | `/.projects/true-recall-v1/` | Original (Redis-based) | -| **memory-qdrant** | `/root/.openclaw/extensions/memory-qdrant/` | Plugin | -| **mem-redis** | `/root/.openclaw/workspace/skills/mem-redis/` | Redis utilities | - ---- - -## Backups - -| File | Backup Location | -|------|-----------------| -| openclaw.json | `/root/.openclaw/openclaw.json.bak.2026-02-24` | -| Plugin | `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24` | - ---- - -## Final Status (2026-02-24 15:08) - -| Component | Status | -|-----------|--------| -| Collection `kimi_memories` → `memories_tr` | ✅ Migrated (12,223 points) | -| Real-time watcher | ✅ **Deployed & verified** (12,228 points, +5 new) | -| Collection `gems_tr` | ✅ Active (5 gems stored) | -| Curator v2 | ✅ Working - tested with 327 turns | -| Config | ✅ Updated to `gems_tr` | -| Cron jobs | ✅ Cleaned | -| Documentation | ✅ Updated | - -### Daily Schedule (Simplified) - -| Time | Job | Flow | -|------|-----|------| -| Continuous | autoCapture | Conversation → `memories_tr` | -| 2:45 AM | Curator v2 | `memories_tr` → `gems_tr` | -| Each turn | Injection | `gems_tr` → Context | - -**Redis usage:** Disabled for memory. Used only for `delayed:notifications` queue. - -**Auto-Capture Status:** ✅ **Working** - Real-time watcher deployed and verified - ---- - -**Last Updated:** 2026-02-24 -**Project Lead:** Rob -**AI Assistant:** Kimi 🎙️ - diff --git a/README.md.neuralstream.bak b/README.md.neuralstream.bak deleted file mode 100644 index 6ff992a..0000000 --- a/README.md.neuralstream.bak +++ /dev/null @@ -1,183 +0,0 @@ -# NeuralStream - -**Neural streaming memory for OpenClaw with gem-based context injection.** - -## Overview - -NeuralStream extracts high-value insights ("gems") from conversation batches using qwen3, stores them in Qdrant, and injects relevant gems into context on each new turn. This creates **infinite effective context** — the active window stays small, but semantically relevant gems from all past conversations are always retrievable. - -## Core Concept - -| Traditional Memory | NeuralStream | -|-------------------|--------------| -| Context lost on `/new` | Gems persist in Qdrant | -| Full history or generic summary | Semantic gem retrieval | -| Static context window | Dynamic injection | -| Survives compaction only | Survives session reset | -| **Limited context** | **Infinite effective context** | - -## How It Works - -### Capture → Extract → Store → Retrieve - -1. **Capture:** Every turn buffered to Redis (reuses mem-redis-watcher) -2. **Extract:** Batch of 5 turns → qwen3 (with 256k context) extracts structured gems -3. **Store:** Gems embedded + stored in Qdrant `neuralstream` -4. **Retrieve:** Each new turn → semantic search → inject top-10 gems - -### Hybrid Triggers (Three-way) - -| Trigger | Condition | Purpose | -|---------|-----------|---------| -| Batch | Every 5 turns | Normal extraction | -| Context | 50% usage (`ctx.getContextUsage()`) | Proactive pre-compaction | -| Timer | 15 min idle | Safety net | - -**Context Awareness:** qwen3 receives up to 256k tokens of history for understanding, but only extracts gems from the last N turns (avoiding current context). - -All gems survive `/new`, `/reset`, and compaction via Qdrant persistence. - -## Architecture - -NeuralStream is the **middle layer** — extraction intelligence on top of existing infrastructure: - -``` -┌─────────────────────────────────────────────────────────┐ -│ EXISTING: mem-redis-watcher │ -│ Every turn → Redis buffer │ -└──────────────────┬──────────────────────────────────────┘ - │ - ┌──────────▼──────────┐ - │ NeuralStream │ - │ - Batch reader │ - │ - Gem extractor │ - │ - Qdrant store │ - └──────────┬──────────┘ - │ - ┌──────────▼──────────┐ - │ EXISTING: │ - │ qdrant-memory │ - │ Semantic search │ - │ Context injection │ - └─────────────────────┘ -``` - -## Technical Reference - -### Native Context Monitoring - -```typescript -// In turn_end hook -const usage = ctx.getContextUsage(); -// usage.tokens, usage.contextWindow, usage.percent -// Trigger extraction when usage.percent >= threshold -``` - -### Primary Hook: turn_end - -```typescript -pi.on("turn_end", async (event, ctx) => { - const { turnIndex, message, toolResults } = event; - - // Buffer turn to Redis - // Check ctx.getContextUsage().percent - // If batch >= 5 OR percent >= 50%: extract -}); -``` - -### Timer Fallback - -```bash -# Cron every 10 min -# Check neuralstream:buffer age > 15 min -# If yes: extract from partial batch -``` - -### Context-Aware Extraction - -- Feed qwen3: Up to 256k tokens (full history for context) -- Extract from: Last `batch_size` turns only -- Benefit: Rich understanding without gemming current context - -## Gem Format - -```json -{ - "gem_id": "uuid", - "content": "Distilled insight/fact/decision", - "summary": "One-line for quick scanning", - "topics": ["docker", "redis", "architecture"], - "importance": 0.9, - "source": { - "session_id": "uuid", - "date": "2026-02-23", - "turn_range": "15-20" - }, - "tags": ["decision", "fact", "preference", "todo", "code"], - "created_at": "2026-02-23T15:26:00Z" -} -``` - -## Configuration (All Tunable) - -| Setting | Default | Description | -|---------|---------|-------------| -| batch_size | 5 | Turns per extraction | -| context_threshold | 50% | Token % trigger (40-80% range) | -| idle_timeout | 15 min | Timer trigger threshold | -| gem_model | qwen3 | Extraction LLM (256k context) | -| max_gems_injected | 10 | Per-turn limit | -| embedding | snowflake-arctic-embed2 | Same as kimi_memories | -| collection | neuralstream | Qdrant (1024 dims, Cosine) | - -## Qdrant Schema - -**Collection:** `neuralstream` -- Vector size: 1024 -- Distance: Cosine -- On-disk payload: true - -## Project Structure - -``` -.projects/neuralstream/ -├── README.md # This file -├── session.md # Development log & state -├── prompt.md # (TBD) qwen3 extraction prompt -└── src/ # (TBD) Implementation - ├── extract.ts # Gem extraction logic - ├── store.ts # Qdrant storage - └── inject.ts # Context injection -``` - -## Status - -- [x] Architecture defined (v2.2 context-aware) -- [x] Native context monitoring validated (ctx.getContextUsage) -- [x] Naming finalized (NeuralStream, alias: ns) -- [x] Hook research completed -- [x] Qdrant collection created (`neuralstream`) -- [x] Gem format proposed -- [x] Infrastructure decision (reuse Redis/Qdrant) -- [ ] Extraction prompt design -- [ ] Implementation -- [ ] Testing - -## Backups - -- Local: `/root/.openclaw/workspace/.projects/neuralstream/` -- Remote: `deb2:/root/.projects/neuralstream/` (build/test only) -- kimi_kb: Research entries stored - -## Related Projects - -- **True Recall:** Gem extraction inspiration -- **OpenClaw:** Host platform -- **kimi_memories:** Shared Qdrant infrastructure -- **mem-redis-watcher:** Existing capture layer - ---- - -**Created:** 2026-02-23 -**Alias:** ns -**Purpose:** Infinite context for LLMs diff --git a/debug_curator.py b/debug_curator.py deleted file mode 100644 index e6b757a..0000000 --- a/debug_curator.py +++ /dev/null @@ -1,106 +0,0 @@ -#!/usr/bin/env python3 -"""Debug curator with real data""" -import json -import requests -import urllib.request - -QDRANT_URL = "http://10.0.0.40:6333" -SOURCE_COLLECTION = "kimi_memories" - -# Get sample turns from real data -filter_data = { - "must": [ - {"key": "user_id", "match": {"value": "rob"}}, - {"key": "date", "match": {"value": "2026-02-23"}} - ] -} - -req = urllib.request.Request( - f"{QDRANT_URL}/collections/{SOURCE_COLLECTION}/points/scroll", - data=json.dumps({"limit": 5, "with_payload": True, "filter": filter_data}).encode(), - headers={"Content-Type": "application/json"}, - method="POST" -) - -with urllib.request.urlopen(req, timeout=30) as response: - result = json.loads(response.read().decode()) - points = result.get("result", {}).get("points", []) - - turns = [] - for point in points: - payload = point.get("payload", {}) - user_msg = payload.get("user_message", "") - ai_msg = payload.get("ai_response", "") - - if user_msg or ai_msg: - turn = { - "turn": payload.get("turn_number", 0), - "user_id": payload.get("user_id", "rob"), - "user": user_msg[:300], # Truncate - "ai": ai_msg[:300], # Truncate - "conversation_id": payload.get("conversation_id", ""), - "timestamp": payload.get("created_at", ""), - "date": payload.get("date", "2026-02-23") - } - turns.append(turn) - - turns.sort(key=lambda x: (x.get("conversation_id", ""), x.get("turn", 0))) - -print(f"Got {len(turns)} turns") -print("Sample:") -for t in turns[:2]: - print(f" User: {t['user'][:100]}...") - print(f" AI: {t['ai'][:100]}...") - -# Now test with curator -with open('/root/.openclaw/workspace/.projects/true-recall-v1/curator-prompt.md') as f: - prompt = f.read() - -conversation_json = json.dumps(turns[:5], indent=2) - -prompt_text = f"""## Input Conversation - -```json -{conversation_json} -``` - -## Output -""" - -response = requests.post( - 'http://10.0.0.10:11434/api/generate', - json={ - 'model': 'qwen3:4b-instruct', - 'system': prompt, - 'prompt': prompt_text, - 'stream': False, - 'options': {'temperature': 0.1, 'num_predict': 3000} - }, - timeout=120 -) - -result = response.json() -output = result.get('response', '').strip() - -print("\n=== CURATOR OUTPUT ===") -print(output[:3000]) -print("\n=== TRYING TO PARSE ===") - -# Try to parse -try: - if '```json' in output: - parsed = output.split('```json')[1].split('```')[0].strip() - gems = json.loads(parsed) - print(f"Parsed {len(gems)} gems") - elif '```' in output: - parsed = output.split('```')[1].split('```')[0].strip() - gems = json.loads(parsed) - print(f"Parsed {len(gems)} gems") - else: - gems = json.loads(output) - print(f"Parsed {len(gems)} gems") -except Exception as e: - print(f"Parse error: {e}") - print("Trying raw parse...") - gems = json.loads(output.strip()) - print(f"Parsed {len(gems)} gems") diff --git a/session.md.bak.2026-02-24-1518 b/session.md.bak.2026-02-24-1518 deleted file mode 100644 index b4ea42f..0000000 --- a/session.md.bak.2026-02-24-1518 +++ /dev/null @@ -1,224 +0,0 @@ -# TrueRecall v2 - Session Notes - -**Last Updated:** 2026-02-24 15:08 CST - ---- - -## Summary of Changes (2026-02-24) - -### Collection Migration - -| Action | From | To | Status | -|--------|------|----|--------| -| **Rename** | `kimi_memories` | `memories_tr` | ✅ Done + data migrated (12,223 points) | -| **Create** | — | `gems_tr` | ✅ Created (empty, ready for curation) | -| **Archive** | `true_recall` | `true_recall` | ✅ Preserved (existing data kept) | - -### Dual Collection Support (NEW - 14:10) - -**Problem:** Auto-capture was saving to `gems_tr` because that's where recall pulled from. This was wrong. - -**Solution:** Added `captureCollection` option to memory-qdrant plugin: -- `collectionName`: `gems_tr` — for recall/injection (gems only) -- `captureCollection`: `memories_tr` — for auto-capture (full conversations) - -**Files Modified:** -1. `/root/.openclaw/extensions/memory-qdrant/config.ts` — Added `captureCollection` option -2. `/root/.openclaw/extensions/memory-qdrant/index.ts` — Uses `dbRecall` and `dbCapture` separately -3. `/root/.openclaw/openclaw.json` — Added `"captureCollection": "memories_tr"` - -### Auto-Capture Architecture (Real-Time Watcher) - -**Mechanism:** `realtime_qdrant_watcher.py` daemon -- Watches OpenClaw session JSONL files in real-time -- Parses each conversation turn -- Embeds with `snowflake-arctic-embed2` via Ollama -- Stores directly to `memories_tr` collection (no Redis) -- Cleans content (removes metadata, thinking tags) - -**Files:** -- Script: `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` -- Service: `/root/.openclaw/workspace/skills/qdrant-memory/mem-qdrant-watcher.service` - -**Status:** ✅ Deployed and running - -**Deployment:** -- Stopped: `mem-redis-watcher` -- Started: `mem-qdrant-watcher` -- Status: Active (PID 84465) -- Verified: ✅ Capturing (points: 12,223 → 12,224) - -### Current State (2026-02-24 15:08) - -| Collection | Purpose | Points | Last Update | -|------------|---------|--------|-------------| -| `memories_tr` | Full text (autoCapture via watcher) | 12,228 | **Live** | -| `gems_tr` | Gems (for injection) | 5 | Feb 24 | - -**Flow:** -``` -OpenClaw Session → Real-Time Watcher → memories_tr (Qdrant) - ↓ - Daily Curator (2:45 AM) - ↓ - gems_tr (Qdrant) - ↓ - memory-qdrant plugin → Injection -``` - -### Files Modified Today - -1. `/root/.openclaw/openclaw.json` — Changed collectionName to `gems_tr` -2. `/root/.openclaw/extensions/memory-qdrant/config.ts` — Default to `memories_tr` -3. `/root/.openclaw/extensions/memory-qdrant/openclaw.plugin.json` — Defaults to `memories_tr` -4. `/root/.openclaw/workspace/skills/mem-redis/scripts/save_mem.py` — Added `--silent` flag -5. `/root/.openclaw/workspace/HEARTBEAT.md` — Updated to use `--silent` -6. `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` — TARGET_COLLECTION = `gems_tr` -7. `/root/.openclaw/workspace/SOUL.md` — Updated collection references -8. `/root/.openclaw/workspace/kb/relevant-memories-ui-issue.md` — Updated documentation - -### Migration Script Created - -- `/root/.openclaw/workspace/.projects/true-recall-v2/migrate_memories.py` -- Migrated 12,223 points from `kimi_memories` → `memories_tr` -- Cleaned noise (metadata, thinking tags) during migration -- Preserved original `kimi_memories` as backup - -### Current Collections (Qdrant) - -| Collection | Purpose | Points | -|------------|---------|--------| -| `memories_tr` | Full text (live capture) | **12,228** | -| `gems_tr` | Gems (for injection) | 5 | -| `true_recall` | Legacy gems archive | existing | -| `kimi_memories` | Original (backup) | 12,223 | - ---- - -## Final Status (2026-02-24) - -| Component | Status | -|-----------|--------| -| **Collections** | 4 active (memories_tr, gems_tr, true_recall, kimi_memories) | -| **Curator v2** | ✅ Tested & working - 327 turns → 5 gems | -| **Config** | ✅ Using `gems_tr` for injection | -| **Cron** | ✅ Simplified - only essential jobs | -| **Redis** | ✅ Only for notifications (not memory) | - -### Collection Summary - -| Collection | Points | Purpose | Status | -|------------|--------|---------|--------| -| `memories_tr` | **12,228** | Full text (live capture) | ✅ Active | -| `gems_tr` | 5 | Gems (curated) | ✅ Active | -| `true_recall` | existing | Legacy archive | 📦 Preserved | -| `kimi_memories` | 12,223 | Original backup | 📦 Preserved | - -### Cron Schedule (Cleaned) - -| Time | Job | -|------|-----| -| 2:45 AM | Curator v2: memories_tr → gems_tr | -| 2:20 AM | File backup | -| 2:00 AM | Log monitoring | - -**Removed:** 2:15 AM Redis backup, 2:50 AM archive, 5-min cron_capture (all redundant) - ---- - -## What We Did Today - -### Issue: `` showing in UI - -**Problem:** The `` block was showing full conversation text in webchat UI. It should be hidden/internal. - -**Investigation:** -1. Initially thought it was `save_mem.py` output (wrong) -2. Then thought it was HTML comment issue (wrong) -3. Found root cause: memory-qdrant plugin was recalling from `memories_tr` (full text) instead of `gems_tr` (gems) - -**Solution Applied:** -Changed `openclaw.json` config: -- Before: `"collectionName": "memories_tr"` -- After: `"collectionName": "gems_tr"` - -**Result:** ✅ Fixed! Now injection uses gems from true_recall, not full text. - -### Issue: Auto-Capture Architecture Change - -**Problem:** Real-time capture was going through Redis, we needed direct Qdrant storage - -**Solution:** -1. Created `realtime_qdrant_watcher.py` — watches session JSONL, embeds, stores directly to Qdrant -2. Created `mem-qdrant-watcher.service` — systemd service for the watcher -3. Deployed and verified: - - Points before: 12,223 - - Points after test messages: 12,228 - - Verified new captures have correct structure (role, content, date, source) - -**Status:** ✅ **Deployed and working** - -### Other Changes Made - -1. **Plugin dual collection support** — Added `captureCollection` option -2. **Session/Readme updates** — Clarified architecture - -### Files Modified - -| File | Change | -|------|--------| -| `/root/.openclaw/openclaw.json` | collectionName: memories_tr → gems_tr, added captureCollection | -| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Added captureCollection option | -| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Dual collection support, debug logging | -| `/root/.openclaw/extensions/memory-qdrant/openclaw.plugin.json` | Added captureCollection to schema | - -### Files Created (NEW) - -| File | Purpose | -|------|---------| -| `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` | Real-time watcher daemon (Qdrant direct) | -| `/root/.openclaw/workspace/skills/qdrant-memory/mem-qdrant-watcher.service` | Systemd service file | - -### Deployment - -| Service | Action | Status | -|---------|--------|--------| -| `mem-redis-watcher` | Stopped | ✅ | -| `mem-qdrant-watcher` | Started | ✅ Active | - -### Backups Created - -- `/root/.openclaw/openclaw.json.bak.2026-02-24` -- `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24` - ---- - -## Current Status - -| Component | Status | -|-----------|--------| -| **Curation (daily)** | v1 cron at 2:45 AM | -| **Injection** | ✅ Working, uses gems_tr | -| **Collection** | gems_tr | -| **KB** | Updated | - ---- - -## What Still Needs Doing - -1. ~~Test autoCapture (cleaned content to memories_tr)~~ ✅ Done -2. Test v2 curator (read from Qdrant, not Redis) — Next step -3. Full validation 2x - ---- - -## Session Recovery - -If starting new session: -1. Read this session.md -2. Read README.md for architecture -3. Read KB for issue history - ---- - -**Next:** Test v2 curator (reads from memories_tr, creates gems in gems_tr) diff --git a/session.md.neuralstream.bak b/session.md.neuralstream.bak deleted file mode 100644 index 3ad9cf5..0000000 --- a/session.md.neuralstream.bak +++ /dev/null @@ -1,150 +0,0 @@ -# NeuralStream Session State - -**Date:** 2026-02-23 -**Status:** Architecture v2.2 - Context-aware hybrid triggers -**Alias:** ns - ---- - -## Architecture v2.2 (Current) - -**Decision:** Three hybrid extraction triggers with full context awareness - -| Trigger | When | Purpose | -|---------|------|---------| -| `turn_end` (N=5) | Every 5 turns | Normal batch extraction | -| Timer (15 min idle) | No new turn for 15 min | Catch partial batches | -| Context (50% threshold) | `ctx.getContextUsage().percent >= threshold` | Proactive pre-compaction | - -**Context Awareness:** -- qwen3 gets **up to 256k tokens** of full conversation history for understanding -- Only extracts **last N turns** (oldest in batch) to avoid gemming current context -- Uses `ctx.getContextUsage()` native API for token monitoring - -**Why Hybrid:** -- Batch extraction = better quality gems (more context) -- Timer safety = never lose important turns if user walks away -- Context trigger = proactive extraction before system forces compaction -- All gems survive `/new` and `/reset` via Qdrant - -**Infrastructure:** Reuse existing Redis/Qdrant — NeuralStream is the "middle layer" only - ---- - -## Core Insight - -NeuralStream enables **infinite effective context** — active window stays small, but semantically relevant gems from all past conversations are queryable and injectable. - ---- - -## Technical Decisions 2026-02-23 - -### Triggers (Three-way Hybrid) -| Trigger | Config | Default | -|---------|--------|---------| -| Batch size | `batch_size` | 5 turns | -| Idle timeout | `idle_timeout` | 15 minutes | -| Context threshold | `context_threshold` | 50% | - -### Context Monitoring (Native API) -- `ctx.getContextUsage()` → `{tokens, contextWindow, percent}` -- Checked in `turn_end` hook -- Triggers extraction when `percent >= context_threshold` - -### Extraction Context Window -- **Feed to qwen3:** Up to 256k tokens (full history for understanding) -- **Extract from:** Last `batch_size` turns only -- **Benefit:** Rich context awareness without gemming current conversation - -### Storage -- **Buffer:** Redis (`neuralstream:buffer` key) -- **Gems:** Qdrant `neuralstream` collection (1024 dims, Cosine) -- **Existing infra:** Reuse mem-redis-watcher + qdrant-memory - -### Gem Format (Proposed) -```json -{ - "gem_id": "uuid", - "content": "Distilled insight/fact/decision", - "summary": "One-line for quick scanning", - "topics": ["docker", "redis", "architecture"], - "importance": 0.9, - "source": { - "session_id": "uuid", - "date": "2026-02-23", - "turn_range": "15-20" - }, - "tags": ["decision", "fact", "preference", "todo", "code"], - "created_at": "2026-02-23T15:26:00Z" -} -``` - -### Extraction Model -- **qwen3** for gem extraction (256k context, cheap) -- **Dedicated prompt** (to be designed) for extracting high-value items - ---- - -## Architecture Layers - -| Layer | Status | Description | -|-------|--------|-------------| -| Capture | ✅ Existing | Every turn → Redis (mem-redis-watcher) | -| **Extract** | ⏳ NeuralStream | Batch → qwen3 → gems → Qdrant | -| Retrieve | ✅ Existing | Semantic search → inject context | - -NeuralStream = Smart extraction layer on top of existing infra. - ---- - -## Open Questions - -- Gem extraction prompt design (deferred) -- Importance scoring: auto vs manual? -- Injection: `turn_start` hook or modify system prompt? -- Semantic search threshold tuning - ---- - -## Next Steps - -| Task | Status | -|------|--------| -| Architecture v2.2 finalized | ✅ | -| Native context monitoring validated | ✅ | -| Gem JSON schema | ✅ Proposed | -| Implement turn_end hook | ⏳ | -| Implement timer/cron check | ⏳ | -| Implement context trigger | ⏳ | -| Create extraction prompt | ⏳ | -| Test gem extraction with qwen3 | ⏳ | -| Implement injection mechanism | ⏳ | - ---- - -## Decisions Log - -| Date | Decision | -|------|----------| -| 2026-02-23 | Switch to turn_end hook (v2) | -| 2026-02-23 | Hybrid triggers with timer (v2.1) | -| 2026-02-23 | Context-aware extraction (v2.2) | -| 2026-02-23 | Native API: ctx.getContextUsage() | -| 2026-02-23 | Full context feed to qwen3 (256k) | -| 2026-02-23 | Reuse existing Redis/Qdrant infrastructure | -| 2026-02-23 | Batch N=5 turns | -| 2026-02-23 | Context threshold = 50% | -| 2026-02-23 | Inactivity timer = 15 min | -| 2026-02-23 | Dedicated qwen3 extraction prompt (deferred) | - ---- - -## Backups - -- Local: `/root/.openclaw/workspace/.projects/neuralstream/` -- Remote: `deb2:/root/.projects/neuralstream/` (build/test only) -- kimi_kb: Research entries stored - ---- - -**Key Insight:** Session resets wipe context but NOT Qdrant. NeuralStream = "Context insurance policy" for infinite LLM memory.