Initial commit: TrueRecall v2.2 with 30b curator and timer-based curation

2026-02-24 20:27:44 -06:00
commit 8bb1abaf18
23 changed files with 4112 additions and 0 deletions
--- a/README.md.bak.2026-02-24-1518-final
+++ b/README.md.bak.2026-02-24-1518-final
@@ -0,0 +1,325 @@
+# TrueRecall v2
+
+**Project:** Gem extraction and memory recall system  
+**Status:** ✅ Active  
+**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/`
+
+---
+
+## Table of Contents
+
+- [Overview](#overview)
+- [Architecture](#architecture)
+- [Collections](#collections-qdrant)
+- [Components](#components)
+- [Process Flow](#process-flow)
+- [Configuration](#configuration)
+- [Validation Commands](#validation-commands)
+- [Troubleshooting](#troubleshooting)
+- [Related Projects](#related-projects)
+- [Backups](#backups)
+- [Final Status](#final-status-2026-02-24-1508)
+
+---
+
+## Overview
+
+TrueRecall extracts "gems" (key insights) from conversations and stores them for context injection. It's the memory curation system that powers Kimi's contextual awareness.
+
+### Current Flow
+
+```
+1. Conversation happens → Real-time watcher → memories_tr (Qdrant)
+2. Daily (2:45 AM) → Curator reads memories_tr → extracts gems
+3. Gems stored → gems_tr collection (Qdrant)
+4. On each turn → memory-qdrant plugin injects gems as context
+```
+
+**Verified:** 2026-02-24 — Real-time watcher capturing (12,223 → 12,228 points)
+
+---
+
+## Architecture
+
+### Collections (Qdrant)
+
+| Collection | Purpose | Content | Status |
+|------------|---------|---------|--------|
+| `memories_tr` | Full text storage | Every conversation turn (migrated from `kimi_memories`) | ✅ Active |
+| `gems_tr` | Gems (extracted insights) | Curated key points (for injection) | ✅ Active |
+| `true_recall` | Legacy gems | Archive of previously extracted gems | 📦 Preserved |
+| `kimi_memories` | Original collection | Backup (12,223 points preserved) | 📦 Preserved |
+
+### Migration Script
+
+| File | Purpose |
+|------|---------|
+| `migrate_memories.py` | Migrate data from `kimi_memories` → `memories_tr` with cleaning |
+
+**Usage:**
+```bash
+python3 migrate_memories.py
+```
+
+**What it does:**
+- Reads all points from `kimi_memories`
+- Cleans content (removes metadata, thinking tags)
+- Stores to `memories_tr`
+- Preserves original `kimi_memories`
+
+---
+
+### Components
+
+| Component | Location | Purpose |
+|-----------|----------|---------|
+| **memory-qdrant plugin** | `/root/.openclaw/extensions/memory-qdrant/` | Injects gems as context |
+| **Curation script** | `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` | Extracts gems |
+| **Curator prompt** | `/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md` | Instructions for gem extraction |
+
+---
+
+## File Locations
+
+### Core Files
+
+```
+/root/.openclaw/workspace/.projects/true-recall-v2/
+├── README.md                           # This file
+├── session.md                          # Development notes
+├── curator-prompt.md                   # Gem extraction prompt
+│
+├── tr-daily/                           # Daily curation
+│   └── curate_from_qdrant.py          # Main curator script
+│
+├── tr-compact/                         # (Reserved for v2 expansion)
+│   └── hook.py                         # Compaction hook (not active)
+│
+├── tr-worker/                          # (Reserved for v2 expansion)
+│   └── worker.py                       # Background worker (not active)
+│
+└── shared/                             # Shared resources
+```
+
+### Plugin Files (Located in OpenClaw Extensions)
+
+> **Note:** These files live in `/root/.openclaw/extensions/`, not in this project folder. Documented here for reference.
+
+| Actual Location | Project Reference |
+|----------------|-------------------|
+| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code (capture + injection) |
+| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
+| `/root/.openclaw/openclaw.json` | Plugin configuration |
+
+### Configuration
+
+| File | Purpose |
+|------|---------|
+| `/root/.openclaw/openclaw.json` | Plugin config (collectionName: gems_tr) |
+| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code |
+| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
+
+### Cron Jobs
+
+| Time | Command | Purpose |
+|------|---------|---------|
+| 2:45 AM | `curate_memories.py` | Daily gem extraction (v1) → stores to gems_tr |
+| 2:50 AM | `archive_to_memories_tr.py` | Archive to memories_tr |
+
+View cron: `crontab -l`
+
+---
+
+## Process Flow
+
+### Step 1: Capture (Real-Time Watcher)
+
+```
+OpenClaw Session ──→ Real-Time Watcher ──→ Qdrant (memories_tr)
+```
+
+**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
+
+**What it does:**
+- Watches session JSONL files in real-time
+- Parses each conversation turn (user + AI)
+- Embeds with `snowflake-arctic-embed2` via Ollama
+- Stores directly to `memories_tr` collection
+- Cleans content (removes metadata, thinking tags)
+
+**Systemd Service:** `mem-qdrant-watcher.service`
+
+**Status:** ✅ Created, ready to deploy
+
+### Step 2: Curation (Daily)
+
+```
+2:45 AM → curate_memories.py → memories_tr → qwen3 → gems_tr
+```
+
+**Location:** `/root/.openclaw/workspace/.projects/true-recall-v1/tr-process/curate_memories.py`
+
+**What it does:**
+1. Reads from Redis buffer (`mem:rob`)
+2. Passes to qwen3 (curator model)
+3. Extracts gems using prompt
+4. Stores gems to `gems_tr` collection
+
+### Step 3: Injection
+
+```
+User message → memory-qdrant plugin → Search gems_tr → Inject as context
+```
+
+**Location:** `/root/.openclaw/extensions/memory-qdrant/index.ts`
+
+**What it does:**
+1. Listens to `before_agent_start` events
+2. Embeds current prompt
+3. Searches `gems_tr` collection
+4. Returns `{ prependContext: "..." }` with gems
+5. Gems appear in my context (hidden from UI)
+
+---
+
+## Configuration
+
+### memory-qdrant Plugin
+
+```json
+{
+  "autoCapture": true,
+  "autoRecall": true,
+  "collectionName": "gems_tr",
+  "embeddingModel": "snowflake-arctic-embed2",
+  "maxRecallResults": 2,
+  "minRecallScore": 0.7
+}
+```
+
+**Key setting:** `collectionName: "gems_tr"` — tells plugin to inject gems, not full text.
+
+---
+
+## Gem Format
+
+```json
+{
+  "gem": "User prefers dark mode for interface",
+  "context": "Discussed UI theme options",
+  "snippet": "rob: I want dark mode\nKimi: Done",
+  "categories": ["preference"],
+  "importance": "high",
+  "confidence": 0.95,
+  "date": "2026-02-24",
+  "conversation_id": "uuid",
+  "turn_range": "5-7"
+}
+```
+
+---
+
+## Validation Commands
+
+### Check Qdrant Collections
+
+```bash
+curl -s "http://10.0.0.40:6333/collections" | python3 -m json.tool
+```
+
+### Check Collection Points
+
+```bash
+# memories_tr (full text)
+curl -s -X POST "http://10.0.0.40:6333/collections/memories_tr/points/scroll" \
+  -H "Content-Type: application/json" \
+  -d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
+
+# gems_tr (gems)
+curl -s -X POST "http://10.0.0.40:6333/collections/gems_tr/points/scroll" \
+  -H "Content-Type: application/json" \
+  -d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
+```
+
+### Check Cron Jobs
+
+```bash
+crontab -l | grep -E "true|recall|curate"
+```
+
+### Check Plugin Logs
+
+```bash
+tail -50 /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant
+```
+
+---
+
+## Troubleshooting
+
+### Issue: `<relevant-memories>` showing in UI
+
+**Cause:** Plugin was using `memories_tr` (full text) instead of `gems_tr` (gems)
+
+**Fix:** Set `"collectionName": "gems_tr"` in openclaw.json
+
+**Reference:** KB entry `relevant-memories-ui-issue.md`
+
+### Issue: No gems being extracted
+
+**Check:**
+1. Is memories_tr populated? `curl .../collections/memories_tr/points/scroll`
+2. Is curator prompt valid? `cat curator-prompt.md`
+3. Is qwen3 available? `curl -s http://10.0.0.10:11434/api/tags`
+
+---
+
+## Related Projects
+
+| Project | Location | Purpose |
+|---------|----------|---------|
+| **true-recall-v1** | `/.projects/true-recall-v1/` | Original (Redis-based) |
+| **memory-qdrant** | `/root/.openclaw/extensions/memory-qdrant/` | Plugin |
+| **mem-redis** | `/root/.openclaw/workspace/skills/mem-redis/` | Redis utilities |
+
+---
+
+## Backups
+
+| File | Backup Location |
+|------|-----------------|
+| openclaw.json | `/root/.openclaw/openclaw.json.bak.2026-02-24` |
+| Plugin | `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24` |
+
+---
+
+## Final Status (2026-02-24 15:08)
+
+| Component | Status |
+|-----------|--------|
+| Collection `kimi_memories` → `memories_tr` | ✅ Migrated (12,223 points) |
+| Real-time watcher | ✅ **Deployed & verified** (12,228 points, +5 new) |
+| Collection `gems_tr` | ✅ Active (5 gems stored) |
+| Curator v2 | ✅ Working - tested with 327 turns |
+| Config | ✅ Updated to `gems_tr` |
+| Cron jobs | ✅ Cleaned |
+| Documentation | ✅ Updated |
+
+### Daily Schedule (Simplified)
+
+| Time | Job | Flow |
+|------|-----|------|
+| Continuous | autoCapture | Conversation → `memories_tr` |
+| 2:45 AM | Curator v2 | `memories_tr` → `gems_tr` |
+| Each turn | Injection | `gems_tr` → Context |
+
+**Redis usage:** Disabled for memory. Used only for `delayed:notifications` queue.
+
+**Auto-Capture Status:** ✅ **Working** - Real-time watcher deployed and verified
+
+---
+
+**Last Updated:** 2026-02-24  
+**Project Lead:** Rob  
+**AI Assistant:** Kimi 🎙️
+