495 lines
14 KiB
Markdown
495 lines
14 KiB
Markdown
# TrueRecall v2 - Session Notes
|
|
|
|
**Last Updated:** 2026-02-24 19:02 CST
|
|
**Status:** ✅ Active & Verified
|
|
**Version:** v2.2 (Timer-based curation deployed)
|
|
|
|
---
|
|
|
|
## Session End (18:09 CST)
|
|
|
|
**Reason:** User starting new session
|
|
|
|
**Current State:**
|
|
- Real-time watcher: ✅ Active (capturing live)
|
|
- Timer curator: ✅ Deployed (every 30 min via cron)
|
|
- Daily curator: ❌ Removed (replaced by timer)
|
|
- Total memories: 12,378 (all tagged with `curated: false`)
|
|
- Gems: 5 (from Feb 18 test)
|
|
|
|
**Next session start:** Read this file, then check:
|
|
```bash
|
|
# Quick status
|
|
python3 /root/.openclaw/workspace/.projects/true-recall-v2/tr-continuous/curator_by_count.py --status
|
|
sudo systemctl status mem-qdrant-watcher
|
|
curl -s http://10.0.0.40:6333/collections/memories_tr | jq '.result.points_count'
|
|
```
|
|
|
|
---
|
|
|
|
## Executive Summary
|
|
|
|
TrueRecall v2 is a complete memory system with real-time capture, daily curation, and context injection. All components are operational.
|
|
|
|
---
|
|
|
|
## Current State (Verified 18:09 CST)
|
|
|
|
### Qdrant Collections
|
|
|
|
| Collection | Points | Purpose | Status |
|
|
|------------|--------|---------|--------|
|
|
| `memories_tr` | **12,378** | Full text (live capture) | ✅ Active |
|
|
| `gems_tr` | **5** | Curated gems (injection) | ✅ Active |
|
|
| `true_recall` | existing | Legacy archive | 📦 Preserved |
|
|
| `kimi_memories` | 12,223 | Original backup | 📦 Preserved |
|
|
|
|
**Note:** All memories tagged with `curated: false` for timer curator.
|
|
|
|
### Services
|
|
|
|
| Service | Status | Uptime |
|
|
|---------|--------|--------|
|
|
| `mem-qdrant-watcher` | ✅ Active | 30+ min |
|
|
| OpenClaw Gateway | ✅ Running | 2026.2.23 |
|
|
| memory-qdrant plugin | ✅ Loaded | recall: gems_tr, capture: memories_tr |
|
|
|
|
---
|
|
|
|
## Architecture
|
|
|
|
### v2.2: Timer-Based Curation (DEPLOYED)
|
|
|
|
**Data Flow:**
|
|
```
|
|
┌─────────────────┐ ┌──────────────────────┐ ┌─────────────┐
|
|
│ OpenClaw Chat │────▶│ Real-Time Watcher │────▶│ memories_tr │
|
|
│ (Session JSONL)│ │ (Python daemon) │ │ (Qdrant) │
|
|
└─────────────────┘ └──────────────────────┘ └──────┬──────┘
|
|
│
|
|
│ Every 30 min
|
|
▼
|
|
┌──────────────────┐
|
|
│ Timer Curator │
|
|
│ (cron/qwen3) │
|
|
└────────┬─────────┘
|
|
│
|
|
▼
|
|
┌──────────────────┐
|
|
│ gems_tr │
|
|
│ (Qdrant) │
|
|
└────────┬─────────┘
|
|
│
|
|
Per turn │
|
|
▼
|
|
┌──────────────────┐
|
|
│ memory-qdrant │
|
|
│ plugin │
|
|
└──────────────────┘
|
|
```
|
|
|
|
**Key Changes:**
|
|
- ✅ Replaced daily 2:45 AM batch with 30-minute timer
|
|
- ✅ All memories tagged `curated: false` on write
|
|
- ✅ Migration completed for 12,378 existing memories
|
|
- ✅ No Redis dependency (direct Qdrant only)
|
|
|
|
---
|
|
|
|
## Components
|
|
|
|
### Curation Mode: Timer-Based (DEPLOYED v2.2)
|
|
|
|
| Setting | Value | Adjustable |
|
|
|---------|-------|------------|
|
|
| **Trigger** | Cron timer | ✅ |
|
|
| **Interval** | 30 minutes | ✅ Config file |
|
|
| **Batch size** | 100 memories max | ✅ Config file |
|
|
| **Minimum** | None (0 is OK) | — |
|
|
|
|
**Config:** `/tr-continuous/curator_config.json`
|
|
```json
|
|
{
|
|
"timer_minutes": 30,
|
|
"max_batch_size": 100,
|
|
"user_id": "rob",
|
|
"source_collection": "memories_tr",
|
|
"target_collection": "gems_tr"
|
|
}
|
|
```
|
|
|
|
**Cron:**
|
|
```
|
|
*/30 * * * * cd .../tr-continuous && python3 curator_timer.py
|
|
```
|
|
|
|
**Old modes deprecated:**
|
|
- ❌ Turn-based (every N turns)
|
|
- ❌ Hybrid (timer + turn)
|
|
- ❌ Daily batch (2:45 AM)
|
|
|
|
### 1. Real-Time Watcher (Primary Capture)
|
|
|
|
**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
|
|
|
**Function:**
|
|
- Watches `/root/.openclaw/agents/main/sessions/*.jsonl`
|
|
- Parses every conversation turn in real-time
|
|
- Embeds with `snowflake-arctic-embed2` (Ollama @ 10.0.0.10)
|
|
- Stores directly to `memories_tr` (no Redis)
|
|
- **Cleans content:** Removes markdown, tables, metadata, thinking tags
|
|
|
|
**Service:** `mem-qdrant-watcher.service`
|
|
- **Status:** Active since 16:46:53 CST
|
|
- **Systemd:** Enabled, auto-restart
|
|
|
|
**Log:** `journalctl -u mem-qdrant-watcher -f`
|
|
|
|
---
|
|
|
|
### 2. Content Cleaner (Existing Data)
|
|
|
|
**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/clean_memories_tr.py`
|
|
|
|
**Function:**
|
|
- Batch-cleans existing `memories_tr` points
|
|
- Removes: `**bold**`, `|tables|`, `` `code` ``, `---` rules, `# headers`
|
|
- Flattens nested content dicts
|
|
- Rate-limited to prevent Qdrant overload
|
|
|
|
**Usage:**
|
|
```bash
|
|
# Dry run (preview)
|
|
python3 clean_memories_tr.py --dry-run
|
|
|
|
# Clean all
|
|
python3 clean_memories_tr.py --execute
|
|
|
|
# Clean limited (test)
|
|
python3 clean_memories_tr.py --execute --limit 100
|
|
```
|
|
|
|
---
|
|
|
|
### 3. Timer Curator (v2.2 - DEPLOYED)
|
|
|
|
**Replaces:** Daily curator (2:45 AM batch) and turn-based curator
|
|
|
|
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/tr-continuous/curator_timer.py`
|
|
|
|
**Schedule:** Every 30 minutes (cron)
|
|
|
|
**Flow:**
|
|
1. Query uncurated memories (`curated: false`)
|
|
2. Send batch to qwen3 (max 100)
|
|
3. Extract gems using curator prompt
|
|
4. Store gems to `gems_tr`
|
|
5. Mark processed memories as `curated: true`
|
|
|
|
**Files:**
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `curator_timer.py` | Main curator script |
|
|
| `curator_config.json` | Adjustable settings |
|
|
| `migrate_add_curated.py` | One-time migration (completed) |
|
|
|
|
**Usage:**
|
|
```bash
|
|
# Dry run (preview)
|
|
python3 curator_timer.py --dry-run
|
|
|
|
# Manual run
|
|
python3 curator_timer.py --config curator_config.json
|
|
```
|
|
|
|
**Status:** ✅ Deployed, first run will process ~12,378 existing memories
|
|
|
|
### 5. Silent Compacting (NEW - Concept)
|
|
|
|
**Idea:** Automatically remove old context from prompt when token limit approached.
|
|
|
|
**Behavior:**
|
|
- Trigger: Context window > 80% full
|
|
- Action: Remove oldest messages (silently)
|
|
- Preserve: Gems always kept, recent N turns kept
|
|
- Result: Seamless conversation without "compacting" notification
|
|
|
|
**Config:**
|
|
```json
|
|
{
|
|
"compacting": {
|
|
"enabled": true,
|
|
"triggerAtPercent": 80,
|
|
"keepRecentTurns": 20,
|
|
"preserveGems": true,
|
|
"silent": true
|
|
}
|
|
}
|
|
```
|
|
|
|
**Status:** ⏳ Concept only - requires OpenClaw core changes
|
|
|
|
### 6. memory-qdrant Plugin
|
|
|
|
**Location:** `/root/.openclaw/extensions/memory-qdrant/`
|
|
|
|
**Config:**
|
|
```json
|
|
{
|
|
"collectionName": "gems_tr",
|
|
"captureCollection": "memories_tr",
|
|
"autoRecall": true,
|
|
"autoCapture": true
|
|
}
|
|
```
|
|
|
|
**Function:**
|
|
- **Recall:** Searches `gems_tr`, injects as context (hidden)
|
|
- **Capture:** Session-level capture to `memories_tr` (backup)
|
|
|
|
**Status:** Loaded, dual collection support working
|
|
|
|
---
|
|
|
|
## Files & Locations
|
|
|
|
### Core Project Files
|
|
|
|
```
|
|
/root/.openclaw/workspace/.projects/true-recall-v2/
|
|
├── README.md # Architecture docs
|
|
├── session.md # This file
|
|
├── curator-prompt.md # Gem extraction prompt
|
|
├── tr-daily/ # Daily batch curation
|
|
│ └── curate_from_qdrant.py # Daily curator (2:45 AM)
|
|
├── tr-continuous/ # Real-time curation (NEW)
|
|
│ ├── curator_by_count.py # Turn-based curator
|
|
│ ├── curator_turn_based.py # Alternative approach
|
|
│ ├── curator_cron.sh # Cron wrapper
|
|
│ ├── turn-curator.service # Systemd service
|
|
│ └── README.md # Documentation
|
|
└── shared/
|
|
└── (shared resources)
|
|
```
|
|
|
|
### New Files (2026-02-24 19:00)
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `tr-continuous/curator_timer.py` | Timer-based curator (deployed) |
|
|
| `tr-continuous/curator_config.json` | Curator settings |
|
|
| `tr-continuous/migrate_add_curated.py` | Migration script (completed) |
|
|
|
|
### Legacy Files (Pre-v2.2)
|
|
|
|
| File | Status | Note |
|
|
|------|--------|------|
|
|
| `tr-daily/curate_from_qdrant.py` | 📦 Archived | Replaced by timer |
|
|
| `tr-continuous/curator_by_count.py` | 📦 Archived | Replaced by timer |
|
|
| `tr-continuous/curator_turn_based.py` | 📦 Archived | Replaced by timer |
|
|
|
|
### System Locations
|
|
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `/root/.openclaw/extensions/memory-qdrant/` | Plugin code |
|
|
| `/root/.openclaw/openclaw.json` | Plugin configuration |
|
|
| `/etc/systemd/system/mem-qdrant-watcher.service` | Systemd service |
|
|
|
|
---
|
|
|
|
## Changes Made Today (2026-02-24 19:00)
|
|
|
|
### 1. Timer Curator Deployed (v2.2)
|
|
|
|
- Created `curator_timer.py` — simplified timer-based curation
|
|
- Created `curator_config.json` — adjustable settings
|
|
- Removed daily 2:45 AM cron job
|
|
- Added `*/30 * * * *` cron timer
|
|
- **Status:** ✅ Deployed, logs to `/var/log/true-recall-timer.log`
|
|
|
|
### 2. Migration Completed
|
|
|
|
- Created `migrate_add_curated.py`
|
|
- Tagged 12,378 existing memories with `curated: false`
|
|
- Updated watcher to add `curated: false` to new memories
|
|
- **Status:** ✅ Complete
|
|
|
|
### 3. Simplified Architecture
|
|
|
|
- ❌ Removed turn-based curator complexity
|
|
- ❌ Removed daily batch processing
|
|
- ✅ Single timer trigger every 30 minutes
|
|
- ✅ No minimum threshold (processes 0-N memories)
|
|
|
|
---
|
|
|
|
## Configuration
|
|
|
|
### memory-qdrant Plugin
|
|
|
|
**File:** `/root/.openclaw/openclaw.json`
|
|
|
|
```json
|
|
{
|
|
"memory-qdrant": {
|
|
"config": {
|
|
"autoCapture": true,
|
|
"autoRecall": true,
|
|
"collectionName": "gems_tr",
|
|
"captureCollection": "memories_tr",
|
|
"embeddingModel": "snowflake-arctic-embed2",
|
|
"maxRecallResults": 2,
|
|
"minRecallScore": 0.7,
|
|
"ollamaUrl": "http://10.0.0.10:11434",
|
|
"qdrantUrl": "http://10.0.0.40:6333"
|
|
},
|
|
"enabled": true
|
|
}
|
|
}
|
|
```
|
|
|
|
### Gateway (OpenClaw Update Fix)
|
|
|
|
```json
|
|
{
|
|
"gateway": {
|
|
"controlUi": {
|
|
"allowedOrigins": ["*"],
|
|
"allowInsecureAuth": false,
|
|
"dangerouslyDisableDeviceAuth": true
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## Validation Commands
|
|
|
|
### Check Collections
|
|
|
|
```bash
|
|
# Points count
|
|
curl -s http://10.0.0.40:6333/collections/memories_tr | jq '.result.points_count'
|
|
curl -s http://10.0.0.40:6333/collections/gems_tr | jq '.result.points_count'
|
|
|
|
# Recent points
|
|
curl -s -X POST http://10.0.0.40:6333/collections/memories_tr/points/scroll \
|
|
-H "Content-Type: application/json" \
|
|
-d '{"limit": 5, "with_payload": true}' | jq '.result.points[].payload.content'
|
|
```
|
|
|
|
### Check Services
|
|
|
|
```bash
|
|
# Watcher status
|
|
sudo systemctl status mem-qdrant-watcher
|
|
|
|
# Watcher logs
|
|
sudo journalctl -u mem-qdrant-watcher -n 20
|
|
|
|
# OpenClaw status
|
|
openclaw status
|
|
```
|
|
|
|
---
|
|
|
|
## Troubleshooting
|
|
|
|
### Issue: Watcher Not Capturing
|
|
|
|
**Check:**
|
|
1. Service running? `systemctl status mem-qdrant-watcher`
|
|
2. Logs: `journalctl -u mem-qdrant-watcher -f`
|
|
3. Qdrant accessible? `curl http://10.0.0.40:6333/`
|
|
4. Ollama accessible? `curl http://10.0.0.10:11434/api/tags`
|
|
|
|
### Issue: Cleaner Fails
|
|
|
|
**Common causes:**
|
|
- Qdrant connection timeout (add `time.sleep(0.1)` between batches)
|
|
- Nested content dicts (handled in updated script)
|
|
- Type errors (non-string content — handled)
|
|
|
|
### Issue: Plugin Not Loading
|
|
|
|
**Check:**
|
|
1. `openclaw.json` syntax valid? `openclaw config validate`
|
|
2. Plugin compiled? `cd /root/.openclaw/extensions/memory-qdrant && npx tsc`
|
|
3. Gateway logs: `tail /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log`
|
|
|
|
---
|
|
|
|
## Cron Schedule (Updated v2.2)
|
|
|
|
| Time | Job | Script | Status |
|
|
|------|-----|--------|--------|
|
|
| Every 30 min | Timer curator | `tr-continuous/curator_timer.py` | ✅ Active |
|
|
| Per turn | Capture | `mem-qdrant-watcher` | ✅ Daemon |
|
|
| Per turn | Injection | `memory-qdrant` plugin | ✅ Active |
|
|
|
|
**Removed:**
|
|
- ❌ 2:45 AM daily curator
|
|
- ❌ Every-minute turn curator check
|
|
|
|
---
|
|
|
|
## Next Steps
|
|
|
|
### Immediate
|
|
- ⏳ Monitor first timer run (logs: `/var/log/true-recall-timer.log`)
|
|
- ⏳ Validate gem extraction quality from timer curator
|
|
- ⏳ Archive old curator scripts if timer works
|
|
|
|
### Completed ✅
|
|
- ✅ **Compactor config** — Minimal overhead: `mode: default`, `reserveTokensFloor: 0`, `memoryFlush: false`
|
|
|
|
### Future
|
|
- ⏳ Curator tuning based on timer results
|
|
- ⏳ Silent compacting (requires OpenClaw core changes)
|
|
|
|
### Planned Features (Backlog)
|
|
- ⏳ **Interactive install script** — Prompts for embedding model, timer interval, batch size, endpoints
|
|
- ⏳ **Single embedding model option** — Use one model for both collections
|
|
- ⏳ **Configurable thresholds** — Per-user customization via prompts
|
|
|
|
**Compactor Settings (Applied):**
|
|
```json5
|
|
{
|
|
agents: {
|
|
defaults: {
|
|
compaction: {
|
|
mode: "default",
|
|
reserveTokensFloor: 0,
|
|
memoryFlush: { enabled: false }
|
|
}
|
|
}
|
|
}
|
|
}
|
|
```
|
|
|
|
**Note:** Only `mode`, `reserveTokensFloor`, and `memoryFlush` are valid under `agents.defaults.compaction`. Other settings are Pi runtime parameters.
|
|
|
|
**Install script prompts:**
|
|
1. Embedding model (snowflake vs mxbai)
|
|
2. Timer interval (5 min / 30 min / hourly)
|
|
3. Batch size (50 / 100 / 500)
|
|
4. Qdrant/Ollama URLs
|
|
5. User ID
|
|
|
|
---
|
|
|
|
## Session Recovery
|
|
|
|
If starting fresh:
|
|
1. Read `README.md` for architecture overview
|
|
2. Check service status: `sudo systemctl status mem-qdrant-watcher`
|
|
3. Check timer curator: `tail /var/log/true-recall-timer.log`
|
|
4. Verify collections: `curl http://10.0.0.40:6333/collections`
|
|
|
|
---
|
|
|
|
*Last Verified: 2026-02-24 19:29 CST*
|
|
*Version: v2.2 (30b curator, install script planned)*
|