SpeedyFoxAi/true-recall-gems
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: simplify README, update validation and curator docs

Browse Source
This commit is contained in:
root
2026-03-10 12:08:53 -05:00
parent 08aaddb4d0
commit 62953e9f39
6 changed files with 261 additions and 813 deletions
Download Patch File Download Diff File Expand all files Collapse all files

694
README.md
View File

@@ -1,613 +1,147 @@
# TrueRecall v2
# TrueRecall Gems (v2)
**Project:** Gem extraction and memory recall system
**Status:** ✅ Active & Verified
**Location:** `~/.openclaw/workspace/.local_projects/true-recall-v2/`
**Last Updated:** 2026-02-25 12:04 CST
**Purpose:** Memory curation (gems) + context injection
---
## Table of Contents
- [Quick Start](#quick-start)
- [Overview](#overview)
- [Current State](#current-state)
- [Architecture](#architecture)
- [Components](#components)
- [Files & Locations](#files--locations)
- [Configuration](#configuration)
- [Validation](#validation)
- [Troubleshooting](#troubleshooting)
- [Status Summary](#status-summary)
---
## Quick Start
```bash
# Check system status
openclaw status
sudo systemctl status mem-qdrant-watcher
# View recent captures
curl -s http://<QDRANT_IP>:6333/collections/memories_tr | jq '.result.points_count'
# Check collections
curl -s http://<QDRANT_IP>:6333/collections | jq '.result.collections[].name'
```
---
## Recent Fixes (2026-02-25 12:41 CST)
| Issue | Root Cause | Fix Applied |
|-------|------------|-------------|
| **Watcher stuck on old session** | Watcher only switched sessions when file deleted, old sessions persisted | ✅ Restarted service, now follows current session |
| **Plugin capture 0 exchanges** | OpenClaw uses OpenAI content format (array of items), plugin expected string | ✅ Added `extractMessageText()` to extract text from `type: "text"` items |
| **Gem ID collision** | Hash used non-existent fields (`conversation_id`, `turn_range`, `gem`) | ✅ Hash now uses `embedding_text_for_hash[:100]` |
| **Meta-gems extracted** | Curator extracted from debug/tool output | ✅ Added SKIP_PATTERNS filter ("gems extracted", "✅", "🔍", etc.) + skip `role: "assistant"` |
| **gems_tr pollution** | 5 meta-gems + 1 real gem | ✅ Cleaned, now 1 real gem only |
| **First-person format** | Third person "User decided..." | ✅ Changed to "I decided..." for better query matching (score 0.746 vs 0.39) |
### Validation Results
**Plugin capture:**
```
Before: parsed 14 user, 84 assistant messages, 0 exchanges
After: parsed 17 user, 116 assistant messages, 9 exchanges ✅
```
**Watcher:**
```
Before: Watching old session (1737142a... from Feb 24)
After: Watching current session (93dc32bf... from Feb 25) ✅
```
---
## Needed Improvements
| Issue | Description | Priority |
|-------|-------------|----------|
| **Semantic Deduplication** | No dedup between similar gems. Same fact phrased differently creates multiple gems. Need semantic similarity check before storage. | High |
| **Search Result Deduplication** | Similar gems both above threshold are both injected, causing redundancy. Need filter to remove near-duplicates from results. | Medium |
| **Gem Quality Scoring** | No quality metric. Some extracted gems may be low value. Need LLM-based quality scoring. | Medium |
| **Temporal Decay** | All gems treated equally regardless of age. Should weight recent gems higher. | Low |
| **Gem Merging/Updating** | When user changes preference, old gem still exists. Need mechanism to update/contradict old gems. | Low |
| **Importance Calibration** | All curator gems marked "medium" importance. Should dynamically assign based on significance. | Low |
### Gem Quality & Model Intelligence
**Gem quality improves significantly with smarter models:**
| Model | Gem Quality | Example |
|-------|-------------|---------|
| **Small models (7B)** | Basic extraction, may miss nuance | "User likes local AI" |
| **Medium models (30B)** | Better categorization, captures intent | "I prefer local AI over cloud services for privacy reasons" |
| **Large models (70B+)** | Rich context, infers significance, better first-person conversion | "I decided to self-host AI tools because I value data privacy and want to avoid vendor lock-in" |
**Example Gem (High Quality):**
```json
{
"text": "I decided to keep the installation simple and not include gems for the basic version",
"category": "decision",
"importance": "high"
}
```
**Current:** Using `qwen3:30b-a3b-instruct` for extraction (good balance of quality/speed).
**Recommendation:** For production use, consider `qwen3:72b` or `deepseek-r1` for higher gem quality.
**Status:** ⚠️ Requires true-recall-base to be installed first
---
## Overview
TrueRecall v2 is a **standalone memory system** that extracts "gems" (key insights) from conversations and injects them as context. It operates independently — not an addon or extension of any previous system.
TrueRecall Gems adds **curation** and **injection** on top of Base's capture foundation.
TrueRecall v2 replaces both Jarvis Memory and TrueRecall v1 with a completely re-architected solution:
| System | Status | Relationship to v2 |
|--------|--------|-------------------|
| **Jarvis Memory** | Legacy | Replaced by v2 |
| **TrueRecall v1** | Deprecated | Replaced by v2 |
| **TrueRecall v2** | ✅ Active | Complete standalone replacement |
### Three-Layer Architecture
1. **Capture** — Real-time watcher saves every turn to `memories_tr`
2. **Curation** — Timer-based curator extracts gems to `gems_tr`
3. **Injection** — Plugin searches `gems_tr` and injects gems per turn
**Key:** v2 requires no components from Jarvis Memory or v1. It is self-contained with its own storage (Qdrant-only), capture mechanism, and injection system.
**Gems is an ADDON:**
- Requires true-recall-base
- Independent from openclaw-true-recall-blocks
- Choose Gems OR Blocks, not both
---
## Current State
### Verified at 19:02 CST
| Collection | Points | Purpose | Status |
|------------|--------|---------|--------|
| `memories_tr` | **12,729** | Full text (live capture) | ✅ Active |
| `gems_tr` | **14+** | Curated gems (injection) | ✅ **WORKING** - Context injection verified |
**All memories tagged with `curated: false` for timer curation.**
### Services Status
| Service | Status | Details |
|---------|--------|---------|
| `mem-qdrant-watcher` | ✅ Active | PID 234, capturing |
| Timer curator | ✅ Deployed | Every 5 min via cron |
| OpenClaw Gateway | ✅ Running | Version 2026.2.23 |
| memory-qdrant plugin | ✅ Loaded | recall: gems_tr |
---
## Comparison: TrueRecall v2 vs Jarvis Memory vs v1
| Feature | Jarvis Memory | TrueRecall v1 | TrueRecall v2 |
|---------|---------------|---------------|---------------|
| **Storage** | Redis | Redis + Qdrant | Qdrant only |
| **Capture** | Session batch | Session batch | Real-time |
| **Curation** | Manual | Daily 2:45 AM | Timer (5 min) ✅ |
| **Embedding** | — | snowflake | snowflake-arctic-embed2 ✅ |
| **Curator LLM** | — | qwen3:4b | qwen3:30b |
| **State tracking** | — | — | `curated` tag |
| **Batch size** | — | 24h worth | Configurable |
| **JSON parsing** | — | Fallback needed | Native (30b) |
**Key Improvements v2:**
- ✅ Real-time capture (no batch delay)
- ✅ Timer-based curation (responsive vs daily)
- ✅ 30b curator (better gems, faster ~3s)
- ✅ `curated` tag (reliable state tracking)
- ✅ No Redis dependency (simpler stack)
---
## Architecture
### v2.2: Timer-Based Curation
## Three-Tier Architecture
```
┌─────────────────┐ ┌──────────────────────┐ ┌─────────────┐
│ OpenClaw Chat │────▶│ Real-Time Watcher │────▶│ memories_tr │
│ (Session JSONL)│ │ (Python daemon) │ │ (Qdrant) │
└─────────────────┘ └──────────────────────┘ └──────┬──────┘
│
│ Every 30 min
▼
┌──────────────────┐
│ Timer Curator │
│ (cron/qwen3) │
└────────┬─────────┘
│
▼
┌──────────────────┐
│ gems_tr │
│ (Qdrant) │
└────────┬─────────┘
│
Per turn │
▼
┌──────────────────┐
│ memory-qdrant │
│ plugin │
└──────────────────┘
```
true-recall-base (REQUIRED)
├── Watcher daemon
└── memories_tr (raw capture)
│
└──▶ true-recall-gems (THIS ADDON)
├── Curator extracts gems
├── gems_tr (curated)
└── Plugin injection
**Key Changes in v2.2:**
- ✅ Timer-based curation (30 min intervals)
- ✅ All memories tagged `curated: false` on capture
- ✅ Migration complete (12,378 memories)
- ❌ Removed daily batch processing (2:45 AM)
---
## Components
### 1. Real-Time Watcher
**File:** `skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
**What it does:**
- Watches `~/.openclaw/agents/main/sessions/*.jsonl`
- Parses each turn (user + AI)
- Embeds with `snowflake-arctic-embed2`
- Stores to `memories_tr` instantly
- **Cleans:** Removes markdown, tables, metadata
**Service:** `mem-qdrant-watcher.service`
**Commands:**
```bash
# Check status
sudo systemctl status mem-qdrant-watcher
# View logs
sudo journalctl -u mem-qdrant-watcher -f
# Restart
sudo systemctl restart mem-qdrant-watcher
Note: Don't install with openclaw-true-recall-blocks.
Choose one addon: Gems OR Blocks.
```
---
### 2. Content Cleaner
## Prerequisites
**File:** `skills/qdrant-memory/scripts/clean_memories_tr.py`
**REQUIRED: Install TrueRecall Base first**
**Purpose:** Batch-clean existing points
**Usage:**
```bash
# Preview changes
python3 clean_memories_tr.py --dry-run
# Clean all
python3 clean_memories_tr.py --execute
# Clean 100 (test)
python3 clean_memories_tr.py --execute --limit 100
```
**Cleans:**
- `**bold**` → plain text
- `|tables|` → removed
- `` `code` `` → plain text
- `---` rules → removed
- `# headers` → removed
---
### 3. Timer Curator
**File:** `tr-continuous/curator_timer.py`
**Schedule:** Every 30 minutes (cron)
**Flow:**
1. Query uncurated memories from `memories_tr`
2. Send batch to qwen3 (max 100)
3. Extract gems → store to `gems_tr`
4. Mark memories as `curated: true`
**Config:** `tr-continuous/curator_config.json`
```json
{
"timer_minutes": 30,
"max_batch_size": 100
}
```
**Logs:** `/var/log/true-recall-timer.log`
---
### 4. Curation Model Comparison
**Current:** `qwen3:4b-instruct`
| Metric | 4b | 30b |
|--------|----|----|
| Speed | ~10-30s per batch | **~3.3s** (tested 2026-02-24) |
| JSON reliability | ⚠️ Needs fallback | ✅ Native |
| Context quality | Basic extraction | ✅ Nuanced |
| Snippet accuracy | ~80% | ✅ Expected: 95%+ |
**30b Benchmark (2026-02-24):**
- Load: 108ms
- Prompt eval: 49ms (1,576 tok/s)
- Generation: 2.9s (233 tokens, 80 tok/s)
- **Total: 3.26s**
**Trade-offs:**
- **4b:** Faster batch processing, lightweight, catches explicit decisions
- **30b:** Deeper context, better inference, ~3x slower but superior quality
**Gem Quality Comparison (Sample Review):**
| Aspect | 4b | 30b |
|--------|----|----|
| **Context depth** | "Extracted via fallback" | Explains *why* decisions were made |
| **Confidence scores** | 0.7-0.85 | 0.9-0.97 |
| **Snippet accuracy** | ~80% (wrong source) | ✅ 95%+ (relevant quotes) |
| **Categories** | Generic "extracted" | Specific: knowledge, technical, decision |
| **Example** | "User implemented BorgBackup" (no context) | "User selected mxbai... due to top MTEB score of 66.5" (explains reasoning) |
**Verdict:** 30b produces significantly higher quality gems — richer context, accurate snippets, and captures architectural intent, not just surface facts.
---
### 5. OpenClaw Compactor Configuration
**Status:** ✅ Applied
**Goal:** Minimal overhead — just remove context, do nothing else.
**Config Applied:**
```json5
{
agents: {
defaults: {
compaction: {
mode: "default", // "default" or "safeguard"
reserveTokensFloor: 0, // Disable safety floor (default: 20000)
memoryFlush: {
enabled: false // Disable silent .md file writes
}
}
}
}
}
```
**What this does:**
- `mode: "default"` — Standard summarization (faster)
- `reserveTokensFloor: 0` — Allow aggressive settings (disables 20k minimum)
- `memoryFlush.enabled: false` — No silent "write memory" turns
**Note:** `reserveTokens` and `keepRecentTokens` are Pi runtime settings, not configurable via `agents.defaults.compaction`. They are set per-model in `contextWindow`/`contextTokens`.
---
### 6. Configuration Options Reference
**All configurable options with defaults:**
| Option | Default | Description |
|--------|---------|-------------|
| **Embedding model** | `mxbai-embed-large` | Model for generating gem embeddings. `mxbai` = higher accuracy (MTEB 66.5). `snowflake` = faster processing. |
| **Timer interval** | `5` minutes | How often the curator runs. `5 min` = fast backlog clearing. `30 min` = balanced. `60 min` = minimal overhead. |
| **Batch size** | `100` | Max memories sent to curator per run. Higher = fewer API calls but more memory usage. |
| **Max gems per run** | *(unlimited)* | Hard limit on gems extracted per batch. Not set by default — extracts all found gems. |
| **Qdrant URL** | `http://<QDRANT_IP>:6333` | Vector database endpoint. Change if Qdrant runs on different host/port. |
| **Ollama URL** | `http://<OLLAMA_IP>:11434` | LLM endpoint for gem extraction. Change if Ollama runs elsewhere. |
| **Curator LLM** | `qwen3:30b-a3b-instruct` | Model for extracting gems. `30b` = best quality (~3s). `4b` = faster but needs JSON fallback. |
| **User ID** | `rob` | Owner identifier for memories. Used for filtering and multi-user setups. |
| **Source collection** | `memories_tr` | Qdrant collection for raw captured memories. |
| **Target collection** | `gems_tr` | Qdrant collection for curated gems (injected into context). |
| **Watcher service** | `enabled` | Real-time capture daemon. Reads session JSONL and writes to Qdrant. |
| **Cron timer** | `enabled` | Periodic curation job. Runs `curator_timer.py` on schedule. |
| **Log path** | `/var/log/true-recall-timer.log` | Where curator output is written. Check with `tail -f`. |
| **Dry-run mode** | `disabled` | Test mode — shows what would be curated without writing to Qdrant. |
**OpenClaw-side options:**
| Option | Default | Description |
|--------|---------|-------------|
| **Compactor mode** | `default` | How context is summarized. `default` = fast standard. `safeguard` = chunked for very long sessions. |
| **Memory flush** | `disabled` | If enabled, writes silent "memory" turn before compaction. Adds overhead — disabled for minimal lag. |
| **Context pruning** | `cache-ttl` | Removes old tool results from context. `cache-ttl` = prunes hourly. `off` = no pruning. |
---
### 7. Embedding Models
**Current Setup:**
- `memories_tr`: `snowflake-arctic-embed2` (capture)
- `gems_tr`: `snowflake-arctic-embed2` (recall) ✅ **FIXED** - Both collections now use same model
**Note:** Previously used `mxbai-embed-large` for gems, but this caused embedding model mismatch. Fixed 2026-02-25.
---
### 6. memory-qdrant Plugin
**Location:** `~/.openclaw/extensions/memory-qdrant/`
**Config (openclaw.json):**
```json
{
"collectionName": "gems_tr",
"captureCollection": "memories_tr",
"autoRecall": true,
"autoCapture": true
}
```
**Functions:**
- **Recall:** Searches `gems_tr`, injects gems (hidden)
- **Capture:** Session-level to `memories_tr` (backup)
---
## Files & Locations
### Core Project
```
~/.openclaw/workspace/.local_projects/true-recall-v2/
├── README.md # This file
├── session.md # Detailed notes
├── curator-prompt.md # Extraction prompt
├── tr-daily/
│ └── curate_from_qdrant.py # Daily curator
└── shared/
```
### New Files (2026-02-24)
| File | Purpose |
|------|---------|
| `tr-continuous/curator_timer.py` | Timer curator (v2.2) |
| `tr-continuous/curator_config.json` | Curator settings |
| `tr-continuous/migrate_add_curated.py` | Migration script |
| `skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` | Capture daemon |
| `skills/qdrant-memory/mem-qdrant-watcher.service` | Systemd service |
### Archived Files (v2.1)
| File | Status | Note |
|------|--------|------|
| `tr-daily/curate_from_qdrant.py` | 📦 Archived | Replaced by timer |
| `tr-continuous/curator_by_count.py` | 📦 Archived | Replaced by timer |
### System Files
| File | Purpose |
|------|---------|
| `~/.openclaw/extensions/memory-qdrant/` | Plugin code |
| `~/.openclaw/openclaw.json` | Configuration |
| `/etc/systemd/system/mem-qdrant-watcher.service` | Service file |
---
## Configuration
### memory-qdrant Plugin
**File:** `~/.openclaw/openclaw.json`
```json
{
"memory-qdrant": {
"config": {
"autoCapture": true,
"autoRecall": true,
"collectionName": "gems_tr",
"captureCollection": "memories_tr",
"embeddingModel": "snowflake-arctic-embed2",
"maxRecallResults": 2,
"minRecallScore": 0.7,
"ollamaUrl": "http://<OLLAMA_IP>:11434",
"qdrantUrl": "http://<QDRANT_IP>:6333"
},
"enabled": true
}
}
```
### Gateway Control UI (OpenClaw 2026.2.23)
```json
{
"gateway": {
"controlUi": {
"allowedOrigins": ["*"],
"allowInsecureAuth": false,
"dangerouslyDisableDeviceAuth": true
}
}
}
```
---
## Validation
### Check Collections
Base provides the capture infrastructure (`memories_tr` collection).
```bash
# Count points
# Verify base is running
curl -s http://<QDRANT_IP>:6333/collections/memories_tr | jq '.result.points_count'
```
## Installation
### 1. Curator Setup
**Install cron job:**
```bash
# Edit path and add to crontab
echo "*/5 * * * * cd <INSTALL_PATH>/true-recall-gems/tr-continuous && /usr/bin/python3 curator_timer.py >> /var/log/true-recall-timer.log 2>&1" | sudo crontab -
sudo touch /var/log/true-recall-timer.log
sudo chmod 644 /var/log/true-recall-timer.log
```
**Configure curator_config.json:**
```json
{
"timer_minutes": 5,
"max_batch_size": 100,
"user_id": "your-user-id",
"source_collection": "memories_tr",
"target_collection": "gems_tr"
}
```
**Edit curator_timer.py:**
- Replace `<QDRANT_IP>`, `<OLLAMA_IP>` with your endpoints
- Replace `<USER_ID>` with your identifier
- Replace `<CURATOR_MODEL>` with your LLM (e.g., `qwen3:30b`)
### 2. Injection Setup
Add to your OpenClaw `openclaw.json`:
```json
{
"plugins": {
"entries": {
"memory-qdrant": {
"config": {
"autoCapture": true,
"autoRecall": true,
"captureCollection": "memories_tr",
"collectionName": "gems_tr",
"embeddingModel": "snowflake-arctic-embed2",
"maxRecallResults": 2,
"minRecallScore": 0.8,
"ollamaUrl": "http://<OLLAMA_IP>:11434",
"qdrantUrl": "http://<QDRANT_IP>:6333"
},
"enabled": true
}
},
"slots": {
"memory": "memory-qdrant"
}
}
}
```
---
## Files
| File | Purpose |
|------|---------|
| `tr-continuous/curator_timer.py` | Timer-based curator |
| `tr-continuous/curator_config.json` | Curator settings template |
---
## Verification
```bash
# Check v1 capture
curl -s http://<QDRANT_IP>:6333/collections/memories_tr | jq '.result.points_count'
# Check v2 curation
curl -s http://<QDRANT_IP>:6333/collections/gems_tr | jq '.result.points_count'
# View recent captures
curl -s -X POST http://<QDRANT_IP>:6333/collections/memories_tr/points/scroll \
-H "Content-Type: application/json" \
-d '{"limit": 3, "with_payload": true}' | jq '.result.points[].payload.content'
```
### Check Services
```bash
# Watcher
sudo systemctl status mem-qdrant-watcher
sudo journalctl -u mem-qdrant-watcher -n 20
# OpenClaw
openclaw status
openclaw gateway status
```
### Test Capture
Send a message, then check:
```bash
# Should increase by 1-2 points
curl -s http://<QDRANT_IP>:6333/collections/memories_tr | jq '.result.points_count'
# Check curator logs
tail -20 /var/log/true-recall-timer.log
```
---
## Troubleshooting
## Dependencies
### Watcher Not Capturing
```bash
# Check logs
sudo journalctl -u mem-qdrant-watcher -f
# Verify dependencies
curl http://<QDRANT_IP>:6333/ # Qdrant
curl http://<OLLAMA_IP>:11434/api/tags # Ollama
```
### Plugin Not Loading
```bash
# Validate config
openclaw config validate
# Check logs
tail /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant
# Restart gateway
openclaw gateway restart
```
### Gateway Won't Start (OpenClaw 2026.2.23+)
**Error:** `non-loopback Control UI requires gateway.controlUi.allowedOrigins`
**Fix:** Add to `openclaw.json`:
```json
"gateway": {
"controlUi": {
"allowedOrigins": ["*"]
}
}
```
| Component | Provided By | Required For |
|-----------|-------------|--------------|
| Capture | v1 | v2 (input) |
| Curation | v2 | Injection |
| Injection | v2 | Context recall |
---
## Status Summary
| Component | Status | Notes |
|-----------|--------|-------|
| Real-time watcher | ✅ Active | PID 1748, capturing |
| memories_tr | ✅ 12,378 pts | All tagged `curated: false` |
| gems_tr | ✅ 5 pts | Injection ready |
| Timer curator | ✅ Deployed | Every 30 min via cron |
| Plugin injection | ✅ **WORKING** | Context injection verified - score 0.587 |
| Migration | ✅ Complete | 12,378 memories |
**Logs:** `tail /var/log/true-recall-timer.log`
**Next:** Monitor first timer run
---
## Roadmap
### Planned Features
| Feature | Status | Description |
|---------|--------|-------------|
| Interactive install script | ⏳ Planned | Prompts for embedding model, timer interval, batch size, endpoints |
| Single embedding model | ⏳ Planned | Option to use one model for both collections |
| Configurable thresholds | ⏳ Planned | Per-user customization via prompts |
**Install script will prompt for:**
1. **Embedding model** — snowflake (fast) vs mxbai (accurate)
2. **Timer interval** — 5 min / 30 min / hourly
3. **Batch size** — 50 / 100 / 500 memories
4. **Endpoints** — Qdrant/Ollama URLs
5. **User ID** — for multi-user setups
---
**Maintained by:** Rob
**AI Assistant:** Kimi 🎙️
**Version:** 2026.02.24-v2.2
**Version:** 2.0
**Requires:** TrueRecall v1
**Collections:** `memories_tr` (v1), `gems_tr` (v2)