true-recall-gems/README.md

# TrueRecall v2

**Project:** Gem extraction and memory recall system  
**Status:** ✅ Active & Verified  
**Location:** `~/.openclaw/workspace/.local_projects/true-recall-v2/`  
**Last Updated:** 2026-02-25 12:04 CST

---

## Table of Contents

- [Quick Start](#quick-start)
- [Overview](#overview)
- [Current State](#current-state)
- [Architecture](#architecture)
- [Components](#components)
- [Files & Locations](#files--locations)
- [Configuration](#configuration)
- [Validation](#validation)
- [Troubleshooting](#troubleshooting)
- [Status Summary](#status-summary)

---

## Quick Start

```bash
# Check system status
openclaw status
sudo systemctl status mem-qdrant-watcher

# View recent captures
curl -s http://<QDRANT_IP>:6333/collections/memories_tr | jq '.result.points_count'

# Check collections
curl -s http://<QDRANT_IP>:6333/collections | jq '.result.collections[].name'
```

---

## Recent Fixes (2026-02-25 12:41 CST)

| Issue | Root Cause | Fix Applied |
|-------|------------|-------------|
| **Watcher stuck on old session** | Watcher only switched sessions when file deleted, old sessions persisted | ✅ Restarted service, now follows current session |
| **Plugin capture 0 exchanges** | OpenClaw uses OpenAI content format (array of items), plugin expected string | ✅ Added `extractMessageText()` to extract text from `type: "text"` items |
| **Gem ID collision** | Hash used non-existent fields (`conversation_id`, `turn_range`, `gem`) | ✅ Hash now uses `embedding_text_for_hash[:100]` |
| **Meta-gems extracted** | Curator extracted from debug/tool output | ✅ Added SKIP_PATTERNS filter ("gems extracted", "✅", "🔍", etc.) + skip `role: "assistant"` |
| **gems_tr pollution** | 5 meta-gems + 1 real gem | ✅ Cleaned, now 1 real gem only |
| **First-person format** | Third person "User decided..." | ✅ Changed to "I decided..." for better query matching (score 0.746 vs 0.39) |

### Validation Results

**Plugin capture:**
```
Before: parsed 14 user, 84 assistant messages, 0 exchanges
After:  parsed 17 user, 116 assistant messages, 9 exchanges ✅
```

**Watcher:**
```
Before: Watching old session (1737142a... from Feb 24)
After:  Watching current session (93dc32bf... from Feb 25) ✅
```

---

## Needed Improvements

| Issue | Description | Priority |
|-------|-------------|----------|
| **Semantic Deduplication** | No dedup between similar gems. Same fact phrased differently creates multiple gems. Need semantic similarity check before storage. | High |
| **Search Result Deduplication** | Similar gems both above threshold are both injected, causing redundancy. Need filter to remove near-duplicates from results. | Medium |
| **Gem Quality Scoring** | No quality metric. Some extracted gems may be low value. Need LLM-based quality scoring. | Medium |
| **Temporal Decay** | All gems treated equally regardless of age. Should weight recent gems higher. | Low |
| **Gem Merging/Updating** | When user changes preference, old gem still exists. Need mechanism to update/contradict old gems. | Low |
| **Importance Calibration** | All curator gems marked "medium" importance. Should dynamically assign based on significance. | Low |

### Gem Quality & Model Intelligence

**Gem quality improves significantly with smarter models:**

| Model | Gem Quality | Example |
|-------|-------------|---------|
| **Small models (7B)** | Basic extraction, may miss nuance | "User likes local AI" |
| **Medium models (30B)** | Better categorization, captures intent | "I prefer local AI over cloud services for privacy reasons" |
| **Large models (70B+)** | Rich context, infers significance, better first-person conversion | "I decided to self-host AI tools because I value data privacy and want to avoid vendor lock-in" |

**Example Gem (High Quality):**
```json
{
  "text": "I decided to keep the installation simple and not include gems for the basic version",
  "category": "decision",
  "importance": "high"
}
```

**Current:** Using `qwen3:30b-a3b-instruct` for extraction (good balance of quality/speed).  
**Recommendation:** For production use, consider `qwen3:72b` or `deepseek-r1` for higher gem quality.

---

## Overview

TrueRecall v2 is a **standalone memory system** that extracts "gems" (key insights) from conversations and injects them as context. It operates independently — not an addon or extension of any previous system.

TrueRecall v2 replaces both Jarvis Memory and TrueRecall v1 with a completely re-architected solution:

| System | Status | Relationship to v2 |
|--------|--------|-------------------|
| **Jarvis Memory** | Legacy | Replaced by v2 |
| **TrueRecall v1** | Deprecated | Replaced by v2 |
| **TrueRecall v2** | ✅ Active | Complete standalone replacement |

### Three-Layer Architecture

1. **Capture** — Real-time watcher saves every turn to `memories_tr`
2. **Curation** — Timer-based curator extracts gems to `gems_tr`
3. **Injection** — Plugin searches `gems_tr` and injects gems per turn

**Key:** v2 requires no components from Jarvis Memory or v1. It is self-contained with its own storage (Qdrant-only), capture mechanism, and injection system.

---

## Current State

### Verified at 19:02 CST

| Collection | Points | Purpose | Status |
|------------|--------|---------|--------|
| `memories_tr` | **12,729** | Full text (live capture) | ✅ Active |
| `gems_tr` | **14+** | Curated gems (injection) | ✅ **WORKING** - Context injection verified |

**All memories tagged with `curated: false` for timer curation.**

### Services Status

| Service | Status | Details |
|---------|--------|---------|
| `mem-qdrant-watcher` | ✅ Active | PID 234, capturing |
| Timer curator | ✅ Deployed | Every 5 min via cron |
| OpenClaw Gateway | ✅ Running | Version 2026.2.23 |
| memory-qdrant plugin | ✅ Loaded | recall: gems_tr |

---

## Comparison: TrueRecall v2 vs Jarvis Memory vs v1

| Feature | Jarvis Memory | TrueRecall v1 | TrueRecall v2 |
|---------|---------------|---------------|---------------|
| **Storage** | Redis | Redis + Qdrant | Qdrant only |
| **Capture** | Session batch | Session batch | Real-time |
| **Curation** | Manual | Daily 2:45 AM | Timer (5 min) ✅ |
| **Embedding** | — | snowflake | snowflake-arctic-embed2 ✅ |
| **Curator LLM** | — | qwen3:4b | qwen3:30b |
| **State tracking** | — | — | `curated` tag |
| **Batch size** | — | 24h worth | Configurable |
| **JSON parsing** | — | Fallback needed | Native (30b) |

**Key Improvements v2:**
- ✅ Real-time capture (no batch delay)
- ✅ Timer-based curation (responsive vs daily)
- ✅ 30b curator (better gems, faster ~3s)
- ✅ `curated` tag (reliable state tracking)
- ✅ No Redis dependency (simpler stack)

---

## Architecture

### v2.2: Timer-Based Curation

```
┌─────────────────┐     ┌──────────────────────┐     ┌─────────────┐
│  OpenClaw Chat  │────▶│ Real-Time Watcher    │────▶│ memories_tr │
│  (Session JSONL)│     │ (Python daemon)      │     │ (Qdrant)    │
└─────────────────┘     └──────────────────────┘     └──────┬──────┘
                                                          │
                                                          │ Every 30 min
                                                          ▼
                                                ┌──────────────────┐
                                                │  Timer Curator   │
                                                │   (cron/qwen3)   │
                                                └────────┬─────────┘
                                                         │
                                                         ▼
                                                ┌──────────────────┐
                                                │    gems_tr       │
                                                │   (Qdrant)       │
                                                └────────┬─────────┘
                                                         │
                                              Per turn   │
                                                         ▼
                                                ┌──────────────────┐
                                                │ memory-qdrant    │
                                                │     plugin       │
                                                └──────────────────┘
```

**Key Changes in v2.2:**
- ✅ Timer-based curation (30 min intervals)
- ✅ All memories tagged `curated: false` on capture
- ✅ Migration complete (12,378 memories)
- ❌ Removed daily batch processing (2:45 AM)

---

## Components

### 1. Real-Time Watcher

**File:** `skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`

**What it does:**
- Watches `~/.openclaw/agents/main/sessions/*.jsonl`
- Parses each turn (user + AI)
- Embeds with `snowflake-arctic-embed2`
- Stores to `memories_tr` instantly
- **Cleans:** Removes markdown, tables, metadata

**Service:** `mem-qdrant-watcher.service`

**Commands:**
```bash
# Check status
sudo systemctl status mem-qdrant-watcher

# View logs
sudo journalctl -u mem-qdrant-watcher -f

# Restart
sudo systemctl restart mem-qdrant-watcher
```

---

### 2. Content Cleaner

**File:** `skills/qdrant-memory/scripts/clean_memories_tr.py`

**Purpose:** Batch-clean existing points

**Usage:**
```bash
# Preview changes
python3 clean_memories_tr.py --dry-run

# Clean all
python3 clean_memories_tr.py --execute

# Clean 100 (test)
python3 clean_memories_tr.py --execute --limit 100
```

**Cleans:**
- `**bold**` → plain text
- `|tables|` → removed
- `` `code` `` → plain text
- `---` rules → removed
- `# headers` → removed

---

### 3. Timer Curator

**File:** `tr-continuous/curator_timer.py`

**Schedule:** Every 30 minutes (cron)

**Flow:**
1. Query uncurated memories from `memories_tr`
2. Send batch to qwen3 (max 100)
3. Extract gems → store to `gems_tr`
4. Mark memories as `curated: true`

**Config:** `tr-continuous/curator_config.json`
```json
{
  "timer_minutes": 30,
  "max_batch_size": 100
}
```

**Logs:** `/var/log/true-recall-timer.log`

---

### 4. Curation Model Comparison

**Current:** `qwen3:4b-instruct`

| Metric | 4b | 30b |
|--------|----|----|
| Speed | ~10-30s per batch | **~3.3s** (tested 2026-02-24) |
| JSON reliability | ⚠️ Needs fallback | ✅ Native |
| Context quality | Basic extraction | ✅ Nuanced |
| Snippet accuracy | ~80% | ✅ Expected: 95%+ |

**30b Benchmark (2026-02-24):**
- Load: 108ms
- Prompt eval: 49ms (1,576 tok/s)
- Generation: 2.9s (233 tokens, 80 tok/s)
- **Total: 3.26s**

**Trade-offs:**
- **4b:** Faster batch processing, lightweight, catches explicit decisions
- **30b:** Deeper context, better inference, ~3x slower but superior quality

**Gem Quality Comparison (Sample Review):**

| Aspect | 4b | 30b |
|--------|----|----|
| **Context depth** | "Extracted via fallback" | Explains *why* decisions were made |
| **Confidence scores** | 0.7-0.85 | 0.9-0.97 |
| **Snippet accuracy** | ~80% (wrong source) | ✅ 95%+ (relevant quotes) |
| **Categories** | Generic "extracted" | Specific: knowledge, technical, decision |
| **Example** | "User implemented BorgBackup" (no context) | "User selected mxbai... due to top MTEB score of 66.5" (explains reasoning) |

**Verdict:** 30b produces significantly higher quality gems — richer context, accurate snippets, and captures architectural intent, not just surface facts.

---

### 5. OpenClaw Compactor Configuration

**Status:** ✅ Applied

**Goal:** Minimal overhead — just remove context, do nothing else.

**Config Applied:**
```json5
{
  agents: {
    defaults: {
      compaction: {
        mode: "default",              // "default" or "safeguard"
        reserveTokensFloor: 0,        // Disable safety floor (default: 20000)
        memoryFlush: {
          enabled: false              // Disable silent .md file writes
        }
      }
    }
  }
}
```

**What this does:**
- `mode: "default"` — Standard summarization (faster)
- `reserveTokensFloor: 0` — Allow aggressive settings (disables 20k minimum)
- `memoryFlush.enabled: false` — No silent "write memory" turns

**Note:** `reserveTokens` and `keepRecentTokens` are Pi runtime settings, not configurable via `agents.defaults.compaction`. They are set per-model in `contextWindow`/`contextTokens`.

---

### 6. Configuration Options Reference

**All configurable options with defaults:**

| Option | Default | Description |
|--------|---------|-------------|
| **Embedding model** | `mxbai-embed-large` | Model for generating gem embeddings. `mxbai` = higher accuracy (MTEB 66.5). `snowflake` = faster processing. |
| **Timer interval** | `5` minutes | How often the curator runs. `5 min` = fast backlog clearing. `30 min` = balanced. `60 min` = minimal overhead. |
| **Batch size** | `100` | Max memories sent to curator per run. Higher = fewer API calls but more memory usage. |
| **Max gems per run** | *(unlimited)* | Hard limit on gems extracted per batch. Not set by default — extracts all found gems. |
| **Qdrant URL** | `http://<QDRANT_IP>:6333` | Vector database endpoint. Change if Qdrant runs on different host/port. |
| **Ollama URL** | `http://<OLLAMA_IP>:11434` | LLM endpoint for gem extraction. Change if Ollama runs elsewhere. |
| **Curator LLM** | `qwen3:30b-a3b-instruct` | Model for extracting gems. `30b` = best quality (~3s). `4b` = faster but needs JSON fallback. |
| **User ID** | `rob` | Owner identifier for memories. Used for filtering and multi-user setups. |
| **Source collection** | `memories_tr` | Qdrant collection for raw captured memories. |
| **Target collection** | `gems_tr` | Qdrant collection for curated gems (injected into context). |
| **Watcher service** | `enabled` | Real-time capture daemon. Reads session JSONL and writes to Qdrant. |
| **Cron timer** | `enabled` | Periodic curation job. Runs `curator_timer.py` on schedule. |
| **Log path** | `/var/log/true-recall-timer.log` | Where curator output is written. Check with `tail -f`. |
| **Dry-run mode** | `disabled` | Test mode — shows what would be curated without writing to Qdrant. |

**OpenClaw-side options:**
| Option | Default | Description |
|--------|---------|-------------|
| **Compactor mode** | `default` | How context is summarized. `default` = fast standard. `safeguard` = chunked for very long sessions. |
| **Memory flush** | `disabled` | If enabled, writes silent "memory" turn before compaction. Adds overhead — disabled for minimal lag. |
| **Context pruning** | `cache-ttl` | Removes old tool results from context. `cache-ttl` = prunes hourly. `off` = no pruning. |

---

### 7. Embedding Models

**Current Setup:**
- `memories_tr`: `snowflake-arctic-embed2` (capture)
- `gems_tr`: `snowflake-arctic-embed2` (recall) ✅ **FIXED** - Both collections now use same model

**Note:** Previously used `mxbai-embed-large` for gems, but this caused embedding model mismatch. Fixed 2026-02-25.

---

### 6. memory-qdrant Plugin

**Location:** `~/.openclaw/extensions/memory-qdrant/`

**Config (openclaw.json):**
```json
{
  "collectionName": "gems_tr",
  "captureCollection": "memories_tr",
  "autoRecall": true,
  "autoCapture": true
}
```

**Functions:**
- **Recall:** Searches `gems_tr`, injects gems (hidden)
- **Capture:** Session-level to `memories_tr` (backup)

---

## Files & Locations

### Core Project

```
~/.openclaw/workspace/.local_projects/true-recall-v2/
├── README.md                    # This file
├── session.md                   # Detailed notes
├── curator-prompt.md            # Extraction prompt
├── tr-daily/
│   └── curate_from_qdrant.py   # Daily curator
└── shared/
```

### New Files (2026-02-24)

| File | Purpose |
|------|---------|
| `tr-continuous/curator_timer.py` | Timer curator (v2.2) |
| `tr-continuous/curator_config.json` | Curator settings |
| `tr-continuous/migrate_add_curated.py` | Migration script |
| `skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` | Capture daemon |
| `skills/qdrant-memory/mem-qdrant-watcher.service` | Systemd service |

### Archived Files (v2.1)

| File | Status | Note |
|------|--------|------|
| `tr-daily/curate_from_qdrant.py` | 📦 Archived | Replaced by timer |
| `tr-continuous/curator_by_count.py` | 📦 Archived | Replaced by timer |

### System Files

| File | Purpose |
|------|---------|
| `~/.openclaw/extensions/memory-qdrant/` | Plugin code |
| `~/.openclaw/openclaw.json` | Configuration |
| `/etc/systemd/system/mem-qdrant-watcher.service` | Service file |

---

## Configuration

### memory-qdrant Plugin

**File:** `~/.openclaw/openclaw.json`

```json
{
  "memory-qdrant": {
    "config": {
      "autoCapture": true,
      "autoRecall": true,
      "collectionName": "gems_tr",
      "captureCollection": "memories_tr",
      "embeddingModel": "snowflake-arctic-embed2",
      "maxRecallResults": 2,
      "minRecallScore": 0.7,
      "ollamaUrl": "http://<OLLAMA_IP>:11434",
      "qdrantUrl": "http://<QDRANT_IP>:6333"
    },
    "enabled": true
  }
}
```

### Gateway Control UI (OpenClaw 2026.2.23)

```json
{
  "gateway": {
    "controlUi": {
      "allowedOrigins": ["*"],
      "allowInsecureAuth": false,
      "dangerouslyDisableDeviceAuth": true
    }
  }
}
```

---

## Validation

### Check Collections

```bash
# Count points
curl -s http://<QDRANT_IP>:6333/collections/memories_tr | jq '.result.points_count'
curl -s http://<QDRANT_IP>:6333/collections/gems_tr | jq '.result.points_count'

# View recent captures
curl -s -X POST http://<QDRANT_IP>:6333/collections/memories_tr/points/scroll \
  -H "Content-Type: application/json" \
  -d '{"limit": 3, "with_payload": true}' | jq '.result.points[].payload.content'
```

### Check Services

```bash
# Watcher
sudo systemctl status mem-qdrant-watcher
sudo journalctl -u mem-qdrant-watcher -n 20

# OpenClaw
openclaw status
openclaw gateway status
```

### Test Capture

Send a message, then check:
```bash
# Should increase by 1-2 points
curl -s http://<QDRANT_IP>:6333/collections/memories_tr | jq '.result.points_count'
```

---

## Troubleshooting

### Watcher Not Capturing

```bash
# Check logs
sudo journalctl -u mem-qdrant-watcher -f

# Verify dependencies
curl http://<QDRANT_IP>:6333/          # Qdrant
curl http://<OLLAMA_IP>:11434/api/tags # Ollama
```

### Plugin Not Loading

```bash
# Validate config
openclaw config validate

# Check logs
tail /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant

# Restart gateway
openclaw gateway restart
```

### Gateway Won't Start (OpenClaw 2026.2.23+)

**Error:** `non-loopback Control UI requires gateway.controlUi.allowedOrigins`

**Fix:** Add to `openclaw.json`:
```json
"gateway": {
  "controlUi": {
    "allowedOrigins": ["*"]
  }
}
```

---

## Status Summary

| Component | Status | Notes |
|-----------|--------|-------|
| Real-time watcher | ✅ Active | PID 1748, capturing |
| memories_tr | ✅ 12,378 pts | All tagged `curated: false` |
| gems_tr | ✅ 5 pts | Injection ready |
| Timer curator | ✅ Deployed | Every 30 min via cron |
| Plugin injection | ✅ **WORKING** | Context injection verified - score 0.587 |
| Migration | ✅ Complete | 12,378 memories |

**Logs:** `tail /var/log/true-recall-timer.log`

**Next:** Monitor first timer run

---

## Roadmap

### Planned Features

| Feature | Status | Description |
|---------|--------|-------------|
| Interactive install script | ⏳ Planned | Prompts for embedding model, timer interval, batch size, endpoints |
| Single embedding model | ⏳ Planned | Option to use one model for both collections |
| Configurable thresholds | ⏳ Planned | Per-user customization via prompts |

**Install script will prompt for:**
1. **Embedding model** — snowflake (fast) vs mxbai (accurate)
2. **Timer interval** — 5 min / 30 min / hourly
3. **Batch size** — 50 / 100 / 500 memories
4. **Endpoints** — Qdrant/Ollama URLs
5. **User ID** — for multi-user setups

---

**Maintained by:** Rob  
**AI Assistant:** Kimi 🎙️  
**Version:** 2026.02.24-v2.2