true-recall/README.md.bak.2026-02-24-1518-final

# TrueRecall v2

**Project:** Gem extraction and memory recall system  
**Status:** ✅ Active  
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/`

---

## Table of Contents

- [Overview](#overview)
- [Architecture](#architecture)
- [Collections](#collections-qdrant)
- [Components](#components)
- [Process Flow](#process-flow)
- [Configuration](#configuration)
- [Validation Commands](#validation-commands)
- [Troubleshooting](#troubleshooting)
- [Related Projects](#related-projects)
- [Backups](#backups)
- [Final Status](#final-status-2026-02-24-1508)

---

## Overview

TrueRecall extracts "gems" (key insights) from conversations and stores them for context injection. It's the memory curation system that powers Kimi's contextual awareness.

### Current Flow

```
1. Conversation happens → Real-time watcher → memories_tr (Qdrant)
2. Daily (2:45 AM) → Curator reads memories_tr → extracts gems
3. Gems stored → gems_tr collection (Qdrant)
4. On each turn → memory-qdrant plugin injects gems as context
```

**Verified:** 2026-02-24 — Real-time watcher capturing (12,223 → 12,228 points)

---

## Architecture

### Collections (Qdrant)

| Collection | Purpose | Content | Status |
|------------|---------|---------|--------|
| `memories_tr` | Full text storage | Every conversation turn (migrated from `kimi_memories`) | ✅ Active |
| `gems_tr` | Gems (extracted insights) | Curated key points (for injection) | ✅ Active |
| `true_recall` | Legacy gems | Archive of previously extracted gems | 📦 Preserved |
| `kimi_memories` | Original collection | Backup (12,223 points preserved) | 📦 Preserved |

### Migration Script

| File | Purpose |
|------|---------|
| `migrate_memories.py` | Migrate data from `kimi_memories` → `memories_tr` with cleaning |

**Usage:**
```bash
python3 migrate_memories.py
```

**What it does:**
- Reads all points from `kimi_memories`
- Cleans content (removes metadata, thinking tags)
- Stores to `memories_tr`
- Preserves original `kimi_memories`

---

### Components

| Component | Location | Purpose |
|-----------|----------|---------|
| **memory-qdrant plugin** | `/root/.openclaw/extensions/memory-qdrant/` | Injects gems as context |
| **Curation script** | `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` | Extracts gems |
| **Curator prompt** | `/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md` | Instructions for gem extraction |

---

## File Locations

### Core Files

```
/root/.openclaw/workspace/.projects/true-recall-v2/
├── README.md                           # This file
├── session.md                          # Development notes
├── curator-prompt.md                   # Gem extraction prompt
│
├── tr-daily/                           # Daily curation
│   └── curate_from_qdrant.py          # Main curator script
│
├── tr-compact/                         # (Reserved for v2 expansion)
│   └── hook.py                         # Compaction hook (not active)
│
├── tr-worker/                          # (Reserved for v2 expansion)
│   └── worker.py                       # Background worker (not active)
│
└── shared/                             # Shared resources
```

### Plugin Files (Located in OpenClaw Extensions)

> **Note:** These files live in `/root/.openclaw/extensions/`, not in this project folder. Documented here for reference.

| Actual Location | Project Reference |
|----------------|-------------------|
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code (capture + injection) |
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
| `/root/.openclaw/openclaw.json` | Plugin configuration |

### Configuration

| File | Purpose |
|------|---------|
| `/root/.openclaw/openclaw.json` | Plugin config (collectionName: gems_tr) |
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code |
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |

### Cron Jobs

| Time | Command | Purpose |
|------|---------|---------|
| 2:45 AM | `curate_memories.py` | Daily gem extraction (v1) → stores to gems_tr |
| 2:50 AM | `archive_to_memories_tr.py` | Archive to memories_tr |

View cron: `crontab -l`

---

## Process Flow

### Step 1: Capture (Real-Time Watcher)

```
OpenClaw Session ──→ Real-Time Watcher ──→ Qdrant (memories_tr)
```

**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`

**What it does:**
- Watches session JSONL files in real-time
- Parses each conversation turn (user + AI)
- Embeds with `snowflake-arctic-embed2` via Ollama
- Stores directly to `memories_tr` collection
- Cleans content (removes metadata, thinking tags)

**Systemd Service:** `mem-qdrant-watcher.service`

**Status:** ✅ Created, ready to deploy

### Step 2: Curation (Daily)

```
2:45 AM → curate_memories.py → memories_tr → qwen3 → gems_tr
```

**Location:** `/root/.openclaw/workspace/.projects/true-recall-v1/tr-process/curate_memories.py`

**What it does:**
1. Reads from Redis buffer (`mem:rob`)
2. Passes to qwen3 (curator model)
3. Extracts gems using prompt
4. Stores gems to `gems_tr` collection

### Step 3: Injection

```
User message → memory-qdrant plugin → Search gems_tr → Inject as context
```

**Location:** `/root/.openclaw/extensions/memory-qdrant/index.ts`

**What it does:**
1. Listens to `before_agent_start` events
2. Embeds current prompt
3. Searches `gems_tr` collection
4. Returns `{ prependContext: "..." }` with gems
5. Gems appear in my context (hidden from UI)

---

## Configuration

### memory-qdrant Plugin

```json
{
  "autoCapture": true,
  "autoRecall": true,
  "collectionName": "gems_tr",
  "embeddingModel": "snowflake-arctic-embed2",
  "maxRecallResults": 2,
  "minRecallScore": 0.7
}
```

**Key setting:** `collectionName: "gems_tr"` — tells plugin to inject gems, not full text.

---

## Gem Format

```json
{
  "gem": "User prefers dark mode for interface",
  "context": "Discussed UI theme options",
  "snippet": "rob: I want dark mode\nKimi: Done",
  "categories": ["preference"],
  "importance": "high",
  "confidence": 0.95,
  "date": "2026-02-24",
  "conversation_id": "uuid",
  "turn_range": "5-7"
}
```

---

## Validation Commands

### Check Qdrant Collections

```bash
curl -s "http://10.0.0.40:6333/collections" | python3 -m json.tool
```

### Check Collection Points

```bash
# memories_tr (full text)
curl -s -X POST "http://10.0.0.40:6333/collections/memories_tr/points/scroll" \
  -H "Content-Type: application/json" \
  -d '{"limit": 5, "with_payload": true}' | python3 -m json.tool

# gems_tr (gems)
curl -s -X POST "http://10.0.0.40:6333/collections/gems_tr/points/scroll" \
  -H "Content-Type: application/json" \
  -d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
```

### Check Cron Jobs

```bash
crontab -l | grep -E "true|recall|curate"
```

### Check Plugin Logs

```bash
tail -50 /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant
```

---

## Troubleshooting

### Issue: `<relevant-memories>` showing in UI

**Cause:** Plugin was using `memories_tr` (full text) instead of `gems_tr` (gems)

**Fix:** Set `"collectionName": "gems_tr"` in openclaw.json

**Reference:** KB entry `relevant-memories-ui-issue.md`

### Issue: No gems being extracted

**Check:**
1. Is memories_tr populated? `curl .../collections/memories_tr/points/scroll`
2. Is curator prompt valid? `cat curator-prompt.md`
3. Is qwen3 available? `curl -s http://10.0.0.10:11434/api/tags`

---

## Related Projects

| Project | Location | Purpose |
|---------|----------|---------|
| **true-recall-v1** | `/.projects/true-recall-v1/` | Original (Redis-based) |
| **memory-qdrant** | `/root/.openclaw/extensions/memory-qdrant/` | Plugin |
| **mem-redis** | `/root/.openclaw/workspace/skills/mem-redis/` | Redis utilities |

---

## Backups

| File | Backup Location |
|------|-----------------|
| openclaw.json | `/root/.openclaw/openclaw.json.bak.2026-02-24` |
| Plugin | `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24` |

---

## Final Status (2026-02-24 15:08)

| Component | Status |
|-----------|--------|
| Collection `kimi_memories` → `memories_tr` | ✅ Migrated (12,223 points) |
| Real-time watcher | ✅ **Deployed & verified** (12,228 points, +5 new) |
| Collection `gems_tr` | ✅ Active (5 gems stored) |
| Curator v2 | ✅ Working - tested with 327 turns |
| Config | ✅ Updated to `gems_tr` |
| Cron jobs | ✅ Cleaned |
| Documentation | ✅ Updated |

### Daily Schedule (Simplified)

| Time | Job | Flow |
|------|-----|------|
| Continuous | autoCapture | Conversation → `memories_tr` |
| 2:45 AM | Curator v2 | `memories_tr` → `gems_tr` |
| Each turn | Injection | `gems_tr` → Context |

**Redis usage:** Disabled for memory. Used only for `delayed:notifications` queue.

**Auto-Capture Status:** ✅ **Working** - Real-time watcher deployed and verified

---

**Last Updated:** 2026-02-24  
**Project Lead:** Rob  
**AI Assistant:** Kimi 🎙️