SpeedyFoxAi/true-recall-gems
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update docs: watcher fix, plugin capture fix (2026-02-25)

Browse Source 
- Fixed watcher stuck on old session bug (restarted service)
- Fixed plugin capture 0 exchanges (added extractMessageText for OpenAI content arrays)
- Updated README, session.md, function_check.md, audit_checklist.md
- Verified: 9 exchanges captured per session
This commit is contained in:
root
2026-02-25 12:45:27 -06:00
parent abc5498f60
commit 87a390901d
8 changed files with 1234 additions and 202 deletions
Download Patch File Download Diff File Expand all files Collapse all files

208
README.md
View File

@@ -3,7 +3,7 @@
**Project:** Gem extraction and memory recall system
**Status:** ✅ Active & Verified
**Location:** `~/.openclaw/workspace/.projects/true-recall-v2/`
**Last Updated:** 2026-02-24 19:02 CST
**Last Updated:** 2026-02-25 12:04 CST
---
@@ -38,6 +38,29 @@ curl -s http://<QDRANT_IP>:6333/collections | jq '.result.collections[].name'
---
## Recent Fixes (2026-02-25 12:41 CST)
| Issue | Root Cause | Fix Applied |
|-------|------------|-------------|
| **Watcher stuck on old session** | Watcher only switched sessions when file deleted, old sessions persisted | ✅ Restarted service, now follows current session |
| **Plugin capture 0 exchanges** | OpenClaw uses OpenAI content format (array of items), plugin expected string | ✅ Added `extractMessageText()` to extract text from `type: "text"` items |
### Validation Results
**Plugin capture:**
```
Before: parsed 14 user, 84 assistant messages, 0 exchanges
After: parsed 17 user, 116 assistant messages, 9 exchanges ✅
```
**Watcher:**
```
Before: Watching old session (old session ID from Feb 24)
After: Watching current session (current session ID from Feb 25) ✅
```
---
## Overview
TrueRecall v2 extracts "gems" (key insights) from conversations and injects them as context. It consists of three layers:
@@ -54,8 +77,8 @@ TrueRecall v2 extracts "gems" (key insights) from conversations and injects them
| Collection | Points | Purpose | Status |
|------------|--------|---------|--------|
| `memories_tr` | **12,378** | Full text (live capture) | ✅ Active |
| `gems_tr` | **5** | Curated gems (injection) | ✅ Active |
| `memories_tr` | **12,729** | Full text (live capture) | ✅ Active |
| `gems_tr` | **14+** | Curated gems (injection) | ✅ **WORKING** - Context injection verified |
**All memories tagged with `curated: false` for timer curation.**
@@ -63,8 +86,8 @@ TrueRecall v2 extracts "gems" (key insights) from conversations and injects them
| Service | Status | Details |
|---------|--------|---------|
| `mem-qdrant-watcher` | ✅ Active | PID 1748, capturing |
| Timer curator | ✅ Deployed | Every 30 min via cron |
| `mem-qdrant-watcher` | ✅ Active | PID 234, capturing |
| Timer curator | ✅ Deployed | Every 5 min via cron |
| OpenClaw Gateway | ✅ Running | Version 2026.2.23 |
| memory-qdrant plugin | ✅ Loaded | recall: gems_tr |
@@ -76,8 +99,8 @@ TrueRecall v2 extracts "gems" (key insights) from conversations and injects them
|---------|---------------|---------------|---------------|
| **Storage** | Redis | Redis + Qdrant | Qdrant only |
| **Capture** | Session batch | Session batch | Real-time |
| **Curation** | Manual | Daily 2:45 AM | Timer (5 min) |
| **Embedding** | — | snowflake | snowflake + mxbai |
| **Curation** | Manual | Daily 2:45 AM | Timer (5 min) ✅ |
| **Embedding** | — | snowflake | snowflake-arctic-embed2 ✅ |
| **Curator LLM** | — | qwen3:4b | qwen3:30b |
| **State tracking** | — | — | `curated` tag |
| **Batch size** | — | 24h worth | Configurable |
@@ -138,7 +161,7 @@ TrueRecall v2 extracts "gems" (key insights) from conversations and injects them
**File:** `skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
**What it does:**
- Watches `~/.openclaw/agents/main/sessions/*.jsonl`
- Watches `/root/.openclaw/agents/main/sessions/*.jsonl`
- Parses each turn (user + AI)
- Embeds with `snowflake-arctic-embed2`
- Stores to `memories_tr` instantly
@@ -246,132 +269,7 @@ python3 clean_memories_tr.py --execute --limit 100
---
### 5. Semantic Deduplication (Similarity Checking)
**Why:** Smaller models (4b) often extract duplicate or near-duplicate gems. Without checking, your `gems_tr` collection fills with redundant entries.
**The Problem:**
- "User decided on Redis" and "User selected Redis for caching" are the same gem
- Smaller models lack nuance — they extract surface variations as separate gems
- Over time, 30-50% of gems may be duplicates
**Solution: Semantic Similarity Check**
Before inserting a new gem:
1. Embed the candidate gem text
2. Search `gems_tr` for similar embeddings (past 24h)
3. If similarity > 0.85, SKIP (don't insert)
4. If similarity 0.70-0.85, MERGE (update existing with richer context)
5. If similarity < 0.70, INSERT (new unique gem)
**Implementation Options:**
#### Option A: Built-in Curator Check (Recommended)
Modify `curator_timer.py` to add pre-insertion similarity check:
```python
import numpy as np
from qdrant_client import QdrantClient
qdrant = QdrantClient("http://<QDRANT_IP>:6333")
def is_duplicate(gem_text: str, user_id: str = "rob", threshold: float = 0.85) -> bool:
"""Check if similar gem exists in past 24h"""
# Embed the candidate
response = requests.post(
"http://<OLLAMA_IP>:11434/api/embeddings",
json={"model": "mxbai-embed-large", "prompt": gem_text}
)
embedding = response.json()["embedding"]
# Search for similar gems
results = qdrant.search(
collection_name="gems_tr",
query_vector=embedding,
limit=3,
query_filter={
"must": [
{"key": "user_id", "match": {"value": user_id}},
{"key": "timestamp", "range": {"gte": "now-24h"}}
]
}
)
# Check similarity scores
for result in results:
if result.score > threshold:
return True # Duplicate found
return False
# In main loop, before inserting:
if is_duplicate(gem["gem"]):
log.info(f"Skipping duplicate gem: {gem['gem'][:50]}...")
continue
```
**Pros:** Catches duplicates at source, no extra jobs
**Cons:** Adds ~50-100ms per gem (embedding call)
#### Option B: Periodic AI Review (Subagent Task)
Have a subagent periodically review and merge duplicates:
```bash
# Run weekly via cron
0 3 * * 0 cd <PROJECT_PATH> && python3 dedup_gems.py
```
**dedup_gems.py approach:**
1. Load all gems from past 7 days
2. Group by semantic similarity (clustering)
3. For each cluster > 1 gem:
- Keep highest confidence gem as primary
- Merge context from others into primary
- Delete duplicates
**Pros:** Can use reasoning model for nuanced merging
**Cons:** Batch job, duplicates exist until cleanup runs
#### Option C: Real-time Watcher Hook
Add deduplication to the real-time watcher before memories are even stored:
```python
# In watcher, before upsert to memories_tr
if is_similar_to_recent(memory_text, window="1h"):
memory["duplicate_of"] = similar_id # Tag but still store
```
**Pros:** Prevents duplicate memories upstream
**Cons:** Memories may differ slightly even if gems would be same
**Recommendation by Model:**
| Model | Recommended Approach | Reason |
|-------|---------------------|--------|
| **4b** | **Option A + B** | Built-in check prevents duplicates; periodic review catches edge cases |
| **30b** | **Option B only** | 30b produces fewer duplicates; weekly review sufficient |
| **Production** | **Option A** | Best balance of prevention and performance |
**Configuration:**
Add to `curator_config.json`:
```json
{
"deduplication": {
"enabled": true,
"similarity_threshold": 0.85,
"lookback_hours": 24,
"mode": "skip" // "skip", "merge", or "flag"
}
}
```
---
### 6. OpenClaw Compactor Configuration
### 5. OpenClaw Compactor Configuration
**Status:** ✅ Applied
@@ -399,25 +297,11 @@ Add to `curator_config.json`:
- `reserveTokensFloor: 0` — Allow aggressive settings (disables 20k minimum)
- `memoryFlush.enabled: false` — No silent "write memory" turns
**Known Issue: UI Glitch During Compaction**
When compaction runs, the Control UI may briefly behave unexpectedly:
- Typed text may not appear immediately after hitting Enter
- Messages may render out of order briefly
- UI "catches up" within 1-2 seconds after compaction completes
**Why:** Compaction replaces the full conversation history with a summary. The UI's WebSocket state can get briefly out of sync during this transition.
**Workaround:**
- Wait 2-3 seconds after hitting Enter during compaction
- Or hard refresh (Ctrl+Shift+R) if UI seems stuck
- **Note:** This is an OpenClaw Control UI limitation — cannot be fixed from TrueRecall side at this time.
**Note:** `reserveTokens` and `keepRecentTokens` are Pi runtime settings, not configurable via `agents.defaults.compaction`. They are set per-model in `contextWindow`/`contextTokens`.
---
### 7. Configuration Options Reference
### 6. Configuration Options Reference
**All configurable options with defaults:**
@@ -447,23 +331,19 @@ When compaction runs, the Control UI may briefly behave unexpectedly:
---
### 8. Embedding Models
### 7. Embedding Models
**Current Setup:**
- `memories_tr`: `snowflake-arctic-embed2` (capture similarity)
- `gems_tr`: `mxbai-embed-large` (recall similarity)
- `memories_tr`: `snowflake-arctic-embed2` (capture)
- `gems_tr`: `snowflake-arctic-embed2` (recall) ✅ **FIXED** - Both collections now use same model
**Rationale:**
- mxbai has higher MTEB score (66.5) for semantic search
- snowflake is faster for high-volume capture
**Note:** For simplicity, a single embedding model could be used for both collections. This would reduce complexity and memory overhead, though with slightly lower recall performance.
**Note:** Previously used `mxbai-embed-large` for gems, but this caused embedding model mismatch. Fixed 2026-02-25.
---
### 9. memory-qdrant Plugin
### 6. memory-qdrant Plugin
**Location:** `~/.openclaw/extensions/memory-qdrant/`
**Location:** `/root/.openclaw/extensions/memory-qdrant/`
**Config (openclaw.json):**
```json
@@ -516,9 +396,9 @@ When compaction runs, the Control UI may briefly behave unexpectedly:
| File | Purpose |
|------|---------|
| `~/.openclaw/extensions/memory-qdrant/` | Plugin code |
| `~/.openclaw/openclaw.json` | Configuration |
| `/etc/systemd/system/mem-qdrant-watcher.service` | Service file |
| `/root/.openclaw/extensions/memory-qdrant/` | Plugin code |
| `/root/.openclaw/openclaw.json` | Configuration |
| `<SYSTEMD_PATH>/mem-qdrant-watcher.service` | Service file |
---
@@ -526,7 +406,7 @@ When compaction runs, the Control UI may briefly behave unexpectedly:
### memory-qdrant Plugin
**File:** `~/.openclaw/openclaw.json`
**File:** `/root/.openclaw/openclaw.json`
```json
{
@@ -649,7 +529,7 @@ openclaw gateway restart
| memories_tr | ✅ 12,378 pts | All tagged `curated: false` |
| gems_tr | ✅ 5 pts | Injection ready |
| Timer curator | ✅ Deployed | Every 30 min via cron |
| Plugin injection | ✅ Working | Uses gems_tr |
| Plugin injection | ✅ **WORKING** | Context injection verified - score 0.587 |
| Migration | ✅ Complete | 12,378 memories |
**Logs:** `tail /var/log/true-recall-timer.log`