true-recall/session.md

TrueRecall v2 - Session Notes

Last Updated: 2026-02-24 19:02 CST Status: ✅ Active & Verified Version: v2.2 (Timer-based curation deployed)

---

Session End (18:09 CST)

Reason: User starting new session

Current State:

• Real-time watcher: ✅ Active (capturing live)
• Timer curator: ✅ Deployed (every 30 min via cron)
• Daily curator: ❌ Removed (replaced by timer)
• Total memories: 12,378 (all tagged with curated: false)
• Gems: 5 (from Feb 18 test)

Next session start: Read this file, then check:

# Quick status
python3 ~/.openclaw/workspace/.projects/true-recall-v2/tr-continuous/curator_by_count.py --status
sudo systemctl status mem-qdrant-watcher
curl -s http://<QDRANT_IP>:6333/collections/memories_tr | jq '.result.points_count'

---

Executive Summary

TrueRecall v2 is a complete memory system with real-time capture, daily curation, and context injection. All components are operational.

---

Current State (Verified 18:09 CST)

Qdrant Collections

Collection	Points	Purpose	Status
memories_tr	12,378	Full text (live capture)	✅ Active
gems_tr	5	Curated gems (injection)	✅ Active
true_recall	existing	Legacy archive	📦 Preserved
kimi_memories	12,223	Original backup	📦 Preserved

Note: All memories tagged with curated: false for timer curator.

Services

Service	Status	Uptime
mem-qdrant-watcher	✅ Active	30+ min
OpenClaw Gateway	✅ Running	2026.2.23
memory-qdrant plugin	✅ Loaded	recall: gems_tr, capture: memories_tr

---

Architecture

v2.2: Timer-Based Curation (DEPLOYED)

Data Flow:

┌─────────────────┐     ┌──────────────────────┐     ┌─────────────┐
│  OpenClaw Chat  │────▶│ Real-Time Watcher    │────▶│ memories_tr │
│  (Session JSONL)│     │ (Python daemon)      │     │ (Qdrant)    │
└─────────────────┘     └──────────────────────┘     └──────┬──────┘
                                                          │
                                                          │ Every 30 min
                                                          ▼
                                                ┌──────────────────┐
                                                │  Timer Curator   │
                                                │   (cron/qwen3)   │
                                                └────────┬─────────┘
                                                         │
                                                         ▼
                                                ┌──────────────────┐
                                                │    gems_tr       │
                                                │   (Qdrant)       │
                                                └────────┬─────────┘
                                                         │
                                              Per turn   │
                                                         ▼
                                                ┌──────────────────┐
                                                │ memory-qdrant    │
                                                │     plugin       │
                                                └──────────────────┘

Key Changes:

• ✅ Replaced daily 2:45 AM batch with 30-minute timer
• ✅ All memories tagged curated: false on write
• ✅ Migration completed for 12,378 existing memories
• ✅ No Redis dependency (direct Qdrant only)

---

Components

Curation Mode: Timer-Based (DEPLOYED v2.2)

Setting	Value	Adjustable
Trigger	Cron timer	✅
Interval	30 minutes	✅ Config file
Batch size	100 memories max	✅ Config file
Minimum	None (0 is OK)	—

Config: /tr-continuous/curator_config.json

{
  "timer_minutes": 30,
  "max_batch_size": 100,
  "user_id": "rob",
  "source_collection": "memories_tr",
  "target_collection": "gems_tr"
}

Cron:

*/30 * * * * cd .../tr-continuous && python3 curator_timer.py

Old modes deprecated:

• ❌ Turn-based (every N turns)
• ❌ Hybrid (timer + turn)
• ❌ Daily batch (2:45 AM)

1. Real-Time Watcher (Primary Capture)

Location: ~/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py

Function:

• Watches ~/.openclaw/agents/main/sessions/*.jsonl
• Parses every conversation turn in real-time
• Embeds with snowflake-arctic-embed2 (Ollama @ <OLLAMA_IP>)
• Stores directly to memories_tr (no Redis)
• Cleans content: Removes markdown, tables, metadata, thinking tags

Service: mem-qdrant-watcher.service

• Status: Active since 16:46:53 CST
• Systemd: Enabled, auto-restart

Log: journalctl -u mem-qdrant-watcher -f

---

2. Content Cleaner (Existing Data)

Location: ~/.openclaw/workspace/skills/qdrant-memory/scripts/clean_memories_tr.py

Function:

• Batch-cleans existing memories_tr points
• Removes: **bold**, |tables|, `code`, --- rules, # headers
• Flattens nested content dicts
• Rate-limited to prevent Qdrant overload

Usage:

# Dry run (preview)
python3 clean_memories_tr.py --dry-run

# Clean all
python3 clean_memories_tr.py --execute

# Clean limited (test)
python3 clean_memories_tr.py --execute --limit 100

---

3. Timer Curator (v2.2 - DEPLOYED)

Replaces: Daily curator (2:45 AM batch) and turn-based curator

Location: ~/.openclaw/workspace/.projects/true-recall-v2/tr-continuous/curator_timer.py

Schedule: Every 30 minutes (cron)

Flow:

1. Query uncurated memories (curated: false)
2. Send batch to qwen3 (max 100)
3. Extract gems using curator prompt
4. Store gems to gems_tr
5. Mark processed memories as curated: true

Files:

File	Purpose
curator_timer.py	Main curator script
curator_config.json	Adjustable settings
migrate_add_curated.py	One-time migration (completed)

Usage:

# Dry run (preview)
python3 curator_timer.py --dry-run

# Manual run
python3 curator_timer.py --config curator_config.json

Status: ✅ Deployed, first run will process ~12,378 existing memories

5. Silent Compacting (NEW - Concept)

Idea: Automatically remove old context from prompt when token limit approached.

Behavior:

• Trigger: Context window > 80% full
• Action: Remove oldest messages (silently)
• Preserve: Gems always kept, recent N turns kept
• Result: Seamless conversation without "compacting" notification

Config:

{
  "compacting": {
    "enabled": true,
    "triggerAtPercent": 80,
    "keepRecentTurns": 20,
    "preserveGems": true,
    "silent": true
  }
}

Status: ⏳ Concept only - requires OpenClaw core changes

6. memory-qdrant Plugin

Location: ~/.openclaw/extensions/memory-qdrant/

Config:

{
  "collectionName": "gems_tr",
  "captureCollection": "memories_tr",
  "autoRecall": true,
  "autoCapture": true
}

Function:

• Recall: Searches gems_tr, injects as context (hidden)
• Capture: Session-level capture to memories_tr (backup)

Status: Loaded, dual collection support working

---

Files & Locations

Core Project Files

~/.openclaw/workspace/.projects/true-recall-v2/
├── README.md                           # Architecture docs
├── session.md                          # This file
├── curator-prompt.md                   # Gem extraction prompt
├── tr-daily/                           # Daily batch curation
│   └── curate_from_qdrant.py          # Daily curator (2:45 AM)
├── tr-continuous/                      # Real-time curation (NEW)
│   ├── curator_by_count.py           # Turn-based curator
│   ├── curator_turn_based.py         # Alternative approach
│   ├── curator_cron.sh               # Cron wrapper
│   ├── turn-curator.service          # Systemd service
│   └── README.md                     # Documentation
└── shared/
    └── (shared resources)

New Files (2026-02-24 19:00)

File	Purpose
tr-continuous/curator_timer.py	Timer-based curator (deployed)
tr-continuous/curator_config.json	Curator settings
tr-continuous/migrate_add_curated.py	Migration script (completed)

Legacy Files (Pre-v2.2)

File	Status	Note
tr-daily/curate_from_qdrant.py	📦 Archived	Replaced by timer
tr-continuous/curator_by_count.py	📦 Archived	Replaced by timer
tr-continuous/curator_turn_based.py	📦 Archived	Replaced by timer

System Locations

File	Purpose
~/.openclaw/extensions/memory-qdrant/	Plugin code
~/.openclaw/openclaw.json	Plugin configuration
/etc/systemd/system/mem-qdrant-watcher.service	Systemd service

---

Changes Made Today (2026-02-24 19:00)

1. Timer Curator Deployed (v2.2)

• Created curator_timer.py — simplified timer-based curation
• Created curator_config.json — adjustable settings
• Removed daily 2:45 AM cron job
• Added */30 * * * * cron timer
• Status: ✅ Deployed, logs to /var/log/true-recall-timer.log

2. Migration Completed

• Created migrate_add_curated.py
• Tagged 12,378 existing memories with curated: false
• Updated watcher to add curated: false to new memories
• Status: ✅ Complete

3. Simplified Architecture

• ❌ Removed turn-based curator complexity
• ❌ Removed daily batch processing
• ✅ Single timer trigger every 30 minutes
• ✅ No minimum threshold (processes 0-N memories)

---

Configuration

memory-qdrant Plugin

File: ~/.openclaw/openclaw.json

{
  "memory-qdrant": {
    "config": {
      "autoCapture": true,
      "autoRecall": true,
      "collectionName": "gems_tr",
      "captureCollection": "memories_tr",
      "embeddingModel": "snowflake-arctic-embed2",
      "maxRecallResults": 2,
      "minRecallScore": 0.7,
      "ollamaUrl": "http://<OLLAMA_IP>:11434",
      "qdrantUrl": "http://<QDRANT_IP>:6333"
    },
    "enabled": true
  }
}

Gateway (OpenClaw Update Fix)

{
  "gateway": {
    "controlUi": {
      "allowedOrigins": ["*"],
      "allowInsecureAuth": false,
      "dangerouslyDisableDeviceAuth": true
    }
  }
}

---

Validation Commands

Check Collections

# Points count
curl -s http://<QDRANT_IP>:6333/collections/memories_tr | jq '.result.points_count'
curl -s http://<QDRANT_IP>:6333/collections/gems_tr | jq '.result.points_count'

# Recent points
curl -s -X POST http://<QDRANT_IP>:6333/collections/memories_tr/points/scroll \
  -H "Content-Type: application/json" \
  -d '{"limit": 5, "with_payload": true}' | jq '.result.points[].payload.content'

Check Services

# Watcher status
sudo systemctl status mem-qdrant-watcher

# Watcher logs
sudo journalctl -u mem-qdrant-watcher -n 20

# OpenClaw status
openclaw status

---

Troubleshooting

Issue: Watcher Not Capturing

Check:

1. Service running? systemctl status mem-qdrant-watcher
2. Logs: journalctl -u mem-qdrant-watcher -f
3. Qdrant accessible? curl http://<QDRANT_IP>:6333/
4. Ollama accessible? curl http://<OLLAMA_IP>:11434/api/tags

Issue: Cleaner Fails

Common causes:

• Qdrant connection timeout (add time.sleep(0.1) between batches)
• Nested content dicts (handled in updated script)
• Type errors (non-string content — handled)

Issue: Plugin Not Loading

Check:

1. openclaw.json syntax valid? openclaw config validate
2. Plugin compiled? cd ~/.openclaw/extensions/memory-qdrant && npx tsc
3. Gateway logs: tail /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log

---

Cron Schedule (Updated v2.2)

Time	Job	Script	Status
Every 30 min	Timer curator	tr-continuous/curator_timer.py	✅ Active
Per turn	Capture	mem-qdrant-watcher	✅ Daemon
Per turn	Injection	memory-qdrant plugin	✅ Active

Removed:

• ❌ 2:45 AM daily curator
• ❌ Every-minute turn curator check

---

Next Steps

Immediate

• ⏳ Monitor first timer run (logs: /var/log/true-recall-timer.log)
• ⏳ Validate gem extraction quality from timer curator
• ⏳ Archive old curator scripts if timer works

Completed ✅

• ✅ Compactor config — Minimal overhead: mode: default, reserveTokensFloor: 0, memoryFlush: false

Future

• ⏳ Curator tuning based on timer results
• ⏳ Silent compacting (requires OpenClaw core changes)

Planned Features (Backlog)

• ⏳ Interactive install script — Prompts for embedding model, timer interval, batch size, endpoints
• ⏳ Single embedding model option — Use one model for both collections
• ⏳ Configurable thresholds — Per-user customization via prompts

Compactor Settings (Applied):

{
  agents: {
    defaults: {
      compaction: {
        mode: "default",
        reserveTokensFloor: 0,
        memoryFlush: { enabled: false }
      }
    }
  }
}

Note: Only mode, reserveTokensFloor, and memoryFlush are valid under agents.defaults.compaction. Other settings are Pi runtime parameters.

Install script prompts:

1. Embedding model (snowflake vs mxbai)
2. Timer interval (5 min / 30 min / hourly)
3. Batch size (50 / 100 / 500)
4. Qdrant/Ollama URLs
5. User ID

---

Session Recovery

If starting fresh:

1. Read README.md for architecture overview
2. Check service status: sudo systemctl status mem-qdrant-watcher
3. Check timer curator: tail /var/log/true-recall-timer.log
4. Verify collections: curl http://<QDRANT_IP>:6333/collections

---

Last Verified: 2026-02-24 19:29 CST Version: v2.2 (30b curator, install script planned)