chore: Remove backup files and add .gitignore
- Remove tracked backup files (*.bak, *.neuralstream.bak) - Remove debug_curator.py - Add comprehensive .gitignore - Prevents future backup/debug files from being committed
This commit is contained in:
root
59
.gitignore
vendored
Normal file
59
.gitignore
vendored
Normal file
@@ -0,0 +1,59 @@
|
|||||||
|
# TrueRecall v2 - .gitignore
|
||||||
|
# Files that should not be committed to version control
|
||||||
|
|
||||||
|
# Backup files
|
||||||
|
*.bak
|
||||||
|
*.bak.*
|
||||||
|
*~
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
*.orig
|
||||||
|
*.save
|
||||||
|
|
||||||
|
# Debug and temporary files
|
||||||
|
debug_*
|
||||||
|
test_*
|
||||||
|
temp_*
|
||||||
|
*.tmp
|
||||||
|
|
||||||
|
# Log files
|
||||||
|
*.log
|
||||||
|
logs/
|
||||||
|
log/
|
||||||
|
|
||||||
|
# Python cache
|
||||||
|
__pycache__/
|
||||||
|
*.py[cod]
|
||||||
|
*$py.class
|
||||||
|
*.so
|
||||||
|
.Python
|
||||||
|
|
||||||
|
# Virtual environments
|
||||||
|
venv/
|
||||||
|
env/
|
||||||
|
ENV/
|
||||||
|
|
||||||
|
# IDE/editor files
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.iml
|
||||||
|
|
||||||
|
# OS files
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
|
|
||||||
|
# Environment/config with secrets (if they exist locally)
|
||||||
|
.env
|
||||||
|
.env.local
|
||||||
|
config.local.json
|
||||||
|
|
||||||
|
# Local data that shouldn't be shared
|
||||||
|
data/
|
||||||
|
datasets/
|
||||||
|
*.db
|
||||||
|
*.sqlite
|
||||||
|
|
||||||
|
# Build artifacts
|
||||||
|
build/
|
||||||
|
dist/
|
||||||
|
*.egg-info/
|
||||||
@@ -1,308 +0,0 @@
|
|||||||
# TrueRecall v2
|
|
||||||
|
|
||||||
**Project:** Gem extraction and memory recall system
|
|
||||||
**Status:** ✅ Active
|
|
||||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
TrueRecall extracts "gems" (key insights) from conversations and stores them for context injection. It's the memory curation system that powers Kimi's contextual awareness.
|
|
||||||
|
|
||||||
### Current Flow
|
|
||||||
|
|
||||||
```
|
|
||||||
1. Conversation happens → Real-time watcher → memories_tr (Qdrant)
|
|
||||||
2. Daily (2:45 AM) → Curator reads memories_tr → extracts gems
|
|
||||||
3. Gems stored → gems_tr collection (Qdrant)
|
|
||||||
4. On each turn → memory-qdrant plugin injects gems as context
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verified:** 2026-02-24 — Real-time watcher capturing (12,223 → 12,228 points)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
### Collections (Qdrant)
|
|
||||||
|
|
||||||
| Collection | Purpose | Content | Status |
|
|
||||||
|------------|---------|---------|--------|
|
|
||||||
| `memories_tr` | Full text storage | Every conversation turn (migrated from `kimi_memories`) | ✅ Active |
|
|
||||||
| `gems_tr` | Gems (extracted insights) | Curated key points (for injection) | ✅ Active |
|
|
||||||
| `true_recall` | Legacy gems | Archive of previously extracted gems | 📦 Preserved |
|
|
||||||
| `kimi_memories` | Original collection | Backup (12,223 points preserved) | 📦 Preserved |
|
|
||||||
|
|
||||||
### Migration Script
|
|
||||||
|
|
||||||
| File | Purpose |
|
|
||||||
|------|---------|
|
|
||||||
| `migrate_memories.py` | Migrate data from `kimi_memories` → `memories_tr` with cleaning |
|
|
||||||
|
|
||||||
**Usage:**
|
|
||||||
```bash
|
|
||||||
python3 migrate_memories.py
|
|
||||||
```
|
|
||||||
|
|
||||||
**What it does:**
|
|
||||||
- Reads all points from `kimi_memories`
|
|
||||||
- Cleans content (removes metadata, thinking tags)
|
|
||||||
- Stores to `memories_tr`
|
|
||||||
- Preserves original `kimi_memories`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Components
|
|
||||||
|
|
||||||
| Component | Location | Purpose |
|
|
||||||
|-----------|----------|---------|
|
|
||||||
| **memory-qdrant plugin** | `/root/.openclaw/extensions/memory-qdrant/` | Injects gems as context |
|
|
||||||
| **Curation script** | `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` | Extracts gems |
|
|
||||||
| **Curator prompt** | `/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md` | Instructions for gem extraction |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## File Locations
|
|
||||||
|
|
||||||
### Core Files
|
|
||||||
|
|
||||||
```
|
|
||||||
/root/.openclaw/workspace/.projects/true-recall-v2/
|
|
||||||
├── README.md # This file
|
|
||||||
├── session.md # Development notes
|
|
||||||
├── curator-prompt.md # Gem extraction prompt
|
|
||||||
│
|
|
||||||
├── tr-daily/ # Daily curation
|
|
||||||
│ └── curate_from_qdrant.py # Main curator script
|
|
||||||
│
|
|
||||||
├── tr-compact/ # (Reserved for v2 expansion)
|
|
||||||
│ └── hook.py # Compaction hook (not active)
|
|
||||||
│
|
|
||||||
├── tr-worker/ # (Reserved for v2 expansion)
|
|
||||||
│ └── worker.py # Background worker (not active)
|
|
||||||
│
|
|
||||||
└── shared/ # Shared resources
|
|
||||||
```
|
|
||||||
|
|
||||||
### Plugin Files (Located in OpenClaw Extensions)
|
|
||||||
|
|
||||||
> **Note:** These files live in `/root/.openclaw/extensions/`, not in this project folder. Documented here for reference.
|
|
||||||
|
|
||||||
| Actual Location | Project Reference |
|
|
||||||
|----------------|-------------------|
|
|
||||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code (capture + injection) |
|
|
||||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
|
||||||
| `/root/.openclaw/openclaw.json` | Plugin configuration |
|
|
||||||
|
|
||||||
### Configuration
|
|
||||||
|
|
||||||
| File | Purpose |
|
|
||||||
|------|---------|
|
|
||||||
| `/root/.openclaw/openclaw.json` | Plugin config (collectionName: gems_tr) |
|
|
||||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code |
|
|
||||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
|
||||||
|
|
||||||
### Cron Jobs
|
|
||||||
|
|
||||||
| Time | Command | Purpose |
|
|
||||||
|------|---------|---------|
|
|
||||||
| 2:45 AM | `curate_memories.py` | Daily gem extraction (v1) → stores to gems_tr |
|
|
||||||
| 2:50 AM | `archive_to_memories_tr.py` | Archive to memories_tr |
|
|
||||||
|
|
||||||
View cron: `crontab -l`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Process Flow
|
|
||||||
|
|
||||||
### Step 1: Capture (Real-Time Watcher)
|
|
||||||
|
|
||||||
```
|
|
||||||
OpenClaw Session ──→ Real-Time Watcher ──→ Qdrant (memories_tr)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
|
||||||
|
|
||||||
**What it does:**
|
|
||||||
- Watches session JSONL files in real-time
|
|
||||||
- Parses each conversation turn (user + AI)
|
|
||||||
- Embeds with `snowflake-arctic-embed2` via Ollama
|
|
||||||
- Stores directly to `memories_tr` collection
|
|
||||||
- Cleans content (removes metadata, thinking tags)
|
|
||||||
|
|
||||||
**Systemd Service:** `mem-qdrant-watcher.service`
|
|
||||||
|
|
||||||
**Status:** ✅ Created, ready to deploy
|
|
||||||
|
|
||||||
### Step 2: Curation (Daily)
|
|
||||||
|
|
||||||
```
|
|
||||||
2:45 AM → curate_memories.py → memories_tr → qwen3 → gems_tr
|
|
||||||
```
|
|
||||||
|
|
||||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v1/tr-process/curate_memories.py`
|
|
||||||
|
|
||||||
**What it does:**
|
|
||||||
1. Reads from Redis buffer (`mem:rob`)
|
|
||||||
2. Passes to qwen3 (curator model)
|
|
||||||
3. Extracts gems using prompt
|
|
||||||
4. Stores gems to `gems_tr` collection
|
|
||||||
|
|
||||||
### Step 3: Injection
|
|
||||||
|
|
||||||
```
|
|
||||||
User message → memory-qdrant plugin → Search gems_tr → Inject as context
|
|
||||||
```
|
|
||||||
|
|
||||||
**Location:** `/root/.openclaw/extensions/memory-qdrant/index.ts`
|
|
||||||
|
|
||||||
**What it does:**
|
|
||||||
1. Listens to `before_agent_start` events
|
|
||||||
2. Embeds current prompt
|
|
||||||
3. Searches `gems_tr` collection
|
|
||||||
4. Returns `{ prependContext: "..." }` with gems
|
|
||||||
5. Gems appear in my context (hidden from UI)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
### memory-qdrant Plugin
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"autoCapture": true,
|
|
||||||
"autoRecall": true,
|
|
||||||
"collectionName": "gems_tr",
|
|
||||||
"embeddingModel": "snowflake-arctic-embed2",
|
|
||||||
"maxRecallResults": 2,
|
|
||||||
"minRecallScore": 0.7
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Key setting:** `collectionName: "gems_tr"` — tells plugin to inject gems, not full text.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Gem Format
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"gem": "User prefers dark mode for interface",
|
|
||||||
"context": "Discussed UI theme options",
|
|
||||||
"snippet": "rob: I want dark mode\nKimi: Done",
|
|
||||||
"categories": ["preference"],
|
|
||||||
"importance": "high",
|
|
||||||
"confidence": 0.95,
|
|
||||||
"date": "2026-02-24",
|
|
||||||
"conversation_id": "uuid",
|
|
||||||
"turn_range": "5-7"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Validation Commands
|
|
||||||
|
|
||||||
### Check Qdrant Collections
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl -s "http://10.0.0.40:6333/collections" | python3 -m json.tool
|
|
||||||
```
|
|
||||||
|
|
||||||
### Check Collection Points
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# memories_tr (full text)
|
|
||||||
curl -s -X POST "http://10.0.0.40:6333/collections/memories_tr/points/scroll" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
|
||||||
|
|
||||||
# gems_tr (gems)
|
|
||||||
curl -s -X POST "http://10.0.0.40:6333/collections/gems_tr/points/scroll" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
|
||||||
```
|
|
||||||
|
|
||||||
### Check Cron Jobs
|
|
||||||
|
|
||||||
```bash
|
|
||||||
crontab -l | grep -E "true|recall|curate"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Check Plugin Logs
|
|
||||||
|
|
||||||
```bash
|
|
||||||
tail -50 /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Issue: `<relevant-memories>` showing in UI
|
|
||||||
|
|
||||||
**Cause:** Plugin was using `memories_tr` (full text) instead of `gems_tr` (gems)
|
|
||||||
|
|
||||||
**Fix:** Set `"collectionName": "gems_tr"` in openclaw.json
|
|
||||||
|
|
||||||
**Reference:** KB entry `relevant-memories-ui-issue.md`
|
|
||||||
|
|
||||||
### Issue: No gems being extracted
|
|
||||||
|
|
||||||
**Check:**
|
|
||||||
1. Is memories_tr populated? `curl .../collections/memories_tr/points/scroll`
|
|
||||||
2. Is curator prompt valid? `cat curator-prompt.md`
|
|
||||||
3. Is qwen3 available? `curl -s http://10.0.0.10:11434/api/tags`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Related Projects
|
|
||||||
|
|
||||||
| Project | Location | Purpose |
|
|
||||||
|---------|----------|---------|
|
|
||||||
| **true-recall-v1** | `/.projects/true-recall-v1/` | Original (Redis-based) |
|
|
||||||
| **memory-qdrant** | `/root/.openclaw/extensions/memory-qdrant/` | Plugin |
|
|
||||||
| **mem-redis** | `/root/.openclaw/workspace/skills/mem-redis/` | Redis utilities |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Backups
|
|
||||||
|
|
||||||
| File | Backup Location |
|
|
||||||
|------|-----------------|
|
|
||||||
| openclaw.json | `/root/.openclaw/openclaw.json.bak.2026-02-24` |
|
|
||||||
| Plugin | `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24` |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Final Status (2026-02-24 15:08)
|
|
||||||
|
|
||||||
| Component | Status |
|
|
||||||
|-----------|--------|
|
|
||||||
| Collection `kimi_memories` → `memories_tr` | ✅ Migrated (12,223 points) |
|
|
||||||
| Real-time watcher | ✅ **Deployed & verified** (12,228 points, +5 new) |
|
|
||||||
| Collection `gems_tr` | ✅ Active (5 gems stored) |
|
|
||||||
| Curator v2 | ✅ Working - tested with 327 turns |
|
|
||||||
| Config | ✅ Updated to `gems_tr` |
|
|
||||||
| Cron jobs | ✅ Cleaned |
|
|
||||||
| Documentation | ✅ Updated |
|
|
||||||
|
|
||||||
### Daily Schedule (Simplified)
|
|
||||||
|
|
||||||
| Time | Job | Flow |
|
|
||||||
|------|-----|------|
|
|
||||||
| Continuous | autoCapture | Conversation → `memories_tr` |
|
|
||||||
| 2:45 AM | Curator v2 | `memories_tr` → `gems_tr` |
|
|
||||||
| Each turn | Injection | `gems_tr` → Context |
|
|
||||||
|
|
||||||
**Redis usage:** Disabled for memory. Used only for `delayed:notifications` queue.
|
|
||||||
|
|
||||||
**Auto-Capture Status:** ✅ **Working** - Real-time watcher deployed and verified
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Last Updated:** 2026-02-24
|
|
||||||
**Project Lead:** Rob
|
|
||||||
**AI Assistant:** Kimi 🎙️
|
|
||||||
@@ -1,325 +0,0 @@
|
|||||||
# TrueRecall v2
|
|
||||||
|
|
||||||
**Project:** Gem extraction and memory recall system
|
|
||||||
**Status:** ✅ Active
|
|
||||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Table of Contents
|
|
||||||
|
|
||||||
- [Overview](#overview)
|
|
||||||
- [Architecture](#architecture)
|
|
||||||
- [Collections](#collections-qdrant)
|
|
||||||
- [Components](#components)
|
|
||||||
- [Process Flow](#process-flow)
|
|
||||||
- [Configuration](#configuration)
|
|
||||||
- [Validation Commands](#validation-commands)
|
|
||||||
- [Troubleshooting](#troubleshooting)
|
|
||||||
- [Related Projects](#related-projects)
|
|
||||||
- [Backups](#backups)
|
|
||||||
- [Final Status](#final-status-2026-02-24-1508)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
TrueRecall extracts "gems" (key insights) from conversations and stores them for context injection. It's the memory curation system that powers Kimi's contextual awareness.
|
|
||||||
|
|
||||||
### Current Flow
|
|
||||||
|
|
||||||
```
|
|
||||||
1. Conversation happens → Real-time watcher → memories_tr (Qdrant)
|
|
||||||
2. Daily (2:45 AM) → Curator reads memories_tr → extracts gems
|
|
||||||
3. Gems stored → gems_tr collection (Qdrant)
|
|
||||||
4. On each turn → memory-qdrant plugin injects gems as context
|
|
||||||
```
|
|
||||||
|
|
||||||
**Verified:** 2026-02-24 — Real-time watcher capturing (12,223 → 12,228 points)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
### Collections (Qdrant)
|
|
||||||
|
|
||||||
| Collection | Purpose | Content | Status |
|
|
||||||
|------------|---------|---------|--------|
|
|
||||||
| `memories_tr` | Full text storage | Every conversation turn (migrated from `kimi_memories`) | ✅ Active |
|
|
||||||
| `gems_tr` | Gems (extracted insights) | Curated key points (for injection) | ✅ Active |
|
|
||||||
| `true_recall` | Legacy gems | Archive of previously extracted gems | 📦 Preserved |
|
|
||||||
| `kimi_memories` | Original collection | Backup (12,223 points preserved) | 📦 Preserved |
|
|
||||||
|
|
||||||
### Migration Script
|
|
||||||
|
|
||||||
| File | Purpose |
|
|
||||||
|------|---------|
|
|
||||||
| `migrate_memories.py` | Migrate data from `kimi_memories` → `memories_tr` with cleaning |
|
|
||||||
|
|
||||||
**Usage:**
|
|
||||||
```bash
|
|
||||||
python3 migrate_memories.py
|
|
||||||
```
|
|
||||||
|
|
||||||
**What it does:**
|
|
||||||
- Reads all points from `kimi_memories`
|
|
||||||
- Cleans content (removes metadata, thinking tags)
|
|
||||||
- Stores to `memories_tr`
|
|
||||||
- Preserves original `kimi_memories`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### Components
|
|
||||||
|
|
||||||
| Component | Location | Purpose |
|
|
||||||
|-----------|----------|---------|
|
|
||||||
| **memory-qdrant plugin** | `/root/.openclaw/extensions/memory-qdrant/` | Injects gems as context |
|
|
||||||
| **Curation script** | `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` | Extracts gems |
|
|
||||||
| **Curator prompt** | `/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md` | Instructions for gem extraction |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## File Locations
|
|
||||||
|
|
||||||
### Core Files
|
|
||||||
|
|
||||||
```
|
|
||||||
/root/.openclaw/workspace/.projects/true-recall-v2/
|
|
||||||
├── README.md # This file
|
|
||||||
├── session.md # Development notes
|
|
||||||
├── curator-prompt.md # Gem extraction prompt
|
|
||||||
│
|
|
||||||
├── tr-daily/ # Daily curation
|
|
||||||
│ └── curate_from_qdrant.py # Main curator script
|
|
||||||
│
|
|
||||||
├── tr-compact/ # (Reserved for v2 expansion)
|
|
||||||
│ └── hook.py # Compaction hook (not active)
|
|
||||||
│
|
|
||||||
├── tr-worker/ # (Reserved for v2 expansion)
|
|
||||||
│ └── worker.py # Background worker (not active)
|
|
||||||
│
|
|
||||||
└── shared/ # Shared resources
|
|
||||||
```
|
|
||||||
|
|
||||||
### Plugin Files (Located in OpenClaw Extensions)
|
|
||||||
|
|
||||||
> **Note:** These files live in `/root/.openclaw/extensions/`, not in this project folder. Documented here for reference.
|
|
||||||
|
|
||||||
| Actual Location | Project Reference |
|
|
||||||
|----------------|-------------------|
|
|
||||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code (capture + injection) |
|
|
||||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
|
||||||
| `/root/.openclaw/openclaw.json` | Plugin configuration |
|
|
||||||
|
|
||||||
### Configuration
|
|
||||||
|
|
||||||
| File | Purpose |
|
|
||||||
|------|---------|
|
|
||||||
| `/root/.openclaw/openclaw.json` | Plugin config (collectionName: gems_tr) |
|
|
||||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code |
|
|
||||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
|
||||||
|
|
||||||
### Cron Jobs
|
|
||||||
|
|
||||||
| Time | Command | Purpose |
|
|
||||||
|------|---------|---------|
|
|
||||||
| 2:45 AM | `curate_memories.py` | Daily gem extraction (v1) → stores to gems_tr |
|
|
||||||
| 2:50 AM | `archive_to_memories_tr.py` | Archive to memories_tr |
|
|
||||||
|
|
||||||
View cron: `crontab -l`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Process Flow
|
|
||||||
|
|
||||||
### Step 1: Capture (Real-Time Watcher)
|
|
||||||
|
|
||||||
```
|
|
||||||
OpenClaw Session ──→ Real-Time Watcher ──→ Qdrant (memories_tr)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
|
||||||
|
|
||||||
**What it does:**
|
|
||||||
- Watches session JSONL files in real-time
|
|
||||||
- Parses each conversation turn (user + AI)
|
|
||||||
- Embeds with `snowflake-arctic-embed2` via Ollama
|
|
||||||
- Stores directly to `memories_tr` collection
|
|
||||||
- Cleans content (removes metadata, thinking tags)
|
|
||||||
|
|
||||||
**Systemd Service:** `mem-qdrant-watcher.service`
|
|
||||||
|
|
||||||
**Status:** ✅ Created, ready to deploy
|
|
||||||
|
|
||||||
### Step 2: Curation (Daily)
|
|
||||||
|
|
||||||
```
|
|
||||||
2:45 AM → curate_memories.py → memories_tr → qwen3 → gems_tr
|
|
||||||
```
|
|
||||||
|
|
||||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v1/tr-process/curate_memories.py`
|
|
||||||
|
|
||||||
**What it does:**
|
|
||||||
1. Reads from Redis buffer (`mem:rob`)
|
|
||||||
2. Passes to qwen3 (curator model)
|
|
||||||
3. Extracts gems using prompt
|
|
||||||
4. Stores gems to `gems_tr` collection
|
|
||||||
|
|
||||||
### Step 3: Injection
|
|
||||||
|
|
||||||
```
|
|
||||||
User message → memory-qdrant plugin → Search gems_tr → Inject as context
|
|
||||||
```
|
|
||||||
|
|
||||||
**Location:** `/root/.openclaw/extensions/memory-qdrant/index.ts`
|
|
||||||
|
|
||||||
**What it does:**
|
|
||||||
1. Listens to `before_agent_start` events
|
|
||||||
2. Embeds current prompt
|
|
||||||
3. Searches `gems_tr` collection
|
|
||||||
4. Returns `{ prependContext: "..." }` with gems
|
|
||||||
5. Gems appear in my context (hidden from UI)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Configuration
|
|
||||||
|
|
||||||
### memory-qdrant Plugin
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"autoCapture": true,
|
|
||||||
"autoRecall": true,
|
|
||||||
"collectionName": "gems_tr",
|
|
||||||
"embeddingModel": "snowflake-arctic-embed2",
|
|
||||||
"maxRecallResults": 2,
|
|
||||||
"minRecallScore": 0.7
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Key setting:** `collectionName: "gems_tr"` — tells plugin to inject gems, not full text.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Gem Format
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"gem": "User prefers dark mode for interface",
|
|
||||||
"context": "Discussed UI theme options",
|
|
||||||
"snippet": "rob: I want dark mode\nKimi: Done",
|
|
||||||
"categories": ["preference"],
|
|
||||||
"importance": "high",
|
|
||||||
"confidence": 0.95,
|
|
||||||
"date": "2026-02-24",
|
|
||||||
"conversation_id": "uuid",
|
|
||||||
"turn_range": "5-7"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Validation Commands
|
|
||||||
|
|
||||||
### Check Qdrant Collections
|
|
||||||
|
|
||||||
```bash
|
|
||||||
curl -s "http://10.0.0.40:6333/collections" | python3 -m json.tool
|
|
||||||
```
|
|
||||||
|
|
||||||
### Check Collection Points
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# memories_tr (full text)
|
|
||||||
curl -s -X POST "http://10.0.0.40:6333/collections/memories_tr/points/scroll" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
|
||||||
|
|
||||||
# gems_tr (gems)
|
|
||||||
curl -s -X POST "http://10.0.0.40:6333/collections/gems_tr/points/scroll" \
|
|
||||||
-H "Content-Type: application/json" \
|
|
||||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
|
||||||
```
|
|
||||||
|
|
||||||
### Check Cron Jobs
|
|
||||||
|
|
||||||
```bash
|
|
||||||
crontab -l | grep -E "true|recall|curate"
|
|
||||||
```
|
|
||||||
|
|
||||||
### Check Plugin Logs
|
|
||||||
|
|
||||||
```bash
|
|
||||||
tail -50 /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Troubleshooting
|
|
||||||
|
|
||||||
### Issue: `<relevant-memories>` showing in UI
|
|
||||||
|
|
||||||
**Cause:** Plugin was using `memories_tr` (full text) instead of `gems_tr` (gems)
|
|
||||||
|
|
||||||
**Fix:** Set `"collectionName": "gems_tr"` in openclaw.json
|
|
||||||
|
|
||||||
**Reference:** KB entry `relevant-memories-ui-issue.md`
|
|
||||||
|
|
||||||
### Issue: No gems being extracted
|
|
||||||
|
|
||||||
**Check:**
|
|
||||||
1. Is memories_tr populated? `curl .../collections/memories_tr/points/scroll`
|
|
||||||
2. Is curator prompt valid? `cat curator-prompt.md`
|
|
||||||
3. Is qwen3 available? `curl -s http://10.0.0.10:11434/api/tags`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Related Projects
|
|
||||||
|
|
||||||
| Project | Location | Purpose |
|
|
||||||
|---------|----------|---------|
|
|
||||||
| **true-recall-v1** | `/.projects/true-recall-v1/` | Original (Redis-based) |
|
|
||||||
| **memory-qdrant** | `/root/.openclaw/extensions/memory-qdrant/` | Plugin |
|
|
||||||
| **mem-redis** | `/root/.openclaw/workspace/skills/mem-redis/` | Redis utilities |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Backups
|
|
||||||
|
|
||||||
| File | Backup Location |
|
|
||||||
|------|-----------------|
|
|
||||||
| openclaw.json | `/root/.openclaw/openclaw.json.bak.2026-02-24` |
|
|
||||||
| Plugin | `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24` |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Final Status (2026-02-24 15:08)
|
|
||||||
|
|
||||||
| Component | Status |
|
|
||||||
|-----------|--------|
|
|
||||||
| Collection `kimi_memories` → `memories_tr` | ✅ Migrated (12,223 points) |
|
|
||||||
| Real-time watcher | ✅ **Deployed & verified** (12,228 points, +5 new) |
|
|
||||||
| Collection `gems_tr` | ✅ Active (5 gems stored) |
|
|
||||||
| Curator v2 | ✅ Working - tested with 327 turns |
|
|
||||||
| Config | ✅ Updated to `gems_tr` |
|
|
||||||
| Cron jobs | ✅ Cleaned |
|
|
||||||
| Documentation | ✅ Updated |
|
|
||||||
|
|
||||||
### Daily Schedule (Simplified)
|
|
||||||
|
|
||||||
| Time | Job | Flow |
|
|
||||||
|------|-----|------|
|
|
||||||
| Continuous | autoCapture | Conversation → `memories_tr` |
|
|
||||||
| 2:45 AM | Curator v2 | `memories_tr` → `gems_tr` |
|
|
||||||
| Each turn | Injection | `gems_tr` → Context |
|
|
||||||
|
|
||||||
**Redis usage:** Disabled for memory. Used only for `delayed:notifications` queue.
|
|
||||||
|
|
||||||
**Auto-Capture Status:** ✅ **Working** - Real-time watcher deployed and verified
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Last Updated:** 2026-02-24
|
|
||||||
**Project Lead:** Rob
|
|
||||||
**AI Assistant:** Kimi 🎙️
|
|
||||||
|
|
||||||
@@ -1,183 +0,0 @@
|
|||||||
# NeuralStream
|
|
||||||
|
|
||||||
**Neural streaming memory for OpenClaw with gem-based context injection.**
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
NeuralStream extracts high-value insights ("gems") from conversation batches using qwen3, stores them in Qdrant, and injects relevant gems into context on each new turn. This creates **infinite effective context** — the active window stays small, but semantically relevant gems from all past conversations are always retrievable.
|
|
||||||
|
|
||||||
## Core Concept
|
|
||||||
|
|
||||||
| Traditional Memory | NeuralStream |
|
|
||||||
|-------------------|--------------|
|
|
||||||
| Context lost on `/new` | Gems persist in Qdrant |
|
|
||||||
| Full history or generic summary | Semantic gem retrieval |
|
|
||||||
| Static context window | Dynamic injection |
|
|
||||||
| Survives compaction only | Survives session reset |
|
|
||||||
| **Limited context** | **Infinite effective context** |
|
|
||||||
|
|
||||||
## How It Works
|
|
||||||
|
|
||||||
### Capture → Extract → Store → Retrieve
|
|
||||||
|
|
||||||
1. **Capture:** Every turn buffered to Redis (reuses mem-redis-watcher)
|
|
||||||
2. **Extract:** Batch of 5 turns → qwen3 (with 256k context) extracts structured gems
|
|
||||||
3. **Store:** Gems embedded + stored in Qdrant `neuralstream`
|
|
||||||
4. **Retrieve:** Each new turn → semantic search → inject top-10 gems
|
|
||||||
|
|
||||||
### Hybrid Triggers (Three-way)
|
|
||||||
|
|
||||||
| Trigger | Condition | Purpose |
|
|
||||||
|---------|-----------|---------|
|
|
||||||
| Batch | Every 5 turns | Normal extraction |
|
|
||||||
| Context | 50% usage (`ctx.getContextUsage()`) | Proactive pre-compaction |
|
|
||||||
| Timer | 15 min idle | Safety net |
|
|
||||||
|
|
||||||
**Context Awareness:** qwen3 receives up to 256k tokens of history for understanding, but only extracts gems from the last N turns (avoiding current context).
|
|
||||||
|
|
||||||
All gems survive `/new`, `/reset`, and compaction via Qdrant persistence.
|
|
||||||
|
|
||||||
## Architecture
|
|
||||||
|
|
||||||
NeuralStream is the **middle layer** — extraction intelligence on top of existing infrastructure:
|
|
||||||
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────┐
|
|
||||||
│ EXISTING: mem-redis-watcher │
|
|
||||||
│ Every turn → Redis buffer │
|
|
||||||
└──────────────────┬──────────────────────────────────────┘
|
|
||||||
│
|
|
||||||
┌──────────▼──────────┐
|
|
||||||
│ NeuralStream │
|
|
||||||
│ - Batch reader │
|
|
||||||
│ - Gem extractor │
|
|
||||||
│ - Qdrant store │
|
|
||||||
└──────────┬──────────┘
|
|
||||||
│
|
|
||||||
┌──────────▼──────────┐
|
|
||||||
│ EXISTING: │
|
|
||||||
│ qdrant-memory │
|
|
||||||
│ Semantic search │
|
|
||||||
│ Context injection │
|
|
||||||
└─────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
## Technical Reference
|
|
||||||
|
|
||||||
### Native Context Monitoring
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
// In turn_end hook
|
|
||||||
const usage = ctx.getContextUsage();
|
|
||||||
// usage.tokens, usage.contextWindow, usage.percent
|
|
||||||
// Trigger extraction when usage.percent >= threshold
|
|
||||||
```
|
|
||||||
|
|
||||||
### Primary Hook: turn_end
|
|
||||||
|
|
||||||
```typescript
|
|
||||||
pi.on("turn_end", async (event, ctx) => {
|
|
||||||
const { turnIndex, message, toolResults } = event;
|
|
||||||
|
|
||||||
// Buffer turn to Redis
|
|
||||||
// Check ctx.getContextUsage().percent
|
|
||||||
// If batch >= 5 OR percent >= 50%: extract
|
|
||||||
});
|
|
||||||
```
|
|
||||||
|
|
||||||
### Timer Fallback
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Cron every 10 min
|
|
||||||
# Check neuralstream:buffer age > 15 min
|
|
||||||
# If yes: extract from partial batch
|
|
||||||
```
|
|
||||||
|
|
||||||
### Context-Aware Extraction
|
|
||||||
|
|
||||||
- Feed qwen3: Up to 256k tokens (full history for context)
|
|
||||||
- Extract from: Last `batch_size` turns only
|
|
||||||
- Benefit: Rich understanding without gemming current context
|
|
||||||
|
|
||||||
## Gem Format
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"gem_id": "uuid",
|
|
||||||
"content": "Distilled insight/fact/decision",
|
|
||||||
"summary": "One-line for quick scanning",
|
|
||||||
"topics": ["docker", "redis", "architecture"],
|
|
||||||
"importance": 0.9,
|
|
||||||
"source": {
|
|
||||||
"session_id": "uuid",
|
|
||||||
"date": "2026-02-23",
|
|
||||||
"turn_range": "15-20"
|
|
||||||
},
|
|
||||||
"tags": ["decision", "fact", "preference", "todo", "code"],
|
|
||||||
"created_at": "2026-02-23T15:26:00Z"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Configuration (All Tunable)
|
|
||||||
|
|
||||||
| Setting | Default | Description |
|
|
||||||
|---------|---------|-------------|
|
|
||||||
| batch_size | 5 | Turns per extraction |
|
|
||||||
| context_threshold | 50% | Token % trigger (40-80% range) |
|
|
||||||
| idle_timeout | 15 min | Timer trigger threshold |
|
|
||||||
| gem_model | qwen3 | Extraction LLM (256k context) |
|
|
||||||
| max_gems_injected | 10 | Per-turn limit |
|
|
||||||
| embedding | snowflake-arctic-embed2 | Same as kimi_memories |
|
|
||||||
| collection | neuralstream | Qdrant (1024 dims, Cosine) |
|
|
||||||
|
|
||||||
## Qdrant Schema
|
|
||||||
|
|
||||||
**Collection:** `neuralstream`
|
|
||||||
- Vector size: 1024
|
|
||||||
- Distance: Cosine
|
|
||||||
- On-disk payload: true
|
|
||||||
|
|
||||||
## Project Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
.projects/neuralstream/
|
|
||||||
├── README.md # This file
|
|
||||||
├── session.md # Development log & state
|
|
||||||
├── prompt.md # (TBD) qwen3 extraction prompt
|
|
||||||
└── src/ # (TBD) Implementation
|
|
||||||
├── extract.ts # Gem extraction logic
|
|
||||||
├── store.ts # Qdrant storage
|
|
||||||
└── inject.ts # Context injection
|
|
||||||
```
|
|
||||||
|
|
||||||
## Status
|
|
||||||
|
|
||||||
- [x] Architecture defined (v2.2 context-aware)
|
|
||||||
- [x] Native context monitoring validated (ctx.getContextUsage)
|
|
||||||
- [x] Naming finalized (NeuralStream, alias: ns)
|
|
||||||
- [x] Hook research completed
|
|
||||||
- [x] Qdrant collection created (`neuralstream`)
|
|
||||||
- [x] Gem format proposed
|
|
||||||
- [x] Infrastructure decision (reuse Redis/Qdrant)
|
|
||||||
- [ ] Extraction prompt design
|
|
||||||
- [ ] Implementation
|
|
||||||
- [ ] Testing
|
|
||||||
|
|
||||||
## Backups
|
|
||||||
|
|
||||||
- Local: `/root/.openclaw/workspace/.projects/neuralstream/`
|
|
||||||
- Remote: `deb2:/root/.projects/neuralstream/` (build/test only)
|
|
||||||
- kimi_kb: Research entries stored
|
|
||||||
|
|
||||||
## Related Projects
|
|
||||||
|
|
||||||
- **True Recall:** Gem extraction inspiration
|
|
||||||
- **OpenClaw:** Host platform
|
|
||||||
- **kimi_memories:** Shared Qdrant infrastructure
|
|
||||||
- **mem-redis-watcher:** Existing capture layer
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Created:** 2026-02-23
|
|
||||||
**Alias:** ns
|
|
||||||
**Purpose:** Infinite context for LLMs
|
|
||||||
106
debug_curator.py
106
debug_curator.py
@@ -1,106 +0,0 @@
|
|||||||
#!/usr/bin/env python3
|
|
||||||
"""Debug curator with real data"""
|
|
||||||
import json
|
|
||||||
import requests
|
|
||||||
import urllib.request
|
|
||||||
|
|
||||||
QDRANT_URL = "http://10.0.0.40:6333"
|
|
||||||
SOURCE_COLLECTION = "kimi_memories"
|
|
||||||
|
|
||||||
# Get sample turns from real data
|
|
||||||
filter_data = {
|
|
||||||
"must": [
|
|
||||||
{"key": "user_id", "match": {"value": "rob"}},
|
|
||||||
{"key": "date", "match": {"value": "2026-02-23"}}
|
|
||||||
]
|
|
||||||
}
|
|
||||||
|
|
||||||
req = urllib.request.Request(
|
|
||||||
f"{QDRANT_URL}/collections/{SOURCE_COLLECTION}/points/scroll",
|
|
||||||
data=json.dumps({"limit": 5, "with_payload": True, "filter": filter_data}).encode(),
|
|
||||||
headers={"Content-Type": "application/json"},
|
|
||||||
method="POST"
|
|
||||||
)
|
|
||||||
|
|
||||||
with urllib.request.urlopen(req, timeout=30) as response:
|
|
||||||
result = json.loads(response.read().decode())
|
|
||||||
points = result.get("result", {}).get("points", [])
|
|
||||||
|
|
||||||
turns = []
|
|
||||||
for point in points:
|
|
||||||
payload = point.get("payload", {})
|
|
||||||
user_msg = payload.get("user_message", "")
|
|
||||||
ai_msg = payload.get("ai_response", "")
|
|
||||||
|
|
||||||
if user_msg or ai_msg:
|
|
||||||
turn = {
|
|
||||||
"turn": payload.get("turn_number", 0),
|
|
||||||
"user_id": payload.get("user_id", "rob"),
|
|
||||||
"user": user_msg[:300], # Truncate
|
|
||||||
"ai": ai_msg[:300], # Truncate
|
|
||||||
"conversation_id": payload.get("conversation_id", ""),
|
|
||||||
"timestamp": payload.get("created_at", ""),
|
|
||||||
"date": payload.get("date", "2026-02-23")
|
|
||||||
}
|
|
||||||
turns.append(turn)
|
|
||||||
|
|
||||||
turns.sort(key=lambda x: (x.get("conversation_id", ""), x.get("turn", 0)))
|
|
||||||
|
|
||||||
print(f"Got {len(turns)} turns")
|
|
||||||
print("Sample:")
|
|
||||||
for t in turns[:2]:
|
|
||||||
print(f" User: {t['user'][:100]}...")
|
|
||||||
print(f" AI: {t['ai'][:100]}...")
|
|
||||||
|
|
||||||
# Now test with curator
|
|
||||||
with open('/root/.openclaw/workspace/.projects/true-recall-v1/curator-prompt.md') as f:
|
|
||||||
prompt = f.read()
|
|
||||||
|
|
||||||
conversation_json = json.dumps(turns[:5], indent=2)
|
|
||||||
|
|
||||||
prompt_text = f"""## Input Conversation
|
|
||||||
|
|
||||||
```json
|
|
||||||
{conversation_json}
|
|
||||||
```
|
|
||||||
|
|
||||||
## Output
|
|
||||||
"""
|
|
||||||
|
|
||||||
response = requests.post(
|
|
||||||
'http://10.0.0.10:11434/api/generate',
|
|
||||||
json={
|
|
||||||
'model': 'qwen3:4b-instruct',
|
|
||||||
'system': prompt,
|
|
||||||
'prompt': prompt_text,
|
|
||||||
'stream': False,
|
|
||||||
'options': {'temperature': 0.1, 'num_predict': 3000}
|
|
||||||
},
|
|
||||||
timeout=120
|
|
||||||
)
|
|
||||||
|
|
||||||
result = response.json()
|
|
||||||
output = result.get('response', '').strip()
|
|
||||||
|
|
||||||
print("\n=== CURATOR OUTPUT ===")
|
|
||||||
print(output[:3000])
|
|
||||||
print("\n=== TRYING TO PARSE ===")
|
|
||||||
|
|
||||||
# Try to parse
|
|
||||||
try:
|
|
||||||
if '```json' in output:
|
|
||||||
parsed = output.split('```json')[1].split('```')[0].strip()
|
|
||||||
gems = json.loads(parsed)
|
|
||||||
print(f"Parsed {len(gems)} gems")
|
|
||||||
elif '```' in output:
|
|
||||||
parsed = output.split('```')[1].split('```')[0].strip()
|
|
||||||
gems = json.loads(parsed)
|
|
||||||
print(f"Parsed {len(gems)} gems")
|
|
||||||
else:
|
|
||||||
gems = json.loads(output)
|
|
||||||
print(f"Parsed {len(gems)} gems")
|
|
||||||
except Exception as e:
|
|
||||||
print(f"Parse error: {e}")
|
|
||||||
print("Trying raw parse...")
|
|
||||||
gems = json.loads(output.strip())
|
|
||||||
print(f"Parsed {len(gems)} gems")
|
|
||||||
@@ -1,224 +0,0 @@
|
|||||||
# TrueRecall v2 - Session Notes
|
|
||||||
|
|
||||||
**Last Updated:** 2026-02-24 15:08 CST
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Summary of Changes (2026-02-24)
|
|
||||||
|
|
||||||
### Collection Migration
|
|
||||||
|
|
||||||
| Action | From | To | Status |
|
|
||||||
|--------|------|----|--------|
|
|
||||||
| **Rename** | `kimi_memories` | `memories_tr` | ✅ Done + data migrated (12,223 points) |
|
|
||||||
| **Create** | — | `gems_tr` | ✅ Created (empty, ready for curation) |
|
|
||||||
| **Archive** | `true_recall` | `true_recall` | ✅ Preserved (existing data kept) |
|
|
||||||
|
|
||||||
### Dual Collection Support (NEW - 14:10)
|
|
||||||
|
|
||||||
**Problem:** Auto-capture was saving to `gems_tr` because that's where recall pulled from. This was wrong.
|
|
||||||
|
|
||||||
**Solution:** Added `captureCollection` option to memory-qdrant plugin:
|
|
||||||
- `collectionName`: `gems_tr` — for recall/injection (gems only)
|
|
||||||
- `captureCollection`: `memories_tr` — for auto-capture (full conversations)
|
|
||||||
|
|
||||||
**Files Modified:**
|
|
||||||
1. `/root/.openclaw/extensions/memory-qdrant/config.ts` — Added `captureCollection` option
|
|
||||||
2. `/root/.openclaw/extensions/memory-qdrant/index.ts` — Uses `dbRecall` and `dbCapture` separately
|
|
||||||
3. `/root/.openclaw/openclaw.json` — Added `"captureCollection": "memories_tr"`
|
|
||||||
|
|
||||||
### Auto-Capture Architecture (Real-Time Watcher)
|
|
||||||
|
|
||||||
**Mechanism:** `realtime_qdrant_watcher.py` daemon
|
|
||||||
- Watches OpenClaw session JSONL files in real-time
|
|
||||||
- Parses each conversation turn
|
|
||||||
- Embeds with `snowflake-arctic-embed2` via Ollama
|
|
||||||
- Stores directly to `memories_tr` collection (no Redis)
|
|
||||||
- Cleans content (removes metadata, thinking tags)
|
|
||||||
|
|
||||||
**Files:**
|
|
||||||
- Script: `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
|
||||||
- Service: `/root/.openclaw/workspace/skills/qdrant-memory/mem-qdrant-watcher.service`
|
|
||||||
|
|
||||||
**Status:** ✅ Deployed and running
|
|
||||||
|
|
||||||
**Deployment:**
|
|
||||||
- Stopped: `mem-redis-watcher`
|
|
||||||
- Started: `mem-qdrant-watcher`
|
|
||||||
- Status: Active (PID 84465)
|
|
||||||
- Verified: ✅ Capturing (points: 12,223 → 12,224)
|
|
||||||
|
|
||||||
### Current State (2026-02-24 15:08)
|
|
||||||
|
|
||||||
| Collection | Purpose | Points | Last Update |
|
|
||||||
|------------|---------|--------|-------------|
|
|
||||||
| `memories_tr` | Full text (autoCapture via watcher) | 12,228 | **Live** |
|
|
||||||
| `gems_tr` | Gems (for injection) | 5 | Feb 24 |
|
|
||||||
|
|
||||||
**Flow:**
|
|
||||||
```
|
|
||||||
OpenClaw Session → Real-Time Watcher → memories_tr (Qdrant)
|
|
||||||
↓
|
|
||||||
Daily Curator (2:45 AM)
|
|
||||||
↓
|
|
||||||
gems_tr (Qdrant)
|
|
||||||
↓
|
|
||||||
memory-qdrant plugin → Injection
|
|
||||||
```
|
|
||||||
|
|
||||||
### Files Modified Today
|
|
||||||
|
|
||||||
1. `/root/.openclaw/openclaw.json` — Changed collectionName to `gems_tr`
|
|
||||||
2. `/root/.openclaw/extensions/memory-qdrant/config.ts` — Default to `memories_tr`
|
|
||||||
3. `/root/.openclaw/extensions/memory-qdrant/openclaw.plugin.json` — Defaults to `memories_tr`
|
|
||||||
4. `/root/.openclaw/workspace/skills/mem-redis/scripts/save_mem.py` — Added `--silent` flag
|
|
||||||
5. `/root/.openclaw/workspace/HEARTBEAT.md` — Updated to use `--silent`
|
|
||||||
6. `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` — TARGET_COLLECTION = `gems_tr`
|
|
||||||
7. `/root/.openclaw/workspace/SOUL.md` — Updated collection references
|
|
||||||
8. `/root/.openclaw/workspace/kb/relevant-memories-ui-issue.md` — Updated documentation
|
|
||||||
|
|
||||||
### Migration Script Created
|
|
||||||
|
|
||||||
- `/root/.openclaw/workspace/.projects/true-recall-v2/migrate_memories.py`
|
|
||||||
- Migrated 12,223 points from `kimi_memories` → `memories_tr`
|
|
||||||
- Cleaned noise (metadata, thinking tags) during migration
|
|
||||||
- Preserved original `kimi_memories` as backup
|
|
||||||
|
|
||||||
### Current Collections (Qdrant)
|
|
||||||
|
|
||||||
| Collection | Purpose | Points |
|
|
||||||
|------------|---------|--------|
|
|
||||||
| `memories_tr` | Full text (live capture) | **12,228** |
|
|
||||||
| `gems_tr` | Gems (for injection) | 5 |
|
|
||||||
| `true_recall` | Legacy gems archive | existing |
|
|
||||||
| `kimi_memories` | Original (backup) | 12,223 |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Final Status (2026-02-24)
|
|
||||||
|
|
||||||
| Component | Status |
|
|
||||||
|-----------|--------|
|
|
||||||
| **Collections** | 4 active (memories_tr, gems_tr, true_recall, kimi_memories) |
|
|
||||||
| **Curator v2** | ✅ Tested & working - 327 turns → 5 gems |
|
|
||||||
| **Config** | ✅ Using `gems_tr` for injection |
|
|
||||||
| **Cron** | ✅ Simplified - only essential jobs |
|
|
||||||
| **Redis** | ✅ Only for notifications (not memory) |
|
|
||||||
|
|
||||||
### Collection Summary
|
|
||||||
|
|
||||||
| Collection | Points | Purpose | Status |
|
|
||||||
|------------|--------|---------|--------|
|
|
||||||
| `memories_tr` | **12,228** | Full text (live capture) | ✅ Active |
|
|
||||||
| `gems_tr` | 5 | Gems (curated) | ✅ Active |
|
|
||||||
| `true_recall` | existing | Legacy archive | 📦 Preserved |
|
|
||||||
| `kimi_memories` | 12,223 | Original backup | 📦 Preserved |
|
|
||||||
|
|
||||||
### Cron Schedule (Cleaned)
|
|
||||||
|
|
||||||
| Time | Job |
|
|
||||||
|------|-----|
|
|
||||||
| 2:45 AM | Curator v2: memories_tr → gems_tr |
|
|
||||||
| 2:20 AM | File backup |
|
|
||||||
| 2:00 AM | Log monitoring |
|
|
||||||
|
|
||||||
**Removed:** 2:15 AM Redis backup, 2:50 AM archive, 5-min cron_capture (all redundant)
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## What We Did Today
|
|
||||||
|
|
||||||
### Issue: `<relevant-memories>` showing in UI
|
|
||||||
|
|
||||||
**Problem:** The `<relevant-memories>` block was showing full conversation text in webchat UI. It should be hidden/internal.
|
|
||||||
|
|
||||||
**Investigation:**
|
|
||||||
1. Initially thought it was `save_mem.py` output (wrong)
|
|
||||||
2. Then thought it was HTML comment issue (wrong)
|
|
||||||
3. Found root cause: memory-qdrant plugin was recalling from `memories_tr` (full text) instead of `gems_tr` (gems)
|
|
||||||
|
|
||||||
**Solution Applied:**
|
|
||||||
Changed `openclaw.json` config:
|
|
||||||
- Before: `"collectionName": "memories_tr"`
|
|
||||||
- After: `"collectionName": "gems_tr"`
|
|
||||||
|
|
||||||
**Result:** ✅ Fixed! Now injection uses gems from true_recall, not full text.
|
|
||||||
|
|
||||||
### Issue: Auto-Capture Architecture Change
|
|
||||||
|
|
||||||
**Problem:** Real-time capture was going through Redis, we needed direct Qdrant storage
|
|
||||||
|
|
||||||
**Solution:**
|
|
||||||
1. Created `realtime_qdrant_watcher.py` — watches session JSONL, embeds, stores directly to Qdrant
|
|
||||||
2. Created `mem-qdrant-watcher.service` — systemd service for the watcher
|
|
||||||
3. Deployed and verified:
|
|
||||||
- Points before: 12,223
|
|
||||||
- Points after test messages: 12,228
|
|
||||||
- Verified new captures have correct structure (role, content, date, source)
|
|
||||||
|
|
||||||
**Status:** ✅ **Deployed and working**
|
|
||||||
|
|
||||||
### Other Changes Made
|
|
||||||
|
|
||||||
1. **Plugin dual collection support** — Added `captureCollection` option
|
|
||||||
2. **Session/Readme updates** — Clarified architecture
|
|
||||||
|
|
||||||
### Files Modified
|
|
||||||
|
|
||||||
| File | Change |
|
|
||||||
|------|--------|
|
|
||||||
| `/root/.openclaw/openclaw.json` | collectionName: memories_tr → gems_tr, added captureCollection |
|
|
||||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Added captureCollection option |
|
|
||||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Dual collection support, debug logging |
|
|
||||||
| `/root/.openclaw/extensions/memory-qdrant/openclaw.plugin.json` | Added captureCollection to schema |
|
|
||||||
|
|
||||||
### Files Created (NEW)
|
|
||||||
|
|
||||||
| File | Purpose |
|
|
||||||
|------|---------|
|
|
||||||
| `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` | Real-time watcher daemon (Qdrant direct) |
|
|
||||||
| `/root/.openclaw/workspace/skills/qdrant-memory/mem-qdrant-watcher.service` | Systemd service file |
|
|
||||||
|
|
||||||
### Deployment
|
|
||||||
|
|
||||||
| Service | Action | Status |
|
|
||||||
|---------|--------|--------|
|
|
||||||
| `mem-redis-watcher` | Stopped | ✅ |
|
|
||||||
| `mem-qdrant-watcher` | Started | ✅ Active |
|
|
||||||
|
|
||||||
### Backups Created
|
|
||||||
|
|
||||||
- `/root/.openclaw/openclaw.json.bak.2026-02-24`
|
|
||||||
- `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24`
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Current Status
|
|
||||||
|
|
||||||
| Component | Status |
|
|
||||||
|-----------|--------|
|
|
||||||
| **Curation (daily)** | v1 cron at 2:45 AM |
|
|
||||||
| **Injection** | ✅ Working, uses gems_tr |
|
|
||||||
| **Collection** | gems_tr |
|
|
||||||
| **KB** | Updated |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## What Still Needs Doing
|
|
||||||
|
|
||||||
1. ~~Test autoCapture (cleaned content to memories_tr)~~ ✅ Done
|
|
||||||
2. Test v2 curator (read from Qdrant, not Redis) — Next step
|
|
||||||
3. Full validation 2x
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Session Recovery
|
|
||||||
|
|
||||||
If starting new session:
|
|
||||||
1. Read this session.md
|
|
||||||
2. Read README.md for architecture
|
|
||||||
3. Read KB for issue history
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Next:** Test v2 curator (reads from memories_tr, creates gems in gems_tr)
|
|
||||||
@@ -1,150 +0,0 @@
|
|||||||
# NeuralStream Session State
|
|
||||||
|
|
||||||
**Date:** 2026-02-23
|
|
||||||
**Status:** Architecture v2.2 - Context-aware hybrid triggers
|
|
||||||
**Alias:** ns
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture v2.2 (Current)
|
|
||||||
|
|
||||||
**Decision:** Three hybrid extraction triggers with full context awareness
|
|
||||||
|
|
||||||
| Trigger | When | Purpose |
|
|
||||||
|---------|------|---------|
|
|
||||||
| `turn_end` (N=5) | Every 5 turns | Normal batch extraction |
|
|
||||||
| Timer (15 min idle) | No new turn for 15 min | Catch partial batches |
|
|
||||||
| Context (50% threshold) | `ctx.getContextUsage().percent >= threshold` | Proactive pre-compaction |
|
|
||||||
|
|
||||||
**Context Awareness:**
|
|
||||||
- qwen3 gets **up to 256k tokens** of full conversation history for understanding
|
|
||||||
- Only extracts **last N turns** (oldest in batch) to avoid gemming current context
|
|
||||||
- Uses `ctx.getContextUsage()` native API for token monitoring
|
|
||||||
|
|
||||||
**Why Hybrid:**
|
|
||||||
- Batch extraction = better quality gems (more context)
|
|
||||||
- Timer safety = never lose important turns if user walks away
|
|
||||||
- Context trigger = proactive extraction before system forces compaction
|
|
||||||
- All gems survive `/new` and `/reset` via Qdrant
|
|
||||||
|
|
||||||
**Infrastructure:** Reuse existing Redis/Qdrant — NeuralStream is the "middle layer" only
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Core Insight
|
|
||||||
|
|
||||||
NeuralStream enables **infinite effective context** — active window stays small, but semantically relevant gems from all past conversations are queryable and injectable.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Technical Decisions 2026-02-23
|
|
||||||
|
|
||||||
### Triggers (Three-way Hybrid)
|
|
||||||
| Trigger | Config | Default |
|
|
||||||
|---------|--------|---------|
|
|
||||||
| Batch size | `batch_size` | 5 turns |
|
|
||||||
| Idle timeout | `idle_timeout` | 15 minutes |
|
|
||||||
| Context threshold | `context_threshold` | 50% |
|
|
||||||
|
|
||||||
### Context Monitoring (Native API)
|
|
||||||
- `ctx.getContextUsage()` → `{tokens, contextWindow, percent}`
|
|
||||||
- Checked in `turn_end` hook
|
|
||||||
- Triggers extraction when `percent >= context_threshold`
|
|
||||||
|
|
||||||
### Extraction Context Window
|
|
||||||
- **Feed to qwen3:** Up to 256k tokens (full history for understanding)
|
|
||||||
- **Extract from:** Last `batch_size` turns only
|
|
||||||
- **Benefit:** Rich context awareness without gemming current conversation
|
|
||||||
|
|
||||||
### Storage
|
|
||||||
- **Buffer:** Redis (`neuralstream:buffer` key)
|
|
||||||
- **Gems:** Qdrant `neuralstream` collection (1024 dims, Cosine)
|
|
||||||
- **Existing infra:** Reuse mem-redis-watcher + qdrant-memory
|
|
||||||
|
|
||||||
### Gem Format (Proposed)
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"gem_id": "uuid",
|
|
||||||
"content": "Distilled insight/fact/decision",
|
|
||||||
"summary": "One-line for quick scanning",
|
|
||||||
"topics": ["docker", "redis", "architecture"],
|
|
||||||
"importance": 0.9,
|
|
||||||
"source": {
|
|
||||||
"session_id": "uuid",
|
|
||||||
"date": "2026-02-23",
|
|
||||||
"turn_range": "15-20"
|
|
||||||
},
|
|
||||||
"tags": ["decision", "fact", "preference", "todo", "code"],
|
|
||||||
"created_at": "2026-02-23T15:26:00Z"
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Extraction Model
|
|
||||||
- **qwen3** for gem extraction (256k context, cheap)
|
|
||||||
- **Dedicated prompt** (to be designed) for extracting high-value items
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture Layers
|
|
||||||
|
|
||||||
| Layer | Status | Description |
|
|
||||||
|-------|--------|-------------|
|
|
||||||
| Capture | ✅ Existing | Every turn → Redis (mem-redis-watcher) |
|
|
||||||
| **Extract** | ⏳ NeuralStream | Batch → qwen3 → gems → Qdrant |
|
|
||||||
| Retrieve | ✅ Existing | Semantic search → inject context |
|
|
||||||
|
|
||||||
NeuralStream = Smart extraction layer on top of existing infra.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Open Questions
|
|
||||||
|
|
||||||
- Gem extraction prompt design (deferred)
|
|
||||||
- Importance scoring: auto vs manual?
|
|
||||||
- Injection: `turn_start` hook or modify system prompt?
|
|
||||||
- Semantic search threshold tuning
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Next Steps
|
|
||||||
|
|
||||||
| Task | Status |
|
|
||||||
|------|--------|
|
|
||||||
| Architecture v2.2 finalized | ✅ |
|
|
||||||
| Native context monitoring validated | ✅ |
|
|
||||||
| Gem JSON schema | ✅ Proposed |
|
|
||||||
| Implement turn_end hook | ⏳ |
|
|
||||||
| Implement timer/cron check | ⏳ |
|
|
||||||
| Implement context trigger | ⏳ |
|
|
||||||
| Create extraction prompt | ⏳ |
|
|
||||||
| Test gem extraction with qwen3 | ⏳ |
|
|
||||||
| Implement injection mechanism | ⏳ |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Decisions Log
|
|
||||||
|
|
||||||
| Date | Decision |
|
|
||||||
|------|----------|
|
|
||||||
| 2026-02-23 | Switch to turn_end hook (v2) |
|
|
||||||
| 2026-02-23 | Hybrid triggers with timer (v2.1) |
|
|
||||||
| 2026-02-23 | Context-aware extraction (v2.2) |
|
|
||||||
| 2026-02-23 | Native API: ctx.getContextUsage() |
|
|
||||||
| 2026-02-23 | Full context feed to qwen3 (256k) |
|
|
||||||
| 2026-02-23 | Reuse existing Redis/Qdrant infrastructure |
|
|
||||||
| 2026-02-23 | Batch N=5 turns |
|
|
||||||
| 2026-02-23 | Context threshold = 50% |
|
|
||||||
| 2026-02-23 | Inactivity timer = 15 min |
|
|
||||||
| 2026-02-23 | Dedicated qwen3 extraction prompt (deferred) |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Backups
|
|
||||||
|
|
||||||
- Local: `/root/.openclaw/workspace/.projects/neuralstream/`
|
|
||||||
- Remote: `deb2:/root/.projects/neuralstream/` (build/test only)
|
|
||||||
- kimi_kb: Research entries stored
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Key Insight:** Session resets wipe context but NOT Qdrant. NeuralStream = "Context insurance policy" for infinite LLM memory.
|
|
||||||
Reference in New Issue
Block a user