chore: Remove backup files and add .gitignore
- Remove tracked backup files (*.bak, *.neuralstream.bak) - Remove debug_curator.py - Add comprehensive .gitignore - Prevents future backup/debug files from being committed
This commit is contained in:
root
59
.gitignore
vendored
Normal file
59
.gitignore
vendored
Normal file
@@ -0,0 +1,59 @@
|
||||
# TrueRecall v2 - .gitignore
|
||||
# Files that should not be committed to version control
|
||||
|
||||
# Backup files
|
||||
*.bak
|
||||
*.bak.*
|
||||
*~
|
||||
*.swp
|
||||
*.swo
|
||||
*.orig
|
||||
*.save
|
||||
|
||||
# Debug and temporary files
|
||||
debug_*
|
||||
test_*
|
||||
temp_*
|
||||
*.tmp
|
||||
|
||||
# Log files
|
||||
*.log
|
||||
logs/
|
||||
log/
|
||||
|
||||
# Python cache
|
||||
__pycache__/
|
||||
*.py[cod]
|
||||
*$py.class
|
||||
*.so
|
||||
.Python
|
||||
|
||||
# Virtual environments
|
||||
venv/
|
||||
env/
|
||||
ENV/
|
||||
|
||||
# IDE/editor files
|
||||
.vscode/
|
||||
.idea/
|
||||
*.iml
|
||||
|
||||
# OS files
|
||||
.DS_Store
|
||||
Thumbs.db
|
||||
|
||||
# Environment/config with secrets (if they exist locally)
|
||||
.env
|
||||
.env.local
|
||||
config.local.json
|
||||
|
||||
# Local data that shouldn't be shared
|
||||
data/
|
||||
datasets/
|
||||
*.db
|
||||
*.sqlite
|
||||
|
||||
# Build artifacts
|
||||
build/
|
||||
dist/
|
||||
*.egg-info/
|
||||
@@ -1,308 +0,0 @@
|
||||
# TrueRecall v2
|
||||
|
||||
**Project:** Gem extraction and memory recall system
|
||||
**Status:** ✅ Active
|
||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/`
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
TrueRecall extracts "gems" (key insights) from conversations and stores them for context injection. It's the memory curation system that powers Kimi's contextual awareness.
|
||||
|
||||
### Current Flow
|
||||
|
||||
```
|
||||
1. Conversation happens → Real-time watcher → memories_tr (Qdrant)
|
||||
2. Daily (2:45 AM) → Curator reads memories_tr → extracts gems
|
||||
3. Gems stored → gems_tr collection (Qdrant)
|
||||
4. On each turn → memory-qdrant plugin injects gems as context
|
||||
```
|
||||
|
||||
**Verified:** 2026-02-24 — Real-time watcher capturing (12,223 → 12,228 points)
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Collections (Qdrant)
|
||||
|
||||
| Collection | Purpose | Content | Status |
|
||||
|------------|---------|---------|--------|
|
||||
| `memories_tr` | Full text storage | Every conversation turn (migrated from `kimi_memories`) | ✅ Active |
|
||||
| `gems_tr` | Gems (extracted insights) | Curated key points (for injection) | ✅ Active |
|
||||
| `true_recall` | Legacy gems | Archive of previously extracted gems | 📦 Preserved |
|
||||
| `kimi_memories` | Original collection | Backup (12,223 points preserved) | 📦 Preserved |
|
||||
|
||||
### Migration Script
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `migrate_memories.py` | Migrate data from `kimi_memories` → `memories_tr` with cleaning |
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python3 migrate_memories.py
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- Reads all points from `kimi_memories`
|
||||
- Cleans content (removes metadata, thinking tags)
|
||||
- Stores to `memories_tr`
|
||||
- Preserves original `kimi_memories`
|
||||
|
||||
---
|
||||
|
||||
### Components
|
||||
|
||||
| Component | Location | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| **memory-qdrant plugin** | `/root/.openclaw/extensions/memory-qdrant/` | Injects gems as context |
|
||||
| **Curation script** | `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` | Extracts gems |
|
||||
| **Curator prompt** | `/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md` | Instructions for gem extraction |
|
||||
|
||||
---
|
||||
|
||||
## File Locations
|
||||
|
||||
### Core Files
|
||||
|
||||
```
|
||||
/root/.openclaw/workspace/.projects/true-recall-v2/
|
||||
├── README.md # This file
|
||||
├── session.md # Development notes
|
||||
├── curator-prompt.md # Gem extraction prompt
|
||||
│
|
||||
├── tr-daily/ # Daily curation
|
||||
│ └── curate_from_qdrant.py # Main curator script
|
||||
│
|
||||
├── tr-compact/ # (Reserved for v2 expansion)
|
||||
│ └── hook.py # Compaction hook (not active)
|
||||
│
|
||||
├── tr-worker/ # (Reserved for v2 expansion)
|
||||
│ └── worker.py # Background worker (not active)
|
||||
│
|
||||
└── shared/ # Shared resources
|
||||
```
|
||||
|
||||
### Plugin Files (Located in OpenClaw Extensions)
|
||||
|
||||
> **Note:** These files live in `/root/.openclaw/extensions/`, not in this project folder. Documented here for reference.
|
||||
|
||||
| Actual Location | Project Reference |
|
||||
|----------------|-------------------|
|
||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code (capture + injection) |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
||||
| `/root/.openclaw/openclaw.json` | Plugin configuration |
|
||||
|
||||
### Configuration
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `/root/.openclaw/openclaw.json` | Plugin config (collectionName: gems_tr) |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
||||
|
||||
### Cron Jobs
|
||||
|
||||
| Time | Command | Purpose |
|
||||
|------|---------|---------|
|
||||
| 2:45 AM | `curate_memories.py` | Daily gem extraction (v1) → stores to gems_tr |
|
||||
| 2:50 AM | `archive_to_memories_tr.py` | Archive to memories_tr |
|
||||
|
||||
View cron: `crontab -l`
|
||||
|
||||
---
|
||||
|
||||
## Process Flow
|
||||
|
||||
### Step 1: Capture (Real-Time Watcher)
|
||||
|
||||
```
|
||||
OpenClaw Session ──→ Real-Time Watcher ──→ Qdrant (memories_tr)
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
||||
|
||||
**What it does:**
|
||||
- Watches session JSONL files in real-time
|
||||
- Parses each conversation turn (user + AI)
|
||||
- Embeds with `snowflake-arctic-embed2` via Ollama
|
||||
- Stores directly to `memories_tr` collection
|
||||
- Cleans content (removes metadata, thinking tags)
|
||||
|
||||
**Systemd Service:** `mem-qdrant-watcher.service`
|
||||
|
||||
**Status:** ✅ Created, ready to deploy
|
||||
|
||||
### Step 2: Curation (Daily)
|
||||
|
||||
```
|
||||
2:45 AM → curate_memories.py → memories_tr → qwen3 → gems_tr
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v1/tr-process/curate_memories.py`
|
||||
|
||||
**What it does:**
|
||||
1. Reads from Redis buffer (`mem:rob`)
|
||||
2. Passes to qwen3 (curator model)
|
||||
3. Extracts gems using prompt
|
||||
4. Stores gems to `gems_tr` collection
|
||||
|
||||
### Step 3: Injection
|
||||
|
||||
```
|
||||
User message → memory-qdrant plugin → Search gems_tr → Inject as context
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/extensions/memory-qdrant/index.ts`
|
||||
|
||||
**What it does:**
|
||||
1. Listens to `before_agent_start` events
|
||||
2. Embeds current prompt
|
||||
3. Searches `gems_tr` collection
|
||||
4. Returns `{ prependContext: "..." }` with gems
|
||||
5. Gems appear in my context (hidden from UI)
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### memory-qdrant Plugin
|
||||
|
||||
```json
|
||||
{
|
||||
"autoCapture": true,
|
||||
"autoRecall": true,
|
||||
"collectionName": "gems_tr",
|
||||
"embeddingModel": "snowflake-arctic-embed2",
|
||||
"maxRecallResults": 2,
|
||||
"minRecallScore": 0.7
|
||||
}
|
||||
```
|
||||
|
||||
**Key setting:** `collectionName: "gems_tr"` — tells plugin to inject gems, not full text.
|
||||
|
||||
---
|
||||
|
||||
## Gem Format
|
||||
|
||||
```json
|
||||
{
|
||||
"gem": "User prefers dark mode for interface",
|
||||
"context": "Discussed UI theme options",
|
||||
"snippet": "rob: I want dark mode\nKimi: Done",
|
||||
"categories": ["preference"],
|
||||
"importance": "high",
|
||||
"confidence": 0.95,
|
||||
"date": "2026-02-24",
|
||||
"conversation_id": "uuid",
|
||||
"turn_range": "5-7"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Commands
|
||||
|
||||
### Check Qdrant Collections
|
||||
|
||||
```bash
|
||||
curl -s "http://10.0.0.40:6333/collections" | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Check Collection Points
|
||||
|
||||
```bash
|
||||
# memories_tr (full text)
|
||||
curl -s -X POST "http://10.0.0.40:6333/collections/memories_tr/points/scroll" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
||||
|
||||
# gems_tr (gems)
|
||||
curl -s -X POST "http://10.0.0.40:6333/collections/gems_tr/points/scroll" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Check Cron Jobs
|
||||
|
||||
```bash
|
||||
crontab -l | grep -E "true|recall|curate"
|
||||
```
|
||||
|
||||
### Check Plugin Logs
|
||||
|
||||
```bash
|
||||
tail -50 /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: `<relevant-memories>` showing in UI
|
||||
|
||||
**Cause:** Plugin was using `memories_tr` (full text) instead of `gems_tr` (gems)
|
||||
|
||||
**Fix:** Set `"collectionName": "gems_tr"` in openclaw.json
|
||||
|
||||
**Reference:** KB entry `relevant-memories-ui-issue.md`
|
||||
|
||||
### Issue: No gems being extracted
|
||||
|
||||
**Check:**
|
||||
1. Is memories_tr populated? `curl .../collections/memories_tr/points/scroll`
|
||||
2. Is curator prompt valid? `cat curator-prompt.md`
|
||||
3. Is qwen3 available? `curl -s http://10.0.0.10:11434/api/tags`
|
||||
|
||||
---
|
||||
|
||||
## Related Projects
|
||||
|
||||
| Project | Location | Purpose |
|
||||
|---------|----------|---------|
|
||||
| **true-recall-v1** | `/.projects/true-recall-v1/` | Original (Redis-based) |
|
||||
| **memory-qdrant** | `/root/.openclaw/extensions/memory-qdrant/` | Plugin |
|
||||
| **mem-redis** | `/root/.openclaw/workspace/skills/mem-redis/` | Redis utilities |
|
||||
|
||||
---
|
||||
|
||||
## Backups
|
||||
|
||||
| File | Backup Location |
|
||||
|------|-----------------|
|
||||
| openclaw.json | `/root/.openclaw/openclaw.json.bak.2026-02-24` |
|
||||
| Plugin | `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24` |
|
||||
|
||||
---
|
||||
|
||||
## Final Status (2026-02-24 15:08)
|
||||
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| Collection `kimi_memories` → `memories_tr` | ✅ Migrated (12,223 points) |
|
||||
| Real-time watcher | ✅ **Deployed & verified** (12,228 points, +5 new) |
|
||||
| Collection `gems_tr` | ✅ Active (5 gems stored) |
|
||||
| Curator v2 | ✅ Working - tested with 327 turns |
|
||||
| Config | ✅ Updated to `gems_tr` |
|
||||
| Cron jobs | ✅ Cleaned |
|
||||
| Documentation | ✅ Updated |
|
||||
|
||||
### Daily Schedule (Simplified)
|
||||
|
||||
| Time | Job | Flow |
|
||||
|------|-----|------|
|
||||
| Continuous | autoCapture | Conversation → `memories_tr` |
|
||||
| 2:45 AM | Curator v2 | `memories_tr` → `gems_tr` |
|
||||
| Each turn | Injection | `gems_tr` → Context |
|
||||
|
||||
**Redis usage:** Disabled for memory. Used only for `delayed:notifications` queue.
|
||||
|
||||
**Auto-Capture Status:** ✅ **Working** - Real-time watcher deployed and verified
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-02-24
|
||||
**Project Lead:** Rob
|
||||
**AI Assistant:** Kimi 🎙️
|
||||
@@ -1,325 +0,0 @@
|
||||
# TrueRecall v2
|
||||
|
||||
**Project:** Gem extraction and memory recall system
|
||||
**Status:** ✅ Active
|
||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/`
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Architecture](#architecture)
|
||||
- [Collections](#collections-qdrant)
|
||||
- [Components](#components)
|
||||
- [Process Flow](#process-flow)
|
||||
- [Configuration](#configuration)
|
||||
- [Validation Commands](#validation-commands)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Related Projects](#related-projects)
|
||||
- [Backups](#backups)
|
||||
- [Final Status](#final-status-2026-02-24-1508)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
TrueRecall extracts "gems" (key insights) from conversations and stores them for context injection. It's the memory curation system that powers Kimi's contextual awareness.
|
||||
|
||||
### Current Flow
|
||||
|
||||
```
|
||||
1. Conversation happens → Real-time watcher → memories_tr (Qdrant)
|
||||
2. Daily (2:45 AM) → Curator reads memories_tr → extracts gems
|
||||
3. Gems stored → gems_tr collection (Qdrant)
|
||||
4. On each turn → memory-qdrant plugin injects gems as context
|
||||
```
|
||||
|
||||
**Verified:** 2026-02-24 — Real-time watcher capturing (12,223 → 12,228 points)
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Collections (Qdrant)
|
||||
|
||||
| Collection | Purpose | Content | Status |
|
||||
|------------|---------|---------|--------|
|
||||
| `memories_tr` | Full text storage | Every conversation turn (migrated from `kimi_memories`) | ✅ Active |
|
||||
| `gems_tr` | Gems (extracted insights) | Curated key points (for injection) | ✅ Active |
|
||||
| `true_recall` | Legacy gems | Archive of previously extracted gems | 📦 Preserved |
|
||||
| `kimi_memories` | Original collection | Backup (12,223 points preserved) | 📦 Preserved |
|
||||
|
||||
### Migration Script
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `migrate_memories.py` | Migrate data from `kimi_memories` → `memories_tr` with cleaning |
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python3 migrate_memories.py
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- Reads all points from `kimi_memories`
|
||||
- Cleans content (removes metadata, thinking tags)
|
||||
- Stores to `memories_tr`
|
||||
- Preserves original `kimi_memories`
|
||||
|
||||
---
|
||||
|
||||
### Components
|
||||
|
||||
| Component | Location | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| **memory-qdrant plugin** | `/root/.openclaw/extensions/memory-qdrant/` | Injects gems as context |
|
||||
| **Curation script** | `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` | Extracts gems |
|
||||
| **Curator prompt** | `/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md` | Instructions for gem extraction |
|
||||
|
||||
---
|
||||
|
||||
## File Locations
|
||||
|
||||
### Core Files
|
||||
|
||||
```
|
||||
/root/.openclaw/workspace/.projects/true-recall-v2/
|
||||
├── README.md # This file
|
||||
├── session.md # Development notes
|
||||
├── curator-prompt.md # Gem extraction prompt
|
||||
│
|
||||
├── tr-daily/ # Daily curation
|
||||
│ └── curate_from_qdrant.py # Main curator script
|
||||
│
|
||||
├── tr-compact/ # (Reserved for v2 expansion)
|
||||
│ └── hook.py # Compaction hook (not active)
|
||||
│
|
||||
├── tr-worker/ # (Reserved for v2 expansion)
|
||||
│ └── worker.py # Background worker (not active)
|
||||
│
|
||||
└── shared/ # Shared resources
|
||||
```
|
||||
|
||||
### Plugin Files (Located in OpenClaw Extensions)
|
||||
|
||||
> **Note:** These files live in `/root/.openclaw/extensions/`, not in this project folder. Documented here for reference.
|
||||
|
||||
| Actual Location | Project Reference |
|
||||
|----------------|-------------------|
|
||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code (capture + injection) |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
||||
| `/root/.openclaw/openclaw.json` | Plugin configuration |
|
||||
|
||||
### Configuration
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `/root/.openclaw/openclaw.json` | Plugin config (collectionName: gems_tr) |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
||||
|
||||
### Cron Jobs
|
||||
|
||||
| Time | Command | Purpose |
|
||||
|------|---------|---------|
|
||||
| 2:45 AM | `curate_memories.py` | Daily gem extraction (v1) → stores to gems_tr |
|
||||
| 2:50 AM | `archive_to_memories_tr.py` | Archive to memories_tr |
|
||||
|
||||
View cron: `crontab -l`
|
||||
|
||||
---
|
||||
|
||||
## Process Flow
|
||||
|
||||
### Step 1: Capture (Real-Time Watcher)
|
||||
|
||||
```
|
||||
OpenClaw Session ──→ Real-Time Watcher ──→ Qdrant (memories_tr)
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
||||
|
||||
**What it does:**
|
||||
- Watches session JSONL files in real-time
|
||||
- Parses each conversation turn (user + AI)
|
||||
- Embeds with `snowflake-arctic-embed2` via Ollama
|
||||
- Stores directly to `memories_tr` collection
|
||||
- Cleans content (removes metadata, thinking tags)
|
||||
|
||||
**Systemd Service:** `mem-qdrant-watcher.service`
|
||||
|
||||
**Status:** ✅ Created, ready to deploy
|
||||
|
||||
### Step 2: Curation (Daily)
|
||||
|
||||
```
|
||||
2:45 AM → curate_memories.py → memories_tr → qwen3 → gems_tr
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v1/tr-process/curate_memories.py`
|
||||
|
||||
**What it does:**
|
||||
1. Reads from Redis buffer (`mem:rob`)
|
||||
2. Passes to qwen3 (curator model)
|
||||
3. Extracts gems using prompt
|
||||
4. Stores gems to `gems_tr` collection
|
||||
|
||||
### Step 3: Injection
|
||||
|
||||
```
|
||||
User message → memory-qdrant plugin → Search gems_tr → Inject as context
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/extensions/memory-qdrant/index.ts`
|
||||
|
||||
**What it does:**
|
||||
1. Listens to `before_agent_start` events
|
||||
2. Embeds current prompt
|
||||
3. Searches `gems_tr` collection
|
||||
4. Returns `{ prependContext: "..." }` with gems
|
||||
5. Gems appear in my context (hidden from UI)
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### memory-qdrant Plugin
|
||||
|
||||
```json
|
||||
{
|
||||
"autoCapture": true,
|
||||
"autoRecall": true,
|
||||
"collectionName": "gems_tr",
|
||||
"embeddingModel": "snowflake-arctic-embed2",
|
||||
"maxRecallResults": 2,
|
||||
"minRecallScore": 0.7
|
||||
}
|
||||
```
|
||||
|
||||
**Key setting:** `collectionName: "gems_tr"` — tells plugin to inject gems, not full text.
|
||||
|
||||
---
|
||||
|
||||
## Gem Format
|
||||
|
||||
```json
|
||||
{
|
||||
"gem": "User prefers dark mode for interface",
|
||||
"context": "Discussed UI theme options",
|
||||
"snippet": "rob: I want dark mode\nKimi: Done",
|
||||
"categories": ["preference"],
|
||||
"importance": "high",
|
||||
"confidence": 0.95,
|
||||
"date": "2026-02-24",
|
||||
"conversation_id": "uuid",
|
||||
"turn_range": "5-7"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Commands
|
||||
|
||||
### Check Qdrant Collections
|
||||
|
||||
```bash
|
||||
curl -s "http://10.0.0.40:6333/collections" | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Check Collection Points
|
||||
|
||||
```bash
|
||||
# memories_tr (full text)
|
||||
curl -s -X POST "http://10.0.0.40:6333/collections/memories_tr/points/scroll" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
||||
|
||||
# gems_tr (gems)
|
||||
curl -s -X POST "http://10.0.0.40:6333/collections/gems_tr/points/scroll" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Check Cron Jobs
|
||||
|
||||
```bash
|
||||
crontab -l | grep -E "true|recall|curate"
|
||||
```
|
||||
|
||||
### Check Plugin Logs
|
||||
|
||||
```bash
|
||||
tail -50 /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: `<relevant-memories>` showing in UI
|
||||
|
||||
**Cause:** Plugin was using `memories_tr` (full text) instead of `gems_tr` (gems)
|
||||
|
||||
**Fix:** Set `"collectionName": "gems_tr"` in openclaw.json
|
||||
|
||||
**Reference:** KB entry `relevant-memories-ui-issue.md`
|
||||
|
||||
### Issue: No gems being extracted
|
||||
|
||||
**Check:**
|
||||
1. Is memories_tr populated? `curl .../collections/memories_tr/points/scroll`
|
||||
2. Is curator prompt valid? `cat curator-prompt.md`
|
||||
3. Is qwen3 available? `curl -s http://10.0.0.10:11434/api/tags`
|
||||
|
||||
---
|
||||
|
||||
## Related Projects
|
||||
|
||||
| Project | Location | Purpose |
|
||||
|---------|----------|---------|
|
||||
| **true-recall-v1** | `/.projects/true-recall-v1/` | Original (Redis-based) |
|
||||
| **memory-qdrant** | `/root/.openclaw/extensions/memory-qdrant/` | Plugin |
|
||||
| **mem-redis** | `/root/.openclaw/workspace/skills/mem-redis/` | Redis utilities |
|
||||
|
||||
---
|
||||
|
||||
## Backups
|
||||
|
||||
| File | Backup Location |
|
||||
|------|-----------------|
|
||||
| openclaw.json | `/root/.openclaw/openclaw.json.bak.2026-02-24` |
|
||||
| Plugin | `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24` |
|
||||
|
||||
---
|
||||
|
||||
## Final Status (2026-02-24 15:08)
|
||||
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| Collection `kimi_memories` → `memories_tr` | ✅ Migrated (12,223 points) |
|
||||
| Real-time watcher | ✅ **Deployed & verified** (12,228 points, +5 new) |
|
||||
| Collection `gems_tr` | ✅ Active (5 gems stored) |
|
||||
| Curator v2 | ✅ Working - tested with 327 turns |
|
||||
| Config | ✅ Updated to `gems_tr` |
|
||||
| Cron jobs | ✅ Cleaned |
|
||||
| Documentation | ✅ Updated |
|
||||
|
||||
### Daily Schedule (Simplified)
|
||||
|
||||
| Time | Job | Flow |
|
||||
|------|-----|------|
|
||||
| Continuous | autoCapture | Conversation → `memories_tr` |
|
||||
| 2:45 AM | Curator v2 | `memories_tr` → `gems_tr` |
|
||||
| Each turn | Injection | `gems_tr` → Context |
|
||||
|
||||
**Redis usage:** Disabled for memory. Used only for `delayed:notifications` queue.
|
||||
|
||||
**Auto-Capture Status:** ✅ **Working** - Real-time watcher deployed and verified
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-02-24
|
||||
**Project Lead:** Rob
|
||||
**AI Assistant:** Kimi 🎙️
|
||||
|
||||
@@ -1,183 +0,0 @@
|
||||
# NeuralStream
|
||||
|
||||
**Neural streaming memory for OpenClaw with gem-based context injection.**
|
||||
|
||||
## Overview
|
||||
|
||||
NeuralStream extracts high-value insights ("gems") from conversation batches using qwen3, stores them in Qdrant, and injects relevant gems into context on each new turn. This creates **infinite effective context** — the active window stays small, but semantically relevant gems from all past conversations are always retrievable.
|
||||
|
||||
## Core Concept
|
||||
|
||||
| Traditional Memory | NeuralStream |
|
||||
|-------------------|--------------|
|
||||
| Context lost on `/new` | Gems persist in Qdrant |
|
||||
| Full history or generic summary | Semantic gem retrieval |
|
||||
| Static context window | Dynamic injection |
|
||||
| Survives compaction only | Survives session reset |
|
||||
| **Limited context** | **Infinite effective context** |
|
||||
|
||||
## How It Works
|
||||
|
||||
### Capture → Extract → Store → Retrieve
|
||||
|
||||
1. **Capture:** Every turn buffered to Redis (reuses mem-redis-watcher)
|
||||
2. **Extract:** Batch of 5 turns → qwen3 (with 256k context) extracts structured gems
|
||||
3. **Store:** Gems embedded + stored in Qdrant `neuralstream`
|
||||
4. **Retrieve:** Each new turn → semantic search → inject top-10 gems
|
||||
|
||||
### Hybrid Triggers (Three-way)
|
||||
|
||||
| Trigger | Condition | Purpose |
|
||||
|---------|-----------|---------|
|
||||
| Batch | Every 5 turns | Normal extraction |
|
||||
| Context | 50% usage (`ctx.getContextUsage()`) | Proactive pre-compaction |
|
||||
| Timer | 15 min idle | Safety net |
|
||||
|
||||
**Context Awareness:** qwen3 receives up to 256k tokens of history for understanding, but only extracts gems from the last N turns (avoiding current context).
|
||||
|
||||
All gems survive `/new`, `/reset`, and compaction via Qdrant persistence.
|
||||
|
||||
## Architecture
|
||||
|
||||
NeuralStream is the **middle layer** — extraction intelligence on top of existing infrastructure:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ EXISTING: mem-redis-watcher │
|
||||
│ Every turn → Redis buffer │
|
||||
└──────────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
┌──────────▼──────────┐
|
||||
│ NeuralStream │
|
||||
│ - Batch reader │
|
||||
│ - Gem extractor │
|
||||
│ - Qdrant store │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
┌──────────▼──────────┐
|
||||
│ EXISTING: │
|
||||
│ qdrant-memory │
|
||||
│ Semantic search │
|
||||
│ Context injection │
|
||||
└─────────────────────┘
|
||||
```
|
||||
|
||||
## Technical Reference
|
||||
|
||||
### Native Context Monitoring
|
||||
|
||||
```typescript
|
||||
// In turn_end hook
|
||||
const usage = ctx.getContextUsage();
|
||||
// usage.tokens, usage.contextWindow, usage.percent
|
||||
// Trigger extraction when usage.percent >= threshold
|
||||
```
|
||||
|
||||
### Primary Hook: turn_end
|
||||
|
||||
```typescript
|
||||
pi.on("turn_end", async (event, ctx) => {
|
||||
const { turnIndex, message, toolResults } = event;
|
||||
|
||||
// Buffer turn to Redis
|
||||
// Check ctx.getContextUsage().percent
|
||||
// If batch >= 5 OR percent >= 50%: extract
|
||||
});
|
||||
```
|
||||
|
||||
### Timer Fallback
|
||||
|
||||
```bash
|
||||
# Cron every 10 min
|
||||
# Check neuralstream:buffer age > 15 min
|
||||
# If yes: extract from partial batch
|
||||
```
|
||||
|
||||
### Context-Aware Extraction
|
||||
|
||||
- Feed qwen3: Up to 256k tokens (full history for context)
|
||||
- Extract from: Last `batch_size` turns only
|
||||
- Benefit: Rich understanding without gemming current context
|
||||
|
||||
## Gem Format
|
||||
|
||||
```json
|
||||
{
|
||||
"gem_id": "uuid",
|
||||
"content": "Distilled insight/fact/decision",
|
||||
"summary": "One-line for quick scanning",
|
||||
"topics": ["docker", "redis", "architecture"],
|
||||
"importance": 0.9,
|
||||
"source": {
|
||||
"session_id": "uuid",
|
||||
"date": "2026-02-23",
|
||||
"turn_range": "15-20"
|
||||
},
|
||||
"tags": ["decision", "fact", "preference", "todo", "code"],
|
||||
"created_at": "2026-02-23T15:26:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration (All Tunable)
|
||||
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| batch_size | 5 | Turns per extraction |
|
||||
| context_threshold | 50% | Token % trigger (40-80% range) |
|
||||
| idle_timeout | 15 min | Timer trigger threshold |
|
||||
| gem_model | qwen3 | Extraction LLM (256k context) |
|
||||
| max_gems_injected | 10 | Per-turn limit |
|
||||
| embedding | snowflake-arctic-embed2 | Same as kimi_memories |
|
||||
| collection | neuralstream | Qdrant (1024 dims, Cosine) |
|
||||
|
||||
## Qdrant Schema
|
||||
|
||||
**Collection:** `neuralstream`
|
||||
- Vector size: 1024
|
||||
- Distance: Cosine
|
||||
- On-disk payload: true
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
.projects/neuralstream/
|
||||
├── README.md # This file
|
||||
├── session.md # Development log & state
|
||||
├── prompt.md # (TBD) qwen3 extraction prompt
|
||||
└── src/ # (TBD) Implementation
|
||||
├── extract.ts # Gem extraction logic
|
||||
├── store.ts # Qdrant storage
|
||||
└── inject.ts # Context injection
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
- [x] Architecture defined (v2.2 context-aware)
|
||||
- [x] Native context monitoring validated (ctx.getContextUsage)
|
||||
- [x] Naming finalized (NeuralStream, alias: ns)
|
||||
- [x] Hook research completed
|
||||
- [x] Qdrant collection created (`neuralstream`)
|
||||
- [x] Gem format proposed
|
||||
- [x] Infrastructure decision (reuse Redis/Qdrant)
|
||||
- [ ] Extraction prompt design
|
||||
- [ ] Implementation
|
||||
- [ ] Testing
|
||||
|
||||
## Backups
|
||||
|
||||
- Local: `/root/.openclaw/workspace/.projects/neuralstream/`
|
||||
- Remote: `deb2:/root/.projects/neuralstream/` (build/test only)
|
||||
- kimi_kb: Research entries stored
|
||||
|
||||
## Related Projects
|
||||
|
||||
- **True Recall:** Gem extraction inspiration
|
||||
- **OpenClaw:** Host platform
|
||||
- **kimi_memories:** Shared Qdrant infrastructure
|
||||
- **mem-redis-watcher:** Existing capture layer
|
||||
|
||||
---
|
||||
|
||||
**Created:** 2026-02-23
|
||||
**Alias:** ns
|
||||
**Purpose:** Infinite context for LLMs
|
||||
106
debug_curator.py
106
debug_curator.py
@@ -1,106 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Debug curator with real data"""
|
||||
import json
|
||||
import requests
|
||||
import urllib.request
|
||||
|
||||
QDRANT_URL = "http://10.0.0.40:6333"
|
||||
SOURCE_COLLECTION = "kimi_memories"
|
||||
|
||||
# Get sample turns from real data
|
||||
filter_data = {
|
||||
"must": [
|
||||
{"key": "user_id", "match": {"value": "rob"}},
|
||||
{"key": "date", "match": {"value": "2026-02-23"}}
|
||||
]
|
||||
}
|
||||
|
||||
req = urllib.request.Request(
|
||||
f"{QDRANT_URL}/collections/{SOURCE_COLLECTION}/points/scroll",
|
||||
data=json.dumps({"limit": 5, "with_payload": True, "filter": filter_data}).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST"
|
||||
)
|
||||
|
||||
with urllib.request.urlopen(req, timeout=30) as response:
|
||||
result = json.loads(response.read().decode())
|
||||
points = result.get("result", {}).get("points", [])
|
||||
|
||||
turns = []
|
||||
for point in points:
|
||||
payload = point.get("payload", {})
|
||||
user_msg = payload.get("user_message", "")
|
||||
ai_msg = payload.get("ai_response", "")
|
||||
|
||||
if user_msg or ai_msg:
|
||||
turn = {
|
||||
"turn": payload.get("turn_number", 0),
|
||||
"user_id": payload.get("user_id", "rob"),
|
||||
"user": user_msg[:300], # Truncate
|
||||
"ai": ai_msg[:300], # Truncate
|
||||
"conversation_id": payload.get("conversation_id", ""),
|
||||
"timestamp": payload.get("created_at", ""),
|
||||
"date": payload.get("date", "2026-02-23")
|
||||
}
|
||||
turns.append(turn)
|
||||
|
||||
turns.sort(key=lambda x: (x.get("conversation_id", ""), x.get("turn", 0)))
|
||||
|
||||
print(f"Got {len(turns)} turns")
|
||||
print("Sample:")
|
||||
for t in turns[:2]:
|
||||
print(f" User: {t['user'][:100]}...")
|
||||
print(f" AI: {t['ai'][:100]}...")
|
||||
|
||||
# Now test with curator
|
||||
with open('/root/.openclaw/workspace/.projects/true-recall-v1/curator-prompt.md') as f:
|
||||
prompt = f.read()
|
||||
|
||||
conversation_json = json.dumps(turns[:5], indent=2)
|
||||
|
||||
prompt_text = f"""## Input Conversation
|
||||
|
||||
```json
|
||||
{conversation_json}
|
||||
```
|
||||
|
||||
## Output
|
||||
"""
|
||||
|
||||
response = requests.post(
|
||||
'http://10.0.0.10:11434/api/generate',
|
||||
json={
|
||||
'model': 'qwen3:4b-instruct',
|
||||
'system': prompt,
|
||||
'prompt': prompt_text,
|
||||
'stream': False,
|
||||
'options': {'temperature': 0.1, 'num_predict': 3000}
|
||||
},
|
||||
timeout=120
|
||||
)
|
||||
|
||||
result = response.json()
|
||||
output = result.get('response', '').strip()
|
||||
|
||||
print("\n=== CURATOR OUTPUT ===")
|
||||
print(output[:3000])
|
||||
print("\n=== TRYING TO PARSE ===")
|
||||
|
||||
# Try to parse
|
||||
try:
|
||||
if '```json' in output:
|
||||
parsed = output.split('```json')[1].split('```')[0].strip()
|
||||
gems = json.loads(parsed)
|
||||
print(f"Parsed {len(gems)} gems")
|
||||
elif '```' in output:
|
||||
parsed = output.split('```')[1].split('```')[0].strip()
|
||||
gems = json.loads(parsed)
|
||||
print(f"Parsed {len(gems)} gems")
|
||||
else:
|
||||
gems = json.loads(output)
|
||||
print(f"Parsed {len(gems)} gems")
|
||||
except Exception as e:
|
||||
print(f"Parse error: {e}")
|
||||
print("Trying raw parse...")
|
||||
gems = json.loads(output.strip())
|
||||
print(f"Parsed {len(gems)} gems")
|
||||
@@ -1,224 +0,0 @@
|
||||
# TrueRecall v2 - Session Notes
|
||||
|
||||
**Last Updated:** 2026-02-24 15:08 CST
|
||||
|
||||
---
|
||||
|
||||
## Summary of Changes (2026-02-24)
|
||||
|
||||
### Collection Migration
|
||||
|
||||
| Action | From | To | Status |
|
||||
|--------|------|----|--------|
|
||||
| **Rename** | `kimi_memories` | `memories_tr` | ✅ Done + data migrated (12,223 points) |
|
||||
| **Create** | — | `gems_tr` | ✅ Created (empty, ready for curation) |
|
||||
| **Archive** | `true_recall` | `true_recall` | ✅ Preserved (existing data kept) |
|
||||
|
||||
### Dual Collection Support (NEW - 14:10)
|
||||
|
||||
**Problem:** Auto-capture was saving to `gems_tr` because that's where recall pulled from. This was wrong.
|
||||
|
||||
**Solution:** Added `captureCollection` option to memory-qdrant plugin:
|
||||
- `collectionName`: `gems_tr` — for recall/injection (gems only)
|
||||
- `captureCollection`: `memories_tr` — for auto-capture (full conversations)
|
||||
|
||||
**Files Modified:**
|
||||
1. `/root/.openclaw/extensions/memory-qdrant/config.ts` — Added `captureCollection` option
|
||||
2. `/root/.openclaw/extensions/memory-qdrant/index.ts` — Uses `dbRecall` and `dbCapture` separately
|
||||
3. `/root/.openclaw/openclaw.json` — Added `"captureCollection": "memories_tr"`
|
||||
|
||||
### Auto-Capture Architecture (Real-Time Watcher)
|
||||
|
||||
**Mechanism:** `realtime_qdrant_watcher.py` daemon
|
||||
- Watches OpenClaw session JSONL files in real-time
|
||||
- Parses each conversation turn
|
||||
- Embeds with `snowflake-arctic-embed2` via Ollama
|
||||
- Stores directly to `memories_tr` collection (no Redis)
|
||||
- Cleans content (removes metadata, thinking tags)
|
||||
|
||||
**Files:**
|
||||
- Script: `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
||||
- Service: `/root/.openclaw/workspace/skills/qdrant-memory/mem-qdrant-watcher.service`
|
||||
|
||||
**Status:** ✅ Deployed and running
|
||||
|
||||
**Deployment:**
|
||||
- Stopped: `mem-redis-watcher`
|
||||
- Started: `mem-qdrant-watcher`
|
||||
- Status: Active (PID 84465)
|
||||
- Verified: ✅ Capturing (points: 12,223 → 12,224)
|
||||
|
||||
### Current State (2026-02-24 15:08)
|
||||
|
||||
| Collection | Purpose | Points | Last Update |
|
||||
|------------|---------|--------|-------------|
|
||||
| `memories_tr` | Full text (autoCapture via watcher) | 12,228 | **Live** |
|
||||
| `gems_tr` | Gems (for injection) | 5 | Feb 24 |
|
||||
|
||||
**Flow:**
|
||||
```
|
||||
OpenClaw Session → Real-Time Watcher → memories_tr (Qdrant)
|
||||
↓
|
||||
Daily Curator (2:45 AM)
|
||||
↓
|
||||
gems_tr (Qdrant)
|
||||
↓
|
||||
memory-qdrant plugin → Injection
|
||||
```
|
||||
|
||||
### Files Modified Today
|
||||
|
||||
1. `/root/.openclaw/openclaw.json` — Changed collectionName to `gems_tr`
|
||||
2. `/root/.openclaw/extensions/memory-qdrant/config.ts` — Default to `memories_tr`
|
||||
3. `/root/.openclaw/extensions/memory-qdrant/openclaw.plugin.json` — Defaults to `memories_tr`
|
||||
4. `/root/.openclaw/workspace/skills/mem-redis/scripts/save_mem.py` — Added `--silent` flag
|
||||
5. `/root/.openclaw/workspace/HEARTBEAT.md` — Updated to use `--silent`
|
||||
6. `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` — TARGET_COLLECTION = `gems_tr`
|
||||
7. `/root/.openclaw/workspace/SOUL.md` — Updated collection references
|
||||
8. `/root/.openclaw/workspace/kb/relevant-memories-ui-issue.md` — Updated documentation
|
||||
|
||||
### Migration Script Created
|
||||
|
||||
- `/root/.openclaw/workspace/.projects/true-recall-v2/migrate_memories.py`
|
||||
- Migrated 12,223 points from `kimi_memories` → `memories_tr`
|
||||
- Cleaned noise (metadata, thinking tags) during migration
|
||||
- Preserved original `kimi_memories` as backup
|
||||
|
||||
### Current Collections (Qdrant)
|
||||
|
||||
| Collection | Purpose | Points |
|
||||
|------------|---------|--------|
|
||||
| `memories_tr` | Full text (live capture) | **12,228** |
|
||||
| `gems_tr` | Gems (for injection) | 5 |
|
||||
| `true_recall` | Legacy gems archive | existing |
|
||||
| `kimi_memories` | Original (backup) | 12,223 |
|
||||
|
||||
---
|
||||
|
||||
## Final Status (2026-02-24)
|
||||
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| **Collections** | 4 active (memories_tr, gems_tr, true_recall, kimi_memories) |
|
||||
| **Curator v2** | ✅ Tested & working - 327 turns → 5 gems |
|
||||
| **Config** | ✅ Using `gems_tr` for injection |
|
||||
| **Cron** | ✅ Simplified - only essential jobs |
|
||||
| **Redis** | ✅ Only for notifications (not memory) |
|
||||
|
||||
### Collection Summary
|
||||
|
||||
| Collection | Points | Purpose | Status |
|
||||
|------------|--------|---------|--------|
|
||||
| `memories_tr` | **12,228** | Full text (live capture) | ✅ Active |
|
||||
| `gems_tr` | 5 | Gems (curated) | ✅ Active |
|
||||
| `true_recall` | existing | Legacy archive | 📦 Preserved |
|
||||
| `kimi_memories` | 12,223 | Original backup | 📦 Preserved |
|
||||
|
||||
### Cron Schedule (Cleaned)
|
||||
|
||||
| Time | Job |
|
||||
|------|-----|
|
||||
| 2:45 AM | Curator v2: memories_tr → gems_tr |
|
||||
| 2:20 AM | File backup |
|
||||
| 2:00 AM | Log monitoring |
|
||||
|
||||
**Removed:** 2:15 AM Redis backup, 2:50 AM archive, 5-min cron_capture (all redundant)
|
||||
|
||||
---
|
||||
|
||||
## What We Did Today
|
||||
|
||||
### Issue: `<relevant-memories>` showing in UI
|
||||
|
||||
**Problem:** The `<relevant-memories>` block was showing full conversation text in webchat UI. It should be hidden/internal.
|
||||
|
||||
**Investigation:**
|
||||
1. Initially thought it was `save_mem.py` output (wrong)
|
||||
2. Then thought it was HTML comment issue (wrong)
|
||||
3. Found root cause: memory-qdrant plugin was recalling from `memories_tr` (full text) instead of `gems_tr` (gems)
|
||||
|
||||
**Solution Applied:**
|
||||
Changed `openclaw.json` config:
|
||||
- Before: `"collectionName": "memories_tr"`
|
||||
- After: `"collectionName": "gems_tr"`
|
||||
|
||||
**Result:** ✅ Fixed! Now injection uses gems from true_recall, not full text.
|
||||
|
||||
### Issue: Auto-Capture Architecture Change
|
||||
|
||||
**Problem:** Real-time capture was going through Redis, we needed direct Qdrant storage
|
||||
|
||||
**Solution:**
|
||||
1. Created `realtime_qdrant_watcher.py` — watches session JSONL, embeds, stores directly to Qdrant
|
||||
2. Created `mem-qdrant-watcher.service` — systemd service for the watcher
|
||||
3. Deployed and verified:
|
||||
- Points before: 12,223
|
||||
- Points after test messages: 12,228
|
||||
- Verified new captures have correct structure (role, content, date, source)
|
||||
|
||||
**Status:** ✅ **Deployed and working**
|
||||
|
||||
### Other Changes Made
|
||||
|
||||
1. **Plugin dual collection support** — Added `captureCollection` option
|
||||
2. **Session/Readme updates** — Clarified architecture
|
||||
|
||||
### Files Modified
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `/root/.openclaw/openclaw.json` | collectionName: memories_tr → gems_tr, added captureCollection |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Added captureCollection option |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Dual collection support, debug logging |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/openclaw.plugin.json` | Added captureCollection to schema |
|
||||
|
||||
### Files Created (NEW)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` | Real-time watcher daemon (Qdrant direct) |
|
||||
| `/root/.openclaw/workspace/skills/qdrant-memory/mem-qdrant-watcher.service` | Systemd service file |
|
||||
|
||||
### Deployment
|
||||
|
||||
| Service | Action | Status |
|
||||
|---------|--------|--------|
|
||||
| `mem-redis-watcher` | Stopped | ✅ |
|
||||
| `mem-qdrant-watcher` | Started | ✅ Active |
|
||||
|
||||
### Backups Created
|
||||
|
||||
- `/root/.openclaw/openclaw.json.bak.2026-02-24`
|
||||
- `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24`
|
||||
|
||||
---
|
||||
|
||||
## Current Status
|
||||
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| **Curation (daily)** | v1 cron at 2:45 AM |
|
||||
| **Injection** | ✅ Working, uses gems_tr |
|
||||
| **Collection** | gems_tr |
|
||||
| **KB** | Updated |
|
||||
|
||||
---
|
||||
|
||||
## What Still Needs Doing
|
||||
|
||||
1. ~~Test autoCapture (cleaned content to memories_tr)~~ ✅ Done
|
||||
2. Test v2 curator (read from Qdrant, not Redis) — Next step
|
||||
3. Full validation 2x
|
||||
|
||||
---
|
||||
|
||||
## Session Recovery
|
||||
|
||||
If starting new session:
|
||||
1. Read this session.md
|
||||
2. Read README.md for architecture
|
||||
3. Read KB for issue history
|
||||
|
||||
---
|
||||
|
||||
**Next:** Test v2 curator (reads from memories_tr, creates gems in gems_tr)
|
||||
@@ -1,150 +0,0 @@
|
||||
# NeuralStream Session State
|
||||
|
||||
**Date:** 2026-02-23
|
||||
**Status:** Architecture v2.2 - Context-aware hybrid triggers
|
||||
**Alias:** ns
|
||||
|
||||
---
|
||||
|
||||
## Architecture v2.2 (Current)
|
||||
|
||||
**Decision:** Three hybrid extraction triggers with full context awareness
|
||||
|
||||
| Trigger | When | Purpose |
|
||||
|---------|------|---------|
|
||||
| `turn_end` (N=5) | Every 5 turns | Normal batch extraction |
|
||||
| Timer (15 min idle) | No new turn for 15 min | Catch partial batches |
|
||||
| Context (50% threshold) | `ctx.getContextUsage().percent >= threshold` | Proactive pre-compaction |
|
||||
|
||||
**Context Awareness:**
|
||||
- qwen3 gets **up to 256k tokens** of full conversation history for understanding
|
||||
- Only extracts **last N turns** (oldest in batch) to avoid gemming current context
|
||||
- Uses `ctx.getContextUsage()` native API for token monitoring
|
||||
|
||||
**Why Hybrid:**
|
||||
- Batch extraction = better quality gems (more context)
|
||||
- Timer safety = never lose important turns if user walks away
|
||||
- Context trigger = proactive extraction before system forces compaction
|
||||
- All gems survive `/new` and `/reset` via Qdrant
|
||||
|
||||
**Infrastructure:** Reuse existing Redis/Qdrant — NeuralStream is the "middle layer" only
|
||||
|
||||
---
|
||||
|
||||
## Core Insight
|
||||
|
||||
NeuralStream enables **infinite effective context** — active window stays small, but semantically relevant gems from all past conversations are queryable and injectable.
|
||||
|
||||
---
|
||||
|
||||
## Technical Decisions 2026-02-23
|
||||
|
||||
### Triggers (Three-way Hybrid)
|
||||
| Trigger | Config | Default |
|
||||
|---------|--------|---------|
|
||||
| Batch size | `batch_size` | 5 turns |
|
||||
| Idle timeout | `idle_timeout` | 15 minutes |
|
||||
| Context threshold | `context_threshold` | 50% |
|
||||
|
||||
### Context Monitoring (Native API)
|
||||
- `ctx.getContextUsage()` → `{tokens, contextWindow, percent}`
|
||||
- Checked in `turn_end` hook
|
||||
- Triggers extraction when `percent >= context_threshold`
|
||||
|
||||
### Extraction Context Window
|
||||
- **Feed to qwen3:** Up to 256k tokens (full history for understanding)
|
||||
- **Extract from:** Last `batch_size` turns only
|
||||
- **Benefit:** Rich context awareness without gemming current conversation
|
||||
|
||||
### Storage
|
||||
- **Buffer:** Redis (`neuralstream:buffer` key)
|
||||
- **Gems:** Qdrant `neuralstream` collection (1024 dims, Cosine)
|
||||
- **Existing infra:** Reuse mem-redis-watcher + qdrant-memory
|
||||
|
||||
### Gem Format (Proposed)
|
||||
```json
|
||||
{
|
||||
"gem_id": "uuid",
|
||||
"content": "Distilled insight/fact/decision",
|
||||
"summary": "One-line for quick scanning",
|
||||
"topics": ["docker", "redis", "architecture"],
|
||||
"importance": 0.9,
|
||||
"source": {
|
||||
"session_id": "uuid",
|
||||
"date": "2026-02-23",
|
||||
"turn_range": "15-20"
|
||||
},
|
||||
"tags": ["decision", "fact", "preference", "todo", "code"],
|
||||
"created_at": "2026-02-23T15:26:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Extraction Model
|
||||
- **qwen3** for gem extraction (256k context, cheap)
|
||||
- **Dedicated prompt** (to be designed) for extracting high-value items
|
||||
|
||||
---
|
||||
|
||||
## Architecture Layers
|
||||
|
||||
| Layer | Status | Description |
|
||||
|-------|--------|-------------|
|
||||
| Capture | ✅ Existing | Every turn → Redis (mem-redis-watcher) |
|
||||
| **Extract** | ⏳ NeuralStream | Batch → qwen3 → gems → Qdrant |
|
||||
| Retrieve | ✅ Existing | Semantic search → inject context |
|
||||
|
||||
NeuralStream = Smart extraction layer on top of existing infra.
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Gem extraction prompt design (deferred)
|
||||
- Importance scoring: auto vs manual?
|
||||
- Injection: `turn_start` hook or modify system prompt?
|
||||
- Semantic search threshold tuning
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
| Task | Status |
|
||||
|------|--------|
|
||||
| Architecture v2.2 finalized | ✅ |
|
||||
| Native context monitoring validated | ✅ |
|
||||
| Gem JSON schema | ✅ Proposed |
|
||||
| Implement turn_end hook | ⏳ |
|
||||
| Implement timer/cron check | ⏳ |
|
||||
| Implement context trigger | ⏳ |
|
||||
| Create extraction prompt | ⏳ |
|
||||
| Test gem extraction with qwen3 | ⏳ |
|
||||
| Implement injection mechanism | ⏳ |
|
||||
|
||||
---
|
||||
|
||||
## Decisions Log
|
||||
|
||||
| Date | Decision |
|
||||
|------|----------|
|
||||
| 2026-02-23 | Switch to turn_end hook (v2) |
|
||||
| 2026-02-23 | Hybrid triggers with timer (v2.1) |
|
||||
| 2026-02-23 | Context-aware extraction (v2.2) |
|
||||
| 2026-02-23 | Native API: ctx.getContextUsage() |
|
||||
| 2026-02-23 | Full context feed to qwen3 (256k) |
|
||||
| 2026-02-23 | Reuse existing Redis/Qdrant infrastructure |
|
||||
| 2026-02-23 | Batch N=5 turns |
|
||||
| 2026-02-23 | Context threshold = 50% |
|
||||
| 2026-02-23 | Inactivity timer = 15 min |
|
||||
| 2026-02-23 | Dedicated qwen3 extraction prompt (deferred) |
|
||||
|
||||
---
|
||||
|
||||
## Backups
|
||||
|
||||
- Local: `/root/.openclaw/workspace/.projects/neuralstream/`
|
||||
- Remote: `deb2:/root/.projects/neuralstream/` (build/test only)
|
||||
- kimi_kb: Research entries stored
|
||||
|
||||
---
|
||||
|
||||
**Key Insight:** Session resets wipe context but NOT Qdrant. NeuralStream = "Context insurance policy" for infinite LLM memory.
|
||||
Reference in New Issue
Block a user