Initial commit: TrueRecall v2.2 with 30b curator and timer-based curation
This commit is contained in:
root
513
README.md
Normal file
513
README.md
Normal file
@@ -0,0 +1,513 @@
|
||||
# TrueRecall v2
|
||||
|
||||
**Project:** Gem extraction and memory recall system
|
||||
**Status:** ✅ Active & Verified
|
||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/`
|
||||
**Last Updated:** 2026-02-24 19:02 CST
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Quick Start](#quick-start)
|
||||
- [Overview](#overview)
|
||||
- [Current State](#current-state)
|
||||
- [Architecture](#architecture)
|
||||
- [Components](#components)
|
||||
- [Files & Locations](#files--locations)
|
||||
- [Configuration](#configuration)
|
||||
- [Validation](#validation)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Status Summary](#status-summary)
|
||||
|
||||
---
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# Check system status
|
||||
openclaw status
|
||||
sudo systemctl status mem-qdrant-watcher
|
||||
|
||||
# View recent captures
|
||||
curl -s http://10.0.0.40:6333/collections/memories_tr | jq '.result.points_count'
|
||||
|
||||
# Check collections
|
||||
curl -s http://10.0.0.40:6333/collections | jq '.result.collections[].name'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
TrueRecall v2 extracts "gems" (key insights) from conversations and injects them as context. It consists of three layers:
|
||||
|
||||
1. **Capture** — Real-time watcher saves every turn to `memories_tr`
|
||||
2. **Curation** — Daily curator extracts gems to `gems_tr`
|
||||
3. **Injection** — Plugin searches `gems_tr` and injects gems per turn
|
||||
|
||||
---
|
||||
|
||||
## Current State
|
||||
|
||||
### Verified at 19:02 CST
|
||||
|
||||
| Collection | Points | Purpose | Status |
|
||||
|------------|--------|---------|--------|
|
||||
| `memories_tr` | **12,378** | Full text (live capture) | ✅ Active |
|
||||
| `gems_tr` | **5** | Curated gems (injection) | ✅ Active |
|
||||
|
||||
**All memories tagged with `curated: false` for timer curation.**
|
||||
|
||||
### Services Status
|
||||
|
||||
| Service | Status | Details |
|
||||
|---------|--------|---------|
|
||||
| `mem-qdrant-watcher` | ✅ Active | PID 1748, capturing |
|
||||
| Timer curator | ✅ Deployed | Every 30 min via cron |
|
||||
| OpenClaw Gateway | ✅ Running | Version 2026.2.23 |
|
||||
| memory-qdrant plugin | ✅ Loaded | recall: gems_tr |
|
||||
|
||||
---
|
||||
|
||||
## Comparison: TrueRecall v2 vs Jarvis Memory vs v1
|
||||
|
||||
| Feature | Jarvis Memory | TrueRecall v1 | TrueRecall v2 |
|
||||
|---------|---------------|---------------|---------------|
|
||||
| **Storage** | Redis | Redis + Qdrant | Qdrant only |
|
||||
| **Capture** | Session batch | Session batch | Real-time |
|
||||
| **Curation** | Manual | Daily 2:45 AM | Timer (5 min) |
|
||||
| **Embedding** | — | snowflake | snowflake + mxbai |
|
||||
| **Curator LLM** | — | qwen3:4b | qwen3:30b |
|
||||
| **State tracking** | — | — | `curated` tag |
|
||||
| **Batch size** | — | 24h worth | Configurable |
|
||||
| **JSON parsing** | — | Fallback needed | Native (30b) |
|
||||
|
||||
**Key Improvements v2:**
|
||||
- ✅ Real-time capture (no batch delay)
|
||||
- ✅ Timer-based curation (responsive vs daily)
|
||||
- ✅ 30b curator (better gems, faster ~3s)
|
||||
- ✅ `curated` tag (reliable state tracking)
|
||||
- ✅ No Redis dependency (simpler stack)
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### v2.2: Timer-Based Curation
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────────┐ ┌─────────────┐
|
||||
│ OpenClaw Chat │────▶│ Real-Time Watcher │────▶│ memories_tr │
|
||||
│ (Session JSONL)│ │ (Python daemon) │ │ (Qdrant) │
|
||||
└─────────────────┘ └──────────────────────┘ └──────┬──────┘
|
||||
│
|
||||
│ Every 30 min
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Timer Curator │
|
||||
│ (cron/qwen3) │
|
||||
└────────┬─────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ gems_tr │
|
||||
│ (Qdrant) │
|
||||
└────────┬─────────┘
|
||||
│
|
||||
Per turn │
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ memory-qdrant │
|
||||
│ plugin │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
**Key Changes in v2.2:**
|
||||
- ✅ Timer-based curation (30 min intervals)
|
||||
- ✅ All memories tagged `curated: false` on capture
|
||||
- ✅ Migration complete (12,378 memories)
|
||||
- ❌ Removed daily batch processing (2:45 AM)
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Real-Time Watcher
|
||||
|
||||
**File:** `skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
||||
|
||||
**What it does:**
|
||||
- Watches `/root/.openclaw/agents/main/sessions/*.jsonl`
|
||||
- Parses each turn (user + AI)
|
||||
- Embeds with `snowflake-arctic-embed2`
|
||||
- Stores to `memories_tr` instantly
|
||||
- **Cleans:** Removes markdown, tables, metadata
|
||||
|
||||
**Service:** `mem-qdrant-watcher.service`
|
||||
|
||||
**Commands:**
|
||||
```bash
|
||||
# Check status
|
||||
sudo systemctl status mem-qdrant-watcher
|
||||
|
||||
# View logs
|
||||
sudo journalctl -u mem-qdrant-watcher -f
|
||||
|
||||
# Restart
|
||||
sudo systemctl restart mem-qdrant-watcher
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 2. Content Cleaner
|
||||
|
||||
**File:** `skills/qdrant-memory/scripts/clean_memories_tr.py`
|
||||
|
||||
**Purpose:** Batch-clean existing points
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Preview changes
|
||||
python3 clean_memories_tr.py --dry-run
|
||||
|
||||
# Clean all
|
||||
python3 clean_memories_tr.py --execute
|
||||
|
||||
# Clean 100 (test)
|
||||
python3 clean_memories_tr.py --execute --limit 100
|
||||
```
|
||||
|
||||
**Cleans:**
|
||||
- `**bold**` → plain text
|
||||
- `|tables|` → removed
|
||||
- `` `code` `` → plain text
|
||||
- `---` rules → removed
|
||||
- `# headers` → removed
|
||||
|
||||
---
|
||||
|
||||
### 3. Timer Curator
|
||||
|
||||
**File:** `tr-continuous/curator_timer.py`
|
||||
|
||||
**Schedule:** Every 30 minutes (cron)
|
||||
|
||||
**Flow:**
|
||||
1. Query uncurated memories from `memories_tr`
|
||||
2. Send batch to qwen3 (max 100)
|
||||
3. Extract gems → store to `gems_tr`
|
||||
4. Mark memories as `curated: true`
|
||||
|
||||
**Config:** `tr-continuous/curator_config.json`
|
||||
```json
|
||||
{
|
||||
"timer_minutes": 30,
|
||||
"max_batch_size": 100
|
||||
}
|
||||
```
|
||||
|
||||
**Logs:** `/var/log/true-recall-timer.log`
|
||||
|
||||
---
|
||||
|
||||
### 4. Curation Model Comparison
|
||||
|
||||
**Current:** `qwen3:4b-instruct`
|
||||
|
||||
| Metric | 4b | 30b |
|
||||
|--------|----|----|
|
||||
| Speed | ~10-30s per batch | **~3.3s** (tested 2026-02-24) |
|
||||
| JSON reliability | ⚠️ Needs fallback | ✅ Native |
|
||||
| Context quality | Basic extraction | ✅ Nuanced |
|
||||
| Snippet accuracy | ~80% | ✅ Expected: 95%+ |
|
||||
|
||||
**30b Benchmark (2026-02-24):**
|
||||
- Load: 108ms
|
||||
- Prompt eval: 49ms (1,576 tok/s)
|
||||
- Generation: 2.9s (233 tokens, 80 tok/s)
|
||||
- **Total: 3.26s**
|
||||
|
||||
**Trade-offs:**
|
||||
- **4b:** Faster batch processing, lightweight, catches explicit decisions
|
||||
- **30b:** Deeper context, better inference, ~3x slower but superior quality
|
||||
|
||||
**Gem Quality Comparison (Sample Review):**
|
||||
|
||||
| Aspect | 4b | 30b |
|
||||
|--------|----|----|
|
||||
| **Context depth** | "Extracted via fallback" | Explains *why* decisions were made |
|
||||
| **Confidence scores** | 0.7-0.85 | 0.9-0.97 |
|
||||
| **Snippet accuracy** | ~80% (wrong source) | ✅ 95%+ (relevant quotes) |
|
||||
| **Categories** | Generic "extracted" | Specific: knowledge, technical, decision |
|
||||
| **Example** | "User implemented BorgBackup" (no context) | "User selected mxbai... due to top MTEB score of 66.5" (explains reasoning) |
|
||||
|
||||
**Verdict:** 30b produces significantly higher quality gems — richer context, accurate snippets, and captures architectural intent, not just surface facts.
|
||||
|
||||
---
|
||||
|
||||
### 5. OpenClaw Compactor Configuration
|
||||
|
||||
**Status:** ✅ Applied
|
||||
|
||||
**Goal:** Minimal overhead — just remove context, do nothing else.
|
||||
|
||||
**Config Applied:**
|
||||
```json5
|
||||
{
|
||||
agents: {
|
||||
defaults: {
|
||||
compaction: {
|
||||
mode: "default", // "default" or "safeguard"
|
||||
reserveTokensFloor: 0, // Disable safety floor (default: 20000)
|
||||
memoryFlush: {
|
||||
enabled: false // Disable silent .md file writes
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**What this does:**
|
||||
- `mode: "default"` — Standard summarization (faster)
|
||||
- `reserveTokensFloor: 0` — Allow aggressive settings (disables 20k minimum)
|
||||
- `memoryFlush.enabled: false` — No silent "write memory" turns
|
||||
|
||||
**Note:** `reserveTokens` and `keepRecentTokens` are Pi runtime settings, not configurable via `agents.defaults.compaction`. They are set per-model in `contextWindow`/`contextTokens`.
|
||||
|
||||
---
|
||||
|
||||
### 6. Embedding Models
|
||||
|
||||
**Current Setup:**
|
||||
- `memories_tr`: `snowflake-arctic-embed2` (capture similarity)
|
||||
- `gems_tr`: `mxbai-embed-large` (recall similarity)
|
||||
|
||||
**Rationale:**
|
||||
- mxbai has higher MTEB score (66.5) for semantic search
|
||||
- snowflake is faster for high-volume capture
|
||||
|
||||
**Note:** For simplicity, a single embedding model could be used for both collections. This would reduce complexity and memory overhead, though with slightly lower recall performance.
|
||||
|
||||
---
|
||||
|
||||
### 6. memory-qdrant Plugin
|
||||
|
||||
**Location:** `/root/.openclaw/extensions/memory-qdrant/`
|
||||
|
||||
**Config (openclaw.json):**
|
||||
```json
|
||||
{
|
||||
"collectionName": "gems_tr",
|
||||
"captureCollection": "memories_tr",
|
||||
"autoRecall": true,
|
||||
"autoCapture": true
|
||||
}
|
||||
```
|
||||
|
||||
**Functions:**
|
||||
- **Recall:** Searches `gems_tr`, injects gems (hidden)
|
||||
- **Capture:** Session-level to `memories_tr` (backup)
|
||||
|
||||
---
|
||||
|
||||
## Files & Locations
|
||||
|
||||
### Core Project
|
||||
|
||||
```
|
||||
/root/.openclaw/workspace/.projects/true-recall-v2/
|
||||
├── README.md # This file
|
||||
├── session.md # Detailed notes
|
||||
├── curator-prompt.md # Extraction prompt
|
||||
├── tr-daily/
|
||||
│ └── curate_from_qdrant.py # Daily curator
|
||||
└── shared/
|
||||
```
|
||||
|
||||
### New Files (2026-02-24)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `tr-continuous/curator_timer.py` | Timer curator (v2.2) |
|
||||
| `tr-continuous/curator_config.json` | Curator settings |
|
||||
| `tr-continuous/migrate_add_curated.py` | Migration script |
|
||||
| `skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` | Capture daemon |
|
||||
| `skills/qdrant-memory/mem-qdrant-watcher.service` | Systemd service |
|
||||
|
||||
### Archived Files (v2.1)
|
||||
|
||||
| File | Status | Note |
|
||||
|------|--------|------|
|
||||
| `tr-daily/curate_from_qdrant.py` | 📦 Archived | Replaced by timer |
|
||||
| `tr-continuous/curator_by_count.py` | 📦 Archived | Replaced by timer |
|
||||
|
||||
### System Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `/root/.openclaw/extensions/memory-qdrant/` | Plugin code |
|
||||
| `/root/.openclaw/openclaw.json` | Configuration |
|
||||
| `/etc/systemd/system/mem-qdrant-watcher.service` | Service file |
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### memory-qdrant Plugin
|
||||
|
||||
**File:** `/root/.openclaw/openclaw.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"memory-qdrant": {
|
||||
"config": {
|
||||
"autoCapture": true,
|
||||
"autoRecall": true,
|
||||
"collectionName": "gems_tr",
|
||||
"captureCollection": "memories_tr",
|
||||
"embeddingModel": "snowflake-arctic-embed2",
|
||||
"maxRecallResults": 2,
|
||||
"minRecallScore": 0.7,
|
||||
"ollamaUrl": "http://10.0.0.10:11434",
|
||||
"qdrantUrl": "http://10.0.0.40:6333"
|
||||
},
|
||||
"enabled": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Gateway Control UI (OpenClaw 2026.2.23)
|
||||
|
||||
```json
|
||||
{
|
||||
"gateway": {
|
||||
"controlUi": {
|
||||
"allowedOrigins": ["*"],
|
||||
"allowInsecureAuth": false,
|
||||
"dangerouslyDisableDeviceAuth": true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation
|
||||
|
||||
### Check Collections
|
||||
|
||||
```bash
|
||||
# Count points
|
||||
curl -s http://10.0.0.40:6333/collections/memories_tr | jq '.result.points_count'
|
||||
curl -s http://10.0.0.40:6333/collections/gems_tr | jq '.result.points_count'
|
||||
|
||||
# View recent captures
|
||||
curl -s -X POST http://10.0.0.40:6333/collections/memories_tr/points/scroll \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"limit": 3, "with_payload": true}' | jq '.result.points[].payload.content'
|
||||
```
|
||||
|
||||
### Check Services
|
||||
|
||||
```bash
|
||||
# Watcher
|
||||
sudo systemctl status mem-qdrant-watcher
|
||||
sudo journalctl -u mem-qdrant-watcher -n 20
|
||||
|
||||
# OpenClaw
|
||||
openclaw status
|
||||
openclaw gateway status
|
||||
```
|
||||
|
||||
### Test Capture
|
||||
|
||||
Send a message, then check:
|
||||
```bash
|
||||
# Should increase by 1-2 points
|
||||
curl -s http://10.0.0.40:6333/collections/memories_tr | jq '.result.points_count'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Watcher Not Capturing
|
||||
|
||||
```bash
|
||||
# Check logs
|
||||
sudo journalctl -u mem-qdrant-watcher -f
|
||||
|
||||
# Verify dependencies
|
||||
curl http://10.0.0.40:6333/ # Qdrant
|
||||
curl http://10.0.0.10:11434/api/tags # Ollama
|
||||
```
|
||||
|
||||
### Plugin Not Loading
|
||||
|
||||
```bash
|
||||
# Validate config
|
||||
openclaw config validate
|
||||
|
||||
# Check logs
|
||||
tail /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant
|
||||
|
||||
# Restart gateway
|
||||
openclaw gateway restart
|
||||
```
|
||||
|
||||
### Gateway Won't Start (OpenClaw 2026.2.23+)
|
||||
|
||||
**Error:** `non-loopback Control UI requires gateway.controlUi.allowedOrigins`
|
||||
|
||||
**Fix:** Add to `openclaw.json`:
|
||||
```json
|
||||
"gateway": {
|
||||
"controlUi": {
|
||||
"allowedOrigins": ["*"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Status Summary
|
||||
|
||||
| Component | Status | Notes |
|
||||
|-----------|--------|-------|
|
||||
| Real-time watcher | ✅ Active | PID 1748, capturing |
|
||||
| memories_tr | ✅ 12,378 pts | All tagged `curated: false` |
|
||||
| gems_tr | ✅ 5 pts | Injection ready |
|
||||
| Timer curator | ✅ Deployed | Every 30 min via cron |
|
||||
| Plugin injection | ✅ Working | Uses gems_tr |
|
||||
| Migration | ✅ Complete | 12,378 memories |
|
||||
|
||||
**Logs:** `tail /var/log/true-recall-timer.log`
|
||||
|
||||
**Next:** Monitor first timer run
|
||||
|
||||
---
|
||||
|
||||
## Roadmap
|
||||
|
||||
### Planned Features
|
||||
|
||||
| Feature | Status | Description |
|
||||
|---------|--------|-------------|
|
||||
| Interactive install script | ⏳ Planned | Prompts for embedding model, timer interval, batch size, endpoints |
|
||||
| Single embedding model | ⏳ Planned | Option to use one model for both collections |
|
||||
| Configurable thresholds | ⏳ Planned | Per-user customization via prompts |
|
||||
|
||||
**Install script will prompt for:**
|
||||
1. **Embedding model** — snowflake (fast) vs mxbai (accurate)
|
||||
2. **Timer interval** — 5 min / 30 min / hourly
|
||||
3. **Batch size** — 50 / 100 / 500 memories
|
||||
4. **Endpoints** — Qdrant/Ollama URLs
|
||||
5. **User ID** — for multi-user setups
|
||||
|
||||
---
|
||||
|
||||
**Maintained by:** Rob
|
||||
**AI Assistant:** Kimi 🎙️
|
||||
**Version:** 2026.02.24-v2.2
|
||||
308
README.md.bak.2026-02-24-1518
Normal file
308
README.md.bak.2026-02-24-1518
Normal file
@@ -0,0 +1,308 @@
|
||||
# TrueRecall v2
|
||||
|
||||
**Project:** Gem extraction and memory recall system
|
||||
**Status:** ✅ Active
|
||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/`
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
TrueRecall extracts "gems" (key insights) from conversations and stores them for context injection. It's the memory curation system that powers Kimi's contextual awareness.
|
||||
|
||||
### Current Flow
|
||||
|
||||
```
|
||||
1. Conversation happens → Real-time watcher → memories_tr (Qdrant)
|
||||
2. Daily (2:45 AM) → Curator reads memories_tr → extracts gems
|
||||
3. Gems stored → gems_tr collection (Qdrant)
|
||||
4. On each turn → memory-qdrant plugin injects gems as context
|
||||
```
|
||||
|
||||
**Verified:** 2026-02-24 — Real-time watcher capturing (12,223 → 12,228 points)
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Collections (Qdrant)
|
||||
|
||||
| Collection | Purpose | Content | Status |
|
||||
|------------|---------|---------|--------|
|
||||
| `memories_tr` | Full text storage | Every conversation turn (migrated from `kimi_memories`) | ✅ Active |
|
||||
| `gems_tr` | Gems (extracted insights) | Curated key points (for injection) | ✅ Active |
|
||||
| `true_recall` | Legacy gems | Archive of previously extracted gems | 📦 Preserved |
|
||||
| `kimi_memories` | Original collection | Backup (12,223 points preserved) | 📦 Preserved |
|
||||
|
||||
### Migration Script
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `migrate_memories.py` | Migrate data from `kimi_memories` → `memories_tr` with cleaning |
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python3 migrate_memories.py
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- Reads all points from `kimi_memories`
|
||||
- Cleans content (removes metadata, thinking tags)
|
||||
- Stores to `memories_tr`
|
||||
- Preserves original `kimi_memories`
|
||||
|
||||
---
|
||||
|
||||
### Components
|
||||
|
||||
| Component | Location | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| **memory-qdrant plugin** | `/root/.openclaw/extensions/memory-qdrant/` | Injects gems as context |
|
||||
| **Curation script** | `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` | Extracts gems |
|
||||
| **Curator prompt** | `/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md` | Instructions for gem extraction |
|
||||
|
||||
---
|
||||
|
||||
## File Locations
|
||||
|
||||
### Core Files
|
||||
|
||||
```
|
||||
/root/.openclaw/workspace/.projects/true-recall-v2/
|
||||
├── README.md # This file
|
||||
├── session.md # Development notes
|
||||
├── curator-prompt.md # Gem extraction prompt
|
||||
│
|
||||
├── tr-daily/ # Daily curation
|
||||
│ └── curate_from_qdrant.py # Main curator script
|
||||
│
|
||||
├── tr-compact/ # (Reserved for v2 expansion)
|
||||
│ └── hook.py # Compaction hook (not active)
|
||||
│
|
||||
├── tr-worker/ # (Reserved for v2 expansion)
|
||||
│ └── worker.py # Background worker (not active)
|
||||
│
|
||||
└── shared/ # Shared resources
|
||||
```
|
||||
|
||||
### Plugin Files (Located in OpenClaw Extensions)
|
||||
|
||||
> **Note:** These files live in `/root/.openclaw/extensions/`, not in this project folder. Documented here for reference.
|
||||
|
||||
| Actual Location | Project Reference |
|
||||
|----------------|-------------------|
|
||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code (capture + injection) |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
||||
| `/root/.openclaw/openclaw.json` | Plugin configuration |
|
||||
|
||||
### Configuration
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `/root/.openclaw/openclaw.json` | Plugin config (collectionName: gems_tr) |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
||||
|
||||
### Cron Jobs
|
||||
|
||||
| Time | Command | Purpose |
|
||||
|------|---------|---------|
|
||||
| 2:45 AM | `curate_memories.py` | Daily gem extraction (v1) → stores to gems_tr |
|
||||
| 2:50 AM | `archive_to_memories_tr.py` | Archive to memories_tr |
|
||||
|
||||
View cron: `crontab -l`
|
||||
|
||||
---
|
||||
|
||||
## Process Flow
|
||||
|
||||
### Step 1: Capture (Real-Time Watcher)
|
||||
|
||||
```
|
||||
OpenClaw Session ──→ Real-Time Watcher ──→ Qdrant (memories_tr)
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
||||
|
||||
**What it does:**
|
||||
- Watches session JSONL files in real-time
|
||||
- Parses each conversation turn (user + AI)
|
||||
- Embeds with `snowflake-arctic-embed2` via Ollama
|
||||
- Stores directly to `memories_tr` collection
|
||||
- Cleans content (removes metadata, thinking tags)
|
||||
|
||||
**Systemd Service:** `mem-qdrant-watcher.service`
|
||||
|
||||
**Status:** ✅ Created, ready to deploy
|
||||
|
||||
### Step 2: Curation (Daily)
|
||||
|
||||
```
|
||||
2:45 AM → curate_memories.py → memories_tr → qwen3 → gems_tr
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v1/tr-process/curate_memories.py`
|
||||
|
||||
**What it does:**
|
||||
1. Reads from Redis buffer (`mem:rob`)
|
||||
2. Passes to qwen3 (curator model)
|
||||
3. Extracts gems using prompt
|
||||
4. Stores gems to `gems_tr` collection
|
||||
|
||||
### Step 3: Injection
|
||||
|
||||
```
|
||||
User message → memory-qdrant plugin → Search gems_tr → Inject as context
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/extensions/memory-qdrant/index.ts`
|
||||
|
||||
**What it does:**
|
||||
1. Listens to `before_agent_start` events
|
||||
2. Embeds current prompt
|
||||
3. Searches `gems_tr` collection
|
||||
4. Returns `{ prependContext: "..." }` with gems
|
||||
5. Gems appear in my context (hidden from UI)
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### memory-qdrant Plugin
|
||||
|
||||
```json
|
||||
{
|
||||
"autoCapture": true,
|
||||
"autoRecall": true,
|
||||
"collectionName": "gems_tr",
|
||||
"embeddingModel": "snowflake-arctic-embed2",
|
||||
"maxRecallResults": 2,
|
||||
"minRecallScore": 0.7
|
||||
}
|
||||
```
|
||||
|
||||
**Key setting:** `collectionName: "gems_tr"` — tells plugin to inject gems, not full text.
|
||||
|
||||
---
|
||||
|
||||
## Gem Format
|
||||
|
||||
```json
|
||||
{
|
||||
"gem": "User prefers dark mode for interface",
|
||||
"context": "Discussed UI theme options",
|
||||
"snippet": "rob: I want dark mode\nKimi: Done",
|
||||
"categories": ["preference"],
|
||||
"importance": "high",
|
||||
"confidence": 0.95,
|
||||
"date": "2026-02-24",
|
||||
"conversation_id": "uuid",
|
||||
"turn_range": "5-7"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Commands
|
||||
|
||||
### Check Qdrant Collections
|
||||
|
||||
```bash
|
||||
curl -s "http://10.0.0.40:6333/collections" | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Check Collection Points
|
||||
|
||||
```bash
|
||||
# memories_tr (full text)
|
||||
curl -s -X POST "http://10.0.0.40:6333/collections/memories_tr/points/scroll" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
||||
|
||||
# gems_tr (gems)
|
||||
curl -s -X POST "http://10.0.0.40:6333/collections/gems_tr/points/scroll" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Check Cron Jobs
|
||||
|
||||
```bash
|
||||
crontab -l | grep -E "true|recall|curate"
|
||||
```
|
||||
|
||||
### Check Plugin Logs
|
||||
|
||||
```bash
|
||||
tail -50 /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: `<relevant-memories>` showing in UI
|
||||
|
||||
**Cause:** Plugin was using `memories_tr` (full text) instead of `gems_tr` (gems)
|
||||
|
||||
**Fix:** Set `"collectionName": "gems_tr"` in openclaw.json
|
||||
|
||||
**Reference:** KB entry `relevant-memories-ui-issue.md`
|
||||
|
||||
### Issue: No gems being extracted
|
||||
|
||||
**Check:**
|
||||
1. Is memories_tr populated? `curl .../collections/memories_tr/points/scroll`
|
||||
2. Is curator prompt valid? `cat curator-prompt.md`
|
||||
3. Is qwen3 available? `curl -s http://10.0.0.10:11434/api/tags`
|
||||
|
||||
---
|
||||
|
||||
## Related Projects
|
||||
|
||||
| Project | Location | Purpose |
|
||||
|---------|----------|---------|
|
||||
| **true-recall-v1** | `/.projects/true-recall-v1/` | Original (Redis-based) |
|
||||
| **memory-qdrant** | `/root/.openclaw/extensions/memory-qdrant/` | Plugin |
|
||||
| **mem-redis** | `/root/.openclaw/workspace/skills/mem-redis/` | Redis utilities |
|
||||
|
||||
---
|
||||
|
||||
## Backups
|
||||
|
||||
| File | Backup Location |
|
||||
|------|-----------------|
|
||||
| openclaw.json | `/root/.openclaw/openclaw.json.bak.2026-02-24` |
|
||||
| Plugin | `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24` |
|
||||
|
||||
---
|
||||
|
||||
## Final Status (2026-02-24 15:08)
|
||||
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| Collection `kimi_memories` → `memories_tr` | ✅ Migrated (12,223 points) |
|
||||
| Real-time watcher | ✅ **Deployed & verified** (12,228 points, +5 new) |
|
||||
| Collection `gems_tr` | ✅ Active (5 gems stored) |
|
||||
| Curator v2 | ✅ Working - tested with 327 turns |
|
||||
| Config | ✅ Updated to `gems_tr` |
|
||||
| Cron jobs | ✅ Cleaned |
|
||||
| Documentation | ✅ Updated |
|
||||
|
||||
### Daily Schedule (Simplified)
|
||||
|
||||
| Time | Job | Flow |
|
||||
|------|-----|------|
|
||||
| Continuous | autoCapture | Conversation → `memories_tr` |
|
||||
| 2:45 AM | Curator v2 | `memories_tr` → `gems_tr` |
|
||||
| Each turn | Injection | `gems_tr` → Context |
|
||||
|
||||
**Redis usage:** Disabled for memory. Used only for `delayed:notifications` queue.
|
||||
|
||||
**Auto-Capture Status:** ✅ **Working** - Real-time watcher deployed and verified
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-02-24
|
||||
**Project Lead:** Rob
|
||||
**AI Assistant:** Kimi 🎙️
|
||||
325
README.md.bak.2026-02-24-1518-final
Normal file
325
README.md.bak.2026-02-24-1518-final
Normal file
@@ -0,0 +1,325 @@
|
||||
# TrueRecall v2
|
||||
|
||||
**Project:** Gem extraction and memory recall system
|
||||
**Status:** ✅ Active
|
||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/`
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
- [Overview](#overview)
|
||||
- [Architecture](#architecture)
|
||||
- [Collections](#collections-qdrant)
|
||||
- [Components](#components)
|
||||
- [Process Flow](#process-flow)
|
||||
- [Configuration](#configuration)
|
||||
- [Validation Commands](#validation-commands)
|
||||
- [Troubleshooting](#troubleshooting)
|
||||
- [Related Projects](#related-projects)
|
||||
- [Backups](#backups)
|
||||
- [Final Status](#final-status-2026-02-24-1508)
|
||||
|
||||
---
|
||||
|
||||
## Overview
|
||||
|
||||
TrueRecall extracts "gems" (key insights) from conversations and stores them for context injection. It's the memory curation system that powers Kimi's contextual awareness.
|
||||
|
||||
### Current Flow
|
||||
|
||||
```
|
||||
1. Conversation happens → Real-time watcher → memories_tr (Qdrant)
|
||||
2. Daily (2:45 AM) → Curator reads memories_tr → extracts gems
|
||||
3. Gems stored → gems_tr collection (Qdrant)
|
||||
4. On each turn → memory-qdrant plugin injects gems as context
|
||||
```
|
||||
|
||||
**Verified:** 2026-02-24 — Real-time watcher capturing (12,223 → 12,228 points)
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### Collections (Qdrant)
|
||||
|
||||
| Collection | Purpose | Content | Status |
|
||||
|------------|---------|---------|--------|
|
||||
| `memories_tr` | Full text storage | Every conversation turn (migrated from `kimi_memories`) | ✅ Active |
|
||||
| `gems_tr` | Gems (extracted insights) | Curated key points (for injection) | ✅ Active |
|
||||
| `true_recall` | Legacy gems | Archive of previously extracted gems | 📦 Preserved |
|
||||
| `kimi_memories` | Original collection | Backup (12,223 points preserved) | 📦 Preserved |
|
||||
|
||||
### Migration Script
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `migrate_memories.py` | Migrate data from `kimi_memories` → `memories_tr` with cleaning |
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
python3 migrate_memories.py
|
||||
```
|
||||
|
||||
**What it does:**
|
||||
- Reads all points from `kimi_memories`
|
||||
- Cleans content (removes metadata, thinking tags)
|
||||
- Stores to `memories_tr`
|
||||
- Preserves original `kimi_memories`
|
||||
|
||||
---
|
||||
|
||||
### Components
|
||||
|
||||
| Component | Location | Purpose |
|
||||
|-----------|----------|---------|
|
||||
| **memory-qdrant plugin** | `/root/.openclaw/extensions/memory-qdrant/` | Injects gems as context |
|
||||
| **Curation script** | `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` | Extracts gems |
|
||||
| **Curator prompt** | `/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md` | Instructions for gem extraction |
|
||||
|
||||
---
|
||||
|
||||
## File Locations
|
||||
|
||||
### Core Files
|
||||
|
||||
```
|
||||
/root/.openclaw/workspace/.projects/true-recall-v2/
|
||||
├── README.md # This file
|
||||
├── session.md # Development notes
|
||||
├── curator-prompt.md # Gem extraction prompt
|
||||
│
|
||||
├── tr-daily/ # Daily curation
|
||||
│ └── curate_from_qdrant.py # Main curator script
|
||||
│
|
||||
├── tr-compact/ # (Reserved for v2 expansion)
|
||||
│ └── hook.py # Compaction hook (not active)
|
||||
│
|
||||
├── tr-worker/ # (Reserved for v2 expansion)
|
||||
│ └── worker.py # Background worker (not active)
|
||||
│
|
||||
└── shared/ # Shared resources
|
||||
```
|
||||
|
||||
### Plugin Files (Located in OpenClaw Extensions)
|
||||
|
||||
> **Note:** These files live in `/root/.openclaw/extensions/`, not in this project folder. Documented here for reference.
|
||||
|
||||
| Actual Location | Project Reference |
|
||||
|----------------|-------------------|
|
||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code (capture + injection) |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
||||
| `/root/.openclaw/openclaw.json` | Plugin configuration |
|
||||
|
||||
### Configuration
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `/root/.openclaw/openclaw.json` | Plugin config (collectionName: gems_tr) |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Plugin code |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Plugin config schema |
|
||||
|
||||
### Cron Jobs
|
||||
|
||||
| Time | Command | Purpose |
|
||||
|------|---------|---------|
|
||||
| 2:45 AM | `curate_memories.py` | Daily gem extraction (v1) → stores to gems_tr |
|
||||
| 2:50 AM | `archive_to_memories_tr.py` | Archive to memories_tr |
|
||||
|
||||
View cron: `crontab -l`
|
||||
|
||||
---
|
||||
|
||||
## Process Flow
|
||||
|
||||
### Step 1: Capture (Real-Time Watcher)
|
||||
|
||||
```
|
||||
OpenClaw Session ──→ Real-Time Watcher ──→ Qdrant (memories_tr)
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
||||
|
||||
**What it does:**
|
||||
- Watches session JSONL files in real-time
|
||||
- Parses each conversation turn (user + AI)
|
||||
- Embeds with `snowflake-arctic-embed2` via Ollama
|
||||
- Stores directly to `memories_tr` collection
|
||||
- Cleans content (removes metadata, thinking tags)
|
||||
|
||||
**Systemd Service:** `mem-qdrant-watcher.service`
|
||||
|
||||
**Status:** ✅ Created, ready to deploy
|
||||
|
||||
### Step 2: Curation (Daily)
|
||||
|
||||
```
|
||||
2:45 AM → curate_memories.py → memories_tr → qwen3 → gems_tr
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v1/tr-process/curate_memories.py`
|
||||
|
||||
**What it does:**
|
||||
1. Reads from Redis buffer (`mem:rob`)
|
||||
2. Passes to qwen3 (curator model)
|
||||
3. Extracts gems using prompt
|
||||
4. Stores gems to `gems_tr` collection
|
||||
|
||||
### Step 3: Injection
|
||||
|
||||
```
|
||||
User message → memory-qdrant plugin → Search gems_tr → Inject as context
|
||||
```
|
||||
|
||||
**Location:** `/root/.openclaw/extensions/memory-qdrant/index.ts`
|
||||
|
||||
**What it does:**
|
||||
1. Listens to `before_agent_start` events
|
||||
2. Embeds current prompt
|
||||
3. Searches `gems_tr` collection
|
||||
4. Returns `{ prependContext: "..." }` with gems
|
||||
5. Gems appear in my context (hidden from UI)
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### memory-qdrant Plugin
|
||||
|
||||
```json
|
||||
{
|
||||
"autoCapture": true,
|
||||
"autoRecall": true,
|
||||
"collectionName": "gems_tr",
|
||||
"embeddingModel": "snowflake-arctic-embed2",
|
||||
"maxRecallResults": 2,
|
||||
"minRecallScore": 0.7
|
||||
}
|
||||
```
|
||||
|
||||
**Key setting:** `collectionName: "gems_tr"` — tells plugin to inject gems, not full text.
|
||||
|
||||
---
|
||||
|
||||
## Gem Format
|
||||
|
||||
```json
|
||||
{
|
||||
"gem": "User prefers dark mode for interface",
|
||||
"context": "Discussed UI theme options",
|
||||
"snippet": "rob: I want dark mode\nKimi: Done",
|
||||
"categories": ["preference"],
|
||||
"importance": "high",
|
||||
"confidence": 0.95,
|
||||
"date": "2026-02-24",
|
||||
"conversation_id": "uuid",
|
||||
"turn_range": "5-7"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Commands
|
||||
|
||||
### Check Qdrant Collections
|
||||
|
||||
```bash
|
||||
curl -s "http://10.0.0.40:6333/collections" | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Check Collection Points
|
||||
|
||||
```bash
|
||||
# memories_tr (full text)
|
||||
curl -s -X POST "http://10.0.0.40:6333/collections/memories_tr/points/scroll" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
||||
|
||||
# gems_tr (gems)
|
||||
curl -s -X POST "http://10.0.0.40:6333/collections/gems_tr/points/scroll" \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"limit": 5, "with_payload": true}' | python3 -m json.tool
|
||||
```
|
||||
|
||||
### Check Cron Jobs
|
||||
|
||||
```bash
|
||||
crontab -l | grep -E "true|recall|curate"
|
||||
```
|
||||
|
||||
### Check Plugin Logs
|
||||
|
||||
```bash
|
||||
tail -50 /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log | grep memory-qdrant
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: `<relevant-memories>` showing in UI
|
||||
|
||||
**Cause:** Plugin was using `memories_tr` (full text) instead of `gems_tr` (gems)
|
||||
|
||||
**Fix:** Set `"collectionName": "gems_tr"` in openclaw.json
|
||||
|
||||
**Reference:** KB entry `relevant-memories-ui-issue.md`
|
||||
|
||||
### Issue: No gems being extracted
|
||||
|
||||
**Check:**
|
||||
1. Is memories_tr populated? `curl .../collections/memories_tr/points/scroll`
|
||||
2. Is curator prompt valid? `cat curator-prompt.md`
|
||||
3. Is qwen3 available? `curl -s http://10.0.0.10:11434/api/tags`
|
||||
|
||||
---
|
||||
|
||||
## Related Projects
|
||||
|
||||
| Project | Location | Purpose |
|
||||
|---------|----------|---------|
|
||||
| **true-recall-v1** | `/.projects/true-recall-v1/` | Original (Redis-based) |
|
||||
| **memory-qdrant** | `/root/.openclaw/extensions/memory-qdrant/` | Plugin |
|
||||
| **mem-redis** | `/root/.openclaw/workspace/skills/mem-redis/` | Redis utilities |
|
||||
|
||||
---
|
||||
|
||||
## Backups
|
||||
|
||||
| File | Backup Location |
|
||||
|------|-----------------|
|
||||
| openclaw.json | `/root/.openclaw/openclaw.json.bak.2026-02-24` |
|
||||
| Plugin | `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24` |
|
||||
|
||||
---
|
||||
|
||||
## Final Status (2026-02-24 15:08)
|
||||
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| Collection `kimi_memories` → `memories_tr` | ✅ Migrated (12,223 points) |
|
||||
| Real-time watcher | ✅ **Deployed & verified** (12,228 points, +5 new) |
|
||||
| Collection `gems_tr` | ✅ Active (5 gems stored) |
|
||||
| Curator v2 | ✅ Working - tested with 327 turns |
|
||||
| Config | ✅ Updated to `gems_tr` |
|
||||
| Cron jobs | ✅ Cleaned |
|
||||
| Documentation | ✅ Updated |
|
||||
|
||||
### Daily Schedule (Simplified)
|
||||
|
||||
| Time | Job | Flow |
|
||||
|------|-----|------|
|
||||
| Continuous | autoCapture | Conversation → `memories_tr` |
|
||||
| 2:45 AM | Curator v2 | `memories_tr` → `gems_tr` |
|
||||
| Each turn | Injection | `gems_tr` → Context |
|
||||
|
||||
**Redis usage:** Disabled for memory. Used only for `delayed:notifications` queue.
|
||||
|
||||
**Auto-Capture Status:** ✅ **Working** - Real-time watcher deployed and verified
|
||||
|
||||
---
|
||||
|
||||
**Last Updated:** 2026-02-24
|
||||
**Project Lead:** Rob
|
||||
**AI Assistant:** Kimi 🎙️
|
||||
|
||||
183
README.md.neuralstream.bak
Normal file
183
README.md.neuralstream.bak
Normal file
@@ -0,0 +1,183 @@
|
||||
# NeuralStream
|
||||
|
||||
**Neural streaming memory for OpenClaw with gem-based context injection.**
|
||||
|
||||
## Overview
|
||||
|
||||
NeuralStream extracts high-value insights ("gems") from conversation batches using qwen3, stores them in Qdrant, and injects relevant gems into context on each new turn. This creates **infinite effective context** — the active window stays small, but semantically relevant gems from all past conversations are always retrievable.
|
||||
|
||||
## Core Concept
|
||||
|
||||
| Traditional Memory | NeuralStream |
|
||||
|-------------------|--------------|
|
||||
| Context lost on `/new` | Gems persist in Qdrant |
|
||||
| Full history or generic summary | Semantic gem retrieval |
|
||||
| Static context window | Dynamic injection |
|
||||
| Survives compaction only | Survives session reset |
|
||||
| **Limited context** | **Infinite effective context** |
|
||||
|
||||
## How It Works
|
||||
|
||||
### Capture → Extract → Store → Retrieve
|
||||
|
||||
1. **Capture:** Every turn buffered to Redis (reuses mem-redis-watcher)
|
||||
2. **Extract:** Batch of 5 turns → qwen3 (with 256k context) extracts structured gems
|
||||
3. **Store:** Gems embedded + stored in Qdrant `neuralstream`
|
||||
4. **Retrieve:** Each new turn → semantic search → inject top-10 gems
|
||||
|
||||
### Hybrid Triggers (Three-way)
|
||||
|
||||
| Trigger | Condition | Purpose |
|
||||
|---------|-----------|---------|
|
||||
| Batch | Every 5 turns | Normal extraction |
|
||||
| Context | 50% usage (`ctx.getContextUsage()`) | Proactive pre-compaction |
|
||||
| Timer | 15 min idle | Safety net |
|
||||
|
||||
**Context Awareness:** qwen3 receives up to 256k tokens of history for understanding, but only extracts gems from the last N turns (avoiding current context).
|
||||
|
||||
All gems survive `/new`, `/reset`, and compaction via Qdrant persistence.
|
||||
|
||||
## Architecture
|
||||
|
||||
NeuralStream is the **middle layer** — extraction intelligence on top of existing infrastructure:
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────────────────────────┐
|
||||
│ EXISTING: mem-redis-watcher │
|
||||
│ Every turn → Redis buffer │
|
||||
└──────────────────┬──────────────────────────────────────┘
|
||||
│
|
||||
┌──────────▼──────────┐
|
||||
│ NeuralStream │
|
||||
│ - Batch reader │
|
||||
│ - Gem extractor │
|
||||
│ - Qdrant store │
|
||||
└──────────┬──────────┘
|
||||
│
|
||||
┌──────────▼──────────┐
|
||||
│ EXISTING: │
|
||||
│ qdrant-memory │
|
||||
│ Semantic search │
|
||||
│ Context injection │
|
||||
└─────────────────────┘
|
||||
```
|
||||
|
||||
## Technical Reference
|
||||
|
||||
### Native Context Monitoring
|
||||
|
||||
```typescript
|
||||
// In turn_end hook
|
||||
const usage = ctx.getContextUsage();
|
||||
// usage.tokens, usage.contextWindow, usage.percent
|
||||
// Trigger extraction when usage.percent >= threshold
|
||||
```
|
||||
|
||||
### Primary Hook: turn_end
|
||||
|
||||
```typescript
|
||||
pi.on("turn_end", async (event, ctx) => {
|
||||
const { turnIndex, message, toolResults } = event;
|
||||
|
||||
// Buffer turn to Redis
|
||||
// Check ctx.getContextUsage().percent
|
||||
// If batch >= 5 OR percent >= 50%: extract
|
||||
});
|
||||
```
|
||||
|
||||
### Timer Fallback
|
||||
|
||||
```bash
|
||||
# Cron every 10 min
|
||||
# Check neuralstream:buffer age > 15 min
|
||||
# If yes: extract from partial batch
|
||||
```
|
||||
|
||||
### Context-Aware Extraction
|
||||
|
||||
- Feed qwen3: Up to 256k tokens (full history for context)
|
||||
- Extract from: Last `batch_size` turns only
|
||||
- Benefit: Rich understanding without gemming current context
|
||||
|
||||
## Gem Format
|
||||
|
||||
```json
|
||||
{
|
||||
"gem_id": "uuid",
|
||||
"content": "Distilled insight/fact/decision",
|
||||
"summary": "One-line for quick scanning",
|
||||
"topics": ["docker", "redis", "architecture"],
|
||||
"importance": 0.9,
|
||||
"source": {
|
||||
"session_id": "uuid",
|
||||
"date": "2026-02-23",
|
||||
"turn_range": "15-20"
|
||||
},
|
||||
"tags": ["decision", "fact", "preference", "todo", "code"],
|
||||
"created_at": "2026-02-23T15:26:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration (All Tunable)
|
||||
|
||||
| Setting | Default | Description |
|
||||
|---------|---------|-------------|
|
||||
| batch_size | 5 | Turns per extraction |
|
||||
| context_threshold | 50% | Token % trigger (40-80% range) |
|
||||
| idle_timeout | 15 min | Timer trigger threshold |
|
||||
| gem_model | qwen3 | Extraction LLM (256k context) |
|
||||
| max_gems_injected | 10 | Per-turn limit |
|
||||
| embedding | snowflake-arctic-embed2 | Same as kimi_memories |
|
||||
| collection | neuralstream | Qdrant (1024 dims, Cosine) |
|
||||
|
||||
## Qdrant Schema
|
||||
|
||||
**Collection:** `neuralstream`
|
||||
- Vector size: 1024
|
||||
- Distance: Cosine
|
||||
- On-disk payload: true
|
||||
|
||||
## Project Structure
|
||||
|
||||
```
|
||||
.projects/neuralstream/
|
||||
├── README.md # This file
|
||||
├── session.md # Development log & state
|
||||
├── prompt.md # (TBD) qwen3 extraction prompt
|
||||
└── src/ # (TBD) Implementation
|
||||
├── extract.ts # Gem extraction logic
|
||||
├── store.ts # Qdrant storage
|
||||
└── inject.ts # Context injection
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
- [x] Architecture defined (v2.2 context-aware)
|
||||
- [x] Native context monitoring validated (ctx.getContextUsage)
|
||||
- [x] Naming finalized (NeuralStream, alias: ns)
|
||||
- [x] Hook research completed
|
||||
- [x] Qdrant collection created (`neuralstream`)
|
||||
- [x] Gem format proposed
|
||||
- [x] Infrastructure decision (reuse Redis/Qdrant)
|
||||
- [ ] Extraction prompt design
|
||||
- [ ] Implementation
|
||||
- [ ] Testing
|
||||
|
||||
## Backups
|
||||
|
||||
- Local: `/root/.openclaw/workspace/.projects/neuralstream/`
|
||||
- Remote: `deb2:/root/.projects/neuralstream/` (build/test only)
|
||||
- kimi_kb: Research entries stored
|
||||
|
||||
## Related Projects
|
||||
|
||||
- **True Recall:** Gem extraction inspiration
|
||||
- **OpenClaw:** Host platform
|
||||
- **kimi_memories:** Shared Qdrant infrastructure
|
||||
- **mem-redis-watcher:** Existing capture layer
|
||||
|
||||
---
|
||||
|
||||
**Created:** 2026-02-23
|
||||
**Alias:** ns
|
||||
**Purpose:** Infinite context for LLMs
|
||||
BIN
__pycache__/migrate_memories.cpython-312.pyc
Normal file
BIN
__pycache__/migrate_memories.cpython-312.pyc
Normal file
Binary file not shown.
41
curator-prompt.md
Normal file
41
curator-prompt.md
Normal file
@@ -0,0 +1,41 @@
|
||||
# The Curator System Prompt
|
||||
|
||||
You are The Curator, a discerning AI expert in memory preservation for True-Recall-Out. Like a museum curator selecting priceless artifacts for an exhibit, you exercise careful judgment to identify and preserve only the most valuable "gems" from conversations—moments that truly matter for long-term recall. You are not a hoarder; you focus on substance, context, and lasting value, discarding noise to create a meaningful archive. You run daily at 3 AM, processing 24 hours of conversation data from Redis (a temporary buffer at REDIS_HOST:REDIS_PORT, key pattern 'mem:user_id', list of JSON strings with 24-hour TTL). You treat the entire input as one cohesive narrative story, not isolated messages, to uncover arcs, patterns, and pivotal moments. After extracting gems, you store them in Qdrant (vector database at http://10.0.0.40:6333, collection 'kimi_memories', using snowflake-arctic-embed2 with 1024 dimensions and cosine similarity; payload is the full gem object). Then, clear the Redis buffer. Your input is a JSON array of conversation turns. Each turn object includes: user_id (speaker), user_message (user's text), ai_response (AI's text), turn (number), timestamp (ISO 8601, e.g., "2026-02-22T14:30:00"), date (YYYY-MM-DD, e.g., "2026-02-22"), conversation_id (unique string, e.g., "abc123"). Example input snippet: [ { "user_id": "rob", "user_message": "Should I use Redis or Postgres for caching?", "ai_response": "For short-term caching, Redis is faster; Postgres is better for persistence.", "turn": 15, "timestamp": "2026-02-22T14:28:00", "date": "2026-02-22", "conversation_id": "abc123" }, { "user_id": "rob", "user_message": "I decided on Redis. Speed matters more for this use case.", "ai_response": "Good choice; Redis will handle the caching layer efficiently.", "turn": 16, "timestamp": "2026-02-22T14:30:00", "date": "2026-02-22", "conversation_id": "abc123" } ] Your task: Read the full narrative, identify gems (important moments like decisions or insights), extract them with rich details, and output a JSON array of gems. If no gems, return an empty array []. Each gem MUST have exactly these 11 required fields (all present, no extras): - "gem": String, 1-2 sentences summarizing the main insight/decision (e.g., "User decided to use Redis over Postgres for memory system caching."). - "context": String, 2-3 sentences explaining why it matters (e.g., "After discussing tradeoffs between persistence versus speed for short-term storage, user prioritized speed over data durability. This choice impacts system performance."). - "snippet": String, raw conversation excerpt (2-3 turns, with speakers, e.g., "rob: Should I use Redis or Postgres for caching? Kimi: For short-term caching, Redis is faster; Postgres is better for persistence. rob: I decided on Redis. Speed matters more for this use case."). - "categories": Array of strings, tags like ["decision", "technical", "preference", "project", "knowledge", "insight", "plan", "architecture", "workflow"] (non-empty, 1-5 items). - "importance": String, "high", "medium", or "low" (must be medium or high for storage). - "confidence": Float, 0.0-1.0 (must be >=0.6; target 0.8+). - "timestamp": String, exact ISO 8601 from the last turn in the range (e.g., "2026-02-22T14:30:00"). - "date": String, YYYY-MM-DD from timestamp (e.g., "2026-02-22"). - "conversation_id": String, from input (e.g., "abc123"). - "turn_range": String, first-last turn (e.g., "15-16"). - "source_turns": Array of integers, all turns involved (e.g., [15, 16]). Output strictly as JSON array, no extra text. ### What Makes a Gem Extract gems only for: - Decisions: User chooses one option (e.g., "I decided on Redis", "Let's go with Mattermost", "I'm switching to Linux"). - Technical solutions: Problem-solving methods (e.g., "Use Python asyncio", "Fix by increasing timeout", "Deploy with Docker Compose"). - Preferences: Likes/dislikes (e.g., "I prefer dark mode", "I hate popups", "Local is better than cloud"). - Projects: Work details (e.g., "Building a memory system", "Setting up True-Recall", "Working on the website"). - Knowledge: Learned facts (e1. **Timestamp:** Use the exact ISO 8601 from the final turn where the gem crystallized (e.g., decision finalized).
|
||||
2. **Date:** Derive as YYYY-MM-DD from timestamp.
|
||||
3. **Conversation_id:** Copy from input (consistent across turns).
|
||||
4. **Turn_range:** "first-last" (e.g., "15-16" for contiguous; "15-16,18" if non-contiguous but prefer contiguous).
|
||||
5. **Source_turns:** List all integers (e.g., [15,16]).
|
||||
|
||||
### Evaluation Process
|
||||
|
||||
Follow these steps strictly:
|
||||
|
||||
**Step 1: Read as Narrative.** Treat the entire JSON array as one story. Scan for arcs (e.g., problem to solution), patterns (e.g., repeated preferences), decisions, insights. Note timestamps for timing.
|
||||
|
||||
**Step 2: Identify Gems.** For each potential:
|
||||
- Worth remembering in 6 months? (Yes = proceed; no = skip).
|
||||
- Has context? (Explain why matters).
|
||||
- **Duplicate Check:** If this expresses the same decision/concept as a previous gem (even re-phrased), MERGE the context instead of creating a new gem. Combine insights from both sources for richer context.
|
||||
- Confidence? (>=0.6 = proceed).
|
||||
- Precise timestamp? (From last relevant turn).
|
||||
|
||||
**Step 3: Extract with Context and Timestamp.**
|
||||
- Gem: Concise 1-2 sentences.
|
||||
- Context: 2-3 explanatory sentences.
|
||||
- Snippet: Raw dialogue (speakers: messages).
|
||||
- Add metadata: Categories (match types), importance (high for critical, medium for useful), confidence, timestamp (last turn), date, conversation_id, turn_range, source_turns.
|
||||
|
||||
**Step 4: Validate.**
|
||||
- Output valid JSON array.
|
||||
- Each gem has all 11 fields.
|
||||
- Timestamp valid ISO 8601.
|
||||
- Date matches timestamp.
|
||||
- Confidence float 0.0-1.0 (>=0.6).
|
||||
- Importance "high"/"medium".
|
||||
- Categories non-empty array.
|
||||
- Snippet has dialogue.
|
||||
- Source_turns matches turn_range.
|
||||
- Conversation_id from input.
|
||||
Fix any issues.
|
||||
|
||||
**Step 5: Output.** Return JSON array of gems (or []). Encourage discernment: Preserve only what adds value, like selecting exhibit pieces that tell a compelling story.
|
||||
106
debug_curator.py
Normal file
106
debug_curator.py
Normal file
@@ -0,0 +1,106 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Debug curator with real data"""
|
||||
import json
|
||||
import requests
|
||||
import urllib.request
|
||||
|
||||
QDRANT_URL = "http://10.0.0.40:6333"
|
||||
SOURCE_COLLECTION = "kimi_memories"
|
||||
|
||||
# Get sample turns from real data
|
||||
filter_data = {
|
||||
"must": [
|
||||
{"key": "user_id", "match": {"value": "rob"}},
|
||||
{"key": "date", "match": {"value": "2026-02-23"}}
|
||||
]
|
||||
}
|
||||
|
||||
req = urllib.request.Request(
|
||||
f"{QDRANT_URL}/collections/{SOURCE_COLLECTION}/points/scroll",
|
||||
data=json.dumps({"limit": 5, "with_payload": True, "filter": filter_data}).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST"
|
||||
)
|
||||
|
||||
with urllib.request.urlopen(req, timeout=30) as response:
|
||||
result = json.loads(response.read().decode())
|
||||
points = result.get("result", {}).get("points", [])
|
||||
|
||||
turns = []
|
||||
for point in points:
|
||||
payload = point.get("payload", {})
|
||||
user_msg = payload.get("user_message", "")
|
||||
ai_msg = payload.get("ai_response", "")
|
||||
|
||||
if user_msg or ai_msg:
|
||||
turn = {
|
||||
"turn": payload.get("turn_number", 0),
|
||||
"user_id": payload.get("user_id", "rob"),
|
||||
"user": user_msg[:300], # Truncate
|
||||
"ai": ai_msg[:300], # Truncate
|
||||
"conversation_id": payload.get("conversation_id", ""),
|
||||
"timestamp": payload.get("created_at", ""),
|
||||
"date": payload.get("date", "2026-02-23")
|
||||
}
|
||||
turns.append(turn)
|
||||
|
||||
turns.sort(key=lambda x: (x.get("conversation_id", ""), x.get("turn", 0)))
|
||||
|
||||
print(f"Got {len(turns)} turns")
|
||||
print("Sample:")
|
||||
for t in turns[:2]:
|
||||
print(f" User: {t['user'][:100]}...")
|
||||
print(f" AI: {t['ai'][:100]}...")
|
||||
|
||||
# Now test with curator
|
||||
with open('/root/.openclaw/workspace/.projects/true-recall-v1/curator-prompt.md') as f:
|
||||
prompt = f.read()
|
||||
|
||||
conversation_json = json.dumps(turns[:5], indent=2)
|
||||
|
||||
prompt_text = f"""## Input Conversation
|
||||
|
||||
```json
|
||||
{conversation_json}
|
||||
```
|
||||
|
||||
## Output
|
||||
"""
|
||||
|
||||
response = requests.post(
|
||||
'http://10.0.0.10:11434/api/generate',
|
||||
json={
|
||||
'model': 'qwen3:4b-instruct',
|
||||
'system': prompt,
|
||||
'prompt': prompt_text,
|
||||
'stream': False,
|
||||
'options': {'temperature': 0.1, 'num_predict': 3000}
|
||||
},
|
||||
timeout=120
|
||||
)
|
||||
|
||||
result = response.json()
|
||||
output = result.get('response', '').strip()
|
||||
|
||||
print("\n=== CURATOR OUTPUT ===")
|
||||
print(output[:3000])
|
||||
print("\n=== TRYING TO PARSE ===")
|
||||
|
||||
# Try to parse
|
||||
try:
|
||||
if '```json' in output:
|
||||
parsed = output.split('```json')[1].split('```')[0].strip()
|
||||
gems = json.loads(parsed)
|
||||
print(f"Parsed {len(gems)} gems")
|
||||
elif '```' in output:
|
||||
parsed = output.split('```')[1].split('```')[0].strip()
|
||||
gems = json.loads(parsed)
|
||||
print(f"Parsed {len(gems)} gems")
|
||||
else:
|
||||
gems = json.loads(output)
|
||||
print(f"Parsed {len(gems)} gems")
|
||||
except Exception as e:
|
||||
print(f"Parse error: {e}")
|
||||
print("Trying raw parse...")
|
||||
gems = json.loads(output.strip())
|
||||
print(f"Parsed {len(gems)} gems")
|
||||
187
migrate_memories.py
Normal file
187
migrate_memories.py
Normal file
@@ -0,0 +1,187 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Migrate memories from kimi_memories to memories_tr
|
||||
- Reads from kimi_memories (Qdrant)
|
||||
- Cleans/strips noise (metadata, thinking tags)
|
||||
- Stores to memories_tr (Qdrant)
|
||||
- Keeps original kimi_memories intact
|
||||
"""
|
||||
|
||||
import json
|
||||
import urllib.request
|
||||
import urllib.error
|
||||
from datetime import datetime
|
||||
from typing import List, Dict, Any
|
||||
|
||||
QDRANT_URL = "http://10.0.0.40:6333"
|
||||
SOURCE_COLLECTION = "kimi_memories"
|
||||
TARGET_COLLECTION = "memories_tr"
|
||||
|
||||
def clean_content(text: str) -> str:
|
||||
"""Clean noise from content"""
|
||||
if not text:
|
||||
return ""
|
||||
|
||||
cleaned = text
|
||||
|
||||
# Remove metadata JSON blocks
|
||||
import re
|
||||
cleaned = re.sub(r'Conversation info \(untrusted metadata\):\s*```json\s*\{[\s\S]*?\}\s*```', '', cleaned)
|
||||
|
||||
# Remove thinking tags
|
||||
cleaned = re.sub(r'\[thinking:[^\]]*\]', '', cleaned)
|
||||
|
||||
# Remove timestamp lines
|
||||
cleaned = re.sub(r'\[\w{3} \d{4}-\d{2}-\d{2} \d{2}:\d{2} [A-Z]{3}\]', '', cleaned)
|
||||
|
||||
# Clean up whitespace
|
||||
cleaned = re.sub(r'\n{3,}', '\n\n', cleaned)
|
||||
cleaned = cleaned.strip()
|
||||
|
||||
return cleaned
|
||||
|
||||
def get_all_points(collection: str) -> List[Dict]:
|
||||
"""Get all points from a collection"""
|
||||
all_points = []
|
||||
offset = None
|
||||
max_iterations = 1000
|
||||
iterations = 0
|
||||
|
||||
while iterations < max_iterations:
|
||||
iterations += 1
|
||||
scroll_data = {
|
||||
"limit": 100,
|
||||
"with_payload": True,
|
||||
"with_vector": True
|
||||
}
|
||||
|
||||
if offset:
|
||||
scroll_data["offset"] = offset
|
||||
|
||||
req = urllib.request.Request(
|
||||
f"{QDRANT_URL}/collections/{collection}/points/scroll",
|
||||
data=json.dumps(scroll_data).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST"
|
||||
)
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=60) as response:
|
||||
result = json.loads(response.read().decode())
|
||||
points = result.get("result", {}).get("points", [])
|
||||
|
||||
if not points:
|
||||
break
|
||||
|
||||
all_points.extend(points)
|
||||
|
||||
offset = result.get("result", {}).get("next_page_offset")
|
||||
if not offset:
|
||||
break
|
||||
except urllib.error.HTTPError as e:
|
||||
print(f"Error: {e}")
|
||||
break
|
||||
|
||||
return all_points
|
||||
|
||||
def store_points(collection: str, points: List[Dict]) -> int:
|
||||
"""Store points to collection"""
|
||||
if not points:
|
||||
return 0
|
||||
|
||||
# Batch upload
|
||||
batch_size = 100
|
||||
stored = 0
|
||||
|
||||
for i in range(0, len(points), batch_size):
|
||||
batch = points[i:i+batch_size]
|
||||
|
||||
points_data = {
|
||||
"points": batch
|
||||
}
|
||||
|
||||
req = urllib.request.Request(
|
||||
f"{QDRANT_URL}/collections/{collection}/points",
|
||||
data=json.dumps(points_data).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="PUT"
|
||||
)
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=60) as response:
|
||||
if response.status == 200:
|
||||
stored += len(batch)
|
||||
except urllib.error.HTTPError as e:
|
||||
print(f"Error storing batch: {e}")
|
||||
|
||||
return stored
|
||||
|
||||
def migrate_point(point: Dict) -> Dict:
|
||||
"""Clean a single point"""
|
||||
payload = point.get("payload", {})
|
||||
|
||||
# Clean user and AI messages
|
||||
user_msg = clean_content(payload.get("user_message", ""))
|
||||
ai_msg = clean_content(payload.get("ai_response", ""))
|
||||
|
||||
# Keep other fields
|
||||
cleaned_payload = {
|
||||
**payload,
|
||||
"user_message": user_msg,
|
||||
"ai_response": ai_msg,
|
||||
"migrated_from": "kimi_memories",
|
||||
"migrated_at": datetime.now().isoformat()
|
||||
}
|
||||
|
||||
return {
|
||||
"id": point.get("id"),
|
||||
"vector": point.get("vector"),
|
||||
"payload": cleaned_payload
|
||||
}
|
||||
|
||||
def main():
|
||||
print("=" * 60)
|
||||
print("Memory Migration: kimi_memories → memories_tr")
|
||||
print("=" * 60)
|
||||
print()
|
||||
|
||||
# Check source
|
||||
print(f"📥 Reading from {SOURCE_COLLECTION}...")
|
||||
source_points = get_all_points(SOURCE_COLLECTION)
|
||||
print(f" Found {len(source_points)} points")
|
||||
|
||||
if not source_points:
|
||||
print("❌ No points to migrate")
|
||||
return
|
||||
|
||||
# Clean points
|
||||
print(f"\n🧹 Cleaning {len(source_points)} points...")
|
||||
cleaned_points = [migrate_point(p) for p in source_points]
|
||||
print(f" ✓ Cleaned")
|
||||
|
||||
# Store to target
|
||||
print(f"\n💾 Storing to {TARGET_COLLECTION}...")
|
||||
stored = store_points(TARGET_COLLECTION, cleaned_points)
|
||||
print(f" ✓ Stored {stored} points")
|
||||
|
||||
# Verify
|
||||
print(f"\n🔍 Verifying...")
|
||||
target_points = get_all_points(TARGET_COLLECTION)
|
||||
print(f" Target now has {len(target_points)} points")
|
||||
|
||||
# Summary
|
||||
print()
|
||||
print("=" * 60)
|
||||
print("Migration Summary:")
|
||||
print(f" Source ({SOURCE_COLLECTION}): {len(source_points)} points")
|
||||
print(f" Target ({TARGET_COLLECTION}): {len(target_points)} points")
|
||||
print(f" Cleaned & migrated: {stored} points")
|
||||
print("=" * 60)
|
||||
|
||||
if stored == len(source_points):
|
||||
print("\n✅ Migration complete!")
|
||||
else:
|
||||
print(f"\n⚠️ Warning: Only migrated {stored}/{len(source_points)} points")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
494
session.md
Normal file
494
session.md
Normal file
@@ -0,0 +1,494 @@
|
||||
# TrueRecall v2 - Session Notes
|
||||
|
||||
**Last Updated:** 2026-02-24 19:02 CST
|
||||
**Status:** ✅ Active & Verified
|
||||
**Version:** v2.2 (Timer-based curation deployed)
|
||||
|
||||
---
|
||||
|
||||
## Session End (18:09 CST)
|
||||
|
||||
**Reason:** User starting new session
|
||||
|
||||
**Current State:**
|
||||
- Real-time watcher: ✅ Active (capturing live)
|
||||
- Timer curator: ✅ Deployed (every 30 min via cron)
|
||||
- Daily curator: ❌ Removed (replaced by timer)
|
||||
- Total memories: 12,378 (all tagged with `curated: false`)
|
||||
- Gems: 5 (from Feb 18 test)
|
||||
|
||||
**Next session start:** Read this file, then check:
|
||||
```bash
|
||||
# Quick status
|
||||
python3 /root/.openclaw/workspace/.projects/true-recall-v2/tr-continuous/curator_by_count.py --status
|
||||
sudo systemctl status mem-qdrant-watcher
|
||||
curl -s http://10.0.0.40:6333/collections/memories_tr | jq '.result.points_count'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
TrueRecall v2 is a complete memory system with real-time capture, daily curation, and context injection. All components are operational.
|
||||
|
||||
---
|
||||
|
||||
## Current State (Verified 18:09 CST)
|
||||
|
||||
### Qdrant Collections
|
||||
|
||||
| Collection | Points | Purpose | Status |
|
||||
|------------|--------|---------|--------|
|
||||
| `memories_tr` | **12,378** | Full text (live capture) | ✅ Active |
|
||||
| `gems_tr` | **5** | Curated gems (injection) | ✅ Active |
|
||||
| `true_recall` | existing | Legacy archive | 📦 Preserved |
|
||||
| `kimi_memories` | 12,223 | Original backup | 📦 Preserved |
|
||||
|
||||
**Note:** All memories tagged with `curated: false` for timer curator.
|
||||
|
||||
### Services
|
||||
|
||||
| Service | Status | Uptime |
|
||||
|---------|--------|--------|
|
||||
| `mem-qdrant-watcher` | ✅ Active | 30+ min |
|
||||
| OpenClaw Gateway | ✅ Running | 2026.2.23 |
|
||||
| memory-qdrant plugin | ✅ Loaded | recall: gems_tr, capture: memories_tr |
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
### v2.2: Timer-Based Curation (DEPLOYED)
|
||||
|
||||
**Data Flow:**
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────────┐ ┌─────────────┐
|
||||
│ OpenClaw Chat │────▶│ Real-Time Watcher │────▶│ memories_tr │
|
||||
│ (Session JSONL)│ │ (Python daemon) │ │ (Qdrant) │
|
||||
└─────────────────┘ └──────────────────────┘ └──────┬──────┘
|
||||
│
|
||||
│ Every 30 min
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ Timer Curator │
|
||||
│ (cron/qwen3) │
|
||||
└────────┬─────────┘
|
||||
│
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ gems_tr │
|
||||
│ (Qdrant) │
|
||||
└────────┬─────────┘
|
||||
│
|
||||
Per turn │
|
||||
▼
|
||||
┌──────────────────┐
|
||||
│ memory-qdrant │
|
||||
│ plugin │
|
||||
└──────────────────┘
|
||||
```
|
||||
|
||||
**Key Changes:**
|
||||
- ✅ Replaced daily 2:45 AM batch with 30-minute timer
|
||||
- ✅ All memories tagged `curated: false` on write
|
||||
- ✅ Migration completed for 12,378 existing memories
|
||||
- ✅ No Redis dependency (direct Qdrant only)
|
||||
|
||||
---
|
||||
|
||||
## Components
|
||||
|
||||
### Curation Mode: Timer-Based (DEPLOYED v2.2)
|
||||
|
||||
| Setting | Value | Adjustable |
|
||||
|---------|-------|------------|
|
||||
| **Trigger** | Cron timer | ✅ |
|
||||
| **Interval** | 30 minutes | ✅ Config file |
|
||||
| **Batch size** | 100 memories max | ✅ Config file |
|
||||
| **Minimum** | None (0 is OK) | — |
|
||||
|
||||
**Config:** `/tr-continuous/curator_config.json`
|
||||
```json
|
||||
{
|
||||
"timer_minutes": 30,
|
||||
"max_batch_size": 100,
|
||||
"user_id": "rob",
|
||||
"source_collection": "memories_tr",
|
||||
"target_collection": "gems_tr"
|
||||
}
|
||||
```
|
||||
|
||||
**Cron:**
|
||||
```
|
||||
*/30 * * * * cd .../tr-continuous && python3 curator_timer.py
|
||||
```
|
||||
|
||||
**Old modes deprecated:**
|
||||
- ❌ Turn-based (every N turns)
|
||||
- ❌ Hybrid (timer + turn)
|
||||
- ❌ Daily batch (2:45 AM)
|
||||
|
||||
### 1. Real-Time Watcher (Primary Capture)
|
||||
|
||||
**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
||||
|
||||
**Function:**
|
||||
- Watches `/root/.openclaw/agents/main/sessions/*.jsonl`
|
||||
- Parses every conversation turn in real-time
|
||||
- Embeds with `snowflake-arctic-embed2` (Ollama @ 10.0.0.10)
|
||||
- Stores directly to `memories_tr` (no Redis)
|
||||
- **Cleans content:** Removes markdown, tables, metadata, thinking tags
|
||||
|
||||
**Service:** `mem-qdrant-watcher.service`
|
||||
- **Status:** Active since 16:46:53 CST
|
||||
- **Systemd:** Enabled, auto-restart
|
||||
|
||||
**Log:** `journalctl -u mem-qdrant-watcher -f`
|
||||
|
||||
---
|
||||
|
||||
### 2. Content Cleaner (Existing Data)
|
||||
|
||||
**Location:** `/root/.openclaw/workspace/skills/qdrant-memory/scripts/clean_memories_tr.py`
|
||||
|
||||
**Function:**
|
||||
- Batch-cleans existing `memories_tr` points
|
||||
- Removes: `**bold**`, `|tables|`, `` `code` ``, `---` rules, `# headers`
|
||||
- Flattens nested content dicts
|
||||
- Rate-limited to prevent Qdrant overload
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Dry run (preview)
|
||||
python3 clean_memories_tr.py --dry-run
|
||||
|
||||
# Clean all
|
||||
python3 clean_memories_tr.py --execute
|
||||
|
||||
# Clean limited (test)
|
||||
python3 clean_memories_tr.py --execute --limit 100
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### 3. Timer Curator (v2.2 - DEPLOYED)
|
||||
|
||||
**Replaces:** Daily curator (2:45 AM batch) and turn-based curator
|
||||
|
||||
**Location:** `/root/.openclaw/workspace/.projects/true-recall-v2/tr-continuous/curator_timer.py`
|
||||
|
||||
**Schedule:** Every 30 minutes (cron)
|
||||
|
||||
**Flow:**
|
||||
1. Query uncurated memories (`curated: false`)
|
||||
2. Send batch to qwen3 (max 100)
|
||||
3. Extract gems using curator prompt
|
||||
4. Store gems to `gems_tr`
|
||||
5. Mark processed memories as `curated: true`
|
||||
|
||||
**Files:**
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `curator_timer.py` | Main curator script |
|
||||
| `curator_config.json` | Adjustable settings |
|
||||
| `migrate_add_curated.py` | One-time migration (completed) |
|
||||
|
||||
**Usage:**
|
||||
```bash
|
||||
# Dry run (preview)
|
||||
python3 curator_timer.py --dry-run
|
||||
|
||||
# Manual run
|
||||
python3 curator_timer.py --config curator_config.json
|
||||
```
|
||||
|
||||
**Status:** ✅ Deployed, first run will process ~12,378 existing memories
|
||||
|
||||
### 5. Silent Compacting (NEW - Concept)
|
||||
|
||||
**Idea:** Automatically remove old context from prompt when token limit approached.
|
||||
|
||||
**Behavior:**
|
||||
- Trigger: Context window > 80% full
|
||||
- Action: Remove oldest messages (silently)
|
||||
- Preserve: Gems always kept, recent N turns kept
|
||||
- Result: Seamless conversation without "compacting" notification
|
||||
|
||||
**Config:**
|
||||
```json
|
||||
{
|
||||
"compacting": {
|
||||
"enabled": true,
|
||||
"triggerAtPercent": 80,
|
||||
"keepRecentTurns": 20,
|
||||
"preserveGems": true,
|
||||
"silent": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Status:** ⏳ Concept only - requires OpenClaw core changes
|
||||
|
||||
### 6. memory-qdrant Plugin
|
||||
|
||||
**Location:** `/root/.openclaw/extensions/memory-qdrant/`
|
||||
|
||||
**Config:**
|
||||
```json
|
||||
{
|
||||
"collectionName": "gems_tr",
|
||||
"captureCollection": "memories_tr",
|
||||
"autoRecall": true,
|
||||
"autoCapture": true
|
||||
}
|
||||
```
|
||||
|
||||
**Function:**
|
||||
- **Recall:** Searches `gems_tr`, injects as context (hidden)
|
||||
- **Capture:** Session-level capture to `memories_tr` (backup)
|
||||
|
||||
**Status:** Loaded, dual collection support working
|
||||
|
||||
---
|
||||
|
||||
## Files & Locations
|
||||
|
||||
### Core Project Files
|
||||
|
||||
```
|
||||
/root/.openclaw/workspace/.projects/true-recall-v2/
|
||||
├── README.md # Architecture docs
|
||||
├── session.md # This file
|
||||
├── curator-prompt.md # Gem extraction prompt
|
||||
├── tr-daily/ # Daily batch curation
|
||||
│ └── curate_from_qdrant.py # Daily curator (2:45 AM)
|
||||
├── tr-continuous/ # Real-time curation (NEW)
|
||||
│ ├── curator_by_count.py # Turn-based curator
|
||||
│ ├── curator_turn_based.py # Alternative approach
|
||||
│ ├── curator_cron.sh # Cron wrapper
|
||||
│ ├── turn-curator.service # Systemd service
|
||||
│ └── README.md # Documentation
|
||||
└── shared/
|
||||
└── (shared resources)
|
||||
```
|
||||
|
||||
### New Files (2026-02-24 19:00)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `tr-continuous/curator_timer.py` | Timer-based curator (deployed) |
|
||||
| `tr-continuous/curator_config.json` | Curator settings |
|
||||
| `tr-continuous/migrate_add_curated.py` | Migration script (completed) |
|
||||
|
||||
### Legacy Files (Pre-v2.2)
|
||||
|
||||
| File | Status | Note |
|
||||
|------|--------|------|
|
||||
| `tr-daily/curate_from_qdrant.py` | 📦 Archived | Replaced by timer |
|
||||
| `tr-continuous/curator_by_count.py` | 📦 Archived | Replaced by timer |
|
||||
| `tr-continuous/curator_turn_based.py` | 📦 Archived | Replaced by timer |
|
||||
|
||||
### System Locations
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `/root/.openclaw/extensions/memory-qdrant/` | Plugin code |
|
||||
| `/root/.openclaw/openclaw.json` | Plugin configuration |
|
||||
| `/etc/systemd/system/mem-qdrant-watcher.service` | Systemd service |
|
||||
|
||||
---
|
||||
|
||||
## Changes Made Today (2026-02-24 19:00)
|
||||
|
||||
### 1. Timer Curator Deployed (v2.2)
|
||||
|
||||
- Created `curator_timer.py` — simplified timer-based curation
|
||||
- Created `curator_config.json` — adjustable settings
|
||||
- Removed daily 2:45 AM cron job
|
||||
- Added `*/30 * * * *` cron timer
|
||||
- **Status:** ✅ Deployed, logs to `/var/log/true-recall-timer.log`
|
||||
|
||||
### 2. Migration Completed
|
||||
|
||||
- Created `migrate_add_curated.py`
|
||||
- Tagged 12,378 existing memories with `curated: false`
|
||||
- Updated watcher to add `curated: false` to new memories
|
||||
- **Status:** ✅ Complete
|
||||
|
||||
### 3. Simplified Architecture
|
||||
|
||||
- ❌ Removed turn-based curator complexity
|
||||
- ❌ Removed daily batch processing
|
||||
- ✅ Single timer trigger every 30 minutes
|
||||
- ✅ No minimum threshold (processes 0-N memories)
|
||||
|
||||
---
|
||||
|
||||
## Configuration
|
||||
|
||||
### memory-qdrant Plugin
|
||||
|
||||
**File:** `/root/.openclaw/openclaw.json`
|
||||
|
||||
```json
|
||||
{
|
||||
"memory-qdrant": {
|
||||
"config": {
|
||||
"autoCapture": true,
|
||||
"autoRecall": true,
|
||||
"collectionName": "gems_tr",
|
||||
"captureCollection": "memories_tr",
|
||||
"embeddingModel": "snowflake-arctic-embed2",
|
||||
"maxRecallResults": 2,
|
||||
"minRecallScore": 0.7,
|
||||
"ollamaUrl": "http://10.0.0.10:11434",
|
||||
"qdrantUrl": "http://10.0.0.40:6333"
|
||||
},
|
||||
"enabled": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Gateway (OpenClaw Update Fix)
|
||||
|
||||
```json
|
||||
{
|
||||
"gateway": {
|
||||
"controlUi": {
|
||||
"allowedOrigins": ["*"],
|
||||
"allowInsecureAuth": false,
|
||||
"dangerouslyDisableDeviceAuth": true
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Validation Commands
|
||||
|
||||
### Check Collections
|
||||
|
||||
```bash
|
||||
# Points count
|
||||
curl -s http://10.0.0.40:6333/collections/memories_tr | jq '.result.points_count'
|
||||
curl -s http://10.0.0.40:6333/collections/gems_tr | jq '.result.points_count'
|
||||
|
||||
# Recent points
|
||||
curl -s -X POST http://10.0.0.40:6333/collections/memories_tr/points/scroll \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"limit": 5, "with_payload": true}' | jq '.result.points[].payload.content'
|
||||
```
|
||||
|
||||
### Check Services
|
||||
|
||||
```bash
|
||||
# Watcher status
|
||||
sudo systemctl status mem-qdrant-watcher
|
||||
|
||||
# Watcher logs
|
||||
sudo journalctl -u mem-qdrant-watcher -n 20
|
||||
|
||||
# OpenClaw status
|
||||
openclaw status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Issue: Watcher Not Capturing
|
||||
|
||||
**Check:**
|
||||
1. Service running? `systemctl status mem-qdrant-watcher`
|
||||
2. Logs: `journalctl -u mem-qdrant-watcher -f`
|
||||
3. Qdrant accessible? `curl http://10.0.0.40:6333/`
|
||||
4. Ollama accessible? `curl http://10.0.0.10:11434/api/tags`
|
||||
|
||||
### Issue: Cleaner Fails
|
||||
|
||||
**Common causes:**
|
||||
- Qdrant connection timeout (add `time.sleep(0.1)` between batches)
|
||||
- Nested content dicts (handled in updated script)
|
||||
- Type errors (non-string content — handled)
|
||||
|
||||
### Issue: Plugin Not Loading
|
||||
|
||||
**Check:**
|
||||
1. `openclaw.json` syntax valid? `openclaw config validate`
|
||||
2. Plugin compiled? `cd /root/.openclaw/extensions/memory-qdrant && npx tsc`
|
||||
3. Gateway logs: `tail /tmp/openclaw/openclaw-$(date +%Y-%m-%d).log`
|
||||
|
||||
---
|
||||
|
||||
## Cron Schedule (Updated v2.2)
|
||||
|
||||
| Time | Job | Script | Status |
|
||||
|------|-----|--------|--------|
|
||||
| Every 30 min | Timer curator | `tr-continuous/curator_timer.py` | ✅ Active |
|
||||
| Per turn | Capture | `mem-qdrant-watcher` | ✅ Daemon |
|
||||
| Per turn | Injection | `memory-qdrant` plugin | ✅ Active |
|
||||
|
||||
**Removed:**
|
||||
- ❌ 2:45 AM daily curator
|
||||
- ❌ Every-minute turn curator check
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
### Immediate
|
||||
- ⏳ Monitor first timer run (logs: `/var/log/true-recall-timer.log`)
|
||||
- ⏳ Validate gem extraction quality from timer curator
|
||||
- ⏳ Archive old curator scripts if timer works
|
||||
|
||||
### Completed ✅
|
||||
- ✅ **Compactor config** — Minimal overhead: `mode: default`, `reserveTokensFloor: 0`, `memoryFlush: false`
|
||||
|
||||
### Future
|
||||
- ⏳ Curator tuning based on timer results
|
||||
- ⏳ Silent compacting (requires OpenClaw core changes)
|
||||
|
||||
### Planned Features (Backlog)
|
||||
- ⏳ **Interactive install script** — Prompts for embedding model, timer interval, batch size, endpoints
|
||||
- ⏳ **Single embedding model option** — Use one model for both collections
|
||||
- ⏳ **Configurable thresholds** — Per-user customization via prompts
|
||||
|
||||
**Compactor Settings (Applied):**
|
||||
```json5
|
||||
{
|
||||
agents: {
|
||||
defaults: {
|
||||
compaction: {
|
||||
mode: "default",
|
||||
reserveTokensFloor: 0,
|
||||
memoryFlush: { enabled: false }
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Note:** Only `mode`, `reserveTokensFloor`, and `memoryFlush` are valid under `agents.defaults.compaction`. Other settings are Pi runtime parameters.
|
||||
|
||||
**Install script prompts:**
|
||||
1. Embedding model (snowflake vs mxbai)
|
||||
2. Timer interval (5 min / 30 min / hourly)
|
||||
3. Batch size (50 / 100 / 500)
|
||||
4. Qdrant/Ollama URLs
|
||||
5. User ID
|
||||
|
||||
---
|
||||
|
||||
## Session Recovery
|
||||
|
||||
If starting fresh:
|
||||
1. Read `README.md` for architecture overview
|
||||
2. Check service status: `sudo systemctl status mem-qdrant-watcher`
|
||||
3. Check timer curator: `tail /var/log/true-recall-timer.log`
|
||||
4. Verify collections: `curl http://10.0.0.40:6333/collections`
|
||||
|
||||
---
|
||||
|
||||
*Last Verified: 2026-02-24 19:29 CST*
|
||||
*Version: v2.2 (30b curator, install script planned)*
|
||||
224
session.md.bak.2026-02-24-1518
Normal file
224
session.md.bak.2026-02-24-1518
Normal file
@@ -0,0 +1,224 @@
|
||||
# TrueRecall v2 - Session Notes
|
||||
|
||||
**Last Updated:** 2026-02-24 15:08 CST
|
||||
|
||||
---
|
||||
|
||||
## Summary of Changes (2026-02-24)
|
||||
|
||||
### Collection Migration
|
||||
|
||||
| Action | From | To | Status |
|
||||
|--------|------|----|--------|
|
||||
| **Rename** | `kimi_memories` | `memories_tr` | ✅ Done + data migrated (12,223 points) |
|
||||
| **Create** | — | `gems_tr` | ✅ Created (empty, ready for curation) |
|
||||
| **Archive** | `true_recall` | `true_recall` | ✅ Preserved (existing data kept) |
|
||||
|
||||
### Dual Collection Support (NEW - 14:10)
|
||||
|
||||
**Problem:** Auto-capture was saving to `gems_tr` because that's where recall pulled from. This was wrong.
|
||||
|
||||
**Solution:** Added `captureCollection` option to memory-qdrant plugin:
|
||||
- `collectionName`: `gems_tr` — for recall/injection (gems only)
|
||||
- `captureCollection`: `memories_tr` — for auto-capture (full conversations)
|
||||
|
||||
**Files Modified:**
|
||||
1. `/root/.openclaw/extensions/memory-qdrant/config.ts` — Added `captureCollection` option
|
||||
2. `/root/.openclaw/extensions/memory-qdrant/index.ts` — Uses `dbRecall` and `dbCapture` separately
|
||||
3. `/root/.openclaw/openclaw.json` — Added `"captureCollection": "memories_tr"`
|
||||
|
||||
### Auto-Capture Architecture (Real-Time Watcher)
|
||||
|
||||
**Mechanism:** `realtime_qdrant_watcher.py` daemon
|
||||
- Watches OpenClaw session JSONL files in real-time
|
||||
- Parses each conversation turn
|
||||
- Embeds with `snowflake-arctic-embed2` via Ollama
|
||||
- Stores directly to `memories_tr` collection (no Redis)
|
||||
- Cleans content (removes metadata, thinking tags)
|
||||
|
||||
**Files:**
|
||||
- Script: `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
|
||||
- Service: `/root/.openclaw/workspace/skills/qdrant-memory/mem-qdrant-watcher.service`
|
||||
|
||||
**Status:** ✅ Deployed and running
|
||||
|
||||
**Deployment:**
|
||||
- Stopped: `mem-redis-watcher`
|
||||
- Started: `mem-qdrant-watcher`
|
||||
- Status: Active (PID 84465)
|
||||
- Verified: ✅ Capturing (points: 12,223 → 12,224)
|
||||
|
||||
### Current State (2026-02-24 15:08)
|
||||
|
||||
| Collection | Purpose | Points | Last Update |
|
||||
|------------|---------|--------|-------------|
|
||||
| `memories_tr` | Full text (autoCapture via watcher) | 12,228 | **Live** |
|
||||
| `gems_tr` | Gems (for injection) | 5 | Feb 24 |
|
||||
|
||||
**Flow:**
|
||||
```
|
||||
OpenClaw Session → Real-Time Watcher → memories_tr (Qdrant)
|
||||
↓
|
||||
Daily Curator (2:45 AM)
|
||||
↓
|
||||
gems_tr (Qdrant)
|
||||
↓
|
||||
memory-qdrant plugin → Injection
|
||||
```
|
||||
|
||||
### Files Modified Today
|
||||
|
||||
1. `/root/.openclaw/openclaw.json` — Changed collectionName to `gems_tr`
|
||||
2. `/root/.openclaw/extensions/memory-qdrant/config.ts` — Default to `memories_tr`
|
||||
3. `/root/.openclaw/extensions/memory-qdrant/openclaw.plugin.json` — Defaults to `memories_tr`
|
||||
4. `/root/.openclaw/workspace/skills/mem-redis/scripts/save_mem.py` — Added `--silent` flag
|
||||
5. `/root/.openclaw/workspace/HEARTBEAT.md` — Updated to use `--silent`
|
||||
6. `/root/.openclaw/workspace/.projects/true-recall-v2/tr-daily/curate_from_qdrant.py` — TARGET_COLLECTION = `gems_tr`
|
||||
7. `/root/.openclaw/workspace/SOUL.md` — Updated collection references
|
||||
8. `/root/.openclaw/workspace/kb/relevant-memories-ui-issue.md` — Updated documentation
|
||||
|
||||
### Migration Script Created
|
||||
|
||||
- `/root/.openclaw/workspace/.projects/true-recall-v2/migrate_memories.py`
|
||||
- Migrated 12,223 points from `kimi_memories` → `memories_tr`
|
||||
- Cleaned noise (metadata, thinking tags) during migration
|
||||
- Preserved original `kimi_memories` as backup
|
||||
|
||||
### Current Collections (Qdrant)
|
||||
|
||||
| Collection | Purpose | Points |
|
||||
|------------|---------|--------|
|
||||
| `memories_tr` | Full text (live capture) | **12,228** |
|
||||
| `gems_tr` | Gems (for injection) | 5 |
|
||||
| `true_recall` | Legacy gems archive | existing |
|
||||
| `kimi_memories` | Original (backup) | 12,223 |
|
||||
|
||||
---
|
||||
|
||||
## Final Status (2026-02-24)
|
||||
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| **Collections** | 4 active (memories_tr, gems_tr, true_recall, kimi_memories) |
|
||||
| **Curator v2** | ✅ Tested & working - 327 turns → 5 gems |
|
||||
| **Config** | ✅ Using `gems_tr` for injection |
|
||||
| **Cron** | ✅ Simplified - only essential jobs |
|
||||
| **Redis** | ✅ Only for notifications (not memory) |
|
||||
|
||||
### Collection Summary
|
||||
|
||||
| Collection | Points | Purpose | Status |
|
||||
|------------|--------|---------|--------|
|
||||
| `memories_tr` | **12,228** | Full text (live capture) | ✅ Active |
|
||||
| `gems_tr` | 5 | Gems (curated) | ✅ Active |
|
||||
| `true_recall` | existing | Legacy archive | 📦 Preserved |
|
||||
| `kimi_memories` | 12,223 | Original backup | 📦 Preserved |
|
||||
|
||||
### Cron Schedule (Cleaned)
|
||||
|
||||
| Time | Job |
|
||||
|------|-----|
|
||||
| 2:45 AM | Curator v2: memories_tr → gems_tr |
|
||||
| 2:20 AM | File backup |
|
||||
| 2:00 AM | Log monitoring |
|
||||
|
||||
**Removed:** 2:15 AM Redis backup, 2:50 AM archive, 5-min cron_capture (all redundant)
|
||||
|
||||
---
|
||||
|
||||
## What We Did Today
|
||||
|
||||
### Issue: `<relevant-memories>` showing in UI
|
||||
|
||||
**Problem:** The `<relevant-memories>` block was showing full conversation text in webchat UI. It should be hidden/internal.
|
||||
|
||||
**Investigation:**
|
||||
1. Initially thought it was `save_mem.py` output (wrong)
|
||||
2. Then thought it was HTML comment issue (wrong)
|
||||
3. Found root cause: memory-qdrant plugin was recalling from `memories_tr` (full text) instead of `gems_tr` (gems)
|
||||
|
||||
**Solution Applied:**
|
||||
Changed `openclaw.json` config:
|
||||
- Before: `"collectionName": "memories_tr"`
|
||||
- After: `"collectionName": "gems_tr"`
|
||||
|
||||
**Result:** ✅ Fixed! Now injection uses gems from true_recall, not full text.
|
||||
|
||||
### Issue: Auto-Capture Architecture Change
|
||||
|
||||
**Problem:** Real-time capture was going through Redis, we needed direct Qdrant storage
|
||||
|
||||
**Solution:**
|
||||
1. Created `realtime_qdrant_watcher.py` — watches session JSONL, embeds, stores directly to Qdrant
|
||||
2. Created `mem-qdrant-watcher.service` — systemd service for the watcher
|
||||
3. Deployed and verified:
|
||||
- Points before: 12,223
|
||||
- Points after test messages: 12,228
|
||||
- Verified new captures have correct structure (role, content, date, source)
|
||||
|
||||
**Status:** ✅ **Deployed and working**
|
||||
|
||||
### Other Changes Made
|
||||
|
||||
1. **Plugin dual collection support** — Added `captureCollection` option
|
||||
2. **Session/Readme updates** — Clarified architecture
|
||||
|
||||
### Files Modified
|
||||
|
||||
| File | Change |
|
||||
|------|--------|
|
||||
| `/root/.openclaw/openclaw.json` | collectionName: memories_tr → gems_tr, added captureCollection |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/config.ts` | Added captureCollection option |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/index.ts` | Dual collection support, debug logging |
|
||||
| `/root/.openclaw/extensions/memory-qdrant/openclaw.plugin.json` | Added captureCollection to schema |
|
||||
|
||||
### Files Created (NEW)
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `/root/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` | Real-time watcher daemon (Qdrant direct) |
|
||||
| `/root/.openclaw/workspace/skills/qdrant-memory/mem-qdrant-watcher.service` | Systemd service file |
|
||||
|
||||
### Deployment
|
||||
|
||||
| Service | Action | Status |
|
||||
|---------|--------|--------|
|
||||
| `mem-redis-watcher` | Stopped | ✅ |
|
||||
| `mem-qdrant-watcher` | Started | ✅ Active |
|
||||
|
||||
### Backups Created
|
||||
|
||||
- `/root/.openclaw/openclaw.json.bak.2026-02-24`
|
||||
- `/root/.openclaw/extensions/memory-qdrant/index.ts.bak.2026-02-24`
|
||||
|
||||
---
|
||||
|
||||
## Current Status
|
||||
|
||||
| Component | Status |
|
||||
|-----------|--------|
|
||||
| **Curation (daily)** | v1 cron at 2:45 AM |
|
||||
| **Injection** | ✅ Working, uses gems_tr |
|
||||
| **Collection** | gems_tr |
|
||||
| **KB** | Updated |
|
||||
|
||||
---
|
||||
|
||||
## What Still Needs Doing
|
||||
|
||||
1. ~~Test autoCapture (cleaned content to memories_tr)~~ ✅ Done
|
||||
2. Test v2 curator (read from Qdrant, not Redis) — Next step
|
||||
3. Full validation 2x
|
||||
|
||||
---
|
||||
|
||||
## Session Recovery
|
||||
|
||||
If starting new session:
|
||||
1. Read this session.md
|
||||
2. Read README.md for architecture
|
||||
3. Read KB for issue history
|
||||
|
||||
---
|
||||
|
||||
**Next:** Test v2 curator (reads from memories_tr, creates gems in gems_tr)
|
||||
150
session.md.neuralstream.bak
Normal file
150
session.md.neuralstream.bak
Normal file
@@ -0,0 +1,150 @@
|
||||
# NeuralStream Session State
|
||||
|
||||
**Date:** 2026-02-23
|
||||
**Status:** Architecture v2.2 - Context-aware hybrid triggers
|
||||
**Alias:** ns
|
||||
|
||||
---
|
||||
|
||||
## Architecture v2.2 (Current)
|
||||
|
||||
**Decision:** Three hybrid extraction triggers with full context awareness
|
||||
|
||||
| Trigger | When | Purpose |
|
||||
|---------|------|---------|
|
||||
| `turn_end` (N=5) | Every 5 turns | Normal batch extraction |
|
||||
| Timer (15 min idle) | No new turn for 15 min | Catch partial batches |
|
||||
| Context (50% threshold) | `ctx.getContextUsage().percent >= threshold` | Proactive pre-compaction |
|
||||
|
||||
**Context Awareness:**
|
||||
- qwen3 gets **up to 256k tokens** of full conversation history for understanding
|
||||
- Only extracts **last N turns** (oldest in batch) to avoid gemming current context
|
||||
- Uses `ctx.getContextUsage()` native API for token monitoring
|
||||
|
||||
**Why Hybrid:**
|
||||
- Batch extraction = better quality gems (more context)
|
||||
- Timer safety = never lose important turns if user walks away
|
||||
- Context trigger = proactive extraction before system forces compaction
|
||||
- All gems survive `/new` and `/reset` via Qdrant
|
||||
|
||||
**Infrastructure:** Reuse existing Redis/Qdrant — NeuralStream is the "middle layer" only
|
||||
|
||||
---
|
||||
|
||||
## Core Insight
|
||||
|
||||
NeuralStream enables **infinite effective context** — active window stays small, but semantically relevant gems from all past conversations are queryable and injectable.
|
||||
|
||||
---
|
||||
|
||||
## Technical Decisions 2026-02-23
|
||||
|
||||
### Triggers (Three-way Hybrid)
|
||||
| Trigger | Config | Default |
|
||||
|---------|--------|---------|
|
||||
| Batch size | `batch_size` | 5 turns |
|
||||
| Idle timeout | `idle_timeout` | 15 minutes |
|
||||
| Context threshold | `context_threshold` | 50% |
|
||||
|
||||
### Context Monitoring (Native API)
|
||||
- `ctx.getContextUsage()` → `{tokens, contextWindow, percent}`
|
||||
- Checked in `turn_end` hook
|
||||
- Triggers extraction when `percent >= context_threshold`
|
||||
|
||||
### Extraction Context Window
|
||||
- **Feed to qwen3:** Up to 256k tokens (full history for understanding)
|
||||
- **Extract from:** Last `batch_size` turns only
|
||||
- **Benefit:** Rich context awareness without gemming current conversation
|
||||
|
||||
### Storage
|
||||
- **Buffer:** Redis (`neuralstream:buffer` key)
|
||||
- **Gems:** Qdrant `neuralstream` collection (1024 dims, Cosine)
|
||||
- **Existing infra:** Reuse mem-redis-watcher + qdrant-memory
|
||||
|
||||
### Gem Format (Proposed)
|
||||
```json
|
||||
{
|
||||
"gem_id": "uuid",
|
||||
"content": "Distilled insight/fact/decision",
|
||||
"summary": "One-line for quick scanning",
|
||||
"topics": ["docker", "redis", "architecture"],
|
||||
"importance": 0.9,
|
||||
"source": {
|
||||
"session_id": "uuid",
|
||||
"date": "2026-02-23",
|
||||
"turn_range": "15-20"
|
||||
},
|
||||
"tags": ["decision", "fact", "preference", "todo", "code"],
|
||||
"created_at": "2026-02-23T15:26:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Extraction Model
|
||||
- **qwen3** for gem extraction (256k context, cheap)
|
||||
- **Dedicated prompt** (to be designed) for extracting high-value items
|
||||
|
||||
---
|
||||
|
||||
## Architecture Layers
|
||||
|
||||
| Layer | Status | Description |
|
||||
|-------|--------|-------------|
|
||||
| Capture | ✅ Existing | Every turn → Redis (mem-redis-watcher) |
|
||||
| **Extract** | ⏳ NeuralStream | Batch → qwen3 → gems → Qdrant |
|
||||
| Retrieve | ✅ Existing | Semantic search → inject context |
|
||||
|
||||
NeuralStream = Smart extraction layer on top of existing infra.
|
||||
|
||||
---
|
||||
|
||||
## Open Questions
|
||||
|
||||
- Gem extraction prompt design (deferred)
|
||||
- Importance scoring: auto vs manual?
|
||||
- Injection: `turn_start` hook or modify system prompt?
|
||||
- Semantic search threshold tuning
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
| Task | Status |
|
||||
|------|--------|
|
||||
| Architecture v2.2 finalized | ✅ |
|
||||
| Native context monitoring validated | ✅ |
|
||||
| Gem JSON schema | ✅ Proposed |
|
||||
| Implement turn_end hook | ⏳ |
|
||||
| Implement timer/cron check | ⏳ |
|
||||
| Implement context trigger | ⏳ |
|
||||
| Create extraction prompt | ⏳ |
|
||||
| Test gem extraction with qwen3 | ⏳ |
|
||||
| Implement injection mechanism | ⏳ |
|
||||
|
||||
---
|
||||
|
||||
## Decisions Log
|
||||
|
||||
| Date | Decision |
|
||||
|------|----------|
|
||||
| 2026-02-23 | Switch to turn_end hook (v2) |
|
||||
| 2026-02-23 | Hybrid triggers with timer (v2.1) |
|
||||
| 2026-02-23 | Context-aware extraction (v2.2) |
|
||||
| 2026-02-23 | Native API: ctx.getContextUsage() |
|
||||
| 2026-02-23 | Full context feed to qwen3 (256k) |
|
||||
| 2026-02-23 | Reuse existing Redis/Qdrant infrastructure |
|
||||
| 2026-02-23 | Batch N=5 turns |
|
||||
| 2026-02-23 | Context threshold = 50% |
|
||||
| 2026-02-23 | Inactivity timer = 15 min |
|
||||
| 2026-02-23 | Dedicated qwen3 extraction prompt (deferred) |
|
||||
|
||||
---
|
||||
|
||||
## Backups
|
||||
|
||||
- Local: `/root/.openclaw/workspace/.projects/neuralstream/`
|
||||
- Remote: `deb2:/root/.projects/neuralstream/` (build/test only)
|
||||
- kimi_kb: Research entries stored
|
||||
|
||||
---
|
||||
|
||||
**Key Insight:** Session resets wipe context but NOT Qdrant. NeuralStream = "Context insurance policy" for infinite LLM memory.
|
||||
64
test_curator.py
Normal file
64
test_curator.py
Normal file
@@ -0,0 +1,64 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Quick test of curator with simple input"""
|
||||
import json
|
||||
import requests
|
||||
|
||||
# Load prompt from v1
|
||||
with open('/root/.openclaw/workspace/.projects/true-recall-v1/curator-prompt.md') as f:
|
||||
prompt = f.read()
|
||||
|
||||
# Test with a simple conversation
|
||||
test_turns = [
|
||||
{
|
||||
'turn': 1,
|
||||
'user_id': 'rob',
|
||||
'user': 'I want to switch from Redis to Qdrant for memory storage',
|
||||
'ai': 'Got it - Qdrant is a good choice for vector storage.',
|
||||
'conversation_id': 'test123',
|
||||
'timestamp': '2026-02-23T10:00:00',
|
||||
'date': '2026-02-23'
|
||||
},
|
||||
{
|
||||
'turn': 2,
|
||||
'user_id': 'rob',
|
||||
'user': 'Yes, and I want the curator to read from Qdrant directly',
|
||||
'ai': 'Makes sense - we can modify the curator to query Qdrant instead of Redis.',
|
||||
'conversation_id': 'test123',
|
||||
'timestamp': '2026-02-23T10:01:00',
|
||||
'date': '2026-02-23'
|
||||
}
|
||||
]
|
||||
|
||||
conversation_json = json.dumps(test_turns, indent=2)
|
||||
|
||||
prompt_text = f"""## Input Conversation
|
||||
|
||||
```json
|
||||
{conversation_json}
|
||||
```
|
||||
|
||||
## Output
|
||||
"""
|
||||
|
||||
response = requests.post(
|
||||
'http://10.0.0.10:11434/api/generate',
|
||||
json={
|
||||
'model': 'qwen3:4b-instruct',
|
||||
'system': prompt,
|
||||
'prompt': prompt_text,
|
||||
'stream': False,
|
||||
'options': {'temperature': 0.1, 'num_predict': 2000}
|
||||
},
|
||||
timeout=120
|
||||
)
|
||||
|
||||
result = response.json()
|
||||
output = result.get('response', '').strip()
|
||||
print('=== RAW OUTPUT ===')
|
||||
print(output[:2000])
|
||||
print()
|
||||
print('=== PARSED ===')
|
||||
# Try to extract JSON
|
||||
if '```json' in output:
|
||||
parsed = output.split('```json')[1].split('```')[0].strip()
|
||||
print(parsed)
|
||||
105
tr-compact/hook.py
Normal file
105
tr-compact/hook.py
Normal file
@@ -0,0 +1,105 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
TrueRecall v2 - Compaction Hook
|
||||
Fast Redis queue push for compaction events
|
||||
|
||||
Called by OpenClaw session_before_compact hook
|
||||
"""
|
||||
|
||||
import json
|
||||
import sys
|
||||
import redis
|
||||
from datetime import datetime
|
||||
from typing import List, Dict, Any
|
||||
|
||||
# Redis config
|
||||
REDIS_HOST = "10.0.0.36"
|
||||
REDIS_PORT = 6379
|
||||
REDIS_DB = 0
|
||||
QUEUE_KEY = "tr:compact_queue"
|
||||
TAG_PREFIX = "tr:processed"
|
||||
|
||||
def get_redis_client():
|
||||
return redis.Redis(
|
||||
host=REDIS_HOST,
|
||||
port=REDIS_PORT,
|
||||
db=REDIS_DB,
|
||||
decode_responses=True
|
||||
)
|
||||
|
||||
def tag_turns(messages: List[Dict], user_id: str = "rob"):
|
||||
"""Tag turns so v1 daily curator skips them"""
|
||||
r = get_redis_client()
|
||||
pipe = r.pipeline()
|
||||
|
||||
for msg in messages:
|
||||
conv_id = msg.get("conversation_id", "unknown")
|
||||
turn = msg.get("turn", 0)
|
||||
tag_key = f"{TAG_PREFIX}:{conv_id}:{turn}"
|
||||
pipe.setex(tag_key, 86400, "1") # 24h TTL
|
||||
|
||||
pipe.execute()
|
||||
|
||||
def queue_messages(messages: List[Dict], user_id: str = "rob"):
|
||||
"""Push messages to Redis queue for background processing"""
|
||||
r = get_redis_client()
|
||||
|
||||
queue_item = {
|
||||
"user_id": user_id,
|
||||
"timestamp": datetime.now().isoformat(),
|
||||
"message_count": len(messages),
|
||||
"messages": messages
|
||||
}
|
||||
|
||||
# LPUSH to queue (newest first)
|
||||
r.lpush(QUEUE_KEY, json.dumps(queue_item))
|
||||
|
||||
return len(messages)
|
||||
|
||||
def process_compaction_event(event_data: Dict):
|
||||
"""
|
||||
Process session_before_compact event from OpenClaw
|
||||
|
||||
Expected event_data:
|
||||
{
|
||||
"session_id": "uuid",
|
||||
"user_id": "rob",
|
||||
"messages_being_compacted": [
|
||||
{"role": "user", "content": "...", "turn": 1, "conversation_id": "..."},
|
||||
...
|
||||
],
|
||||
"compaction_reason": "context_limit"
|
||||
}
|
||||
"""
|
||||
user_id = event_data.get("user_id", "rob")
|
||||
messages = event_data.get("messages_being_compacted", [])
|
||||
|
||||
if not messages:
|
||||
return {"status": "ok", "queued": 0, "reason": "no_messages"}
|
||||
|
||||
# Tag turns for v1 coordination
|
||||
tag_turns(messages, user_id)
|
||||
|
||||
# Queue for background processing
|
||||
count = queue_messages(messages, user_id)
|
||||
|
||||
return {
|
||||
"status": "ok",
|
||||
"queued": count,
|
||||
"user_id": user_id,
|
||||
"queue_key": QUEUE_KEY
|
||||
}
|
||||
|
||||
def main():
|
||||
"""CLI entry point - reads JSON from stdin"""
|
||||
try:
|
||||
event_data = json.load(sys.stdin)
|
||||
result = process_compaction_event(event_data)
|
||||
print(json.dumps(result))
|
||||
sys.exit(0)
|
||||
except Exception as e:
|
||||
print(json.dumps({"status": "error", "error": str(e)}))
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
101
tr-continuous/README.md
Normal file
101
tr-continuous/README.md
Normal file
@@ -0,0 +1,101 @@
|
||||
# Turn-Based Curator
|
||||
|
||||
Extract gems every N turns instead of waiting for daily curation.
|
||||
|
||||
## Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `curator_turn_based.py` | Main script - checks turn count, extracts gems |
|
||||
| `curator_cron.sh` | Cron wrapper to run every minute |
|
||||
| `turn-curator.service` | Alternative systemd service (runs on-demand) |
|
||||
|
||||
## Usage
|
||||
|
||||
### Manual Run
|
||||
|
||||
```bash
|
||||
# Check current status
|
||||
python3 curator_turn_based.py --status
|
||||
|
||||
# Preview what would be curated
|
||||
python3 curator_turn_based.py --threshold 10 --dry-run
|
||||
|
||||
# Execute curation
|
||||
python3 curator_turn_based.py --threshold 10 --execute
|
||||
```
|
||||
|
||||
### Automatic (Cron)
|
||||
|
||||
Add to crontab:
|
||||
```bash
|
||||
* * * * * /root/.openclaw/workspace/.projects/true-recall-v2/tr-continuous/curator_cron.sh
|
||||
```
|
||||
|
||||
Or use systemd timer:
|
||||
```bash
|
||||
sudo cp turn-curator.service /etc/systemd/system/
|
||||
sudo systemctl enable turn-curator.timer # If you create a timer
|
||||
```
|
||||
|
||||
### Automatic (Integrated)
|
||||
|
||||
Alternative: Modify `realtime_qdrant_watcher.py` to trigger curation every 10 turns.
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **Tracks turn count** - Stores last curation turn in `/tmp/curator_turn_state.json`
|
||||
2. **Monitors delta** - Compares current turn count vs last curation
|
||||
3. **Triggers at threshold** - When 10+ new turns exist, runs curation
|
||||
4. **Extracts gems** - Sends conversation to qwen3, gets gems
|
||||
5. **Stores results** - Saves gems to `gems_tr` collection
|
||||
|
||||
## State File
|
||||
|
||||
`/tmp/curator_turn_state.json`:
|
||||
```json
|
||||
{
|
||||
"last_turn": 150,
|
||||
"last_curation": "2026-02-24T17:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
## Comparison with Daily Curator
|
||||
|
||||
| Feature | Daily Curator | Turn-Based Curator |
|
||||
|---------|--------------|-------------------|
|
||||
| Schedule | 2:45 AM daily | Every 10 turns (dynamic) |
|
||||
| Time window | 24 hours | Variable (depends on chat frequency) |
|
||||
| Trigger | Cron | Turn threshold |
|
||||
| Use case | Nightly batch | Real-time-ish extraction |
|
||||
| Overlap | Low | Possible with daily curator |
|
||||
|
||||
## Recommendation
|
||||
|
||||
Use **BOTH**:
|
||||
- **Turn-based**: Every 10 turns for active conversations
|
||||
- **Daily**: 2:45 AM as backup/catch-all
|
||||
|
||||
They'll deduplicate automatically (same embeddings → skipped).
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Simulate 10 turns
|
||||
for i in {1..10}; do
|
||||
echo "Test message $i" > /dev/null
|
||||
done
|
||||
|
||||
# Check status
|
||||
python3 curator_turn_based.py --status
|
||||
|
||||
# Run manually
|
||||
python3 curator_turn_based.py --threshold 10 --execute
|
||||
```
|
||||
|
||||
## Status
|
||||
|
||||
- ✅ Script created: `curator_turn_based.py`
|
||||
- ✅ Cron wrapper: `curator_cron.sh`
|
||||
- ⏳ Deployment: Optional (manual or cron)
|
||||
- ⏳ Testing: Pending
|
||||
194
tr-continuous/curator_by_count.py
Normal file
194
tr-continuous/curator_by_count.py
Normal file
@@ -0,0 +1,194 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Turn-Based Curator: Extract gems every N new memories (turns).
|
||||
|
||||
Usage:
|
||||
python3 curator_by_count.py --threshold 10 --dry-run
|
||||
python3 curator_by_count.py --threshold 10 --execute
|
||||
python3 curator_by_count.py --status
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import requests
|
||||
import sys
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from pathlib import Path
|
||||
|
||||
QDRANT_URL = "http://10.0.0.40:6333"
|
||||
MEMORIES = "memories_tr"
|
||||
GEMS = "gems_tr"
|
||||
OLLAMA = "http://10.0.0.10:11434"
|
||||
MODEL = "ollama-remote/qwen3:30b-a3b-instruct-2507-q8_0"
|
||||
STATE_FILE = Path("/tmp/curator_count_state.json")
|
||||
|
||||
def load_state():
|
||||
if STATE_FILE.exists():
|
||||
with open(STATE_FILE) as f:
|
||||
return json.load(f)
|
||||
return {"last_count": 0, "last_time": None}
|
||||
|
||||
def save_state(state):
|
||||
with open(STATE_FILE, 'w') as f:
|
||||
json.dump(state, f)
|
||||
|
||||
def get_total_count():
|
||||
try:
|
||||
r = requests.get(f"{QDRANT_URL}/collections/{MEMORIES}", timeout=10)
|
||||
return r.json()["result"]["points_count"]
|
||||
except:
|
||||
return 0
|
||||
|
||||
def get_recent_memories(hours=1):
|
||||
"""Get memories from last N hours."""
|
||||
since = (datetime.now(timezone.utc) - timedelta(hours=hours)).isoformat()
|
||||
try:
|
||||
r = requests.post(
|
||||
f"{QDRANT_URL}/collections/{MEMORIES}/points/scroll",
|
||||
json={"limit": 1000, "with_payload": True},
|
||||
timeout=30
|
||||
)
|
||||
points = r.json()["result"]["points"]
|
||||
# Filter by timestamp
|
||||
recent = [p for p in points if p.get("payload", {}).get("timestamp", "") > since]
|
||||
return recent
|
||||
except:
|
||||
return []
|
||||
|
||||
def extract_gems(memories):
|
||||
"""Send to LLM for gem extraction."""
|
||||
if not memories:
|
||||
return []
|
||||
|
||||
# Build conversation
|
||||
parts = []
|
||||
for m in memories:
|
||||
role = m["payload"].get("role", "unknown")
|
||||
content = m["payload"].get("content", "")[:500] # Limit per message
|
||||
parts.append(f"{role.upper()}: {content}")
|
||||
|
||||
conversation = "\n\n".join(parts[:20]) # Max 20 messages
|
||||
|
||||
prompt = f"""Extract 3-5 key gems (insights, decisions, facts) from this conversation.
|
||||
|
||||
Conversation:
|
||||
{conversation}
|
||||
|
||||
Return JSON: [{{"text": "gem", "category": "decision|fact|preference"}}]"""
|
||||
|
||||
try:
|
||||
r = requests.post(
|
||||
f"{OLLAMA}/v1/chat/completions",
|
||||
json={
|
||||
"model": MODEL,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"temperature": 0.3
|
||||
},
|
||||
timeout=120
|
||||
)
|
||||
content = r.json()["choices"][0]["message"]["content"]
|
||||
|
||||
# Parse JSON
|
||||
start = content.find('[')
|
||||
end = content.rfind(']')
|
||||
if start >= 0 and end > start:
|
||||
return json.loads(content[start:end+1])
|
||||
except:
|
||||
pass
|
||||
return []
|
||||
|
||||
def store_gem(gem):
|
||||
"""Store gem to gems_tr."""
|
||||
try:
|
||||
# Get embedding
|
||||
r = requests.post(
|
||||
f"{OLLAMA}/api/embeddings",
|
||||
json={"model": "snowflake-arctic-embed2", "prompt": gem["text"]},
|
||||
timeout=30
|
||||
)
|
||||
vector = r.json()["embedding"]
|
||||
|
||||
# Store
|
||||
r = requests.put(
|
||||
f"{QDRANT_URL}/collections/{GEMS}/points",
|
||||
json={
|
||||
"points": [{
|
||||
"id": abs(hash(gem["text"])) % (2**63),
|
||||
"vector": vector,
|
||||
"payload": {
|
||||
"text": gem["text"],
|
||||
"category": gem.get("category", "other"),
|
||||
"createdAt": datetime.now(timezone.utc).isoformat(),
|
||||
"source": "turn_curator"
|
||||
}
|
||||
}]
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
return r.status_code == 200
|
||||
except:
|
||||
return False
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--threshold", "-t", type=int, default=10)
|
||||
parser.add_argument("--execute", "-e", action="store_true")
|
||||
parser.add_argument("--dry-run", "-n", action="store_true")
|
||||
parser.add_argument("--status", "-s", action="store_true")
|
||||
args = parser.parse_args()
|
||||
|
||||
state = load_state()
|
||||
current = get_total_count()
|
||||
new_points = current - state.get("last_count", 0)
|
||||
|
||||
if args.status:
|
||||
print(f"Total memories: {current}")
|
||||
print(f"Last curated: {state.get('last_count', 0)}")
|
||||
print(f"New since last: {new_points}")
|
||||
print(f"Threshold: {args.threshold}")
|
||||
print(f"Ready: {'YES' if new_points >= args.threshold else 'NO'}")
|
||||
return
|
||||
|
||||
print(f"Curator: {new_points} new / {args.threshold} threshold")
|
||||
|
||||
if new_points < args.threshold:
|
||||
print("Not enough new memories")
|
||||
return
|
||||
|
||||
# Get recent memories (last hour should cover the new points)
|
||||
memories = get_recent_memories(hours=1)
|
||||
print(f"Fetched {len(memories)} recent memories")
|
||||
|
||||
if not memories:
|
||||
print("No memories to process")
|
||||
return
|
||||
|
||||
if args.dry_run:
|
||||
print(f"[DRY RUN] Would process {len(memories)} memories")
|
||||
return
|
||||
|
||||
if not args.execute:
|
||||
print("Use --execute to run or --dry-run to preview")
|
||||
return
|
||||
|
||||
# Extract gems
|
||||
print("Extracting gems...")
|
||||
gems = extract_gems(memories)
|
||||
print(f"Extracted {len(gems)} gems")
|
||||
|
||||
# Store
|
||||
success = 0
|
||||
for gem in gems:
|
||||
if store_gem(gem):
|
||||
success += 1
|
||||
print(f" Stored: {gem['text'][:60]}...")
|
||||
|
||||
# Update state
|
||||
state["last_count"] = current
|
||||
state["last_time"] = datetime.now(timezone.utc).isoformat()
|
||||
save_state(state)
|
||||
|
||||
print(f"Done: {success}/{len(gems)} gems stored")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
7
tr-continuous/curator_config.json
Normal file
7
tr-continuous/curator_config.json
Normal file
@@ -0,0 +1,7 @@
|
||||
{
|
||||
"timer_minutes": 5,
|
||||
"max_batch_size": 100,
|
||||
"user_id": "rob",
|
||||
"source_collection": "memories_tr",
|
||||
"target_collection": "gems_tr"
|
||||
}
|
||||
12
tr-continuous/curator_cron.sh
Normal file
12
tr-continuous/curator_cron.sh
Normal file
@@ -0,0 +1,12 @@
|
||||
#!/bin/bash
|
||||
# Turn-based curator cron - runs every minute to check if 10 turns reached
|
||||
|
||||
SCRIPT_DIR="/root/.openclaw/workspace/.projects/true-recall-v2/tr-continuous"
|
||||
|
||||
# Check if enough turns accumulated
|
||||
/usr/bin/python3 "${SCRIPT_DIR}/curator_turn_based.py" --threshold 10 --status 2>/dev/null | grep -q "Ready to curate: YES"
|
||||
|
||||
if [ $? -eq 0 ]; then
|
||||
# Run curation
|
||||
/usr/bin/python3 "${SCRIPT_DIR}/curator_turn_based.py" --threshold 10 --execute 2>&1 | logger -t turn-curator
|
||||
fi
|
||||
350
tr-continuous/curator_timer.py
Executable file
350
tr-continuous/curator_timer.py
Executable file
@@ -0,0 +1,350 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
TrueRecall Timer Curator: Runs every 30 minutes via cron.
|
||||
|
||||
- Queries all uncurated memories from memories_tr
|
||||
- Sends batch to qwen3 for gem extraction
|
||||
- Stores gems to gems_tr
|
||||
- Marks processed memories as curated=true
|
||||
|
||||
Usage:
|
||||
python3 curator_timer.py --config curator_config.json
|
||||
python3 curator_timer.py --config curator_config.json --dry-run
|
||||
"""
|
||||
|
||||
import os
|
||||
import sys
|
||||
import json
|
||||
import argparse
|
||||
import requests
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Any, Optional
|
||||
import hashlib
|
||||
|
||||
# Load config
|
||||
def load_config(config_path: str) -> Dict[str, Any]:
|
||||
with open(config_path, 'r') as f:
|
||||
return json.load(f)
|
||||
|
||||
# Default paths
|
||||
SCRIPT_DIR = Path(__file__).parent
|
||||
DEFAULT_CONFIG = SCRIPT_DIR / "curator_config.json"
|
||||
|
||||
# Curator prompt path
|
||||
CURATOR_PROMPT_PATH = Path("/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md")
|
||||
|
||||
|
||||
def load_curator_prompt() -> str:
|
||||
"""Load the curator system prompt."""
|
||||
try:
|
||||
with open(CURATOR_PROMPT_PATH, 'r') as f:
|
||||
return f.read()
|
||||
except FileNotFoundError:
|
||||
print(f"⚠️ Curator prompt not found at {CURATOR_PROMPT_PATH}")
|
||||
return """You are The Curator. Extract meaningful gems from conversation history.
|
||||
Extract facts, insights, decisions, preferences, and context that would be valuable to remember.
|
||||
Output a JSON array of gems with fields: gem, context, snippet, categories, importance (1-5), confidence (0-0.99)."""
|
||||
|
||||
|
||||
def get_uncurated_memories(qdrant_url: str, collection: str, user_id: str, max_batch: int) -> List[Dict[str, Any]]:
|
||||
"""Query Qdrant for uncurated memories."""
|
||||
filter_data = {
|
||||
"must": [
|
||||
{"key": "user_id", "match": {"value": user_id}},
|
||||
{"key": "curated", "match": {"value": False}}
|
||||
]
|
||||
}
|
||||
|
||||
all_points = []
|
||||
offset = None
|
||||
iterations = 0
|
||||
max_iterations = 10
|
||||
|
||||
while len(all_points) < max_batch and iterations < max_iterations:
|
||||
iterations += 1
|
||||
scroll_data = {
|
||||
"limit": min(100, max_batch - len(all_points)),
|
||||
"with_payload": True,
|
||||
"filter": filter_data
|
||||
}
|
||||
|
||||
if offset:
|
||||
scroll_data["offset"] = offset
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{qdrant_url}/collections/{collection}/points/scroll",
|
||||
json=scroll_data,
|
||||
headers={"Content-Type": "application/json"},
|
||||
timeout=30
|
||||
)
|
||||
response.raise_for_status()
|
||||
result = response.json()
|
||||
points = result.get("result", {}).get("points", [])
|
||||
|
||||
if not points:
|
||||
break
|
||||
|
||||
all_points.extend(points)
|
||||
offset = result.get("result", {}).get("next_page_offset")
|
||||
if not offset:
|
||||
break
|
||||
except Exception as e:
|
||||
print(f"Error querying Qdrant: {e}", file=sys.stderr)
|
||||
break
|
||||
|
||||
# Convert to simple dicts
|
||||
memories = []
|
||||
for point in all_points:
|
||||
payload = point.get("payload", {})
|
||||
memories.append({
|
||||
"id": point.get("id"),
|
||||
"content": payload.get("content", ""),
|
||||
"role": payload.get("role", ""),
|
||||
"timestamp": payload.get("timestamp", ""),
|
||||
"turn": payload.get("turn", 0),
|
||||
**payload
|
||||
})
|
||||
|
||||
return memories[:max_batch]
|
||||
|
||||
|
||||
def extract_gems(memories: List[Dict[str, Any]], ollama_url: str) -> List[Dict[str, Any]]:
|
||||
"""Send memories to qwen3 for gem extraction."""
|
||||
if not memories:
|
||||
return []
|
||||
|
||||
prompt = load_curator_prompt()
|
||||
|
||||
# Build conversation from memories
|
||||
conversation_lines = []
|
||||
for mem in memories:
|
||||
role = mem.get("role", "unknown")
|
||||
content = mem.get("content", "")
|
||||
if content:
|
||||
conversation_lines.append(f"{role}: {content}")
|
||||
|
||||
conversation_text = "\n".join(conversation_lines)
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{ollama_url}/api/generate",
|
||||
json={
|
||||
"model": "qwen3:30b-a3b-instruct-2507-q8_0",
|
||||
"system": prompt,
|
||||
"prompt": f"## Input Conversation\n\n{conversation_text}\n\n## Output\n",
|
||||
"stream": False,
|
||||
"options": {
|
||||
"temperature": 0.1,
|
||||
"num_predict": 4000
|
||||
}
|
||||
},
|
||||
timeout=120
|
||||
)
|
||||
response.raise_for_status()
|
||||
except Exception as e:
|
||||
print(f"Error calling Ollama: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
result = response.json()
|
||||
output = result.get('response', '').strip()
|
||||
|
||||
# Extract JSON from output
|
||||
if '```json' in output:
|
||||
output = output.split('```json')[1].split('```')[0].strip()
|
||||
elif '```' in output:
|
||||
output = output.split('```')[1].split('```')[0].strip()
|
||||
|
||||
try:
|
||||
start_idx = output.find('[')
|
||||
end_idx = output.rfind(']')
|
||||
if start_idx != -1 and end_idx != -1 and end_idx > start_idx:
|
||||
output = output[start_idx:end_idx+1]
|
||||
|
||||
# Fix common JSON issues from LLM output
|
||||
# Replace problematic escape sequences
|
||||
output = output.replace('\\n', '\n').replace('\\t', '\t')
|
||||
# Fix single quotes in content that break JSON
|
||||
output = output.replace("\\'", "'")
|
||||
|
||||
gems = json.loads(output)
|
||||
if not isinstance(gems, list):
|
||||
gems = [gems] if gems else []
|
||||
return gems
|
||||
except json.JSONDecodeError as e:
|
||||
# Try to extract gems with regex fallback
|
||||
import re
|
||||
gem_matches = re.findall(r'"gem"\s*:\s*"([^"]+)"', output)
|
||||
if gem_matches:
|
||||
gems = []
|
||||
for gem_text in gem_matches:
|
||||
gems.append({
|
||||
"gem": gem_text,
|
||||
"context": "Extracted via fallback",
|
||||
"categories": ["extracted"],
|
||||
"importance": 3,
|
||||
"confidence": 0.7
|
||||
})
|
||||
print(f"⚠️ Fallback extraction: {len(gems)} gems", file=sys.stderr)
|
||||
return gems
|
||||
print(f"Error parsing curator output: {e}", file=sys.stderr)
|
||||
print(f"Raw output: {repr(output[:500])}...", file=sys.stderr)
|
||||
return []
|
||||
|
||||
|
||||
def get_embedding(text: str, ollama_url: str) -> Optional[List[float]]:
|
||||
"""Get embedding from Ollama."""
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{ollama_url}/api/embeddings",
|
||||
json={"model": "mxbai-embed-large", "prompt": text},
|
||||
timeout=30
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json()['embedding']
|
||||
except Exception as e:
|
||||
print(f"Error getting embedding: {e}", file=sys.stderr)
|
||||
return None
|
||||
|
||||
|
||||
def store_gem(gem: Dict[str, Any], user_id: str, qdrant_url: str, target_collection: str, ollama_url: str) -> bool:
|
||||
"""Store a single gem to Qdrant."""
|
||||
embedding_text = f"{gem.get('gem', '')} {gem.get('context', '')} {gem.get('snippet', '')}"
|
||||
vector = get_embedding(embedding_text, ollama_url)
|
||||
|
||||
if vector is None:
|
||||
return False
|
||||
|
||||
# Generate ID
|
||||
hash_content = f"{user_id}:{gem.get('conversation_id', '')}:{gem.get('turn_range', '')}:{gem.get('gem', '')[:50]}"
|
||||
hash_bytes = hashlib.sha256(hash_content.encode()).digest()[:8]
|
||||
gem_id = int.from_bytes(hash_bytes, byteorder='big') % (2**63)
|
||||
|
||||
payload = {
|
||||
"user_id": user_id,
|
||||
**gem,
|
||||
"curated_at": datetime.now(timezone.utc).isoformat()
|
||||
}
|
||||
|
||||
try:
|
||||
response = requests.put(
|
||||
f"{qdrant_url}/collections/{target_collection}/points",
|
||||
json={
|
||||
"points": [{
|
||||
"id": abs(gem_id),
|
||||
"vector": vector,
|
||||
"payload": payload
|
||||
}]
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
response.raise_for_status()
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Error storing gem: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
|
||||
def mark_curated(memory_ids: List, qdrant_url: str, collection: str) -> bool:
|
||||
"""Mark memories as curated in Qdrant using POST /points/payload format."""
|
||||
if not memory_ids:
|
||||
return True
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{qdrant_url}/collections/{collection}/points/payload",
|
||||
json={
|
||||
"points": memory_ids,
|
||||
"payload": {
|
||||
"curated": True,
|
||||
"curated_at": datetime.now(timezone.utc).isoformat()
|
||||
}
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
response.raise_for_status()
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Error marking curated: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="TrueRecall Timer Curator")
|
||||
parser.add_argument("--config", "-c", default=str(DEFAULT_CONFIG), help="Config file path")
|
||||
parser.add_argument("--dry-run", "-n", action="store_true", help="Don't write, just preview")
|
||||
args = parser.parse_args()
|
||||
|
||||
config = load_config(args.config)
|
||||
|
||||
qdrant_url = os.getenv("QDRANT_URL", "http://10.0.0.40:6333")
|
||||
ollama_url = os.getenv("OLLAMA_URL", "http://10.0.0.10:11434")
|
||||
|
||||
user_id = config.get("user_id", "rob")
|
||||
source_collection = config.get("source_collection", "memories_tr")
|
||||
target_collection = config.get("target_collection", "gems_tr")
|
||||
max_batch = config.get("max_batch_size", 100)
|
||||
|
||||
print(f"🔍 TrueRecall Timer Curator")
|
||||
print(f"👤 User: {user_id}")
|
||||
print(f"📥 Source: {source_collection}")
|
||||
print(f"💎 Target: {target_collection}")
|
||||
print(f"📦 Max batch: {max_batch}")
|
||||
if args.dry_run:
|
||||
print("🏃 DRY RUN MODE")
|
||||
print()
|
||||
|
||||
# Get uncurated memories
|
||||
print("📥 Fetching uncurated memories...")
|
||||
memories = get_uncurated_memories(qdrant_url, source_collection, user_id, max_batch)
|
||||
print(f"✅ Found {len(memories)} uncurated memories")
|
||||
|
||||
if not memories:
|
||||
print("🤷 Nothing to curate. Exiting.")
|
||||
return
|
||||
|
||||
# Extract gems
|
||||
print(f"\n🧠 Sending {len(memories)} memories to curator...")
|
||||
gems = extract_gems(memories, ollama_url)
|
||||
print(f"✅ Extracted {len(gems)} gems")
|
||||
|
||||
if not gems:
|
||||
print("⚠️ No gems extracted. Nothing to store.")
|
||||
# Still mark as curated so we don't reprocess
|
||||
memory_ids = [m["id"] for m in memories] # Keep as integers
|
||||
mark_curated(memory_ids, qdrant_url, source_collection)
|
||||
return
|
||||
|
||||
# Preview
|
||||
print("\n💎 Gems preview:")
|
||||
for i, gem in enumerate(gems[:3], 1):
|
||||
print(f" {i}. {gem.get('gem', 'N/A')[:80]}...")
|
||||
if len(gems) > 3:
|
||||
print(f" ... and {len(gems) - 3} more")
|
||||
|
||||
if args.dry_run:
|
||||
print("\n🏃 DRY RUN: Not storing gems or marking curated.")
|
||||
return
|
||||
|
||||
# Store gems
|
||||
print(f"\n💾 Storing {len(gems)} gems...")
|
||||
stored = 0
|
||||
for gem in gems:
|
||||
if store_gem(gem, user_id, qdrant_url, target_collection, ollama_url):
|
||||
stored += 1
|
||||
print(f"✅ Stored: {stored}/{len(gems)}")
|
||||
|
||||
# Mark memories as curated
|
||||
print("\n📝 Marking memories as curated...")
|
||||
memory_ids = [m["id"] for m in memories] # Keep as integers
|
||||
if mark_curated(memory_ids, qdrant_url, source_collection):
|
||||
print(f"✅ Marked {len(memory_ids)} memories as curated")
|
||||
else:
|
||||
print(f"⚠️ Failed to mark some memories as curated")
|
||||
|
||||
print("\n🎉 Curation complete!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
291
tr-continuous/curator_turn_based.py
Normal file
291
tr-continuous/curator_turn_based.py
Normal file
@@ -0,0 +1,291 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Turn-Based Curator: Extract gems every N turns (instead of daily).
|
||||
|
||||
Usage:
|
||||
python3 curator_turn_based.py --threshold 10 --dry-run
|
||||
python3 curator_turn_based.py --threshold 10 --execute
|
||||
python3 curator_turn_based.py --status # Show turn counts
|
||||
|
||||
This tracks turn count since last curation and runs when threshold is reached.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import requests
|
||||
import sys
|
||||
from datetime import datetime, timezone, timedelta
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Any, Optional
|
||||
|
||||
# Config
|
||||
QDRANT_URL = "http://10.0.0.40:6333"
|
||||
MEMORIES_COLLECTION = "memories_tr"
|
||||
GEMS_COLLECTION = "gems_tr"
|
||||
OLLAMA_URL = "http://10.0.0.10:11434"
|
||||
CURATOR_MODEL = "ollama-remote/qwen3:30b-a3b-instruct-2507-q8_0"
|
||||
|
||||
# State file tracks last curation
|
||||
STATE_FILE = Path("/tmp/curator_turn_state.json")
|
||||
|
||||
def get_curator_prompt(conversation_text: str) -> str:
|
||||
"""Generate prompt for gem extraction."""
|
||||
return f"""You are a memory curator. Extract only the most valuable gems (key insights) from this conversation.
|
||||
|
||||
Rules:
|
||||
1. Extract only genuinely important information (decisions, preferences, key facts)
|
||||
2. Skip transient/trivial content (greetings, questions, temporary requests)
|
||||
3. Each gem should be self-contained and useful for future context
|
||||
4. Format: concise, factual statements
|
||||
5. Max 3-5 gems total
|
||||
|
||||
Conversation to curate:
|
||||
---
|
||||
{conversation_text}
|
||||
---
|
||||
|
||||
Return ONLY a JSON array of gems like:
|
||||
[{{"text": "User decided to use X approach for Y", "category": "decision"}}]
|
||||
|
||||
Categories: preference, fact, decision, entity, other
|
||||
|
||||
JSON:"""
|
||||
|
||||
def load_state() -> Dict[str, Any]:
|
||||
"""Load curation state."""
|
||||
if STATE_FILE.exists():
|
||||
try:
|
||||
with open(STATE_FILE) as f:
|
||||
return json.load(f)
|
||||
except:
|
||||
pass
|
||||
return {"last_turn": 0, "last_curation": None}
|
||||
|
||||
def save_state(state: Dict[str, Any]):
|
||||
"""Save curation state."""
|
||||
with open(STATE_FILE, 'w') as f:
|
||||
json.dump(state, f, indent=2)
|
||||
|
||||
def get_point_count_since(last_time: str) -> int:
|
||||
"""Get count of points since last curation time."""
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{QDRANT_URL}/collections/{MEMORIES_COLLECTION}/points/count",
|
||||
json={
|
||||
"filter": {
|
||||
"must": [
|
||||
{
|
||||
"key": "timestamp",
|
||||
"range": {
|
||||
"gt": last_time
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
response.raise_for_status()
|
||||
return response.json().get("result", {}).get("count", 0)
|
||||
except Exception as e:
|
||||
print(f"Error getting count: {e}", file=sys.stderr)
|
||||
return 0
|
||||
|
||||
def get_turns_since(last_turn: int, limit: int = 100) -> List[Dict[str, Any]]:
|
||||
"""Get all turns since last curation."""
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{QDRANT_URL}/collections/{MEMORIES_COLLECTION}/points/scroll",
|
||||
json={"limit": limit, "with_payload": True},
|
||||
timeout=30
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
turns = []
|
||||
for point in data.get("result", {}).get("points", []):
|
||||
turn_num = point.get("payload", {}).get("turn", 0)
|
||||
if turn_num > last_turn:
|
||||
turns.append(point)
|
||||
|
||||
# Sort by turn number
|
||||
turns.sort(key=lambda x: x.get("payload", {}).get("turn", 0))
|
||||
return turns
|
||||
except Exception as e:
|
||||
print(f"Error fetching turns: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
def extract_gems_with_llm(conversation_text: str) -> List[Dict[str, str]]:
|
||||
"""Send conversation to LLM for gem extraction."""
|
||||
prompt = get_curator_prompt(conversation_text)
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{OLLAMA_URL}/v1/chat/completions",
|
||||
json={
|
||||
"model": CURATOR_MODEL,
|
||||
"messages": [{"role": "user", "content": prompt}],
|
||||
"temperature": 0.3,
|
||||
"max_tokens": 1000
|
||||
},
|
||||
timeout=120
|
||||
)
|
||||
response.raise_for_status()
|
||||
data = response.json()
|
||||
|
||||
content = data.get("choices", [{}])[0].get("message", {}).get("content", "[]")
|
||||
|
||||
# Extract JSON from response
|
||||
try:
|
||||
# Try to find JSON array in response
|
||||
start = content.find('[')
|
||||
end = content.rfind(']')
|
||||
if start != -1 and end != -1:
|
||||
json_str = content[start:end+1]
|
||||
gems = json.loads(json_str)
|
||||
if isinstance(gems, list):
|
||||
return gems
|
||||
except:
|
||||
pass
|
||||
|
||||
return []
|
||||
except Exception as e:
|
||||
print(f"Error calling LLM: {e}", file=sys.stderr)
|
||||
return []
|
||||
|
||||
def store_gem(gem: Dict[str, str]) -> bool:
|
||||
"""Store a single gem to gems_tr."""
|
||||
try:
|
||||
# Get embedding for gem
|
||||
response = requests.post(
|
||||
f"{OLLAMA_URL}/api/embeddings",
|
||||
json={"model": "snowflake-arctic-embed2", "prompt": gem["text"]},
|
||||
timeout=30
|
||||
)
|
||||
response.raise_for_status()
|
||||
vector = response.json().get("embedding", [])
|
||||
|
||||
if not vector:
|
||||
return False
|
||||
|
||||
# Store to gems_tr
|
||||
response = requests.put(
|
||||
f"{QDRANT_URL}/collections/{GEMS_COLLECTION}/points",
|
||||
json={
|
||||
"points": [{
|
||||
"id": hash(gem["text"]) % (2**63),
|
||||
"vector": vector,
|
||||
"payload": {
|
||||
"text": gem["text"],
|
||||
"category": gem.get("category", "other"),
|
||||
"createdAt": datetime.now(timezone.utc).isoformat(),
|
||||
"source": "turn_based_curator"
|
||||
}
|
||||
}]
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
response.raise_for_status()
|
||||
return True
|
||||
except Exception as e:
|
||||
print(f"Error storing gem: {e}", file=sys.stderr)
|
||||
return False
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="Turn-based curator")
|
||||
parser.add_argument("--threshold", "-t", type=int, default=10,
|
||||
help="Run curation every N turns (default: 10)")
|
||||
parser.add_argument("--execute", "-e", action="store_true",
|
||||
help="Execute curation")
|
||||
parser.add_argument("--dry-run", "-n", action="store_true",
|
||||
help="Preview what would be curated")
|
||||
parser.add_argument("--status", "-s", action="store_true",
|
||||
help="Show current turn status")
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
# Load state
|
||||
state = load_state()
|
||||
current_turn = get_current_turn_count()
|
||||
turns_since = current_turn - state["last_turn"]
|
||||
|
||||
if args.status:
|
||||
print(f"Current turn: {current_turn}")
|
||||
print(f"Last curation: {state['last_turn']}")
|
||||
print(f"Turns since last curation: {turns_since}")
|
||||
print(f"Threshold: {args.threshold}")
|
||||
print(f"Ready to curate: {'YES' if turns_since >= args.threshold else 'NO'}")
|
||||
return
|
||||
|
||||
print(f"Turn-based Curator")
|
||||
print(f"Current turn: {current_turn}")
|
||||
print(f"Last curation: {state['last_turn']}")
|
||||
print(f"Turns since: {turns_since}")
|
||||
print(f"Threshold: {args.threshold}")
|
||||
print()
|
||||
|
||||
if turns_since < args.threshold:
|
||||
print(f"Not enough turns. Need {args.threshold}, have {turns_since}")
|
||||
return
|
||||
|
||||
# Get turns to process
|
||||
print(f"Fetching {turns_since} turns...")
|
||||
turns = get_turns_since(state["last_turn"], limit=turns_since + 10)
|
||||
|
||||
if not turns:
|
||||
print("No new turns found")
|
||||
return
|
||||
|
||||
# Build conversation text
|
||||
conversation_parts = []
|
||||
for turn in turns:
|
||||
role = turn.get("payload", {}).get("role", "unknown")
|
||||
content = turn.get("payload", {}).get("content", "")
|
||||
conversation_parts.append(f"{role.upper()}: {content}")
|
||||
|
||||
conversation_text = "\n\n".join(conversation_parts)
|
||||
|
||||
print(f"Processing {len(turns)} turns ({len(conversation_text)} chars)")
|
||||
print()
|
||||
|
||||
if args.dry_run:
|
||||
print("=== CONVERSATION TEXT ===")
|
||||
print(conversation_text[:500] + "..." if len(conversation_text) > 500 else conversation_text)
|
||||
print()
|
||||
print("[DRY RUN] Would extract gems and store to gems_tr")
|
||||
return
|
||||
|
||||
if not args.execute:
|
||||
print("Use --execute to run curation or --dry-run to preview")
|
||||
return
|
||||
|
||||
# Extract gems
|
||||
print("Extracting gems with LLM...")
|
||||
gems = extract_gems_with_llm(conversation_text)
|
||||
|
||||
if not gems:
|
||||
print("No gems extracted")
|
||||
return
|
||||
|
||||
print(f"Extracted {len(gems)} gems:")
|
||||
for i, gem in enumerate(gems, 1):
|
||||
print(f" {i}. [{gem.get('category', 'other')}] {gem['text'][:80]}...")
|
||||
print()
|
||||
|
||||
# Store gems
|
||||
print("Storing gems...")
|
||||
success = 0
|
||||
for gem in gems:
|
||||
if store_gem(gem):
|
||||
success += 1
|
||||
|
||||
# Update state
|
||||
state["last_turn"] = current_turn
|
||||
state["last_curation"] = datetime.now(timezone.utc).isoformat()
|
||||
save_state(state)
|
||||
|
||||
print(f"Done! Stored {success}/{len(gems)} gems")
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
85
tr-continuous/migrate_add_curated.py
Executable file
85
tr-continuous/migrate_add_curated.py
Executable file
@@ -0,0 +1,85 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
Migration: Add 'curated: false' to existing memories_tr entries.
|
||||
|
||||
Run once to update all existing memories for the new timer curator.
|
||||
Uses POST /collections/{name}/points/payload with {"points": [ids], "payload": {...}}
|
||||
"""
|
||||
|
||||
import requests
|
||||
import time
|
||||
import sys
|
||||
|
||||
QDRANT_URL = "http://10.0.0.40:6333"
|
||||
COLLECTION = "memories_tr"
|
||||
|
||||
def update_existing_memories():
|
||||
"""Add curated=false to all memories that don't have the field."""
|
||||
print("🔧 Migrating existing memories...")
|
||||
|
||||
offset = None
|
||||
updated = 0
|
||||
batch_size = 100
|
||||
max_iterations = 200
|
||||
iterations = 0
|
||||
|
||||
while iterations < max_iterations:
|
||||
iterations += 1
|
||||
|
||||
scroll_data = {
|
||||
"limit": batch_size,
|
||||
"with_payload": True
|
||||
}
|
||||
if offset:
|
||||
scroll_data["offset"] = offset
|
||||
|
||||
try:
|
||||
response = requests.post(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/scroll",
|
||||
json=scroll_data,
|
||||
headers={"Content-Type": "application/json"},
|
||||
timeout=30
|
||||
)
|
||||
response.raise_for_status()
|
||||
result = response.json()
|
||||
points = result.get("result", {}).get("points", [])
|
||||
|
||||
if not points:
|
||||
break
|
||||
|
||||
# Collect IDs that need curated=false
|
||||
ids_to_update = []
|
||||
for point in points:
|
||||
payload = point.get("payload", {})
|
||||
if "curated" not in payload:
|
||||
ids_to_update.append(point["id"])
|
||||
|
||||
if ids_to_update:
|
||||
# POST /points/payload with {"points": [ids], "payload": {...}}
|
||||
update_response = requests.post(
|
||||
f"{QDRANT_URL}/collections/{COLLECTION}/points/payload",
|
||||
json={
|
||||
"points": ids_to_update,
|
||||
"payload": {"curated": False}
|
||||
},
|
||||
timeout=30
|
||||
)
|
||||
update_response.raise_for_status()
|
||||
updated += len(ids_to_update)
|
||||
print(f" Updated batch: {len(ids_to_update)} memories (total: {updated})")
|
||||
time.sleep(0.05)
|
||||
|
||||
offset = result.get("result", {}).get("next_page_offset")
|
||||
if not offset:
|
||||
break
|
||||
|
||||
except Exception as e:
|
||||
print(f"Error: {e}", file=sys.stderr)
|
||||
import traceback
|
||||
traceback.print_exc()
|
||||
break
|
||||
|
||||
print(f"✅ Migration complete: {updated} memories updated with curated=false")
|
||||
|
||||
if __name__ == "__main__":
|
||||
update_existing_memories()
|
||||
14
tr-continuous/turn-curator.service
Normal file
14
tr-continuous/turn-curator.service
Normal file
@@ -0,0 +1,14 @@
|
||||
[Unit]
|
||||
Description=TrueRecall Turn-Based Curator (every 10 turns)
|
||||
After=network.target mem-qdrant-watcher.service
|
||||
|
||||
[Service]
|
||||
Type=simple
|
||||
User=root
|
||||
WorkingDirectory=/root/.openclaw/workspace/.projects/true-recall-v2/tr-continuous
|
||||
ExecStart=/usr/bin/python3 /root/.openclaw/workspace/.projects/true-recall-v2/tr-continuous/curator_turn_based.py --threshold 10 --execute
|
||||
Restart=on-failure
|
||||
RestartSec=60
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
BIN
tr-daily/__pycache__/curate_from_qdrant.cpython-312.pyc
Normal file
BIN
tr-daily/__pycache__/curate_from_qdrant.cpython-312.pyc
Normal file
Binary file not shown.
358
tr-daily/curate_from_qdrant.py
Normal file
358
tr-daily/curate_from_qdrant.py
Normal file
@@ -0,0 +1,358 @@
|
||||
#!/usr/bin/env python3
|
||||
"""
|
||||
True-Recall v2 Curator: Reads from Qdrant kimi_memories
|
||||
|
||||
Reads 24 hours of conversation from Qdrant kimi_memories collection,
|
||||
extracts contextual gems using qwen3, stores to Qdrant gems_tr with mxbai embeddings.
|
||||
|
||||
Usage:
|
||||
python curate_from_qdrant.py --user-id rob
|
||||
python curate_from_qdrant.py --user-id rob --date 2026-02-23
|
||||
"""
|
||||
|
||||
import json
|
||||
import argparse
|
||||
import requests
|
||||
import urllib.request
|
||||
from datetime import datetime, timedelta
|
||||
from pathlib import Path
|
||||
from typing import List, Dict, Any, Optional
|
||||
import hashlib
|
||||
|
||||
# Configuration
|
||||
QDRANT_URL = "http://10.0.0.40:6333"
|
||||
SOURCE_COLLECTION = "memories_tr"
|
||||
TARGET_COLLECTION = "gems_tr"
|
||||
|
||||
OLLAMA_URL = "http://10.0.0.10:11434"
|
||||
EMBEDDING_MODEL = "mxbai-embed-large"
|
||||
CURATION_MODEL = "qwen3:4b-instruct"
|
||||
|
||||
# Load curator prompt
|
||||
CURATOR_PROMPT_PATH = "/root/.openclaw/workspace/.projects/true-recall/curator-prompt.md"
|
||||
|
||||
|
||||
def load_curator_prompt() -> str:
|
||||
"""Load the curator system prompt."""
|
||||
try:
|
||||
with open(CURATOR_PROMPT_PATH, 'r') as f:
|
||||
return f.read()
|
||||
except FileNotFoundError:
|
||||
# Fallback to v2 location
|
||||
CURATOR_PROMPT_PATH_V2 = "/root/.openclaw/workspace/.projects/true-recall-v2/curator-prompt.md"
|
||||
with open(CURATOR_PROMPT_PATH_V2, 'r') as f:
|
||||
return f.read()
|
||||
|
||||
|
||||
def get_turns_from_qdrant(user_id: str, date_str: str) -> List[Dict[str, Any]]:
|
||||
"""
|
||||
Get all conversation turns from Qdrant for a specific user and date.
|
||||
|
||||
Returns turns sorted by conversation_id and turn_number.
|
||||
"""
|
||||
# Build filter for user_id and date
|
||||
filter_data = {
|
||||
"must": [
|
||||
{"key": "user_id", "match": {"value": user_id}},
|
||||
{"key": "date", "match": {"value": date_str}}
|
||||
]
|
||||
}
|
||||
|
||||
# Use scroll API to get all matching points
|
||||
all_points = []
|
||||
offset = None
|
||||
max_iterations = 100 # Safety limit
|
||||
iterations = 0
|
||||
|
||||
while iterations < max_iterations:
|
||||
iterations += 1
|
||||
scroll_data = {
|
||||
"limit": 100,
|
||||
"with_payload": True,
|
||||
"filter": filter_data
|
||||
}
|
||||
|
||||
if offset:
|
||||
scroll_data["offset"] = offset
|
||||
|
||||
req = urllib.request.Request(
|
||||
f"{QDRANT_URL}/collections/{SOURCE_COLLECTION}/points/scroll",
|
||||
data=json.dumps(scroll_data).encode(),
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="POST"
|
||||
)
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as response:
|
||||
result = json.loads(response.read().decode())
|
||||
points = result.get("result", {}).get("points", [])
|
||||
|
||||
if not points:
|
||||
break
|
||||
|
||||
all_points.extend(points)
|
||||
|
||||
# Check if there's more
|
||||
offset = result.get("result", {}).get("next_page_offset")
|
||||
if not offset:
|
||||
break
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404:
|
||||
print(f"⚠️ Collection {SOURCE_COLLECTION} not found")
|
||||
return []
|
||||
raise
|
||||
|
||||
# Convert points to turn format (harvested summaries)
|
||||
turns = []
|
||||
for point in all_points:
|
||||
payload = point.get("payload", {})
|
||||
|
||||
# Extract user and AI messages
|
||||
user_msg = payload.get("user_message", "")
|
||||
ai_msg = payload.get("ai_response", "")
|
||||
|
||||
# Get timestamp from created_at
|
||||
created_at = payload.get("created_at", "")
|
||||
|
||||
turn = {
|
||||
"turn": payload.get("turn_number", 0),
|
||||
"user_id": payload.get("user_id", user_id),
|
||||
"user": user_msg,
|
||||
"ai": ai_msg,
|
||||
"conversation_id": payload.get("conversation_id", ""),
|
||||
"session_id": payload.get("session_id", ""),
|
||||
"timestamp": created_at,
|
||||
"date": payload.get("date", date_str),
|
||||
"content_hash": payload.get("content_hash", "")
|
||||
}
|
||||
|
||||
# Skip if no content
|
||||
if turn["user"] or turn["ai"]:
|
||||
turns.append(turn)
|
||||
|
||||
# Sort by conversation_id, then by turn number
|
||||
turns.sort(key=lambda x: (x.get("conversation_id", ""), x.get("turn", 0)))
|
||||
|
||||
return turns
|
||||
|
||||
|
||||
def extract_gems_with_curator(turns: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
|
||||
"""Use qwen3 to extract gems from conversation turns."""
|
||||
if not turns:
|
||||
return []
|
||||
|
||||
prompt = load_curator_prompt()
|
||||
|
||||
# Build the conversation input
|
||||
conversation_json = json.dumps(turns, indent=2)
|
||||
|
||||
# Call Ollama with native system prompt
|
||||
response = requests.post(
|
||||
f"{OLLAMA_URL}/api/generate",
|
||||
json={
|
||||
"model": CURATION_MODEL,
|
||||
"system": prompt,
|
||||
"prompt": f"## Input Conversation\n\n```json\n{conversation_json}\n```\n\n## Output\n",
|
||||
"stream": False,
|
||||
"options": {
|
||||
"temperature": 0.1,
|
||||
"num_predict": 4000
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
raise RuntimeError(f"Curation failed: {response.text}")
|
||||
|
||||
result = response.json()
|
||||
output = result.get('response', '').strip()
|
||||
|
||||
# Extract JSON from output (handle markdown code blocks)
|
||||
if '```json' in output:
|
||||
output = output.split('```json')[1].split('```')[0].strip()
|
||||
elif '```' in output:
|
||||
output = output.split('```')[1].split('```')[0].strip()
|
||||
|
||||
try:
|
||||
# Extract JSON array - find first [ and last ]
|
||||
start_idx = output.find('[')
|
||||
end_idx = output.rfind(']')
|
||||
if start_idx != -1 and end_idx != -1 and end_idx > start_idx:
|
||||
output = output[start_idx:end_idx+1]
|
||||
|
||||
gems = json.loads(output)
|
||||
if not isinstance(gems, list):
|
||||
print(f"Warning: Curator returned non-list, wrapping: {type(gems)}")
|
||||
gems = [gems] if gems else []
|
||||
return gems
|
||||
except json.JSONDecodeError as e:
|
||||
print(f"Error parsing curator output: {e}")
|
||||
print(f"Raw output: {output[:500]}...")
|
||||
return []
|
||||
|
||||
|
||||
def get_embedding(text: str) -> List[float]:
|
||||
"""Get embedding vector from Ollama using mxbai-embed-large."""
|
||||
response = requests.post(
|
||||
f"{OLLAMA_URL}/api/embeddings",
|
||||
json={
|
||||
"model": EMBEDDING_MODEL,
|
||||
"prompt": text
|
||||
}
|
||||
)
|
||||
|
||||
if response.status_code != 200:
|
||||
raise RuntimeError(f"Embedding failed: {response.text}")
|
||||
|
||||
return response.json()['embedding']
|
||||
|
||||
|
||||
def get_gem_id(gem: Dict[str, Any], user_id: str) -> int:
|
||||
"""Generate deterministic integer ID for a gem."""
|
||||
hash_bytes = hashlib.sha256(
|
||||
f"{user_id}:{gem.get('conversation_id', '')}:{gem.get('turn_range', '')}".encode()
|
||||
).digest()[:8]
|
||||
return int.from_bytes(hash_bytes, byteorder='big') % (2**63)
|
||||
|
||||
|
||||
def check_duplicate(gem: Dict[str, Any], user_id: str) -> bool:
|
||||
"""Check if a similar gem already exists in gems_tr."""
|
||||
gem_id = get_gem_id(gem, user_id)
|
||||
|
||||
# Check if point exists
|
||||
try:
|
||||
req = urllib.request.Request(
|
||||
f"{QDRANT_URL}/collections/{TARGET_COLLECTION}/points/{gem_id}",
|
||||
headers={"Content-Type": "application/json"},
|
||||
method="GET"
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=10) as response:
|
||||
return True # Point exists
|
||||
except urllib.error.HTTPError as e:
|
||||
if e.code == 404:
|
||||
return False # Point doesn't exist
|
||||
raise
|
||||
|
||||
|
||||
def store_gem_to_qdrant(gem: Dict[str, Any], user_id: str) -> bool:
|
||||
"""Store a gem to Qdrant with embedding."""
|
||||
# Create embedding from gem text
|
||||
embedding_text = f"{gem.get('gem', '')} {gem.get('context', '')} {gem.get('snippet', '')}"
|
||||
vector = get_embedding(embedding_text)
|
||||
|
||||
# Prepare payload
|
||||
payload = {
|
||||
"user_id": user_id,
|
||||
**gem
|
||||
}
|
||||
|
||||
# Generate deterministic integer ID
|
||||
gem_id = get_gem_id(gem, user_id)
|
||||
|
||||
# Store to Qdrant
|
||||
response = requests.put(
|
||||
f"{QDRANT_URL}/collections/{TARGET_COLLECTION}/points",
|
||||
json={
|
||||
"points": [{
|
||||
"id": gem_id,
|
||||
"vector": vector,
|
||||
"payload": payload
|
||||
}]
|
||||
}
|
||||
)
|
||||
|
||||
return response.status_code == 200
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description="True-Recall Curator v2 - Reads from Qdrant")
|
||||
parser.add_argument("--user-id", required=True, help="User ID to process")
|
||||
parser.add_argument("--date", help="Specific date to process (YYYY-MM-DD), defaults to yesterday")
|
||||
parser.add_argument("--dry-run", action="store_true", help="Don't store, just preview")
|
||||
args = parser.parse_args()
|
||||
|
||||
# Determine date (yesterday by default)
|
||||
if args.date:
|
||||
date_str = args.date
|
||||
else:
|
||||
yesterday = datetime.now() - timedelta(days=1)
|
||||
date_str = yesterday.strftime("%Y-%m-%d")
|
||||
|
||||
print(f"🔍 True-Recall Curator v2 for {args.user_id}")
|
||||
print(f"📅 Processing date: {date_str}")
|
||||
print(f"🧠 Embedding model: {EMBEDDING_MODEL}")
|
||||
print(f"💎 Target collection: {TARGET_COLLECTION}")
|
||||
print()
|
||||
|
||||
# Get turns from Qdrant
|
||||
print(f"📥 Fetching conversation turns from {SOURCE_COLLECTION}...")
|
||||
turns = get_turns_from_qdrant(args.user_id, date_str)
|
||||
print(f"✅ Found {len(turns)} turns")
|
||||
|
||||
if not turns:
|
||||
print("⚠️ No turns to process. Exiting.")
|
||||
return
|
||||
|
||||
# Show sample
|
||||
print("\n📄 Sample turns:")
|
||||
for i, turn in enumerate(turns[:3], 1):
|
||||
user_msg = turn.get("user", "")[:60]
|
||||
ai_msg = turn.get("ai", "")[:60]
|
||||
print(f" Turn {turn.get('turn')}: User: {user_msg}...")
|
||||
print(f" AI: {ai_msg}...")
|
||||
if len(turns) > 3:
|
||||
print(f" ... and {len(turns) - 3} more")
|
||||
|
||||
# Extract gems
|
||||
print("\n🧠 Extracting gems with The Curator (qwen3)...")
|
||||
gems = extract_gems_with_curator(turns)
|
||||
print(f"✅ Extracted {len(gems)} gems")
|
||||
|
||||
if not gems:
|
||||
print("⚠️ No gems extracted. Exiting.")
|
||||
return
|
||||
|
||||
# Preview gems
|
||||
print("\n💎 Preview of extracted gems:")
|
||||
for i, gem in enumerate(gems[:3], 1):
|
||||
print(f"\n--- Gem {i} ---")
|
||||
print(f"Gem: {gem.get('gem', 'N/A')[:100]}...")
|
||||
print(f"Categories: {gem.get('categories', [])}")
|
||||
print(f"Importance: {gem.get('importance', 'N/A')}")
|
||||
print(f"Confidence: {gem.get('confidence', 'N/A')}")
|
||||
|
||||
if len(gems) > 3:
|
||||
print(f"\n... and {len(gems) - 3} more gems")
|
||||
|
||||
if args.dry_run:
|
||||
print("\n🏃 DRY RUN: Not storing gems.")
|
||||
return
|
||||
|
||||
# Check for duplicates and store
|
||||
print("\n💾 Storing gems to Qdrant...")
|
||||
stored = 0
|
||||
skipped = 0
|
||||
failed = 0
|
||||
|
||||
for gem in gems:
|
||||
# Check for duplicates
|
||||
if check_duplicate(gem, args.user_id):
|
||||
print(f" ⏭️ Skipping duplicate: {gem.get('gem', 'N/A')[:50]}...")
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
if store_gem_to_qdrant(gem, args.user_id):
|
||||
stored += 1
|
||||
else:
|
||||
print(f" ⚠️ Failed to store gem: {gem.get('gem', 'N/A')[:50]}...")
|
||||
failed += 1
|
||||
|
||||
print(f"\n✅ Stored: {stored}")
|
||||
print(f"⏭️ Skipped (duplicates): {skipped}")
|
||||
print(f"❌ Failed: {failed}")
|
||||
|
||||
print("\n🎉 Curation complete!")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user