Initial commit: Jarvis Memory system

2026-02-23 12:13:04 -06:00
commit e8854cd959
72 changed files with 14801 additions and 0 deletions
--- a/skills/qdrant-memory/HARVEST.md
+++ b/skills/qdrant-memory/HARVEST.md
@@ -0,0 +1,137 @@
+# Session Harvest Instructions
+
+## What is Session Harvesting?
+
+Session harvesting extracts conversation turns from OpenClaw session JSONL files and stores them to Qdrant long-term memory with proper embeddings and user_id linking.
+
+## When to Use
+
+- **After setting up a new memory system** — harvest existing sessions
+- **After discovering missed backups** — recover data from session files
+- **Periodically** — if cron jobs missed any data
+
+## Scripts
+
+| Script | Purpose | Usage |
+|--------|---------|-------|
+| `harvest_sessions.py` | Harvest all sessions (auto-sorts by mtime) | Limited by memory, may timeout |
+| `harvest_newest.py` | Harvest specific sessions by name | Recommended for batch control |
+
+## Location
+
+```
+/root/.openclaw/workspace/skills/qdrant-memory/scripts/
+├── harvest_sessions.py   # Auto-harvest (use --limit to control)
+└── harvest_newest.py     # Manual batch (specify session names)
+```
+
+## Usage
+
+### Method 1: Auto-Harvest with Limit
+
+```bash
+# Harvest oldest 10 sessions (default sort)
+python3 harvest_sessions.py --user-id rob --limit 10
+
+# Dry run to see what would be stored
+python3 harvest_sessions.py --user-id rob --dry-run --limit 5
+```
+
+### Method 2: Batch by Session Name (Recommended)
+
+```bash
+# Harvest specific sessions (newest first recommended)
+python3 harvest_newest.py --user-id rob \
+  session-uuid-1.jsonl \
+  session-uuid-2.jsonl \
+  session-uuid-3.jsonl
+```
+
+### Finding Newest Sessions
+
+```bash
+# List 20 newest session files
+ls -t /root/.openclaw/agents/main/sessions/*.jsonl | head -20
+
+# Get just filenames for copy-paste
+ls -t /root/.openclaw/agents/main/sessions/*.jsonl | head -20 | xargs -I{} basename {}
+```
+
+## How It Works
+
+1. **Parse** — Reads JSONL session file, extracts user/AI turns
+2. **Pair** — Matches user message with next AI response
+3. **Embed** — Generates 3 embeddings (user, AI, summary) via Ollama
+4. **Deduplicate** — Checks content_hash before storing
+5. **Store** — Upserts to Qdrant with user_id, conversation_id, turn_number
+
+## Deduplication
+
+- Uses MD5 hash of `user_message::ai_response`
+- Checks Qdrant for existing `user_id + content_hash`
+- Skips if already stored (returns "duplicate")
+- Safe to run multiple times on same sessions
+
+## Output Format
+
+```
+[1] session-uuid.jsonl
+  Stored: 10, Skipped: 6
+
+Total: 44 stored, 6 skipped
+```
+
+- **Stored** = New memories added to Qdrant
+- **Skipped** = Duplicates (already in Qdrant)
+
+## Troubleshooting
+
+### Timeout / SIGKILL
+
+The embedding process is CPU-intensive. If killed:
+
+```bash
+# Use smaller batches
+python3 harvest_newest.py --user-id rob session1.jsonl session2.jsonl
+```
+
+### Check Qdrant Status
+
+```bash
+curl -s http://10.0.0.40:6333/collections/kimi_memories | \
+  python3 -c "import sys,json; d=json.load(sys.stdin); print(d['result']['points_count'])"
+```
+
+### Check Session Content
+
+```bash
+# Count turns in a session
+python3 -c "
+import json
+from pathlib import Path
+f = Path('/root/.openclaw/agents/main/sessions/YOUR-SESSION.jsonl')
+count = sum(1 for line in open(f) if 'user' in line or 'assistant' in line)
+print(f'~{count} messages')
+"
+```
+
+## Memory Architecture
+
+```
+Session JSONL (raw)
+       │
+       ▼
+  harvest_*.py
+       │
+       ├──► Embeddings (Ollama snowflake-arctic-embed2)
+       │
+       ▼
+  Qdrant kimi_memories
+       │
+       └──► Searchable via user_id: "rob"
+```
+
+---
+
+**Created:** February 17, 2026  
+**Author:** Kimi (audit session)