diff --git a/checklist.md b/checklist.md index 7176cfe..5eca5af 100644 --- a/checklist.md +++ b/checklist.md @@ -1,216 +1,477 @@ -# TrueRecall v2 - Installation Checklist +# TrueRecall v2 - Exhaustive Installation & Validation Checklist -Pre-install and post-install validation steps. +Comprehensive pre-install, install, and post-install validation steps. --- -## Pre-Installation Checks +## Table of Contents -### 1. System Requirements - -| Check | Command | Expected Result | -|-------|---------|---------------| -| Python 3.8+ | `python3 --version` | 3.8 or higher | -| pip available | `pip3 --version` | Working | -| curl available | `curl --version` | Working | -| git available | `git --version` | Working | - -### 2. Qdrant Verification - -| Check | Command | Expected Result | -|-------|---------|---------------| -| Qdrant running | `curl http://localhost:6333` | Returns version | -| Collections accessible | `curl http://localhost:6333/collections` | Returns list | -| Network reachable | `curl http://127.0.0.1:6333` | Same result | - -### 3. Ollama Verification - -| Check | Command | Expected Result | -|-------|---------|---------------| -| Ollama running | `curl http://localhost:11434/api/tags` | Returns models | -| Embedding model | Check in list | snowflake or mxbai | -| Curator model | Check in list | qwen3:4b or 30b | -| Network reachable | `curl http://127.0.0.1:11434/api/tags` | Same result | - -### 4. OpenClaw Status - -| Check | Command | Expected Result | -|-------|---------|---------------| -| Gateway running | `openclaw gateway status` | Active | -| Config valid | `openclaw doctor` | No errors | -| Plugin available | `openclaw plugins list` | memory-qdrant present | +1. [Pre-Installation Checks](#1-pre-installation-checks) +2. [System Dependencies](#2-system-dependencies) +3. [Network & Connectivity](#3-network--connectivity) +4. [Installation Validation](#4-installation-validation) +5. [Post-Installation Runtime](#5-post-installation-runtime) +6. [Data Integrity & Quality](#6-data-integrity--quality) +7. [Security & Privacy](#7-security--privacy) +8. [Performance & Scaling](#8-performance--scaling) +9. [Git & Backup](#9-git--backup) +10. [Troubleshooting](#10-troubleshooting) +11. [Quick Reference](#11-quick-reference) --- -## Installation Steps +## 1. Pre-Installation Checks -### Step 1: Choose Mode +### 1.1 Environment Prerequisites -| Mode | Description | Use Case | -|------|-------------|----------| -| **full** | Watcher + Curator + Gems | Complete TrueRecall v2 | -| **simple** | Watcher only | Just capture memories | +| # | Check | Command | Expected Result | Critical | +|---|-------|---------|-----------------|----------| +| 1.1.1 | Python 3.8+ | `python3 --version` | `3.8.x` or higher | ✅ Yes | +| 1.1.2 | pip available | `pip3 --version` | Version displayed | ✅ Yes | +| 1.1.3 | curl available | `curl --version` | Version displayed | ✅ Yes | +| 1.1.4 | git available | `git --version` | Version displayed | ⚠️ Medium | +| 1.1.5 | jq available | `jq --version` | Version displayed | ⚠️ Medium | +| 1.1.6 | virtualenv | `python3 -m venv --help` | Help text shown | ❌ No | -### Step 2: Full Mode Prompts +### 1.2 TrueRecall v1 Detection -| # | Prompt | Default | Validate | -|---|--------|---------|----------| -| 1 | Installation mode | `full` | Must be "full" or "simple" | -| 2 | Embedding model | `mxbai-embed-large` | Model must exist in Ollama | -| 3 | Timer interval | `5` minutes | 5, 30, 60, or custom | -| 4 | Batch size | `100` | 50, 100, 500, or custom | -| 5 | Qdrant URL | `http://localhost:6333` | Must be reachable | -| 6 | Ollama URL | `http://localhost:11434` | Must be reachable | -| 7 | Curator LLM | `qwen3:30b-a3b-instruct` | Model must exist in Ollama | -| 8 | User ID | `rob` | Non-empty string | - -### Step 3: Simple Mode Prompts - -| # | Prompt | Default | Validate | -|---|--------|---------|----------| -| 1 | Installation mode | `full` | Must be "simple" | -| 2 | Embedding model | `snowflake-arctic-embed2` | Model must exist in Ollama | -| 3 | Qdrant URL | `http://localhost:6333` | Must be reachable | -| 4 | User ID | `rob` | Non-empty string | +| # | Check | Command/Method | Expected Result | +|---|-------|---------------|-----------------| +| 1.2.1 | v1 folder exists | `ls ~/.openclaw/workspace/.projects/true-recall/tr-daily/` | Should NOT exist | +| 1.2.2 | v1 cron exists | `crontab -l \| grep "tr-daily"` | Should be empty | +| 1.2.3 | v1 service exists | `systemctl status memories-qdrant` | Should be "not found" | +| 1.2.4 | v1 Redis queue | `redis-cli llen "tr:compact_queue"` | Should be 0 or error | --- -## Post-Installation Validation +## 2. System Dependencies -### 1. Config Files Created +### 2.1 Qdrant Verification -| File | Location | Check | -|------|----------|-------| -| `curator_config.json` | `tr-continuous/` | Contains all settings | -| `curator_timer.py` | `tr-continuous/` | Executable, no syntax errors | -| `mem-qdrant-watcher.service` | systemd path | Loaded and enabled | +| # | Check | Command | Expected Result | Critical | +|---|-------|---------|-----------------|----------| +| 2.1.1 | Qdrant HTTP port | `curl http://:6333` | Returns JSON version | ✅ Yes | +| 2.1.2 | Qdrant gRPC port | `curl http://:6334` | Should connect | ⚠️ Medium | +| 2.1.3 | Collections API | `curl http://:6333/collections` | Returns list | ✅ Yes | +| 2.1.4 | Health check | `curl http://:6333/healthz` | Returns "ok" | ✅ Yes | +| 2.1.5 | HTTPS fallback | `curl -k https://:6333` | Check if TLS enabled | ❌ No | +| 2.1.6 | Authentication | `curl -H "api-key: XXX" ` | Check if auth required | ❌ No | +| 2.1.7 | Cluster status | `curl http://:6333/cluster` | Returns peers | ❌ No | -### 2. Watcher Service +### 2.2 Ollama Verification -| Check | Command | Expected Result | -|-------|---------|---------------| -| Service installed | `systemctl status mem-qdrant-watcher` | Loaded | -| Service running | `systemctl is-active mem-qdrant-watcher` | `active` | -| Logs available | `sudo journalctl -u mem-qdrant-watcher -n 5` | Recent entries | +| # | Check | Command | Expected Result | Critical | +|---|-------|---------|-----------------|----------| +| 2.2.1 | Ollama running | `curl http://:11434/api/tags` | Returns models list | ✅ Yes | +| 2.2.2 | Version check | `curl http://:11434/api/version` | Version string | ⚠️ Medium | +| 2.2.3 | Embedding model pulled | Check in tags response | `mxbai-embed-large` or `snowflake-arctic-embed2` | ✅ Yes | +| 2.2.4 | Curator model pulled | Check in tags response | `qwen3:30b-a3b-instruct` or `qwen3:4b-instruct` | ✅ Yes | +| 2.2.5 | GPU acceleration | `curl http://:11434/api/ps` | Shows GPU if available | ❌ No | +| 2.2.6 | Model info | `curl http://:11434/api/show -d '{"name":"mxbai-embed-large"}'` | Returns details | ⚠️ Medium | +| 2.2.7 | Generate test | Test embedding generation | Returns vector | ✅ Yes | -### 3. Cron Job (Full Mode Only) +### 2.3 OpenClaw Verification -| Check | Command | Expected Result | -|-------|---------|---------------| -| Cron entry exists | `crontab -l \| grep true-recall` | Entry present | -| Syntax valid | Visual check | No errors | -| Timer matches | Compare to config | Same interval | - -### 4. Collections Created - -| Check | Command | Expected Result | -|-------|---------|---------------| -| memories_tr exists | `curl http://localhost:6333/collections/memories_tr` | 200 OK | -| gems_tr exists (full) | `curl http://localhost:6333/collections/gems_tr` | 200 OK | -| Can write | Test point insertion | Success | -| Can read | Test point query | Success | - -### 5. First Run (Full Mode) - -| Check | Command | Expected Result | -|-------|---------|---------------| -| Curator runs | Wait for first cron | Log entry created | -| Log file exists | `ls /var/log/true-recall-timer.log` | File present | -| Gems extracted | Check `gems_tr` count | Increased from 0 | -| Memories tagged | Query with `curated: false` | All have field | - -### 6. Injection Test - -| Check | Method | Expected Result | -|-------|--------|-----------------| -| Plugin loads | `openclaw plugins list` | memory-qdrant active | -| Recall works | Send test message | Gems injected in context | -| Capture works | Check `memories_tr` | New point added | +| # | Check | Command | Expected Result | Critical | +|---|-------|---------|-----------------|----------| +| 2.3.1 | Gateway running | `openclaw gateway status` | "active" | ✅ Yes | +| 2.3.2 | Config valid | `openclaw doctor` | No errors | ✅ Yes | +| 2.3.3 | Plugin available | `openclaw plugins list` | `memory-qdrant` present | ✅ Yes | +| 2.3.4 | Plugin enabled | `openclaw config get plugins.entries.memory-qdrant.enabled` | `true` | ✅ Yes | +| 2.3.5 | Gateway API | `curl http://:18789/api/status` | Returns status JSON | ⚠️ Medium | +| 2.3.6 | Control UI | `curl http://:18789` | Returns HTML | ⚠️ Medium | +| 2.3.7 | Sessions path | `ls ~/.openclaw/agents/main/sessions/` | Directory exists | ✅ Yes | +| 2.3.8 | Compactor config | `openclaw config get agents.defaults.compaction` | Mode and settings | ✅ Yes | --- -## Troubleshooting +## 3. Network & Connectivity -### Common Issues +### 3.1 Service Reachability -| Symptom | Likely Cause | Fix | -|---------|--------------|-----| -| Qdrant unreachable | Not running | `sudo systemctl start qdrant` | -| Ollama unreachable | Not running | `ollama serve` | -| Model not found | Not pulled | `ollama pull mxbai-embed-large` | -| Permission denied | Wrong user | Run with `sudo` or check ownership | -| Cron not running | Service stopped | `sudo systemctl start cron` | -| Watcher not capturing | Session path wrong | Check `session.jsonl` location | +| # | Check | Command | Expected Result | +|---|-------|---------|-----------------| +| 3.1.1 | Qdrant from host | `curl http://:6333` | Response < 100ms | +| 3.1.2 | Ollama from host | `curl http://:11434/api/tags` | Response < 100ms | +| 3.1.3 | Gateway from host | `curl http://:18789/api/status` | Response < 100ms | +| 3.1.4 | Qdrant from container | `docker exec openclaw curl ` | Same as host | +| 3.1.5 | DNS resolution | `nslookup ` | Resolves or fails fast | +| 3.1.6 | Firewall ports | `nc -zv 6333` | "succeeded" | + +### 3.2 URL Validation + +| # | Check | Pattern | Example | +|---|-------|---------|---------| +| 3.2.1 | Qdrant URL format | `http://IP:PORT` | `http://localhost:6333` | +| 3.2.2 | Ollama URL format | `http://IP:PORT` | `http://localhost:11434` | +| 3.2.3 | No trailing slash | `sed 's|/$||'` | `http://localhost:6333` not `...6333/` | +| 3.2.4 | No protocol mismatch | Check for `https://` when should be `http://` | Match service config | --- -## Validation Commands Quick Reference +## 4. Installation Validation -```bash -# Check all services -openclaw status && curl -s http://localhost:6333 && curl -s http://localhost:11434/api/tags +### 4.1 Installer Execution -# Check collections -curl -s http://localhost:6333/collections/memories_tr | jq .result.points_count -curl -s http://localhost:6333/collections/gems_tr | jq .result.points_count +| # | Check | What to Verify | Expected Result | +|---|-------|---------------|-----------------| +| 4.1.1 | v1 detected | Installer warning shown | v1 components listed | +| 4.1.2 | User acknowledgment | "Continue at your own risk?" | Y/N prompt | +| 4.1.3 | Mode selection | Full vs Simple | User choice recorded | +| 4.1.4 | All prompts answered | No skipped prompts | Complete config | +| 4.1.5 | Go-back navigation | Can navigate between prompts | Works correctly | +| 4.1.6 | Custom input | Can enter custom values | Accepted and validated | +| 4.1.7 | Summary displayed | Final config shown | All fields present | +| 4.1.8 | Confirmation | User confirms install | Proceeds or aborts | -# Check watcher -sudo systemctl status mem-qdrant-watcher +### 4.2 Config File Generation -# Check cron -crontab -l | grep true-recall +| # | File | Location | Validation | Critical | +|---|------|----------|------------|----------| +| 4.2.1 | `curator_config.json` | `tr-continuous/` | Valid JSON, all fields | ✅ Yes | +| 4.2.2 | `curator_timer.py` | `tr-continuous/` | Python syntax valid | ✅ Yes | +| 4.2.3 | `content_cleaner.py` | `tr-continuous/` | Python syntax valid | ⚠️ Medium | +| 4.2.4 | `curator-prompt.md` | `tr-continuous/` | File exists, readable | ✅ Yes | +| 4.2.5 | `push-all.sh` | Root | Executable, valid bash | ⚠️ Medium | +| 4.2.6 | `plugin-config.json` | Root (full mode) | Valid JSON | ✅ Yes | -# Check logs -tail -f /var/log/true-recall-timer.log -sudo journalctl -u mem-qdrant-watcher -f -``` +### 4.3 Config File Content Validation + +| # | Field | Type | Validation Rule | Example | +|---|-------|------|-----------------|---------| +| 4.3.1 | `timer_minutes` | integer | > 0 && < 1440 | 5, 30, 60 | +| 4.3.2 | `batch_size` | integer | > 0 && <= 1000 | 50, 100, 500 | +| 4.3.3 | `user_id` | string | non-empty | "rob" | +| 4.3.4 | `source_collection` | string | non-empty | "memories_tr" | +| 4.3.5 | `target_collection` | string | non-empty | "gems_tr" | +| 4.3.6 | `embedding_model` | string | in Ollama model list | "mxbai-embed-large" | +| 4.3.7 | `curator_model` | string | in Ollama model list | "qwen3:30b-a3b-instruct" | +| 4.3.8 | `qdrant_url` | URL | reachable | "http://localhost:6333" | +| 4.3.9 | `ollama_url` | URL | reachable | "http://localhost:11434" | +| 4.3.10 | `max_recall_results` | integer | > 0 && <= 20 | 2, 5, 10 | +| 4.3.11 | `min_recall_score` | float | >= 0.0 && <= 1.0 | 0.7, 0.85 | --- -## Security & Privacy Review (Pre-Commit) +## 5. Post-Installation Runtime -### Before Committing to Git +### 5.1 Watcher Service -| Check | Command | What to Look For | -|-------|---------|------------------| -| **Internal IPs** | `grep -rE "(10\.[0-9]+\.[0-9]+\.[0-9]+\|192\.168\.[0-9]+\.[0-9]+\|172\.(1[6-9]\|2[0-9]\|3[01])\.[0-9]+\.[0-9]+)"` | Private network addresses | -| **Passwords/Tokens** | `grep -riE "(password|token|secret|api.?key)"` | Hardcoded credentials | -| **SSH Keys** | `grep -rE "(BEGIN.*PRIVATE KEY|ssh-rsa|ssh-ed25519)"` | Private keys | -| **Absolute Paths** | `grep -rE "(/root/|/home/[a-z]+/|\.ssh/)"` | System-specific paths | -| **Timestamps** | `grep -rE "[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}"` | Non-project dates/times | -| **Usernames** | `grep -riE "(user: [a-z]+\|username: [a-z]+)"` | System usernames | +| # | Check | Command | Expected Result | Critical | +|---|-------|---------|-----------------|----------| +| 5.1.1 | Service file exists | `ls /etc/systemd/system/mem-qdrant-watcher.service` | File present | ✅ Yes | +| 5.1.2 | Service loaded | `systemctl daemon-reload` | No errors | ✅ Yes | +| 5.1.3 | Service enabled | `systemctl is-enabled mem-qdrant-watcher` | `enabled` | ✅ Yes | +| 5.1.4 | Service running | `systemctl is-active mem-qdrant-watcher` | `active` | ✅ Yes | +| 5.1.5 | Process running | `ps aux \| grep "realtime_qdrant_watcher"` | Process found | ✅ Yes | +| 5.1.6 | Logs accessible | `sudo journalctl -u mem-qdrant-watcher -n 5` | Entries shown | ✅ Yes | +| 5.1.7 | No errors | `sudo journalctl -u mem-qdrant-watcher \| grep -i error` | Empty or minimal | ✅ Yes | +| 5.1.8 | PID file | Check if PID tracking enabled | File exists | ❌ No | +| 5.1.9 | Restart on crash | `systemctl cat mem-qdrant-watcher \| grep Restart` | `Restart=always` | ⚠️ Medium | -### Replace With Placeholders +### 5.2 Timer Curator (Full Mode) -| Found | Replace With | -|-------|--------------| -| `10.0.0.x` | `` or `` | -| `192.168.x.x` | `` | -| `/root/.openclaw/` | `~/.openclaw/` or `` | -| `/home/username/` | `~/` or `` | -| Real tokens | `` or `` | -| Specific dates | `` or `` | +| # | Check | Command | Expected Result | Critical | +|---|-------|---------|-----------------|----------| +| 5.2.1 | Cron entry exists | `crontab -l \| grep "curator_timer.py"` | Entry found | ✅ Yes | +| 5.2.2 | Cron syntax valid | `crontab -l` | No syntax errors | ✅ Yes | +| 5.2.3 | Timer matches config | Compare crontab vs config | Match | ✅ Yes | +| 5.2.4 | Log file created | `ls /var/log/true-recall-timer.log` | File exists | ✅ Yes | +| 5.2.5 | Log writable | `touch /var/log/true-recall-timer.log` | No permission error | ✅ Yes | +| 5.2.6 | Log rotation | `ls /var/log/true-recall-timer.log*` | Rotated files | ❌ No | +| 5.2.7 | First run completed | `tail /var/log/true-recall-timer.log` | "Completed" or similar | ✅ Yes | +| 5.2.8 | No Python errors | `grep -i "traceback\|error" /var/log/true-recall-timer.log` | Empty | ✅ Yes | +| 5.2.9 | Process not stuck | `pgrep -f "curator_timer.py" \| wc -l` | 0 or 1 (not many) | ✅ Yes | -### Files to Review +### 5.3 Collection Creation + +| # | Check | Command | Expected Result | Critical | +|---|-------|---------|-----------------|----------| +| 5.3.1 | memories_tr exists | `curl /collections/memories_tr` | HTTP 200 | ✅ Yes | +| 5.3.2 | gems_tr exists (full) | `curl /collections/gems_tr` | HTTP 200 | ✅ Yes | +| 5.3.3 | Collection config | `curl /collections/memories_tr` | Correct vector size | ✅ Yes | +| 5.3.4 | Embedding size | Check collection config | 768 (mxbai) or 1024 (snowflake) | ✅ Yes | +| 5.3.5 | Distance metric | Check collection config | `cosine` | ✅ Yes | +| 5.3.6 | Index created | Query points | Returns results | ✅ Yes | +| 5.3.7 | Can write points | Test insert | HTTP 200 | ✅ Yes | +| 5.3.8 | Can read points | Test query | Returns points | ✅ Yes | +| 5.3.9 | Can scroll | Scroll API | Returns batch | ✅ Yes | +| 5.3.10 | Payload indexing | Check index config | Fields indexed | ⚠️ Medium | + +--- + +## 6. Data Integrity & Quality + +### 6.1 Memory Capture + +| # | Check | Method | Expected Result | +|---|-------|--------|-----------------| +| 6.1.1 | Watcher capturing | Send message, check memories_tr | Point added within 1s | +| 6.1.2 | Session ID stored | Query point payload | `session_id` present | +| 6.1.3 | Timestamp valid | Query point payload | ISO 8601 format | +| 6.1.4 | Content not empty | Query point payload | `text` non-empty | +| 6.1.5 | User ID set | Query point payload | `user_id` matches config | +| 6.1.6 | curated flag | Query point payload | `curated: false` on new | +| 6.1.7 | Embedding generated | Query point vector | Vector non-null | +| 6.1.8 | Vector correct size | Check dimensions | 768 or 1024 | + +### 6.2 Gem Curation + +| # | Check | Method | Expected Result | +|---|-------|--------|-----------------| +| 6.2.1 | Gems being created | Check gems_tr count | Increasing over time | +| 6.2.2 | curated flag updated | Query memories_tr | `curated: true` after process | +| 6.2.3 | Gem has 11 fields | Query gem payload | All fields present | +| 6.2.4 | importance valid | Check gem payload | "high", "medium", or "low" (not "3") | +| 6.2.5 | confidence valid | Check gem payload | Float 0.0-1.0 (not "0.7") | +| 6.2.6 | gem field valid | Check gem payload | Not starting with "User confirmed" | +| 6.2.7 | categories valid | Check gem payload | Non-empty array | +| 6.2.8 | snippet accurate | Check gem payload | Matches source turns | +| 6.2.9 | timestamp valid | Check gem payload | ISO 8601 from source turn | +| 6.2.10 | context richness | Review gem manually | 2-3 explanatory sentences | + +### 6.3 Duplication Check + +| # | Check | Method | Expected Result | +|---|-------|--------|-----------------| +| 6.3.1 | Similarity score | Search similar gems | < 0.85 for unique | +| 6.3.2 | 24h window check | Query recent gems | No near-duplicates | +| 6.3.3 | Duplicate rate | Sample 100 gems | < 10% duplicates | + +--- + +## 7. Security & Privacy + +### 7.1 Pre-Commit Checks + +| # | Check | Command | What to Look For | Severity | +|---|-------|---------|------------------|----------| +| 7.1.1 | Internal IPs | `grep -rE "(10\.[0-9]+\.[0-9]+\.[0-9]+\|192\.168\.[0-9]+\.[0-9]+)"` | Private network addresses | 🔴 Critical | +| 7.1.2 | Passwords/Tokens | `grep -riE "(password|token|secret|api.?key)"` | Hardcoded credentials | 🔴 Critical | +| 7.1.3 | SSH Keys | `grep -rE "(BEGIN.*PRIVATE KEY|ssh-rsa|ssh-ed25519)"` | Private keys | 🔴 Critical | +| 7.1.4 | Absolute Paths | `grep -rE "(/root/|/home/[a-z]+/|\.ssh/)"` | System-specific paths | 🟡 Medium | +| 7.1.5 | Timestamps | `grep -rE "[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}"` | Non-project dates | 🟡 Medium | +| 7.1.6 | Usernames | `grep -riE "user: [a-z]+"` | System usernames | 🟡 Medium | +| 7.1.7 | Email addresses | `grep -rE "[a-z]+@[a-z]+\.[a-z]+"` | Personal emails | 🟡 Medium | +| 7.1.8 | URLs with credentials | `grep -rE "https?://[^:]+:[^@]+@"` | Embedded auth | 🔴 Critical | +| 7.1.9 | Database credentials | `grep -riE "(mongodb://\|postgres://\|mysql://)"` | Connection strings | 🔴 Critical | +| 7.1.10 | AWS/Azure/GCP keys | `grep -rE "(AKIA[0-9A-Z]{16}\|azure|gcp)"` | Cloud credentials | 🔴 Critical | + +### 7.2 Placeholder Replacements + +| # | Found | Replace With | +|---|-------|--------------| +| 7.2.1 | `10.0.0.x` | `` or `` | +| 7.2.2 | `192.168.x.x` | `` | +| 7.2.3 | `172.16-31.x.x` | `` | +| 7.2.4 | `/root/.openclaw/` | `~/.openclaw/` or `` | +| 7.2.5 | `/home/username/` | `~/` or `` | +| 7.2.6 | Real tokens | `` or `` | +| 7.2.7 | Specific dates | `` or `` | +| 7.2.8 | Real passwords | `` | +| 7.2.9 | Hostnames | `` if internal | + +### 7.3 Files to Review Before Commit - [ ] All `.py` files - [ ] All `.md` files - [ ] All `.json` files - [ ] All `.sh` files +- [ ] All `.yaml` / `.yml` files +- [ ] All `.toml` files - [ ] Any log files (should be in `.gitignore`) - [ ] Any config files with paths +- [ ] `.env` files (should be in `.gitignore`) +- [ ] Dockerfiles +- [ ] Makefile / justfile --- -## Sign-off +## 8. Performance & Scaling -| Check | Status | Date | -|-------|--------|------| -| Pre-install checks passed | ⏳ | | -| Installation completed | ⏳ | | -| Post-install validation passed | ⏳ | | -| First curator run successful | ⏳ | | -| Injection test passed | ⏳ | | +### 8.1 Timing Benchmarks -**Installer:** _______________ -**Date:** _______________ +| # | Operation | Target Time | Max Acceptable | +|---|-----------|-------------|----------------| +| 8.1.1 | Watcher capture | < 100ms | 500ms | +| 8.1.2 | Embedding generation | < 50ms | 200ms | +| 8.1.3 | Qdrant upsert | < 50ms | 100ms | +| 8.1.4 | Curator per batch (4b) | < 30s | 60s | +| 8.1.5 | Curator per batch (30b) | < 5s | 15s | +| 8.1.6 | Plugin recall query | < 100ms | 500ms | +| 8.1.7 | Memory search | < 200ms | 1s | +| 8.1.8 | Log write | < 10ms | 50ms | + +### 8.2 Resource Usage + +| # | Check | Command | Expected | +|---|-------|---------|----------| +| 8.2.1 | Watcher CPU | `htop` or `ps` | < 5% idle | +| 8.2.2 | Watcher RAM | `ps aux` | < 100MB | +| 8.2.3 | Curator CPU | During run | Spike to 100% OK | +| 8.2.4 | Curator RAM | During run | < 2GB | +| 8.2.5 | Qdrant memory | `curl /metrics` | Depends on data | +| 8.2.6 | Disk usage | `df -h` | > 10% free | +| 8.2.7 | Log size | `du -sh /var/log/true-recall*` | Rotate if > 100MB | +| 8.2.8 | Collection size | Check Qdrant metrics | Monitor growth | + +### 8.3 Scaling Considerations + +| # | Check | Threshold | Action | +|---|-------|-----------|--------| +| 8.3.1 | memories_tr points | > 100,000 | Consider indexing optimization | +| 8.3.2 | gems_tr points | > 10,000 | Monitor recall performance | +| 8.3.3 | Daily memories | > 1,000 | Increase batch size | +| 8.3.4 | Curator runtime | > 10 min | Decrease timer interval | +| 8.3.5 | Qdrant memory | > 80% | Scale Qdrant or archive old data | +| 8.3.6 | Log rotation | Weekly | Verify logrotate working | + +--- + +## 9. Git & Backup + +### 9.1 Multi-Remote Configuration + +| # | Check | Command | Expected | +|---|-------|---------|----------| +| 9.1.1 | Gitea remote | `git remote get-url gitea` | `http://10.0.0.61:3000/...` | +| 9.1.2 | GitLab remote | `git remote get-url gitlab` | `https://gitlab.com/...` | +| 9.1.3 | GitHub remote | `git remote get-url github` | `https://github.com/...` | +| 9.1.4 | All remotes accessible | `git fetch --all` | No errors | +| 9.1.5 | push-all.sh exists | `ls push-all.sh` | File present | +| 9.1.6 | push-all.sh executable | `test -x push-all.sh` | True | +| 9.1.7 | GitLab token valid | `curl -H "PRIVATE-TOKEN:..." gitlab.com/api/v4/user` | Returns user | +| 9.1.8 | GitHub auth (if active) | `ssh -T git@github.com` | "successfully authenticated" | + +### 9.2 Backup Validation + +| # | Check | Method | Expected | +|---|-------|--------|----------| +| 9.2.1 | Code backup | `git push gitea` | Success | +| 9.2.2 | Code backup | `git push gitlab` | Success | +| 9.2.3 | Config backup | Copy to deb2 | Files match | +| 9.2.4 | Qdrant backup | Export collections | Can restore | +| 9.2.5 | Last backup time | Check deb2 timestamp | < 24 hours | +| 9.2.6 | Backup integrity | Hash comparison | Match | + +--- + +## 10. Troubleshooting + +### 10.1 Common Issues & Fixes + +| Symptom | Diagnostic Command | Likely Cause | Fix | +|---------|-------------------|--------------|-----| +| Qdrant unreachable | `curl http://:6333` | Not running | `sudo systemctl start qdrant` | +| Ollama unreachable | `curl http://:11434/api/tags` | Not running | `ollama serve` | +| Model not found | `ollama list` | Not pulled | `ollama pull ` | +| Watcher not running | `systemctl status mem-qdrant-watcher` | Service stopped | `sudo systemctl start mem-qdrant-watcher` | +| No gems created | `tail /var/log/true-recall-timer.log` | Curator error | Check logs, fix config | +| Permission denied | `ls -la` | Wrong ownership | `sudo chown user:user` | +| Cron not running | `sudo systemctl status cron` | Service stopped | `sudo systemctl start cron` | +| Plugin not loading | `openclaw plugins list` | Not enabled | `openclaw config set plugins.entries.memory-qdrant.enabled true` | +| High memory usage | `ps aux` | Leak or large batch | Restart services, reduce batch size | +| Slow curator | Check Ollama GPU usage | CPU fallback | Ensure GPU available, check model quantization | + +### 10.2 Log Locations + +| Service | Log Path | Check | +|---------|----------|-------| +| Watcher | `sudo journalctl -u mem-qdrant-watcher` | `journalctl -u mem-qdrant-watcher -f` | +| Curator | `/var/log/true-recall-timer.log` | `tail -f /var/log/true-recall-timer.log` | +| Qdrant | Qdrant stdout/systemd | `sudo journalctl -u qdrant` | +| Ollama | Ollama stdout | `journalctl -u ollama` or stdout | +| OpenClaw Gateway | `sudo journalctl -u openclaw-gateway` | `journalctl -u openclaw-gateway -f` | +| Git sync | Manual script output | Run `push-all.sh` | + +### 10.3 Health Check Script + +```bash +#!/bin/bash +# health-check.sh - Quick validation script + +echo "=== TrueRecall v2 Health Check ===" + +# Check Qdrant +if curl -s http://localhost:6333/healthz > /dev/null; then + echo "✅ Qdrant: OK" +else + echo "❌ Qdrant: UNREACHABLE" +fi + +# Check Ollama +if curl -s http://localhost:11434/api/tags > /dev/null; then + echo "✅ Ollama: OK" +else + echo "❌ Ollama: UNREACHABLE" +fi + +# Check Watcher +if systemctl is-active mem-qdrant-watcher > /dev/null; then + echo "✅ Watcher: RUNNING" +else + echo "❌ Watcher: STOPPED" +fi + +# Check collections +MEM_COUNT=$(curl -s http://localhost:6333/collections/memories_tr | jq -r '.result.points_count // 0') +GEM_COUNT=$(curl -s http://localhost:6333/collections/gems_tr | jq -r '.result.points_count // 0') +echo "📊 memories_tr: $MEM_COUNT points" +echo "📊 gems_tr: $GEM_COUNT points" + +echo "=== Done ===" +``` + +--- + +## 11. Quick Reference + +### 11.1 Validation Commands + +```bash +# Check all services +openclaw status && \ +curl -s http://localhost:6333 && \ +curl -s http://localhost:11434/api/tags && \ +systemctl is-active mem-qdrant-watcher + +# Check collections +curl -s http://localhost:6333/collections/memories_tr | jq .result.points_count +curl -s http://localhost:6333/collections/gems_tr | jq .result.points_count + +# Check watcher logs +sudo journalctl -u mem-qdrant-watcher -n 10 --no-pager + +# Check curator logs +tail -20 /var/log/true-recall-timer.log + +# Check cron +crontab -l | grep true-recall + +# Check plugin +openclaw config get plugins.entries.memory-qdrant + +# Push to all remotes +./push-all.sh +``` + +### 11.2 Sign-off Checklist + +| Section | Status | Date | Checked By | +|---------|--------|------|------------| +| Pre-install checks | ⏳ | | | +| System dependencies | ⏳ | | | +| Network connectivity | ⏳ | | | +| Installation validation | ⏳ | | | +| Watcher service | ⏳ | | | +| Timer curator (if full) | ⏳ | | | +| Collection creation | ⏳ | | | +| First curator run | ⏳ | | | +| Data quality | ⏳ | | | +| Security review | ⏳ | | | +| Multi-remote git | ⏳ | | | +| Backup verified | ⏳ | | | + +**Final Sign-off:** _______________ +**Date:** _______________ +**Version:** v2.0