Files
true-recall/checklist.md
root abc5498f60 docs: Add Known Issues section to checklist
Document all issues discovered during TrueRecall v2 deployment:
- Gem quality issues (malformed fields, duplicates, 4b vs 30b)
- OpenClaw UI glitches during compaction
- v1 migration conflicts and detection
- Security issues (IPs, paths, backup files)
- Multi-remote git configuration
- Model-specific issues
- Pre-release checklist for future deployments
2026-02-24 21:43:57 -06:00

25 KiB

TrueRecall v2 - Exhaustive Installation & Validation Checklist

Comprehensive pre-install, install, and post-install validation steps.


Table of Contents

  1. Pre-Installation Checks
  2. System Dependencies
  3. Network & Connectivity
  4. Installation Validation
  5. Post-Installation Runtime
  6. Data Integrity & Quality
  7. Security & Privacy
  8. Performance & Scaling
  9. Git & Backup
  10. Troubleshooting
  11. Quick Reference

1. Pre-Installation Checks

1.1 Environment Prerequisites

# Check Command Expected Result Critical
1.1.1 Python 3.8+ python3 --version 3.8.x or higher Yes
1.1.2 pip available pip3 --version Version displayed Yes
1.1.3 curl available curl --version Version displayed Yes
1.1.4 git available git --version Version displayed ⚠️ Medium
1.1.5 jq available jq --version Version displayed ⚠️ Medium
1.1.6 virtualenv python3 -m venv --help Help text shown No

1.2 TrueRecall v1 Detection

# Check Command/Method Expected Result
1.2.1 v1 folder exists ls ~/.openclaw/workspace/.projects/true-recall/tr-daily/ Should NOT exist
1.2.2 v1 cron exists crontab -l | grep "tr-daily" Should be empty
1.2.3 v1 service exists systemctl status memories-qdrant Should be "not found"
1.2.4 v1 Redis queue redis-cli llen "tr:compact_queue" Should be 0 or error

2. System Dependencies

2.1 Qdrant Verification

# Check Command Expected Result Critical
2.1.1 Qdrant HTTP port curl http://<QDRANT_IP>:6333 Returns JSON version Yes
2.1.2 Qdrant gRPC port curl http://<QDRANT_IP>:6334 Should connect ⚠️ Medium
2.1.3 Collections API curl http://<QDRANT_IP>:6333/collections Returns list Yes
2.1.4 Health check curl http://<QDRANT_IP>:6333/healthz Returns "ok" Yes
2.1.5 HTTPS fallback curl -k https://<QDRANT_IP>:6333 Check if TLS enabled No
2.1.6 Authentication curl -H "api-key: XXX" <QDRANT_URL> Check if auth required No
2.1.7 Cluster status curl http://<QDRANT_IP>:6333/cluster Returns peers No

2.2 Ollama Verification

# Check Command Expected Result Critical
2.2.1 Ollama running curl http://<OLLAMA_IP>:11434/api/tags Returns models list Yes
2.2.2 Version check curl http://<OLLAMA_IP>:11434/api/version Version string ⚠️ Medium
2.2.3 Embedding model pulled Check in tags response mxbai-embed-large or snowflake-arctic-embed2 Yes
2.2.4 Curator model pulled Check in tags response qwen3:30b-a3b-instruct or qwen3:4b-instruct Yes
2.2.5 GPU acceleration curl http://<OLLAMA_IP>:11434/api/ps Shows GPU if available No
2.2.6 Model info curl http://<OLLAMA_IP>:11434/api/show -d '{"name":"mxbai-embed-large"}' Returns details ⚠️ Medium
2.2.7 Generate test Test embedding generation Returns vector Yes

2.3 OpenClaw Verification

# Check Command Expected Result Critical
2.3.1 Gateway running openclaw gateway status "active" Yes
2.3.2 Config valid openclaw doctor No errors Yes
2.3.3 Plugin available openclaw plugins list memory-qdrant present Yes
2.3.4 Plugin enabled openclaw config get plugins.entries.memory-qdrant.enabled true Yes
2.3.5 Gateway API curl http://<GATEWAY_IP>:18789/api/status Returns status JSON ⚠️ Medium
2.3.6 Control UI curl http://<GATEWAY_IP>:18789 Returns HTML ⚠️ Medium
2.3.7 Sessions path ls ~/.openclaw/agents/main/sessions/ Directory exists Yes
2.3.8 Compactor config openclaw config get agents.defaults.compaction Mode and settings Yes

3. Network & Connectivity

3.1 Service Reachability

# Check Command Expected Result
3.1.1 Qdrant from host curl http://<QDRANT_IP>:6333 Response < 100ms
3.1.2 Ollama from host curl http://<OLLAMA_IP>:11434/api/tags Response < 100ms
3.1.3 Gateway from host curl http://<GATEWAY_IP>:18789/api/status Response < 100ms
3.1.4 Qdrant from container docker exec openclaw curl <QDRANT_URL> Same as host
3.1.5 DNS resolution nslookup <QDRANT_IP> Resolves or fails fast
3.1.6 Firewall ports nc -zv <QDRANT_IP> 6333 "succeeded"

3.2 URL Validation

# Check Pattern Example
3.2.1 Qdrant URL format http://IP:PORT http://localhost:6333
3.2.2 Ollama URL format http://IP:PORT http://localhost:11434
3.2.3 No trailing slash `sed 's /$
3.2.4 No protocol mismatch Check for https:// when should be http:// Match service config

4. Installation Validation

4.1 Installer Execution

# Check What to Verify Expected Result
4.1.1 v1 detected Installer warning shown v1 components listed
4.1.2 User acknowledgment "Continue at your own risk?" Y/N prompt
4.1.3 Mode selection Full vs Simple User choice recorded
4.1.4 All prompts answered No skipped prompts Complete config
4.1.5 Go-back navigation Can navigate between prompts Works correctly
4.1.6 Custom input Can enter custom values Accepted and validated
4.1.7 Summary displayed Final config shown All fields present
4.1.8 Confirmation User confirms install Proceeds or aborts

4.2 Config File Generation

# File Location Validation Critical
4.2.1 curator_config.json tr-continuous/ Valid JSON, all fields Yes
4.2.2 curator_timer.py tr-continuous/ Python syntax valid Yes
4.2.3 content_cleaner.py tr-continuous/ Python syntax valid ⚠️ Medium
4.2.4 curator-prompt.md tr-continuous/ File exists, readable Yes
4.2.5 push-all.sh Root Executable, valid bash ⚠️ Medium
4.2.6 plugin-config.json Root (full mode) Valid JSON Yes

4.3 Config File Content Validation

# Field Type Validation Rule Example
4.3.1 timer_minutes integer > 0 && < 1440 5, 30, 60
4.3.2 batch_size integer > 0 && <= 1000 50, 100, 500
4.3.3 user_id string non-empty "rob"
4.3.4 source_collection string non-empty "memories_tr"
4.3.5 target_collection string non-empty "gems_tr"
4.3.6 embedding_model string in Ollama model list "mxbai-embed-large"
4.3.7 curator_model string in Ollama model list "qwen3:30b-a3b-instruct"
4.3.8 qdrant_url URL reachable "http://localhost:6333"
4.3.9 ollama_url URL reachable "http://localhost:11434"
4.3.10 max_recall_results integer > 0 && <= 20 2, 5, 10
4.3.11 min_recall_score float >= 0.0 && <= 1.0 0.7, 0.85

5. Post-Installation Runtime

5.1 Watcher Service

# Check Command Expected Result Critical
5.1.1 Service file exists ls /etc/systemd/system/mem-qdrant-watcher.service File present Yes
5.1.2 Service loaded systemctl daemon-reload No errors Yes
5.1.3 Service enabled systemctl is-enabled mem-qdrant-watcher enabled Yes
5.1.4 Service running systemctl is-active mem-qdrant-watcher active Yes
5.1.5 Process running ps aux | grep "realtime_qdrant_watcher" Process found Yes
5.1.6 Logs accessible sudo journalctl -u mem-qdrant-watcher -n 5 Entries shown Yes
5.1.7 No errors sudo journalctl -u mem-qdrant-watcher | grep -i error Empty or minimal Yes
5.1.8 PID file Check if PID tracking enabled File exists No
5.1.9 Restart on crash systemctl cat mem-qdrant-watcher | grep Restart Restart=always ⚠️ Medium

5.2 Timer Curator (Full Mode)

# Check Command Expected Result Critical
5.2.1 Cron entry exists crontab -l | grep "curator_timer.py" Entry found Yes
5.2.2 Cron syntax valid crontab -l No syntax errors Yes
5.2.3 Timer matches config Compare crontab vs config Match Yes
5.2.4 Log file created ls /var/log/true-recall-timer.log File exists Yes
5.2.5 Log writable touch /var/log/true-recall-timer.log No permission error Yes
5.2.6 Log rotation ls /var/log/true-recall-timer.log* Rotated files No
5.2.7 First run completed tail /var/log/true-recall-timer.log "Completed" or similar Yes
5.2.8 No Python errors grep -i "traceback|error" /var/log/true-recall-timer.log Empty Yes
5.2.9 Process not stuck pgrep -f "curator_timer.py" | wc -l 0 or 1 (not many) Yes

5.3 Collection Creation

# Check Command Expected Result Critical
5.3.1 memories_tr exists curl <QDRANT_URL>/collections/memories_tr HTTP 200 Yes
5.3.2 gems_tr exists (full) curl <QDRANT_URL>/collections/gems_tr HTTP 200 Yes
5.3.3 Collection config curl <QDRANT_URL>/collections/memories_tr Correct vector size Yes
5.3.4 Embedding size Check collection config 768 (mxbai) or 1024 (snowflake) Yes
5.3.5 Distance metric Check collection config cosine Yes
5.3.6 Index created Query points Returns results Yes
5.3.7 Can write points Test insert HTTP 200 Yes
5.3.8 Can read points Test query Returns points Yes
5.3.9 Can scroll Scroll API Returns batch Yes
5.3.10 Payload indexing Check index config Fields indexed ⚠️ Medium

6. Data Integrity & Quality

6.1 Memory Capture

# Check Method Expected Result
6.1.1 Watcher capturing Send message, check memories_tr Point added within 1s
6.1.2 Session ID stored Query point payload session_id present
6.1.3 Timestamp valid Query point payload ISO 8601 format
6.1.4 Content not empty Query point payload text non-empty
6.1.5 User ID set Query point payload user_id matches config
6.1.6 curated flag Query point payload curated: false on new
6.1.7 Embedding generated Query point vector Vector non-null
6.1.8 Vector correct size Check dimensions 768 or 1024

6.2 Gem Curation

# Check Method Expected Result
6.2.1 Gems being created Check gems_tr count Increasing over time
6.2.2 curated flag updated Query memories_tr curated: true after process
6.2.3 Gem has 11 fields Query gem payload All fields present
6.2.4 importance valid Check gem payload "high", "medium", or "low" (not "3")
6.2.5 confidence valid Check gem payload Float 0.0-1.0 (not "0.7")
6.2.6 gem field valid Check gem payload Not starting with "User confirmed"
6.2.7 categories valid Check gem payload Non-empty array
6.2.8 snippet accurate Check gem payload Matches source turns
6.2.9 timestamp valid Check gem payload ISO 8601 from source turn
6.2.10 context richness Review gem manually 2-3 explanatory sentences

6.3 Duplication Check

# Check Method Expected Result
6.3.1 Similarity score Search similar gems < 0.85 for unique
6.3.2 24h window check Query recent gems No near-duplicates
6.3.3 Duplicate rate Sample 100 gems < 10% duplicates

7. Security & Privacy

7.1 Pre-Commit Checks

# Check Command What to Look For Severity
7.1.1 Internal IPs grep -rE "(10\.[0-9]+\.[0-9]+\.[0-9]+|192\.168\.[0-9]+\.[0-9]+)" Private network addresses 🔴 Critical
7.1.2 Passwords/Tokens `grep -riE "(password token secret
7.1.3 SSH Keys `grep -rE "(BEGIN.*PRIVATE KEY ssh-rsa ssh-ed25519)"`
7.1.4 Absolute Paths `grep -rE "(/root/ /home/[a-z]+/ .ssh/)"`
7.1.5 Timestamps grep -rE "[0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}" Non-project dates 🟡 Medium
7.1.6 Usernames grep -riE "user: [a-z]+" System usernames 🟡 Medium
7.1.7 Email addresses grep -rE "[a-z]+@[a-z]+\.[a-z]+" Personal emails 🟡 Medium
7.1.8 URLs with credentials grep -rE "https?://[^:]+:[^@]+@" Embedded auth 🔴 Critical
7.1.9 Database credentials grep -riE "(mongodb://|postgres://|mysql://)" Connection strings 🔴 Critical
7.1.10 AWS/Azure/GCP keys `grep -rE "(AKIA[0-9A-Z]{16}|azure gcp)"` Cloud credentials

7.2 Placeholder Replacements

# Found Replace With
7.2.1 10.0.0.x <QDRANT_IP> or <OLLAMA_IP>
7.2.2 192.168.x.x <LOCAL_NETWORK_IP>
7.2.3 172.16-31.x.x <PRIVATE_IP>
7.2.4 /root/.openclaw/ ~/.openclaw/ or <OPENCLAW_PATH>
7.2.5 /home/username/ ~/ or <USER_HOME>
7.2.6 Real tokens <TOKEN> or <API_KEY>
7.2.7 Specific dates <DATE> or <TIMESTAMP>
7.2.8 Real passwords <PASSWORD>
7.2.9 Hostnames <HOSTNAME> if internal

7.3 Files to Review Before Commit

  • All .py files
  • All .md files
  • All .json files
  • All .sh files
  • All .yaml / .yml files
  • All .toml files
  • Any log files (should be in .gitignore)
  • Any config files with paths
  • .env files (should be in .gitignore)
  • Dockerfiles
  • Makefile / justfile

8. Performance & Scaling

8.1 Timing Benchmarks

# Operation Target Time Max Acceptable
8.1.1 Watcher capture < 100ms 500ms
8.1.2 Embedding generation < 50ms 200ms
8.1.3 Qdrant upsert < 50ms 100ms
8.1.4 Curator per batch (4b) < 30s 60s
8.1.5 Curator per batch (30b) < 5s 15s
8.1.6 Plugin recall query < 100ms 500ms
8.1.7 Memory search < 200ms 1s
8.1.8 Log write < 10ms 50ms

8.2 Resource Usage

# Check Command Expected
8.2.1 Watcher CPU htop or ps < 5% idle
8.2.2 Watcher RAM ps aux < 100MB
8.2.3 Curator CPU During run Spike to 100% OK
8.2.4 Curator RAM During run < 2GB
8.2.5 Qdrant memory curl <QDRANT_URL>/metrics Depends on data
8.2.6 Disk usage df -h > 10% free
8.2.7 Log size du -sh /var/log/true-recall* Rotate if > 100MB
8.2.8 Collection size Check Qdrant metrics Monitor growth

8.3 Scaling Considerations

# Check Threshold Action
8.3.1 memories_tr points > 100,000 Consider indexing optimization
8.3.2 gems_tr points > 10,000 Monitor recall performance
8.3.3 Daily memories > 1,000 Increase batch size
8.3.4 Curator runtime > 10 min Decrease timer interval
8.3.5 Qdrant memory > 80% Scale Qdrant or archive old data
8.3.6 Log rotation Weekly Verify logrotate working

9. Git & Backup

9.1 Multi-Remote Configuration

# Check Command Expected
9.1.1 Gitea remote git remote get-url gitea http://10.0.0.61:3000/...
9.1.2 GitLab remote git remote get-url gitlab https://gitlab.com/...
9.1.3 GitHub remote git remote get-url github https://github.com/...
9.1.4 All remotes accessible git fetch --all No errors
9.1.5 push-all.sh exists ls push-all.sh File present
9.1.6 push-all.sh executable test -x push-all.sh True
9.1.7 GitLab token valid curl -H "PRIVATE-TOKEN:..." gitlab.com/api/v4/user Returns user
9.1.8 GitHub auth (if active) ssh -T git@github.com "successfully authenticated"

9.2 Backup Validation

# Check Method Expected
9.2.1 Code backup git push gitea Success
9.2.2 Code backup git push gitlab Success
9.2.3 Config backup Copy to deb2 Files match
9.2.4 Qdrant backup Export collections Can restore
9.2.5 Last backup time Check deb2 timestamp < 24 hours
9.2.6 Backup integrity Hash comparison Match

10. Troubleshooting

10.1 Common Issues & Fixes

Symptom Diagnostic Command Likely Cause Fix
Qdrant unreachable curl http://<QDRANT_IP>:6333 Not running sudo systemctl start qdrant
Ollama unreachable curl http://<OLLAMA_IP>:11434/api/tags Not running ollama serve
Model not found ollama list Not pulled ollama pull <model>
Watcher not running systemctl status mem-qdrant-watcher Service stopped sudo systemctl start mem-qdrant-watcher
No gems created tail /var/log/true-recall-timer.log Curator error Check logs, fix config
Permission denied ls -la Wrong ownership sudo chown user:user
Cron not running sudo systemctl status cron Service stopped sudo systemctl start cron
Plugin not loading openclaw plugins list Not enabled openclaw config set plugins.entries.memory-qdrant.enabled true
High memory usage ps aux Leak or large batch Restart services, reduce batch size
Slow curator Check Ollama GPU usage CPU fallback Ensure GPU available, check model quantization

10.2 Log Locations

Service Log Path Check
Watcher sudo journalctl -u mem-qdrant-watcher journalctl -u mem-qdrant-watcher -f
Curator /var/log/true-recall-timer.log tail -f /var/log/true-recall-timer.log
Qdrant Qdrant stdout/systemd sudo journalctl -u qdrant
Ollama Ollama stdout journalctl -u ollama or stdout
OpenClaw Gateway sudo journalctl -u openclaw-gateway journalctl -u openclaw-gateway -f
Git sync Manual script output Run push-all.sh

10.3 Health Check Script

#!/bin/bash
# health-check.sh - Quick validation script

echo "=== TrueRecall v2 Health Check ==="

# Check Qdrant
if curl -s http://localhost:6333/healthz > /dev/null; then
    echo "✅ Qdrant: OK"
else
    echo "❌ Qdrant: UNREACHABLE"
fi

# Check Ollama
if curl -s http://localhost:11434/api/tags > /dev/null; then
    echo "✅ Ollama: OK"
else
    echo "❌ Ollama: UNREACHABLE"
fi

# Check Watcher
if systemctl is-active mem-qdrant-watcher > /dev/null; then
    echo "✅ Watcher: RUNNING"
else
    echo "❌ Watcher: STOPPED"
fi

# Check collections
MEM_COUNT=$(curl -s http://localhost:6333/collections/memories_tr | jq -r '.result.points_count // 0')
GEM_COUNT=$(curl -s http://localhost:6333/collections/gems_tr | jq -r '.result.points_count // 0')
echo "📊 memories_tr: $MEM_COUNT points"
echo "📊 gems_tr: $GEM_COUNT points"

echo "=== Done ==="

11. Quick Reference

11.1 Validation Commands

# Check all services
openclaw status && \
curl -s http://localhost:6333 && \
curl -s http://localhost:11434/api/tags && \
systemctl is-active mem-qdrant-watcher

# Check collections
curl -s http://localhost:6333/collections/memories_tr | jq .result.points_count
curl -s http://localhost:6333/collections/gems_tr | jq .result.points_count

# Check watcher logs
sudo journalctl -u mem-qdrant-watcher -n 10 --no-pager

# Check curator logs
tail -20 /var/log/true-recall-timer.log

# Check cron
crontab -l | grep true-recall

# Check plugin
openclaw config get plugins.entries.memory-qdrant

# Push to all remotes
./push-all.sh

12. Known Issues & Solutions (Discovered During Deployment)

12.1 Gem Quality Issues

Issue Cause Solution Status
Malformed importance Model outputs 3 instead of "high" Strengthen prompt: "MUST be string, NEVER 1/2/3" Fixed in curator-prompt.md
Malformed confidence Model outputs "0.7" instead of 0.7 Strengthen prompt: "MUST be float, NEVER integers" Fixed in curator-prompt.md
"User confirmed..." pattern Model wraps insights in passive voice Add rule: "NEVER start with 'User confirmed/asked/said'" Fixed in curator-prompt.md
Duplicate gems Same decisions extracted multiple times Add deduplication check (similarity >0.85 = skip) Documented in README
Low-quality 4b gems 4b model lacks nuance Use 30b curator for production Documented in README

Prevention: See Section 6.3 (Duplication Check) and curator-prompt.md "Critical Validation Rules"

12.2 OpenClaw UI Issues

Issue Cause Workaround Status
Text invisible after compaction WebSocket sync during history replacement Wait 2-3s or hard refresh (Ctrl+Shift+R) ⚠️ Known limitation
Messages out of order UI state gets briefly desynced Hard refresh if stuck ⚠️ Known limitation

Note: These are OpenClaw Control UI limitations, not fixable from TrueRecall side.

12.3 v1 Migration Issues

Issue Cause Solution Status
v1 conflicts Both v1 and v2 running simultaneously Remove v1 completely before installing v2 Installer detects v1 and warns
Duplicate cron jobs v1 curator still running Remove v1 cron entries Documented in installer
Collection naming v1 uses memories, v2 uses memories_tr Collections are separate, no data conflict By design

Prevention: Installer now detects v1 and requires acknowledgment before proceeding.

12.4 Repository & Security Issues

Issue Cause Solution Status
Private IPs in code Hardcoded 10.0.0.x addresses Replace with placeholders (<QDRANT_IP>, etc.) Fixed in main_projects
Absolute paths /root/, /home/n8n/ in docs Replace with ~/ or placeholders Fixed in main_projects
Backup files committed *.bak, *.neuralstream.bak Removed + added .gitignore Fixed
Session notes in repo session.md with timestamps Removed from git, kept local Fixed
Debug files committed debug_curator.py, test_*.py Removed + .gitignore Fixed
Repo visibility Initially set to private (should have asked) Changed to public for sharing Fixed

Prevention: See Section 7 (Security & Privacy Review) and .gitignore

12.5 Multi-Remote Git Issues

Issue Cause Solution Status
GitHub blocked Account issues Skip GitHub in push-all.sh Configured
Token in remote URL Security risk Remove after push, use credential helper Fixed
Remote naming origin vs github Renamed to github for clarity Fixed

12.6 Model-Specific Issues

Issue Cause Solution Status
4b JSON reliability Smaller model struggles with JSON output Add JSON fallback/parsing retry Documented
30b speed 30b slower than 4b Acceptable for quality (~3s vs ~10s) Documented
Embedding model choice snowflake vs mxbai mxbai = higher accuracy, snowflake = faster Documented

12.7 Checklist for Future Deployments

Before releasing/sharing:

  • Run all commands in Section 7.1 (Security Checks)
  • Verify no *.bak files exist: find . -name "*.bak"
  • Verify no __pycache__ committed
  • Verify no session-specific timestamps in markdown
  • Verify no debug/test files in repo
  • Check .gitignore is comprehensive
  • Test installer in dry-run mode
  • Verify v1 detection works
  • Check all remotes push successfully
  • Verify public repo has no sensitive data

13. Sign-off Checklist

Section Status Date Checked By
Pre-install checks
System dependencies
Network connectivity
Installation validation
Watcher service
Timer curator (if full)
Collection creation
First curator run
Data quality
Security review
Multi-remote git
Backup verified
Known issues reviewed

Final Sign-off: _______________
Date: _______________
Version: v2.0