diff --git a/README.md b/README.md index 1600a33..120db82 100644 --- a/README.md +++ b/README.md @@ -65,6 +65,19 @@ After: Watching current session (93dc32bf... from Feb 25) ✅ --- +## Needed Improvements + +| Issue | Description | Priority | +|-------|-------------|----------| +| **Semantic Deduplication** | No dedup between similar gems. Same fact phrased differently creates multiple gems. Need semantic similarity check before storage. | High | +| **Search Result Deduplication** | Similar gems both above threshold are both injected, causing redundancy. Need filter to remove near-duplicates from results. | Medium | +| **Gem Quality Scoring** | No quality metric. Some extracted gems may be low value. Need LLM-based quality scoring. | Medium | +| **Temporal Decay** | All gems treated equally regardless of age. Should weight recent gems higher. | Low | +| **Gem Merging/Updating** | When user changes preference, old gem still exists. Need mechanism to update/contradict old gems. | Low | +| **Importance Calibration** | All curator gems marked "medium" importance. Should dynamically assign based on significance. | Low | + +--- + ## Overview TrueRecall v2 extracts "gems" (key insights) from conversations and injects them as context. It consists of three layers: diff --git a/audit_checklist.md b/audit_checklist.md index f200b10..7ec12bf 100644 --- a/audit_checklist.md +++ b/audit_checklist.md @@ -1,6 +1,6 @@ -# TrueRecall v2 - Master Audit Checklist (GIT/PUBLIC) +# TrueRecall v2 - Master Audit Checklist (LOCAL) -**For:** `.git_projects/true-recall-v2/` (Sanitized Public Directory) +**For:** `.local_projects/true-recall-v2/` (Working/Development Directory) **Version:** 2.2 **Last Updated:** 2026-02-25 10:07 CST @@ -8,7 +8,7 @@ ## Overview -This checklist validates the **git/public directory** is properly sanitized with placeholders, no credentials, and ready for public release. Use this before every git push. +This checklist validates the **local working directory** with real IPs, paths, and credentials. Use this for development, debugging, and local testing. --- @@ -19,374 +19,282 @@ This checklist validates the **git/public directory** is properly sanitized with | Embedding model mismatch | ✅ Fixed | Changed curator to `snowflake-arctic-embed2` | | Gems had no vectors | ✅ Fixed | Updated `store_gem()` to use `text` field | | JSON parsing errors | ✅ Fixed | Simplified extraction prompt | -| Watcher stuck on old session | ✅ **Fixed** | Restarted watcher service | -| Plugin capture 0 exchanges | ✅ **Fixed** | Added `extractMessageText()` for array content | -| Plugin exchanges working | ✅ **Verified** | 9 exchanges extracted per session | +| Watcher stuck on old session | ✅ **Fixed 12:22** | Restarted watcher service | +| Plugin capture 0 exchanges | ✅ **Fixed 12:34** | Added `extractMessageText()` for array content | +| Plugin exchanges working | ✅ **Verified 12:41** | 9 exchanges extracted per session | + +### Needed Improvements + +| Issue | Description | Priority | +|-------|-------------|----------| +| **Semantic Deduplication** | No dedup between similar gems. Same fact phrased differently creates multiple gems. | High | +| **Search Result Deduplication** | Similar gems both injected, causing redundancy. | Medium | +| **Gem Quality Scoring** | No quality metric for gems. | Medium | +| **Temporal Decay** | All gems treated equally regardless of age. | Low | +| **Gem Merging/Updating** | Old gems not updated when preferences change. | Low | +| **Importance Calibration** | All curator gems marked "medium" importance. | Low | --- -## SECTION 1: Pre-Push Security Checks +## SECTION 1: System Requirements -### 1.1 Critical Security Scan +### 1.1 Python Environment | # | Check | Command | Expected | Status | |---|-------|---------|----------|--------| -| 1.1.1 | No hardcoded IPs | `grep -rE "10\.[0-9]+\.[0-9]+\.[0-9]+" --include="*"` | 0 results | ☐ | -| 1.1.2 | No 192.168.x.x | `grep -rE "192\.168\.[0-9]+\.[0-9]+" --include="*"` | 0 results | ☐ | -| 1.1.3 | No 172.16-31.x.x | `grep -rE "172\.(1[6-9]|2[0-9]|3[01])\.[0-9]+\.[0-9]+" --include="*"` | 0 results | ☐ | -| 1.1.4 | No localhost IPs | `grep -rE "127\.0\.0\.[0-9]+" --include="*"` | 0 results | ☐ | -| 1.1.5 | No IPv6 locals | `grep -rE "\[?::1\]?" --include="*"` | 0 results | ☐ | +| 1.1.1 | Python version | `python3 --version` | 3.8+ | ☐ | +| 1.1.2 | pip available | `pip3 --version` | Working | ☐ | +| 1.1.3 | curl available | `curl --version` | Working | ☐ | +| 1.1.4 | jq available | `jq --version` | Working | ☐ | -### 1.2 Credentials Scan +### 1.2 Network Services | # | Check | Command | Expected | Status | |---|-------|---------|----------|--------| -| 1.2.1 | No passwords | `grep -ri "password" --include="*.py" --include="*.md" --include="*.sh"` | 0 results | ☐ | -| 1.2.2 | No tokens | `grep -ri "token" --include="*.py" --include="*.md" --include="*.json"` | 0 results | ☐ | -| 1.2.3 | No API keys | `grep -riE "api[_-]?key|apikey" --include="*"` | 0 results | ☐ | -| 1.2.4 | No secrets | `grep -ri "secret" --include="*.py" --include="*.md"` | 0 results | ☐ | -| 1.2.5 | No private keys | `grep -ri "private.*key\|privkey" --include="*"` | 0 results | ☐ | -| 1.2.6 | No auth strings | `grep -riE "auth[^o]" --include="*.py" --include="*.json"` | 0 results | ☐ | +| 1.2.1 | Qdrant reachable | `curl -s http://:6333` | Returns version | ☐ | +| 1.2.2 | Ollama reachable | `curl -s http://:11434/api/tags` | Returns models | ☐ | +| 1.2.3 | Redis reachable | `redis-cli -h ping` | PONG | ☐ | +| 1.2.4 | Kokoro reachable | `curl -s http://:8880` | 200 OK | ☐ | -### 1.3 .git/config Security - CRITICAL +### 1.3 OpenClaw | # | Check | Command | Expected | Status | |---|-------|---------|----------|--------| -| 1.3.1 | No tokens in URLs | `grep "url = " .git/config` | No `user:token@` pattern | ☐ | -| 1.3.2 | No HTTP auth | `grep "url = " .git/config | grep -v "^http://[^/]*$"` | Clean URLs | ☐ | -| 1.3.3 | HTTPS remotes | `grep "url = " .git/config` | All HTTPS or SSH | ☐ | -| 1.3.4 | Remote sanity | `git remote -v` | 2-3 remotes, no tokens | ☐ | -| **1.3.5** | **⚠️ NO TOKENS IN CREDENTIAL HELPER** | `grep -E "(password|token|ghp_|github_pat)" .git/config` | **MUST BE 0** | ☐ | -| **1.3.6** | **⚠️ NO CREDENTIAL HELPER WITH SECRETS** | `cat .git/config | grep -A5 "\[credential\]"` | **NO HARDCODED PASSWORDS** | ☐ | - -**CRITICAL WARNING:** Kimi has accidentally pushed tokens TWICE before. **ALWAYS** verify 1.3.5 and 1.3.6 before pushing! - -### 1.4 File Scan - -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 1.4.1 | No .env files | 0 .env files | ☐ | -| 1.4.2 | No .pem files | 0 .pem files | ☐ | -| 1.4.3 | No .key files | 0 .key files | ☐ | -| 1.4.4 | No id_rsa files | 0 id_rsa files | ☐ | -| 1.4.5 | No .p12 files | 0 .p12 files | ☐ | -| 1.4.6 | No .pfx files | 0 .pfx files | ☐ | +| 1.3.1 | Gateway status | `openclaw gateway status` | Active | ☐ | +| 1.3.2 | Config valid | `openclaw doctor` | No errors | ☐ | +| 1.3.3 | Plugin loaded | `openclaw status | grep memory-qdrant` | Enabled | ☐ | --- -## SECTION 2: Placeholder Verification +## SECTION 2: Project Files (Local) -### 2.1 IP Placeholders +### 2.1 Core Files Exist -| # | Placeholder | Used For | Found? | Status | -|---|-------------|----------|--------|--------| -| 2.1.1 | `` | Qdrant endpoint | ☐ | ☐ | -| 2.1.2 | `` | Ollama endpoint | ☐ | ☐ | -| 2.1.3 | `` | Redis endpoint | ☐ | ☐ | -| 2.1.4 | `` | Gitea server | ☐ | ☐ | -| 2.1.5 | `` | OpenClaw gateway | ☐ | ☐ | +| # | File | Path | Status | +|---|------|------|--------| +| 2.1.1 | README.md | `.local_projects/true-recall-v2/README.md` | ☐ | +| 2.1.2 | session.md | `.local_projects/true-recall-v2/session.md` | ☐ | +| 2.1.3 | checklist.md | `.local_projects/true-recall-v2/checklist.md` | ☐ | +| 2.1.4 | curator-prompt.md | `.local_projects/true-recall-v2/curator-prompt.md` | ☐ | -### 2.2 Path Placeholders +### 2.2 Scripts Exist -| # | Placeholder | Used For | Found? | Status | -|---|-------------|----------|--------|--------| -| 2.2.1 | `~/` | Home directory | ☐ | ☐ | -| 2.2.2 | `` | OpenClaw install | ☐ | ☐ | -| 2.2.3 | `` | User home | ☐ | ☐ | -| 2.2.4 | `` | systemd location | ☐ | ☐ | +| # | File | Path | Status | +|---|------|------|--------| +| 2.2.1 | curator_timer.py | `.local_projects/true-recall-v2/tr-continuous/curator_timer.py` | ☐ | +| 2.2.2 | curator_config.json | `.local_projects/true-recall-v2/tr-continuous/curator_config.json` | ☐ | +| 2.2.3 | install.py | `.local_projects/true-recall-v2/install.py` | ☐ | -### 2.3 Config Placeholders +### 2.3 Watcher Files -| # | Placeholder | Used For | Found? | Status | -|---|-------------|----------|--------|--------| -| 2.3.1 | `` | API key example | ☐ | ☐ | -| 2.3.2 | `` | Token example | ☐ | ☐ | -| 2.3.3 | `` | Password example | ☐ | ☐ | -| 2.3.4 | `` | Date example | ☐ | ☐ | -| 2.3.5 | `` | Timestamp example | ☐ | ☐ | +| # | File | Path | Status | +|---|------|------|--------| +| 2.3.1 | realtime_qdrant_watcher.py | `skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` | ☐ | +| 2.3.2 | mem-qdrant-watcher.service | `/etc/systemd/system/mem-qdrant-watcher.service` | ☐ | --- -## SECTION 3: File Completeness +## SECTION 3: Configuration Validation -### 3.1 Required Files Present +### 3.1 curator_config.json -| # | File | Purpose | Status | -|---|------|---------|--------| -| 3.1.1 | `README.md` | Main documentation | ☐ | -| 3.1.2 | `session.md` | Session notes | ☐ | -| 3.1.3 | `checklist.md` | Installation checklist | ☐ | -| 3.1.4 | `curator-prompt.md` | Curation prompt | ☐ | -| 3.1.5 | `install.py` | Installation script | ☐ | -| 3.1.6 | `push-all.sh` | Push script | ☐ | +| # | Setting | Key | Expected | Status | +|---|---------|-----|----------|--------| +| 3.1.1 | Timer minutes | `timer_minutes` | 5 | ☐ | +| 3.1.2 | Batch size | `max_batch_size` | 100 | ☐ | +| 3.1.3 | User ID | `user_id` | rob | ☐ | +| 3.1.4 | Source collection | `source_collection` | memories_tr | ☐ | +| 3.1.5 | Target collection | `target_collection` | gems_tr | ☐ | -### 3.2 Scripts Directory +### 3.2 openclaw.json Plugin Config -| # | File | Purpose | Status | -|---|------|---------|--------| -| 3.2.1 | `tr-continuous/curator_timer.py` | Timer curator | ☐ | -| 3.2.2 | `tr-continuous/curator_config.json` | Curator config | ☐ | - -### 3.3 No Local-Only Files - -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 3.3.1 | No debug_curator.py | Not in git | ☐ | -| 3.3.2 | No test_curator.py | Not in git | ☐ | -| 3.3.3 | No migrate_*.py | Not in git | ☐ | -| 3.3.4 | No tr-daily/ | Not in git (archived) | ☐ | -| 3.3.5 | No tr-compact/ | Not in git (concept) | ☐ | +| # | Setting | Key | Expected | Status | +|---|---------|-----|----------|--------| +| 3.2.1 | Qdrant URL | `qdrantUrl` | http://:6333 | ☐ | +| 3.2.2 | Ollama URL | `ollamaUrl` | http://:11434 | ☐ | +| 3.2.3 | Embedding model | `embeddingModel` | snowflake-arctic-embed2 | ☐ | +| 3.2.4 | Capture collection | `captureCollection` | memories_tr | ☐ | +| 3.2.5 | Recall collection | `collectionName` | gems_tr | ☐ | +| 3.2.6 | Auto capture | `autoCapture` | true | ☐ | +| 3.2.7 | Auto recall | `autoRecall` | true | ☐ | --- -## SECTION 4: Script Validation +## SECTION 4: Qdrant Collections -### 4.1 curator_timer.py - -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 4.1.1 | No hardcoded IPs | Uses env vars | ☐ | -| 4.1.2 | No absolute paths | Uses `~/` | ☐ | -| 4.1.3 | Syntax valid | `python3 -m py_compile` passes | ☐ | -| 4.1.4 | Executable bit | `chmod +x` set | ☐ | -| 4.1.5 | Uses placeholders | ``, `` | ☐ | - -### 4.2 install.py - -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 4.2.1 | No hardcoded IPs | Uses prompts | ☐ | -| 4.2.2 | No absolute paths | Uses defaults | ☐ | -| 4.2.3 | Syntax valid | `python3 -m py_compile` passes | ☐ | -| 4.2.4 | Interactive prompts | Asks for URLs | ☐ | - -### 4.3 push-all.sh - -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 4.3.1 | No hardcoded paths | Uses `$PWD` | ☐ | -| 4.3.2 | No tokens | Clean script | ☐ | -| 4.3.3 | Syntax valid | `bash -n` passes | ☐ | -| 4.3.4 | Executable bit | `chmod +x` set | ☐ | - ---- - -## SECTION 5: Documentation Quality - -### 5.1 README.md - -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 5.1.1 | Uses placeholders | ``, `` | ☐ | -| 5.1.2 | No hardcoded paths | `~/` not `/root/` | ☐ | -| 5.1.3 | Clear instructions | Step-by-step | ☐ | -| 5.1.4 | Config examples | Generic examples | ☐ | -| 5.1.5 | Troubleshooting | Common issues listed | ☐ | - -### 5.2 session.md - -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 5.2.1 | Uses placeholders | ``, `` | ☐ | -| 5.2.2 | No hardcoded paths | `~/` not `/root/` | ☐ | -| 5.2.3 | Current state | Up to date | ☐ | -| 5.2.4 | Validation commands | Generic commands | ☐ | - -### 5.3 checklist.md - -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 5.3.1 | Uses placeholders | ``, etc. | ☐ | -| 5.3.2 | Pre-install checks | Generic commands | ☐ | -| 5.3.3 | Post-install validation | Generic commands | ☐ | -| 5.3.4 | Troubleshooting | Common issues | ☐ | - -### 5.4 curator-prompt.md - -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 5.4.1 | Uses placeholders | `` | ☐ | -| 5.4.2 | No hardcoded IPs | Placeholders only | ☐ | -| 5.4.3 | Updated architecture | No Redis refs | ☐ | -| 5.4.4 | Correct collection | `memories_tr` not `kimi_memories` | ☐ | - ---- - -## SECTION 6: Git Hygiene - -### 6.1 Git Status +### 4.1 Collection Status | # | Check | Command | Expected | Status | |---|-------|---------|----------|--------| -| 6.1.1 | Clean working tree | `git status` | No uncommitted changes | ☐ | -| 6.1.2 | No untracked files | `git status` | 0 untracked or added | ☐ | -| 6.1.3 | Proper .gitignore | `cat .gitignore` | Blocks sensitive files | ☐ | -| 6.1.4 | No large files | `find . -size +10M` | 0 large files | ☐ | +| 4.1.1 | memories_tr exists | `curl -s http://:6333/collections/memories_tr | jq .result.status` | green | ☐ | +| 4.1.2 | gems_tr exists | `curl -s http://:6333/collections/gems_tr | jq .result.status` | green | ☐ | +| 4.1.3 | memories_tr points | `curl -s http://:6333/collections/memories_tr | jq .result.points_count` | 12000+ | ☐ | +| 4.1.4 | gems_tr points | `curl -s http://:6333/collections/gems_tr | jq .result.points_count` | 70+ | ☐ | -### 6.2 Commit Quality +### 4.2 Data Integrity -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 6.2.1 | Descriptive message | Clear summary | ☐ | -| 6.2.2 | Atomic changes | One feature per commit | ☐ | -| 6.2.3 | Signed (optional) | GPG signed | ☐ | - -### 6.3 Remote Configuration - -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 6.3.1 | GitHub remote | Configured | ☐ | -| 6.3.2 | Gitea remote | Configured | ☐ | -| 6.3.3 | GitLab remote | Configured | ☐ | -| 6.3.4 | All clean | No tokens in URLs | ☐ | +| # | Check | Command | Expected | Status | +|---|-------|---------|----------|--------| +| 4.2.1 | Uncurated count | Count `curated: false` | 1500+ | ☐ | +| 4.2.2 | Curated count | Count `curated: true` | 11000+ | ☐ | +| 4.2.3 | Can write points | Test insert | Success | ☐ | +| 4.2.4 | Can read points | Test query | Success | ☐ | --- -## SECTION 7: Error Prevention +## SECTION 5: Services -### 7.1 Common Mistakes +### 5.1 Watcher Service -| # | Mistake | Prevention | Check | Status | -|---|---------|------------|-------|--------| -| 7.1.1 | Forgetting to sanitize | Run this checklist | ☐ | ☐ | -| 7.1.2 | Leaving tokens | Scan with grep | ☐ | ☐ | -| 7.1.3 | Hardcoding IPs | Use placeholders | ☐ | ☐ | -| 7.1.4 | Absolute paths | Use `~/` | ☐ | ☐ | -| 7.1.5 | Local-only files | Check 3.3.1-3.3.5 | ☐ | ☐ | +| # | Check | Command | Expected | Status | +|---|-------|---------|----------|--------| +| 5.1.1 | Service loaded | `systemctl status mem-qdrant-watcher | grep Loaded` | loaded | ☐ | +| 5.1.2 | Service active | `systemctl is-active mem-qdrant-watcher` | active | ☐ | +| 5.1.3 | Service enabled | `systemctl is-enabled mem-qdrant-watcher` | enabled | ☐ | +| 5.1.4 | Process running | `pgrep -f realtime_qdrant_watcher` | PID exists | ☐ | +| 5.1.5 | Logs available | `journalctl -u mem-qdrant-watcher -n 5` | Recent entries | ☐ | -### 7.2 Pre-Push Checklist - MANDATORY +### 5.2 Timer Curator -| # | Step | Command | Status | -|---|------|---------|--------| -| **7.2.1** | **🔴 CHECK .git/config FOR TOKENS** | `grep -E "(password|token|ghp_|github_pat)" .git/config` | ☐ **MUST PASS** | -| **7.2.2** | **🔴 VERIFY NO CREDENTIAL HELPER SECRETS** | `cat .git/config | grep -A5 "\[credential\]"` | ☐ **MUST PASS** | -| 7.2.3 | Run security scan | Section 1.1-1.2 | ☐ | -| 7.2.4 | Verify placeholders | Section 2.1-2.3 | ☐ | -| 7.2.5 | Check file completeness | Section 3.1-3.3 | ☐ | -| 7.2.6 | Validate scripts | Section 4.1-4.3 | ☐ | -| 7.2.7 | Review docs | Section 5.1-5.4 | ☐ | -| 7.2.8 | Check git hygiene | Section 6.1-6.3 | ☐ | +| # | Check | Command | Expected | Status | +|---|-------|---------|----------|--------| +| 5.2.1 | Cron job exists | `crontab -l | grep true-recall` | Entry present | ☐ | +| 5.2.2 | Cron interval | Visual check | */5 * * * * | ☐ | +| 5.2.3 | Log file exists | `ls -la /var/log/true-recall-timer.log` | File exists | ☐ | +| 5.2.4 | Recent activity | `tail -5 /var/log/true-recall-timer.log` | Recent timestamp | ☐ | +| 5.2.5 | Script executable | `test -x curator_timer.py` | Yes | ☐ | --- -## SECTION 8: Function Verification (Generic) +## SECTION 6: Function Tests -### 8.1 Config Validity +### 6.1 Capture Test -| # | File | Check | Expected | Status | -|---|------|-------|----------|--------| -| 8.1.1 | `curator_config.json` | JSON syntax | Valid JSON | ☐ | -| 8.1.2 | `curator_config.json` | Required keys | All present | ☐ | -| 8.1.3 | `curator_config.json` | Value types | Correct types | ☐ | +| # | Step | Expected | Status | +|---|------|----------|--------| +| 6.1.1 | Send test message | Message captured | ☐ | +| 6.1.2 | Wait 10 seconds | Processing time | ☐ | +| 6.1.3 | Check memories_tr count | Increased by 2 | ☐ | +| 6.1.4 | Verify content | Content matches | ☐ | -### 8.2 Script Syntax +### 6.2 Curation Test -| # | File | Check | Command | Status | -|---|------|-------|---------|--------| -| 8.2.1 | `curator_timer.py` | Python syntax | `python3 -m py_compile` | ☐ | -| 8.2.2 | `install.py` | Python syntax | `python3 -m py_compile` | ☐ | -| 8.2.3 | `push-all.sh` | Bash syntax | `bash -n push-all.sh` | ☐ | +| # | Step | Expected | Status | +|---|------|----------|--------| +| 6.2.1 | Note uncurated count | Baseline | ☐ | +| 6.2.2 | Run curator manually | Completes | ☐ | +| 6.2.3 | Check gems_tr | New gems added | ☐ | +| 6.2.4 | Verify curated flag | Marked true | ☐ | -### 8.3 Documentation Links +### 6.3 Recall Test -| # | Check | Expected | Status | -|---|-------|----------|--------| -| 8.3.1 | Internal links valid | All `#section` work | ☐ | -| 8.3.2 | No broken references | No `TODO` or `FIXME` | ☐ | -| 8.3.3 | Consistent formatting | Same style throughout | ☐ | +| # | Step | Expected | Status | +|---|------|----------|--------| +| 6.3.1 | Start new conversation | Context loaded | ☐ | +| 6.3.2 | Send relevant query | Gems injected | ☐ | +| 6.3.3 | Verify injection | Context visible | ☐ | --- -## SECTION 9: Comparison with Local +## SECTION 7: Error Checks -### 9.1 Sync Status +### 7.1 Common Errors -| # | Check | Local | Git | Match? | +| # | Error | Check | Fix | Status | |---|-------|-------|-----|--------| -| 9.1.1 | README structure | Same | Same | ☐ | -| 9.1.2 | session structure | Same | Same | ☐ | -| 9.1.3 | checklist structure | Same | Same | ☐ | -| 9.1.4 | Config structure | Same | Same | ☐ | +| 7.1.1 | Qdrant unreachable | `curl http://:6333` | Start Qdrant | ☐ | +| 7.1.2 | Ollama unreachable | `curl http://:11434` | Start Ollama | ☐ | +| 7.1.3 | Watcher not running | `systemctl status mem-qdrant-watcher` | Restart service | ☐ | +| 7.1.4 | Curator not running | `tail /var/log/true-recall-timer.log` | Check cron | ☐ | +| 7.1.5 | No gems extracted | Check config.json | Verify model | ☐ | -### 9.2 Content Differences +### 7.2 Log Analysis -| # | Check | Local (Real) | Git (Placeholder) | Expected | -|---|-------|--------------|-------------------|----------| -| 9.2.1 | Qdrant IP | 10.0.0.40 | `` | ✅ | -| 9.2.2 | Ollama IP | 10.0.0.10 | `` | ✅ | -| 9.2.3 | Paths | /root/... | ~/... | ✅ | -| 9.2.4 | Usernames | rob | rob or generic | ✅ | +| # | Log | Location | Check For | Status | +|---|-----|----------|-----------|--------| +| 7.2.1 | Watcher log | `journalctl -u mem-qdrant-watcher` | Errors, crashes | ☐ | +| 7.2.2 | Curator log | `/var/log/true-recall-timer.log` | Failures, 0 gems | ☐ | +| 7.2.3 | OpenClaw log | `/tmp/openclaw/openclaw-*.log` | Plugin errors | ☐ | +| 7.2.4 | System log | `journalctl -n 50` | Service failures | ☐ | --- -## SECTION 10: Final Review +## SECTION 8: Security (Local - Expected) -### 10.1 Sign-Off +### 8.1 Private Info (Acceptable in Local) -| # | Reviewer | Date | Notes | Signature | -|---|----------|------|-------|-----------| -| 10.1.1 | Security scan | | | | -| 10.1.2 | Sanitization | | | | -| 10.1.3 | Functionality | | | | -| 10.1.4 | Documentation | | | | +| # | Item | Location | Expected | Status | +|---|------|----------|----------|--------| +| 8.1.1 | Private IPs | Scripts | 10.0.0.x | ✅ OK | +| 8.1.2 | Absolute paths | Scripts | /root/... | ✅ OK | +| 8.1.3 | Usernames | Config | rob | ✅ OK | +| 8.1.4 | Internal URLs | Config | http://10.0.0.x | ✅ OK | -### 10.2 Ready to Push - MANDATORY CHECKS +### 8.2 Credentials (Should NOT Be in Code) -| # | Check | Status | -|---|-------|--------| -| **10.2.1** | **🔴 .git/config contains NO tokens** (Section 1.3.5-1.3.6) | ☐ **MUST PASS** | -| **10.2.2** | **🔴 No credential helper with secrets** (Section 7.2.1-7.2.2) | ☐ **MUST PASS** | -| 10.2.3 | All Section 1 checks passed | ☐ | -| 10.2.4 | All Section 2 checks passed | ☐ | -| 10.2.5 | All Section 3 checks passed | ☐ | -| 10.2.6 | All Section 4 checks passed | ☐ | -| 10.2.7 | All Section 5 checks passed | ☐ | -| 10.2.8 | All Section 6 checks passed | ☐ | -| 10.2.9 | All Section 7 checks passed | ☐ | +| # | Check | Command | Expected | Status | +|---|-------|---------|----------|--------| +| 8.2.1 | No tokens in .py | `grep -r "token" *.py` | Only env vars | ☐ | +| 8.2.2 | No passwords | `grep -r "password" *.py` | None found | ☐ | +| 8.2.3 | No API keys | `grep -rE "[a-zA-Z0-9]{32,}" *.py` | None found | ☐ | +| 8.2.4 | .git/config clean | `cat .git/config | grep url` | No tokens | ☐ | -### 10.3 Push Command +--- + +## SECTION 9: Sync Check (Local vs Git) + +### 9.1 Compare Directories + +| # | Check | Command | Expected | Status | +|---|-------|---------|----------|--------| +| 9.1.1 | File count match | Compare `.local_projects/` vs `.git_projects/` | Similar | ☐ | +| 9.1.2 | Key files exist | README, session, checklist in both | Yes | ☐ | +| 9.1.3 | Scripts in git | curator_timer.py in git | Yes | ☐ | +| 9.1.4 | Config in git | curator_config.json in git | Yes | ☐ | + +### 9.2 Sanitization Verification + +| # | Check | Local | Git | Status | +|---|-------|-------|-----|--------| +| 9.2.1 | IPs in local | 10.0.0.x | ✅ Expected | - | +| 9.2.2 | IPs in git | Placeholders | ✅ Expected | - | +| 9.2.3 | Paths in local | /root/... | ✅ Expected | - | +| 9.2.4 | Paths in git | ~/... | ✅ Expected | - | + +--- + +## Sign-Off + +| Role | Name | Date | Signature | +|------|------|------|-----------| +| Auditor | | | | +| Developer | | | | +| Reviewer | | | | + +--- + +## Quick Commands Reference ```bash -# After all checks pass: -cd ~/.openclaw/workspace/.git_projects/true-recall-v2 -./push-all.sh "Your descriptive commit message" +# Check all services +systemctl status mem-qdrant-watcher +curl -s http://:6333/collections/memories_tr | jq .result.points_count +curl -s http://:6333/collections/gems_tr | jq .result.points_count +tail -20 /var/log/true-recall-timer.log + +# Check Qdrant for curated status +curl -s -X POST http://:6333/collections/memories_tr/points/count \ + -d '{"filter":{"must":[{"key":"curated","match":{"value":false}}]}}' + +# Manual curator run +cd ~/.openclaw/workspace/.local_projects/true-recall-v2/tr-continuous +python3 curator_timer.py --dry-run + +# Restart services +sudo systemctl restart mem-qdrant-watcher ``` --- -## Quick Reference: Security Scan Commands - -```bash -# Scan for private IPs -grep -rE "10\.[0-9]+\.[0-9]+\.[0-9]+" --include="*" -grep -rE "192\.168\.[0-9]+\.[0-9]+" --include="*" -grep -rE "172\.(1[6-9]|2[0-9]|3[01])\.[0-9]+\.[0-9]+" --include="*" - -# Scan for credentials -grep -ri "password\|token\|secret\|api.?key" --include="*" - -# Scan for absolute paths -grep -rE "/(root|home)/[a-z]+" --include="*" - -# Check .git/config -cat .git/config | grep url - -# Find sensitive files -find . -name "*.pem" -o -name "*.key" -o -name ".env*" -o -name "id_rsa" -``` - ---- - -## Emergency: Found Sensitive Data - -If you find sensitive data after pushing: - -1. **Immediately** revoke the exposed credential -2. Remove from git history: `git filter-branch` or BFG Repo-Cleaner -3. Force push to all remotes -4. Notify affected parties - ---- - -*This checklist is for GIT/PUBLIC directory validation only.* -*For local development checks, see `audit_checklist.md` in `.local_projects/true-recall-v2/`* +*This checklist is for LOCAL working directory validation only.* +*For git/public checks, see `audit_checklist.md` in `.git_projects/true-recall-v2/`* diff --git a/function_check.md b/function_check.md index 1b692a6..bf4b507 100644 --- a/function_check.md +++ b/function_check.md @@ -185,6 +185,17 @@ journalctl -u mem-qdrant-watcher -n 50 --no-pager | **gems_tr cleaned** | ✅ **Done 12:53** | Removed 5 meta-gems, kept 1 real gem | | **Gem format (1st person)** | ✅ **Fixed 13:15** | Changed from "User decided..." to "I decided..." for better query matching | +### Needed Improvements + +| Issue | Description | Priority | +|-------|-------------|----------| +| **Semantic Deduplication** | No dedup between similar gems. Same fact phrased differently creates multiple gems. | High | +| **Search Result Deduplication** | Similar gems both injected, causing redundancy. | Medium | +| **Gem Quality Scoring** | No quality metric for gems. | Medium | +| **Temporal Decay** | All gems treated equally regardless of age. | Low | +| **Gem Merging/Updating** | Old gems not updated when preferences change. | Low | +| **Importance Calibration** | All curator gems marked "medium" importance. | Low | + **Result:** Context injection now functional. Gems are embedded and searchable. Both watcher and plugin capture working. | Check | Date | Status | diff --git a/session.md b/session.md index 4175da6..5b3dddb 100644 --- a/session.md +++ b/session.md @@ -46,6 +46,19 @@ --- +## Needed Improvements + +| Issue | Description | Priority | +|-------|-------------|----------| +| **Semantic Deduplication** | No dedup between similar gems. Same fact phrased differently creates multiple gems. | High | +| **Search Result Deduplication** | Similar gems above threshold both injected, causing redundancy. | Medium | +| **Gem Quality Scoring** | No quality metric. Some gems may be low value. | Medium | +| **Temporal Decay** | All gems treated equally regardless of age. | Low | +| **Gem Merging/Updating** | When user changes preference, old gem still exists. | Low | +| **Importance Calibration** | All curator gems marked "medium". Should be dynamic. | Low | + +--- + ## Session End (18:09 CST) **Reason:** User starting new session