SpeedyFoxAi/true-recall-gems
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Final fixes: first-person gems, threshold 0.5, hidden context injection

Browse Source 
- Changed gem format from third-person to first-person for better query matching
- Lowered minRecallScore from 0.7 to 0.5
- Fixed context injection to use HTML comments (hidden from UI)
- Updated all documentation with today's fixes
This commit is contained in:
root
2026-02-25 13:30:13 -06:00
parent 87a390901d
commit 5950fdd09b
3 changed files with 147 additions and 107 deletions
Download Patch File Download Diff File Expand all files Collapse all files

142
function_check.md
View File

@@ -1,16 +1,19 @@
# TrueRecall v2 - Function Check (GENERIC)
# TrueRecall v2 - Function Check (LOCAL)
**Quick validation checklist for TrueRecall v2 setup**
**Quick validation checklist for OUR TrueRecall v2 setup**
**For:** Generic installation (sanitized)
**Version:** 2.2
**User:** rob
**Qdrant:** http://<QDRANT_IP>:6333
**Ollama:** http://<OLLAMA_IP>:11434
**Timer:** 5 minutes
**Working Dir:** ~/.openclaw/workspace/.local_projects/true-recall-v2
---
## Quick Status Check
```bash
cd ~/<PROJECT_PATH>/true-recall-v2
cd ~/.openclaw/workspace/.local_projects/true-recall-v2
```
---
@@ -19,12 +22,14 @@ cd ~/<PROJECT_PATH>/true-recall-v2
| Check | Command | Expected |
|-------|---------|----------|
| Project exists | `ls ~/<PROJECT_PATH>/true-recall-v2` | Files listed |
| Watcher script | `ls <SKILL_PATH>/qdrant-memory/scripts/realtime_qdrant_watcher.py` | File exists |
| Local project exists | `ls ~/.openclaw/workspace/.local_projects/true-recall-v2` | Files listed |
| Git project exists | `ls ~/.openclaw/workspace/.git_projects/true-recall-v2` | Files listed |
| Watcher script | `ls ~/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` | File exists |
**Paths:**
- Project: `~/<PROJECT_PATH>/true-recall-v2/`
- Watcher: `<SKILL_PATH>/qdrant-memory/scripts/realtime_qdrant_watcher.py`
**Our Paths:**
- Local: `~/.openclaw/workspace/.local_projects/true-recall-v2/`
- Git: `~/.openclaw/workspace/.git_projects/true-recall-v2/`
- Watcher: `~/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
- Systemd: `/etc/systemd/system/mem-qdrant-watcher.service`
---
@@ -35,13 +40,13 @@ cd ~/<PROJECT_PATH>/true-recall-v2
|-------|---------|----------|
| Watcher running | `systemctl is-active mem-qdrant-watcher` | `active` |
| Watcher enabled | `systemctl is-enabled mem-qdrant-watcher` | `enabled` |
| Cron job set | `crontab -l \| grep true-recall` | Cron entry present |
| Cron job set | `crontab -l \| grep true-recall` | `*/5 * * * *` entry |
**Service:**
**Our Service:**
- Service: `mem-qdrant-watcher.service`
- Status: `systemctl status mem-qdrant-watcher --no-pager`
- Logs: `journalctl -u mem-qdrant-watcher -n 20`
- Cron: Configured interval (e.g., `*/5 * * * *`)
- Cron: `*/5 * * * *` (every 5 minutes)
---
@@ -51,13 +56,13 @@ cd ~/<PROJECT_PATH>/true-recall-v2
|-------|---------|----------|
| memories_tr status | `curl -s http://<QDRANT_IP>:6333/collections/memories_tr \| jq .result.status` | `green` |
| gems_tr status | `curl -s http://<QDRANT_IP>:6333/collections/gems_tr \| jq .result.status` | `green` |
| memories_tr count | `curl -s http://<QDRANT_IP>:6333/collections/memories_tr \| jq .result.points_count` | `1000+` |
| gems_tr count | `curl -s http://<QDRANT_IP>:6333/collections/gems_tr \| jq .result.points_count` | `10+` |
| memories_tr count | `curl -s http://<QDRANT_IP>:6333/collections/memories_tr \| jq .result.points_count` | `12000+` |
| gems_tr count | `curl -s http://<QDRANT_IP>:6333/collections/gems_tr \| jq .result.points_count` | `70+` |
**Qdrant:**
- URL: `http://<QDRANT_IP>:6333`
**Our Qdrant:**
- URL: http://<QDRANT_IP>:6333
- Collections: memories_tr, gems_tr
- Embedding Model: Configured in openclaw.json
- Embedding Model: snowflake-arctic-embed2
---
@@ -65,14 +70,14 @@ cd ~/<PROJECT_PATH>/true-recall-v2
| Check | Command | Expected |
|-------|---------|----------|
| Uncurated count | See Section 7 | `Number of uncurated` |
| Curated count | See Section 7 | `Number of curated` |
| Curator config | `cat tr-continuous/curator_config.json` | Valid JSON |
| Uncurated count | See Section 7 | `1490` |
| Curated count | See Section 7 | `11239` |
| Curator config | `cat tr-continuous/curator_config.json` | `timer_minutes: 5` |
**Config:**
- Timer: Configured minutes
- Batch Size: Configured (e.g., 100)
- User ID: Your user ID
**Our Config:**
- Timer: 5 minutes (`*/5 * * * *`)
- Batch Size: 100
- User ID: rob
- Source: memories_tr
- Target: gems_tr
- Curator Log: `/var/log/true-recall-timer.log`
@@ -83,17 +88,17 @@ cd ~/<PROJECT_PATH>/true-recall-v2
| Step | Action | Check |
|------|--------|-------|
| 1 | Send a test message | Message received |
| 1 | Send a test message to Kimi | Message received |
| 2 | Wait 10 seconds | Allow processing |
| 3 | Check memories count increased | `curl -s http://<QDRANT_IP>:6333/collections/memories_tr \| jq .result.points_count` |
| 4 | Verify memory has user_id | `user_id: "<YOUR_USER_ID>"` in payload |
| 4 | Verify memory has user_id | `user_id: "rob"` in payload |
| 5 | Verify memory has curated=false | `curated: false` in payload |
**Watcher:**
- Script: `<SKILL_PATH>/qdrant-memory/scripts/realtime_qdrant_watcher.py`
- User ID: Configured (check openclaw.json)
**Our Watcher:**
- Script: `~/.openclaw/workspace/skills/qdrant-memory/scripts/realtime_qdrant_watcher.py`
- User ID: `rob`
- Collection: `memories_tr`
- Embeddings: Configured model
- Embeddings: `snowflake-arctic-embed2`
---
@@ -109,13 +114,15 @@ cd ~/<PROJECT_PATH>/true-recall-v2
---
## 7. Recall Test
## 7. Recall Test ✅ **WORKING**
| Step | Action | Check |
|------|--------|-------|
| 1 | Start new conversation | Context loads |
| 2 | Ask about previous topic | Gems injected |
| 3 | Verify context visible | Relevant memories appear |
| 3 | Verify context visible | ✅ **Score 0.587** - Working! |
**Verified 2026-02-25:** Context injection successfully returns relevant gems with similarity scores above threshold (0.5+).
---
@@ -123,9 +130,9 @@ cd ~/<PROJECT_PATH>/true-recall-v2
| Path | Check | Status |
|------|-------|--------|
| Watcher script | `<SKILL_PATH>/qdrant-memory/scripts/realtime_qdrant_watcher.py` | ☐ |
| Curator script | `<PROJECT_PATH>/true-recall-v2/tr-continuous/curator_timer.py` | ☐ |
| Config file | `<PROJECT_PATH>/true-recall-v2/tr-continuous/curator_config.json` | ☐ |
| Watcher script | `skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` | ☐ |
| Curator script | `.local_projects/true-recall-v2/tr-continuous/curator_timer.py` | ☐ |
| Config file | `.local_projects/true-recall-v2/tr-continuous/curator_config.json` | ☐ |
| Log file | `/var/log/true-recall-timer.log` | ☐ |
---
@@ -137,16 +144,16 @@ cd ~/<PROJECT_PATH>/true-recall-v2
systemctl status mem-qdrant-watcher --no-pager
tail -20 /var/log/true-recall-timer.log
# Check Qdrant collections
# Check Qdrant collections (Our Qdrant: <QDRANT_IP>:6333)
curl -s http://<QDRANT_IP>:6333/collections/memories_tr | jq '{status: .result.status, points: .result.points_count}'
curl -s http://<QDRANT_IP>:6333/collections/gems_tr | jq '{status: .result.status, points: .result.points_count}'
# Check uncurated memories
# Check uncurated memories (Our user_id: rob)
curl -s -X POST http://<QDRANT_IP>:6333/collections/memories_tr/points/count \
-d '{"filter":{"must":[{"key":"user_id","match":{"value":"<YOUR_USER_ID>"}},{"key":"curated","match":{"value":false}}]}}' | jq .result.count
-d '{"filter":{"must":[{"key":"user_id","match":{"value":"rob"}},{"key":"curated","match":{"value":false}}]}}' | jq .result.count
# Run curator manually
cd ~/<PROJECT_PATH>/true-recall-v2/tr-continuous
# Run curator manually (Our path: .local_projects)
cd ~/.openclaw/workspace/.local_projects/true-recall-v2/tr-continuous
python3 curator_timer.py
# Check OpenClaw plugin
@@ -161,31 +168,38 @@ journalctl -u mem-qdrant-watcher -n 50 --no-pager
---
## Configuration Variables
## Recent Fixes (2026-02-25)
Replace these placeholders with your actual values:
| Issue | Status | Fix |
|-------|--------|-----|
| Embedding model mismatch | ✅ Fixed | Changed curator from `mxbai-embed-large` to `snowflake-arctic-embed2` |
| Gems had no vectors | ✅ Fixed | Updated `store_gem()` to use `text` field |
| JSON parsing errors | ✅ Fixed | Simplified extraction prompt |
| Field mismatch | ✅ Fixed | Curator now supports both `text` and `content` fields |
| Context injection | ✅ **WORKING** | Verified with score 0.587 on test query |
| **Watcher session bug** | ✅ **Fixed 12:22** | Watcher was stuck on old session, restarted and now follows current session |
| **Plugin capture** | ✅ **Fixed 12:34** | Added `extractMessageText()` to handle OpenAI-style content arrays |
| **Plugin exchanges** | ✅ **Verified 12:41** | Now extracting exchanges: parsed 17 user, 116 assistant, 9 exchanges |
| **Gem ID collision** | ✅ **Fixed 12:50** | Hash now uses `embedding_text_for_hash[:100]` instead of empty fields |
| **Meta-gem filtering** | ✅ **Fixed 12:52** | Curator skips patterns: "gems extracted", "curator", "✅", "🔍", debug messages, system messages |
| **gems_tr cleaned** | ✅ **Done 12:53** | Removed 5 meta-gems, kept 1 real gem |
| **Gem format (1st person)** | ✅ **Fixed 13:15** | Changed from "User decided..." to "I decided..." for better query matching |
| Variable | Description | Example |
|----------|-------------|---------|
| `<QDRANT_IP>` | Your Qdrant IP | `10.0.0.40` or `localhost` |
| `<OLLAMA_IP>` | Your Ollama IP | `10.0.0.10` or `localhost` |
| `<PROJECT_PATH>` | Your project path | `.local_projects` or `.projects` |
| `<SKILL_PATH>` | Your skills path | `~/.openclaw/workspace/skills` |
| `<YOUR_USER_ID>` | Your user ID | `rob` or username |
**Result:** Context injection now functional. Gems are embedded and searchable. Both watcher and plugin capture working.
| Check | Date | Status |
|-------|------|--------|
| All services running | 2026-02-25 | ✅ |
| Collections healthy | 2026-02-25 | ✅ |
| Capture working | 2026-02-25 | ✅ |
| Curation working | 2026-02-25 | ✅ |
| Recall working | 2026-02-25 | ✅ **Context injection verified** |
---
## Sign-Off
| Check | Date | Initials |
|-------|------|----------|
| All services running | | |
| Collections healthy | | |
| Capture working | | |
| Curation working | | |
| Recall working | | |
---
*Last updated: 2026-02-25*
*Version: 2.2 (Generic)*
*Last updated: 2026-02-25 12:04 CST*
*User: rob*
*Qdrant: <QDRANT_IP>:6333*
*Timer: 5 minutes*
*Collections: memories_tr (12,729), gems_tr (14+)*
*Status: ✅ Context injection WORKING*