Compare commits
10 Commits
98d14be03b
...
c780a24847
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
c780a24847 | ||
|
|
5c2014cb11 | ||
|
|
a053ec1c3d | ||
|
|
1c24618ad9 | ||
|
|
70f5aec465 | ||
|
|
97a95bd3af | ||
|
|
9769839a67 | ||
|
|
59225f0d1b | ||
|
|
a8299b6db7 | ||
|
|
648aa7f016 |
@@ -0,0 +1,67 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Backfill memory files to Qdrant memories_tr collection."""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import json
|
||||||
|
from datetime import datetime
|
||||||
|
|
||||||
|
QDRANT_URL = "http://10.0.0.40:6333"
|
||||||
|
MEMORY_DIR = "/root/.openclaw/workspace/memory"
|
||||||
|
|
||||||
|
def get_memory_files():
|
||||||
|
"""Get all memory files sorted by date."""
|
||||||
|
files = []
|
||||||
|
for f in os.listdir(MEMORY_DIR):
|
||||||
|
if f.startswith("2026-") and f.endswith(".md"):
|
||||||
|
date = f.replace(".md", "")
|
||||||
|
files.append((date, f))
|
||||||
|
return sorted(files, key=lambda x: x[0])
|
||||||
|
|
||||||
|
def backfill_file(date, filename):
|
||||||
|
"""Backfill a single memory file to Qdrant."""
|
||||||
|
filepath = os.path.join(MEMORY_DIR, filename)
|
||||||
|
with open(filepath, 'r') as f:
|
||||||
|
content = f.read()
|
||||||
|
|
||||||
|
# Truncate if too long for payload
|
||||||
|
payload = {
|
||||||
|
"content": content[:50000], # Limit size
|
||||||
|
"date": date,
|
||||||
|
"source": "memory_file",
|
||||||
|
"curated": False,
|
||||||
|
"role": "system",
|
||||||
|
"user_id": "rob"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Add to Qdrant
|
||||||
|
import requests
|
||||||
|
point_id = hash(f"memory_{date}") % 10000000000
|
||||||
|
resp = requests.post(
|
||||||
|
f"{QDRANT_URL}/collections/memories_tr/points",
|
||||||
|
json={
|
||||||
|
"points": [{
|
||||||
|
"id": point_id,
|
||||||
|
"payload": payload
|
||||||
|
}],
|
||||||
|
"ids": [point_id]
|
||||||
|
}
|
||||||
|
)
|
||||||
|
return resp.status_code == 200
|
||||||
|
|
||||||
|
def main():
|
||||||
|
files = get_memory_files()
|
||||||
|
print(f"Found {len(files)} memory files to backfill")
|
||||||
|
|
||||||
|
count = 0
|
||||||
|
for date, filename in files:
|
||||||
|
print(f"Backfilling {filename}...", end=" ")
|
||||||
|
if backfill_file(date, filename):
|
||||||
|
print("✓")
|
||||||
|
count += 1
|
||||||
|
else:
|
||||||
|
print("✗")
|
||||||
|
|
||||||
|
print(f"\nBackfilled {count}/{len(files)} files")
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
566
.local_projects/true-recall-base/README.md
Normal file
566
.local_projects/true-recall-base/README.md
Normal file
@@ -0,0 +1,566 @@
|
|||||||
|
# TrueRecall Base
|
||||||
|
|
||||||
|
**Purpose:** Real-time memory capture → Qdrant `memories_tr`
|
||||||
|
|
||||||
|
**Status:** ✅ Standalone capture system
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
TrueRecall Base is the **foundation**. It watches OpenClaw sessions in real-time and stores every turn to Qdrant's `memories_tr` collection.
|
||||||
|
|
||||||
|
This is **required** for both addons: **Gems** and **Blocks**.
|
||||||
|
|
||||||
|
**Base does NOT include:**
|
||||||
|
- ❌ Curation (gem extraction)
|
||||||
|
- ❌ Topic clustering (blocks)
|
||||||
|
- ❌ Injection (context recall)
|
||||||
|
|
||||||
|
**For those features, install an addon after base.**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
**Vector Database**
|
||||||
|
|
||||||
|
TrueRecall Base requires a vector database to store conversation embeddings. This can be:
|
||||||
|
- **Local** - Self-hosted Qdrant (recommended for privacy)
|
||||||
|
- **Cloud** - Managed Qdrant Cloud or similar service
|
||||||
|
- **Any IP-accessible** Qdrant instance
|
||||||
|
|
||||||
|
In this version, we use a **local Qdrant database** (`http://<QDRANT_IP>:6333`). The database must be reachable from the machine running the watcher daemon.
|
||||||
|
|
||||||
|
**Additional Requirements:**
|
||||||
|
- **Ollama** - For generating text embeddings (local or remote)
|
||||||
|
- **OpenClaw** - The session files to monitor
|
||||||
|
- **Linux systemd** - For running the watcher as a service
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Gotchas & Known Limitations
|
||||||
|
|
||||||
|
> ⚠️ **Embedding Dimensions:** `snowflake-arctic-embed2` outputs **1024 dimensions**, not 768. Ensure your Qdrant collection is configured with `"size": 1024`.
|
||||||
|
|
||||||
|
> ⚠️ **Hardcoded Sessions Path:** `SESSIONS_DIR` is hardcoded to `/root/.openclaw/agents/main/sessions`. To use a different path, modify `realtime_qdrant_watcher.py` to read from an environment variable:
|
||||||
|
> ```python
|
||||||
|
> SESSIONS_DIR = Path(os.getenv("OPENCLAW_SESSIONS_DIR", "/root/.openclaw/agents/main/sessions"))
|
||||||
|
> ```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Three-Tier Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
true-recall-base (REQUIRED)
|
||||||
|
├── Core: Watcher daemon
|
||||||
|
└── Stores: memories_tr
|
||||||
|
│
|
||||||
|
├──▶ true-recall-gems (ADDON)
|
||||||
|
│ ├── Curator extracts gems → gems_tr
|
||||||
|
│ └── Plugin injects gems into prompts
|
||||||
|
│
|
||||||
|
└──▶ true-recall-blocks (ADDON)
|
||||||
|
├── Topic clustering → topic_blocks_tr
|
||||||
|
└── Contextual block retrieval
|
||||||
|
|
||||||
|
Note: Gems and Blocks are INDEPENDENT addons.
|
||||||
|
They both require Base, but don't work together.
|
||||||
|
Choose one: Gems OR Blocks (not both).
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Option 1: Quick Install (Recommended)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /path/to/true-recall-base
|
||||||
|
./install.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
#### What the Installer Does (Step-by-Step)
|
||||||
|
|
||||||
|
The `install.sh` script automates the entire setup process. Here's exactly what happens:
|
||||||
|
|
||||||
|
**Step 1: Interactive Configuration**
|
||||||
|
```
|
||||||
|
Configuration (press Enter for defaults):
|
||||||
|
|
||||||
|
Examples:
|
||||||
|
Qdrant: 10.0.0.40:6333 (remote) or localhost:6333 (local)
|
||||||
|
Ollama: 10.0.0.10:11434 (remote) or localhost:11434 (local)
|
||||||
|
|
||||||
|
Qdrant host:port [localhost:6333]: _
|
||||||
|
Ollama host:port [localhost:11434]: _
|
||||||
|
User ID [user]: _
|
||||||
|
```
|
||||||
|
- Prompts for Qdrant host:port (default: `localhost:6333`)
|
||||||
|
- Prompts for Ollama host:port (default: `localhost:11434`)
|
||||||
|
- Prompts for User ID (default: `user`)
|
||||||
|
- Press Enter to accept defaults, or type custom values
|
||||||
|
|
||||||
|
**Step 2: Configuration Confirmation**
|
||||||
|
```
|
||||||
|
Configuration:
|
||||||
|
Qdrant: http://localhost:6333
|
||||||
|
Ollama: http://localhost:11434
|
||||||
|
User ID: user
|
||||||
|
|
||||||
|
Proceed? [Y/n]: _
|
||||||
|
```
|
||||||
|
- Shows the complete configuration
|
||||||
|
- Asks for confirmation (type `n` to cancel, Enter or `Y` to proceed)
|
||||||
|
- Exits cleanly if cancelled, no changes made
|
||||||
|
|
||||||
|
**Step 3: Systemd Service Generation**
|
||||||
|
- Creates a temporary service file at `/tmp/mem-qdrant-watcher.service`
|
||||||
|
- Inserts your configuration values (IPs, ports, user ID)
|
||||||
|
- Uses absolute path for the script location (handles spaces in paths)
|
||||||
|
- Sets up automatic restart on failure
|
||||||
|
|
||||||
|
**Step 4: Service Installation**
|
||||||
|
```bash
|
||||||
|
sudo cp /tmp/mem-qdrant-watcher.service /etc/systemd/system/
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
```
|
||||||
|
- Copies the service file to systemd directory
|
||||||
|
- Reloads systemd to recognize the new service
|
||||||
|
|
||||||
|
**Step 5: Service Activation**
|
||||||
|
```bash
|
||||||
|
sudo systemctl enable --now mem-qdrant-watcher
|
||||||
|
```
|
||||||
|
- Enables the service to start on boot (`enable`)
|
||||||
|
- Starts the service immediately (`now`)
|
||||||
|
|
||||||
|
**Step 6: Verification**
|
||||||
|
```
|
||||||
|
==========================================
|
||||||
|
Installation Complete!
|
||||||
|
==========================================
|
||||||
|
|
||||||
|
Status:
|
||||||
|
● mem-qdrant-watcher.service - TrueRecall Base...
|
||||||
|
Active: active (running)
|
||||||
|
```
|
||||||
|
- Displays the service status
|
||||||
|
- Shows it's active and running
|
||||||
|
- Provides commands to verify and monitor
|
||||||
|
|
||||||
|
**Post-Installation Commands:**
|
||||||
|
```bash
|
||||||
|
# Check service status anytime
|
||||||
|
sudo systemctl status mem-qdrant-watcher
|
||||||
|
|
||||||
|
# View live logs
|
||||||
|
sudo journalctl -u mem-qdrant-watcher -f
|
||||||
|
|
||||||
|
# Verify Qdrant collection
|
||||||
|
curl -s http://localhost:6333/collections/memories_tr | jq '.result.points_count'
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Installer Requirements
|
||||||
|
- Must run as root or with sudo (for systemd operations)
|
||||||
|
- Must have execute permissions (`chmod +x install.sh`)
|
||||||
|
- Script must be run from the true-recall-base directory
|
||||||
|
|
||||||
|
### Option 2: Manual Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd /path/to/true-recall-base
|
||||||
|
|
||||||
|
# Copy service file
|
||||||
|
sudo cp watcher/mem-qdrant-watcher.service /etc/systemd/system/
|
||||||
|
|
||||||
|
# Edit the service file to set your IPs and user
|
||||||
|
sudo nano /etc/systemd/system/mem-qdrant-watcher.service
|
||||||
|
|
||||||
|
# Reload and start
|
||||||
|
sudo systemctl daemon-reload
|
||||||
|
sudo systemctl enable --now mem-qdrant-watcher
|
||||||
|
```
|
||||||
|
|
||||||
|
### Verify Installation
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Check service status
|
||||||
|
sudo systemctl status mem-qdrant-watcher
|
||||||
|
|
||||||
|
# Check collection
|
||||||
|
curl -s http://<QDRANT_IP>:6333/collections/memories_tr | jq '.result.points_count'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Files
|
||||||
|
|
||||||
|
| File | Purpose |
|
||||||
|
|------|---------|
|
||||||
|
| `watcher/realtime_qdrant_watcher.py` | Capture daemon |
|
||||||
|
| `watcher/mem-qdrant-watcher.service` | Systemd service |
|
||||||
|
| `config.json` | Configuration template |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Edit `config.json` or set environment variables:
|
||||||
|
|
||||||
|
| Variable | Default | Description |
|
||||||
|
|----------|---------|-------------|
|
||||||
|
| `QDRANT_URL` | `http://<QDRANT_IP>:6333` | Qdrant endpoint |
|
||||||
|
| `OLLAMA_URL` | `http://<OLLAMA_IP>:11434` | Ollama endpoint |
|
||||||
|
| `EMBEDDING_MODEL` | `snowflake-arctic-embed2` | Embedding model |
|
||||||
|
| `USER_ID` | `<USER_ID>` | User identifier |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## How It Works
|
||||||
|
|
||||||
|
### Architecture Overview
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────┐ ┌──────────────────┐ ┌─────────────────┐
|
||||||
|
│ OpenClaw Chat │────▶│ Session JSONL │────▶│ Base Watcher │
|
||||||
|
│ (You talking) │ │ (/sessions/*.jsonl) │ │ (This daemon) │
|
||||||
|
└─────────────────┘ └──────────────────┘ └────────┬────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌────────────────────────────────────────────────────────────────────┐
|
||||||
|
│ PROCESSING PIPELINE │
|
||||||
|
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌───────────┐ │
|
||||||
|
│ │ Watch File │─▶│ Parse Turn │─▶│ Clean Text │─▶│ Embed │ │
|
||||||
|
│ │ (inotify) │ │ (JSON→dict) │ │ (strip md) │ │ (Ollama) │ │
|
||||||
|
│ └──────────────┘ └──────────────┘ └──────────────┘ └─────┬─────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ┌───────────────────────────────────────────────────────────┘ │
|
||||||
|
│ │ │
|
||||||
|
│ ▼ │
|
||||||
|
│ ┌──────────────┐ ┌──────────────┐ │
|
||||||
|
│ │ Store to │─▶│ Qdrant │ │
|
||||||
|
│ │ memories_tr │ │ (vector DB) │ │
|
||||||
|
│ └──────────────┘ └──────────────┘ │
|
||||||
|
└────────────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
### Step-by-Step Process
|
||||||
|
|
||||||
|
#### Step 1: File Watching
|
||||||
|
|
||||||
|
The watcher monitors OpenClaw session files in real-time:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# From realtime_qdrant_watcher.py
|
||||||
|
SESSIONS_DIR = Path("/root/.openclaw/agents/main/sessions")
|
||||||
|
```
|
||||||
|
|
||||||
|
> ⚠️ **Known Limitation:** `SESSIONS_DIR` is currently hardcoded. To use a different path, patch the watcher script to read from an environment variable (e.g., `os.getenv("OPENCLAW_SESSIONS_DIR", "/root/.openclaw/agents/main/sessions")`).
|
||||||
|
|
||||||
|
**What happens:**
|
||||||
|
- Uses `inotify` or polling to watch the sessions directory
|
||||||
|
- Automatically detects the most recently modified `.jsonl` file
|
||||||
|
- Handles session rotation (when OpenClaw starts a new session)
|
||||||
|
- Maintains position in file to avoid re-processing old lines
|
||||||
|
|
||||||
|
#### Step 2: Turn Parsing
|
||||||
|
|
||||||
|
Each conversation turn is extracted from the JSONL file:
|
||||||
|
|
||||||
|
```json
|
||||||
|
// Example session file entry
|
||||||
|
{
|
||||||
|
"type": "message",
|
||||||
|
"message": {
|
||||||
|
"role": "user",
|
||||||
|
"content": "Hello, can you help me?",
|
||||||
|
"timestamp": "2026-02-27T09:30:00Z"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**What happens:**
|
||||||
|
- Reads new lines appended to the session file
|
||||||
|
- Parses JSON to extract role (user/assistant/system)
|
||||||
|
- Extracts content text
|
||||||
|
- Captures timestamp
|
||||||
|
- Generates unique turn ID from content hash + timestamp
|
||||||
|
|
||||||
|
**Code flow:**
|
||||||
|
```python
|
||||||
|
def parse_turn(line: str) -> Optional[Dict]:
|
||||||
|
data = json.loads(line)
|
||||||
|
if data.get("type") != "message":
|
||||||
|
return None # Skip non-message entries
|
||||||
|
|
||||||
|
return {
|
||||||
|
"id": hashlib.md5(f"{content}{timestamp}".encode()).hexdigest()[:16],
|
||||||
|
"role": role,
|
||||||
|
"content": content,
|
||||||
|
"timestamp": timestamp,
|
||||||
|
"user_id": os.getenv("USER_ID", "default")
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Step 3: Content Cleaning
|
||||||
|
|
||||||
|
Before storage, content is normalized:
|
||||||
|
|
||||||
|
**Strips:**
|
||||||
|
- Markdown tables (`| column | column |`)
|
||||||
|
- Bold/italic markers (`**text**`, `*text*`)
|
||||||
|
- Inline code (`` `code` ``)
|
||||||
|
- Code blocks (```code```)
|
||||||
|
- Multiple consecutive spaces
|
||||||
|
- Leading/trailing whitespace
|
||||||
|
|
||||||
|
**Example:**
|
||||||
|
```
|
||||||
|
Input: "Check this **important** table: | col1 | col2 |"
|
||||||
|
Output: "Check this important table"
|
||||||
|
```
|
||||||
|
|
||||||
|
**Why:** Clean text improves embedding quality and searchability.
|
||||||
|
|
||||||
|
#### Step 4: Embedding Generation
|
||||||
|
|
||||||
|
The cleaned content is converted to a vector embedding:
|
||||||
|
|
||||||
|
```python
|
||||||
|
def get_embedding(text: str) -> List[float]:
|
||||||
|
response = requests.post(
|
||||||
|
f"{OLLAMA_URL}/api/embeddings",
|
||||||
|
json={"model": EMBEDDING_MODEL, "prompt": text}
|
||||||
|
)
|
||||||
|
return response.json()["embedding"]
|
||||||
|
```
|
||||||
|
|
||||||
|
**What happens:**
|
||||||
|
- Sends text to Ollama API (10.0.0.10:11434)
|
||||||
|
- Uses `snowflake-arctic-embed2` model
|
||||||
|
- Returns **1024-dimensional vector** (not 768)
|
||||||
|
- Falls back gracefully if Ollama is unavailable
|
||||||
|
|
||||||
|
#### Step 5: Qdrant Storage
|
||||||
|
|
||||||
|
The complete turn data is stored to Qdrant:
|
||||||
|
|
||||||
|
```python
|
||||||
|
payload = {
|
||||||
|
"user_id": user_id,
|
||||||
|
"role": turn["role"],
|
||||||
|
"content": cleaned_content[:2000], # Size limit
|
||||||
|
"timestamp": turn["timestamp"],
|
||||||
|
"session_id": session_id,
|
||||||
|
"source": "true-recall-base"
|
||||||
|
}
|
||||||
|
|
||||||
|
requests.put(
|
||||||
|
f"{QDRANT_URL}/collections/memories_tr/points",
|
||||||
|
json={"points": [{"id": turn_id, "vector": embedding, "payload": payload}]}
|
||||||
|
)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Storage format:**
|
||||||
|
| Field | Type | Description |
|
||||||
|
|-------|------|-------------|
|
||||||
|
| `user_id` | string | User identifier |
|
||||||
|
| `role` | string | user/assistant/system |
|
||||||
|
| `content` | string | Cleaned text (max 2000 chars) |
|
||||||
|
| `timestamp` | string | ISO 8601 timestamp |
|
||||||
|
| `session_id` | string | Source session file |
|
||||||
|
| `source` | string | "true-recall-base" |
|
||||||
|
|
||||||
|
### Real-Time Performance
|
||||||
|
|
||||||
|
| Metric | Target | Actual |
|
||||||
|
|--------|--------|--------|
|
||||||
|
| Latency | < 500ms | ~100-200ms |
|
||||||
|
| Throughput | > 10 turns/sec | > 50 turns/sec |
|
||||||
|
| Embedding time | < 300ms | ~50-100ms |
|
||||||
|
| Qdrant write | < 100ms | ~10-50ms |
|
||||||
|
|
||||||
|
### Session Rotation Handling
|
||||||
|
|
||||||
|
When OpenClaw starts a new session:
|
||||||
|
|
||||||
|
1. New `.jsonl` file created in sessions directory
|
||||||
|
2. Watcher detects file change via `inotify`
|
||||||
|
3. Identifies most recently modified file
|
||||||
|
4. Switches to watching new file
|
||||||
|
5. Continues from position 0 of new file
|
||||||
|
6. Old file remains in `memories_tr` (already captured)
|
||||||
|
|
||||||
|
### Error Handling
|
||||||
|
|
||||||
|
**Qdrant unavailable:**
|
||||||
|
- Retries with exponential backoff
|
||||||
|
- Logs error, continues watching
|
||||||
|
- Next turn attempts storage again
|
||||||
|
|
||||||
|
**Ollama unavailable:**
|
||||||
|
- Cannot generate embeddings
|
||||||
|
- Logs error, skips turn
|
||||||
|
- Continues watching (no data loss in file)
|
||||||
|
|
||||||
|
**File access errors:**
|
||||||
|
- Handles permission issues gracefully
|
||||||
|
- Retries on temporary failures
|
||||||
|
|
||||||
|
### Collection Schema
|
||||||
|
|
||||||
|
**Qdrant collection: `memories_tr`**
|
||||||
|
|
||||||
|
```python
|
||||||
|
{
|
||||||
|
"name": "memories_tr",
|
||||||
|
"vectors": {
|
||||||
|
"size": 1024, # snowflake-arctic-embed2 dimension (1024, not 768)
|
||||||
|
"distance": "Cosine" # Similarity metric
|
||||||
|
},
|
||||||
|
"payload_schema": {
|
||||||
|
"user_id": "keyword", # Filterable
|
||||||
|
"role": "keyword", # Filterable
|
||||||
|
"timestamp": "datetime", # Range filterable
|
||||||
|
"content": "text" # Full-text searchable
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Security Notes
|
||||||
|
|
||||||
|
- **No credential storage** in code
|
||||||
|
- All sensitive values via environment variables
|
||||||
|
- `USER_ID` isolates memories per user
|
||||||
|
- Cleaned content removes PII markers (but review your data)
|
||||||
|
- HTTPS recommended for production Qdrant/Ollama
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Using Memories with OpenClaw
|
||||||
|
|
||||||
|
### The "q" Command
|
||||||
|
|
||||||
|
**"q"** refers to your Qdrant memory system (`memories_tr` collection).
|
||||||
|
|
||||||
|
When interacting with OpenClaw agents, you can search your stored memories using:
|
||||||
|
- `search q <topic>` - Semantic search for past conversations
|
||||||
|
- `q <topic>` - Shortcut for the same
|
||||||
|
|
||||||
|
### Context Injection Instructions
|
||||||
|
|
||||||
|
**For OpenClaw System Prompt:**
|
||||||
|
|
||||||
|
Add these lines to your agent's system context to enable memory-aware responses:
|
||||||
|
|
||||||
|
```
|
||||||
|
## Memory System (q)
|
||||||
|
|
||||||
|
**"q" = Qdrant collection `memories_tr`** — your conversation history database.
|
||||||
|
|
||||||
|
### Memory Retrieval Rules
|
||||||
|
|
||||||
|
**Before saying "I don't know" or "I can't do that":**
|
||||||
|
1. **ALWAYS search q first** using the topic/keywords from the user's request
|
||||||
|
2. Incorporate findings INTO your response (not as footnotes)
|
||||||
|
3. Reference specific dates/details: "Based on our Feb 27th discussion..."
|
||||||
|
|
||||||
|
**Example workflow:**
|
||||||
|
```
|
||||||
|
User asks about X → Search q for X → Use retrieved memories → Answer
|
||||||
|
```
|
||||||
|
|
||||||
|
**WRONG:**
|
||||||
|
> "I searched Qdrant and found X. [Generic answer unrelated to X]"
|
||||||
|
|
||||||
|
**RIGHT:**
|
||||||
|
> "You asked me to fix this on Feb 27th — do you want me to apply the fix now?"
|
||||||
|
|
||||||
|
### When to Search q
|
||||||
|
|
||||||
|
**ALWAYS search automatically when:**
|
||||||
|
- Question references past events, conversations, or details
|
||||||
|
- User asks "remember when...", "what did we discuss...", "what did I tell you..."
|
||||||
|
- You're unsure if you have relevant context
|
||||||
|
- ANY question about configuration, memories, or past interactions
|
||||||
|
|
||||||
|
**DO NOT search for:**
|
||||||
|
- General knowledge questions you can answer directly
|
||||||
|
- Current time, weather, or factual queries
|
||||||
|
- Simple requests like "check my email" or "run a command"
|
||||||
|
- When you already have sufficient context in the conversation
|
||||||
|
```
|
||||||
|
|
||||||
|
### Search Priority
|
||||||
|
|
||||||
|
| Order | Source | When to Use |
|
||||||
|
|-------|--------|-------------|
|
||||||
|
| 1 | **q (Qdrant)** | First - semantic search of all conversations |
|
||||||
|
| 2 | `memory/` files | Fallback if q yields no results |
|
||||||
|
| 3 | Web search | Last resort |
|
||||||
|
| 4 | "I don't know" | Only after all above |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Step
|
||||||
|
|
||||||
|
### ✅ Base is Complete
|
||||||
|
|
||||||
|
**You don't need to upgrade.** TrueRecall Base is a **fully functional, standalone memory system**. If you're happy with real-time capture and manual search via the `q` command, you can stop here.
|
||||||
|
|
||||||
|
Base gives you:
|
||||||
|
- ✅ Complete conversation history in Qdrant
|
||||||
|
- ✅ Semantic search via `search q <topic>`
|
||||||
|
- ✅ Full-text search capabilities
|
||||||
|
- ✅ Permanent storage of all conversations
|
||||||
|
|
||||||
|
**Upgrade only if** you want automatic context injection into prompts.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Optional Addons
|
||||||
|
|
||||||
|
Install an **addon** for automatic curation and injection:
|
||||||
|
|
||||||
|
| Addon | Purpose | Status |
|
||||||
|
|-------|---------|--------|
|
||||||
|
| **Gems** | Extracts atomic gems from memories, injects into context | 🚧 Coming Soon |
|
||||||
|
| **Blocks** | Topic clustering, contextual block retrieval | 🚧 Coming Soon |
|
||||||
|
|
||||||
|
### Upgrade Paths
|
||||||
|
|
||||||
|
Once Base is running, you have two upgrade options:
|
||||||
|
|
||||||
|
#### Option 1: Gems (Atomic Memory)
|
||||||
|
**Best for:** Conversational context, quick recall
|
||||||
|
|
||||||
|
- **Curator** extracts "gems" (key insights) from `memories_tr`
|
||||||
|
- Stores curated gems in `gems_tr` collection
|
||||||
|
- **Injection plugin** recalls relevant gems into prompts automatically
|
||||||
|
- Optimized for: Chat assistants, help bots, personal memory
|
||||||
|
|
||||||
|
**Workflow:**
|
||||||
|
```
|
||||||
|
memories_tr → Curator → gems_tr → Injection → Context
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Option 2: Blocks (Topic Clustering)
|
||||||
|
**Best for:** Document organization, topic-based retrieval
|
||||||
|
|
||||||
|
- Clusters conversations by topic automatically
|
||||||
|
- Creates `topic_blocks_tr` collection
|
||||||
|
- Retrieves entire contextual blocks on query
|
||||||
|
- Optimized for: Knowledge bases, document systems
|
||||||
|
|
||||||
|
**Workflow:**
|
||||||
|
```
|
||||||
|
memories_tr → Topic Engine → topic_blocks_tr → Retrieval → Context
|
||||||
|
```
|
||||||
|
|
||||||
|
**Note:** Gems and Blocks are **independent** addons. They both require Base, but you choose one based on your use case.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Prerequisite for:** TrueRecall Gems, TrueRecall Blocks
|
||||||
14
.local_projects/true-recall-base/config.json
Normal file
14
.local_projects/true-recall-base/config.json
Normal file
@@ -0,0 +1,14 @@
|
|||||||
|
{
|
||||||
|
"version": "1.1",
|
||||||
|
"description": "TrueRecall v1.1 - Memory capture with session rotation fix",
|
||||||
|
"components": ["watcher"],
|
||||||
|
"collections": {
|
||||||
|
"memories": "memories_tr"
|
||||||
|
},
|
||||||
|
"qdrant_url": "http://10.0.0.40:6333",
|
||||||
|
"ollama_url": "http://localhost:11434",
|
||||||
|
"embedding_model": "snowflake-arctic-embed2",
|
||||||
|
"embedding_dimensions": 1024,
|
||||||
|
"user_id": "rob",
|
||||||
|
"notes": "Ensure memories_tr collection is created with size=1024 for snowflake-arctic-embed2"
|
||||||
|
}
|
||||||
@@ -0,0 +1,367 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
TrueRecall v1.2 - Real-time Qdrant Watcher
|
||||||
|
Monitors OpenClaw sessions and stores to memories_tr instantly.
|
||||||
|
|
||||||
|
This is the CAPTURE component. For curation and injection, install v2.
|
||||||
|
|
||||||
|
Changelog:
|
||||||
|
- v1.2: Fixed session rotation bug - added inactivity detection (30s threshold)
|
||||||
|
and improved file scoring to properly detect new sessions on /new or /reset
|
||||||
|
- v1.1: Added 1-second mtime polling for session rotation
|
||||||
|
- v1.0: Initial release
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import json
|
||||||
|
import time
|
||||||
|
import signal
|
||||||
|
import hashlib
|
||||||
|
import argparse
|
||||||
|
import requests
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Dict, Any, Optional, List
|
||||||
|
|
||||||
|
# Config
|
||||||
|
QDRANT_URL = os.getenv("QDRANT_URL", "http://10.0.0.40:6333")
|
||||||
|
QDRANT_COLLECTION = os.getenv("QDRANT_COLLECTION", "memories_tr")
|
||||||
|
OLLAMA_URL = os.getenv("OLLAMA_URL", "http://localhost:11434")
|
||||||
|
EMBEDDING_MODEL = os.getenv("EMBEDDING_MODEL", "snowflake-arctic-embed2")
|
||||||
|
USER_ID = os.getenv("USER_ID", "rob")
|
||||||
|
|
||||||
|
# Paths
|
||||||
|
SESSIONS_DIR = Path(os.getenv("OPENCLAW_SESSIONS_DIR", "/root/.openclaw/agents/main/sessions"))
|
||||||
|
|
||||||
|
# State
|
||||||
|
running = True
|
||||||
|
last_position = 0
|
||||||
|
current_file = None
|
||||||
|
turn_counter = 0
|
||||||
|
|
||||||
|
|
||||||
|
def signal_handler(signum, frame):
|
||||||
|
global running
|
||||||
|
print(f"\nReceived signal {signum}, shutting down...", file=sys.stderr)
|
||||||
|
running = False
|
||||||
|
|
||||||
|
|
||||||
|
def get_embedding(text: str) -> List[float]:
|
||||||
|
try:
|
||||||
|
response = requests.post(
|
||||||
|
f"{OLLAMA_URL}/api/embeddings",
|
||||||
|
json={"model": EMBEDDING_MODEL, "prompt": text},
|
||||||
|
timeout=30
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()["embedding"]
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error getting embedding: {e}", file=sys.stderr)
|
||||||
|
return None
|
||||||
|
|
||||||
|
|
||||||
|
def clean_content(text: str) -> str:
|
||||||
|
import re
|
||||||
|
|
||||||
|
# Remove metadata JSON blocks
|
||||||
|
text = re.sub(r'Conversation info \(untrusted metadata\):\s*```json\s*\{[\s\S]*?\}\s*```', '', text)
|
||||||
|
|
||||||
|
# Remove thinking tags
|
||||||
|
text = re.sub(r'\[thinking:[^\]]*\]', '', text)
|
||||||
|
|
||||||
|
# Remove timestamp lines
|
||||||
|
text = re.sub(r'\[\w{3} \d{4}-\d{2}-\d{2} \d{2}:\d{2} [A-Z]{3}\]', '', text)
|
||||||
|
|
||||||
|
# Remove markdown tables
|
||||||
|
text = re.sub(r'\|[^\n]*\|', '', text)
|
||||||
|
text = re.sub(r'\|[-:]+\|', '', text)
|
||||||
|
|
||||||
|
# Remove markdown formatting
|
||||||
|
text = re.sub(r'\*\*([^*]+)\*\*', r'\1', text)
|
||||||
|
text = re.sub(r'\*([^*]+)\*', r'\1', text)
|
||||||
|
text = re.sub(r'`([^`]+)`', r'\1', text)
|
||||||
|
text = re.sub(r'```[\s\S]*?```', '', text)
|
||||||
|
|
||||||
|
# Remove horizontal rules
|
||||||
|
text = re.sub(r'---+', '', text)
|
||||||
|
text = re.sub(r'\*\*\*+', '', text)
|
||||||
|
|
||||||
|
# Remove excess whitespace
|
||||||
|
text = re.sub(r'\n{3,}', '\n', text)
|
||||||
|
text = re.sub(r'[ \t]+', ' ', text)
|
||||||
|
|
||||||
|
return text.strip()
|
||||||
|
|
||||||
|
|
||||||
|
def store_to_qdrant(turn: Dict[str, Any], dry_run: bool = False) -> bool:
|
||||||
|
if dry_run:
|
||||||
|
print(f"[DRY RUN] Would store turn {turn['turn']} ({turn['role']}): {turn['content'][:60]}...")
|
||||||
|
return True
|
||||||
|
|
||||||
|
vector = get_embedding(turn['content'])
|
||||||
|
if vector is None:
|
||||||
|
print(f"Failed to get embedding for turn {turn['turn']}", file=sys.stderr)
|
||||||
|
return False
|
||||||
|
|
||||||
|
payload = {
|
||||||
|
"user_id": turn.get('user_id', USER_ID),
|
||||||
|
"role": turn['role'],
|
||||||
|
"content": turn['content'],
|
||||||
|
"turn": turn['turn'],
|
||||||
|
"timestamp": turn.get('timestamp', datetime.now(timezone.utc).isoformat()),
|
||||||
|
"date": datetime.now(timezone.utc).strftime('%Y-%m-%d'),
|
||||||
|
"source": "true-recall-base",
|
||||||
|
"curated": False
|
||||||
|
}
|
||||||
|
|
||||||
|
# Generate deterministic ID
|
||||||
|
turn_id = turn.get('turn', 0)
|
||||||
|
hash_bytes = hashlib.sha256(f"{USER_ID}:turn:{turn_id}:{datetime.now().strftime('%H%M%S')}".encode()).digest()[:8]
|
||||||
|
point_id = int.from_bytes(hash_bytes, byteorder='big') % (2**63)
|
||||||
|
|
||||||
|
try:
|
||||||
|
response = requests.put(
|
||||||
|
f"{QDRANT_URL}/collections/{QDRANT_COLLECTION}/points",
|
||||||
|
json={
|
||||||
|
"points": [{
|
||||||
|
"id": abs(point_id),
|
||||||
|
"vector": vector,
|
||||||
|
"payload": payload
|
||||||
|
}]
|
||||||
|
},
|
||||||
|
timeout=30
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return True
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error writing to Qdrant: {e}", file=sys.stderr)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
def get_current_session_file():
|
||||||
|
"""Find the most recently active session file.
|
||||||
|
|
||||||
|
Uses a combination of creation time and modification time to handle
|
||||||
|
session rotation when /new or /reset is used.
|
||||||
|
"""
|
||||||
|
if not SESSIONS_DIR.exists():
|
||||||
|
return None
|
||||||
|
|
||||||
|
files = list(SESSIONS_DIR.glob("*.jsonl"))
|
||||||
|
if not files:
|
||||||
|
return None
|
||||||
|
|
||||||
|
# Score files by: recency (mtime) + size activity
|
||||||
|
# Files with very recent mtime AND non-zero size are likely active
|
||||||
|
def file_score(p: Path) -> float:
|
||||||
|
try:
|
||||||
|
stat = p.stat()
|
||||||
|
mtime = stat.st_mtime
|
||||||
|
size = stat.st_size
|
||||||
|
# Prefer files with recent mtime and non-zero size
|
||||||
|
# Add small bonus for larger files (active sessions grow)
|
||||||
|
return mtime + (size / 1e9) # size bonus is tiny vs mtime
|
||||||
|
except Exception:
|
||||||
|
return 0
|
||||||
|
|
||||||
|
return max(files, key=file_score)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_turn(line: str, session_name: str) -> Optional[Dict[str, Any]]:
|
||||||
|
global turn_counter
|
||||||
|
|
||||||
|
try:
|
||||||
|
entry = json.loads(line.strip())
|
||||||
|
except json.JSONDecodeError:
|
||||||
|
return None
|
||||||
|
|
||||||
|
if entry.get('type') != 'message' or 'message' not in entry:
|
||||||
|
return None
|
||||||
|
|
||||||
|
msg = entry['message']
|
||||||
|
role = msg.get('role')
|
||||||
|
|
||||||
|
if role in ('toolResult', 'system', 'developer'):
|
||||||
|
return None
|
||||||
|
|
||||||
|
if role not in ('user', 'assistant'):
|
||||||
|
return None
|
||||||
|
|
||||||
|
content = ""
|
||||||
|
if isinstance(msg.get('content'), list):
|
||||||
|
for item in msg['content']:
|
||||||
|
if isinstance(item, dict) and 'text' in item:
|
||||||
|
content += item['text']
|
||||||
|
elif isinstance(msg.get('content'), str):
|
||||||
|
content = msg['content']
|
||||||
|
|
||||||
|
if not content:
|
||||||
|
return None
|
||||||
|
|
||||||
|
content = clean_content(content)
|
||||||
|
if not content or len(content) < 5:
|
||||||
|
return None
|
||||||
|
|
||||||
|
turn_counter += 1
|
||||||
|
|
||||||
|
return {
|
||||||
|
'turn': turn_counter,
|
||||||
|
'role': role,
|
||||||
|
'content': content[:2000],
|
||||||
|
'timestamp': entry.get('timestamp', datetime.now(timezone.utc).isoformat()),
|
||||||
|
'user_id': USER_ID
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def process_new_lines(f, session_name: str, dry_run: bool = False):
|
||||||
|
global last_position
|
||||||
|
|
||||||
|
f.seek(last_position)
|
||||||
|
|
||||||
|
for line in f:
|
||||||
|
line = line.strip()
|
||||||
|
if not line:
|
||||||
|
continue
|
||||||
|
|
||||||
|
turn = parse_turn(line, session_name)
|
||||||
|
if turn:
|
||||||
|
if store_to_qdrant(turn, dry_run):
|
||||||
|
print(f"✅ Turn {turn['turn']} ({turn['role']}) → Qdrant")
|
||||||
|
|
||||||
|
last_position = f.tell()
|
||||||
|
|
||||||
|
|
||||||
|
def watch_session(session_file: Path, dry_run: bool = False):
|
||||||
|
global last_position, turn_counter
|
||||||
|
|
||||||
|
session_name = session_file.name.replace('.jsonl', '')
|
||||||
|
print(f"Watching session: {session_file.name}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
with open(session_file, 'r') as f:
|
||||||
|
for line in f:
|
||||||
|
turn_counter += 1
|
||||||
|
last_position = session_file.stat().st_size
|
||||||
|
print(f"Session has {turn_counter} existing turns, starting from position {last_position}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Warning: Could not read existing turns: {e}", file=sys.stderr)
|
||||||
|
last_position = 0
|
||||||
|
|
||||||
|
last_session_check = time.time()
|
||||||
|
last_data_time = time.time() # Track when we last saw new data
|
||||||
|
last_file_size = session_file.stat().st_size if session_file.exists() else 0
|
||||||
|
|
||||||
|
INACTIVITY_THRESHOLD = 30 # seconds - if no data for 30s, check for new session
|
||||||
|
|
||||||
|
with open(session_file, 'r') as f:
|
||||||
|
while running:
|
||||||
|
if not session_file.exists():
|
||||||
|
print("Session file removed, looking for new session...")
|
||||||
|
return None
|
||||||
|
|
||||||
|
current_time = time.time()
|
||||||
|
|
||||||
|
# Check for newer session every 1 second
|
||||||
|
if current_time - last_session_check > 1.0:
|
||||||
|
last_session_check = current_time
|
||||||
|
newest_session = get_current_session_file()
|
||||||
|
if newest_session and newest_session != session_file:
|
||||||
|
print(f"Newer session detected: {newest_session.name}")
|
||||||
|
return newest_session
|
||||||
|
|
||||||
|
# Check if current file is stale (no new data for threshold)
|
||||||
|
if current_time - last_data_time > INACTIVITY_THRESHOLD:
|
||||||
|
try:
|
||||||
|
current_size = session_file.stat().st_size
|
||||||
|
# If file hasn't grown, check if another session is active
|
||||||
|
if current_size == last_file_size:
|
||||||
|
newest_session = get_current_session_file()
|
||||||
|
if newest_session and newest_session != session_file:
|
||||||
|
print(f"Current session inactive, switching to: {newest_session.name}")
|
||||||
|
return newest_session
|
||||||
|
else:
|
||||||
|
# File grew, update tracking
|
||||||
|
last_file_size = current_size
|
||||||
|
last_data_time = current_time
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
# Process new lines and update activity tracking
|
||||||
|
old_position = last_position
|
||||||
|
process_new_lines(f, session_name, dry_run)
|
||||||
|
|
||||||
|
# If we processed new data, update activity timestamp
|
||||||
|
if last_position > old_position:
|
||||||
|
last_data_time = current_time
|
||||||
|
try:
|
||||||
|
last_file_size = session_file.stat().st_size
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
|
||||||
|
time.sleep(0.1)
|
||||||
|
|
||||||
|
return session_file
|
||||||
|
|
||||||
|
|
||||||
|
def watch_loop(dry_run: bool = False):
|
||||||
|
global current_file, turn_counter
|
||||||
|
|
||||||
|
while running:
|
||||||
|
session_file = get_current_session_file()
|
||||||
|
|
||||||
|
if session_file is None:
|
||||||
|
print("No active session found, waiting...")
|
||||||
|
time.sleep(1)
|
||||||
|
continue
|
||||||
|
|
||||||
|
if current_file != session_file:
|
||||||
|
print(f"\nNew session detected: {session_file.name}")
|
||||||
|
current_file = session_file
|
||||||
|
turn_counter = 0
|
||||||
|
last_position = 0
|
||||||
|
|
||||||
|
result = watch_session(session_file, dry_run)
|
||||||
|
|
||||||
|
if result is None:
|
||||||
|
current_file = None
|
||||||
|
time.sleep(0.5)
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
global USER_ID
|
||||||
|
|
||||||
|
parser = argparse.ArgumentParser(description="TrueRecall v1.1 - Real-time Memory Capture")
|
||||||
|
parser.add_argument("--daemon", "-d", action="store_true", help="Run as daemon")
|
||||||
|
parser.add_argument("--once", "-o", action="store_true", help="Process once then exit")
|
||||||
|
parser.add_argument("--dry-run", "-n", action="store_true", help="Don't write to Qdrant")
|
||||||
|
parser.add_argument("--user-id", "-u", default=USER_ID, help=f"User ID (default: {USER_ID})")
|
||||||
|
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
signal.signal(signal.SIGINT, signal_handler)
|
||||||
|
signal.signal(signal.SIGTERM, signal_handler)
|
||||||
|
|
||||||
|
if args.user_id:
|
||||||
|
USER_ID = args.user_id
|
||||||
|
|
||||||
|
print(f"🔍 TrueRecall v1.1 - Real-time Memory Capture")
|
||||||
|
print(f"📍 Qdrant: {QDRANT_URL}/{QDRANT_COLLECTION}")
|
||||||
|
print(f"🧠 Ollama: {OLLAMA_URL}/{EMBEDDING_MODEL}")
|
||||||
|
print(f"👤 User: {USER_ID}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
if args.once:
|
||||||
|
print("Running once...")
|
||||||
|
session_file = get_current_session_file()
|
||||||
|
if session_file:
|
||||||
|
watch_session(session_file, args.dry_run)
|
||||||
|
else:
|
||||||
|
print("No session found")
|
||||||
|
else:
|
||||||
|
print("Running as daemon (Ctrl+C to stop)...")
|
||||||
|
watch_loop(args.dry_run)
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
@@ -153,5 +153,15 @@ sessions_spawn({
|
|||||||
|
|
||||||
**Status:** Configured and ready
|
**Status:** Configured and ready
|
||||||
|
|
||||||
|
## Git Repository Initialized
|
||||||
|
|
||||||
|
**Setup:** Git repo initialized for workspace version control
|
||||||
|
|
||||||
|
**Commits:**
|
||||||
|
- `d1357c5` — Initial commit: 77 files, 10,822 insertions (workspace setup)
|
||||||
|
- `98d14be` — MEMORY.md updated with sub-agent and git config
|
||||||
|
|
||||||
|
**Status:** Clean working tree, tracking active
|
||||||
|
|
||||||
---
|
---
|
||||||
*Stored for long-term memory retention*
|
*Stored for long-term memory retention*
|
||||||
|
|||||||
168
memory/2026-02-14.md
Normal file
168
memory/2026-02-14.md
Normal file
@@ -0,0 +1,168 @@
|
|||||||
|
# Website Details - SpeedyFoxAI
|
||||||
|
|
||||||
|
**Domain:** speedyfoxai.com
|
||||||
|
**Hosted on:** deb2 (10.0.0.39)
|
||||||
|
**Web Root:** /root/html/ (Nginx serves from here, NOT /var/www/html/)
|
||||||
|
**Created:** February 13, 2026
|
||||||
|
**Created by:** Kimi (OpenClaw + Ollama) via SSH
|
||||||
|
|
||||||
|
## Critical Discovery
|
||||||
|
Nginx config (`/etc/nginx/sites-enabled/default`) sets `root /root/html;`
|
||||||
|
This means `/var/www/html/` is NOT the live document root - `/root/html/` is.
|
||||||
|
|
||||||
|
## Visitor Counter System
|
||||||
|
|
||||||
|
### Current Status
|
||||||
|
- **Count:** 288 (restored from Feb 13 backup)
|
||||||
|
- **Location:** `/root/html/count.txt`
|
||||||
|
- **Persistent storage:** `/root/html/.counter_total`
|
||||||
|
- **Script:** `/root/html/update_count_persistent.sh`
|
||||||
|
|
||||||
|
### Why It Reset
|
||||||
|
Old script read from nginx access.log which gets rotated by logrotate. When logs rotate, count drops to near zero. Lost ~350 visits (was ~400+, dropped to 46).
|
||||||
|
|
||||||
|
### Fix Applied
|
||||||
|
New persistent counter that:
|
||||||
|
1. Stores total in `.counter_total` file
|
||||||
|
2. Tracks last log line counted in `.counter_last_line`
|
||||||
|
3. Only adds NEW visits since last run
|
||||||
|
4. Handles log rotation gracefully
|
||||||
|
|
||||||
|
## Site Versions
|
||||||
|
|
||||||
|
### Current Live Version
|
||||||
|
- **File:** `/root/html/index.html` (served by nginx)
|
||||||
|
- **Source:** `/var/www/html/index.html` (edit here, copy to /root/html/)
|
||||||
|
- **Style:** Simple HTML/CSS (Rob's Tech Lab theme)
|
||||||
|
- **Features:** Visitor counter, 3 embedded YouTube videos
|
||||||
|
|
||||||
|
### Backup Version (Full Featured)
|
||||||
|
- **Location:** `/root/html_backup/20260213_155707/index.html`
|
||||||
|
- **Style:** Tailwind CSS, full SpeedyFoxAI branding
|
||||||
|
- **Features:** Dark mode, FAQ section, navigation, full counter
|
||||||
|
|
||||||
|
## YouTube Channel
|
||||||
|
**Name:** SpeedyFoxAI
|
||||||
|
**URL:** https://www.youtube.com/@SpeedyFoxAi
|
||||||
|
**Stats:** 5K+ subscribers, 51+ videos
|
||||||
|
|
||||||
|
### Embedded Videos
|
||||||
|
1. DIY AI Assistant Setup (kz-4l5roK6k)
|
||||||
|
2. Self-Hosted Tools Deep Dive (9IYNGK44EyM)
|
||||||
|
3. OpenClaw + Ollama Workflow (8Fncc5Sg2yg)
|
||||||
|
|
||||||
|
## File Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
/root/html/ # LIVE site (nginx root)
|
||||||
|
├── index.html # Main page
|
||||||
|
├── count.txt # Visitor count (288)
|
||||||
|
├── .counter_total # Persistent count storage
|
||||||
|
├── .counter_last_line # Log line tracking
|
||||||
|
├── update_count_persistent.sh # Counter script
|
||||||
|
├── websitememory.md # Documentation
|
||||||
|
├── downloads.html
|
||||||
|
├── fox720.jpg
|
||||||
|
└── favicon.png
|
||||||
|
|
||||||
|
/root/html_backup/ # Backups with timestamps
|
||||||
|
├── 20260214_071243/ # Pre-counter-script backup
|
||||||
|
├── 20260214_070713/ # Before counter fix
|
||||||
|
├── 20260214_070536/ # Full backup
|
||||||
|
└── 20260213_155707/ # Full version with counter
|
||||||
|
|
||||||
|
/var/www/html/ # Edit source (copy to /root/html/)
|
||||||
|
└── index.html
|
||||||
|
```
|
||||||
|
|
||||||
|
## Technical Details
|
||||||
|
|
||||||
|
### Counter JavaScript
|
||||||
|
```javascript
|
||||||
|
fetch("/count.txt?t=" + Date.now())
|
||||||
|
.then(r => r.text())
|
||||||
|
.then(n => {
|
||||||
|
document.getElementById("visit-count").textContent =
|
||||||
|
parseInt(n || 0).toString().padStart(6, "0");
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Counter Display
|
||||||
|
- Location: Footer
|
||||||
|
- Format: "Visitors: 000288"
|
||||||
|
- Style: 10px font, opacity 0.5
|
||||||
|
|
||||||
|
### Backup Strategy
|
||||||
|
```bash
|
||||||
|
DT=$(date +%Y%m%d_%H%M%S)
|
||||||
|
mkdir -p /root/html_backup/${DT}
|
||||||
|
cp -r /root/html/* /root/html_backup/${DT}/
|
||||||
|
cp /var/www/html/* /root/html_backup/${DT}/
|
||||||
|
```
|
||||||
|
|
||||||
|
## SEO & Content
|
||||||
|
- **Title:** Rob's Tech Lab | Local AI & Self-Hosted Tools
|
||||||
|
- **Meta:** None (simple version)
|
||||||
|
- **Schema:** None (simple version)
|
||||||
|
- **Full version has:** Schema.org FAQPage, structured data
|
||||||
|
|
||||||
|
## Social Links
|
||||||
|
- YouTube: @SpeedyFoxAi
|
||||||
|
- Discord: mdkrush
|
||||||
|
- GitHub: mdkrush
|
||||||
|
- Twitter: mdkrush
|
||||||
|
|
||||||
|
## Creator
|
||||||
|
**Name:** Rob
|
||||||
|
**Brand:** SpeedyFoxAI
|
||||||
|
**Focus:** Self-hosting, local AI, automation tools
|
||||||
|
**Personality:** Comical/structured humor
|
||||||
|
|
||||||
|
## Status
|
||||||
|
- **Counter:** Working (shows 000288)
|
||||||
|
- **HTML:** Valid structure, no misaligned code
|
||||||
|
- **Backups:** Multiple timestamps available
|
||||||
|
- **Documentation:** /root/html/websitememory.md
|
||||||
|
|
||||||
|
Stored: February 14, 2026
|
||||||
|
|
||||||
|
|
||||||
|
## Relationship to YouTube
|
||||||
|
The SpeedyFoxAI.com website complements the YouTube channel @SpeedyFoxAi. It serves as a hub for video content, resources, and contact info, with embedded videos linking directly to the channel. Design and branding are consistent across both platforms.
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## UPDATE - Feb 14, 07:21
|
||||||
|
|
||||||
|
### Counter Reset Issue - ROOT CAUSE FOUND
|
||||||
|
**Problem:** Count kept resetting to current nginx log lines (86, 89, etc.)
|
||||||
|
|
||||||
|
**Root Cause:** Old script `/root/html/update_count.sh` still existed and was running:
|
||||||
|
```bash
|
||||||
|
#!/bin/bash
|
||||||
|
COUNT=$(wc -l < /var/log/nginx/access.log 2>/dev/null || echo 0)
|
||||||
|
echo "$COUNT" > /root/html/count.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
This script was periodically overwriting count.txt with nginx log line count, overriding the persistent counter.
|
||||||
|
|
||||||
|
**Fix Applied:**
|
||||||
|
- Removed `/root/html/update_count.sh`
|
||||||
|
- Restored count.txt to 288
|
||||||
|
- Persistent counter now working correctly
|
||||||
|
|
||||||
|
**Lesson:** Check for competing scripts before implementing fixes.
|
||||||
|
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rule Added - Feb 14, 2026
|
||||||
|
**Always validate after changes.** No exceptions.
|
||||||
|
- Test functionality
|
||||||
|
- Verify file integrity
|
||||||
|
- Check permissions
|
||||||
|
- Confirm expected output
|
||||||
|
|
||||||
|
Applied retroactively to today’s counter fix.
|
||||||
|
|
||||||
Reference in New Issue
Block a user