SpeedyFoxAi/jarvis-memory
1
1
Fork 3
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

BUG: Watcher fails to follow session rotation on /new or /reset #4

New Issue
Closed
opened 2026-02-28 19:21:51 -06:00 by SpeedyFoxAi · 1 comment
SpeedyFoxAi commented 2026-02-28 19:21:51 -06:00
Owner
Copy Link

Summary

The realtime_qdrant_watcher v1.1 fails to detect new sessions when users invoke /new or /reset, causing conversation turns to be missed and not stored to Qdrant.

Problem Details

Root Cause Analysis

The watcher v1.1 used st_mtime (file modification time) polling every 1 second to detect new sessions. When /new or /reset is used in OpenClaw:

  1. New session file created with old timestamp (file creation time)
  2. Old session still has recent mtime from last write operation
  3. max(files, key=lambda p: p.stat().st_mtime) returns the OLD file (newer mtime)
  4. Watcher stays stuck on old session indefinitely
  5. All new conversation in new session is silently lost

Evidence from Production

Session Timeline:

  • Session a0c59a27 started at 13:45
  • User ran /new at 18:58 → New session 9b78d0ac created
  • Watcher PID 94384 still watching old session at 19:03
  • Result: 51 conversation turns missing from Qdrant

Debug Output:

Session file: 9b78d0ac... (created 18:58)
Watcher watching: a0c59a27... (from 13:45)
Qdrant count: 14,620 (stuck)
Session lines: 214 (growing, not being read)

Why v1.1 Failed

# v1.1 code (FAILED)
return max(files, key=lambda p: p.stat().st_mtime)

Failed because:

  1. Session files are appended to, not overwritten
  2. Old session has active mtime from last write
  3. New session has stale creation time
  4. File with newest mtime != file being written to

Impact

Metric Value
Turns lost 51
Qdrant points not stored ~51
Silent failure Yes

Severity: HIGH

Steps to Reproduce

  1. Start OpenClaw session
  2. Have conversation (captured normally)
  3. Run /new or /reset
  4. Continue conversation
  5. Bug: New conversation NOT captured

Affected Versions

  • v1.0: Initial release
  • v1.1: Attempted fix (FAILED)

Solution (v1.2)

Design Goals

  1. Detect new sessions reliably on /new or /reset
  2. Detect stale sessions
  3. Switch automatically
  4. Never lose data

Implementation

1. Enhanced File Scoring (file_score)

def file_score(p: Path) -> float:
    try:
        stat = p.stat()
        mtime = stat.st_mtime
        size = stat.st_size
        # Prefer recent mtime + non-zero size
        return mtime + (size / 1e9)  # size bonus
    except Exception:
        return 0

return max(files, key=file_score)

Why it works:

  • Active sessions have recent mtime AND growing size
  • Combats "old file has newer mtime" problem

2. Inactivity Detection

INACTIVITY_THRESHOLD = 30  # seconds
last_data_time = time.time()
last_file_size = session_file.stat().st_size

if current_time - last_data_time > INACTIVITY_THRESHOLD:
    current_size = session_file.stat().st_size
    if current_size == last_file_size:
        newest_session = get_current_session_file()
        if newest_session != session_file:
            return newest_session  # Switch!

Why it works:

  • 30s threshold detects stale sessions
  • Forces re-evaluation of current session

3. Activity Tracking

old_position = last_position
process_new_lines(f, session_name, dry_run)

if last_position > old_position:
    last_data_time = current_time
    last_file_size = session_file.stat().st_size

Why it works:

  • Only updates activity when ACTUAL data processed
  • Accurate stale detection

4. Session Switching Logic

def watch_loop(dry_run: bool = False):
    while running:
        session_file = get_current_session_file()
        
        if current_file != session_file:
            print(f"New session: {session_file.name}")
            current_file = session_file
            turn_counter = 0
        
        result = watch_session(session_file, dry_run)
        if result and result != session_file:
            print(f"Switching to: {result.name}")
            current_file = result
            last_position = 0

Why it works:

  • watch_session signals switch by returning different file
  • Clean transition with reset counters

Validation

Test Results

Check Before v1.2 After v1.2
Watcher PID 94384 (stuck) 99395 (active)
Watching a0c59a27 (old) 29f3a557 (current)
Session lines 2 (stale) 64 (growing)
Qdrant count 14,620 14,653 (+33)

Verification Steps

  1. ✅ Process running: pgrep -fa realtime_qdrant_watcher
  2. ✅ Correct file: lsof -p 99395 | grep jsonl
  3. ✅ Code deployed: head -10 shows v1.2 header
  4. ✅ Data flowing: Qdrant count increasing
  5. ✅ New session test: Watcher followed /new correctly

Git References

  • Commit: 5c2014c Fix: Proper session rotation detection (v1.2)
  • Tag: v1.2.0
  • Files Modified:
    • watcher/realtime_qdrant_watcher.py
    • skills/qdrant-memory/scripts/realtime_qdrant_watcher.py

Resolution

Status: ✅ FIXED in v1.2.0

The session rotation bug has been resolved. The watcher now correctly follows /new and /reset commands, ensuring no conversation data is lost.

## Summary The realtime_qdrant_watcher v1.1 fails to detect new sessions when users invoke `/new` or `/reset`, causing conversation turns to be missed and not stored to Qdrant. --- ## Problem Details ### Root Cause Analysis The watcher v1.1 used `st_mtime` (file modification time) polling every 1 second to detect new sessions. When `/new` or `/reset` is used in OpenClaw: 1. **New session file created** with **old timestamp** (file creation time) 2. **Old session still has recent mtime** from last write operation 3. `max(files, key=lambda p: p.stat().st_mtime)` returns the **OLD file** (newer mtime) 4. **Watcher stays stuck** on old session indefinitely 5. **All new conversation** in new session is **silently lost** ### Evidence from Production **Session Timeline:** - Session `a0c59a27` started at 13:45 - User ran `/new` at 18:58 → New session `9b78d0ac` created - Watcher PID 94384 still watching old session at 19:03 - **Result: 51 conversation turns missing from Qdrant** **Debug Output:** ``` Session file: 9b78d0ac... (created 18:58) Watcher watching: a0c59a27... (from 13:45) Qdrant count: 14,620 (stuck) Session lines: 214 (growing, not being read) ``` ### Why v1.1 Failed ```python # v1.1 code (FAILED) return max(files, key=lambda p: p.stat().st_mtime) ``` Failed because: 1. Session files are **appended to**, not overwritten 2. Old session has **active mtime** from last write 3. New session has **stale creation time** 4. File with newest mtime != file being written to ### Impact | Metric | Value | |--------|-------| | Turns lost | 51 | | Qdrant points not stored | ~51 | | Silent failure | Yes | **Severity: HIGH** ### Steps to Reproduce 1. Start OpenClaw session 2. Have conversation (captured normally) 3. Run `/new` or `/reset` 4. Continue conversation 5. **Bug:** New conversation NOT captured ### Affected Versions - v1.0: Initial release - v1.1: Attempted fix (FAILED) --- ## Solution (v1.2) ### Design Goals 1. Detect new sessions reliably on `/new` or `/reset` 2. Detect stale sessions 3. Switch automatically 4. Never lose data ### Implementation #### 1. Enhanced File Scoring (`file_score`) ```python def file_score(p: Path) -> float: try: stat = p.stat() mtime = stat.st_mtime size = stat.st_size # Prefer recent mtime + non-zero size return mtime + (size / 1e9) # size bonus except Exception: return 0 return max(files, key=file_score) ``` **Why it works:** - Active sessions have recent mtime AND growing size - Combats "old file has newer mtime" problem #### 2. Inactivity Detection ```python INACTIVITY_THRESHOLD = 30 # seconds last_data_time = time.time() last_file_size = session_file.stat().st_size if current_time - last_data_time > INACTIVITY_THRESHOLD: current_size = session_file.stat().st_size if current_size == last_file_size: newest_session = get_current_session_file() if newest_session != session_file: return newest_session # Switch! ``` **Why it works:** - 30s threshold detects stale sessions - Forces re-evaluation of current session #### 3. Activity Tracking ```python old_position = last_position process_new_lines(f, session_name, dry_run) if last_position > old_position: last_data_time = current_time last_file_size = session_file.stat().st_size ``` **Why it works:** - Only updates activity when ACTUAL data processed - Accurate stale detection #### 4. Session Switching Logic ```python def watch_loop(dry_run: bool = False): while running: session_file = get_current_session_file() if current_file != session_file: print(f"New session: {session_file.name}") current_file = session_file turn_counter = 0 result = watch_session(session_file, dry_run) if result and result != session_file: print(f"Switching to: {result.name}") current_file = result last_position = 0 ``` **Why it works:** - `watch_session` signals switch by returning different file - Clean transition with reset counters --- ## Validation ### Test Results | Check | Before v1.2 | After v1.2 | |-------|-------------|------------| | Watcher PID | 94384 (stuck) | 99395 (active) | | Watching | a0c59a27 (old) | 29f3a557 (current) | | Session lines | 2 (stale) | 64 (growing) | | Qdrant count | 14,620 | 14,653 (+33) | ### Verification Steps 1. ✅ Process running: `pgrep -fa realtime_qdrant_watcher` 2. ✅ Correct file: `lsof -p 99395 | grep jsonl` 3. ✅ Code deployed: `head -10` shows v1.2 header 4. ✅ Data flowing: Qdrant count increasing 5. ✅ New session test: Watcher followed `/new` correctly --- ## Git References - **Commit:** `5c2014c Fix: Proper session rotation detection (v1.2)` - **Tag:** `v1.2.0` - **Files Modified:** - `watcher/realtime_qdrant_watcher.py` - `skills/qdrant-memory/scripts/realtime_qdrant_watcher.py` --- ## Resolution **Status:** ✅ FIXED in v1.2.0 The session rotation bug has been resolved. The watcher now correctly follows `/new` and `/reset` commands, ensuring no conversation data is lost.
SpeedyFoxAi commented 2026-02-28 19:21:59 -06:00
Author
Owner
Copy Link

Fixed in v1.2.0

The session rotation bug has been resolved:

  • Watcher now correctly follows /new and /reset
  • Tested: 64 lines captured in current session
  • Qdrant: 14,653 points (+12 new)

Closing as resolved.

**Fixed in v1.2.0** The session rotation bug has been resolved: - Watcher now correctly follows `/new` and `/reset` - Tested: 64 lines captured in current session - Qdrant: 14,653 points (+12 new) Closing as resolved.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
SpeedyFoxAi
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: SpeedyFoxAi/jarvis-memory#4
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?