---
name: hermes-cron-management
description: Diagnose, consolidate, and fix Hermes cron jobs across profiles. Covers the gateway dependency, stuck-job detection, cross-profile migration, and direct jobs.json editing when the cronjob tool can't reach other profiles.
version: 1.1.0
author: Hermes Agent
platforms: [linux]
---

# Hermes Cron Management

Diagnose and fix Hermes cron jobs — especially when jobs are stuck because their profile has no running gateway.

## When to Use

- Cron jobs have `next_run_at` timestamps in the past
- Jobs were created in a profile that no longer has a gateway running
- Consolidating scattered cron jobs into a single profile with a reliable gateway
- Validating that all cron jobs across all profiles are progressing properly

## Core Rule: Gateway Dependency

**Hermes cron jobs only execute when a gateway process is running for that profile.** A job defined in `~/.hermes/profiles/<name>/cron/jobs.json` will never fire unless a gateway process is active for `<name>`. The `hermes.service` systemd unit or a `hermes gateway run --profile <name>` process must be running.

This is the #1 cause of stuck cron jobs. Before debugging anything else, check: is a gateway running for this profile?

## Diagnosis Workflow

### 1. List all profiles and their cron jobs

```bash
for p in $(ls ~/.hermes/profiles/); do
  f="$HOME/.hermes/profiles/$p/cron/jobs.json"
  [ -f "$f" ] && echo "=== $p ===" && python3 -c "
import json
with open('$f') as fh:
    data = json.load(fh)
for j in data.get('jobs', []):
    print(f'  {j[\"id\"][:12]}  {j[\"name\"]}  enabled={j[\"enabled\"]}  state={j[\"state\"]}  next={j.get(\"next_run_at\",\"?\")}  last_status={j.get(\"last_status\",\"?\")}')
"
done
```

### 2. Check which profiles have running gateways

```bash
ps aux | grep 'hermes.*gateway run' | grep -v grep
```

### 3. Cross-reference

Any profile with cron jobs but NO gateway process → those jobs are stuck. Their `next_run_at` will be frozen in the past.

## Fix: Consolidate into a Profile with a Running Gateway
The reliable fix: move orphaned jobs into a profile that has a running gateway.

### Step 1: Verify the target profile

```bash
# Confirm gateway is running
ps aux | grep 'hermes.*gateway.*general' | grep -v grep

# Confirm cron scheduler is active in gateway logs
grep 'cron\|tick' ~/.hermes/profiles/general/logs/gateway.log | tail -5

# Confirm required models are available
curl -s localhost:11434/api/tags | python3 -c "import sys,json; [print(m['name']) for m in json.load(sys.stdin).get('models',[])]"
```

### Step 2: Verify dependencies exist in target profile

Before migrating, confirm the target profile has:
- Required skills (check `~/.hermes/profiles/<target>/skills/`)
- Required scripts (check `~/.hermes/profiles/<target>/scripts/`)
- Workdir paths exist on disk

### Step 3: Create the job in the target profile

Use `cronjob(action='create', ...)` from within the target profile's session. Match the original job's schedule, model, provider, skills, toolsets, workdir, and deliver settings.

**For no_agent script jobs:** The `cronjob` tool requires scripts to be in `~/.hermes/scripts/` relative to the profile. Copy the script from the source profile first:

```bash
cp ~/.hermes/profiles/<source>/scripts/<script>.sh ~/.hermes/profiles/<target>/scripts/<script>.sh
```

Then create with `no_agent=true` and `script="<script>.sh"`.

### Step 4: Disable the source job

The `cronjob` tool only manages jobs in the current profile. To disable jobs in OTHER profiles, edit their `jobs.json` directly:

```python
import json
with open('/home/n8n/.hermes/profiles/<source>/cron/jobs.json') as f:
    data = json.load(f)
for j in data['jobs']:
    j['enabled'] = False
    j['state'] = 'paused'
    j['paused_reason'] = 'Migrated to <target> profile (job <new_id>)'
with open('/home/n8n/.hermes/profiles/<source>/cron/jobs.json', 'w') as f:
    json.dump(data, f, indent=2)
```

### Step 5: Verify

```bash
# Target: all jobs enabled and scheduled
python3 -c "import json; d=json.load(open('$HOME/.hermes/profiles/<target>/cron/jobs.json')); [print(j['name'], j['enabled'], j['state']) for j in d['jobs']]"

# Source: all jobs disabled
python3 -c "import json; d=json.load(open('$HOME/.hermes/profiles/<source>/cron/jobs.json')); [print(j['name'], j['enabled'], j['state']) for j in d['jobs']]"
```

## Common Failure Patterns

### Finance profile with wrong base_url

The finance profile had `base_url: http://10.0.0.26:8000/v1` (epyc server, powered off) but `model: deepseek-v4-pro:cloud` with `provider: custom:ollama`. The base_url should have been `http://localhost:11434/v1`. The model and provider were correct — only the base_url was wrong. Fix:

```yaml
model:
  base_url: http://localhost:11434/v1  # was 10.0.0.26:8000/v1
  default: deepseek-v4-pro:cloud
  provider: custom:ollama
```

### Cron job has no model configured

If `last_error` says "has no model configured (job.model=None)", the job was created without a model override AND the profile's config.yaml has no `model.default` set. Fix: update the job with an explicit model, or set `model.default` in the profile's config.yaml.

## Pitfalls

- **Cron jobs need a gateway.** This is the #1 cause of stuck jobs. Always check gateway status first.
- **The `cronjob` tool is profile-scoped.** It only manages jobs in the current session's profile. Cross-profile operations require direct `jobs.json` editing.
- **Script jobs need scripts in the profile's own scripts dir.** The `cronjob` tool rejects absolute paths and paths from other profiles. Copy the script first.
- **Don't ask Claude to validate local infrastructure.** Claude runs on 10.0.0.28 and has no access to local `.hermes` trees, gateway processes, or Ollama. Validate locally.
- **Gateway logs are per-profile.** Check `~/.hermes/profiles/<name>/logs/gateway.log`, not the base `~/.hermes/logs/gateway.log` (which may be stale from an old gateway process).
- **Multiple gateways can't share a Telegram bot token.** If the general gateway shows "Telegram bot token already in use (PID X)", another profile's gateway already claimed it. This is normal — the general gateway still runs cron jobs.
- **Config cleanup can break cron.** Removing `cron:`, `gateway:`, or `streaming:` sections from a profile's config.yaml during cleanup will prevent the scheduler from initializing even if a gateway is running. CLI-only profiles that still have cron jobs need these sections.
- **State files may use profile-specific paths.** Scripts may hardcode state file paths to a specific profile (e.g., `~/.hermes/profiles/finance/cron_email_state.json`). When migrating jobs between profiles, check for hardcoded profile paths in scripts — they may need updating or the state file may need copying.
- **`cronjob(action='pause')` only works within the current profile.** The tool searches the current session's profile only. To disable jobs in OTHER profiles, edit their `jobs.json` directly (see Step 4 in Consolidation workflow). The tool will return "Job with ID not found" for cross-profile job IDs — that's expected, not a bug.
- **State file debugging: check what the script actually writes vs. what you expect.** If a state field shows a stale/garbage value (e.g., `last_ingested_at: 24`), grep the script for that field name. If it's only in `init_state()` and never set in the hot path, the field was never being updated. The fix is adding the write in the processing function, not patching the state file. See `references/email-ingest-state-debug.md` for a worked example.
- **State field consistency across creation paths.** When adding a new field to folder state dicts, add it to ALL three creation paths: `init_state()`, the folder-refresh path in `_main()`, and any migration code. Use `setdefault` in the hot path as defensive fallback, but seed the field at creation time so it's never missing. A field present in only some creation paths causes asymmetry (e.g., `sum(folder.ingested) != stats.ingested` because pre-fix folders lack the key).
- **Bash `set -e` + `((VAR++))` kills scripts on first zero-valued counter**: When a script uses `set -euo pipefail` and increments counters with `((SKIPPED++))`, the post-increment evaluates to the OLD value. If `SKIPPED=0` and the first iteration hits the skip branch, `((SKIPPED++))` evaluates to 0 (falsy in bash), `set -e` treats it as a command failure, and the script dies immediately. Only the header prints — no summary, no further processing. Fix: replace all `((VAR++))` with `VAR=$((VAR + 1))`. The `$((...))` form always returns exit code 0 regardless of the computed value. Same applies to `((VAR--))` → `VAR=$((VAR - 1))`. This is a classic `set -e` gotcha, not specific to cron scripts — any bash script with counters under `set -e` is vulnerable.
- **Monolithic agent prompts silently block downstream phases**: When a cron job uses an LLM agent to execute multiple sequential phases, a single sticky phase (e.g., Docker container recreation failing) can consume the entire tool-call budget. Later phases never run, and the report (typically the last phase) is never written — so the failure is silent. The agent's `last_status` may even report `ok` because the phase that ran didn't error. Fix: move multi-phase work into a `no_agent` bash script with real `||` error handling, per-phase timeouts, and a `FAILED` flag. The script guarantees every phase runs regardless of prior failures. See `references/monolithic-agent-prompt-pitfall.md`.
- **Docker container recreation via `docker inspect` reconstruction is fragile**: Capturing HostConfig and reconstructing a `docker run` command fails on port conflicts (new container can't bind while old holds the port), config gaps (missing env/cmd/args), and flag drift (new image drops old flags). Fix: migrate standalone containers to `docker compose` — config lives in a checked-in `compose.yaml`, `docker compose up -d` handles recreation correctly. See `references/docker-container-update-pitfall.md`.
- **Qdrant point IDs from `points_count` cause silent data loss**: Using `points_count` as the next point ID breaks when points are deleted (count < max ID → collision) and under concurrency (two runs get same count → overwrite). Fix: use deterministic UUIDv5 IDs keyed on content (`uuid5(namespace, f"{message_id}:{chunk_idx}")`). Same input always maps to same ID — idempotent upserts, no collisions. See `references/qdrant-deterministic-point-ids.md`.
- **Verify job output content, not just `last_status: ok`.** A job can report `ok` while producing garbage output or silently failing. Always read the latest output file and check external side effects (Qdrant point counts, state file timestamps, actual data written). See `references/verify-job-output.md`.
- **Cross-profile batch validation workflow.** When auditing all cron jobs across all profiles, use `execute_code` to batch-query `hermes -p <name> cron list`, then read each job's latest output and verify external side effects. See `references/cross-profile-cron-validation.md`.
- **Parallel delegation validation.** When the user asks to validate each job individually ("ask dev to validate, one at a time"), dispatch jobs to subagents in parallel batches of up to 3 using `delegate_task(tasks=[...])`. Each subagent reads output files, verifies Qdrant side effects, and reports WORKING/BROKEN with evidence. See `references/parallel-delegation-validation.md`.
- **Docker container recreation can break on CLI flag changes.** When a cron job updates Docker images and recreates containers, the new image may drop flags that the old image supported. The container crash-loops with `error: unknown option`. Compare `--help` between old and new images, then either roll back or adapt. See `references/docker-container-update-pitfall.md`.
- **Claude validation loop for script fixes.** When fixing a cron job's script, use the loop: fix → ask Claude to validate → apply Claude's feedback → re-validate until Claude says "CLEAN — no issues." Claude catches edge cases (missing init_state fields, folder-refresh path gaps, counter semantics) that are easy to miss in a single pass. See `references/claude-validation-loop.md`.
- **`last_status: error` can be stale — re-run the script manually before diagnosing.** A job may show `last_status: error` from a transient failure days ago while the script runs clean today. The error could have been a one-time environmental issue (malformed state file, network blip, OOM) that self-resolved. Always run the script manually (`bash ~/.hermes/profiles/<p>/scripts/<name>.sh`) before concluding the job is broken. If it runs clean, the error was transient — note it and move on. Only investigate further if the manual run also fails.\n- **Transient I/O contention on shared state files.** A script that reads a JSON state file (e.g., `.usage.json`) can fail with exit 1 and truncated output if another process (curator, usage tracker) is writing the file at the same moment. The inline Python JSON parse fails, `2>/dev/null` suppresses the error, and `set -e` kills the script mid-loop. If the script runs clean on retry (both interactive and in `env -i` cron simulation), the failure was transient — no code fix needed. See `references/simulate-cron-environment.md` for the reproduction recipe.\n- **Simulate cron's minimal environment to isolate env-vs-code bugs.** When a script fails in cron but works in an interactive shell, use `env -i HOME=/home/n8n PATH=/usr/bin:/bin SHELL=/bin/bash bash <script>` to reproduce cron's stripped environment. If it still works, the failure is environmental (transient I/O, TZ/locale, resource contention). If it fails, the script has a hidden dependency on interactive-shell state. See `references/simulate-cron-environment.md`.
- **Docker Compose `external: true` network + missing `networks:` block on a service → DNS failure.** If a service is missing its `networks:` block, it lands on the default network and can't resolve other services by container/service name. The `external: true` flag itself doesn't break DNS — the missing `networks:` block does. Fix: add `networks: - <name>` to the service. If no outside containers depend on the external network, switch to compose-managed (`driver: bridge`) for simpler DNS. Before switching, verify the old external network is empty: `docker network inspect <name>` and check `Containers: {}`.
- **Bash script Claude validation: specific gotchas Claude catches.** When converting an agent-driven cron job to a `no_agent` bash script, run it through the Claude validation loop (fix → scp → ask.sh → apply → re-validate). Claude reliably catches: unguarded `git commit` under `set -e` (exits on "nothing to commit"), missing `mkdir -p` before report file redirects (set -e kill before any phase runs), `exit 1` in error handlers that skip the summary, push-after-commit-fail false positives (push runs even when commit failed → misleading PASS), and `pip` vs `python3 -m pip` naming fragility. See `references/bash-script-claude-validation.md` for the full gotcha catalog from a 3-round validation session.

## Bulk Model Updates

To update all agent-driven jobs to a new model at once:

```bash
# From within the target profile's session
for job_id in <id1> <id2> <id3>; do
  cronjob(action='update', job_id="$job_id", model={"model": "kimi-k2.7-code:cloud", "provider": "custom:ollama"})
done
```

Skip `no_agent` jobs — they have `model: null` and don't need a model.

## Pre-Migration Dependency Checklist

Before migrating a job into a target profile, verify every dependency exists:

| Check | Command |
|-------|---------|
| Skills exist | `find ~/.hermes/profiles/<target>/skills -name "SKILL.md" -path "*/<skill>/*"` |
| Toolsets in config | `grep -A20 'platform_toolsets:' ~/.hermes/profiles/<target>/config.yaml` |
| Workdir exists | `ls -d /home/n8n/workspace/<workdir>` |
| Scripts (for no_agent) | `ls ~/.hermes/profiles/<target>/scripts/<script>.sh` |
| Model in Ollama | `curl -s localhost:11434/api/tags \| python3 -c "import sys,json; print([m['name'] for m in json.load(sys.stdin)['models']])"` |
| External services | Qdrant: `curl -s http://10.0.0.22:6333/collections`, SearXNG: `curl -s "http://10.0.0.8:8888/search?q=test&format=json"`, Ollama embed: `curl -s localhost:11434/api/embed -d '{"model":"snowflake-arctic-embed2:latest","input":"test"}'` |
| CLI tools | `which himalaya`, `himalaya --version` |

## Verifying a Job Is Actually Working

Don't trust `last_status: ok` alone. Check the output:

```bash
# List recent output files
ls -lt ~/.hermes/profiles/<profile>/cron/output/<job_id>/

# Read the latest
cat $(ls -t ~/.hermes/profiles/<profile>/cron/output/<job_id>/*.md | head -1) | tail -40
```

For jobs that write to external state (Qdrant, databases), verify the side effect:
```bash
# Qdrant point count
curl -s -X POST http://10.0.0.22:6333/collections/<name>/points/count -H "Content-Type: application/json" -d '{}'
```