hermes-skills/hermes-cron-management/SKILL.md

name, description, version, author, platforms

name	description	version	author	platforms
hermes-cron-management	Diagnose, consolidate, and fix Hermes cron jobs across profiles. Covers the gateway dependency, stuck-job detection, cross-profile migration, and direct jobs.json editing when the cronjob tool can't reach other profiles.	1.1.0	Hermes Agent	linux

Hermes Cron Management

Diagnose and fix Hermes cron jobs — especially when jobs are stuck because their profile has no running gateway.

When to Use

• Cron jobs have next_run_at timestamps in the past
• Jobs were created in a profile that no longer has a gateway running
• Consolidating scattered cron jobs into a single profile with a reliable gateway
• Validating that all cron jobs across all profiles are progressing properly

Core Rule: Gateway Dependency

Hermes cron jobs only execute when a gateway process is running for that profile. A job defined in ~/.hermes/profiles/<name>/cron/jobs.json will never fire unless a gateway process is active for <name>. The hermes.service systemd unit or a hermes gateway run --profile <name> process must be running.

This is the #1 cause of stuck cron jobs. Before debugging anything else, check: is a gateway running for this profile?

Diagnosis Workflow

1. List all profiles and their cron jobs

for p in $(ls ~/.hermes/profiles/); do
  f="$HOME/.hermes/profiles/$p/cron/jobs.json"
  [ -f "$f" ] && echo "=== $p ===" && python3 -c "
import json
with open('$f') as fh:
    data = json.load(fh)
for j in data.get('jobs', []):
    print(f'  {j[\"id\"][:12]}  {j[\"name\"]}  enabled={j[\"enabled\"]}  state={j[\"state\"]}  next={j.get(\"next_run_at\",\"?\")}  last_status={j.get(\"last_status\",\"?\")}')
"
done

2. Check which profiles have running gateways

ps aux | grep 'hermes.*gateway run' | grep -v grep

3. Cross-reference

Any profile with cron jobs but NO gateway process → those jobs are stuck. Their next_run_at will be frozen in the past.

Fix: Consolidate into a Profile with a Running Gateway

The reliable fix: move orphaned jobs into a profile that has a running gateway.

Step 1: Verify the target profile

# Confirm gateway is running
ps aux | grep 'hermes.*gateway.*general' | grep -v grep

# Confirm cron scheduler is active in gateway logs
grep 'cron\|tick' ~/.hermes/profiles/general/logs/gateway.log | tail -5

# Confirm required models are available
curl -s localhost:11434/api/tags | python3 -c "import sys,json; [print(m['name']) for m in json.load(sys.stdin).get('models',[])]"

Step 2: Verify dependencies exist in target profile

Before migrating, confirm the target profile has:

• Required skills (check ~/.hermes/profiles/<target>/skills/)
• Required scripts (check ~/.hermes/profiles/<target>/scripts/)
• Workdir paths exist on disk

Step 3: Create the job in the target profile

Use cronjob(action='create', ...) from within the target profile's session. Match the original job's schedule, model, provider, skills, toolsets, workdir, and deliver settings.

For no_agent script jobs: The cronjob tool requires scripts to be in ~/.hermes/scripts/ relative to the profile. Copy the script from the source profile first:

cp ~/.hermes/profiles/<source>/scripts/<script>.sh ~/.hermes/profiles/<target>/scripts/<script>.sh

Then create with no_agent=true and script="<script>.sh".

Step 4: Disable the source job

The cronjob tool only manages jobs in the current profile. To disable jobs in OTHER profiles, edit their jobs.json directly:

import json
with open('/home/n8n/.hermes/profiles/<source>/cron/jobs.json') as f:
    data = json.load(f)
for j in data['jobs']:
    j['enabled'] = False
    j['state'] = 'paused'
    j['paused_reason'] = 'Migrated to <target> profile (job <new_id>)'
with open('/home/n8n/.hermes/profiles/<source>/cron/jobs.json', 'w') as f:
    json.dump(data, f, indent=2)

Step 5: Verify

# Target: all jobs enabled and scheduled
python3 -c "import json; d=json.load(open('$HOME/.hermes/profiles/<target>/cron/jobs.json')); [print(j['name'], j['enabled'], j['state']) for j in d['jobs']]"

# Source: all jobs disabled
python3 -c "import json; d=json.load(open('$HOME/.hermes/profiles/<source>/cron/jobs.json')); [print(j['name'], j['enabled'], j['state']) for j in d['jobs']]"

Common Failure Patterns

Finance profile with wrong base_url

The finance profile had base_url: http://10.0.0.26:8000/v1 (epyc server, powered off) but model: deepseek-v4-pro:cloud with provider: custom:ollama. The base_url should have been http://localhost:11434/v1. The model and provider were correct — only the base_url was wrong. Fix:

model:
  base_url: http://localhost:11434/v1  # was 10.0.0.26:8000/v1
  default: deepseek-v4-pro:cloud
  provider: custom:ollama

Cron job has no model configured

If last_error says "has no model configured (job.model=None)", the job was created without a model override AND the profile's config.yaml has no model.default set. Fix: update the job with an explicit model, or set model.default in the profile's config.yaml.

Pitfalls

• Cron jobs need a gateway. This is the #1 cause of stuck jobs. Always check gateway status first.
• The cronjob tool is profile-scoped. It only manages jobs in the current session's profile. Cross-profile operations require direct jobs.json editing.
• Script jobs need scripts in the profile's own scripts dir. The cronjob tool rejects absolute paths and paths from other profiles. Copy the script first.
• Don't ask Claude to validate local infrastructure. Claude runs on 10.0.0.28 and has no access to local .hermes trees, gateway processes, or Ollama. Validate locally.
• Gateway logs are per-profile. Check ~/.hermes/profiles/<name>/logs/gateway.log, not the base ~/.hermes/logs/gateway.log (which may be stale from an old gateway process).
• Multiple gateways can't share a Telegram bot token. If the general gateway shows "Telegram bot token already in use (PID X)", another profile's gateway already claimed it. This is normal — the general gateway still runs cron jobs.
• Config cleanup can break cron. Removing cron:, gateway:, or streaming: sections from a profile's config.yaml during cleanup will prevent the scheduler from initializing even if a gateway is running. CLI-only profiles that still have cron jobs need these sections.
• State files may use profile-specific paths. Scripts may hardcode state file paths to a specific profile (e.g., ~/.hermes/profiles/finance/cron_email_state.json). When migrating jobs between profiles, check for hardcoded profile paths in scripts — they may need updating or the state file may need copying.
• cronjob(action='pause') only works within the current profile. The tool searches the current session's profile only. To disable jobs in OTHER profiles, edit their jobs.json directly (see Step 4 in Consolidation workflow). The tool will return "Job with ID not found" for cross-profile job IDs — that's expected, not a bug.
• State file debugging: check what the script actually writes vs. what you expect. If a state field shows a stale/garbage value (e.g., last_ingested_at: 24), grep the script for that field name. If it's only in init_state() and never set in the hot path, the field was never being updated. The fix is adding the write in the processing function, not patching the state file. See references/email-ingest-state-debug.md for a worked example.
• State field consistency across creation paths. When adding a new field to folder state dicts, add it to ALL three creation paths: init_state(), the folder-refresh path in _main(), and any migration code. Use setdefault in the hot path as defensive fallback, but seed the field at creation time so it's never missing. A field present in only some creation paths causes asymmetry (e.g., sum(folder.ingested) != stats.ingested because pre-fix folders lack the key).
• Bash set -e + ((VAR++)) kills scripts on first zero-valued counter: When a script uses set -euo pipefail and increments counters with ((SKIPPED++)), the post-increment evaluates to the OLD value. If SKIPPED=0 and the first iteration hits the skip branch, ((SKIPPED++)) evaluates to 0 (falsy in bash), set -e treats it as a command failure, and the script dies immediately. Only the header prints — no summary, no further processing. Fix: replace all ((VAR++)) with VAR=$((VAR + 1)). The $((...)) form always returns exit code 0 regardless of the computed value. Same applies to ((VAR--)) → VAR=$((VAR - 1)). This is a classic set -e gotcha, not specific to cron scripts — any bash script with counters under set -e is vulnerable.
• Monolithic agent prompts silently block downstream phases: When a cron job uses an LLM agent to execute multiple sequential phases, a single sticky phase (e.g., Docker container recreation failing) can consume the entire tool-call budget. Later phases never run, and the report (typically the last phase) is never written — so the failure is silent. The agent's last_status may even report ok because the phase that ran didn't error. Fix: move multi-phase work into a no_agent bash script with real || error handling, per-phase timeouts, and a FAILED flag. The script guarantees every phase runs regardless of prior failures. See references/monolithic-agent-prompt-pitfall.md.
• Docker container recreation via docker inspect reconstruction is fragile: Capturing HostConfig and reconstructing a docker run command fails on port conflicts (new container can't bind while old holds the port), config gaps (missing env/cmd/args), and flag drift (new image drops old flags). Fix: migrate standalone containers to docker compose — config lives in a checked-in compose.yaml, docker compose up -d handles recreation correctly. See references/docker-container-update-pitfall.md.
• Qdrant point IDs from points_count cause silent data loss: Using points_count as the next point ID breaks when points are deleted (count < max ID → collision) and under concurrency (two runs get same count → overwrite). Fix: use deterministic UUIDv5 IDs keyed on content (uuid5(namespace, f"{message_id}:{chunk_idx}")). Same input always maps to same ID — idempotent upserts, no collisions. See references/qdrant-deterministic-point-ids.md.
• Verify job output content, not just last_status: ok. A job can report ok while producing garbage output or silently failing. Always read the latest output file and check external side effects (Qdrant point counts, state file timestamps, actual data written). See references/verify-job-output.md.
• Cross-profile batch validation workflow. When auditing all cron jobs across all profiles, use execute_code to batch-query hermes -p <name> cron list, then read each job's latest output and verify external side effects. See references/cross-profile-cron-validation.md.
• Parallel delegation validation. When the user asks to validate each job individually ("ask dev to validate, one at a time"), dispatch jobs to subagents in parallel batches of up to 3 using delegate_task(tasks=[...]). Each subagent reads output files, verifies Qdrant side effects, and reports WORKING/BROKEN with evidence. See references/parallel-delegation-validation.md.
• Docker container recreation can break on CLI flag changes. When a cron job updates Docker images and recreates containers, the new image may drop flags that the old image supported. The container crash-loops with error: unknown option. Compare --help between old and new images, then either roll back or adapt. See references/docker-container-update-pitfall.md.
• Claude validation loop for script fixes. When fixing a cron job's script, use the loop: fix → ask Claude to validate → apply Claude's feedback → re-validate until Claude says "CLEAN — no issues." Claude catches edge cases (missing init_state fields, folder-refresh path gaps, counter semantics) that are easy to miss in a single pass. See references/claude-validation-loop.md.
• last_status: error can be stale — re-run the script manually before diagnosing. A job may show last_status: error from a transient failure days ago while the script runs clean today. The error could have been a one-time environmental issue (malformed state file, network blip, OOM) that self-resolved. Always run the script manually (bash ~/.hermes/profiles/<p>/scripts/<name>.sh) before concluding the job is broken. If it runs clean, the error was transient — note it and move on. Only investigate further if the manual run also fails.\n- Transient I/O contention on shared state files. A script that reads a JSON state file (e.g., .usage.json) can fail with exit 1 and truncated output if another process (curator, usage tracker) is writing the file at the same moment. The inline Python JSON parse fails, 2>/dev/null suppresses the error, and set -e kills the script mid-loop. If the script runs clean on retry (both interactive and in env -i cron simulation), the failure was transient — no code fix needed. See references/simulate-cron-environment.md for the reproduction recipe.\n- Simulate cron's minimal environment to isolate env-vs-code bugs. When a script fails in cron but works in an interactive shell, use env -i HOME=/home/n8n PATH=/usr/bin:/bin SHELL=/bin/bash bash <script> to reproduce cron's stripped environment. If it still works, the failure is environmental (transient I/O, TZ/locale, resource contention). If it fails, the script has a hidden dependency on interactive-shell state. See references/simulate-cron-environment.md.
• Docker Compose external: true network + missing networks: block on a service → DNS failure. If a service is missing its networks: block, it lands on the default network and can't resolve other services by container/service name. The external: true flag itself doesn't break DNS — the missing networks: block does. Fix: add networks: - <name> to the service. If no outside containers depend on the external network, switch to compose-managed (driver: bridge) for simpler DNS. Before switching, verify the old external network is empty: docker network inspect <name> and check Containers: {}.
• Bash script Claude validation: specific gotchas Claude catches. When converting an agent-driven cron job to a no_agent bash script, run it through the Claude validation loop (fix → scp → ask.sh → apply → re-validate). Claude reliably catches: unguarded git commit under set -e (exits on "nothing to commit"), missing mkdir -p before report file redirects (set -e kill before any phase runs), exit 1 in error handlers that skip the summary, push-after-commit-fail false positives (push runs even when commit failed → misleading PASS), and pip vs python3 -m pip naming fragility. See references/bash-script-claude-validation.md for the full gotcha catalog from a 3-round validation session.

Bulk Model Updates

To update all agent-driven jobs to a new model at once:

# From within the target profile's session
for job_id in <id1> <id2> <id3>; do
  cronjob(action='update', job_id="$job_id", model={"model": "kimi-k2.7-code:cloud", "provider": "custom:ollama"})
done

Skip no_agent jobs — they have model: null and don't need a model.

Pre-Migration Dependency Checklist

Before migrating a job into a target profile, verify every dependency exists:

Check	Command
Skills exist	find ~/.hermes/profiles/<target>/skills -name "SKILL.md" -path "*/<skill>/*"
Toolsets in config	grep -A20 'platform_toolsets:' ~/.hermes/profiles/<target>/config.yaml
Workdir exists	ls -d /home/n8n/workspace/<workdir>
Scripts (for no_agent)	ls ~/.hermes/profiles/<target>/scripts/<script>.sh
Model in Ollama	curl -s localhost:11434/api/tags | python3 -c "import sys,json; print([m['name'] for m in json.load(sys.stdin)['models']])"
External services	Qdrant: curl -s http://10.0.0.22:6333/collections, SearXNG: curl -s "http://10.0.0.8:8888/search?q=test&format=json", Ollama embed: curl -s localhost:11434/api/embed -d '{"model":"snowflake-arctic-embed2:latest","input":"test"}'
CLI tools	which himalaya, himalaya --version

Verifying a Job Is Actually Working

Don't trust last_status: ok alone. Check the output:

# List recent output files
ls -lt ~/.hermes/profiles/<profile>/cron/output/<job_id>/

# Read the latest
cat $(ls -t ~/.hermes/profiles/<profile>/cron/output/<job_id>/*.md | head -1) | tail -40

For jobs that write to external state (Qdrant, databases), verify the side effect:

# Qdrant point count
curl -s -X POST http://10.0.0.22:6333/collections/<name>/points/count -H "Content-Type: application/json" -d '{}'