121 lines
7.0 KiB
Markdown
121 lines
7.0 KiB
Markdown
---
|
|
name: model-capability-benchmark
|
|
description: "Use when picking a model for a capability. Benchmark."
|
|
version: 1.0.0
|
|
author: Hermes Agent
|
|
license: MIT
|
|
platforms: [linux]
|
|
metadata:
|
|
hermes:
|
|
tags: [benchmark, evaluation, model-selection, ollama, extraction]
|
|
related_skills: [agent-routing, cognee-brain, save, save-q-memory]
|
|
---
|
|
|
|
# Model Capability Benchmark
|
|
|
|
Use when the user needs to pick which model to use for a specific capability (memory
|
|
extraction, entity/relation extraction, classification, summarization quality, etc.).
|
|
The goal is a data-backed model choice, not a vibe check.
|
|
|
|
## When to Use
|
|
|
|
Load this skill when the user needs to pick which model to use for a specific
|
|
capability (memory extraction, entity/relation extraction, classification,
|
|
summarization quality, etc.). The goal is a data-backed model choice, not a vibe check.
|
|
|
|
Triggers:
|
|
- "which model should I use for X"
|
|
- "run a test of the models we have"
|
|
- "compare models on <capability>"
|
|
- Selecting a model for a pipeline (memory extraction, Cognee-style work, etc.)
|
|
|
|
## Method
|
|
|
|
0. **Run the SAME test on every candidate — no variations.** When the user says
|
|
"test these models" or "do the same test", send the identical prompt, identical
|
|
`temperature`, identical system message to every model. Do NOT add toggles
|
|
(thinking on/off, different prompts, extra modes) unless the user explicitly asks
|
|
for a comparison of those modes. The user corrected this directly: "WHY DIDN'T you
|
|
just do the same test??? DO not turn off thinking. Just DO THE SAME TEST." A
|
|
benchmark's value is comparability; a variation you introduce silently breaks it.
|
|
|
|
1. **Enumerate the candidate models.** For Ollama cloud models, `ollama list` shows
|
|
them; `curl -s http://localhost:11434/api/tags` reveals `remote_model`/`remote_host`
|
|
(cloud models have a `remote_host`; local models have a real byte size). Filter to
|
|
the requested scope (e.g. "cloud only" = has `remote_host`). For a non-Ollama
|
|
endpoint (e.g. a llama.cpp server on another box), hit its OpenAI-compatible
|
|
`/v1/chat/completions` directly with `urllib` — same system prompt, same
|
|
`temperature: 0`, same test cases. The harness just needs a `chat(messages)`
|
|
function; the scoring is identical regardless of backend.
|
|
|
|
2. **Write a test-case set.** Each case = input turns + expected extraction + explicit
|
|
fail conditions. Cover the hard cases for the capability, not just happy paths. For
|
|
memory extraction the canonical hard cases are: identity/stable facts, preference
|
|
vs one-off, update/contradiction, relative time, negation, specificity, multi-entity
|
|
relation, pronoun resolution (needs prior turn), plan vs completed fact, abstention
|
|
(no facts), assistant contamination, quantity/attribute, soft preference + intensity,
|
|
two-people-easy-to-merge.
|
|
|
|
3. **Build a harness** that sends each case to each model with a fixed system prompt
|
|
demanding a clean structured list (JSON array), `temperature=0`, and captures raw
|
|
output. Save raw outputs to disk (JSON) — never score in the same pass that runs.
|
|
|
|
4. **Score kept / missed / invented** per case, then total. A good extractor is high
|
|
recall (kept) with near-zero inventions. For Cognee-style work also reject answers
|
|
that aren't clean entity/relation lists.
|
|
|
|
5. **Pick the winner** on recall + zero inventions, and report per-case detail for the
|
|
interesting failures (not just totals).
|
|
|
|
## Pitfalls
|
|
|
|
- **Normalize BOTH sides before substring scoring.** Expected facts and model output
|
|
must go through the same `re.sub(r'[^a-z0-9 ]', ' ', s.lower())` — otherwise
|
|
hyphenated facts ("sci-fi", "15-gauge", "fine-tune", "gpt-oss-120b") are falsely
|
|
marked missed. This bit the first scoring pass.
|
|
- **Retired cloud models return HTTP 410.** A model in `ollama list` can still be
|
|
retired upstream; every call returns `ERROR: ... was retired at ... (status code: 410)`.
|
|
Detect this and exclude the model rather than scoring it as zero-kept.
|
|
- **Verify relative-time anchors yourself.** "last Tuesday" relative to a reference
|
|
date must be computed with `date -d <ref> +%A` etc. The user's expected answer may
|
|
itself be wrong (e.g. "Aug 19" when Aug 19 is a Wednesday) — compute the correct
|
|
date and flag the discrepancy rather than silently scoring against a wrong target.
|
|
- **Contamination test needs an assistant-role turn.** To test that the model ignores
|
|
assistant-suggested facts, the middle turn must be `role: "assistant"`, not user.
|
|
- **Substring scoring can give false credit.** A model that *answers* instead of
|
|
*extracting* (e.g. recommends vector DBs) may contain the expected keywords in prose
|
|
and score as "kept" when it actually failed the format. Inspect raw output for the
|
|
cases that matter before trusting the score.
|
|
- **Score from raw output, not from a live re-run.** Models are non-deterministic even
|
|
at temperature 0; re-running changes results. Persist raw outputs and score the file.
|
|
- **Large cloud models are slow — run sequentially in the background.** A 397b/675b/120b
|
|
cloud model can take ~20s per test case, so 14 cases ≈ 5 min per model. A single
|
|
foreground run of several models will hit the terminal timeout. Run one model per
|
|
background process (or a `for` loop over models in ONE background process), and poll
|
|
the error log for `DONE <model>` markers rather than blocking on `wait`. Background
|
|
processes that get killed mid-run leave a partial JSON — check which models actually
|
|
completed before scoring.
|
|
- **Strip fences/whitespace in the abstention check.** A model that correctly returns
|
|
an empty list may wrap it as ```json\n[]\n``` or `[ ]`. The abstention check must
|
|
strip ``` fences and whitespace and accept `[]`, `[ ]`, `""`, `null` — otherwise a
|
|
correct abstention is falsely scored as a failure.
|
|
- **Negation FORBID substrings false-positive.** A FORBID fact like "allergic to
|
|
shellfish" will match inside a *correct* negation ("not allergic to shellfish") and
|
|
be falsely scored as invented. For negation cases, either check for the negated form
|
|
explicitly (e.g. forbid "allergic to shellfish" only when NOT preceded by "not"), or
|
|
inspect the raw output manually before trusting an "invented" flag on a negation test.
|
|
- **Non-Ollama endpoints need their own auth.** A llama.cpp server (e.g. e1's
|
|
qwen38-27b at `http://10.0.0.26:8099/v1`) may require a Bearer key while `/health`
|
|
and `/v1/models` stay public — only a real completion proves auth. Reuse the same
|
|
harness by swapping the transport; don't assume the `ollama` python client works for it.
|
|
|
|
## Support Files
|
|
|
|
- `references/memory-extraction-results.md` — 2026-08-26 benchmark of 11 Ollama cloud
|
|
models on memory extraction, with per-model findings and the winner.
|
|
- `references/recommended-sampling-params.md` — official temperature/top_p guidance
|
|
per model (DeepSeek V4 Pro = 1.0/1.0, MiniMax M3 = 1.0/0.95, API use-case table),
|
|
plus how temperature is actually set for Hermes custom providers
|
|
(`extra_body.temperature`, not a top-level key) and how to research user-experience
|
|
threads (HN Algolia API). Load when the user asks about optimal temperature for a model.
|