hermes-skills/qdrant-collection-management/SKILL.md

name, description, version, author, license, metadata

name	description	version	author	license	metadata
qdrant-collection-management	Manage Qdrant collections — consolidation, migration, deduplication, registry, and cleanup. Covers the scroll→transform→upsert→validate→delete pattern, batch processing with curl subprocess, and the collection registry file convention.	1.1.0	Hermes Agent	MIT	hermes tags related_skills qdrant collections migration dedup consolidation registry local-vector-memory

Qdrant Collection Management

Manage Qdrant collections at scale — consolidate scattered collections into a unified store, deduplicate near-duplicate points, maintain a registry file, and clean up stale data.

When to Use

• Consolidating multiple collections into one (e.g., merging kimi, vera into memories)
• Deduplicating points by cosine similarity
• Cleaning up stale/dormant collections
• Maintaining a collection registry with tags, ownership, and protection status
• Migrating points between collections (including re-embedding for dimension mismatches)

MCP vs Curl

The MCP Qdrant tools (mcp_better_qdrant_*) only expose: list, search, add, delete. They do NOT cover: collection info (dimensions, config), point scrolling, batch search, or point count. For these operations, use curl directly against the Qdrant REST API.

Always prefer MCP for what it covers (list, search, add, delete). Fall back to curl for everything else.

Consolidation Workflow

The proven pattern for merging collections:

1. Read source (scroll all points with vectors)

curl -s -X POST http://10.0.0.22:6333/collections/{source}/points/scroll \
  -H "Content-Type: application/json" \
  -d '{"limit": 100, "with_payload": true, "with_vector": true}'

Paginate using next_page_offset until it returns null. For large collections (10K+), use a Python script with curl subprocess — the requests library can hit port exhaustion on the sandbox.

2. Transform payloads

Add source and migrated_at fields to every point. Normalize text fields: if the source uses content or data instead of text, copy to text for unified search.

payload["source"] = "source_collection_name"
payload["migrated_at"] = "2026-06-28T00:00:00Z"
if "content" in payload and "text" not in payload:
    payload["text"] = payload["content"]

3. Upsert to target

curl -s -X PUT http://10.0.0.22:6333/collections/{target}/points \
  -H "Content-Type: application/json" \
  -d '{"points": [{"id": "...", "vector": [...], "payload": {...}}]}'

Batch in groups of 100. For large payloads, write to temp file and use -d @file.json to avoid argument list overflow.

4. Validate

curl -s -X POST http://10.0.0.22:6333/collections/{target}/points/count \
  -H "Content-Type: application/json" \
  -d '{"filter": {"must": [{"key": "source", "match": {"value": "source_name"}}]}}'

Count must match the source collection's point count.

5. Delete source

curl -s -X DELETE http://10.0.0.22:6333/collections/{source}

Or use mcp_better_qdrant_delete_collection for MCP-tracked deletions.

Dimension Mismatch

If source has different vector dimensions than target (e.g., 768-dim → 1024-dim), re-embed using Ollama before upserting:

r = subprocess.run(
    ["curl", "-s", "http://localhost:11434/api/embed",
     "-d", json.dumps({"model": "snowflake-arctic-embed2:latest", "input": text})],
    capture_output=True, text=True
)
vec = json.loads(r.stdout)["embeddings"][0]

Deduplication

Use cosine similarity search to find near-duplicates. The batch search API is efficient for large collections:

# Build batch searches
searches = []
for pt in chunk:
    searches.append({
        "vector": pt["vector"],
        "limit": 5,
        "score_threshold": 0.97,
        "filter": {"must_not": [{"has_id": [pt["id"]]}]}
    })

# Execute batch
curl -X POST /collections/{name}/points/search/batch -d '{"searches": [...]}'

Threshold guidance:

• 0.99: exact duplicates (same text, different source)
• 0.97: near-duplicates (same fact, slightly different wording)
• 0.95: semantic duplicates (same meaning, different phrasing)

At 0.97 on a 30K-point consolidated collection, expect ~20% duplicates from overlapping sources.

Important: When using Python for dedup, the requests library may fail with [Errno 99] Cannot assign requested address from the sandbox. Use subprocess.run(["curl", ...]) instead — curl works reliably. See references/dedup-script.py for the full implementation.

Collection Registry

Maintain a registry file at ~/workspace/general/qdrant-collections.json with:

{
  "version": "1.0.0",
  "collections": {
    "name": {
      "tags": ["project", "kb"],
      "description": "...",
      "owner": "project_name",
      "status": "active",
      "protected": false
    }
  },
  "tag_index": {
    "project": ["col1", "col2"],
    "kb": ["col1"]
  },
  "deleted_collections": {
    "old_name": {
      "date": "2026-06-28",
      "reason": "Merged into memories",
      "points": 1234
    }
  }
}

Protected collections ("protected": true) must never be modified or deleted without explicit user direction. Known protected collections: girls_mom, memories, pandaneuro, private_court_docs.

Update the registry after every collection change (merge, delete, create). Bump the version number.

Unified Memory Schema

The memories collection uses a 10-type classification schema (replacing the old Hindsight-aligned 4-type schema). See references/memory-schema-gc-proposal.md for the full design with type taxonomy, garbage-collection strategy, and quality scoring.

Current 10 types: fact, infrastructure, entity, preference, procedure, decision, event, experience, observation, reference — each with a mem_class (semantic/episodic/procedural) for retrieval routing.

Key payload fields:

{
  "text": "Normalized content — the only field that gets embedded",
  "type": "fact | infrastructure | entity | preference | procedure | decision | event | experience | observation | reference",
  "mem_class": "semantic | episodic | procedural",
  "entities": ["Rob", "10.0.0.6", "Qwen3 DFlash"],
  "tags": ["infrastructure", "dgx", "vllm"],
  "importance": 0.85,
  "quality_score": 0.72,
  "confidence": 0.9,
  "timestamp": "2026-03-27T12:50:37Z",
  "organized_at": "2026-07-15T09:00:00Z",
  "source": "memories_tr",
  "migrated_at": "2026-06-28T00:00:00Z",
  "merged_from": ["id1", "id2"],
  "contradiction_with": ["id3"],
  "supersedes": ["id4"],
  "superseded_by": "id5",
  "_source": { /* original payload, untouched */ }
}

Key design decisions:

• type + mem_class replace the old Hindsight-aligned 4-type schema (world/experience/observation/reference). Hindsight is no longer used.
• _source is nested (not flattened) to preserve original payloads from 13 source schemas without polluting the filterable namespace.
• importance is LLM-assigned (0.0-1.0) — a heartbeat check is 0.1, a court document is 0.9.
• contradiction_with tracks conflicting facts (e.g., "model is qwen" vs "model is deepseek").
• supersedes/superseded_by chain facts through versions — semantic/procedural types are superseded, never deleted on staleness alone.
• quality_score is weighted: 30% richness + 25% actionability + 20% uniqueness + 15% freshness + 10% confidence.
• Long-term types (8): fact, infrastructure, entity, preference, procedure, decision, observation, reference — NEVER deleted, only versioned via supersedes/superseded_by. The only exception is hard-duplicate merging.
• Short-term types (2): event, experience — consolidated into observations, then deleted after 90 days.
• Garbage collection deletes: noise/trivial (regex patterns, short text, no entities), hard duplicates (cosine >=0.97), and consolidated episodics (>90d, already rolled into an observation). Capped at 5 per run (5% of 100). Long-term types are NEVER deleted — only versioned via supersedes/superseded_by.
• Error log: /home/n8n/workspace/general/cron_qdrant_errors.md — append-only, timestamped entries. Only touched when errors occur; if no errors, the file is not created or modified. If it already exists from a prior run, append to it.

When the user says "save to memory", save to the Qdrant memories collection (10.0.0.22:6333, 1024-dim, snowflake-arctic-embed2). This is the consolidated primary memory store.

At save time, the agent MUST classify the content and include these fields in the payload:

• type — one of 10 types (fact, infrastructure, entity, preference, procedure, decision, event, experience, observation, reference)
• mem_class — semantic, episodic, or procedural
• entities — people, servers, IPs, tools, projects, models mentioned
• tags — 2-3 category tags for filtering
• importance — 0.0-1.0 (0.9+ critical, 0.5-0.8 useful, 0.1-0.4 noise)
• confidence — 1.0 user-stated, 0.7 agent-inferred, 0.5 single-episode
• created_at / updated_at — ISO 8601 timestamps
• source — "agent", "user", "skill", or "tool"
• ttl_hint — "slow" for semantic/procedural, "fast" for episodic
• quality_score — set to None (cron organizer computes this)

The agent reasons about the content inline — no extra LLM call needed. The cron organizer later backfills quality_score and handles cross-point work (dedup, supersession, contradiction, garbage collection).

Use the pattern in references/individual-statement-upsert.md for the curl + Ollama embed + upsert workflow. Use mcp_better_qdrant_add_documents only for file-based bulk saves.

Pitfalls

• Check before creating. Before creating a new collection, always list existing collections first. The user may already have one with that name and purpose.

• requests library fails from sandbox. The Hermes execute_code sandbox can't reach 10.0.0.22 via Python's requests. Use subprocess.run(["curl", ...]) instead — curl works from both sandbox and terminal.

• Batch search needs vectors. Qdrant's search API requires a vector — you can't search by ID alone. Scroll points with with_vector: true first.

• indexed_vectors_count is NOT an activity indicator. Qdrant's indexing_threshold is 10,000 — any collection under that shows 0 indexed regardless of usage. Don't use it to determine if a collection is active.

• One-at-a-time with validation. Never batch-delete collections. Migrate one, validate the count, then delete. This prevents silent data loss.

• Never touch pandaneuro. It's permanently off-limits.

• KBs should NOT be merged into memories. Project-scoped knowledge bases (comfyui_kb, hermes_kb, etc.) need isolation for scoped recall. Only merge conversation memories and raw data.

• Cron job model pinning. When creating a cron job, always pass both model and provider explicitly. Passing only provider causes the job to inherit the session model, which may not be the intended model for the cron workload.

• Holographic sync supersedes manual saves. With the Holographic→Qdrant sync cron active, the agent no longer needs to manually save facts to Qdrant. The "save to memory" workflow remains available for file-based bulk saves (PDFs, documents) via mcp_better_qdrant_add_documents.

• UPSERT vs SET_PAYLOAD — critical distinction. PUT /collections/{name}/points (upsert) requires a vector and replaces the entire payload. Using it for payload-only updates (e.g., setting type on existing points) will fail with "missing field vector" or silently wipe all other payload fields. For payload-only updates, use POST /collections/{name}/points/payload?wait=true (set_payload) which merges the new payload fields without touching vectors or existing fields. The consolidation workflow in this skill uses upsert because it's writing new points with vectors — that's correct. But any operation that updates payload on existing points MUST use set_payload.

• BigInt point IDs break in jq/JS pipelines. Point IDs derived from SHA-256 hashing are ~19-digit uint64s (>2^53, exceeding Number.MAX_SAFE_INTEGER). Any JS-based JSON tool (jq ≤1.6, node, most shell pipelines) silently rounds them to a different integer. When building JSON for Qdrant API calls, use Python's json.dumps which preserves arbitrary-precision integers exactly. Never pipe a point ID through jq. The id field in the JSON body must be a bare integer (not quoted) — Python's json.dumps emits it correctly.

• is_empty filter matches absent keys. {"is_empty": {"key": "type"}} matches points where type does not exist, is null, or is []. This is the correct filter for finding untyped points. Do NOT use must_not: [match type=...] — Qdrant issue #9255 shows match conditions incorrectly return points whose key is absent, so the inverse would wrongly exclude untyped points. Verify is_empty behavior on your Qdrant version with a count probe before relying on it in production.

• LLM agent non-compliance: move mechanics to code, not more instructions. When an LLM-driven cron job ignores its own prompt (e.g., using upsert instead of set_payload, unfiltered scroll instead of is_empty), adding more instructions is the wrong fix — it makes the prompt longer and the compliance problem worse. The reliable fix is to move all mechanical operations (scroll, write-back, checkpoint, lease, logging) into a deterministic Python wrapper script that shells out to curl for network calls. Reduce the LLM to classification-only: text in, type out. No HTTP access, no endpoint to pick, no checkpoint to bookkeep. See references/classify-batch-wrapper.md for the production implementation with lease, validation, poison-pill guard, and stall tracking.

• Wrapper script pitfalls (from production deployment): When building a classify-batch wrapper, these 11 bugs were found by adversarial peer review and must be avoided: (1) Double release_lease() in poison-pill path — use a single atomic write. (2) lease_expires set to now_iso() instead of now + timedelta(minutes=N). (3) No defense against LLM string IDs — coerce int(item["id"]) and reject float/bool. (4) Empty LLM output advancing the checkpoint — release lease WITHOUT advancing. (5) if offset truthiness treats point ID 0 as null — use if offset is not None. (6) valid_ids type mismatch — force int() on both sides. (7) No stall reset on success — stall.pop(offset_key) + prune + cap. (8) No list guard on classifications — check isinstance(raw_classifications, list). (9) Unknown types defaulting to experience — REJECT unknown types; defaulting permanently misclassifies. (10) Hallucinated IDs not validated — check pid in valid_ids. (11) __null__ dropped by 50-entry cap — pop before sort, re-add after.

References

• Dedup script: references/dedup-script.py — full Python dedup implementation using curl subprocess, batch search, and checkpointing
• Registry template: references/registry-template.json — starter collection registry file
• Individual statement upsert: references/individual-statement-upsert.md — pattern for upserting discrete facts as independent searchable points (one per statement) with deterministic IDs, source/category tags, and typo-correction workflow. Use when MCP add_documents chunking is too coarse.