hermes-skills/qdrant-collection-management/SKILL.md

---
name: qdrant-collection-management
description: "Manage Qdrant collections — consolidation, migration, deduplication, registry, and cleanup. Covers the scroll→transform→upsert→validate→delete pattern, batch processing with curl subprocess, and the collection registry file convention."
version: 1.1.0
author: Hermes Agent
license: MIT
metadata:
  hermes:
    tags: [qdrant, collections, migration, dedup, consolidation, registry]
    related_skills: [local-vector-memory]
---

# Qdrant Collection Management

Manage Qdrant collections at scale — consolidate scattered collections into a unified store, deduplicate near-duplicate points, maintain a registry file, and clean up stale data.

## When to Use

- Consolidating multiple collections into one (e.g., merging kimi, vera into `memories`)
- Deduplicating points by cosine similarity
- Cleaning up stale/dormant collections
- Maintaining a collection registry with tags, ownership, and protection status
- Migrating points between collections (including re-embedding for dimension mismatches)

## MCP vs Curl

The MCP Qdrant tools (`mcp_better_qdrant_*`) only expose: list, search, add, delete. They do NOT cover: collection info (dimensions, config), point scrolling, batch search, or point count. For these operations, use curl directly against the Qdrant REST API.

**Always prefer MCP for what it covers** (list, search, add, delete). Fall back to curl for everything else.

## Consolidation Workflow

The proven pattern for merging collections:

### 1. Read source (scroll all points with vectors)

```bash
curl -s -X POST http://10.0.0.22:6333/collections/{source}/points/scroll \
  -H "Content-Type: application/json" \
  -d '{"limit": 100, "with_payload": true, "with_vector": true}'
```

Paginate using `next_page_offset` until it returns null. For large collections (10K+), use a Python script with curl subprocess — the `requests` library can hit port exhaustion on the sandbox.

### 2. Transform payloads

Add `source` and `migrated_at` fields to every point. Normalize text fields: if the source uses `content` or `data` instead of `text`, copy to `text` for unified search.

```python
payload["source"] = "source_collection_name"
payload["migrated_at"] = "2026-06-28T00:00:00Z"
if "content" in payload and "text" not in payload:
    payload["text"] = payload["content"]
```

### 3. Upsert to target

```bash
curl -s -X PUT http://10.0.0.22:6333/collections/{target}/points \
  -H "Content-Type: application/json" \
  -d '{"points": [{"id": "...", "vector": [...], "payload": {...}}]}'
```

Batch in groups of 100. For large payloads, write to temp file and use `-d @file.json` to avoid argument list overflow.

### 4. Validate

```bash
curl -s -X POST http://10.0.0.22:6333/collections/{target}/points/count \
  -H "Content-Type: application/json" \
  -d '{"filter": {"must": [{"key": "source", "match": {"value": "source_name"}}]}}'
```

Count must match the source collection's point count.

### 5. Delete source

```bash
curl -s -X DELETE http://10.0.0.22:6333/collections/{source}
```

Or use `mcp_better_qdrant_delete_collection` for MCP-tracked deletions.

### Dimension Mismatch

If source has different vector dimensions than target (e.g., 768-dim → 1024-dim), re-embed using Ollama before upserting:

```python
r = subprocess.run(
    ["curl", "-s", "http://localhost:11434/api/embed",
     "-d", json.dumps({"model": "snowflake-arctic-embed2:latest", "input": text})],
    capture_output=True, text=True
)
vec = json.loads(r.stdout)["embeddings"][0]
```

## Deduplication

Use cosine similarity search to find near-duplicates. The batch search API is efficient for large collections:

```python
# Build batch searches
searches = []
for pt in chunk:
    searches.append({
        "vector": pt["vector"],
        "limit": 5,
        "score_threshold": 0.97,
        "filter": {"must_not": [{"has_id": [pt["id"]]}]}
    })

# Execute batch
curl -X POST /collections/{name}/points/search/batch -d '{"searches": [...]}'
```

**Threshold guidance:**
- 0.99: exact duplicates (same text, different source)
- 0.97: near-duplicates (same fact, slightly different wording)
- 0.95: semantic duplicates (same meaning, different phrasing)

At 0.97 on a 30K-point consolidated collection, expect ~20% duplicates from overlapping sources.

**Important:** When using Python for dedup, the `requests` library may fail with `[Errno 99] Cannot assign requested address` from the sandbox. Use `subprocess.run(["curl", ...])` instead — curl works reliably. See `references/dedup-script.py` for the full implementation.

## Collection Registry

Maintain a registry file at `~/workspace/general/qdrant-collections.json` with:

```json
{
  "version": "1.0.0",
  "collections": {
    "name": {
      "tags": ["project", "kb"],
      "description": "...",
      "owner": "project_name",
      "status": "active",
      "protected": false
    }
  },
  "tag_index": {
    "project": ["col1", "col2"],
    "kb": ["col1"]
  },
  "deleted_collections": {
    "old_name": {
      "date": "2026-06-28",
      "reason": "Merged into memories",
      "points": 1234
    }
  }
}
```

**Protected collections** (`"protected": true`) must never be modified or deleted without explicit user direction. Known protected collections: `girls_mom`, `memories`, `pandaneuro`, `private_court_docs`.

Update the registry after every collection change (merge, delete, create). Bump the version number.

## Unified Memory Schema

The `memories` collection uses a 10-type classification schema (replacing the old Hindsight-aligned 4-type schema). See `references/memory-schema-gc-proposal.md` for the full design with type taxonomy, garbage-collection strategy, and quality scoring.

**Current 10 types:** `fact`, `infrastructure`, `entity`, `preference`, `procedure`, `decision`, `event`, `experience`, `observation`, `reference` — each with a `mem_class` (semantic/episodic/procedural) for retrieval routing.

**Key payload fields:**
```json
{
  "text": "Normalized content — the only field that gets embedded",
  "type": "fact | infrastructure | entity | preference | procedure | decision | event | experience | observation | reference",
  "mem_class": "semantic | episodic | procedural",
  "entities": ["Rob", "10.0.0.6", "Qwen3 DFlash"],
  "tags": ["infrastructure", "dgx", "vllm"],
  "importance": 0.85,
  "quality_score": 0.72,
  "confidence": 0.9,
  "timestamp": "2026-03-27T12:50:37Z",
  "organized_at": "2026-07-15T09:00:00Z",
  "source": "memories_tr",
  "migrated_at": "2026-06-28T00:00:00Z",
  "merged_from": ["id1", "id2"],
  "contradiction_with": ["id3"],
  "supersedes": ["id4"],
  "superseded_by": "id5",
  "_source": { /* original payload, untouched */ }
}
```

**Key design decisions:**
- `type` + `mem_class` replace the old Hindsight-aligned 4-type schema (world/experience/observation/reference). Hindsight is no longer used.
- `_source` is nested (not flattened) to preserve original payloads from 13 source schemas without polluting the filterable namespace.
- `importance` is LLM-assigned (0.0-1.0) — a heartbeat check is 0.1, a court document is 0.9.
- `contradiction_with` tracks conflicting facts (e.g., "model is qwen" vs "model is deepseek").
- `supersedes`/`superseded_by` chain facts through versions — semantic/procedural types are superseded, never deleted on staleness alone.
- `quality_score` is weighted: 30% richness + 25% actionability + 20% uniqueness + 15% freshness + 10% confidence.
- **Long-term types (8):** fact, infrastructure, entity, preference, procedure, decision, observation, reference — NEVER deleted, only versioned via supersedes/superseded_by. The only exception is hard-duplicate merging.
- **Short-term types (2):** event, experience — consolidated into observations, then deleted after 90 days.
- Garbage collection deletes: noise/trivial (regex patterns, short text, no entities), hard duplicates (cosine >=0.97), and consolidated episodics (>90d, already rolled into an observation). Capped at 5 per run (5% of 100). Long-term types are NEVER deleted — only versioned via supersedes/superseded_by.
- **Error log:** `/home/n8n/workspace/general/cron_qdrant_errors.md` — append-only, timestamped entries. Only touched when errors occur; if no errors, the file is not created or modified. If it already exists from a prior run, append to it.

When the user says "save to memory", save to the Qdrant `memories` collection (10.0.0.22:6333, 1024-dim, snowflake-arctic-embed2). This is the consolidated primary memory store.

**At save time, the agent MUST classify the content and include these fields in the payload:**
- `type` — one of 10 types (fact, infrastructure, entity, preference, procedure, decision, event, experience, observation, reference)
- `mem_class` — semantic, episodic, or procedural
- `entities` — people, servers, IPs, tools, projects, models mentioned
- `tags` — 2-3 category tags for filtering
- `importance` — 0.0-1.0 (0.9+ critical, 0.5-0.8 useful, 0.1-0.4 noise)
- `confidence` — 1.0 user-stated, 0.7 agent-inferred, 0.5 single-episode
- `created_at` / `updated_at` — ISO 8601 timestamps
- `source` — "agent", "user", "skill", or "tool"
- `ttl_hint` — "slow" for semantic/procedural, "fast" for episodic
- `quality_score` — set to `None` (cron organizer computes this)

The agent reasons about the content inline — no extra LLM call needed. The cron organizer later backfills `quality_score` and handles cross-point work (dedup, supersession, contradiction, garbage collection).

Use the pattern in `references/individual-statement-upsert.md` for the curl + Ollama embed + upsert workflow. Use `mcp_better_qdrant_add_documents` only for file-based bulk saves.

## Pitfalls

- **Check before creating.** Before creating a new collection, always list existing collections first. The user may already have one with that name and purpose.
- **`requests` library fails from sandbox.** The Hermes execute_code sandbox can't reach 10.0.0.22 via Python's requests. Use `subprocess.run(["curl", ...])` instead — curl works from both sandbox and terminal.
- **Batch search needs vectors.** Qdrant's search API requires a vector — you can't search by ID alone. Scroll points with `with_vector: true` first.
- **`indexed_vectors_count` is NOT an activity indicator.** Qdrant's `indexing_threshold` is 10,000 — any collection under that shows 0 indexed regardless of usage. Don't use it to determine if a collection is active.
- **One-at-a-time with validation.** Never batch-delete collections. Migrate one, validate the count, then delete. This prevents silent data loss.
- **Never touch `pandaneuro`.** It's permanently off-limits.
- **KBs should NOT be merged into `memories`.** Project-scoped knowledge bases (comfyui_kb, hermes_kb, etc.) need isolation for scoped recall. Only merge conversation memories and raw data.
- **Cron job model pinning.** When creating a cron job, always pass both `model` and `provider` explicitly. Passing only `provider` causes the job to inherit the session model, which may not be the intended model for the cron workload.
- **Holographic sync supersedes manual saves.** With the Holographic→Qdrant sync cron active, the agent no longer needs to manually save facts to Qdrant. The "save to memory" workflow remains available for file-based bulk saves (PDFs, documents) via `mcp_better_qdrant_add_documents`.
- **UPSERT vs SET_PAYLOAD — critical distinction.** `PUT /collections/{name}/points` (upsert) **requires a vector** and **replaces the entire payload**. Using it for payload-only updates (e.g., setting `type` on existing points) will fail with "missing field `vector`" or silently wipe all other payload fields. For payload-only updates, use `POST /collections/{name}/points/payload?wait=true` (set_payload) which **merges** the new payload fields without touching vectors or existing fields. The consolidation workflow in this skill uses upsert because it's writing new points with vectors — that's correct. But any operation that updates payload on existing points MUST use set_payload.
- **BigInt point IDs break in jq/JS pipelines.** Point IDs derived from SHA-256 hashing are ~19-digit uint64s (>2^53, exceeding `Number.MAX_SAFE_INTEGER`). Any JS-based JSON tool (jq ≤1.6, node, most shell pipelines) silently rounds them to a different integer. When building JSON for Qdrant API calls, use Python's `json.dumps` which preserves arbitrary-precision integers exactly. Never pipe a point ID through jq. The `id` field in the JSON body must be a bare integer (not quoted) — Python's `json.dumps` emits it correctly.
- **`is_empty` filter matches absent keys.** `{"is_empty": {"key": "type"}}` matches points where `type` does not exist, is null, or is `[]`. This is the correct filter for finding untyped points. Do NOT use `must_not: [match type=...]` — Qdrant issue #9255 shows `match` conditions incorrectly return points whose key is absent, so the inverse would wrongly exclude untyped points. Verify `is_empty` behavior on your Qdrant version with a count probe before relying on it in production.
- **LLM agent non-compliance: move mechanics to code, not more instructions.** When an LLM-driven cron job ignores its own prompt (e.g., using upsert instead of set_payload, unfiltered scroll instead of `is_empty`), adding more instructions is the wrong fix — it makes the prompt longer and the compliance problem worse. The reliable fix is to move all mechanical operations (scroll, write-back, checkpoint, lease, logging) into a deterministic Python wrapper script that shells out to curl for network calls. Reduce the LLM to classification-only: text in, type out. No HTTP access, no endpoint to pick, no checkpoint to bookkeep. See `references/classify-batch-wrapper.md` for the production implementation with lease, validation, poison-pill guard, and stall tracking.

- **Wrapper script pitfalls (from production deployment):** When building a classify-batch wrapper, these 11 bugs were found by adversarial peer review and must be avoided: (1) Double `release_lease()` in poison-pill path — use a single atomic write. (2) `lease_expires` set to `now_iso()` instead of `now + timedelta(minutes=N)`. (3) No defense against LLM string IDs — coerce `int(item["id"])` and reject `float`/`bool`. (4) Empty LLM output advancing the checkpoint — release lease WITHOUT advancing. (5) `if offset` truthiness treats point ID 0 as null — use `if offset is not None`. (6) `valid_ids` type mismatch — force `int()` on both sides. (7) No stall reset on success — `stall.pop(offset_key)` + prune + cap. (8) No list guard on classifications — check `isinstance(raw_classifications, list)`. (9) Unknown types defaulting to `experience` — REJECT unknown types; defaulting permanently misclassifies. (10) Hallucinated IDs not validated — check `pid in valid_ids`. (11) `__null__` dropped by 50-entry cap — pop before sort, re-add after.

## References

- **Dedup script**: `references/dedup-script.py` — full Python dedup implementation using curl subprocess, batch search, and checkpointing
- **Registry template**: `references/registry-template.json` — starter collection registry file
- **Individual statement upsert**: `references/individual-statement-upsert.md` — pattern for upserting discrete facts as independent searchable points (one per statement) with deterministic IDs, source/category tags, and typo-correction workflow. Use when MCP `add_documents` chunking is too coarse.