stock-search v1.0.0 (new) + stock-search-research v1.0.0 (new) + ltx-video-pipeline, better-search, better-search-research, deep-web-research, ai-brain-kb, ai-vid-stock updates (2026-07-22)
This commit is contained in:
Hermes Agent
@@ -0,0 +1,215 @@
|
||||
---
|
||||
name: stock-search-research
|
||||
description: Stock material search methodology — 3-move flow (search → evaluate → download+classify+upload to TrueNAS), 3-loop cap, /tmp ledger. Opt-in skill for the research profile.
|
||||
version: 1.0.0
|
||||
author: Hermes Agent
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [research, search, stock, download, truenas, methodology]
|
||||
related_skills: [better-search-research, deep-web-research]
|
||||
---
|
||||
|
||||
# stock-search-research — Stock Material Search + Download Methodology
|
||||
|
||||
Loaded explicitly via `-s stock-search-research`. Not loaded during normal
|
||||
interactive use of the research profile. This skill enforces a 3-move research
|
||||
flow with a hard 3-loop cap, ending in file download and TrueNAS upload.
|
||||
|
||||
## §1 Overview
|
||||
|
||||
This skill performs a 3-move research flow:
|
||||
|
||||
1. **Move 1: Initial Search** — 2-3 SearXNG searches for downloadable stock
|
||||
materials matching the query, read top results, write structured summary to
|
||||
`/tmp/stock-<sid>.md`.
|
||||
2. **Move 2: AI Evaluation + Refine** — Read the ledger from disk, self-evaluate
|
||||
for quality/relevance/license, optionally run 1-2 refinement searches.
|
||||
3. **Move 3: Download + Classify + Upload** — Read full ledger, download top
|
||||
files to `/tmp/`, classify by type, upload to TrueNAS via smbclient, update
|
||||
inventory.
|
||||
|
||||
**Hard loop cap: 3** (1 initial search + up to 2 refinements). The cap is
|
||||
enforced by the ledger — at most 1 `## Search` block and 2 `## Refinement`
|
||||
blocks. No saturation-based continuation. No `--resume` — every dispatch is a
|
||||
fresh session with a new ledger and a new 3-loop budget.
|
||||
|
||||
**Total budget: 50 turns.** The 3-loop cap is the real limit; 50 turns is a
|
||||
safety net. If the agent hits 50, deliver partial results with a note.
|
||||
|
||||
**Ledger:** `/tmp/stock-<sid>.md` — a flat structured file. The ledger forces a
|
||||
re-read from disk at each move so details that scrolled out of context are not
|
||||
lost.
|
||||
|
||||
**TrueNAS target:** `//10.0.0.117/proxmoxBackup/ai_vid_stock_material/`
|
||||
Access via `smbclient -N //10.0.0.117/proxmoxBackup`.
|
||||
|
||||
## §2 Parsing the Target Category
|
||||
|
||||
The question string contains `--target <category>` at the end. Parse it:
|
||||
|
||||
- `start_frames` → `ai_vid_stock_material/start_frames/`
|
||||
- `character_refs` → `ai_vid_stock_material/character_refs/`
|
||||
- `audio` → `ai_vid_stock_material/audio/`
|
||||
- `misc_images` → `ai_vid_stock_material/misc_images/`
|
||||
- `all` or missing → classify by file extension (see §4)
|
||||
|
||||
Strip `--target <category>` from the question before using it as the search query.
|
||||
|
||||
## §3 Move 1: Initial Search
|
||||
|
||||
**Turns 1-5.** Gather downloadable stock materials.
|
||||
|
||||
1. Run 2-3 SearXNG searches with different framings:
|
||||
- One broad: `<query> free download stock image`
|
||||
- One specific: `<query> site:pexels.com OR site:unsplash.com OR site:pixabay.com`
|
||||
- One targeted: `<query> site:huggingface.co OR site:civitai.com` (for AI-specific stock)
|
||||
- For audio: add `site:freesound.org OR site:epidemicsound.com`
|
||||
- Use `filetype:png OR filetype:jpg` when searching for images
|
||||
2. Read top 1-3 results per search with `mcp_searxng_web_url_read`.
|
||||
3. For each result, identify:
|
||||
- Direct download URL (not a page — the actual file URL)
|
||||
- File type (image, audio, video)
|
||||
- License (free/commercial/attribution required)
|
||||
- Resolution/quality notes
|
||||
4. Write a structured summary to `/tmp/stock-<sid>.md`:
|
||||
|
||||
```
|
||||
## Question
|
||||
<verbatim question with --target stripped>
|
||||
|
||||
## Target Category
|
||||
<category>
|
||||
|
||||
## Search 1: <query>
|
||||
- Source: <url>
|
||||
- Download URL: <direct file URL>
|
||||
- Type: image/png | image/jpg | audio/mp3 | audio/wav
|
||||
- License: free | commercial | attribution | unknown
|
||||
- Notes: <resolution, quality, relevance>
|
||||
|
||||
## Search 2: <query>
|
||||
...
|
||||
```
|
||||
|
||||
**SearXNG error handling:** Same as better-search-research §2 — VPN reconnect
|
||||
on first failure, different framing on second, proceed to Move 3 on third.
|
||||
|
||||
## §4 Move 2: AI Evaluation + Refine
|
||||
|
||||
**Turns 6-15.** Evaluate quality and fill gaps.
|
||||
|
||||
1. Read `/tmp/stock-<sid>.md` from disk.
|
||||
2. Self-evaluate using these criteria (write the eval to the ledger):
|
||||
|
||||
- **Relevance:** Does each result match the query intent?
|
||||
- **Quality:** Resolution sufficient? (Images: min 512px. Audio: min 44.1kHz.)
|
||||
- **License:** Safe to use? Prefer free/commercial. Flag attribution-required.
|
||||
- **Downloadability:** Is there a direct download URL? Skip pages that only
|
||||
show previews.
|
||||
- **Coverage:** Enough results? (Aim for 3-10 good files.)
|
||||
|
||||
3. IF gaps → write `## Refinement <N>: <new query>` to ledger, run 1-2 more
|
||||
searches, append findings.
|
||||
4. Hard cap: 2 refinements total. Track count in the ledger.
|
||||
5. IF no gaps (or cap hit) → proceed to Move 3.
|
||||
|
||||
**Ledger format for refinements:**
|
||||
|
||||
```
|
||||
## Refinement 1: <new query>
|
||||
- Source: <url>
|
||||
- Download URL: <direct file URL>
|
||||
- Type: <file type>
|
||||
- License: <license>
|
||||
- Notes: <notes>
|
||||
```
|
||||
|
||||
## §5 Move 3: Download + Classify + Upload
|
||||
|
||||
**Turns 16-50.** Download files and upload to TrueNAS.
|
||||
|
||||
1. Read full ledger from `/tmp/stock-<sid>.md`.
|
||||
2. Create a temp directory: `mkdir -p /tmp/stock-dl-<sid>/`
|
||||
3. For each result with a direct download URL:
|
||||
- Download: `wget -q -P /tmp/stock-dl-<sid>/ "<download_url>"`
|
||||
- Skip if download fails (3 retries max, then blacklist)
|
||||
- Skip if file is < 1KB (likely an error page)
|
||||
4. Classify each downloaded file:
|
||||
- `.png`, `.jpg`, `.jpeg`, `.webp` → `start_frames/` (if target is `all` or `start_frames`)
|
||||
- `.mp3`, `.wav`, `.ogg`, `.flac` → `audio/`
|
||||
- `.mp4`, `.webm`, `.mov` → `misc_images/` (no video category yet)
|
||||
- Everything else → `misc_images/`
|
||||
- If a specific `--target` was given, ALL files go to that category
|
||||
5. Upload each file to TrueNAS:
|
||||
```bash
|
||||
smbclient -N //10.0.0.117/proxmoxBackup -c "cd ai_vid_stock_material\\<category>; put /tmp/stock-dl-<sid>/<filename> <filename>"
|
||||
```
|
||||
- Use backslashes in SMB paths: `ai_vid_stock_material\\start_frames`
|
||||
- If a file with the same name exists, append `-<N>` before the extension
|
||||
6. Update the inventory README on TrueNAS:
|
||||
```bash
|
||||
# Download current README
|
||||
smbclient -N //10.0.0.117/proxmoxBackup -c "cd ai_vid_stock_material; get README.md /tmp/stock-dl-<sid>/README.md"
|
||||
# Append new entries
|
||||
echo "## Stock Search: <date> — <query>" >> /tmp/stock-dl-<sid>/README.md
|
||||
for each uploaded file:
|
||||
echo "- <category>/<filename> — <source_url> — <license>" >> /tmp/stock-dl-<sid>/README.md
|
||||
# Upload updated README
|
||||
smbclient -N //10.0.0.117/proxmoxBackup -c "cd ai_vid_stock_material; put /tmp/stock-dl-<sid>/README.md README.md"
|
||||
```
|
||||
7. Clean up: `rm -rf /tmp/stock-dl-<sid>/`
|
||||
8. Report to stdout:
|
||||
```
|
||||
Downloaded N files to TrueNAS:
|
||||
- ai_vid_stock_material/<category>/<filename> (source: <url>, license: <license>)
|
||||
- ...
|
||||
```
|
||||
|
||||
**Download safety:**
|
||||
- Max 20 files per dispatch (prevents runaway downloads)
|
||||
- Max 500MB total per dispatch
|
||||
- Skip any URL that redirects to a different domain (potential malware)
|
||||
- Skip `.exe`, `.dmg`, `.pkg`, `.msi` files (executables)
|
||||
- If smbclient fails, report the error — do not retry more than 3 times
|
||||
|
||||
## §6 Safety Boundaries
|
||||
|
||||
These persist across all turns — they are in the skill, not in fading context:
|
||||
|
||||
- **Confined to /tmp.** All downloads go to `/tmp/stock-dl-<sid>/`. Never write
|
||||
outside /tmp except for TrueNAS uploads.
|
||||
- **No self-provisioning.** Never install software. No pip, npm, apt, docker, or
|
||||
any package manager. Use only what's already configured (wget, smbclient, ffprobe).
|
||||
- **No repeat searches.** If you catch yourself searching the same thing twice,
|
||||
stop. That sub-question is saturated.
|
||||
- **Blacklist after 3 failures.** If a URL returns an error 3 times, blacklist it
|
||||
and move on. Do not retry indefinitely.
|
||||
- **Local and free only.** No internet-based paid services, no SaaS APIs with
|
||||
billing, no metered endpoints.
|
||||
- **Max 20 files, 500MB total per dispatch.** Hard cap to prevent runaway downloads.
|
||||
- **No executables.** Skip `.exe`, `.dmg`, `.pkg`, `.msi`.
|
||||
|
||||
## §7 Cap-Hit Behavior
|
||||
|
||||
When the 3-loop cap is hit (1 initial search + 2 refinements used) without
|
||||
satisfaction:
|
||||
|
||||
1. Proceed directly to Move 3 — download what you have.
|
||||
2. In the stdout report, add a note:
|
||||
> **Note:** Loop cap reached (3 loops / 1 initial + 2 refinements). Some
|
||||
> results may not be fully explored. For exhaustive coverage, re-dispatch with
|
||||
> a refined question or use `deep-research`.
|
||||
|
||||
When the 50-turn ceiling is hit:
|
||||
|
||||
1. Download whatever files have been identified so far.
|
||||
2. Report with a header note: "incomplete — turn ceiling hit at Move N."
|
||||
|
||||
## §8 See Also
|
||||
|
||||
- `better-search-research` — medium-depth research without download (report-only)
|
||||
- `deep-web-research` — exhaustive multi-source research with disconfirmation
|
||||
- `stock-search` (dispatcher) — the operator-facing skill that triggers this
|
||||
methodology. Installed on all profiles; delegates to the research profile via
|
||||
`research -s stock-search-research`.
|
||||
- `ai-vid-stock` — manual stock management on TrueNAS (list, add, remove)
|
||||
Reference in New Issue
Block a user