107 lines
3.8 KiB
Markdown
107 lines
3.8 KiB
Markdown
---
|
|||
|
|
name: youtube-knowledge-ingestion
|
||
|
|
description: "Use when user drops a YouTube link. Transcript→brain→NAS."
|
||
|
|
version: 1.0.0
|
||
|
|
platforms: [linux]
|
||
|
|
---
|
||
|
|
|
||
|
|
# YouTube Knowledge Ingestion Pipeline
|
||
|
|
|
||
|
|
## When to use
|
||
|
|
|
||
|
|
User drops a YouTube link for AI/ML video content (ComfyUI workflows, LTX, MiniMax,
|
||
|
|
Krea, model tutorials, prompting guides). Run the full three-step pipeline — never skip
|
||
|
|
any step.
|
||
|
|
|
||
|
|
## Pipeline (always all three steps)
|
||
|
|
|
||
|
|
### 1. Transcript
|
||
|
|
|
||
|
|
Load the `youtube-content` skill and fetch:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
uv run python3 <SKILL_DIR>/scripts/fetch_transcript.py "URL" --text-only --timestamps
|
||
|
|
```
|
||
|
|
|
||
|
|
If `youtube-transcript-api` is missing, install with `pip3 install --user youtube-transcript-api`.
|
||
|
|
|
||
|
|
### 2. Brain Injection (MANDATORY)
|
||
|
|
|
||
|
|
Load the `ai-brain-kb` skill. Always dedup-first, then ingest:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Dedup check
|
||
|
|
python3 /home/n8n/bin/ai_brain_kb.py search --query "<key topic>" --limit 5
|
||
|
|
|
||
|
|
# Ingest — use --type workflow for tutorials
|
||
|
|
python3 /home/n8n/bin/ai_brain_kb.py add \
|
||
|
|
--type workflow \
|
||
|
|
--title "Descriptive Title — Key Topics" \
|
||
|
|
--stage <t2v|i2v|upscale|...> \
|
||
|
|
--tool <ltx-video|minimax-h3|krea2|...> \
|
||
|
|
--host 10.0.0.202 \
|
||
|
|
--trust community \
|
||
|
|
--importance 0.7-0.85 \
|
||
|
|
--tags "comma,separated,keywords,creator-name" \
|
||
|
|
--url "https://www.youtube.com/watch?v=VIDEO_ID" \
|
||
|
|
--content "..." \
|
||
|
|
--json
|
||
|
|
|
||
|
|
# Verify BM25 leg
|
||
|
|
python3 /home/n8n/bin/ai_brain_kb.py search --query "<distinctive phrase>" \
|
||
|
|
--mode bm25 --doc-id <doc_id>
|
||
|
|
```
|
||
|
|
|
||
|
|
**Content format**: Dense, factual technical summary. Use CAPITALIZED section headers.
|
||
|
|
Include specific settings, model names, thresholds, commands, and failure modes.
|
||
|
|
This is a technical reference, not a blog post.
|
||
|
|
|
||
|
|
**MANDATORY: workflow links in the brain content.** Every brain entry must include a
|
||
|
|
`## WORKFLOW LINKS` section listing the DOWNLOADED workflow files on NAS (TrueNAS
|
||
|
|
10.0.0.117, proxmoxBackup/ai_vid_stock_material/workflows/, exact filenames + the
|
||
|
|
smbclient get command), the source repo URL, and the video URL. The NAS copies are
|
||
|
|
the primary links — the user wants links to the downloaded workflows, not upstream
|
||
|
|
URLs. If the description links elsewhere (Patreon etc.) and is unreachable, say so
|
||
|
|
in the content. Append as a separate chunk via `--doc-id` if the main content was
|
||
|
|
already ingested.
|
||
|
|
|
||
|
|
**Tag strategy**: model name, tool name, creator name, key techniques. Tags are
|
||
|
|
exact-match keyword indexes.
|
||
|
|
|
||
|
|
### 3. NAS Workflow Download
|
||
|
|
|
||
|
|
If the video description contains workflow links (GitHub repos, direct JSON files),
|
||
|
|
extract the description with `yt-dlp --print description "URL"`, clone the repos,
|
||
|
|
and copy workflow JSONs to the NAS:
|
||
|
|
|
||
|
|
```bash
|
||
|
|
# Clone
|
||
|
|
cd /tmp && rm -rf <repo> 2>/dev/null
|
||
|
|
git clone --depth 1 <repo-url>
|
||
|
|
|
||
|
|
# Find workflows
|
||
|
|
find /tmp/<repo> -name "*.json"
|
||
|
|
|
||
|
|
# Copy to NAS with creator prefix for disambiguation
|
||
|
|
smbclient -N //10.0.0.117/proxmoxBackup \
|
||
|
|
-c 'cd ai_vid_stock_material/workflows; put "<local>" "<creator>-<descriptive>.json"'
|
||
|
|
```
|
||
|
|
|
||
|
|
**Naming convention**: `<creator>-<model>-<workflow-type>.json`
|
||
|
|
Examples: `amao2001-ltx2.5-video_ltx2_5_t2v1.json`, `vionex-krea2-film-studio-v01.json`
|
||
|
|
|
||
|
|
**Target**: `proxmoxBackup/ai_vid_stock_material/workflows/` on TrueNAS (10.0.0.117).
|
||
|
|
|
||
|
|
## Pitfalls
|
||
|
|
|
||
|
|
- **Never skip the brain injection step.** The user's memory directive explicitly
|
||
|
|
requires both transcript AND brain injection for every YouTube link.
|
||
|
|
- **Never skip the BM25 verification.** A write that can't be found via BM25 is
|
||
|
|
effectively invisible — the `ai_brain_kb.py` helper is the only interface that
|
||
|
|
produces searchable points.
|
||
|
|
- **Don't overwrite existing NAS workflows.** Use creator prefixes to avoid collisions
|
||
|
|
with workflows already in the directory.
|
||
|
|
- **The `youtube-content` and `ai-brain-kb` skills are protected** — load them for
|
||
|
|
reference but don't attempt to patch them. This skill is the integration layer
|
||
|
|
between them.
|