YouTube Knowledge Ingestion Pipeline

When to use

User drops a YouTube link for AI/ML video content (ComfyUI workflows, LTX, MiniMax, Krea, model tutorials, prompting guides). Run the full three-step pipeline — never skip any step.

Pipeline (always all three steps)

1. Transcript

Load the youtube-content skill and fetch:

uv run python3 <SKILL_DIR>/scripts/fetch_transcript.py "URL" --text-only --timestamps

If youtube-transcript-api is missing, install with pip3 install --user youtube-transcript-api.

2. Brain Injection (MANDATORY)

Load the ai-brain-kb skill. Always dedup-first, then ingest:

# Dedup check
python3 /home/n8n/bin/ai_brain_kb.py search --query "<key topic>" --limit 5

# Ingest — use --type workflow for tutorials
python3 /home/n8n/bin/ai_brain_kb.py add \
  --type workflow \
  --title "Descriptive Title — Key Topics" \
  --stage <t2v|i2v|upscale|...> \
  --tool <ltx-video|minimax-h3|krea2|...> \
  --host 10.0.0.202 \
  --trust community \
  --importance 0.7-0.85 \
  --tags "comma,separated,keywords,creator-name" \
  --url "https://www.youtube.com/watch?v=VIDEO_ID" \
  --content "..." \
  --json

# Verify BM25 leg
python3 /home/n8n/bin/ai_brain_kb.py search --query "<distinctive phrase>" \
  --mode bm25 --doc-id <doc_id>

Content format: Dense, factual technical summary. Use CAPITALIZED section headers. Include specific settings, model names, thresholds, commands, and failure modes. This is a technical reference, not a blog post.

MANDATORY: workflow links in the brain content. Every brain entry must include a ## WORKFLOW LINKS section listing the DOWNLOADED workflow files on NAS (TrueNAS 10.0.0.117, proxmoxBackup/ai_vid_stock_material/workflows/, exact filenames + the smbclient get command), the source repo URL, and the video URL. The NAS copies are the primary links — the user wants links to the downloaded workflows, not upstream URLs. If the description links elsewhere (Patreon etc.) and is unreachable, say so in the content. Append as a separate chunk via --doc-id if the main content was already ingested.

Tag strategy: model name, tool name, creator name, key techniques. Tags are exact-match keyword indexes.

3. NAS Workflow Download

If the video description contains workflow links (GitHub repos, direct JSON files), extract the description with yt-dlp --print description "URL", clone the repos, and copy workflow JSONs to the NAS:

# Clone
cd /tmp && rm -rf <repo> 2>/dev/null
git clone --depth 1 <repo-url>

# Find workflows
find /tmp/<repo> -name "*.json"

# Copy to NAS with creator prefix for disambiguation
smbclient -N //10.0.0.117/proxmoxBackup \
  -c 'cd ai_vid_stock_material/workflows; put "<local>" "<creator>-<descriptive>.json"'

Naming convention: <creator>-<model>-<workflow-type>.json Examples: amao2001-ltx2.5-video_ltx2_5_t2v1.json, vionex-krea2-film-studio-v01.json

Target: proxmoxBackup/ai_vid_stock_material/workflows/ on TrueNAS (10.0.0.117).

Pitfalls

Never skip the brain injection step. The user's memory directive explicitly requires both transcript AND brain injection for every YouTube link.
Never skip the BM25 verification. A write that can't be found via BM25 is effectively invisible — the ai_brain_kb.py helper is the only interface that produces searchable points.
Don't overwrite existing NAS workflows. Use creator prefixes to avoid collisions with workflows already in the directory.
The youtube-content and ai-brain-kb skills are protected — load them for reference but don't attempt to patch them. This skill is the integration layer between them.

3.8 KiB Raw Blame History