{ "source": "./your-clip.mp4", "duration_ms": 124800, "scenes": [ { "id": 0, "start_ms": 0, "end_ms": 4200, "shot_type": "wide" }, { "id": 1, "start_ms": 4200, "end_ms": 9800, "shot_type": "medium" } ], "transcript": [ { "start_ms": 120, "end_ms": 2800, "text": "Welcome back to the show." }, { "start_ms": 3100, "end_ms": 5400, "text": "Today we're breaking down..." } ], "entities": [ { "tracking_id": "p1", "label": "person", "confidence": 0.93, "appearances": [ { "scene_id": 0, "start_ms": 0, "end_ms": 4200, "bboxes": [{ "t_ms": 2100, "bbox": [120, 88, 412, 720] }] }, { "scene_id": 1, "start_ms": 4200, "end_ms": 9800, "bboxes": [{ "t_ms": 6500, "bbox": [200, 100, 380, 700] }] } ] } ], "actions": [{ "scene_id": 1, "start_ms": 5400, "end_ms": 7200, "label": "pointing at chart", "confidence": 0.87 }], "ocr": [{ "start_ms": 0, "end_ms": 4200, "text": "LIVE • Q3 EARNINGS", "bbox": [40, 20, 320, 60] }], "keyframes": [{ "scene_id": 0, "t_ms": 2100, "jpeg_b64": "..." }], "_version": "1.4.1", "_generated_at": "2026-05-14T22:14:08.901Z" } ``` Now Claude can say: > *"At 5.4s — second scene — the speaker points at a chart (action confidence 0.87) and says 'today we're breaking down...'. The same speaker (tracked as p1 across both scenes) was on a wide shot for the first 4.2 seconds."* …instead of: > *"The video appears to show some kind of presentation."* That's the whole product. --- ## Just give me a video You need **Node 20+**. That's it. ```bash npm install -g vzt-video-intel vintel analyze ./demo.mp4 # first run prompts you to pick a mode ``` The CLI auto-detects what's available and picks the best execution path. Two modes ship out of the box: ### 🪶 Lite mode — free, offline, zero API key ```bash npm install -g vzt-video-intel vintel analyze ./demo.mp4 ``` Pure-Node WASM pipeline. **Verified working on a fresh Windows machine with no GPU, no API key:** | Stage | Backend | Verified | |---|---|---| | Transcript | `@xenova/transformers` Whisper-tiny (ONNX) | 4.6s for a 12s clip | | Scenes | `ffmpeg-static` content-aware filter | < 1s | | OCR | `tesseract.js` (WASM) | 93% confidence, accurate bboxes | | Keyframes | `ffmpeg-static` | < 1s | | Semantic search | `@xenova/transformers` CLIP ViT-B/32 (ONNX) | ~7s for a 12s clip | Heavy backends (Qwen-VL action recognition, SAM2 entity tracking) skip gracefully in lite mode — set a Replicate token to enable them. No native compilation. No Python. Runs on macOS / Linux / Windows identically. ### 🌩 Cloud mode — full pipeline, ~$0.06/min ```bash npm install -g vzt-video-intel vintel login # paste a Replicate token (https://replicate.com/account/api-tokens) vintel analyze https://example.com/clip.mp4 ``` Heavy stages (Qwen2.5-VL, SAM2) run on Replicate. Light stages (scenes, keyframes) still run locally via ffmpeg-static — no point spending cloud cycles on them. Works on a fresh MacBook, a Codespace, a Lambda. ### Auto — let it pick ```bash vintel auto # prints recommended mode + per-stage routing vintel auto --apply # persists the recommendation ``` `vintel auto` checks for ffmpeg and a Replicate token, then picks the best mode automatically. The first-run wizard runs the same flow interactively the first time you call `vintel analyze`. > Prefer not to install globally? `npx vzt-video-intel analyze ./demo.mp4` works too — or clone the repo and run `node bin/vzt-video-intel.js`. The output schema is **identical** across both modes — only the execution path changes. Scene graphs you produce in lite mode are byte-for-byte compatible with cloud-mode scene graphs (minus the entities/actions arrays when those stages skip). --- ## Verified end-to-end Every stage was smoke-tested before tagging v1.4.1. On a fresh Windows machine with no GPU, no Replicate token: ``` $ npm install -g vzt-video-intel $ vintel auto Environment: ✓ ffmpeg ✗ REPLICATE_API_TOKEN Resolved mode: lite no cloud key — falling back to pure-Node lite mode (free + offline) Per-stage routing: transcribe → lite scenes → lite ocr → lite clip → lite entities → skip actions → skip $ vintel analyze ./demo.mp4 { "source": "./demo.mp4", "duration_ms": 12000, "scenes": [{ "id": 0, "start_ms": 0, "end_ms": 8000 }, { "id": 1, "start_ms": 8000, "end_ms": 12000 }], "transcript": [{ "start_ms": 0, "end_ms": 12000, "text": "[Music]" }], "ocr": [{ "start_ms": 0, "end_ms": 1000, "text": "SCENE", "bbox": [80, 98, 64, 21], "confidence": 0.93 }, ... 23 more], "keyframes": [{ "scene_id": 0, "t_ms": 4000, "width": 320, "height": 240, "jpeg_b64": "..." }, ...], "entities": [], "actions": [], "_version": "1.4.1" } real 0m4.620s ``` 12-second clip, full lite pipeline, **4.6 seconds wall-clock on CPU**. See [CHANGELOG.md](CHANGELOG.md) for the bugs caught + fixed during smoke testing. --- ## The cost math — per video, not per question Native video APIs charge **per question**: every time you ask about a video, the whole clip is re-ingested and re-billed. VZT Video-Intel charges **per video, once** — analyze it, and every subsequent query reads the cached scene graph for $0. Say you ask 10 questions about a 1-hour video: | Approach | Analyze | 10 queries | Total | |---|---|---|---| | **Gemini 3.1 native** | — | re-ingests the hour each time, ~$2.80 × 10 | **~$28** | | **VZT Video-Intel — cloud mode** | ~$3.60 once | cached scene graph, $0 each | **~$3.60** | | **VZT Video-Intel — lite mode** | $0 (your CPU) | cached scene graph, $0 each | **$0** | The structured output is the point: lite mode is free and offline, and *every* mode hands you a Claude-citable graph you analyze once and query forever. The more you interrogate a video, the wider the gap. --- ## Architecture ```mermaid flowchart LR A[Video file or URL] --> B[CLI / MCP server] B --> C[Pipeline orchestrator] C --> D[Whisper
transcription] C --> E[ffmpeg
scenes + keyframes] C --> F[Tesseract / cloud
OCR] C --> G[SAM2 cloud
entity tracking] C --> H[VLM caption lite / Qwen2.5-VL cloud
actions + chapters] C --> I[CLIP
semantic search] D --> J[Scene Graph JSON] E --> J F --> J G --> J H --> J I --> J J --> K[Claude / your code] ``` **Stage 1** (parallel): scenes + transcript + OCR — fully independent, fire concurrently. **Stage 2** (per-scene): entity tracking + action recognition + keyframe extraction. **Stage 3** (on demand): CLIP semantic search ("find me the moment when X happens"). Each stage has a **lite** (pure-Node WASM) and a **cloud** (Replicate) adapter. The orchestrator picks per stage based on the resolved runtime mode — same JSON output either way. See [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). --- ## CLI reference ``` vintel [options] # vzt-video-intel also works analyze full pipeline → scene graph JSON observe watch + listen → one fused perception timeline transcribe Whisper transcription scenes scene boundaries (ffmpeg) entities SAM2 entity tracking (cloud) keyframes per-scene keyframes (base64 JPEG) ocr on-screen text search CLIP semantic moment search chapters chapter generation (lite captions / cloud Qwen2.5-VL) auto [--apply] detect environment + recommend the best mode config [show|set k=v] show or edit persisted config cache [list|clear|path] inspect the persistent scene-graph store login [token] store a Replicate API token mcp run as MCP stdio server (for Claude Code, Cursor, OpenCode) ``` All commands accept `--help` for full option lists. ### Examples ```bash # Full pipeline, skip the expensive entity tracking and action recognition vintel analyze ./game.mp4 --no-entities --no-actions # Watch AND listen — one timeline fusing speech, visuals, on-screen text + scenes vintel observe ./talk.mp4 --format=text # Re-analyzing is instant — the scene graph is cached. --refresh forces a re-run. vintel analyze ./game.mp4 # second call returns from the cache vintel cache # list cached scene graphs # Stream the scene graph as JSONL — each track the moment it's produced vintel analyze ./long-keynote.mp4 --stream | jq -c 'select(.type=="scene_analysis")' # Build a library, then search across ALL of it at once (see docs/CORPUS.md) vintel index ./clips # analyze every video under ./clips (cached ones are instant) vintel search "whiteboard diagram" # cross-video search — 1 arg = corpus vintel search ./highlight.mp4 "ball crossing the goal line" --top-k=5 # 2 args = single-video CLIP # Score the pipeline against gold fixtures — make a model swap measurable (see docs/EVAL.md) vintel eval --ci # Transcribe only, Spanish hint vintel transcribe ./meeting.m4a --language=es # YouTube-style chapters (requires Replicate token) vintel chapters ./lecture.mp4 --style=course --count=12 # Pipe straight to jq vintel transcribe ./call.mp3 | jq '.segments[] | .text' ``` --- ## Connect it to your AI assistant (MCP) VZT Video-Intel is also an **MCP server** — the standard way an AI assistant calls an external tool. Once connected, you don't run CLI commands; you just *ask* your assistant about a video and it calls the tools, gets the scene graph, and answers with real timestamps. It works in **Claude Code, Claude Desktop, Cursor, Codex, GitHub Copilot, and Antigravity**. **1. Install globally** (puts `vintel` on your PATH and makes the assistant launch instantly): ```bash npm install -g vzt-video-intel ``` **2. Wire it into your app** — one command writes the right config file, in the right format, preserving any servers you already have: ```bash vintel install claude # Claude Code → ~/.claude.json vintel install claude-desktop # Claude Desktop → per-OS app config vintel install cursor # Cursor → ~/.cursor/mcp.json vintel install codex # Codex → ~/.codex/config.toml vintel install antigravity # Antigravity → ~/.gemini/config/mcp_config.json vintel install copilot # VS Code Copilot → .vscode/mcp.json (project) vintel install all # everything except project-local copilot vintel install copilot --global # prints VS Code user-level setup instructions vintel install claude --print # show the config snippet, write nothing ``` **3. Activate it** (each app picks up MCP config differently): | App | After install, do this | |---|---| | **Claude Code** | reloads automatically | | **Claude Desktop** | **fully quit** (system-tray → Quit) and reopen — closing the window isn't enough | | **Cursor** | Settings → MCP (the server should show green) | | **Codex** | run `codex`, then `/mcp` to confirm | | **Antigravity** | auto-reloads; open an Agent session | Then just ask: *"Analyze ./game.mp4 and tell me what happens at the 2-minute mark"*, or *"Index my ./clips folder, then find every moment someone mentions pricing."* > **Windows note:** desktop apps spawn the server without a shell, so `vintel install` automatically writes the Windows-safe `cmd /c vzt-video-intel mcp` launch form (it uses plain `npx` on macOS/Linux). You don't have to do anything — it just works. **Cloud vs lite for your assistant:** run `vintel login` once and **every** connected app inherits cloud mode from `~/.vzt-video-intel/config.json` — **no token goes into any editor config**. Without a token, the assistant runs in free lite mode. (If you'd rather embed an explicit token in one app's config, pass `vintel install --token r8_…`.) Prefer to wire it by hand instead of `vintel install`? Add this to `~/.claude.json` (or your project's `.mcp.json`); on Windows use `"command": "cmd", "args": ["/c", "vzt-video-intel", "mcp"]`: ```json { "mcpServers": { "vzt-video-intel": { "command": "npx", "args": ["vzt-video-intel", "mcp"] } } }

VZT Video-Intel

Installation