Yapper

What Yapper does

Yapper takes a full-length film and returns a ten-minute commentary short: condensed footage from the movie playing under an AI voiceover that retells the story and riffs on it. The format is a genre of its own on Douyin and Kuaishou — 电影解說. Made by hand it is a day of work per film: watch it, choose the beats, write a script that stays funny, record it, cut the picture to the narration.

You give Yapper the file and the output language. Nothing else. Roughly fifteen minutes later there is a finished, subtitled MP4.

• In — a full-length movie file, plus the language you want the commentary in
• Out — a 10-12 minute cut with condensed footage, a cloned-voice narrator, and burned-in subtitles
• Between — 13 pipeline stages across three compute zones: a local orchestrator, a 6-GPU box, and a cloud LLM

Watch me run it

Two minutes, start to finish: submit a film, then watch the run move through its stages. Every stage reports its own state and duration, so a run in flight tells you exactly where it is instead of showing a spinner.

The result

This is a finished cut, exactly as the pipeline produced it — no manual editing after the run. The narration is voice-cloned TTS reading a script the pipeline wrote from the film itself; the picture is footage the pipeline selected, ordered and conformed to that narration.

End-to-end flow

Thirteen stages, split into a front half that builds an understanding of the film and a back half that writes and produces the commentary. Each stage is a pure function: it consumes one validated artifact and produces the next. Twelve of them cache that artifact to disk, so a stage whose output already exists is skipped on re-run — which is why changing a prompt costs the back half, not the whole film.

The source film never leaves the local machine. Only derived data crosses a network boundary: 16 kHz audio and keyframe JPEGs go out, transcripts and voiceover WAVs come back.

The load-bearing idea: the model never invents a timestamp

The obvious way to build this is to ask a model for narration plus the timecodes to show under it. It does not work — timecodes are exactly the kind of detail a language model will produce confidently and wrongly, and a hallucinated one is not visible until it has already been paid for in GPU time and shows up as the wrong shot in the finished render.

So Yapper never lets the model emit a timestamp. Stage s04 groups roughly 1-2.5k detected shots into ~300 narrative scenes, gives each a stable id — clip_0042 — and writes them to a SQLite index with their real timecodes. The model sees clips by id and references them by id. One function, resolve_clip_refs, is the only path from an id back to a time, and it raises on any id it does not recognise.

That turns a whole class of failure into a cheap deterministic check. Stage s08 validates every reference in the script against the index before a single line goes to TTS, so a hallucinated clip id kills the run at the point where it costs nothing rather than after several GPU-minutes of speech synthesis. The same stage recomputes the spoken duration from the text and trims the script to its time budget, protecting the climax and closing beats.

The editing rule is the mirror image of the same instinct. Screen time for a narration line equals the measured duration of its synthesized voiceover — measured with ffprobe after the fact, never estimated beforehand — so footage is conformed to the narration instead of the narration being time-stretched to fit the footage.

How it runs as a service

The CLI pipeline is the engine; the hosted version wraps it in a small distributed system built around one constraint — GPU memory is finite and model loads are slow, so GPU work has to be serialized without serializing everything else.

Work is dispatched to five Celery queues whose concurrency encodes the physical limits: cpu for ffmpeg and scene work, asr and tts for the two GPU services, llm for the API-bound scripting stages, and render sized to the core count. ASR and TTS sit on different physical GPUs, so those two lanes run concurrently while each stays serialized within itself.

GPU processes are not kept resident. A daemon called gpud runs always-on with no model loaded and owns a bounded pool of ASR and TTS model servers. A worker leases an instance for the duration of one task; gpud either hands back a warm idle process or starts one on a GPU with enough free VRAM, returns its address, and reaps it after an idle grace once released. Leases carry a heartbeat, so a worker that dies mid-task cannot strand a GPU. Model traffic goes worker-to-server directly over gRPC — gpud is control plane only, never in the data path.

• Orchestration — Celery over five queues (cpu · asr · llm · tts · render), concurrency set to the physical limit of each resource
• GPU supervision — gpud leases model-server instances per task, with heartbeats, idle reaping, and VRAM-aware placement
• Transport — gRPC to the ASR and TTS servers; the LLM stages call an OpenAI-compatible endpoint (MiniMax-M3)
• Models — WhisperX for forced-aligned transcription, CosyVoice2 for seed-pinned voice cloning
• Caching — every stage artifact is content-addressed, so re-runs resume rather than restart
• Deployment — Docker and Kubernetes manifests, Terraform for infrastructure, Prometheus and Grafana for telemetry

What I watch while it runs

GPU seconds and token spend are the entire cost model, so the dashboard is built around them rather than around generic cluster health. Per-stage duration, queue depth per lane, VRAM per GPU, and cumulative LLM spend broken out by stage answer almost every question I have about a run.

The render realtime-factor panel is the one I check first: output seconds divided by render seconds, so 1.0 means the pipeline produces a minute of finished video per minute of wall clock. It makes runs of different lengths comparable, which a raw duration never does.

Per-stage cost visibility earns its keep by being surprising. Understanding the film costs more than writing the commentary about it — 44.6k tokens in against the script stage's 13.3k, and about a third more spend — which is not the split you would guess, and it points at where optimisation effort actually pays.

What I would do next

• Streaming preview — play the first rendered segment while later stages are still running
• Autoscale each lane against its own queue depth instead of a fixed worker count
• A human-in-the-loop pass on the script stage that keeps the cached front half intact

Stack