Video Semantic Search
Ask a question in plain language, get the exact timestamp — a Go/Python distributed system over Whisper and Qwen3-VL.
Query unit
Timestamped segment
Modalities
Speech + vision
Ingest
Kafka-driven
Store
PostgreSQL + vectors
The problem
Video is the least searchable format in common use. Filename and title get you to the file; nothing gets you to the moment. The unit people actually want is 'the part where they explain the retry logic', which is a semantic query against a span of time.
So the system's atom is not a video — it is a timestamped segment, indexed by what is said in it and what is visible in it.
End-to-end flow — natural-language query to the exact timestamp
Screen capture: type a question, jump straight to the moment
Architecture
Two planes with different shapes, so they get different languages. Ingest is a slow, GPU-bound, failure-prone Python pipeline. Serving is a fast, concurrent, IO-bound Go service. Kafka is the seam between them: ingest emits segment records, the indexer consumes them, and neither side blocks the other.
- Go backend — query API, ranking, segment lookup, playback offsets
- Python ML services — Whisper for transcription, Qwen3-VL for visual description of sampled frames
- Kafka — decouples upload/ingest from indexing so a GPU backlog never stalls the API
- PostgreSQL — segment metadata plus vector search in one store
- Next.js frontend — query, results, and deep-linked playback
System diagram — ingest plane, Kafka, index, query plane
Making the two modalities agree
Speech and vision produce very different text. Transcripts are verbatim and dense; VLM frame descriptions are sparse and interpretive. Embedding both into one index and ranking them naively lets whichever modality is wordier dominate the results.
The fix is treating them as separate signals over the same segment key and merging at rank time, so a segment that is weakly supported by both beats one that is loudly supported by only the chattier modality.
Results panel — segment hits with scores and thumbnails
What I would do next
- Adaptive frame sampling — more frames where the scene changes, fewer on static shots
- Speaker diarization so 'who said it' becomes a filter
- Result caching keyed on query embedding rather than raw string