video-search-and-summarization
NVIDIA-AI-Blueprints · NVIDIA-AI-Blueprints/video-search-and-summarization
This NVIDIA AI Blueprint provides an MCP for building GPU-accelerated video analytics agents that search, summarize, and reason over live or recorded video using natural language.
NVIDIA AI Blueprint for video search and summarization (VSS) is a GPU-accelerated reference architecture for building video analytics agents with real-time verified alerts, visual Q&A, and automated reporting. The VSS Blueprint uses vision language models (VLMs) such as NVIDIA Cosmos, LLMs such as NVIDIA Nemotron, RAG, and NVIDIA NIMs.
Preview
What you can do with it
- Search through hours of video footage using natural language queries.
- Automatically summarize long videos or extract key events.
- Implement visual Q&A systems or real-time alert verification for video streams.
Safety check
Usable, but read the notes first.
- ●No clear license: check with the author before using it in a commercial product.
Automated scan of the repo files. Always review code you run.
From the README
NVIDIA AI Blueprint: Video Search and Summarization (VSS)
Build GPU-accelerated video AI agents that search, analyze, summarize, and reason over live or recorded video using natural language.
NVIDIA AI Blueprint for Video Search and Summarization (VSS) combines vision-language models, RAG, and NVIDIA NIM microservices to deliver real-time video analytics, visual Q&A, alert verification, clip retrieval, and long-video summarization.
- Search video streams or archives using natural language queries
- Summarize hours of video
- Ask visual questions and automatically generate reports
- Detect and verify real-time alerts with VLMs
🚀 Try the Demo · ⚡ Quickstart · 📚 Documentation · 🏗️ Architecture · 📦 Latest Release
Table of Contents
- Overview
- Use Case / Problem Description
- Agent Workflows
- Software Components
- Target Audience
- Repository Structure Overview
- Documentation
- Prerequisites
- Hardware Requirements
- Quickstart Guide
- Contributing
- License
Overview
The NVIDIA Blueprint for Video Search and Summarization (VSS) provides a suite of reference architectures for building vision agents and AI-powered video analytics applications. Those architectures bring together accelerated vision microservices, vision language models (VLMs), and large language models (LLMs) so you can use them in existing applications, as standalone microservices, or as part of a larger vision agent.
VSS is organized into three areas of processing and analysis: real-time video intelligence (feature extraction, embeddings, and stream understanding with results published to a message broker), downstream analytics (enrichment of metadata into trajectories, incidents, and verified alerts), and agentic and offline processing (orchestrated tools for search, Q&A, summarization, and clip retrieval, including via the Model Context Protocol).
This repository implements the blueprint and powers the NVIDIA build experience for natural-language video agents—search, summarization, visual Q&A, and related workflows—backed by generative AI, VLMs and LLMs, and NVIDIA NIM microservices as configured in the stacks below.
Use Case / Problem Description
The NVIDIA AI Blueprint for Video Search and Summarization addresses the challenge of deploying visual agents capable of interacting with large volumes of video data, both stored and streamed. This can be used to create vision AI agents, that can be applied to a multitude of use cases such as monitoring smart spaces, warehouse automation, and SOP validation. This is important where quick and accurate video analysis can lead to better decision-making and enhanced operational efficiency.
Agent Workflows
We provide multiple reference Agent Workflows which demonstrate how the individual components can be leveraged by an agent:
| Workflow | Description |
|---|---|
| Q&A and Report Generation (Quickstart) | Video retrieval, VLM-based Q&A, and report generation on short video clips |
| Alert Verification | Realtime processing of videos using perception (object detection, tracking) and behavior analytics to generate alerts, which are subsequently verified with VLM to reduce false positives |
| Real-Time Alerts | Continuous processing of video streams through VLM for anomaly detection |
| Video Search | Natural language search across video archives using video embeddings (alpha) |
| Long Video Summarization | Analysis and summarization of extended video recordings through chunking and aggregation of dense captions |
Software Components
-
NIM microservices: Here are models used in this blueprint:
-
Real-time video intelligence: The Real-Time Video Intelligence layer extracts rich visual features, semantic embeddings, and contextual understanding from video data in real-time, publishing results to a message broker for downstream analytics and agentic workflows. It provides three core microservices for processing video streams.
-
Downstream analytics: The Downstream Analytics layer processes and enriches the metadata streams generated by real-time video intelligence microservices, transforming raw detections into actionable insights and verified alerts.
-
Agent and offline processing: The top-level agent leverages the Model Context Protocol (MCP) to access video analytics data, incident records, and vision processing capabilities through a unified tool interface. It integrates multiple vision-based tools including video understanding with Vision Language Models (VLMs), semantic video search using embeddings, long video summarization for extended footage analysis, and video snapshot/clip retrieval.
Target Audience
This blueprint is designed for ease of setup with exten