VideoQnA Application

VideoQnA is a framework that retrieves video based on provided user prompt. It uses only the video embeddings to perform vector similarity search in Intel’s VDMS vector database and performs all operations on Intel Xeon CPU. The pipeline supports long form videos and time-based search.

VideoQnA is implemented on top of GenAIComps, with the architecture flow chart shows below:

flowchart LR %% Colors %% classDef blue fill:#ADD8E6,stroke:#ADD8E6,stroke-width:2px,fill-opacity:0.5 classDef orange fill:#FBAA60,stroke:#ADD8E6,stroke-width:2px,fill-opacity:0.5 classDef orchid fill:#C26DBC,stroke:#ADD8E6,stroke-width:2px,fill-opacity:0.5 classDef invisible fill:transparent,stroke:transparent; style VideoQnA-MegaService stroke:#000000 %% Subgraphs %% subgraph VideoQnA-MegaService["VideoQnA-MegaService"] direction LR EM([Embedding <br>]):::blue RET([Retrieval <br>]):::blue RER([Rerank <br>]):::blue LLM([LLM <br>]):::blue end subgraph User Interface direction TB a([User Input Query]):::orchid UI([UI server<br>]):::orchid Ingest([Ingest<br>]):::orchid end subgraph VideoQnA GateWay direction LR invisible1[ ]:::invisible GW([VideoQnA GateWay<br>]):::orange end subgraph . X([OPEA Micsrservice]):::blue Y{{Open Source Service}} Z([OPEA Gateway]):::orange Z1([UI]):::orchid end LOCAL_RER{{Reranking service<br>}} CLIP_EM{{Embedding service <br>}} VDB{{Vector DB<br><br>}} V_RET{{Retriever service <br>}} Ingest{{Ingest data <br>}} DP([Data Preparation<br>]):::blue LVM_gen{{LLM Service <br>}} %% Data Preparation flow %% Ingest data flow direction LR Ingest[Ingest data] -->|a| DP DP <-.->|b| CLIP_EM %% Questions interaction direction LR a[User Input Query] -->|1| UI UI -->|2| GW GW <==>|3| VideoQnA-MegaService EM ==>|4| RET RET ==>|5| RER RER ==>|6| LLM %% Embedding service flow direction TB EM <-.->|3'| CLIP_EM RET <-.->|4'| V_RET RER <-.->|5'| LOCAL_RER LLM <-.->|6'| LVM_gen direction TB %% Vector DB interaction V_RET <-.->|d|VDB DP <-.->|d|VDB

  • This project implements a Retrieval-Augmented Generation (RAG) workflow using LangChain, Intel VDMS VectorDB, and Text Generation Inference, optimized for Intel Xeon Scalable Processors.

  • Video Processing: Videos are converted into feature vectors using mean aggregation and stored in the VDMS vector store.

  • Query Handling: When a user submits a query, the system performs a similarity search in the vector store to retrieve the best-matching videos.

  • Contextual Inference: The retrieved videos are then sent to the Large Vision Model (LVM) for inference, providing supplemental context for the query.

Deploy VideoQnA Service

The VideoQnA service can be effortlessly deployed on Intel Xeon Scalable Processors.

Required Models

By default, the embedding and LVM models are set to a default value as listed below:

Service

Model

Embedding

openai/clip-vit-base-patch32

LVM

DAMO-NLP-SG/Video-LLaMA

Deploy VideoQnA on Xeon

For full instruction of deployment, please check Guide

Currently we support deploying VideoQnA services with docker compose, using the docker images built from source. Find the corresponding compose.yaml.