OPEA™ Logo
1.2
OPEA Project v: 1.2
Document Versions
latest
1.0
1.1
OPEA Project links
Project Home
Wiki
  • Documentation Home
  • OPEA Overview
    • OPEA Project Architecture
      • Microservices: Flexible and Scalable Architecture
      • Megaservices: A Comprehensive Solution
      • Gateways: Customized Access to Mega- and Microservices
    • Next Step
      • Open Platform for Enterprise AI (OPEA) Framework Draft Proposal
        • 1. Summary
        • 2. Introduction
        • 3. Framework Components, Architecture and Flow
        • 4. Assessing GenAI components and flows
        • 5. Grading Structure
        • 6. Reference flows
        • Appendix A – Draft OPEA Specifications
  • Getting Started with OPEA
    • Understanding OPEA’s Core Components
    • Prerequisites
    • Create and Configure a Virtual Server
    • Deploy the ChatQnA Solution
      • Interact with ChatQnA
    • What’s Next
      • Get Involved
  • GenAI Examples
    • AgentQnA Sample Guide
      • Overview
      • Purpose
      • How It Works
      • Deployment
        • Single Node
    • ChatQnA Sample Guide
      • Overview
      • Purpose
      • Key Implementation Details
      • How It Works
        • Expected Output
        • Validation Matrix and Prerequisites
      • Architecture
        • Microservice Outline and Diagram
      • Deployment
      • Single Node
        • Xeon Scalable Processor
        • Gaudi AI Accelerator
        • Nvidia GPU
        • AI PC
      • Kubernetes
      • Cloud Native
      • Troubleshooting
      • Monitoring
        • Set Up the Prometheus Server
        • Set Up the Grafana Dashboard
        • Summary and Next Steps
    • Codegen Sample Guide
      • Overview
      • Purpose
      • How It Works
      • Deployment
        • Intel® Xeon® Scalable processor
        • Gaudi AI Accelerator
    • Code Translation Sample Guide
      • Overview
      • Purpose
      • How It Works
      • Deployment
        • Single Node
    • Docsum Sample Guide
      • Overview
      • Purpose
      • How It Works
      • Deployment
        • Intel® Xeon® Scalable processor
        • Gaudi AI Accelerator
    • Generative AI Examples
      • Introduction
      • Architecture
      • Getting Started
        • Deployment Guide
      • Supported Examples
      • Contributing to OPEA
      • Additional Content
    • Examples
      • AgentQnA Application
        • Agents for Question Answering
        • Single node on-prem deployment with Docker Compose on AMD GPU
        • Single node on-prem deployment with Docker Compose on Xeon Scalable processors
        • Single node on-prem deployment AgentQnA on Gaudi
        • Deploy AgentQnA on Kubernetes cluster
        • Retrieval tool for agent
        • AgentQnA
      • AudioQnA Application
        • AudioQnA Application
        • AudioQnA Accuracy
        • AudioQnA Benchmarking
        • Build Mega Service of AudioQnA on AMD ROCm GPU
        • Build Mega Service of AudioQnA on Xeon
        • Build Mega Service of AudioQnA on Gaudi
        • Deploy AudioQnA in Kubernetes Cluster on Xeon and Gaudi
        • Deploy AudioQnA on Kubernetes cluster
        • AudioQnA
      • AvatarChatbot Application
        • AvatarChatbot Application
        • Build Mega Service of AvatarChatbot on Xeon
        • Build Mega Service of AvatarChatbot on Gaudi
      • ChatQnA Application
        • ChatQnA Application
        • ChatQnA Accuracy
        • ChatQnA Benchmarking
        • Build and deploy CodeGen Application on AMD GPU (ROCm)
        • Build Mega Service of ChatQnA on AIPC
        • Build Mega Service of ChatQnA on Xeon
        • Build Mega Service of ChatQnA on Xeon
        • Build Mega Service of ChatQnA (with Qdrant) on Xeon
        • Build MegaService of ChatQnA on Gaudi
        • How to Check and Validate Micro Service in the GenAI Example
        • Build MegaService of ChatQnA on NVIDIA GPU
        • Deploy ChatQnA in Kubernetes Cluster on Xeon and Gaudi
        • Deploy ChatQnA on Kubernetes cluster
        • ChatQnA Conversational UI
        • ChatQnA Customized UI
      • CodeGen Application
        • Code Generation Application
        • CodeGen Accuracy
        • CodeGen Benchmarking
        • Build and deploy CodeGen Application on AMD GPU (ROCm)
        • Validate the MicroServices and MegaService
        • Build MegaService of CodeGen on Xeon
        • Build MegaService of CodeGen on Gaudi
        • Deploy CodeGen in a Kubernetes Cluster
        • Deploy CodeGen on Kubernetes cluster
        • Code Gen
        • Code Gen
      • CodeTrans Application
        • Code Translation Application
        • CodeTrans Benchmarking
        • Build and deploy CodeTrans Application on AMD GPU (ROCm)
        • Validate the MicroServices and MegaService
        • Build Mega Service of CodeTrans on Xeon
        • Build Mega Service of CodeTrans on Gaudi
        • Deploy CodeTrans in a Kubernetes Cluster
        • Deploy CodeTrans on Kubernetes cluster
        • Code Translation
      • DBQnA Application
        • DBQnA Application
        • Deploy on Intel Xeon Processor
        • DBQnA React Application
      • DocIndexRetriever Application
        • DocRetriever Application
        • DocRetriever Application with Docker
        • DocRetriever Application with Docker
      • DocSum Application
        • Document Summarization Application
        • Build and deploy DocSum Application on AMD GPU (ROCm)
        • Build Mega Service of Document Summarization on Intel Xeon Processor
        • Build MegaService of Document Summarization on Gaudi
        • Deploy DocSum in Kubernetes Cluster
        • Deploy DocSum on Kubernetes cluster
        • Document Summary
        • Doc Summary React
        • Doc Summary
      • EdgeCraftRAG Application
        • Edge Craft Retrieval-Augmented Generation
      • FaqGen Application
        • FAQ Generation Application
        • FaqGen Accuracy
        • FaqGen Benchmarking
        • Build and deploy FaqGen Application on AMD GPU (ROCm)
        • Build Mega Service of FAQ Generation on Intel Xeon Processor
        • Build MegaService of FAQ Generation on Gaudi
        • Deploy FaqGen on Kubernetes cluster
        • Doc Summary React
        • FAQ Generation
      • GraphRAG Application
        • GraphRAG Application
        • ChatQnA Conversational UI
        • ChatQnA Customized UI
      • InstructionTuning Application
        • Instruction Tuning
        • Deploy Instruction Tuning Service on Xeon
        • Deploy Instruction Tuning Service on Gaudi
      • MultimodalQnA Application
        • MultimodalQnA Application
        • Build Mega Service of MultimodalQnA for AMD ROCm
        • Build Mega Service of MultimodalQnA on Xeon
        • Build Mega Service of MultimodalQnA on Gaudi
      • ProductivitySuite Application
        • Productivity Suite Application
        • Build Mega Service of Productivity Suite on Xeon
        • 🔐 Keycloak Configuration Setup
        • Productivity Suite React UI
      • RerankFinetuning Application
        • Rerank Model Finetuning
        • Deploy Rerank Model Finetuning Service on Xeon
        • Deploy Rerank Model Finetuning Service on Gaudi
      • SearchQnA Application
        • SearchQnA Application
        • Build Mega Service of SearchQnA on Xeon
        • Build Mega Service of SearchQnA on Gaudi
        • Deploy SearchQnA in a Kubernetes Cluster
        • Deploy SearchQnA on Kubernetes cluster
        • Neural Chat
      • Text2Image Application
        • Text-to-Image Microservice
        • Deploy Text-to-Image Service on Xeon
        • Deploy Text-to-Image Service on Gaudi
        • Deploy txt2img on Kubernetes cluster
        • Text2Image Customized UI
      • Translation Application
        • Translation Application
        • Build and deploy Translation Application on AMD GPU (ROCm)
        • Validate the MicroServices and MegaService
        • Build Mega Service of Translation on Xeon
        • Build MegaService of Translation on Gaudi
        • Deploy Translation in a Kubernetes Cluster
        • Language Translation
      • VideoQnA Application
        • VideoQnA Application
        • Build Mega Service of VideoQnA on Xeon
      • VisualQnA Application
        • Visual Question and Answering
        • VisualQnA Benchmarking
        • Build Mega Service of VisualQnA on AMD ROCm
        • Build Mega Service of VisualQnA on Xeon
        • Build MegaService of VisualQnA on Gaudi
        • Deploy VisualQnA in a Kubernetes Cluster
        • Deploy VisualQnA on Kubernetes cluster
      • WorkflowExecAgent Application
        • Workflow Executor Agent
        • Validate Workflow Agent Microservice
    • Legal Information
      • License
      • Citation
    • ChatQnA Benchmarking
      • Purpose
      • Table of Contents
      • Prerequisites
      • Data Preparation
      • Overview
        • Using deploy_and_benchmark.py (Recommended)
    • Docker Images
      • Example images
      • Microservice images
    • Supported Examples
      • ChatQnA
        • CodeGen
        • CodeTrans
        • DocSum
        • Language Translation
        • SearchQnA
        • VisualQnA
        • VideoQnA
        • RerankFinetuning
        • InstructionTuning
        • DocIndexRetriever
        • AgentQnA
        • AudioQnA
        • FaqGen
        • MultimodalQnA
        • ProductivitySuite
  • GenAI Microservices
    • Generative AI Components (GenAIComps)
      • GenAIComps
        • Installation
      • MicroService
      • MegaService
      • Contributing to OPEA
      • Additional Content
    • Legal Information
      • License
      • Citation
    • Agent Microservice
      • Deploy Agent microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Agent Microservice
        • 1. Overview
        • 🚀2. Start Agent Microservice
        • 🚀 3. Validate Microservice
        • 🚀 4. Provide your own tools
        • 5. Customize agent strategy
      • Plan Execute
      • RAG Agent
      • SQL Agents
        • Overview of sql_agent_llama
        • Overview of sql_agent
        • Limitations
    • Animation Microservice
      • Avatar Animation Microservice
      • 🚀1. Start Microservice with Docker (option 1)
        • 1.1 Build the Docker images
        • 1.2. Set environment variables
      • 🚀2. Run the Docker container
        • 2.1 Run Wav2Lip Microservice
        • 2.2 Run Animation Microservice
      • 🚀3. Start Microservice with Docker Compose
      • 🚀4. Validate Microservice
        • 4.1 Validate Wav2Lip service
        • 4.2 Validate Animation service
    • Asr Microservice
      • Deploy ASR microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • ASR Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Start Microservice with Docker Compose (Option 3)
    • Chathistory Microservice
      • 📝 Chat History Microservice
        • 🛠️ Features
        • ⚙️ Implementation
      • Deploy chathistory microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • 📝 Chat History Microservice with MongoDB
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
        • ✅ Invoke Microservice
    • Cores Microservice
      • Telemetry for OPEA
        • Metrics
        • Tracing
        • Visualization
    • Dataprep Microservice
      • Dataprep Microservice
        • Install Requirements
        • Use LVM (Large Vision Model) for Summarizing Image Data
        • Dataprep Microservice with Redis
        • Dataprep Microservice with Milvus
        • Dataprep Microservice with Qdrant
        • Dataprep Microservice with Pinecone
        • Dataprep Microservice with PGVector
        • Dataprep Microservice with VDMS
        • Dataprep Microservice with Multimodal
        • Dataprep Microservice with ElasticSearch
        • Dataprep Microservice with OpenSearch
        • Dataprep Microservice with neo4j
      • Deploy dataprep microservice on Kubernetes cluster
        • Deploy on Kubernetes with redis VectorDB
        • Deploy on Kubernetes with milvus VectorDB
      • Dataprep Microservice with Elasticsearch
        • 🚀1. Start Microservice with Python(Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Microservice
      • Dataprep Microservice with Milvus
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Microservice
        • 🚀4. Troubleshooting
      • Dataprep Microservice for Multimodal Data with Redis
        • 🚀1. Start Microservice with Python(Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Status Microservice
        • 🚀4. Consume Microservice
      • Dataprep Microservice with Neo4J
        • 🚀Start Microservice with Python
        • 🚀Start Microservice with Docker
        • Invoke Microservice
      • Dataprep Microservice with Neo4J
        • Setup Environment Variables
        • 🚀Start Microservice with Docker
        • Invoke Microservice
      • Dataprep Microservice with OpenSearch
        • 🚀1. Start Microservice with Python(Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Status Microservice
        • 🚀4. Consume Microservice
      • Dataprep Microservice with PGVector
        • 🚀1. Start Microservice with Python(Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Microservice
      • Dataprep Microservice with Pinecone
        • 🚀Start Microservice with Python
        • 🚀Start Microservice with Docker
        • Invoke Microservice
      • Dataprep Microservice with Qdrant
        • 🚀Start Microservice with Python
        • 🚀Start Microservice with Docker
        • Invoke Microservice
      • Dataprep Microservice with Redis
        • 🚀1. Start Microservice with Python(Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Status Microservice
        • 🚀4. Consume Microservice
      • Dataprep Microservice with VDMS
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Status Microservice
        • 🚀4. Consume Microservice
    • Embeddings Microservice
      • Deploy Embedding microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Embeddings Microservice
        • Embeddings Microservice with TEI
        • Embeddings Microservice with Prediction Guard
        • Embeddings Microservice with Multimodal
      • Multimodal Embeddings Microservice
        • 📦 1. Start Microservice
        • 📦 2. Consume Embedding Service
      • Embedding Microservice with Prediction Guard
        • 📦 1. Start Microservice with docker run
        • 📦 2. Start Microservice with docker compose
        • 📦 3. Consume Embedding Service
        • ✨ Additional Notes
      • 🌟 Embedding Microservice with TEI
        • 📦 1. Start Microservice with docker run
        • 📦 2. Start Microservice with docker compose
        • 📦 3. Consume Embedding Service
        • ✨ Tips for Better Understanding:
    • Feedback_management Microservice
      • 🗨 Feedback Management Microservice
        • 🛠️ Features
        • ⚙️ Implementation
      • 🗨 Feedback Management Microservice with MongoDB
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
    • Finetuning Microservice
      • Fine-tuning Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Finetuning Service
        • 🚀4. Descriptions for Finetuning parameters
    • Guardrails Microservice
      • Trust and Safety with LLM
      • Deploy guardrails microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Bias Detection Microservice
        • Introduction
        • Future Development
        • 🚀1. Start Microservice with Python(Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Get Status of Microservice
        • 🚀4. Consume Microservice Pre-LLM/Post-LLM
      • Factuality Check Prediction Guard Microservice
      • 🚀 Start Microservice with Docker
        • Setup Environment Variables
        • Build Docker Images
        • Start Service
      • 🚀 Consume Factuality Check Service
      • Guardrails Microservice
        • LlamaGuard
        • WildGuard
      • Hallucination Detection Microservice
        • Introduction
        • 🚀1. Start Microservice based on vLLM endpoint on Intel Gaudi Accelerator
        • 2. Set up Hallucination Microservice
        • 🚀3. Get Status of Hallucination Microservice
        • 🚀4. Consume Guardrail Micorservice Post-LLM
      • PII Detection Prediction Guard Microservice
      • 🚀 Start Microservice with Docker
        • Setup Environment Variables
        • Build Docker Images
        • Start Service
      • 🚀 Consume PII Detection Service
      • Prompt Injection Detection Prediction Guard Microservice
      • 🚀 Start Microservice with Docker
        • Setup Environment Variables
        • Build Docker Images
        • Start Service
      • 🚀 Consume Prompt Injection Detection Service
      • Toxicity Detection Microservice
        • Introduction
        • 🚀1. Start Microservice with Python(Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Get Status of Microservice
        • 🚀4. Consume Microservice Pre-LLM/Post-LLM
    • Image2image Microservice
      • Image-to-Image Microservice
      • 🚀1. Start Microservice with Python (Option 1)
        • 1.1 Install Requirements
        • 1.2 Start Image-to-Image Microservice
      • 🚀2. Start Microservice with Docker (Option 2)
        • 2.1 Build Images
        • 2.2 Start Image-to-Image Service with Docker
      • 🚀3. Start Image-to-Image with Docker Compose
      • 4 Test Image-to-Image Service
    • Image2video Microservice
      • Image-to-Video Microservice
      • 🚀1. Start Microservice with Python (Option 1)
        • 1.1 Install Requirements
        • 1.2 Start Image-to-Video Microservice
      • 🚀2. Start Microservice with Docker (Option 2)
        • 2.1 Build Images
        • 2.2 Start Image-to-Video Service
        • 2.3 Test
    • Llms Microservice
      • Deploy LLM microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Document Summary LLM Microservice
        • 🚀1. Start Microservice with Docker 🐳
        • 🚀3. Consume LLM Service
      • FAQGen LLM Microservice
        • 🚀1. Start Microservice with Docker
        • 🚀2. Consume LLM Service
      • LLM text generation Microservice
        • Validated LLM Models
        • Support integrations
        • Clone OPEA GenAIComps
        • Prerequisites
        • 🚀Start Microservice with Docker
        • 🚀3. Consume LLM Service
      • Introduction
        • Get Started
        • Setup Environment Variables
        • Build Docker Image
        • Run the Bedrock Microservice
        • Consume the Bedrock Microservice
      • LLM Native Microservice
        • 🚀1. Start Microservice
        • 🚀2. Consume LLM Service
      • Prediction Guard Introduction
        • Get Started
        • Consume the Prediction Guard Microservice
      • LM-Eval Microservice
        • CPU service
    • Lvms Microservice
      • Deploy LVM microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • LVM Microservice
        • 🚀1. Start Microservice with Docker (Option 1)
        • 🚀1. Start Microservice with Docker Compose (Option 2)
        • Test
      • LVM Microservice
        • 🚀 Start Microservice with Docker
      • LVM Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
      • LVM Prediction Guard Microservice
        • 🚀1. Start Microservice with Python
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume LVM Service
      • LVM Microservice
        • 🚀1. Start Microservice with Docker
        • ✅ 2. Test
        • ♻️ 3. Clean
    • Prompt_registry Microservice
      • 🧾 Prompt Registry Microservice
        • 🛠️ Features
        • ⚙️ Implementation
      • Deploy prompt microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • 🧾 Prompt Registry Microservice with MongoDB
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
    • Rerankings Microservice
      • Deploy reranking microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Reranking Microservice
        • 🛠️ Features
        • ⚙️ Implementation
      • 🌟 Reranking Microservice with TEI
        • 📦 1. Start Microservice with Docker
        • 📦 2. Start Microservice with docker compose
        • 📦 3. Consume Reranking Service
        • ✨ Tips for Better Understanding:
      • 🌟 Reranking Microservice with VideoQnA
        • 📦 1. Start Microservice with Docker
        • 📦 2. Start Microservice with docker compose
        • 📦 3. Consume Reranking Service
        • ✨ Tips for Better Understanding:
    • Retrievers Microservice
      • Retriever Microservice
        • Retriever Microservice with Redis
        • Retriever Microservice with Milvus
        • Retriever Microservice with Qdrant
        • Retriever Microservice with PGVector
        • Retriever Microservice with VDMS
        • Retriever Microservice with ElasticSearch
        • Retriever Microservice with OpenSearch
        • Retriever Microservice with neo4j
        • Retriever Microservice with Pathway
      • Deploy retriever microservice on Kubernetes cluster
        • Deploy on Kubernetes with redis vector DB
        • Deploy on Kubernetes with milvus vector DB
      • Retriever Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
      • Retriever Microservice with Milvus
        • 🚀Start Microservice with Python
        • 🚀Start Microservice with Docker
        • 🚀3. Consume Retriever Service
      • Retriever Microservice with Neo4J
        • 🚀Start Microservice with Docker
        • Invoke Microservice
      • Retriever Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
      • Retriever Microservice with Pathway
        • 🚀Start Microservices
      • Retriever Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
      • Retriever Microservice with Qdrant
        • 1. 🚀Start Microservice with Python (Option 1)
        • 2. 🚀Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
      • Retriever Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
      • Retriever Microservice
        • Visual Data Management System (VDMS)
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
    • Text2image Microservice
      • Deploy text2image on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • Text-to-Image Microservice
      • 🚀1. Start Microservice with Python (Option 1)
        • 1.1 Install Requirements
        • 1.2 Start Text-to-Image Microservice
      • 🚀2. Start Microservice with Docker (Option 2)
        • 2.1 Build Images
        • 2.2 Start Text-to-Image Service
      • 3 Test Text-to-Image Service
    • Text2sql Microservice
      • 🛢 Text-to-SQL Microservice
        • 🛠️ Features
        • ⚙️ Implementation
    • Third_parties Microservice
      • Deploy mm-embedding on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • Multimodal Embeddings Microservice with BridgeTower
        • 🚀1. Start MMEI on Gaudi2 HPU
        • 🚀2. Start MMEI on Xeon CPU
        • 🚀3. Access the service
      • Multimodal Embeddings Microservice with CLIP
        • 🚀1. Start Microservice with Docker
        • 🚀2. Consume Embedding Service
      • Start Elasticsearch server
        • 1. Download Elasticsearch image
        • 2. Run Elasticsearch service
      • Deploy gpt-sovits on kubernetes cluster
        • Deploy on Xeon
      • Start Milvus server
        • 1. Configuration
        • 2. Run Milvus service
      • Deploy MongoDB on kubernetes cluster
        • Deploy on Xeon
      • Start Neo4J Server
        • 1. Download Neo4J image
        • 2. Configure the username, password and dbname
        • 3. Run Neo4J service
      • Deploy nginx on kubernetes cluster
        • Deploy on Xeon
      • Introduction
        • Get Started
      • Start Opensearch server
        • Prerequisites
        • Instructions
        • Troubleshooting
      • Start the Pathway Vector DB Server
        • Configuration
        • Building and running
        • Health check the vector store
      • Start PGVector server
        • 1. Download Pgvector image
        • 2. Configure the username, password and dbname
        • 3. Run Pgvector service
      • Start PGVector server
        • 1. Download Pgvector image
        • 2. Configure the username, password and dbname
        • 3. Run Pgvector service
      • Deploy RedisDB on kubernetes cluster
        • Deploy on Xeon
      • Start Redis server
        • 1. Download Redis image
        • 2. Run Redis service
      • Deploy speecht5 on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • Deploy TEI on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • Deploy TEIRERANK on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • TGI LLM Microservice
        • Start TGI with docker compose
      • Deploy TGI on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • Start VDMS server
        • 1. Download VDMS image
        • 2. Run VDMS service
      • vLLM Endpoint Service
        • 🚀1. Set up Environment Variables
        • 🚀2. Set up vLLM Service
        • 🚀3. Set up LLM microservice
      • Deploy vllm on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • Deploy whisper on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
    • Tts Microservice
      • Deploy tts microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • TTS Microservice
        • 1.2 Start SpeechT5 Service/Test
        • 1.3 Start TTS Service/Test
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Start Microservice with Docker Compose (Option 3)
      • GPT-SoVITS Microservice
        • Build the Image
        • Start the Service
        • Test
    • Web_retrievers Microservice
      • Deploy web-retriever microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Web Retriever Microservice
        • 🚀1. Start Microservice with Docker (Option 1)
        • 🚀2. Start Microservice with Docker Compose (Option 2)
        • 🚀3. Consume Web Retriever Service
  • Deploying GenAI
    • GenAIInfra
      • Overview
      • Prerequisite
        • Setup Kubernetes cluster
        • (Optional) To run GenAIInfra on Intel Gaudi product
      • Usages
        • Use GenAI Microservices Connector (GMC) to deploy and adjust GenAIExamples
        • Use helm charts to deploy
        • Use terraform to deploy on cloud service providers
      • Additional Content
    • Development
      • Prerequisites
      • Testing
      • pre-commit testing
    • Legal Information
      • License
      • Citation
    • Release Branches
      • 1. Create release candidate branch
      • 2. Create images with release tag
      • 3. Test helm charts
      • 4. Test GMC
      • 5. Publish images
    • Installation Guides
      • GenAI-microservices-connector(GMC) Installation
        • GenAI-microservices-connector(GMC)
        • Install GMC
        • Use GMC to compose a chatQnA Pipeline
      • Kubernetes Installation Options
      • Kubernetes Installation using AWS EKS Cluster
        • Prerequisites
        • Create AWS EKS Cluster in AWS Console
        • Uploading images to an AWS Private Registry
      • Kubernetes installation demo using kubeadm
        • Node configuration
        • Step 0. Clean up the environment
        • Step 1. Install relevant components
        • Step 2. Create the k8s cluster
        • Step 3 (optional) Reset Kubernetes cluster
        • NOTES
      • Kubernetes installation using Kubespray
        • Node preparation
        • Prerequisites
        • Step 1. Set up Kubespray and Ansible
        • Step 2. Build your own inventory
        • Step 3. Define Kubernetes configuration
        • Step 4. Deploy Kubernetes
        • Step 5. Create kubectl configuration
        • Quick reference
    • Cloud Service Provider
      • OPEA applications Azure AKS deployment guide
        • Prerequisites
        • Setup
        • AKS cluster
        • Cosmos DB
        • Persistent Volume Claim
        • OPEA Applications
    • Authentication and Authorization
      • Authentication and authorization
        • Istio based implementation for cloud native environments
        • APISIX based implementation for cloud native environments
      • Authentication and Authorization with APISIX and OIDC based Identity provider (Keycloak)
        • Start ChatQnA service
        • Starting and Configuring Keycloak
        • Update values
        • Install APISIX
        • Usage
        • Uninstall
      • Leveraging Istio to compose an OPEA Pipeline with authentication and authorization enabled
        • Prerequisite
        • Perform authentication and authorization via Bearer JWT tokens and curl
        • Perform authentication and authorization via oauth2-proxy and OIDC provider and UI
        • Uninstall Istio
    • Helm Charts
      • Helm charts for deploying GenAI Components and Examples
        • Table of Contents
        • Helm Charts
        • Deploy with Helm charts
        • Helm Charts Options
        • Using HPA (autoscaling)
        • Using Persistent Volume
        • Using Private Docker Hub
        • Generate manifests from Helm Charts
      • CI guidelines for helm charts
        • Table of Contents
        • Infra Setup
        • Add new test case
      • HorizontalPodAutoscaler (HPA) support
        • Table of Contents
        • Introduction
        • Pre-conditions
        • Gotchas
        • Enable HPA
        • Verify
      • Monitoring support
        • Table of Contents
        • Introduction
        • Pre-conditions
        • Install
        • Verify
      • agent
        • Deploy
        • Verify
        • Options
      • asr
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • chathistory-usvc
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • data-prep
        • Installing the chart
        • Verify
        • Values
        • Milvus support
      • embedding-usvc
        • Installing the chart
        • Verify
        • Values
      • gpt-sovits
        • Install the chart
        • Verify
        • Values
      • guardrails-usvc
        • Installing the chart
        • Verify
        • Values
      • llm-uservice
        • Installing the chart
        • Verify
        • Values
      • OPEA lvm-serve microservice
        • Installing the Chart
        • Verify
        • Values
      • lvm-uservice
        • Installing the chart
        • Verify
        • Values
      • OPEA mm-embedding microservice
        • Installing the Chart
        • Verify
        • Values
      • mongodb
        • Install the Chart
        • Verify
        • Values
      • OPEA nginx microservice
        • Using nginx chart
      • prompt-usvc
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • redis-vector-db
        • Install the Chart
        • Verify
        • Values
      • reranking-usvc
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • retriever-usvc
        • Installing the chart
        • Verify
        • Values
        • Milvus support
      • speecht5
        • Installing the Chart
        • Verify
        • Values
      • tei
        • Installing the Chart
        • Verify
        • Values
      • teirerank
        • Installing the Chart
        • Verify
        • Values
      • OPEA text2image microservice
        • Installing the Chart
        • Verify
        • Values
      • tgi
        • Installing the Chart
        • Verify
        • Values
      • tts
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • vllm
        • Installing the Chart
        • Verify
        • Values
      • web-retriever
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • whisper
        • Installing the Chart
        • Verify
        • Values
      • AgentQnA
        • Deploy
        • Verify
      • AudioQnA
        • Installing the Chart
        • Verify
        • Values
      • ChatQnA
        • Installing the Chart
        • Verify
        • Values
        • Troubleshooting
      • ChatQnA Troubleshooting
        • a function to get the endpoint of service
        • define the namespace of service
        • Update a file to database
        • get the embedding of input
        • get the retriever docs
        • reranking the docs
        • TGI Q and A
        • REF
      • CodeGen
        • Installing the Chart
        • Verify
        • Values
      • CodeTrans
        • Installing the Chart
        • Verify
        • Values
      • DocSum
        • Installing the Chart
        • Verify
        • Values
      • FaqGen
        • Verify
        • Values
      • SearchQnA
        • Installing the Chart
        • Verify
        • Values
      • txt2img
        • Installing the Chart
        • Verify
        • Values
      • VisualQnA
        • Verify
        • Values
    • Kubernetes Addons
      • Deploy Kubernetes add-ons for OPEA
      • How-To Setup Observability for OPEA Workload in Kubernetes
        • Prepare
        • 1. Setup Prometheus & Grafana
        • 2. Metrics for Gaudi Hardware (v1.16.2)
        • 3. Metrics for OPEA applications
        • 4. Metrics for PCM (Intel® Performance Counter Monitor)
        • More dashboards
      • OPEA with Istio
        • Introduction
        • Deployment
        • Create Istio gateway with TLS and virtual service
        • Enforce mTLS between OPEA pods
        • Cleanup
      • memory bandwidth exporter
        • Setup
        • More flags about memory bandwidth exporter
    • Microservices Connector
      • genai-microservices-connector(GMC)
        • Description
        • Architecture
        • Personas
        • Getting Started
      • Troubleshooting GMC Custom Resource(CR)
      • Usage guide for genai-microservices-connector(GMC)
        • Use GMC to compose a chatQnA Pipeline
        • Use GMC to adjust the chatQnA Pipeline
        • Use GMC to delete the chatQnA Pipeline
        • Use GMC and Istio to compose an OPEA Pipeline with authentication and authorization enabled
      • ChatQnA Use Cases in Kubernetes Cluster via GMC
        • Using prebuilt images
        • Deploy ChatQnA pipeline
      • Helm chart for genai-microservices-connector(GMC)
        • Installing the GMC Helm Chart
        • Check the installation result
        • Next step
        • Uninstall
    • Pipeline Proxy
      • OPEA Pipeline Proxy
        • Features
        • Build
        • Deployment
        • Development
      • Guardrails
        • Architecture
        • Deployment
    • Scripts
      • Scripts and tools
      • NVIDIA GPU Quick-Start Guide
        • Prerequisite
        • Usages
        • FAQ and Troubleshooting
      • Deploy Autoscaling Ray Cluster with KubeRay in Kubernetes Cluster
        • Install KubeRay
        • Start Ray Cluster with Autoscaling
        • Delete Ray Cluster with Autoscaling
        • Uninstall KubeRay
  • Evaluating GenAI
    • GenAIEval
      • Installation
      • Evaluation
        • lm-evaluation-harness
        • bigcode-evaluation-harness
        • Kubernetes platform optimization
      • Benchmark
        • Features
        • How to use
        • Grafana Dashboards
      • Additional Content
    • Legal Information
      • License
      • Citation
    • Kubernetes Platform Optimization with Resource Management
      • Introduction
      • NRI Plugins
      • Install
      • Validate policy status
      • Configure
      • Validate CPU affinity and hardware alignment in containers
      • Remove a policy
      • NRI topology-aware resource policy
    • OPEA Benchmark Tool
      • Features
      • Table of Contents
      • Installation
        • Prerequisites
      • Usage
      • Configuration
        • Test Suite Configuration
        • Test Cases
    • Auto-Tuning for ChatQnA: Optimizing Resource Allocation in Kubernetes
      • Key Features
    • Usage
      • Configuration Files
        • Hardrware_info.json
        • chatqna_neuralchat_rerank_latest.yaml
        • Tuning Config Parameters
      • Output
    • Auto-Tuning for ChatQnA: Optimizing Accuracy by Tuning Model Related Parameters
      • Prepare Dataset
      • Run the Tuning script
    • Setup Prometheus and Grafana to visualize microservice metrics
      • 1. Setup Prometheus
        • 1.1 Node Metrics (optional)
        • 1.2 Intel® Gaudi® Metrics (optional)
      • 2. Setup Grafana
      • 3. Import Grafana Dashboard
    • StressCli
      • stresscli.py
        • Prerequirements
        • Installation
        • Usage
    • How to benchmark pubmed datasets by send query randomly
      • 1. prepare the pubmed datasets
        • 1.1 get pubmed data
        • 1.2 use script to extract data
        • 1.3 prepare 4 dataset files
        • 1.4 prepare the query list
      • 2. How to use pubmed qlist
    • locust scripts for OPEA ChatQnA
      • Configuration file
      • Basic Usage
    • HELMET: How to Evaluate Long-context Language Models Effectively and Thoroughly HELMET
      • Quick Links
      • Setup
      • Data
      • Running evaluation
        • Model-based evaluation
      • Adding new models
      • Adding new tasks
      • Others
      • Contacts
      • Citation
    • CRAG Benchmark for Agent QnA systems
      • Overview
      • Getting started
      • CRAG dataset
      • Launch agent QnA system
      • Run CRAG benchmark
      • Use LLM-as-judge to grade the answers
        • Validation of LLM-as-judge
      • Benchmark results for OPEA RAG Agents
        • Comparison with GPT-4o-mini
    • AutoRAG to evaluate the RAG system performance
      • Service preparation
      • RAG evaluation
      • Notes
    • 🚀 QuickStart
      • Installation
      • Launch Service of LLM-as-a-Judge
      • Writing your first test case
      • Acknowledgements
    • 🚀 QuickStart
      • Installation
      • Launch a LLM Service
        • Example 1: TGI
        • Example 2: OPEA LLM
      • Predict
      • Evaluate
    • Evaluation Methodology
      • Introduction
      • Prerequisite
        • Environment
      • MultiHop (English dataset)
        • Launch Service of RAG System
        • Launch Service of LLM-as-a-Judge
        • Prepare Dataset
        • Evaluation
      • CRUD (Chinese dataset)
        • Prepare Dataset
        • Launch Service of RAG System
        • Evaluation
      • Acknowledgements
    • Metric Card for BLEU
      • Metric Description
      • Intended Uses
      • How to Use
        • Inputs
        • Output Values
        • Examples
      • Limitations and Bias
      • Citation
      • Further References
    • RAGAAF (RAG assessment - Annotation Free)
      • Key features
      • Run RAGAAF
        • 1. Data
        • 2. Launch endpoint on Gaudi
        • 3. Model
        • 4. Metrics
        • 5. Evaluation
      • Customizations
    • OPEA adaption of ragas (LLM-as-a-judge evaluation of Retrieval Augmented Generation)
      • User data
      • Launch HuggingFace endpoint on Intel’s Gaudi machines
      • Run OPEA ragas pipeline using your desired list of metrics
      • Troubleshooting
  • Developer Guides
    • Coding Guides
      • OPEA API Service Spec (v1.0)
        • OPEA Mega Service API
        • OPEA Micro Service API
    • Documentation Guides
      • Documentation Guidelines
        • Markdown vs. RestructuredText
        • Documentation Organization
        • Headings
        • Content Highlighting
        • Lists
        • Multi-Column Lists
        • Tables
        • File Names and Commands
        • Branch-Specific File Links
        • Internal Cross-Reference Linking
        • Non-ASCII Characters
        • Include Content from Other Files
        • Code and Command Examples
        • Images
        • Tabs, Spaces, and Indenting
        • Background Colors
        • Drawings
        • Alternative Tabbed Content
        • Instruction Steps
        • First Instruction Step
        • Second Instruction Step
        • Documentation Generation
      • Drawings Using Graphviz
        • Simple Directed Graph
        • Adding Edge Labels
        • Tables
        • Finite-State Machine
      • OPEA Documentation Generation
        • Documentation Overview
        • Set Up the Documentation Working Folders
        • Install the Documentation Tools
        • Documentation Presentation Theme
        • Run the Documentation Processors
        • Doc Build Troubleshooting
        • Publish Content
        • Document Versioning
        • Filter Expected Warnings
  • OPEA Community
    • Community Support
    • Resources
    • Contributing Guides
      • Contribution Guidelines
        • All The Ways To Contribute
        • Support
        • Contributor Covenant Code of Conduct
      • OPEA Project Code Owners
        • GenAIComps Repository Code Owners
        • GenAIEval Repository Code Owners
        • GenAIExamples Repository Code Owners
        • GenAIInfra Repository Code Owners
        • docs Repository Code Owners
        • Continuous Integration (CICD) owners
      • Reporting a Vulnerability
        • Script Usage Notice
    • Roadmaps
      • OPEA 2024 - 2025 Roadmap
        • May 2024
        • June 2024
        • July 2024
        • Aug 2024
        • Sep 2024
        • Q4 2024
        • Q1 2025
      • OPEA CI/CD Roadmap
        • Milestone 1 (May, Done)
        • Milestone 2 (June)
        • Milestone 3 (July)
        • Milestone 4 (Aug)
    • Project Governance
      • Technical Charter (the “Charter”) for OPEA a Series of LF Projects, LLC
        • 1. Mission and Scope of the Project
        • 2. Technical Steering Committee
        • 3. TSC Voting
        • 4. Compliance with Policies
        • 5. Community Assets
        • 6. General Rules and Operations.
        • 7. Intellectual Property Policy
        • 8. Amendments
      • Technical Steering Committee (TSC)
        • Technical Steering Committee Members
      • Contributor Covenant Code of Conduct
        • Our Pledge
        • Our Standards
        • Enforcement Responsibilities
        • Scope
        • Enforcement
        • Enforcement Guidelines
        • Attribution
      • Reporting a Vulnerability
        • Script Usage Notice
    • RFC Proposals
      • Request for Comments (RFCs)
        • 24-05-16 GenAIExamples-001 Using MicroService to Implement ChatQnA
        • 24-05-16 OPEA-001 Overall Design
        • 24-05-24 OPEA-001 Code Structure
        • 24-06-21-OPEA-001-DocSum_Video_Audio
        • 24-06-21-OPEA-001-Guardrails-Gateway
        • 24-07-11-OPEA-Agent
        • 24-08-02-OPEA-AIAvatarChatbot
        • 24-08-07 OPEA-001 OPEA GenAIStudio
        • 24-08-20-OPEA-001-AI Gateway API
        • 24-08-21-GenAIExample-002-Edge Craft RAG
        • 24-10-02-GenAIExamples-001-Image_and_Audio_Support_in_MultimodalQnA
        • RFC Template
  • Release Notes
    • OPEA Release Guide
      • Release Cadence
      • General Overview
      • Feature Freeze
      • Code/Doc Freeze, and Creating the RC Branch
        • Preparing Creating RC Branch
        • Creating RC Branch
      • Cherry Pick Critical Code/Doc Fix
        • How to do Cherry Picking
      • Creating Tag from RC Branch
      • Deliver Docker Images, Helm Charts, and PyPi Binaries
    • OPEA Release Notes v1.2
      • What’s New in OPEA v1.2
        • Highlights
      • Deprecations and Behavior Changes
        • GenAIComps
        • GenAIExamples
        • GenAIEval
        • GenAIInfra
        • Docker Images
      • Notable Changes
      • Full Changelogs
      • Contributors
        • Contributing Organizations
        • Individual Contributors
    • OPEA Release Notes v1.1
      • What’s New in OPEA v1.1
        • Highlights
        • Notable Changes
        • Full Changelogs
      • Contributors
        • Contributing Organizations
        • Individual Contributors
    • OPEA Release Notes v1.0
      • What’s New in OPEA v1.0
      • Details
    • OPEA Release Notes v0.9
      • What’s New in OPEA v0.9
      • Details
    • OPEA Release Notes v0.8
      • What’s New in OPEA v0.8
      • Details
      • Thanks to these contributors
    • OPEA Release Notes v0.7
      • OPEA Highlights
      • GenAIExamples
      • GenAIComps
      • GenAIEvals
      • GenAIInfra
    • OPEA Release Notes v0.6
      • OPEA Highlight
      • GenAIExamples
      • GenAIComps
      • GenAIEvals
      • GenAIInfra
  • Contributing to OPEA
  • Additional Content
  • OPEA Frequently Asked Questions
    • What is OPEA’s mission?
    • What is OPEA?
    • What problems are faced by GenAI deployments within the enterprise?
    • Why now?
    • How does it compare to other options for deploying Gen AI solutions within the enterprise?
    • Will OPEA reference implementations work with proprietary components?
    • What does OPEA acronym stand for?
    • How do I pronounce OPEA?
    • What initial companies and open-source projects joined OPEA?
    • What is Intel contributing?
    • When you say Technical Conceptual Framework, what components are included?
    • What are the different ways partners can contribute to OPEA?
    • Where can partners see the latest draft of the Conceptual Framework spec?
    • Is there a cost for joining?
    • Do I need to be a Linux Foundation member to join?
    • Where can I report a bug or vulnerability?
OPEA™
  • 1.2 »
  • Deploying GenAI »
  • Legal Information
  • View page source

Legal Information¶

  1. License

  2. Citation

License¶

Generative AI Examples is licensed under Apache License Version 2.0. This software includes components that have separate copyright notices and licensing terms. Your use of the source code for these components is subject to the terms and conditions of the following licenses.

See the accompanying license file for full license text and copyright notices.

Citation¶

If you use Generative AI Infrastructure in your research, use the following BibTeX entry.

@misc{Generative AI Infrastructure,
  author =       {Jianfeng Ding},
  title =        {Generative AI Infrastructure},
  howpublished = {\url{https://github.com/opea-project/GenAIInfra}},
  year =         {2024}
}
Previous Next

© Copyright 2024-2025 OPEA™, a Series of LF Projects, LLC. Published on Jan 27, 2025.