OPEA™ Logo
latest
OPEA Project v: latest
Document Versions
latest
1.0
1.1
1.2
1.3
OPEA Project links
Project Home
Wiki
  • Documentation Home
  • OPEA Overview
    • OPEA Project Architecture
      • Microservices: Flexible and Scalable Architecture
      • Megaservices: A Comprehensive Solution
      • Gateways: Customized Access to Mega- and Microservices
    • Next Step
      • Open Platform for Enterprise AI (OPEA) Framework Draft Proposal
        • 1. Summary
        • 2. Introduction
        • 3. Framework Components, Architecture and Flow
        • 4. Assessing GenAI components and flows
        • 5. Grading Structure
        • 6. Reference flows
        • Appendix A – Draft OPEA Specifications
  • Getting Started with OPEA
    • Understanding OPEA’s Core Components
    • Prerequisites
    • Create and Configure a Virtual Server
    • Deploy the ChatQnA Solution
      • Interact with ChatQnA
    • What’s Next
      • Get Involved
  • OPEA Tutorials
    • AgentQnA
      • Overview
      • Purpose
      • How It Works
      • Deployment
        • Single Node
    • AudioQnA
      • Overview
      • Purpose
      • Key Implementation Details
      • How It Works
      • Deployment
        • Single Node
    • ChatQnA
      • Overview
      • Purpose
      • Key Implementation Details
      • How It Works
        • Customize with new VectorDB
        • Expected Output
        • Validation Matrix and Prerequisites
      • Architecture
        • Microservice Outline and Diagram
      • Deployment
      • Single Node
        • Xeon Scalable Processor
        • Gaudi AI Accelerator
        • Nvidia GPU
        • AI PC
      • Kubernetes
        • Getting Started
        • Kubernetes Deployment with Helm on Xeon
      • Cloud Native
      • Troubleshooting
      • Monitoring
        • Set Up the Prometheus Server
        • Set Up the Grafana Dashboard
        • Summary and Next Steps
    • CodeGen
      • Overview
      • Purpose
      • How It Works
      • Deployment
        • Intel® Xeon® Scalable processor
        • Gaudi AI Accelerator
    • Code Translation
      • Overview
      • Purpose
      • How It Works
      • Deployment
        • Single Node
    • DocSum
      • Overview
      • Purpose
      • How It Works
      • Deployment
        • Intel® Xeon® Scalable processor
        • Gaudi AI Accelerator
    • DocIndexRetriever
      • Overview
      • Purpose
      • Key Implementation Details
      • How It Works
      • Deployment
        • Single Node
    • VideoQnA
      • Overview
      • Purpose
      • How It Works
      • Deployment
    • Enterprise Inference Guide
      • Overview
      • How It Works
      • Setting Up a Remote Server or Cluster
      • Using Remote Endpoints on OPEA GenAIExamples
        • 1. Endpoints with Megaservices
        • 2. Endpoints with Microservices
      • Next Steps
    • OpenTelemetry on OPEA Guide
      • Overview
      • How It Works
      • How to Monitor
        • 1. Prometheus
        • 2. Grafana
        • 3. Jaeger
      • Code Instrumentations for OPEA Tracing
      • OpenTelemetry on GenAIExamples
        • ChatQnA
        • AgentQnA
  • GenAI Examples
    • Generative AI Examples
      • Introduction
      • Architecture
      • Use Cases
      • Documentation
      • Getting Started
        • Deployment Guide
      • Supported Examples
      • Validated Configurations
      • Contributing to OPEA
      • Additional Content
    • Examples
      • AgentQnA Application
        • Agents for Question and Answering Application
        • Agents for Question and Answering Application
        • Build Service Docker Image
        • Deploy AgentQnA on AMD GPU (ROCm)
        • Deploying AgentQnA on Intel® Xeon® Processors
        • Deploying AgentQnA on Intel® Gaudi® Processors
        • Deploy AgentQnA on Kubernetes cluster
        • Retrieval tool for agent
      • ArbPostHearingAssistant Application
        • Arbitration Post-Hearing Assistant
        • Table of Contents
        • Deploy Arbitration Post-Hearing Assistant Application on AMD EPYC™ Processors with Docker Compose
        • Example Arbitration Post-Hearing Assistant deployments on AMD GPU (ROCm)
        • Example Arbitration Post-Hearing Assistant deployments on Intel Xeon Processor
        • Example Arbitration Post-Hearing Assistant deployments on Intel® Gaudi® Platform
        • DocSum E2E test scripts
        • Arbitration Post-Hearing Assistant
      • AudioQnA Application
        • AudioQnA Application
        • AudioQnA Docker Image Build
        • AudioQnA Accuracy
        • AudioQnA Benchmarking
        • Deploying AudioQnA on AMD EPYC™ Processors
        • Deploying AudioQnA on AMD ROCm GPU
        • Deploying AudioQnA on Intel® Xeon® Processors
        • Deploy AudioQnA application
        • Deploying AudioQnA on Intel® Gaudi® Processors
        • Deploy AudioQnA in Kubernetes Cluster on Xeon and Gaudi
        • Deploy AudioQnA on Kubernetes cluster
        • AudioQnA E2E test scripts
        • AudioQnA
      • AvatarChatbot Application
        • AvatarChatbot Application
        • Build Mega Service of AvatarChatbot on AMD GPU
        • Example AvatarChatbot Deployment on Intel® Xeon® Platform
        • Example AvatarChatbot Deployment on Intel® Gaudi® Platform
        • AvatarChatbot E2E test scripts
      • BrowserUseAgent Application
        • Browser-use Agent Application
        • Example BrowserUseAgent deployments on an Intel® Gaudi® Platform
        • Setup Scripts for Webarena
      • ChatQnA Application
        • ChatQnA Application
        • ChatQnA Docker Image Build
        • ChatQnA Benchmark Results
        • ChatQnA Accuracy
        • FaqGen Accuracy
        • FaqGen Benchmarking
        • Deploying ChatQnA on AMD EPYC™ Processors
        • Deploying FAQ Generation on AMD EPYC™ Processors
        • Deploying ChatQnA with Pinecone on AMD EPYC™ Processors
        • Deploying ChatQnA with Qdrant on AMD EPYC™ Processors
        • Deploying ChatQnA on AMD ROCm GPU
        • Build Mega Service of ChatQnA on AIPC
        • Deploying ChatQnA on Intel® Xeon® Processors
        • Deploying FAQ Generation on Intel® Xeon® Processors
        • Deploying ChatQnA with MariaDB Vector on Intel® Xeon® Processors
        • Deploying ChatQnA with Pinecone on Intel® Xeon® Processors
        • Deploying ChatQnA with Qdrant on Intel® Xeon® Processors
        • Example ChatQnA deployments on an Intel® Gaudi® Platform
        • How to Check and Validate Micro Service in the GenAI Example
        • Build MegaService of ChatQnA on NVIDIA GPU
        • Deploy ChatQnA in Kubernetes Cluster on Xeon and Gaudi
        • Deploy ChatQnA on Kubernetes cluster
        • ChatQnA E2E test scripts
        • ChatQnA Conversational UI
        • ChatQnA Customized UI
      • CodeGen Application
        • Code Generation Example (CodeGen)
        • CodeGen Accuracy Benchmark
        • CodeGen Performance Benchmark
        • Deploy CodeGen Application on AMD EPYC™ Processors with Docker Compose
        • Deploy CodeGen Application on AMD GPU (ROCm) with Docker Compose
        • To deploy the CodeGen services, execute the docker compose up command with the appropriate arguments. For a TGI deployment, execute:
        • Deploy CodeGen Application on Intel Xeon CPU with Docker Compose
        • Deploy CodeGen Application on Intel Gaudi HPU with Docker Compose
        • Deploy CodeGen using Kubernetes Microservices Connector (GMC)
        • Deploy CodeGen on Kubernetes using Helm
        • CodeGen E2E test scripts
        • Document Summary
        • Code Gen
        • Code Gen
      • CodeTrans Application
        • Code Translation Application
        • CodeTrans Docker Image Build
        • CodeTrans Benchmarking
        • Deploy CodeTrans Application on AMD EPYC™ Processors with Docker Compose
        • Deploying CodeTrans on AMD ROCm GPU
        • Deploying CodeTrans on Intel® Xeon® Processors
        • Deploying CodeTrans on Intel® Gaudi® Processors
        • Deploy CodeTrans in a Kubernetes Cluster
        • Deploy CodeTrans on Kubernetes cluster
        • CodeTrans E2E test scripts
        • Code Translation
      • DBQnA Application
        • DBQnA Application
        • Example DBQnA Deployment on AMD GPU (ROCm)
        • Example DBQnA Deployment on Intel® Xeon® Platform
        • DBQnA E2E test scripts
        • DBQnA React Application
      • DeepResearchAgent Application
        • Deep Research Agent Application
        • Deep Research Agent Benchmarks
      • DocIndexRetriever Application
        • DocRetriever Application
        • DocRetriever Application with Docker
        • DocRetriever Application with Docker
        • DocIndexRetriever E2E test scripts
      • DocSum Application
        • Document Summarization Application
        • Table of Contents
        • Deploy DocSum Application on AMD EPYC™ Processors with Docker Compose
        • Example DocSum deployments on AMD GPU (ROCm)
        • Example DocSum deployments on Intel Xeon Processor
        • Example DocSum deployments on Intel® Gaudi® Platform
        • Deploy DocSum in Kubernetes Cluster
        • Deploy DocSum on Kubernetes cluster
        • DocSum E2E test scripts
        • Document Summary
        • Doc Summary React
        • Doc Summary
      • EdgeCraftRAG Application
        • Edge Craft Retrieval-Augmented Generation Application
        • Example Edge Craft Retrieval-Augmented Generation Deployment on Intel® Arc® Platform
        • Edge Craft Retrieval-Augmented Generation API guide
        • Edge Craft Retrieval-Augmented Generation Advanced Setup
        • Explore Edge Craft RAG
        • Query Search
        • 快速启动指南
        • EdgeCraftRAG E2E test scripts
        • EdgeCraftRAG one-click deployment script
      • FinanceAgent Application
        • Finance Agent Example
        • Deploy Finance Agent on Intel® Xeon® Scalable processors with Docker Compose
        • Deploy Finance Agent on Intel® Gaudi® AI Accelerator with Docker Compose
      • GraphRAG Application
        • GraphRAG Application
        • About GraphRAG LLMs
        • GraphRAG Application
        • Example GraphRAG Deployment on Intel® Gaudi® Platform
        • GraphRAG Example Datasets
        • GraphRAG E2E test scripts
        • ChatQnA Conversational UI
        • ChatQnA Customized UI
      • HybridRAG Application
        • HybridRAG Application
        • Notice for FFmpeg:
        • Example HybridRAG deployments on an Intel® Gaudi® Platform
        • ChatQnA Customized UI
      • InstructionTuning Application
        • Instruction Tuning
        • Table of contents
        • Deploy Instruction Tuning Service on Intel® Xeon® Processors
        • Table of contents
        • Deploy Instruction Tuning Service on Intel® Gaudi® Platform
        • Table of contents
        • Translation E2E test scripts
      • MultimodalQnA Application
        • MultimodalQnA Application
        • MultimodalQnA Docker Image Build
        • Build Mega Service of MultimodalQnA on AMD EPYC™ Processors
        • Deploying MultimodalQnA on AMD GPU (ROCm)
        • Table of Contents
        • Deploying MultimodalQnA on Intel® Xeon® Processors
        • Deploying MultimodalQnA on Intel® Gaudi® Processors
        • MultimodalQnA E2E test scripts
      • ProductivitySuite Application
        • Productivity Suite Application
        • Build Mega Service of Productivity Suite on AMD EPYC™ Processors
        • 🔐 Keycloak Configuration Setup
        • Example Productivity Suite Deployment on Intel® Xeon® Platform
        • 🔐 Keycloak Configuration Setup
        • ProductivitySuite E2E test scripts
        • Productivity Suite React UI
      • RerankFinetuning Application
        • Rerank Model Finetuning
        • Deploy Rerank Model Finetuning Service on Xeon
        • Deploy Rerank Model Finetuning Service on Gaudi
      • SearchQnA Application
        • SearchQnA Application
        • SearchQnA Docker Image Build
        • Deploying SearchQnA on AMD EPYC™ Processors
        • Deploying SearchQnA on AMD ROCm Platform
        • Deploying SearchQnA on Intel® Xeon® Processors
        • Deploying searchqna on Intel® Gaudi® Processors
        • Deploy SearchQnA in a Kubernetes Cluster
        • Deploy SearchQnA on Kubernetes cluster
        • SearchQnA E2E test scripts
        • Neural Chat
      • Text2Image Application
        • Text-to-Image Microservice
        • Deploy Text-to-Image Service on AMD EPYC™ Processors
        • Deploy Text-to-Image Service on Xeon
        • Deploy Text-to-Image Service on Gaudi
        • Deploy txt2img on Kubernetes cluster
        • Text2Image E2E test scripts
        • Text2Image Customized UI
      • Translation Application
        • Translation Application
        • Example Translation Deployment on AMD EPYC™ Platform
        • Example Translation Deployment on AMD GPU (ROCm)
        • Example Translation Deployment on Intel® Xeon® Platform
        • Example Translation Deployment on Intel® Gaudi® Platform
        • Deploy Translation in a Kubernetes Cluster
        • Translation E2E test scripts
        • Language Translation
      • VideoQnA Application
        • VideoQnA Application
        • Build Mega Service of VideoQnA on Xeon
      • VisualQnA Application
        • Visual Question and Answering
        • VisualQnA Docker Image Build
        • VisualQnA Benchmarking
        • Build Mega Service of VisualQnA on AMD EPYC™ Processors
        • Deploy VisualQnA on AMD GPU (ROCm)
        • Deploying VisualQnA on Intel® Xeon® Processors
        • Deploying VisualQnA on Intel® Gaudi® Processors
        • Deploy VisualQnA in a Kubernetes Cluster
        • Deploy VisualQnA on Kubernetes cluster
        • VisualQnA E2E test scripts
      • WorkflowExecAgent Application
        • Workflow Executor Agent
        • Deploying WorkflowExecAgent on Intel® Xeon® Processors
        • Validate Workflow Agent Microservice
      • one_click_deploy Application
        • One-Click Deployment for GenAI Examples
    • Legal Information
      • License
      • Citation
    • Deploy and Benchmark
      • Purpose
        • Support Example List
      • Table of Contents
      • Prerequisites
      • Data Preparation
      • Running Deploy and Benchmark Tests
        • Running the Tests
        • Test Modes
        • Troubleshooting
      • ChatQnA Release Data
    • Docker Images
      • Example images
      • Microservice images
    • Supported Examples
      • ChatQnA
        • CodeGen
        • CodeTrans
        • DocSum
        • Language Translation
        • SearchQnA
        • VisualQnA
        • VideoQnA
        • RerankFinetuning
        • InstructionTuning
        • DocIndexRetriever
        • AgentQnA
        • AudioQnA
        • MultimodalQnA
        • ProductivitySuite
        • DBQnA
        • Text2Image
        • AvatarChatbot
    • Validated Configurations
      • v1.3 Release Test Config Overview
      • v1.4 Release Test Config Overview
  • GenAI Microservices
    • Generative AI Components (GenAIComps)
      • GenAIComps
        • Installation
      • MicroService
      • MegaService
      • Contributing to OPEA
        • uv pip compile usage
      • Additional Content
    • Legal Information
      • License
      • Citation
    • Agent Microservice
      • Deploy Agent microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Agent Microservice
        • Table of Contents
        • Quick Start Deployment
        • Supported agent types
        • Docker Compose Files
        • Other Supported Features
        • Customizations
      • Plan Execute Agent Microservice
      • RAG Agent Microservice
      • SQL Agent Microservice
        • Table of Contents
        • Overview of sql_agent_llama
        • Overview of sql_agent
        • Limitations
    • Animation Microservice
      • Deploying Avatar Animation Service
        • 🚀 1. Quick Start with Docker Compose
        • 🚀 2. Manual Step-by-Step Deployment (Advanced)
      • Avatar Animation Microservice
        • Table of contents
        • Architecture
        • Deployment Options
        • Validated Configurations
    • Arb_post_hearing_assistant Microservice
      • Arbitrator Post-Hearing Assistant
        • Features
        • Benefits
        • Table of Contents
        • Start Microservice
    • Asr Microservice
      • Deploy ASR microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • ASR Microservice
        • Table of contents
        • Architecture
        • Deployment Options
        • Validated Configurations
      • Deploying ASR Service
        • Table of contents
        • 🚀 1. Quick Start with Docker Compose
        • 🚀 2. Manual Step-by-Step Deployment (Advanced)
        • 🚀 3. Start Microservice with Python
      • Deploying ASR Service
        • Table of contents
        • 🚀 1. Quick Start with Docker Compose
        • 🚀 2. Manual Step-by-Step Deployment (Advanced)
        • 🚀 3. Start Microservice with Python
    • Chathistory Microservice
      • 📝 Chat History Microservice
        • 🛠️ Features
        • ⚙️ Deployment Options
      • 📝 Chat History Microservice with MongoDB
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
        • ✅ Invoke Microservice
      • 📝 Chat History Microservice with ArangoDB
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
        • ✅ Invoke Microservice
      • 📝 Chat History Microservice with Redis
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
        • ✅ Invoke Microservice
      • Deploy chathistory microservice on Kubernetes cluster
        • Deploy on Kubernetes
    • Cores Microservice
      • OPEA MCP Tool
        • OPEA MCP Tool Overview
        • API Usage
        • Examples
      • Contributing to Storage
        • Storage
        • Adding a New Storage Backend
        • Example
      • Telemetry for OPEA
        • Metrics
        • Statistics
        • Tracing
        • Visualization
    • Dataprep Microservice
      • Dataprep Microservice
        • Table of contents
        • Install Requirements
        • Summarizing Image Data with LVM
        • Dataprep Microservice on Various Databases
        • Running in the air gapped environment
      • Deploy dataprep microservice on Kubernetes cluster
        • Table of contents
        • Deploy on Kubernetes with redis VectorDB
        • Deploy on Kubernetes with milvus VectorDB
      • Dataprep Microservice with ArangoDB
        • Table of contents
        • 🚀Start Microservice with Docker
        • 🚀Consume Dataprep Service
      • Dataprep Microservice with Elasticsearch
        • Table of contents
        • 🚀1. Start Microservice with Docker
        • 🚀2. Consume Microservice
      • Dataprep microservice for financial domain data
        • Table of contents
        • 1. Overview
        • 2. Deploy with docker
        • 3. Consume Microservice
      • Dataprep Microservice with MariaDB Vector
        • Table of contents
        • 🚀1. Start Microservice with Docker
        • 🚀2. Consume Microservice
      • Dataprep Microservice with Milvus
        • Table of contents
        • 🚀1. Start Microservice with Docker
        • 🚀2. Consume Microservice
        • 🚀3. Troubleshooting
        • Running in the air gapped environment
      • Dataprep Microservice for Multimodal Data with Redis
        • Table of contents
        • 🚀1. Start Microservice with Docker
        • 🚀2. Status Microservice
        • 🚀3. Consume Microservice
      • Dataprep Microservice with Neo4J
        • Table of contents
        • Start Microservice with Docker
        • Invoke Microservice
      • Dataprep Microservice with Neo4J
        • Table of contents
        • Start Microservice with Docker
        • Invoke Microservice
      • Dataprep Microservice with OpenSearch
        • Table of contents
        • 🚀1. Start Microservice with Docker
        • 🚀2. Status Microservice
        • 🚀3. Consume Microservice
      • Dataprep Microservice with PGVector
        • Table of contents
        • 🚀1. Start Microservice with Docker
        • 🚀2. Consume Microservice
      • Dataprep Microservice with Pinecone
        • Table of contents
        • 🚀Start Microservice with Docker
        • Invoke Microservice
      • Dataprep Microservice with Qdrant
        • Table of contents
        • 🚀Start Microservice with Docker
        • Invoke Microservice
        • Running in the air gapped environment
      • Dataprep Microservice with Redis
        • Table of contents
        • 🚀1. Start Microservice with Docker
        • 🚀2. Status Microservice
        • 🚀3. Consume Microservice
        • Running in the air gapped environment
      • Dataprep Microservice with VDMS
        • Table of contents
        • 🚀1. Start Microservice with Docker (Option 2)
        • 🚀2. Status Microservice
        • 🚀3. Consume Microservice
    • Embeddings Microservice
      • Embeddings Microservice
        • Table of Contents
        • Overview
        • Key Features
        • Supported Implementations
      • Deploy Embedding microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Multimodal Embeddings Microservice
        • Table of Contents
        • Start Microservice
        • Consume Embedding Service
        • Running in the air gapped environment
      • Multimodal CLIP Embedding Microservice
        • Table of Contents
        • Start Microservice
        • Consume Embedding Service
      • Embedding Microservice with OpenVINO Model Server (OVMS)
        • Table of Contents
        • Start Microservice with docker run
        • Start Microservice with Docker Compose
        • Consume Embedding Service
        • Tips for Better Understanding
      • Embedding Microservice with Prediction Guard
        • Table of Contents
        • Start Microservice with docker run
        • Start Microservice with Docker Compose
        • Consume Embedding Service
        • Tips for Better Understanding
      • Embedding Microservice with TEI
        • Table of Contents
        • Start Microservice with docker run
        • Start Microservice with Docker Compose
        • Consume Embedding Service
        • Tips for Better Understanding
    • Feedback_management Microservice
      • 🗨 Feedback Management Microservice
        • 🛠️ Features
        • ⚙️ Deployment Options
      • 🗨 Feedback Management Microservice with MongoDB
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
      • 🗨 Feedback Management Microservice with ArangoDB
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
      • 🗨 Feedback Management Microservice with Redis
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
    • Finetuning Microservice
      • Fine-tuning Microservice
        • Table of contents
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Finetuning Service
        • 🚀4. Descriptions for Finetuning parameters
      • XTune - Model finetune tool for Intel GPU
        • Table of contents
        • Installation
        • Data Preparation
        • Fine-Tuning with LLaMA Board GUI (powered by Gradio)
        • Fine-Tuning with Shell instead of GUI
        • Xtune Examples
        • Citation
        • Acknowledgement
      • Dataset
        • Dataset for CLIP
        • Dataset for AdaCLIP
        • Preprocessing
        • Dataset for Qwen2-VL Finetune
        • Update dataset_info.json
      • Qwen2-VL Training and Hyperparameter Optimization
        • 🚀 Features
        • 📋 Running Modes
        • 🛠️ Usage
        • ⚙️ Configuration Parameters
        • 🔍 Optuna features
        • 💡 Log Control
        • 📊 Output and Results
        • 📝 Complete Examples
        • Finetune qwen2-vl with logging eval loss
      • AdaCLIP-Finetune
      • Prerequisites
      • How to Install
        • Install on NVIDIA
        • Install on Arc A770
      • Prepare Datasets
        • Datasets
        • Frame Extraction
        • Dataset JSON prepare
      • Implemented finetune methods
        • BitFit & SSF
        • Importance Based Selection (IBS)
        • Performance of different finetune methods
      • How to Finetune
        • Finetune on NVIDIA
        • Finetune on Arc A770
      • Use optuna to automatic get the best param
        • Visualization
      • Install linux kernel
      • Install Driver
        • Optional:
      • The core features for clip finetune tool:
      • How to config features for clip finetune tool:
        • Basic yaml for vit_b16.yaml
        • Angle-Based Selection for full-finetune
        • customize trained layer for partial-finetune
        • fixed cache size for tip-adapter
    • Guardrails Microservice
      • Trust and Safety with LLM
      • Deploy guardrails microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Bias Detection Microservice
        • Table of Contents
        • Introduction
        • Future Development
        • Start Microservice
      • Factuality Check Prediction Guard Microservice
        • Table of Contents
        • Introduction
        • Start Microservice with Docker
        • Consume Factuality Check Service
      • Guardrails Microservice
        • Table of Contents
        • Introduction
        • Setups
        • Start Microservice
        • Consume Microservice
      • Hallucination Detection Microservice
        • Table of Contents
        • Introduction
        • Start Microservice
        • Set up Microservice
        • Consume Guardrail Micorservice
      • PII Detection Microservice
        • Table of Contents
        • Introduction
        • Environment Setup
        • Start Microservice
        • Get Status of Microservice
        • Consume Microservice
      • Prompt Injection and Jailbreak Detection Microservice
        • Table of Contents
        • Introduction
        • Environment Setup
        • Start Microservice with Docker
        • Get Status of Microservice
        • Consume Prompt Injection Detection Service
      • Toxicity Detection Microservice
        • Table of Contents
        • Introduction
        • Environment Setup
        • Start Microservice with Python（Option 1）
        • Start Microservice with Docker (Option 2)
        • Get Status of Microservice
        • Consume Microservice Pre-LLM/Post-LLM
    • Image2image Microservice
      • Deploying Image-to-Image Service
        • 🚀 1. Quick Start with Docker Compose
        • 🚀 2. Manual Step-by-Step Deployment (Advanced)
      • Image-to-Image Microservice
        • Table of contents
        • Architecture
        • Deployment Options
        • Validated Configurations
    • Image2video Microservice
      • Deploying Image-to-Video Service
        • 🚀 1. Quick Start with Docker Compose
        • 🚀 2. Manual Step-by-Step Deployment (Advanced)
      • Image-to-Video Microservice
        • Table of contents
        • Architecture
        • Deployment Options
        • Validated Configurations
    • Language_detection Microservice
      • Deploying language-detection Service
        • 🚀 1. Quick Start with Docker Compose
        • 🚀 2. Manual Step-by-Step Deployment (Advanced)
        • 🚀 3. Start Language Detection Microservice with Python
      • Language Detection microservice
        • Table of contents
        • Architecture
        • Deployment Options
        • Validated Configurations
    • Llms Microservice
      • Deploy LLM microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Document Summary LLM Microservice
        • Table of Contents
        • Start Microservice
        • Consume Microservice
        • Air-Gapped Environment Usage
      • FAQGen LLM Microservice
        • Table of Contents
        • Start Microservice
        • Consume Microservice
      • LLM Text Generation Microservice
        • Table of Contents
        • Validated LLM Models
        • Start Microservice
        • Consume Microservice
      • OPEA Text Generation Microservice (Amazon Bedrock)
        • Table of Contents
        • Setup Environment Variables
        • Build Docker Image
        • Start Microservice
        • Consume Microservice
      • OPEA Text Generation Microservice (OpenAI-Compatible Endpoint)
        • Table of Contents
        • Prepare Docker Image
        • Setup Environment Variables
        • Start Microservice
        • Consume Microservice
      • LLM Native Microservice
        • Table of Contents
        • Start Microservice
        • Consume Microservice
      • LLM OVMS Microservice
        • Table of Contents
        • Start OVMS Microservice
        • Start OPEA LLM Microservice
        • Consume Microservice
        • Tips
      • Prediction Guard Microservice
        • Table of Contents
        • Start Microservice
        • Consume Microservice
      • LLM Microservice with vLLM on Intel XPU
        • Table of Contents
        • Prerequisites
        • Start Microservice
        • Consume LLM Service
      • LM-Eval Microservice
        • CPU service
    • Lvms Microservice
      • LVM Microservice
        • Table of Contents
        • Overview
        • Key Features
        • Supported Implementations
      • Deploy LVM microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • LVM Microservice with LLaMA-Vision
        • Table of Contents
        • Start Microservice
        • Consume LVM Service
      • LVM Microservice with LLaVA
        • Table of Contents
        • Start Microservice
        • Consume LVM Service
      • LVM Microservice with TGI-accelerated LLaVA
        • Table of Contents
        • Start Microservice
        • Consume LVM Service
      • LVM Microservice with PredictionGuard
        • Table of Contents
        • Start Microservice
        • Consume LVM Service
      • LVM Microservice with Video-LLaMA
        • Table of Contents
        • Start Microservice
        • Consume LVM Service
      • LVM Microservice with vLLM on Gaudi
        • Table of Contents
        • Prerequisites
        • Start Microservice
        • Consume LVM Service
      • LVM Microservice with vLLM on Intel XPU
        • Table of Contents
        • Prerequisites
        • Start Microservice
        • Consume LVM Service
    • Prompt_registry Microservice
      • 🧾 Prompt Registry Microservice
        • 🛠️ Features
        • ⚙️ Deployment Options
      • 🧾 Prompt Registry Microservice with MongoDB
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
        • 🤖 MCP (Model Context Protocol) Usage
      • 🧾 Prompt Registry Microservice with ArangoDB
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
        • 🤖 MCP (Model Context Protocol) Usage
      • 🧾 Prompt Registry Microservice with Redis
        • Setup Environment Variables
        • 🚀 Start Microservice with Docker (Option 1)
        • 🚀 Start Microservice with Docker Compose (Option 2)
        • 🤖 MCP (Model Context Protocol) Usage
      • Deploy prompt microservice on Kubernetes cluster
        • Deploy on Kubernetes
    • Prompt_template Microservice
      • Prompt Template microservice
        • Getting started
    • Rerankings Microservice
      • Deploy reranking microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Reranking Microservice
        • 🛠️ Features
        • ⚙️ Implementation
      • 🌟 Reranking Microservice with OVMS
        • 📦 1. Prepare the model in the model repository
        • 📦 2. Start Microservice with Docker
        • 📦 3. Start Microservice with docker compose
        • 📦 4. Consume Reranking Service
        • ✨ Tips for Better Understanding:
      • 🌟 Reranking Microservice with TEI
        • 📦 1. Start Microservice with Docker
        • 📦 2. Start Microservice with docker compose
        • 📦 3. Consume Reranking Service
        • ✨ Tips for Better Understanding:
      • 🌟 Reranking Microservice with VideoQnA
        • 📦 1. Start Microservice with Docker
        • 📦 2. Start Microservice with docker compose
        • 📦 3. Consume Reranking Service
        • ✨ Tips for Better Understanding:
    • Retrievers Microservice
      • Retriever Microservice
        • Retriever Microservice with Redis
        • Retriever Microservice with Milvus
        • Retriever Microservice with Qdrant
        • Retriever Microservice with PGVector
        • Retriever Microservice with VDMS
        • Retriever Microservice with ElasticSearch
        • Retriever Microservice with OpenSearch
        • Retriever Microservice with neo4j
        • Retriever Microservice with Pathway
        • Retriever Microservice with MariaDB Vector
        • Retriever Microservice with ArangoDB
      • Deploy retriever microservice on Kubernetes cluster
        • Deploy on Kubernetes with redis vector DB
        • Deploy on Kubernetes with milvus vector DB
      • Retriever Microservice with ArangoDB
        • 🚀Start Microservice with Docker
        • 🚀3. Consume Retriever Service
      • Retriever Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
      • Retriever Microservice
        • 🚀1. Start Microservice with Docker
        • 🚀2. Consume Retriever Service
      • Retriever Microservice with Milvus
        • 🚀Start Microservice with Python
        • 🚀Start Microservice with Docker
        • 🚀3. Consume Retriever Service
      • Retriever Microservice with Neo4J
        • 🚀Start Microservice with Docker
        • Invoke Microservice
      • Retriever Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
      • Retriever Microservice with Pathway
        • 🚀Start Microservices
      • Retriever Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
      • Retriever Microservice with Qdrant
        • 1. 🚀Start Microservice with Python (Option 1)
        • 2. 🚀Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
      • Retriever Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
      • Retriever Microservice
        • Visual Data Management System (VDMS)
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume Retriever Service
    • Router Microservice
      • Router Microservice
        • Build
        • RouteLLM compatibility patch
        • API Usage
        • Configuration Reference
        • Troubleshooting
        • Testing
    • Struct2graph Microservice
      • Struct2Graph Microservice
        • Features
        • Implementation
        • 🚀1. Start Microservice with docker run
        • 🚀2. Start Microservice with docker compose
        • 3. Validate the service using API endpoint
    • Text2cypher Microservice
      • Deploy text2cypher microservice using docker-compose
        • Deploy on Intel Gaudi
      • ⚠️ Deprecation Notice: text2cypher
      • 🛢 Text-to-Cypher Microservice
        • 🛠️ Features
        • ⚙️ Implementation
    • Text2graph Microservice
      • Text to graph triplet extractor
        • Decoder Models
        • Encoder-decoder models
      • Features
        • Implementation
      • 🚀1. Start Microservice with Docker
        • Install Requirements
        • Environment variables : Configure LLM Parameters based on the model selected.
      • Validation and testing
        • Text to triplets
        • Check if services are up
    • Text2image Microservice
      • Deploy text2image on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • Text-to-Image Microservice
      • 🚀1. Start Microservice with Python (Option 1)
        • 1.1 Install Requirements
        • 1.2 Start Text-to-Image Microservice
      • 🚀2. Start Microservice with Docker (Option 2)
        • 2.1 Build Images
        • 2.2 Start Text-to-Image Service
      • 3 Test Text-to-Image Service
    • Text2kg Microservice
      • Text to knowledge graph (text2kg) microservice
        • Decoder-Only Models
        • Features
        • 🚀 1. Start individual microservices using docker cli (Option 1)
        • 🚀 2. Start text2kg and dependent microservices with docker-compose (Option 2)
        • 3. Check the service using API endpoint
    • Text2query Microservice
      • 🛢 Text-to-Query Microservice
        • 🛠️ Features
        • ⚙️ Supported Implementations
      • 🛢 Text-to-Cypher Microservice
        • Setup Environment Variables.
        • 🚀 Start Text2Cypher Microservice with Python（Option 1）
        • 🚀 Start Microservice with Docker (Option 2)
        • ✅ Invoke the microservice.
      • Text-to-SQL Microservice
        • 🚀 Start Microservice with Python（Option 1）
        • 🚀 Start Microservice with Docker (Option 2)
        • ✅ Invoke the microservice.
    • Text2sql Microservice
      • ⚠️ Deprecation Notice: text2sql
      • 🛢 Text-to-SQL Microservice
        • 🛠️ Features
        • ⚙️ Implementation
    • Third_parties Microservice
      • Start ArangoDB Server
        • 1. Download ArangoDB image
        • 2. Configure the password
        • 3. Run ArangoDB service
      • Deploy mm-embedding on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • Multimodal Embeddings Microservice with BridgeTower
        • 🚀1. Start MMEI on Gaudi2 HPU
        • 🚀2. Start MMEI on Xeon CPU
        • Running in the air gapped environment
        • 🚀3. Access the service
      • Multimodal Embeddings Microservice with CLIP
        • 🚀1. Start Microservice with Docker
        • 🚀2. Consume Embedding Service
      • Start Elasticsearch server
        • 1. Download Elasticsearch image
        • 2. Run Elasticsearch service
      • Deploy gpt-sovits on kubernetes cluster
        • Deploy on Xeon
      • GPT-SoVITS Microservice
        • Build the Image
        • Start the Service
        • Test
      • IPEX Serving microservice
        • 🚀1. Build the Docker Image
        • 🚀2. Start the microservice
        • 🚀3. Access the service
      • LVM Microservice
        • 🚀 Start Microservice with Docker
      • LVM Microservice
        • 🚀1. Start Microservice with Python (Option 1)
        • 🚀2. Start Microservice with Docker (Option 2)
      • Start MariaDB Server
        • 1. Configure the server
        • 2. Run MariaDB Server
      • Start Milvus server
        • 1. Configuration
        • 2. Run Milvus service
      • Deploy MongoDB on kubernetes cluster
        • Deploy on Xeon
      • Start Neo4J Server
        • 1. Download Neo4J image
        • 2. Configure the username, password and dbname
        • 3. Run Neo4J service
      • Deploy nginx on kubernetes cluster
        • Deploy on Xeon
      • Introduction
        • Get Started
      • Start Opensearch server
        • Prerequisites
        • Instructions
        • Troubleshooting
      • Deploy OVMS on kubernetes cluster
        • Deploy on Xeon
      • Start the Pathway Vector DB Server
        • Configuration
        • Building and running
        • Health check the vector store
      • Start PGVector server
        • 1. Download Pgvector image
        • 2. Configure the username, password and dbname
        • 3. Run Pgvector service
      • LVM Prediction Guard Microservice
        • 🚀1. Start Microservice with Python
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Consume LVM Service
      • Start PGVector server
        • 1. Download Pgvector image
        • 2. Configure the username, password and dbname
        • 3. Run Pgvector service
      • Deploy RedisDB on kubernetes cluster
        • Deploy on Xeon
      • Start Redis server
        • 1. Download Redis image
        • 2. Run Redis service
      • SGLang Serving microservice
        • 🚀1. Build the Docker Image
        • 🚀2. Start the microservice
        • 🚀3. Access the service
      • Deploy speecht5 on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • Deploy TEI on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • TGI LLM Microservice
        • Start TGI with docker compose
      • Deploy TGI on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • Start VDMS server
        • 1. Download VDMS image
        • 2. Run VDMS service
      • LVM Microservice
        • 🚀1. Start Microservice with Docker
        • ✅ 2. Test
        • ♻️ 3. Clean
      • vLLM Endpoint Service
        • 🚀1. Set up Environment Variables
        • 🚀2. Set up vLLM Service
        • 🚀3. Set up LLM microservice
      • Deploy vllm on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
      • Deploy whisper on kubernetes cluster
        • Deploy on Xeon
        • Deploy on Gaudi
    • Tts Microservice
      • Deploy tts microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • TTS Microservice
        • 1.2 Start SpeechT5 Service/Test
        • 1.3 Start TTS Service/Test
        • 🚀2. Start Microservice with Docker (Option 2)
        • 🚀3. Start Microservice with Docker Compose (Option 3)
    • Web_retrievers Microservice
      • Deploy web-retriever microservice on Kubernetes cluster
        • Deploy on Kubernetes
      • Web Retriever Microservice
        • 🚀1. Start Microservice with Docker (Option 1)
        • 🚀2. Start Microservice with Docker Compose (Option 2)
        • 🚀3. Consume Web Retriever Service
  • Deploying GenAI
    • GenAIInfra
      • Overview
      • Prerequisite
        • Setup Kubernetes cluster
        • (Optional) To run GenAIInfra on Intel Gaudi product
      • Usages
        • Use GenAI Microservices Connector (GMC) to deploy and adjust GenAIExamples
        • Use helm charts to deploy
        • Use terraform to deploy on cloud service providers
      • Additional Content
    • Development
      • Prerequisites
      • Creating new helm chart for OPEA services
      • Testing
      • pre-commit testing
    • Legal Information
      • License
      • Citation
    • Release Branches
      • 1. Create release candidate branch
      • 2. Create images with release tag
      • 3. Test helm charts
      • 4. Test GMC
      • 5. Publish images
    • Installation Guides
      • GenAI-microservices-connector(GMC) Installation
        • GenAI-microservices-connector(GMC)
        • Install GMC
        • Use GMC to compose a chatQnA Pipeline
      • Kubernetes Installation Options
      • Kubernetes Installation using AWS EKS Cluster
        • Prerequisites
        • Create AWS EKS Cluster in AWS Console
        • Uploading images to an AWS Private Registry
      • Kubernetes installation demo using kubeadm
        • Node configuration
        • Step 0. Clean up the environment
        • Step 1. Install relevant components
        • Step 2. Create the k8s cluster
        • Step 3 (optional) Reset Kubernetes cluster
        • NOTES
      • Kubernetes installation using Kubespray
        • Node preparation
        • Prerequisites
        • Step 1. Set up Kubespray and Ansible
        • Step 2. Build your own inventory
        • Step 3. Define Kubernetes configuration
        • Step 4. Deploy Kubernetes
        • Step 5. Create kubectl configuration
        • Quick reference
    • CI/CD ENV Setup
      • Setup Test Environment for CI/CD
        • Install Habana Driver (Gaudi Only)
        • Install Docker
        • Install Conda
        • Install K8S
        • Setup Image Registry for K8S Test
        • Setup GHA ENV for CI/CD
        • Setup Action Runner Controller (ARC)
    • Cloud Service Provider
      • OPEA applications AWS EKS deployment guide
        • Prerequisites
        • Setup
        • EKS cluster
        • Persistent Volume Claim
        • OPEA Applications
      • OPEA applications Azure AKS deployment guide
        • Prerequisites
        • Setup
        • AKS cluster
        • Cosmos DB
        • Persistent Volume Claim
        • OPEA Applications
      • OPEA applications GCP GKE deployment guide
        • Prerequisites
        • Setup
        • Update your opea-chatqna.tfvars file
        • GKE cluster
        • Persistent Volume Claim
        • OPEA Applications
    • Authentication and Authorization
      • Authentication and authorization
        • Istio based implementation for cloud native environments
        • APISIX based implementation for cloud native environments
      • Authentication and Authorization with APISIX and OIDC based Identity provider (Keycloak)
        • Start ChatQnA service
        • Starting and Configuring Keycloak
        • Update values
        • Install APISIX
        • Usage
        • Uninstall
      • Leveraging Istio to compose an OPEA Pipeline with authentication and authorization enabled
        • Prerequisite
        • Perform authentication and authorization via Bearer JWT tokens and curl
        • Perform authentication and authorization via oauth2-proxy and OIDC provider and UI
        • Uninstall Istio
    • Helm Charts
      • Helm charts for deploying GenAI Components and Examples
        • Table of Contents
        • Helm Charts
        • Deploy with Helm charts
        • Helm Charts Options
        • Deploy the Helm Charts on Intel® Xeon® Processors with Intel® Trust Domain Extensions (Intel® TDX)
        • Using HPA (autoscaling)
        • Using Persistent Volume
        • Using Private Docker Hub
        • Generate manifests from Helm Charts
      • CI guidelines for helm charts
        • Table of Contents
        • Infra Setup
        • Add new test case
      • HorizontalPodAutoscaler (HPA) support
        • Table of Contents
        • Introduction
        • Pre-conditions
        • Gotchas
        • Enable HPA
        • Verify
        • Scaling metric considerations
      • Deployment of the Helm Charts on Intel® Xeon® Processors with Intel® Trust Domain Extensions (Intel® TDX)
        • Technical background
        • Prerequisites
        • Getting Started
      • Observability for OPEA Workloads in Kubernetes
        • Table of Contents
        • Introduction
        • Pre-conditions
        • Install
        • Verify
      • agent
        • Deploy
        • Verify
        • Options
      • asr
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • chathistory-usvc
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • Dashboards
        • Installing the Chart
        • Values
      • data-prep
        • Installing the chart
        • Verify
        • Values
        • Milvus support
      • embedding-usvc
        • Installing the chart
        • Verify
        • Values
      • gpt-sovits
        • Install the chart
        • Verify
        • Values
      • guardrails-usvc
        • Installing the chart
        • Verify
        • Values
      • llm-uservice
        • Installing the chart
        • Verify
        • Values
      • OPEA lvm-serve microservice
        • Installing the Chart
        • Verify
        • Values
      • lvm-uservice
        • Installing the chart
        • Verify
        • Values
      • OPEA mm-embedding microservice
        • Installing the Chart
        • Verify
        • Values
      • mongodb
        • Install the Chart
        • Verify
        • Values
      • OPEA nginx microservice
        • Integration with OPEA Applications
        • Gateway Mode (Central Router)
        • Deployment Modes
      • ollama
        • Installing the Chart
        • Verify
        • Values
      • prompt-usvc
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • redis-vector-db
        • Install the Chart
        • Verify
        • Values
      • reranking-usvc
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • retriever-usvc
        • Installing the chart
        • Verify
        • Values
        • Milvus support
      • speecht5
        • Installing the Chart
        • Verify
        • Values
      • tei
        • Installing the Chart
        • Verify
        • Values
      • teirerank
        • Installing the Chart
        • Verify
        • Values
      • OPEA text2image microservice
        • Installing the Chart
        • Verify
        • Values
      • tgi
        • Installing the Chart
        • Verify
        • Values
      • tts
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • vllm
        • Installing the Chart
        • Verify
        • Values
      • web-retriever
        • (Option1): Installing the chart separately
        • (Option2): Installing the chart with dependencies automatically
        • Verify
        • Values
      • whisper
        • Installing the Chart
        • Verify
        • Values
      • AgentQnA
        • Deploy
        • Verify
      • AudioQnA
        • Installing the Chart
        • Verify
        • Values
      • ChatQnA
        • Installing the Chart
        • Verify
        • Values
        • Troubleshooting
      • ChatQnA Troubleshooting
        • a function to get the endpoint of service
        • define the namespace of service
        • Update a file to database
        • get the embedding of input
        • get the retriever docs
        • reranking the docs
        • TGI Q and A
        • REF
      • CodeGen
        • Installing the Chart
        • Verify
        • Values
      • CodeTrans
        • Installing the Chart
        • Verify
        • Values
      • DocSum
        • Installing the Chart
        • Verify
        • Values
      • FinanceAgent
        • Deploy
        • Verify
      • SearchQnA
        • Installing the Chart
        • Verify
        • Values
      • txt2img
        • Installing the Chart
        • Verify
        • Values
      • VisualQnA
        • Installing the Chart
        • Verify
        • Values
    • KubeAI Operator
      • KubeAI for OPEA
        • Features
        • Models
      • Installation
        • Prerequisites
        • Install KubeAI
      • Deploying the Models
        • Text Generation with Llama-3 on CPU
        • Text Generation with Llama-3 on Gaudi
        • Text Embeddings with BGE on CPU
      • Using the Models
      • CPU Performance Optimization with NRI
        • Overview
        • Installation of Balloons Policy Plugin
        • Configuration of Balloons Policy Plugin
      • Observability
      • Model Files
        • Benchmarking
        • Additional Notes
    • Kubernetes Addons
      • Deploy Kubernetes add-ons for OPEA
      • Metrics / visualization add-ons
        • Pre-conditions
        • Device metrics for Gaudi HW
        • Extra metrics for OPEA applications
        • CPU mmetrics from PCM
        • Importing dashboards to Grafana
        • More dashboards
      • OPEA with Istio
        • Introduction
        • Deployment
        • Create Istio gateway with TLS and virtual service
        • Enforce mTLS between OPEA pods
        • Cleanup
      • memory bandwidth exporter
        • Setup
        • More flags about memory bandwidth exporter
    • Microservices Connector
      • genai-microservices-connector(GMC)
        • Description
        • Architecture
        • Personas
        • Getting Started
      • Troubleshooting GMC Custom Resource（CR）
      • Usage guide for genai-microservices-connector(GMC)
        • Use GMC to compose a chatQnA Pipeline
        • Use GMC to adjust the chatQnA Pipeline
        • Use GMC to delete the chatQnA Pipeline
        • Use GMC and Istio to compose an OPEA Pipeline with authentication and authorization enabled
      • ChatQnA Use Cases in Kubernetes Cluster via GMC
        • Using prebuilt images
        • Deploy ChatQnA pipeline
      • Helm chart for genai-microservices-connector(GMC)
        • Installing the GMC Helm Chart
        • Check the installation result
        • Next step
        • Uninstall
    • Pipeline Proxy
      • OPEA Pipeline Proxy
        • Features
        • Build
        • Deployment
        • Development
      • Guardrails
        • Architecture
        • Deployment
    • Scripts
      • Scripts and tools
      • NVIDIA GPU Quick-Start Guide
        • Prerequisite
        • Usages
        • FAQ and Troubleshooting
      • Deploy Autoscaling Ray Cluster with KubeRay in Kubernetes Cluster
        • Install KubeRay
        • Start Ray Cluster with Autoscaling
        • Delete Ray Cluster with Autoscaling
        • Uninstall KubeRay
    • Contributing
      • Development
        • Helm Starter Chart
        • NOTE
      • OPEA microservice
        • Installing the Chart
        • Verify
        • Values
  • Evaluating GenAI
    • GenAIEval
      • Installation
      • Evaluation
        • lm-evaluation-harness
        • bigcode-evaluation-harness
        • Kubernetes platform optimization
      • Benchmark
        • Features
        • How to use
        • Grafana Dashboards
      • Additional Content
    • Legal Information
      • License
      • Citation
    • Kubernetes Platform Optimization with Resource Management
      • Introduction
      • NRI Plugins
      • Install
      • Validate policy status
      • Configure
      • Validate CPU affinity and hardware alignment in containers
      • Remove a policy
      • NRI topology-aware resource policy
      • Recommendations
    • GenAIEval Dockerfiles
      • Gaudi Requirements
      • Run GenAIEval on Gaudi
        • Docker Build
        • Docker Run
      • Benchmarking OPEA Examples on Intel® Gaudi® AI Processor and Xeon® Processor
        • Docker Build
        • Run a OPEA Example using docker compose
        • Docker Run
    • OPEA Benchmark Tool
      • Features
      • Table of Contents
      • Installation
        • Prerequisites
      • Usage
      • Configuration
        • Test Suite Configuration
        • Test Cases
    • Auto-Tuning for ChatQnA: Optimizing Resource Allocation in Kubernetes
      • Key Features
    • Usage
      • Configuration Files
        • Hardrware_info.json
        • chatqna_neuralchat_rerank_latest.yaml
        • Tuning Config Parameters
      • Output
    • Auto-Tuning for ChatQnA: Optimizing Accuracy by Tuning Model Related Parameters
      • Prepare Dataset
      • Run the Tuning script
    • Setup Prometheus and Grafana to visualize microservice metrics
      • 1. Setup Prometheus
        • 1.1 Node Metrics (optional)
        • 1.2 Intel® Gaudi® Metrics (optional)
      • 2. Setup Grafana
      • 3. Import Grafana Dashboard
    • StressCli
      • stresscli.py
        • Prerequirements
        • Installation
        • Usage
    • How to benchmark pubmed datasets by send query randomly
      • 1. prepare the pubmed datasets
        • 1.1 get pubmed data
        • 1.2 use script to extract data
        • 1.3 prepare 4 dataset files
        • 1.4 prepare the query list
      • 2. How to use pubmed qlist
    • locust scripts for OPEA ChatQnA
      • Configuration file
      • Basic Usage
    • HELMET HELMET: How to Evaluate Long-context Language Models Effectively and Thoroughly
      • Quick Links
      • Release Progress
      • Setup
      • Data
      • Running evaluation
        • Run on Intel Gaudi
        • Model-based evaluation
      • Adding new models
      • Adding new tasks
      • Dataset correlation analysis
      • Others
      • Contacts
      • Citation
    • Benchmarks for agentic applications
    • TAG-Bench for evaluating SQL agents
      • Overview of TAG-Bench
      • Getting started
      • Launch your SQL agent
      • Run the benchmark
      • Benchmark results
    • CRAG Benchmark for Agent QnA systems
      • Overview
      • Getting started
      • CRAG dataset
      • Launch Agent QnA system
      • Run CRAG benchmark
      • Use LLM-as-judge to grade the answers
        • Validation of LLM-as-judge
      • Benchmark results for OPEA RAG Agents
        • Comparison with GPT-4o-mini
    • AutoRAG to evaluate the RAG system performance
      • Service preparation
      • RAG evaluation
      • Notes
    • 🚀 QuickStart
      • Installation
      • Launch Service of LLM-as-a-Judge
      • Writing your first test case
      • Acknowledgements
    • Model Card Generator
      • Steps to generate a Model Card
    • 🚀 QuickStart
      • Installation
      • Launch a LLM Service
        • Example 1: TGI
        • Example 2: OPEA LLM
      • Predict
      • Evaluate
    • Evaluation Methodology
      • Introduction
      • Prerequisite
        • Environment
      • MultiHop (English dataset)
        • Launch Service of RAG System
        • Launch Service of LLM-as-a-Judge
        • Prepare Dataset
        • Evaluation
      • CRUD (Chinese dataset)
        • Prepare Dataset
        • Launch Service of RAG System
        • Evaluation
      • Acknowledgements
    • 🚀 RAG Pilot - A RAG Pipeline Tuning Tool
      • 📖 Overview
      • 🌐 Quickstart Guide
        • ⚙️ Dependencies and Environment Setup
    • 🚀 RAG Pilot - A RAG Pipeline Tuning Tool
      • 📖 Overview
      • 🧠 Available Tuners
      • 🚦 How to use RAG Pilot
        • 🔹Input file: QA List File (your_queries.csv)
        • ▶️ Use RAG Pilot with UI
        • ▶️ Use RAG Pilot with RESTful API
      • 🔧 How to Adjust RAG Pilot to Tune Your RAG Solution
        • 🧩 What’s Nodes and Modules
        • ⚙️ How to Configure Nodes and Modules
        • 🧑‍💻 How to Use Nodes and Modules
    • Toxicity Detection Accuracy
      • Get Started on Gaudi 2 Accelerator
        • Requirements
        • Setup
        • Evaluation
      • Get Started on CPU
        • Requirements
        • Installation
        • Evaluation
    • Metric Card for BLEU
      • Metric Description
      • Intended Uses
      • How to Use
        • Inputs
        • Output Values
        • Examples
      • Limitations and Bias
      • Citation
      • Further References
    • RAGAAF (RAG assessment - Annotation Free)
      • Key features
      • Run RAGAAF
        • 1. Data
        • 2. Launch endpoint on Gaudi
        • 3. Model
        • 4. Metrics
        • 5. Evaluation
      • Customizations
    • OPEA adaption of ragas (LLM-as-a-judge evaluation of Retrieval Augmented Generation)
      • User data
      • Launch HuggingFace endpoint on Intel’s Gaudi machines
      • Run OPEA ragas pipeline using your desired list of metrics
      • Troubleshooting
  • Developer Guides
    • Coding Guides
      • OPEA API Service Spec (v1.0)
        • OPEA Mega Service API
        • OPEA Micro Service API
    • Documentation Guides
      • Documentation Guidelines
        • Markdown vs. RestructuredText
        • Documentation Organization
        • Headings
        • Content Highlighting
        • Lists
        • Multi-Column Lists
        • Tables
        • File Names and Commands
        • Branch-Specific File Links
        • Internal Cross-Reference Linking
        • Non-ASCII Characters
        • Include Content from Other Files
        • Code and Command Examples
        • Images
        • Tabs, Spaces, and Indenting
        • Background Colors
        • Drawings
        • Alternative Tabbed Content
        • Instruction Steps
        • First Instruction Step
        • Second Instruction Step
        • Documentation Generation
      • README Guidelines for GenAIExamples
        • Prepare the Primary README File
        • Prepare the Secondary README File
      • Drawings Using Graphviz
        • Simple Directed Graph
        • Adding Edge Labels
        • Tables
        • Finite-State Machine
      • OPEA Documentation Generation
        • Documentation Overview
        • Set Up the Documentation Working Folders
        • Install the Documentation Tools
        • Documentation Presentation Theme
        • Run the Documentation Processors
        • Doc Build Troubleshooting
        • Publish Content
        • Document Versioning
        • Filter Expected Warnings
  • OPEA Community
    • Community Support
    • Resources
    • Contributing Guides
      • Contribution Guidelines
        • All The Ways To Contribute
        • Support
        • Contributor Covenant Code of Conduct
      • Contributing A New Vector Database to OPEA
        • Contributing a new Vector Database to GenAIComps
        • ChatQnA GenAIExample Vector Database Usage
      • OPEA Project Code Owners
        • docs Repository Code Owners
        • GenAIComps Repository Code Owners
        • GenAIEval Repository Code Owners
        • GenAIExamples Repository Code Owners
        • GenAIInfra Repository Code Owners
        • Continuous Integration (CICD) owners
      • OPEA Emeritus Code Owners
      • Reporting a Vulnerability
        • Script Usage Notice
    • Roadmaps
      • OPEA 2024 - 2025 Roadmap
        • May 2024
        • June 2024
        • July 2024
        • Aug 2024
        • Sep 2024
        • Q4 2024
      • OPEA 2025 Roadmap
        • Release Cadence
        • Q1 2025 (v1.2 release)
        • Q2 2025 (v1.3 release)
        • Q3 2025 (v1.4 release)
        • Q4 2025 (v1.5 release)
        • Q1 2026 (v1.6 release)
      • OPEA CI/CD Roadmap
        • Milestone 1 (May, Done)
        • Milestone 2 (June)
        • Milestone 3 (July)
        • Milestone 4 (Aug)
    • Project Governance
      • Technical Charter (the “Charter”) for OPEA a Series of LF Projects, LLC
        • 1. Mission and Scope of the Project
        • 2. Technical Steering Committee
        • 3. TSC Voting
        • 4. Compliance with Policies
        • 5. Community Assets
        • 6. General Rules and Operations.
        • 7. Intellectual Property Policy
        • 8. Amendments
      • Technical Steering Committee (TSC)
        • Technical Steering Committee Members
      • Contributor Covenant Code of Conduct
        • Our Pledge
        • Our Standards
        • Enforcement Responsibilities
        • Scope
        • Enforcement
        • Enforcement Guidelines
        • Attribution
      • Reporting a Vulnerability
        • Script Usage Notice
    • RFC Proposals
      • Request for Comments (RFCs)
        • 24-05-16 GenAIExamples-001 Using MicroService to Implement ChatQnA
        • 24-05-16 OPEA-001 Overall Design
        • 24-05-24 OPEA-001 Code Structure
        • 24-06-21-OPEA-001-DocSum_Video_Audio
        • 24-06-21-OPEA-001-Guardrails-Gateway
        • 24-07-11-OPEA-Agent
        • 24-08-02-OPEA-AIAvatarChatbot
        • 24-08-05 OPEA Workflow Executor Example
        • 24-08-07 OPEA-001 OPEA GenAIStudio
        • 24-08-20-OPEA-001-AI Gateway API
        • 24-08-21-GenAIExample-002-Edge Craft RAG
        • 24-10-02-GenAIExamples-001-Image_and_Audio_Support_in_MultimodalQnA
        • 24-10-20-OPEA-001-Haystack-Integration
        • 24-11-25-GenAIExamples-Ollama_Support_for_CPU_Server
        • Purpose
        • Design
        • Routing AgentRFC
        • RFC: OPEA Inference Microservices (OIM)
        • 25-03-14-GenAIExamples-001-CodeTrans-with-Agents
        • Add Streaming Automatic Speech Recognition (ASR) support in GenAIComps
        • 25-15-01-GenAIExamples-001-Code-Generation-Using-RAG-and-Agents
        • HybridRAG
        • Support Air-gapped environment
        • RFC Template
  • Publications
    • News
    • Events
    • Blogs
    • Demos and Videos
  • Release Notes
    • OPEA Release Guide
      • Release Cadence
      • General Overview
      • Feature Freeze
      • Code/Doc Freeze, and Creating the RC Branch
        • Preparing Creating RC Branch
        • Creating RC Branch
      • Cherry Pick Critical Code/Doc Fix
        • How to do Cherry Picking
      • Creating Tag from RC Branch
      • Deliver Docker Images, Helm Charts, and PyPi Binaries
    • OPEA Release Notes v1.4
      • Table of Contents
      • What’s New in OPEA v1.4
        • Advanced Agent Capabilities
        • Components as MCP Servers
        • KubeAI Operator for OPEA
        • New GenAI Capabilities
        • Better User Experience
        • Newly Supported Models
        • Newly Supported Hardware
        • Newly Supported OS
      • Updated Dependencies
      • Changes to Default Behavior
      • Validated Hardware
      • Validated Software
      • Known Issues
      • Full Changelogs
      • Contributors
        • Contributing Organizations
        • Individual Contributors
    • OPEA Release Notes v1.3
      • Table of Contents
      • What’s New in OPEA v1.3
        • Advanced Agent Capabilities
        • Performance and Scalability
        • Ecosystem Integrations
        • New GenAI Capabilities
        • Enhanced Evaluation
        • Observability
        • Better User Experience
        • Exploration
        • Newly Supported Models
        • Newly Supported Hardware
        • Other Notable Changes
      • Deprecations
        • Deprecated Examples
        • Deprecated Docker Images
        • Deprecated GenAIExample Variables
        • Deprecated GenAIComps Parameters
      • Updated Dependencies
      • Changes to Default Behavior
      • Validated Hardware
      • Validated Software
      • Known Issues
      • Full Changelogs
      • Contributors
        • Contributing Organizations
        • Individual Contributors
    • OPEA Release Notes v1.2
      • What’s New in OPEA v1.2
        • Highlights
      • Deprecations and Behavior Changes
        • GenAIComps
        • GenAIExamples
        • GenAIEval
        • GenAIInfra
        • Docker Images
      • Notable Changes
      • Full Changelogs
      • Contributors
        • Contributing Organizations
        • Individual Contributors
    • OPEA Release Notes v1.1
      • What’s New in OPEA v1.1
        • Highlights
        • Notable Changes
        • Full Changelogs
      • Contributors
        • Contributing Organizations
        • Individual Contributors
    • OPEA Release Notes v1.0
      • What’s New in OPEA v1.0
      • Details
    • OPEA Release Notes v0.9
      • What’s New in OPEA v0.9
      • Details
    • OPEA Release Notes v0.8
      • What’s New in OPEA v0.8
      • Details
      • Thanks to these contributors
    • OPEA Release Notes v0.7
      • OPEA Highlights
      • GenAIExamples
      • GenAIComps
      • GenAIEvals
      • GenAIInfra
    • OPEA Release Notes v0.6
      • OPEA Highlight
      • GenAIExamples
      • GenAIComps
      • GenAIEvals
      • GenAIInfra
  • Contributing to OPEA
    • Additional Content
  • OPEA Frequently Asked Questions
    • What is OPEA’s mission?
    • What is OPEA?
    • What problems are faced by GenAI deployments within the enterprise?
    • Why now?
    • How does it compare to other options for deploying Gen AI solutions within the enterprise?
    • Will OPEA reference implementations work with proprietary components?
    • What does OPEA acronym stand for?
    • How do I pronounce OPEA?
    • What initial companies and open-source projects joined OPEA?
    • What is Intel contributing?
    • When you say Technical Conceptual Framework, what components are included?
    • What are the different ways partners can contribute to OPEA?
    • Where can partners see the latest draft of the Conceptual Framework spec?
    • Is there a cost for joining?
    • Do I need to be a Linux Foundation member to join?
    • Where can I report a bug or vulnerability?
OPEA™
  • Latest »
  • Search

© Copyright 2024-2025 OPEA™, a Series of LF Projects, LLC. Published on Oct 27, 2025.