OPEA™
latest
OPEA Project
v: latest
Document Versions
latest
1.0
1.1
OPEA Project links
Project Home
Wiki
Documentation Home
OPEA Overview
OPEA Project Architecture
Microservices: Flexible and Scalable Architecture
Megaservices: A Comprehensive Solution
Gateways: Customized Access to Mega- and Microservices
Next Step
Open Platform for Enterprise AI (OPEA) Framework Draft Proposal
1. Summary
2. Introduction
3. Framework Components, Architecture and Flow
4. Assessing GenAI components and flows
5. Grading Structure
6. Reference flows
Appendix A – Draft OPEA Specifications
Getting Started with OPEA
Understanding OPEA’s Core Components
Prerequisites
Create and Configure a Virtual Server
Deploy the ChatQnA Solution
Interact with ChatQnA
What’s Next
Get Involved
GenAI Examples
AgentQnA Sample Guide
Overview
Purpose
How It Works
Deployment
Single Node
ChatQnA Sample Guide
Overview
Purpose
Key Implementation Details
How It Works
Expected Output
Validation Matrix and Prerequisites
Architecture
Microservice Outline and Diagram
Deployment
Single Node
Xeon Scalable Processor
Gaudi AI Accelerator
Nvidia GPU
AI PC
Kubernetes
Cloud Native
Troubleshooting
Monitoring
Set Up the Prometheus Server
Set Up the Grafana Dashboard
Summary and Next Steps
Codegen Sample Guide
Overview
Purpose
How It Works
Deployment
Gaudi AI Accelerator
Generative AI Examples
Introduction
Architecture
Getting Started
Deployment Guide
Supported Examples
Contributing to OPEA
Additional Content
Examples
AgentQnA Application
Agents for Question Answering
Single node on-prem deployment with Docker Compose on Xeon Scalable processors
Single node on-prem deployment AgentQnA on Gaudi
Deploy AgentQnA on Kubernetes cluster
Retrieval tool for agent
AgentQnA
AudioQnA Application
AudioQnA Application
AudioQnA Accuracy
AudioQnA Benchmarking
Build Mega Service of AudioQnA on AMD ROCm GPU
Build Mega Service of AudioQnA on Xeon
Build Mega Service of AudioQnA on Gaudi
Deploy AudioQnA in Kubernetes Cluster on Xeon and Gaudi
Deploy AudioQnA on Kubernetes cluster
AudioQnA
AvatarChatbot Application
AvatarChatbot Application
Build Mega Service of AvatarChatbot on Xeon
Build Mega Service of AvatarChatbot on Gaudi
ChatQnA Application
ChatQnA Application
ChatQnA Accuracy
ChatQnA Benchmarking
Build and deploy CodeGen Application on AMD GPU (ROCm)
Build Mega Service of ChatQnA on AIPC
Build Mega Service of ChatQnA on Xeon
Build Mega Service of ChatQnA on Xeon
Build Mega Service of ChatQnA (with Qdrant) on Xeon
Build MegaService of ChatQnA on Gaudi
How to Check and Validate Micro Service in the GenAI Example
Build MegaService of ChatQnA on NVIDIA GPU
Deploy ChatQnA in Kubernetes Cluster on Xeon and Gaudi
Deploy ChatQnA on Kubernetes cluster
ChatQnA Conversational UI
ChatQnA Customized UI
CodeGen Application
Code Generation Application
CodeGen Accuracy
CodeGen Benchmarking
Build and deploy CodeGen Application on AMD GPU (ROCm)
Validate the MicroServices and MegaService
Build MegaService of CodeGen on Xeon
Build MegaService of CodeGen on Gaudi
Deploy CodeGen in a Kubernetes Cluster
Deploy CodeGen on Kubernetes cluster
Code Gen
Code Gen
CodeTrans Application
Code Translation Application
CodeTrans Benchmarking
Build and deploy CodeTrans Application on AMD GPU (ROCm)
Validate the MicroServices and MegaService
Build Mega Service of CodeTrans on Xeon
Build Mega Service of CodeTrans on Gaudi
Deploy CodeTrans in a Kubernetes Cluster
Deploy CodeTrans on Kubernetes cluster
Code Translation
DBQnA Application
DBQnA Application
Deploy on Intel Xeon Processor
DBQnA React Application
DocIndexRetriever Application
DocRetriever Application
DocRetriever Application with Docker
DocRetriever Application with Docker
DocSum Application
Document Summarization Application
Build and deploy DocSum Application on AMD GPU (ROCm)
Build Mega Service of Document Summarization on Intel Xeon Processor
Build MegaService of Document Summarization on Gaudi
Deploy DocSum in Kubernetes Cluster
Deploy DocSum on Kubernetes cluster
Document Summary
Doc Summary React
Doc Summary
EdgeCraftRAG Application
Edge Craft Retrieval-Augmented Generation
FaqGen Application
FAQ Generation Application
FaqGen Accuracy
FaqGen Benchmarking
Build and deploy FaqGen Application on AMD GPU (ROCm)
Build Mega Service of FAQ Generation on Intel Xeon Processor
Build MegaService of FAQ Generation on Gaudi
Deploy FaqGen on Kubernetes cluster
Doc Summary React
FAQ Generation
GraphRAG Application
GraphRAG Application
ChatQnA Conversational UI
ChatQnA Customized UI
InstructionTuning Application
Instruction Tuning
Deploy Instruction Tuning Service on Xeon
Deploy Instruction Tuning Service on Gaudi
MultimodalQnA Application
MultimodalQnA Application
Build Mega Service of MultimodalQnA for AMD ROCm
Build Mega Service of MultimodalQnA on Xeon
Build Mega Service of MultimodalQnA on Gaudi
ProductivitySuite Application
Productivity Suite Application
Build Mega Service of Productivity Suite on Xeon
🔐 Keycloak Configuration Setup
🚀 Deploy ProductivitySuite with ReactUI
Productivity Suite React UI
RerankFinetuning Application
Rerank Model Finetuning
Deploy Rerank Model Finetuning Service on Xeon
Deploy Rerank Model Finetuning Service on Gaudi
SearchQnA Application
SearchQnA Application
Build Mega Service of SearchQnA on Xeon
Build Mega Service of SearchQnA on Gaudi
Deploy SearchQnA in a Kubernetes Cluster
Neural Chat
Text2Image Application
Text-to-Image Microservice
Deploy Text-to-Image Service on Xeon
Deploy Text-to-Image Service on Gaudi
Text2Image Customized UI
Translation Application
Translation Application
Build and deploy Translation Application on AMD GPU (ROCm)
Validate the MicroServices and MegaService
Build Mega Service of Translation on Xeon
Build MegaService of Translation on Gaudi
Deploy Translation in a Kubernetes Cluster
Deploy Translation in Kubernetes Cluster
Language Translation
VideoQnA Application
VideoQnA Application
Build Mega Service of VideoQnA on Xeon
VisualQnA Application
Visual Question and Answering
VisualQnA Benchmarking
Build Mega Service of VisualQnA on AMD ROCm
Build Mega Service of VisualQnA on Xeon
Build MegaService of VisualQnA on Gaudi
Deploy VisualQnA in a Kubernetes Cluster
Deploy VisualQnA on Kubernetes cluster
WorkflowExecAgent Application
Workflow Executor Agent
Validate Workflow Agent Microservice
Legal Information
License
Citation
Docker Images
Example images
Microservice images
Supported Examples
ChatQnA
CodeGen
CodeTrans
DocSum
Language Translation
SearchQnA
VisualQnA
VideoQnA
RerankFinetuning
InstructionTuning
DocIndexRetriever
AgentQnA
AudioQnA
FaqGen
MultimodalQnA
ProductivitySuite
GenAI Microservices
Generative AI Components (GenAIComps)
GenAIComps
Installation
MicroService
MegaService
Contributing to OPEA
Additional Content
Legal Information
License
Citation
Agent Microservice
Deploy Agent microservice on Kubernetes cluster
Deploy on Kubernetes
Agent Microservice
1. Overview
🚀2. Start Agent Microservice
🚀 3. Validate Microservice
🚀 4. Provide your own tools
5. Customize agent strategy
Plan Execute
RAG Agent
SQL Agents
Overview of sql_agent_llama
Overview of sql_agent
Limitations
Animation Microservice
Avatar Animation Microservice
🚀1. Start Microservice with Docker (option 1)
1.1 Build the Docker images
1.2. Set environment variables
🚀2. Run the Docker container
2.1 Run Wav2Lip Microservice
2.2 Run Animation Microservice
🚀3. Validate Microservice
3.1 Validate Wav2Lip service
3.2 Validate Animation service
Asr Microservice
Deploy ASR microservice on Kubernetes cluster
Deploy on Kubernetes
ASR Microservice
🚀1. Start Microservice with Python (Option 1)
🚀2. Start Microservice with Docker (Option 2)
Chathistory Microservice
📝 Chat History Microservice
🛠️ Features
⚙️ Implementation
Deploy chathistory microservice on Kubernetes cluster
Deploy on Kubernetes
📝 Chat History Microservice with MongoDB
Setup Environment Variables
🚀Start Microservice with Docker
✅ Invoke Microservice
Cores Microservice
Telemetry for OPEA
Metrics
Tracing
Visualization
Dataprep Microservice
Dataprep Microservice
Install Requirements
Use LVM (Large Vision Model) for Summarizing Image Data
Dataprep Microservice with Redis
Dataprep Microservice with Milvus
Dataprep Microservice with Qdrant
Dataprep Microservice with Pinecone
Dataprep Microservice with PGVector
Dataprep Microservice with VDMS
Dataprep Microservice with Multimodal
Deploy dataprep microservice on Kubernetes cluster
Deploy on Kubernetes with redis VectorDB
Deploy on Kubernetes with milvus VectorDB
Dataprep Microservice with Elasticsearch
🚀1. Start Microservice with Python（Option 1）
🚀2. Start Microservice with Docker (Option 2)
🚀3. Consume Microservice
Dataprep Microservice with Milvus
🚀1. Start Microservice with Python (Option 1)
🚀2. Start Microservice with Docker (Option 2)
🚀3. Consume Microservice
🚀4. Troubleshooting
Dataprep Microservice for Multimodal Data with Redis
🚀1. Start Microservice with Python（Option 1）
🚀2. Start Microservice with Docker (Option 2)
🚀3. Status Microservice
🚀4. Consume Microservice
Dataprep Microservice with Neo4J
🚀Start Microservice with Python
🚀Start Microservice with Docker
Invoke Microservice
Dataprep Microservice with Neo4J
Setup Environment Variables
🚀Start Microservice with Docker
Invoke Microservice
Dataprep Microservice with OpenSearch
🚀1. Start Microservice with Python（Option 1）
🚀2. Start Microservice with Docker (Option 2)
🚀3. Status Microservice
🚀4. Consume Microservice
Dataprep Microservice with PGVector
🚀1. Start Microservice with Python（Option 1）
🚀2. Start Microservice with Docker (Option 2)
🚀3. Consume Microservice
Dataprep Microservice with Pinecone
🚀Start Microservice with Python
🚀Start Microservice with Docker
Invoke Microservice
Dataprep Microservice with Qdrant
🚀Start Microservice with Python
🚀Start Microservice with Docker
Invoke Microservice
Dataprep Microservice with Redis
🚀1. Start Microservice with Python（Option 1）
🚀2. Start Microservice with Docker (Option 2)
🚀3. Status Microservice
🚀4. Consume Microservice
Dataprep Microservice
Install Requirements
Use LVM (Large Vision Model) for Summarizing Image Data
Dataprep Microservice with VDMS
🚀1. Start Microservice with Python (Option 1)
🚀2. Start Microservice with Docker (Option 2)
🚀3. Status Microservice
🚀4. Consume Microservice
Multimodal Dataprep Microservice with VDMS
🚀1. Start Microservice with Python (Option 1)
🚀2. Start Microservice with Docker (Option 2)
🚀3. Status Microservice
🚀4. Consume Microservice
Embeddings Microservice
Deploy Embedding microservice on Kubernetes cluster
Deploy on Kubernetes
Embeddings Microservice
Feedback_management Microservice
🗨 Feedback Management Microservice
🛠️ Features
⚙️ Implementation
🗨 Feedback Management Microservice with MongoDB
Setup Environment Variables
🚀Start Microservice with Docker
Finetuning Microservice
Fine-tuning Microservice
🚀1. Start Microservice with Python (Option 1)
🚀2. Start Microservice with Docker (Option 2)
🚀3. Consume Finetuning Service
🚀4. Descriptions for Finetuning parameters
Guardrails Microservice
Trust and Safety with LLM
Deploy guardrails microservice on Kubernetes cluster
Deploy on Kubernetes
Bias Detection Microservice
Introduction
Future Development
🚀1. Start Microservice with Python（Option 1）
🚀2. Start Microservice with Docker (Option 2)
🚀3. Get Status of Microservice
🚀4. Consume Microservice Pre-LLM/Post-LLM
Factuality Check Prediction Guard Microservice
🚀 Start Microservice with Docker
Setup Environment Variables
Build Docker Images
Start Service
🚀 Consume Factuality Check Service
Guardrails Microservice
LlamaGuard
WildGuard
Hallucination Detection Microservice
Introduction
🚀1. Start Microservice based on vLLM endpoint on Intel Gaudi Accelerator
2. Set up Hallucination Microservice
🚀3. Get Status of Hallucination Microservice
🚀4. Consume Guardrail Micorservice Post-LLM
PII Detection Prediction Guard Microservice
🚀 Start Microservice with Docker
Setup Environment Variables
Build Docker Images
Start Service
🚀 Consume PII Detection Service
Prompt Injection Detection Prediction Guard Microservice
🚀 Start Microservice with Docker
Setup Environment Variables
Build Docker Images
Start Service
🚀 Consume Prompt Injection Detection Service
Toxicity Detection Microservice
Introduction
🚀1. Start Microservice with Python（Option 1）
🚀2. Start Microservice with Docker (Option 2)
🚀3. Get Status of Microservice
🚀4. Consume Microservice Pre-LLM/Post-LLM
Image2image Microservice
Image-to-Image Microservice
🚀1. Start Microservice with Python (Option 1)
1.1 Install Requirements
1.2 Start Image-to-Image Microservice
🚀2. Start Microservice with Docker (Option 2)
2.1 Build Images
2.2 Start Image-to-Image Service
3 Test Image-to-Image Service
Image2video Microservice
Image-to-Video Microservice
🚀1. Start Microservice with Python (Option 1)
1.1 Install Requirements
1.2 Start Image-to-Video Microservice
🚀2. Start Microservice with Docker (Option 2)
2.1 Build Images
2.2 Start Image-to-Video Service
2.3 Test
Intent_detection Microservice
Intent Detection Microservice by TGI
🚀1. Start Microservice with Python（Option 1）
🚀2. Start Microservice with Docker (Option 2)
🚀3. Consume Microservice
Llms Microservice
Deploy LLM microservice on Kubernetes cluster
Deploy on Kubernetes
Document Summary LLM Microservice
🚀1. Start Microservice with Docker 🐳
🚀3. Consume LLM Service
FAQGen LLM Microservice
🚀1. Start Microservice with Docker
🚀2. Consume LLM Service
TGI LLM Microservice
🚀1. Start Microservice with Python (Option 1)
🚀2. Start Microservice with Docker (Option 2)
🚀3. Consume LLM Service
LLM Native Microservice
🚀1. Start Microservice
🚀2. Consume LLM Service
Introduction
Get Started
Build Docker Image
Run the Ollama Microservice
Consume the Ollama Microservice
Prediction Guard Introduction
Get Started
Consume the Prediction Guard Microservice
LLM Microservice
Validated LLM Models
Clone OPEA GenAIComps
Prerequisites
🚀1. Start Microservice with Python (Option 1)
🚀2. Start Microservice with Docker (Option 2)
🚀3. Consume LLM Service
LM-Eval Microservice
CPU service
Lvms Microservice
Deploy LVM microservice on Kubernetes cluster
Deploy on Kubernetes
LVM Microservice
Build Image & Run
Test
LVM Microservice
🚀 Start Microservice with Docker
LVM Microservice
🚀1. Start Microservice with Python (Option 1)
🚀2. Start Microservice with Docker (Option 2)
LVM Prediction Guard Microservice
🚀1. Start Microservice with Python
🚀2. Start Microservice with Docker (Option 2)
🚀3. Consume LVM Service
LVM Microservice
🚀1. Start Microservice with Docker
✅ 2. Test
♻️ 3. Clean
Prompt_registry Microservice
🧾 Prompt Registry Microservice
🛠️ Features
⚙️ Implementation
Deploy prompt microservice on Kubernetes cluster
Deploy on Kubernetes
🧾 Prompt Registry Microservice with MongoDB
Setup Environment Variables
🚀Start Microservice with Docker
Rerankings Microservice
Deploy reranking microservice on Kubernetes cluster
Deploy on Kubernetes
Reranking Microservice
Retrievers Microservice
Deploy retriever microservice on Kubernetes cluster
Deploy on Kubernetes with redis vector DB
Deploy on Kubernetes with milvus vector DB
Retriever Microservice
Text2image Microservice
Text-to-Image Microservice
🚀1. Start Microservice with Python (Option 1)
1.1 Install Requirements
1.2 Start Text-to-Image Microservice
🚀2. Start Microservice with Docker (Option 2)
2.1 Build Images
2.2 Start Text-to-Image Service
3 Test Text-to-Image Service
Text2sql Microservice
🛢 Text-to-SQL Microservice
🛠️ Features
⚙️ Implementation
Third_parties Microservice
Multimodal Embeddings Microservice with BridgePower
🚀1. Start MMEI on Gaudi2 HPU
🚀2. Start MMEI on Xeon CPU
🚀3. Access the service
Multimodal Embeddings Microservice with CLIP
🚀1. Start Microservice with Docker
🚀2. Consume Embedding Service
Start Elasticsearch server
1. Download Elasticsearch image
2. Run Elasticsearch service
Deploy gpt-sovits on kubernetes cluster
Deploy on Xeon
Start Milvus server
1. Configuration
2. Run Milvus service
Deploy MongoDB on kubernetes cluster
Deploy on Xeon
Deploy nginx on kubernetes cluster
Deploy on Xeon
Start Opensearch server
Prerequisites
Instructions
Troubleshooting
Start the Pathway Vector DB Server
Configuration
Building and running
Health check the vector store
Start PGVector server
1. Download Pgvector image
2. Configure the username, password and dbname
3. Run Pgvector service
Deploy RedisDB on kubernetes cluster
Deploy on Xeon
Start Redis server
1. Download Redis image
2. Run Redis service
Deploy speecht5 on kubernetes cluster
Deploy on Xeon
Deploy on Gaudi
Deploy TEI on kubernetes cluster
Deploy on Xeon
Deploy on Gaudi
Deploy TEIRERANK on kubernetes cluster
Deploy on Xeon
Deploy on Gaudi
Deploy TGI on kubernetes cluster
Deploy on Xeon
Deploy on Gaudi
Start VDMS server
1. Download VDMS image
2. Run VDMS service
vLLM Endpoint Service
🚀1. Set up Environment Variables
🚀2. Set up vLLM Service
🚀3. Set up LLM microservice
Deploy vllm on kubernetes cluster
Deploy on Xeon
Deploy on Gaudi
Deploy whisper on kubernetes cluster
Deploy on Xeon
Deploy on Gaudi
Tts Microservice
Deploy tts microservice on Kubernetes cluster
Deploy on Kubernetes
TTS Microservice
1.2 Start SpeechT5 Service/Test
1.3 Start TTS Service/Test
🚀2. Start Microservice with Docker (Option 2)
GPT-SoVITS Microservice
Build the Image
Start the Service
Test
Web_retrievers Microservice
Deploy web-retriever microservice on Kubernetes cluster
Deploy on Kubernetes
Web Retriever Microservice
Start Microservice with Docker
Deploying GenAI
GenAIInfra
Overview
Prerequisite
Setup Kubernetes cluster
(Optional) To run GenAIInfra on Intel Gaudi product
Usages
Use GenAI Microservices Connector (GMC) to deploy and adjust GenAIExamples
Use helm charts to deploy
Use terraform to deploy on cloud service providers
Additional Content
Development
Prerequisites
Testing
pre-commit testing
Legal Information
License
Citation
Release Branches
1. Create release candidate branch
2. Create images with release tag
3. Test helm charts
4. Test GMC
5. Publish images
Installation Guides
GenAI-microservices-connector(GMC) Installation
GenAI-microservices-connector(GMC)
Install GMC
Use GMC to compose a chatQnA Pipeline
Kubernetes Installation Options
Kubernetes Installation using AWS EKS Cluster
Prerequisites
Create AWS EKS Cluster in AWS Console
Uploading images to an AWS Private Registry
Kubernetes installation demo using kubeadm
Node configuration
Step 0. Clean up the environment
Step 1. Install relevant components
Step 2. Create the k8s cluster
Step 3 (optional) Reset Kubernetes cluster
NOTES
Kubernetes installation using Kubespray
Node preparation
Prerequisites
Step 1. Set up Kubespray and Ansible
Step 2. Build your own inventory
Step 3. Define Kubernetes configuration
Step 4. Deploy Kubernetes
Step 5. Create kubectl configuration
Quick reference
Cloud Service Provider
OPEA applications Azure AKS deployment guide
Prerequisites
Setup
AKS cluster
Persistent Volume Claim
OPEA Applications
Authentication and Authorization
Authentication and authorization
Istio based implementation for cloud native environments
APISIX based implementation for cloud native environments
Authentication and Authorization with APISIX and OIDC based Identity provider (Keycloak)
Prerequisites
Update values
Install
Usage
Uninstall
Leveraging Istio to compose an OPEA Pipeline with authentication and authorization enabled
Prerequisite
Perform authentication and authorization via Bearer JWT tokens and curl
Perform authentication and authorization via oauth2-proxy and OIDC provider and UI
Helm Charts
Helm charts for deploying GenAI Components and Examples
Table of Contents
Helm Charts
Deploy with Helm charts
Helm Charts Options
Using HPA (autoscaling)
Using Persistent Volume
Using Private Docker Hub
Generate manifests from Helm Charts
CI guidelines for helm charts
Table of Contents
Infra Setup
Add new test case
HorizontalPodAutoscaler (HPA) support
Table of Contents
Introduction
Pre-conditions
Gotchas
Enable HPA
Verify
Monitoring support
Table of Contents
Introduction
Pre-conditions
Install
Verify
agent
Deploy
Verify
Options
asr
(Option1): Installing the chart separately
(Option2): Installing the chart with dependencies automatically
Verify
Values
chathistory-usvc
(Option1): Installing the chart separately
(Option2): Installing the chart with dependencies automatically
Verify
Values
data-prep
(Option1): Installing the chart separately
(Option2): Installing the chart with dependencies automatically
Verify
Values
Milvus support
embedding-usvc
Installing the chart
Verify
Values
gpt-sovits
Install the chart
Verify
Values
guardrails-usvc
Installing the chart
Verify
Values
llm-uservice
Installing the chart
Verify
Values
OPEA lvm-serve microservice
Installing the Chart
Verify
Values
lvm-uservice
Installing the chart
Verify
Values
OPEA mm-embedding microservice
Installing the Chart
Verify
Values
mongodb
Install the Chart
Verify
Values
OPEA nginx microservice
Using nginx chart
prompt-usvc
(Option1): Installing the chart separately
(Option2): Installing the chart with dependencies automatically
Verify
Values
redis-vector-db
Install the Chart
Verify
Values
reranking-usvc
(Option1): Installing the chart separately
(Option2): Installing the chart with dependencies automatically
Verify
Values
retriever-usvc
Installing the chart
Verify
Values
Milvus support
speecht5
Installing the Chart
Verify
Values
tei
Installing the Chart
Verify
Values
teirerank
Installing the Chart
Verify
Values
OPEA text2image microservice
Installing the Chart
Verify
Values
tgi
Installing the Chart
Verify
Values
tts
(Option1): Installing the chart separately
(Option2): Installing the chart with dependencies automatically
Verify
Values
vllm
Installing the Chart
Verify
Values
web-retriever
(Option1): Installing the chart separately
(Option2): Installing the chart with dependencies automatically
Verify
Values
whisper
Installing the Chart
Verify
Values
AgentQnA
Deploy
Verify
AudioQnA
Installing the Chart
Verify
Values
ChatQnA
Installing the Chart
Verify
Values
Troubleshooting
ChatQnA Troubleshooting
a function to get the endpoint of service
define the namespace of service
Update a file to database
get the embedding of input
get the retriever docs
reranking the docs
TGI Q and A
REF
CodeGen
Installing the Chart
Verify
Values
CodeTrans
Installing the Chart
Verify
Values
DocSum
Installing the Chart
Verify
Values
FaqGen
Verify
Values
SearchQnA
Installing the Chart
Verify
Values
VisualQnA
Verify
Values
Kubernetes Addons
Deploy Kubernetes add-ons for OPEA
How-To Setup Observability for OPEA Workload in Kubernetes
Prepare
1. Setup Prometheus & Grafana
2. Metrics for Gaudi Hardware (v1.16.2)
3. Metrics for OPEA applications
4. Metrics for PCM (Intel® Performance Counter Monitor)
More dashboards
OPEA with Istio
Introduction
Deployment
Create Istio gateway with TLS and virtual service
Enforce mTLS between OPEA pods
Cleanup
memory bandwidth exporter
Setup
More flags about memory bandwidth exporter
Microservices Connector
genai-microservices-connector(GMC)
Description
Architecture
Personas
Getting Started
Troubleshooting GMC Custom Resource（CR）
Usage guide for genai-microservices-connector(GMC)
Use GMC to compose a chatQnA Pipeline
Use GMC to adjust the chatQnA Pipeline
Use GMC to delete the chatQnA Pipeline
Use GMC and Istio to compose an OPEA Pipeline with authentication and authorization enabled
ChatQnA Use Cases in Kubernetes Cluster via GMC
Using prebuilt images
Deploy ChatQnA pipeline
Helm chart for genai-microservices-connector(GMC)
Installing the GMC Helm Chart
Check the installation result
Next step
Uninstall
Pipeline Proxy
OPEA Pipeline Proxy
Features
Build
Deployment
Development
Guardrails
Architecture
Deployment
Scripts
Scripts and tools
NVIDIA GPU Quick-Start Guide
Prerequisite
Usages
FAQ and Troubleshooting
Deploy Autoscaling Ray Cluster with KubeRay in Kubernetes Cluster
Install KubeRay
Start Ray Cluster with Autoscaling
Delete Ray Cluster with Autoscaling
Uninstall KubeRay
Evaluating GenAI
GenAIEval
Installation
Evaluation
lm-evaluation-harness
bigcode-evaluation-harness
Kubernetes platform optimization
Benchmark
Features
How to use
Grafana Dashboards
Additional Content
Legal Information
License
Citation
Kubernetes Platform Optimization with Resource Management
Introduction
NRI Plugins
Install
Validate policy status
Configure
Validate CPU affinity and hardware alignment in containers
Remove a policy
NRI topology-aware resource policy
OPEA Benchmark Tool
Features
Table of Contents
Installation
Prerequisites
Usage
Configuration
Test Suite Configuration
Test Cases
Auto-Tuning for ChatQnA: Optimizing Resource Allocation in Kubernetes
Key Features
Usage
Configuration Files
Hardrware_info.json
chatqna_neuralchat_rerank_latest.yaml
Tuning Config Parameters
Output
Auto-Tuning for ChatQnA: Optimizing Accuracy by Tuning Model Related Parameters
Prepare Dataset
Run the Tuning script
Setup Prometheus and Grafana to visualize microservice metrics
1. Setup Prometheus
1.1 Node Metrics (optional)
1.2 Intel® Gaudi® Metrics (optional)
2. Setup Grafana
3. Import Grafana Dashboard
StressCli
stresscli.py
Prerequirements
Installation
Usage
How to benchmark pubmed datasets by send query randomly
1. prepare the pubmed datasets
1.1 get pubmed data
1.2 use script to extract data
1.3 prepare 4 dataset files
1.4 prepare the query list
2. How to use pubmed qlist
locust scripts for OPEA ChatQnA
Configuration file
Basic Usage
HELMET: How to Evaluate Long-context Language Models Effectively and Thoroughly
Quick Links
Setup
Data
Running evaluation
Model-based evaluation
Adding new models
Adding new tasks
Others
Contacts
Citation
CRAG Benchmark for Agent QnA systems
Overview
Getting started
CRAG dataset
Launch agent QnA system
Run CRAG benchmark
Use LLM-as-judge to grade the answers
Validation of LLM-as-judge
Benchmark results for OPEA RAG Agents
Comparison with GPT-4o-mini
AutoRAG to evaluate the RAG system performance
Service preparation
RAG evaluation
Notes
🚀 QuickStart
Installation
Launch Service of LLM-as-a-Judge
Writing your first test case
Acknowledgements
🚀 QuickStart
Installation
Launch a LLM Service
Example 1: TGI
Example 2: OPEA LLM
Predict
Evaluate
Evaluation Methodology
Introduction
Prerequisite
Environment
MultiHop (English dataset)
Launch Service of RAG System
Launch Service of LLM-as-a-Judge
Prepare Dataset
Evaluation
CRUD (Chinese dataset)
Prepare Dataset
Launch Service of RAG System
Evaluation
Acknowledgements
Metric Card for BLEU
Metric Description
Intended Uses
How to Use
Inputs
Output Values
Examples
Limitations and Bias
Citation
Further References
RAGAAF (RAG assessment - Annotation Free)
Key features
Run RAGAAF
1. Data
2. Launch endpoint on Gaudi
3. Model
4. Metrics
5. Evaluation
Customizations
OPEA adaption of ragas (LLM-as-a-judge evaluation of Retrieval Augmented Generation)
User data
Launch HuggingFace endpoint on Intel’s Gaudi machines
Run OPEA ragas pipeline using your desired list of metrics
Troubleshooting
Developer Guides
Coding Guides
OPEA API Service Spec (v1.0)
OPEA Mega Service API
OPEA Micro Service API
Documentation Guides
Documentation Guidelines
Markdown vs. RestructuredText
Documentation Organization
Headings
Content Highlighting
Lists
Multi-Column Lists
Tables
File Names and Commands
Branch-Specific File Links
Internal Cross-Reference Linking
Non-ASCII Characters
Include Content from Other Files
Code and Command Examples
Images
Tabs, Spaces, and Indenting
Background Colors
Drawings
Alternative Tabbed Content
Instruction Steps
First Instruction Step
Second Instruction Step
Documentation Generation
Drawings Using Graphviz
Simple Directed Graph
Adding Edge Labels
Tables
Finite-State Machine
OPEA Documentation Generation
Documentation Overview
Set Up the Documentation Working Folders
Install the Documentation Tools
Documentation Presentation Theme
Run the Documentation Processors
Doc Build Troubleshooting
Publish Content
Document Versioning
Filter Expected Warnings
OPEA Community
Community Support
Resources
Contributing Guides
Contribution Guidelines
All The Ways To Contribute
Support
Contributor Covenant Code of Conduct
OPEA Project Code Owners
docs Repository Code Owners
GenAIComps Repository Code Owners
GenAIEval Repository Code Owners
GenAIExamples Repository Code Owners
GenAIInfra Repository Code Owners
Continuous Integration (CICD) owners
Reporting a Vulnerability
Script Usage Notice
Roadmaps
OPEA 2024 - 2025 Roadmap
May 2024
June 2024
July 2024
Aug 2024
Sep 2024
Q4 2024
Q1 2025
OPEA CI/CD Roadmap
Milestone 1 (May, Done)
Milestone 2 (June)
Milestone 3 (July)
Milestone 4 (Aug)
Project Governance
Technical Charter (the “Charter”) for OPEA a Series of LF Projects, LLC
1. Mission and Scope of the Project
2. Technical Steering Committee
3. TSC Voting
4. Compliance with Policies
5. Community Assets
6. General Rules and Operations.
7. Intellectual Property Policy
8. Amendments
Technical Steering Committee (TSC)
Technical Steering Committee Members
Contributor Covenant Code of Conduct
Our Pledge
Our Standards
Enforcement Responsibilities
Scope
Enforcement
Enforcement Guidelines
Attribution
Reporting a Vulnerability
Script Usage Notice
RFC Proposals
Request for Comments (RFCs)
24-05-16 GenAIExamples-001 Using MicroService to Implement ChatQnA
24-05-16 OPEA-001 Overall Design
24-05-24 OPEA-001 Code Structure
24-06-21-OPEA-001-DocSum_Video_Audio
24-06-21-OPEA-001-Guardrails-Gateway
24-07-11-OPEA-Agent
24-08-02-OPEA-AIAvatarChatbot
24-08-07 OPEA-001 OPEA GenAIStudio
24-08-20-OPEA-001-AI Gateway API
24-08-21-GenAIExample-002-Edge Craft RAG
24-10-02-GenAIExamples-001-Image_and_Audio_Support_in_MultimodalQnA
RFC Template
Release Notes
OPEA Release Guide
Release Cadence
General Overview
Feature Freeze
Code/Doc Freeze, and Creating the RC Branch
Preparing Creating RC Branch
Creating RC Branch
Cherry Pick Critical Code/Doc Fix
How to do Cherry Picking
Creating Tag from RC Branch
Deliver Docker Images, Helm Charts, and PyPi Binaries
OPEA Release Notes v1.1
What’s New in OPEA v1.1
Highlights
Notable Changes
Full Changelogs
Contributors
Contributing Organizations
Individual Contributors
OPEA Release Notes v1.0
What’s New in OPEA v1.0
Details
OPEA Release Notes v0.9
What’s New in OPEA v0.9
Details
OPEA Release Notes v0.8
What’s New in OPEA v0.8
Details
Thanks to these contributors
OPEA Release Notes v0.7
OPEA Highlights
GenAIExamples
GenAIComps
GenAIEvals
GenAIInfra
OPEA Release Notes v0.6
OPEA Highlight
GenAIExamples
GenAIComps
GenAIEvals
GenAIInfra
Contributing to OPEA
Additional Content
OPEA Frequently Asked Questions
What is OPEA’s mission?
What is OPEA?
What problems are faced by GenAI deployments within the enterprise?
Why now?
How does it compare to other options for deploying Gen AI solutions within the enterprise?
Will OPEA reference implementations work with proprietary components?
What does OPEA acronym stand for?
How do I pronounce OPEA?
What initial companies and open-source projects joined OPEA?
What is Intel contributing?
When you say Technical Conceptual Framework, what components are included?
What are the different ways partners can contribute to OPEA?
Where can partners see the latest draft of the Conceptual Framework spec?
Is there a cost for joining?
Do I need to be a Linux Foundation member to join?
Where can I report a bug or vulnerability?
OPEA™
Latest
»
Release Notes
View page source
Release Notes
¶
Release plan & guide.
OPEA Release Guide
Release notes for the current and previous releases are archived here.
OPEA Release Notes v1.1
OPEA Release Notes v1.0
OPEA Release Notes v0.9
OPEA Release Notes v0.8
OPEA Release Notes v0.7
OPEA Release Notes v0.6