Generative AI Examples¶

Introduction¶

GenAIExamples are designed to give developers an easy entry into generative AI, featuring microservice-based samples that simplify the processes of deploying, testing, and scaling GenAI applications. All examples are fully compatible with Docker and Kubernetes, supporting a wide range of hardware platforms such as Gaudi, Xeon, and NVIDIA GPU, and other hardwares, ensuring flexibility and efficiency for your GenAI adoption.

Architecture¶

GenAIComps is a service-based tool that includes microservice components such as llm, embedding, reranking, and so on. Using these components, various examples in GenAIExample can be constructed, including ChatQnA, DocSum, etc.

GenAIInfra, part of the OPEA containerization and cloud-native suite, enables quick and efficient deployment of GenAIExamples in the cloud.

GenAIEval measures service performance metrics such as throughput, latency, and accuracy for GenAIExamples. This feature helps users compare performance across various hardware configurations easily.

Getting Started¶

GenAIExamples offers flexible deployment options that cater to different user needs, enabling efficient use and deployment in various environments. Here’s a brief overview of the three primary methods: Python startup, Docker Compose, and Kubernetes.

Users can choose the most suitable approach based on ease of setup, scalability needs, and the environment in which they are operating.

Deployment Guide¶

Deployment are based on released docker images by default, check docker image list for detailed information. You can also build your own images following instructions.

Prerequisite¶

For Docker Compose based deployment, you should have docker compose installed. Refer to docker compose install.
For Kubernetes based deployment, we provide 3 ways from the easiest manifests to powerful GMC based deployment.
- You should have a kubernetes cluster ready for use. If not, you can refer to k8s install to deploy one.
- (Optional) You should have GMC installed to your kubernetes cluster if you want to try with GMC. Refer to GMC install for more information.
- (Optional) You should have Helm (version >= 3.15) installed if you want to deploy with Helm Charts. Refer to the Helm Installation Guide for more information.

Deploy Examples¶

Use Case	Docker Compose Deployment on Xeon	Docker Compose Deployment on Gaudi	Kubernetes with Manifests	Kubernetes with Helm Charts	Kubernetes with GMC
ChatQnA	Xeon Instructions	Gaudi Instructions	ChatQnA with Manifests	ChatQnA with Helm Charts	ChatQnA with GMC
CodeGen	Xeon Instructions	Gaudi Instructions	CodeGen with Manifests	CodeGen with Helm Charts	CodeGen with GMC
CodeTrans	Xeon Instructions	Gaudi Instructions	CodeTrans with Manifests	CodeTrans with Helm Charts	CodeTrans with GMC
DocSum	Xeon Instructions	Gaudi Instructions	DocSum with Manifests	DocSum with Helm Charts	DocSum with GMC
SearchQnA	Xeon Instructions	Gaudi Instructions	Not Supported	Not Supported	SearchQnA with GMC
FaqGen	Xeon Instructions	Gaudi Instructions	FaqGen with Manifests	Not Supported	FaqGen with GMC
Translation	Xeon Instructions	Gaudi Instructions	Not Supported	Not Supported	Translation with GMC
AudioQnA	Xeon Instructions	Gaudi Instructions	AudioQnA with Manifests	Not Supported	AudioQnA with GMC
VisualQnA	Xeon Instructions	Gaudi Instructions	VisualQnA with Manifests	Not Supported	VisualQnA with GMC
ProductivitySuite	Xeon Instructions	Not Supported	ProductivitySuite with Manifests	Not Supported	Not Supported

Supported Examples¶

Check here for detailed information of supported examples, models, hardwares, etc.