Deploy DocSum in Kubernetes Cluster¶
This document outlines the deployment process for a Document Summary (DocSum) application that utilizes the GenAIComps microservice components on Intel Xeon servers and Gaudi machines. The DocSum Service leverages a Kubernetes operator called genai-microservices-connector(GMC). GMC supports connecting microservices to create pipelines based on the specification in the pipeline yaml file, in addition it allows the user to dynamically control which model is used in a service such as an LLM or embedder. The underlying pipeline language also supports using external services that may be running in public or private clouds elsewhere.
Install GMC in your Kubernetes cluster, if you have not already done so, by following the steps in Section “Getting Started” at GMC Install. We will soon publish images to Docker Hub, at which point no builds will be required, further simplifying install.
The DocSum application is defined as a Custom Resource (CR) file that the above GMC operator acts upon. It first checks if the microservices listed in the CR yaml file are running, if not it starts them and then proceeds to connect them. When the DocSum RAG pipeline is ready, the service endpoint details are returned, letting you use the application. Should you use “kubectl get pods” commands you will see all the component microservices, in particular embedding, retriever, rerank, and llm.
The DocSum pipeline uses prebuilt images. The Xeon version uses the prebuilt image llm-docsum-tgi:latest
which internally leverages the
the image ghcr.io/huggingface/text-generation-inference:sha-e4201f4-intel-cpu
. The service is called tgi-svc. Meanwhile, the Gaudi version launches the
service tgi-gaudi-svc, which uses the image ghcr.io/huggingface/tgi-gaudi:2.0.5
. Both TGI model services serve the model specified in the LLM_MODEL_ID variable that is exported by you. In the below example we use Intel/neural-chat-7b-v3-3
.
[NOTE] Refer to Docker Xeon README or Docker Gaudi README to build the OPEA images. These will be available on Docker Hub soon, simplifying installation.
Deploy the RAG pipeline¶
This involves deploying the application pipeline custom resource. You can use docsum_xeon.yaml
if you have just a Xeon cluster or docsum_gaudi.yaml
if you have a Gaudi cluster.
Setup Environment variables. These are specific to the user. Skip the proxy settings if you are not operating behind one.
We use “Intel/neural-chat-7b-v3-3” as an example. If you want to use other models, change “LLM_MODEL_ID” in following setting and change “MODEL_ID” in manifests yaml file.
export no_proxy=${your_no_proxy} export http_proxy=${your_http_proxy} export https_proxy=${your_http_proxy} export LLM_MODEL_ID="Intel/neural-chat-7b-v3-3" export HUGGINGFACEHUB_API_TOKEN=${your_hf_api_token} export ns=${docsum}
Create namespace for the application and deploy it
kubectl create ns ${ns} kubectl apply -f $(pwd)/docsum_xeon.yaml
GMC will reconcile the custom resource and get all related components/services ready. Confirm the service status using below command
kubectl get service -n ${ns}
Obtain the custom resource/pipeline access URL
kubectl get gmconnectors.gmc.opea.io -n ${ns} NAME URL READY AGE docsum http://router-service.docsum.svc.cluster.local:8080 8/0/8 3m
Deploy a client pod to test the application
kubectl create deployment client-test -n ${ns} --image=python:3.8.13 -- sleep infinity
Access the pipeline using the above URL from the client pod and execute a request
export CLIENT_POD=$(kubectl get pod -n ${ns} -l app=client-test -o jsonpath={.items..metadata.name}) export accessUrl=$(kubectl get gmc -n $ns -o jsonpath="{.items[?(@.metadata.name=='docsum')].status.accessUrl}") kubectl exec "$CLIENT_POD" -n $ns -- curl -s --no-buffer $accessUrl -X POST -d '{"query":"Text Embeddings Inference (TEI) is a toolkit for deploying and serving open source text embeddings and sequence classification models. TEI enables high-performance extraction for the most popular models, including FlagEmbedding, Ember, GTE and E5."}' -H 'Content-Type: application/json'
Clean up. Use standard Kubernetes custom resource remove commands. Confirm cleaned by retrieving pods in application namespace.