Deploying SearchQnA on AMD EPYC™ Processors¶

This document details the deployment process of the SearchQnA application on a single node, leveraging the GenAIComps microservices, optimized for AMD EPYC™ Processors.

Table of Contents¶

SearchQnA Quick Start Deployment
SearchQnA Docker Compose Files
Validate Microservices
Conclusion

SearchQnA Quick Start Deployment¶

This section describes how to quickly deploy and test the SearchQnA service manually on an AMD EPYC™ processor. The basic steps are:

Access the Code
Install Docker
Determine your host’s external IP address
Generate a HuggingFace Access Token
Configure the Deployment Environment
Deploy the Services Using Docker Compose
Check the Deployment Status
Validate the Pipeline
Cleanup the Deployment

Access the Code¶

Clone the GenAIExample repository and access the SearchQnA AMD EPYC™ platform Docker Compose files and supporting scripts:

git clone https://github.com/opea-project/GenAIExamples.git
cd GenAIExamples/SearchQnA/docker_compose/amd/cpu/epyc

Install Docker¶

Ensure Docker is installed on your system. If Docker is not already installed, use the provided script to set it up:

source ./install_docker.sh

This script installs Docker and its dependencies. After running it, verify the installation by checking the Docker version:

docker --version

If Docker is already installed, this step can be skipped.

Determine your host external IP address¶

Run the following command in your terminal to list network interfaces:

ifconfig

Look for the inet address associated with your active network interface (e.g., enp99s0). For example:

enp99s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
    inet 10.101.16.119  netmask 255.255.255.0  broadcast 10.101.16.255

In this example, the (host_ip) would be (10.101.16.119).

# Replace with your host's external IP address
export host_ip="your_external_ip_address"

Generate a HuggingFace Access Token¶

Some HuggingFace resources, such as some models, are only accessible if you have an access token. If you do not already have a HuggingFace access token, you can create one by first creating an account by following the steps provided at HuggingFace and then generating a user access token.

export HF_TOKEN="your_huggingface_token"

Configure the Deployment Environment¶

The model_cache directory, by default, stores models in the ./data directory. To change this, use the following command:

# Optional
export model_cache=/home/documentation/data_searchqna/data # Path to save cache models

To set up environment variables for deploying SearchQnA services, set up some parameters specific to the deployment environment and then source the set_env.sh script in this directory:

The environment variables GOOGLE_CSE_ID and GOOGLE_API_KEY must be set. To create an API key:

Open the (Google Cloud Console: Credentials.)[(https://console.cloud.google.com/apis/credentials)]
Click Create credentials → API key

To enable the Custom Search API:

To enable the Custom Search API on a Google account follow here

export GOOGLE_API_KEY="your google api key"
export GOOGLE_CSE_ID="your cse id"

export http_proxy="Your_HTTP_Proxy"           # http proxy if any
export https_proxy="Your_HTTPs_Proxy"         # https proxy if any
export no_proxy=localhost,127.0.0.1,$host_ip  # additional no proxies if needed
export NGINX_PORT=${your_nginx_port}          # your usable port for nginx, 80 for example

Finally set the other environment variables

source ./set_env.sh

Deploy the Services Using Docker Compose¶

To deploy the SearchQnA services, execute the docker compose up command with the appropriate arguments. For a default deployment, execute the command below. It uses the ‘compose.yaml’ file.

docker compose -f compose.yaml up -d

Note: developers should build docker image from source when:

Developing off the git main branch (as the container’s ports in the repo may be different > from the published docker image).

Unable to download the docker image.

Use a specific version of Docker image.

Please refer to the table below to build different microservices from source:

Microservice	Deployment Guide
Embedding	Embedding build guide
Retriever	Retriever build guide
Reranking	Reranking build guide
LLM	LLM build guide
MegaService	MegaService build guide
UI	Basic UI build guide

Check the Deployment Status¶

After running docker compose, check if all the containers launched via docker compose have started:

docker ps -a

For the default deployment, the following containers should have started

If any issues are encountered during deployment, refer to the Troubleshooting section.

Validate the Pipeline¶

Once the SearchQnA services are running, test the pipeline using the following command:

curl http://${host_ip}:3008/v1/searchqna -H "Content-Type: application/json" -d '{
     "messages": "What is the latest news? Give me also the source link.",
     "stream": "true"
     }'

Note : Access the SearchQnA UI by web browser through this URL: http://${host_ip}:80. Please confirm the 80 port is opened in the firewall. To validate each microservice used in the pipeline refer to the Validate Microservices section.

Cleanup the Deployment¶

To stop the containers associated with the deployment, execute the following command:

docker compose -f compose.yaml down

SearchQnA Docker Compose Files¶

When deploying a SearchQnA pipeline on an AMD EPYC™ platform, different large language model serving frameworks can be selected. The table below outlines the available configurations included in the application. These configurations can serve as templates and be extended to other components available in GenAIComps.

File	Description
compose.yaml	Default compose file using vllm as serving framework and redis as vector database

Validate Microservices¶

Embedding backend Service

 curl http://${host_ip}:3001/embed \
     -X POST \
     -d '{"inputs":"What is Deep Learning?"}' \
     -H 'Content-Type: application/json'

Embedding Microservice

 curl http://${host_ip}:3002/v1/embeddings\
   -X POST \
   -d '{"text":"hello"}' \
   -H 'Content-Type: application/json'

Web Retriever Microservice

 export your_embedding=$(python3 -c "import random; embedding = [random.uniform(-1, 1) for _ in range(768)]; print(embedding)")
 curl http://${host_ip}:3003/v1/web_retrieval \
   -X POST \
   -d "{\"text\":\"What is the 2024 holiday schedule?\",\"embedding\":${your_embedding}}" \
   -H 'Content-Type: application/json'

Reranking backend Service

 # TEI Reranking service
  curl http://${host_ip}:3004/rerank \
      -X POST \
      -d '{"query":"What is Deep Learning?", "texts": ["Deep Learning is not...", "Deep learning is..."]}' \
      -H 'Content-Type: application/json'

Reranking Microservice

  curl http://${host_ip}:3005/v1/reranking\
    -X POST \
    -d '{"initial_query":"What is Deep Learning?", "retrieved_docs": [{"text":"Deep Learning is not..."}, {"text":"Deep learning is..."}]}' \
    -H 'Content-Type: application/json'

LLM backend Service

 # TGI service
  curl http://${host_ip}:3006/generate \
    -X POST \
    -d '{"inputs":"What is Deep Learning?","parameters":{"max_new_tokens":17, "do_sample": true}}' \
    -H 'Content-Type: application/json'

LLM Microservice

 curl http://${host_ip}:3007/v1/chat/completions\
   -X POST \
   -d '{"query":"What is Deep Learning?","max_tokens":17,"top_k":10,"top_p":0.95,"typical_p":0.95,"temperature":0.01,"repetition_penalty":1.03,"stream":true}' \
   -H 'Content-Type: application/json'

MegaService

 curl http://${host_ip}:3008/v1/searchqna -H "Content-Type: application/json" -d '{
     "messages": "What is the latest news? Give me also the source link.",
     "stream": "true"
     }'

Nginx Service

curl http://${host_ip}:${NGINX_PORT}/v1/searchqna \
    -H "Content-Type: application/json" \
    -d '{
     "messages": "What is the latest news? Give me also the source link.",
     "stream": "true"
     }'

Conclusion¶

This guide should enable developer to deploy the default configuration or any of the other compose yaml files for different configurations. It also highlights the configurable parameters that can be set before deployment.