Deploying MultimodalQnA on AMD GPU (ROCm)¶
This document outlines the deployment process for a MultimodalQnA application utilizing the GenAIComps microservice pipeline on AMD server with ROCm GPUs. The steps include Docker image creation, container deployment via Docker Compose, and service execution to integrate microservices such as multimodal_embedding
that employs BridgeTower model as embedding model, multimodal_retriever
, lvm
, and multimodal-data-prep
.
Table of Contents¶
MultimodalQnA Quick Start Deployment¶
This section describes how to quickly deploy and test the MultimodalQnA service manually on an AMD GPU (ROCm) processor. The basic steps are:
Access the Code¶
Clone the GenAIExamples repository and access the MultimodalQnA AMD GPU (ROCm) platform Docker Compose files and supporting scripts:
git clone https://github.com/opea-project/GenAIExamples.git
cd GenAIExamples/MultimodalQnA
Then checkout a released version, such as v1.3:
git checkout v1.3
Configure the Deployment Environment¶
To set up environment variables for deploying MultimodalQnA services, set up some parameters specific to the deployment environment and source the set_env_*.sh
script in this directory:
if used vLLM - set_env_vllm.sh
if used TGI - set_env.sh
Set the values of the variables:
HOST_IP, HOST_IP_EXTERNAL - These variables are used to configure the name/address of the service in the operating system environment for the application services to interact with each other and with the outside world.
If your server uses only an internal address and is not accessible from the Internet, then the values for these two variables will be the same and the value will be equal to the server’s internal name/address.
If your server uses only an external, Internet-accessible address, then the values for these two variables will be the same and the value will be equal to the server’s external name/address.
If your server is located on an internal network, has an internal address, but is accessible from the Internet via a proxy/firewall/load balancer, then the HOST_IP variable will have a value equal to the internal name/address of the server, and the EXTERNAL_HOST_IP variable will have a value equal to the external name/address of the proxy/firewall/load balancer behind which the server is located.
We set these values in the file set_env****.sh
Variables with names like “******_PORT”** - These variables set the IP port numbers for establishing network connections to the application services. The values shown in the file set_env.sh or set_env_vllm they are the values used for the development and testing of the application, as well as configured for the environment in which the development is performed. These values must be configured in accordance with the rules of network access to your environment’s server, and must not overlap with the IP ports of other applications that are already in use.
Setting variables in the operating system environment:
export HF_TOKEN="Your_HuggingFace_API_Token"
source ./set_env_*.sh # replace the script name with the appropriate one
Consult the section on MultimodalQnA Service configuration for information on how service specific configuration parameters affect deployments.
Deploy the Services Using Docker Compose¶
To deploy the MultimodalQnA services, execute the docker compose up
command with the appropriate arguments. For a default deployment with TGI, execute the command below. It uses the ‘compose.yaml’ file.
cd docker_compose/amd/gpu/rocm
# if used TGI
docker compose -f compose.yaml up -d
# if used vLLM
# docker compose -f compose_vllm.yaml up -d
To enable GPU support for AMD GPUs, the following configuration is added to the Docker Compose file:
compose_vllm.yaml - for vLLM-based application
compose.yaml - for TGI-based
shm_size: 1g
devices:
- /dev/kfd:/dev/kfd
- /dev/dri/:/dev/dri/
cap_add:
- SYS_PTRACE
group_add:
- video
security_opt:
- seccomp:unconfined
This configuration forwards all available GPUs to the container. To use a specific GPU, specify its cardN and renderN device IDs. For example:
shm_size: 1g
devices:
- /dev/kfd:/dev/kfd
- /dev/dri/card0:/dev/dri/card0
- /dev/dri/render128:/dev/dri/render128
cap_add:
- SYS_PTRACE
group_add:
- video
security_opt:
- seccomp:unconfined
How to Identify GPU Device IDs:
Use AMD GPU driver utilities to determine the correct cardN
and renderN
IDs for your GPU.
Note: developers should build docker image from source when:
Developing off the git main branch (as the container’s ports in the repo may be different > from the published docker image).
Unable to download the docker image.
Use a specific version of Docker image.
Check the Deployment Status¶
Check if all the containers launched via docker compose have started:
docker ps -a
For the default deployment with TGI, the following 10 containers should have started:
CONTAINER ID IMAGE COMMAND STATUS PORTS NAMES
3bfa91d2ac8c opea/multimodalqna-ui:latest "docker-entrypoint.sh" Up 2 minutes 0.0.0.0:5173->5173/tcp multimodalqna-gradio-ui-server
1e93a6f60b7e opea/multimodalqna:latest "docker-entrypoint.sh" Up 2 minutes 0.0.0.0:8888->8888/tcp multimodalqna-backend-server
98b1a8a7ef23 opea/lvm:latest "docker-entrypoint.sh" Up 3 minutes 0.0.0.0:9399->9399/tcp lvm
a743dcdfb3d7 ghcr.io/huggingface/text-generation-inference:2.4.1-rocm "/entrypoint.sh ..." Up 3 minutes 0.0.0.0:8399->80/tcp tgi-llava-rocm-server
ba0f72a62e4b opea/retriever:latest "docker-entrypoint.sh" Up 3 minutes 0.0.0.0:7000->7000/tcp retriever-redis
e5a429aac4f7 opea/embedding:latest "docker-entrypoint.sh" Up 3 minutes 0.0.0.0:7061->7061/tcp embedding
ad25a3fc3cdd opea/embedding-multimodal-bridgetower:latest "python bridgetower..." Up 3 minutes 0.0.0.0:7050->7050/tcp embedding-multimodal-bridgetower
d834adc71bd4 opea/dataprep:latest "docker-entrypoint.sh" Up 3 minutes 0.0.0.0:6007->5000/tcp dataprep-multimodal-redis
4fd73dabc267 redis/redis-stack:7.2.0-v9 "redis-stack-server" Up 4 minutes 0.0.0.0:6379->6379/tcp, 8001->8001/tcp redis-vector-db
dfdf41dcd8e1 opea/whisper:latest "docker-entrypoint.sh" Up 4 minutes 0.0.0.0:7066->7066/tcp whisper-service
if used vLLM:
CONTAINER ID IMAGE COMMAND STATUS PORTS NAMES
cf3193a3e7c1 opea/multimodalqna-ui:latest "docker-entrypoint.sh" Up 2 minutes 0.0.0.0:5173->5173/tcp multimodalqna-gradio-ui-server
a14a529b06d2 opea/multimodalqna:latest "docker-entrypoint.sh" Up 2 minutes 0.0.0.0:8888->8888/tcp multimodalqna-backend-server
e91f81b6dc27 opea/lvm:latest "docker-entrypoint.sh" Up 3 minutes 0.0.0.0:9399->9399/tcp lvm
de5f2a4024bb opea/vllm-rocm:latest "/bin/sh -c '--model…" Up 3 minutes 0.0.0.0:8081->8011/tcp multimodalqna-vllm-service
f9918f9cba12 opea/retriever:latest "docker-entrypoint.sh" Up 3 minutes 0.0.0.0:7000->7000/tcp retriever-redis
d4a3a6e31fc2 opea/embedding:latest "docker-entrypoint.sh" Up 3 minutes 0.0.0.0:7061->7061/tcp embedding
9cd19b2fc1f0 opea/embedding-multimodal-bridgetower:latest "python bridgetower…" Up 3 minutes 0.0.0.0:7050->7050/tcp embedding-multimodal-bridgetower
b3e9135a9c23 opea/dataprep:latest "docker-entrypoint.sh" Up 3 minutes 0.0.0.0:6007->5000/tcp dataprep-multimodal-redis
ab00bc56fa67 redis/redis-stack:7.2.0-v9 "redis-stack-server" Up 4 minutes 0.0.0.0:6379->6379/tcp, 8001->8001/tcp redis-vector-db
d9f2172bb875 opea/whisper:latest "docker-entrypoint.sh" Up 4 minutes 0.0.0.0:7066->7066/tcp whisper-service
If any issues are encountered during deployment, refer to the Troubleshooting section.
Validate the Pipeline¶
Once the MultimodalQnA services are running, test the pipeline using the following command:
DATA='{"messages": [{"role": "user", "content": [{"type": "audio", "audio": "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}]}]}'
curl http://${HOST_IP}:8888/v1/multimodalqna \
-H "Content-Type: application/json" \
-d "$DATA"
Cleanup the Deployment¶
To stop the containers associated with the deployment, execute the following command:
# if used TGI
docker compose -f compose.yaml down
# if used vLLM
# docker compose -f compose_vllm.yaml down
MultimodalQnA Docker Compose Files¶
In the context of deploying a MultimodalQnA pipeline on an AMD GPU (ROCm) platform, we can pick and choose different large language model serving frameworks. The table below outlines the various configurations that are available as part of the application. These configurations can be used as templates and can be extended to different components available in GenAIComps.
File |
Description |
---|---|
Default compose file using TGI as serving framework |
|
The LLM serving framework is vLLM. All other configurations remain the same as the default |
Validate Microservices¶
embedding-multimodal-bridgetower¶
Text example:
curl http://${host_ip}:${EMM_BRIDGETOWER_PORT}/v1/encode \
-X POST \
-H "Content-Type:application/json" \
-d '{"text":"This is example"}'
Checking the response from the service. The response should be similar to text:
{"embedding":[0.036936961114406586,-0.0022056063171476126,0.0891181230545044,-0.019263656809926033,-0.049174826592206955,-0.05129311606287956,-0.07172256708145142,0.04365323856472969,0.03275766223669052,0.0059910244308412075,-0.0301326...,-0.0031989417038857937,0.042092420160770416]}
Image example:
curl http://${host_ip}:${EMM_BRIDGETOWER_PORT}/v1/encode \
-X POST \
-H "Content-Type:application/json" \
-d '{"text":"This is example", "img_b64_str": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC"}'
Checking the response from the service. The response should be similar to text:
{"embedding":[0.024372786283493042,-0.003916610032320023,0.07578050345182419,...,-0.046543147414922714]}
embedding¶
Text example:
curl http://${host_ip}:$MM_EMBEDDING_PORT_MICROSERVICE/v1/embeddings \
-X POST \
-H "Content-Type: application/json" \
-d '{"text" : "This is some sample text."}'
Checking the response from the service. The response should be similar to text:
{"id":"4fb722012a2719e38188190e1cb37ed3","text":"This is some sample text.","embedding":[0.043303076177835464,-0.051807764917612076,...,-0.0005179636646062136,-0.0027774290647357702],"search_type":"similarity","k":4,"distance_threshold":null,"fetch_k":20,"lambda_mult":0.5,"score_threshold":0.2,"constraints":null,"url":null,"base64_image":null}
Image example:
curl http://${host_ip}:${EMM_BRIDGETOWER_PORT}/v1/encode \
-X POST \
-H "Content-Type:application/json" \
-d '{"text":"This is example", "img_b64_str": "iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mP8/5+hnoEIwDiqkL4KAcT9GO0U4BxoAAAAAElFTkSuQmCC"}'
Checking the response from the service. The response should be similar to text:
{"id":"cce4eab623255c4c632fb920e277dcf7","text":"This is some sample text.","embedding":[0.02613169699907303,-0.049398183822631836,...,0.03544217720627785],"search_type":"similarity","k":4,"distance_threshold":null,"fetch_k":20,"lambda_mult":0.5,"score_threshold":0.2,"constraints":null,"url":"https://github.com/docarray/docarray/blob/main/tests/toydata/image-data/apple.png?raw=true","base64_image":"iVBORw0KGgoAAAANSUhEUgAAAoEAAAJqCAMAAABjDmrLAAAABGdBTUEAALGPC/.../BCU5wghOc4AQnOMEJTnCCE5zgBCc4wQlOcILzqvO/ARWd2ns+lvHkAAAAAElFTkSuQmCC"}
retriever-multimodal-redis¶
set “your_embedding” variable:
export your_embedding=$(python3 -c "import random; embedding = [random.uniform(-1, 1) for _ in range(512)]; print(embedding)")
Test Redis retriever
curl http://${host_ip}:${REDIS_RETRIEVER_PORT}/v1/retrieval \
-X POST \
-H "Content-Type: application/json" \
-d "{\"text\":\"test\",\"embedding\":${your_embedding}}"
Checking the response from the service. The response should be similar to text:
{"id":"80a4f3fc5f5d5cd31ab1e3912f6b6042","retrieved_docs":[],"initial_query":"test","top_n":1,"metadata":[]}
whisper service¶
curl http://${host_ip}:7066/v1/asr \
-X POST \
-d '{"audio": "UklGRigAAABXQVZFZm10IBIAAAABAAEARKwAAIhYAQACABAAAABkYXRhAgAAAAEA"}' \
-H 'Content-Type: application/json'
Checking the response from the service. The response should be similar to text:
{"asr_result":"you"}
Conclusion¶
This guide should enable developer to deploy the default configuration or any of the other compose yaml files for different configurations. It also highlights the configurable parameters that can be set before deployment.