Customize with new VectorDB

Please refer Contributing A New Vector Database to OPEA

Customize the VectorDB for the ChatQnA Example

The OPEA sub-project GenAIExamples houses multiple GenAI RAG sample applications such as chatbots, document summarization, code generation, and code translation to name a few. The ChatQnA application is the primary example and contains instructions to deploy on a variety of hardware (such as Intel CPUs and Gaudi accelerator and AMD’s ROCm), in environments such as Docker and Kubernetes, including how to customize an application pipeline using different vector database backends.

Add new VectorDB to ChatQnA Example

This deployment section covers how to add a new Vector DB to ChatQnA example with OPEA comps. Here we will be showcasing how to build an (end-to-end) e2e ChatQnA with a new VectorDB.

Overview

There are several ways to setup a ChatQnA use case with different VectorDBs. Here in this tutorial, we will walk through how to enable a new VectorDB with the below list of microservices from OPEA: GenAIComps to setup a ChatQnA.

1. Data Prep
2. Embedding
3. Retriever
4. Reranking
5. LLM with Ollama

To add a new VectorDB to OPEA involves minimal changes to OPEA sub-project GenAIComps that covers installation, launch, usage, and tests. The necessary customizations are covered in detail [here]

Prerequisites

We start by cloning the GenAIExamples and GenAIComps projects. GenAIComps is the fundamental and necessary component used to build the examples examples you find in GenAIExamples and deploy them as microservices. Next, set an environment variable for the desired release version with the number only (i.e. 1.0, 1.1, etc) and checkout using the tag with that version. The GenAIComps should contain the customized components(third_party, dataprep, retrievers) for the VectorDB as mentioned before.

# Set workspace
export WORKSPACE=<path>
cd $WORKSPACE

# Set desired release version - number only
export RELEASE_VERSION=<insert-release-version>

# GenAIComps
git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps
git checkout tags/v${RELEASE_VERSION}
cd ..
# GenAIExamples
git clone https://github.com/opea-project/GenAIExamples.git
cd GenAIExamples
git checkout tags/v${RELEASE_VERSION}
cd ..

To customize ChatQnA with the new VectorDB the changes are in GenAIExamples/ChatQnA

ChatQnA
|__docker_compose
   |__intel
      |__cpu/xeon
      |  |__compose_<Vector_DB>.yaml
      |  |__<Vector_DB>.yaml
      |  |__README_<Vector_DB>.md
      |__hpu/gaudi
         |__compose_<Vector_DB>.yaml
         |__README_<Vector_DB>.md

VectorDB.yaml adds the VectorDB specific configurations

compose_<Vector_DB>.yaml contains all the necessary configs to launch a ChatQnA pipeline with the VectorDB. The different microservices are configured in different sections

    services: VectorDB specific services and healthcheck

    dataprep-<Vector_DB>-service: this references opea-project/GenAIComps/comps/dataprep/ opea-project/GenAIComps/comps/dataprep/deployment/docker_compose/compose.yaml 

    retriever-<Vector_DB>-service: this references opea-project/GenAIComps/comps/retrievers/deployment/docker_compose/compose.yaml

    tei-embedding-service: references TEI component

    tei-reranking-service: references ReRanking component

    vllm-service: the inference Serving service

    chatqna-xeon-backend-server: For Xeon only

    chatqna-xeon-ui-server: for Xeon only

    chatqna-xeon-nginx-server: Load balancer for Xeon only

    chatqna-gaudi-backend-server: for Gaudi only

    chatqna-gaudi-ui-server: for Gaudi only

    chatqna-gaudi-nginx-server:Load balancer for Gaudi only

README_<Vector_DB>.md adds details to start the Mega service of ChatQnA on Xeon in respective folders

README_<Vector_DB>.md adds details to start the Mega service of ChatQnA on Gaudi in respective folders.

Following are the contents of README_<Vector_DB>.md

Build Mega Service of ChatQnA (with VectorDB)

This document outlines the deployment process for a ChatQnA application utilizing the GenAIComps microservice pipeline on Intel Xeon server. The steps include Docker image creation, container deployment via Docker Compose, and service execution to integrate microservices such as embedding, retriever, rerank, and llm.

Quick Start:

    1. Set up the environment variables.
    2. Run Docker Compose.
    3. Consume the ChatQnA Service.

The default pipeline deploys with vLLM as the LLM serving component and leverages the re-rank component.

Note: The default LLM is meta-llama/Meta-Llama-38B-Instruct. Before deploying the aplication, please make sure either you’ve requested and have been granted access to it on HuggingFace or you’ve downloaded the model locally from ModelScope.

Quick Start: 1. Setup Environment Variable

To set up environment variables for deploying ChatQnA services, follow these steps:

Set the required environment variables:

# Example: host_ip="192.168.1.1"
export host_ip="External_Public_IP"
export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"

If you are in a proxy environment, also set the proxy-related environment variables:

export http_proxy="Your_HTTP_Proxy"
export https_proxy="Your_HTTPs_Proxy"
# Example: no_proxy="localhost, 127.0.0.1, 192.168.1.1"
export no_proxy="Your_No_Proxy",chatqna-xeon-ui-server,chatqna-xeon-backend-server,dataprep-pinecone-service,tei-embedding-service,retriever,tei-reranking-service,tgi-service,vllm-service

Set up other environment variables, make sure to update the INDEX_NAME variable to Pinecone index value:

source ./set_env.sh

Quick Start: 2.Run Docker Compose

docker compose -f compose_<Vector_DB>.yaml up -d

It will automatically download the docker image on docker hub:

docker pull opea/chatqna:latest
docker pull opea/chatqna-ui:latest

Note: You should build docker image from source by yourself if:

• You are developing off the git main branch (as the container’s ports in the repo may be different from the published docker image).

• You can’t download the docker image.

• You want to use a specific version of Docker image.

QuickStart: 3.Consume the ChatQnA Service

curl http://${host_ip}:8888/v1/chatqna \
    -H "Content-Type: application/json" \
    -d '{
        "messages": "What is the revenue of Nike in 2023?"
    }'

Build Docker Images

First of all, you need to build Docker Images locally and install the python package of it.

git clone https://github.com/opea-project/GenAIComps.git
cd GenAIComps

Build Retriever Image

docker build --no-cache -t opea/retriever:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/retrievers/src/Dockerfile .

Build Dataprep Image

docker build --no-cache -t opea/dataprep:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/src/Dockerfile .
cd ..

Build MegaService Docker Image

Option 1. MegaService with Rerank To construct the Mega Service with Rerank, we utilize the GenAIComps microservice pipeline within the chatqna.py Python script. Build MegaService Docker image via below command:

```bash
git clone https://github.com/opea-project/GenAIExamples.git
cd GenAIExamples/ChatQnA
docker build --no-cache -t opea/chatqna:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile .
```

Option 2. MegaService without Rerank To construct the Mega Service without Rerank, we utilize the GenAIComps microservice pipeline within the chatqna_without_rerank.py Python script. Build MegaService Docker image via below command:

```bash
git clone https://github.com/opea-project/GenAIExamples.git
cd GenAIExamples/ChatQnA
docker build --no-cache -t opea/chatqna-without-rerank:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f Dockerfile.without_rerank .
```

Build UI Docker Image

Build frontend Docker image via below command:

cd GenAIExamples/ChatQnA/ui
docker build --no-cache -t opea/chatqna-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile .

Build Conversational React UI Docker Image (Optional)

Build frontend Docker image that enables Conversational experience with ChatQnA megaservice via below command:

cd GenAIExamples/ChatQnA/ui
docker build --no-cache -t opea/chatqna-conversation-ui:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f ./docker/Dockerfile.react .

Build Nginx Docker Image

cd GenAIComps
docker build -t opea/nginx:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/third_parties/nginx/src/Dockerfile .

Then run the command docker images, you will have the following 5 Docker Images:

1. `opea/dataprep:latest`
2. `opea/retriever:latest`
3. `opea/chatqna:latest` or `opea/chatqna-without-rerank:latest`
4. `opea/chatqna-ui:latest`
5. `opea/nginx:latest`

Start Microservices

Required Models

By default, the embedding, reranking and LLM models are set to a default value as listed below:

| Service   | Model                               |
| --------- | ----------------------------------- |
| Embedding | BAAI/bge-base-en-v1.5               |
| Reranking | BAAI/bge-reranker-base              |
| LLM       | meta-llama/Meta-Llama-3-8B-Instruct |

Change the xxx_MODEL_ID below for your needs. For users in China who are unable to download models directly from Huggingface, you can use ModelScope or a Huggingface mirror to download models. The vLLM can load the models either online or offline as described below:

1. Online

   export HF_TOKEN=${your_hf_token}
   export HF_ENDPOINT="https://hf-mirror.com"
   model_name="meta-llama/Meta-Llama-3-8B-Instruct"
   docker run -p 8008:80 -v ./data:/data --name vllm-service -e HF_ENDPOINT=$HF_ENDPOINT -e http_proxy=$http_proxy -e https_proxy=$https_proxy --shm-size 128g opea/vllm:latest --model $model_name --host 0.0.0.0 --port 80
   

2. Offline

   • Search your model name in ModelScope. For example, check this page for model Meta-Llama-3-8B-Instruct.

   • Click on Download this model button, and choose one way to download the model to your local path /path/to/model.

   • Run the following command to start the LLM service.

   export HF_TOKEN=${your_hf_token}
   export model_path="/path/to/model"
   docker run -p 8008:80 -v $model_path:/data --name vllm-service --shm-size 128g opea/vllm:latest --model /data --host 0.0.0.0 --port 80
   

Setup Environment Variables

1. Set the required environment variables:

   # Example: host_ip="192.168.1.1"
   export host_ip="External_Public_IP"
   export HUGGINGFACEHUB_API_TOKEN="Your_Huggingface_API_Token"
   # Example: NGINX_PORT=80
   export NGINX_PORT=${your_nginx_port}
   

2. If you are in a proxy environment, also set the proxy-related environment variables:

   export http_proxy="Your_HTTP_Proxy"
   export https_proxy="Your_HTTPs_Proxy"
   # Example: no_proxy="localhost, 127.0.0.1, 192.168.1.1"
   export no_proxy="Your_No_Proxy",chatqna-xeon-ui-server,chatqna-xeon-backend-server,dataprep-pinecone-service,tei-embedding-service,retriever,tei-reranking-service,tgi-service,vllm-service
   

3. Set up other environment variables:

   source ./set_env.sh
   

Start all the services

Before running the docker compose command, you need to be in the folder that has the docker compose yaml file

```bash
cd GenAIExamples/ChatQnA/docker_compose/intel/cpu/xeon/
```

Start ChatQnA with Rerank Pipeline

```bash
docker compose -f compose_<vectorDB>.yaml up -d
```

Validate Microservices

Note, when verify the microservices by curl or API from remote client, please make sure the ports of the microservices are opened in the firewall of the cloud node. Follow the instructions to validate MicroServices.

TEI Embedding Service

 curl ${host_ip}:6006/embed \
    -X POST \
    -d '{"inputs":"What is Deep Learning?"}' \
    -H 'Content-Type: application/json'

Retriever Microservice

To consume the retriever microservice, you need to generate a mock embedding vector by Python script. The length of embedding vector is determined by the embedding model. Here we use the model EMBEDDING_MODEL_ID="BAAI/bge-base-en-v1.5", which vector size is 768. Check the vector dimension of your embedding model, set your_embedding dimension equals to it.

export your_embedding=$(python3 -c "import random; embedding = [random.uniform(-1, 1) for _ in range(768)]; print(embedding)")
curl http://${host_ip}:7000/v1/retrieval \
  -X POST \
  -d "{\"text\":\"test\",\"embedding\":${your_embedding}}" \
  -H 'Content-Type: application/json'

TEI Reranking Service

Skip for ChatQnA without Rerank pipeline

 curl http://${host_ip}:8808/rerank \
    -X POST \
    -d '{"query":"What is Deep Learning?", "texts": ["Deep Learning is not...", "Deep learning is..."]}' \
    -H 'Content-Type: application/json'

LLM backend Service

In the first startup, this service will take more time to download, load and warm up the model. After it’s finished, the service will be ready. Try the command below to check whether the LLM serving is ready.

docker logs vllm-service 2>&1 | grep complete

If the service is ready, you will get the response like below.

INFO: Application startup complete.

Then try the cURL command below to validate services.

 curl http://${host_ip}:9009/v1/chat/completions \
     -X POST \
     -d '{"model": "meta-llama/Meta-Llama-3-8B-Instruct", "messages": [{"role": "user", "content": "What is Deep Learning?"}], "max_tokens":17}' \
     -H 'Content-Type: application/json'

MegaService

 curl http://${host_ip}:8888/v1/chatqna -H "Content-Type: application/json" -d '{
       "messages": "What is the revenue of Nike in 2023?"
       }'

Nginx Service

 curl http://${host_ip}:${NGINX_PORT}/v1/chatqna \
    -H "Content-Type: application/json" \
    -d '{"messages": "What is the revenue of Nike in 2023?"}'

Dataprep Microservice（Optional）

If you want to update the default knowledge base, you can use the following commands: Update Knowledge Base via Local File nke-10k-2023.pdf. Or click here to download the file via any web browser Or run this command to get the file on a terminal.

wget https://raw.githubusercontent.com/opea-project/GenAIComps/v1.1/comps/retrievers/redis/data/nke-10k-2023.pdf

Upload:

    curl -X POST "http://${host_ip}:6007/v1/dataprep/ingest" \
        -H "Content-Type: multipart/form-data" \
        -F "files=@./nke-10k-2023.pdf"

This command updates a knowledge base by uploading a local file for processing. Update the file path according to your environment. Add Knowledge Base via HTTP Links:

    curl -X POST "http://${host_ip}:6007/v1/dataprep/ingest" \
        -H "Content-Type: multipart/form-data" \
        -F 'link_list=["https://opea.dev"]'

This command updates a knowledge base by submitting a list of HTTP links for processing. To delete the files/link you uploaded:

    curl -X POST "http://${host_ip}:6009/v1/dataprep/delete" \
        -d '{"file_path": "all"}' \
        -H "Content-Type: application/json"

Tests for ChatQnA with new VectorDB

This should go under GenAIExamples/ChatQnA/tests

Test files to create - below examples give a skeleton for test files.

Tests for Xeon

test_compose_<Vector_DB>_on_xeon.sh

build_docker_images()
        echo "Building Docker Images...."
        
        if [ ! -d "GenAIComps" ] ; then
            git clone --single-branch --branch "${opea_branch:-"main"}" https://github.com/opea-project/GenAIComps.git
        fi
        
        service_list="dataprep embedding retriever reranking ChatQnA"
        docker compose -f build.yaml build ${service_list} --no-cache 
        docker pull ghcr.io/huggingface/text-embeddings-inference:cpu-1.5
        docker pull <Vector_DB> specific images
        docker images && sleep 1s
	    echo "Docker images built!"

start_services()
        echo "Starting Docker Services...."
   
        export EMBEDDING_MODEL_ID="BAAI/bge-base-en-v1.5"
        export RERANK_MODEL_ID="BAAI/bge-reranker-base"
        export TEI_EMBEDDING_ENDPOINT="http://${ip_address}:6006"
        export TEI_RERANKING_ENDPOINT="http://${ip_address}:8808"
        export TGI_LLM_ENDPOINT="http://${ip_address}:8008"
        export MILVUS_HOST=${ip_address}
        export HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
        export MEGA_SERVICE_HOST_IP=${ip_address}
        export EMBEDDING_SERVICE_HOST_IP=${ip_address}
        export RETRIEVER_SERVICE_HOST_IP=${ip_address}
        export RERANK_SERVICE_HOST_IP=${ip_address}
        export LLM_SERVICE_HOST_IP=${ip_address}
        export host_ip=${ip_address}
        export DATAPREP_SERVICE_ENDPOINT="http://${host_ip}:6007/v1/dataprep/ingest"
        export RERANK_TYPE="tei"
        export LOGFLAG=true

        # Start Docker Containers
        docker compose -f compose_<Vector_DB>.yaml up -d
        sleep 2m
        echo "Docker services started!"

validate_megaservice()
        echo "===========Ingest data=================="
        
        local CONTENT=$(http_proxy="" curl -X POST "http://${ip_address}:6007/v1/dataprep/ingest" \
        -H "Content-Type: multipart/form-data" \
        -F 'link_list=["https://opea.dev/"]')
        local EXIT_CODE=$(validate "$CONTENT" "Data preparation succeeded" "dataprep-<Vector_DB>-service-xeon")
        echo "$EXIT_CODE"

        # Curl the Mega Service
        echo "================Testing retriever service: Text Request ================"

        local CONTENT=$(http_proxy="" curl http://${ip_address}:8889/v1/retrievaltool -X POST -H "Content-Type: application/json" -d '{
        "text": "Explain the OPEA project?"
        }')

Tests for Gaudi

test_compose_<Vector_DB>_on_gaudi.sh

build_docker_images()
        echo "Building Docker Images...."
            
        if [ ! -d "GenAIComps" ] ; then
            git clone --single-branch --branch "${opea_branch:-"main"}" https://github.com/opea-project/GenAIComps.git
        fi
            
        service_list="dataprep embedding retriever reranking ChatQnA"
        docker compose -f build.yaml build ${service_list} --no-cache 
        docker pull ghcr.io/huggingface/text-embeddings-inference:hpu-1.5
        docker pull <Vector_DB> specific images
        docker images && sleep 1s
        echo "Docker images built!"

start_services()
        echo "Starting Docker Services...."
   
        export EMBEDDING_MODEL_ID="BAAI/bge-base-en-v1.5"
        export RERANK_MODEL_ID="BAAI/bge-reranker-base"
        export TEI_EMBEDDING_ENDPOINT="http://${ip_address}:6006"
        export TEI_RERANKING_ENDPOINT="http://${ip_address}:8808"
        export TGI_LLM_ENDPOINT="http://${ip_address}:8008"
        export MILVUS_HOST=${ip_address}
        export HUGGINGFACEHUB_API_TOKEN=${HUGGINGFACEHUB_API_TOKEN}
        export MEGA_SERVICE_HOST_IP=${ip_address}
        export EMBEDDING_SERVICE_HOST_IP=${ip_address}
        export RETRIEVER_SERVICE_HOST_IP=${ip_address}
        export RERANK_SERVICE_HOST_IP=${ip_address}
        export LLM_SERVICE_HOST_IP=${ip_address}
        export host_ip=${ip_address}
        export DATAPREP_SERVICE_ENDPOINT="http://${host_ip}:6007/v1/dataprep/ingest"
        export RERANK_TYPE="tei"
        export LOGFLAG=true

        # Start Docker Containers
        docker compose -f compose_<Vector_DB>.yaml up -d
        sleep 2m
        echo "Docker services started!"

validate_megaservice()
        echo "===========Ingest data=================="
        
        local CONTENT=$(http_proxy="" curl -X POST "http://${ip_address}:6007/v1/dataprep/ingest" \
        -H "Content-Type: multipart/form-data" \
        -F 'link_list=["https://opea.dev/"]')
        local EXIT_CODE=$(validate "$CONTENT" "Data preparation succeeded" "dataprep-<Vector_DB>-service-gaudi")
        echo "$EXIT_CODE"

        # Curl the Mega Service
        echo "================Testing retriever service: Text Request ================"

        local CONTENT=$(http_proxy="" curl http://${ip_address}:8889/v1/retrievaltool -X POST -H "Content-Type: application/json" -d '{
        "text": "Explain the OPEA project?"
        }')