Dataprep Microservice with Pinecone¶

🚀Start Microservice with Python¶

Install Requirements¶

pip install -r requirements.txt

Start Pinecone Server¶

Create Pinecone account from the below link

https://app.pinecone.io/

More details from Pinecone quick start guide https://docs.pinecone.io/guides/get-started/quickstart

Get API key

API Key is needed to make the API calls. API key can get it from the Project -> Manage -> API keys

Create the index in https://app.pinecone.io/

Following details are to be provided

- Index name
- Based on the embedding model selected, following has to be provided
    a. Dimensions
    b. Metric

Setup Environment Variables¶

export http_proxy=${your_http_proxy}
export https_proxy=${your_http_proxy}
export PINECONE_API_KEY=${PINECONE_API_KEY}
export PINECONE_INDEX_NAME=${PINECONE_INDEX_NAME}

Start Document Preparation Microservice for Pinecone with Python Script¶

Start document preparation microservice for Pinecone with below command.

python prepare_doc_pinecone.py

🚀Start Microservice with Docker¶

Build Docker Image¶

cd ../../../../
docker build -t opea/dataprep-pinecone:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy -f comps/dataprep/pinecone/langchain/Dockerfile .

Run Docker with CLI¶

docker run -d --name="dataprep-pinecone-server" -p 6007:6007 --ipc=host -e http_proxy=$http_proxy -e https_proxy=$https_proxy opea/dataprep-pinecone:latest

Setup Environment Variables¶

export http_proxy=${your_http_proxy}
export https_proxy=${your_http_proxy}
export PINECONE_API_KEY=${PINECONE_API_KEY}
export PINECONE_INDEX_NAME=${PINECONE_INDEX_NAME}

Run Docker with Docker Compose¶

cd comps/dataprep/pinecone/langchain
docker compose -f docker-compose-dataprep-pinecone.yaml up -d

Invoke Microservice¶

Once document preparation microservice for Pinecone is started, user can use below command to invoke the microservice to convert the document to embedding and save to the database.

curl -X POST -H "Content-Type: application/json" -d '{"path":"/path/to/document"}' http://localhost:6007/v1/dataprep