Dataprep Microservice¶

The Dataprep Microservice aims to preprocess the data from various sources (either structured or unstructured data) to text data, and convert the text data to embedding vectors then store them in the database.

Install Requirements¶

apt-get update
apt-get install libreoffice

Use LVM (Large Vision Model) for Summarizing Image Data¶

Occasionally unstructured data will contain image data, to convert the image data to the text data, LVM can be used to summarize the image. To leverage LVM, please refer to this readme to start the LVM microservice first and then set the below environment variable, before starting any dataprep microservice.

export SUMMARIZE_IMAGE_VIA_LVM=1

Dataprep Microservice with Redis¶

For details, please refer to this readme

Dataprep Microservice with Milvus¶

For details, please refer to this readme

Dataprep Microservice with Qdrant¶

For details, please refer to this readme

Dataprep Microservice with Pinecone¶

For details, please refer to this readme

Dataprep Microservice with PGVector¶

For details, please refer to this readme

Dataprep Microservice with VDMS¶

For details, please refer to this readme

Dataprep Microservice with Multimodal¶

For details, please refer to this readme

Dataprep Microservice with ElasticSearch¶

For details, please refer to this readme

Dataprep Microservice with OpenSearch¶

For details, please refer to this readme

Dataprep Microservice with neo4j¶

For details, please refer to this readme

Dataprep Microservice for financial domain data¶

For details, please refer to this readme

Dataprep Microservice with MariaDB Vector¶

For details, please refer to this readme

Running in the air gapped environment¶

The following steps are common for running the dataprep microservice in an air gapped environment (a.k.a. environment with no internet access), for all DB backends.

Download the following models, e.g. huggingface-cli download --cache-dir <model data directory> <model>

microsoft/table-transformer-structure-recognition
timm/resnet18.a1_in1k
unstructuredio/yolo_x_layout

launch the dataprep microservice with the following settings:

mount the model data directory as the /data directory within the dataprep container
set environment variable HF_HUB_OFFLINE to 1 when launching the dataprep microservice

e.g. docker run -d -v <model data directory>:/data -e HF_HUB_OFFLINE=1 ... ...