XTune - Model finetune tool for Intel GPU¶

Xtune is an model finetune tool for Intel GPU(Intel Arc 770)

[!NOTE]

Xtune incorporates with Llama-Factory to offer various methods for finetuning visual models (CLIP, AdaCLIP), LLM and Multi-modal models. It makes easier to choose the method and to set fine-tuning parameters.

The core features include:

Four finetune method for CLIP, details in CLIP
Three finetune method for AdaCLIP, details in AdaCLIP
Automatic hyperparameter searching enabled by Optuna Optuna
Distillation from large models with Intel ARC GPU
Incorporate with Llama-Factory UI
Finetune methods for multi-modal models (to be supported)

You can use this UI to easily access basic functions(merge two tool into one UI),

or use the command line to use tools separately which is easier to customize parameters and has more comprehensive functionality.

Installation¶

[!IMPORTANT] Installation is mandatory.

Please install git first and make sure git clone can work.

Please fololow install_dependency to install Driver for Arc 770

1. Install xtune on native¶

Run install_xtune.sh to prepare component.

conda create -n xtune python=3.10 -y
conda activate xtune
apt install -y rsync
# open webui as default
bash prepare_xtune.sh
# this way it will not open webui
# bash prepare_xtune.sh false

Blow command is in prepare_xtune.sh. You can ignore it if you don’t want to update lib manually.

# if you want to run on NVIDIA GPU
    conda install pytorch torchvision cudatoolkit=10.2 -c pytorch
# else run on A770
# You can refer to https://github.com/intel/intel-extension-for-pytorch for latest command to update lib
    python -m pip install torch==2.5.1+cxx11.abi torchvision==0.20.1+cxx11.abi torchaudio==2.5.1+cxx11.abi  --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/
    python -m pip install intel-extension-for-pytorch==2.5.10+xpu oneccl_bind_pt==2.5.0+xpu --extra-index-url https://pytorch-extension.intel.com/release-whl/stable/xpu/us/

2. Install xtune on docker¶

2.1 Build Docker Image¶

Build docker image with below command:

cd ../../../deployment/docker_compose
export DATA="where to find dataset"
docker build -t opea/finetuning-xtune:latest --build-arg https_proxy=$https_proxy --build-arg http_proxy=$http_proxy --build-arg HF_TOKEN=$HF_TOKEN --build-arg DATA=$DATA -f comps/finetuning/src/Dockerfile.xtune .

2.2 Run Docker with CLI¶

Suse docker compose with below command:

export HF_TOKEN=${your_huggingface_token}
export DATA="where to find dataset"
cd ../../../deployment/docker_compose
docker compose -f compose.yaml up finetuning-xtune -d

Data Preparation¶

Please refer to data/Prepare_dataset.md for checking the details about the dataset files.

[!NOTE] Please update dataset_info.json to use your custom dataset.

Prepare dataset info for caltech101 make caltech101.json in your dataset directory

[]

then make dataset_info.json in your dataset directory

{
  "caltech101": {
    "file_name": "caltech101.json"
  }
}

Fine-Tuning with LLaMA Board GUI (powered by Gradio)¶

[!NOTE] We don’t support multi-card in GUI now, will add it later.

When run with prepare_xtune.sh, it will automatic run ZE_AFFINITY_MASK=0 llamafactory-cli webui.

If you see “server start successfully” in terminal. You can access in web through http://localhost:7860/

The UI component information can be seen in doc/ui_component.md after run with prepare_xtune.sh.

When run with prepare_xtune.sh, it will automatic run ZE_AFFINITY_MASK=0 llamafactory-cli webui.

If you see “server start successfully” in terminal. You can access in web through http://localhost:7860/

The UI component information can be seen in doc/ui_component.md after run with prepare_xtune.sh.

 Run with A100:
 CUDA_VISIBLE_DEVICES=0 llamafactory-cli webui
 Run with ARC770:
 ZE_AFFINITY_MASK=0 llamafactory-cli webui

 Then access in web through http://localhost:7860/

Fine-Tuning with Shell instead of GUI¶

After run prepare_xtune.sh, it will download all related file. And open webui as default.

You can run bash prepare_xtune.sh false to close webui. Then you can run fine-tune with shell.

Below are examples.

CLIP¶

Please see doc for how to config feature

cd src/llamafactory/clip_finetune
# Please see README.md in src/llamafactory/clip_finetune for detail

AdaCLIP¶

cd src/llamafactory/adaclip_finetune
# Please see README.md in src/llamafactory/adaclip_finetune for detail

DeepSeek-R1 Distillation(not main function)¶

Please see doc for details

Step 1: Download existing CoT synthetic dataset from huggingface¶

Dataset link: https://huggingface.co/datasets/Magpie-Align/Magpie-Reasoning-V2-250K-CoT-Deepseek-R1-Llama-70B

Step 2: Convert to sharegpt format¶

cd data
import json
from datasets import load_dataset
# Load the dataset
dataset = load_dataset("Magpie-Align/Magpie-Reasoning-V2-""250K-CoT-Deepseek-R1-Llama-70B")
dataset = dataset["train"]
# Filter dataset
## Change the filter conditions according to your needs
dataset = dataset.filter(lambda example: len(example['response']) <= 1024)
# Save as sharegpt format
with open("Magpie-Reasoning-V2-250K-CoT-Deepseek-R1-Llama-70B-response1024.json",
'w') as f:
  json.dump(list(dataset), f, ensure_ascii=False, indent=4)

Step 3: Register CoT dataset LLAMA-Factory dataset_info.json¶

cd data
vim dataset_info.json

# make sure the file is put under `xtune/data`
"deepseek-r1-distill-sample": {
  "file_name": "Magpie-Reasoning-V2-250K-CoT-Deepseek-R1-Llama-70B-response1024.json",
  "formatting": "sharegpt",
  "columns": {
    "messages": "conversations"
  }
}

Step 4: Use the accelerate command to enable training on XPU plugin¶

accelerate config

For Single GPU:
  Which type of machine are you using?
  No distributed training
  Do you want to run your training on CPU only (even if a GPU / Apple Silicon / Ascend NPU device is available)? [yes/NO]:NO
  Do you want to use XPU plugin to speed up training on XPU? [yes/NO]:yes
  Do you wish to optimize your script with torch dynamo?[yes/NO]:NO
  Do you want to use DeepSpeed? [yes/NO]: NO
  What GPU(s) (by id) should be used for training on this machine as a comma-seperated list? [all]:all
  Would you like to enable numa efficiency? (Currently only supported on NVIDIA hardware). [yes/NO]:
  Do you wish to use mixed precision?
  bf16
For Multi-GPU with FSDP:
  Which type of machine are you using?
  multi-XPU
  How many different machines will you use (use more than 1 for multi-node training)? [1]: 1
  Should distributed operations be checked while running for errors? This can avoid timeout issues but will be slower. [yes/NO]: NO
  Do you want to use XPU plugin to speed up training on XPU? [yes/NO]:yes
  Do you wish to optimize your script with torch dynamo?[yes/NO]:NO
  Do you want to use DeepSpeed? [yes/NO]: NO
  Do you want to use FullyShardedDataParallel? [yes/NO]: yes
  What should be your sharding strategy?
  FULL_SHARD
  Do you want to offload parameters and gradients to CPU? [yes/NO]: NO
  What should be your auto wrap policy?
  TRANSFORMER_BASED_WRAP
  Do you want to use the model's `_no_split_modules` to wrap. Only applicable for Transformers [yes/NO]: yes
  What should be your FSDP's backward prefetch policy?
  BACKWARD_PRE
  What should be your FSDP's state dict type?
  SHARDED_STATE_DICT
  Do you want to enable FSDP's forward prefetch policy? [yes/NO]: yes
  Do you want to enable FSDP's `use_orig_params` feature? [YES/no]: yes
  Do you want to enable CPU RAM efficient model loading? Only applicable for Transformers models. [YES/no]: yes
  Do you want to enable FSDP activation checkpointing? [yes/NO]: yes
  How many GPU(s) should be used for distributed training? [1]:2
  Do you wish to use mixed precision?
  bf16

Step 5: Run with train script as follows¶

export ONEAPI_DEVICE_SELECTOR="level_zero:0"
MODEL_ID="microsoft/Phi-3-mini-4k-instruct"
EXP_NAME="Phi-3-mini-4k-instruct-r1-distill-finetuned"
DATASET_NAME="deepseek-r1-distill-sample"
export OUTPUT_DIR="where to put output"
accelerate launch src/train.py --stage sft --do_train --use_fast_tokenizer --new_special_tokens "<think>,</think>" --resize_vocab --flash_attn auto --model_name_or_path ${MODEL_ID} --dataset ${DATASET_NAME} --template phi --finetuning_type lora --lora_rank 8 --lora_alpha 16 --lora_target q_proj,v_proj,k_proj,o_proj --additional_target lm_head,embed_tokens --output_dir $OUTPUT_DIR --overwrite_cache --overwrite_output_dir --warmup_steps 100 --weight_decay 0.1 --per_device_train_batch_size 1 --gradient_accumulation_steps 4 --ddp_timeout 9000 --learning_rate 5e-6 --lr_scheduler_type cosine --logging_steps 1 --save_steps 1000 --plot_loss --num_train_epochs 3 --torch_empty_cache_steps 10 --bf16

`Xtune` Examples¶

See screenshot of running CLIP and AdaCLIP finetune on Intel Arc A770 in README_XTUNE.md.

Citation¶

@inproceedings{zheng2024llamafactory,
  title={LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models},
  author={Yaowei Zheng and Richong Zhang and Junhao Zhang and Yanhan Ye and Zheyan Luo and Zhangchi Feng and Yongqiang Ma},
  booktitle={Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  address={Bangkok, Thailand},
  publisher={Association for Computational Linguistics},
  year={2024},
  url={http://arxiv.org/abs/2403.13372}
}

Acknowledgement¶

This repo benefits from LLaMA-Factory, CLIP-Adapter and CoOp. Thanks for their wonderful works.