vllm 0.11 and A770

[savvadesogle]

Since Intel has so far abandoned ipex-llm and Arc cards...

vllm v0.11.1rc2.dev221+g49c00fe30 works together with A770 (4x)

You can build a Docker container from the vllm repository sources (Dockerfile.xpu)
https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.xpu

docker build -f docker/Dockerfile.xpu -t vllm-xpu-0110 --shm-size=32g .

But I do not know how to properly configure it for the 4x A770, and I am sure that the performance could be higher
2 req/s -> 10+ req/s.

Llama3.1 8b Instruct FP8
Sometimes the request processing speed reaches 12 requests/s, but there are problems with the process "hanging up" and then speeding up. I haven't figured out the reason yet.
1024 in, 512 out for configuration

--max-model-len "2000" 
--max-num-batched-tokens "3000"

test

vllm bench serve \
    --model /llm/models/LLM-Research/Meta-Llama-3.1-8B-Instruct \
    --served-model-name Meta-Llama-3.1-8B-Instruct \
    --dataset-name random \
    --random-input-len 1024 \
    --random-output-len 512 \
    --ignore-eos \
    --num-prompt 1500 \
    --trust-remote-code \
    --request-rate inf \
    --backend vllm \
    --port 8000

Ubuntu 25.10, 6,17.3 kernel
my numbers for 4x A770, 2x Xeon 2699 V3 is:

115 requests

1500 requests

[xiangyuT]

Hi @savvadesogle,

For A770 devices, you can refer to this quickstart and upgrade vllm inside intelanalytics/ipex-llm-serving-xpu:0.8.3-b19. The dockerfile is here and you can get the corresponding code base with following commandlines:

cd /llm && \
wget https://raw.githubusercontent.com/intel/ipex-llm/refs/heads/main/docker/llm/serving/xpu/docker/vllm_for_multi_arc.patch && \
git clone -b v0.8.3 https://github.com/vllm-project/vllm /llm/vllm && \
cd /llm/vllm && \
git apply /llm/vllm_for_multi_arc.patch

[savvadesogle]

For A770 devices, you can refer to this quickstart and upgrade vllm inside

Thank you for your response! ❤️
But won't updating the vllm in the b19 container to the current (v 0.11) version break it?

Did I get it right:

1. pip install -U vllm in the container b19
2. apply patches like in your example

Should I use the b19 image specifically?

[savvadesogle]

Suddenly, someone is interested in what parameters it works with.

export USE_XETLA=OFF
export SYCL_CACHE_PERSISTENT=1
export SYCL_PI_LEVEL_ZERO_USE_IMMEDIATE_COMMANDLISTS=0
#export SYCL_PI_LEVEL_ZERO_BATCH_SIZE=48 try to change numbers
export FI_PROVIDER=shm
#export TORCH_LLM_ALLREDUCE=0

export CCL_WORKER_COUNT=1      
export CCL_ATL_TRANSPORT=ofi
export CCL_ZE_IPC_EXCHANGE=sockets
export CCL_ATL_SHM=1
export CCL_SAME_STREAM=1
export CCL_BLOCKING_WAIT=0
# export CCL_DG2_USM=1     

#export CCL_DG2_ALLREDUCE=1
#export VLLM_USE_V1=0  

#source /opt/intel/1ccl-wks/setvars.sh

python3 -m vllm.entrypoints.openai.api_server \
  --port 8000 \
  --model /llm/models/Meta-Llama-3.1-8B-Instruct  \
  --served-model-name Meta-Llama-3.1-8B-Instruct \
  --trust-remote-code \
  --gpu-memory-utilization "0.95" \
  --dtype float16 \
  --enforce-eager \
  --quantization fp8 \
  --max-model-len "3000" \
  --max-num-batched-tokens "2000" \
  --max-num-seqs "256" \
  --tensor-parallel-size "1" \
  --pipeline-parallel-size "4" \
  --block-size 32 

any number TP is not working for me.. pp 1-4 is working
block-size 8 not working anymore

[savvadesogle]

For an example, Deepseek OCR doesnt work for me.

RuntimeError: Input type (c10::BFloat16) and bias type (c10::Half) should be the same

ValueError: Intel Arc A770 have bfloat16 accuracy known issue. You can use float16 instead by explicitly setting the `dtype` flag in CLI, for example: --dtype=half`

And recommended using Pytorch 2.9.
intel/intel-extension-for-pytorch#867 (comment)

It's such a deep rabbit hole.

[jakegibson]

Thanks! And you are using the vllm-project/vllm repo 11.rc2 with their instructions (below) for building? That's too bad on tensor parallel but matches my experience both on vllm and SYCL backend of llama.cpp. I have 7 a770s in a rig and would love to use ~4 of them for a bigger MoE model (gpt-oss or glm 4.5 air).

docker build -f docker/Dockerfile.xpu -t vllm-xpu-env --shm-size=4g .
docker run -it \
             --rm \
             --network=host \
             --device /dev/dri \
             -v /dev/dri/by-path:/dev/dri/by-path \
             vllm-xpu-env