While upstream Python-based vLLM requires complex container layers, heavy virtual environments, and multi-gigabyte PyTorch dependencies (Kwon et al., 2023; Paszke et al., 2019), mudler/vllm.cpp compiles directly via modern CMake into an embeddable, standalone C++20 serving engine with zero Python dependencies (Di Giacinto, 2026).
Below is the complete engineering walkthrough for compiling vllm.cpp across modern Linux distributions (such as Fedora), configuring hardware acceleration backends—including NVIDIA CUDA, cross-platform Vulkan, and native AMD ROCm / HIP targeting RDNA3 (gfx1100) and RDNA3.5 (gfx1151)—serving compact models like Qwen 2.5 / 3.5, tuning the RadixTree prefix-caching subsystem, establishing an automated systemd background daemon, and evaluating compatibility with TurboQuant (Di Giacinto, 2026; Zandieh et al., 2025; Zheng et al., 2024).
+-------------------------------------------------------------+
| vllm.cpp ARCHITECTURE OVERVIEW |
+-------------------------------------------------------------+
| INPUT FORMATS → GGUF (Q4_0, Q4_K_M, IQ4_XS), |
| EXL3 Trellis, SafeTensors. |
| ↓ |
| SERVING ENGINE → C++20 Continuous Batching Loop, |
| (Pure Native Core) Block-Paged KV Cache (FP8/FP16),|
| RadixTree Prefix Caching / LPM. |
| ↓ |
| COMPUTE BACKENDS → CUDA, Vulkan, ROCm/HIP, Metal. |
| ↓ |
| DISTRIBUTION → Single ~66 MiB Standalone Binary |
| (Zero Python / Zero PyTorch). |
+-------------------------------------------------------------+
Step 1: System Prerequisites & Dependencies
On Fedora Linux, install the baseline C++20 compiler (gcc-c++ 13+ or clang), CMake build system, and Ninja generator:
Bash
sudo dnf install -y git cmake ninja-build gcc-c++ libstdc++-devel
Acceleration Backend Toolkits
- NVIDIA CUDA: Install the NVIDIA CUDA Toolkit (
cuda,cuda-devel). Ensurenvcc --versionis accessible in$PATH. - Vulkan: Install the Vulkan loader, header packages, and development tools:Bash
sudo dnf install -y vulkan-loader-devel vulkan-headers vulkan-toolsVerify device visibility by executingvulkaninfo --summary. - AMD ROCm / HIP (for RDNA3 / RDNA3.5): Install the ROCm development stack (
rocm-hip-sdk,hipblas-devel,rocminfo):Bashsudo dnf install -y rocm-hip-sdk hipblas-devel rocminfoEnsurehipcc --versionexecutes cleanly and verify that the target GPU node is enumerated withrocminfo.
Clone the repository with submodules:
Bash
git clone --recurse-submodules https://github.com/mudler/vllm.cpp.git
cd vllm.cpp
Step 2: Compiling the Engine (CUDA, Vulkan, and AMD ROCm)
vllm.cpp supports native GPU execution backends through dedicated CMake flags:
+-------------------------------------------------------------+
| vllm.cpp BUILD CONFIGURATIONS |
+-------------------------------------------------------------+
| NVIDIA CUDA (sm_75 to sm_120a) |
| cmake -DVLLM_CPP_CUDA=ON -DVLLM_CPP_CUDA_ARCHITECTURES=... |
| OR |
| AMD ROCm / HIP (gfx1100, gfx1151) |
| cmake -DVLLM_CPP_HIP=ON -DVLLM_CPP_HIP_ARCHITECTURES=... |
| OR |
| VULKAN ACCELERATION (Cross-Vendor: AMD / Intel / APUs) |
| cmake -DVLLM_CPP_VULKAN=ON -DVLLM_CPP_CUDA=OFF |
+-------------------------------------------------------------+
Variant A: NVIDIA CUDA Compilation
When targeting NVIDIA GPUs, configure CMake with the target Streaming Multiprocessor (SM) compute capabilities:
Bash
# Broad build covering Turing (75), Ampere (80/86), Ada (89), Hopper (90a), and Blackwell (120a)
cmake -S . -B build-cuda \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DVLLM_CPP_CUDA=ON \
-DVLLM_CPP_CUDA_ARCHITECTURES="75;80;86;89;90a;120a"
cmake --build build-cuda --target vllm-server -j$(nproc)
(Note: To minimize compile times when building for a single machine, supply only that card’s architecture, such as 89 for an RTX 4090 or 86 for an RTX 3050).
Variant B: AMD ROCm / HIP Compilation (RDNA3 gfx1100 & RDNA3.5 gfx1151)
vllm.cpp provides a native ROCm backend utilizing HIP kernels for matrix-multiplication operations and memory transfers (Di Giacinto, 2026). This enables high-performance execution on discrete AMD RDNA3 GPUs and mobile/desktop RDNA3.5 APUs (Di Giacinto, 2026):
gfx1100: Discrete AMD Radeon RX 7900 XTX / 7900 XT / 7900 GRE (RDNA3).gfx1151: AMD Strix Point / Strix Halo APUs (e.g., Ryzen AI 9 HX 370 with Radeon 890M integrated graphics running RDNA3.5).
Bash
# Set HIP compiler to hipcc
export CXX=hipcc
cmake -S . -B build-rocm \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DVLLM_CPP_HIP=ON \
-DVLLM_CPP_HIP_ARCHITECTURES="gfx1100;gfx1151"
cmake --build build-rocm --target vllm-server -j$(nproc)
Operational Stability Note for gfx1151: On unified-memory AMD Strix APUs where system RAM is dynamically shared, export HSA_ENABLE_SDMA=0 prior to execution if DMA engine memory faults or driver timeouts occur under sustained concurrent load (Di Giacinto, 2026).
Variant C: Vulkan Compilation (Universal Fallback & Consumer Discrete)
The Vulkan backend serves as an agnostic compute path across AMD Radeon, Intel Arc, Apple Silicon, and NVIDIA cards without proprietary SDK toolchains:
Bash
cmake -S . -B build-vulkan \
-G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DVLLM_CPP_VULKAN=ON \
-DVLLM_CPP_CUDA=OFF \
-DVLLM_CPP_HIP=OFF
cmake --build build-vulkan --target vllm-server -j$(nproc)
The resulting binary (vllm-server) will be compiled in build-*/examples/vllm-server with an uncompressed binary footprint of roughly 60 to 70 MiB (Di Giacinto, 2026).
Step 3: RadixTree Prefix Caching and Cache-Aware Scheduling
A primary feature in vllm.cpp is RadixTree prefix caching, adapted from SGLang’s memory management engine (Di Giacinto, 2026; Zheng et al., 2024).
Standard token-serving architectures discard KV states across completed calls, forcing the engine to recompute prompt tokens for repetitive system instructions, large few-shot context windows, and multi-turn conversational agents (Kwon et al., 2023; Zheng et al., 2024). vllm.cpp maps an explicit radix trie directly over its physical block-paged KV allocator (Di Giacinto, 2026; Zheng et al., 2024).
+-------------------------------------------------------------+
| RADIXTREE KV CACHE ALLOCATION PIPELINE |
+-------------------------------------------------------------+
| REQUEST A: "System: You are an AI code reviewer... [Code]" |
| |
| [ Root: System Instructions (Tokens 0-256) ] |
| | |
| +---> [ Branch A: Code Reviewer #1 (Tokens 257-512) ] |
| | |
| +---> [ Branch B: Code Reviewer #2 (Tokens 257-640) ] |
| |
| * Exact matching prefix retains warm KV pages. |
| * Zero TTFT latency on repeated prompt templates. |
| * Eviction policy: Least Recently Used (LRU) node culling. |
+-------------------------------------------------------------+
1. Technical Mechanics of the Radix Cache
- Radix Prefix Matching: The tokenized prompt is looked up inside a radix trie where internal nodes index continuous physical token blocks (Zheng et al., 2024).
- Zero-Copy Cache Hits: If an incoming prompt shares a prefix with existing branches, the engine binds the new sequence’s block table directly to existing physical memory pages, eliminating the prompt-prefill step for that segment (Di Giacinto, 2026; Zheng et al., 2024).
- Longest Prefix Match (LPM) Scheduling: The request scheduler prioritizes queue items that share the longest matching warm prefix, maximizing hardware cache reuse and reducing compute cycles (Di Giacinto, 2026; Zheng et al., 2024).
- LRU Eviction: When GPU physical memory fills under concurrent load, the RadixTree drops leaf nodes with the oldest access timestamps, returning memory blocks to the global page pool without fragmentation (Di Giacinto, 2026; Zheng et al., 2024).
2. CLI Configuration Parameters
| Flag | Parameter Type | Default | Description & Operational Impact |
--scheduling-policy | String (fcfs / lpm) | fcfs | Configures scheduling logic; lpm sorts the incoming request queue by Longest Prefix Match to maximize cache hit rates (Di Giacinto, 2026; Zheng et al., 2024). |
--max-num-seqs | Integer | 32 | Maximum concurrent sequences executed per decoding step; limits memory pressure on 4GB GPUs (Di Giacinto, 2026). |
--kv-cache-dtype | String (fp16 / fp8) | fp16 | Precision of cached KV pages; fp8 halves page size to 1 byte per element, doubling cache capacity on constrained GPUs (Di Giacinto, 2026). |
--gpu-memory-utilization | Float (0.0 to 1.0) | 0.90 | Fraction of total VRAM allocated to the block-paged KV buffer pool (Di Giacinto, 2026; Kwon et al., 2023). |
Step 4: Serving Qwen 2B / 3B Models (CUDA, ROCm, Vulkan)
The Qwen model family provides high reasoning density at small parameter footprints (Yang et al., 2024). In vllm.cpp, GGUF is treated as a native storage format rather than a secondary conversion target (Di Giacinto, 2026).
+-------------------------------------------------------------+
| QWEN RUNTIME MEMORY ALLOCATION |
+-------------------------------------------------------------+
| Qwen 2.5/3.5 ~2B-3B Model Weights (Q4_K_M) ~ 1.95 GiB |
| Engine Execution & Compute Scratchpad ~ 0.25 GiB |
| Paged Key-Value Cache (--kv-cache-dtype fp8) ~ 1.10 GiB |
| Headroom / Activation Buffers ~ 0.40 GiB |
| |
| Total Peak VRAM Footprint ~ 3.70 GiB |
| (Completely fits inside standard 4GB VRAM graphics cards) |
+-------------------------------------------------------------+
1. Download Model Weights
Retrieve model weights directly in GGUF format:
Bash
curl -L -o qwen2.5-3b-instruct-q4_k_m.gguf \
"https://huggingface.co/Qwen/Qwen2.5-3B-Instruct-GGUF/resolve/main/qwen2.5-3b-instruct-q4_k_m.gguf"
2. Launch Commands by Acceleration Target
On an NVIDIA GPU (CUDA):
Bash
./build-cuda/examples/vllm-server \
--model ./qwen2.5-3b-instruct-q4_k_m.gguf \
--kv-cache-dtype fp8 \
--block-size 16 \
--max-num-seqs 16 \
--max-model-len 4096 \
--gpu-memory-utilization 0.90 \
--scheduling-policy lpm \
--port 8000
On AMD RDNA3 / RDNA3.5 (ROCm gfx1100 / gfx1151):
Bash
# Optional SDMA workaround for APU integrated graphics
export HSA_ENABLE_SDMA=0
./build-rocm/examples/vllm-server \
--model ./qwen2.5-3b-instruct-q4_k_m.gguf \
--kv-cache-dtype fp8 \
--block-size 16 \
--max-num-seqs 16 \
--max-model-len 4096 \
--gpu-memory-utilization 0.88 \
--scheduling-policy lpm \
--port 8000
On Vulkan (Universal Discrete & APU fallback):
Bash
./build-vulkan/examples/vllm-server \
--model ./qwen2.5-3b-instruct-q4_k_m.gguf \
--kv-cache-dtype fp8 \
--block-size 16 \
--max-num-seqs 16 \
--max-model-len 4096 \
--gpu-memory-utilization 0.90 \
--scheduling-policy lpm \
--port 8000
3. Test with the OpenAI-Compatible API
Bash
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "qwen2.5-3b-instruct-q4_k_m.gguf",
"messages": [
{"role": "system", "content": "You are a concise systems programming assistant."},
{"role": "user", "content": "Explain paged KV caches in 2 sentences."}
],
"temperature": 0.2
}'
Step 5: Production Systemd Service Configuration
To ensure continuous operation on a Linux workstation or server running Fedora, deploy vllm.cpp as a managed background daemon using systemd.
1. Directory Structure and Binary Installation
Create dedicated directories for configuration, binaries, and model artifacts:
Bash
sudo mkdir -p /opt/vllm-cpp/bin
sudo mkdir -p /opt/vllm-cpp/models
sudo mkdir -p /etc/vllm-cpp
# Copy the compiled binary (e.g., from build-cuda, build-rocm, or build-vulkan)
sudo cp build-cuda/examples/vllm-server /opt/vllm-cpp/bin/
sudo cp qwen2.5-3b-instruct-q4_k_m.gguf /opt/vllm-cpp/models/
Create a system user with access to GPU acceleration devices:
Bash
sudo useradd -r -s /sbin/nologin -d /opt/vllm-cpp vllm
sudo usermod -a -G video,render vllm
sudo chown -R vllm:vllm /opt/vllm-cpp
2. Environment Configuration File
Create an environment file /etc/vllm-cpp/vllm-server.env:
Ini, TOML
# /etc/vllm-cpp/vllm-server.env
# Hardware & Runtime Variables
CUDA_VISIBLE_DEVICES=0
HIP_VISIBLE_DEVICES=0
HSA_ENABLE_SDMA=0
VULKAN_DEVICE_INDEX=0
# Server Bind Configuration
HOST=0.0.0.0
PORT=8000
# Model Paths & Allocation Bounds
MODEL_PATH=/opt/vllm-cpp/models/qwen2.5-3b-instruct-q4_k_m.gguf
KV_CACHE_DTYPE=fp8
BLOCK_SIZE=16
MAX_NUM_SEQS=16
MAX_MODEL_LEN=4096
GPU_MEMORY_UTILIZATION=0.90
SCHEDULING_POLICY=lpm
3. Systemd Unit File Definition
Create the unit file /etc/systemd/system/vllm-server.service:
Ini, TOML
[Unit]
Description=vllm.cpp High-Throughput LLM Server (NVIDIA CUDA / AMD ROCm / Vulkan)
After=network.target local-fs.target
Wants=network-online.target
[Service]
Type=simple
User=vllm
Group=vllm
SupplementaryGroups=video render
WorkingDirectory=/opt/vllm-cpp
# Environment injection
EnvironmentFile=/etc/vllm-cpp/vllm-server.env
# Execution Command with RadixTree Caching Flags
ExecStart=/opt/vllm-cpp/bin/vllm-server \
--model ${MODEL_PATH} \
--host ${HOST} \
--port ${PORT} \
--kv-cache-dtype ${KV_CACHE_DTYPE} \
--block-size ${BLOCK_SIZE} \
--max-num-seqs ${MAX_NUM_SEQS} \
--max-model-len ${MAX_MODEL_LEN} \
--gpu-memory-utilization ${GPU_MEMORY_UTILIZATION} \
--scheduling-policy ${SCHEDULING_POLICY}
# Restart policy and error handling
Restart=always
RestartSec=3s
# Process and resource headroom
LimitNOFILE=65536
LimitMEMLOCK=infinity
StandardOutput=journal
StandardError=journal
SyslogIdentifier=vllm-cpp
# Sandboxing compatible with DRM and render nodes
ProtectSystem=full
ProtectHome=true
PrivateTmp=true
[Install]
WantedBy=multi-user.target
4. Enable and Verify Service
Reload systemd, enable the unit to start at system boot, and inspect the logs:
Bash
sudo systemctl daemon-reload
sudo systemctl enable --now vllm-server.service
# Check active service status
systemctl status vllm-server.service
# Tail runtime logs
journalctl -u vllm-server.service -f
Step 6: Status of TurboQuant Support
TurboQuant is an online, vector-quantized key-value compression scheme introduced by researchers at Google Research and Google DeepMind (Zandieh et al., 2025). Rather than relying on simple scalar clamping or rounding down to low bits, TurboQuant applies a random rotation matrix across high-dimensional KV activation vectors to smooth out activation spikes, followed by an optimal scalar quantizer paired with a 1-bit residual correction for unbiased inner product preservation (Zandieh et al., 2025).
+-------------------------------------------------------------+
| TURBOQUANT ALGORITHM DESIGN |
+-------------------------------------------------------------+
| 1. INPUT KV VECTOR |
| Spiky high-dimensional activations from attention heads.|
| ↓ |
| 2. RANDOM ROTATION (WHT / Orthogonal Projection) |
| Spreads outliers evenly across coordinates. |
| ↓ |
| 3. TURBOQUANT MSE (3-Bit Scalar Quantizer) |
| Quantizes coordinate dimensions with minimal MSE loss. |
| ↓ |
| 4. TURBOQUANT PROD (1-Bit Residual Correction) |
| Corrects angular distortion; unbiased cosine similarity.|
+-------------------------------------------------------------+
| Engine Implementation | TurboQuant KV Status | Native KV Precision Options | Architecture Notes |
mudler/vllm.cpp | In Roadmap / Secondary Branch | Native FP8 (--kv-cache-dtype fp8), FP16 | Prioritizes E4M3/E5M2 block-paged KV allocation and RadixAttention (Di Giacinto, 2026; Zheng et al., 2024). |
| LocalAI Ecosystem | Integrated via turboquant backend | 2-bit to 4-bit TurboQuant KV | Available in the LocalAI model registry as an isolated fork backend alongside vllm-cpp (Di Giacinto, 2026; LocalAI, 2026). |
llama.cpp Forks | Experimental Primitives | FP16, Q8_0, Q4_0, TurboQuant patches | Requires custom kernel compilation for fast Walsh-Hadamard Transforms (Gerganov, 2023; Zandieh et al., 2025). |
In the main line of mudler/vllm.cpp, high-density KV compression on 4GB cards is achieved primarily through native FP8 (--kv-cache-dtype fp8) page allocation (Di Giacinto, 2026). This cuts standard 16-bit key-value storage requirements in half without requiring the rotational projections and residual inner-product corrections used in TurboQuant pipelines (Di Giacinto, 2026; Zandieh et al., 2025).
Comparative Feature Matrix Across Backends
| Serving Capability | vllm.cpp (CUDA) | vllm.cpp (ROCm / HIP) | vllm.cpp (Vulkan) | Upstream Python vLLM |
| Target Architectures | sm_75 to sm_120a (Di Giacinto, 2026) | gfx1100, gfx1151 (Di Giacinto, 2026) | Cross-Vendor (Di Giacinto, 2026) | Datacenter CUDA / ROCm (Kwon et al., 2023) |
| Minimum Hardware Floor | 4GB VRAM (Di Giacinto, 2026) | 4GB VRAM / APU Unified (Di Giacinto, 2026) | 4GB VRAM / APU Unified (Di Giacinto, 2026) | 8GB–16GB VRAM (Kwon et al., 2023) |
| Direct GGUF Execution | Yes (Di Giacinto, 2026) | Yes (Di Giacinto, 2026) | Yes (Di Giacinto, 2026) | Experimental / Secondary (Kwon et al., 2023) |
| Continuous Batching | Yes (Di Giacinto, 2026) | Yes (Di Giacinto, 2026) | Yes (Di Giacinto, 2026) | Yes (Kwon et al., 2023) |
| Radix Prefix Caching | Yes (Di Giacinto, 2026; Zheng et al., 2024) | Yes (Di Giacinto, 2026; Zheng et al., 2024) | Yes (Di Giacinto, 2026; Zheng et al., 2024) | APC only (Kwon et al., 2023) |
| Runtime Binary Footprint | ~66 MiB (Di Giacinto, 2026) | ~66 MiB (Di Giacinto, 2026) | ~66 MiB (Di Giacinto, 2026) | 7.5 GiB – 12 GiB (Kwon et al., 2023) |
| Python / PyTorch Required | No (Di Giacinto, 2026) | No (Di Giacinto, 2026) | No (Di Giacinto, 2026) | Yes (Paszke et al., 2019) |
References
- Di Giacinto, E. (2026). vllm.cpp: A community oriented 1:1, vLLM-alike engine in C++ with additional features (Version 0.2.x) [Computer software]. GitHub. https://github.com/mudler/vllm.cpp
- Gerganov, G. (2023). llama.cpp: Port of Facebook’s LLaMA model in C/C++ [Computer software]. GitHub. https://github.com/ggerganov/llama.cpp
- Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient memory management for large language model serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (SOSP ’23) (pp. 611–626). Association for Computing Machinery. https://doi.org/10.1145/3600006.3613165
- Leviathan, Y., Kalman, M., & Matias, Y. (2023). Fast inference from transformers via speculative decoding. In International Conference on Machine Learning (ICML 2023) (pp. 19274–19286). PMLR. https://proceedings.mlr.press/v202/leviathan23a.html
- LocalAI. (2026). Model compatibility and backend acceleration table. LocalAI Documentation. https://localai.io/docs/model-compatibility/
- Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshteyn, N., Antiga, L., Desmaison, A., Köpf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., … Chintala, S. (2019). PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems (NeurIPS 2019) (Vol. 32, pp. 8024–8035). Curran Associates, Inc. https://proceedings.neurips.cc/paper/2019/hash/bdbca288fee7f92f2bfa9f70127277b0-Abstract.html
- Shoeybi, M., Patwary, M., Puri, R., LeGresley, P., Casper, J., & Catanzaro, B. (2019). Megatron-LM: Training multi-billion parameter language models using model parallelism. arXiv. https://doi.org/10.48550/arXiv.1909.08053
- Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., & Lample, G. (2023). LLaMA: Open and efficient foundation language models. arXiv. https://doi.org/10.48550/arXiv.2302.13971
- Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F., Dong, G., Wei, H., Lin, H., Tang, J., Wang, J., Yang, J., Tu, J., Zhang, J., Ma, J., … Zeng, Z. (2024). Qwen2 technical report. arXiv. https://doi.org/10.48550/arXiv.2407.10671
- Zandieh, A., Daliri, M., Hadian, M., & Mirrokni, V. (2025). TurboQuant: Online vector quantization with near-optimal distortion rate. arXiv. https://doi.org/10.48550/arXiv.2504.19874
- Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, H., Yu, C. H., Cao, S., Christaballen, C., Duan, C., Wang, H., Lu, J., Wu, B., Zhu, B., Zhu, Y., Zhang, W., Shen, L., Yao, Y., Xu, C., Lin, J., … Stoica, I. (2024). SGLang: Efficient execution of structured language model programs. In Advances in Neural Information Processing Systems (NeurIPS 2024) (Vol. 37, pp. 62584–62618). Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2024/hash/703cecb68853b0dfb2f3473f3246eb7a-Abstract-Conference.html