HTML 77.2%
TypeScript 10.5%
Python 9.6%
JavaScript 2.5%
1<!-- markdownlint-disable MD001 MD041 -->2<p align="center">3 <picture>4 <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/assets/logos/vllm-logo-text-dark.png">5 <img alt="vLLM" src="https://raw.githubusercontent.com/vllm-project/vllm/main/docs/assets/logos/vllm-logo-text-light.png" width=55%>6 </picture>7</p>89<h3 align="center">10Easy, fast, and cheap LLM serving for everyone11</h3>1213<p align="center">14| <a href="https://docs.vllm.ai"><b>Documentation</b></a> | <a href="https://blog.vllm.ai/"><b>Blog</b></a> | <a href="https://arxiv.org/abs/2309.06180"><b>Paper</b></a> | <a href="https://x.com/vllm_project"><b>Twitter/X</b></a> | <a href="https://discuss.vllm.ai"><b>User Forum</b></a> | <a href="https://slack.vllm.ai"><b>Developer Slack</b></a> |15</p>1617🔥 We have built a vLLM website to help you get started with vLLM. Please visit [vllm.ai](https://vllm.ai) to learn more.18For events, please visit [vllm.ai/events](https://vllm.ai/events) to join us.1920---2122## About2324vLLM is a fast and easy-to-use library for LLM inference and serving.2526Originally developed in the [Sky Computing Lab](https://sky.cs.berkeley.edu) at UC Berkeley, vLLM has grown into one of the most active open-source AI projects built and maintained by a diverse community of many dozens of academic institutions and companies from over 2000 contributors.2728vLLM is fast with:2930- State-of-the-art serving throughput31- Efficient management of attention key and value memory with [**PagedAttention**](https://blog.vllm.ai/2023/06/20/vllm.html)32- Continuous batching of incoming requests, chunked prefill, prefix caching33- Fast and flexible model execution with piecewise and full CUDA/HIP graphs34- Quantization: FP8, MXFP8/MXFP4, NVFP4, INT8, INT4, GPTQ/AWQ, GGUF, compressed-tensors, ModelOpt, TorchAO, and [more](https://docs.vllm.ai/en/latest/features/quantization/index.html)35- Optimized attention kernels including FlashAttention, FlashInfer, TRTLLM-GEN, FlashMLA, and Triton36- Optimized GEMM/MoE kernels for various precisions using CUTLASS, TRTLLM-GEN, CuTeDSL37- Speculative decoding including n-gram, suffix, EAGLE, DFlash38- Automatic kernel generation and graph-level transformations using torch.compile39- Disaggregated prefill, decode, and encode4041vLLM is flexible and easy to use with:4243- Seamless integration with popular Hugging Face models44- High-throughput serving with various decoding algorithms, including *parallel sampling*, *beam search*, and more45- Tensor, pipeline, data, expert, and context parallelism for distributed inference46- Streaming outputs47- Generation of structured outputs using xgrammar or guidance48- Tool calling and reasoning parsers49- OpenAI-compatible API server, plus Anthropic Messages API and gRPC support50- Efficient multi-LoRA support for dense and MoE layers51- Support for NVIDIA GPUs, AMD GPUs, Intel GPUs, and x86/ARM/PowerPC CPUs. Additionally, diverse hardware plugins such as Google TPUs, Intel Gaudi, IBM Spyre, Huawei Ascend, Rebellions NPU, Apple Silicon, MetaX GPU, and more.5253vLLM seamlessly supports 200+ model architectures on Hugging Face, including:5455- Decoder-only LLMs (e.g., Llama, Qwen, Gemma)56- Mixture-of-Expert LLMs (e.g., Mixtral, DeepSeek-V3, Qwen-MoE, GPT-OSS)57- Hybrid attention and state-space models (e.g., Mamba, Qwen3.5)58- Multi-modal models (e.g., LLaVA, Qwen-VL, Pixtral)59- Embedding and retrieval models (e.g., E5-Mistral, GTE, ColBERT)60- Reward and classification models (e.g., Qwen-Math)6162Find the full list of supported models [here](https://docs.vllm.ai/en/latest/models/supported_models.html).6364## Getting Started6566Install vLLM with [`uv`](https://docs.astral.sh/uv/) (recommended) or `pip`:6768```bash69uv pip install vllm70```7172Or [build from source](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/index.html#build-wheel-from-source) for development.7374Visit our [documentation](https://docs.vllm.ai/en/latest/) to learn more.7576- [Installation](https://docs.vllm.ai/en/latest/getting_started/installation.html)77- [Quickstart](https://docs.vllm.ai/en/latest/getting_started/quickstart.html)78- [List of Supported Models](https://docs.vllm.ai/en/latest/models/supported_models.html)7980## Contributing8182We welcome and value any contributions and collaborations.83Please check out [Contributing to vLLM](https://docs.vllm.ai/en/latest/contributing/index.html) for how to get involved.8485## Citation8687If you use vLLM for your research, please cite our [paper](https://arxiv.org/abs/2309.06180):8889```bibtex90@inproceedings{kwon2023efficient,91 title={Efficient Memory Management for Large Language Model Serving with PagedAttention},92 author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},93 booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},94 year={2023}95}96```9798## Contact Us99100<!-- --8<-- [start:contact-us] -->101- For technical questions and feature requests, please use GitHub [Issues](https://github.com/vllm-project/vllm/issues)102- For discussing with fellow users, please use the [vLLM Forum](https://discuss.vllm.ai)103- For coordinating contributions and development, please use [Slack](https://slack.vllm.ai)104- For security disclosures, please use GitHub's [Security Advisories](https://github.com/vllm-project/vllm/security/advisories) feature105- For collaborations and partnerships, please contact us at [collaboration@vllm.ai](mailto:collaboration@vllm.ai)106<!-- --8<-- [end:contact-us] -->107108## Media Kit109110- If you wish to use vLLM's logo, please refer to [our media kit repo](https://github.com/vllm-project/media-kit)111