FlashInfer 0.2: customizable attention kernels for LLM serving
In one sentence On December 16, 2024 the FlashInfer community releases 0.2 of its kernel library for LLM inference: FlashAttention-3 templates for Hopper GPUs, block- and vector-sparse attention, JIT-compiled attention variants, MLA decoding, and CUDAGraph and torch.compile compatibility.
FlashInfer is a GPU kernel library used by engines that serve language models in production: it mainly handles attention computation and management of the KV cache, the memory where the model keeps the conversation context.
Version 0.2 brings FlashAttention-3's optimized kernels to NVIDIA Hopper GPUs, treats the paged KV cache as a case of block-sparse attention, and lets you define new attention variants that are compiled on the fly (JIT) instead of recompiling the whole library.
It also adds a kernel for Multi-head Latent Attention (MLA) decoding, and compatibility with CUDAGraph and torch.compile. SGLang is cited as a user of the vector-sparse mode.
Companies
FlashInfer
Tools
FlashInfer, SGLang, PyTorch, CUDA
Tags
Sources