High-Throughput Generative Inference of Large Language Models with a Single GPU

[Submitted on 13 Mar 2023]

Download PDF

Abstract: The high computational and memory requirements of large language model (LLM)
inference traditionally make it feasible only with multiple high-end
accelerators. Motivated by the emerging demand for latency-insensitive tasks
with batched processing, this paper initiates the study of high-throughput LLM
inference using limited resources, such as a single commodity GPU. We present
FlexGen, a high-throughput generation engine for running LLMs with limited GPU
memory. FlexGen can be flexibly configured under various hardware resource
constraints by aggregating memory and computation from the GPU, CPU, and disk.
Through a linear programming optimizer, it searches for efficient patterns to
store and access tensors. FlexGen further compresses these weights and the
attention cache to 4 bits with negligible accuracy loss. These techniques
enable FlexGen to have a larger space of batch size choices and thus
significantly increase maximum throughput. As a result, when running OPT-175B
on a single 16GB GPU, FlexGen achieves significantly higher throughput compared
to state-of-the-art offloading systems, reaching a generation throughput of 1
token/s for the first time with an effective batch size of 144. On the HELM
benchmark, FlexGen can benchmark a 30B model with a 16GB GPU on 7
representative sub-scenarios in 21 hours. The code is available at
this https URL