Training
Quantization
Storing a model's weights with fewer bits (for example 4 or 8 instead of 16) to cut memory use and speed it up, with a small loss in quality.
Quantization is how large open-weight models fit on consumer GPUs and laptops. Labels like Q4 or int8 describe how aggressively a model has been compressed.
Frequently asked questions
What does Quantization mean?
Storing a model's weights with fewer bits (for example 4 or 8 instead of 16) to cut memory use and speed it up, with a small loss in quality. Quantization is how large open-weight models fit on consumer GPUs and laptops. Labels like Q4 or int8 describe how aggressively a model has been compressed.