High throughput makes long answers appear quickly. It depends on the model, the provider's hardware and current load, so measured speeds vary over time.
Frequently asked questions
What does Throughput mean?
How fast a model produces output, usually in tokens per second. High throughput makes long answers appear quickly. It depends on the model, the provider's hardware and current load, so measured speeds vary over time.