Inference
Rate limit
The cap a provider sets on how many requests or tokens you can use per minute or per day.
Limits usually rise with your spending tier. Applications at scale need retries with back-off and sometimes several providers to stay within limits.
Frequently asked questions
What does Rate limit mean?
The cap a provider sets on how many requests or tokens you can use per minute or per day. Limits usually rise with your spending tier. Applications at scale need retries with back-off and sometimes several providers to stay within limits.