Inference
Streaming
Sending a model's output to the user token by token as it is generated, instead of waiting for the full answer.
Streaming doesn't make generation faster, but it makes an app feel faster because the user starts reading immediately.
Frequently asked questions
What does Streaming mean?
Sending a model's output to the user token by token as it is generated, instead of waiting for the full answer. Streaming doesn't make generation faster, but it makes an app feel faster because the user starts reading immediately.