← Home

Continuous Batching

Step 1 / 15
▲ Static Batching waits for full batch to finish
▼ Continuous Batching slots in new requests each iteration
Static Batching
Batch 1 ends: t = 8
Batch 2 ends: t = 13
Idle slot-steps: 8 of 39 total
GPU utilization ≈ 79%
Continuous Batching
All 6 requests done: t = 11
2 iterations faster than static
Idle slot-steps: 0 while busy
GPU utilization ≈ 85%