AI latency measures how long a request or token takes, while throughput measures how much work a serving system completes per ...