Milliseconds Are a Product Decision: Drafting a Real Latency Budget for Generative Media
Real-time generative media has a way of making infrastructure debates feel philosophical. A demo runs at 24 frames per second on a laptop, everyone nods, and then someone asks why the production avatar feels like it is thinking about every sentence. The answer is rarely one slow model. It is a stack of small delays that nobody assigned to a budget.
At Hedra AI, we have been testing a deliberately boring pipeline: a 720p webcam feed driving a 512×512 talking-head avatar, plus a separate text-to-image path that renders 1024×1024 stills. We ran it in three places—a cloud GPU in us-east-1, an edge PoP with GPU capacity, and a local laptop with an NPU—and logged every stage. This is not a vendor scorecard. It is a worksheet for product teams who need to promise 'instant' without lying.
The user-visible promise comes first
Latency budgets fail when they start with GPUs. Start with the moment a user notices. For a conversational avatar, mouth-to-pixel delay under 300 ms feels tight; 300–700 ms feels acceptable if the avatar has natural pauses; past 900 ms, people start talking over it. For image generation, 1–2 seconds feels responsive for a draft, while anything over 4 seconds needs progress UI and a cancel button. Write those numbers down before you benchmark anything.
The test rig
Our avatar path had eight stages: capture, preprocess, uplink, queue, model inference, postprocess, encode, and client render. The image path swapped capture for prompt parsing and added a safety check. We measured p50 and p95 over 2,000 sessions per deployment, with warm models and a cold-start run set aside. The numbers below are illustrative from our rig, not a universal law.
| Stage | Cloud p50 | Edge p50 | On-device p50 |
|---|---|---|---|
| Capture + preprocess | 18 ms | 18 ms | 14 ms |
| Uplink | 42 ms | 12 ms | 0 ms |
| Queue | 120 ms | 22 ms | 2 ms |
| Model inference | 85 ms | 110 ms | 170 ms |
| Postprocess + encode | 54 ms | 46 ms | 46 ms |
| Downlink + render | 55 ms | 22 ms | 8 ms |
| Total | 374 ms | 230 ms | 240 ms |
The p95 tells the more honest story: cloud hit 690 ms, edge 410 ms, and on-device 360 ms before thermal throttling. The cloud number was not dominated by the model. Queue time and network jitter were the expensive parts. The edge deployment cut transport delay but paid for it with less headroom and a smaller capacity pool. On-device looked great on a cool laptop and less great after 12 minutes of continuous use.
For 1024×1024 image generation, the spread was wider: roughly 2.8 seconds cloud, 4.1 seconds edge, and 9.5 seconds on-device for 30 steps. That changes the product. A cloud image feature can feel like a slot machine; an on-device one needs streaming previews or a queue.
Where the dollars actually go
Compute is the visible bill. Idle capacity is the expensive one. Cloud GPUs charge by the hour whether or not a user is mid-sentence; edge capacity costs more per unit but can be sized closer to demand; on-device shifts marginal cost to the user's battery and your support team. In our rig, the relative cost per 1,000 avatar-minutes landed roughly 1.0× cloud, 1.4× edge, and 0.2× on-device marginal cost—before device fragmentation, app size, and model updates. That last column is not free. It is just someone else's invoice.
The practical move is to separate cost per session from cost per idle minute. A real-time feature with spiky usage needs warm pools, pre-warmed models, and a queue that degrades gracefully. A batch image feature can tolerate cold starts if the UI sets expectations. Same model, different budget.
A latency budget worksheet you can steal
Pick one user promise and cap the whole path. Then allocate stages, not vibes.
| Line item | Example target | Notes |
|---|---|---|
| User-visible promise | 500 ms p50, 900 ms p95 | Avatar conversation |
| Capture + preprocess | 40 ms | Client-side crop and resize |
| Network round trip | 120 ms | Use p95, not speed test |
| Queue + scheduling | 100 ms | Include cold starts separately |
| Model inference | 180 ms | Warm model, batch size 1 |
| Postprocess + encode | 60 ms | Measure codec, not just model |
| Headroom | 20% | Reserve for jitter and retries |
If the total exceeds the promise, cut scope before you cut quality: lower resolution, fewer steps, shorter context, or a draft-first UI. A latency budget is a product design artifact. Put it in the same doc as the acceptance criteria.
What broke in our runs
Queue time was the sneaky one. A model that took 85 ms felt slow because it waited 120 ms behind a batch. Cold starts added 1.8–4.5 seconds and were invisible in our warm benchmarks. Client rendering added 20–40 ms on older laptops. And retries—one dropped frame, one reconnect—turned a 374 ms p50 into a 1.1 second p95.
The fix was not a faster model. It was instrumentation with one trace ID across client, edge, and GPU; a queue metric that separated wait from inference; and a rule that every new feature had to declare its latency budget before launch. Boring, but it kept us honest.
The takeaway
Real-time generative media is a scheduling problem wearing a model costume. Benchmark the whole path, publish p95 next to p50, and price idle capacity as part of the feature. In 2026, the teams that ship convincing real-time experiences will not be the ones with the biggest GPUs. They will be the ones who know exactly where their milliseconds go—and who decided, on purpose, which ones users will never feel.
