Today we're sharing the architecture behind our Persona models: photoreal, expressive digital humans that hold a video call in real time. The design follows from one observation. Nearly all of the bits in a video frame describe appearance, which barely changes over a conversation; nearly all of the difficulty is in behavior, which changes every frame. We factor the avatar accordingly: a small behavior model decides what the face does, a generative renderer produces the pixels, and the two communicate through a compact learned representation we call semantic codes.

This factorization is why our Personas are photoreal, expressive, and fast at the same time. It is also why a single GPU serves many conversations at once, which is what brings the price as low as $0.06/minute.

The pixel tax

A conversational avatar produces a photoreal frame every 40 milliseconds, in sync with audio that arrives as the user speaks. That budget bounds how much computation each frame can receive, per user, for as long as the conversation lasts. Existing systems spend it in one of two ways.

End-to-end video diffusion. A single model maps audio directly to pixels.¹ Everything is learned and everything is modeled, so in principle nothing about a face or a scene is out of reach. However, expense follows from the same property: whether the model diffuses pixels or VAE latents, the representation must describe the entire scene — identity, lighting, background — not just what the person is doing, so a better-looking avatar means a larger model or more sampling steps, and therefore more latency and a higher cost per stream. Real-time deployments manage this with aggressive step distillation and constrained resolutions.

Fixed intermediate representations. The alternative drives an explicit representation from audio: facial keypoints, blendshape or 3DMM coefficients,² or more recently Gaussian-splat rigs.³ Rendering is cheap, so real time comes easily, and appearance is stable by default. The limitation is the representation itself. It is built from scan collections, annotation schemes, or mesh geometry rather than from conversational video itself, and it acts as a ceiling: facial behavior it cannot express — asymmetries, micro-expressions, the mannerisms particular to one person — is lost before the driving model ever trains, and no amount of capability in that model can recover it.

The bitter lesson⁴ favors the first approach: learned components eventually outperform designed ones. We agree. It does not follow, though, that the whole system must be one model. We keep the factored structure of the second family and replace its fixed representation with a learned one — the same structure latent diffusion uses for image generation, where the generative model operates in a learned latent space and a decoder produces the pixels.⁵ Generation still happens in a learned space, but one sized for behavior rather than for video.

Three architectures: end-to-end diffusion generates the entire scene every frame, fixed rigs cap what behavior can be expressed, and semantic codes carry only behavior into an amortized renderer

Behavior is small

A video frame carries megabits of information; the behavior in it — head pose, gaze, mouth shape, the set of the eyebrows — amounts to orders of magnitude less. Most of what a video call transmits is appearance and background, restated 25 times a second.

We make the factorization literal.⁶ From conversational video we learn a compact code describing what a face is doing at each moment, and a diffusion renderer⁷ produces the full frame given the codes and the persona's appearance.

Nothing dictates what the codes represent — there is no landmark detector, blendshape basis, or 3D prior in the loop — yet after training they separate pose, gaze, and expression from identity, lighting, and background. By the time the behavior model trains, the code space is as fixed as any rig; what distinguishes it is that it was learned from conversational video in the first place, so its capacity reflects what faces actually do.

The factorization buys three things:

A fast behavior model. Generating codes is a sequence-modeling problem thousands of times smaller than generating pixels, so the latency between hearing audio and deciding what the face does is a few milliseconds.

An amortized renderer. Photorealism lives entirely in the renderer, and the code space acts as a stable interface: a better renderer can be trained against the same codes and swapped in without touching the behavior model.

Consistency by construction. Autoregression happens only in code space, and every frame is rendered fresh from codes and appearance, so identity and texture cannot drift.⁸

The same code trajectory rendered under four appearance conditions — behavior is generated once, in a space shared by every persona

Generating behavior

The behavior model is chunked-autoregressive: it generates the next window of semantic codes from the conversation's audio, an emotion target, and the codes it has already produced.

We treat this as conditional diffusion over code sequences: a transformer denoises windows of future codes, conditioned on the audio itself rather than on a transcript or phonemes. Working from raw audio makes the system agnostic to the TTS and language driving it, and lets it respond to non-verbal cues like laughter and hesitation.

The model is trained as a continuation model: it learns to continue a sequence of codes already in place, the same task it performs in deployment, so chunk boundaries are seamless.

The loss also penalizes errors in the motion rather than treating frames independently; without this, output that looks correct frame by frame still jitters.

Streaming generation: each window of codes is generated from the live audio and the codes already committed, and the renderer consumes them as they land

Emotion is an input

Prosody already carries emotional information, and most audio-driven systems rely on it alone. For some deployments this is limiting: an agent may need to look sympathetic while a neutral TTS voice delivers bad news, or stay composed while the audio gets excited. We therefore make emotion an explicit conditioning signal to the behavior model, supplied per window at inference, so the agent driving the avatar can steer expression mid-conversation.

During training, each conditioning signal — audio, emotion, motion prefix — is independently dropped to a learned null embedding.⁹ At inference this yields a separate classifier-free guidance scale per signal, composed leave-one-out in a single batched forward pass:

v_{\text{guided}} = v_{\text{cond}} + \sum_{s} (\gamma_s - 1)\big(v_{\text{cond}} - v_{\neg s}\big)

where $v_{\neg s}$ is the prediction with signal $s$ nulled and the others intact. The three scales control how strongly the face commits to the audio, how intensely it expresses the emotion target, and how much recent motion constrains the present; we set them with a preference-optimization pass before each release.

Per-signal guidance: one denoising step runs four conditioning branches in a single batched forward pass, each nulled branch contributing a scaled difference

The learned nulls also make the extremes well behaved: with audio nulled, the model produces natural idle behavior — blinks, small shifts in gaze and posture — rather than freezing.

The latency budget

A conversational turn reads as instant somewhere under half a second,¹⁰ and the avatar shares that budget with transport and with the voice stack driving it. The avatar model itself adds about 180 ms from receiving audio to emitting video; end to end — including transport and the leading audio the first chunk needs before inference can begin — a typical turn lands around 500 ms.

The behavior model runs a handful of denoising steps per window, with all guidance branches batched into one forward pass; it costs single-digit milliseconds. The renderer is the dominant per-frame cost, and a fixed one: it does not grow with conversation length or re-encode history. It also distills easily — the codes and the appearance conditioning already specify most of the frame — so it runs at a few steps.

Frames come out significantly faster than real time on a single GPU, so one GPU can serve many concurrent streams, and production deployments scale to hundreds of simultaneous conversations.

One conversational turn as a latency waterfall: the avatar model spends about 180 ms of a roughly 500 ms turn, most of it in the renderer

Limitations and future work

Dyadic conversation. The behavior model generates one side of the call: a Persona's behavior is driven by its own audio. Listening is a modeling problem of its own — reacting to the user's speech, backchanneling, holding attention between turns — and it is where much of our current research effort goes.

Richer affect. Discrete emotion targets cover a lot of product surface, but real affect is a blend with dynamics of its own; continuous emotion conditioning is the natural next step.

Longer-horizon behavior. Continuation training makes chunk-to-chunk streaming seamless, but behavior also has structure at the scale of minutes — postures you return to, mannerisms, a temperament. Extending the behavior model's memory without spending the latency budget is one of the problems we find most interesting right now.

Try it

persona-1.5-live is live today — create an avatar with it and embed it in a website in minutes, or drive it through our API with your own agent stack.

Citation

Please cite this work as:

Keyframe Labs, "Semantic Codes: Decoupling Behavior from Appearance in Real-Time Avatars",
Keyframe Labs Research Blog, March 2026.

Or use the BibTeX citation:

@article{keyframe2026semanticcodes,
  author = {Keyframe Labs},
  title = {Semantic Codes: Decoupling Behavior from Appearance in Real-Time Avatars},
  journal = {Keyframe Labs Research Blog},
  year = {2026},
  month = {March},
  note = {https://keyframelabs.com/blog/persona-1-5-live},
}

For the audio-to-video diffusion family in the offline setting, see EMO (Tian et al., 2024) and OmniHuman-1 (Lin et al., 2025). Real-time variants additionally distill the sampler to a handful of steps, e.g. via distribution matching distillation (Yin et al., 2024). ↩
Morphable models and blendshape bases are typically PCA fits to collections of 3D face scans — Blanz & Vetter (1999); FLAME (Li et al., 2017). The basis is learned, but linearly, from scans rather than conversational video, and independently of the task it will be asked to serve. ↩
3D Gaussian Splatting (Kerbl et al., 2023); rigged-avatar variants such as GaussianAvatars (Qian et al., 2024). ↩
Sutton, R. The Bitter Lesson, 2019. ↩
Rombach et al., High-Resolution Image Synthesis with Latent Diffusion Models, 2022. ↩
Learned motion latents have precedent in talking-head research — e.g. MegaPortraits (Drobyshev et al., 2022) and VASA-1 (Xu et al., 2024), which build them on reenactment objectives and explicit disentanglement machinery. Our codes come from a different construction and are built to a different specification: windowed generation under guidance. ↩
Ho et al., Denoising Diffusion Probabilistic Models, 2020; Song et al., Score-Based Generative Modeling through Stochastic Differential Equations, 2021. ↩
Error accumulation is the standard failure mode of long autoregressive rollouts in pixel or latent video space; representative mitigations include Diffusion Forcing (Chen et al., 2024), CausVid (Yin et al., 2025), Self Forcing (Huang et al., 2025), and FramePack (Zhang & Agrawala, 2025). ↩
Ho & Salimans, Classifier-Free Diffusion Guidance, 2022. ↩
Median gaps between turns in human conversation are around 200 ms across languages (Stivers et al., 2009) — faster than deliberate speech planning, because listeners project turn ends rather than react to them. In practice, half a second is roughly where an artificial interlocutor stops feeling laggy. ↩