EmoteBench

For the headline results on one page, see the EmoteBench benchmark page. This post is the full report.

A conversational avatar is judged by its face. When an agent delivers bad news, the person on the other end expects to see concern; when the conversation turns, they expect the expression to turn with it; and throughout, they expect the face to move the way a person's would. Simply put, immersion requires presence. These qualities are hard to judge from a demo, though, because the voice supplies most of what a viewer perceives, so we built a study that isolates the face.

EmoteBench takes five real-time avatar APIs, gives each the same audio for the same scripted lines, drives each through its own emotion controls, and asks blinded human raters three kinds of question: which emotion a silent face is showing; how natural, how strong, and how clear its expression is for a named emotion; and whether, and how, it changed emotion part-way through a line. Those are the three things a face has to do in a conversation, and they are what we mean when we say an avatar is expressive, natural, and controllable.

Keyframe leads every other system on every measure.

The rest of this post describes how the study was built and walks through the evidence behind each number. Every figure is interactive: hover or tap for exact values, and use the toggles to change views.

Setup

A fair comparison of avatar systems has to hold everything constant except the face. Each provider ships its own voice pipeline, identities, resolution, and emotion controls, and each of those can move a rating. We made the following choices.

Every system receives the identical audio file for a given line, generated once with ElevenLabs, so that the only thing differing between two clips of the same line is the face. Before the run we also drove Anam and Tavus through their own text-and-tag pipelines and saw no advantage over the shared audio. Each provider contributes three stock identities, two women and one man, each paired with a voice of its gender. Clips are captured live from each provider's streaming session, at whatever resolution and framing the provider delivers, and aligned to the source audio afterward.

The material is 182 scripted lines across seven emotions—happiness, sadness, anger, fear, surprise, disgust, and neutral—written in a conversational register. The six beyond neutral are the basic emotions of Ekman's account, chosen because they are the ones people recognize most consistently from the face alone.¹ Single-emotion lines run five to seven seconds. Switch lines run seven to nine seconds and change emotion once, at a clause boundary, between two emotions that follow each other in ordinary conversation: anger settling to neutral, say, or happiness interrupted by surprise.

Raters were recruited through an online research panel and rated on Mabyduck. A session is 78 screens drawn without replacement, so that no rater sees the same line twice, plus three attention checks. Each screen shows a single clip, with the system identified only by a letter, and which system's clip appears is drawn at random; over the study, every system is rated on every line by many raters. HeyGen's API has no emotion control, so it is scored only on rating screens for neutral and happy lines. Session statistics, the attention checks, and the details of how each number is computed are in the appendix.

Tasks

Recognition

The most direct test of whether a face expresses an emotion is whether someone can name it without help. In the recognition task the clip plays once with the sound off, and the rater chooses one of the seven emotions and says how sure they are, from guessing to certain. Because the voice is removed, the only information about the emotion is in the face.

Rating

Recognition tells you whether an emotion is legible, but not whether it is the right size for the moment or whether the face looks like a person while showing it. In the rating task the clip plays with the sound on, the heading names the emotion the face is meant to show, and the rater is asked to judge the face rather than the voice on three scales: how natural the movement looks, whether the amount of expression is too little, about right, or too much for what is being said, and how clearly the face shows the named emotion.

Switching

An avatar driven by an agent has to change expression when the agent does, and hold steady when it does not. In the switching task the clip plays once without sound and the rater says whether the face changed emotion, which emotion came first and which second, and whether the change looked natural. Of the 42 switching lines, 28 cue the face to switch; 7 change the voice's emotion while cueing the face to hold, and 7 change nothing at all. The two kinds of control let us separate a face that responds to direction from one that simply moves a lot.

Results

Expressiveness

With no sound, raters identified the intended emotion on 81% of Keyframe's clips. For the other three systems the figure lies between 32% and 37%, against a chance rate of one in seven. The difference is not a matter of degree on a shared scale; it is a difference in kind. An avatar without the right emotional profile cannot serve as a skeptical buyer in a sales role-play, nor as an appropriately enthusiastic tutor in a language-learning app.

The per-emotion breakdown shows where the gap comes from. The other systems do best on the two easiest targets: Tavus reads as happy 94% of the time, and Anam and LemonSlice read as neutral about three times in four. On fear, all three fall to chance or below while Keyframe is at 79%, and sadness and surprise look similar. Keyframe's weakest emotion is disgust, which raters read as anger about two times in five.

Confidence agrees with accuracy. Raters marked themselves certain on 41% of Keyframe's clips and on about a tenth of the others'. On Keyframe, confidence was more than half a point higher when raters were right than when they were wrong; on the other systems the gap was under a third of a point. Raters could tell when the face was giving them information.

Calibration and naturalness

A common concern with expressive avatars is that expression comes at the cost of realism: a face that shows emotion clearly may look overdone. The rating task checks this directly, because the same raters judged clarity, amount, and naturalness on the same clips.

With the audio on and the intended emotion named, 79% of Keyframe's clips were judged to show it clearly and 5% to show it slightly or not at all; the other systems split the opposite way, with roughly a quarter to a third clear and about half not. The amount-of-expression panel explains why. Raters judged around 40% of the other systems' clips as showing too little expression for the words, compared with 7% of Keyframe's. Two thirds of Keyframe's clips were judged just right, and its misses lean toward too much rather than too little.

Naturalness does not suffer. Keyframe's clips were rated natural, 4 or 5 on the scale, 61% of the time and unnatural 15% of the time; the other systems range from a fifth to a third natural and from a third to three fifths unnatural. Expression and naturalness rise together rather than trading against each other. This matters for any product where one avatar has to handle both routine and difficult moments, such as a patient intake followed by a bad result, or a product demo followed by a refund dispute. The pattern holds on neutral and happy lines alone, where the other systems score highest and HeyGen can be included: Keyframe scores 3.65 on naturalness against 2.34 to 3.19.

Broken out by face, Keyframe's three identities occupy the top three positions, and its lowest, Caspian at 3.42, sits above the best face from any other provider, Anam's Mia at 3.03. Within every provider the face paired with the male voice is rated lowest, by two to five tenths of a point against that provider's female faces, which suggests that part of the spread across faces belongs to the lines and the voice rather than to the faces themselves.

Controllability

An agent needs the avatar to show the emotion it asked for, when it asked for it. The switching task scores this strictly: a rater has to notice that the face changed and name both the starting and the ending emotion, which by chance would happen about two percent of the time.

Raters saw the cued switch and named both emotions on 53% of Keyframe's switch clips. Tavus reached 13%, LemonSlice 5%, and Anam 3%. The control clips measure the other direction: on lines where the face was cued to hold, or where nothing changed, raters still reported a switch on roughly a third of the other systems' clips, and on 14% of Keyframe's. Summarized as a detection index, Keyframe's d′ is 1.98 and the others fall between 0.39 and 0.62. For three of the four systems, raters could barely tell a cued switch from no switch. In a conversation, this is the difference between a face that turns sympathetic when the news turns bad and one that reacts to volume.

One number in this task is easy to misread. If you look only at clips where a rater said the face switched, and ask how often they also called the switch natural, Anam scores higher than Keyframe. The outcome bars show why the two numbers are not comparable. On 45% of Anam's switch clips, the rater noticed some movement, named emotions other than the ones cued, and judged that small movement natural; the intended switch was seen and judged natural on 2%. Keyframe's visible transitions are larger and more often the ones that were asked for, and raters called them natural three times in four. The remaining quarter is a limitation we discuss below.

Consistency

Averages over thousands of ratings can conceal a result that depends on a few lines or a few raters, so we checked the lead two ways. First, line by line: for each line, we compared Keyframe's clip with each competitor's clip of the same line and counted how often Keyframe's was rated higher. Second, rater by rater: for each of the 250 raters, we compared their average rating of Keyframe with their average rating of each competitor and counted how often Keyframe came out ahead.

Read the first table as: on 100% of the lines both systems performed, raters found Keyframe's clip more natural than Tavus's. Read the second as: 87% of raters, averaging over everything they saw, rated Keyframe more natural than Tavus. HeyGen has no recognition entry because it was not tested on that task. A stricter comparison, between individual ratings rather than averages, gives the same ordering with smaller margins: a single Keyframe naturalness rating exceeds a competitor's rating of the same line 69% to 80% of the time, and a clarity rating 79% to 87% of the time. Restricting the analysis to the sessions that passed an attention check changes none of the comparisons; the appendix has the details.

Limitations

The study points at three things we would like to improve. On switches that raters clearly saw, about one transition in four was judged too abrupt; the faces execute the change the agent asked for, but not always at a pace a person would. Disgust and surprise trail the other emotions in silent recognition, at 58% and 69%, with disgust most often read as anger. And our three identities span four tenths of a point on naturalness, which is a wider range than we would like from one model.

The study also has limits of its own. Three identities per provider and one voice per gender are small samples of each, and the male-identity effect, which appears in every provider, could as easily come from the lines and the voice as from the faces. Seven discrete emotions leave out blends and intensity, which matter in practice. Clips were recorded and then rated, not rated during a live conversation, so responsiveness, turn-taking, and listening are outside the scope; the published studies from Tavus and LemonSlice made the same choice for the same reason, that a blinded comparison requires both clips to show identical content.² Raters are a crowd panel rather than people who use these products. The results describe the versions of each system available in late October 2026. And the attention checks placed their instruction in a line raters had learned to skip, so most sessions failed them; the appendix describes this, and a future round will put the instruction in the question itself.

One assumption remains untested. The recognition task runs without sound on the premise that the voice would otherwise carry much of the emotion. That is plausible, and it motivated the design, but we did not measure it; a sound-on recognition arm and a voice-only arm would.

Code

The lines and emotion tags, identity assignments, capture scripts, Mabyduck configuration, answer key, and analysis will be released soon.

Appendix

Sessions

250 raters completed one session each, for 19,500 rated screens after attention checks are removed. The median session took 27 minutes, at a median of 14 seconds per screen. Each clip played once at native resolution and could not be scrubbed; silent-task clips carried no audio track. Within a session, no rater saw the same line on two systems. Across the study, each system's clip of a line was rated by a median of 26 raters, with a range of 6 to 72 because screens were drawn at random.

Attention checks

Each session included three attention checks, one per task. A check was a normal screen for its task, with the required answers written in the line beneath the heading, where every other screen repeated the same routine instruction ("No sound on this one. Watch the whole clip, then answer."). The recognition check asked for Sadness at full confidence over a clip of a happy face; the rating check asked for all three sliders at the far left; the switching check asked for Yes, Fear, Surprise, No in order.

Raters passed 296 of 750 checks, and 130 of 250 sessions passed at least one. The failures were not inattention to the clips. On the recognition check, 115 of the 156 raters who failed answered Happiness, the emotion the face was showing; on the switching check, 73 of 147 gave the switch that was on screen. Raters were answering the clip and skipping the instruction. We therefore report results on all sessions, and show below that restricting to the 130 passing sessions changes none of the comparisons: the recognition lead, for instance, is 44 to 50 points either way.

How the numbers are computed

Intervals are 95% bootstrap intervals that resample raters and lines together, so that they account for a result depending on a few raters or a few lines. For a system's mean, the sample is every rating of that system on the task; for a lead, it is Keyframe's ratings and the other system's on the lines both appeared on.

The detection index d′ in the switching task is the standard normal quantile of the hit rate minus that of the false-alarm rate, where a hit is a rater reporting a switch on a clip cued to switch and a false alarm is a rater reporting a switch on a hold or no-change clip. Rates use the log-linear correction, so a system with no false alarms still gets a finite d′.

In the consistency tables, the line-level comparison uses each system's mean rating of a line and counts a tie as half a win. It covers the 56 rating lines and 84 recognition lines on which the two systems both appeared; for HeyGen, the 16 neutral and happy rating lines. The rater-level comparison uses each rater's mean rating of each system within their own session. The stricter rating-level figure compares every individual Keyframe rating of a line against every individual competitor rating of the same line.

Pipeline checks

Before the run, we drove Anam and Tavus through their own text-to-speech pipelines with inline emotion tags, on a sample of lines, and compared the output with the shared-audio path by eye; we saw no case where the shared audio put either at a disadvantage. All captures were aligned to the source audio by cross-correlation after recording. No clip was cropped or re-framed.