TurnBench: measuring the timing of conversation

We believe being a good conversation partner comes down to three fundamental skills: saying the right thing, in the right way, at the right time. In our last technical post, Crossing the uncanny valley of conversational voice, we focused on the importance of unlocking emotionally rich voices through delivery and prosody.

Human conversation is a complex social dance, where timing matters as much as content and delivery. The typical gap between turns is only ~200 milliseconds [1], which is shorter than the ~600 ms it takes a person to plan a single word [2]. This means listeners aren’t just reacting to silence, but predicting when a turn will end based on prosodic and linguistic cues. A good conversation partner must handle turn changes, backchannels, overlaps, and interruptions without a second thought.

This coordination is so deeply ingrained that we tend to notice it only when it breaks. An assistant that interrupts mid-thought, talks over you, or leaves an awkward silence doesn’t feel right, no matter how natural the voice itself is. This skill—knowing when to speak, when to yield, and when to stay silent—is called turn-taking. Its absence is a common giveaway that you’re talking to a machine.

Today we’re releasing TurnBench, an open-source benchmark for evaluating turn-taking dynamics in spoken dialogue. It reflects our internal evaluation methodology with the same taxonomy, tasks, and scoring protocol. It features a public leaderboard, an interactive conversation viewer, and self-serve scoring on the dev set.

Turning this methodology into a community resource has been a team effort. We partnered with Carnegie Mellon University, Mundo AI, and Oto to record and annotate an open dataset. Additionally, researchers from National Taiwan University, Academia Sinica, and Brno University of Technology helped shape the benchmark and contributed turn-taking models for evaluation.

Why turn-taking is hard

From the perspective of a conversational agent, turn-taking is a decision the system has to make continuously, at fine temporal resolution: speak or stay silent. The correct decision depends on who holds the conversational floor, and the floor is rarely signaled explicitly.

Take these two very common transition points in a conversation:

Ideal turn-taking systems need to navigate this ambiguity. They must be both fast and accurate, capable of recognizing user intention based on the same prosodic and linguistic cues that humans use. For example, they should understand that a rising pitch at the end of a question means they should anticipate a turn-change. They should also recognize the semantic difference between an interrupting user and a backchanneling user.

To build such a system, we need a way to measure the nuances of turn-taking. Meaningful measurement requires more than a single accuracy number: how often does the system fire during a mid-turn pause? Does it mistake a backchannel for an interruption? How quickly does it commit once a turn has actually ended? When we looked for an evaluation framework that could answer these questions, we found that none existed.

Where current benchmarks fall short

One problem is a lack of domain diversity. For the past few decades, the default evaluation set for turn-taking has been Switchboard, a collection of phone calls between strangers recorded in the early 90s. The problem is that those calls have a very specific rhythm: they leave short gaps, overlap in predictable ways, and have very similar content. Day-to-day conversation (say, talking over dinner with two friends) sounds different. A model that performs well on Switchboard may nevertheless perform poorly in other situations.

The second problem is that turn-taking events are hard to label. For example, most papers identify backchannels using heuristic rules or a predefined list, but deterministic rules don’t capture how these events vary across conversational contexts. Instead, we should base labeling methodology on linguistic research and train annotators to label events accordingly.

What TurnBench measures

TurnBench grounds its labels in conversation analysis, a framework in applied linguistics that studies how speakers coordinate the floor. Its core concept is the transition-relevance place: the structural point at which a change of speaker becomes legitimate. Around these critical junction points, four distinct events can happen:

• Turn: A contiguous set of vocalizations by one speaker, bounded by a shift in who holds the floor.
• Interruption: A vocalization where the listener intends to take the floor mid-turn.
• Backchannel: A vocalization that signals acknowledgment without claiming the floor.
• Mid-turn pause: A silence within a turn where the speaker doesn’t actually yield the floor.

Our scoring ultimately reduces to these four events. Beneath that, annotators labeled a finer taxonomy of 17 categories that are preserved in the open dataset for deeper research.

The challenge of turn-taking boils down to determining when it’s appropriate to speak and when it’s not. This yields two evaluation tracks:

• End-of-Turn (EOT) detection: For each speaker, the system declares the times at which that speaker gives up the floor.
  • True Positives (TP): Real end-of-turn floor transitions.
  • False Positives (FP): Mid-turn pauses.
• Interruption (INT) detection: For each speaker, the system declares the times at which the listener took the floor mid-turn.
  • True Positives (TP): Legitimate barge-ins.
  • False Positives (FP): Backchannels, noise, channel bleed.

The corpus

TurnBench is built on 30 hours of dual-channel dyadic English speech: 154 dialogues featuring 106 voice actors in 53 pairs. We recorded the conversations in a professional studio with one speaker per channel, in separate sound-isolated booths, so the two sides can be scored independently. Every dialogue is triple-annotated under the above protocol, with strong inter-annotator agreement (Fleiss’ κ = 0.78).

The corpus is balanced across six conversation types: casual social talk, task-oriented dialogue, instructional exchanges, collaborative problem-solving, argumentative disagreement, and narrative storytelling. Each targets a different distribution of turn-taking behavior; for example, argumentative dialogue is dense with competitive interruptions while instructional conversations feature longer turns and fewer turn changes.

Alongside the evaluation dataset, we’re also releasing a 104-hour training set, hand-labeled under the same protocol, so that anyone building a supervised turn-taking model can train and evaluate on consistent ground. The benchmark itself is split into a public dev set for self-serve scoring and a hold-out test set, with labels withheld to prevent overfitting to the leaderboard.

Evaluation protocol

To evaluate a model’s responses, we search a window around each human-labeled event (0.25s before to 3s after) and check whether the model committed a matching event within that window. A false end-of-turn (EOT) that fires mid-turn usually means the model mistook a pause for the speaker being done. On the other hand, a false interruption (INT) is the opposite: where the model thought someone was cutting in, but it was really just noise or a backchannel. We report recall, false-positive rate, and the signed latency of each decision at the 10th, 50th, and 90th percentiles. In this case, negative latency means the model fired before the event start time. Submissions are ranked by recall, capped by a maximum false-positive rate, so an oversensitive model cannot buy recall by firing constantly.

We score committed events rather than probability curves. A deployed turn-taking model must act causally at a single operating point: at each moment either taking the turn or yielding. Scoring the decision at each moment matches that reality. Furthermore, operating on decisions gives us a common ground for every model: those that emit probabilities can simply be thresholded into decisions, while those that never expose them, like Moshi or the Gemini API, can be evaluated directly from their behavior. This way, any model capable of turn-taking can be compared under the same protocol.

What we learned benchmarking 14 turn-taking systems

We ran TurnBench on 14 turn-taking systems and configurations spanning the real-world applications being built today: from a simple voice-activity-detection (VAD) baseline, to commercial endpointers trained to detect turn-ends, to full-duplex dialogue models evaluated in zero-shot settings. Every system was frozen at a single operating point tuned on the development set, then scored once on the test set, to best reflect deployment conditions.

Three findings stand out:

1. End-of-turn detection is consistent across conversation types, but false positives are not. Systems detect turn-ends equally well across conversation types, but misfires vary. False interruptions peak in backchannel-heavy casual talk. This highlights that casual conversation is actually the hardest mode for current conversational systems to handle.
2. There’s a tradeoff between speed and accuracy. Purely acoustic systems that fire on the first frame of listener speech catch interruptions quickly but imprecisely because, in that instant, a backchannel and a real interruption look the same. Systems that wait for more evidence are precise but slow. This also applies to the EOT track.
3. The best system is still slow where it matters. Voice Activity Projection (VAP) leads both tracks (0.845 recall at 0.055 false-positive rate on end-of-turn, 0.945 at 0.107 on interruption) but takes nearly a second on average to commit an interruption. Though systems around 200ms exist, they are trigger-happy on false positives. No system is fast, selective, and high-recall at the same time.

Limitations

First, our interruption track scores the user’s audio. This works for the endpointer-style and cascaded systems we evaluate, but full-duplex models generate and listen at the same time, so scoring their interruptions requires a protocol that accounts for the agent’s own speech.

Second, TurnBench is English-only. The structure of turn-taking is universal, but the timing norms differ: average gaps range from ~7 ms in Japanese to ~470 ms in Danish [1]. A model calibrated on English norms might feel sluggish to a Japanese speaker and pushy to a Danish one.

Third, the corpus is studio-recorded and dyadic. In-the-wild audio is remarkably noisy, frequently featuring background noise, echo, and background chatter. Clean-room turn-taking scores may not generalize to those conditions.

Conclusion

What we’ve described here is the same methodology we used to develop our own models, providing a reliable signal to separate real gains from noise. With TurnBench, we are now making that open to anyone.

We believe turn-taking is at the same inflection point that speech synthesis reached a year ago, when standard quality benchmarks became saturated and could no longer distinguish good models from great ones.

By open-sourcing our benchmark methodology alongside a corpus built specifically for this purpose, we hope to accelerate progress in this area. We invite the community to build upon and improve upon these results. The corpus, training set, scoring code, and leaderboard are available here:

• Leaderboard and interactive viewer: turnbench.sesame.com
• Self-serve dev-set scoring and submission format: turnbench.sesame.com/dev
• Paper (arXiv preprint): arxiv.org/abs/2608.25218

The data is distributed under a non-commercial license that prohibits voice cloning.

Join us

Turn-taking is one piece of our effort to build voice agents that feel genuinely present. If you’re excited about the timing, structure, and dynamics of natural conversation, and about the models and measurements that get us there, check out our open roles. We’re hiring.