Back to Blog
Open Source

Sesame CSM Explained: The Open Conversational Speech Model

Sesame CSM is an open conversational speech model: a Llama backbone plus Mimi codec under Apache 2.0. Learn what CSM-1B does, how to run it and its limits.

8 min read
By OndaLabs Team

Sesame CSM is a speech generation model that turns text, plus optional earlier conversation audio, into speech. The open release, CSM-1B, is a base model under Apache-2.0. It is not the fine-tuned model behind Sesame's voice demo, it cannot generate text, and it does not model turn-taking. Those limits matter as much as the license.

What is Sesame CSM?

CSM stands for conversational speech model. Sesame's research post, dated 2025-02-27, frames the goal as "voice presence", which it breaks into four parts: emotional intelligence, conversational dynamics, contextual awareness and consistent personality. CSM is the speech generation piece of that effort. According to the GitHub repository, it generates "RVQ audio codes from text and audio inputs", and its architecture "employs a Llama backbone and a smaller audio decoder that produces Mimi audio codes".

Sesame's current website focuses on personal AI agents and smart glasses, and it does not mention CSM. No 2026 successor to the open model was found.

How CSM works

CSM runs two autoregressive transformers, both in the Llama family. The process runs in four steps:

  1. Text is tokenized with a Llama tokenizer.
  2. Audio is represented with Mimi, a split-RVQ codec that produces one semantic codebook and N-1 acoustic codebooks per frame at 12.5 Hz.
  3. The backbone predicts the zeroth codebook for each frame.
  4. A separate, smaller audio decoder predicts the remaining codebooks. To save compute, that decoder trains on a random 1/16 of audio frames.

The Hugging Face Transformers documentation describes CSM as "the first open-source contextual text-to-speech model released by Sesame". It also says the model uses Kyutai's Mimi codec, the same codec family used by Kyutai's own open TTS model.

Model sizes and what was released

The research post lists three sizes. Only the smallest was released as an open model.

| Research size | Backbone | Audio decoder | Released as open model? | |---|---|---|---| | Tiny | 1B | 100M | Yes, as CSM-1B | | Small | 3B | 250M | No | | Medium | 8B | 300M | No |

Training used about one million hours of mostly English audio from publicly available sources, with sequences of 2048 tokens (about 2 minutes) and five epochs. Be careful with the parameter count. GitHub and the research post describe CSM-1B as 1B, meaning the backbone. The Hugging Face page lists 2B total parameters in F32, so say "a 1B-parameter backbone plus a smaller decoder" if precision matters.

What Sesame actually released

  • The CSM-1B base model and its code, under Apache-2.0. The model was released on 2025/03/13.
  • Transformers support from version 4.52.1, announced 2025/05/20.

Not released:

  • The fine-tuned variant that powers Sesame's interactive voice demo. The GitHub README says that variant "is not part of this release".
  • The 3B and 8B sizes described in the research post.
  • The 20-plus languages the research post plans to add.

The README describes the open model as "a base generation model" that "has not been fine-tuned on any specific voice".

How to run CSM-1B

The setup is more involved than a pip install:

  • Hardware: a CUDA-compatible GPU. The repository says it was tested on CUDA 12.4 and 12.6. Sesame publishes no official memory figure.
  • Python: 3.10 is recommended, and ffmpeg may be needed.
  • Hugging Face access: the Llama-3.2-1B and CSM-1B repositories are gated, so you must agree to share contact details before downloading.
  • Windows: the README recommends triton-windows.
  • Context prompts: speaker prompting uses Segment objects that pair a transcript with audio. In Transformers, the text prompt uses a speaker id tag such as [0], and the processor accepts conversation turns that include audio.

What CSM can and cannot do

| Capability | CSM-1B status | |---|---| | Generate speech from text | Yes | | Use earlier audio as context | Yes, through speaker prompting | | Generate text or hold a conversation alone | No. Pair it with a separate language model | | Model turn-taking and pauses | Not modeled, according to the research post | | Clone a specific voice | Not documented. The base model is not fine-tuned on any voice | | Non-English speech | Some capacity, but the Hugging Face page lists English only |

The research post says CSM "models the content of speech and text, not conversation structure such as turn-taking and pauses". The Decoder, a secondary source, reports unnatural pauses and audio artifacts in longer dialogues.

How good is it?

Sesame's research post reports a subjective test on the Expresso dataset using CSM-Medium, the 8B model. Eighty paid raters each judged about 15 examples. Without context, the win-loss ratio was 50:50, meaning no clear preference. With 90 seconds of prior audio and text as context, raters "consistently favored the human recordings". Sesame also describes word error rate and speaker similarity as saturated and close to human performance, but those figures appear only in charts.

These are vendor results for the 8B model, not for the 1B release that people can download. Read them as a claim to test, not a benchmark you can rely on.

Safety and acceptable use

The Hugging Face model page says the model is "intended for research and educational purposes". Its prohibited uses include impersonation or fraud, including mimicking real people without explicit consent, misinformation or deception, and illegal or harmful uses. The GitHub README says: "Do not use this model to generate speech that mimics real individuals without their explicit consent." The Decoder reports that the only safeguard is guidelines, with no technical controls.

The license is the tricky part. Apache-2.0 normally permits commercial use, but the model card's research wording points the other way. Read both before you build a product on CSM-1B.

CSM compared with other open models

CSM-1B does a different job from Kokoro's prebuilt voices or Chatterbox's voice cloning. It is a context-aware base model that needs a GPU and gated model access. For a side-by-side view of licenses and sizes across open models, see the open source TTS roundup, and for a smaller model with prebuilt voices, see Kokoro TTS explained.

Where hosted dialogue fits

If you need a two-voice scene and do not want to manage a gated GPU model, a hosted dialogue tool is simpler. OndaLabs dialogue is a two-speaker tool that is included on the free plan, and delivery is directed in plain words and with supported expressive tags. The free plan is $0 with 2,000 credits per month, about 2 minutes of speech. OndaLabs dialogue does not use conversation context the way CSM does, and this article does not claim it matches CSM quality. Compare plans on the pricing page before you sign up.

Frequently asked questions

What is Sesame CSM?

Sesame CSM is a conversational speech model that generates speech from text and optional audio context. The open release, CSM-1B, uses a Llama backbone and the Mimi codec and is licensed under Apache-2.0. It is a base model and does not generate text.

Is Sesame CSM open source and free for commercial use?

CSM-1B is released under Apache-2.0, but its Hugging Face card says the model is intended for research and educational purposes. Read the card and the license together before any commercial use. The card also prohibits impersonation and mimicking real people without consent.

How do I run CSM-1B locally?

You need a CUDA-compatible GPU, Python 3.10 is recommended, and you must have Hugging Face access to the gated Llama-3.2-1B and CSM-1B repositories. Install the code from the GitHub repository or use the Transformers implementation from version 4.52.1 onward. Sesame publishes no official memory figure, so test on your own hardware.

Does CSM clone voices?

The README documents prompting with audio segments, not a voice cloning feature. The base model "has not been fine-tuned on any specific voice". Do not plan a project around cloning a specific person with CSM-1B.

What languages does CSM support?

The Hugging Face page lists English only. The research post says the model has "some capacity for non-English languages" due to data contamination, but that it likely will not do well. Expansion to 20-plus languages is a plan in the research post, not a release.

Can CSM hold a conversation by itself?

No. CSM generates speech and cannot generate text, so it needs a separate language model to produce replies. It also does not model turn-taking or pauses, so conversation timing must be handled by your application.