Open Whisper-style Speech Models (OWSM)
Overview
Open Whisper-style Speech Models (OWSM, pronounced as “awesome”) are a series of speech foundation models developed by WAVLab at Carnegie Mellon University. We reproduce Whisper-style training using publicly available data and our open-source toolkit ESPnet. By publicly releasing data preparation scripts, training and inference code, pre-trained model weights and training logs, we aim to promote transparency and open science in large-scale speech pre-training.
Demo
Both demos take the same audio, so the autoregressive and non-autoregressive models can be compared directly.
Pre-trained models
We publicly release a series of pre-trained models. The training logs are also available for major models. We recommend OWSM v4, which outperforms OWSM v3.1 on language identification and multilingual ASR.
Data details
OWSM v4 is trained on 320k hours. That is the OWSM v3.1 mixture below plus YODAS_OWSMv4, a newly curated 166k-hour subset of YODAS2 covering 75 languages, which we release as well.
YODAS2 is web-sourced, so it carries inaccurate language labels and misaligned audio-text pairs, and in our preliminary experiments that noise hurt the downstream ASR models. The curated subset is the output of a three-stage cleaning pipeline built from public toolkits; the scripts are in the recipe. Scaling and cleaning the data is what the gain over v3.1 comes from – the architecture is unchanged apart from the subsampling noted below.
OWSM v4 also subsamples the log-Mel features 8 times rather than 4, so the encoder works at 80 ms rather than 40 ms.
The OWSM v3.1 models are trained on a diverse combination of public datasets as listed below.
OWSM v3.1 training data mixtures
- AIDATATANG
- AISHELL-1
- AMI
- Babel
- Common Voice
- Googlei18n
- CoVoST2
- Fisher Callhome Spanish
- Fisher (Switchboard)
- FLEURS
- GigaSpeech
- GigaST
- KsponSpeech
- LibriSpeech
- MagicData
- Multilingual LibriSpeech
- MuST-C
- ReazonSpeech
- Russian Open STT
- SPGISpeech
- TEDLIUM3
- VCTK
- VoxForge
- VoxPopuli
- WenetSpeech
The low-restriction model is trained on a subset of the above data with “more flexible licenses”.
OWSM v3.1 low-restriction data
- AMI: CC-BY-4.0
- Common Voice: CC0-1.0
- FLEURS: CC-BY-4.0
- KsponSpeech: MIT
- LibriSpeech: CC-BY-4.0
- Multilingual LibriSpeech: CC-BY-4.0
- VCTK: CC-BY-4.0
Inference
Similar to other ESPnet models, the pre-trained OWSM models can be easily downloaded and used in a python script. Below are some examples using OWSM v4. For earlier versions (v2 and before), the language code should follow the two-letter format (e.g., <en>, <de>).
With OWSM v4, leave maxlenratio at its default of 0.0 and avoid small positive values such as 0.5, which truncate the output. 0.0 allows the same maximum length as 1.0 – the number of encoder frames – and it also turns on the beam search’s end detection, so decoding stops once the search has converged instead of running to the maximum length.
Language Identification
We pass the Hugging Face model tag when initializing Speech2Language. The model will be automatically downloaded from Hugging Face to a local cache directory.
from espnet2.bin.s2t_inference_language import Speech2Language
s2l = Speech2Language.from_pretrained(
model_tag="espnet/owsm_v4_medium_1B",
device="cuda",
nbest=3, # return nbest prediction and probability
)
import soundfile as sf
speech, rate = sf.read("audio.wav")
result = s2l(speech)
print(result)
# list of tuples (language, probability)
# [('<eng>', 0.9994348883628845), ('<jpn>', 0.00010286537144565955), ('<rus>', 6.185896199895069e-05)]
Speech Recognition or Translation
We use Speech2Text for speech recognition or translation. We also pass the model tag so that the model can be automatically downloaded. When initializing this object, we set the default values for lang_sym, task_sym and predict_time. These variables can be overwritten later, which provides more flexibility. Note that the language must be known to use this functionality. If it is unknown, one can first perform language identification and then recognition or translation.
from espnet2.bin.s2t_inference import Speech2Text
s2t = Speech2Text.from_pretrained(
model_tag="espnet/owsm_v4_medium_1B",
device="cuda",
beam_size=5,
ctc_weight=0.0,
maxlenratio=0.0,
# below are default values which can be overwritten in __call__
lang_sym="<eng>",
task_sym="<asr>",
predict_time=False,
)
import soundfile as sf
speech, rate = sf.read("audio.wav")
result = s2t(speech)[0][-2]
# an optional text prompt can be passed
result = s2t(
speech,
text_prev="this is an optional prompt"
)[0][-2]
# lang_sym, task_sym, predict_time can be overwritten
result = s2t(
speech,
lang_sym="<eng>",
task_sym="<st_zho>", # translation into Chinese
predict_time=True,
)[0][-2]
Long-form Speech Recognition or Translation
OWSM processes an entire audio recording in a chunk-by-chunk manner. Each chunk has a fixed length of 30s. The chunk is shifted based on the predicted timestamps. We still use Speech2Text but we call its decode_long method.
from espnet2.bin.s2t_inference import Speech2Text
s2t = Speech2Text.from_pretrained(
model_tag="espnet/owsm_v4_medium_1B",
device="cuda",
beam_size=5,
ctc_weight=0.0,
maxlenratio=0.0,
# below are default values which can be overwritten in __call__
lang_sym="<eng>",
task_sym="<asr>",
)
import soundfile as sf
speech, rate = sf.read("covid.wav")
result = s2t.decode_long(speech)
# list of tuples (start_time, end_time, text)
Fine-tuning on custom data
The s2t1 template carries a worked guide:
How to fine-tune pre-trained OWSM.
It fine-tunes on AISHELL-1 end to end, in three steps:
- Set up an
s2t1recipe under your dataset withegs2/TEMPLATE/s2t1/setup.sh egs2/<your_data>/s2t1, then pull the pre-trained model down with./s2t.sh --download_model <hf_tag> --stage 100 ..., which leaves anexpdirectory linking the checkpoint, config, stats and BPE model. - Prepare the data in OWSM format — see
Data preparation
for the
text,text.prevandtext.ctclayout. Fine-tuning does not build a new vocabulary: reuse the token list and BPE model from the pre-trained model, and keep the special tokens such as language codes consistent with it. - Train, with
init_parampointing at the pre-trained checkpoint. Collect shapes withextract_feats_in_collect_stats=falseand copy the existing mean and variance across rather than recomputing them.
The learning rate and batch size in that config are a starting point, not a recommendation. A model this size overfits a small training set quickly.
For a Python-only route rather than the shell recipes, see “ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration” (SLT 2024).
Papers
Please cite our papers if you find OWSM helpful.
- INTERSPEECH 2025: OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning — Best Student Paper Award
- ACL 2024: OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identification
- INTERSPEECH 2024: On the Effects of Heterogeneous Data Sources on Speech-to-Text Foundation Models
- INTERSPEECH 2024: OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformer
- ASRU 2023: Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Data
We also collect other papers related to OWSM. Please contact Yifan Peng (yifanpen@andrew.cmu.edu) if you use OWSM in your work and want to include it here.
OWSM applications
- ASRU 2023 SPARKS Workshop: SLUE-PERB: A Spoken Language Understanding Performance Benchmark and Toolkit
Foundational work used by OWSM
- INTERSPEECH 2023: A Comparative Study on E-Branchformer vs Conformer in Speech Recognition, Translation, and Understanding Tasks
- SLT 2022: E-Branchformer: Branchformer with Enhanced merging for speech recognition
- ICML 2022: Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding