Overview

Open Whisper-style Speech Models (OWSM, pronounced as “awesome”) are a series of speech foundation models developed by WAVLab at Carnegie Mellon University. We reproduce Whisper-style training using publicly available data and our open-source toolkit ESPnet. By publicly releasing data preparation scripts, training and inference code, pre-trained model weights and training logs, we aim to promote transparency and open science in large-scale speech pre-training.

Demo

  • OWSM v4 (encoder-decoder, 1B): Static Badge
  • OWSM-CTC v4 (encoder-only, 1B): Static Badge
  • Colab notebook: Open All Collab

Both demos take the same audio, so the autoregressive and non-autoregressive models can be compared directly.

Pre-trained models

We publicly release a series of pre-trained models. The training logs are also available for major models. We recommend OWSM v4, which outperforms OWSM v3.1 on language identification and multilingual ASR.

Name Data (hours) Encoder Parameters Model Link ESPnet Recipe
OWSM v1 38k Transformer 272M espnet/owsm_v1 egs2/owsm_v1/s2t1
OWSM v2 129k Transformer 712M espnet/owsm_v2 egs2/owsm_v2/s2t1
OWSM v2 129k E-Branchformer 739M espnet/owsm_v2_ebranchformer egs2/owsm_v2/s2t1
OWSM v3 180k Transformer 889M espnet/owsm_v3 egs2/owsm_v3/s2t1
OWSM v3.1 base 180k E-Branchformer 101M espnet/owsm_v3.1_ebf_base egs2/owsm_v3.1/s2t1
OWSM v3.1 small 180k E-Branchformer 367M espnet/owsm_v3.1_ebf_small egs2/owsm_v3.1/s2t1
OWSM v3.1 medium 180k E-Branchformer 1.02B espnet/owsm_v3.1_ebf egs2/owsm_v3.1/s2t1
OWSM v3.1 small low-restriction 70k E-Branchformer 367M espnet/owsm_v3.1_ebf_small_lowrestriction egs2/owsm_v3.1/s2t1
OWSM-CTC v3.1 medium 180k E-Branchformer 1.01B pyf98/owsm_ctc_v3.1_1B Check model page
OWSM v3.2 small 180k E-Branchformer 367M espnet/owsm_v3.2 Coming soon
OWSM-CTC v3.2 medium 180k E-Branchformer 1.01B espnet/owsm_ctc_v3.2_ft_1B egs2/owsm_ctc_v3.1/s2t1
OWSM v4 base 320k E-Branchformer 102M espnet/owsm_v4_base_102M egs2/owsm_v4/s2t1
OWSM v4 small 320k E-Branchformer 370M espnet/owsm_v4_small_370M egs2/owsm_v4/s2t1
OWSM v4 medium 320k E-Branchformer 1.02B espnet/owsm_v4_medium_1B egs2/owsm_v4/s2t1
OWSM-CTC v4 medium 320k E-Branchformer 1.01B espnet/owsm_ctc_v4_1B egs2/owsm_ctc_v4/s2t1

Data details

OWSM v4 is trained on 320k hours. That is the OWSM v3.1 mixture below plus YODAS_OWSMv4, a newly curated 166k-hour subset of YODAS2 covering 75 languages, which we release as well.

YODAS2 is web-sourced, so it carries inaccurate language labels and misaligned audio-text pairs, and in our preliminary experiments that noise hurt the downstream ASR models. The curated subset is the output of a three-stage cleaning pipeline built from public toolkits; the scripts are in the recipe. Scaling and cleaning the data is what the gain over v3.1 comes from – the architecture is unchanged apart from the subsampling noted below.

OWSM v4 also subsamples the log-Mel features 8 times rather than 4, so the encoder works at 80 ms rather than 40 ms.

The OWSM v3.1 models are trained on a diverse combination of public datasets as listed below.

OWSM v3.1 training data mixtures
  • AIDATATANG
  • AISHELL-1
  • AMI
  • Babel
  • Common Voice
  • Googlei18n
  • CoVoST2
  • Fisher Callhome Spanish
  • Fisher (Switchboard)
  • FLEURS
  • GigaSpeech
  • GigaST
  • KsponSpeech
  • LibriSpeech
  • MagicData
  • Multilingual LibriSpeech
  • MuST-C
  • ReazonSpeech
  • Russian Open STT
  • SPGISpeech
  • TEDLIUM3
  • VCTK
  • VoxForge
  • VoxPopuli
  • WenetSpeech

The low-restriction model is trained on a subset of the above data with “more flexible licenses”.

OWSM v3.1 low-restriction data
  • AMI: CC-BY-4.0
  • Common Voice: CC0-1.0
  • FLEURS: CC-BY-4.0
  • KsponSpeech: MIT
  • LibriSpeech: CC-BY-4.0
  • Multilingual LibriSpeech: CC-BY-4.0
  • VCTK: CC-BY-4.0

Inference

Similar to other ESPnet models, the pre-trained OWSM models can be easily downloaded and used in a python script. Below are some examples using OWSM v4. For earlier versions (v2 and before), the language code should follow the two-letter format (e.g., <en>, <de>).

With OWSM v4, leave maxlenratio at its default of 0.0 and avoid small positive values such as 0.5, which truncate the output. 0.0 allows the same maximum length as 1.0 – the number of encoder frames – and it also turns on the beam search’s end detection, so decoding stops once the search has converged instead of running to the maximum length.

Language Identification

We pass the Hugging Face model tag when initializing Speech2Language. The model will be automatically downloaded from Hugging Face to a local cache directory.

from espnet2.bin.s2t_inference_language import Speech2Language
s2l = Speech2Language.from_pretrained(
    model_tag="espnet/owsm_v4_medium_1B",
    device="cuda",
    nbest=3,  # return nbest prediction and probability
)

import soundfile as sf
speech, rate = sf.read("audio.wav")

result = s2l(speech)
print(result)
# list of tuples (language, probability)
# [('<eng>', 0.9994348883628845), ('<jpn>', 0.00010286537144565955), ('<rus>', 6.185896199895069e-05)]

Speech Recognition or Translation

We use Speech2Text for speech recognition or translation. We also pass the model tag so that the model can be automatically downloaded. When initializing this object, we set the default values for lang_sym, task_sym and predict_time. These variables can be overwritten later, which provides more flexibility. Note that the language must be known to use this functionality. If it is unknown, one can first perform language identification and then recognition or translation.

from espnet2.bin.s2t_inference import Speech2Text
s2t = Speech2Text.from_pretrained(
    model_tag="espnet/owsm_v4_medium_1B",
    device="cuda",
    beam_size=5,
    ctc_weight=0.0,
    maxlenratio=0.0,
    # below are default values which can be overwritten in __call__
    lang_sym="<eng>",
    task_sym="<asr>",
    predict_time=False,
)

import soundfile as sf
speech, rate = sf.read("audio.wav")


result = s2t(speech)[0][-2]

# an optional text prompt can be passed
result = s2t(
    speech,
    text_prev="this is an optional prompt"
)[0][-2]

# lang_sym, task_sym, predict_time can be overwritten
result = s2t(
    speech,
    lang_sym="<eng>",
    task_sym="<st_zho>",    # translation into Chinese
    predict_time=True,
)[0][-2]

Long-form Speech Recognition or Translation

OWSM processes an entire audio recording in a chunk-by-chunk manner. Each chunk has a fixed length of 30s. The chunk is shifted based on the predicted timestamps. We still use Speech2Text but we call its decode_long method.

from espnet2.bin.s2t_inference import Speech2Text
s2t = Speech2Text.from_pretrained(
    model_tag="espnet/owsm_v4_medium_1B",
    device="cuda",
    beam_size=5,
    ctc_weight=0.0,
    maxlenratio=0.0,
    # below are default values which can be overwritten in __call__
    lang_sym="<eng>",
    task_sym="<asr>",
)

import soundfile as sf
speech, rate = sf.read("covid.wav")

result = s2t.decode_long(speech)
# list of tuples (start_time, end_time, text)

Fine-tuning on custom data

The s2t1 template carries a worked guide: How to fine-tune pre-trained OWSM. It fine-tunes on AISHELL-1 end to end, in three steps:

  1. Set up an s2t1 recipe under your dataset with egs2/TEMPLATE/s2t1/setup.sh egs2/<your_data>/s2t1, then pull the pre-trained model down with ./s2t.sh --download_model <hf_tag> --stage 100 ..., which leaves an exp directory linking the checkpoint, config, stats and BPE model.
  2. Prepare the data in OWSM format — see Data preparation for the text, text.prev and text.ctc layout. Fine-tuning does not build a new vocabulary: reuse the token list and BPE model from the pre-trained model, and keep the special tokens such as language codes consistent with it.
  3. Train, with init_param pointing at the pre-trained checkpoint. Collect shapes with extract_feats_in_collect_stats=false and copy the existing mean and variance across rather than recomputing them.

The learning rate and batch size in that config are a starting point, not a recommendation. A model this size overfits a small training set quickly.

For a Python-only route rather than the shell recipes, see “ESPnet-EZ: Python-only ESPnet for Easy Fine-tuning and Integration” (SLT 2024).

Papers

Please cite our papers if you find OWSM helpful.

We also collect other papers related to OWSM. Please contact Yifan Peng (yifanpen@andrew.cmu.edu) if you use OWSM in your work and want to include it here.

OWSM applications
Foundational work used by OWSM