Whisper is a general-purpose speech recognition model trained via large-scale weak supervision. It performs multilingual transcription, speech-to-English translation, and language identification using a single multitasking sequence-to-sequence model.
Project overview
It replaces multiple stages of a traditional speech-processing pipeline by jointly representing transcription, translation, and identification tasks as a sequence of tokens.
Developers and AI engineers who need to process diverse audio files for multilingual speech recognition, spoken language identification, or speech translation to English text.
Key capabilities
Transcribes speech from diverse audio into text using a multitasking model.
Translates non-English speech into English text.
Identifies the spoken language from an audio input.
Performs voice activity detection as part of a joint sequence-to-sequence training format.
Limitations and risks
The turbo model is not trained for translation tasks.
English-only models tend to perform better only for English-only applications; the difference becomes less significant for small.en and medium.en models.
You may need Rust installed if tiktoken does not provide a pre-built wheel for your platform.
Getting started
Installation difficulty is rated as easy. Install openai-whisper via pip, install ffmpeg via your system package manager, and run whisper audio.flac. Coding is optional, with interaction supported via CLI or as a library.
Alternatives and comparisons
A toolkit for building speech recognition systems using C++ code and example builds.
Small offline speech recognition models providing continuous large vocabulary transcription and zero-latency streaming.
A scalable generative AI framework for researchers and PyTorch developers to build and deploy speech models.
README: Whisper is a general-purpose speech recognition model. It is trained on a large dataset of diverse audio and is also a multitasking model that can perform multilingual speech reco…
README: A Transformer sequence-to-sequence model is trained on various speech processing tasks, including multilingual speech recognition, speech translation, spoken language identificati…
GitHub project description: Robust Speech Recognition via Large-Scale Weak Supervision
README: pip install -U openai-whisper
README: There are six model sizes, four with English-only versions, offering speed and accuracy tradeoffs.