WhisperX adds batched inference, word-level alignment, and speaker diarization to Whisper-based transcription. It targets developers and researchers who need faster processing and more precise timestamps than the base Whisper model provides.
Project overview
Batched inference with whisper large-v2 reaches 70x realtime, and wav2vec2 alignment adds word-level timestamps, addressing known gaps in base Whisper utterance timing.
Project type
Audio & Speech
Use cases
Meeting Notes · Audio & Speech
Deployment
Refer to project documentation
License
BSD-2-Clause
Best for
Developers and researchers needing word-level timestamps and batched inference for audio files.
Users requiring multispeaker ASR with speaker ID labels.
Key capabilities
Batched inference for 70x realtime transcription using whisper large-v2 with faster-whisper backend.
Accurate word-level timestamps using wav2vec2 alignment.
Multispeaker ASR using speaker diarization from pyannote-audio with speaker ID labels.
Voice Activity Detection preprocessing is designed to reduce hallucination and enable batching with no WER degradation.
Allows loading models, transcribing audio, aligning output, and assigning speaker labels via Python code.
Command line tool to run transcription, alignment, and diarization on audio files.
Option to run inference on CPU instead of GPU using compute_type int8.
Limitations and risks
Transcript words lacking characters in the alignment model's dictionary cannot be aligned and will not receive a timing.
Overlapping speech is not handled particularly well.
Diarization is far from perfect.
A language-specific wav2vec2 model is needed for alignment.
The development version may contain experimental features and bugs; the stable PyPI release is recommended for production.
Getting started
Install easily via pip or uvx; requires ffmpeg and CUDA for GPU usage.
Install CUDA toolkit 12.8 if using GPU, run pip install whisperx, and execute whisperx path/to/audio.wav.
Alternatives and comparisons
A general-purpose speech recognition model that can perform multilingual speech recognition, speech translation, and language identification.
Provide high-performance, dependency-free inference for OpenAI Whisper speech-recognition models across supported platforms.
README: This repository provides fast automatic speech recognition (70x realtime with large-v2) with word-level timestamps and speaker diarization.
README: Whilst it does produces highly accurate transcriptions, the corresponding timestamps are at the utterance-level, not per word, and can be inaccurate by several seconds. OpenAI's w…
README: 1st place at [Ego4d transcription challenge](https://eval.ai/web/challenges/challenge-page/1637/leaderboard/3931/WER) 🏆