WhisperX

What does WhisperX do?

WhisperX "provides fast automatic speech recognition (70x realtime with large-v2) with word-level timestamps and speaker diarization". Diarization is the term for labelling who spoke when. Three things sit on top of Whisper: batched inference through the faster-whisper backend, which the README says needs under 8 GB of GPU memory for large-v2; forced alignment with a wav2vec2 model, which gives each word a start and end time; and speaker labels through pyannote-audio, with flags for the minimum and maximum number of speakers.

Voice activity detection runs first, which the README says reduces hallucination and enables batching without an accuracy cost. Alignment models ship for English, French, German, Spanish and Italian, with more on Hugging Face; alignment is language-specific.

Key facts

• Licence: BSD-2-Clause on the code. WhisperX ships no weights of its own.
• Batched inference through the faster-whisper backend; the README claims 70 times real time with large-v2 and under 8 GB of GPU memory at beam size 5.
• Word-level timestamps by forced alignment with a wav2vec2 model, which is language-specific; models ship for English, French, German, Spanish and Italian with more on Hugging Face.
• Speaker diarization through pyannote-audio, with flags for the minimum and maximum number of speakers; the model is CC-BY-4.0 and needs a Hugging Face token and an accepted agreement.
• Voice activity detection runs first, which the README says reduces hallucination and enables batching with no accuracy cost.
• Install: pip install whisperx or uvx whisperx; GPU use needs CUDA toolkit 12.8; CPU and macOS run with --compute_type int8 --device cpu.
• No Docker image or HTTP server is described; the maintainer links third-party demos on Replicate.
• Latest tagged release when read: v3.8.6 on 2026-05-25.

What does it replace, and where does it fall short?

WhisperX replaces Deepgram and AssemblyAI where the output you depend on is a transcript with word timestamps and speaker labels, which is what those APIs return. It is the closest open-source match to that shape on this site.

Where it falls short: the diarization model is CC-BY-4.0, needs a Hugging Face token and an accepted agreement, and the README says "Diarization is far from perfect." Overlapping speech "is not handled particularly well by whisper nor whisperx". Words with characters outside the alignment dictionary, such as "2014." or "£13.60", get no timing.

How does WhisperX run?

pip install whisperx or uvx whisperx, then the whisperx command or the Python API. GPU use needs CUDA toolkit 12.8. The documented way to run on a CPU or on a Mac is --compute_type int8 --device cpu. No Docker image or HTTP server is described; the maintainer links third-party demos on Replicate for people without a GPU.

Who is WhisperX for?

A team transcribing recordings in bulk on a GPU that needs timings and speakers, and is prepared to accept pyannote's licence and the README's own caveats about it. A team that needs live audio should look elsewhere on this list.

What limits does the README state?

The four items in its own "Limitations" section, quoted above: unaligned tokens, overlapping speech, imperfect diarization, and the need for a language-specific alignment model.

Questions people ask

Is WhisperX open source?

The code is, under BSD-2-Clause. The speaker labelling depends on pyannote's diarization model, which is CC-BY-4.0, a Creative Commons licence that is not on the OSI list and is gated behind a Hugging Face agreement. Alignment models come from torchaudio or Hugging Face with their own licences.

Can I use WhisperX commercially?

The BSD-2-Clause code, yes. The pyannote diarization model is CC-BY-4.0, which permits commercial use with attribution, but you must accept its Hugging Face agreement first. Check each alignment model's licence separately if you ship them.

Does WhisperX run on a CPU or a Mac?

Yes, with the documented flags: --compute_type int8 --device cpu is the README's way to run on a CPU and on macOS. GPU use is the main path, with CUDA toolkit 12.8 and under 8 GB of GPU memory for large-v2.

How does WhisperX compare with AssemblyAI?

WhisperX returns the shape AssemblyAI returns, a transcript with word timings and speaker labels, on your own hardware with no per-hour bill. AssemblyAI adds summaries and understanding in the same call and handles overlapping speech as a service. The alternatives to AssemblyAI page places WhisperX first for that reason.

What does the WhisperX README admit it does badly?

Four things, in its own limitations section: words with characters outside the alignment dictionary, such as "2014." or "£13.60", get no timing; overlapping speech is not handled well by Whisper or WhisperX; diarization is far from perfect; and a language-specific wav2vec2 model is needed for alignment.

Sources

• WhisperX README and BSD-2-Clause LICENSE: github.com/m-bain/whisperX, read 2026-09-16. The top of the README carries a paid sponsor block from a meeting-transcription API; it is advertising and nothing on this page comes from it.