What does this PR do?

This PR adds the long-form transcription as originally proposed in the Whisper codebase: https://github.com/openai/whisper and in the paper: https://cdn.openai.com/papers/whisper.pdf

To better understand long-form transcription, please have a look at Section 4.5: Strategies for Reliable Long-form Transcription of the paper.

Before this PR transformers only had "chunked" long-form transcription which trades speed against accuracy (see Table below). In this PR we add the best-performing long-form transcription to Transformers.

Usage:

One can use long-form transcription now easily with the pipeline object simply passing long-form audio. Previously, long-form audio was truncated to just 30 seconds. This PR makes sure that long audio is not cut when passed to the pipeline:

from datasets import load_dataset
from datasets import Audio
import numpy as np
from transformers import WhisperForConditionalGeneration, AutoProcessor, pipeline
import torch

processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en", torch_dtype=torch.float16)
model = model.to("cuda")

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    torch_dtype=torch.float16,
    device="cuda:0",
)

ds = load_dataset("distil-whisper/meanwhile", "default")["test"]
ds = ds.cast_column("audio", Audio(sampling_rate=16000))
audio = ds[:8]["audio"]

result = pipe(audio)

The pipeline is great for "easy-to-set-up" code but lacks customization and readability. For example the pipeline currently does not allow running the model with batch sizes > 1 and instead runs each audio 1-by-1, thus being very suboptimal regarding speed. To use long-form transcription for batch size > 1, you can use the following snippet:

from datasets import load_dataset
from datasets import Audio
import numpy as np
from transformers import WhisperForConditionalGeneration, AutoProcessor, pipeline
import torch

processor = AutoProcessor.from_pretrained("openai/whisper-tiny.en")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny.en", torch_dtype=torch.float16)
model = model.to("cuda")

ds = load_dataset("distil-whisper/meanwhile", "default")["test"]
ds = ds.cast_column("audio", Audio(sampling_rate=16000))
audio = [x["array"] for x in ds[:8]["audio"]]

inputs = processor(audio, return_tensors="pt", truncation=False, padding="longest", return_attention_mask=True, sampling_rate=16_000)
inputs = inputs.to("cuda", torch.float16)

result = model.generate(**inputs, return_timestamps=True)
decoded = processor.batch_decode(result, skip_special_tokens=True)

print(decoded)

Docs have been

added to give examples for both short- and longform transcription:

But I don't think that this is enough for people to notice this method. We should in my opinion create much better guides for Whisper (will be done in a follow-up PR).

Credits

IMPORTANT: Most of the code added from this PR was copied & tweaked from the original whisper code: https://github.com/openai/whisper/blob/main/whisper/transcribe.py . Therefore 90% of the credit of this PR goes to @jongwook as the original author of the transcribe.py code.

Why copy the code?!: We originally weren't planning on integrating the full long-form transcription algorithm to transformers but a couple of reasons forced us now to add it:

• Original codebase is very slow. The original whisper codebase is very slow and doesn't take advantaged of recent advances such as Flash Attention, Torch compile etc...
• Batched long-form inference is not supported see here: model.transcribe() modified to perform batch inference on audio files openai/whisper#662
• We want to integrate whisper with normal decoding mechanisms such as speculative decoding

Next steps:

When looking at all long-form generation strategies:

Transformers has now support for the following:

• beam search
• initial timestamp constraint

In a follow-up PR we will add: "temperature fallback", "voice activity detection", and "previous text conditioning".

Results:

Note: that for "chunked transformers" the numbers are crossed-through because the original results from the whisper paper seem to have been slightly incorrect. Re-running the eval gives better results.

Here the results for openai/whisper-tiny.en

Here the results for openai/whisper-small.en

Here the results for openai/whisper-large-v2

Update:

It seems like the number we measured in the distil-whisper paper for chunked long-form are a bit off. Re-running them gives the following:

Here the results for openai/whisper-tiny.en

=> New algo is on avg. 0.01 WER abs points worse which means it's identical

Here the results for openai/whisper-small.en

=> New algo is on avg. 0.005 WER abs points worse which means it's identical

Here the results for openai/whisper-large-v2

=> New algo is on avg. 0225 WER abs points better which means it's identical or (tiny tiny bit better)

Batch size > 1

The code now fully functions for batch size > 1 (made sure that results on the four datasets is within +/- 0.1 % WER). When using batch size = 8, there is a 4x speed-up for large-v2, 2x speed-up for small (and 1.5x speed-up for tiny). The bigger the model, the larger the speed-up!

One should definitely use larger batch sizes when doing long-form timestamp prediction!