How to Run IBM Granite Speech 5.0 Locally for Transcription

By /

Documentation-verified scope: I checked this guide against IBM’s Granite Speech 5.0 model cards and release material on 26 August 2026. I have not installed the model or measured its accuracy, memory use or speed. The commands below follow the official Transformers example and adapt it to a local PCM WAV file.

IBM Granite Speech 5.0 TurboCTC is a 470-million-parameter English speech-recognition model. You can run it locally with Python, PyTorch and Hugging Face Transformers, then send a WAV file through the model and save the transcript as text.

Use the standard repository, ibm-granite/granite-speech-5.0-470m-turboctc, for this guide. It carries an Apache 2.0 licence. A separate repository ending in -nc is limited to non-commercial use under CC-BY-NC-SA-4.0.

The best first test is a short, clear English recording in PCM WAV format. The script here does not add speaker names, word timestamps or translation, and IBM’s published evaluation focuses on short-form English speech.

What Granite Speech 5.0 TurboCTC does

TurboCTC uses a Conformer acoustic encoder and Connectionist Temporal Classification rather than a language-model backbone. IBM says it was trained on roughly 60,000 hours of public or synthetic English speech. Decoding is non-autoregressive and greedy, which keeps the transcription path compact.

IBM positions the model for low-latency and high-throughput English transcription on edge devices and larger servers. Its release material reports throughput near 12,600 RTFx in a test using one NVIDIA H200. That is a vendor result on datacentre hardware, not an expected speed for a Windows PC, Mac or Linux laptop, and I have not reproduced it independently. IBM has not published a universal consumer minimum for RAM or a laptop benchmark.

This guide therefore avoids a speed promise. Run the same short clip on your own machine and record the elapsed time before planning a large transcription job.

Before you install it

  • Python: use a current Python 3 release supported by the current PyTorch and Transformers builds.
  • Storage: leave several gigabytes free for the virtual environment, PyTorch, model files and Hugging Face cache. The model card does not publish a complete installed-size figure.
  • First-run internet: the Python packages, processor files and model weights have to download before local inference can begin.
  • Audio: start with a short English PCM WAV file that you recorded or have permission to process.
  • Audio decoder: current TorchAudio uses TorchCodec for torchaudio.load. TorchCodec needs a compatible PyTorch version and working FFmpeg shared libraries, so install FFmpeg before the import check below.
  • GPU: optional for following the code. A compatible accelerator can improve speed, but support depends on the exact PyTorch build and driver.

Do this in a new folder and virtual environment. The model currently needs Transformers installed from its source repository until native support reaches the next stable release. A separate environment makes that temporary dependency easier to replace.

Step 1: create a virtual environment

Windows PowerShell

mkdir granite-speech
cd granite-speech
py -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip

If PowerShell blocks the activation script, do not disable script checks across the computer. Open Command Prompt and activate the same environment with:

.venv\Scripts\activate.bat

macOS or Linux

mkdir granite-speech
cd granite-speech
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip

Step 2: install PyTorch and Granite Speech support

For the package-manager baseline, install matching PyTorch, TorchAudio and TorchCodec packages, then add Accelerate and the current Transformers source:

python -m pip install torch torchaudio torchcodec accelerate
python -m pip install git+https://github.com/huggingface/transformers.git

For an NVIDIA or other accelerated build, use the selector on the official PyTorch installation page instead of guessing a CUDA package URL. The PyTorch, TorchAudio and TorchCodec versions must be compatible; check the current official TorchCodec installation table if the latest packages do not resolve together.

Check that the active environment can import the main packages:

python -c "import torch, torchaudio, torchcodec, transformers; print('torch', torch.__version__); print('transformers', transformers.__version__)"

Because the second install command follows the current Transformers development branch, it is a moving dependency. For a production service, record the Git commit that was reviewed and tested. Move back to a stable Transformers release when its release notes confirm Granite Speech 5.0 support.

Step 3: create the local transcription script

Create a file named transcribe_granite.py in the project folder and paste in this code:

from pathlib import Path
import sys

import torch
import torchaudio
from transformers import AutoModelForCTC, AutoProcessor

MODEL_ID = "ibm-granite/granite-speech-5.0-470m-turboctc"

if len(sys.argv) != 2:
    raise SystemExit("Usage: python transcribe_granite.py path/to/audio.wav")

audio_path = Path(sys.argv[1]).expanduser()
if not audio_path.is_file():
    raise SystemExit(f"Audio file not found: {audio_path}")
if audio_path.suffix.lower() != ".wav":
    raise SystemExit("Convert the first test to PCM WAV before running this script.")

processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForCTC.from_pretrained(MODEL_ID, device_map="auto")

waveform, source_rate = torchaudio.load(str(audio_path))
waveform = waveform.mean(dim=0)
target_rate = processor.feature_extractor.sampling_rate

if source_rate != target_rate:
    waveform = torchaudio.functional.resample(
        waveform, source_rate, target_rate
    )

samples = waveform.numpy()
inputs = processor(
    [samples],
    sampling_rate=target_rate,
    device=model.device,
)
inputs.to(model.device, dtype=model.dtype)

with torch.inference_mode():
    output_ids = model.generate(**inputs)

transcript = processor.batch_decode(
    output_ids, skip_special_tokens=True
)[0].strip()

output_path = audio_path.with_suffix(".txt")
output_path.write_text(transcript + "\n", encoding="utf-8")

print(transcript)
print(f"\nSaved transcript to: {output_path}")

The model and processor download on the first run and are then kept in the Hugging Face cache. The script downmixes stereo to mono and resamples the waveform to the rate requested by the processor.

Step 4: transcribe a short WAV file

Put a short file such as sample.wav in the project folder, then run:

python transcribe_granite.py sample.wav

The transcript prints in the terminal and is also written to sample.txt. Read it while listening to the source. Names, numbers, accents, background noise and specialised terms are useful things to check before you trust a batch job.

The first run includes downloads and should not be used as a speed measurement. Time a second run of the same clip after the cache is warm. Record the computer, operating system, PyTorch build, processor or GPU, clip duration and elapsed time if you plan to compare results.

Convert MP3, M4A or another audio format

The supported codecs behind torchaudio can vary by operating system. Converting the first test to PCM WAV removes that variable. With an official FFmpeg installation available in the terminal:

ffmpeg -i input.m4a -ar 16000 -ac 1 -c:a pcm_s16le sample.wav

The script still reads the processor’s requested sample rate and resamples if necessary. The explicit 16 kHz mono conversion is useful because it also gives you a predictable file for troubleshooting.

Handle a long recording in short chunks

IBM reports the model on short-form ASR test sets. Sending an entire meeting through the basic script can also increase memory use and gives you one transcript without timestamps. Start by splitting a copy into 30-second WAV files:

mkdir chunks
ffmpeg -i meeting.wav -f segment -segment_time 30 -ar 16000 -ac 1 -c:a pcm_s16le chunks/part-%03d.wav

Transcribe the chunks in filename order and keep their boundaries if you need to locate an error later. Thirty seconds is a conservative workflow choice for this script, not an IBM model limit. A proper production pipeline should add voice-activity detection, timestamps, retries and an explicit rule for joining text across boundaries.

Run from the cache for private recordings

The Python script processes the waveform on the selected local device; it contains no audio-upload call. The package installation and initial from_pretrained calls still contact GitHub and Hugging Face to fetch code and model files.

For confidential material, complete setup with a harmless test clip, confirm the model is cached, review the environment and then change both loads to include local_files_only=True:

processor = AutoProcessor.from_pretrained(
    MODEL_ID, local_files_only=True
)
model = AutoModelForCTC.from_pretrained(
    MODEL_ID,
    device_map="auto",
    local_files_only=True,
)

Disconnect the network or monitor it during the test if offline operation is a requirement. This guide has not audited every Python dependency, operating-system service or cache location.

Fix common Granite Speech 5.0 problems

Transformers does not recognise granite_speech5_ctc

The stable Transformers package is probably older than the model. Activate the intended virtual environment and reinstall the current source build:

python -m pip uninstall -y transformers
python -m pip install git+https://github.com/huggingface/transformers.git

Run the version check again. Avoid changing a shared system Python installation to fix one project.

device_map asks for Accelerate

python -m pip install --upgrade accelerate

Close and reopen the activated environment if the same import error continues.

TorchAudio says TorchCodec or FFmpeg is missing

As of TorchAudio 2.9, torchaudio.load uses TorchCodec. Install a TorchCodec version compatible with the active PyTorch build, install FFmpeg using its official package for your operating system, then repeat the import check from Step 2. Do not solve this by mixing unrelated PyTorch, TorchAudio and TorchCodec versions in the same environment.

CUDA is unavailable or the wrong build was installed

Check what PyTorch sees:

python -c "import torch; print(torch.cuda.is_available()); print(torch.version.cuda)"

If the first value is False, compare the installed driver and PyTorch command with the current official selector. Do not install a random wheel from a forum post. You can continue on CPU while fixing the accelerator.

The model runs out of memory

Close other memory-heavy programs and retry with a short clip. To force CPU placement, replace the model-loading line with:

model = AutoModelForCTC.from_pretrained(
    MODEL_ID,
    dtype=torch.float32,
).to("cpu")

The explicit float32 setting favours broad CPU compatibility and uses more memory than lower-precision weights. CPU mode may also be slower. IBM does not publish a universal memory requirement for this Python workflow, so measure peak use on the target machine.

The transcript is blank or badly wrong

  • Confirm the recording contains audible English speech and is not mostly silence or music.
  • Convert a short section to 16 kHz, mono, PCM WAV with the FFmpeg command above.
  • Listen to the converted file before blaming the model.
  • Try one speaker in a quiet recording. The basic decoder does not separate speakers.
  • Check names, numbers and specialised terms manually; the model has no keyword-biasing step in this script.

Git or HTTPS errors stop the Transformers install

Confirm that Git is installed and that the computer can reach the official Hugging Face Transformers repository. A company proxy may require an approved certificate or package mirror. Do not disable TLS verification. If the environment cannot install a reviewed source dependency, wait for the next stable Transformers release rather than copying an unknown package.

Licence choice: standard versus NC

The standard model used here is labelled Apache 2.0. The separate ibm-granite/granite-speech-5.0-470m-turboctc-nc model card says research and non-commercial use only and lists CC-BY-NC-SA-4.0.

Do not switch to the NC repository because a benchmark looks better without reviewing that licence. A permissive model licence also does not grant rights to record people, process workplace calls or redistribute the source audio. Apply the consent, privacy, employment and copyright rules that cover the recording.

Granite Speech 5.0 questions

Is Granite Speech 5.0 English only?

Yes. Both TurboCTC model cards list English as the supported language. Use a model with documented support for the target language rather than expecting translation from this decoder.

Does it need an NVIDIA GPU?

The model card describes deployment on laptops and edge devices, and the code can be placed on CPU. IBM’s published high-throughput benchmark used one datacentre-class NVIDIA H200. That vendor benchmark cannot answer how fast your CPU, Apple Silicon Mac or consumer GPU will be, and I have not reproduced it independently.

Can it identify speakers or create subtitles?

Not with the basic CTC script. It returns text for the supplied waveform. Speaker diarisation, word-level timing and subtitle formatting need additional components and their own verification.

Can it transcribe a live microphone?

IBM describes streaming and provides a WebGPU demonstration, but this article uses a saved WAV file so the input and output are repeatable. A live application needs audio buffering, voice-activity detection and latency testing.

Is Granite Speech 5.0 better than Whisper?

No general claim is safe from the release material alone. Compare the same authorised clips, languages, noise, names, timestamps, memory use and hardware. Granite Speech 5.0 is English-only and compact; Whisper has different model sizes, language coverage and tooling.

Can I use the standard model for paid work?

The standard repository is marked Apache 2.0. The NC repository is not. Review the exact model card and licence in your dependency record, and separately confirm the rights and privacy rules attached to the audio.

Related local speech guides

The NeMo-Speech.cpp local transcription guide covers another on-device route with a different runtime. If a raw transcript needs local cleanup, the Superwhisper S1-mini guide covers a compact text-processing model. Keep the original transcript so any cleanup remains auditable.

For the first Granite Speech run, use the Apache repository, a short English PCM WAV and a clean virtual environment. Check the saved text against the recording before scaling the job.

Primary sources

Leave a Reply

Scroll to Top

Discover more from Lachie's Lifestyle

Subscribe now to keep reading and get access to the full archive.

Continue reading