Verified project record

NVIDIA-NeMo/Speech

NeMo Speech provides a scalable PyTorch framework for researchers and developers to create, customize, and deploy Speech AI models across Automatic Speech Recognition, Text-to-Speech, and Speech Large Language Models. The project requires an NVIDIA GPU for training and relies on pre-trained model checkpoints for deployment and inference.

Project overview

The project provides tools for both offline and streaming speech recognition inference alongside full-duplex VoiceChat multimodal LLM capabilities.

Project type: Model Development · Model Runtime · Audio & Speech · Infrastructure
Use cases: Meeting Notes · Translation & Subtitles · Audio & Speech
Deployment: Refer to project documentation
License: Apache-2.0

Best for

Researchers, developers, and AI engineers who have an NVIDIA GPU and need a PyTorch framework to build, customize, or run ASR, TTS, or Speech LLMs.

Key capabilities

Tools to create, customize, and deploy speech recognition models, supporting both offline and streaming inference.
Tools to create, customize, and deploy speech synthesis models (e.g., MagpieTTS).
Capabilities for building and running Speech Large Language Models (e.g., full-duplex VoiceChat) and multimodal LLMs.

Limitations and risks

The prebuilt NGC container image for NeMo Speech is not yet published and is listed as coming soon.
Loading model checkpoints with weights_only=False can risk arbitrary code execution if loaded from untrusted sources.
The repository is undergoing transformation and pivoting its focus, which may impact stability during the transition.

Getting started

Setup requires a pre-configured Python 3.12+, PyTorch 2.7+, and CUDA stack. Users must clone the repository, select an installation path (source, pip, or docker), configure environment extras (cu12/cu13), and run commands such as 'uv sync --extra all --extra cu13' or 'pip install nemo-toolkit[asr,tts]' before importing NeMo in Python.

Alternatives and comparisons

A tokenizer-free text-to-speech model supporting multiple languages, voice design from text descriptions, and accurate voice cloning as an alternative for speech synthesis.
An open-source PyTorch-based holistic framework offering training recipes and pretrained models for diverse speech, text, and related tasks.
An open-source text-to-speech engine supporting bilingual Chinese and English synthesis with prompt-controlled emotional speech generation.

Project comparisons

Evidence and sources

GitHub project description: A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speec…
README: Project Status: Active -- The project has reached a stable, usable state and is being actively developed.
README: Checkout our HuggingFace🤗 collection for the latest open weight checkpoints and demos!
README: Nemotron-3.5-ASR-Streaming-0.6B has been released with 40 languages supported, controllable latency 80ms-1s, and 240-2400 1xH100 concurrent streams.
README: NVIDIA NeMo Speech is built for researchers and PyTorch developers working on Speech models including Automatic Speech Recognition (ASR), Text to Speech (TTS), and Speech LLMs. It…

Reviewed at: 2026-07-17T13:05:48Z

Open original GitHub project