NVIDIA-NeMo/Speech

Project overview

A PyTorch-based generative AI framework for researchers and developers to build, customize, and deploy Automatic Speech Recognition, Text-to-Speech, and Speech Large Language Models. The project requires a pre-configured Python environment with CUDA-capable NVIDIA GPUs.

Project type

Model Development · Model Runtime · Audio & Speech · Infrastructure

Use cases

Meeting Notes · Audio & Speech

Deployment

Refer to project documentation

License

Apache-2.0

Best for

• Researchers, developers, and AI engineers building and deploying speech recognition, speech synthesis, and multimodal audio models.

Key capabilities

• Tools to create, customize, and deploy speech recognition models, supporting both offline and streaming inference.
• Tools to create, customize, and deploy speech synthesis models (e.g., MagpieTTS).
• Capabilities for building and running Speech Large Language Models (e.g., full-duplex VoiceChat) and multimodal LLMs.

Limitations and risks

• The prebuilt NGC container image for the framework is not yet published.
• Loading model checkpoints with weights_only=False can risk arbitrary code execution if from untrusted sources.
• The repository is undergoing transformation and pivoting its focus, which may impact stability during the transition.

Getting started

• Clone the repository, install Python 3.12+, PyTorch 2.7+ and CUDA. Select an installation path (source, pip, or docker) and set up environment extras like cu12 or cu13.
• Run 'uv sync --extra all --extra cu13' or 'pip install nemo-toolkit[asr,tts]', then import the framework in Python or run model scripts.

Evidence and sources

• GitHub project description: A scalable generative AI framework built for researchers and developers working on Large Language Models, Multimodal, and Speech AI (Automatic Speech Recognition and Text-to-Speec…
• README: Project Status: Active -- The project has reached a stable, usable state and is being actively developed.
• README: Checkout our HuggingFace🤗 collection for the latest open weight checkpoints and demos!
• README: Nemotron-3.5-ASR-Streaming-0.6B has been released with 40 languages supported, controllable latency 80ms-1s, and 240-2400 1xH100 concurrent streams.
• README: NVIDIA NeMo Speech is built for researchers and PyTorch developers working on Speech models including Automatic Speech Recognition (ASR), Text to Speech (TTS), and Speech LLMs. It…

Reviewed at: 2026-07-17T13:05:48Z