yue provides open-source foundation models that transform text lyrics into complete songs lasting several minutes, including both vocal and accompaniment tracks. The models can handle diverse genres, languages, and vocal techniques.
Project overview
yue distinguishes itself as an open-source foundation model dedicated to full-song generation from lyrics, covering both vocals and accompaniment, which directly addresses the challenge of producing complete multi-minute songs from text input.
Project type
Audio & Speech
Use cases
Audio & Speech
Deployment
Refer to project documentation
License
Apache-2.0
Best for
Creators who want to generate complete songs with vocals and accompaniment from text lyrics.
Developers and researchers seeking open-source foundation models for full-song generation that they can run locally or self-host.
Key capabilities
Generates a complete song lasting several minutes including both a vocal track and accompaniment track from lyrics.
Capable of modeling diverse genres, languages, and vocal techniques.
Supports extending and continuing generated music.
Supports fine-tuning the model using LoRA adapters.
Support for stemgen mode is planned.
Support for vLLM and sgLang serving backends is planned.
Limitations and risks
Generating long sequences requires significant GPU memory.
The [intro] label is less stable during generation.
There is a risk of plagiarism or unlawful reproduction of existing material when using generated output.
Getting started
The first success path involves three steps: install the environment and dependencies, download the infer code and tokenizer, and run the inference. Users should be prepared for manual CUDA version matching and installation of dependencies such as flash attention 2. Coding is required.
GPUs with 24GB memory or less support up to 2 sessions, while GPUs with at least 80GB memory are needed for full song generation.
The project requires model checkpoints including YuE-s1-7B-anneal-en-cot and YuE-s2-1B-general.
Evidence and sources
README: YuE is a groundbreaking series of open-source foundation models designed for music generation, specifically for transforming lyrics into full songs (lyrics2song). It can generate…
README: YuE-s1-7B-anneal-en-cot 🤗
README: Now YuE supports LoRA finetune.
README: We now support dual-track ICL mode! You can prompt the model with a reference song, and it will generate a new song in a similar style (voice cloning demo by @abrakjamson, music s…
README: Now YuE supports music continuation and Google Colab!