Verified project record

microsoft/unilm

unilm is an open-source project that ships a large collection of pre-trained foundation models, architectures, and tools for diverse AI tasks spanning NLP, vision, speech, and document AI. It addresses the challenge of scaling self-supervised pre-training across tasks, languages, and modalities.

Project overview

By providing unified foundation models that span text, image, audio, and document modalities, unilm consolidates advanced pre-training research into a single repository, making cross-task and cross-modal AI development more accessible.

Project type: Document Processing · Image & Vision · Infrastructure
Use cases: Documents & Office · Image Processing · Audio & Speech
Deployment: Refer to project documentation
License: MIT

Best for

AI engineers and researchers who need access to a broad collection of pre-trained foundation models spanning NLP, vision, speech, and document AI.

Key capabilities

A multimodal document foundation model that processes text together with layout, format, and image information for Document AI tasks such as scanned documents and PDFs.

Limitations and risks

The repository contains multiple advanced research models, each with varying dependencies, so setup difficulty depends on the specific sub-project chosen.
Deployment methods, self-hosting details, local runnability, and managed service availability are not documented, requiring teams to determine their own operational approach.

Getting started

Because the repository contains multiple advanced research models with varying dependencies, the path to a first successful run is to clone the repository, install dependencies for the specific sub-project, download pre-trained model weights, and run inference or fine-tuning scripts.

Evidence and sources

GitHub project description: Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
README: - December, 2024: [**RedStone**](https://github.com/microsoft/redstone) was released!
README: > [**LayoutLM**](https://github.com/microsoft/unilm/tree/master/layoutlm)/[**LayoutLMv2**](https://github.com/microsoft/unilm/tree/master/layoutlmv2)/[**LayoutLMv3**](https://gith…
README: - January, 2023: [**VALL-E**](https://arxiv.org/abs/2301.02111) a language modeling approach for text to speech synthesis (TTS), which achieves state-of-the-art zero-shot TTS perf…
README: > [**BitNet**](https://arxiv.org/abs/2310.11453): 1-bit Transformers for Large Language Models

Reviewed at: 2026-07-17T03:45:44Z