Verified project record

microsoft/unilm

microsoft/unilm is a repository of pre-trained foundation models, architectures, and tools spanning NLP, vision, speech, and document AI. It targets scaling self-supervised pre-training across tasks, languages, and modalities.

Project overview

It ships a large collection of foundation models covering diverse AI tasks and modalities, including a multimodal Document Foundation Model that jointly processes text, layout/format, and image inputs for Document AI.

Project type: Document Processing · Image & Vision · Infrastructure
Use cases: Documents & Office · Image Processing · Audio & Speech
Deployment: Refer to project documentation
License: MIT

Best for

AI engineers, researchers, and data teams working on pre-trained foundation models across NLP, vision, speech, and document AI tasks.

Key capabilities

Multimodal (text + layout/format + image) Document Foundation Model for Document AI tasks such as scanned documents and PDF processing.

Limitations and risks

The repository contains multiple advanced research models with varying dependencies, which affects setup clarity and difficulty.

Getting started

Clone the repository, then install dependencies for the specific sub-project of interest. Download pre-trained model weights and run inference or fine-tuning scripts. The overall setup difficulty is unclear because the repository contains multiple advanced research models with varying dependencies.

Evidence and sources

GitHub project description: Large-scale Self-supervised Pre-training Across Tasks, Languages, and Modalities
README: - December, 2024: [**RedStone**](https://github.com/microsoft/redstone) was released!
README: > [**LayoutLM**](https://github.com/microsoft/unilm/tree/master/layoutlm)/[**LayoutLMv2**](https://github.com/microsoft/unilm/tree/master/layoutlmv2)/[**LayoutLMv3**](https://gith…
README: - January, 2023: [**VALL-E**](https://arxiv.org/abs/2301.02111) a language modeling approach for text to speech synthesis (TTS), which achieves state-of-the-art zero-shot TTS perf…
README: > [**BitNet**](https://arxiv.org/abs/2310.11453): 1-bit Transformers for Large Language Models

Reviewed at: 2026-07-17T03:45:44Z

Open original GitHub project