A learning-based system that converts images of math equations into LaTeX code using a ViT encoder with a ResNet backbone and a Transformer decoder. It runs locally via CLI, GUI, or Python library.
Project overview
A learning-based system that converts images of math equations into LaTeX code using a ViT encoder with a ResNet backbone and a Transformer decoder. It runs locally via CLI, GUI, or Python library.
Project type
Document Processing · Image & Vision
Use cases
Documents & Office
Deployment
Refer to project documentation
License
MIT
Best for
General users and developers needing a local tool to convert images of math formulas into LaTeX code.
Users who need to process smaller resolution images via desktop GUI, CLI, or Python library.
Key capabilities
Takes an image of a math formula and returns the corresponding LaTeX code using a trained model.
Provides a graphical user interface where users can take a screenshot of a formula and get the LaTeX code rendered and copied to their clipboard.
Allows users to parse existing images from the disk or clipboard via the pix2tex command line tool.
Can be imported and used directly within Python scripts to apply OCR on PIL Image objects.
Uses an auxiliary neural network to predict the optimal resolution and automatically resize the input image to increase performance.
Supports training the model on custom datasets using provided dataset utilities and configuration files.
Planned capability to add beam search to model decoding.
Planned capability to support handwritten formula recognition.
Limitations and risks
Works best with smaller resolution images and might not handle huge images optimally.
The model might not be perfect, output different predictions on retries, and requires the user to always double check the result.
Getting started
Install the package via pip using pix2tex[gui] and execute the first call by running latexocr. A Docker image is also available.
Evidence and sources
README: The goal of this project is to create a learning based system that takes an image of a math formula and returns corresponding LaTeX code.
README: Install the package `pix2tex`: ``` pip install "pix2tex[gui]" ``` Model checkpoints will be downloaded automatically. There are three ways to get a prediction from an image. 1. Yo…
README: Install a couple of dependencies `pip install "pix2tex[train]"`. 1. First we need to combine the images with their ground truth labels. I wrote a dataset class (which needs furthe…
README: - [x] add more evaluation metrics - [x] create a GUI - [ ] add beam search - [ ] support handwritten formulae (kinda done, see training colab notebook) - [ ] reduce model size (di…