Verified project record

adbar/trafilatura

A modular library and command-line tool for web scraping, crawling, and converting raw HTML into structured text and metadata without requiring a database. It outputs in multiple formats including TXT, JSON, and XML.

Project overview

Project type: Data Processing · Infrastructure
Use cases: Search & Research
Deployment: Refer to project documentation
License: Apache-2.0

Best for

Developers, researchers, and data teams who need to convert raw HTML or web sources into structured text and metadata without configuring a database.

Key capabilities

Supports sitemaps (TXT, XML) and feeds (ATOM, JSON, RSS), along with smart crawling and URL management features including filtering and deduplication.
Extracts main text using common patterns and generic algorithms, alongside metadata fields such as title, author, date, site name, categories, and tags.
Extracts document structure and formatting including paragraphs, titles, lists, quotes, code, line breaks, in-line text formatting, and optional elements like comments, links, images, and tables.
Converts extracted data and content into TXT, Markdown, CSV, JSON, HTML, XML, and XML-TEI output formats.
Provides an optional add-on that performs language detection on extracted text content.
Processes online and offline input in parallel, supporting live URLs, download queues, previously downloaded HTML files, and parsed HTML trees.
Available for use with R.

Limitations and risks

The future development and maintenance of the tool depends on community support and sponsorships.

Getting started

Install the package from PyPI via pip; no code is strictly required as it provides a command-line tool, though it can also be imported as a Python library.

Evidence and sources

GitHub project description: Python & Command-line tool to gather text and metadata on the Web: Crawling, scraping, extraction, output as CSV, JSON, HTML, MD, TXT, XML
README: Trafilatura is a cutting-edge **Python package and command-line tool** designed to **gather text on the Web and simplify the process of turning raw HTML into structured, meaningfu…
README: Going from HTML bulk to essential parts can alleviate many problems related to text quality, by **focusing on the actual content**, **avoiding the noise** caused by recurring elem…
README: The extractor strikes a balance between limiting noise (precision) and including all valid parts (recall). It is **robust and reasonably fast**.
README: Trafilatura consistently outperforms other open-source libraries in text extraction benchmarks, showcasing its efficiency and accuracy in extracting web content.

Reviewed at: 2026-07-16T20:15:39Z