Pathway is a Python framework for building scalable ETL pipelines that process batch and streaming data through the same code. It uses a Rust engine for execution and supports real-time analytics, LLM pipelines, and RAG applications.
Project overview
It allows developers to maintain a single Python codebase for local development, CI/CD tests, batch jobs, and stream processing by executing the code with a Rust engine based on Differential Dataflow.
Project type
RAG · Workflow · Data Processing
Use cases
Knowledge Q&A · Data Analysis
Deployment
Refer to project documentation
License
License pending
Best for
Developers, data teams, and AI engineers who need to build scalable Python ETL pipelines for stream processing, real-time analytics, LLM pipelines, and RAG.
Teams seeking to use the same Python code for local development, CI/CD tests, batch jobs, stream replays, and processing data streams.
Key capabilities
Processes both batch and streaming data effectively using the same Python code.
Executes Python code using a Rust engine based on Differential Dataflow for multithreading, multiprocessing, and distributed computations.
A built-in dashboard to track the number of messages sent by connectors and system latency, including log messages.
Limitations and risks
Available only on MacOS and Linux; users of other systems need a Virtual Machine.
Requires Python 3.10 or above.
Distributed under BSL 1.1 License which automatically converts to Open Source (Apache 2.0) after 4 years.
Getting started
Installation difficulty is rated as easy because it is installable via a single pip command.
Evidence and sources
GitHub project description: Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
README: Pathway Live Data Framework code is versatile and robust: **you can use it in both development and production environments, handling both batch and streaming data effectively**.
README: With its unified engine for batch and streaming and its full Python compatibility, Pathway Live Data Framework makes data processing as easy as possible. It's the ideal solution f…
README: Pathway Live Data Framework provides dedicated LLM tooling to build live LLM and RAG pipelines. Wrappers for most common LLM services and utilities are included, making working wi…
README: - **Stateless and stateful transformations**: Pathway Live Data Framework supports stateful transformations such as joins, windowing, and sorting. It provides many transformations…