A Python framework with a Rust engine for building scalable ETL, streaming, real-time analytics, and RAG pipelines. It processes batch and streaming data using the same Python code.
Project overview
A Python framework with a Rust engine for building scalable ETL, streaming, real-time analytics, and RAG pipelines. It processes batch and streaming data using the same Python code.
Project type
RAG · Workflow · Data Processing
Use cases
Data Analysis
Deployment
Refer to project documentation
License
License pending
Best for
Developers and AI engineers who need to build scalable Python ETL pipelines for stream processing, real-time analytics, LLM pipelines, and RAG.
Key capabilities
Processes both batch and streaming data effectively using the same Python code.
Executes Python code using a Rust engine based on Differential Dataflow for multithreading, multiprocessing, and distributed computations.
A built-in dashboard to track the number of messages sent by connectors and system latency, including log messages.
Limitations and risks
Available only on MacOS and Linux; users of other systems need a Virtual Machine.
Requires Python 3.10 or above.
Distributed under BSL 1.1 License which automatically converts to Open Source (Apache 2.0) after 4 years.
Getting started
Installable via a single pip command (pip install -U pathway) and can be started by running a main Python script (python main.py).
Evidence and sources
GitHub project description: Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
README: Pathway Live Data Framework code is versatile and robust: **you can use it in both development and production environments, handling both batch and streaming data effectively**.
README: With its unified engine for batch and streaming and its full Python compatibility, Pathway Live Data Framework makes data processing as easy as possible. It's the ideal solution f…
README: Pathway Live Data Framework provides dedicated LLM tooling to build live LLM and RAG pipelines. Wrappers for most common LLM services and utilities are included, making working wi…
README: - **Stateless and stateful transformations**: Pathway Live Data Framework supports stateful transformations such as joins, windowing, and sorting. It provides many transformations…