Data Processing Open-Source Projects

Showing 40 indexable projects from the current snapshot.

• koala73/worldmonitor

  A situational awareness application that consolidates over 65 external data providers and 500 news feeds into AI-synthesized briefs and multi-layered geospatial maps. It provides a native desktop interface, API access via an MCP server, an…

• HKUDS/Vibe-Trading

  An open-source research workspace that translates natural-language prompts into market-data loading, strategy drafting, backtesting, and report generation. It targets developers and data teams building research and analysis workflows acros…

• tensorflow/tensorflow

  TensorFlow provides an end-to-end open source machine learning platform with tools and libraries for building and training ML models. It supports CPU and GPU execution across Linux, macOS, Windows, Android, and Raspberry Pi, and can be ins…

• f/prompts.chat

  An open-source prompt library providing browsable, downloadable prompt examples for AI chat models, along with a CLI-driven path to deploy a private, branded instance. It targets general users, developers, and educators who need to find, s…

• firecrawl/firecrawl

  Firecrawl is a hosted web scraping, crawling, and search API built for AI applications. It converts web pages into Markdown, structured JSON, screenshots, and HTML while handling rotating proxies and rate limits.

• Snailclimb/JavaGuide

  An open-source knowledge repository and learning path covering Java backend interview topics, from Java basics and computer science fundamentals through databases, distributed systems, Spring, development tools, and AI application developm…

• run-llama/llama_index

  A data framework for augmenting LLMs with private data and building agentic applications, offering data connectors, indexing, and retrieval/query interfaces. It is designed for development teams that need to ingest existing documents, APIs…

• unclecode/crawl4ai

  Crawl4ai converts web pages into clean, LLM-ready Markdown and structured JSON for retrieval pipelines and AI agents. It provides controllable browser-based crawling and data extraction without mandatory API keys, operating as a self-hoste…

• supabase/supabase

  Supabase provides an open source, Firebase-like developer experience built around a dedicated Postgres database for web, mobile, and AI applications. It integrates database, authentication, and storage services, supporting self-hosted or m…

• TauricResearch/TradingAgents

  A multi-agent LLM framework that mirrors a real-world trading firm by deploying specialized agents to collaboratively evaluate market conditions and inform trading decisions. It deploys specialized agents for fundamentals, sentiment, news,…

• microsoft/ML-For-Beginners

  A 12-week, 26-lesson project-based curriculum covering classic machine learning techniques using Scikit-learn. It includes quizzes, assignments, and solutions while avoiding deep learning topics.

• Unstructured-IO/unstructured

  This project provides open-source components for ingesting and pre-processing unstructured data into structured outputs for large language models. It offers modular document partitioning across multiple file types and includes a Model Cont…

• krahets/hello-algo

  An open-source, beginner-friendly data structures and algorithms tutorial featuring animated illustrations and runnable code in multiple programming languages. It is designed to provide an accessible introduction for learners and educators.

• opendatalab/MinerU

  MinerU converts PDF, image, DOCX, PPTX, and XLSX files into Markdown and JSON for retrieval and extraction workflows. It runs locally across multiple backends via CLI or WebUI and requires at least 16GB of RAM, with optional GPU accelerati…

• hiyouga/LlamaFactory

  A framework that provides zero-code fine-tuning for over 100 large language models and vision-language models through command-line and web interfaces. It enables continuous pre-training, supervised fine-tuning, and preference alignment usi…

• D4Vinci/Scrapling

  An adaptive web scraping framework for Python that relocates elements after website changes, bypasses anti-bot systems, and automates dynamic content extraction. It supports self-hosted local runs via Python or Docker and requires Python 3…

• OpenBB-finance/OpenBB

  OpenBB provides a connect-once infrastructure layer that consolidates proprietary, licensed, and public financial data for consumption across Python environments, OpenBB Workspace, Excel, REST APIs, and MCP servers for AI agents.

• pathwaycom/pathway

  A Python framework with a Rust engine for building scalable ETL, streaming, real-time analytics, and RAG pipelines. It processes batch and streaming data using the same Python code.

• sansan0/TrendRadar

  A lightweight public opinion and trending topic monitor that aggregates multi-platform data and RSS feeds with keyword and AI-based filtering, routing alerts to channels such as Telegram. It relies on external APIs for data and an LLM via…

• ZhuLinsen/daily_stock_analysis

  The project consolidates multi-market stock data and news into daily, decision-oriented analysis reports. It provides an LLM-powered decision dashboard, an agent for multi-turn stock questions, and automated notifications via GitHub Action…

• Avik-Jain/100-Days-Of-ML-Code

  A 100-day educational machine learning track providing daily Markdown tutorials, Python code snippets, and visual infographics for common algorithms. The repository is designed for self-paced learning and is not intended for production dep…

• microsoft/qlib

  Qlib is a modular framework for applying AI and machine learning to quantitative investment research. It supports building complete workflows from data collection through model training and backtesting, with configurable components that ca…

• apache/airflow

  Apache Airflow is a platform for authoring, scheduling, and monitoring workflows as code. It targets data teams that need to orchestrate tasks and data pipelines, but does not provide native Windows support or handle streaming workloads.

• siyuan-note/siyuan

  SiYuan is a self-hosted, local-first personal knowledge management system providing Markdown WYSIWYG editing and fine-grained block-level references. It operates locally without requiring coding skills, with optional AI features available…

• aymericdamien/TensorFlow-Examples

  This repository provides concise tutorials and source code for learning TensorFlow v1 and v2, covering basic models, neural networks, data management, and hardware. It is designed for developers and educators who need clear, executable exa…

• ray-project/ray

  Ray provides a unified framework for scaling Python and AI applications from a laptop to a cluster. It offers distributed abstractions for tasks and actors, alongside libraries for data processing, training, tuning, reinforcement learning,…

• deepspeedai/DeepSpeed

  DeepSpeed is a PyTorch-integrated library that applies system innovations such as ZeRO, ZeRO-Infinity, 3D-Parallelism, and Ulysses Sequence Parallelism to make large-scale deep learning training and inference more efficient and effective.

• pingcap/tidb

  TiDB is an open-source, cloud-native distributed SQL database that handles transactions, analytics, and vector search. It provides horizontal scalability and MySQL compatibility while maintaining ACID guarantees across multiple nodes.

• HKUDS/LightRAG

  LightRAG is a graph-based retrieval-augmented generation framework that reduces computational overhead and LLM calls during indexing and querying compared to traditional GraphRAG. It supports incremental knowledge base updates without glob…

• PostHog/posthog

  PostHog provides a unified platform that combines product analytics, session replay, feature flags, experimentation, error tracking, logs, and surveys. It processes product data and user sessions to generate analytical visualizations, aler…

• ashishpatel26/500-AI-Machine-learning-Deep-learning-Computer-vision-NLP-Projects-with-code

  A curated directory providing categorized links to over 500 AI, ML, deep learning, computer vision, and NLP projects. It is a static list designed for discovery rather than a runtime application or library.

• microsoft/graphrag

  GraphRAG uses LLMs to extract structured data from unstructured text and applies knowledge graph memory structures to enhance LLM reasoning over private data. It is a Python library and demonstration project that requires CLI usage, prompt…

• shiyu-coder/Kronos

  A pre-trained foundation model specialized for financial candlestick (K-line) sequences, supporting forecasting and fine-tuning for quantitative data analysis. It is designed for data teams and researchers who need to process high-noise fi…

• explosion/spaCy

  A Python library providing Natural Language Processing capabilities including tagging, parsing, named entity recognition, and text classification. It includes pretrained pipelines for over 70 languages and a training system designed for re…

• eriklindernoren/ML-From-Scratch

  eriklindernoren/ml-from-scratch provides bare-bones NumPy implementations of fundamental machine learning models and algorithms. It is intended for educational use to help learners understand how algorithms work internally rather than to p…

• SigNoz/signoz

  SigNoz provides a single platform for logs, metrics, traces, alerts, and dashboards. It is built on OpenTelemetry and ClickHouse, offering self-hosted deployment alongside a managed cloud service with usage-based pricing.

• AMAI-GmbH/AI-Expert-Roadmap

  A community-shared set of roadmap charts demonstrating the learning paths and technologies for becoming a data scientist, machine learning, or AI expert. The content is explicitly targeted for becoming an AI expert in 2022, and an interact…

• hsliuping/TradingAgents-CN

  hsliuping/tradingagents-cn is a Chinese-localized, multi-agent LLM-based stock analysis learning platform supporting A-share, HK, and US stock research without providing real trading instructions.

• eugeneyan/applied-ml

  A curated directory that organizes real-world machine learning papers and engineering blogs into 31 specific categories. It serves data teams and researchers investigating how organizations implement ML in production.

• academic/awesome-datascience

  This project is a curated, open-source repository that organizes learning resources, algorithms, tools, and literature for data science. It provides a structured study path from fundamental Python programming through deep learning.