Track Awesome Streaming Updates Daily
a curated list of awesome streaming frameworks, applications, etc
🏠 Home · 🔍 Search · 🔥 Feed · 📮 Subscribe · ❤️ Sponsor · 😺 manuzhang/awesome-streaming · ⭐ 3K · 🏷️ Big Data
[ Daily / Weekly / Overview ]
Table of Contents / Engines and Platforms
- Aeron (⭐8.8k)
- Reliable UDP unicast, multicast, and IPC message transport.
- Apache Heron (⭐3.6k)
- Retired distributed, fault-tolerant stream processing engine originally developed at Twitter.
- Azure Stream Analytics
- Fully managed service for serverless real-time analytics in the cloud and at the edge.
- Concord
- Historical distributed stream processing framework built on Apache Mesos.
- eKuiper (⭐1.7k)
- Lightweight data stream processing engine for resource-constrained IoT edge devices.
- Fluvio (⭐5.2k)
- Composable, stateful data streaming system with programmable in-line computation.
- IBM Streams
- Discontinued platform for distributed stream processing and real-time analytics.
- LaminarDB (⭐42)
- Embeddable streaming SQL engine built on Apache Arrow and DataFusion.
- LogDevice (⭐1.9k) - Facebook's archived distributed storage system for sequential data.
- Mantis (⭐1.5k) - Netflix platform for building real-time, operations-focused stream processing applications.
- NSQ (⭐26k) - Real-time distributed messaging platform designed to operate at scale.
- PipelineDB (⭐2.7k)
- Unmaintained PostgreSQL extension for continuous SQL aggregation over time-series data.
- S4 (⭐43) - Retired distributed platform for processing continuous unbounded data streams.
- Squall (⭐273) - Online SQL query processing engine built on Apache Storm.
- Tigon (⭐284)
- High-throughput real-time stream processing framework built on Hadoop and HBase.
- Trill (⭐1.3k)
- Single-node query processor for temporal and streaming data from Microsoft Research.
- Wally (⭐1.5k)
- Distributed stream processing framework formerly named Wallaroo.
Table of Contents / Libraries, SDKs, and Programming Models
- Akka (⭐13k)
- Toolkit and runtime for concurrent, distributed, resilient applications on the JVM.
- Apache Edgent (⭐222) - Retired programming model and runtime for streaming analytics on gateways and edge devices.
- Apache Pekko (⭐1.6k) - Open-source toolkit for concurrent, distributed, resilient applications, forked from Akka 2.6.
- FastStream (⭐5.3k)
- Asynchronous framework for building event-driven applications with Kafka, RabbitMQ, NATS, Redis, and MQTT.
- FS2 (⭐2.4k)
- Compositional streaming I/O library for Scala.
- Jubatus (⭐707) - Distributed processing framework and library for online machine learning.
- MediaPipe (⭐37k)
- Cross-platform, customizable ML solutions for live and streaming media.
- Monix (⭐1.9k) - High-performance Scala and Scala.js library for asynchronous and event-based programs.
- Numalogic (⭐172) - Collection of machine learning models and tools for anomaly detection and forecasting on operational time-series data.
- NVIDIA DeepStream SDK
- Open-source GStreamer-based toolkit for real-time AI streaming analytics and multi-sensor processing.
- Pulsar (⭐1.8k) - Actor-based event-driven concurrency framework for Python.
- Summingbird (⭐2.1k) - Retired library for expressing streaming MapReduce computations over Storm and Scalding.
- Tributary (⭐466) - Python library for constructing synchronous, reactive, and lazy dataflow graphs.
- WindFlow (⭐88) - Data stream processing parallel library for multi-core CPUs and GPUs.
- Yurita (⭐109) - PayPal anomaly detection framework built on Spark Structured Streaming.
Table of Contents / Data Integration and Pipelines
- Apache Flume (⭐2.6k) - Distributed service for collecting, aggregating, and moving large amounts of log-like data.
- Brooklin (⭐962) - Distributed system for reliable nearline data streaming between heterogeneous systems at scale.
- Camus (⭐879) - LinkedIn's retired, previous-generation Kafka-to-HDFS pipeline.
- Databus (⭐3.7k) - LinkedIn source-agnostic distributed change data capture system.
- faucet-stream (⭐7) - Config-driven platform and embeddable library for ETL, CDC, and streaming data movement.
- inGestr (⭐3.8k)
- Command-line application and Python SDK for copying data between databases, SaaS applications, and data warehouses.
- Redpanda Connect (⭐8.7k) - Declarative stream processor for moving, enriching, transforming, and filtering data between sources and sinks.
- Suro (⭐796) - Netflix data pipeline for collecting, aggregating, and dispatching application events.
Table of Contents / Applications and Tools
- Streamdal (⭐615)
- Code-native data privacy controls for detecting PII in application data flows.
Table of Contents / Benchmarks
- Flotilla (⭐238) - Automated message queue orchestration for scaled-up benchmarking.
Table of Contents / Readings
Table of Contents / Engines and Platforms
- Wingfoil (⭐199)
- Graph-based stream processing engine for latency-critical systems.
Table of Contents / Libraries, SDKs, and Programming Models
- CapyMOA (⭐140)
- Machine learning library for data streams with a Python API and MOA backend.
Table of Contents / Applications and Tools
- Eventum (⭐46)
- Data generation platform for producing synthetic event streams.
Table of Contents / Applications and Tools
- beava (⭐139)
- Single-binary feature server for querying fresh per-entity counters and aggregates without a message broker.
Table of Contents / Libraries, SDKs, and Programming Models
- Kzmlabs StateFun Actors (⭐13) - Stateful actors on Apache Flink 2.x with durable per-key state, exactly-once messaging, and Kafka and Kinesis I/O.
Table of Contents / Data Integration and Pipelines
- Bruin (⭐1.7k) - End-to-end data pipeline tool combining ingestion, SQL and Python transformations, and data quality checks.
Table of Contents / Readings
Table of Contents / Engines and Platforms
Table of Contents / Engines and Platforms
- ArkFlow (⭐1.3k) - High-performance stream processing engine with pluggable sources, processors, and sinks.
Table of Contents / Data Integration and Pipelines
- CocoIndex (⭐11k) - Incremental data transformation engine for continuously updated AI and agent workloads.
Table of Contents / Engines and Platforms
- Apache Storm (⭐6.7k) - Distributed real-time computation system for reliably processing unbounded streams of data.
- Arroyo (⭐5k) - Distributed stream processing engine for stateful SQL computations over unbounded data.
Table of Contents / Engines and Platforms
- RisingWave (⭐9.2k) - PostgreSQL-compatible streaming database for event-driven applications, real-time ETL, and continuous analytics.
Table of Contents / Engines and Platforms
- AutoMQ (⭐10k)
- Diskless Kafka-compatible streaming platform that stores durable data in object storage.
Table of Contents / Engines and Platforms
- Numaflow (⭐2.8k)
- Kubernetes-native, language-agnostic platform for scalable event-driven applications.
Table of Contents / Engines and Platforms
- Proton (⭐2.2k) - Unified streaming and historical data analytics database powered by ClickHouse.
Table of Contents / Data Integration and Pipelines
- Pathway (⭐63k)
- Python ETL framework for stream processing, real-time analytics, and AI data pipelines.
Table of Contents / Applications and Tools
- javactrl-kafka (⭐18) - Code-first distributed workflow engine for microservice orchestration and business process automation.
Table of Contents / Applications and Tools
- Zilla (⭐693) - Multi-protocol gateway for connecting applications, APIs, agents, and devices to event streams.
Table of Contents / Applications and Tools
- Substation (⭐403) - Toolkit for routing, normalizing, and enriching security event and audit logs.
Table of Contents / Libraries, SDKs, and Programming Models
- Quix Streams (⭐1.6k) - Python framework for real-time data engineering, analytics, and machine learning on Apache Kafka.
Table of Contents / Engines and Platforms
- Bytewax (⭐2k) - Community-maintained Python framework with a Rust-based distributed engine for stateful stream processing.
Table of Contents / Libraries, SDKs, and Programming Models
Table of Contents / Engines and Platforms
Table of Contents / Libraries, SDKs, and Programming Models
Table of Contents / Engines and Platforms
- Redpanda (⭐12k) - Kafka API-compatible streaming data platform without ZooKeeper or a JVM.
Table of Contents / Applications and Tools
Table of Contents / Engines and Platforms
Table of Contents / Libraries, SDKs, and Programming Models
- Daggy (⭐165) - Utility and developer library for aggregating and capturing data streams.
Table of Contents / Readings
Table of Contents / Readings
Table of Contents / Engines and Platforms
Table of Contents / Engines and Platforms
- ksqlDB (⭐312) - Source-available database purpose-built for stream processing applications.
- Materialize - Incremental SQL engine for maintaining continuously updated views over changing data.
Table of Contents / Engines and Platforms
- HStreamDB (⭐721)
- Cloud-native streaming database for IoT data storage and real-time processing.
Table of Contents / Engines and Platforms
- Maki Nage (⭐42) - Stream processing framework for data scientists based on Kafka and ReactiveX.
Table of Contents / Readings
Table of Contents / Data Integration and Pipelines
- RudderStack (⭐4.5k) - Developer-focused customer data platform for event streaming and cloud-to-warehouse data pipelines.
Table of Contents / Engines and Platforms
- Gazette (⭐794) - Cloud-storage-backed streaming infrastructure that combines SQL, batch, and millisecond-latency stream processing.
Table of Contents / Engines and Platforms
- LightSaber (⭐74) - Multi-core stream processing engine using code generation for window aggregation.
Table of Contents / Libraries, SDKs, and Programming Models
- Apache Beam (⭐8.6k)
- Unified programming model and language-specific SDKs for batch and streaming data processing.
Table of Contents / Libraries, SDKs, and Programming Models
- Stream Ops (⭐50) - Embeddable data streaming engine and stream processing API for Java.
Table of Contents / Engines and Platforms
- Gearpump (⭐756) - Lightweight real-time distributed streaming engine built on Akka.
Table of Contents / Libraries, SDKs, and Programming Models
- Streamz (⭐1.3k) - Library for building continuous data pipelines with branching, joining, flow control, and back pressure.
Table of Contents / Libraries, SDKs, and Programming Models
- StormCV (⭐174) - Computer vision operations and data models for Apache Storm.
- streamDM (⭐497) - Huawei library for mining data streams with Spark Streaming.
Table of Contents / Engines and Platforms
- StreamCQL (⭐0) - Storm-based continuous query language for real-time computation.
Table of Contents / Engines and Platforms
- Apache Flink (⭐26k) - Distributed engine for stateful computation over bounded and unbounded data streams.
- Apache Kafka (⭐33k) - Distributed event streaming platform for high-performance data pipelines and applications.
- Apache Samza (⭐846) - Distributed stream processing framework built on Apache Kafka with standalone and YARN deployment options.
- Esper (⭐875) - Complex event processing, Streaming SQL, and event series analysis engine.
- hailstorm (⭐94) - Distributed stream processing with exactly-once semantics based on Storm.
- Hazelcast Jet (⭐1.1k) - Stream and batch processing engine whose development moved into Hazelcast Platform.
- Onyx (⭐2.1k)
- Distributed, masterless, fault-tolerant data processing platform.
- SABER (⭐44)
- Window-based hybrid CPU/GPU stream processing engine.
- SPQR (⭐30) - Dynamic framework for processing high-volume data streams through pipelines.
- Teknek (⭐10) - Stream processing platform with the interactive Stream Operator Language shell.
Table of Contents / Libraries, SDKs, and Programming Models
- coast (⭐60) - DSL that builds exactly-once dataflow graphs on top of Apache Samza.
- Faust (⭐6.8k) - Deprecated Python stream processing library inspired by Kafka Streams.
- samza-luwak (⭐100) - Integration of Apache Samza with the Luwak stored-query engine for full-text stream search.
- Swave (⭐173) - Lightweight Reactive Streams infrastructure toolkit for Scala.
Table of Contents / Applications and Tools
- StreamAlert (⭐2.9k) - Airbnb serverless framework for real-time security log analysis and alerting.
- StreamingBandit (⭐85) - Web application for setting up and evaluating contextual multi-armed bandit experiments.
- Streamline (⭐167) - Hortonworks visual stream analytics framework built around engines such as Apache Storm.
- Turbine (⭐833) - Netflix tool for aggregating Server-Sent Event JSON streams.
Table of Contents / Benchmarks
Table of Contents / Readings
- Self-service industrial IoT platform for connecting, analyzing, and exploring data streams.
- Distributed runtime for running and managing data processing programs on edge, server, and cloud infrastructure.
- Scalable, fault-tolerant stream processing on Apache Spark.