Track Awesome Streaming Updates Weekly

a curated list of awesome streaming frameworks, applications, etc

🏠 Home · 🔍 Search · 🔥 Feed · 📮 Subscribe · ❤️ Sponsor · 😺 manuzhang/awesome-streaming · ⭐ 3K · 🏷️ Big Data

[ Daily / Weekly / Overview ]

Aug 17 - Aug 23, 2026

Table of Contents / Engines and Platforms

• Aeron (⭐8.8k) - Reliable UDP unicast, multicast, and IPC message transport.

• Amazon Kinesis Data Streams - Fully managed service for ingesting and processing real-time data streams on AWS.

• Apache Heron (⭐3.6k) - Retired distributed, fault-tolerant stream processing engine originally developed at Twitter.

• Apache Pulsar (⭐15k) - Distributed pub-sub messaging and event streaming platform.

• Apache StreamPipes (⭐734) - Self-service industrial IoT platform for connecting, analyzing, and exploring data streams.

• AthenaX (⭐1.2k) - Uber's retired SQL-based streaming analytics platform.

• Azure Stream Analytics - Fully managed service for serverless real-time analytics in the cloud and at the edge.

• Concord - Historical distributed stream processing framework built on Apache Mesos.

• eKuiper (⭐1.7k) - Lightweight data stream processing engine for resource-constrained IoT edge devices.

• Fluvio (⭐5.2k) - Composable, stateful data streaming system with programmable in-line computation.

• Google Cloud Dataflow - Fully managed service for running Apache Beam batch and streaming pipelines.

• IBM Streams - Discontinued platform for distributed stream processing and real-time analytics.

• LaminarDB (⭐42) - Embeddable streaming SQL engine built on Apache Arrow and DataFusion.

• LogDevice (⭐1.9k) - Facebook's archived distributed storage system for sequential data.

• Mantis (⭐1.5k) - Netflix platform for building real-time, operations-focused stream processing applications.

• Metaq (⭐1.3k) - Taobao distributed messaging system.

• mupd8 (Muppet) (⭐128) - MapReduce-style framework for processing fast-moving data streams.

• NATS Streaming (⭐2.5k) - Deprecated disk-backed messaging system superseded by NATS JetStream.

• NSQ (⭐26k) - Real-time distributed messaging platform designed to operate at scale.

• PipelineDB (⭐2.7k) - Unmaintained PostgreSQL extension for continuous SQL aggregation over time-series data.

• S4 (⭐43) - Retired distributed platform for processing continuous unbounded data streams.

• SensorBee (⭐231) - Discontinued lightweight stream processing engine for IoT.

• Squall (⭐273) - Online SQL query processing engine built on Apache Storm.

• Tigon (⭐284) - High-throughput real-time stream processing framework built on Hadoop and HBase.

• Trill (⭐1.3k) - Single-node query processor for temporal and streaming data from Microsoft Research.

• Wally (⭐1.5k) - Distributed stream processing framework formerly named Wallaroo.

Table of Contents / Libraries, SDKs, and Programming Models

• Akka (⭐13k) - Toolkit and runtime for concurrent, distributed, resilient applications on the JVM.

• Akka Streams (⭐13k) - Reactive Streams implementation built on Akka actors.

• Apache Edgent (⭐222) - Retired programming model and runtime for streaming analytics on gateways and edge devices.

• Apache Pekko (⭐1.6k) - Open-source toolkit for concurrent, distributed, resilient applications, forked from Akka 2.6.

• Apache SAMOA (⭐251) - Retired distributed streaming machine learning framework.

• Apache StormCrawler (⭐993) - Scalable web crawler SDK based on Apache Storm.

• DataSketches (⭐959) - Apache library of stochastic streaming algorithms known as sketches.

• FastStream (⭐5.3k) - Asynchronous framework for building event-driven applications with Kafka, RabbitMQ, NATS, Redis, and MQTT.

• FS2 (⭐2.4k) - Compositional streaming I/O library for Scala.

• Jubatus (⭐707) - Distributed processing framework and library for online machine learning.

• MediaPipe (⭐37k) - Cross-platform, customizable ML solutions for live and streaming media.

• Monix (⭐1.9k) - High-performance Scala and Scala.js library for asynchronous and event-based programs.

• Numalogic (⭐172) - Collection of machine learning models and tools for anomaly detection and forecasting on operational time-series data.

• NVIDIA DeepStream SDK - Open-source GStreamer-based toolkit for real-time AI streaming analytics and multi-sensor processing.

• Pulsar (⭐1.8k) - Actor-based event-driven concurrency framework for Python.

• Scramjet TypeScript (⭐41) - Reactive stream programming framework for asynchronous JavaScript and TypeScript applications.

• Streamiz.Kafka.Net (⭐543) - .NET stream processing library for Apache Kafka.

• Streamparse (⭐1.5k) - Python API, command-line tools, and topology DSL for Apache Storm.

• Summingbird (⭐2.1k) - Retired library for expressing streaming MapReduce computations over Storm and Scalding.

• Tributary (⭐466) - Python library for constructing synchronous, reactive, and lazy dataflow graphs.

• WindFlow (⭐88) - Data stream processing parallel library for multi-core CPUs and GPUs.

• Yurita (⭐109) - PayPal anomaly detection framework built on Spark Structured Streaming.

Table of Contents / Data Integration and Pipelines

• Apache Flume (⭐2.6k) - Distributed service for collecting, aggregating, and moving large amounts of log-like data.

• Brooklin (⭐962) - Distributed system for reliable nearline data streaming between heterogeneous systems at scale.

• Camus (⭐879) - LinkedIn's retired, previous-generation Kafka-to-HDFS pipeline.

• Databus (⭐3.7k) - LinkedIn source-agnostic distributed change data capture system.

• faucet-stream (⭐7) - Config-driven platform and embeddable library for ETL, CDC, and streaming data movement.

• inGestr (⭐3.8k) - Command-line application and Python SDK for copying data between databases, SaaS applications, and data warehouses.

• Redpanda Connect (⭐8.7k) - Declarative stream processor for moving, enriching, transforming, and filtering data between sources and sinks.

• Suro (⭐796) - Netflix data pipeline for collecting, aggregating, and dispatching application events.

Table of Contents / Applications and Tools

• Streamdal (⭐615) - Code-native data privacy controls for detecting PII in application data flows.

Table of Contents / Benchmarks

• Flotilla (⭐238) - Automated message queue orchestration for scaled-up benchmarking.

Table of Contents / Readings

• MillWheel: Fault-Tolerant Stream Processing at Internet Scale

Aug 10 - Aug 16, 2026

Table of Contents / Engines and Platforms

• Wingfoil (⭐199) - Graph-based stream processing engine for latency-critical systems.

Table of Contents / Libraries, SDKs, and Programming Models

• CapyMOA (⭐140) - Machine learning library for data streams with a Python API and MOA backend.

May 18 - May 24, 2026

Table of Contents / Applications and Tools

• beava (⭐139) - Single-binary feature server for querying fresh per-entity counters and aggregates without a message broker.

• Eventum (⭐46) - Data generation platform for producing synthetic event streams.

May 11 - May 17, 2026

Table of Contents / Libraries, SDKs, and Programming Models

• Kzmlabs StateFun Actors (⭐13) - Stateful actors on Apache Flink 2.x with durable per-key state, exactly-once messaging, and Kafka and Kinesis I/O.

Feb 16 - Feb 22, 2026

Table of Contents / Data Integration and Pipelines

• Bruin (⭐1.7k) - End-to-end data pipeline tool combining ingestion, SQL and Python transformations, and data quality checks.

Table of Contents / Readings

• Real Time Analytics: Algorithms and Systems (VLDB 2015)

Jul 28 - Aug 03, 2025

Table of Contents / Engines and Platforms

• NebulaStream (⭐90) - Data management system for cloud-edge-sensor environments.

May 12 - May 18, 2025

Table of Contents / Engines and Platforms

• ArkFlow (⭐1.3k) - High-performance stream processing engine with pluggable sources, processors, and sinks.

Mar 17 - Mar 23, 2025

Table of Contents / Data Integration and Pipelines

• CocoIndex (⭐11k) - Incremental data transformation engine for continuously updated AI and agent workloads.

Jan 06 - Jan 12, 2025

Table of Contents / Engines and Platforms

• Apache Storm (⭐6.7k) - Distributed real-time computation system for reliably processing unbounded streams of data.

• Arroyo (⭐5k) - Distributed stream processing engine for stateful SQL computations over unbounded data.

Nov 18 - Nov 24, 2024

Table of Contents / Engines and Platforms

• RisingWave (⭐9.2k) - PostgreSQL-compatible streaming database for event-driven applications, real-time ETL, and continuous analytics.

Aug 19 - Aug 25, 2024

Table of Contents / Engines and Platforms

• AutoMQ (⭐10k) - Diskless Kafka-compatible streaming platform that stores durable data in object storage.

Feb 05 - Feb 11, 2024

Table of Contents / Engines and Platforms

• Numaflow (⭐2.8k) - Kubernetes-native, language-agnostic platform for scalable event-driven applications.

Dec 25 - Dec 31, 2023

Table of Contents / Engines and Platforms

• Proton (⭐2.2k) - Unified streaming and historical data analytics database powered by ClickHouse.

Nov 06 - Nov 12, 2023

Table of Contents / Data Integration and Pipelines

• Pathway (⭐63k) - Python ETL framework for stream processing, real-time analytics, and AI data pipelines.

Jul 03 - Jul 09, 2023

Table of Contents / Applications and Tools

• javactrl-kafka (⭐18) - Code-first distributed workflow engine for microservice orchestration and business process automation.

Jun 05 - Jun 11, 2023

Table of Contents / Applications and Tools

• Zilla (⭐693) - Multi-protocol gateway for connecting applications, APIs, agents, and devices to event streams.

May 08 - May 14, 2023

Table of Contents / Applications and Tools

• Substation (⭐403) - Toolkit for routing, normalizing, and enriching security event and audit logs.

Feb 20 - Feb 26, 2023

Table of Contents / Libraries, SDKs, and Programming Models

• Quix Streams (⭐1.6k) - Python framework for real-time data engineering, analytics, and machine learning on Apache Kafka.

Nov 21 - Nov 27, 2022

Table of Contents / Engines and Platforms

• Bytewax (⭐2k) - Community-maintained Python framework with a Rust-based distributed engine for stateful stream processing.

Table of Contents / Libraries, SDKs, and Programming Models

• River (⭐5.9k) - Online machine learning library for Python.

Aug 01 - Aug 07, 2022

Table of Contents / Engines and Platforms

• Scramjet Cloud Platform (⭐71) - Distributed runtime for running and managing data processing programs on edge, server, and cloud infrastructure.

Table of Contents / Libraries, SDKs, and Programming Models

• Scramjet C++ (⭐4) - Reactive stream programming framework for C++.

• Scramjet Python (⭐35) - Reactive stream programming framework for Python.

Mar 14 - Mar 20, 2022

Table of Contents / Engines and Platforms

• Redpanda (⭐12k) - Kafka API-compatible streaming data platform without ZooKeeper or a JVM.

Feb 07 - Feb 13, 2022

Table of Contents / Engines and Platforms

• Apache RocketMQ (⭐23k) - Cloud-native messaging and streaming platform for event-driven applications.

Table of Contents / Applications and Tools

• Nussknacker (⭐739) - Visual tool for defining and running real-time decision algorithms.

Jan 31 - Feb 06, 2022

Table of Contents / Libraries, SDKs, and Programming Models

• Daggy (⭐165) - Utility and developer library for aggregating and capturing data streams.

Oct 25 - Oct 31, 2021

Table of Contents / Readings

• Data Pipelines with Apache Airflow by Bas P. Harenslak and Julian Rutger de Ruiter

Sep 27 - Oct 03, 2021

Table of Contents / Readings

• Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing by Reuven Lax, Slava Chernyak, and Tyler Akidau

May 03 - May 09, 2021

Table of Contents / Engines and Platforms

• Siddhi (⭐1.6k) - Streaming SQL and complex event processing engine.

Mar 29 - Apr 04, 2021

Table of Contents / Engines and Platforms

• ksqlDB (⭐312) - Source-available database purpose-built for stream processing applications.

• Materialize - Incremental SQL engine for maintaining continuously updated views over changing data.

Mar 22 - Mar 28, 2021

Table of Contents / Engines and Platforms

• HStreamDB (⭐721) - Cloud-native streaming database for IoT data storage and real-time processing.

Feb 15 - Feb 21, 2021

Table of Contents / Engines and Platforms

• Maki Nage (⭐42) - Stream processing framework for data scientists based on Kafka and ReactiveX.

Jun 15 - Jun 21, 2020

Table of Contents / Readings

• Grokking Streaming Systems by Josh Fischer & Ning Wang

May 11 - May 17, 2020

Table of Contents / Data Integration and Pipelines

• RudderStack (⭐4.5k) - Developer-focused customer data platform for event streaming and cloud-to-warehouse data pipelines.

May 04 - May 10, 2020

Table of Contents / Engines and Platforms

• Gazette (⭐794) - Cloud-storage-backed streaming infrastructure that combines SQL, batch, and millisecond-latency stream processing.

Mar 23 - Mar 29, 2020

Table of Contents / Engines and Platforms

• LightSaber (⭐74) - Multi-core stream processing engine using code generation for window aggregation.

Oct 14 - Oct 20, 2019

Table of Contents / Libraries, SDKs, and Programming Models

• Apache Beam (⭐8.6k) - Unified programming model and language-specific SDKs for batch and streaming data processing.

Sep 09 - Sep 15, 2019

Table of Contents / Libraries, SDKs, and Programming Models

• Stream Ops (⭐50) - Embeddable data streaming engine and stream processing API for Java.

Sep 02 - Sep 08, 2019

Table of Contents / Engines and Platforms

• Gearpump (⭐756) - Lightweight real-time distributed streaming engine built on Akka.

Aug 12 - Aug 18, 2019

Table of Contents / Libraries, SDKs, and Programming Models

• Streamz (⭐1.3k) - Library for building continuous data pipelines with branching, joining, flow control, and back pressure.

Jul 29 - Aug 04, 2019

Table of Contents / Libraries, SDKs, and Programming Models

• StormCV (⭐174) - Computer vision operations and data models for Apache Storm.

• streamDM (⭐497) - Huawei library for mining data streams with Spark Streaming.

• trident-ml (⭐383) - Online machine learning library based on Apache Storm Trident.

Jul 15 - Jul 21, 2019

Table of Contents / Engines and Platforms

• StreamCQL (⭐0) - Storm-based continuous query language for real-time computation.

Apr 15 - Apr 21, 2019

Table of Contents / Engines and Platforms

• Apache Apex (⭐350) - Unified platform for big data stream and batch processing.

• Apache Flink (⭐26k) - Distributed engine for stateful computation over bounded and unbounded data streams.

• Apache Kafka (⭐33k) - Distributed event streaming platform for high-performance data pipelines and applications.

• Apache Samza (⭐846) - Distributed stream processing framework built on Apache Kafka with standalone and YARN deployment options.

• Apache Spark Streaming (⭐44k) - Scalable, fault-tolerant stream processing on Apache Spark.

• Esper (⭐875) - Complex event processing, Streaming SQL, and event series analysis engine.

• hailstorm (⭐94) - Distributed stream processing with exactly-once semantics based on Storm.

• Hazelcast Jet (⭐1.1k) - Stream and batch processing engine whose development moved into Hazelcast Platform.

• Onyx (⭐2.1k) - Distributed, masterless, fault-tolerant data processing platform.

• SABER (⭐44) - Window-based hybrid CPU/GPU stream processing engine.

• SPQR (⭐30) - Dynamic framework for processing high-volume data streams through pipelines.

• Teknek (⭐10) - Stream processing platform with the interactive Stream Operator Language shell.

Table of Contents / Libraries, SDKs, and Programming Models

• Apache Kafka Streams (⭐33k) - Stream processing library included with Apache Kafka.

• coast (⭐60) - DSL that builds exactly-once dataflow graphs on top of Apache Samza.

• Faust (⭐6.8k) - Deprecated Python stream processing library inspired by Kafka Streams.

• samza-luwak (⭐100) - Integration of Apache Samza with the Luwak stored-query engine for full-text stream search.

• Swave (⭐173) - Lightweight Reactive Streams infrastructure toolkit for Scala.

Table of Contents / Applications and Tools

• straw (⭐103) - Platform for real-time streaming search.

• StreamAlert (⭐2.9k) - Airbnb serverless framework for real-time security log analysis and alerting.

• StreamFlow (⭐256) - Tool for building and monitoring stream processing workflows.

• StreamingBandit (⭐85) - Web application for setting up and evaluating contextual multi-armed bandit experiments.

• Streamline (⭐167) - Hortonworks visual stream analytics framework built around engines such as Apache Storm.

• Turbine (⭐833) - Netflix tool for aggregating Server-Sent Event JSON streams.

Table of Contents / Benchmarks

• storm-perf-test (⭐75) - Apache Storm performance and stress test.

• streaming-benchmarks (⭐647) - Benchmarks for low-latency stream processing systems including Storm, Spark, and Flink.

Feb 08 - Feb 14, 2016

Table of Contents / Readings

• In-Stream Big Data Processing

• The world beyond batch: Streaming 101 by Tyler Akidau.