Awesome Streaming Overview

a curated list of awesome streaming frameworks, applications, etc

🏠 Home · 🔥 Feed · 📮 Subscribe · ❤️ Sponsor · 😺 manuzhang/awesome-streaming · ⭐ 3K · 🏷️ Big Data

[ Daily / Weekly / Overview ]

Awesome Streaming

A curated list of awesome streaming (stream processing) frameworks, applications, readings and other resources. Inspired by other awesome projects (⭐494k).

Website

https://manuzhang.github.io/awesome-streaming/ is a more dynamic website where you can find updates of the awesome projects here.

Table of Contents

• Engines and Platforms
• Libraries, SDKs, and Programming Models
• Data Integration and Pipelines
• Applications and Tools
• Benchmarks
• Readings

Engines and Platforms

• Aeron (⭐8.8k) - Reliable UDP unicast, multicast, and IPC message transport.
• Amazon Kinesis Data Streams - Fully managed service for ingesting and processing real-time data streams on AWS.
• Apache Apex (⭐350) - Unified platform for big data stream and batch processing.
• Apache Flink (⭐26k) - Distributed engine for stateful computation over bounded and unbounded data streams.
• Apache Heron (⭐3.6k) - Retired distributed, fault-tolerant stream processing engine originally developed at Twitter.
• Apache Kafka (⭐33k) - Distributed event streaming platform for high-performance data pipelines and applications.
• Apache Pulsar (⭐15k) - Distributed pub-sub messaging and event streaming platform.
• Apache RocketMQ (⭐23k) - Cloud-native messaging and streaming platform for event-driven applications.
• Apache Samza (⭐846) - Distributed stream processing framework built on Apache Kafka with standalone and YARN deployment options.
• Apache Spark Streaming (⭐44k) - Scalable, fault-tolerant stream processing on Apache Spark.
• Apache Storm (⭐6.7k) - Distributed real-time computation system for reliably processing unbounded streams of data.
• Apache StreamPipes (⭐734) - Self-service industrial IoT platform for connecting, analyzing, and exploring data streams.
• ArkFlow (⭐1.3k) - High-performance stream processing engine with pluggable sources, processors, and sinks.
• Arroyo (⭐5k) - Distributed stream processing engine for stateful SQL computations over unbounded data.
• AthenaX (⭐1.2k) - Uber's retired SQL-based streaming analytics platform.
• AutoMQ (⭐10k) - Diskless Kafka-compatible streaming platform that stores durable data in object storage.
• Azure Stream Analytics - Fully managed service for serverless real-time analytics in the cloud and at the edge.
• Bytewax (⭐2k) - Community-maintained Python framework with a Rust-based distributed engine for stateful stream processing.
• Concord - Historical distributed stream processing framework built on Apache Mesos.
• eKuiper (⭐1.7k) - Lightweight data stream processing engine for resource-constrained IoT edge devices.
• Esper (⭐875) - Complex event processing, Streaming SQL, and event series analysis engine.
• Fluvio (⭐5.2k) - Composable, stateful data streaming system with programmable in-line computation.
• Gazette (⭐794) - Cloud-storage-backed streaming infrastructure that combines SQL, batch, and millisecond-latency stream processing.
• Gearpump (⭐756) - Lightweight real-time distributed streaming engine built on Akka.
• Google Cloud Dataflow - Fully managed service for running Apache Beam batch and streaming pipelines.
• hailstorm (⭐94) - Distributed stream processing with exactly-once semantics based on Storm.
• Hazelcast Jet (⭐1.1k) - Stream and batch processing engine whose development moved into Hazelcast Platform.
• HStreamDB (⭐721) - Cloud-native streaming database for IoT data storage and real-time processing.
• IBM Streams - Discontinued platform for distributed stream processing and real-time analytics.
• ksqlDB (⭐312) - Source-available database purpose-built for stream processing applications.
• LaminarDB (⭐42) - Embeddable streaming SQL engine built on Apache Arrow and DataFusion.
• LightSaber (⭐74) - Multi-core stream processing engine using code generation for window aggregation.
• LogDevice (⭐1.9k) - Facebook's archived distributed storage system for sequential data.
• Maki Nage (⭐42) - Stream processing framework for data scientists based on Kafka and ReactiveX.
• Mantis (⭐1.5k) - Netflix platform for building real-time, operations-focused stream processing applications.
• Materialize - Incremental SQL engine for maintaining continuously updated views over changing data.
• Metaq (⭐1.3k) - Taobao distributed messaging system.
• mupd8 (Muppet) (⭐128) - MapReduce-style framework for processing fast-moving data streams.
• NATS Streaming (⭐2.5k) - Deprecated disk-backed messaging system superseded by NATS JetStream.
• NebulaStream (⭐90) - Data management system for cloud-edge-sensor environments.
• NSQ (⭐26k) - Real-time distributed messaging platform designed to operate at scale.
• Numaflow (⭐2.8k) - Kubernetes-native, language-agnostic platform for scalable event-driven applications.
• Onyx (⭐2.1k) - Distributed, masterless, fault-tolerant data processing platform.
• PipelineDB (⭐2.7k) - Unmaintained PostgreSQL extension for continuous SQL aggregation over time-series data.
• Proton (⭐2.2k) - Unified streaming and historical data analytics database powered by ClickHouse.
• Redpanda (⭐12k) - Kafka API-compatible streaming data platform without ZooKeeper or a JVM.
• RisingWave (⭐9.2k) - PostgreSQL-compatible streaming database for event-driven applications, real-time ETL, and continuous analytics.
• S4 (⭐43) - Retired distributed platform for processing continuous unbounded data streams.
• SABER (⭐44) - Window-based hybrid CPU/GPU stream processing engine.
• Scramjet Cloud Platform (⭐71) - Distributed runtime for running and managing data processing programs on edge, server, and cloud infrastructure.
• SensorBee (⭐231) - Discontinued lightweight stream processing engine for IoT.
• Siddhi (⭐1.6k) - Streaming SQL and complex event processing engine.
• SPQR (⭐30) - Dynamic framework for processing high-volume data streams through pipelines.
• Squall (⭐273) - Online SQL query processing engine built on Apache Storm.
• StreamCQL (⭐0) - Storm-based continuous query language for real-time computation.
• Teknek (⭐10) - Stream processing platform with the interactive Stream Operator Language shell.
• Tigon (⭐284) - High-throughput real-time stream processing framework built on Hadoop and HBase.
• Trill (⭐1.3k) - Single-node query processor for temporal and streaming data from Microsoft Research.
• Wally (⭐1.5k) - Distributed stream processing framework formerly named Wallaroo.
• Wingfoil (⭐199) - Graph-based stream processing engine for latency-critical systems.

Libraries, SDKs, and Programming Models

• Akka (⭐13k) - Toolkit and runtime for concurrent, distributed, resilient applications on the JVM.
• Akka Streams (⭐13k) - Reactive Streams implementation built on Akka actors.
• Apache Beam (⭐8.6k) - Unified programming model and language-specific SDKs for batch and streaming data processing.
• Apache Edgent (⭐222) - Retired programming model and runtime for streaming analytics on gateways and edge devices.
• Apache Kafka Streams (⭐33k) - Stream processing library included with Apache Kafka.
• Apache Pekko (⭐1.6k) - Open-source toolkit for concurrent, distributed, resilient applications, forked from Akka 2.6.
• Apache SAMOA (⭐251) - Retired distributed streaming machine learning framework.
• Apache StormCrawler (⭐993) - Scalable web crawler SDK based on Apache Storm.
• CapyMOA (⭐140) - Machine learning library for data streams with a Python API and MOA backend.
• coast (⭐60) - DSL that builds exactly-once dataflow graphs on top of Apache Samza.
• Daggy (⭐165) - Utility and developer library for aggregating and capturing data streams.
• DataSketches (⭐959) - Apache library of stochastic streaming algorithms known as sketches.
• FastStream (⭐5.3k) - Asynchronous framework for building event-driven applications with Kafka, RabbitMQ, NATS, Redis, and MQTT.
• Faust (⭐6.8k) - Deprecated Python stream processing library inspired by Kafka Streams.
• FS2 (⭐2.4k) - Compositional streaming I/O library for Scala.
• Jubatus (⭐707) - Distributed processing framework and library for online machine learning.
• Kzmlabs StateFun Actors (⭐13) - Stateful actors on Apache Flink 2.x with durable per-key state, exactly-once messaging, and Kafka and Kinesis I/O.
• MediaPipe (⭐37k) - Cross-platform, customizable ML solutions for live and streaming media.
• Monix (⭐1.9k) - High-performance Scala and Scala.js library for asynchronous and event-based programs.
• Numalogic (⭐172) - Collection of machine learning models and tools for anomaly detection and forecasting on operational time-series data.
• NVIDIA DeepStream SDK - Open-source GStreamer-based toolkit for real-time AI streaming analytics and multi-sensor processing.
• Pulsar (⭐1.8k) - Actor-based event-driven concurrency framework for Python.
• Quix Streams (⭐1.6k) - Python framework for real-time data engineering, analytics, and machine learning on Apache Kafka.
• River (⭐5.9k) - Online machine learning library for Python.
• samza-luwak (⭐100) - Integration of Apache Samza with the Luwak stored-query engine for full-text stream search.
• Scramjet C++ (⭐4) - Reactive stream programming framework for C++.
• Scramjet Python (⭐35) - Reactive stream programming framework for Python.
• Scramjet TypeScript (⭐41) - Reactive stream programming framework for asynchronous JavaScript and TypeScript applications.
• StormCV (⭐174) - Computer vision operations and data models for Apache Storm.
• Stream Ops (⭐50) - Embeddable data streaming engine and stream processing API for Java.
• streamDM (⭐497) - Huawei library for mining data streams with Spark Streaming.
• Streamiz.Kafka.Net (⭐543) - .NET stream processing library for Apache Kafka.
• Streamparse (⭐1.5k) - Python API, command-line tools, and topology DSL for Apache Storm.
• Streamz (⭐1.3k) - Library for building continuous data pipelines with branching, joining, flow control, and back pressure.
• Summingbird (⭐2.1k) - Retired library for expressing streaming MapReduce computations over Storm and Scalding.
• Swave (⭐173) - Lightweight Reactive Streams infrastructure toolkit for Scala.
• Tributary (⭐466) - Python library for constructing synchronous, reactive, and lazy dataflow graphs.
• trident-ml (⭐383) - Online machine learning library based on Apache Storm Trident.
• WindFlow (⭐88) - Data stream processing parallel library for multi-core CPUs and GPUs.
• Yurita (⭐109) - PayPal anomaly detection framework built on Spark Structured Streaming.

Data Integration and Pipelines

• Apache Flume (⭐2.6k) - Distributed service for collecting, aggregating, and moving large amounts of log-like data.
• Brooklin (⭐962) - Distributed system for reliable nearline data streaming between heterogeneous systems at scale.
• Bruin (⭐1.7k) - End-to-end data pipeline tool combining ingestion, SQL and Python transformations, and data quality checks.
• Camus (⭐879) - LinkedIn's retired, previous-generation Kafka-to-HDFS pipeline.
• CocoIndex (⭐11k) - Incremental data transformation engine for continuously updated AI and agent workloads.
• Databus (⭐3.7k) - LinkedIn source-agnostic distributed change data capture system.
• faucet-stream (⭐7) - Config-driven platform and embeddable library for ETL, CDC, and streaming data movement.
• inGestr (⭐3.8k) - Command-line application and Python SDK for copying data between databases, SaaS applications, and data warehouses.
• Pathway (⭐63k) - Python ETL framework for stream processing, real-time analytics, and AI data pipelines.
• Redpanda Connect (⭐8.7k) - Declarative stream processor for moving, enriching, transforming, and filtering data between sources and sinks.
• RudderStack (⭐4.5k) - Developer-focused customer data platform for event streaming and cloud-to-warehouse data pipelines.
• Suro (⭐796) - Netflix data pipeline for collecting, aggregating, and dispatching application events.

Applications and Tools

• beava (⭐139) - Single-binary feature server for querying fresh per-entity counters and aggregates without a message broker.
• Eventum (⭐46) - Data generation platform for producing synthetic event streams.
• javactrl-kafka (⭐18) - Code-first distributed workflow engine for microservice orchestration and business process automation.
• Nussknacker (⭐739) - Visual tool for defining and running real-time decision algorithms.
• straw (⭐103) - Platform for real-time streaming search.
• StreamAlert (⭐2.9k) - Airbnb serverless framework for real-time security log analysis and alerting.
• Streamdal (⭐615) - Code-native data privacy controls for detecting PII in application data flows.
• StreamFlow (⭐256) - Tool for building and monitoring stream processing workflows.
• StreamingBandit (⭐85) - Web application for setting up and evaluating contextual multi-armed bandit experiments.
• Streamline (⭐167) - Hortonworks visual stream analytics framework built around engines such as Apache Storm.
• Substation (⭐403) - Toolkit for routing, normalizing, and enriching security event and audit logs.
• Turbine (⭐833) - Netflix tool for aggregating Server-Sent Event JSON streams.
• Zilla (⭐693) - Multi-protocol gateway for connecting applications, APIs, agents, and devices to event streams.

Benchmarks

• Flotilla (⭐238) - Automated message queue orchestration for scaled-up benchmarking.
• storm-perf-test (⭐75) - Apache Storm performance and stress test.
• streaming-benchmarks (⭐647) - Benchmarks for low-latency stream processing systems including Storm, Spark, and Flink.

Readings

1. In-Stream Big Data Processing
2. The world beyond batch: Streaming 101 by Tyler Akidau.
3. Real Time Analytics: Algorithms and Systems (VLDB 2015)
4. Grokking Streaming Systems by Josh Fischer & Ning Wang
5. Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing by Reuven Lax, Slava Chernyak, and Tyler Akidau
6. Data Pipelines with Apache Airflow by Bas P. Harenslak and Julian Rutger de Ruiter
7. MillWheel: Fault-Tolerant Stream Processing at Internet Scale

License

Licensed under a Creative Commons Attribution-ShareAlike 4.0 International License