README.md

Siddhi - Cloud Native Stream Processor

Geting Started | Download | User Guide | Contribute

Siddhi is a cloud native Streaming and Complex Event Processing engine that understands Streaming SQL queries in order to capture events from diverse data sources, process them, detect complex conditions, and publish output to various endpoints in real time.

Siddhi can run as an embedded Java library, and as a micro service on bare metal, VM, Docker and natively in Kubernetes. It also has a graphical and text editor for building Streaming Data Integration and Streaming Analytics applications.

Distributions

And more installation options

Overview

Siddhi supports:

• Streaming Data Integration

  • Retrieving data from various event sources (NATS, Kafka, JMS, HTTP, CDC, etc)
  • Map events to and from multiple event formats (JSON, XML, Text, Avro, etc)
  • Data preprocessing & cleaning
  • Joining multiple data streams
  • Integrate streaming data with databases (RDBMS, Cassandra, HBase, Redis, etc)
  • Integrate with external services
  • Publish data to multiple event sinks (Email, JMS, HTTP, etc)

• Streaming Data Analytics

  • Generating alerts based on thresholds
  • Calculate aggregations over a short windows (time, length, session, unique, etc) or a long time period
  • Calculate aggregations over long time periods with seconds, minutes, hours, days, months & years granularity
  • Correlating data while finding missing and erroneous events
  • Detecting temporal event patterns
  • Analyzing trends (rise, fall, turn, tipple bottom)
  • Run pretreated machine learning models (PMML, Tensorflow)
  • Learn and predict at runtime using online machine learning models

• Adaptive Intelligence

  • Static rule processing
  • Stateful rule processing
  • Decision making through synchronous stream processing
  • Query tables, windows and aggregations

• And many more ... For more information, see Patterns of Streaming Realtime Analytics

Siddhi is free and open source, released under Apache Software License v2.0.

Why use Siddhi ?

• Fast. UBER uses it to process 20 Billion events per day (300,000 events per second).
• Lightweight (core Siddhi libs are <2MB), and embeddable in Android, Python and RaspberryPi.
• Has over 50 Siddhi Extensions
• Used by over 60 companies including many Fortune 500 companies in production. Following are some examples:
  • WSO2 uses Siddhi for the following purposes:
    • To provide distributed and high available stream processing capabilities via WSO2 Stream Processor. It is named as a strong performer in The Forrester Wave: Big Data Streaming Analytics, Q1 2016 (Report).
    • As the edge analytics library of WSO2 IoT Server.
    • As the core of WSO2 API Manager's throttling.
    • As the core of WSO2 products' analytics.
  • UBER uses Siddhi for fraud analytics.
  • Apache Eagle uses Siddhi as a policy engine.
  • Also used by Punch Platform, Sqooba, and SiteWhere
• Solutions based on Siddhi have been finalists at ACM DEBS Grand Challenge Stream Processing competitions in 2014, 2015, 2016, 2017.
• Siddhi has been the basis of many academic research projects and has over 60 citations.

If you are a Siddhi user, we would love to hear more on how you use Siddhi? Please share your experience and feedback via the Siddhi user Google group.

Get Started!

Get started with Siddhi in a few minutes by following the Siddhi Quick Start Guide

Siddhi Development Environment

Siddhi Tooling

Siddhi provides siddhi-tooling that supports following features to develop and test stream processing applications:

• Text Query Editor with syntax highlighting and advanced auto completion support.

• Event Simulator and Debugger to test Siddhi Applications.

• Graphical Query Editor with drag and drop query building support.

IntelliJ IDEA Plugin

Install IDEA plugin to get the following features:

• Siddhi Query Editor with syntax highlighting and with basic auto completion
• Siddhi Runner and Debugger support to test Siddhi Application

Siddhi Versions

• Active development version of Siddhi : v5.0.0 built on Java 8 & 11.

  Find the released Siddhi libraries here.

  Siddhi Query Guide for Siddhi v5.x.x

  Architecture of Siddhi v5.x.x

• Latest Stable Release of Siddhi v4.x.x : v4.4.8 built on Java 8. (Recommended for production use)

  Find the released Siddhi libraries here.

  Siddhi Query Guide for Siddhi v4.x.x

  Architecture of Siddhi v4.x.x

• Latest Stable Release of Siddhi v3.x.x : v3.2.3 built on Java 7.

  Find the released Siddhi libraries here.

  Siddhi Query Guide for Siddhi v3.x.x

Latest API Docs

Latest API Docs is 5.0.0.

Contact us

• Post your questions with the "Siddhi" tag in Stackoverflow.
• For questions and feedback please connect via the Siddhi user Google group.
• Engage in community development through Siddhi dev Google group.

How to Contribute

Find the detail information on asking questions, providing feedback, reporting issues, building and contributing code on How to contribute? section.

Roadmap

• Support Kafka
• Support NATS
• Siddhi Runner Distribution
• Siddhi Tooling (Editor)
• Siddhi Kubernetes CRD
• Periodic incremental state persistence
• Support Prometheus for metrics collection
• Support high available Siddhi deployment with NATS via Kubernetes CRD
• Support distributed Siddhi deployment with NATS via Kubernetes CRD

Events

Support

WSO2 provides production, and query support for Siddhi and its extensions. For more details contact via http://wso2.com/support/

Siddhi is joint research project initiated by WSO2 and University of Moratuwa, Sri Lanka.