1. Project Description
What it does
EMQ X Kuiper is an edge lightweight IoT data analytics/streaming software implemented by Golang, and it can be run at all kinds of resource-constrained edge devices. One goal of Kuiper is to migrate the cloud streaming software frameworks (such as Apache Spark, Apache Flink) to edge side. Kuiper references these cloud streaming frameworks, and also considered a special requirement of edge analytics, and introduced rule engine, which is based on Source, SQL (business logic) and Sinks, the rule engine is used for developing streaming applications at edge side.
Why it is valuable
The core server package is only about 4.5M, the initial memory footprint is about 10MB
CPU Arch：X86 AMD * 32, X86 AMD * 64; ARM * 32, ARM * 64; PPC
The popular Linux distributions, OpenWrt Linux, MacOS and Docker
Industrial PC, Raspberry Pi, industrial gateway, home gateway, MEC edge cloud server
Data analysis support
Support data extract, transform and filter through SQL
Data order, group, aggregation, and join
60+ functions include mathematical, string, aggregate and hash, etc
4-time windows & count window
Plugin system is provided, and it supports to extend at Source, SQL functions and Sink.
Source: embedded support for MQTT, and provide extension points for sources
Sink: embedded support for MQTT and HTTP, and provide extension points for sinks
UDF functions: embedded support for 60+ functions, and provide extension points for SQL functions
A web-based management dashboard (opens new window)for nodes, plugins, streams & rules management
Plugins, streams and rules management through CLI & REST API
Integration with EMQ X Edge
Seamless integration with EMQ X Neuron & EMQ X Edge, and provided an end-to-end solution from messaging to analytics.
Origin and history
The project is originated from a fully-functional streaming software created and donated by EMQ. It is developed with feedback and contributions from partners, customers and community users. Below is the milestones of the project.
22, Oct 2020: 1.0.0 - The 1st stable version.
21, Feb 2020: 0.3.2 - The 1st version integrated with the LF EdgeX Foundry project.
23, Oct 2019: 0.2.0 - The 1st released version.
2. Statement of mission
The mission of Kuiper is aligned with LF Edge:
Build Kuiper as a lightweight IoT data analytics / streaming software in edge side
Enable organizations to accelerate the adoption of edge computing by providing handy tools and frameworks
Integrate with relevant edge projects by providing built-in support
Promote edge computing ecosystem by providing mechanisms for extension and integration
3. Project synergy
EdgeX Foundry rules engine, it's reference implementation edge analytics service that performs if-then conditional actuation at the edge based on sensor data collected by the EdgeX instance. Kuiper has partnered with EdgeX Foundry to provide the default rule engine service.
Baetyl project integration. Baetyl project also integrated Kuiper, which can help users to perform data analytics within Baetyle framework.
Fledge: Kuiper has a flexible and powerful data analytic feature that compliments Fledge. One possible way to harmonize both products is that Fledge collects data from all kinds of sensors and then deliver to Kuiper for analysis.
4. Leadership team and decision making process
The Kuiper core runtime architecture consist a pipeline of sources (built-in or extended), SQL layer and sinks (built-in or extended) as shown below.
The SQL layer components include:
SQL/Rule Parser: Parse commands to orchestrate the pipeline of processors
SQL Processors: The working operators to do the SQL interpreted data transformation
Streaming/SQL runtime: The runtime to deal with streaming like window, event time/watermark and effective once semantic.
Storage: Components to persist rule state and system data.
There are other supportive components include:
Plugin management: Manage plugin creation, loading and deletion for source, function and sink.
Configuration management: Manage system configuration.
Monitoring: provide metrics for rules.
REST API and CLI: expose management API
6. Project Template
Name of Project
EMQ X Kuiper
Project Description (what it does, why it is valuable, origin and history)
EMQ X Kuiper is an edge lightweight IoT data analytics / streaming software implemented by Golang, and it can be run at all kinds of resource constrained edge devices. Considered special requirement of edge analytics, and introduced rule engine, which is based on Source, SQL (business logic) and Sink, rule engine is used for developing streaming applications at edge side. Kuiper is created by EMQ X and open sourced since July 2019. It is then developed with feedback and contributions from partners, customers and community users.
Statement on alignment with Foundation Mission Statement
The mission of Kuiper is aligned with LF Edge:
High level assessment of project synergy with existing projects under LF Edge, including how the project compliments/overlaps with existing projects, and potential ways to harmonize over time. Responses may be included both here and/or in accompanying documentation.
Kuiper is a neutral edge streaming analytics framework, and it can be easily integrated with IoT edge frameworks that require streaming analytics.
1. EdgeX Foundry rules engine, it's reference implementation edge analytics service that performs if-then conditional actuation at the edge based on sensor data collected by the EdgeX instance. Kuiper has partnered with EdgeX Foundry to provide the default rule engine service.
2. Baetyl project integration. Baetyl project also integrated Kuiper, which can help users to perform data analytics within Baetyle framework.
3. Fledge: Kuiper has a flexible and powerful data analytic feature which compliments Fledge. One possible way to harmonize both products is that Fledge collects data from all kinds of sensors and then deliver to Kuiper for analysis.
Link to current Code of Conduct
2 TAC Sponsors, if identified (Sponsors help mentor projects) - See full definition on Project Stages: Definitions and Expectations
Leding Li, Jianguo Zeng
Source control (GitHub by default)
Issue tracker (GitHub by default)
External dependencies (including licenses)
https://github.com/PaesslerAG/gval (BSD 3-Clause "New" or "Revised" License)
https://github.com/PaesslerAG/jsonpath (BSD 3-Clause "New" or "Revised" License)
https://github.com/benbjohnson/clock (MIT License)
Release methodology and mechanics
Use the Github actions to automatically building new versions.
Names of initial committers, if different from those submitting proposal
Current number of code contributors to proposed project
Current number of organizations contributing to proposed project
Briefly describe the project's leadership team and decision-making process
The Technical Steering Committee (TSC) is a committee composed of technical leaders from the open source project responsible for oversight of the technical codebase, the technical community and release process.
TSC team will discuss any proposal, and make decision based on the discussion.
List of project's official communication channels (slack, irc, mailing lists)
slack kuiper channel
Link to project's website
Links to social media accounts
Existing financial sponsorship
EMQ Technologies Co., Ltd.
Infrastructure needs or requests (to include GitHub/Gerrit, CI/CD, Jenkins, Nexus, JIRA, other ...)
Currently Supported Architecture
Planned Architecture Support
Project logo in svg format (see see https://github.com/lf-edge/lfedge-landscape#logos - Connect to preview for guidelines)
Kuiper trademark is applied in P.R China.
Does the project have a Core Infrastructure Initiative security best practices badge? (See: https://bestpractices.coreinfrastructure.org)
Any additional information the TAC and Board should take into consideration when reviewing your proposal?