Status

• Current Project Stage: Pending submission to the LF Edge TAC (TAC@lists.lfedge.org) for Stage 1 - At Large consideration (as of March 29, 2021)

1. Project Description

What it does

EMQ X Kuiper is an edge lightweight IoT data analytics/streaming software implemented by Golang, and it can be run at all kinds of resource-constrained edge devices. One goal of Kuiper is to migrate the cloud streaming software frameworks (such as Apache Spark, Apache Flink) to edge side. Kuiper references these cloud streaming frameworks, and also considered a special requirement of edge analytics, and introduced rule engine, which is based on Source, SQL (business logic) and Sinks, the rule engine is used for developing streaming applications at edge side.

Why it is valuable

• Lightweight

  • The core server package is only about 4.5M, the initial memory footprint is about 10MB

• Cross-platform

  • CPU Arch：X86 AMD * 32, X86 AMD * 64; ARM * 32, ARM * 64; PPC

  • The popular Linux distributions, OpenWrt Linux, MacOS and Docker

  • Industrial PC, Raspberry Pi, industrial gateway, home gateway, MEC edge cloud server

• Data analysis support

  • Support data extract, transform and filter through SQL

  • Data order, group, aggregation, and join

  • 60+ functions include mathematical, string, aggregate and hash, etc

  • 4-time windows & count window

• Highly extensible

  Plugin system is provided, and it supports to extend at Source, SQL functions and Sink.

  • Source: embedded support for MQTT, and provide extension points for sources

  • Sink: embedded support for MQTT and HTTP, and provide extension points for sinks

  • UDF functions: embedded support for 60+ functions, and provide extension points for SQL functions

• Management

  • A web-based management dashboard (opens new window)for nodes, plugins, streams & rules management

  • Plugins, streams and rules management through CLI & REST API

  • Easily be integrate with KubeEdge (opens new window), K3s (opens new window) and Baetyl (opens new window), which bases Kubernetes

• Integration with EMQ X Edge

  Seamless integration with EMQ X Neuron & EMQ X Edge, and provided an end-to-end solution from messaging to analytics.

Origin and history

The project is originated from a fully-functional streaming software created and donated by EMQ. It is developed with feedback and contributions from partners, customers and community users. Below is the milestones of the project.

• 22, Oct 2020: 1.0.0 - The 1st stable version.

• 21, Feb 2020: 0.3.2 - The 1st version integrated with the LF EdgeX Foundry project.

• 23, Oct 2019: 0.2.0 - The 1st released version.

2. Statement of mission

The mission of Kuiper is aligned with LF Edge:

• Build Kuiper as a lightweight IoT data analytics / streaming software in edge side

• Enable organizations to accelerate the adoption of edge computing by providing handy tools and frameworks

• Integrate with relevant edge projects by providing built-in support

• Promote edge computing ecosystem by providing mechanisms for extension and integration

3. Project synergy

1. EdgeX Foundry rules engine, it's reference implementation edge analytics service that performs if-then conditional actuation at the edge based on sensor data collected by the EdgeX instance. Kuiper has partnered with EdgeX Foundry to provide the default rule engine service.

2. Baetyl project integration. Baetyl project also integrated Kuiper, which can help users to perform data analytics within Baetyle framework.

3. Fledge: Kuiper has a flexible and powerful data analytic feature that compliments Fledge. One possible way to harmonize both products is that Fledge collects data from all kinds of sensors and then deliver to Kuiper for analysis.

4. Leadership team and decision making process

5. Architecture

The Kuiper core runtime architecture consist a pipeline of sources (built-in or extended), SQL layer and sinks (built-in or extended) as shown below.

The SQL layer components include:

• SQL/Rule Parser: Parse commands to orchestrate the pipeline of processors

• SQL Processors: The working operators to do the SQL interpreted data transformation

• Streaming/SQL runtime: The runtime to deal with streaming like window, event time/watermark and effective once semantic.

• Storage: Components to persist rule state and system data.

There are other supportive components include:

• Plugin management: Manage plugin creation, loading and deletion for source, function and sink.

• Configuration management: Manage system configuration.

• Monitoring: provide metrics for rules.

• REST API and CLI: expose management API

6. Project Template

Mapping Criteria and Data:

Stage 1: At Large Projects (formerly 'Sandbox')

Taxonomy Data:

Functions (Provide, Consume, Facilitate, or N/A; Add context as needed)

Deployment & Industry Verticals (Support, Possible, N/A; Add context as needed)

Deployment & Industry Verticals (✔ or X; Add context as needed)

Deployments (static v dynamic, connectivity, physical placement) - (✔ or X; Add context as needed)

Compute Stack Layers (architecture classification) - (Provide, Require, or N/A; Add context as needed)

Cloud Stack Layers (architecture classification) - (Provide, Require, or N/A; Add context as needed)

成为首赞

无标签