Stackable Operator for Apache NiFi
This Operator manages Apache NiFi clusters on Kubernetes. Apache NiFi is an open-source data integration tool that provides a web-based interface for designing, monitoring and managing data flows between various systems and devices, using a visual programming approach. It supports a wide range of data sources, formats and features such as data provenance, security and clustering.
Getting started
Get started with Apache NiFi and the Stackable Operator by following the Getting started guide. It will guide you through the installation process and connect to the NiFi web interface. Afterwards have a look at the Usage guide to learn how to configure your NiFi instance to your needs or run some demos to learn more about using NiFi with other components.
Operator Model
The Operator manages the NifiCluster custom resource. NiFi only has a single process that it needs to run, so the
NifiCluster has only a single role: node. This role can be divided in
multiple role groups.
For every role group the Operator creates a ConfigMap and StatefulSet which can have multiple replicas (Pods). Every role group is accessible through it’s own Service, and there is a Service for the whole Cluster.
Dependencies
Apache NiFi depends on Apache ZooKeeper which you can run in Kubernetes with the Stackable Operator for Apache ZooKeeper.
Demos
NiFi is often a good choice as a first step in a data pipeline when it comes to fetching the data in various formats from various sources. The data-lakehouse-iceberg-trino-spark demo uses NiFi to fetch six different datasets in various formats. The data is then ingested into a Kafka topic. Apache Kafka is also part of the Stackable platform.
The nifi-kafka-druid-earthquake-data and nifi-kafka-druid-water-level-data demo use NiFi in the same way, both demos showcase downloading data from web APIs and ingesting it into Kafka.