Concepts

• 1: Resource
• 2: Labels and annotations
• 3: Reconciliation
• 4: Selectors
• 5: Transformations
• 6: Validations
• 7: Template
• 8: Collectors
• 9: Controllers
• 10: Extensions
• 11: Reporters
• 12: Actions

1 - Resource

Resource Objects

All resources can be distinguished between persistent objects, which are used to describe the desired state of your system, and transient objects, which are only used to enrich or provide additional capabilities for the definition of persistent objects.

A resource is an object with a type (called a Kind) and a concrete model that describe the associated data. All resource are scoped by an API Group and Version.

Resource Definition

Resources are described in YAML format.

Here is a sample resource that described a Kafka Topic.

apiVersion: "kafka.jikkou.io/v1beta2"
kind: KafkaTopic
metadata:
  name: 'my-topic'
  labels:
    environment: test
  annotations: {}
spec:
  partitions: 1
  replicas: 1
  configs:
    min.insync.replicas: 1
    cleanup.policy: 'delete'

Resource Properties

The following are the properties that can be set to describe a resource:

2 - Labels and annotations

Labels

You can use labels to attach arbitrary identifying metadata to objects.

Labels are key/value maps:

metadata:
  labels:
    "key1": "value-1"
    "key2": "value-2"

Example

metadata:
  labels:
    environment: "stating"

Annotations

You can use annotations to attach arbitrary non-identifying metadata to objects.

Annotations are key/value maps:

metadata:
  annotations:
    key1: "value-1"
    key2: "value-2"

Built-in Annotations

jikkou.io/ignore

Used on: All Objects.

This annotation indicates whether the object should be ignored for reconciliation.

jikkou.io/bypass-validations

Used on: All Objects.

This annotation indicates whether the object should bypass the validation chain. In other words, no validations will be applied on the object.

jikkou.io/delete

Used on: All Objects.

This annotation indicates (when set to true) that the object should be deleted from your system.

jikkou.io/resource-location

Used by jikkou.

This annotation is automatically added by Jikkou to an object when loaded from your local filesystem.

jikkou.io/items-count

Used by jikkou.

This annotation is automatically added by Jikkou to an object collection grouping several resources of homogeneous type.

3 - Reconciliation

Changes

A Change represents a difference, detected during reconciliation, between two objects that can reconciled or corrected by adding, updating, or deleting an object or property attached to the actual state of the system.

A Change represents a detected difference between two objects during the reconciliation process. These differences can be reconciled or corrected by adding, updating, or deleting an object or property associated with the actual state of the system

• Jikkou identifies four types of changes:

• ADD: Indicates the addition of a new object or property to an existing object.

• UPDATE: Indicates modifications made to an existing object or property of an existing object.

• DELETE: Indicates the removal of an existing object or property of an existing object.

• NONE: Indicates that no changes were made to an existing object or property.

Reconciliation Modes

Depending on the chosen reconciliation mode, only specific types of changes will be applied.

Jikkou provides four distinct reconciliation modes that determine the types of changes to be applied:

• CREATE: This mode only applies changes that create new resource objects in your system.
• DELETE: This mode only applies changes that delete existing resource objects in your system.
• UPDATE: This mode only applies changes that create or update existing resource objects in your system.
• APPLY_ALL: This mode applies all changes to ensure that the actual state of a resource in the cluster matches the desired state defined in your resource definition file, regardless of the specific type of change.

Each mode corresponds to a command offered by the Jikkou CLI (i.e., create, update, delete, and apply). Choose the appropriate mode based on your requirements.

Reconciliation Options

Depending on the type of resources being reconciled, the controller that will be involved in the reconciliation process might accept some options (i.e., using --options argument).

Mark Resource for Deletion

To delete all the states associated with resource’s entities, you must add the following annotation to the resource definition:

metadata:
  annotations:
    jikkou.io/delete: true

4 - Selectors

Selector Expressions

Selectors are passed as arguments to Jikkou as expression strings in the following form:

• <SELECTOR>: <KEY> <OPERATOR> VALUE
• <SELECTOR>: <KEY> <OPERATOR> (VALUE[, VALUES])

or (using default field selector):

• <KEY> <OPERATOR> VALUE
• <KEY> <OPERATOR> (VALUE[, VALUES])

Selectors

Field (default)

Jikkou packs with a built-in FieldSelector allowing to filter resource objects based on a field key.

For example, the expression below shows you how to select only resource having a label environement equals to either staging or production.

metadata.labels.environement IN (staging, production)

Note: In the above example, we have omitted the selector because field is the default selector.

Expression Operators

Five kinds of operators are supported:

• IN
• NOTIN
• EXISTS
• MATCHES
• DOESNOTMATCH

Matching Strategies

Jikkou allows you to use multiple selector expressions. To indicate how these expressions are to be combined, you can pass one of the following matching strategies:

• ALL: A resource is selected if it matches all selectors.
• ANY: A resource is selected if it matches one of the selectors.
• NONE: A resource is selected if it matches none of the selectors.

Example:

jikkou get kafkatopics \
--selector 'metadata.name MATCHES (^__.*)' \
--selector 'metadata.name IN (_schemas)' \
--selector-match ANY

5 - Transformations

Available Transformations

You can list all the available transformations using the Jikkou CLI command:

jikkou extensions list --type=Transformation [-kinds <a resource kind to filter returned results>]

Transformation chain

When using Jikkou CLI, you can configure a transformation chain that will be applied to every resource. This chain consists of multiple transformations, each designed to handle different types of resources. Jikkou ensures that a transformation is executed only for the resource types it supports. In cases where a resource is not accepted by a transformation, it is passed to the next transformation in the chain. This process continues until a suitable transformation is found or until all transformations have been attempted.

Configuration

jikkou {
  # The list of transformations to execute
  transformations: [
    {
      # Simple or fully qualified class name of the transformation extension.
      type = ""
      # Priority to be used for executing this transformation extension.
      # The lowest value has the highest priority, so it's run first. Minimum value is -2^31 (highest) and a maximum value is 2^31-1 (lowest).
      # Usually, values under 0 should be reserved for internal transformation extensions.
      priority = 0
      config = {
        # Configuration properties for this transformation
      }
    }
  ]
}

Example

jikkou {
  # The list of transformations to execute
  transformations: [
    {
      # Enforce a minimum number of replicas for a kafka topic
      type = KafkaTopicMinReplicasTransformation
      priority = 100
      config = {
        minReplicationFactor = 4
      }
    },
    {
      # Enforce a {@code min.insync.replicas} for a kafka topic.
      type = KafkaTopicMinInSyncReplicasTransformation
      priority = 100
      config = {
        minInSyncReplicas = 2
      }
    }
  ]
}

6 - Validations

Available Validations

You can list all the available validations using the Jikkou CLI command:

jikkou api-extensions list --category=validation [--kinds <a resource kind to filter returned results>]

Validation chain

When using Jikkou CLI, you can configure a validation chain that will be applied to every resource. This chain consists of multiple validations, each designed to handle different types of resources. Jikkou ensures that a validation is executed only for the resource types it supports. In cases where a resource is not accepted by a validation, it is passed to the next validation in the chain. This process continues until a suitable validation is found or until all validations have been attempted.

Configuration

jikkou {
  # The list of validations to execute
  validations: [
    {
      # Custom name for the validation rule
      name = ""
      # Simple or fully qualified class name of the validation extension.
      type = ""
      config = {
        # Configuration properties for this validation
      }
    }
  ]
}

Example

jikkou {
  # The list of transformations to execute
  validations: [
    {
      # Custom name for the validation rule
      name = topicMustBePrefixedWithRegion
      # Simple or fully qualified class name of the validation extension.
      type = TopicNameRegexValidation
      # The config values that will be passed to the validation.
      config = {
        topicNameRegex = "(europe|northamerica|asiapacific)-.+"
      }
    }
  ]
}

7 - Template

Template Engine

Jikkou provides a simple templating mechanism based-on Jinjava, a Jinja template engine for Java.

Read the official documentation of Jinja to learn more about the syntax and semantics of the template engine.

How Does It Work ?

Jikkou performs the rendering of your template in two phases:

1. First, an initial rendering is performed using only the values and labels passed through the command-lines arguments.
   • Thus, it is perfectly OK if your resource file is not initially a valid YAML file.
2. Then, a second and final rendering is performed after parsing the YAML resource file using the additional values and labels as defined into the YAML resource file.
   • Therefore, it’s important that your resource file is converted into a valid YAML file after the first rendering.

Variables

Jikkou defines a number of top-level variables that are passed to the template engine.

• values:

  • The values passed into the template through the command-line --values-files and/or --set-value arguments
  • In addition, values can be defined into the application.conf file and directly into the template file using the property template.values.
  • By default, values is empty.

• labels:

  • The labels passed into the template through the command-line argument: --set-label.
  • In addition, labels can be defined into the template file using the property metadata.labels.
  • By default, labels is empty.

• system.env:

  • This provides access to all environment variables.

• system.props:

  • This provides access to all system properties.

Template Values

When using templating, a resource definition file may contain the additional property template. fields:

apiVersion: The api version (required)
kind: The resource kind (required)
metadata:
  labels: The set of key/value pairs that you can use to describe your resource file (optional)
  annotations: The set of key/value pairs automatically generated by the tool (optional)
template:
  values: The set of key/value pairs to be passed to the template engine (optional)
spec: Specification of the resource

Values Data File

Values Data File are used to define all the necessary values (i.e., the variables) to be used for generating a template.

Example

# file: ./values.yaml
topicConfigs:
  partitions: 4
  replicas: 3
topicPrefix: "{{ system.env.TOPIC_PREFIX | default('test', true) }}"
countryCodes:
  - fr
  - be
  - de
  - es
  - uk
  - us

Template Resource File

Example

# file: ./kafka-topics.tpl
apiVersion: 'kafka.jikkou.io/v1beta2'
kind: 'KafkaTopicList'
items:
  { % for country in values.countryCodes % }
  - metadata:
      name: "{{ values.topicPrefix}}-iot-events-{{ country }}"
    spec:
      partitions: { { values.topicConfigs.partitions } }
      replicas: { { values.topicConfigs.replicas } }
      configMapRefs:
        - TopicConfig
    { % endfor % }
---
apiVersion: "core.jikkou.io/v1beta2"
kind: "ConfigMap"
metadata:
  name: TopicConfig
template:
  values:
    default_min_insync_replicas: "{{ values.topicConfigs.replicas | default(3, true) | int | add(-1) }}"
data:
  retention.ms: 3600000
  max.message.bytes: 20971520
  min.insync.replicas: '{% raw %}{{ values.default_min_insync_replicas }}{% endraw %}'

Command

$  TOPIC_PREFIX=local jikkou validate --files topics.tpl --values-files values.yaml 

(Output)

---
apiVersion: "kafka.jikkou.io/v1beta2"
kind: "KafkaTopicList"
metadata:
  labels: { }
  annotations:
    jikkou.io/resource-location: "file:///tmp/jikkou/topics.tpl"
spec:
  topics:
    - metadata:
        name: "local-iot-events-fr"
      spec:
        partitions: 4
        replicas: 3
        configs:
          min.insync.replicas: "2"
          retention.ms: 3600000
          max.message.bytes: 20971520
    - metadata:
        name: "local-iot-events-be"
      spec:
        partitions: 4
        replicas: 3
        configs:
          min.insync.replicas: "2"
          retention.ms: 3600000
          max.message.bytes: 20971520
    - metadata:
        name: "local-iot-events-de"
      spec:
        partitions: 4
        replicas: 3
        configs:
          min.insync.replicas: "2"
          max.message.bytes: 20971520
          retention.ms: 3600000
    - metadata:
        name: "local-iot-events-es"
      spec:
        partitions: 4
        replicas: 3
        configs:
          min.insync.replicas: "2"
          max.message.bytes: 20971520
          retention.ms: 3600000
    - metadata:
        name: "local-iot-events-uk"
      spec:
        partitions: 4
        replicas: 3
        configs:
          min.insync.replicas: "2"
          max.message.bytes: 20971520
          retention.ms: 3600000
    - metadata:
        name: "local-iot-events-us"
      spec:
        partitions: 4
        replicas: 3
        configs:
          min.insync.replicas: "2"
          max.message.bytes: 20971520
          retention.ms: 3600000

Configuration

jinja {
  # Enable/Disable recursive macro calls for rendering
  enableRecursiveMacroCalls = false
}

8 - Collectors

Available Collectors

You can list all the available collectors using the Jikkou CLI command:

jikkou extensions list --type=Collector [-kinds <a resource kind to filter returned results>]

9 - Controllers

Available Controllers

You can list all the available controllers using the Jikkou CLI command:

jikkou extensions list --type=Controller [-kinds <a resource kind to filter returned results>]

10 - Extensions

Extension Providers

Most of the Jikkou’s features are provided by Jikkou Extension Providers. A provider is a module providing a set of extensions used to manage one or more resources.

Built-in Extension Providers

Jikkou ships with a number of extension providers:

• Aiven
• Apache Kafka
• Core
• Kafka Connect
• Schema Registry

11 - Reporters

Configuration

Jikkou allows you to configure multiple reporters as follows:

jikkou {
  # The list of reporters to execute
  reporters: [
    {
      # Custom name for the reporter
      name = ""
      # Simple or fully qualified class name of the transformation extension.
      type = ""
      config = {
        # Configuration properties for this reporter
      }
    }
  ]
}

Built-in implementations

Jikkou packs with some built-in ChangeReporter implementations:

KafkaChangeReporter

The KafkaChangeReporter can be used to send change results into a given kafka topic. Changes will be published as Cloud Events.

Configuration

The below example shows how to configure the KafkaChangeReporter.

jikkou {
  # The default custom reporters to report applied changes.
  reporters = [
    {
      name = "kafka-reporter"
      type = io.streamthoughts.jikkou.kafka.reporter.KafkaChangeReporter
      config = {
        # The 'source' of the event that will be generated.
        event.source = "jikkou/cli"
        kafka = {
          # If 'true', topic will be automatically created if it does not already exist.
          topic.creation.enabled = true
          # The default replication factor used for creating topic.
          topic.creation.defaultReplicationFactor = 1
          # The name of the topic the events will be sent.
          topic.name = "jikkou-resource-change-event"
          # The configuration settings for Kafka Producer and AdminClient
          client = ${jikkou.kafka.client} {
            client.id = "jikkou-reporter-producer"
          }
        }
      }
    }
  ]
}

12 - Actions

Available Actions (CLI)

You can list all the available actions using the Jikkou CLI command:

jikkou api-extensions list --category=action [-kinds <a resource kind to filter returned results>]

Execution Actions (CLI)

You can execute a specific extension using the Jikkou CLI command:

jikkou action <ACTION_NAME> execute [<options>]