This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Reference

Technical reference for the Jikkou CLI, the API Server, and every Extension Provider.

Look here for precise, descriptive information: command flags, configuration keys, resource specifications, and provider options. Reference is for looking things up — if you want to learn or accomplish a task, see the Tutorials and How-to Guides.

1 - Jikkou CLI

Complete reference for the Jikkou command-line interface.

Hands-on: Try the Jikkou Get Started tutorials.

The Jikkou CLI (jikkou) is the primary interface for managing infrastructure resources. This reference covers:

  • CLI Features — Global options, version check, and shell tab-completion
  • CLI Configuration — Contexts, configuration files, and HOCON settings
  • Commands — Complete reference for every command

To automate Jikkou in CI/CD, see the Automating guide.

1.1 - Overview

Global options, version information, and shell tab-completion.

The command line interface to Jikkou is the jikkou command, which accepts a variety of subcommands such as jikkou apply or jikkou validate.

To view a list of the commands available in your current Jikkou version, run jikkou with no additional arguments:

Usage: 
jikkou [-hV] [--logger-level=<level>] [COMMAND]


Jikkou CLI:: A command-line client designed to provide an efficient and easy way to manage, automate, and provision resources.

Find more information at: https://www.jikkou.io/.

OPTIONS:

  -h, --help      Show this help message and exit.
      --logger-level=<level>
                  Specify the log level verbosity to be used while running a command.
                  Valid level values are: TRACE, DEBUG, INFO, WARN, ERROR.
                  For example, `--logger-level=INFO`
  -V, --version   Print version information and exit.

CORE COMMANDS:
  apply                     Update the resources as described by the resource definition files.
  create                    Create resources from the resource definition files (only non-existing resources will be created).
  delete                    Delete resources that are no longer described by the resource definition files.
  diff                      Show resource changes required by the current resource definitions.
  get                       Display one or many specific resources.
  patch                     Execute all changes for the specified reconciliation mode.
  prepare                   Prepare the resource definition files for validation.
  replace                   Replace all resources.
  update                    Create or update resources from the resource definition files
  validate                  Check whether the resources definitions meet all validation requirements.

SYSTEM MANAGEMENT COMMANDS:
  action                    List/execute actions.
  health                    Print or describe health indicators.

ADDITIONAL COMMANDS:
  api-extensions            Print the supported API extensions
  api-resources             Print the supported API resources
  config                    Sets or retrieves the configuration of this client
  generate-completion       Generate bash/zsh completion script for jikkou.
  help                      Display help information about the specified command.

See 'jikkou --help' for more information about a command.

(The output from your current Jikkou version may be different than the above example.)

For detailed options and usage examples for each command, see the Commands Reference.

Checking Jikkou Version

Run the jikkou --version to display your current installation version:

Jikkou version "0.37.0" 2026-02-17
JVM: 25.0.1 (GraalVM Community Substrate VM 25.0.1+8)

Shell Tab-completion

It is recommended to install the bash/zsh completion script jikkou_completion.

The completion script can be downloaded from the project Github repository:

wget https://raw.githubusercontent.com/streamthoughts/jikkou/main/jikkou_completion -O jikkou_completion

or alternatively, you can run the following command to generate it.

source <(jikkou generate-completion)

1.2 - Configuration

Contexts, configuration files, and HOCON settings.

Configuration

To set up the configuration settings used by Jikkou CLI, you will need create a jikkou config file, which is created automatically when you create a configuration context using:

jikkou config set-context <context-name> [--config-file=<config-file>] [--config-props=<config-value>]

By default, the configuration of jikkou is located under the path $HOME/.jikkou/config.

This jikkou config file defines all the contexts that can be used by jikkou CLI.

For example, below is the config file created during the Getting Started.

{
  "currentContext": "localhost",
  "localhost": {
    "configFile": null,
    "configProps": {
      "provider.kafka.client.bootstrap.servers": "localhost:9092"
    }
  }
}

Most of the time, a context does not directly contain the configuration properties to be used, but rather points to a specific HOCON (Human-Optimized Config Object Notation) through the configFile property.

Then, the configProps allows you to override some of the property define by this file.

In addition, if no configuration file path is specified, Jikkou will lookup for an application.conf to those following locations:

  • ./application.conf
  • $HOME/.jikkou/application.conf

Finally, Jikkou always fallback to a reference.conf file that you can use as a template to define your own configuration.

reference.conf:

jikkou {

  # Configure Jikkou Proxy Mode
  # proxy {
  #  url = "http://localhost:8080"
  # }

  # Configure Extension Providers
  extension {
  }

  # Core
  provider.core {
    enabled = true
  }
  # Apache Kafka Provider
  provider.kafka {
    enabled = true
    type = io.jikkou.kafka.KafkaExtensionProvider
    config = {
      # The default Kafka Client configuration
      client {
        bootstrap.servers = "localhost:9092"
        bootstrap.servers = ${?JIKKOU_DEFAULT_KAFKA_BOOTSTRAP_SERVERS}
      }

      brokers {
        # If 'True'
        waitForEnabled = true
        waitForEnabled = ${?JIKKOU_KAFKA_BROKERS_WAIT_FOR_ENABLED}
        # The minimal number of brokers that should be alive for the CLI stops waiting.
        waitForMinAvailable = 1
        waitForMinAvailable = ${?JIKKOU_KAFKA_BROKERS_WAIT_FOR_MIN_AVAILABLE}
        # The amount of time to wait before verifying that brokers are available.
        waitForRetryBackoffMs = 1000
        waitForRetryBackoffMs = ${?JIKKOU_KAFKA_BROKERS_WAIT_FOR_RETRY_BACKOFF_MS}
        # Wait until brokers are available or this timeout is reached.
        waitForTimeoutMs = 60000
        waitForTimeoutMs = ${?JIKKOU_KAFKA_BROKERS_WAIT_FOR_TIMEOUT_MS}
      }
    }
  }
  # Schema Registry Provider
  provider.schemaregistry {
    enabled = true
    type = io.jikkou.schema.registry.SchemaRegistryExtensionProvider
    config = {
      url = "http://localhost:8081"
      url = ${?JIKKOU_DEFAULT_SCHEMA_REGISTRY_URL}
    }
  }

  # The default custom transformations to apply on any resources.
  transformations = []

  # The default custom validations to apply on any resources.
  validations = [
    {
      name = "topicMustHaveValidName"
      type = io.jikkou.kafka.validation.TopicNameRegexValidation
      priority = 100
      config = {
        topicNameRegex = "[a-zA-Z0-9\\._\\-]+"
        topicNameRegex = ${?VALIDATION_DEFAULT_TOPIC_NAME_REGEX}
      }
    },
    {
      name = "topicMustHavePartitionsEqualsOrGreaterThanOne"
      type = io.jikkou.kafka.validation.TopicMinNumPartitionsValidation
      priority = 100
      config = {
        topicMinNumPartitions = 1
        topicMinNumPartitions = ${?VALIDATION_DEFAULT_TOPIC_MIN_NUM_PARTITIONS}
      }
    },
    {
      name = "topicMustHaveReplicasEqualsOrGreaterThanOne"
      type = io.jikkou.kafka.validation.TopicMinReplicationFactorValidation
      priority = 100
      config = {
        topicMinReplicationFactor = 1
        topicMinReplicationFactor = ${?VALIDATION_DEFAULT_TOPIC_MIN_REPLICATION_FACTOR}
      }
    }
  ]
  # The default custom reporters to report applied changes.
  reporters = [
    # Uncomment following lines to enable default kafka reporter
    #    {
    #     name = "default"
    #      type = io.jikkou.kafka.reporter.KafkaChangeReporter
    #      config = {
    #        event.source = "jikkou/cli"
    #        kafka = {
    #          topic.creation.enabled = true
    #          topic.creation.defaultReplicationFactor = 1
    #          topic.name = "jikkou-resource-change-event"
    #          client = ${jikkou.kafka.client} {
    #            client.id = "jikkou-reporter-producer"
    #          }
    #        }
    #      }
    #    }
  ]

  jinja {
    enableRecursiveMacroCalls = false
  }
}

Managing Contexts

For detailed usage of all configuration commands, see the Commands Reference:

1.3 - Commands

Comprehensive reference for all Jikkou CLI commands.

This section contains the complete reference for every Jikkou CLI command, including synopsis, options, usage examples, and related commands.

You can also run jikkou <command> --help from the terminal to view built-in help for any command.

Global Options

The following options are available on all commands:

FlagDescription
--logger-level=<level>Specify the log level verbosity. Valid values: TRACE, DEBUG, INFO, WARN, ERROR
-h, --helpShow help message and exit
-V, --versionPrint version information and exit

Core Commands

Commands for reconciling and inspecting resources against your target platform.

Reconciliation

These commands apply changes to your target platform based on resource definition files. Each command corresponds to a specific reconciliation mode.

CommandModeDescription
jikkou applyFULLCreate, update, and delete resources to match the desired state
jikkou createCREATECreate only non-existing resources
jikkou updateUPDATECreate new and update existing resources (no deletions)
jikkou deleteDELETEDelete resources no longer described in definition files
jikkou patch(explicit)Reconcile with an explicitly specified mode
jikkou replaceDelete and recreate all resources from scratch

Inspect & Validate

CommandDescription
jikkou diffPreview resource changes without applying them
jikkou getDisplay one or many resources from the target platform
jikkou validateCheck resource definitions against all validation requirements
jikkou prepareRender templates and apply transformations without validating

System Management Commands

CommandDescription
jikkou actionList and execute provider actions
jikkou healthPrint or describe health indicators for target environments
jikkou server-infoDisplay Jikkou API server information (proxy mode only)

Configuration & Discovery Commands

CommandDescription
jikkou configManage CLI configuration contexts
jikkou api-resourcesList the supported API resource types and their verbs
jikkou api-extensionsList and inspect registered API extensions

1.3.1 - jikkou action

List and execute actions.

Synopsis

The action command manages Jikkou actions. Actions are named operations that can be executed on the target platform. Like the get command, action uses dynamic subcommands based on the actions available from your configured providers.

Each action is registered as a subcommand under jikkou action, and has an execute subcommand to run it.

jikkou action <action-name> execute [flags]

Examples

# List all available actions (shown as subcommands in help)
jikkou action --help

# Execute an action
jikkou action KafkaConsumerGroupsResetOffsets execute --options topic=my-topic --options group=my-group

# Execute an action with YAML output
jikkou action KafkaConsumerGroupsResetOffsets execute -o YAML

# Execute an action targeting a specific provider
jikkou action KafkaConsumerGroupsResetOffsets execute --provider kafka-prod --options topic=my-topic

Options (execute subcommand)

FlagShortDefaultDescription
--output-oYAMLOutput format. Valid values: JSON, YAML
--providerSelect a specific provider instance

Additional options may be available depending on the action. These are dynamically registered based on the action’s ApiOptionSpec definitions. Use jikkou api-extensions get <action-name> to view options for a specific action.

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.2 - jikkou api-extensions

Print the supported API extensions.

Synopsis

The api-extensions command lists and inspects the API extensions registered in Jikkou. Extensions include resource collectors, transformations, validations, actions, health indicators, and controllers.

This command has two subcommands:

  • api-extensions list - List all extensions
  • api-extensions get - Get details about a specific extension

Subcommands

jikkou api-extensions list

Print a summary table of all supported API extensions.

jikkou api-extensions list [flags]

Examples

# List all extensions
jikkou api-extensions list

# List extensions of a specific category
jikkou api-extensions list --category Transformation

# List extensions from a specific provider
jikkou api-extensions list --provider kafka

# List extensions that support a specific resource kind
jikkou api-extensions list --kind KafkaTopic

Options

FlagDefaultDescription
--categoryLimit to extensions of the specified category
--providerLimit to extensions of the specified provider
--kindLimit to extensions that support the specified resource kind

jikkou api-extensions get

Print detailed information about a specific API extension, including its title, description, configuration options, and usage examples.

jikkou api-extensions get <name> [flags]

Examples

# Get details about an extension in WIDE format (default)
jikkou api-extensions get KafkaTopicMaxRetentionMs

# Get details in JSON format
jikkou api-extensions get KafkaTopicMaxRetentionMs -o JSON

# Get details in YAML format
jikkou api-extensions get KafkaTopicMaxRetentionMs -o YAML

Options

FlagShortDefaultDescription
--output-oWIDEOutput format. Valid values: JSON, YAML, WIDE

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.3 - jikkou api-resources

Print the supported API resources.

Synopsis

List the API resources supported by the Jikkou CLI or Jikkou API Server (in proxy mode). This command shows the resource name, short names, API version, kind, and supported verbs for each resource type.

Use this command to discover which resource types are available with your current configuration and providers.

jikkou api-resources [flags]

Examples

# List all available API resources
jikkou api-resources

# List resources for a specific API group
jikkou api-resources --api-group kafka.jikkou.io

# List resources that support specific verbs
jikkou api-resources --verbs LIST,GET

# List resources that support CREATE
jikkou api-resources --verbs CREATE

# Combine filters
jikkou api-resources --api-group kafka.jikkou.io --verbs LIST

Sample output

NAME               SHORTNAMES   APIVERSION               KIND                  VERBS
kafkatopics        kt           kafka.jikkou.io/v1beta2  KafkaTopic            CREATE, DELETE, GET, LIST, UPDATE
kafkaconsumergroups              kafka.jikkou.io/v1beta2  KafkaConsumerGroup    DELETE, GET, LIST

Options

FlagDefaultDescription
--api-groupLimit to resources in the specified API group
--verbsLimit to resources that support the specified verbs (comma-separated)

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.4 - jikkou apply

Update the resources aand as described by the resource definition files.

Synopsis

Reconciles the target platform so that the resources match the resource definition files passed as arguments. This command uses the FULL reconciliation mode, meaning it will create, update, and delete resources as needed to match the desired state defined in your resource files.

jikkou apply [flags]

Examples

# Apply resources defined in a YAML file
jikkou apply -f my-resources.yaml

# Apply resources from a directory
jikkou apply -f ./resources/

# Apply resources with dry-run to preview changes
jikkou apply -f my-resources.yaml --dry-run

# Apply resources with specific selector
jikkou apply -f my-resources.yaml --selector 'metadata.name IN (my-topic)'

# Apply resources with template values
jikkou apply -f my-resources.yaml --values-files values.yaml

# Apply with inline template variable
jikkou apply -f my-resources.yaml -v topicName=my-topic

# Apply with custom labels
jikkou apply -f my-resources.yaml -l environment=production

# Apply with controller options
jikkou apply -f my-resources.yaml --options delete-orphans=true

# Apply targeting a specific provider
jikkou apply -f my-resources.yaml --provider kafka-prod

# Apply with JSON output format
jikkou apply -f my-resources.yaml -o JSON --pretty

Options

FlagShortDefaultDescription
--files-fResource definition file or directory locations (one or more required)
--file-name-n**/*.{yaml,yml}Glob pattern to filter resource files when using directories
--values-filesTemplate values file locations (one or more)
--values-file-name**/*.{yaml,yml}Glob pattern to filter values files
--set-label-lSet labels on resources (key=value, repeatable)
--set-annotationSet annotations on resources (key=value, repeatable)
--set-value-vSet template variables for the built-in Values object (key=value, repeatable)
--selector-sSelector expression for including or excluding resources
--selector-matchALLSelector matching strategy. Valid values: ALL, ANY, NONE
--optionsController configuration options (key=value, repeatable)
--providerSelect a specific provider instance
--output-oTEXTOutput format. Valid values: TEXT, COMPACT, JSON, YAML
--prettyfalsePretty print JSON output
--dry-runfalseExecute command in dry-run mode (preview changes without applying)

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.5 - jikkou config

Sets or retrieves the configuration of this client.

Synopsis

The config command manages Jikkou CLI configuration contexts. A context defines the configuration settings (config file path, inline properties, provider bindings) used when running Jikkou commands.

Configuration is stored in the Jikkou config file, located by default at $HOME/.jikkou/config.

Subcommands

CommandDescription
jikkou config set-contextConfigure a context with the provided arguments
jikkou config get-contextsList all configured contexts
jikkou config current-contextDisplay the current context
jikkou config use-contextSwitch to a specified context
jikkou config viewShow merged configuration settings

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.6 - jikkou config current-context

Display the current context.

Synopsis

Displays the current context used by Jikkou CLI, including the configuration file path and inline configuration properties associated with it.

jikkou config current-context

Examples

# Show the current context
$ jikkou config current-context
Using context 'localhost'

 KEY          VALUE
 ConfigFile
 ConfigProps  {"provider.kafka.config.client.bootstrap.servers":"localhost:9092"}

Options

This command has no specific options.

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.7 - jikkou config get-contexts

List all configured contexts.

Synopsis

Get all contexts defined in the Jikkou config file. The current context is marked with an asterisk (*).

jikkou config get-contexts

Examples

# List all contexts
$ jikkou config get-contexts

 NAME
 localhost *
 development
 staging
 production

Options

This command has no specific options.

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.8 - jikkou config set-context

Configure a context with the provided arguments.

Synopsis

Configures the specified context with the provided arguments. A context stores a reference to a configuration file and/or inline configuration properties. If the context already exists, it will be updated.

After setting a context, use jikkou config use-context to switch to it.

jikkou config set-context <context-name> [flags]

Examples

# Create a context with inline Kafka bootstrap server
jikkou config set-context localhost \
  --config-props=provider.kafka.config.client.bootstrap.servers=localhost:9092

# Create a context pointing to a configuration file
jikkou config set-context production --config-file=/path/to/production.conf

# Create a context with a properties file
jikkou config set-context staging --config-props-file=/path/to/staging.properties

# Create a context with provider-scoped properties
jikkou config set-context dev \
  --provider kafka \
  --config-props=client.bootstrap.servers=kafka-dev:9092

# Create a context with a config prefix
jikkou config set-context dev \
  --config-prefix=provider.kafka.config \
  --config-props=client.bootstrap.servers=kafka-dev:9092

Options

FlagDefaultDescription
--config-filePath to a Jikkou configuration file
--config-propsInline configuration properties as key=value pairs (repeatable)
--config-props-filePath(s) to one or more configuration properties files (repeatable)
--providerName of the provider to which this configuration should be attached
--config-prefixPrefix to apply to all configuration property keys

Arguments

ArgumentDescription
context-name(required) The name of the context to configure

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.9 - jikkou config use-context

Switch to a specified context.

Synopsis

Configures Jikkou to use the specified context. All subsequent commands will use the configuration associated with this context.

jikkou config use-context <context-name>

Examples

# Switch to the production context
$ jikkou config use-context production
Using context 'production'

# If already using the specified context
$ jikkou config use-context production
Already using context production

Arguments

ArgumentDescription
context-name(required) The name of the context to switch to

Options

This command has no specific options.

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.10 - jikkou config view

Show merged configuration settings.

Synopsis

Show the merged Jikkou configuration settings for the current context (or a named context). The configuration is rendered in HOCON format, showing all resolved values from the config file, inline properties, and defaults.

Use --debug to see the origin of each setting, or --comments to include human-written comments from the configuration files.

jikkou config view [flags]

Examples

# View the current configuration
jikkou config view

# View configuration for a specific context
jikkou config view --name production

# View configuration with origin comments (useful for debugging)
jikkou config view --debug

# View configuration with human-written comments
jikkou config view --comments

Options

FlagDefaultDescription
--nameThe name of the context configuration to view (defaults to current context)
--debugfalsePrint configuration with the origin of each setting as comments
--commentsfalsePrint configuration with human-written comments

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.11 - jikkou create

Create resources from the resource definition files (only non-existing resources will be created).

Synopsis

Reconcile the target platform by creating all non-existing resources that are described by the resource definition files passed as arguments. This command uses the CREATE reconciliation mode, meaning only new resources will be created. Existing resources will not be updated or deleted.

jikkou create [flags]

Examples

# Create resources defined in a YAML file
jikkou create -f my-resources.yaml

# Preview what would be created
jikkou create -f my-resources.yaml --dry-run

# Create resources from a directory
jikkou create -f ./resources/

# Create resources with template values
jikkou create -f my-resources.yaml --values-files values.yaml

# Create resources targeting a specific provider
jikkou create -f my-resources.yaml --provider kafka-dev

Options

FlagShortDefaultDescription
--files-fResource definition file or directory locations (one or more required)
--file-name-n**/*.{yaml,yml}Glob pattern to filter resource files when using directories
--values-filesTemplate values file locations (one or more)
--values-file-name**/*.{yaml,yml}Glob pattern to filter values files
--set-label-lSet labels on resources (key=value, repeatable)
--set-annotationSet annotations on resources (key=value, repeatable)
--set-value-vSet template variables for the built-in Values object (key=value, repeatable)
--selector-sSelector expression for including or excluding resources
--selector-matchALLSelector matching strategy. Valid values: ALL, ANY, NONE
--optionsController configuration options (key=value, repeatable)
--providerSelect a specific provider instance
--output-oTEXTOutput format. Valid values: TEXT, COMPACT, JSON, YAML
--prettyfalsePretty print JSON output
--dry-runfalseExecute command in dry-run mode (preview changes without applying)

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.12 - jikkou delete

Delete resources that are no longer described by the resource definition files.

Synopsis

Reconcile the target platform by deleting all existing resources that are no longer described by the resource definition files passed as arguments. This command uses the DELETE reconciliation mode, meaning only deletions will be performed. No resources will be created or updated.

jikkou delete [flags]

Examples

# Delete resources no longer defined in a YAML file
jikkou delete -f my-resources.yaml

# Preview what would be deleted
jikkou delete -f my-resources.yaml --dry-run

# Delete resources from a directory
jikkou delete -f ./resources/

# Delete with selector to target specific resources
jikkou delete -f my-resources.yaml -s 'metadata.name IN (old-topic)'

# Delete resources targeting a specific provider
jikkou delete -f my-resources.yaml --provider kafka-dev

Options

FlagShortDefaultDescription
--files-fResource definition file or directory locations (one or more required)
--file-name-n**/*.{yaml,yml}Glob pattern to filter resource files when using directories
--values-filesTemplate values file locations (one or more)
--values-file-name**/*.{yaml,yml}Glob pattern to filter values files
--set-label-lSet labels on resources (key=value, repeatable)
--set-annotationSet annotations on resources (key=value, repeatable)
--set-value-vSet template variables for the built-in Values object (key=value, repeatable)
--selector-sSelector expression for including or excluding resources
--selector-matchALLSelector matching strategy. Valid values: ALL, ANY, NONE
--optionsController configuration options (key=value, repeatable)
--providerSelect a specific provider instance
--output-oTEXTOutput format. Valid values: TEXT, COMPACT, JSON, YAML
--prettyfalsePretty print JSON output
--dry-runfalseExecute command in dry-run mode (preview changes without applying)

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.13 - jikkou diff

Show resource changes required by the current resource definitions.

Synopsis

Generates a speculative reconciliation plan, showing the resource changes Jikkou would apply to reconcile the resource definitions. This command does not actually perform the reconciliation actions.

Use diff to preview what changes would be made before running apply, create, update, or delete.

jikkou diff [flags]

Examples

# Show changes required by a resource definition file
jikkou diff -f my-resources.yaml

# Show only resources that would be created
jikkou diff -f my-resources.yaml --filter-resource-op CREATE

# Show only resources that would be created or deleted
jikkou diff -f my-resources.yaml --filter-resource-op CREATE,DELETE

# Filter changes to show only update operations
jikkou diff -f my-resources.yaml --filter-change-op UPDATE

# Output as a resource list
jikkou diff -f my-resources.yaml --list

# Output in JSON format
jikkou diff -f my-resources.yaml -o JSON

# Diff with a specific provider
jikkou diff -f my-resources.yaml --provider kafka-prod

# Diff with selector
jikkou diff -f my-resources.yaml -s 'metadata.name MATCHES (my-topic-.*)'

Options

FlagShortDefaultDescription
--files-fResource definition file or directory locations (one or more required)
--file-name-n**/*.{yaml,yml}Glob pattern to filter resource files when using directories
--values-filesTemplate values file locations (one or more)
--values-file-name**/*.{yaml,yml}Glob pattern to filter values files
--set-label-lSet labels on resources (key=value, repeatable)
--set-annotationSet annotations on resources (key=value, repeatable)
--set-value-vSet template variables for the built-in Values object (key=value, repeatable)
--selector-sSelector expression for including or excluding resources
--selector-matchALLSelector matching strategy. Valid values: ALL, ANY, NONE
--optionsController configuration options (key=value, repeatable)
--providerSelect a specific provider instance
--output-oYAMLOutput format. Valid values: JSON, YAML
--filter-resource-opFilter resources by operation (comma-separated). Valid values: NONE, CREATE, DELETE, REPLACE, UPDATE
--filter-change-opFilter state changes by operation (comma-separated). Valid values: NONE, CREATE, DELETE, REPLACE, UPDATE
--listfalseOutput resources as an ApiResourceChangeList

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.14 - jikkou get

Display one or many specific resources.

Synopsis

Display one or many resources of a given type from the target platform. The get command groups resources under their provider (e.g. kafka, schemaregistry, aiven), so invocations follow the form jikkou get <provider> <kind>. Use jikkou api-providers list to discover the available providers and jikkou api-resources to discover the available resource types.

When a resource name is specified via --name, it retrieves that specific resource. Otherwise, it lists all resources of the given type.

jikkou get <provider> <resource-type> [flags]

Resource naming. Under a provider, each resource is exposed by its provider-local name when one is declared (e.g. topics under kafka, subjects under schemaregistry). The full plural name (e.g. kafkatopics, schemaregistrysubjects) and any short names (e.g. kt, sr) remain valid aliases inside the provider scope.

Backward compatibility. The flat form jikkou get <resource-type> is still supported but prints a deprecation notice on stderr and will be removed in a future release. Prefer the qualified form jikkou get <provider> <resource-type>.

Examples

# List all Kafka topics (canonical form)
jikkou get kafka topics

# Same, using the plural name as alias
jikkou get kafka kafkatopics

# Same, using the short name as alias
jikkou get kafka kt

# Get a specific Kafka topic by name
jikkou get kafka topics --name my-topic

# List resources with a selector
jikkou get kafka topics -s 'metadata.name MATCHES (my-.*)'

# List resources with JSON output
jikkou get kafka topics -o JSON

# List resources as a ResourceListObject
jikkou get kafka topics --list

# List resources with custom options
jikkou get kafka topics --options describe-default-configs=true

# List resources targeting a specific provider instance
jikkou get kafka topics --provider kafka-prod

# List schema registry subjects
jikkou get schemaregistry subjects

# Discover available resource types
jikkou api-resources --verbs LIST,GET

Options

FlagShortDefaultDescription
--selector-sSelector expression for including or excluding resources
--selector-matchALLSelector matching strategy. Valid values: ALL, ANY, NONE
--output-oYAMLOutput format. Valid values: JSON, YAML
--listfalseOutput resources as a ResourceListObject
--providerSelect a specific provider instance

Additional options may be available depending on the resource type. These are dynamically registered based on the provider’s ApiOptionSpec definitions. Use jikkou api-extensions get <extension-name> to view options for a specific resource collector.

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.15 - jikkou health

Print or describe health indicators.

Synopsis

The health command provides information about the health of target environments. It has two subcommands:

  • health get - Retrieve health status for one or all indicators
  • health get-indicators - List all available health indicators

Subcommands

jikkou health get

Get health information for a specific indicator or all indicators.

jikkou health get <indicator|all> [flags]

Examples

# Get health for all indicators
jikkou health get all

# Get health for a specific indicator
jikkou health get kafka

# Get health with a custom timeout
jikkou health get all --timeout-ms 5000

# Get health in JSON format
jikkou health get all -o JSON

# Get health targeting a specific provider
jikkou health get kafka --provider kafka-prod

Options

FlagShortDefaultDescription
--output-oYAMLOutput format. Valid values: JSON, YAML
--timeout-ms2000Timeout in milliseconds for retrieving health indicators
--providerSelect a specific provider instance

The command exits with code 0 if the health status is UP, or a non-zero code otherwise.


jikkou health get-indicators

List all available health indicators. This command takes no options and displays the name and description of each registered health indicator.

jikkou health get-indicators

Examples

# List all health indicators
jikkou health get-indicators

Options

This subcommand has no specific options.

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.16 - jikkou patch

Execute all changes for the specified reconciliation mode.

Synopsis

Reconcile resources by applying all the changes as defined in the resource descriptor files passed through the arguments. Unlike the other reconciliation commands (apply, create, update, delete), patch requires you to explicitly specify the reconciliation mode using the --mode flag.

The patch command applies changes to existing resources on the platform by computing a diff between the current and desired states, then executing only the operations allowed by the selected mode.

jikkou patch --mode <mode> [flags]

Examples

# Patch resources using FULL reconciliation mode
jikkou patch --mode FULL -f my-resources.yaml

# Patch resources - create only
jikkou patch --mode CREATE -f my-resources.yaml

# Patch resources - update only
jikkou patch --mode UPDATE -f my-resources.yaml

# Preview patches without applying
jikkou patch --mode FULL -f my-resources.yaml --dry-run

# Patch with a specific provider
jikkou patch --mode UPDATE -f my-resources.yaml --provider kafka-prod

Options

FlagShortDefaultDescription
--mode(required) The reconciliation mode. Valid values: CREATE, DELETE, UPDATE, FULL
--files-fResource definition file or directory locations (one or more required)
--file-name-n**/*.{yaml,yml}Glob pattern to filter resource files when using directories
--values-filesTemplate values file locations (one or more)
--values-file-name**/*.{yaml,yml}Glob pattern to filter values files
--set-label-lSet labels on resources (key=value, repeatable)
--set-annotationSet annotations on resources (key=value, repeatable)
--set-value-vSet template variables for the built-in Values object (key=value, repeatable)
--selector-sSelector expression for including or excluding resources
--selector-matchALLSelector matching strategy. Valid values: ALL, ANY, NONE
--optionsController configuration options (key=value, repeatable)
--providerSelect a specific provider instance
--output-oTEXTOutput format. Valid values: TEXT, COMPACT, JSON, YAML
--prettyfalsePretty print JSON output
--dry-runfalseExecute command in dry-run mode (preview changes without applying)

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.17 - jikkou prepare

Prepare the resource definition files for validation.

Synopsis

Prepare the resource definition files specified through the command line arguments for validation. This command applies all configured transformations to the resource definitions and outputs the result, without running validation rules or performing reconciliation.

Use prepare to inspect how your resources look after template rendering and transformations, before validation and reconciliation happen.

jikkou prepare [flags]

Examples

# Prepare resources from a YAML file
jikkou prepare -f my-resources.yaml

# Prepare resources from a directory
jikkou prepare -f ./resources/

# Prepare with template values
jikkou prepare -f my-resources.yaml --values-files values.yaml

# Prepare with JSON output
jikkou prepare -f my-resources.yaml -o JSON

# Prepare with selector
jikkou prepare -f my-resources.yaml -s 'metadata.name IN (my-topic)'

# Prepare targeting a specific provider
jikkou prepare -f my-resources.yaml --provider kafka-prod

Options

FlagShortDefaultDescription
--files-fResource definition file or directory locations (one or more required)
--file-name-n**/*.{yaml,yml}Glob pattern to filter resource files when using directories
--values-filesTemplate values file locations (one or more)
--values-file-name**/*.{yaml,yml}Glob pattern to filter values files
--set-label-lSet labels on resources (key=value, repeatable)
--set-annotationSet annotations on resources (key=value, repeatable)
--set-value-vSet template variables for the built-in Values object (key=value, repeatable)
--selector-sSelector expression for including or excluding resources
--selector-matchALLSelector matching strategy. Valid values: ALL, ANY, NONE
--optionsController configuration options (key=value, repeatable)
--providerSelect a specific provider instance
--output-oYAMLOutput format. Valid values: JSON, YAML

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.18 - jikkou replace

Replace all resources.

Synopsis

Replaces resources by deleting and (re)creating all the resources as defined in the resource descriptor files passed through the arguments. Unlike apply, which performs incremental updates, replace performs a full replacement of resources: existing resources are deleted first and then recreated from scratch.

Use replace when you need to ensure resources exactly match the desired state without any leftover configuration from previous versions.

jikkou replace [flags]

Examples

# Replace resources defined in a YAML file
jikkou replace -f my-resources.yaml

# Preview what would be replaced
jikkou replace -f my-resources.yaml --dry-run

# Replace resources from a directory
jikkou replace -f ./resources/

# Replace resources targeting a specific provider
jikkou replace -f my-resources.yaml --provider kafka-prod

# Replace with JSON output
jikkou replace -f my-resources.yaml -o JSON --pretty

Options

FlagShortDefaultDescription
--files-fResource definition file or directory locations (one or more required)
--file-name-n**/*.{yaml,yml}Glob pattern to filter resource files when using directories
--values-filesTemplate values file locations (one or more)
--values-file-name**/*.{yaml,yml}Glob pattern to filter values files
--set-label-lSet labels on resources (key=value, repeatable)
--set-annotationSet annotations on resources (key=value, repeatable)
--set-value-vSet template variables for the built-in Values object (key=value, repeatable)
--selector-sSelector expression for including or excluding resources
--selector-matchALLSelector matching strategy. Valid values: ALL, ANY, NONE
--optionsController configuration options (key=value, repeatable)
--providerSelect a specific provider instance
--output-oTEXTOutput format. Valid values: TEXT, COMPACT, JSON, YAML
--prettyfalsePretty print JSON output
--dry-runfalseExecute command in dry-run mode (preview changes without applying)

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.19 - jikkou server-info

Display Jikkou API server information.

Synopsis

Print information about the Jikkou API server. This command is only available when Jikkou is configured in proxy mode (i.e., when jikkou.proxy.url is set in your configuration).

jikkou server-info [flags]

Examples

# Get server information in YAML format (default)
jikkou server-info

# Get server information in JSON format
jikkou server-info -o JSON

Options

FlagShortDefaultDescription
--output-oYAMLOutput format. Valid values: JSON, YAML

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.20 - jikkou update

Create or update resources from the resource definition files.

Synopsis

Reconcile the target platform by creating or updating resources that are described by the resource definition files passed as arguments. This command uses the UPDATE reconciliation mode, meaning new resources will be created and existing resources will be updated. Resources that exist on the platform but are not described in the resource files will not be deleted.

jikkou update [flags]

Examples

# Update resources defined in a YAML file
jikkou update -f my-resources.yaml

# Preview what would be created or updated
jikkou update -f my-resources.yaml --dry-run

# Update resources from a directory
jikkou update -f ./resources/

# Update resources with selector to filter specific resources
jikkou update -f my-resources.yaml -s 'metadata.name MATCHES (my-topic-.*)'

# Update resources targeting a specific provider
jikkou update -f my-resources.yaml --provider kafka-prod

Options

FlagShortDefaultDescription
--files-fResource definition file or directory locations (one or more required)
--file-name-n**/*.{yaml,yml}Glob pattern to filter resource files when using directories
--values-filesTemplate values file locations (one or more)
--values-file-name**/*.{yaml,yml}Glob pattern to filter values files
--set-label-lSet labels on resources (key=value, repeatable)
--set-annotationSet annotations on resources (key=value, repeatable)
--set-value-vSet template variables for the built-in Values object (key=value, repeatable)
--selector-sSelector expression for including or excluding resources
--selector-matchALLSelector matching strategy. Valid values: ALL, ANY, NONE
--optionsController configuration options (key=value, repeatable)
--providerSelect a specific provider instance
--output-oTEXTOutput format. Valid values: TEXT, COMPACT, JSON, YAML
--prettyfalsePretty print JSON output
--dry-runfalseExecute command in dry-run mode (preview changes without applying)

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

1.3.21 - jikkou validate

Check whether the resources definitions meet all validation requirements.

Synopsis

Validate the resource definition files specified through the command line arguments.

Validate runs all the user-defined validation requirements after performing any relevant resource transformations. Validation rules are applied only to resources matching the selectors passed through the command line arguments.

If validation passes, the command outputs the validated resources. If validation fails, the command prints the validation errors and exits with a non-zero status code.

jikkou validate [flags]

Examples

# Validate resources defined in a YAML file
jikkou validate -f my-resources.yaml

# Validate resources from a directory
jikkou validate -f ./resources/

# Validate with a selector to filter specific resources
jikkou validate -f my-resources.yaml -s 'metadata.name MATCHES (my-topic-.*)'

# Validate with template values
jikkou validate -f my-resources.yaml --values-files values.yaml

# Validate with JSON output
jikkou validate -f my-resources.yaml -o JSON

# Validate targeting a specific provider
jikkou validate -f my-resources.yaml --provider kafka-prod

Options

FlagShortDefaultDescription
--files-fResource definition file or directory locations (one or more required)
--file-name-n**/*.{yaml,yml}Glob pattern to filter resource files when using directories
--values-filesTemplate values file locations (one or more)
--values-file-name**/*.{yaml,yml}Glob pattern to filter values files
--set-label-lSet labels on resources (key=value, repeatable)
--set-annotationSet annotations on resources (key=value, repeatable)
--set-value-vSet template variables for the built-in Values object (key=value, repeatable)
--selector-sSelector expression for including or excluding resources
--selector-matchALLSelector matching strategy. Valid values: ALL, ANY, NONE
--optionsController configuration options (key=value, repeatable)
--providerSelect a specific provider instance
--output-oYAMLOutput format. Valid values: JSON, YAML

Options inherited from parent commands

FlagDescription
--logger-level=<level>Log level: TRACE, DEBUG, INFO, WARN, ERROR

SEE ALSO

2 - Jikkou API Server Documentation

Learn Jikkou’s API Server usages.

Jikkou API Server provides a REST interface to any platform supported by Jikkou, making it even easier to manage, automate and visualise all your data platform assets.

Jikkou CLI can be used in combination with Jikkou API Server by configuring it in proxy mode. In this mode, the CLI no longer connects directly to your various platforms, but forwards all operations to the API server. This deployment method allows you to enhance the overall security of the platforms managed through Jikkou.

2.1 - Configurations

Learn how to configure Jikkou API Server.

Jikkou API Server is built with Micronaut Framework.

The default configuration file is located in the installation directory of you server under the path /etc/application.yaml.

You can either modify this configuration file directly or create a new one. Then, your configuration file path can be targeted through the MICRONAUT_CONFIG_FILES environment variable.

A YAML Configuration file example can be found here: application.yaml

2.1.1 - API Server

Learn how to configure the Jikkou API server.

Running Server on a Specific Port

By default, the server runs on port 28082. However, you can set the server to run on a specific port:

# ./etc/application.yaml
micronaut:
  server:
    port: 80  # Port used to access APIs

endpoints:
  all:
    port: 80  # Port used to access Health endpoints

Enabling Specific Extension Providers

By default, the server is configured to run only with the core and kafka extension providers. However, you can enable (or disable) additional providers:

jikkou:
  # The providers
  provider:
    # Core
    core:
      enabled: true
      type: io.jikkou.core.CoreExtensionProvider
      
    # Default configuration for Apache Kafka
    kafka:
      enabled: true
      type: io.jikkou.kafka.KafkaExtensionProvider
      config:
        client:
          bootstrap.servers: localhost:9092
          
    # Default configuration for Schema Registry
    schemaregistry:
      enabled: true
      type: io.jikkou.schema.registry.SchemaRegistryExtensionProvider
      config:
        url: http://localhost:8081
        
    # Default configuration for Kafka Connect
    kafkaconnect:
      enabled: true
      type: io.jikkou.kafka.connect.KafkaConnectExtensionProvider
      config:
        clusters:
          - name: localhost
            url: http://localhost:8083

2.1.2 - Authentication

Learn how to secure access to Jikkou API server.

Enable Security

To enable secure access to the API Server:

Configuration File

Update the configuration file (i.e., application.yaml) of the server with:

micronaut:
  security:
    enabled: true

Environment Variable

As an alternative, you can set the following environment variable MICRONAUT_SECUTIRY_ENABLED=true.

Unauthorized Access

When accessing a secured path, the server will return the following response if access is not authorized:

{
  "message": "Unauthorized",
  "errors": [
    {
      "status": 401,
      "error_code": "authentication_user_unauthorized",
      "message": "Unauthorized"
    }
  ]
}

2.1.2.1 - Basic Auth

Learn how to secure Jikkou API Server using Basic HTTP Authentication Scheme.

Jikkou API Server can be secured using a Basic HTTP Authentication Scheme.

RFC7617 defines the “Basic” Hypertext Transfer Protocol (HTTP) authentication scheme, which transmits credentials as user-id/password pairs, encoded using Base64.

Basic Authentication should be used over a secured connection using HTTPS.

Configure Basic HTTP Authentication

Step1: Enable security

Add the following configuration to your server configuration.

# ./etc/application.yaml
micronaut:
  security:
    enabled: true

Step2: Configure the list of users

The list of username/password authorized to connect to the API server can be configured as follows:

# ./etc/application.yaml
jikkou:
  security:
    basic-auth:
      - username: "admin"
        password: "{noop}password"

For production environment, password must not be configured in plaintext. Password can be passed encoded in bcrypt, scrypt, argon2, and sha256.

Example

echo -n password | sha256sum
5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8
# ./etc/application.yaml
jikkou:
  security:
    basic-auth:
      - username: "admin"
        password: "{sha256}5e884898da28047151d0e56f8dc6292773603d0d6aabbdd62a11ef721d1542d8"

Step3: Validate authentication

Encode credentials

echo -n "admin:password" | base64 
YWRtaW46cGFzc3dvcmQ=

Send request

curl -IX GET http://localhost:28082/apis/kafka.jikkou.io/v1beta2/kafkabrokers \
-H "Accept: application/json" \
-H "Authorization: Basic YWRtaW46cGFzc3dvcmQ"

HTTP/1.1 200 OK
Content-Type: application/hal+json
content-length: 576

2.1.2.2 - JWT

Learn how to secure Jikkou API Server using JWT (JSON Web Token) Authentication.

Jikkou API Server can be secured using JWT (JSON Web Token) Authentication.

Configure JWT

Step1: Set JWT signature secret

Add the following configuration to your server configuration.

# ./etc/application.yaml
micronaut:
  security:
    enabled: true
    authentication: bearer <1>
    token:
      enabled: true
      jwt:
        signatures:
          secret:
            generator:
              secret: ${JWT_GENERATOR_SIGNATURE_SECRET:pleaseChangeThisSecretForANewOne} <2>
  • <1> Set authentication to bearer to receive a JSON response from the login endpoint.
  • <2> Change this to your own secret and keep it safe (do not store this in your VCS).

Step2: Generate a Token

Generate a valid JSON Web Token on https://jwt.io/ using your secret.

Example with pleaseChangeThisSecretForANewOne as signature secret.

TOKEN=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.6cD3MnZmX2xyEAWyh-GgGD11TX8SmvmHVLknuAIJ8yE

Step3: Validate authentication

$ curl -I -X GET http://localhost:28082/apis/kafka.jikkou.io/v1beta2/kafkabrokers \
-H "Accept: application/json" \
-H "Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiIxMjM0NTY3ODkwIiwibmFtZSI6IkpvaG4gRG9lIiwiaWF0IjoxNTE2MjM5MDIyfQ.6cD3MnZmX2xyEAWyh-GgGD11TX8SmvmHVLknuAIJ8yE"

HTTP/1.1 200 OK
Content-Type: application/hal+json
content-length: 576

2.1.3 - CLI Proxy Mode

Learn how to configure Jikkou CLI in proxy mode.

Configuration

Step 1: Enable Proxy Mode

To enable proxy mode so that the CLI communicates directly with your API Server, add the following parameters to your configuration:

jikkou {
  # Proxy Configuration
  proxy {
    # Specify whether proxy mode is enabled (default: false).
    enabled = true
    # URL of the API Server
    url = "http://localhost:28082"
    # Specifcy whether HTTP request debugging should be enabled (default: false)
    debugging = false
    # The connect timeout in millisecond (if not configured used ` default-timeout` ).
    connect-timeout = 10000
    # The read timeout in millisecond (if not configured used ` default-timeout` ).
    read-timeout = 10000
    # The write timeout in millisecond (if not configured used ` default-timeout` ).
    write-timeout = 10000
    # The default timeout (i.e., for read/connect) in millisecond (default: 10000)
    default-timeout = 10000
    # Security settings to authenticate to the API Server.
    security = {
      # For Token based Authentication.
      # access-token = ""
      # For Username/Password Basic-Authentication.
      # basic-auth = {
      #   username = ""
      #   password = ""
      # }
    }
  }
}

Step 2: Check connection

When enabling Proxy Mode, Jikkou CLI provides the additional command server-info. You can use it to verify the connectivity with teh server.

$ jikkou server-info -o JSON | jq

{
  "version": "0.31.0",
  "build_time": "2023-11-15T10:35:22+0100",
  "commit_id": "f3384d38e606fb32599c175895d0cbef28258540"
}

2.2 - Jikkou - API References

Jikkou - API References

3 - Extension Providers

Learn how to use Jikkou Extension Provider to provision and manage configuration assets on your data infrastructure.

The section helps you learn more about the built-in Extension Providers for Jikkou.

3.1 - Core Extensions

Built-in Jikkou core extensions — resource repositories, validations, and transformations available in every installation.

Here, you will find information to use the Core extensions.

More information:

3.1.1 - Resource Repositories

Core resource repositories supported by Jikkou for loading configurations from various sources.

Here, you will find the list of core Resource Repositories supported for Jikkou.

Core Repositories

More information:

3.1.1.1 - Github Resource Repository

Load resources from a public or private GitHub repository.

Overview

The GithubResourceRepository can be used to load resources from a public or private GitHub repository.

Configuration

jikkou {
  repositories = [
    {
      # Name of your local repositories  
      name = "<string>"
      # The fully qualified class name (FQCN) of the repository
      type = io.jikkou.core.repository.GithubResourceRepository
      config {
        # Specify the GitHub repository in the format 'owner/repo'
        repository = "<string>"
  
        # Specify the branch or ref to load resources from
        branch = main
  
        # Specify the paths/directories in the repository containing the resource definitions
        paths = []    # Default: **/*.{yaml,yml}
  
        # Specify the pattern used to match YAML file paths.
        # Pattern should be passed in the form of 'syntax:pattern'. The "glob" and "regex" syntaxes are supported (e.g.: **/*.{yaml,yml}).
        # If no syntax is specified the 'glob' syntax is used.
        file-pattern = "**/*.{yaml,yml}"
        
        # Specify the locations of the values-files containing the variables to pass into the template engine built-in object 'Values'.
        values-files = []
        
        # Specify the pattern used to match YAML file paths when one or multiple directories are given through the `values-files` property.
        # Pattern should be passed in the form of 'syntax:pattern'. The "glob" and "regex" syntaxes are supported (e.g.: **/*.{yaml,yml}).
        # If no syntax is specified the 'glob' syntax is used.
        values-file-name = "<string>" # Default: **/*.{yaml,yml}
        
        # The labels to be added to all resources loaded from the repository
        labels {
          <label_key> = <label_value>
        }
      }
    }
  ]
}

3.1.1.2 - Local Resource Repository

Load resources from local files or directories.

Overview

The LocalResourceRepository can be used to load resources from local files or directories.

Configuration

jikkou {
  repositories = [
    {
      # Name of your local repositories  
      name = "<string>"
      # The fully qualified class name (FQCN) of the repository
      type = io.jikkou.core.repository.LocalResourceRepository
      config {
        # Specify the locations containing the definitions for resources in a YAML file, a directory.
        files = []
        
        # Specify the pattern used to match YAML file paths when one or multiple directories are given through the `files` property.
        # Pattern should be passed in the form of 'syntax:pattern'. The "glob" and "regex" syntaxes are supported (e.g.: **/*.{yaml,yml}).
        # If no syntax is specified the 'glob' syntax is used.
        file-name = "<string>"    # Default: **/*.{yaml,yml}
        
        # Specify the locations of the values-files containing the variables to pass into the template engine built-in object 'Values'.
        values-files = []
        
        # Specify the pattern used to match YAML file paths when one or multiple directories are given through the `values-files` property.
        # Pattern should be passed in the form of 'syntax:pattern'. The "glob" and "regex" syntaxes are supported (e.g.: **/*.{yaml,yml}).
        # If no syntax is specified the 'glob' syntax is used.
        values-file-name = "<string>" # Default: **/*.{yaml,yml}
        
        # The labels to add to all resources loaded from the repository
        labels {
          <label_key> = <label_value>
        }
      }
    }
  ]
}

3.1.2 - Resources

Core resources supported by Jikkou for infrastructure management.

Here, you will find the list of core resources supported for Jikkou.

Core Resources

More information:

3.1.2.1 - ConfigMap

Learn how to use ConfigMap objects.

You can use a ConfigMap to define reusable data in the form of key/value pairs that can then be referenced and used by other resources.

Specification

---
apiVersion: "core.jikkou.io/v1beta2"
kind: ConfigMap
metadata:
  name: '<CONFIG-MAP-NAME>'   # Name of the ConfigMap (required)
data:                         # Map of key-value pairs (required)
  <KEY_1>: "<VALUE_1>" 

Example

For example, the below ConfigMap show how to define default config properties namedcKafkaTopicConfig that can then reference and used to define multiple KafkaTopic. resources.

---
apiVersion: "core.jikkou.io/v1beta2"
kind: ConfigMap
metadata:
name: 'KafkaTopicConfig'
data:
  cleanup.policy: 'delete'
  min.insync.replicas: 2
  retention.ms: 86400000 # (1 day)

3.1.2.2 - ValidatingResourcePolicy

The ValidatingResourcePolicy resource defines validation rules and selection strategies for applying policies to resources in Jikkou.

The ValidatingResourcePolicy resource is used to define validation rules applied to resources or resource changes before they are applied by Jikkou.
It allows enforcing organizational policies, validating constraints, or filtering out undesired operations.

Each policy can select one or more resource kinds and define rules expressed in Google CEL (Common Expression Language).
Rules can either fail the execution or filter the invalid resources, depending on the configured failurePolicy.


Specification

apiVersion: core.jikkou.io/v1
kind: ValidatingResourcePolicy
metadata:
  name: <string> # Required. Unique policy name.
spec:
  failurePolicy: <string> # Required. One of: FAIL | FILTER
  selector:
    matchingStrategy: <string> # Optional. One of: ALL | ANY (default: ALL)
    matchResources:
      - apiVersion: <string> # Optional. API version to match (e.g., core.jikkou.io/v1)
        kind: <string>       # Required. Resource kind (e.g., KafkaTopic)
    matchLabels:
      - key: <string>       # Label key to match
        operator: <string>  # One of: In | NotIn | Exists | DoesNotExist
        values: [<string>]  # Optional list of values
    matchExpressions:
      - <string> # CEL expression
  rules:
    - name: <string> # Required. Rule identifier.
      expression: <string> # Required. A CEL expression evaluated against the resource.
      message: <string> # Optional. Static message returned when the rule fails.
      messageExpression: <string> # Optional. CEL expression to generate a dynamic error message.

Fields

FieldTypeRequiredDescription
spec.failurePolicystringYesDefines the policy behavior when validation fails. Possible values:
FAIL → stop execution with an error.
FILTER → skip the invalid resource(s) but continue processing others.
spec.selector.matchingStrategystringNoStrategy for combining multiple selectors. Possible values:
ALL → resource must match all conditions.
ANY → resource must match at least one condition.
Default: ALL.
spec.selector.matchResourceslistNoSelects resources by API version and kind.
spec.selector.matchLabelslistNoSelects resources based on labels, using operators (In, NotIn, Exists, DoesNotExist).
spec.selector.matchExpressionslistNoSelects resources using CEL expressions for advanced filtering.
spec.ruleslistYesA list of validation rules.
spec.rules[].namestringYesA unique identifier for the rule.
spec.rules[].expressionstringYesA CEL expression evaluated against the resource. The rule fails when the expression evaluates to true.
spec.rules[].messagestringNoStatic error message returned when validation fails.
spec.rules[].messageExpressionstringNoCEL expression returning a dynamic error message string.

Resource Selection

Policies define which resources they apply to using a selector.
A selector can combine multiple strategies to target resources based on:

  • Resource metadata (kind, apiVersion).
  • Labels (with operators like In, NotIn, Exists, DoesNotExist).
  • CEL expressions (arbitrary conditions on resource content).

Matching Strategy

StrategyDescription
ALLThe resource must match all specified selectors (matchResources, matchLabels, and matchExpressions).
ANYThe resource is selected if it matches at least one of the specified selectors.

Default: ALL


matchResources

Selects resources by API version and/or kind.

matchResources:
  - apiVersion: core.jikkou.io/v1
    kind: KafkaTopic
  • apiVersion → Optional. Restricts matching to a specific API group/version.
  • kind → Required. Matches the resource kind (e.g. KafkaTopic, KafkaTopicChange).

matchLabels

Selects resources based on their metadata labels using operators.

matchLabels:
  - key: environment
    operator: In
    values: ["prod", "staging"]
  - key: team
    operator: NotIn
    values: ["test"]
  - key: critical
    operator: Exists

Supported operators:

OperatorDescription
InMatches if the label value is in the list of values.
NotInMatches if the label value is not in the list of values.
ExistsMatches if the label key is defined (value doesn’t matter).
DoesNotExistMatches if the label key is not defined.

matchExpressions

Selects resources using CEL expressions for maximum flexibility.

matchExpressions:
  - "resource.metadata.name.startsWith('topic-')"
  - "resource.spec.partitions > 10"

Examples:

  • Match resources with names starting with topic-.
  • Match topics with more than 10 partitions.

Examples

Example 1: Filtering DELETE operations on KafkaTopic resources

apiVersion: core.jikkou.io/v1
kind: ValidatingResourcePolicy
metadata:
  name: KafkaTopicPolicy
spec:
  failurePolicy: FILTER
  selector:
    matchResources:
      - kind: KafkaTopicChange
  rules:
    - name: FilterDeleteOperation
      expression: "size(resource.spec.changes) > 0 && resource.spec.op == 'DELETE'"
      messageExpression: "'Operation ' + resource.spec.op + ' on topics is not authorized'"

This policy prevents delete operations on Kafka topics from being executed by filtering them out.


Example 2: Validating partitions count for KafkaTopic

apiVersion: core.jikkou.io/v1
kind: ValidatingResourcePolicy
metadata:
  name: KafkaTopicPolicy
spec:
  failurePolicy: FAIL
  selector:
    matchResources:
      - kind: KafkaTopic
  rules:
    - name: MaxTopicPartitions
      expression: "resource.spec.partitions >= 50"
      messageExpression: "'Topic partition MUST be inferior to 50, but was: ' + string(resource.spec.partitions)"

    - name: MinTopicPartitions
      expression: "resource.spec.partitions < 3"
      message: "Topic must have at-least 3 partitions"

This policy enforces a minimum of 3 partitions and a maximum of 49 partitions for Kafka topics.


Example 3: Match only KafkaTopic in prod environment

selector:
  matchingStrategy: ALL
  matchResources:
    - kind: KafkaTopic
  matchLabels:
    - key: environment
      operator: In
      values: ["prod"]

Example 4: Match any KafkaTopic OR resources with label critical=true

selector:
  matchingStrategy: ANY
  matchResources:
    - kind: KafkaTopic
  matchLabels:
    - key: critical
      operator: In
      values: ["true"]

Example 5: Match using CEL expression

selector:
  matchExpressions:
    - "resource.spec.replicationFactor < 3"

Use cases

  • Preventing destructive operations (e.g., deleting topics, removing configs).
  • Enforcing resource limits (e.g., partition count, replication factor).
  • Ensuring naming conventions or metadata compliance.
  • Dynamically generating error messages with contextual information.

3.2 - Apache Kafka

Lean how to use the Jikkou Extension Provider for Apache Kafka.

Here, you will find information to use the Apache Kafka extensions.

More information:

3.2.1 - Configuration

Learn how to configure the extensions for Apache Kafka.

Here, you will find the list of resources supported for Apache Kafka.

Configuration

The Apache Kafka extension is built on top of the Kafka Admin Client. You can configure the properties to be passed to kafka client through the Jikkou client configuration property jikkou.provider.kafka.

Example:

jikkou {
  provider.kafka {
    enabled = true
    type = io.jikkou.kafka.KafkaExtensionProvider
    config = {
      client {
        bootstrap.servers = "localhost:9092"
        security.protocol = "SSL"
        ssl.keystore.location = "/tmp/client.keystore.p12"
        ssl.keystore.password = "password"
        ssl.keystore.type = "PKCS12"
        ssl.truststore.location = "/tmp/client.truststore.jks"
        ssl.truststore.password = "password"
        ssl.key.password = "password"
      }
    }
  }
}

In addition, the extension support configuration settings to wait for at least a minimal number of brokers before processing.

jikkou {
  provider.kafka {
    enabled = true
    type = io.jikkou.kafka.KafkaExtensionProvider
    config = {
      brokers {
        # If 'True' 
        waitForEnabled = true
        waitForEnabled = ${?JIKKOU_KAFKA_BROKERS_WAIT_FOR_ENABLED}
        # The minimal number of brokers that should be alive for the CLI stops waiting.
        waitForMinAvailable = 1
        waitForMinAvailable = ${?JIKKOU_KAFKA_BROKERS_WAIT_FOR_MIN_AVAILABLE}
        # The amount of time to wait before verifying that brokers are available.
        waitForRetryBackoffMs = 1000
        waitForRetryBackoffMs = ${?JIKKOU_KAFKA_BROKERS_WAIT_FOR_RETRY_BACKOFF_MS}
        # Wait until brokers are available or this timeout is reached.
        waitForTimeoutMs = 60000
        waitForTimeoutMs = ${?JIKKOU_KAFKA_BROKERS_WAIT_FOR_TIMEOUT_MS}
      }
    }
  }
}

3.2.2 - Resources

Learn how to use the built-in resources provided by the Extension Provider for Apache Kafka.

Here, you will find the list of resources supported for Apache Kafka.

Apache Kafka Resources

More information:

3.2.2.1 - Kafka Brokers

Learn how to manage Kafka Brokers.

This section describes the resource definition format for kafkabrokers entities, which can be used to define the brokers you plan to manage on a specific Kafka cluster.

Listing KafkaBroker

You can retrieve the state of Kafka Consumer Groups using the jikkou get kafkabrokers (or jikkou get kb) command.

Usage

Usage:

Get all 'KafkaBroker' resources.

jikkou get kafkabrokers [-hV] [--default-configs] [--dynamic-broker-configs]
                        [--list] [--static-broker-configs]
                        [--logger-level=<level>] [-o=<format>]
                        [-s=<expressions>]...

DESCRIPTION:

Use jikkou get kafkabrokers when you want to describe the state of all
resources of type 'KafkaBroker'.

OPTIONS:

      --default-configs   Describe built-in default configuration for configs
                            that have a default value.
      --dynamic-broker-configs
                          Describe dynamic configs that are configured as
                            default for all brokers or for specific broker in
                            the cluster.
  -h, --help              Show this help message and exit.
      --list              Get resources as ResourceListObject.
      --logger-level=<level>
                          Specify the log level verbosity to be used while
                            running a command.
                          Valid level values are: TRACE, DEBUG, INFO, WARN,
                            ERROR.
                          For example, `--logger-level=INFO`
  -o, --output=<format>   Prints the output in the specified format. Allowed
                            values: json, yaml (default yaml).
  -s, --selector=<expressions>
                          The selector expression used for including or
                            excluding resources.
      --static-broker-configs
                          Describe static configs provided as broker properties
                            at start up (e.g. server.properties file).
  -V, --version           Print version information and exit.

(The output from your current Jikkou version may be different from the above example.)

Examples

(command)

$ jikkou get kafkabrokers --static-broker-configs

(output)

---
apiVersion: "kafka.jikkou.io/v1beta2"
kind: "KafkaBroker"
metadata:
  name: "101"
  labels: {}
  annotations:
    kafka.jikkou.io/cluster-id: "xtzWWN4bTjitpL3kfd9s5g"
spec:
  id: "101"
  host: "localhost"
  port: 9092
  configs:
    advertised.listeners: "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://localhost:9092"
    authorizer.class.name: "org.apache.kafka.metadata.authorizer.StandardAuthorizer"
    broker.id: "101"
    controller.listener.names: "CONTROLLER"
    controller.quorum.voters: "101@kafka:29093"
    inter.broker.listener.name: "PLAINTEXT"
    listener.security.protocol.map: "CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT"
    listeners: "PLAINTEXT://kafka:29092,PLAINTEXT_HOST://0.0.0.0:9092,CONTROLLER://kafka:29093"
    log.dirs: "/var/lib/kafka/data"
    node.id: "101"
    offsets.topic.replication.factor: "1"
    process.roles: "broker,controller"
    transaction.state.log.replication.factor: "1"
    zookeeper.connect: ""

3.2.2.2 - Kafka Consumer Groups

Learn how to manage Kafka Consumer Groups.

This section describes the resource definition format for KafkaConsumerGroup entities, which can be used to define the consumer groups you plan to manage on a specific Kafka cluster.

Listing KafkaConsumerGroup

You can retrieve the state of Kafka Consumer Groups using the jikkou get kafkaconsumergroups (or jikkou get kcg) command.

Usage

$ jikkou get kafkaconsumergroups --help

Usage:

Get all 'KafkaConsumerGroup' resources.

jikkou get kafkaconsumergroups [-hV] [--list] [--offsets]
                               [--logger-level=<level>] [-o=<format>]
                               [--in-states=PARAM]... [-s=<expressions>]...

DESCRIPTION:

Use jikkou get kafkaconsumergroups when you want to describe the state of all
resources of type 'KafkaConsumerGroup'.

OPTIONS:

  -h, --help              Show this help message and exit.
      --in-states=PARAM   If states is set, only groups in these states will be
                            returned. Otherwise, all groups are returned. This
                            operation is supported by brokers with version
                            2.6.0 or later
      --list              Get resources as ResourceListObject.
      --logger-level=<level>
                          Specify the log level verbosity to be used while
                            running a command.
                          Valid level values are: TRACE, DEBUG, INFO, WARN,
                            ERROR.
                          For example, `--logger-level=INFO
  -o, --output=<format>   Prints the output in the specified format. Allowed
                            values: json, yaml (default yaml).
      --offsets           Specify whether consumer group offsets should be
                            described.
  -s, --selector=<expressions>
                          The selector expression used for including or
                            excluding resources.
  -V, --version           Print version information and exit

(The output from your current Jikkou version may be different from the above example.)

Examples

(command)

$ jikkou get kafkaconsumergroups --in-states STABLE --offsets

(output)

---
apiVersion: "kafka.jikkou.io/v1beta1"
kind: "KafkaConsumerGroup"
metadata:
  name: "my-group"
  labels:
    kafka.jikkou.io/is-simple-consumer: false
status:
  state: "STABLE"
  members:
    - memberId: "console-consumer-b103994e-bcd5-4236-9d03-97065057e594"
      clientId: "console-consumer"
      host: "/127.0.0.1"
      assignments:
        - "my-topic-0"
      offsets:
        - topic: "my-topic"
          partition: 0
          offset: 0
  coordinator:
    id: "101"
    host: "localhost"
    port: 9092

Managing consumer group configuration

Since Kafka 4.x, consumer groups support dynamic group-level configuration. You can manage these declaratively through the spec.configs map, which is reconciled against the Kafka GROUP configuration resource (drift detection + incremental alter), just like topic configs.

apiVersion: "kafka.jikkou.io/v1"
kind: "KafkaConsumerGroup"
metadata:
  name: "my-group"
spec:
  configs:
    consumer.session.timeout.ms: 50000

By default, group configs present on the cluster but absent from the resource are removed (config-delete-orphans=true). Set it to false to leave externally-managed configs untouched.

3.2.2.3 - Kafka Topics

Learn how to manage Kafka Topics.

KafkaTopic resources are used to define the topics you want to manage on your Kafka Cluster(s). A KafkaTopic resource defines the number of partitions, the replication factor, and the configuration properties to be associated to a topics.

KafkaTopic

Specification

Here is the resource definition file for defining a KafkaTopic.

apiVersion: "kafka.jikkou.io/v1beta2"  # The api version (required)
kind: "KafkaTopic"                     # The resource kind (required)
metadata:
  name: <The name of the topic>        # (required)
  labels: { }
  annotations: { }
spec:
  partitions: <Number of partitions>   # (optional)
  replicas: <Number of replicas>       # (optional)
  configs:
    <config_key>: <Config Value>       # The topic config properties keyed by name to override (optional)
  configMapRefs: [ ]                   # The list of ConfigMap to be applied to this topic (optional)

The metadata.name property is mandatory and specifies the name of the kafka topic.

To use the cluster default values for the number of partitions and replicas you can set the property spec.partitions and spec.replicas to -1.

Example

Here is a simple example that shows how to define a single YAML file containing two topic definition using the KafkaTopic resource type.

file: kafka-topics.yaml

---
apiVersion: "kafka.jikkou.io/v1beta2"
kind: KafkaTopic
metadata:
  name: 'my-topic-p1-r1'  # Name of the topic
  labels:
    environment: example
spec:
  partitions: 1           # Number of topic partitions (use -1 to use cluster default)
  replicas: 1             # Replication factor per partition (use -1 to use cluster default)
  configs:
    min.insync.replicas: 1
    cleanup.policy: 'delete'
---
apiVersion: "kafka.jikkou.io/v1beta2"
kind: KafkaTopic
metadata:
  name: 'my-topic-p2-r1'   # Name of the topic 
  labels:
    environment: example
spec:
  partitions: 2             # Number of topic partitions (use -1 to use cluster default)
  replicas: 1               # Replication factor per partition (use -1 to use cluster default)
  configs:
    min.insync.replicas: 1
    cleanup.policy: 'delete'

See official Apache Kafka documentation for details about the topic-level configs.

KafkaTopicList

If you need to define multiple topics (e.g. using a template), it may be easier to use a KafkaTopicList resource.

Specification

Here the resource definition file for defining a KafkaTopicList.

apiVersion: "kafka.jikkou.io/v1beta2"  # The api version (required)
kind: "KafkaTopicList"                 # The resource kind (required)
metadata: # (optional)
  labels: { }
  annotations: { }
items: [ ]                             # An array of KafkaTopic

Example

Here is a simple example that shows how to define a single YAML file containing two topic definitions using the KafkaTopicList resource type. In addition, the example uses a ConfigMap object to define the topic configuration only once.

file: kafka-topics.yaml

---
apiVersion: "kafka.jikkou.io/v1beta2"
kind: KafkaTopicList
metadata:
  labels:
    environment: example
items:
  - metadata:
      name: 'my-topic-p1-r1'
    spec:
      partitions: 1
      replicas: 1
      configMapRefs: [ "TopicConfig" ]

  - metadata:
      name: 'my-topic-p2-r1'
    spec:
      partitions: 2
      replicas: 1
      configMapRefs: [ "TopicConfig" ]
---
apiVersion: "core.jikkou.io/v1beta2"
kind: ConfigMap
metadata:
  name: 'TopicConfig'
data:
  min.insync.replicas: 1
  cleanup.policy: 'delete'

3.2.2.4 - Kafka Users

Learn how to manage Kafka Users.

This section describes the resource definition format for KafkaUser entities, which can be used to manage SCRAM Users for Apache Kafka.

Definition Format of KafkaUser

Below is the overall structure of the KafkaUser resource.

---
apiVersion: kafka.jikkou.io/v1  # The api version (required)
kind: KafkaUser                 # The resource kind (required)
metadata:
  name: <string>
  annotations:
    # force update
    kafka.jikkou.io/force-password-renewal: <boolean>
spec:
  authentications:
    - type: <enum> # or 
      password: <string>  # leave empty to generate secure password

See below for details about all these fields.

Metadata

metadata.name [required]

The name of the User.

kafka.jikkou.io/force-password-renewal [optional]

Specification

spec.authentications [required]

The list of authentications to manage for the user.

spec.authentications[].type [required]

The authentication type:

  • scram-sha-256
  • scram-sha-512

spec.authentications[].password [required]

The password of the user.

Examples

The following is an example of a resource describing a User:

---
# Example: file: kafka-scram-users.yaml
apiVersion: "kafka.jikkou.io/v1"
kind: "User"
metadata:
  name: "Bob"
spec:
  authentications:
    - type: scram-sha-256
      password: null
    - type: scram-sha-512
      password: null

Listing Kafka Users

You can retrieve the SCRAM users of a Kafka cluster using the jikkou get kafkausers (or jikkou get ku) command.

Usage

$ jikkou get kc --help

Usage:

Get all 'KafkaUser' resources.

jikkou get kafkausers [-hV] [--list] [--logger-level=<level>] [--name=<name>]
                      [-o=<format>]
                      [--selector-match=<selectorMatchingStrategy>]
                      [-s=<expressions>]...

DESCRIPTION:

Use jikkou get kafkausers when you want to describe the state of all resources
of type 'KafkaUser'.

OPTIONS:

  -h, --help              Show this help message and exit.
      --list              Get resources as ResourceListObject (default: false).
      --logger-level=<level>
                          Specify the log level verbosity to be used while
                            running a command.
                          Valid level values are: TRACE, DEBUG, INFO, WARN,
                            ERROR.
                          For example, `--logger-level=INFO`
      --name=<name>       The name of the resource.
  -o, --output=<format>   Prints the output in the specified format. Valid
                            values: JSON, YAML (default: YAML).
  -s, --selector=<expressions>
                          The selector expression used for including or
                            excluding resources.
      --selector-match=<selectorMatchingStrategy>
                          The selector matching strategy. Valid values: NONE,
                            ALL, ANY (default: ALL)
  -V, --version           Print version information and exit.

(The output from your current Jikkou version may be different from the above example.)

Examples

(command)

$ jikkou get ku

(output)

apiVersion: "kafka.jikkou.io/v1"
kind: "KafkaUser"
metadata:
  name: "Bob"
  labels: {}
  annotations:
    kafka.jikkou.io/cluster-id: "xtzWWN4bTjitpL3kfd9s5g"
spec:
  authentications:
    - type: "scram-sha-256"
      iterations: 8192
    - type: "scram-sha-512"
      iterations: 8192

3.2.2.5 - Kafka Share Groups

Learn how to manage Kafka Share Groups (Queues).

This section describes the resource definition format for KafkaShareGroup entities, which can be used to manage the share groups (Queues, KIP-932) of a specific Kafka cluster.

Listing KafkaShareGroup

You can retrieve the state of Kafka Share Groups using the jikkou get kafkasharegroups (or jikkou get ksg) command.

Examples

(command)

$ jikkou get kafkasharegroups --in-states STABLE --offsets

(output)

---
apiVersion: "kafka.jikkou.io/v1"
kind: "KafkaShareGroup"
metadata:
  name: "my-share-group"
spec:
  configs:
    share.auto.offset.reset: "earliest"
status:
  state: "STABLE"
  members:
    - memberId: "share-consumer-b103994e-bcd5-4236-9d03-97065057e594"
      clientId: "share-consumer"
      host: "/127.0.0.1"
      rackId: "rack-1"
      assignments:
        - "my-topic-0"
  offsets:
    - topic: "my-topic"
      partition: 0
      offset: 0
  coordinator:
    id: "101"
    host: "localhost"
    port: 9092

Managing share group configuration

The spec.configs map lets you declaratively manage the dynamic group configuration of a share group (e.g. share.auto.offset.reset, share.record.lock.duration.ms, share.isolation.level). These are applied against the Kafka GROUP configuration resource and are reconciled like topic configs (drift detection + incremental alter).

apiVersion: "kafka.jikkou.io/v1"
kind: "KafkaShareGroup"
metadata:
  name: "orders-queue"
spec:
  configs:
    share.record.lock.duration.ms: 30000
    share.auto.offset.reset: "earliest"

Apply it with:

$ jikkou apply --files orders-queue.yaml

By default, group configs present on the cluster but absent from the resource are removed (config-delete-orphans=true). Set it to false to leave externally-managed configs untouched.

Deleting KafkaShareGroup

A share group can be deleted by reconciling a resource flagged for deletion (jikkou delete), which removes the group via the AdminClient deleteShareGroups API.

Resetting share group offsets

See the KafkaShareGroupsResetOffsets action to reset the Share-Partition Start Offset (SPSO) of one or more share groups.

3.2.2.6 - Kafka Authorizations

Learn how to manage Kafka Authorizations and ACLs.

KafkaPrincipalAuthorization resources are used to define Access Control Lists (ACLs) for principals authenticated to your Kafka Cluster.

Jikkou can be used to describe all ACL policies that need to be created on Kafka Cluster

KafkaPrincipalAuthorization

Specification

---
apiVersion: "kafka.jikkou.io/v1beta2"
kind: "KafkaPrincipalAuthorization"
metadata:
  name: "User:Alice"
spec:
  roles: [ ]                     # List of roles to be added to the principal (optional)
  acls:                          # List of KafkaPrincipalACL (required)
    - resource:
        type: <The type of the resource>                            #  (required)
        pattern: <The pattern to be used for matching resources>    #  (required) 
        patternType: <The pattern type>                             #  (required) 
      type: <The type of this ACL>     # ALLOW or DENY (required) 
      operations: [ ]                  # Operation that will be allowed or denied (required) 
      host: <HOST>                     # IP address from which principal will have access or will be denied (optional)

For more information on how to define authorization and ACLs, see the official Apache Kafka documentation: Security

Operations

The list below describes the valid values for the spec.acls.[].operations property :

  • READ
  • WRITE
  • CREATE
  • DELETE
  • ALTER
  • DESCRIBE
  • CLUSTER_ACTION
  • DESCRIBE_CONFIGS
  • ALTER_CONFIGS
  • IDEMPOTENT_WRITE
  • CREATE_TOKEN
  • DESCRIBE_TOKENS
  • ALL

For more information see official Apache Kafka documentation: Operations in Kafka

Resource Types

The list below describes the valid values for the spec.acls.[].resource.type property :

  • TOPIC
  • GROUP
  • CLUSTER
  • USER
  • TRANSACTIONAL_ID

For more information see official Apache Kafka documentation: Resources in Kafka

Pattern Types

The list below describes the valid values for the spec.acls.[].resource.patternType property :

  • LITERAL: Use to allow or denied a principal to have access to a specific resource name.
  • MATCH: Use to allow or denied a principal to have access to all resources matching the given regex.
  • PREFIXED: Use to allow or denied a principal to have access to all resources having the given prefix.

Example

---
apiVersion: "kafka.jikkou.io/v1beta2"    # The api version (required)
kind: "KafkaPrincipalAuthorization"      # The resource kind (required)
metadata:
  name: "User:Alice"
spec:
  acls:
    - resource:
        type: 'topic'
        pattern: 'my-topic-'
        patternType: 'PREFIXED'
      type: "ALLOW"
      operations: [ 'READ', 'WRITE' ]
      host: "*"
    - resource:
        type: 'topic'
        pattern: 'my-other-topic-.*'
        patternType: 'MATCH'
      type: 'ALLOW'
      operations: [ 'READ' ]
      host: "*"
---
apiVersion: "kafka.jikkou.io/v1beta2"
kind: "KafkaPrincipalAuthorization"
metadata:
  name: "User:Bob"
spec:
  acls:
    - resource:
        type: 'topic'
        pattern: 'my-topic-'
        patternType: 'PREFIXED'
      type: 'ALLOW'
      operations: [ 'READ', 'WRITE' ]
      host: "*"

KafkaPrincipalRole

Specification

apiVersion: "kafka.jikkou.io/v1beta2"    # The api version (required)
kind: "KafkaPrincipalRole"               # The resource kind (required)
metadata:
  name: <Name of role>                   # The name of the role (required)  
spec:
acls: [ ]                                # A list of KafkaPrincipalACL (required)

Example

---
apiVersion: "kafka.jikkou.io/v1beta2"
kind: "KafkaPrincipalRole"
metadata:
  name: "KafkaTopicPublicRead"
spec:
  acls:
    - type: "ALLOW"
      operations: [ 'READ' ]
      resource:
        type: 'topic'
        pattern: '/public-([.-])*/'
        patternType: 'MATCH'
      host: "*"
---
apiVersion: "kafka.jikkou.io/v1beta2"
kind: "KafkaPrincipalRole"
metadata:
  name: "KafkaTopicPublicWrite"
spec:
  acls:
    - type: "ALLOW"
      operations: [ 'WRITE' ]
      resource:
        type: 'topic'
        pattern: '/public-([.-])*/'
        patternType: 'MATCH'
      host: "*"
---

apiVersion: "kafka.jikkou.io/v1beta2"
kind: "KafkaPrincipalAuthorization"
metadata:
  name: "User:Alice"
spec:
  roles:
    - "KafkaTopicPublicRead"
    - "KafkaTopicPublicWrite"
---
apiVersion: "kafka.jikkou.io/v1beta2"
kind: "KafkaPrincipalAuthorization"
metadata:
  name: "User:Bob"
spec:
  roles:
    - "KafkaTopicPublicRead"

3.2.2.7 - Kafka Quotas

Learn how to manage Kafka Client Quotas

KafkaClientQuota resources are used to define the quota limits to be applied on Kafka consumers and producers. A KafkaClientQuota resource can be used to apply limit to consumers and/or producers identified by a client-id or a user principal.

KafkaClientQuota

Specification

Here is the resource definition file for defining a KafkaClientQuota.

apiVersion: "kafka.jikkou.io/v1beta2" # The api version (required)
kind: "KafkaClientQuota"              # The resource kind (required)
metadata: # (optional)
  labels: { }
  annotations: { }
spec:
  type: <The quota type> # (required)       
  entity:
    clientId: <The id of the client>    # (required depending on the quota type).
    user: <The principal of the user>   # (required depending on the quota type).
  configs:
    requestPercentage: <The quota in percentage (%) of total requests>      # (optional)
    producerByteRate: <The quota in bytes for restricting data production>  # (optional)
    consumerByteRate: <The quota in bytes for restricting data consumption> # (optional)

Quota Types

The list below describes the supported quota types:

  • USERS_DEFAULT: Set default quotas for all users.
  • USER: Set quotas for a specific user principal.
  • USER_CLIENT: Set quotas for a specific user principal and a specific client-id.
  • USER_ALL_CLIENTS: Set default quotas for a specific user and all clients.
  • CLIENT: Set default quotas for a specific client.
  • CLIENTS_DEFAULT: Set default quotas for all clients.

Example

Here is a simple example that shows how to define a single YAML file containing two quota definitions using the KafkaClientQuota resource type.

file: kafka-quotas.yaml

---
apiVersion: 'kafka.jikkou.io/v1beta2'
kind: 'KafkaClientQuota'
metadata:
  labels: { }
  annotations: { }
spec:
  type: 'CLIENT'
  entity:
    clientId: 'my-client'
  configs:
    requestPercentage: 10
    producerByteRate: 1024
    consumerByteRate: 1024
---
apiVersion: 'kafka.jikkou.io/v1beta2'
kind: 'KafkaClientQuota'
metadata:
  labels: { }
  annotations: { }
spec:
  type: 'USER'
  entity:
    user: 'my-user'
  configs:
    requestPercentage: 10
    producerByteRate: 1024
    consumerByteRate: 1024

KafkaClientQuotaList

If you need to define multiple topics (e.g. using a template), it may be easier to use a KafkaClientQuotaList resource.

Specification

Here the resource definition file for defining a KafkaTopicList.

apiVersion: "kafka.jikkou.io/v1beta2"  # The api version (required)
kind: "KafkaClientQuotaList"           # The resource kind (required)
metadata: # (optional)
  name: <The name of the topic>
  labels: { }
  annotations: { }
items: [ ]                              # An array of KafkaClientQuota

Example

Here is a simple example that shows how to define a single YAML file containing two KafkaClientQuota definition using the KafkaClientQuotaList resource type.

apiVersion: 'kafka.jikkou.io/v1beta2'
kind: 'KafkaClientQuotaList'
metadata:
  labels: { }
  annotations: { }
items:
  - spec:
    type: 'CLIENT'
    entity:
      clientId: 'my-client'
    configs:
      requestPercentage: 10
      producerByteRate: 1024
      consumerByteRate: 1024

  - spec:
      type: 'USER'
      entity:
        user: 'my-user'
      configs:
        requestPercentage: 10
        producerByteRate: 1024
        consumerByteRate: 1024

3.2.2.8 - Kafka Table Records

Learn how to manage a KTable Topic Records

A KafkaTableRecord resource can be used to produce a key/value record into a given compacted topic, i.e., a topic with cleanup.policy=compact (a.k.a. KTable).

KafkaTableRecord

Specification

Here is the resource definition file for defining a KafkaTableRecord.

apiVersion: "kafka.jikkou.io/v1beta1" # The api version (required)
kind: "KafkaTableRecord"              # The resource kind (required)
metadata:
  labels: { }
  annotations: { }
spec:
  type: <string>      # The topic name (required)        
  headers: # The list of headers
    - name: <string>
      value: <string>
  key: # The record-key (required)
    type: <string>                   # The record-key type. Must be one of: BINARY, STRING, JSON (required)
    data: # The record-key in JSON serialized form.
      $ref: <url or path>            # Or an url to a local file containing the JSON string value.
  value: # The record-value (required)
    type: <string>                   # The record-value type. Must be one of: BINARY, STRING, JSON (required)
    data: # The record-value in JSON serialized form.
      $ref: <url or path>            # Or an url to a local file containing the JSON string value.

Usage

The KafkaTableRecord resource has been designed primarily to manage reference data published and shared via Kafka. Therefore, it is highly recommended to use this resource only with compacted Kafka topics containing a small amount of data.

Examples

Here are some examples that show how to a KafkaTableRecord using the different supported data type.

STRING:

---
apiVersion: "kafka.jikkou.io/v1beta1"
kind: "KafkaTableRecord"
spec:
  topic: "my-topic"
  headers:
    - name: "content-type"
      value: "application/text"
  key:
    type: STRING
    data: |
      "bar"      
  value:
    type: STRING
    data: |
      "foo"      

JSON:

---
apiVersion: "kafka.jikkou.io/v1beta1"
kind: "KafkaTableRecord"
spec:
  topic: "my-topic"
  headers:
    - name: "content-type"
      value: "application/text"
  key:
    type: STRING
    data: |
      "bar"      
  value:
    type: JSON
    data: |
      {
        "foo": "bar"
      }      

BINARY:

---
apiVersion: "kafka.jikkou.io/v1beta1"
kind: "KafkaTableRecord"
spec:
  topic: "my-topic"
  headers:
    - name: "content-type"
      value: "application/text"
  key:
    type: STRING
    data: |
      "bar"      
  value:
    type: BINARY
    data: |
      "eyJmb28iOiAiYmFyIn0K"      

3.2.3 - Transformations

Learn how to use the built-in transformation provided by the Extension Provider for Apache Kafka.

Here, you will find information to use the built-in transformations for Apache Kafka resources.

More information:

3.2.3.1 - KafkaTopicMaxNumPartitions

Enforce a maximum partition count for Kafka topics using this Jikkou transformation.

This transformation can be used to enforce a maximum value for the number of partitions of kafka topics.

Configuration

NameTypeDescriptionDefault
maxNumPartitionsIntmaximum value for the number of partitions to be used for Kafka Topics

Example

jikkou {
  transformations: [
    {
      type = io.jikkou.kafka.transform.KafkaTopicMaxNumPartitions
      priority = 100
      config = {
        maxNumPartitions = 50
      }
    }
  ]
}

3.2.3.2 - KafkaTopicMaxRetentionMs

Enforce a maximum retention.ms value for Kafka topics using this Jikkou transformation.

This transformation can be used to enforce a maximum value for the retention.ms property of kafka topics.

Configuration

NameTypeDescriptionDefault
maxRetentionMsIntMinimum value of retention.ms to be used for Kafka Topics

Example

jikkou {
  transformations: [
    {
      type = io.jikkou.kafka.transform.KafkaTopicMinRetentionMsTransformation
      priority = 100
      config = {
        maxRetentionMs = 2592000000 # 30 days
      }
    }
  ]
}

3.2.3.3 - KafkaTopicMinInSyncReplicas

Enforce a minimum min.insync.replicas value for Kafka topics using this Jikkou transformation.

This transformation can be used to enforce a minimum value for the min.insync.replicas property of kafka topics.

Configuration

NameTypeDescriptionDefault
minInSyncReplicasIntMinimum value of min.insync.replicas to be used for Kafka Topics

Example

jikkou {
  transformations: [
    {
      type = io.jikkou.kafka.transform.KafkaTopicMinInSyncReplicasTransformation
      priority = 100
      config = {
        minInSyncReplicas = 2
      }
    }
  ]
}

3.2.3.4 - KafkaTopicMinReplicas

Enforce a minimum replication factor for Kafka topics using this Jikkou transformation.

This transformation can be used to enforce a minimum value for the replication factor of kafka topics.

Configuration

NameTypeDescriptionDefault
minReplicationFactorIntMinimum value of replication factor to be used for Kafka Topics

Example

jikkou {
  transformations: [
    {
      type = io.jikkou.kafka.transform.KafkaTopicMinReplicasTransformation
      priority = 100
      config = {
        minReplicationFactor = 3
      }
    }
  ]
}

3.2.3.5 - KafkaTopicMinRetentionMs

Enforce a minimum retention.ms value for Kafka topics using this Jikkou transformation.

This transformation can be used to enforce a minimum value for the retention.ms property of kafka topics.

Configuration

NameTypeDescriptionDefault
minRetentionMsIntMinimum value of retention.ms to be used for Kafka Topics

Example

jikkou {
  transformations: [
    {
      type = io.jikkou.kafka.transform.KafkaTopicMinRetentionMsTransformation
      priority = 100
      config = {
        minRetentionMs = 604800000 # 7 days
      }
    }
  ]
}

3.2.4 - Validations

Learn how to use the built-in validations provided by the Extension Provider for Apache Kafka.

Jikkou ships with the following built-in validations:

Topics

NoDuplicateTopicsAllowedValidation

(auto registered)

TopicConfigKeysValidation

(auto registered)

type = io.jikkou.kafka.validation.TopicConfigKeysValidation

The TopicConfigKeysValidation allows checking if the specified topic configs are all valid.

TopicMinNumPartitions

type = io.jikkou.kafka.validation.TopicMinNumPartitionsValidation

The TopicMinNumPartitions allows checking if the specified number of partitions for a topic is not less than the minimum required.

Configuration

NameTypeDescriptionDefault
topicMinNumPartitionsIntMinimum number of partitions allowed

TopicMaxNumPartitions

type = io.jikkou.kafka.validation.TopicMaxNumPartitions

The TopicMaxNumPartitions allows checking if the number of partitions for a topic is not greater than the maximum configured.

Configuration

NameTypeDescriptionDefault
topicMaxNumPartitionsIntMaximum number of partitions allowed

TopicMinReplicationFactor

type = io.jikkou.kafka.validation.TopicMinReplicationFactor

The TopicMinReplicationFactor allows checking if the specified replication factor for a topic is not less than the minimum required.

Configuration

NameTypeDescriptionDefault
topicMinReplicationFactorIntMinimum replication factor allowed

TopicMaxReplicationFactor

type = io.jikkou.kafka.validation.TopicMaxReplicationFactor

The TopicMaxReplicationFactor allows checking if the specified replication factor for a topic is not greater than the maximum configured.

Configuration

NameTypeDescriptionDefault
topicMaxReplicationFactorIntMaximum replication factor allowed

TopicNamePrefix

type = io.jikkou.kafka.validation.TopicNamePrefix

The TopicNamePrefix allows checking if the specified name for a topic starts with one of the configured suffixes.

Configuration

NameTypeDescriptionDefault
topicNamePrefixesListList of topic name prefixes allows

TopicNameSuffix

type = io.jikkou.kafka.validation.TopicNameSuffix

The TopicNameSuffix allows checking if the specified name for a topic ends with one of the configured suffixes.

Configuration

NameTypeDescriptionDefault
topicNameSuffixesListList of topic name suffixes allows

ACLs

NoDuplicateUsersAllowedValidation

(auto registered)

NoDuplicateRolesAllowedValidation

(auto registered)

Quotas

QuotasEntityValidation

(auto registered)

3.2.5 - Annotations

Learn how to use the metadata annotations provided by the extensions for Apache Kafka.

Here, you will find information about the annotations provided the Apache Kafka extension for Jikkou.

List of built-in annotations

kafka.jikkou.io/cluster-id

Used by jikkou.

The annotation is automatically added by Jikkou to a describe object part of an Apache Kafka cluster.

3.2.6 - Actions

Learn how to use the actions provided by the Extension Provider for Apache Kafka.

Here, you will find the list of actions provided by the Extension Provider for Apache Kafka.

Apache Kafka Action

More information:

3.2.6.1 - KafkaConsumerGroupsResetOffsets

Learn how to use the KafkaConsumerGroupsResetOffsets action.

The KafkaConsumerGroupsResetOffsets action allows resetting offsets of consumer group. It supports one consumer group at the time, and group should be in EMPTY state.

Usage (CLI)

Usage:

Execute the action.

jikkou action KafkaConsumerGroupsResetOffsets execute [-hV] [--all] [--dry-run]
[--to-earliest] [--to-latest] [--group=PARAM] [--logger-level=<level>]
[-o=<format>] [--to-datetime=PARAM] [--to-offset=PARAM] [--excludes=PARAM]...
[--groups=PARAM]... [--includes=PARAM]... --topic=PARAM [--topic=PARAM]...

DESCRIPTION:

Reset offsets of consumer group. Supports multiple consumer groups, and groups
should be in EMPTY state.
You must choose one of the following reset specifications: to-datetime,
by-duration, to-earliest, to-latest, to-offset.


OPTIONS:

      --all                 Specifies to act on all consumer groups.
      --dry-run             Only show results without executing changes on
                              Consumer Groups.
      --excludes=PARAM      List of patterns to match the consumer groups that
                              must be excluded from the reset-offset action.
      --group=PARAM         The consumer group to act on.
      --groups=PARAM        The consumer groups to act on.
  -h, --help                Show this help message and exit.
      --includes=PARAM      List of patterns to match the consumer groups that
                              must be included in the reset-offset action.
      --logger-level=<level>
                            Specify the log level verbosity to be used while
                              running a command.
                            Valid level values are: TRACE, DEBUG, INFO, WARN,
                              ERROR.
                            For example, `--logger-level=INFO`
  -o, --output=<format>     Prints the output in the specified format. Allowed
                              values: JSON, YAML (default YAML).
      --to-datetime=PARAM   Reset offsets to offset from datetime. Format:
                              'YYYY-MM-DDTHH:mm:SS.sss'
      --to-earliest         Reset offsets to earliest offset.
      --to-latest           Reset offsets to latest offset.
      --to-offset=PARAM     Reset offsets to a specific offset.
      --topic=PARAM         The topic whose partitions must be included in the
                              reset-offset action.
  -V, --version             Print version information and exit.

Examples

Reset Single Consumer Group to the earliest offsets

jikkou action kafkaconsumergroupresetoffsets execute \
--group my-group \
--topic test \
--to-earliest

(output)

---
kind: "ApiActionResultSet"
apiVersion: "core.jikkou.io/v1"
metadata:
  labels: {}
  annotations:
    configs.jikkou.io/to-earliest: "true"
    configs.jikkou.io/group: "my-group"
    configs.jikkou.io/dry-run: "false"
    configs.jikkou.io/topic: 
        - "test"
results:
- status: "SUCCEEDED"
  errors: []
  data:
    apiVersion: "kafka.jikkou.io/v1beta1"
    kind: "KafkaConsumerGroup"
    metadata:
      name: "my-group"
      labels:
        kafka.jikkou.io/is-simple-consumer: false
      annotations: {}
    status:
      state: "EMPTY"
      members: []
      offsets:
      - topic: "test"
        partition: 1
        offset: 0
      - topic: "test"
        partition: 0
        offset: 0
      - topic: "test"
        partition: 2
        offset: 0
      - topic: "--test"
        partition: 0
        offset: 0
      coordinator:
        id: "101"
        host: "localhost"
        port: 9092

Reset All Consumer Groups to the earliest offsets

jikkou action kafkaconsumergroupresetoffsets execute \
--all \
--topic test \
--to-earliest

3.2.6.2 - KafkaShareGroupsResetOffsets

Learn how to reset the offsets of Kafka Share Groups.

This section describes the KafkaShareGroupsResetOffsets action, used to reset the Share-Partition Start Offset (SPSO) of one or more Kafka share groups.

Usage

You must choose one of the following reset specifications: to-earliest, to-latest, to-offset, or delete-offsets.

OptionDescription
--groupThe share group to act on.
--groupsThe share groups to act on.
--allAct on all share groups.
--topicTopics to include. Each entry is topic (all partitions) or topic:partition.
--to-earliestReset offsets to the earliest offset.
--to-latestReset offsets to the latest offset.
--to-offsetReset offsets to a specific offset.
--delete-offsetsDelete the share-partition offsets for the given topics.
--includesPatterns of share groups to include.
--excludesPatterns of share groups to exclude.
--dry-runOnly show results without executing changes.

Examples

Reset a single share group to the earliest offset for a topic:

$ jikkou action KafkaShareGroupsResetOffsets execute \
    --group my-share-group \
    --topic my-topic \
    --to-earliest

Delete the share-partition offsets for a topic:

$ jikkou action KafkaShareGroupsResetOffsets execute \
    --group my-share-group \
    --topic my-topic \
    --delete-offsets

3.2.7 - Compatibility

Compatibility for Apache Kafka.

The Apache Kafka extension for Jikkou utilizes the Kafka Admin Client which is compatible with any Kafka infrastructure, such as :

  • Aiven
  • Apache Kafka
  • Confluent Cloud
  • MSK
  • Redpanda
  • etc.

In addition, Kafka Protocol has a “bidirectional” client compatibility policy. In other words, new clients can talk to old servers, and old clients can talk to new servers.

3.3 - Apache Kafka Connect

Lean how to use the Jikkou Extension Provider for Apache Kafka Connect.

Here, you will find information to use the Kafka Connect extension for Jikkou.

More information:

3.3.1 - Configuration

Learn how to configure the extensions for Kafka Connect.

This section describes how to configure the Kafka Connect extension.

Configuration

You can configure the properties to be used to connect the Kafka Connect cluster through the Jikkou client configuration property: jikkou.provider.kafkaconnect.

Example:

jikkou {
  provider.kafkaconnect {
    enabled = true
    type = io.jikkou.kafka.connect.KafkaConnectExtensionProvider
    config = {
      # Array of Kafka Connect clusters configurations.
      clusters = [
        {
          # Name of the cluster (e.g., dev, staging, production, etc.)
          name = "localhost"
          # URL of the Kafka Connect service
          url = "http://localhost:8083"
          # Method to use for authenticating on Kafka Connect. Available values are: [none, basicauth, ssl]
          authMethod = none
          # Use when 'authMethod' is 'basicauth' to specify the username for Authorization Basic header
          basicAuthUser = null
          # Use when 'authMethod' is 'basicauth' to specify the password for Authorization Basic header
          basicAuthPassword = null
          # Enable debug logging
          debugLoggingEnabled = false

          # Ssl Config: Use when 'authMethod' is 'ssl'
          # The location of the key store file.
          sslKeyStoreLocation = "/certs/registry.keystore.jks"
          # The file format of the key store file.
          sslKeyStoreType = "JKS"
          # The password for the key store file.
          sslKeyStorePassword = "password"
          # The password of the private key in the key store file.
          sslKeyPassword = "password"
          # The location of the trust store file.
          sslTrustStoreLocation = "/certs/registry.truststore.jks"
          # The file format of the trust store file.
          sslTrustStoreType = "JKS"
          # The password for the trust store file.
          sslTrustStorePassword = "password"
          # Specifies whether to ignore the hostname verification.
          sslIgnoreHostnameVerification = true

          # HTTP proxy: route requests to the Kafka Connect REST API through a forward proxy.
          # Proxy URL, e.g. 'http://proxy.example.com:3128'. When empty, JVM proxy system properties are used.
          proxyUrl = "http://proxy.example.com:3128"
          # Username for proxy Basic authentication (optional).
          proxyUsername = null
          # Password for proxy Basic authentication (optional).
          proxyPassword = null
          # Comma-separated hosts that bypass the proxy, e.g. 'localhost,127.0.0.1,*.internal'.
          nonProxyHosts = "localhost,127.0.0.1"
        }
      ]
    }
  }
}

HTTP proxy

Jikkou can reach the Kafka Connect REST API through an HTTP/HTTPS forward proxy.

PropertyDescription
proxyUrlProxy URL, e.g. http://proxy.example.com:3128. When empty, JVM proxy system properties are used.
proxyUsernameUsername for proxy Basic authentication (optional).
proxyPasswordPassword for proxy Basic authentication (optional).
nonProxyHostsComma-separated hosts that bypass the proxy, e.g. localhost,127.0.0.1,*.internal.

If proxyUrl is not set, Jikkou honors the standard JVM proxy system properties (-Dhttps.proxyHost, -Dhttp.proxyHost, -Dhttp.proxyUser, -Dhttp.proxyPassword, -Dhttp.nonProxyHosts), which can be supplied via JAVA_TOOL_OPTIONS.

3.3.2 - Resources

Learn how to use the resources provided by the Kafka Connect extension.

Here, you will find the list of resources supported by the Kafka Connect Extension.

Kafka Connect Resources

More information:

3.3.2.1 - KafkaConnectors

Learn how to manage Kafka Connectors.

This section describes the resource definition format for KafkaConnector entities, which can be used to define the configuration and status of connectors you plan to create and manage on specific Kafka Connect clusters.

Definition Format of KafkaConnector

Below is the overall structure of the KafkaConnector resource.

---
apiVersion: "kafka.jikkou.io/v1beta1"  # The api version (required)
kind: "KafkaConnector"                 # The resource kind (required)
metadata:
  name: <string>                       # The name of the connector (required)
  labels:
    # Name of the Kafka Connect cluster to create the connector instance in (required).
    kafka.jikkou.io/connect-cluster: <string>
  annotations:
    # Override client properties to connect to Kafka Connect cluster (optional).
    jikkou.io/config-override: | 
      <json>
spec:
  connectorClass: <string>            # Name or alias of the class for this connector.
  tasksMax: <integer>                 # The maximum number of tasks for the Kafka Connector.
  config:                             # Configuration properties of the connector.
    <key>: <value>
  state: <string>                     # The state the connector should be in. Defaults to running.

See below for details about all these fields.

Metadata

metadata.name [required]

The name of the connector.

labels.kafka.jikkou.io/connect-cluster [required]

The name of the Kafka Connect cluster to create the connector instance in. The cluster name must be configured through the kafkaConnect.clusters[] Jikkou’s configuration setting (see: Configuration).

jikkou.io/config-override: [optional]

The JSON client configurations to override for connecting to the Kafka Connect cluster. The configuration properties passed through this annotation override any cluster properties defined in the Jikkou’s configuration setting (see: Configuration).

apiVersion: "kafka.jikkou.io/v1beta1"
kind: "KafkaConnector"
metadata:
  name: "my-connector"
  labels:
    kafka.jikkou.io/connect-cluster: "my-connect-cluster"
  annotations:
    jikkou.io/config-override: |
      { "url": "http://localhost:8083" }      

Specification

spec.connectorClass [required]

The name or alias of the class for this connector.

spec.tasksMax [optional]

The maximum number of tasks for the Kafka Connector. Default is 1.

spec.config [required]

The connector’s configuration properties.

spec.state [optional]

The state the connector should be in. Defaults to running.

Below are the valid values:

  • running: Transition the connector and its tasks to RUNNING state.
  • paused: Pause the connector and its tasks, which stops message processing until the connector is resumed.
  • stopped: Completely shut down the connector and its tasks. The connector config remains present in the config topic of the cluster (if running in distributed mode), unmodified.

Examples

The following is an example of a resource describing a Kafka connector:

---
# Example: file: kafka-connector-filestream-sink.yaml
apiVersion: "kafka.jikkou.io/v1beta1"
kind: "KafkaConnector"
metadata:
  name: "local-file-sink"
  labels:
    kafka.jikkou.io/connect-cluster: "my-connect-cluster"
spec:
  connectorClass: "FileStreamSink"
  tasksMax: 1
  config:
    file: "/tmp/test.sink.txt"
    topics: "connect-test"
  state: "RUNNING"

Listing KafkaConnector

You can retrieve the state of Kafka Connector instances running on your Kafka Connect clusters using the jikkou get kafkaconnectors (or jikkou get kc) command.

Usage

$jikkou get kc --help

Usage:

Get all 'KafkaConnector' resources.

jikkou get kafkaconnectors [-hV] [--expand-status] [-o=<format>]
                           [-s=<expressions>]...

Description:

Use jikkou get kafkaconnectors when you want to describe the state of all
resources of type 'KafkaConnector'.

Options:

      --expand-status     Retrieves additional information about the status of
                            the connector and its tasks.
  -h, --help              Show this help message and exit.
  -o, --output=<format>   Prints the output in the specified format. Allowed
                            values: json, yaml (default yaml).
  -s, --selector=<expressions>
                          The selector expression use for including or
                            excluding resources.
  -V, --version           Print version information and exit.

(The output from your current Jikkou version may be different from the above example.)

Examples

(command)

$ jikkou get kc --expand-status 

(output)

apiVersion: "kafka.jikkou.io/v1beta1"
kind: "KafkaConnector"
metadata:
  name: "local-file-sink"
  labels:
    kafka.jikkou.io/connect-cluster: "localhost"
spec:
  connectorClass: "FileStreamSink"
  tasksMax: 1
  config:
    file: "/tmp/test.sink.txt"
    topics: "connect-test"
  state: "RUNNING"
status:
  connectorStatus:
    name: "local-file-sink"
    connector:
      state: "RUNNING"
      worker_id: "localhost:8083"
    tasks:
      id: 1
      state: "RUNNING"
      worker_id: "localhost:8083"

The status.connectorStatus provides the connector status, as reported by the Kafka Connect REST API.

3.3.3 - Validations

Learn how to use the validations provided by the Kafka Connect extension.

Jikkou ships with the following built-in validations:

3.3.4 - Annotations

Learn how to use the metadata annotations provided by the Kafka Connect extension.

This section lists a number of well known annotations, that have defined semantics. They can be attached to KafkaConnect resources through the metadata.annotations field and consumed as needed by extensions (i.e., validations, transformations, controller, collector, etc.).

List of built-in annotations

3.3.5 - Labels

Learn how to use the metadata labels provided by the Kafka Connect extension.

This section lists a number of well known labels, that have defined semantics. They can be attached to KafkaConnect resources through the metadata.labels field and consumed as needed by extensions (i.e., validations, transformations, controller, collector, etc.).

Labels

kafka.jikkou.io/connect-cluster

# Example
---
apiVersion: "kafka.jikkou.io/v1beta1"
kind: "KafkaConnector"
metadata:
  labels:
    kafka.jikkou.io/connect-cluster: 'my-connect-cluster'

The value of this label defined the name of the Kafka Connect cluster to create the connector instance in. The cluster name must be configured through the kafkaConnect.clusters[] Jikkou’s configuration setting (see: Configuration).

3.3.6 - Actions

Learn how to use the actions provided by the Extension Provider for Kafka Connect.

Here, you will find the list of actions provided by the Extension Provider for Kafka Connect.

Kafka Connect Action

More information:

3.3.6.1 - KafkaConnectRestartConnectors

Learn how to use the KafkaConnectRestartConnector action.

The KafkaConnectRestartConnectors action allows a user to restart all or just the failed Connector and Task instances for one or multiple named connectors.

Usage (CLI)

Usage:

Execute the action.

jikkou action KafkaConnectRestartConnectors execute [-hV] [--include-tasks]
[--only-failed] [--connect-cluster=PARAM] [--logger-level=<level>]
[-o=<format>] [--connector-name=PARAM]...

DESCRIPTION:

The KafkaConnectRestartConnectors action a user to restart all or just the
failed Connector and Task instances for one or multiple named connectors.

OPTIONS:

      --connect-cluster=PARAM
                          The name of the connect cluster.
      --connector-name=PARAM
                          The connector's name.
  -h, --help              Show this help message and exit.
      --include-tasks     Specifies whether to restart the connector instance
                            and task instances (includeTasks=true) or just the
                            connector instance (includeTasks=false)
      --logger-level=<level>
                          Specify the log level verbosity to be used while
                            running a command.
                          Valid level values are: TRACE, DEBUG, INFO, WARN,
                            ERROR.
                          For example, `--logger-level=INFO`
  -o, --output=<format>   Prints the output in the specified format. Allowed
                            values: JSON, YAML (default YAML).
      --only-failed       Specifies whether to restart just the instances with
                            a FAILED status (onlyFailed=true) or all instances
                            (onlyFailed=false)
  -V, --version           Print version information and exit.

Examples

Restart all connectors for all Kafka Connect clusters.

jikkou action kafkaconnectrestartconnectors execute

(output)

---
kind: "ApiActionResultSet"
apiVersion: "core.jikkou.io/v1"
metadata:
  labels: {}
  annotations: {}
results:
- status: "SUCCEEDED"
  data:
    apiVersion: "kafka.jikkou.io/v1beta1"
    kind: "KafkaConnector"
    metadata:
      name: "local-file-sink"
      labels:
        kafka.jikkou.io/connect-cluster: "my-connect-cluster"
      annotations: {}
    spec:
      connectorClass: "FileStreamSink"
      tasksMax: 1
      config:
        file: "/tmp/test.sink.txt"
        topics: "connect-test"
      state: "RUNNING"
    status:
      connectorStatus:
        name: "local-file-sink"
        connector:
          state: "RUNNING"
          workerId: "connect:8083"
        tasks:
        - id: 0
          state: "RUNNING"
          workerId: "connect:8083"

Restart all connectors with a FAILED status on all Kafka Connect clusters.

jikkou action kafkaconnectrestartconnectors execute \
--only-failed

Restart specific connector and tasks for on Kafka Connect cluster

jikkou action kafkaconnectrestartconnectors execute \
--cluster-name my-connect-cluster
--connector-name local-file-sink \
--include-tasks

3.4 - Schema Registry

Lean how to use the Jikkou Extension Provider for Schema Registry.

Here, you will find information to use the Schema Registry extensions.

More information:

3.4.1 - Configuration

Learn how to configure the extensions for SchemaRegistry.

Here, you will find the list of resources supported for SchemaRegistry.

Configuration

You can configure the properties to be used to connect the SchemaRegistry service through the Jikkou client configuration property jikkou.provider.schemaregistry.

Example:

jikkou {
  provider.schemaregistry {
    enabled = true
    type = io.jikkou.schema.registry.SchemaRegistryExtensionProvider
    config = {
      # Comma-separated list of URLs for schema registry instances that can be used to register or look up schemas
      url = "http://localhost:8081"
      # The name of the schema registry implementation vendor - can be any value
      vendor = generic
      # Method to use for authenticating on Schema Registry. Available values are: [none, basicauth, ssl]
      authMethod = none
      # Use when 'schemaRegistry.authMethod' is 'basicauth' to specify the username for Authorization Basic header
      basicAuthUser = null
      # Use when 'schemaRegistry.authMethod' is 'basicauth' to specify the password for Authorization Basic header
      basicAuthPassword = null
      # Enable debug logging
      debugLoggingEnabled = false

      # Ssl Config: Use when 'authMethod' is 'ssl'
      # The location of the key store file.
      sslKeyStoreLocation = "/certs/registry.keystore.jks"
      # The file format of the key store file.
      sslKeyStoreType = "JKS"
      # The password for the key store file.
      sslKeyStorePassword = "password"
      # The password of the private key in the key store file.
      sslKeyPassword = "password"
      # The location of the trust store file.
      sslTrustStoreLocation = "/certs/registry.truststore.jks"
      # The file format of the trust store file.
      sslTrustStoreType = "JKS"
      # The password for the trust store file.
      sslTrustStorePassword = "password"
      # Specifies whether to ignore the hostname verification.
      sslIgnoreHostnameVerification = true

      # HTTP proxy: route requests to the Schema Registry through a forward proxy.
      # Proxy URL, e.g. 'http://proxy.example.com:3128'. When empty, JVM proxy system properties are used.
      proxyUrl = "http://proxy.example.com:3128"
      # Username for proxy Basic authentication (optional).
      proxyUsername = null
      # Password for proxy Basic authentication (optional).
      proxyPassword = null
      # Comma-separated hosts that bypass the proxy, e.g. 'localhost,127.0.0.1,*.internal'.
      nonProxyHosts = "localhost,127.0.0.1"
    }
  }
}

HTTP proxy

Jikkou can reach the Schema Registry REST API through an HTTP/HTTPS forward proxy.

PropertyDescription
proxyUrlProxy URL, e.g. http://proxy.example.com:3128. When empty, JVM proxy system properties are used.
proxyUsernameUsername for proxy Basic authentication (optional).
proxyPasswordPassword for proxy Basic authentication (optional).
nonProxyHostsComma-separated hosts that bypass the proxy, e.g. localhost,127.0.0.1,*.internal.

If proxyUrl is not set, Jikkou honors the standard JVM proxy system properties (-Dhttps.proxyHost, -Dhttp.proxyHost, -Dhttp.proxyUser, -Dhttp.proxyPassword, -Dhttp.nonProxyHosts), which can be supplied via JAVA_TOOL_OPTIONS.

3.4.2 - Resources

Learn how to use the built-in resources provided by the extensions for Schema Registry.

Here, you will find the list of resources supported for Schema Registry.

Schema Registry Resources

More information:

3.4.2.1 - Schema Registry Subjects

Learn how to manage SchemaRegistry Subjects.

SchemaRegistrySubject resources are used to define the subject schemas you want to manage on your SchemaRegistry. A SchemaRegistrySubject resource defines the schema, the compatibility level, and the references to be associated with a subject version.

Definition Format of SchemaRegistrySubject

Below is the overall structure of the SchemaRegistrySubject resource.

apiVersion: "schemaregistry.jikkou.io/v1beta2"  # The api version (required)
kind: "SchemaRegistrySubject"                   # The resource kind (required)
metadata:
  name: <The name of the subject>               # (required)
  labels: { }
  annotations: { }
spec:
  schemaRegistry:
    vendor: <vendor_name>                       # (optional) The vendor of the SchemaRegistry, e.g., Confluent, Karapace, etc
  compatibilityLevel: <compatibility_level>     # (optional) The schema compatibility level for this subject.
  schemaType: <The schema format>               # (required) Accepted values are: AVRO, PROTOBUF, JSON
  schema:
    $ref: <url or path>  # 
  references: # Specifies the names of referenced schemas (optional array).
    - name: <string>                             # The name for the reference.
      subject: <string>                          # The subject under which the referenced schema is registered.
      version: <string>                          # The exact version of the schema under the registered subject.
]

Metadata

The metadata.name property is mandatory and specifies the name of the Subject.

Specification

To use the SchemaRegistry default values for the compatibilityLevel you can omit the property.

Example

Here is a simple example that shows how to define a single subject AVRO schema for type using the SchemaRegistrySubject resource type.

file: subject-user.yaml

---
apiVersion: "schemaregistry.jikkou.io/v1beta2"
kind: "SchemaRegistrySubject"
metadata:
  name: "User"
  labels: { }
  annotations:
    schemaregistry.jikkou.io/normalize-schema: true
spec:
  compatibilityLevel: "FULL_TRANSITIVE"
  schemaType: "AVRO"
  schema:
    $ref: ./user-schema.avsc

file: user-schema.avsc

---
{
  "namespace": "example.avro",
  "type": "record",
  "name": "User",
  "fields": [
    {
      "name": "name",
      "type": [ "null", "string" ],
      "default": null,
    },
    {
      "name": "favorite_number",
      "type": [ "null", "int" ],
      "default": null
    },
    {
      "name": "favorite_color",
      "type": [ "null", "string" ],
      "default": null
    }
  ]
}

Alternatively, we can directly pass the Avro schema as follows :

file: subject-user.yaml

---
apiVersion: "schemaregistry.jikkou.io/v1beta2"
kind: "SchemaRegistrySubject"
metadata:
  name: "User"
  labels: { }
  annotations:
    schemaregistry.jikkou.io/normalize-schema: true
spec:
  compatibilityLevel: "FULL_TRANSITIVE"
  schemaType: "AVRO"
  schema: |
    {
      "namespace": "example.avro",
      "type": "record",
      "name": "User",
      "fields": [
        {
          "name": "name",
          "type": [ "null", "string" ],
          "default": null
        },
        {
          "name": "favorite_number",
          "type": [ "null", "int" ],
          "default": null
        },
        {
          "name": "favorite_color",
          "type": [ "null", "string"],
          "default": null
        }
      ]
    }    

SchemaRegistrySubjectList

If you need to manage multiple Schemas at once (e.g. using a template), it may be more suitable to use the resource collection SchemaRegistrySubjectList.

Specification

Here the resource definition file for defining a SchemaRegistrySubjectList.

apiVersion: "schemaregistry.jikkou.io/v1beta2"  # The api version (required)
kind: "SchemaRegistrySubjectList"      # The resource kind (required)
metadata: # (optional)
  labels: { }
  annotations: { }
items: [ ]                             # The array of SchemaRegistrySubject

3.4.3 - Validations

Learn how to use the built-in validations provided by the extensions for Schema Registry.

Jikkou ships with the following built-in validations:

Subject

SchemaCompatibilityValidation

type = io.jikkou.schema.registry.validation.SchemaCompatibilityValidation

The SchemaCompatibilityValidation allows testing the compatibility of the schema with the latest version already registered in the Schema Registry using the provided compatibility-level.

AvroSchemaValidation

The AvroSchemaValidation allows checking if the specified Avro schema matches some specific avro schema definition rules;

type = io.jikkou.schema.registry.validation.AvroSchemaValidation

Configuration

NameTypeDescriptionDefault
fieldsMustHaveDocBooleanVerify that all record fields have a doc propertyfalse
fieldsMustBeNullableBooleanVerify that all record fields are nullablefalse
fieldsMustBeOptionalBooleanVerify that all record fields are optionalfalse

3.4.4 - Annotations

Learn how to use the metadata annotations provided by the extensions for Schema Registry.

Here, you will find information about the annotations provided by the Schema Registry extension for Jikkou.

List of built-in annotations

schemaregistry.jikkou.io/url

Used by jikkou.

The annotation is automatically added by Jikkou to describe the SchemaRegistry URL from which a subject schema is retrieved.

schemaregistry.jikkou.io/schema-version

Used by jikkou.

The annotation is automatically added by Jikkou to describe the version of a subject schema.

schemaregistry.jikkou.io/schema-id

Used by jikkou.

The annotation is automatically added by Jikkou to describe the version of a subject id.

schemaregistry.jikkou.io/normalize-schema

Used on: schemaregistry.jikkou.io/v1beta2:SchemaRegistrySubject

This annotation can be used to normalize the schema on SchemaRegistry server side. Note, that Jikkou will attempt to normalize AVRO and JSON schema.

See: Confluent SchemaRegistry API Reference

schemaregistry.jikkou.io/permanante-delete

Used on: schemaregistry.jikkou.io/v1beta2:SchemaRegistrySubject

The annotation can be used to specify a hard delete of the subject, which removes all associated metadata including the schema ID. The default is false. If the flag is not included, a soft delete is performed. You must perform a soft delete first, then the hard delete.

See: Confluent SchemaRegistry API Reference

schemaregistry.jikkou.io/use-canonical-fingerprint

This annotation can be used to use a canonical fingerprint to compare schemas (only supported for Avro schema).

3.5 - Aiven

Lean how to use the Jikkou Extensions Providers for Aiven.

Here, you will find information to use the Aiven for Kafka extensions.

More information:

3.5.1 - Configuration

Learn how to configure the extensions for Aiven.

Here, you will find the list of resources supported by the extension for Aiven.

Configuration

You can configure the properties to be used to connect the Aiven service through the Jikkou client configuration property jikkou.provider.aiven.

Example:

jikkou {
  provider.aiven {
    enabled = true
    type = io.jikkou.extension.aiven.AivenExtensionProvider
    config = {
      # Aiven project name
      project = "http://localhost:8081"
      # Aiven service name
      service = generic
      # URL to the Aiven REST API.
      apiUrl = "https://api.aiven.io/v1/"
      # Aiven Bearer Token. Tokens can be obtained from your Aiven profile page
      tokenAuth = null
      # HTTP proxy URL, e.g. 'http://proxy.example.com:3128'. When empty, JVM proxy system properties are used.
      proxyUrl = "http://proxy.example.com:3128"
      # Username for proxy Basic authentication (optional).
      proxyUsername = null
      # Password for proxy Basic authentication (optional).
      proxyPassword = null
      # Comma-separated hosts that bypass the proxy, e.g. 'localhost,127.0.0.1,*.internal'.
      nonProxyHosts = "localhost,127.0.0.1"
      # Enable debug logging
      debugLoggingEnabled = false
    }
  }
}

HTTP proxy

Jikkou can reach the Aiven REST API through an HTTP/HTTPS forward proxy.

PropertyDescription
proxyUrlProxy URL, e.g. http://proxy.example.com:3128. When empty, JVM proxy system properties are used.
proxyUsernameUsername for proxy Basic authentication (optional).
proxyPasswordPassword for proxy Basic authentication (optional).
nonProxyHostsComma-separated hosts that bypass the proxy, e.g. localhost,127.0.0.1,*.internal.

If proxyUrl is not set, Jikkou honors the standard JVM proxy system properties (-Dhttps.proxyHost, -Dhttp.proxyHost, -Dhttp.proxyUser, -Dhttp.proxyPassword, -Dhttp.nonProxyHosts), which can be supplied via JAVA_TOOL_OPTIONS.

3.5.2 - Resources

Learn how to use the built-in resources provided by the extensions for Aiven.

Here, you will find the list of resources supported by the extensions for Aiven.

Aiven for Apache Kafka Resources

More information:

3.5.2.1 - ACL for Aiven Apache Kafka®

Learn how to manage Access Control Lists (ACLs) in Aiven for Apache Kafka®

The KafkaTopicAclEntry resources are used to manage the Access Control Lists in Aiven for Apache Kafka®. A KafkaTopicAclEntry resource defines the permission to be granted to a user for one or more kafka topics.

KafkaTopicAclEntry

Specification

Here is the resource definition file for defining a KafkaTopicAclEntry.

---
apiVersion: "kafka.aiven.io/v1beta1"   # The api version (required)
kind: "KafkaTopicAclEntry"             # The resource kind (required)
metadata:
  labels: { }
  annotations: { }
spec:
  permission: <>               # The permission. Accepted values are: READ, WRITE, READWRITE, ADMIN
  username: <>                 # The username
  topic: <>                    # Topic name or glob pattern

Example

Here is a simple example that shows how to define a single ACL entry using the KafkaTopicAclEntry resource type.

file: kafka-topic-acl-entry.yaml

---
apiVersion: "kafka.aiven.io/v1beta1"
kind: "KafkaTopicAclEntry"
metadata:
  labels: { }
  annotations: { }
spec:
  permission: "READWRITE"
  username: "alice"
  topic: "public-*"

KafkaTopicAclEntryList

If you need to define multiple ACL entries (e.g. using a template), it may be easier to use a KafkaTopicAclEntryList resource.

Specification

Here the resource definition file for defining a KafkaTopicList.

---
apiVersion: "kafka.aiven.io/v1beta1"    # The api version (required)
kind: "KafkaTopicAclEntryList"          # The resource kind (required)
metadata: # (optional)
  name: <The name of the topic>
  labels: { }
  annotations: { }
items: [ ]                             # An array of KafkaTopicAclEntry

Example

Here is a simple example that shows how to define a single YAML file containing two ACL entry definitions using the KafkaTopicAclEntryList resource type.

---
apiVersion: "kafka.aiven.io/v1beta1"
kind: "KafkaTopicAclEntryList"
items:
  - spec:
      permission: "READWRITE"
      username: "alice"
      topic: "public-*"
  - spec:
      permission: "READ"
      username: "bob"
      topic: "public-*"

3.5.2.2 - Quotas for Aiven Apache Kafka®

Learn how to manage Quotas in Aiven for Apache Kafka®

The KafkaQuota resources are used to manage the Quotas in Aiven for Apache Kafka® service. For more details, see https://docs.aiven.io/docs/products/kafka/concepts/kafka-quotas

KafkaQuota

Specification

Here is the resource definition file for defining a KafkaQuota.

---
apiVersion: "kafka.aiven.io/v1beta1"   # The api version (required)
kind: "KafkaQuota"                     # The resource kind (required)
metadata:
  labels: { }
  annotations: { }
spec:
  user: <string>                     # The username: (Optional: 'default' if null)
  clientId: <string>                 # The client-id
  consumerByteRate: <number>         # The quota in bytes for restricting data consumption
  producerByteRate: <number>         # The quota in bytes for restricting data production
  requestPercentage: <number>

Example

Here is a simple example that shows how to define a single ACL entry using the KafkaQuota resource type.

file: kafka-quotas.yaml

---
apiVersion: "kafka.aiven.io/v1beta1"
kind: "KafkaQuota"
spec:
  user: "default"
  clientId: "default"
  consumerByteRate: 1048576
  producerByteRate: 1048576
  requestPercentage: 25

KafkaQuotaList

If you need to define multiple Kafka quotas (e.g. using a template), it may be easier to use a KafkaQuotaList resource.

Specification

Here the resource definition file for defining a KafkaTopicList.

---
apiVersion: "kafka.aiven.io/v1beta1"    # The api version (required)
kind: "KafkaQuotaList"                  # The resource kind (required)
metadata: # (optional)
  labels: { }
  annotations: { }
items: [ ]                             # An array of KafkaQuotaList

Example

Here is a simple example that shows how to define a single YAML file containing two ACL entry definitions using the KafkaQuotaList resource type.

---
apiVersion: "kafka.aiven.io/v1beta1"
kind: "KafkaQuotaList"
items:
  - spec:
      user: "default"
      clientId: "default"
      consumerByteRate: 1048576
      producerByteRate: 1048576
      requestPercentage: 5
  - spec:
      user: "avnadmin"
      consumerByteRate: 5242880
      producerByteRate: 5242880
      requestPercentage: 25

3.5.2.3 - ACL for Aiven Schema Registry

Learn how to manage Access Control Lists (ACLs) in Aiven for Schema Registry

The SchemaRegistryAclEntry resources are used to manage the Access Control Lists in Aiven for Schema Registry. A SchemaRegistryAclEntry resource defines the permission to be granted to a user for one or more Schema Registry Subjects.

SchemaRegistryAclEntry

Specification

Here is the resource definition file for defining a SchemaRegistryAclEntry.

---
apiVersion: "kafka.aiven.io/v1beta1"   # The api version (required)
kind: "SchemaRegistryAclEntry"         # The resource kind (required)
metadata:
  labels: { }
  annotations: { }
spec:
  permission: <>               # The permission. Accepted values are: READ, WRITE
  username: <>                 # The username
  resource: <>                 # The Schema Registry ACL entry resource name pattern

NOTE: The resource name pattern should be Config: or Subject:<subject_name> where subject_name must consist of alpha-numeric characters, underscores, dashes, dots and glob characters * and ?.

Example

Here is an example that shows how to define a simple ACL entry using the SchemaRegistryAclEntry resource type.

file: schema-registry-acl-entry.yaml

---
apiVersion: "kafka.aiven.io/v1beta1"
kind: "SchemaRegistryAclEntry"
spec:
  permission: "READ"
  username: "Alice"
  resource: "Subject:*"

SchemaRegistryAclEntryList

If you need to define multiple ACL entries (e.g. using a template), it may be easier to use a SchemaRegistryAclEntryList resource.

Specification

Here the resource definition file for defining a SchemaRegistryAclEntryList.

---
apiVersion: "kafka.aiven.io/v1beta1"    # The api version (required)
kind: "SchemaRegistryAclEntryList"      # The resource kind (required)
metadata: # (optional)
  labels: { }
  annotations: { }
items: [ ]                              # An array of SchemaRegistryAclEntry

Example

Here is a simple example that shows how to define a single YAML file containing two ACL entry definitions using the SchemaRegistryAclEntryList resource type.

---
apiVersion: "kafka.aiven.io/v1beta1"
kind: "SchemaRegistryAclEntryList"
items:
  - spec:
      permission: "READ"
      username: "alice"
      resource: "Config:"
  - spec:
      permission: "WRITE"
      username: "alice"
      resource: "Subject:*"

3.5.2.4 - Subject for Aiven Schema Registry

Learn how to manage Schema Registry Subject Schema in Aiven.

SchemaRegistrySubject resources are used to define the subject schemas you want to manage on your Schema Registry. A SchemaRegistrySubject resource defines the schema, the compatibility level, and the references to be associated with a subject version.

SchemaRegistrySubject

Specification

Here is the resource definition file for defining a SchemaRegistrySubject.

apiVersion: "kafka.aiven.io/v1beta1"  # The api version (required)
kind: "SchemaRegistrySubject"                   # The resource kind (required)
metadata:
  name: <The name of the subject>               # (required)
  labels: { }
  annotations: { }
spec:
  schemaRegistry:
    vendor: 'Karapace'                          # (optional) The vendor of the Schema Registry
  compatibilityLevel: <compatibility_level>     # (optional) The schema compatibility level for this subject.
  schemaType: <The schema format>               # (required) Accepted values are: AVRO, PROTOBUF, JSON
  schema:
    $ref: <url or path>  # 
  references:                                   # Specifies the names of referenced schemas (optional array).
    - name: <>                                  # The name for the reference.
      subject: <>                               # The subject under which the referenced schema is registered.
      version: <>                               # The exact version of the schema under the registered subject.
]

The metadata.name property is mandatory and specifies the name of the Subject.

To use the SchemaRegistry default values for the compatibilityLevel you can omit the property.

Example

Here is a simple example that shows how to define a single subject AVRO schema for type using the SchemaRegistrySubject resource type.

file: subject-user.yaml

---
apiVersion: "kafka.aiven.io/v1beta1"
kind: "SchemaRegistrySubject"
metadata:
  name: "User"
  labels: { }
  annotations:
    schemaregistry.jikkou.io/normalize-schema: true
spec:
  compatibilityLevel: "FULL_TRANSITIVE"
  schemaType: "AVRO"
  schema:
    $ref: ./user-schema.avsc

file: user-schema.avsc

---
{
  "namespace": "example.avro",
  "type": "record",
  "name": "User",
  "fields": [
    {
      "name": "name",
      "type": [ "null", "string" ],
      "default": null,
    },
    {
      "name": "favorite_number",
      "type": [ "null", "int" ],
      "default": null
    },
    {
      "name": "favorite_color",
      "type": [ "null", "string" ],
      "default": null
    }
  ]
}

Alternatively, we can directly pass the Avro schema as follows :

file: subject-user.yaml

---
apiVersion: "kafka.aiven.io/v1beta1"
kind: "SchemaRegistrySubject"
metadata:
  name: "User"
  labels: { }
  annotations:
    schemaregistry.jikkou.io/normalize-schema: true
spec:
  compatibilityLevel: "FULL_TRANSITIVE"
  schemaType: "AVRO"
  schema: |
    {
      "namespace": "example.avro",
      "type": "record",
      "name": "User",
      "fields": [
        {
          "name": "name",
          "type": [ "null", "string" ],
          "default": null
        },
        {
          "name": "favorite_number",
          "type": [ "null", "int" ],
          "default": null
        },
        {
          "name": "favorite_color",
          "type": [ "null", "string"],
          "default": null
        }
      ]
    }    

3.5.3 - Validations

Learn how to use the built-in validations provided by the extensions for Aiven.

Jikkou ships with the following built-in validations:

No validation

3.5.4 - Annotations

Learn how to use the metadata annotations provided by the extensions for Aiven.

Here, you will find information about the annotations provided by the Aiven extension for Jikkou.

List of built-in annotations

kafka.aiven.io/acl-entry-id

Used by jikkou.

The annotation is automatically added by Jikkou to describe the ID of an ACL entry.

3.6 - AWS

Learn how to use the Jikkou extension provider for AWS — manage AWS Glue Schema Registry resources as code.

Here, you will find information to use the AWS extensions (Jikkou v0.36).

More information:

3.6.1 - Configuration

Learn how to configure the extensions for AWS.

Here, you will find the list of resources supported by the extension for AWS.

Configuration

You can configure the properties to be used to connect the AWS services through the Jikkou client configuration property jikkou.providers.

Example:

jikkou {
  # AWS
  provider.aws {
    enabled = true
    config = {
      # The AWS S3 Region, e.g. us-east-1
      aws.client.region = ""
      # The AWS Access Key ID.
      aws.client.accessKeyId = ""
      # The AWS Secret Access Key.
      aws.client.secretAccessKey = ""
      # The AWS session token.
      aws.client.sessionToken = ""
      # The endpoint with which the SDK should communicate allowing you to use a different S3 compatible service
      aws.client.endpointOverride = ""
      # The name of the registries. Used only for lookup.
      aws.glue.registryNames = ""
    }
  }
}

3.6.2 - Resources

Learn how to use the built-in resources provided by the extensions for AWS.

Here, you will find the list of resources supported by the extensions for AWS.

AWS Resources

More information:

3.6.2.1 - Schema for AWS Glue Schema Registry

Learn how to manage Schema in AWS Glue Schema Registry.

AwsGlueSchema resources are used to define the schemas you want to manage on your AWS Glue Schema registry. A AwsGlueSchema resource defines the schema, and the compatibility mode to be associated with a subject definition.

AwsGlueSchema

Specification

Here is the resource definition file for defining a AwsGlueSchema.

apiVersion: "aws.jikkou.io/v1"                  # The api version (required)
kind: "AwsGlueSchema"                           # The resource kind (required)    
metadata:
  name: <The name of the subject>               # The schema name (required)
  labels:
    glue.aws.amazon.com/registry-name:          # The registry name (required)
  annotations: { }
spec:
  compatibility: <compatibility>                # The schema compatibility level for this subject (required).
  dataFormat: <The data format>                 # Accepted values are: AVRO, PROTOBUF, JSON (required).
  schemaDefinition:
    $ref: <url or path>  # 
]

The metadata.name property is mandatory and specifies the name of the Subject.

Compatibility

Supported compatibility mode are:

  • BACKWARD (recommended) — Consumer can read both current and previous version.
  • BACKWARD_ALL — Consumer can read current and all previous versions.
  • FORWARD — Consumer can read both current and subsequent version.
  • FORWARD_ALL — Consumer can read both current and all subsequent versions.
  • FULL — Combination of Backward and Forward.
  • FULL_ALL — Combination of Backward all and Forward all.
  • NONE — No compatibility checks are performed.
  • DISABLED — Prevent any versioning for this schema

Example

Here is a simple example that shows how to define a single subject AVRO schema for type using the AwsGlueSchema resource type.

file: subject-user.yaml

---
apiVersion: "aws.jikkou.io/v1"
kind: "AwsGlueSchema"
metadata:
  name: "User"
  labels:
    glue.aws.amazon.com/registry-name: Test
  annotations:
    glue.aws.amazon.com/normalize-schema: true
spec:
  compatibility: "BACKWARD"
  dataFormat: "AVRO"
  schema:
    $ref: ./user-schema.avsc

file: user-schema.avsc

---
{
  "namespace": "example.avro",
  "type": "record",
  "name": "User",
  "fields": [
    {
      "name": "name",
      "type": [ "null", "string" ],
      "default": null,
    },
    {
      "name": "favorite_number",
      "type": [ "null", "int" ],
      "default": null
    },
    {
      "name": "favorite_color",
      "type": [ "null", "string" ],
      "default": null
    }
  ]
}

Alternatively, we can directly pass the Avro schema as follows :

file: subject-user.yaml

---
apiVersion: "aws.jikkou.io/v1"
kind: "AwsGlueSchema"
metadata:
  name: "User"
  labels: { }
  annotations:
    glue.aws.amazon.com/normalize-schema: true
spec:
  compatibility: "BACKWARD"
  dataFormat: "AVRO"
  schema: |
    {
      "namespace": "example.avro",
      "type": "record",
      "name": "User",
      "fields": [
        {
          "name": "name",
          "type": [ "null", "string" ],
          "default": null
        },
        {
          "name": "favorite_number",
          "type": [ "null", "int" ],
          "default": null
        },
        {
          "name": "favorite_color",
          "type": [ "null", "string"],
          "default": null
        }
      ]
    }    

3.6.3 - Validations

Learn how to use the built-in validations provided by the extensions for AWS.

Jikkou ships with the following built-in validations:

No validation

3.6.4 - Annotations

Learn how to use the metadata annotations provided by the extensions for AWS.

Here, you will find information about the annotations provided by the AWS extension for Jikkou.

List of built-in annotations

glue.aws.amazon.com/created-time

The date and time the schema was created.

The annotation is automatically added by Jikkou.

glue.aws.amazon.com/updated-time

The date and time the schema was updated.

The annotation is automatically added by Jikkou.

glue.aws.amazon.com/registry-name

The name of the registry.

The annotation is automatically added by Jikkou.

glue.aws.amazon.com/registry-arn

The Amazon Resource Name (ARN) of the registry.

The annotation is automatically added by Jikkou.

glue.aws.amazon.com/schema-arn

The Amazon Resource Name (ARN) of the schema.

The annotation is automatically added by Jikkou.

glue.aws.amazon.com/schema-version-id

The SchemaVersionId of the schema version.

The annotation is automatically added by Jikkou.

glue.aws.amazon.com/use-canonical-fingerprint

This annotation can be used to use a canonical fingerprint to compare schemas (only supported for Avro schema).

3.7 - Apache Iceberg

Learn how to use the Jikkou extension provider for Apache Iceberg — manage Iceberg namespaces, tables, and views as code.

Here, you will find information to use the Apache Iceberg extensions.

More information:

3.7.1 - Configuration

Learn how to configure the extensions for Apache Iceberg.

Here, you will find the configuration properties for the Apache Iceberg extension.

Configuration

The Apache Iceberg extension connects to an Iceberg catalog. You configure it through the Jikkou client configuration property jikkou.provider.iceberg.

Example (JDBC catalog with PostgreSQL):

jikkou {
  provider.iceberg {
    enabled = true
    type = io.jikkou.iceberg.IcebergExtensionProvider
    config = {
      # Required — type of Iceberg catalog.
      # Accepted values: rest, hive, jdbc, glue, nessie, hadoop
      catalogType = "jdbc"

      # The catalog name used to identify this catalog instance.
      catalogName = "default"

      # The URI of the catalog endpoint (REST URL, Hive Metastore URI, JDBC URL, etc.)
      catalogUri = "jdbc:postgresql://localhost:5432/iceberg"

      # The warehouse root location (e.g., local path, S3 bucket path, HDFS path)
      warehouse = "/tmp/iceberg-warehouse"

      # Extra catalog-specific properties passed directly to CatalogUtil.buildIcebergCatalog()
      catalogProperties {
        jdbc.user     = "iceberg"
        jdbc.password = "iceberg"
      }

      # Enable verbose debug logging for catalog operations (default: false)
      debugLoggingEnabled = false
    }
  }
}

Configuration Properties

PropertyTypeRequiredDefaultDescription
catalogTypeStringyesIceberg catalog type: rest, hive, jdbc, glue, nessie, hadoop
catalogNameStringnodefaultThe catalog instance name
catalogUriStringnoCatalog endpoint URI (REST API URL, Hive Metastore thrift URI, JDBC URL, Nessie server URL)
warehouseStringnoWarehouse root location (e.g., s3://bucket/warehouse, /tmp/iceberg)
catalogPropertiesMapnoAdditional catalog properties forwarded verbatim to the Iceberg CatalogUtil
debugLoggingEnabledBooleannofalseEnable debug-level logging for catalog operations

Catalog Types

JDBC Catalog (PostgreSQL)

Stores catalog metadata (namespaces, table specs) in a relational database. The PostgreSQL JDBC driver is bundled in the Jikkou CLI distribution.

config = {
  catalogType = "jdbc"
  catalogUri  = "jdbc:postgresql://localhost:5432/iceberg"
  warehouse   = "/tmp/iceberg-warehouse"
  catalogProperties {
    jdbc.user     = "iceberg"
    jdbc.password = "iceberg"
  }
}

REST Catalog

Connects to any Iceberg REST Catalog API (e.g., Polaris, Gravitino, Unity Catalog):

config = {
  catalogType = "rest"
  catalogUri  = "https://polaris.example.com/api/catalog"
  warehouse   = "s3://my-bucket/warehouse"
  catalogProperties {
    rest.signing-name   = "execute-api"
    rest.signing-region = "us-east-1"
  }
}

Hive Metastore

Connects to an Apache Hive Metastore (requires iceberg-hive-metastore on the classpath):

config = {
  catalogType = "hive"
  catalogUri  = "thrift://hive-metastore:9083"
  warehouse   = "hdfs://namenode:8020/user/hive/warehouse"
}

AWS Glue

Connects to AWS Glue Data Catalog (requires iceberg-aws on the classpath):

config = {
  catalogType = "glue"
  warehouse   = "s3://my-bucket/warehouse"
  catalogProperties {
    glue.region = "us-east-1"
  }
}

Nessie

Nessie exposes a standard Iceberg REST catalog endpoint at /iceberg. Using catalogType = "rest" is recommended because it relies only on iceberg-core (always bundled in the Jikkou CLI). The catalogType = "nessie" variant requires the optional iceberg-nessie JAR on the classpath.

# Recommended: use Nessie's built-in Iceberg REST endpoint
config = {
  catalogType = "rest"
  catalogUri  = "http://nessie:19120/iceberg"
  warehouse   = "s3://my-bucket/warehouse"
  catalogProperties {
    prefix = "main"   # Nessie branch
  }
}

Controller Settings

The table and view controllers expose additional options to control reconciliation behaviour. These are set inside the provider config block.

Table Controller

PropertyTypeDefaultDescription
delete-orphansBooleanfalseDrop tables that exist in the catalog but are not defined in any resource
delete-orphan-columnsBooleanfalseDrop columns present in the live table but absent from the spec
delete-purgeBooleanfalsePurge underlying data files when dropping a table (irreversible)
tables.deletion.excludeList<Pattern>[]Regex patterns — matching table names are never deleted

View Controller

PropertyTypeDefaultDescription
delete-orphansBooleanfalseDrop views that exist in the catalog but are not defined in any resource
views.deletion.excludeList<Pattern>[]Regex patterns — matching view names are never deleted

Example:

jikkou {
  provider.iceberg {
    enabled = true
    type = io.jikkou.iceberg.IcebergExtensionProvider
    config = {
      catalogType = "rest"
      catalogUri  = "http://localhost:8181"
      warehouse   = "s3://my-bucket/warehouse"

      # Table reconciliation safety settings
      delete-orphans        = false
      delete-orphan-columns = false
      delete-purge          = false

      # Never delete tables whose name starts with "audit_"
      tables.deletion.exclude = ["^audit_.*"]

      # Never delete views whose name starts with "v_core_"
      views.deletion.exclude = ["^v_core_.*"]
    }
  }
}

3.7.2 - Resources

Learn how to use the built-in resources provided by the extensions for Apache Iceberg.

Here, you will find the list of resources supported by the extensions for Apache Iceberg.

Iceberg Resources

The Apache Iceberg extension provides the following resource types:

ResourceDescription
IcebergNamespaceManage namespaces (databases) in an Iceberg catalog
IcebergTableManage tables with schema evolution, partitioning, and sort order
IcebergViewManage SQL views backed by one or more dialect-specific queries

More information:

3.7.2.1 - Iceberg Namespace

Learn how to manage Apache Iceberg Namespaces.

IcebergNamespace resources are used to define the namespaces (databases) you want to manage in your Iceberg catalog. A namespace groups tables and carries metadata as key/value properties.

IcebergNamespace

Specification

Here is the resource definition file for defining an IcebergNamespace.

apiVersion: "iceberg.jikkou.io/v1beta1"  # The api version (required)
kind: "IcebergNamespace"                  # The resource kind (required)
metadata:
  name: <namespace name>                  # Dot-separated namespace path (required)
  labels: { }
  annotations: { }
spec:
  properties:                             # Namespace-level metadata (optional)
    <key>: <value>

The metadata.name property is mandatory. Nested namespaces are expressed using dot notation — for example, analytics.events creates a namespace events inside analytics.

Example

file: iceberg-namespaces.yaml

---
apiVersion: "iceberg.jikkou.io/v1beta1"
kind: "IcebergNamespace"
metadata:
  name: "analytics"
spec:
  properties:
    owner: "data-team"
    environment: "production"
---
apiVersion: "iceberg.jikkou.io/v1beta1"
kind: "IcebergNamespace"
metadata:
  name: "analytics.events"
spec:
  properties:
    owner: "data-team"
    team: "platform"

IcebergNamespaceList

If you need to define multiple namespaces (e.g., using a template), it may be easier to use an IcebergNamespaceList resource.

Specification

apiVersion: "iceberg.jikkou.io/v1beta1"  # The api version (required)
kind: "IcebergNamespaceList"              # The resource kind (required)
metadata: { }
items: [ ]                                # An array of IcebergNamespace

Example

---
apiVersion: "iceberg.jikkou.io/v1beta1"
kind: "IcebergNamespaceList"
items:
  - metadata:
      name: "analytics"
    spec:
      properties:
        owner: "data-team"
  - metadata:
      name: "analytics.events"
    spec:
      properties:
        owner: "platform-team"

3.7.2.2 - Iceberg Table

Learn how to manage Apache Iceberg Tables, including schema evolution.

IcebergTable resources are used to define the tables you want to manage in your Iceberg catalog. An IcebergTable resource defines the schema, partition layout, sort order, and table-level properties. Jikkou performs safe schema evolution — adding, renaming, updating, and dropping columns — without data loss.

IcebergTable

Specification

Here is the resource definition file for defining an IcebergTable.

apiVersion: "iceberg.jikkou.io/v1beta1"  # The api version (required)
kind: "IcebergTable"                      # The resource kind (required)
metadata:
  name: <namespace>.<table>              # Fully qualified table name (required)
  labels: { }
  annotations: { }
spec:
  location: <storage path>               # Override the default table location (optional)
  schema:
    identifierFields:                    # Primary-key columns for MERGE/UPSERT semantics (optional)
      - <column name>
    columns:                             # Ordered list of columns (required)
      - name: <column name>              # Column name (required)
        type: <type>                     # Column type (required) — see type reference below
        required: <true|false>           # Whether the column is non-nullable (default: false)
        doc: <description>              # Documentation string (optional)
        default: <value>                 # Initial default value, immutable after creation (optional)
        writeDefault: <value>            # Write-default for absent values, updatable (optional)
        previousName: <old name>         # Triggers a safe rename instead of drop+add (optional)
  partitionFields:                       # Partition layout (optional)
    - sourceColumn: <column>             # Source column to partition on (required)
      transform: <transform>             # Partition transform (required) — see transforms below
      name: <partition field name>       # Custom partition field name (optional)
  sortFields:                            # Default write sort order (optional)
    - column: <column>                   # Column name (mutually exclusive with term)
      term: <expression>                 # Sort expression e.g. bucket[16](user_id) (mutually exclusive with column)
      direction: <asc|desc>              # Sort direction (default: asc)
      nullOrder: <first|last>            # Null placement (default: last)
  properties:                            # Table-level metadata properties (optional)
    <key>: <value>

Column Types

Jikkou maps a set of type strings to the underlying Iceberg types:

Type stringIceberg typeNotes
booleanBooleanType
int / integerIntegerType
longLongType
floatFloatType
doubleDoubleType
dateDateType
timeTimeType
timestampTimestampType (without tz)
timestamptzTimestampType (with tz)
stringStringType
uuidUUIDType
binaryBinaryType
fixed[N]FixedType(N)e.g. fixed[16]
decimal(P,S)DecimalType(P,S)e.g. decimal(18,2)

Complex types (struct, list, map) can be expressed as nested objects using the following format:

Struct:

type:
  type: "struct"
  fields:
    - name: "field_name"
      type: "string"       # Any type (primitive or nested)
      required: true        # default: false
      doc: "description"    # optional

List:

type:
  type: "list"
  elementType: "string"     # Any type (primitive or nested)
  elementRequired: false    # default: false

Map:

type:
  type: "map"
  keyType: "string"         # Any type (primitive or nested)
  valueType: "long"         # Any type (primitive or nested)
  valueRequired: false      # default: false

Partition Transforms

TransformExampleDescription
identityidentityPartition by the exact column value
yearyearExtract year from a date/timestamp column
monthmonthExtract year-month
daydayExtract calendar date
hourhourExtract date-hour
bucket[N]bucket[16]Hash-bucket into N buckets
truncate[W]truncate[8]Truncate string or integer to width W
voidvoidAlways-null partition (marks dropped fields) — not yet supported

Schema Evolution

Jikkou applies schema changes in a safe, deterministic order:

  1. Incompatible change check — verify annotation before proceeding
  2. Renames — processed first to preserve Iceberg field IDs
  3. Column additions — new columns appended
  4. Column updates — type promotion, documentation, nullability, write-default changes
  5. Column deletions — processed after updates to avoid conflicts
  6. Identifier field changes
  7. Schema commit — all schema changes are committed atomically
  8. Partition spec replacement
  9. Sort order replacement
  10. Properties update

Safe Column Rename

To rename a column without losing its Iceberg field ID (which would break existing readers), set the previousName field to the old column name:

columns:
  - name: "user_identifier"   # new name
    previousName: "user_id"   # old name — triggers a rename, not drop+add
    type: "long"
    required: true

Incompatible Changes

By default, Jikkou rejects type changes that are not safe promotions (e.g., int → long is safe; string → int is not). To allow incompatible changes on a specific resource, set the annotation iceberg.jikkou.io/allow-incompatible-changes: "true".


Examples

Simple table with day partitioning

file: iceberg-page-views.yaml

---
apiVersion: "iceberg.jikkou.io/v1beta1"
kind: "IcebergTable"
metadata:
  name: "analytics.events.page_views"
spec:
  schema:
    columns:
      - name: "event_id"
        type: "uuid"
        required: true
        doc: "Unique event identifier"
      - name: "user_id"
        type: "long"
        required: true
        doc: "The user who triggered the event"
      - name: "page_url"
        type: "string"
        required: true
        doc: "URL of the viewed page"
      - name: "event_time"
        type: "timestamptz"
        required: true
        doc: "Timestamp when the event occurred (UTC)"
      - name: "duration_ms"
        type: "long"
        doc: "Time spent on the page in milliseconds"
  partitionFields:
    - sourceColumn: "event_time"
      transform: "day"
  sortFields:
    - column: "event_time"
      direction: "asc"
      nullOrder: "last"
  properties:
    write.format.default: "parquet"
    write.parquet.compression-codec: "zstd"

Table with bucket partitioning and identifier fields

file: iceberg-orders.yaml

---
apiVersion: "iceberg.jikkou.io/v1beta1"
kind: "IcebergTable"
metadata:
  name: "analytics.events.orders"
  annotations:
    iceberg.jikkou.io/allow-incompatible-changes: "false"
spec:
  schema:
    columns:
      - name: "order_id"
        type: "long"
        required: true
        doc: "Unique order identifier"
      - name: "customer_id"
        type: "long"
        required: true
        doc: "Customer who placed the order"
      - name: "order_date"
        type: "date"
        required: true
        doc: "Date the order was placed"
      - name: "status"
        type: "string"
        required: true
        doc: "Order status"
      - name: "total_amount"
        type: "decimal(18,2)"
        required: true
        doc: "Total order amount"
    identifierFields:
      - "order_id"
  partitionFields:
    - sourceColumn: "order_date"
      transform: "month"
    - sourceColumn: "customer_id"
      transform: "bucket[16]"
  sortFields:
    - column: "order_date"
      direction: "desc"
      nullOrder: "last"
    - column: "customer_id"
      direction: "asc"
      nullOrder: "last"
  properties:
    write.format.default: "parquet"

IcebergTableList

If you need to define multiple tables (e.g., using a template), it may be easier to use an IcebergTableList resource.

Specification

apiVersion: "iceberg.jikkou.io/v1beta1"  # The api version (required)
kind: "IcebergTableList"                  # The resource kind (required)
metadata: { }
items: [ ]                                # An array of IcebergTable

Example

---
apiVersion: "iceberg.jikkou.io/v1beta1"
kind: "IcebergTableList"
items:
  - metadata:
      name: "analytics.raw.clicks"
    spec:
      schema:
        columns:
          - name: "click_id"
            type: "uuid"
            required: true
          - name: "ts"
            type: "timestamptz"
            required: true
      partitionFields:
        - sourceColumn: "ts"
          transform: "day"

  - metadata:
      name: "analytics.raw.impressions"
    spec:
      schema:
        columns:
          - name: "impression_id"
            type: "uuid"
            required: true
          - name: "ts"
            type: "timestamptz"
            required: true
      partitionFields:
        - sourceColumn: "ts"
          transform: "day"

3.7.2.3 - Iceberg View

Learn how to manage Apache Iceberg Views.

IcebergView resources are used to define SQL views in your Iceberg catalog. A view is a logical definition backed by one or more SQL queries — the output schema is inferred by the engine and populated on collect.

IcebergView

Specification

Here is the resource definition file for defining an IcebergView.

apiVersion: "iceberg.jikkou.io/v1beta1"  # The api version (required)
kind: "IcebergView"                      # The resource kind (required)
metadata:
  name: <namespace>.<view>              # Fully qualified view name (required)
  labels: { }
  annotations: { }
spec:
  schema:                                # Output schema (read-only, inferred by the engine)
    columns:
      - name: <column name>
        type: <type>
        required: <true|false>
        doc: <description>
  queries:                               # SQL query definitions (at least one required)
    - sql: <SQL SELECT statement>        # The SQL defining the view (required)
      dialect: <dialect>                 # SQL dialect e.g. 'spark', 'trino', 'presto', 'hive' (required)
  defaultNamespace: <namespace>          # Default namespace for unqualified table references (optional)
  defaultCatalog: <catalog>              # Default catalog for unqualified table references (optional)
  properties:                            # View-level metadata properties (optional)
    <key>: <value>

Examples

Simple view with daily aggregation

file: iceberg-daily-page-stats.yaml

---
apiVersion: "iceberg.jikkou.io/v1beta1"
kind: "IcebergView"
metadata:
  name: "analytics.events.daily_page_stats"
spec:
  queries:
    - sql: >-
        SELECT
            CAST(event_time AS DATE) AS view_date,
            page_url,
            COUNT(*) AS view_count,
            COUNT(DISTINCT user_id) AS unique_users
        FROM analytics.events.page_views
        GROUP BY CAST(event_time AS DATE), page_url        
      dialect: "spark"
  defaultNamespace: "analytics.events"
  properties:
    comment: "Daily page view statistics aggregated from raw events"

View with multiple SQL dialects

file: iceberg-multi-dialect-view.yaml

---
apiVersion: "iceberg.jikkou.io/v1beta1"
kind: "IcebergView"
metadata:
  name: "analytics.events.active_users"
spec:
  queries:
    - sql: >-
        SELECT user_id, COUNT(*) AS event_count
        FROM analytics.events.page_views
        WHERE event_time >= current_date() - INTERVAL 30 DAYS
        GROUP BY user_id        
      dialect: "spark"
    - sql: >-
        SELECT user_id, COUNT(*) AS event_count
        FROM analytics.events.page_views
        WHERE event_time >= current_date - INTERVAL '30' DAY
        GROUP BY user_id        
      dialect: "trino"
  defaultNamespace: "analytics.events"
  properties:
    comment: "Users active in the last 30 days"

IcebergViewList

If you need to define multiple views (e.g., using a template), it may be easier to use an IcebergViewList resource.

Specification

apiVersion: "iceberg.jikkou.io/v1beta1"  # The api version (required)
kind: "IcebergViewList"                  # The resource kind (required)
metadata: { }
items: [ ]                               # An array of IcebergView

Example

---
apiVersion: "iceberg.jikkou.io/v1beta1"
kind: "IcebergViewList"
items:
  - metadata:
      name: "analytics.events.daily_page_stats"
    spec:
      queries:
        - sql: >-
            SELECT CAST(event_time AS DATE) AS view_date, page_url, COUNT(*) AS view_count
            FROM analytics.events.page_views
            GROUP BY CAST(event_time AS DATE), page_url            
          dialect: "spark"
      defaultNamespace: "analytics.events"

  - metadata:
      name: "analytics.events.active_users"
    spec:
      queries:
        - sql: >-
            SELECT user_id, COUNT(*) AS event_count
            FROM analytics.events.page_views
            WHERE event_time >= current_date() - INTERVAL 30 DAYS
            GROUP BY user_id            
          dialect: "spark"
      defaultNamespace: "analytics.events"

3.7.3 - Annotations

Learn how to use the metadata annotations provided by the extensions for Apache Iceberg.

Here, you will find information about the annotations provided by the Apache Iceberg extension for Jikkou.

List of built-in annotations

iceberg.jikkou.io/allow-incompatible-changes

Controls whether incompatible schema changes are allowed when reconciling an IcebergTable.

By default, Jikkou rejects type changes that cannot be safely promoted (for example, changing a column from string to int). Setting this annotation to "true" on a specific table resource lifts that restriction for that resource only.

metadata:
  annotations:
    iceberg.jikkou.io/allow-incompatible-changes: "true"

iceberg.jikkou.io/namespace-location

Read-only annotation populated by the namespace collector. Contains the storage location of a namespace as reported by the catalog. This annotation is set automatically when collecting existing namespaces and does not need to be specified in resource files.

iceberg.jikkou.io/table-location

Reserved annotation for the storage location of a table. Currently, the table location is stored in the spec.location field rather than as an annotation. This annotation key is defined for future use.

iceberg.jikkou.io/view-location

Read-only annotation populated by the view collector. Contains the storage location of a view as reported by the catalog. This annotation is set automatically when collecting existing views and does not need to be specified in resource files.

3.8 - Confluent Cloud

Learn how to use the Jikkou Extensions Provider for Confluent Cloud.

Here, you will find information to use the Confluent Cloud extensions.

More information:

3.8.1 - Configuration

Learn how to configure the extensions for Confluent Cloud.

Here, you will find the configuration for the Confluent Cloud extension.

Configuration

You can configure the properties to connect to Confluent Cloud through the Jikkou client configuration property jikkou.provider.confluent-cloud.

Example:

jikkou {
  provider.confluent-cloud {
    enabled = true
    type = io.jikkou.extension.confluent.ConfluentCloudExtensionProvider
    config = {
      # URL to the Confluent Cloud REST API (default: https://api.confluent.cloud)
      apiUrl = "https://api.confluent.cloud"
      # Confluent Cloud API Key (must be a Cloud API Key, not a Cluster API Key)
      apiKey = ${CONFLUENT_CLOUD_API_KEY}
      # Confluent Cloud API Secret
      apiSecret = ${CONFLUENT_CLOUD_API_SECRET}
      # CRN pattern used to scope role binding list operations
      crnPattern = ${CONFLUENT_CLOUD_CRN_PATTERN}
      # HTTP proxy URL, e.g. 'http://proxy.example.com:3128' (optional)
      proxyUrl = "http://proxy.example.com:3128"
      # Username for proxy Basic authentication (optional)
      proxyUsername = null
      # Password for proxy Basic authentication (optional)
      proxyPassword = null
      # Comma-separated hosts that bypass the proxy (optional)
      nonProxyHosts = "localhost,127.0.0.1"
      # Enable debug logging (default: false)
      debugLoggingEnabled = false
    }
  }
}

Configuration Properties

PropertyTypeRequiredDefaultDescription
apiUrlStringNohttps://api.confluent.cloudURL to the Confluent Cloud REST API.
apiKeyStringYesCloud API Key. Must be a Cloud API Key, not a Cluster API Key.
apiSecretStringYesCloud API Secret.
crnPatternStringYesCRN pattern to scope role binding list operations.
proxyUrlStringNoHTTP proxy URL, e.g. http://proxy.example.com:3128. When empty, JVM proxy system properties are used.
proxyUsernameStringNoUsername for proxy Basic authentication.
proxyPasswordStringNoPassword for proxy Basic authentication.
nonProxyHostsStringNoComma-separated hosts that bypass the proxy, e.g. localhost,*.internal.
debugLoggingEnabledBooleanNofalseEnable debug logging for REST API calls.

If proxyUrl is not set, Jikkou honors the standard JVM proxy system properties (-Dhttps.proxyHost, -Dhttp.proxyHost, -Dhttp.proxyUser, -Dhttp.proxyPassword, -Dhttp.nonProxyHosts), which can be supplied via JAVA_TOOL_OPTIONS. The OS-level http_proxy / https_proxy environment variables are not read by the JVM and have no effect.

Creating a Cloud API Key

Cloud API Keys can be created using the Confluent Cloud CLI:

confluent api-key create --resource cloud --description "Jikkou role binding management"

Important: You must use a Cloud API Key (organization-level), not a Cluster API Key. Cluster API Keys will result in a 401 Unauthorized error.

CRN Pattern

The crnPattern property is required and scopes all list operations to a specific part of your organization hierarchy. Examples:

ScopeCRN Pattern
Organizationcrn://confluent.cloud/organization=org-abc123
Environmentcrn://confluent.cloud/organization=org-abc123/environment=env-def456
Kafka Clustercrn://confluent.cloud/organization=org-abc123/environment=env-def456/cloud-cluster=lkc-789

3.8.2 - Resources

Learn about the resources supported by the Confluent Cloud extension.

Here, you will find the list of resources supported by the extension for Confluent Cloud.

3.8.2.1 - Role Bindings for Confluent Cloud

Learn how to manage RBAC Role Bindings in Confluent Cloud.

The RoleBinding resources are used to manage RBAC role bindings in Confluent Cloud. A RoleBinding resource defines which role is granted to a principal for a specific scope (identified by a CRN pattern).

RoleBinding

Specification

Here is the resource definition file for defining a RoleBinding.

---
apiVersion: "iam.confluent.cloud/v1"    # The api version (required)
kind: "RoleBinding"                     # The resource kind (required)
metadata:
  labels: { }
  annotations: { }
spec:
  principal: <>              # The principal (e.g., User:sa-abc123 or User:u-xyz789)
  roleName: <>               # The role name (e.g., CloudClusterAdmin, DeveloperRead)
  crnPattern: <>             # The Confluent Resource Name pattern (e.g., crn://confluent.cloud/...)

Fields

FieldTypeRequiredDescription
principalStringYesThe principal. Pattern: User:<user-id> or Group:<group-name>.
roleNameStringYesThe role to bind. See Confluent Cloud RBAC roles.
crnPatternStringYesThe Confluent Resource Name (CRN) pattern defining the scope of the binding.

Common Role Names

RoleDescription
OrganizationAdminFull access to the organization.
EnvironmentAdminFull access to an environment.
CloudClusterAdminFull access to a Kafka cluster.
DeveloperManageManage topics and schemas.
DeveloperReadRead from topics and view schemas.
DeveloperWriteWrite to topics and manage schemas.
ResourceOwnerFull access to a specific resource.

Example

Here is a simple example that shows how to define a single role binding using the RoleBinding resource type.

file: role-binding.yaml

---
apiVersion: "iam.confluent.cloud/v1"
kind: "RoleBinding"
metadata:
  labels: { }
  annotations: { }
spec:
  principal: "User:sa-abc123"
  roleName: "CloudClusterAdmin"
  crnPattern: "crn://confluent.cloud/organization=org-123/environment=env-456/cloud-cluster=lkc-789"

Usage

# List all role bindings
jikkou get ccloud-rbs

# Apply role bindings from a file
jikkou apply --files ./role-binding.yaml

# Delete orphan role bindings not defined in the file
jikkou apply --files ./role-binding.yaml -o delete-orphans=true

# Dry-run to preview changes without applying
jikkou diff --files ./role-binding.yaml

Metadata Labels

When listing role bindings, Jikkou automatically enriches each resource with metadata labels to help identify principals:

LabelDescription
confluent.cloud/principal-nameThe display name of the user or service account.
confluent.cloud/principal-emailThe email of the user (not set for service accounts).

Example output:

apiVersion: "iam.confluent.cloud/v1"
kind: "RoleBinding"
metadata:
  labels:
    confluent.cloud/principal-name: "Florian Hussonnois"
    confluent.cloud/principal-email: "florian@example.com"
  annotations:
    confluent.cloud/role-binding-id: "rb-NBl9kE"
spec:
  principal: "User:u-rrnm2g9"
  roleName: "OrganizationAdmin"
  crnPattern: "crn://confluent.cloud/organization=d497af93-23f5-434a-a008-0547797be410"

RoleBindingList

If you need to define multiple role bindings (e.g., using a template), it may be easier to use a RoleBindingList resource.

Specification

Here is the resource definition file for defining a RoleBindingList.

---
apiVersion: "iam.confluent.cloud/v1"    # The api version (required)
kind: "RoleBindingList"                 # The resource kind (required)
metadata: # (optional)
  labels: { }
  annotations: { }
items: [ ]                             # An array of RoleBinding

Example

---
apiVersion: "iam.confluent.cloud/v1"
kind: "RoleBindingList"
items:
  - spec:
      principal: "User:sa-abc123"
      roleName: "CloudClusterAdmin"
      crnPattern: "crn://confluent.cloud/organization=org-123/environment=env-456/cloud-cluster=lkc-789"
  - spec:
      principal: "User:sa-abc123"
      roleName: "DeveloperRead"
      crnPattern: "crn://confluent.cloud/organization=org-123/environment=env-456/cloud-cluster=lkc-789/kafka=lkc-789/topic=my-topic"
  - spec:
      principal: "User:u-xyz789"
      roleName: "OrganizationAdmin"
      crnPattern: "crn://confluent.cloud/organization=org-123"

Note: Role bindings are immutable in the Confluent Cloud API. If you need to change a role binding, delete the old one and create a new one. Jikkou only supports CREATE and DELETE operations (no UPDATE).

3.8.3 - Annotations

Learn how to use the metadata annotations provided by the extensions for Confluent Cloud.

Here, you will find information about the annotations provided by the Confluent Cloud extension for Jikkou.

List of built-in annotations

confluent.cloud/role-binding-id

Used by Jikkou.

The annotation is automatically added by Jikkou to store the Confluent Cloud role binding ID (e.g., rb-NBl9kE). This ID is used internally when deleting role bindings.