Schema Registry

• 1: Configuration
• 2: Resources
• 2.1: Schema Registry Subjects
• 3: Validations
• 4: Annotations

1 - Configuration

Configuration

You can configure the properties to be used to connect the SchemaRegistry service through the Jikkou client configuration property jikkou.provider.schemaregistry.

Example:

jikkou {
  provider.schemaregistry {
    enabled = true
    type = io.streamthoughts.jikkou.schema.registry.SchemaRegistryExtensionProvider
    config = {
      # Comma-separated list of URLs for schema registry instances that can be used to register or look up schemas
      url = "http://localhost:8081"
      # The name of the schema registry implementation vendor - can be any value
      vendor = generic
      # Method to use for authenticating on Schema Registry. Available values are: [none, basicauth, ssl]
      authMethod = none
      # Use when 'schemaRegistry.authMethod' is 'basicauth' to specify the username for Authorization Basic header
      basicAuthUser = null
      # Use when 'schemaRegistry.authMethod' is 'basicauth' to specify the password for Authorization Basic header
      basicAuthPassword = null
      # Enable debug logging
      debugLoggingEnabled = false

      # Ssl Config: Use when 'authMethod' is 'ssl'
      # The location of the key store file.
      sslKeyStoreLocation = "/certs/registry.keystore.jks"
      # The file format of the key store file.
      sslKeyStoreType = "JKS"
      # The password for the key store file.
      sslKeyStorePassword = "password"
      # The password of the private key in the key store file.
      sslKeyPassword = "password"
      # The location of the trust store file.
      sslTrustStoreLocation = "/certs/registry.truststore.jks"
      # The file format of the trust store file.
      sslTrustStoreType = "JKS"
      # The password for the trust store file.
      sslTrustStorePassword = "password"
      # Specifies whether to ignore the hostname verification.
      sslIgnoreHostnameVerification = true
    }
  }
}

2 - Resources

Schema Registry Resources

More information:

2.1 - Schema Registry Subjects

Definition Format of SchemaRegistrySubject

Below is the overall structure of the SchemaRegistrySubject resource.

apiVersion: "schemaregistry.jikkou.io/v1beta2"  # The api version (required)
kind: "SchemaRegistrySubject"                   # The resource kind (required)
metadata:
  name: <The name of the subject>               # (required)
  labels: { }
  annotations: { }
spec:
  schemaRegistry:
    vendor: <vendor_name>                       # (optional) The vendor of the SchemaRegistry, e.g., Confluent, Karapace, etc
  compatibilityLevel: <compatibility_level>     # (optional) The schema compatibility level for this subject.
  schemaType: <The schema format>               # (required) Accepted values are: AVRO, PROTOBUF, JSON
  schema:
    $ref: <url or path>  # 
  references: # Specifies the names of referenced schemas (optional array).
    - name: <string>                             # The name for the reference.
      subject: <string>                          # The subject under which the referenced schema is registered.
      version: <string>                          # The exact version of the schema under the registered subject.
]

Metadata

The metadata.name property is mandatory and specifies the name of the Subject.

Specification

To use the SchemaRegistry default values for the compatibilityLevel you can omit the property.

Example

Here is a simple example that shows how to define a single subject AVRO schema for type using the SchemaRegistrySubject resource type.

file: subject-user.yaml

---
apiVersion: "schemaregistry.jikkou.io/v1beta2"
kind: "SchemaRegistrySubject"
metadata:
  name: "User"
  labels: { }
  annotations:
    schemaregistry.jikkou.io/normalize-schema: true
spec:
  compatibilityLevel: "FULL_TRANSITIVE"
  schemaType: "AVRO"
  schema:
    $ref: ./user-schema.avsc

file: user-schema.avsc

---
{
  "namespace": "example.avro",
  "type": "record",
  "name": "User",
  "fields": [
    {
      "name": "name",
      "type": [ "null", "string" ],
      "default": null,
    },
    {
      "name": "favorite_number",
      "type": [ "null", "int" ],
      "default": null
    },
    {
      "name": "favorite_color",
      "type": [ "null", "string" ],
      "default": null
    }
  ]
}

Alternatively, we can directly pass the Avro schema as follows :

file: subject-user.yaml

---
apiVersion: "schemaregistry.jikkou.io/v1beta2"
kind: "SchemaRegistrySubject"
metadata:
  name: "User"
  labels: { }
  annotations:
    schemaregistry.jikkou.io/normalize-schema: true
spec:
  compatibilityLevel: "FULL_TRANSITIVE"
  schemaType: "AVRO"
  schema: |
    {
      "namespace": "example.avro",
      "type": "record",
      "name": "User",
      "fields": [
        {
          "name": "name",
          "type": [ "null", "string" ],
          "default": null
        },
        {
          "name": "favorite_number",
          "type": [ "null", "int" ],
          "default": null
        },
        {
          "name": "favorite_color",
          "type": [ "null", "string"],
          "default": null
        }
      ]
    }    

SchemaRegistrySubjectList

If you need to manage multiple Schemas at once (e.g. using a template), it may be more suitable to use the resource collection SchemaRegistrySubjectList.

Specification

Here the resource definition file for defining a SchemaRegistrySubjectList.

apiVersion: "schemaregistry.jikkou.io/v1beta2"  # The api version (required)
kind: "SchemaRegistrySubjectList"      # The resource kind (required)
metadata: # (optional)
  labels: { }
  annotations: { }
items: [ ]                             # The array of SchemaRegistrySubject

3 - Validations

Jikkou ships with the following built-in validations:

Subject

SchemaCompatibilityValidation

type = io.streamthoughts.jikkou.schema.registry.validation.SchemaCompatibilityValidation

The SchemaCompatibilityValidation allows testing the compatibility of the schema with the latest version already registered in the Schema Registry using the provided compatibility-level.

AvroSchemaValidation

The AvroSchemaValidation allows checking if the specified Avro schema matches some specific avro schema definition rules;

type = io.streamthoughts.jikkou.schema.registry.validation.AvroSchemaValidation

Configuration

4 - Annotations

List of built-in annotations

schemaregistry.jikkou.io/url

Used by jikkou.

The annotation is automatically added by Jikkou to describe the SchemaRegistry URL from which a subject schema is retrieved.

schemaregistry.jikkou.io/schema-version

Used by jikkou.

The annotation is automatically added by Jikkou to describe the version of a subject schema.

schemaregistry.jikkou.io/schema-id

Used by jikkou.

The annotation is automatically added by Jikkou to describe the version of a subject id.

schemaregistry.jikkou.io/normalize-schema

Used on: schemaregistry.jikkou.io/v1beta2:SchemaRegistrySubject

This annotation can be used to normalize the schema on SchemaRegistry server side. Note, that Jikkou will attempt to normalize AVRO and JSON schema.

See: Confluent SchemaRegistry API Reference

schemaregistry.jikkou.io/permanante-delete

Used on: schemaregistry.jikkou.io/v1beta2:SchemaRegistrySubject

The annotation can be used to specify a hard delete of the subject, which removes all associated metadata including the schema ID. The default is false. If the flag is not included, a soft delete is performed. You must perform a soft delete first, then the hard delete.

See: Confluent SchemaRegistry API Reference

schemaregistry.jikkou.io/use-canonical-fingerprint

This annotation can be used to use a canonical fingerprint to compare schemas (only supported for Avro schema).