This is the multi-page printable view of this section. Click here to print.

Return to the regular view of this page.

Schema Registry

Lean how to use the Jikkou Extension Provider for Schema Registry.

Here, you will find information to use the Schema Registry extensions.

More information:

1 - Configuration

Learn how to configure the extensions for SchemaRegistry.

Here, you will find the list of resources supported for SchemaRegistry.

Configuration

You can configure the properties to be used to connect the SchemaRegistry service through the Jikkou client configuration property jikkou.schemaRegistry.

Example:

jikkou {
  schemaRegistry {
    # Comma-separated list of URLs for schema registry instances that can be used to register or look up schemas
    url = "http://localhost:8081"
    # The name of the schema registry implementation vendor - can be any value
    vendor = generic
    # Method to use for authenticating on Schema Registry. Available values are: [none, basicauth, ssl]
    authMethod = none
    # Use when 'schemaRegistry.authMethod' is 'basicauth' to specify the username for Authorization Basic header
    basicAuthUser = null
    # Use when 'schemaRegistry.authMethod' is 'basicauth' to specify the password for Authorization Basic header
    basicAuthPassword = null
    # Enable debug logging
    debugLoggingEnabled = false

    # Ssl Config: Use when 'authMethod' is 'ssl'
    # The location of the key store file.
    sslKeyStoreLocation = "/certs/registry.keystore.jks"
    # The file format of the key store file.
    sslKeyStoreType = "JKS"
    # The password for the key store file.
    sslKeyStorePassword = "password"
    # The password of the private key in the key store file.
    sslKeyPassword = "password"
    # The location of the trust store file.
    sslTrustStoreLocation = "/certs/registry.truststore.jks"
    # The file format of the trust store file.
    sslTrustStoreType = "JKS"
    # The password for the trust store file.
    sslTrustStorePassword = "password"
    # Specifies whether to ignore the hostname verification.
    sslIgnoreHostnameVerification = true
  }
}

2 - Resources

Learn how to use the built-in resources provided by the extensions for Schema Registry.

Here, you will find the list of resources supported for Schema Registry.

Schema Registry Resources

More information:

2.1 - Schema Registry Subjects

Learn how to manage SchemaRegistry Subjects.

SchemaRegistrySubject resources are used to define the subject schemas you want to manage on your SchemaRegistry. A SchemaRegistrySubject resource defines the schema, the compatibility level, and the references to be associated with a subject version.

SchemaRegistrySubject

Specification

Here is the resource definition file for defining a SchemaRegistrySubject.

apiVersion: "schemaregistry.jikkou.io/v1beta2"  # The api version (required)
kind: "SchemaRegistrySubject"                   # The resource kind (required)
metadata:
  name: <The name of the subject>               # (required)
  labels: { }
  annotations: { }
spec:
  schemaRegistry:
    vendor: <vendor_name>                       # (optional) The vendor of the SchemaRegistry, e.g., Confluent, Karapace, etc
  compatibilityLevel: <compatibility_level>     # (optional) The schema compatibility level for this subject.
  schemaType: <The schema format>               # (required) Accepted values are: AVRO, PROTOBUF, JSON
  schema:
    $ref: <url or path>  # 
  references: # Specifies the names of referenced schemas (optional array).
    - name: <>                                  # The name for the reference.
      subject: <>                               # The subject under which the referenced schema is registered.
      version: <>                               # The exact version of the schema under the registered subject.
]

The metadata.name property is mandatory and specifies the name of the Subject.

To use the SchemaRegistry default values for the compatibilityLevel you can omit the property.

Example

Here is a simple example that shows how to define a single subject AVRO schema for type using the SchemaRegistrySubject resource type.

file: subject-user.yaml

---
apiVersion: "schemaregistry.jikkou.io/v1beta2"
kind: "SchemaRegistrySubject"
metadata:
  name: "User"
  labels: { }
  annotations:
    schemaregistry.jikkou.io/normalize-schema: true
spec:
  compatibilityLevel: "FULL_TRANSITIVE"
  schemaType: "AVRO"
  schema:
    $ref: ./user-schema.avsc

file: user-schema.avsc

---
{
  "namespace": "example.avro",
  "type": "record",
  "name": "User",
  "fields": [
    {
      "name": "name",
      "type": [ "null", "string" ],
      "default": null,
    },
    {
      "name": "favorite_number",
      "type": [ "null", "int" ],
      "default": null
    },
    {
      "name": "favorite_color",
      "type": [ "null", "string" ],
      "default": null
    }
  ]
}

Alternatively, we can directly pass the Avro schema as follows :

file: subject-user.yaml

---
apiVersion: "schemaregistry.jikkou.io/v1beta2"
kind: "SchemaRegistrySubject"
metadata:
  name: "User"
  labels: { }
  annotations:
    schemaregistry.jikkou.io/normalize-schema: true
spec:
  compatibilityLevel: "FULL_TRANSITIVE"
  schemaType: "AVRO"
  schema: |
    {
      "namespace": "example.avro",
      "type": "record",
      "name": "User",
      "fields": [
        {
          "name": "name",
          "type": [ "null", "string" ],
          "default": null
        },
        {
          "name": "favorite_number",
          "type": [ "null", "int" ],
          "default": null
        },
        {
          "name": "favorite_color",
          "type": [ "null", "string"],
          "default": null
        }
      ]
    }    

3 - Validations

Learn how to use the built-in validations provided by the extensions for Schema Registry.

Jikkou ships with the following built-in validations:

Subject

SchemaCompatibilityValidation

type = io.streamthoughts.jikkou.schema.registry.validation.SchemaCompatibilityValidation

The SchemaCompatibilityValidation allows testing the compatibility of the schema with the latest version already registered in the Schema Registry using the provided compatibility-level.

AvroSchemaValidation

The AvroSchemaValidation allows checking if the specified Avro schema matches some specific avro schema definition rules;

type = io.streamthoughts.jikkou.schema.registry.validation.AvroSchemaValidation

Configuration

NameTypeDescriptionDefault
fieldsMustHaveDocBooleanVerify that all record fields have a doc propertyfalse
fieldsMustBeNullableBooleanVerify that all record fields are nullablefalse
fieldsMustBeOptionalBooleanVerify that all record fields are optionalfalse

4 - Annotations

Learn how to use the metadata annotations provided by the extensions for Schema Registry.

Here, you will find information about the annotations provided by the Schema Registry extension for Jikkou.

List of built-in annotations

schemaregistry.jikkou.io/url

Used by jikkou.

The annotation is automatically added by Jikkou to describe the SchemaRegistry URL from which a subject schema is retrieved.

schemaregistry.jikkou.io/schema-version

Used by jikkou.

The annotation is automatically added by Jikkou to describe the version of a subject schema.

schemaregistry.jikkou.io/schema-id

Used by jikkou.

The annotation is automatically added by Jikkou to describe the version of a subject id.

schemaregistry.jikkou.io/normalize-schema

Used on: schemaregistry.jikkou.io/v1beta2:SchemaRegistrySubject

This annotation can be used to normalize the schema on SchemaRegistry server side. Note, that Jikkou will attempt to normalize AVRO and JSON schema.

See: Confluent SchemaRegistry API Reference

schemaregistry.jikkou.io/permanante-delete

Used on: schemaregistry.jikkou.io/v1beta2:SchemaRegistrySubject

The annotation can be used to specify a hard delete of the subject, which removes all associated metadata including the schema ID. The default is false. If the flag is not included, a soft delete is performed. You must perform a soft delete first, then the hard delete.

See: Confluent SchemaRegistry API Reference