Aws

Lean how to use the Jikkou Extensions Providers for “Aws”.

1: Configuration

2: Resources

2.1: Schema for AWS Glue Schema Registry

3: Validations

4: Annotations

Here, you will find information to use the Aws extensions (Jikkou v0.36).

More information:

1 - Configuration

Learn how to configure the extensions for AWS.

Here, you will find the list of resources supported by the extension for AWS.

Configuration

You can configure the properties to be used to connect the AWS services through the Jikkou client configuration property jikkou.providers.

Example:

jikkou {
  providers = [
    # AWS
    {
      enabled = true
      name = aws
      type = io.streamthoughts.jikkou.aws.AwsExtensionProvider
      config = {
        # The AWS S3 Region, e.g. us-east-1
        aws.client.region = ""          
        # The AWS Access Key ID.
        aws.client.accessKeyId = ""    
        # The AWS Secret Access Key.
        aws.client.secretAccessKey = ""
        # The AWS session token.
        aws.client.sessionToken = ""
        # The endpoint with which the SDK should communicate allowing you to use a different S3 compatible service
        aws.client.endpointOverride = ""
        # The name of the registries. Used only for lookup.
        aws.glue.registryNames = ""
      }
    },
  ]
}

2 - Resources

Learn how to use the built-in resources provided by the extensions for AWS.

Here, you will find the list of resources supported by the extensions for AWS.

AWS Resources

More information:

2.1 - Schema for AWS Glue Schema Registry

Learn how to manage Schema in AWS Glue Schema Registry.

AwsGlueSchema resources are used to define the schemas you want to manage on your AWS Glue Schema registry. A AwsGlueSchema resource defines the schema, and the compatibility mode to be associated with a subject definition.

`AwsGlueSchema`

Specification

Here is the resource definition file for defining a AwsGlueSchema.

apiVersion: "aws.jikkou.io/v1"                  # The api version (required)
kind: "AwsGlueSchema"                           # The resource kind (required)    
metadata:
  name: <The name of the subject>               # The schema name (required)
  labels:
    glue.aws.amazon.com/registry-name:          # The registry name (required)
  annotations: { }
spec:
  compatibility: <compatibility>                # The schema compatibility level for this subject (required).
  dataFormat: <The data format>                 # Accepted values are: AVRO, PROTOBUF, JSON (required).
  schemaDefinition:
    $ref: <url or path>  # 
]

The metadata.name property is mandatory and specifies the name of the Subject.

Compatibility

Supported compatibility mode are:

BACKWARD (recommended) — Consumer can read both current and previous version.
BACKWARD_ALL — Consumer can read current and all previous versions.
FORWARD — Consumer can read both current and subsequent version.
FORWARD_ALL — Consumer can read both current and all subsequent versions.
FULL — Combination of Backward and Forward.
FULL_ALL — Combination of Backward all and Forward all.
NONE — No compatibility checks are performed.
DISABLED — Prevent any versioning for this schema

Example

Here is a simple example that shows how to define a single subject AVRO schema for type using the AwsGlueSchema resource type.

file: subject-user.yaml

---
apiVersion: "aws.jikkou.io/v1"
kind: "AwsGlueSchema"
metadata:
  name: "User"
  labels:
    glue.aws.amazon.com/registry-name: Test
  annotations:
    glue.aws.amazon.com/normalize-schema: true
spec:
  compatibility: "BACKWARD"
  dataFormat: "AVRO"
  schema:
    $ref: ./user-schema.avsc

file: user-schema.avsc

---
{
  "namespace": "example.avro",
  "type": "record",
  "name": "User",
  "fields": [
    {
      "name": "name",
      "type": [ "null", "string" ],
      "default": null,
    },
    {
      "name": "favorite_number",
      "type": [ "null", "int" ],
      "default": null
    },
    {
      "name": "favorite_color",
      "type": [ "null", "string" ],
      "default": null
    }
  ]
}

Alternatively, we can directly pass the Avro schema as follows :

file: subject-user.yaml

---
apiVersion: "aws.jikkou.io/v1"
kind: "AwsGlueSchema"
metadata:
  name: "User"
  labels: { }
  annotations:
    glue.aws.amazon.com/normalize-schema: true
spec:
  compatibility: "BACKWARD"
  dataFormat: "AVRO"
  schema: |
    {
      "namespace": "example.avro",
      "type": "record",
      "name": "User",
      "fields": [
        {
          "name": "name",
          "type": [ "null", "string" ],
          "default": null
        },
        {
          "name": "favorite_number",
          "type": [ "null", "int" ],
          "default": null
        },
        {
          "name": "favorite_color",
          "type": [ "null", "string"],
          "default": null
        }
      ]
    }

3 - Validations

Learn how to use the built-in validations provided by the extensions for AWS.

Jikkou ships with the following built-in validations:

No validation

4 - Annotations

Learn how to use the metadata annotations provided by the extensions for AWS.

Here, you will find information about the annotations provided by the AWS extension for Jikkou.