Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

EDU-1710: Integrations consistenecy #2485

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

EDU-1710: Integrations consistenecy #2485

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
144 changes: 116 additions & 28 deletions content/integrations/inbound/kafka-connector.textile
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,76 +1,164 @@
---
title: Ably Kafka Connector
meta_description: "The Ably Kafka Connector enables you to send data from Kafka to an Ably channel."
meta_description: "The Ably Kafka Connector sends data from Kafka to an Ably channel in realtime."
meta_keywords: "Kafka, Kafka Connector, channel"
languages:
- none
redirect_from:
- /docs/general/kafka-connector
---

The Ably Kafka Connector provides a ready-made integration between "Kafka":https://kafka.apache.org/ and Ably. It allows for realtime event distribution from Kafka to web, mobile, and IoT clients, over Ably's feature-rich, multi-protocol pub/sub channels. If instead you need to send data from Ably to Kafka, use a "Kafka rule":/docs/integrations/streaming/kafka.
The Ably Kafka Connector integrates "Kafka":https://kafka.apache.org/ with Ably for realtime event distribution to web, mobile, and IoT clients via Ably's pub/sub "channels":/docs/channels.

The Ably Kafka Connector is verified by Confluent as "Gold":https://www.confluent.io/hub/ably/kafka-connect-ably, following the guidelines set forth by Confluent's "Verified Integrations Program":https://www.confluent.io/verified-integrations-program/.
To send data from Ably to Kafka, use a "Kafka rule":/docs/integrations/streaming/kafka.

You can use the Ably Kafka Connector to send data from one or more "Kafka topics":https://developer.confluent.io/learn-kafka/apache-kafka/topics/ into Ably "channels":/docs/channels. The following diagram illustrates a typical deployment:
This connector is Confluent "Gold":https://www.confluent.io/hub/ably/kafka-connect-ably Verified, ensuring compliance with Confluent's "Verified Integrations Program":https://www.confluent.io/partners/connect/. You can use it to send data from one or more "Kafka topics":https://www.confluent.io/partners/connect/ to one or multiple Ably channels.

<a href="@content/diagrams/ably-kafka-connector-overview.png" target="_blank">
<img src="@content/diagrams/ably-kafka-connector-overview.png" style="width: 100%"
title="Click to enlarge" alt="Ably Kafka Connector overview">
</a>

h2(#how). How does the Ably Kafka Connector work?
The Ably Kafka Connector is a sink connector built on top of Kafka "Connect":/https://docs.confluent.io/platform/current/connect/index.html#how-kafka-connect-works. You can deploy it in two ways:
* Self-hosted - Run within your own infrastructure.
* Third-party hosted - Available on the Confluent Platform.

The Ably Kafka Connector is a sink connector built on top of Kafka Connect. It can be self-hosted or hosted with a third-party provider, the most common being the Confluent Platform. You can download it from either "GitHub":https://github.com/ably/kafka-connect-ably or "Confluent Hub":https://www.confluent.io/hub/ably/kafka-connect-ably and install it into your Kafka Connect workers.
Install the Ably Kafka Connector from:
* "GitHub":https://github.com/ably/kafka-connect-ably
* "Confluent Hub":https://www.confluent.io/hub/ably/kafka-connect-ably

The Ably Kafka Connector provides a ready-made integration between Kafka and Ably via your Ably API key. Once installed, you can configure the connector with your Ably API key to enable data from one or more Kafka topics to be published into a single Ably channel or multiple Ably channels. Events are then distributed in realtime to web, mobile, and IoT clients over feature-rich, multi-protocol pub/sub Ably channels optimized for last-mile delivery.
Once installed, configure it with your "Ably API key":/docs/auth#api-keys. This allows Kafka topics to publish data into Ably channels, making messages available in realtime for web, mobile, and IoT clients.

h2(#mapping-options). Mapping options
h2(#mapping). Mapping

The Ably Kafka Connector supports two mapping options:
The Ably Kafka Connector supports two mapping methods:
# "Static":#static - Assigns messages to a fixed Ably channel.
* "Pattern-based":#pattern - Dynamically assigns messages based on the topic or record key.

* Static
* Pattern-based
h3(#static). Static mapping

h3(#static-mapping). Static mapping
Static mapping assigns one or more Kafka topics to a single Ably channel. The channel name stays the same, regardless of the Kafka record.

Static mapping enables the connection of one or multiple Kafka topics to a single Ably channel. Here you provide a string for a configuration, such as for channel name. This stays the same no matter what record the connector is consuming. For example if you configure a channel with @channel="global"@, all Kafka records will be published through that channel. This is useful if, for example, you are a sports site broadcasting updates to millions of subscribers through a single channel. The following configuration provides an example of this:
For example, a sports website streaming live updates can set @channel = basketball@, ensuring all Kafka records, regardless of their Kafka topic, publish to the basketball channel. The following example maps all Kafka topics to the basketball channel:

```
```[javascript]
channel = basketball
message.name = news_update
```

h3(#pattern-mapping). Pattern-based mapping
h3(#pattern). Pattern-based mapping

Pattern-based mapping gives you the ability to map many Kafka topics to many Ably channels following set rules. Pattern-based mapping enables each message, a Kafka record, to be checked to determine the recipient, and then determine which channel to send it through. Pattern-based mapping provides customers with the ability to adjust their configuration by interpolating across record key and record topic values. For example, where the channel name is configured as @channel="channel_#{topic}"@, the message will be published to a channel according to the topic name. There are two configurations that support pattern-based mapping: @channel@ and @message.name@.
Pattern-based mapping dynamically maps multiple Kafka topics to different Ably channels. Each Kafka record determines the target channel. Additionally each message is published to the Ably channel corresponding to its Kafka topic. For example, setting @channel = channel_#{topic}@ routes Kafka messages to a channel matching their topic name.

```
The following configuration maps Kafka topics to Ably channels based on the topic name:

```[javascript]
channel = channel_#{topic}
message.name = message_#{key}
```

For an @events@ topic, the channel would translate as @channel_events@. For a @remove@ key, the @message.name@ would be configured as @message_remove@.
h3(#mixed). Mixed mapping

It is important to note that you can also use static mapping in conjunction with pattern-based mapping. For example, you could define the channel as @channel_#{topic}@, but @message.name@ as @single_message@. In this instance, the channel will be mapped by pattern and the message name will be mapped statically, as shown in the following configuration:
Mixed mapping combines static and pattern-based mapping. For example, you can dynamically map the channel while keeping the message name static, ensuring messages are routed to topic-specific channels while maintaining a consistent message name. The following configuration maps Kafka topics to Ably channels based on the topic name, while keeping the message name static:

```
```[javascript]
channel = channel_#{topic}
message.name = single_message
```

h2(#schema). Publishing messages with a schema
h2(#publish). Publish messages with a schema

The Ably Kafka Connector supports messages that include schema information. It converts these messages to JSON before publishing them to Ably. You can also use "Kafka schema Registry":https://docs.confluent.io/platform/current/schema-registry/connect.html and supported converters.

h3(#configure). Configure a schema conversion

If messages on the Kafka topic are serialized using "Avro":https://developer.confluent.io/confluent-tutorials/kafka-producer-application/kafka/#produce-avro-records, and schemas are registered in a Schema Registry, configure the connector to convert Avro to JSON.

Set the following properties in your Kafka Connect configuration:

```[text]
value.converter=io.confluent.connect.avro.AvroConverter
value.converter.schema.registry.url=https://<your-schema-registry-host>
```

This configuration ensures Kafka messages are correctly deserialized and transformed before reaching Ably.

h3(#test). Test Avro messages with Docker compose

If you're running the Ably Kafka Connector locally using Docker Compose, you can test Avro-serialized messages using the kafka-avro-console-producer CLI.

Run the following command:

<pre><code>
docker-compose exec -T schema-registry kafka-avro-console-producer \
--topic topic1 \
--broker-list kafka:9092 \
--property key.schema='{"type":"string"}' \
--property parse.key=true \
--property key.separator=":" \
--property value.schema='{"type":"record","name":"myrecord","fields":[{"name":"count","type":"int"}]}' \
--property schema.registry.url=http://schema-registry:8081 <<EOF
"key1":{"count":1} EOF
</code></pre>

This Docker Compose command is used to produce a Kafka record with both an Avro-encoded key and value. It relies on Kafka Schema Registry to validate the schema before sending the message to the Kafka topic named @topic1@.

When you subscribe to the Ably channel, you should receive the following message:

```[json]

{
"clientId": "Ably-Kafka-Connector",
"connectionId": "VSuDXysgaz",
"data": {
"count": 1
},
"extras": {
"kafka": {
"key": "AAAAAKEIa2V5MQ=="
}
},
"id": "-868034334:0:351",
"name": "topic1_message",
"timestamp": 1653923422360
}
```

The following explains how Ably processes the Kafka message structure:

* The Avro-formatted key appears in base64 encoding under the @extras.kafka.key@ field.
* The actual message data, which contains @count: 1@, is extracted and formatted as JSON.
* The @name@ field, @topic1_message@, corresponds directly to the Kafka topic.

<aside data-type='note'>
<p>Note that configuring the schema registry and appropriate key or value converters also enables referencing of record field data within "Dynamic Channel Configuration.":https://github.com/ably/kafka-connect-ably?tab=readme-ov-file#dynamic-channel-configuration </p>
</aside>

h2(#publish). Configure the Kafka connector

The Ably Kafka Connector supports "messages that contain schema information":https://github.com/ably/kafka-connect-ably#publishing-messages-with-schema by converting them to JSON before publishing them to Ably. To learn how to use schema registry and supported converters, see "Using Kafka Connect with Schema Registry":https://docs.confluent.io/platform/current/schema-registry/connect.html.
The Ably Kafka Connector sends Kafka messages to Ably channels in real time. You can configure it to define:

h2(#install). Install the connector
* Kafka topics to pull data from.
* Ably channels to send data to.

The Ably Kafka Connector is available to install from "GitHub":https://github.com/ably/kafka-connect-ably or "Confluent Hub":https://www.confluent.io/hub/ably/kafka-connect-ably.
|_. Installation Method |_. Configuration Steps |
| @Docker@ | Create a @docker-compose-connector.properties@ file inside the @/config@ directory. \n - An example file is already available in the repository. |
| @Single connect worker@ | - Provide a configuration file as a command-line argument when running the worker. |
| @Distributed connect workers@ | - Use the Confluent REST API @/connectors@ endpoint. \n - Pass the configuration as JSON. |

View detailed instructions for installation on "GitHub":https://github.com/ably/kafka-connect-ably#install.
h3. Connector configuration properties

h2(#configure). Configure the connector
You must configure these core properties to get the connector working.

Configure the connector to set "properties":https://github.com/ably/kafka-connect-ably#configuration such as which Kafka topics to send data from, and which Ably channel should receive the data.
|_. Property |_. Description |
| @channel@ | The Ably channel to which messages are published. Supports "Dynamic Channel Configuration:":#dynamic-channel-configuration |
| @client.key@ | An Ably API key used for authentication. Must have *publish* capability for the specified channel. |
| @client.id@ | The Ably client ID the connector uses. Defaults to @"kafka-connect-ably-example"@. |
| @name@ | A globally unique name for the connector. Defaults to @"ably-channel-sink"@. |
| @topics@ | A comma-separated list of Kafka topics to publish from. |
| @tasks.max@ | The maximum number of tasks the connector should run. Defaults to @1@. |
| @connector.class@ | The class name for the connector. Must be a subclass of @org.apache.kafka.connect.connector@. Defaults to @io.ably.kafka.connect.ChannelSinkConnector@. |

Note that the configuration method differs depending on whether you are running a single or distributed set of "connect workers":https://docs.confluent.io/home/connect/self-managed/userguide.html#configuring-and-running-workers.
<aside data-type='note'>
<p>For a full list of configuration options, refer to the Ably Kafka Connector "properties.":https://github.com/ably/kafka-connect-ably?tab=readme-ov-file#configuration-properties</p>
</aside>
Loading