Kafka Trigger Node Stops Listening After Some Time

[hyusufyavuz]

Bug Description

The Kafka Trigger node in n8n stops listening to Kafka messages after a certain period of time. Once the node stops listening, it does not resume or reconnect to Kafka unless the workflow is manually restarted. This issue makes it unreliable for long-running workflows that rely on continuous message consumption from Kafka.

The Kafka topic is storing data from Debezium CDC. It's common for the topic to receive no messages for up to half a day. Despite this, the Kafka Trigger node fails to resume listening when new messages eventually arrive after a long idle period.

Additionally, the Kafka connection is established using SASL authentication, which may affect how connections are managed and re-established.

When monitoring the consumer groups with Conduktor, it can be observed that after some time, the consumer groups fall into an empty state, and the lag starts increasing, indicating that the consumer is no longer actively consuming messages from the topic.

To Reproduce

1. Set up a Kafka Trigger node in an n8n workflow to listen to messages from a Kafka topic receiving data from Debezium CDC.
2. Deploy the workflow and leave it running for an extended period of time (several hours or longer).
3. Allow the Kafka topic to have periods of no new messages for several hours.
4. Observe that the node stops receiving messages when new data eventually arrives, with no errors or logs indicating the cause.
5. Manually restart the workflow to restore message consumption.

Expected behavior

The Kafka Trigger node should continuously listen to Kafka messages without interruption unless explicitly stopped by the user, even when the Kafka topic remains idle for extended periods.

Operating System

Ubuntu 22.04

n8n Version

1.72.1

Node.js Version

18.20.5

Database

PostgreSQL

Execution mode

queue

[Joffcom]

Hey @hyusufyavuz,

We have created an internal ticket to look into this which we will be tracking as "GHC-1142"