Replies: 1 comment
-
Will this be implemented? I often encounter situations where I need to load multiple topics using regex, and I have to use Fluentd, which isn't as fast. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
It would be tremendously useful to be able to subscribe to multiple topics via a wildcard. The config file would then look something like this:
In a world where you may have X number of "channel-topics", you could elegantly subscribe to all of them via a wildcard. The current approach, where you have to list a comma-separated list of topics, is not practical for a large amount of topics.
Also, architecturally speaking, having a wildcard is extremely useful when more topics get created in the future, because they'll never be missed. Example: say you introduce 5 new "channel-topics" in the future; with the current setup, unless you remember to go and add them to the comma-separated list, they will not be consumed by fluentbit. But if you use wildcard chars, they would automatically be monitored. If you follow good naming conventions in your project, your topics will have well-defined names, so this would be a no-brainer.
Lastly, wildcard subscription should be supported by Kafka libraries:
https://github.com/dpkp/kafka-python/blob/68c8fa4ad01f8fef38708f257cb1c261cfac01ab/kafka/consumer/group.py#L717
Beta Was this translation helpful? Give feedback.
All reactions