Using multiple sources with Transactional.sink

@Vasily wrote:

Hello,

I’ve noticed that it’s impossible to use Transactional.sink with data coming from different sources (different topics from the same kafka cluster) because there’s a check in TransactionalProducerStage that ensures that "Transaction batch must contain messages from a single source".

I wonder why does this check exist. Because kafka definitely supports transactions spanning multiple topics. I’ve written a simple implementation of multi-topic transnational sink (not really a sink, it’s really a .groupedWithin().foreach()), just as a proof of concept, and it seems to work fine, producing correct results.

What’s the catch here? What’s the reason for this restriction? Is there a way to have a Transactional.sink() that takes messages from different topics/partitions in the same cluster?

Posts: 1

Participants: 1

Read full topic