In our use case, we have hundreds of streams consuming from hundreds of Kafka topics (thousands of partitions in total) and writing that data to just one topic. In the downstream topic (where our streams are producing the data) we see a huge number of TCP connections getting created. We did some analysis and found out that the number of connections is equal to the number of partitions the stream is reading from (here we have thousands of partitions as explained above).
We are using alpakka kafka 3.0.1 jar. We are using Producer.flexiFlow to produce the messages.
1 post - 1 participant