How Do I Pause consumption from kafka source when using groupBy/groupedWithin

I have a flow that looks something like the code below and what is happening is I don’t have great control of memory usage. What are some options to cause backpressure and pause retrieving more data from the source.

        commitableKafkaSource
            // TODO: add some sort of stoplight that causes backpressure under certain condition
            .map(messageIntake::convert)
            .filter(Optional::isPresent)
            .map(Optional::get)
            .groupBy(Integer.MAX_VALUE, Pair::first)
            .map(Pair::second)
            .groupedWithin(config.getMaxBatchSizeInAkkaStream(),
                Duration.ofSeconds(config.getMaxTimeWindowBySeconds()))
            .map(batchProcessor::processBatch)
            .filter(Optional::isPresent)
            .map(Optional::get)
            .mergeSubstreams()
            .toMat(KafkaSink.committableSink(config.getOutputStream()), Keep.both())
            .run(materializer);

4 posts - 2 participants

Read full topic