We noticed that when the writes volume is heavy during the night, Alpakka Elastic Search fails to write to Elastic Search. There are no errors logged when this happens, not in the supervision strategy or in the (!result.success) block. We use the following versions,
“com.lightbend.akka” %% “akka-stream-alpakka-elasticsearch” % “2.0.2”
“com.typesafe.akka” %% “akka-stream-kafka” % “2.0.6”
Akka version = “2.5.31”
On the client write side, we have tested with buffer sizes set to 20, 100, 1000 and 1. Of all these combinations only when the buffer is set to 1, we did not lose any documents on ES. Is this a known issue? What is the recommended buffer size?
Our stream includes a simple logic which
- Listens to Kafka -> 2. validate -> 3. send to Kafka -> 4. also write to ES -> 5. commit offset back to kafka
1 post - 1 participant