Actor slowness in Akka Cluster when multiple threads send messages to ShardRegion with large Message size

We implemented Akka clustering with cluster sharding for a use case.
When we doing load testing for that, We created 1000 entity actors in a Node by cluster Sharding. And we sends messages to that entity actors from a Proxy Node (Shard Region Proxy on other Node).

Kindly check the update Section on the question too

What we done

Using
akka.cluster.shard.remember.entities=ddata

And dedicated dispatcher given for entity actors with 1000 threads too.
We produced from single thread and multiple threads to check the behavior.

The Issue we faced

When sending from a single thread in Proxy Node the delay time in Actor Receive method in Entity actor is very minimal.

But when sending messages to the entity actors with multiple threads (same no of messages) the delay time is high (>100ms) with respect to no of threads using.

Which is similar when we use ActorSelection in Proxy Node. When checking for an entity actor availability with ActorSelection in Proxy node. Based on no of parallel threads that delay too gets increasing.

Need some clarification on

How the internal mechanism works when sending messages to entity actors in ShardRegion proxy with multiple threads.

• Is there any performance tuning has to be done or i missing any important configuration.?
• Why the delay is increasing when more no of parallel threads tries to send messages to Cluster entity actors compared to single thread sending.?

Lets say an example…

I have 1000 entity actors spitted evenly to 3 shards.
And I’m sending messages to entity actors with 100 threads, each threads sends a message to 10 entity actors.
After sending a message successfully then only next message will be send.
So at a time there are only 100 messages will send to that ShardRegion in parallel.

I will get reply from an entity actor after successful reply by using Pattern.ask

After getting reply only next message send to another entity.

First messages are received by the entity actors instantly but when consecutive messages the delay between proxy to entity Actor on receive is getting increase.

Update on the Question

The actual reason for the delay is Message size.When i send 1 KB message with 100 threads there is not much delay.
But When i send 100KB message from 100 Threads the delay is high.
By reading some blogs and akka documentation and reference.conf find out some params

akka {
 remote {
  netty.tcp {
   # Sets the send buffer size of the Sockets,
   # set to 0b for platform default
   send-buffer-size = 256000b

   # Sets the receive buffer size of the Sockets,
   # set to 0b for platform default
   receive-buffer-size = 256000b
  }
 }
}

We have increased this from 2 MB to even 100MB but there is no gain in the performance in large message size.

Is that i’m missing any important configuration ?

2 posts - 2 participants

Read full topic