Actor processes messages way too late, dispatcher configuration problem?

Hi

We use akka actors with persistence (postgresql database connected with akka-persistence-jdbc version 3.5.3) and cluster version 2.6.9 with scala version 2.13.

There are two actors A and B which live on the same node. I send messages from actor A to actor B and log:

• right before the tell() from A to B happens
• on receiving the message in actor B

Actor A is dispatched by the default-dispatcher and extends AbstractActor.
Actor B is dispatched by a custom blocking-persistence-dispatcher and extends AbstractPersistentActor. This actor persists the received message (but first I log the message).

After a while (not so long), the messages are sent in time as expected but actor B processes them way too late (9 seconds later). I think this problem could be related to a problem with the dispatchers.

Here is the whole dispatcher configuration setup:

# Applied to actors doing blocking input/output operations.
  blocking-io-dispatcher {
    type = Dispatcher
    executor = "thread-pool-executor"
    thread-pool-executor {
      fixed-pool-size = 32
    }
    # Throughput defines the maximum number of messages to be
    # processed per actor before the thread jumps to the next actor.
    # Set to 1 for as fair as possible.
    throughput = 1
  }

  # Applied to persistent actors
  blocking-persistence-dispatcher {
    type = Dispatcher
    executor = "thread-pool-executor"
    thread-pool-executor {
      fixed-pool-size = 32
    }
    # Throughput defines the maximum number of messages to be
    # processed per actor before the thread jumps to the next actor.
    # Set to 1 for as fair as possible.
    throughput = 1
  }

and here is the persistence configuration (with *** instead of sensitive information):

slick {
  profile = "slick.jdbc.PostgresProfile$"
  db {
    url = "jdbc:postgresql://localhost:5432/***"
    user = ***
    password = ***
    driver = "org.postgresql.Driver"

    // hikariCP
    numThreads = 4
    maxConnections = 4 // limit maxConnections to numThreads to avoid deadlocks
  }
}

1. is this a sensible configuration setup if run on a raspberry pi compute module 3?
2. how can I determine whether it’s the dispatcher’s fault?
3. how can I fix this issue?

I created gc logs and analyzed them with gceasy.io and everything is fine. Pausing the application cannot be the root of the problem (max pause time is 100 ms).

Thanks!

1 post - 1 participant

Read full topic