Thread starvation

Hi

I use akka with persistence (akka-persistence-jdbc version 5.0.0) and cluster version 2.6.10, scala version 2.13

After two days of running a 2 node cluster setup with raspberry pi compute module’s, the system is insanely slow i.e. nothing really happens.
I get the following warn log:
[WARN ] [com.zaxxer.hikari.pool.HikariPool] [] - slick.db - Thread starvation or clock leap detected (housekeeper delta=58s277ms542µs362ns).
The worst delta is:
[WARN ] [com.zaxxer.hikari.pool.HikariPool] [] - slick.db - Thread starvation or clock leap detected (housekeeper delta=6h51m20s16ms852µs738ns).

also, if I restart the cluster, the akka scheduler often starts to trigger events too late after one day already.
Since this lasts until I restart the cluster, I exclude a clock leap problem (it’s very unlikely to have so many clock leaps and I checked the clock multiple times).

I monitored CPU and memory of the OS but this is ok (less than 50% total CPU and still enough available memory).
I also analyzed a lot the logs but could not find anything which would point out a thread starvation problem root.

So next, I took thread dumps of the jvm and analyzed them using jstack.review.
Since I’m not an expert with analyzing thread dumps or akka dispatchers, I cannot tell whether it is normal to have such high numbers at the end of the dispatcher name and whether the overview states some problem(s):

highNumberForDispatcher1388×187 40.3 KB

overview1816×315 43.3 KB

Here is the configuration in application.conf file for the custom dispatcher:

# Applied to actors doing blocking input/output operations.
  akka.blocking-io-dispatcher {
    type = Dispatcher
    executor = "thread-pool-executor"
    thread-pool-executor {
      fixed-pool-size = 32
    }
    # Throughput defines the maximum number of messages to be
    # processed per actor before the thread jumps to the next actor.
    # Set to 1 for as fair as possible.
    throughput = 1
  }

Questions

1. Is it ok to have a blocking-io-dispatcher-84632 or does this point to a thread starvation problem?
2. Does the overview look fine or is this problematic for an akka application?
3. What else could I do to approach the problem described above?

Thank you very much,
Simon

1 post - 1 participant

Read full topic