Optimize CPU usage for sttp backend with cats when retrying

[vuhainguyen]

Hi all,
We build our Retryingbackend using cats effect and sttp backend as below:
https://github.com/cognitedata/cognite-sdk-scala/blob/master/src/main/scala/com/cognite/sdk/scala/sttp/RetryingBackend.scala
And we use Spark to run jobs that send request using the backend above. I noticed that during a period that we have many failed request that we need to retry, the % CPU usage of Spark executors is down because the thread is blocked to sleep and wait til next moment of retry so this slow down our Spark application. I wonder if there's a way to optimize, telling the thread that "hey, you know that you will do nothing so instead of waiting for the next retry, why don't you you take some thing else to do".
I tried to use this flag ActiveProcessorCount as part of the spark.executor.extraJavaOptions when submitting Spark, but doesn't look like it helps.
Any ideas on how to optimize the code of RetryingBackend maybe?
Thanks in advance for your help :)

[armanbilge]

I don't think there's a problem in the code you linked. Are you doing unsafeRunSync() somewhere?

[vuhainguyen]

unsafeRunSync() is used mainly in tests only, here's how the RetryingBackend is used as part of our Spark Data Source to create the client.

[armanbilge]

I see multiple unsafeRunSync() in here:
https://github.com/cognitedata/cdp-spark-datasource/blob/9f8a7ec8c48fd0baba55b4809f0143ee8305d48d/src/main/scala/cognite/spark/v1/udf/CogniteUdfs.scala

unsafeRunSync() completely blocks a thread while it waits for a result.

[vuhainguyen]

ah those CogniteUdfs is for experimental only, it's not used in our Spark application 😁
Maybe this makeQueue. But it's a good hint with unsafeRunSync(), I will look to see if I can find it somewhere else in our code

[vuhainguyen]

Anything below that look specious for you?
https://github.com/cognitedata/cdp-spark-datasource/blob/9f8a7ec8c48fd0baba55b4809f0143ee8305d48d/src/main/scala/cognite/spark/v1/DefaultSource.scala#L304
https://github.com/cognitedata/cdp-spark-datasource/blob/1e0161e41ce797641541291692eb8f5fe26c2a92/src/main/scala/cognite/spark/v1/StreamIterator.scala#L47
https://github.com/cognitedata/cdp-spark-datasource/blob/a37972422c7cf3ff2c63459c61a05d226322028a/src/main/scala/cognite/spark/v1/SdkV1Relation.scala#L61 (this has an import CdpConnector._)

[ironbat77]

FWIW we tried to use this flag with a higher number of processor requested and got better usage on the CPU stats :)
https://learn.microsoft.com/en-us/azure/developer/java/containers/kubernetes#understand-jvm-available-processors

[armanbilge]

You may find this tuning guide helpful :)

https://typelevel.org/cats-effect/docs/core/starvation-and-tuning