Performance issues and faster implementation in Scala

[pmeheut]

Hi,
Just to let you know, I encountered CPU and memory issues when running the Python code on long time series with big windows.
Also, the Python multiprocessing is efficient on some processor such as Core i7/i9 but not at all on Xeon multiprocessor machines.

So I implemented the algorithm in Scala and it runs 5 to 15 times faster depending on the computer (tested up to a 48 Cpu AWS machine).
It is based on a June version of the Python code but it could easily be adapted and using Scalapy, I can even read and write Panda dataframes.

Let me know if someone is interested.

[Joshwani]

Hi,

Thanks for sharing. I'd be interested in looking into this. Do you have any shareable code?

Thanks!

Josh

[pmeheut]

You can look at it here: https://github.com/pmeheut/lppls_scala
Code is in src/main/scala/Lppls.scala

Main class sample is ComputeIndicators: it reads a pandas dataframe from a feather file in the home directory called spx.feather
containing a field SPX_INDEX_PX_LAST (i.e. S&P) and writes the result in lppls.feather.

Let me know if you need more informations.

[Joshwani]

Nice, thanks I'll take a look!