resource consumption increase since 2.0.1

[seru71]

First of all thank you for the great work with the cgpwgs pipeline. We have been using version 2.0.1 and consider moving to 2.1.0 now, mainly due to the ASCAT/Caveman bug, but also performance improvements.

In the test runs we noticed however a considerable increase in resources needed to complete the analysis:

1. runtime jumped from ~24 to ~40 hrs on 32 threads
2. memory consumption raised as well (can report on precise numbers if needed)

Is there a reason for such increase in resource use considering changes introduced in 2.1.0 (since 2.0.1)?

[keiranmraine]

Although performance enhancements were made to some elements of the container we later identified that the SNV post processing update for fragment based counting had some performance issues.

We have an improved version in the stand-alone containers but have not done a test cycle to release a new version of the combined containers.

There are 2 PRs of note:

• Fixes an edgecase bug: Feature/strand count bug cgpCaVEManPostProcessing#36
• Performance: Change to use low-level api where possible and reduce fetch calls cgpCaVEManPostProcessing#37

I'll raise an item in our management meeting regarding an updated release and resource.

(edit: removed ye-olde-english style typo)

[seru71]

Thank you. We'll wait with the update then until you have released this new version.

…

On Tue, May 19, 2020 at 10:11 AM Keiran Raine ***@***.***> wrote: Although performance enhancements were made to some elements of the container we later identified that the SNV post processing update for fragment based counting had some performance issues. We have an improved version in thee stand-alone containers but have not done a test cycle to release a new version of the combined containers. There are 2 PRs of note: - Fixes an edgecase bug: cancerit/cgpCaVEManPostProcessing#36 <cancerit/cgpCaVEManPostProcessing#36> - Performance: cancerit/cgpCaVEManPostProcessing#37 <cancerit/cgpCaVEManPostProcessing#37> I'll raise an item in our management meeting regarding an updated release and resource. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAV4RK3VOBKFDT7DJIQQG4TRSI5MXANCNFSM4NEXNP6Q> .

[keiranmraine]

Would you mind providing information as to your project/institute?

[seru71]

The project is run at http://zawlab.home.amu.edu.pl

…

On Tue, May 19, 2020 at 12:09 PM Keiran Raine ***@***.***> wrote: Would you mind providing information as to your project/institute? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#51 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAV4RKZHYFWYVN345XTTDXLRSJLG3ANCNFSM4NEXNP6Q> .