Transform stage occasionally crashes with >16 threads #75
Labels
bug
Something isn't working
Comments
|
Seems like there's an issue reading the flyer output in this |
|
Thanks for reporting! I think this is a problem with the Flye polisher, somehow it generated @Mikes92 and @Ge0rges if you could send us the corresponding flye folders with input and output - this would help a lot to reproduce and fix the issue on the Flye side! Thanks, |
|
Hi Misha, thanks for getting back on this. If i understand correctly, you'd like the flye_input and flye_output folders for those clusters/edges? or do you want all flye_input and all flye_output files, (if so, they're about 5GB each). thanks, |
|
Hi MIke - Just the failed cluster would be sufficient, i.e. Thank you! |
|
flye_consensus_edge_248_s1_2010009_7143.zip Let me know if this is what you needed! |
|
Thank you! It was indeed a corner case for Flye - I've just pushed a fix to the Flye repository, and updated the submodule in Strainy. Could you please try the version from |
|
tried again from the Mike |
|
Thanks! Looks like it went ahead, but this new error message is not super informative unfortunately, because of python multiprocessing quirks. I've pushed a new version to If it does crash, could you also try running the original command line from scratch with a separate output folder? |
|
Hey! Sorry for this delay, it took a little while to run all this on my end. I pulled the updated version and re-ran the same command lines with same input/output adding I then re-ran from the beginning with new input/output files with Finally, I reran the entire workflow with new input/output with Mike |
|
Thank you! I'm glad that it finished with 1 thread. For the crashed run, do you mind sharing @atabeerk what do you think? since running in 1 thread works, but multiple threads consistently crashes, we may have a race condition in the transform part somehow? |
|
Hey! Here are the requested log files. I'll send you an email to clarify the best way to get you the raw data. Mike |
|
Thanks a lot! |
|
Hi Mike, Would you mind trying the new version in Misha |
|
Hey Misha,
Of course! I’ll get this set up today, Hopefully it will all be done
running by Monday.
Talk to you soon!
Mike
Mike Sadler
Pronouns: he/him/his
Graduate Student
School of Oceanography
University of Washington
…On Fri, Mar 22, 2024 at 6:44 AM Mikhail Kolmogorov ***@***.***> wrote:
Hi Mike,
Would you mind trying the new version in rc_2024 with both single and
multiple threads, restarting with transform and from scratch? You'll need
to pull git submodule update again. Thanks a lot!
Misha
—
Reply to this email directly, view it on GitHub
<https://urldefense.com/v3/__https://github.com/katerinakazantseva/stRainy/issues/75*issuecomment-2015141309__;Iw!!K-Hz7m0Vt54!jOSh2F63uZVwzQ24IVa0EAkbEMbBUY4ah7HBW24idJ0Te3zs7sUEOf63sT0HzqEORdVV9AIKSzCumOW1_2ilXAd6$>,
or unsubscribe
<https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/A2BOA76DGM24TWWG5YTATQDYZQYVJAVCNFSM6AAAAABEFTO4JOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAMJVGE2DCMZQHE__;!!K-Hz7m0Vt54!jOSh2F63uZVwzQ24IVa0EAkbEMbBUY4ah7HBW24idJ0Te3zs7sUEOf63sT0HzqEORdVV9AIKSzCumOW1_-DvDYeC$>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
|
Hi Misha, Sorry for this delay. I reinstalled stRainy (but kept the same conda environment) and was re-running on my dataset. However, I only realized today that I accidentally reinstalled strainy from the So I reinstalled from the Mike |
|
Hi Mike, Sorry forgot to mention that - it is a new argument that controls the sensitivity of strain clustering. Please use |
|
Hi Misha, I pulled from Mike
|
|
@Mikes92 thanks for checking! we are trying to debug this on our side. If you have a moment - could you please try running with 10 threads? It seems it might be a hardcoded limit of 16 threads somewhere in Python multiprocessing module.. |
|
And one more request.. Which python version are you using? It seems that this issue may have been fixed in 3.11. If you have older version - do you mind trying with Py 3.11+ and 30 threads once you have a chance? |
|
Hey Misha, Just finished trying to run these. I can confirm that I was running on I then tried changing to Mike |
|
Thank you! Good to know that Linking the CPython change for the sake of completeness: python/cpython#101225 Thanks a lot for testing! In the meantime, we'll put a hard limit of 16 threads for the transform stage, until we can transition to Python 3.11. |
mikolmogorov
changed the title
mikolmogorov
changed the title
|
of course! Happy to help. Please let me know if there is anything else I can do for you, and I'll also let you know when this dataset becomes public. I think it would make a really great/challenging test case for stRainy! Mike |
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hello!
I'm trying to run stRainy on a collection of 14 related genomes to see if it will be useful to me in the future. i sequenced these 14 genomes independently and know what their genomes should look like in the end. Maybe this dataset is too large and i should be using strainyMAG to reduce the number of edges? the error below
When I look into the files it looks like other flye_consensus_edge folders have the correct base_coverage.bed.gz file, however the folder at issue contains a bubbles_1.fasta file instead. a polished_1.fasta also exists. However, both these files (bubbles_1.fasta, and polished_1.fasta) are empty
installed with conda, initial assembly was performed with the suggested flye parameters, and I was able to successfully run stRainy on the test dataset that came with.
running in Linux
command :
./strainy.py -g /scratch/mikes92/strainy_sar11/assembly_graph.gfa -q /scratch/mikes92/strainy_sar11/uisw_sar11_reads.fastq.gz -o /scratch/mikes92/strainy_sar11/strainy_test -m nano -t 20Thanks!
Mike
The text was updated successfully, but these errors were encountered: