fix: Serialize walkAsync so isBinaryFile doesn't throw EMFILE: too many open files. #286
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Hey! Thanks for the great library – ran into an issue and have a small fix proposal.
I know this has already been raised here: #248
However, for #reasons we need to disable ASAR packaging for our product. This means that
walkAsync
has to crawl all ofnode_modules
looking for binary files. BecausegetFilePathIfBinary
actually opens and reads the first few thousand bytes of every file, the recursivePromise.all
quickly throws withEMFILE: too many open files
, no matter how high I set the open file limit viaulimit
.Conveniently, just putting things into a simple for loop fixes things by serializing the file reads, giving Node a chance to release each file before reading the next one. IIRC, this shouldn't impact performance much because of how Node's FS access works. Anecdotally, we also have this change deployed to our CI and it's performing very quickly without any errors!