Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improved build speed of main.js #1411

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

keVIMena98
Copy link

Running the build command to generate 10,000 images can take a pretty long while to finish. This is true especially if you are running this program o a low budget computer with very little memory.

I noticed that the saveImage function is utilizing a writeFileSync in node is an asynchronous function that blocks the entire thread until the file is done writing to disc. To resolve this just converted this to a promise and moved some code around.

Another but very minimal change was adjusting for the deprecation of fs.rmdirSync(buildDir, { recursive: true }); by instead using fs.rmSync(buildDir, { recursive: true });.

@DaWe35
Copy link

DaWe35 commented Dec 10, 2022

TLDR: it is actually slower while eating RAM like insane

I tested it and generated 20 images.

Env:
Windows 11
i5-10200H
Micron 2210 NVMe SSD
16 GB RAM
9+ image layers, 4 variations

Results:
Sim-Labs-LLC:build_speed - 2:44
HashLips:main - 2:34

With Sim-Labs-LLC:build_speed, for some reason, image weren't showing up in the image folder until the code finished.
My RAM usage also went up to an insane 99% level (terminal used 10 GBs of RAM alone), compared to 3 GB on hashlips:main)

So overall, this PR not only made my build time slower, it has a memory leak too. I can image a situation when Sim-Labs-LLC:build_speed is faster, but I couldn't get why would it be better for low-RAM computers.

@nhonx
Copy link

nhonx commented Mar 7, 2023

I think instead of blindly converting asynchronous writeFileSync to Promise, which can cause RAM eating, we should rewrite the code to allow writing file parallelly in multiple threads.
TLDR:
Original: write file one by one
OP solution: write all file at once
Suggesting: write X files a times, X can be configured depends on the machine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants