[Feature]: Compile warmup take too long

[Zjq9409]

🚀 The feature, motivation and pitch

INFO 08-26 07:31:47 habana_model_runner.py:1192] [Warmup][Prompt][1/56] batch_size:64 seq_len:1024 free_mem:13.93 GiB
INFO 08-26 07:32:25 habana_model_runner.py:1192] [Warmup][Prompt][2/56] batch_size:64 seq_len:896 free_mem:13.86 GiB
INFO 08-26 07:32:59 habana_model_runner.py:1192] [Warmup][Prompt][3/56] batch_size:64 seq_len:768 free_mem:13.86 GiB
INFO 08-26 07:33:27 habana_model_runner.py:1192] [Warmup][Prompt][4/56] batch_size:64 seq_len:640 free_mem:13.86 GiB
INFO 08-26 07:33:52 habana_model_runner.py:1192] [Warmup][Prompt][5/56] batch_size:32 seq_len:1024 free_mem:13.86 GiB
INFO 08-26 07:34:08 habana_model_runner.py:1192] [Warmup][Prompt][6/56] batch_size:64 seq_len:512 free_mem:13.86 GiB
INFO 08-26 07:34:25 habana_model_runner.py:1192] [Warmup][Prompt][7/56] batch_size:32 seq_len:896 free_mem:13.86 GiB
INFO 08-26 07:34:40 habana_model_runner.py:1192] [Warmup][Prompt][8/56] batch_size:32 seq_len:768 free_mem:13.86 GiB

Could it be compiled with multiple threads, if the graph compilation use CPU? During my compilation process, CPU utilization is very low.

Alternatives

No response

Additional context

No response

[mswiniarsk]

@Zjq9409 yes, we are planning to decrease warmup time in several directions:

• decrease compilation time, so that single graph takes less time
• more sophisticated way of selecting buckets, so that we are not compiling graphs that are not truly needed
• parallel compilation