Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

推理的时候GPU才用到30%左右，有办法全部吃满GPU吗？ #1211

Open
tuoerye opened this issue Jun 20, 2024 · 2 comments
Open

推理的时候GPU才用到30%左右，有办法全部吃满GPU吗？ #1211

tuoerye opened this issue Jun 20, 2024 · 2 comments
Labels
In follow-up

Comments

@tuoerye
Copy link

tuoerye commented Jun 20, 2024

如题，求教，推理时想用满GPU提高生成速度，有什么办法吗？
@RVC-Boss
Copy link
Owner

1、fast inference分支会高一些，batch size加大了占用率也会更高
2、需要找工程大佬利用TensorRT（我随便举例一个）等特定硬件上的加速框架进行加速提高利用率
3、torch.compile也许可以提升一点利用率
4、CPU的延时也有可能是瓶颈

@tuoerye
Copy link
Author

tuoerye commented Jun 25, 2024

1、fast inference分支会高一些，batch size加大了占用率也会更高 2、需要找工程大佬利用TensorRT（我随便举例一个）等特定硬件上的加速框架进行加速提高利用率 3、torch.compile也许可以提升一点利用率 4、CPU的延时也有可能是瓶颈

谢谢大佬，感恩，我试试

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
In follow-up
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tuoerye @RVC-Boss