Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor inference time measurement through common interface #536

Open
PGijsbers opened this issue Jun 18, 2023 · 0 comments
Open

Refactor inference time measurement through common interface #536

PGijsbers opened this issue Jun 18, 2023 · 0 comments
Labels
quality Something needs to be tested, keep the app functional

Comments

@PGijsbers
Copy link
Collaborator

PGijsbers commented Jun 18, 2023

PR #532 introduced the option to measure inference time for many frameworks, but it introduced a lot of code duplication along the way. The PR should be "finished" by:

  • introducing a common interface that abstracts away a lot of the duplicated boilerplate code
  • extend inference time measurement support to frameworks that don't have it yet
  • add documentation

Another open issue (that may be tackled independently altogether), is that the inference time measurement can take a considerable amount of time. This may lead to the benchmark aborting a job because of the overhead, while it would otherwise would have been successful. There should be the option to do "best effort" inference time measurement within the time limit, or perhaps configure it dynamically. I don't have a concrete idea on how to do this robustly.

@PGijsbers PGijsbers added the quality Something needs to be tested, keep the app functional label Jun 18, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
quality Something needs to be tested, keep the app functional
Projects
None yet
Development

No branches or pull requests

1 participant