Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move the performance-tests into separate units #5

Open
ghost opened this issue Oct 9, 2023 · 3 comments
Open

Move the performance-tests into separate units #5

ghost opened this issue Oct 9, 2023 · 3 comments

Comments

@ghost
Copy link

ghost commented Oct 9, 2023

latest results are too good:

[info] 1 http zio = 666ms / req
[info] 10 http zio = 74ms / req
[info] 1 https zio = 237ms / req
[info] 10 https zio = 64ms / req

This is because the different runs test-requests reuse earlier instantiated stuff.

Reproduce

  • run the speetests
  • disable the http test, run again
  • disable the netty tests, run again

Workaround

  • disable the http tests for now
  • disable solo-netty-tests for now
  • in general: enable only one test

Solution

  • semi: flush/destroy everything befor each test (ZIO clients, scopes, netty)
  • final: move each performance-test (curl, java-http, netty, zio-http, zio-https, ...) into its own unit (file/class/object)
    • instantiate each performance-test from a central runner (or individually via manual run)
@jamesward
Copy link
Owner

It is curious that stuff gets reused despite providing Client.default each time - maybe some state that gets initialized there. Perhaps there is another way to initialize new state each time. Worth investigating instead of turning parts of selectively.

@ghost
Copy link
Author

ghost commented Oct 10, 2023

Even netty is reused, independent of zio framework.

yes, this needs investigation/documentation, but for now (with the limited bounty/time budget), the workaround to enable just one test does it

https://github.com/abebeos/zio-http-client-perf/blob/main/src/main/scala/App.scala

Final solution to keep speedtests in different files would be more elegant though.

Up to you to apply it on your code here.

@ghost
Copy link
Author

ghost commented Oct 10, 2023

And as a sidenote: the problem is mostly with the development-mentality of the zio project. Things are not strict enough, e.g. the speedtests should be already there, right within the CI, ideally with regression-alarms and comparative-alarms (e.g. "if zio is over 50% slower that java-http, alarm").

Anyways, me out at this point, I do not expect to get even an ok for a bounty claim (voices: "where is your PR?").

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant