Discussion: Benchmarking

[Neitsch]

Hi all,

as we discussed in the previous WG, we want to focus on performance. To do so we will need to introduce some sort of benchmarking. Is there any recommended way how to go about this? Should the benchmark run as part of our CI? Are there any widely used packages for benchmarking?

In the next 1 month, I'd like to write some tests that demonstrate slow behavior of the GraphQL LSP, so I'm looking for best practices here :)

[acao]

As a gut-check for the web, we could run this against the GraphiQL netlify deploy on every PR:
https://github.com/GoogleChromeLabs/lighthousebot. It would among other things determine the TTI of GraphiQL on fetching/loading a remote schema, though the current SWAPI remote GraphQL Schema is probably not a sizeable enough schema to benchmark for these purposes.

Which would become relelvant to the point you raise once codemirror-graphql was using all the latest versions of the language services.

So some additional specific benchmarking scripts that we could run with our end to end suite would be nice... Are you interested in server side benchmarking as well, for the LSP, or mostly just browser environment benchmarking? (My bad, just saw that you mentioned the LSP specifically)

As far as end to end suite is concerned, we've been wanting to adopt Cypress.io, but we have an offer from Browserstack for an open source account which would allow us to show browser compatibility for codemirror-graphql (eventually monaco-graphql) and graphiql. Thinking we could use nightwatch, nightmare, or testcafe as the runner with browserstack.

Are there any best practices that you are aware of already?

[acao]

@Neitsch as per server benchmarking, the node project itself has a pretty neat tool for running its own benchmarks on node.js core features
https://github.com/nodejs/node-chakracore/blob/master/doc/guides/writing-and-running-benchmarks.md
Includes instructions for usage with CI
We have conversations open with various hosting providers FYI. So we should have free hosting with Digital Ocean, Amazon, soon etc if we need lambdas or long running instances or anything

[acao]

I've used autocannon for writing benchmarks for http APIs, but testing the expected behavior of an LSP server would be somewhat different I imagine. A single user with potentially many json-rpc requests.

I couldn't find much on benchmarking with LSPs specifically, but here is an example of a benchmark from the kotlin LSP if that helps at all?

[IvanGoncharov]

@Neitsch Thanks for pinging 👍
It's a very interesting topic for me since for the last year or so I'm working on a benchmark for graphql-js as a side project.

---

The most important trick that I learned in all these attempts is to always disable Turbo Boost on my Macbook before running any benchmarks: https://github.com/rugarciap/Turbo-Boost-Switcher

---

We had this discussion in the context of graphql-js: graphql/graphql-js#1167
And decided to build our own wrapper around benchmark.js:
https://github.com/graphql/graphql-js/blob/master/resources/benchmark.js

But we quickly discovered limitations of benchmark.js since results have ~5% difference between the run on the same machine.
IMHO, the source code of benchmark.js is unreadable so it's to trace what causing this dispersion.

I also looked at what other projects are using and it looks like for benchmarking things that running ~1s benchmark.js is a very popular choice (e.g. https://github.com/v8/web-tooling-benchmark). But all other projects I looked into were using custom benchmark script fine-tuned to the particular project.

I found Node.js benchmark to be the most similar to what's graphql-js needs:
https://github.com/nodejs/node/blob/master/benchmark/common.js
The core idea here is that functions are constantly being optimized/deoptimize and memory being collected in separate threads so it's hard to isolate different versions of the same code to get reliable results.
Node.js framework solves this by doing fork for every benchmark and executing code only inside a child process that way all code cache, gc state, and any other side effects are simply cleaned up after child process finish:
https://github.com/nodejs/node/blob/master/benchmark/common.js#L33-L35

Also, it uses process.nextTick to allow a newly created child process to stabilize before running the benchmark loop.

In addition to that, I'm currently experimenting with Node.js's --predictable flag recommended by V8 engineer: nodejs/node#11683 (comment)

And it looks like after by combining all these approaches I finally got a decent result:

So the difference between running two instances of the same code is < 0.5%.

[Neitsch]

Hey all,

I appreciate the suggestions of both of you. To stay in line with what graphql-js is doing, let's reuse the benchmark script as much as possible. I hope to get around implementing a simple benchmark later this week :)

[IvanGoncharov]

@Neitsch Please note graphql-js benchmark is an experiment in progress so please treat it critically. Also would be great if could develop it further together and so please ping me in all PRs and issues related to it.

[IvanGoncharov]

Not sure how useful it for GraphiQL but I just added memory profiling to graphql-js benchmark:
graphql/graphql-js#2088

[acao]

Not sure how useful it for GraphiQL but I just added memory profiling to graphql-js benchmark:

very nice! we should do some benchmarks all the way up to interface, and eventually have benchmarks for the LSP server over different transports