Load Testing

[pwdel]

• General Load testing, no clear specification at the onset of what is needed for this ticket, just generally known that we need so produce some sort of results for our potential users to show how much traffic this site can potentially handle under certain conditions and servers.

[pwdel]

Re: from @astrosnat , Discord

Using pre- v0.0.3, deployed on Digital Ocean.

FYI https://docs.digitalocean.com/products/droplets/how-to/graphs/

Then...

[pwdel]

Created +10 markets, for a total of 11 on https://openprediction.xyz/

This is running on Digital Ocean.

Specs for the box it's running on?

Suggest running a load test again, see where we are at, and then based upon the results of that test, increase the markets up to the point that is significant but perhaps doesn't break it yet, repeat until able to break the website.

[astrosnat]

Graphs & specs pre load test - 4GB RAM 80 GB basic hard drive Ubuntu 22.04 LTS

Thank you so much for doing this, load testing starts after coffee

Do you have an idea of what will be significant @pwdel? Like say a rough % of CPU usage or something?

[astrosnat]

Load testing with 10 markets: requests start to fail at around 100k requests total with a concurrency of 1000 (think 1000 users trying to access the site at the same time)

20240728_LoadTest3.csv
20240728_LoadTest2.csv
20240728_LoadTest1.csv
20240728_LoadTest3_logs.txt
20240728_LoadTest4_logs.txt
20240728_LoadTest4.csv

Here is how CPU usage etc. got affected:

Note that CPU % usage goes above 90% with 10m requests and a concurrency of 1000

[pwdel]

Edit... I may have read your post above wrong and the math below here might be incorrect. My assumptions were based upon 100k requests, the CPU maxing out at close to 90% along with other metrics you show.

If it was in fact 10M requests, then we can potentially multiply all of my calculations below by 100, e.g. 400 markets on the front page, CPU fails or gets close to failing at 2500 users.

---

OK so this means that there is a huge potential for improvement in serving static files.

That being said ... I wonder how geometric this would be. So if the markets/ page displayed 10X more markets, would that be the equivalent of 100 users accessing the site at the same time?

E.g., so does this mean we're only rated for displaying around say 200 markets on the front page, with 50 users simultaneously right now?

Or, the equivalent of 400 markets on the front page, 25 users (not that we would need 400 markets, but something equivalent, processing wise).

I don't necessarily see us needing to support 25 users with that much activity by say, September, but I could ask more from @jg-you and his class as a hypothetical. There could be a safety factor that we build in.

Just trying to think about project planning and whether it's worth it to build in, "serve static files for prod," right away now or keep it more as a dev product.

[astrosnat]

Well, I can create 90 dummy markets and see what happens. My feeling is that it makes sense to build in "serve static files for prod". I would be targeting ~100 scientists to use this - for example I could talk to the UK Reproducibility Network (a group of scientists focused on improving reproducibility) and the University of Sheffield metascience department. It also fits in pretty nicely with UKRI's new pivot to metascience. (UKRI is the umbrella research funder for the UK.)

Wonder if we could share the SocialPredict software with the Open Science Foundation too. I'm sure some OSF folks would be really into it.

[astrosnat]

So I didn't manage to create 90 extra dummy markets...I got up to about 50 and ran out of extra credits. And then when I tried to create a new account, this happened when trying to change its password:

Load testing today:
20240730_LoadTest2_logs.txt
20240730_LoadTest2.csv
20240730_LoadTest1_logs.txt
20240730_LoadTest1.csv

and screenshots:

[pwdel]

hey, I had already commented this on Discord but putting it here for posterity.

Get the logs from the various containers, probably frontend first, to see where that error came from. Post the error here when you can and we'll generate a ticket based off of that.

https://github.com/openpredictionmarkets/socialpredict/blob/main/README/LOCAL_SETUP.md#getting-logs-from-different-containers

[astrosnat]

20240802_ErrorLogs.txt
logs from frontend + backend

[astrosnat]

Load testing on openprediction.xyz after QoL fix to leverage vite build (PR #341)

1GB RAM | 25GB Disk

20240930_LoadTest5_logs.txt

20240930_DOMetrics.txt
(I'll make the metrics into a nice graph later)
20240930_LoadTest5.csv

10000 requests, 1000 requests at a time

Note: we are already at 32.1% disk usage, which might imply needing to resize the droplet at some point in time

[pwdel]

Looks like less memory usage as well... down to ~55% of RAM from the previous 70%.

[pwdel]

From Discord:

BrierFoxForecast ... using old version of master from late August:

• With 6 markets

Disk:

Memory:

CPU: