Performance improvement ideas

[AlliBalliBaba]

This thread is for gathering some recent and potential performance improvements in frankenphp so I won't forget them:

• better thread management perf: improve PHP thread management #898
• only calling ts_resource(0) once per thread perf: cgi-mode 1700% improvement #933
• Reducing logger allocs perf: prevent useless logger allocs #1005
• Logger improvements in Caddy (once new caddy version is released) 1 2 3
• Caching document root resolution perf: cache document root resolution when possible #1002
• Also caching document root resolution in worker mode (path.abs and symlink resolution)
• Lock free handles or 'no handles' approach as mentioned in Lock free handles #934
• Cache environment variables in worker and/or cgi (php_import_environment_variables creates locks )
• maybe: Automatically scale the number of threads with the incoming load to reduce context switches and optimize long requests
• maybe: only register global variables in worker-mode if $_SERVER is accessed (currently only true for cgi mode)
• maybe: optimize frankenphp globals registration (not sure how)

Here's a flamegraph I labeled in MS Paint to showcase the current work done in a worker 'Hello World' request without file_server:

Here's a graph showcasing how one can finangle the number of workers for a maximum RPS for a 'Hello world' (kudos to chatgpt). This was done with 20 CPU cores on a custom branch with php_import_environment_variables disabled and no handles. I cannot yet explain why contention significantly increases on higher worker counts.

Note that the plain Caddy respond "Hello World" will currently manage around 180.000 RPS in the same setup.

[nickchomey]

Improvements are always good, but isn't 90,000 rps far beyond anything anyone is realistically going to need? The bottleneck very much doesn't seem like the hand-off from caddy to php at that point (which is what a hello world is effectively measuring, no?)

So, I wonder if there should be a focus on setting up some standardized benchmarking suite of real world php workloads - laravel (normal and octane), some sort of fphp workers web app, WordPress without workers etc...? And compare it to php-fpm with caddy and nginx, openlitespeed etc

Whatever tests like that reveal - be it that fphp is faster or has some genuine bottlenecks - would be very useful to know

[AlliBalliBaba]

You are right, a 'Hello World' benchmark is as synthetic as it gets. The point of these benchmarks is not to showcase the potential RPS frankenphp can do, it's about identifying potential bottlenecks in the runtime itself. Removing symlink checking on every request might matter very little or a lot to your actual application, depending on what your bottlenecks are.

That's why you should generally only trust benchmarks conducted on your actual productive system (that will probably run on less than 20 CPU cores).

It's worth noting that these optimization also aim at improving the techempower benchmarks, which are slightly less synthetic.

[AlliBalliBaba]

I'll try to create a flamegraph with a Laravel application though, that sound like a good idea 👍

[nickchomey]

Yes, I'm not at all against improving the runtime/guts of caddy/frankenphp, but I'm instead advocating for taking a step back and assessing what is worth putting limited time and resources towards.

90,000 rps is 0.01 milliseconds per request, or 10 microseconds. That's completely imperceptible in the context of any real application or even just basic api endpoint that is surely going to be many orders of magnitude slower. And then there's the unavoidable network latency that'll add another 2-4 orders of magnitude of latency... The techempower benchmarks are completely irrelevant in this context - perhaps they're worthwhile if you're doing high frequency trading and every microsecond counts.

I think benchmarks on real-world use-cases - eg. laravel - would be infinitely more useful. Given that the webserver is basically instantaneous, it would be FAR more revelatory to evaluate differences between frankenphp and php-fpm with caddy or nginx, or with lsphp and openlitespeed. I'd honestly expect them all to be pretty much the same - if they aren't then THAT gives reason for investigation into why php is running slower or with more ram usage than other implementations.

Also, given that a real application uses real computing resources, the results would be far more informative for evaluating things like number of workers per core, than something completely synthetic and irrelevant.

I realize that I'm just yelling from the cheap seats, but I very much intend to actually help contribute to this sort of stuff at some point when time permits me. Until then, I'm just hoping to help those who are currently active with development and testing to make best use of their limited time.

Other important things that could be developed are better docs, tooling, docker images, addons, etc...

[dunglas]

For the record, there is a more "real-life" benchmark in the Symfony demo app, and the Laravel team created one for Laravel too:

1. https://github.com/dunglas/frankenphp-demo/tree/main/benchmark
2. https://blog.laravel.com/octane-frankenphp

[withinboredom]

but I'm instead advocating for taking a step back and assessing what is worth putting limited time and resources towards.

That's the thing. We are using "regular PHP" once it gets past a certain point. Those types of benchmarks apply regardless of what server you are using. While yes, some of us are working on improvements in PHP itself (which everyone benefits from), we can only control everything up until the point it gets into "regular PHP" and when it returns. These benchmarks specifically focus on exactly that. We aren't interested in "regular PHP" performance, but rather FrankenPHP/Caddy itself, which surrounds "regular PHP."

I understand your concerns, but there is very little we can realistically do about applications and what they do in "regular PHP-land." I hope that makes sense.

[dunglas]

The document root is also cached in worker mode (#1002 added cache for both modes).

[AlliBalliBaba]

Hmm it somehow was still showing up in the flamegraph, I'll check again

[AlliBalliBaba]

Oh yeah you are right, but the root directory needs to be added explicitly in order to get cached
Here the root is resolved beforehand:

route {
	root /go/src/app/testdata
        php {
                root /go/src/app/testdata
        }
}

Here it is not (since it's {http.vars.root} when provisioning)

route {
	root /go/src/app/testdata
        php
}

[dunglas]

Good catch! Maybe should we add an exception for this placeholder as it's the default and we know (I think) its value cannot change?

[AlliBalliBaba]

I think the value won't be resolved at the time of provisioning since it comes from outside of the module. I guess it would be possible to store the resolved document roots in a map once we're already in a request

[AlliBalliBaba]

I think I realized why 10 workers is the magic number where the 'Hello World' is most efficient. It's basically about CPU context switching and to some degree unavoidable.
With a 'Hello World' roughly 50% of work happens on the Caddy side and the other 50% in the FrankenPHP threads (see Flamegraphs). Therefore we have the least context switches if Caddy and Frankenphp can exactly utilize half of the CPU cores each (10 for Caddy, 10 for PHP Threads)

I think the only way to make this more efficient would be to autoscale the number of PHP Threads based on the load instead of having a fixed number. While this would be a cool feature, it's probably not trivial to implement. I think FPM also has a similar mode.

[dunglas]

Starting threads only when needed may not be that hard to implement. The hard part will actually be to detect the when. Maybe #966 can help with that?

Another approach that I would like to try is to not use C threads at all, but run PHP directly in goroutines.

To do so we need to:

• get rid of the use of signals in PHP (they are used for PHP timeouts), @arnaud-lb made a POC using kevent instead of signals and it should even be possible to directly use Go's time module if we add the proper abstraction for that in PHP
• write a TSRM implementation compatible with goroutines

It's some work, for sure, but this doesn't look unfeasible at all.

[withinboredom]

I still think #841 would work without those changes. I'm still reasonably confident that taking over a go thread is perfectly fine. :D

Working with go through TSRM is a whole different beast. Go doesn't have a concept of thread-local storage, and TSRM is all about thread-local storage. That isn't to say it is impossible, but to get it working will be an engineering challenge, for sure.

[dunglas]

Because how TSRM is implemented (a global hash map with the "thread" ID as key), it's not that complicated. We "just" need to assign a different ID per PHP "thread".

[withinboredom]

That's what I mean though. There is no thread id on the go side. There is nothing to match it to. As an aside, we also will need to remove any real threadlocal variables and the same goes for extensions.

[withinboredom]

🤦 You don't need a thread id on the go side. It's more like a "request id" that is bound to an executing thread pool. I think you can do this today, actually (from digging into tsrm) with very little changes. The bigger issue is then threadlocal variables.

I will give this a try next week.