Changing the server parameters

[bensilver95]

Is there a way to change the parameters of the server such that it can receive more HTTP requests in a short period of time? I keep getting timeout and bad request errors because I'm doing multiprocessing with a long list of addresses to geocode. I was hoping that by installing my own instance of Nominatim I could get rid of this issue.

[mtmail]

This issue would need more detail: hardware, size of the data/database, how many requests we're talking about, how many parallel requests, if the machine is CPU-bound, disc-bound, running out of memory.

[bensilver95]

Sure I can provide more detail! I'm running the script on a Virtual Machine with Ubuntu 18 that has ~ 115GB of memory. The VM has 32 cores, but I am only using 16 processes for multiprocessing while testing (although would like to do more if I can get it to work). The data is ~3.5 million locations. I'm not sure how to determine if my machine is CPU-bound. I'm including a screenshot from htop if that's helpful.

[mtmail]

I see. Big machine, lots of data. The bottleneck is probably not hardware but the webserver. The default Apache configuration won't start enough threads to serve all incoming requests. In your /var/log/apache2/error.log you might see warnings like "increase number of servers" (or workers) when lots of requests come in in parallel.

There's a couple of tutorials on how to scale Apache on bigger hardware to make sure of more CPU cores. https://www.liquidweb.com/kb/apache-performance-tuning-mpm-directives/

As a start increase ServerLimit and ThreadsPerChild. https://httpd.apache.org/docs/current/mod/mpm_common.html#startservers I can't make specific recommendations, usually it requires a bit of testing with real data.

The guides often also recommend to use php_fpm or other application servers. There's an example in https://github.com/osm-search/Nominatim/blob/master/vagrant/Install-on-Ubuntu-18-nginx.sh but based on the nginx webserver so that would require a lot more changes.

[soyfrien]

Hi @mtmail,

Do you have any advice on how to switch from mpm_prefork to mpm_worker?

root@888df80187cd:/app# a2enmod mpm_worker
Considering conflict mpm_event for mpm_worker:
Considering conflict mpm_prefork for mpm_worker:
ERROR: Module mpm_prefork is enabled - cannot proceed due to conflicts. It needs to be disabled first!
root@888df80187cd:/app# a2dismod php 7.3 mpm_prefork
root@888df80187cd:/app# a2dismod mpm_worker
Enabling module mpm_worker.
To activate the new configuration, you need to run:
  service apache2 restart
root@888df80187cd:/app# a2enmod php7.3
Considering dependency mpm_prefork for php7.3:
Considering conflict mpm_event for mpm_prefork:
Considering conflict mpm_worker for mpm_prefork:
ERROR: Module mpm_worker is enabled - cannot proceed due to conflicts. It needs to be disabled first!
ERROR: Could not enable dependency mpm_prefork for php7.3, aborting
root@888df80187cd:/app# service apache2 restart
 * Restarting Apache httpd web server apache2                                                                           AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.17.0.2. Set the 'ServerName' directive globally to suppress this message
                                                                                                                 [ OK ]
root@888df80187cd:/app# a2enmod php7.3
Considering dependency mpm_prefork for php7.3:
Considering conflict mpm_event for mpm_prefork:
Considering conflict mpm_worker for mpm_prefork:
ERROR: Module mpm_worker is enabled - cannot proceed due to conflicts. It needs to be disabled first!
ERROR: Could not enable dependency mpm_prefork for php7.3, aborting

The prefork mod seems to be missing the directives you mentioned.

Is the idea to not enable php7.3 when using mpm_worker? Does it fall back on something?

Many thanks,
Louis

[mtmail]

Hmm, maybe I mixed up the mpm_* module names, I'm using something different. If you get mpm_event enabled you can try these settings https://github.com/openstreetmap/chef/blob/master/roles/nominatim.rb#L15 as a start.

[soyfrien]

Hi @mtmail,

Thank you for the suggestion. Actually increasing the values in the conf file for mpm_prefork seems to have helped a great deal. So I am not sure if I will need to try this.

Many thanks again,
Louis

[soyfrien]

Dear @mtmail,

I haven't had any luck trying to tweak the container to get mpm_event working.

Would you please do lend me a teaching moment and point me in the direction of how to use that Chef file.

I learned about Chef from your link, and to be honest I have no idea where to begin.

If not, no worries.

Best,
Louis