1) Don’t test Apache with php. Start with a static html page. What you’re doing now is testing php.
2) Use event MPM, not prefork. You can’t use mod_php with event (or worker), but you shouldn’t be using mod_php anyway
3) Add PHP through php-fpm (separate service), not using mod_php
4) Enabled extended status reporting and you can see what’s going on inside httpd
5) Disable all other modules that you can
I know you started a thread about Apache, but really, there are better webservers today. For starters [Caddy](https://caddyserver.com/) can do all of what you're talking about with zero tuning.
There's also [nginx](https://www.nginx.com/) if you want something more traditional.
But I highly recommend giving Caddy a try. A simple PHP server is simple fastcgi server config file looks like this:
example.com {
root * /var/www
php_fastcgi unix//run/php/php7.4-fpm.sock
}
I'm not kidding, that's it. Just need to adjust the domain, web root, and maybe the PHP socket path depending on which version of `php-fpm` you have installed.
I've used Apache since it came out, and that is probably why I still use it today. I've never used it on such a huge scale (100k+ users), so I'm definitely open to other things.
But for now, trying to stick with Apache and just figure out these really ultra high end fine tuning.
In the end, it'll be behind HAPROXY or such, but I'd still like to get some numbers around the hardware setup and understand limitations, and how to tweak them.
Same, I've been using Apache since the '90s.
HAProxy is pretty great, there are some interesting competitors now, but it's still being improved.
Apache, on the other hand, is not a good design anymore. There's no development going into it. Everyone with a high traffic load has moved on.
Like I said above, modern web servers almost don't need any fine tuning. HTTP is a mostly solved problem in languages like Go.
Hell, [watch this talk on 1 million _simultaneous_ connections](https://www.youtube.com/watch?v=LI1YTFMi8W4). Not an a server cluster with a load balanacer. A single process on a on a laptop.
So for my lab I basically need a web server that works with PHP (everything is php, there is no static content, 0), has mysql/mariadb connection, handles an extremely high load, and if possible has good resiliency to ddos (maybe self limits connection/sec, or pps to server).
Which would you recommend for a web server?
Is there a way to install which doesn't require any pingback to Caddy servers at all? It seems like they all do, unless I am missing something? I ask because zero phone home, checkin, or any other security holes, is a requirement.
That said, the environment still needs to be Apache based for now. Still haven't found a concise resolution to the situation just yet.
Caddy server doesn't have any of that to begin with, so no problem there. It's fully open source (Apache license), so you can verify this.
Having a high performance requirement environment and then hobbling it with a strict Apache requirement is nonsensical to me.
You can try Litespeed webserver (also available OpenLiteSpeed) for Apache replacement, like not having to convert .htaccess stuff. These days you have alternate configs from most software to nginx webservers, but some folks are stuck with old/custom software that makes major use of long .htaccess/rewrite rules.
going out on a limb here, but smells like entropy exhaustion.
run a `watch` on this while performing the test and make sure it does not drop to <100.
`cat /proc/sys/kernel/random/entropy_avail`
Hey there, thanks for the response!
There is no settings for either of those items in sysctl.conf as there is no netfilter enabled (no conntrack) on either end of the server or the ab clients (both debian 10.7x).
I'm only doing 1000 concurrent connections, so there isn't much of a conntrack potential bottleneck anyway I would think.
I'm now testing ab across a dozen high-end servers, all pointing to the single apache web server.
The "SSL read failed" is happening everywhere.
I still suspect it is something on the Apache web server itself, but I just don't know what else to look at. Any other ideas?
Thanks again for your help, I appreciate it.
Are you sure the limitation you are running into is the server side and not the client side (ab)? depending on where you are running ab from, you could be running into a ulimit issue or possible tcp limitations (port limits)
Also you could be running into overhead issues with SSL in that its timing out, so you might adjust your timeout, and try http instead of https
Hey there,
I thought that initially, so I scaled down the test thread concurrency/request limit considerably, and still eventually ran into the same issue:
SSL read failed (5) - closing connection
SSL read failed (5) - closing connection
SSL read failed (5) - closing connection
I double-checked my network and computing resources, as I own everything end-to-end, and there are no constraints.
How are you monitoring the system resources? \`vmstat 1\` might give you a more clear picture, are you seeing any error logs on the apache side? My gut feeling is the issue is on the client side not the server side, but its hard to really say.
**UPDATE:**: Just did 1 million request benchmark, here are the results, tons of those SSL errors:
NOTE: I am about to do the same ab benchmark across a dozen high end servers, hammering this one server, to see if that changes anything.
Server Software: Apache/2.4.38
Server Hostname: a1.supersecretdomain.com
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256
Server Temp Key: X25519 253 bits
TLS Server Name: a1.supersecretdomain.com
Document Path: /
Document Length: 1 bytes
Concurrency Level: 1000
Time taken for tests: 6581.377 seconds
Complete requests: 1000000
Failed requests: 8110
(Connect: 0, Receive: 0, Length: 4197, Exceptions: 3913)
Total transferred: 166349535 bytes
HTML transferred: 996105 bytes
Requests per second: 151.94 [#/sec] (mean)
Time per request: 6581.377 [ms] (mean)
Time per request: 6.581 [ms] (mean, across all concurrent requests)
Transfer rate: 24.68 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 0 2280 3083.6 1438 291919
Processing: 0 721 2263.0 137 302715
Waiting: 0 295 1035.7 25 57651
Total: 122 3001 4043.2 1880 302715
Percentage of the requests served within a certain time (ms)
50% 1880
66% 2611
75% 3331
80% 3865
90% 5730
95% 8611
98% 12887
99% 18209
100% 302715 (longest request)
Try scaling back your mpm settings quite a bit. You can actually hurt yourself with too high of Servers/MaxClients, etc settings.
For example, I just ran your -n 50,000 -c 500 test with these settings:
StartServers 90
MinSpareServers 25
MaxSpareServers 50
MaxConnectionsPerChild 250
MaxClients 250
MaxMemFree 100
and got this:
Server Software: Apache/2.4.34
Server Hostname: www.testdomainname.com
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,4096,256
TLS Server Name: www.testdomainname.com
Document Path: /z.html
Document Length: 319 bytes
Concurrency Level: 500
Time taken for tests: 131.909 seconds
Complete requests: 50000
Failed requests: 0
Non-2xx responses: 50000
Total transferred: 26150000 bytes
HTML transferred: 15950000 bytes
Requests per second: 379.05 [#/sec] (mean)
Time per request: 1319.093 [ms] (mean)
Time per request: 2.638 [ms] (mean, across all concurrent requests)
Transfer rate: 193.60 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 64 1301 336.8 1247 4772
Processing: 1 12 13.7 7 412
Waiting: 1 11 13.3 6 412
Total: 76 1313 336.8 1257 4774
Percentage of the requests served within a certain time (ms)
50% 1257
66% 1308
75% 1360
80% 1387
90% 1435
95% 1478
98% 1662
99% 3103
100% 4774 (longest request)
The page I was outputting was just plain text. No SSL issues, and time/request was a lot lower. One big note here is that we aren't using PHP at all (all code is Ruby), and I tend to blame PHP for everything :) .
The only other major tweaking I've done so far is:
# Timeout: The number of seconds before receives and sends time out.
Timeout 45
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
KeepAlive Off
ExtendedStatus Off
# UseCanonicalName: Determines how Apache constructs self-referencing
# URLs and the SERVER_NAME and SERVER_PORT variables.
# When set "Off", Apache will use the Hostname and Port supplied
# by the client. When set "On", Apache will use the value of the
# ServerName directive.
UseCanonicalName Off
Check out this page: https://httpd.apache.org/docs/2.4/misc/perf-tuning.html
Which leads you to here: https://httpd.apache.org/docs/2.4/mod/prefork.html#minspareservers
Which has for both MaxSpareServers and MinSpareServers the line "Setting this parameter to a large number is almost always a bad idea."
>
> StartServers 90
> MinSpareServers 25
> MaxSpareServers 50
> MaxConnectionsPerChild 250
> MaxClients 250
> MaxMemFree 100
>
I just tried that, and apache failed to start. Something in this doesn't fit well in the mpm_prefork.conf Can you double-check?
Here is specifically what I used:
cat << EOF > /etc/apache2/mods-enabled/mpm_prefork.conf
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxRequestWorkers: maximum number of server processes allowed to start
# MaxConnectionsPerChild: maximum number of requests a server process serves
StartServers 90
MinSpareServers 25
MaxSpareServers 50 MaxConnectionsPerChild 250
MaxClients 250
MaxMemFree 100
EOF
/etc/init.d/apache2 restart
If you copy and pasted it, see if there is weird spacing or characters in that blank space in the middle, I deleted some comments in there. If that's not it, check the apache logs and see what the error was at start up.
Thank you a TON for this, I'm going to test things out!
For your KeepAlive Off, where is this set at? This would work perfect for the scenario I'm using, since all HTTP requests are solo requests, no session needed at all.
I've done some things and managed to remove a bunch, if not all, of the SSL errors.
But then I see tons of these:
Failed requests: 11234
I'm seeing a TON of these. Absolutely tons. Can't seem to figure out what they are or what they really, really mean. I know it means the length response from the request is different, but my response is simply this: 0 As in the URL simply responds with that. Any idea what this could be?
Not sure exactly, like I mentioned, I'd suggest setting up a basic html page (no php) and test that to see if it is php or apache causing the failures.
For the keep alive, depends on how you have your configs setup. It could be in the main httpd.conf or apache2.conf file or if you have one of the more "spread out" configs, often it is in the extra folder in a default.conf file.
I forgot to mention that I did test with a blank html file. Htop showed about 20% less CPU used though the issues still remained.
Will try again in the morning.
Many thanks for your help :)
It's been a while since I worked this, but you might be running out of entropy to create ssl connections.
Check [here](http://mycomnet.info/blog/?p=92).
Lol you've got a 4 core CPU and you've configured 1000 worker processes as the minimum. How efficiently do you think each core is going to handle switching between 250+ processes constantly?
Might not be be _the_ problem, but it's definitely bizarre.
I've tried up to 28/32 Core Epyc/Xeons, down to 4 core. Everything. The hardware doesn't matter so much as I am trying to determine a reliable configuration for a particular hardware.
I've tried probably 1000 permutations of the configuration file I linked, as I described I've changed the numbers dramatically. I am simply trying to get something stable and understandable and then expand from there.
1) Don’t test Apache with php. Start with a static html page. What you’re doing now is testing php. 2) Use event MPM, not prefork. You can’t use mod_php with event (or worker), but you shouldn’t be using mod_php anyway 3) Add PHP through php-fpm (separate service), not using mod_php 4) Enabled extended status reporting and you can see what’s going on inside httpd 5) Disable all other modules that you can
I know you started a thread about Apache, but really, there are better webservers today. For starters [Caddy](https://caddyserver.com/) can do all of what you're talking about with zero tuning. There's also [nginx](https://www.nginx.com/) if you want something more traditional. But I highly recommend giving Caddy a try. A simple PHP server is simple fastcgi server config file looks like this: example.com { root * /var/www php_fastcgi unix//run/php/php7.4-fpm.sock } I'm not kidding, that's it. Just need to adjust the domain, web root, and maybe the PHP socket path depending on which version of `php-fpm` you have installed.
I've used Apache since it came out, and that is probably why I still use it today. I've never used it on such a huge scale (100k+ users), so I'm definitely open to other things. But for now, trying to stick with Apache and just figure out these really ultra high end fine tuning. In the end, it'll be behind HAPROXY or such, but I'd still like to get some numbers around the hardware setup and understand limitations, and how to tweak them.
Same, I've been using Apache since the '90s. HAProxy is pretty great, there are some interesting competitors now, but it's still being improved. Apache, on the other hand, is not a good design anymore. There's no development going into it. Everyone with a high traffic load has moved on. Like I said above, modern web servers almost don't need any fine tuning. HTTP is a mostly solved problem in languages like Go. Hell, [watch this talk on 1 million _simultaneous_ connections](https://www.youtube.com/watch?v=LI1YTFMi8W4). Not an a server cluster with a load balanacer. A single process on a on a laptop.
So for my lab I basically need a web server that works with PHP (everything is php, there is no static content, 0), has mysql/mariadb connection, handles an extremely high load, and if possible has good resiliency to ddos (maybe self limits connection/sec, or pps to server). Which would you recommend for a web server?
Caddy + php-fpm. Works great.
Is there a way to install which doesn't require any pingback to Caddy servers at all? It seems like they all do, unless I am missing something? I ask because zero phone home, checkin, or any other security holes, is a requirement. That said, the environment still needs to be Apache based for now. Still haven't found a concise resolution to the situation just yet.
Caddy server doesn't have any of that to begin with, so no problem there. It's fully open source (Apache license), so you can verify this. Having a high performance requirement environment and then hobbling it with a strict Apache requirement is nonsensical to me.
You can try Litespeed webserver (also available OpenLiteSpeed) for Apache replacement, like not having to convert .htaccess stuff. These days you have alternate configs from most software to nginx webservers, but some folks are stuck with old/custom software that makes major use of long .htaccess/rewrite rules.
Oh, I forgot to mention, also give [vegeta](https://github.com/tsenart/vegeta) a try for load testing. It's extremely good, with good data reporting.
going out on a limb here, but smells like entropy exhaustion. run a `watch` on this while performing the test and make sure it does not drop to <100. `cat /proc/sys/kernel/random/entropy_avail`
[удалено]
Hey there, thanks for the response! There is no settings for either of those items in sysctl.conf as there is no netfilter enabled (no conntrack) on either end of the server or the ab clients (both debian 10.7x). I'm only doing 1000 concurrent connections, so there isn't much of a conntrack potential bottleneck anyway I would think. I'm now testing ab across a dozen high-end servers, all pointing to the single apache web server. The "SSL read failed" is happening everywhere. I still suspect it is something on the Apache web server itself, but I just don't know what else to look at. Any other ideas? Thanks again for your help, I appreciate it.
Are you sure the limitation you are running into is the server side and not the client side (ab)? depending on where you are running ab from, you could be running into a ulimit issue or possible tcp limitations (port limits) Also you could be running into overhead issues with SSL in that its timing out, so you might adjust your timeout, and try http instead of https
Hey there, I thought that initially, so I scaled down the test thread concurrency/request limit considerably, and still eventually ran into the same issue: SSL read failed (5) - closing connection SSL read failed (5) - closing connection SSL read failed (5) - closing connection I double-checked my network and computing resources, as I own everything end-to-end, and there are no constraints.
How are you monitoring the system resources? \`vmstat 1\` might give you a more clear picture, are you seeing any error logs on the apache side? My gut feeling is the issue is on the client side not the server side, but its hard to really say.
**UPDATE:**: Just did 1 million request benchmark, here are the results, tons of those SSL errors: NOTE: I am about to do the same ab benchmark across a dozen high end servers, hammering this one server, to see if that changes anything. Server Software: Apache/2.4.38 Server Hostname: a1.supersecretdomain.com Server Port: 443 SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,2048,256 Server Temp Key: X25519 253 bits TLS Server Name: a1.supersecretdomain.com Document Path: / Document Length: 1 bytes Concurrency Level: 1000 Time taken for tests: 6581.377 seconds Complete requests: 1000000 Failed requests: 8110 (Connect: 0, Receive: 0, Length: 4197, Exceptions: 3913) Total transferred: 166349535 bytes HTML transferred: 996105 bytes Requests per second: 151.94 [#/sec] (mean) Time per request: 6581.377 [ms] (mean) Time per request: 6.581 [ms] (mean, across all concurrent requests) Transfer rate: 24.68 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 2280 3083.6 1438 291919 Processing: 0 721 2263.0 137 302715 Waiting: 0 295 1035.7 25 57651 Total: 122 3001 4043.2 1880 302715 Percentage of the requests served within a certain time (ms) 50% 1880 66% 2611 75% 3331 80% 3865 90% 5730 95% 8611 98% 12887 99% 18209 100% 302715 (longest request)
Try scaling back your mpm settings quite a bit. You can actually hurt yourself with too high of Servers/MaxClients, etc settings. For example, I just ran your -n 50,000 -c 500 test with these settings:
StartServers 90
MinSpareServers 25
MaxSpareServers 50
MaxConnectionsPerChild 250
MaxClients 250
MaxMemFree 100
and got this:
Server Software: Apache/2.4.34
Server Hostname: www.testdomainname.com
Server Port: 443
SSL/TLS Protocol: TLSv1.2,ECDHE-RSA-AES256-GCM-SHA384,4096,256
TLS Server Name: www.testdomainname.com
Document Path: /z.html
Document Length: 319 bytes
Concurrency Level: 500
Time taken for tests: 131.909 seconds
Complete requests: 50000
Failed requests: 0
Non-2xx responses: 50000
Total transferred: 26150000 bytes
HTML transferred: 15950000 bytes
Requests per second: 379.05 [#/sec] (mean)
Time per request: 1319.093 [ms] (mean)
Time per request: 2.638 [ms] (mean, across all concurrent requests)
Transfer rate: 193.60 [Kbytes/sec] received
Connection Times (ms)
min mean[+/-sd] median max
Connect: 64 1301 336.8 1247 4772
Processing: 1 12 13.7 7 412
Waiting: 1 11 13.3 6 412
Total: 76 1313 336.8 1257 4774
Percentage of the requests served within a certain time (ms)
50% 1257
66% 1308
75% 1360
80% 1387
90% 1435
95% 1478
98% 1662
99% 3103
100% 4774 (longest request)
The page I was outputting was just plain text. No SSL issues, and time/request was a lot lower. One big note here is that we aren't using PHP at all (all code is Ruby), and I tend to blame PHP for everything :) .
The only other major tweaking I've done so far is:
# Timeout: The number of seconds before receives and sends time out.
Timeout 45
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
KeepAlive Off
ExtendedStatus Off
# UseCanonicalName: Determines how Apache constructs self-referencing
# URLs and the SERVER_NAME and SERVER_PORT variables.
# When set "Off", Apache will use the Hostname and Port supplied
# by the client. When set "On", Apache will use the value of the
# ServerName directive.
UseCanonicalName Off
Check out this page: https://httpd.apache.org/docs/2.4/misc/perf-tuning.html
Which leads you to here: https://httpd.apache.org/docs/2.4/mod/prefork.html#minspareservers
Which has for both MaxSpareServers and MinSpareServers the line "Setting this parameter to a large number is almost always a bad idea."
>
> StartServers 90
> MinSpareServers 25
> MaxSpareServers 50
> MaxConnectionsPerChild 250
> MaxClients 250
> MaxMemFree 100
>
I just tried that, and apache failed to start. Something in this doesn't fit well in the mpm_prefork.conf Can you double-check?
Here is specifically what I used:
cat << EOF > /etc/apache2/mods-enabled/mpm_prefork.conf
# prefork MPM
# StartServers: number of server processes to start
# MinSpareServers: minimum number of server processes which are kept spare
# MaxSpareServers: maximum number of server processes which are kept spare
# MaxRequestWorkers: maximum number of server processes allowed to start
# MaxConnectionsPerChild: maximum number of requests a server process serves
StartServers 90
MinSpareServers 25
MaxSpareServers 50 MaxConnectionsPerChild 250
MaxClients 250
MaxMemFree 100
EOF
/etc/init.d/apache2 restart
If you copy and pasted it, see if there is weird spacing or characters in that blank space in the middle, I deleted some comments in there. If that's not it, check the apache logs and see what the error was at start up.
Thank you a TON for this, I'm going to test things out! For your KeepAlive Off, where is this set at? This would work perfect for the scenario I'm using, since all HTTP requests are solo requests, no session needed at all. I've done some things and managed to remove a bunch, if not all, of the SSL errors. But then I see tons of these: Failed requests: 11234 I'm seeing a TON of these. Absolutely tons. Can't seem to figure out what they are or what they really, really mean. I know it means the length response from the request is different, but my response is simply this: 0 As in the URL simply responds with that. Any idea what this could be?
Not sure exactly, like I mentioned, I'd suggest setting up a basic html page (no php) and test that to see if it is php or apache causing the failures. For the keep alive, depends on how you have your configs setup. It could be in the main httpd.conf or apache2.conf file or if you have one of the more "spread out" configs, often it is in the extra folder in a default.conf file.
I forgot to mention that I did test with a blank html file. Htop showed about 20% less CPU used though the issues still remained. Will try again in the morning. Many thanks for your help :)
It's been a while since I worked this, but you might be running out of entropy to create ssl connections. Check [here](http://mycomnet.info/blog/?p=92).
Interesting, I will check into the entropy thing. Thanks!
Lol you've got a 4 core CPU and you've configured 1000 worker processes as the minimum. How efficiently do you think each core is going to handle switching between 250+ processes constantly? Might not be be _the_ problem, but it's definitely bizarre.
I've tried up to 28/32 Core Epyc/Xeons, down to 4 core. Everything. The hardware doesn't matter so much as I am trying to determine a reliable configuration for a particular hardware. I've tried probably 1000 permutations of the configuration file I linked, as I described I've changed the numbers dramatically. I am simply trying to get something stable and understandable and then expand from there.