GizmoLA.com

Well this looks promising --- httpd -l should give us a list of modules!





So as you can see, there's a small group of core modules compiled into Apache, and in this case the server is running prefork and NOT running worker (aka the threaded server).

What is Prefork?

In unix, "forking" is a way for a parent program to essentially clone a copy of itself as a seperate child program. For this reason when you run ps or top you'll see a number of different httpd processes running.




There are some interesting things to note about this output. First off we have one httpd running as root. Then we have 20 httpd's running as apache.

On this particular server, if you look in the httpd.conf file which is being loaded when the server starts, you find this configuration section:


So as you can see, the MaxSpareServers of 20 matches the number of child processes we have, and this is because this particular server has been running for a while, and has seen some periods where it was receiving enough traffic that apache forked additional child processes, keeping those 20 child servers around. If however, we restart apache, what will we find (assuming there is little to no load when it restarts).


I'm guessing nobody will be surprised to find that we have exactly 8 child processes thanks to the "StartServers 8" configuration item. Now you might be interested in the other parameters. Take the time to read about them in the apache documentation! I am going to talk about the "MaxRequestsPerChild 4000" parameter.

MaxRequestsPerChild

So you might be wondering at this point if apache ever kills one of its child processes?

The answer is yes! Controlling the killing of child processes is what the MaxRequestsPerChild parameter does. Any child will service 4k HTTP requests before it is killed by the parent, and replaced by a new forked child process.

You also might be wondering why Apache doesn't just keep a child process going forever. One reason, is to return memory back to the operating system.

If we run ps aux | more, so we can see the column headers, we'll find that several of the headers are displaying memory statistics.


With these flags for the ps command, we can now see the VSZ and RSS columns, which display memory usage for the process, are showing a variety of different numbers (all of which are in KB) to indicate the amount of memory the process requires. Or do they?

As it turns out these numbers give you a general idea of what the process would require in a vacuum, but doesn't really give you accurate information. This is because substantial amounts of memory use needed for the code segments of the program are in shared libraries which can be shared by all the processes that need them, without having to make a seperate copy of the code (hence the explanation for the word "shared" in the name). We need a better utility to look at the real memory usage.

We can use pmap to look at a specific child process by passing in it's process id:




[root@penny conf]# pmap -d 30548
30548: /usr/sbin/httpd
Address Kbytes Mode Offset Device Mapping
00002b937f999000 312 r-x-- 0000000000000000 0fd:00002 httpd
00002b937f9e7000 100 rw-s- 0000000000000000 000:0007c [ anon ]
00002b937fa41000 1340 rw--- 00002b937fa41000 000:00000 [ anon ]
00002b937fbe6000 16 rw--- 000000000004d000 0fd:00002 httpd
00002b937fbea000 12 rw--- 00002b937fbea000 000:00000 [ anon ]
00002b937fbed000 112 r-x-- 0000000000000000 0fd:00002 ld-2.5.so
00002b937fc09000 1032 rw--- 00002b937fc09000 000:00000 [ anon ]
00002b937fd0b000 88 rw-s- 0000000000000000 000:0007c [ anon ]
00002b937fe09000 4 r---- 000000000001c000 0fd:00002 ld-2.5.so
00002b937fe0a000 4 rw--- 000000000001d000 0fd:00002 ld-2.5.so
00002b937fe0b000 520 r-x-- 0000000000000000 0fd:00002 libm-2.5.so
00002b937fe8d000 2044 ----- 0000000000082000 0fd:00002 libm-2.5.so
00002b938008c000 4 r---- 0000000000081000 0fd:00002 libm-2.5.so
00002b938008d000 4 rw--- 0000000000082000 0fd:00002 libm-2.5.so
00002b938008e000 120 r-x-- 0000000000000000 0fd:00002 libpcre.so.0.0.1
00002b93800ac000 2048 ----- 000000000001e000 0fd:00002 libpcre.so.0.0.1
00002b93802ac000 4 rw--- 000000000001e000 0fd:00002 libpcre.so.0.0.1
00002b93802ad000 84 r-x-- 0000000000000000 0fd:00002 libselinux.so.1
00002b93802c2000 2048 ----- 0000000000015000 0fd:00002 libselinux.so.1
00002b93804c2000 8 rw--- 0000000000015000 0fd:00002 libselinux.so.1
00002b93804c4000 8 rw--- 00002b93804c4000 000:00000 [ anon ]
00002b93804c6000 100 r-x-- 0000000000000000 0fd:00002 libaprutil-1.so.0.2.7
00002b93804df000 2048 ----- 0000000000019000 0fd:00002 libaprutil-1.so.0.2.7
00002b93806df000 4 rw--- 0000000000019000 0fd:00002 libaprutil-1.so.0.2.7
00002b93806e0000 36 r-x-- 0000000000000000 0fd:00002 libcrypt-2.5.so
00002b93806e9000 2044 ----- 0000000000009000 0fd:00002 libcrypt-2.5.so
00002b93808e8000 4 r---- 0000000000008000 0fd:00002 libcrypt-2.5.so
00002b93808e9000 4 rw--- 0000000000009000 0fd:00002 libcrypt-2.5.so
00002b93808ea000 184 rw--- 00002b93808ea000 000:00000 [ anon ]
00002b9380918000 228 r-x-- 0000000000000000 0fd:00002 libldap-2.3.so.0.2.31
00002b9380951000 2048 ----- 0000000000039000 0fd:00002 libldap-2.3.so.0.2.31
00002b9380b51000 8 rw--- 0000000000039000 0fd:00002 libldap-2.3.so.0.2.31
...
00002b9383942000 8 r-x-- 0000000000000000 0fd:00002 mod_auth_basic.so
00002b9383944000 2044 ----- 0000000000002000 0fd:00002 mod_auth_basic.so
00002b9383b43000 8 rw--- 0000000000001000 0fd:00002 mod_auth_basic.so
00002b9383b45000 24 r-x-- 0000000000000000 0fd:00002 mod_auth_digest.so
00002b9383b4b000 2044 ----- 0000000000006000 0fd:00002 mod_auth_digest.so
00002b9383d4a000 8 rw--- 0000000000005000 0fd:00002 mod_auth_digest.so
00002b9383d4c000 8 r-x-- 0000000000000000 0fd:00002 mod_authn_file.so
00002b9383d4e000 2044 ----- 0000000000002000 0fd:00002 mod_authn_file.so
00002b9383f4d000 8 rw--- 0000000000001000 0fd:00002 mod_authn_file.so
00002b9383f4f000 8 r-x-- 0000000000000000 0fd:00002 mod_authn_alias.so
00002b9383f51000 2044 ----- 0000000000002000 0fd:00002 mod_authn_alias.so
00002b9384150000 8 rw--- 0000000000001000 0fd:00002 mod_authn_alias.so
00002b9384152000 4 r-x-- 0000000000000000 0fd:00002 mod_authn_anon.so
00002b9384153000 2048 ----- 0000000000001000 0fd:00002 mod_authn_anon.so
00002b9384353000 8 rw--- 0000000000001000 0fd:00002 mod_authn_anon.so
00002b9384355000 4 r-x-- 0000000000000000 0fd:00002 mod_authn_dbm.so
00002b9384356000 2048 ----- 0000000000001000 0fd:00002 mod_authn_dbm.so
00002b9384556000 8 rw--- 0000000000001000 0fd:00002 mod_authn_dbm.so
00002b9384558000 4 r-x-- 0000000000000000 0fd:00002 mod_authn_default.so
00002b9384559000 2044 ----- 0000000000001000 0fd:00002 mod_authn_default.so
00002b9384758000 8 rw--- 0000000000000000 0fd:00002 mod_authn_default.so
00002b938475a000 8 r-x-- 0000000000000000 0fd:00002 mod_authz_host.so
00002b938475c000 2044 ----- 0000000000002000 0fd:00002 mod_authz_host.so
00002b938495b000 8 rw--- 0000000000001000 0fd:00002 mod_authz_host.so
00002b938495d000 4 r-x-- 0000000000000000 0fd:00002 mod_authz_user.so
00002b938495e000 2044 ----- 0000000000001000 0fd:00002 mod_authz_user.so
00002b9384b5d000 8 rw--- 0000000000000000 0fd:00002 mod_authz_user.so
00002b9384b5f000 8 r-x-- 0000000000000000 0fd:00002 mod_authz_owner.so
00002b9384b61000 2044 ----- 0000000000002000 0fd:00002 mod_authz_owner.so
00002b9384d60000 8 rw--- 0000000000001000 0fd:00002 mod_authz_owner.so
00002b9384d62000 8 r-x-- 0000000000000000 0fd:00002 mod_authz_groupfile.so
00002b9384d64000 2044 ----- 0000000000002000 0fd:00002 mod_authz_groupfile.so
00002b9384f63000 8 rw--- 0000000000001000 0fd:00002 mod_authz_groupfile.so
00002b9384f65000 8 r-x-- 0000000000000000 0fd:00002 mod_authz_dbm.so
00002b9384f67000 2044 ----- 0000000000002000 0fd:00002 mod_authz_dbm.so
00002b9385166000 8 rw--- 0000000000001000 0fd:00002 mod_authz_dbm.so
00002b9385168000 4 r-x-- 0000000000000000 0fd:00002 mod_authz_default.so
00002b9385169000 2044 ----- 0000000000001000 0fd:00002 mod_authz_default.so
00002b9385368000 8 rw--- 0000000000000000 0fd:00002 mod_authz_default.so
00002b938536a000 48 r-x-- 0000000000000000 0fd:00002 mod_ldap.so
00002b9385376000 2044 ----- 000000000000c000 0fd:00002 mod_ldap.so
00002b9385575000 8 rw--- 000000000000b000 0fd:00002 mod_ldap.so
00002b9385577000 24 r-x-- 0000000000000000 0fd:00002 mod_authnz_ldap.so
00002b938557d000 2048 ----- 0000000000006000 0fd:00002 mod_authnz_ldap.so
00002b938577d000 8 rw--- 0000000000006000 0fd:00002 mod_authnz_ldap.so
00002b938577f000 36 r-x-- 0000000000000000 0fd:00002 mod_include.so
00002b9385788000 2048 ----- 0000000000009000 0fd:00002 mod_include.so
00002b9385988000 8 rw--- 0000000000009000 0fd:00002 mod_include.so
00002b938598a000 20 r-x-- 0000000000000000 0fd:00002 mod_log_config.so
00002b938598f000 2044 ----- 0000000000005000 0fd:00002 mod_log_config.so
00002b9385b8e000 8 rw--- 0000000000004000 0fd:00002 mod_log_config.so
00002b9385b90000 8 r-x-- 0000000000000000 0fd:00002 mod_logio.so
00002b9385b92000 2044 ----- 0000000000002000 0fd:00002 mod_logio.so
00002b9385d91000 8 rw--- 0000000000001000 0fd:00002 mod_logio.so
00002b9385d93000 4 r-x-- 0000000000000000 0fd:00002 mod_env.so
00002b9385d94000 2048 ----- 0000000000001000 0fd:00002 mod_env.so
00002b9385f94000 8 rw--- 0000000000001000 0fd:00002 mod_env.so
00002b9385f96000 16 r-x-- 0000000000000000 0fd:00002 mod_ext_filter.so
00002b9385f9a000 2044 ----- 0000000000004000 0fd:00002 mod_ext_filter.so
00002b9386199000 8 rw--- 0000000000003000 0fd:00002 mod_ext_filter.so
00002b938619b000 20 r-x-- 0000000000000000 0fd:00002 mod_mime_magic.so
00002b93861a0000 2044 ----- 0000000000005000 0fd:00002 mod_mime_magic.so
00002b938639f000 8 rw--- 0000000000004000 0fd:00002 mod_mime_magic.so
00002b93863a1000 8 r-x-- 0000000000000000 0fd:00002 mod_expires.so
00002b93863a3000 2044 ----- 0000000000002000 0fd:00002 mod_expires.so
00002b93865a2000 8 rw--- 0000000000001000 0fd:00002 mod_expires.so
00002b93865a4000 20 r-x-- 0000000000000000 0fd:00002 mod_deflate.so
00002b93865a9000 2044 ----- 0000000000005000 0fd:00002 mod_deflate.so
00002b93867a8000 8 rw--- 0000000000004000 0fd:00002 mod_deflate.so
00002b93867aa000 12 r-x-- 0000000000000000 0fd:00002 mod_headers.so
00002b93867ad000 2048 ----- 0000000000003000 0fd:00002 mod_headers.so
00002b93869ad000 8 rw--- 0000000000003000 0fd:00002 mod_headers.so
00002b93869af000 8 r-x-- 0000000000000000 0fd:00002 mod_usertrack.so
00002b93869b1000 2048 ----- 0000000000002000 0fd:00002 mod_usertrack.so
00002b9386bb1000 8 rw--- 0000000000002000 0fd:00002 mod_usertrack.so
00002b9386bb3000 8 r-x-- 0000000000000000 0fd:00002 mod_setenvif.so
00002b9386bb5000 2048 ----- 0000000000002000 0fd:00002 mod_setenvif.so
00002b9386db5000 8 rw--- 0000000000002000 0fd:00002 mod_setenvif.so
00002b9386db7000 16 r-x-- 0000000000000000 0fd:00002 mod_mime.so
00002b9386dbb000 2044 ----- 0000000000004000 0fd:00002 mod_mime.so
00002b9386fba000 8 rw--- 0000000000003000 0fd:00002 mod_mime.so
00002b9386fbc000 84 r-x-- 0000000000000000 0fd:00002 mod_dav.so
00002b9386fd1000 2044 ----- 0000000000015000 0fd:00002 mod_dav.so
00002b93871d0000 8 rw--- 0000000000014000 0fd:00002 mod_dav.so
00002b93871d2000 16 r-x-- 0000000000000000 0fd:00002 mod_status.so
00002b93871d6000 2044 ----- 0000000000004000 0fd:00002 mod_status.so
00002b93873d5000 8 rw--- 0000000000003000 0fd:00002 mod_status.so
00002b93873d7000 32 r-x-- 0000000000000000 0fd:00002 mod_autoindex.so
00002b93873df000 2044 ----- 0000000000008000 0fd:00002 mod_autoindex.so
00002b93875de000 8 rw--- 0000000000007000 0fd:00002 mod_autoindex.so
00002b93875e0000 16 r-x-- 0000000000000000 0fd:00002 mod_info.so
00002b93875e4000 2044 ----- 0000000000004000 0fd:00002 mod_info.so
00002b93877e3000 8 rw--- 0000000000003000 0fd:00002 mod_info.so
00002b93877e5000 44 r-x-- 0000000000000000 0fd:00002 mod_dav_fs.so
00002b93877f0000 2044 ----- 000000000000b000 0fd:00002 mod_dav_fs.so
00002b93879ef000 8 rw--- 000000000000a000 0fd:00002 mod_dav_fs.so
00002b93879f1000 8 r-x-- 0000000000000000 0fd:00002 mod_vhost_alias.so
00002b93879f3000 2044 ----- 0000000000002000 0fd:00002 mod_vhost_alias.so
00002b9387bf2000 8 rw--- 0000000000001000 0fd:00002 mod_vhost_alias.so
00002b9387bf4000 28 r-x-- 0000000000000000 0fd:00002 mod_negotiation.so
00002b9387bfb000 2044 ----- 0000000000007000 0fd:00002 mod_negotiation.so
00002b9387dfa000 8 rw--- 0000000000006000 0fd:00002 mod_negotiation.so
00002b9387dfc000 8 r-x-- 0000000000000000 0fd:00002 mod_dir.so
00002b9387dfe000 2044 ----- 0000000000002000 0fd:00002 mod_dir.so
00002b9387ffd000 8 rw--- 0000000000001000 0fd:00002 mod_dir.so
00002b9387fff000 8 r-x-- 0000000000000000 0fd:00002 mod_actions.so
00002b9388001000 2044 ----- 0000000000002000 0fd:00002 mod_actions.so
00002b9388200000 8 rw--- 0000000000001000 0fd:00002 mod_actions.so
00002b9388202000 8 r-x-- 0000000000000000 0fd:00002 mod_speling.so
00002b9388204000 2048 ----- 0000000000002000 0fd:00002 mod_speling.so
00002b9388404000 8 rw--- 0000000000002000 0fd:00002 mod_speling.so
00002b9388406000 8 r-x-- 0000000000000000 0fd:00002 mod_userdir.so
00002b9388408000 2044 ----- 0000000000002000 0fd:00002 mod_userdir.so
00002b9388607000 8 rw--- 0000000000001000 0fd:00002 mod_userdir.so
00002b9388609000 12 r-x-- 0000000000000000 0fd:00002 mod_alias.so
00002b938860c000 2044 ----- 0000000000003000 0fd:00002 mod_alias.so
00002b938880b000 8 rw--- 0000000000002000 0fd:00002 mod_alias.so
00002b938880d000 56 r-x-- 0000000000000000 0fd:00002 mod_rewrite.so
00002b938881b000 2044 ----- 000000000000e000 0fd:00002 mod_rewrite.so
00002b9388a1a000 8 rw--- 000000000000d000 0fd:00002 mod_rewrite.so
00002b9388a1c000 76 r-x-- 0000000000000000 0fd:00002 mod_proxy.so
00002b9388a2f000 2044 ----- 0000000000013000 0fd:00002 mod_proxy.so
00002b9388c2e000 8 rw--- 0000000000012000 0fd:00002 mod_proxy.so
00002b9388c30000 20 r-x-- 0000000000000000 0fd:00002 mod_proxy_balancer.so
00002b9388c35000 2048 ----- 0000000000005000 0fd:00002 mod_proxy_balancer.so
00002b9388e35000 8 rw--- 0000000000005000 0fd:00002 mod_proxy_balancer.so
00002b9388e37000 28 r-x-- 0000000000000000 0fd:00002 mod_proxy_ftp.so
00002b9388e3e000 2044 ----- 0000000000007000 0fd:00002 mod_proxy_ftp.so
00002b938903d000 8 rw--- 0000000000006000 0fd:00002 mod_proxy_ftp.so
00002b938903f000 28 r-x-- 0000000000000000 0fd:00002 mod_proxy_http.so
00002b9389046000 2048 ----- 0000000000007000 0fd:00002 mod_proxy_http.so
00002b9389246000 8 rw--- 0000000000007000 0fd:00002 mod_proxy_http.so
00002b9389248000 8 r-x-- 0000000000000000 0fd:00002 mod_proxy_connect.so
00002b938924a000 2044 ----- 0000000000002000 0fd:00002 mod_proxy_connect.so
00002b9389449000 8 rw--- 0000000000001000 0fd:00002 mod_proxy_connect.so
...
0002b938cf4e000 508 r-x-- 0000000000000000 0fd:00002 libfreetype.so.6.3.10
00002b938cfcd000 2048 ----- 000000000007f000 0fd:00002 libfreetype.so.6.3.10
00002b938d1cd000 20 rw--- 000000000007f000 0fd:00002 libfreetype.so.6.3.10
00002b938d1d2000 1044 r-x-- 0000000000000000 0fd:00002 libX11.so.6.2.0
00002b938d2d7000 2048 ----- 0000000000105000 0fd:00002 libX11.so.6.2.0
00002b938d4d7000 28 rw--- 0000000000105000 0fd:00002 libX11.so.6.2.0
00002b938d4de000 64 r-x-- 0000000000000000 0fd:00002 libXpm.so.4.11.0
00002b938d4ee000 2048 ----- 0000000000010000 0fd:00002 libXpm.so.4.11.0
00002b938d6ee000 4 rw--- 0000000000010000 0fd:00002 libXpm.so.4.11.0
00002b938d6ef000 140 r-x-- 0000000000000000 0fd:00002 libpng12.so.0.10.0
00002b938d712000 2048 ----- 0000000000023000 0fd:00002 libpng12.so.0.10.0
00002b938d912000 4 rw--- 0000000000023000 0fd:00002 libpng12.so.0.10.0
00002b938d913000 132 r-x-- 0000000000000000 0fd:00002 libjpeg.so.62.0.0
00002b938d934000 2044 ----- 0000000000021000 0fd:00002 libjpeg.so.62.0.0
00002b938db33000 4 rw--- 0000000000020000 0fd:00002 libjpeg.so.62.0.0
00002b938db34000 8 r-x-- 0000000000000000 0fd:00002 libXau.so.6.0.0
00002b938db36000 2044 ----- 0000000000002000 0fd:00002 libXau.so.6.0.0
00002b938dd35000 4 rw--- 0000000000001000 0fd:00002 libXau.so.6.0.0
00002b938dd36000 20 r-x-- 0000000000000000 0fd:00002 libXdmcp.so.6.0.0
00002b938dd3b000 2044 ----- 0000000000005000 0fd:00002 libXdmcp.so.6.0.0
00002b938df3a000 4 rw--- 0000000000004000 0fd:00002 libXdmcp.so.6.0.0
00002b938df3b000 32 r-x-- 0000000000000000 0fd:00002 json.so
00002b938df43000 2044 ----- 0000000000008000 0fd:00002 json.so
00002b938e142000 4 rw--- 0000000000007000 0fd:00002 json.so
00002b938e143000 2052 r-x-- 0000000000000000 0fd:00002 mbstring.so
00002b938e344000 2044 ----- 0000000000201000 0fd:00002 mbstring.so
00002b938e543000 48 rw--- 0000000000200000 0fd:00002 mbstring.so
00002b938e54f000 44 r-x-- 0000000000000000 0fd:00002 mysql.so
00002b938e55a000 2048 ----- 000000000000b000 0fd:00002 mysql.so
00002b938e75a000 8 rw--- 000000000000b000 0fd:00002 mysql.so
00002b938e766000 2384 r-x-- 0000000000000000 0fd:00002 libmysqlclient.so.18.0.0
00002b938e9ba000 2044 ----- 0000000000254000 0fd:00002 libmysqlclient.so.18.0.0
00002b938ebb9000 516 rw--- 0000000000253000 0fd:00002 libmysqlclient.so.18.0.0
00002b938ec3a000 20 rw--- 00002b938ec3a000 000:00000 [ anon ]
...
0002b939116f000 80 r-x-- 0000000000000000 0fd:00002 zip.so
00002b9391183000 2044 ----- 0000000000014000 0fd:00002 zip.so
00002b9391382000 12 rw--- 0000000000013000 0fd:00002 zip.so
00002b9391385000 1024 rw--- 00002b9391385000 000:00000 [ anon ]
00002b9391485000 32768 rw-s- 0000000000000000 000:0007c [ anon ]
00002b9393485000 4 ----- 00002b9393485000 000:00000 [ anon ]
00002b9393486000 12288 rw--- 00002b9393486000 000:00000 [ anon ]
00002b9394086000 55136 r---- 0000000000000000 0fd:00002 locale-archive
00002b939765e000 2048 rw--- 00002b939765e000 000:00000 [ anon ]
00002b939785e000 28 r--s- 0000000000000000 0fd:00002 gconv-modules.cache
00002b9397868000 16 r-x-- 0000000000000000 0fd:00002 libnss_dns-2.5.so
...
00007fff63676000 84 rw--- 00007ffffffe9000 000:00000 [ stack ]
00007fff637fb000 12 r-x-- 00007fff637fb000 000:00000 [ anon ]
ffffffffff600000 8192 ----- 0000000000000000 000:00000 [ anon ]
mapped: 401636K writeable/private: 23188K shared: 32984K

Notice here at the very bottom of the pmap, is a summary breaking down memory use into mapped, private and shared memory.

If we key in on the writeable/private number, we come away with an understanding that this particular process is using about 23mb of memory. If we assume that most of the other apache processes will be in the same range, then our total memory use for 20 processes, is going to be about 500mb+. While not completely accurate, it's closer to the truth than if we just looked at what ps was telling us and added those numbers up.

The other thing that should be obvious in scanning through the pmap, is that there are a lot of different libraries being used, and also a lot of apache modules. This is a place where you can start looking at the list, and asking some questions about what those modules do, and whether or not they are appropriate for your site.

I'll suggest if you want to explore memory utilization further, that you locate a copy of this amazing python memory use script written by Pádraig Brady, who was an engineer for Redhat and now works for Facebook.

It reports the total memory use by a process -- in essence summarizing the use by all the forked children. Give it a try.

Armed with this information you now have a way of determining how to configure your apache server conf file, to make the most efficient use of available server memory, as well as gain insight into whether you have enough resources to serve the amount of traffic you expect and encounter.

---

Notice from the map how many modules there are in memory, and start going through your list of apache modules, investigating whether you even want or need many of the modules that are part of the default. The more you disable, the more memory you will save.