June 1st, 2008 posted by Bender Rodríguez
Wherein we discuss the sweet spot for memory optimization in the Apache Web Server and Django powered websites.
When using Django under Apache and mod_python or mod_wsgi, you notice straight away how memory intensive even one website can be amongst all the other sites that do not use python. Even if your Django applications are solid, the overhead at the Apache layer can be quite disconserting if you plan to host dozens, if not hundreds of sites on one server, so forget about leaving your Apache configuration as you found it upon install.
We are using Red Hat Enterprise 5 in production, and CentOS 5 in staging, and Ubuntu 8 for everyday development, and across all three platforms you see the same results in terms of memory consumption. At one point, in production, each Django site would take up about 200 megs of memory after several days in operation. Someone had suggest to me that I perform a pkill -HUP on the apache process every hour or so, and that seemed to alleviate some of the problem, but it was not a long term solution.
Even though I have worked with Apache since 1.2, I am still a novice when it comes to the secret voodoo that makes up the httpd.conf file. So I studied the docs under performance and ServerLimit and all that, experimented on several combinations, and after several weeks of trial and error I think that I finally stumbled upon a viable incantation.
Here is what the default was for prefork MPM:
StartServers 15
MinSpareServers 15
MaxSpareServers 35
ServerLimit 256
MaxClients 256
MaxRequestsPerChild 10000
and here is what we are running with today:
StartServers 5
MinSpareServers 5
MaxSpareServers 6
ServerLimit 30
MaxClients 30
MaxRequestsPerChild 150
I am still not certain if I could improve on that combination of values or not, but it has worked wonders thus far. We curently have 15 sites running Django with mod_wsgi and memory usage vascilates between 600 and 800. Before the tuning, memory usage would never go down until we did a kill -HUP or restarted Apache. Now, depending on load, the memory goes up and down, never more than 900 and as little as 300 at times. And the response time for requests is tremendous.
We'll see what happens when we double the number of Django sites in the coming weeks.
"It was our own moral failure and not any accident of chance, that while preserving the appearance of the Republic we lost its reality."
Marcus Tullius Cicero