Atop – Apache Top, for keeping tabs on the web servers

When I first became a systems administrator of a large web server, I wanted to know what the current traffic to all of the virtual hosts (vhosts) looked like. I wanted to see which domains were getting the most traffic and where that traffic was coming from. So began my long search for a sufficient tool. There are many out there (apache-top, Apachetop, wtop, htop, IPTraf, etc). But they didn’t do all of the things I wanted. Basically they were just command line versions of the output of Apache mod_status, or they did complex log analysis.

I wanted more. The ability to search, or show only a certain domain name, see a list of IP address and how many connections from that IP address (to detect botnet attacks), and more.

So in true sys admin fashion, I built the tool myself. It is sufficiently stable and usable enough to warrant a blog post and hopefully engender some usage by others, which hopefully will encourage ideas and improvements from the community. Go ahead and grab a copy from the github repo, https://github.com/mossiso/atop

My idea is not much different than some of the tools I linked to. I’m basically writing a wrapper around the Apache mod_status output, but this tool has the ability to do more. So here’s a little walk through of what this tool does.

Requirements

  • Apache with mod_status: This tool is built around the Apache mod_status output, so that obviously has to be installed and set up. The ExtendedStatus option has to be enabled in the httpd.conf file.
  • links: This is a command line based web browser of sorts. Using the -dump flag, it just spits out the page to the command line.
  • netstat: This is used for one of the options to display all of the IPs connected to the webserver (via port 80).

 

This tool is just a BASH script, so once you download the “atop” file, just plop it anywhere in your home directory on your web server, change the permissions so it is executable

[code lang=”bash”]chmod 700 atop[/code]

and run it

[code lang=”bash”]./atop[/code]

There are now several options you can sort the results by:

==============================================================
a = sort all threads by time
c = sort by CPU, no GCRK_
i = list IPs connected to port 80 (uses Apache Server Status)
k = sort by K (Keep alives)
l = list IPs connected to all ports (uses netstat)
n = list IPs connected to port 80 (uses netstat)
o = sort open connections by CPU
p = sort only POST threads by time
r = raw apache status output (good with limit of at least 50)
s = search for a term, returns raw Apache Server Status results
w = sort by inactive workers
q = quit

To see the list of options while the command is running, just type any key on the keyboard.

Getting the BASH script to be responsive to the keyboard was tricky, and took me the longest time to figure out. For a while I could get the results to be displayed and refresh every N seconds, I could even get it to do the sort options, but only if I started the script with that option. So I was super excited to figure out the logic to get the script to respond to input.

The trick lies in setting the output commands in an infinite while loop. At the end of the loop it does a regular bash prompt using “read”. Normally this waits for a response, but the timeout feature allows you to set that to one second, which then goes through the while loop again. If a key is pressed, it breaks the while loop and prints the options message. When an option is selected it goes through that while loop.

Features

Some of the sort options I use most often are POST (p), CPU (c), IPs according to Apache (i), and IPs according to the server (n). I walk through those one by one.

POST

POST-listing

This is probably the most helpful of the options. Usually, when a website is getting hammered, it’s because it is getting comment spam or login attempts. These all require POST requests. If you see a large number of POST requests for a single vhost, then look at the IP addresses sending the requests; you can bet if all the requests are from the same IP, that it should be blocked.

CPU

CPU-list

This is a pretty good overview of what Apache traffic your server is handling. It shows GET and POST requests and sorts them with the most heavy CPU usage requests on the bottom. It filters out open processes with no connections, and a few other things like closing connections.

IPs (Apache)

IP-Apache-list

This one is great, too. It shows the IP addresses that are connected to Apache, and sorts them by how many connections are being made. The IPs with the most connections are at the bottom. If you see an IP address with over 10 connections for a few minutes, you can bet they are up to no good. Double check with the POST option to see if they are spamming.

IPs (Netstat)

IP-Netstat-list

This option gets all traffic to port 80 using netstat. It filters out local traffic (and GMU traffic, but you can edit that out), and then does the sorting and organizing by how many IP addresses are connecting. This gives a little more detail than the other IP option.

If you find any bugs in the script or have a great idea for other options, feel free to fork or submit patches, or report bugs on the github repo.

Python, Trac, virtualenv and CentOS

I’ve just spent too much time figuring this out. I’ve had to piece it together from many other sites.

I need to set up Trac .12 on CentOS 5, but want to do that without interfering with the current setup of Trac and Subversion on the system.

So in comes virtualenv. This allows you to create a virtual environment for python. Like a separate install. The beauty is, once this is set up you can install different versions of python packages (like Trac and Subversion) that don’t have to interact with each other.

To install virtualenv was pretty simple. With root permissions do

easy_install virtualenv

Now, as your normal user, you can install a virtual environment.

virtualenv --no-site-packages foo

This will create a new folder called foo with a virtual environment for python. It won’t reference any of the other installed python packages (like the old Trac version).

Now upgrade the Genshi package with

easy_install --upgrade Genshi

Then, to install Trac do

easy_install Trac==0.12

That’s the part that always hung me up. I would just do easy_install Trac  which would cough up this ugly error:

Searching for trac
Reading http://pypi.python.org/simple/trac/
Reading http://trac.edgewall.org/
Reading http://trac.edgewall.org/wiki/TracDownload
Reading http://trac.edgewall.com/
Reading http://projects.edgewall.com/trac
Reading http://projects.edgewall.com/trac/wiki/TracDownload
Best match: Trac 0.12.2
Downloading ftp://ftp.edgewall.com/pub/trac/Trac-0.12.2.zip
Processing Trac-0.12.2.zip
Running setup.py -q bdist_egg --dist-dir trac-dir/trac/egg-dist-tmp-JmdQXW
Traceback (most recent call last):
File "/home/ammon/foo/bin/easy_install", line 7, in ?
sys.exit(
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/command/easy_install.py", line 1712, in main
with_ei_usage(lambda:
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/command/easy_install.py", line 1700, in with_ei_u
sage
return f()
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/command/easy_install.py", line 1716, in <lambda>
distclass=DistributionWithoutHelpCommands, **kw
File "/usr/lib64/python2.4/distutils/core.py", line 149, in setup
dist.run_commands()
File "/usr/lib64/python2.4/distutils/dist.py", line 946, in run_commands
self.run_command(cmd)
File "/usr/lib64/python2.4/distutils/dist.py", line 966, in run_command
cmd_obj.run()
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/command/easy_install.py", line 211, in run
self.easy_install(spec, not self.no_deps)
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/command/easy_install.py", line 446, in easy_insta
ll
return self.install_item(spec, dist.location, tmpdir, deps)
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/command/easy_install.py", line 476, in install_it
em
dists = self.install_eggs(spec, download, tmpdir)
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/command/easy_install.py", line 655, in install_eg
gs
return self.build_and_install(setup_script, setup_base)
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/command/easy_install.py", line 930, in build_and_
install
self.run_setup(setup_script, setup_base, args)
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/command/easy_install.py", line 919, in run_setup
run_setup(setup_script, args)
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/sandbox.py", line 61, in run_setup
DirectorySandbox(setup_dir).run(
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/sandbox.py", line 105, in run
return func()
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/sandbox.py", line 64, in <lambda>
{'__file__':setup_script, '__name__':'__main__'}
File "setup.py", line 110, in ?
File "/usr/lib64/python2.4/distutils/core.py", line 110, in setup
_setup_distribution = dist = klass(attrs)
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/dist.py", line 260, in __init__
self.fetch_build_eggs(attrs.pop('setup_requires'))
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/setuptools/dist.py", line 283, in fetch_build_eggs
for dist in working_set.resolve(
File "/home/ammon/foo/lib/python2.4/site-packages/setuptools-0.6c12dev_r88795-py2.4.egg/pkg_resources.py", line 569, in resolve
raise VersionConflict(dist,req) # XXX put more info here
pkg_resources.VersionConflict: (Genshi 0.6dev (/usr/local/lib/python2.4/site-packages/Genshi-0.6dev-py2.4-linux-x86_64.egg), Requirement.parse('Genshi
>=0.6'))

Notice the last line referencing a version conflict with the “old” Genshi at /usr/local/lib/python2.4/site-packages. That’s the system-wide default install. So making explicit that you want to install Trac==0.12 is the way to get it installed in a virtual environment.

Now I just need to figure out how to configure Trac and Subversion using this virtual environment, and copy over a live older version of each.