Getting a Grip on your Operations with Munin

Ever taken a midnight drive down a dirt road without any headlights on? While it’s certainly a thrilling (and stupid) thing to do, you shouldn’t do it with your data center. But do you know what the load of your web servers were this morning? Could you tell me how many MySQL “slow queries” were executed on your database yesterday? Have customers and co-workers praised you for your website’s performance?

If you answered “No” to any of these questions, I’d like to introduce you to an old friend – Munin. Munin is an open source, server load analysis tool. It uses Tobi Oetiker’s outstanding RRDtool for “real time” (at least within 5 minutes per default config) server information.

Installation and configuration is a breeze under Debian:

sudo apt-get install munin munin-node

‘munin’ is the server application, so you’ll only need to install ‘munin-node’ on the client machines.

Now, let’s configure the master instance (192.168.0.200) by identifying the client nodes it should poll:

dan@192.168.0.200:~$ sudo vi /etc/munin/munin.conf
...
# a simple host tree
[web-server-1]
    192.168.0.1
    use_node_name yes
[web-server-2]
    192.168.0.2
    use_node_name yes
[db-server]
    192.168.0.100
    use_node_name yes

For more detailed info check out how to configure the Munin master.

Setup the client node by instructing it which interface(s) it should collect data on (this will be the same IP as entered in the master config above) and which master(s) are allowed to poll:

dan@192.168.0.1:~$ sudo vi /etc/munin/munin-node.conf 
...
# Which port to bind to
host 192.168.0.1
...
allow ^192.168.0.200$

Check out the official node configuration page for more details. And remember to restart the client nodes after setup so they start writing data by:

dan@192.168.0.1:~$ sudo /etc/init.d/munin-node restart
Stopping Munin-Node: done.
Starting Munin-Node: done.

Wait five minutes or so and then check the master data files:

dan@192.168.0.200:$ less /var/lib/munin/munin-update.stats
...
FS|yourdoma.in|web-server-1|ntp_amspitz_at|0.07
FN|yourdoma.in|web-server-1|3.57
UN|yourdoma.in|web-server-1|0.01
UD|yourdoma.in|0.01
UT|7.77

After setting up a virtual host pointing to our master (DocumentRoot /var/www/munin), I had instant data pr0n:

While it helps to have all the servers within the same network, any intervening firewalls can be setup to allow the Munin master connection to the client nodes on port 4949. Check out the Munin Monitoring Wiki page for even more helpful tips and tricks!

Munin also has basic monitoring support, and there’s a Nagios plugin available to send alerts. Together, Munin and Nagios go a long way towards helping get your data center under control by allowing you to do real capacity planning. For network usage, check out John Kimball’s overview on free bandwidth monitors.

“Happy Hunting” and remember to share any interesting graphs (or successful optimizations) on the flickr Web Ops Visualization Pool!