Saturday, November 1, 2008

Monitoring AWS Ubuntu Instances using Munin

Another munin setup blog? Why?
There are way too many blog posts out there with incorrect info on how to setup munin. My process to get it setup infuriated me enough to break from work and write the post. If any blog tells you to run:
/usr/bin/munin-cron --force-root
or any other munin command using --force-root, you should instead run like hell in the opposite direction. Doing so will only cause you hours of hell figuring out why your graphs stopped updating. It turns out that by using --force-root you just hijacked ownership of the .rrd, .png, and .html files that munin uses; for more details, see here. Furthermore, anybody that wrote a blog telling you to do that clearly didn't wait to see whether their setup was successful before posting (or they made the unwise choice of running munin as root all the time).

There also is some bad information on the web about using the setting use_node_name when setting up groups and hosts in the munin master config file. Nearly every blog I found included enabling use_node_name despite the munin documentation saying very specifically,
"You will almost never want to set [use_node_name] to anything, as setting it to "yes" tends to break things." (source)
Doesn't that just scream "Don't do it!" And yet nearly everybody is blindly recommending it. The thing is that by enabling it, you can make a setup work where it otherwise wouldn't have if you screwed up the config by using inconsistent host names between the munin master and munin nodes. So, what should you do? Is it okay to enable use_host_name? My inclination is no, because it's just a naive hack around the fact that you screwed up the config. If the people that build munin say don't use it, then I say don't use it; just learn how to configure munin correctly.

So, without further introduction, here's a more comprehensive guide to setting up a single munin master to monitor multiple munin nodes, and since I did this on AWS, there are some snippets about do's and don'ts on AWS.

On the Munin Master
Quite simple, run:
apt-get install -y munin apache2
There. Munin master is installed. Easy, no? That just setup the appropriate directories, cron jobs, and permissions, so all you really need to do next is edit /etc/munin/munin.conf but we'll wait to do that until after the nodes are ready. Additionally, it installed apache so that you can view munin reports on the web.

On all Munin Nodes
Again, simple, run:
apt-get install -y munin-node
Told you that would be easy. Great, now time to configure each node. There are only a few things we need to configure on the node, and they are all located in /etc/munin/munin-node.conf. The properties to set are:
host_name my.hostname.com
allow ^120\.0\.0\.1$
The host_name setting will be the name by which your munin-node will report itself to the munin master. This is very important, though I didn't find it mentioned in any other blog, for reasons I will explain below when I show the master munin.conf config file. The short version is that it doesn't really matter what you call the node, just choose something that makes sense to you and remember it. For a database layer, something rational might be mysql.yourdomain.com for the master, and slave0.yourdomain.com and slave1.yourdomain.com for the slaves.

The allow setting specifies what master host(s) are allowed to connect to this munin node. Due to some perl specifics, this value has to be a regex, though that can come in handy. You can repeat the allow line as much as necessary.

This is where the AWS issue comes in: what value do we put for allow? Do we use the external address, such as ec2-127-0-0-1.compute-1.amazonaws.com? Do we use the internal address, domU-00-00-00-00-00-00.compute-1.internal? Or, do we use the ip address directly, say 127.0.0.1? This will resurface later, too, in how the master will connect to the nodes.

Which one you choose does matter. For example, if you use the ip address or the external AWS address, your AWS account will be charged for all data transfer and you will have to authorize port 4949 to allow the munin master to get through the firewall. Ergo, it really makes sense to use the internal representation. Furthermore, this allows you to keep the firewall closed, in which case you can put a nice little regex in the allow line to allow any of your instances to connect, thus saving yourself headache if your master external address changes, if you decide to switch the master to a different address, or for some reason add another master. So, on AWS, the easiest allow line entry is:
allow ^.*$
If you're not familiar with regex, that literally allows any munin master to connect, which means that it would be ridiculously stupid to do in any situation where your machines are not all behind a common firewall. Make sure to restart each munin-node (/etc/init.d/munin-node restart) after editing the config file to ensure the changes are live. Okay, now repeat that on as many machines/instances as you want to monitor. For more options configuring a node, see the munin documentation. When finished, continue.

Configuring the Munin Master
The config file for the munin master is /etc/munin/munin.conf. The basic idea here is to define groups and hosts. A simple group looks like:

[db.yourdomain.com]
address domU-00-00-00-00-00-00.compute-1.internal
And here's what that all means: You defined a group "yourdomain.com" and then the host "db.yourdomain.com" under that group. Then, you said that the munin node located at that address is the host named db.yourdomain.com in the group yourdomain.com. So, what is all this group nonsense? The simple version is that it's just how hosts are organized, and that makes for a nice display on the web. If you want to get more complex, you can configure different behaviors for groups or show totals by group. For more info on this file, see the munin documentation.

So, on to things that matter. The name of the host (db.yourdomain.com) MUST match the value of host_name that the munin node at the address specified will report. If it doesn't, then you won't see any data collected for the node. The exception is if you enable use_node_name, then graphs will show up because that setting tells munin to fetch "all configured plugins associated with the given node, both plugins that report the nodes host_name and none at all," (source). But, since use_node_name is not recommended, just make sure that the values of host_name in munin-node.conf an the name of the host in munin.conf are the same.

Now Wait
You only have to wait 5 minutes at the most. After that, you will be able to see the data collected from the web. This is where a lot of blogs get impatient and tell you to run:
/usr/bin/munin-cron --force-root
which will update munin databases right away, but will also hijack the permissions from user "munin" to the root user, thereby denying the munin user access to those files at a later time, which will stop all updates because the cron job installed in /etc/cron.d/munin runs /usr/bin/munin-cron as munin, not root. You know, if I recommended doing that in my blog, I'd feel like a jerk for leading people astray.

Okay, patience is not one of your virtues and you don't want to wait 5 minutes? Fine. Just do this instead:

root@xxx:/root$ su munin --shell=/bin/bash
munin@xxx:/root$ /usr/bin/munin-cron
There, that will let you run munin-cron as "munin", which lets you see your update right away without hijacking file permissions. Just hit ctrl+d when you're done to go back to whatever user you previously were.

Alright, hope that saves somebody some frustration.

13 comments:

Tim Parkin said...

Many thanks for this one.. the default ubuntu install used use_node_name and didn't work out the box.. changing the reference in both munin.conf and munin-node.conf did the trick (oh and commenting out use-node-name)

Many thanks..

Anonymous said...

I believe the allow any regex should be

^.*$

Otherwise it means "zero or more ."

Ryan said...

Typo, it's corrected now. Thanks for catching that.

Unknown said...

Correct me if I'm wrong but setting "allow ^.*$" will allow any machine in your EC2 region to connect to your Munin stats - unless you're using Amazon VPC.

Anonymous said...

su munin --shell=/bin/bash -c munin-cron

karya ismail said...

This is a great article, Thanks for giving me this information. Keep posting
Mebel Jati :
Mebel Jati :
Kursi Tamu Minimalis :
Mebel Jepara Murah :
Mebel Jati Jepara :
Mebel Jepara Online :
Mebel Jati Jepara :
Mebel Jepara :
Kursi Tamu Minimalis :
Furniture Jati Jepara :
Karya Priboemi :
Kursi Tamu Jati Murah :
Mebel jepara Online :
Mebel jepara Online :
Meja Makan Jati :
Mebel Jepara Murah :
Sofa Tamu Mewah
Meja Makan Jati Minimalis
Tempat Tidur Jati
Mebel Jati Jepara
Mebel Minimalis
Mebel Jepara Murah
Mebel Jepara Murah
Kamar Set Minimalis

Muhammad Taslimin said...

This is an amazing post actually provide valuable information, I will always be waiting for the next post, thank you.
Greetings from Jepara,
Jepara town is a small town located in Central Java Indonesia.
From the milestone Jepara has a very high historical value of cultural diversity, including Jepara furniture that is already known to foreign tourists, namely:
Mebel Jati Jepara
by Product Highlights:
Jual Mebel Murah - Toko Mebel Jepara - Interior Furniture Minimalis - Mebel Kayu jati jepara - Toko Mebel Jepara - Furniture Jepara Murah - Furniture Kayu Jati - Mebel Jepara - Furniture Minimalis - Jual Mebel - Mebel Kayu Jati - Jual Furniture - Furniture Jati - Tempat Tidur Minimalis Lemari Pakaian Minimalis Jati - Furniture Minimalis Jakarta - Kreasi Jepara - Griya Mebel Jepara - Jual Mebel Murah - Interior Jati - Furniture Murah
the most interesting part of the cultural heritage of Jepara is Ancestor Indonesia in the manufacture of traditional fabrics, namely:
Tenun Troso Jepara
by Product Highlights: - Kerajinan Rotan - Kerajinan Monel - Kaligrafi Kuningan - Kain Tenun Troso - Tenun Rang Rang - Tenun Bali - Tenun Baron - Terapi Ion Elektrik - Jamu Tradisional Jawa
Thanks.

Muhammad Taslimin said...

This is an amazing post actually provide valuable information, I will always be waiting for the next post, thank you.
Greetings from Jepara,
Jepara town is a small town located in Central Java Indonesia.
From the milestone Jepara has a very high historical value of cultural diversity, including Jepara furniture that is already known to foreign tourists, namely:
Mebel Jati Jepara
by Product Highlights:
Jual Mebel Murah - Toko Mebel Jepara - Interior Furniture Minimalis - Mebel Kayu jati jepara - Toko Mebel Jepara - Furniture Jepara Murah - Furniture Kayu Jati - Mebel Jepara - Furniture Minimalis - Jual Mebel - Mebel Kayu Jati - Jual Furniture - Furniture Jati - Tempat Tidur Minimalis Lemari Pakaian Minimalis Jati - Furniture Minimalis Jakarta - Kreasi Jepara - Griya Mebel Jepara - Jual Mebel Murah - Interior Jati - Furniture Murah
the most interesting part of the cultural heritage of Jepara is Ancestor Indonesia in the manufacture of traditional fabrics, namely:
Tenun Troso Jepara
by Product Highlights: - Kerajinan Rotan - Kerajinan Monel - Kaligrafi Kuningan - Kain Tenun Troso - Tenun Rang Rang - Tenun Bali - Tenun Baron - Terapi Ion Elektrik - Jamu Tradisional Jawa
Thanks.

farid anwar said...

the article is very nice and useful for readers.
by product highlights : Mebel Jepara Berkualitas, Interior & eksterior, Furniture Outdoor, Mebel Lemari Jepara, Mebel Jepara Minimalis.
Thank's

aaa kitty20101122 said...

pandora charms sale
chrome hearts online store
burberry scarf
adidas nmd
michael kors outlet
Kanye West shoes
yeezy boost
true religion jeans
michael kors handbags
asics running shoes

aaa kitty20101122 said...

michael kors outlet store
true religion jeans
air jordan
michael kors outlet
mlb jerseys
fitflops clearance
adidas superstar shoes
tory burch shoes
tory burch shoes
michael kors handbags

chenlina said...

ray ban
ugg outlet
coach outlet online
canada goose
ugg boots
pandora jewelry
vibram fivefingers
oakley vault
ralph lauren outlet
christian louboutin outlet
chenlina20180103

Pervez Joarder said...

Specifically special to learn this kind of. We should ensure you together with thank you for anyone results an individual necessary designed for causing the method beautiful site. At present stick to the following payday loans Respect only for outstanding site articles proper.