Nagios and Icinga plugin for VCS

Description

The plugin could be kind of nasty depending on how familiar you are with
programming.  The more of a programmer you are, the more nasty it might
look so I apologize since my background is not in programming and I only
do so on a need-be basis.   

This is not a step by step guide on how to implement the plugin, but a
guide to the files and execution procedure.  All the file included are
RTF files so they can just be opened, but will need to rename them
accordingly.  If you have questions, please let me know.

Step 1: Grab the information from the VCS 

The file *1vcsstatus_grabSH.rtf *is a bash script that when executed
will grab status information from the VCS listed in the script.  The
main line is:

*echo 'xstatus zones' | ssh -l admin a.b.c.d >>
/home/some_user/vcs_scripts/navcsStatus;*

Notice there is no password.  The VCS's will not allow you to store keys
from servers and I did not store the password in the file.  So when you
execute it, you will need to manually enter the password for the admin
account on the VCS.  This is the largest downside to the plugin method.
If we can add ssh keys to the VCS's, then this part could be automated
securely.

#!/bin/bash
echo 'xstatus zones' | ssh -l admin a.b.c.d >> /home/some_user/vcs_scripts/navcsStatus;
exit

Step 2:  Parse the output from Step 1

The file *2statusparsePL.rtf *is a perl script that takes the output
from Step 1, and formats into a simple readable method that will be used
by the actual plugin.  When you look at the script, I tried to add as
much comment code as possible but if you have questions, again, just ask.

The output will be a sequential list of zones that look like this:
 *1:Internet2 Commons - Old:Active*

Here you will need to make sure you pay attention to the number and
name.  When you go to create the host file for Nagios/Icinga, the number
will be important in making sure it executes properly for that zone.  If
not, you may end up with false reports.

#!/usr/bin/perl
use vars;
$file = "/home/gds/navcsStatus";
if (!open (FILE, $file)) {
        print "ERROR: Status file not found : $file";
        exit(0);
    }
#$new_file = "./vcsstatus.php";
#if (-e $new_file) {
#       unlink($new_file);
#}
$stat_line = '';
$Name = '';
$Value = '';
$val_name = '';
$val_status = '';
$val_zone_num = '';
while(<FILE>) {
        my($line) = $_;
        chomp($line);          # Get rid of the trailling \n
        $line =~ s/^\s*//;     # Remove spaces at the start of the line
        $line =~ s/\s*$//;     # Remove spaces at the end of the line
        if ( ($line !~ /^#/) && ($line ne "") ){    # Ignore lines starting with # and blank lines
                ($Name, $Value) = split (/:/, $line);          # Split each line into name value pairs
                if( ($Name eq "Name") or ($Name eq "Status") or ($Name =~ m/^Zone*/)){          # Checks to see if $Name is Name or Status
                        $Value =~ s/^\s*//; # Removes spaces from the start of the value
                        $Value =~ s/\s*$//; # Removes spaces from the end of the value
                        if($Name =~ m/^Zone\s*/){ # Searches for the line that starts with "Zone X"
                                my($z_zone, $z_number); # Defines local variables for parsing and seperating Zone and X
                                $Name =~ s/^\s*//; # Removes extra space from beginning
                                $Name =~ s/\s*$//; # Removes extra space from the end
                                $Name =~ s/[:]*$//; # Removes the : fron the end of "Zone X:"
                                ($z_zone, $z_number) = split (/\s/,$Name); # Splits "Zone X" into "Zone" and "X"
                                $val_zone_num = $z_number; # Assigns "X" to variable $val_zone_num
                        }
                        if($Name eq "Name"){ #Checks for what the name of the line is, if it is "Name", it continutes
                                $Value =~ s/^["]*//; # Removes quotes from beginning of Name
                                $Value =~ s/["]*$//; # Removes quotes from end of Name
                                $val_name = $Value;  # Assigned the name of the zone to $val_name
                        }
                        if($Name eq "Status"){ #Checks for what the name of the line is.  It is "Status", it contintues
                                $val_status = $Value; #Assigned the value of "Status" ot $val_status
                        }
                        if( ($val_name ne '') && ($val_status ne '') ){ #Since it cycles through the lines a few times before it gets to the next variable, it checks to see if both Name and Status have values applies (IE not empty), If both values have values assigned, it continues in.  If not, it goes to the next line retraining any value that has been assiend to either $val_name or $val_status.
                                $stat_line .= "$val_zone_num:$val_name:$val_status\r\n"; # write the line giving the name and status of the zone.  Output will look like: XXX: Zone_name: Status.  IE:  1: Some_Zone: Active
                                $val_name = ''; # Clears out the varible for next loop through.
                                $val_status = ''; # Clears out the variable for the next loop through
                        }
                }
        }
        if($line eq "*s/end"){
                last;
        }
    }
        close FILE; # Closes the input file it read from.
        open FILE2, ">/home/gds/vcs_scripts/vcsstatus.cfg" or die $!; #Opens the file 'vcsstatus.php' for writing to.
        print FILE2 $stat_line; # writes the value of $stat_line to the file
        close FILE2; # Closes the file.

Step 3:  The Plugin

The plugin is the actual script that Nagios/Icinga will execute when it
reads it host configurations.  The file for the plugin script
is *3check_neighbor_status-PLUGIN.rtf*.  This should go into the plugins
folder of Nagios/Icinga and defined in the appropriate files.
 

#!/usr/bin/perl
use vars;
$file = "/home/gds/vcs_scripts/vcsstatus.cfg";
if (!open (FILE, $file)) {
        print "ERROR: Status file not found : $file\r\n";
        exit(0);
    }
#$new_file = "./vcsstatus.php";
#if (-e $new_file) {
#       unlink($new_file);
#}
$val_name = '';
$val_status = '';
$val_zone_num = '';
$ex_status = '';
$ne_id = $ARGV[0];
while(<FILE>) {
        my($line) = $_;
        chomp($line);          # Get rid of the trailling \n
        $line =~ s/^\s*//;     # Remove spaces at the start of the line
        $line =~ s/\s*$//;     # Remove spaces at the end of the line
        if($line =~ m/^($ne_id)/){
                ($val_zone_num, $val_name, $val_status) = split (/:/, $line);          # Split each line into name value pairs
                if( $val_zone_num = $ne_id ){
                        if( $val_status eq "Active"){
                                print "OK\n";
                                $ex_status = 0;
                                last;
                        }
                        elsif( $val_status eq "Warning"){
                                print "Warning!\n";
                                $ex_status = 1;
                                last;
                        }
                        elsif( $val_status eq "Failed"){
                                print "Critical!\n";
                                $ex_status = 2;
                                last;
                        }
                }
        }
        }
close FILE;
exit $ex_status;

Step 4: The Host Configuration

From here, you will still need to define your host configuration that
Nagios/Icinga will actually use to populate its system and know what
checks to run.  The file *host_definition_example.rtf* is sort of a
template you can use for this.  Notice the check command line:

*check_command check_neighbor_status!X*

The "X" here is the number of the host as a result from the output of
Step 2.  So for here I would name my first host "Internet2 Commons –
Old", and when I get to check_command line, it would be:

*check_command                   check_neighbor_status!1*
I already had my host configuration set but if you are doing this all
from scratch, you could easily write a perl script to take the names and
zone number from the output of Step 2 and write your entire host
configuration file from it.  I did not need to, but it shouldn't be too
difficult when using the host configuration template.

That is it for the plugin and files.  Let me know your thoughts.  Feel
free to modify.  Any major changes you make that you think will improve
upon it, let me know so I can keep it up to date.  If this is something
everyone thinks is very usefully, I can more formalize the files and
implementation procedure and place it on the wiki.
 

define host{
host_name                       ZoneName
alias                           ZoneAlias
hostgroups                      zone_group
address                         x.x.x.x
check_command                   check_neighbor_status!X
check_interval                  5
retry_interval                  1
max_check_attempts              5
check_period                    24x7
process_perf_data               0
retain_nonstatus_information    0
contact_groups                  contact_group
notification_interval           30
notification_period             24x7
notification_options            d,u,r
}

Management and Other Things

About 3 years ago I started a sort of GDS White Pages of sorts for us
here in North America/ USA.  It was never formerly announced and lost
traction to other projects and priorities but still may be of use.

The whole system is PHP + MySQL and based on an open source user
management project called Deadlock.  The front end user interface isn't
very impressive since only a handful of people ever registered for our
testing use, but could be very powerful.  It has ability to assign users
and manage users, as well as assign them zone management roles so they
can update their own zones and information.  When we were using GnuGK, I
used this site and DB to generate all of the GnuGK config files but not
any long since I have not had time to write any Perl and PHP scripts to
generate the configs for VCS.  Because I still use the back admin area
for other things I cannot share the admin login, but if during our next
meeting anyone would like to see the whole thing in and out I am more
than happy to share it.

You can view the front end here: 
https://das0.commons.internet2.edu/

There is no need to register because it was designed for North America
but the general setup could in theory be modified and recycled into a
larger project for management of all users/systems/servers.  Just a thought.

Feedback

Comments and feedback to the above can be sent to Nick nick (at) internet2.edu