Manage Nagios with Scripts

Working at many different organisations over the past 10 years, I have been involved in the implementation and maintenance of many different monitoring implementations. These include commercial and open source implementations, such as :

– Nagios
– IP Monitor
– Uptime
– OpenNMS
– Zabbix

Although Nagios may not be the most scalable or dynamic solution, for some organisations that perhaps have 1-100 servers, Nagios may be the best solution.

Additionally, the ability to write custom plugins, as well as the inherent SSL / TLS encryption of the NRPE checks, it may be the most viable. There are pro’s and con’s for each solution out there, and it is completely dependant on the skill level, nature of environment and available time for management / maintenance.

During the course of utilising Nagios, we noticed that one of the most time consuming tasks was maintaining the flat file configuration for adding, removing and modifying hosts within Nagios.

As a result, it was decided to write a quick Perl based script to manage the day-to-day tasks of adding and removing hosts within Nagios. When all is said and done, it really does save ALOT of time. This script can be integrated with existing control based management situations or other automation scripts / solutions where command line options and external scripting / plugins are possible. This way, you can encompass a more rounded, standardised and reliable way of managing your systems in Nagios.

In order for the script to work, you need to have 3 types of servers :

– Windows
– Unix/Linux
– VPS (Virtual Private Server)

Obviously you can modify the script to encompass an unlimited number of categories. Basically the script has defined three pre-existing hosts in the nagios hosts.cfg / hostgroups.cfg and services.cfg files to model them when adding the new server, based on your input.

Please take a look at the script, hopefully it will help make your life a little easier! 😉

#!/usr/bin/perl

# Don't break me, I'm used by automated scripts.

###############################################################################
# Star Dot Hosting : www.stardothosting.com
# Nagios Config Manager
# Description: This program will add/remove entries from nagios.
# The files will be backed up in a archive before any changes are made.
###############################################################################
# Perl Libraries
use File::Copy;
use Switch;
###############################################################################
# Variables

###############################################################################
# Nagios file handlers
my $host_file = "/usr/local/nagios/etc/objects/hosts.cfg";
my $group_file = "/usr/local/nagios/etc/objects/hostgroups.cfg";
my $services_file = "/usr/local/nagios/etc/objects/services.cfg";
my $unixmatch = "sdh-unix" ;
my $windowsmatch  = "sdh-windows";
my $vpsmatch = "vps-server";
my $date = `date "+%d%m%y-%H%M%S"`;

###############################################################################
# Verify Arguments
if ((!$ARGV[0]) || (!$ARGV[1])) {
        &usage;
}

if (length($ARGV[1]) gt 1 ) {   print "Command options too long!n"; &usage; }

# Verify Nagios is working before we start
my $nagios = `nagios -v /usr/local/nagios/etc/nagios.cfg`;
        if ($nagios =~ /One or more problems was encountered while processing the config files/) {
        print "CRITICAL ERROR!nnNagios is already broken and we cannot continue!nPlease fix it!n";
                @error_array = split(/./, $nagios);
                for $error (@error_array) {
                $error=~s/^n//g;
                        print "$errorn" if  $error=~ /Error:/;
             }
        die "nnProgram Aborting before even starting due to nagios config error!n"

}


# Clean up any old tmp files.
unlink("/tmp/hosts.cfg.tmp");
unlink("/tmp/hostgroups.cfg.tmp");
unlink("/tmp/services.cfg.tmp");

###############################################################################
# The Main Program control statement.
###############################################################################
switch ($ARGV[1]) {

        case /d/i {     &delete;        }
        case /x/i {     &addEntry("x"); }
        case /w/i {     &addEntry("w"); }
        case /v/i {     &addEntry("v"); }
        else {
                print "Option: $ARGV[1] not found n";
                &usage;
        }
}
###############################################################################

###############################################################################
# Subroutines
###############################################################################

###############################################################################
## sub backup - Backs up the nagios config files that are to be modified
###############################################################################
sub backup {

# Backup The Nagios files into an archive.
        $date =~s/n//g;

        mkdir("/var/backup/nagios/$date", 0755 ) || die "Cannot create directory /var/backup/nagios/$daten";

        copy($host_file, "/var/backup/nagios/$date/hosts.cfg.bck"); #|| die "Cannot copy $host_file to /var/backup/nagios/$date/hosts.cfg.bckn";
        copy($group_file, "/var/backup/nagios/$date/hostgroups.cfg.bck"); #|| die "Cannot copy $host_file to /var/backup/nagios/$date/hostgroups.cfg.bckn";
        copy($services_file, "/var/backup/nagios/$date/services.cfg.bck"); #|| die "Cannot copy $service_file to /var/backup/nagios/$date/services.cfg.bckn";
}

###############################################################################
## sub openFile($filename) - returns the file to a buffer for parsing
###############################################################################
sub openFile {
        my $blob;
        my $file = shift;
        open (F, "< $file") or die "Can't open $file : $!";

        while(  ) {
                $blob .= $_;
        }

        close(F);
return $blob;

}

###############################################################################
###############################################################################


###############################################################################
## sub delete - Deletes the servername from the config files.
###############################################################################
sub delete {
        &backup;                # Backup the files before we do anything to them.
        &delete_host;
        &delete_hostgroup;
        &delete_services;
        &checkNagios;
}

###############################################################################
## sub delete_host - deletes the host entry from hosts.cfg
###############################################################################
sub delete_host {
        my $host_str = &openFile($host_file);
        my $pattern=$ARGV[0];   # The parser doesn't like the array so we just pass it to a variable.

        # parse the hosts.cfg file first
        # This regular expression is a defined host entry, if it can't find it
        # and assert that the hostname is part of that context, it will die.
        if ($host_str =~/define[^_]*.name.*(?s-i:$pattern)[^}]*./i) {
                print "command: $ARGV[1] : Deleting $ARGV[0] $1n" if $host_str =~s/define[^_]*.name.*(?s-i:$pattern)[^}]*.//g;
                print "Match: $ARGV[0]n" if $host_str =~/define[^_]*.name.*(?s-i:$pattern)[^}]*./i;
                print "Deleted $ARGV[0] from hosts.cfgn";

        # Write the successfull deleteion to a tmp file.
        open(HF, ">/tmp/hosts.cfg.tmp") || die "Cannot open /tmp/hosts.cfg.tmp";
        print HF $host_str;
    close(HF);

        } else { die "Could not find and entry for $ARGV[0] in hosts.cfgn"; };
}

###############################################################################
## sub delete_hostgroup - deletes the hostgroup entry
###############################################################################
sub delete_hostgroup {

        my $hostgrp_str = &openFile($group_file);
        my $pattern=$ARGV[0];   # The parser doesn't like the array so we just pass it to a variable.

        # search/replace the hostgroup.cfg file
        if ($hostgrp_str =~ /$pattern/i) {
                # If the server has a comma after it, we need to remove that too.. or breakage.
                if ($hostgrp_str =~ /$pattern,/i ) {
                        print "Deleted $ARGV[0], from hostgroups.cfgn" if $hostgrp_str =~ s/$pattern,//g;
                }

                        print "Deleted $ARGV[0] from hostgroups.cfgn" if $hostgrp_str =~ s/$pattern//g;

        } else {
                die "Could not find and entry for $ARGV[0] in hostgroups.cfgn";
        }
        open(HGF, ">/tmp/hostgroups.cfg.tmp") || die "Cannot open /tmp/hostgroups.cfg.tmp";
        print HGF $hostgrp_str;
        close(HGF);
}

###############################################################################
## sub delete_services - delete the serivices.cfg entry
###############################################################################
sub delete_services {

        my $services_str= &openFile($services_file);
        my $pattern=$ARGV[0];   # The parser doesn't like the array so we just pass it to a variable.


        # search/replace the hostgroup.cfg file
        if ($services_str =~ /$pattern/i) {
                # If the server has a comma after it, we need to remove that too.. or breakage.
                if ($services_str =~ /$pattern,/i ) {
                        print "Deleted $ARGV[0], from services.cfgn" if $services_str =~ s/$pattern,//g;
                }
                        print "Deleted $ARGV[0] from services.cfgn" if $services_str =~ s/$pattern//g;
         } else {
                die "Could not find and entry for $ARGV[0] in services.cfgn";
        }
        open(SF, ">/tmp/services.cfg.tmp") || die "Cannot open /tmp/services.cfg.tmp";
        print SF $services_str;
        close(SF);
}

###############################################################################
## sub checkNagios - checks nagios for errors and rolesback if so.
###############################################################################
sub checkNagios {


        copy("/tmp/hosts.cfg.tmp", $host_file) || print "Cannot copy /tmp/hosts.cfg.tmp to  $host_filen";
        copy("/tmp/hostgroups.cfg.tmp", $group_file) || print "Cannot copy /tmp/hostgroups.cfg.tmp to $host_filen";
        copy("/tmp/services.cfg.tmp", $services_file) || print "Cannot copy /tmp/services.cfg.tmp $service_filen";
        my $success = `nagios -v /etc/nagios/nagios.cfg`;

        if ($success =~ /One or more problems was encountered while processing the config files/) {
                print "CRITICAL FAILURE - See Errors!n";
                @error_array = split(/./, $success);
                for $error (@error_array) {
                $error=~s/^n//g;
                        print "$errorn" if  $error=~ /Error:/;
                        }
                print "nRestoring from backupnCheck /tmp/hosts.cfg /tmp/hostgroup.cfg /tmp/service.cfgn";
        copy("/var/backup/nagios/$date/hosts.cfg.bck", $host_file) || die "Cannot copy /var/backup/nagios/$date/hosts.cfg.bck to $host_filen";
        copy("/var/backup/nagios/$date/hostgroups.cfg.bck", $group_file) || die "Cannot copy /var/backup/nagios/$date/hostgroups.cfg.bck to $group_filen";
        copy("/var/backup/nagios/$date/services.cfg.bck", $services_file) || die "Cannot copy /var/backup/nagios/$date/services.cfg.bck $services_filen";

        } else {
                print "Nagios config reports success, restarting nagiosn";
                my $restart = `/etc/init.d/nagios reload`;
                print $restart;
        }

}

###############################################################################
## sub addEntry - adds the unix or windows host entry.
###############################################################################
sub addEntry {

        my $type = shift;
        my $pattern = $ARGV[0];
        my $host_str = &openFile($host_file);
        my $hostgrp_str = &openFile($group_file);
        my $services_str= &openFile($services_file);

        if ($host_str=~/$pattern/) { die "$ARGV[0] already in hosts.cfg, aborting!n"; }
        if ($hostgrp_str=~/$pattern/) { die "$ARGV[0] already in hostgroups.cfg, aborting!n"; }
if(($type eq 'w') || ($type eq 'x')) {
        if ($services_str=~/$pattern/) { die "$ARGV[0] already in services.cfg, aborting!n"; }
}
        # Some sanity checks to help prevent data entry errors
        if (!$ARGV[2]) { print "nNo Server Alias, aborting!nn"; &usage; }
        #if ($ARGV[2]=~/[0-9]{5,8}$/i) {} else { print "No Member ID!n"; exit 0}
        if (!$ARGV[3]) { print "nNo IP Address specified, aborting!nn"; &usage; }
        if ($ARGV[3]=~/[a-z]/i) { print "nIP Address $ARGV[3] is invalid, please double checknn"; exit 0; }
        if ($ARGV[3]=~/[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}/) {
                @ip = split(/./, $ARGV[3]);
        } else {
                print "nIP Address $ARGV[3] is invalid, please double checknn"; exit 0;
        }

        if (($ip[0] > 254) || ($ip[1] > 254) ||
            ($ip[3] > 254) || ($ip[4] > 254)) {

        print "nIP Address $ARGV[3] is invalid, please double checknn"; exit 0;
        }

# Passes sanity checks, back up the mo fo.
&backup;
# Check if windows or unix

switch ($type) {

        case "x" {      print "Unix!n";
                        $hostgrp_str =~ s/$unixmatch/$unixmatch,$pattern/g;
                        $services_str =~ s/$unixmatch/$unixmatch,$pattern/g;  }
        case "w" {      print "Windowsn";
                        $hostgrp_str =~ s/$windowsmatch/$windowsmatch,$pattern/g;
                        $services_str =~ s/$windowsmatch/$windowsmatch,$pattern/g; }

        else {          print "VPSn";
                        $hostgrp_str =~ s/$vpsmatch/$vpsmatch,$pattern/g;
                                }
        } # end switch


        # Add it to the host_str buffer.
        $host_str .= "define host{
        use                     sdh-dedicated
        host_name               $ARGV[0]
        alias                   $ARGV[2]
        address                 $ARGV[3]
        }nn";
        open(HF, ">/tmp/hosts.cfg.tmp") || die "Cannot open /tmp/hosts.cfg.tmp";
        print HF $host_str;
        close(HF);
        open(HGF, ">/tmp/hostgroups.cfg.tmp") || die "Cannot open /tmp/hostgroups.cfg.tmp";
        print HGF $hostgrp_str;
        close(HGF);

if(($type eq 'w') || ($type eq 'x')) {
        open(SF, ">/tmp/services.cfg.tmp") || die "Cannot open /tmp/services.cfg.tmp";
        print SF $services_str;
        close(SF);
        }
        &checkNagios;

}
###############################################################################
## sub usage - prints the usage when things don't add up from args
###############################################################################
sub usage{
        print "Usage: /usr/local/bin/nagios-add.pl    nn";
        print "Optional Flags:n";
        print "td delete a servern";
        print "tw add a windows servern";
        print "tx add a unix servern";
        print "tv add a VPS servernn";
        print "eg delete:t./usr/local/bin/nagios-add.pl sdh-server12 dn";
        print "eg add:tt./usr/local/bin/nagios-add.pl sdh-server12 x "sdh-server12 sdh-server12.stardothosting.com MemID:155" 192.168.111.10n";
        exit 0;
}

Shell Script to Report On Hacking Attempts

It is always a good idea , when implementing open source firewall implementations (iptables, pf, etc), to build in as much reporting and verbosity as possible.

Having verbose reports on the state of your firewall, intrusion attempts and other information is key to ensuring the health and integrity of your network.

Somewhere along the line, we wrote a script to provide daily reports on intrusion attempts to penetrate our network — this usually happens when someone exceeds certain connection thresholds.

It may not be the most informative data, but the script can be modified to provide other important statistical information. It can also be modified to be used with other firewall implementations. I’m certain it wouldn’t be hard to convert this script to utilise iptables.

Below you will find the script itself — it can be set to run daily as a cronjob perhaps. Also note that the script tries to resolve a hostname for the IP address to at least provide some quick & easy information to the security administrators when determining coordinated attacks or attacks coming from compromised systems.

#!/bin/bash
# SDH PFCTL Daily Hack Table check

yesterday1=`date -v -1d +"%b"`
yesterday2=`date -v -1d +"%e"`
yesterday_display=`date -v -1d +"%b %d %Y"`

echo "" > /var/log/tablecheck.log

/sbin/pfctl -vvsTables > /var/log/pfctltables.log

echo "Firewall Table Audit: " $yesterday_display >> /var/log/tablecheck.log
echo -e "----------------------------------">> /var/log/tablecheck.log
echo -e "" >> /var/log/tablecheck.log

for obj0 in $(cat /var/log/pfctltables.log | grep "-pa-r-" | awk -F "t" '{printf "%sn", $2}');
do
echo -e $obj0 "TABLE" >> /var/log/tablecheck.log
echo -e "--------------" >> /var/log/tablecheck.log

# this is because the date command outputs single digit non-aligned right, but pfctl doesnt display that way :(
if [ "$yesterday2" -le 9 ]
then
        /sbin/pfctl -t $obj0 -Tshow -vv | grep -A 4 -B 1 "$yesterday1  $yesterday2" >> /var/log/tablecheck.log 2>&1
else
        /sbin/pfctl -t $obj0 -Tshow -vv | grep -A 4 -B 1 "$yesterday1 $yesterday2" >> /var/log/tablecheck.log 2>&1
fi

if [ "$?" -eq 1 ]
then
        echo -e "No values found for yesterday" >> /var/log/tablecheck.log
        echo -e "" >> /var/log/tablecheck.log
else
        echo -e "Hostnames :" >> /var/log/tablecheck.log
        for obj1 in $(/sbin/pfctl -t $obj0 -Tshow -vv | grep -B 1 "$yesterday1 $yesterday2" | grep -v "Cleared" | grep -v "--");
        do
        iphostnm=`/usr/bin/nslookup $obj1 | grep -A1 "Non-authoritative answer" | grep "name" | awk -F "=" '{printf "%sn", $2}'`
        if [ "$?" -eq 0 ]
        then
                echo -e "$obj1 / $iphostnm" >> /var/log/tablecheck.log
        else
                echo -e "$obj1 / No host name found" >> /var/log/tablecheck.log
        fi
        done
       echo -e "" >> /var/log/tablecheck.log
fi


done

cat /var/log/tablecheck.log | mail -s "Firewall Table Report" you@youremail.com

Enjoy!

Network Audit Bash Script Using Netbios and Nmap

Working in a large office, it is sometimes necessary to use different network audit tools in order to properly assess the integrity and security of networks.

In order to quickly audit a network , I created this script to scan selected IPs, read from a configuration file, and compile a simple report to be emailed. The script can be modified to suit your needs, such as exporting the data to a database or perhaps an HTML report for a web based reporting site.

The script itself doesn’t do anything particularly special, however it has proven useful when you want to do a quick & dirty network audit.

There are other tools out there, such as OpenAudit, Nessus and Nmap that could do similar tasks. However, the important thing to remember here is that those tools (with the exception of open audit perhaps) can be incorporated into this script to perform regular scheduled audits.

This script could actually be updated to utilize nmap v5.0 — utilizing the new features plus ndiff could turn this script into a very powerful network analysis tool.

Hopefully some of you will find some use out of the script! Enjoy!

#!/bin/sh


# Basic Information Gathering
currentmonth=`date "+%Y-%m-%d"`

rm lindows.log

echo "Hostname Identification Audit: " $currentmonth >> lindows.log
echo -e "------------------------------------------" >> lindows.log
echo -e >> lindows.log
for obj0 in $(grep -v "^#" all_linux_windows_ips.txt);
do


# Check if windows
check=`nmap -e bge0 -p 3389 $obj0 | grep open`

if [ "$?" -eq 0 ]
        then
        windowshost=`nbtscan -v -s , $obj0 | head -n 1 | awk -F"," '{printf "%s", $2}'`
        if [ -n "${windowshost:+x}" ]
                then
                echo -e "$windowshostt: $obj0t: WINDOWS" >> lindows.log
                else
                echo -e "NETBIOS UNKOWNt: $obj0t: WINDOWS" >> lindows.log
        fi
        else
        # Check if linux or freebsd
        ssh_get=`ssh -l ims $obj0 '(uname | sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/' && hostname | sed 'y/abcdefghijklmnopqrstuvwxyz/ABCDEFGHIJKLMNOPQRSTUVWXYZ/')'`
        if [ "$?" -eq 0 ]
                then
                uname=`echo $ssh_get | awk -F" " '{printf "%s", $1}'`
                hostname1=`echo $ssh_get | awk -F" " '{printf "%s", $2}'`
                hostname2=`echo $hostname1 | awk -F"." '{printf "%s", $1}'`
                echo -e "$hostname2t: $obj0t: $uname" >> lindows.log
                else
                echo -e "UNKNOWN ERRORt: $obj0t: PLEASE CHECK HOST" >> lindows.log
        fi
fi
done

cat lindows.log | mail -s 'Windows/FreeBSD/Linux Host Audit' your@email.com

Note that the “all_windows_linux_ips.txt” is just a text file with the ip addresses of all hostnames on your network. It can be modified to simply utilize whole subnets to make it easier to perform the audit.

How to setup a slave DNS Nameserver with Bind

When implementing redundancy as far as DNS is concerned, automated is always better. In a hosting environment, new zone files are constantly being created.

This need for a DNS master/slave implementation where new zone files are transferred between the master nameserver and the slave became apparent as operations grew and geographic DNS redundancy became apparent.

Obviously some commercial dns products provide this type of functionality out-of-the-box, but I will show you how to do this with a simple Bind DNS distribution.

I wrote this tutorial to help you, hopefully, to create an automated DNS slave / zone file transfer environment. Obviously you can create as many slave servers as you feel necessary.

MASTER Server

1. Edit /etc/named.conf and add the following to the options section where xx.xx.xx.xx is the ip of your slave server.:

allow-transfer { xx.xx.xx.xx; };

2. Create a script with the following, where somedirectory is the directory on your SLAVE server to store the slave zones and where yy.yy.yy.yy is your MASTER server ip and somewwwdir is a directory browsable via http and finally someslavefile.conf is the output file to write you slave config:

#!/bin/sh
#
for domain in `/bin/grep ^zone /etc/named.conf |/bin/grep "type master" |/bin/awk '{print $2}' |/bin/awk -F" '{print $2}'`

do

/usr/bin/printf "zone "${domain}" { type slave; file "/var/named/slaves/somedirectory/${domain}.db"; masters { yy.yy.yy.yy; }; };n"

done > /var/www/html/somewwwdir/someslavefile.conf

3. Test the script to ensure it is writing out the appropriate format.

4. Run the script as any user with permission to write to an http visible directory via cron.

0 4 * * * /path/to/script > /dev/null 2>&1

SLAVE SERVER

1. Transfer the rndc.key file from your master server to the slave :

scp MASTERSERVER:/etc/rndc.key /etc/ns1rndc.key

2. Edit ns1rndc.key and change the name of the key definition.

3. Edit named.conf and add the following to the options section:

allow-transfer { zz.zz.zz.zz; };

4. Append the following to the named.conf file:

include "/etc/ns1rndc.key";
include "/path/to/someslavefile.conf";

5. Run the following commands

touch /path/to/someslavefile.conf
mkdir /var/named/slaves/somedirectory/
chown -R named:named /var/named/slaves/somedirectory/
/etc/init.d/named restart

6. Create a script:

#!/bin/sh
/usr/bin/wget http://yy.yy.yy.yy/somewwwdir/someslavefile.conf  -O /var/named/slaves/someslavefile.conf
/etc/init.d/named restart

7. Add to root’s crontab

0 4 * * * /path/to/script

In the second slave script, you see that the transfer is done via wget. This can be replaced by many other more secure methods. If ssh based key authentication is employed, a simple scp or even rsync can be utilized to accomplish the actual zone transfer.

Log compression Bash script

In my experience as a Systems Administrator, it has come up quite often to create a script to rotate and compress rather large log files.

These log files could be anything: java logs, apache logs (apache should have its own log rotation built in) and mail logs for example. This script has two modes : daily and monthly.

The daily mode is intended to be run daily (obviously) , gzipping the previous days log file. The monthly mode, run monthly (obviously), then tar’s up all the previous month’s gzip files into one big tarball.

Note that this script assumes the log filenames are assorted by the filename + date (year/month/day). This can obviously be modified to suit the specific syntax of your log file names.

Here is the script :

#!/bin/sh
# Rotate / compress old logs
# Star Dot Hosting

yesterday=`date --date='1 day ago' +%Y-%m-%d`
lastmonth=`date --date='1 month ago' +%Y-%m`
lasttwomonth=`date --date='2 months ago' +%Y-%m`
currentmonth=`date "+%Y-%m-%d"`
logdir="/path/to/log/directory"
logfilename="log-file-name"

#gzip yesterdays log
if [ "$1" = "daily" ]
then
        gzip $logdir/$logfilename.$yesterday.log
        exit 0

#tar all last month's logs on the 1st of each month
elif [ "$1" = "monthly" ]
then
        tar -C $logdir -cf $logdir/$logfilename.$lastmonth.tar $logdir/$logfilename.$lastmonth-*.log.gz && rm -f $logdir/$logfilename.$lastmonth-*.log.gz
        exit 0
else
        echo "no or invalid arguments given."
        echo "syntax : ./logcompress.sh daily or ./logcompress.sh monthly"
        exit 1
fi

I simply make two crontab entries :

0 3 * * * /bin/sh /usr/local/bin/logcompress.sh daily
0 5 1 * * /bin/sh /usr/local/bin/logcompress.sh monthly

The above entries run the script daily at 3:00am, and monthly on the 1st of every month at 5:00am, this ensures the script isn’t run at the same time on the 1st as the daily job.

That’s it!

Menu