Patch Scanning / Information Gathering Script for RedHat / CentOS

With all the patch management solutions, local repositories and other options, it is rarely necessary to manually scan all servers on your network to build a “report” of the patch levels in your environment.

Sometimes it is, however. For instance, if you are brought into an environment that has not been properly managed and require some quick audits to evaluate how much actual work needs to be done bringing all the patch levels up to standard, then there are ways to produce these reports with simple bash scripting.

I have developed such a script for similar situations — quick reporting is sometimes necessary even when you are evaluating a large commercial patch management solution. It can even be implemented to coincide such solutions, for independent reporting perhaps.

This script would work well either by distributing it to each server and running the script via ssh key based authentication for centralized reporting. Alternatively, you could modify this script to perform each command via SSH over the network to gather information that way. It is probably more ideal to centrally distribute the script to each server so only one ssh command is executed per server.

Find the script below — note that it only works with RedHat / CentOS systems. Obviously if you are paying for Red Hat enterprise support you already are using satellite; If you are using CentOS then this script may be useful for you.

Enjoy!

#!/bin/sh

# Basic Information Gathering
# Star Dot Hosting
# https://www.stackstar.com

HOSTNAME=`hostname`
UNAME=`uname -a | awk '{print $3}'`

# Begin Package Scanning


# SSH

SSHON="0"
SSHRUN="NULL"
SSHRPM="NULL"
SSHMATCH="NULL"


if [ -f /usr/sbin/sshd ]
then
        SSHON="1"
	SSHMATCH="0"
        SSHRUN=`ssh -V 2>&1 | awk 'BEGIN { FS = "_" } ; { print $2 }' | awk '{print $1}' | cut -b 0-5`
	TESTRPM=`rpm -qa openssh`
	if [ "$TESTRPM" <> 0  ]
	then
	        SSHRPM=`rpm -qa openssh | awk 'BEGIN { FS = "-" } ; { print $2 }'`
	fi
        if [ "$SSHRUN" == "$SSHRPM" ]
        then
                SSHMATCH="1"
        fi

fi

# Apache

HTTPDON="0"
HTTPDRUN="NULL"
HTTPDRPM="NULL"
HTTPDMATCH="NULL"


if [ -f /usr/sbin/httpd ]
then
        HTTPDON="1"
	HTTPDMATCH="0"
        HTTPDRUN=`httpd -v | grep version | awk 'BEGIN {FS="/"};{print$2}'`
	TESTRPM=`rpm -qa httpd`
	if [ "$TESTRPM" <> 0  ]
	then
        	HTTPDRPM=`rpm -qa httpd | awk 'BEGIN { FS = "-" } ; { print $2 }'`
	fi
        if [ "$HTTPDRUN" == "$HTTPDRPM" ]
        then
                HTTPDMATCH="1"
        fi
fi

# MySQL

MYSQLON="0"
MYSQLRUN="NULL"
MYSQLRPM="NULL"
MYSQLMATCH="NULL"


if [ -f /usr/bin/mysql ]
then
        MYSQLON="1"
	MYSQLMATCH="0"
        MYSQLRUN=`mysql -V | awk '{print $5}' | cut -b 0-6`
	TESTRPM=`rpm -qa mysql`
	if [ "$TESTRPM" <> 0  ]
	then
        	MYSQLRPM=`rpm -qa mysql | awk 'BEGIN { FS = "-" } ; { print $2 }'`
	fi
        if [ "$MYSQLRUN" == "$MYSQLRPM" ]
        then
                MYSQLMATCH="1"
        fi
fi

# PHP

PHPON="0"
PHPRUN="NULL"
PHPRPM="NULL"
PHPMATCH="NULL"


if [ -f /usr/bin/php ]
then
        PHPON="1"
	PHPMATCH="0"
        PHPRUN=`php -v | grep built | awk '{print $2 }'`
	TESTRPM=`rpm -qa php`
	if [ "$TESTRPM" <> 0  ]
	then
        	PHPRPM=`rpm -qa php | awk 'BEGIN { FS = "-" } ; { print $2 }'`
	fi
        if [ "$PHPRUN" == "$PHPRPM" ]
        then
                PHPMATCH="1"
        fi
fi

# Exim
# Needs to be tested on RH box

EXIMON="0"
EXIMRUN="NULL"
EXIMRPM="NULL"
EXIMMATCH="NULL"


if [ -f /usr/sbin/exim ]
then
        EXIMON="1"
	EXIMMATCH="0"
        EXIMRUN=`exim -bV | grep version | awk '{print $3}'`
	TESTRPM=`rpm -qa exim`
	if [ "$TESTRPM" <> 0  ]
	then
        	EXIMRPM=`rpm -qa exim | awk 'BEGIN { FS = "-" } ; { print $2 }'`
	fi
        if [ "$EXIMRUN" == "$EXIMRPM" ]
        then
                EXIMMATCH="1"
        fi
fi

# OpenSSL

OSSLON="0"
OSSLRUN="NULL"
OSSLRPM="NULL"
OSSLMATCH="NULL"


if [ -f /usr/bin/openssl ]
then
        OSSLON="1"
	OSSLMATCH="0"
        OSSLRUN=`openssl version | awk '{print $2}'`
	TESTRPM=`rpm -qa openssl`
	if [ "$TESTRPM" <> 0  ]
	then
        	OSSLRPM=`rpm -qa openssl | awk 'BEGIN { FS = "-" } ; { print $2 }'`
	fi
        if [ "$OSSLRUN" == "$OSSLRPM" ]
        then
                OSSLMATCH="1"
        fi
fi

# PERL

PERLON="0"
PERLRUN="NULL"
PERLRPM="NULL"
PERLMATCH="NULL"


if [ -f /usr/bin/perl ]
then
        PERLON="1"
	PERLMATCH="0"
        PERLRUN=`perl -v | grep built | awk '{print $4}' | awk 'BEGIN { FS = "v" } ; { print $2 }'`
	TESTRPM=`rpm -qa perl`
	if [ "$TESTRPM" <> 0  ]
	then
        	PERLRPM=`rpm -qa perl | awk 'BEGIN { FS = "-" } ; { print $2 }'`
	fi
        if [ "$PERLRUN" == "$PERLRPM" ]
        then
                PERLMATCH="1"
        fi
fi


# PYTHON

PYON="0"
PYRUN="NULL"
PYRPM="NULL"
PYMATCH="NULL"


if [ -f /usr/bin/python ]
then
        PYON="1"
	PYMATCH="0"
        PYRUN=`python -V 2>&1 | awk '{print $2}'`
	TESTRPM=`rpm -qa python`
	if [ "$TESTRPM" <> 0  ]
	then
        	PYRPM=`rpm -qa python | awk 'BEGIN { FS = "-" } ; { print $2 }'`
	fi
        if [ "$PYRUN" == "$PYRPM" ]
        then
                PYMATCH="1"
        fi
fi

# GPG

GPGON="0"
GPGRUN="NULL"
GPGRPM="NULL"
GPGMATCH="NULL"


if [ -f /usr/bin/gpg ]
then
        GPGON="1"
	GPGMATCH="0"
        GPGRUN=`gpg --version | grep gpg | awk '{print $3}'`
	TESTRPM=`rpm -qa gnupg`
	if [ "$TESTRPM" <> 0  ]
	then
        	GPGRPM=`rpm -qa gnupg | awk 'BEGIN { FS = "-" } ; { print $2 }'`
	fi
        if [ "$GPGRUN" == "$GPGRPM" ]
        then
                GPGMATCH="1"
        fi
fi

# RPM

RPMON="0"
RPMRUN="NULL"
RPMRPM="NULL"
RPMMATCH="NULL"


if [ -f /bin/rpm ]
then
        RPMON="1"
	RPMMATCH="0"
        RPMRUN=`rpm --version | awk '{print $3}'`
	TESTRPM=`rpm -qa rpm`
	if [ "$TESTRPM" <> 0  ]
	then
        	RPMRPM=`rpm -qa rpm | awk 'BEGIN { FS = "-" } ; { print $2 }'`
	fi
        if [ "$RPMRUN" == "$RPMRPM" ]
        then
                RPMMATCH="1"
        fi
fi

# SENDMAIL

SENDON="0"
SENDRUN="NULL"
SENDRPM="NULL"
SENDMATCH="NULL"


if [ -f /usr/sbin/sendmail ]
then
        SENDON="1"
        SENDMATCH="0"
        SENDRUN=`echo 'quit' | nc localhost 25 | grep Sendmail | awk '{print $5}' | awk 'BEGIN { FS = "/" } ; { print $1 }'`
	TESTRPM=`rpm -qa sendmail`
	if [ "$TESTRPM" <> 0  ]
	then
	        SENDRPM=`rpm -qa sendmail | awk 'BEGIN { FS = "-" } ; { print $2 }'`
	fi
        if [ "$SENDRUN" == "$SENDRPM" ]
        then
                SENDMATCH="1"
        fi
fi

### Non running packages

# bind-libs

BINDLIB="NULL"
TESTRPM=`rpm -qa bind-libs`
if [ "$TESTRPM" <> 0  ]
then
	BINDLIB=`rpm -qa bind-libs | awk 'BEGIN { FS = "-" } ; { print $3 }'`
fi


# bind-utils

BINDUTIL="NULL"
TESTRPM=`rpm -qa bind-utils`
if [ "$TESTRPM" <> 0  ]
then
	BINDUTIL=`rpm -qa bind-utils | awk 'BEGIN { FS = "-" } ; { print $3 }'`
fi

# coreutils

COREUTIL="NULL"
TESTRPM=`rpm -qa coreutils`
if [ "$TESTRPM" <> 0  ]
then
	COREUTIL=`rpm -qa coreutils | awk 'BEGIN { FS = "-" } ; { print $2 }'`
fi

# chkconfig

CHKCONFIG="NULL"
TESTRPM=`rpm -qa chkconfig`
if [ "$TESTRPM" <> 0  ]
then
	CHKCONFIG=`rpm -qa chkconfig | awk 'BEGIN { FS = "-" } ; { print $2 }'`
fi

# initscripts

INITSCR="NULL"
TESTRPM=`rpm -qa initscripts`
if [ "$TESTRPM" <> 0  ]
then
	INITSCR=`rpm -qa initscripts | awk 'BEGIN { FS = "-" } ; { print $2 }'`
fi

# redhat-release

RHRELEASE="NULL"
TESTRPM=`rpm -qa redhat-release`
if [ "$TESTRPM" <> 0  ]
then
	RHRELEASE=`rpm -qa redhat-release | awk 'BEGIN { FS = "-" } ; { print $3"-"$4 }'`
fi



echo $HOSTNAME,$UNAME,$SSHMATCH,$HTTPDMATCH,$MYSQLMATCH,$PHPMATCH,$EXIMMATCH,$OSSLMATCH,$PYMATCH,$PERLMATCH,$GPGMATCH,
$RPMMATCH,$SENDMATCH,$BINDLIB,$BINDUTIL,$COREUTIL,$CHKCONFIG,$INITSCR,$RHRELEASE,$SSHON,$SSHRUN,$SSHRPM,$HTTPDON,$HTTPDRUN,
$HTTPDRPM,$MYSQLON,$MYSQLRUN,$MYSQLRPM,$PHPON,$PHPRUN,$PHPRPM,$EXIMON,$EXIMRUN,$EXIMRPM,$OSSLON,$OSSLRUN,$OSSLRPM,$PERLON,
$PERLRUN,$PERLRPM,$PYON,$PYRUN,$PYRPM,$GPGON,$GPGRUN,$GPGRPM,$RPMON,$RPMRUN,$RPMRPM,$SENDON,$SENDRUN,$SENDRPM

Note that you can modify the echo output to produce whatever output you need in order to present it in a nice human readable report.

Scheduled antivirus scans to prevent viral injections on user generated content

When dealing with high traffic sites, especially media based or community based sites, there is always the risk of javascript, virus, XSS or other malicious injection of badness when giving a community of users the ability to upload files to your site.

There are several things to consider when evaluating all “points of entry” that are available to the public, into your systems.

Most content management and community based systems use libraries such as Imagemagick to process images (such as profile pictures) into their proper format and size.

Believe it or not, it is hard to actually inject code or other malicious data into the actual image to survive this sanitizing process. There is still risks , however. The library version you are running may be vulnerable to exploits itself.

As always, a good rule of thumb is to ensure all possible aspects of your systems are up to date and that you are aware of any security vulnerabilities as they come out so they can either be patched or addressed in some other way.

One thing to consider, especially when dealing with thousands of users and even more uploads is a scheduled scan of your user uploads using free virus scanning tools such as clamscan. This is an endpoint reporting strategy that can at least cover your ass in the event that something else was missed or a 0day vulnerability exploited.

It should be noted that the virus scans themselves aren’t intended to protect the linux systems themselves, but rather the opportunistic ‘spreading’ of compromised images and code that having an infected file on a public community based system can provide.

Its very simple to implement clamav (daemonization is not necessary), clamscan is all we need to execute regular scans at 10, 15, 30 or 60 minute intervals.

Once clamscan is implemented, definitions updated (and regular update cronjobs in place) you can roll out a script similar to the one we have here to implement the scheduled scans :

#!/bin/bash
# Scheduled Scan of user uploaded files
# Usage : ./virusscan.sh /folder


SUBJECT="[VIRUS DETECTED] ON `hostname` !"
EMAIL="you@yourdomain.com"
LOG=/var/log/clamav/scan.log

# Clear out old logs -- the email alerts should be archived if we need to go back to old alerts
echo "" > $LOG

# Check if the folder is empty -- only scan if this is an active node in a clustered system
# look for empty dir
if [ "$(ls -A $1)" ]
then
        # Scan files 
        clamscan $1 -r --infected --scan-pdf --scan-elf --log=$LOG

        # Check the last set of results. If there are any "Infected" counts that aren't zero, we have a problem.
        cat $LOG | grep Infected | grep -v 0
        if [ $? = 0 ]
        then
                cat $LOG | mail -s "$SUBJECT" $EMAIL -- -F Antivirus -f antivirus@yourdomain.com
        fi

else
        echo "directory empty -- doing nothing"
        exit 0;
fi

The actual cronjob entry can look something like this :

0 */1 * * * /bin/bash /usr/local/bin/virusscan.sh "/your/path/to/user/uploaded/files/" > /dev/null 2>&1

It seems like a rather simple solution — but it does provide a venue for additional sanitizing of user input. In our experience , it is best to only report on anything that clamscan might find. You can, however tell clamscan to simply delete any suspected infections it finds.

Backup a live FreeBSD filesystem and remotely migrate to another server

Lately we’ve been all about live migrations / backups here at *.hosting. And why not? with the advent of such concepts as “self healing blade cloud environment” , we have made a point to testing / scripting live migration scenarios.

Following on our last post of backing up LVM volumes, we have decided to make a simple post for ‘dumping’ a live freebsd filesystem, compressing it mid-stream, and over the network (encrpyted through ssh of course) , before being saved as a file (or restored to a waiting live-cd mounted system).

By default in FreeBSD, it partitions your var, usr, root Filesystem Size Used Avail Capacity Mounted on /dev/sd0s1a 989M 445M 465M 49% / /dev/sd0s1f 9.7G 5.2G 3.7G 59% /usr /dev/sd0s1e 19G 1.5G 16G 9% /var

So lets dump the root partition since its the smallest :

dump -0uanL -f - /dev/sd0s1a | bzip2 | ssh user@0.0.0.0 "dd of=dump-root.bzip2"

Lets break down the options so you can fully understand what its doing :

-0 // dump level 0 = full backup
-u // update the dumpdates file after a successful dump
-a // bypass all tape length considerations; autosize
-n // notify if attention is required
-L // tell dump that it is a live filesystem for a consistent dump; it will take a snapshot

Alternatively you could dump the filesystem to a local file :

dump -0uanL -f - /dev/sd0s1a | bzip2 | dd of=/home/backups/dump-root.bzip2

If you wanted to dump from server1 and restore on server2 :

dump -0uanL -f - /dev/sd0s1a | ssh user@0.0.0.0 "restore rf -"

Again , this is a straightforward command. It is typically fast (within reason). You could script this for automated dumps/snapshots of your filesystem for full restores in a disaster scenario.

Menu