Log compression Bash script

In my experience as a Systems Administrator, it has come up quite often to create a script to rotate and compress rather large log files.

These log files could be anything: java logs, apache logs (apache should have its own log rotation built in) and mail logs for example. This script has two modes : daily and monthly.

The daily mode is intended to be run daily (obviously) , gzipping the previous days log file. The monthly mode, run monthly (obviously), then tar’s up all the previous month’s gzip files into one big tarball.

Note that this script assumes the log filenames are assorted by the filename + date (year/month/day). This can obviously be modified to suit the specific syntax of your log file names.

Here is the script :

#!/bin/sh
# Rotate / compress old logs
# Star Dot Hosting

yesterday=`date --date='1 day ago' +%Y-%m-%d`
lastmonth=`date --date='1 month ago' +%Y-%m`
lasttwomonth=`date --date='2 months ago' +%Y-%m`
currentmonth=`date "+%Y-%m-%d"`
logdir="/path/to/log/directory"
logfilename="log-file-name"

#gzip yesterdays log
if [ "$1" = "daily" ]
then
        gzip $logdir/$logfilename.$yesterday.log
        exit 0

#tar all last month's logs on the 1st of each month
elif [ "$1" = "monthly" ]
then
        tar -C $logdir -cf $logdir/$logfilename.$lastmonth.tar $logdir/$logfilename.$lastmonth-*.log.gz && rm -f $logdir/$logfilename.$lastmonth-*.log.gz
        exit 0
else
        echo "no or invalid arguments given."
        echo "syntax : ./logcompress.sh daily or ./logcompress.sh monthly"
        exit 1
fi

I simply make two crontab entries :

0 3 * * * /bin/sh /usr/local/bin/logcompress.sh daily
0 5 1 * * /bin/sh /usr/local/bin/logcompress.sh monthly

The above entries run the script daily at 3:00am, and monthly on the 1st of every month at 5:00am, this ensures the script isn’t run at the same time on the 1st as the daily job.

That’s it!

Software RAID in Linux

Several occasions have arisen where a client requested software raid-1 between two IDE drives in their server.

Obviously the servers in question had no hardware raid capabilities, and compromising disk I/O read/write times for increased redundancy was more important.

Below is a simple tutorial for setting up software raid in Linux, using MDADM. The specific examples in these instructions are with Debian, but can be applied to any linux distribution.

Linux Software RAID-1

1. Verify that you are working with two identical hard drives:

      # cat /proc/ide/hd?/model
      ST320413A
      ST320413A

2. Copy the partition table over to the second drive:

      dd if=/dev/hda of=/dev/hdc bs=1024k count=10

Edit the partition table of the second drive so that all of the partitions, except #3, have type ‘fd’.

      # fdisk -l /dev/hda /dev/hdc

Disk /dev/hda (Sun disk label): 16 heads, 63 sectors, 38792 cylinders
      Units = cylinders of 1008 * 512 bytes

         Device Flag    Start       End    Blocks   Id  System
      /dev/hda1             0       248    124992   83  Linux native
      /dev/hda2           248      2232    999936   83  Linux native
      /dev/hda3             0     38792  19551168    5  Whole disk
      /dev/hda4          2232     10169   4000248   83  Linux native
      /dev/hda5         10169     12153    999936   83  Linux native
      /dev/hda6         12153     18105   2999808   82  Linux swap
      /dev/hda7         18105     28026   5000184   83  Linux native
      /dev/hda8         28026     38792   5426064   83  Linux native

      Disk /dev/hdc (Sun disk label): 16 heads, 63 sectors, 38792 cylinders
      Units = cylinders of 1008 * 512 bytes

         Device Flag    Start       End    Blocks   Id  System
      /dev/hdc1             0       248    124992   fd  Linux raid autodetect
      /dev/hdc2           248      2232    999936   fd  Linux raid autodetect
      /dev/hdc3             0     38792  19551168    5  Whole disk
      /dev/hdc4          2232     10169   4000248   fd  Linux raid autodetect
      /dev/hdc5         10169     12153    999936   fd  Linux raid autodetect
      /dev/hdc6         12153     18105   2999808   fd  Linux raid autodetect
      /dev/hdc7         18105     28026   5000184   fd  Linux raid autodetect
      /dev/hdc8         28026     38792   5426064   fd  Linux raid autodetect

3. Install mdadm to manage the arrays.

      apt-get install mdadm

It’ll ask you a series of questions that are highly dependent on your needs. One key one is: “Yes, automatically start RAID arrays”

4. Load the RAID1 module:

      modprobe raid1

5. Create the RAID1 volumes. Note that we’re setting one mirror as “missing” here. We’ll add the second half of the mirror later because we’re using it right now.

      mdadm --create -n 2 -l 1 /dev/md0 /dev/hdc1 missing
      mdadm --create -n 2 -l 1 /dev/md1 /dev/hdc2 missing
      mdadm --create -n 2 -l 1 /dev/md2 /dev/hdc4 missing
      mdadm --create -n 2 -l 1 /dev/md3 /dev/hdc5 missing
      mdadm --create -n 2 -l 1 /dev/md4 /dev/hdc6 missing
      mdadm --create -n 2 -l 1 /dev/md5 /dev/hdc7 missing
      mdadm --create -n 2 -l 1 /dev/md6 /dev/hdc8 missing

6. Make the filesystems:

      mke2fs -j /dev/md0
      mke2fs -j /dev/md1
      mke2fs -j /dev/md2
      mke2fs -j /dev/md3
      mkswap /dev/md4
      mke2fs -j /dev/md5
      mke2fs -j /dev/md6

7. Install the dump package:

      apt-get install dump

8. Mount the new volumes, dump & restore from the running copies:

      mount /dev/md1 /mnt
      cd /mnt
      dump 0f - / | restore rf -
      rm restoresymtable

      mount /dev/md0 /mnt/boot
      cd /mnt/boot
      dump 0f - /boot | restore rf -
      rm restoresymtable

      mount /dev/md2 /mnt/usr
      cd /mnt/usr
      dump 0f - /usr | restore rf -
      rm restoresymtable

      mount /dev/md3 /mnt/tmp
      cd /mnt/tmp
      dump 0f - /tmp | restore rf -
      rm restoresymtable

      mount /dev/md5 /mnt/var
      cd /mnt/var
      dump 0f - /var | restore rf -
      rm restoresymtable

      mount /dev/md6 /mnt/export
      cd /mnt/export
      dump 0f - /export | restore rf -
      rm restoresymtable

9. Set up the chroot environment:

      mount -t proc none /mnt/proc

      chroot /mnt /bin/bash

10. Edit /boot/silo.conf, and change the following line:

      root=/dev/md1

11. Edit /etc/fstab, and point them to the MD devices:

      # /etc/fstab: static file system information.
      #
      #                
      proc            /proc           proc    defaults        0       0
      /dev/md1        /               ext3    defaults,errors=remount-ro 0       1
      /dev/md0        /boot           ext3    defaults        0       2
      /dev/md6        /export         ext3    defaults        0       2
      /dev/md3        /tmp            ext3    defaults        0       2
      /dev/md2        /usr            ext3    defaults        0       2
      /dev/md5        /var            ext3    defaults        0       2
      /dev/md4        none            swap    sw              0       0
      /dev/hdc        /media/cdrom0   iso9660 ro,user,noauto  0       0
      /dev/fd0        /media/floppy0  auto    rw,user,noauto  0       0

12. Save the MD information to /etc/mdadm/mdadm.conf:

      echo DEVICE partitions >> /etc/mdadm/mdadm.conf

      mdadm -D -s >> /etc/mdadm/mdadm.conf

13. Rebuild the initrd (to add the RAID modules, and boot/root RAID startup information):

     mkinitramfs -o /boot/initrd.img-`uname -r` `uname -r`

14. Leave the chroot environment:

      exit

15. Unmount /boot. klogd uses the System.map file, and we need to kill it to unmount /boot.

      pkill klogd
      # wait a few seconds
      umount /boot

16. Add /dev/hda1 to /dev/md0 — the /boot mirror

      mdadm --add /dev/md0 /dev/hda1
      watch cat /proc/mdstat

Wait until the mirror is complete. CTRL-C to exit watch.

17. Mount the mirrored /boot:

      umount /mnt/boot
      mount /dev/md0 /boot

18. Stamp the boot loader onto both disks, and reboot:

      silo -C /boot/silo.conf && reboot

19. Assuming it booted up correctly, verify that we’re running on the mirrored copies:

      df -h

If so, add the other partitions into their respective mirrors:

      mdadm --add /dev/md0 /dev/hda1
      mdadm --add /dev/md1 /dev/hda2
      mdadm --add /dev/md2 /dev/hda4
      mdadm --add /dev/md3 /dev/hda5
      mdadm --add /dev/md4 /dev/hda6
      mdadm --add /dev/md5 /dev/hda7
      mdadm --add /dev/md6 /dev/hda8
      watch cat /proc/mdstat

And wait until the the mirrors are done building.

20. Edit /etc/mdadm/mdadm.conf and remove any references to the RAID volumes. Refresh the mdadm.conf information:

      mdadm -D -s >> /etc/mdadm/mdadm.conf

21. Rebuild the initrd one more time. The previous time only included one half of each mirror for root and swap.

      mkinitramfs -o /boot/initrd.img-`uname -r` `uname -r`

22. Reboot one more time for good measure. You now have software RAID1.

Testing the Software Raid & simulating a drive failure

Newer versions of raidtools come with a raidsetfaulty command. By using raidsetfaulty you can just simulate a drive failure without unplugging things off.

Just running the command

mdadm --manage --set-faulty /dev/md1 /dev/sdc2

First, you should see something like the first line of this on your system’s log. Something like the second line will appear if you have spare disks configured.

kernel: raid1: Disk failure on sdc2, disabling device. 
kernel: md1: resyncing spare disk sdb7 to replace failed disk

Checking /proc/mdstat out will show the degraded array. If there was a spare disk available, reconstruction should have started.

Try with :

mdadm --detail /dev/md1

Now you’ve seen how it goes when a device fails. Let’s fix things up.

First, we will remove the failed disk from the array. Run the command

mdadm /dev/md1 -r /dev/sdc2

Now we have a /dev/md1 which has just lost a device. This could be a degraded RAID or perhaps a system in the middle of a reconstruction process. We wait until recovery ends before setting things back to normal.

We re-establish /dev/sdc2 back into the array.

mdadm /dev/md1 -a /dev/sdc2

As disk returns to the array, we’ll see it becoming an active member of /dev/md1 if necessary. If not, it will be marked as an spare disk.

Checking for errors and alerting

Steps for setting up e-mail alerting of errors with mdadm:

E-mail error alerting with mdadm can be accomplished in several ways:

1. Using a command line directly

2. Using the /etc/mdadm.conf file to specify an e-mail address

NOTE: e-mails are only sent when the following events occur:

Fail, FailSpare, DegradedArray, and TestMessage

Specifying an e-mail address using the mdadm command line

Using the command line simply involves including the e-mail address in the command. The following explains the mdadm command and how to set it up so that it will load every time the system is started.

mdadm --monitor --scan --daemonize --mail=jdoe@somemail.com

The command could be put /etc/init.d/boot.local so that it was loaded every time the system was started.

Verification that mdadm is running can be verified by typing the following in a terminal window:

ps aux | grep mdadm

Creating a FreeBSD wireless access point

Access points are essentially wireless switches or hubs. Just like a switch or a hub, all clients communicate through the access point. FreeBSD allows us to easily create an access point with just very little configuration and just the right hardware

To set up a wireless access point using FreeBSD, you need to have a compatible wireless card. We are using a Prism 2-based chipset. For a complete list of cards that are supported, consult the man page for wi, or visit the Wireless Network Interface Section of the FreeBSD documentation site.

    • Configuring the kernel
  • Depending on how you wish to set up the access point will determine what options need to be added to the kernel config file. If the wireless network device is being installed on a server that is currently running as a Firewall/NAT, then we only need to compile the wireless device driver into the kernel:

    # Wireless NIC cards
    device          wlan            # 802.11 support
    device          an              # Aironet 4500/4800 802.11 wireless NICs.
    device          awi             # BayStack 660 and others
    device          wi              # WaveLAN/Intersil/Symbol 802.11 wireless NICs.
    device          wl              # Older non 802.11 Wavelan wireless NIC.

    Choose the appropriate driver for your card from the list and include the wlan device, then recompile and install your kernel.

    If this the wireless network device is going to be installed on a system that does not serve as a Firewall/NAT, then we would want to include the BRIDGE option, along with the appropriate wireless device driver in the kernel config file.

    # Ethernet bridging support
    option		BRIDGE
    
    # Wireless NIC cards
    device          wlan            # 802.11 support
    device          an              # Aironet 4500/4800 802.11 wireless NICs.
    device          awi             # BayStack 660 and others
    device          wi              # WaveLAN/Intersil/Symbol 802.11 wireless NICs.
    device          wl              # Older non 802.11 Wavelan wireless NIC.
    

    The bridging option will allow the wireless device to communicate with the wired ethernet interface. We must also add a couple of options to the /etc/sysctl.conf file in order to establish the bridge between the two interfaces:

    net.inet.ip.forwarding=1
    net.link.ether.bridge.enable=1
    net.link.ether.bridge.config=wi0,fxp0

    Be sure and replace fxp0 with whatever wired ethernet interface you are using with your FreeBSD installation. For information on bridging, consult the Bridging Section of the FreeBSD Handbook.

      Configuring the Wireless Interface

    The configuration of the wireless interface is fairly straightforward, we just need to add a few more options than if it were a wired ethernet interface. The following is an example of ifconfig options for a wireless interface:

    ifconfig wi0 inet 10.0.0.5 netmask 255.255.255.0 
    ifconfig wi0 ssid My_Network channel 11 media DS/11Mbps mediaopt hostap up stationname "My Network"

    Of course this can all be setup in the /etc/rc.conf file so that these settings are retained every time the system boots. From this point, your access point should be up and broadcasting. There are just a couple more options to consider

      Post Configuration

    As stated earlier, if the wireless interface is installed in a server that is functioning as a Firewall/NAT, then the bridging option is unecessary. We just need to add a couple of rules to our firewall configuration files to allow traffic to be passed from the wireless interface.

    If you are using PF as your Firewall/NAT solution, simply add the following lines to your /etc/pf.conf file

    pass in on wi0 from wi0:network to any keep state
    pass out on wi0 from any to $wi0:network keep state

    Replace wi0 with the appropriate interface name of your wireless card

    If you are using IPfilter as your Firewall/NAT solution, then simply add the following lines to your /etc/ipf.rules file

    pass in on wi0 from any to any keep state
    pass out on wi0 from any to any keep state

    Again, replace wi0 with the appropriate interface name of your wireless card.

      Administration

    Once the access point is configured and operational, we will want to see the clients that are associated with the access point. We can type the following command to get this information:

    root@host# wicontrol -l
    1 station:
    ap[0]:
            netname (SSID):                 [ My_Network ]
            BSSID:                          [ 00:04:23:60:89:d9 ]
            Channel:                        [ 11 ]
            Quality/Signal/Noise [signal]:  [ 0 / 51 / 0 ]
                                    [dBm]:  [ 0 / -98 / -149 ]
            BSS Beacon Interval [msec]:     [ 10 ]
            Capinfo:                        [ ESS ]

    Now you should have a complete functioning access point up and running. You are encouraged to read more about the wicontrol and wi commands for further information.

    Monitoring raw traffic on a Juniper Netscreen

    Occasionally I will run into situations where the only way to definitively diagnose network related problems is to perform raw traffic dumps on a main internal / external interface.

    The reasons for needing to perform this could be anything. I thought I’d share the quick and easy steps to perform in order to do a quick network traffic capture.

    Be warned though, that it is easy to overflow the console buffer and subsequently crash your firewall if you don’t narrow the scope of your capture enough.

    There exists a command on the juniper netscreen console that works the same way that tcpdump would, called “snoop”.

    To view the current snoop settings :

    snoop info

    To monitor all traffic from a particular ip address going to a particular port :

    snoop filter ip src-ip x.x.x.x dst-port 23

    To monitor all traffic on the network going to a particular ip address :

    snoop filter ip dst-ip x.x.x.x

    The above commands only SET the filter. You have to turn the filter on and monitor the buffer to actually view the results. Note that you should ensure that the scope of your filters are quite narrow as there is the risk of overflowing the console buffer and crashing the firewall if you are monitoring a wide scope.

    To view the filters and turn on snoop :

    clear dbuf
    snoop
    get dbuf stream

    Dont forget to clear the filters , dbuf stream and turn off snoop when your done :

    snoop off
    clear dbuf
    snoop filter delete

    That’s it!

    Dual Monitors in FreeBSD

    To those (few) of you out there that actually use FreeBSD as a workstation (myself included) , you may have had the opportunity to utilize dual monitors.

    As a Systems Administrator who is usually working on 3-4 things simultaneously , it is crucial to be able to function with enough screen space.

    One of the headaches I’ve encountered is trying to get my Dual monitors working with my Ati/Radeon video card in FreeBSD. I’ve written a little tutorial to help those who may need help or are thinking of implementing a second monitor.

    I found out my video driver and pci configuration by executing the following commands :

    X -configure
    
    pciconf -l

    I entered the following as my xorg.conf (ATI / RADEON video driver):

    Section "Files"
        RgbPath     "/usr/X11R6/lib/X11/rgb"
        FontPath   "/usr/X11R6/lib/X11/fonts/misc/"
        FontPath   "/usr/X11R6/lib/X11/fonts/TTF/"
        FontPath   "/usr/X11R6/lib/X11/fonts/Type1/"
        FontPath   "/usr/X11R6/lib/X11/fonts/75dpi/"
        FontPath   "/usr/X11R6/lib/X11/fonts/100dpi/"
        FontPath   "/usr/X11R6/lib/X11/fonts/local/"
    EndSection
    
    Section "Module"
            Load    "GLcore"
            Load    "i2c"
            Load    "bitmap"
            Load    "ddc"
            Load    "dri"
            Load    "extmod"
            Load    "freetype"
            Load    "glx"
            Load    "int10"
            Load    "type1"
            Load    "vbe"
    EndSection
    
    Section "InputDevice"
            Identifier      "Generic Keyboard"
            Driver          "kbd"
    EndSection
    
    Section "InputDevice"
            Identifier  "Configured Mouse"
            Driver      "mouse"
            Option      "Protocol" "auto"
            Option      "Device" "/dev/sysmouse"
            Option      "ZAxisMapping" "4 5 6 7"
    EndSection
    
    Section "Device"
            Identifier      "ATI Technologies, Inc. Radeon 9600 (R300 AP)"
            Driver          "ati"
            BusID           "PCI:1:0:0"
    Option "MergedFB" "true" #Enable MergedFB function
     Option "MonitorLayout" "TMDS, CRT" # Use LCD and CRT even if you have 2 LCD's or CRT's
     Option "OverlayOnCRTC2" "true"
     Option "CRT2Position" "LeftOf" #Physical location of your secondary monitor in relationship to your primary monitor.
     Option "MetaModes" "1280x1024-1280x1024" #Monitor Resolutions for Primary-Secondary monitors
    Option "MergedXineramaCRT2IsScreen0" "true" #determines which screen is going to be the primary screen; value can be "true" or "false
    EndSection
    
    Section "Monitor"
            Identifier      "Monitor gen'erico"
            Option          "DPMS"
            HorizSync       28-64
            VertRefresh     43-60
    EndSection
    Section "Screen"
            Identifier      "Default Screen"
            Device          "ATI Technologies, Inc. Radeon 9600 (R300 AP)"
            Monitor         "Monitor gen'erico"
            DefaultDepth    24
            SubSection "Display"
                    Depth           1
                    Modes           "1280x960" "1024x768" "800x600" "640x480"
            EndSubSection
            SubSection "Display"
                    Depth           4
                    Modes           "1280x960" "1024x768" "800x600" "640x480"
            EndSubSection
            SubSection "Display"
                    Depth           8
                    Modes           "1280x960" "1024x768" "800x600" "640x480"
            EndSubSection
            SubSection "Display"
                    Depth           15
                    Modes           "1280x960" "1024x768" "800x600" "640x480"
            EndSubSection
            SubSection "Display"
                    Depth           16
                    Modes           "1280x960" "1024x768" "800x600" "640x480"
            EndSubSection
            SubSection "Display"
                    Depth           24
                    Modes           "1280x1024" "1280x960" "1024x768" "800x600" "640x480"
                    Virtual     2624 1200
            EndSubSection
    EndSection
    
    Section "ServerLayout"
            Identifier      "Default Layout"
            Screen          "Default Screen"
            InputDevice     "Generic Keyboard"
            InputDevice     "Configured Mouse"
    EndSection
    
    Section "DRI"
            Mode    0666
    EndSection

    Starting KDE / Xorg initially displays two identical monitors. You need to use the “xrandr” utility to utilize the dual monitor configuration. The following script, when run after starting KDE will do this for you :

    #!/bin/sh
    xrandr --output DVI-1 --auto --output DVI-0 --auto --left-of DVI-1

    Optimizing the FreeBSD kernel

    Often we are asked by VPS clients utilizing the FreeBSD operating system, how can they trim down the kernel in order to utilize the full memory footprint potential.

    Without getting into too much detail, here are several things that we usually “omit” from the kernel options during make buildworld / buildkernel to provide for a 60-70% kernel footprint reduction in 7.1-PRERELEASE :

    makeoptions     DEBUG=-g                # Build kernel with gdb(1) debug symbols
    options         MSDOSFS                 # MSDOS Filesystem

    # Wireless NIC cards
    device          wlan            # 802.11 support
    device          wlan_wep        # 802.11 WEP support
    device          wlan_ccmp       # 802.11 CCMP support
    device          wlan_tkip       # 802.11 TKIP support
    device          wlan_amrr       # AMRR transmit rate control algorithm
    device          wlan_scan_ap    # 802.11 AP mode scanning
    device          wlan_scan_sta   # 802.11 STA mode scanning
    device          an              # Aironet 4500/4800 802.11 wireless NICs.
    device          ath             # Atheros pci/cardbus NIC’s
    device          ath_hal         # Atheros HAL (Hardware Access Layer)
    device          ath_rate_sample # SampleRate tx rate control for ath
    device          awi             # BayStack 660 and others
    device          ral             # Ralink Technology RT2500 wireless NICs.
    device          wi              # WaveLAN/Intersil/Symbol 802.11 wireless NICs.
    #device         wl              # Older non 802.11 Wavelan wireless NIC.

    device          ural            # Ralink Technology RT2500USB wireless NICs
    device          rum             # Ralink Technology RT2501USB wireless NICs

    You can remove more ,but that should reduce your kernel size significantly. You should be able to recompile the kernel as per the FreeBSD documentation

    Menu