Compress files and folders over the network without using rsync

The following command ssh’s to your remote server, tar + gzips a directory, and then outputs the compressed stream to your local machine.

This is a good alternative to rsync. Even though rsync can compress the transfer mid stream, the receiving end is still the un-extracted copy.

ssh -l username '(cd /home/mysql-backups/ && tar -czf - . -C /home/mysql-backups)' >> test.tar.gz 2>&1

To do the above command, and extract it on your end (after transferring the compressed file over the network), simply do the following :

ssh -l username '(cd /home/mysql-backups/ && tar -czf - . -C /home/mysql-backups)' | tar -xzf -

These commands could theoretically incorporate pgp encryption to encrypt and compress the archive before it travels across the network, for increased security. That is why this alternative to rsync may be preferential to some.

Obviously you could locally encrpyt + compress , then rsync, but its always a good idea to not utilize local storage for this process and keep all the storage capacity on the centralized storage system that you have already allocated.

Log compression Bash script

In my experience as a Systems Administrator, it has come up quite often to create a script to rotate and compress rather large log files.

These log files could be anything: java logs, apache logs (apache should have its own log rotation built in) and mail logs for example. This script has two modes : daily and monthly.

The daily mode is intended to be run daily (obviously) , gzipping the previous days log file. The monthly mode, run monthly (obviously), then tar’s up all the previous month’s gzip files into one big tarball.

Note that this script assumes the log filenames are assorted by the filename + date (year/month/day). This can obviously be modified to suit the specific syntax of your log file names.

Here is the script :

# Rotate / compress old logs
# Star Dot Hosting

yesterday=`date --date='1 day ago' +%Y-%m-%d`
lastmonth=`date --date='1 month ago' +%Y-%m`
lasttwomonth=`date --date='2 months ago' +%Y-%m`
currentmonth=`date "+%Y-%m-%d"`

#gzip yesterdays log
if [ "$1" = "daily" ]
        gzip $logdir/$logfilename.$yesterday.log
        exit 0

#tar all last month's logs on the 1st of each month
elif [ "$1" = "monthly" ]
        tar -C $logdir -cf $logdir/$logfilename.$lastmonth.tar $logdir/$logfilename.$lastmonth-*.log.gz && rm -f $logdir/$logfilename.$lastmonth-*.log.gz
        exit 0
        echo "no or invalid arguments given."
        echo "syntax : ./ daily or ./ monthly"
        exit 1

I simply make two crontab entries :

0 3 * * * /bin/sh /usr/local/bin/ daily
0 5 1 * * /bin/sh /usr/local/bin/ monthly

The above entries run the script daily at 3:00am, and monthly on the 1st of every month at 5:00am, this ensures the script isn’t run at the same time on the 1st as the daily job.

That’s it!