Centralized Backup Script

Hello There!

I thought I’d share a backup script that was written to consolidate backups onto one server instead of spreading the backup process across several servers. The advantages are somewhat obvious to consolidating the script onto one server, namely being that editing or making changes is much easier as you only have one script to edit.

The environment where this may be ideal would be for environments with 15-20 servers or less. I’d recommend a complete end-to-end backup solution for servers that exceed that number such as Bacula perhaps.

The bash shell script that I pasted below is very straightforward and takes two arguments. The first is the hostname or ip address of the destination server you are backing up. The next (and potentially unlimited) arguments will be single quote encased folders which you would want backed up.

This script is dependent on the server the script is running on having ssh key based authentication enabled and implemented for the root user. Security considerations can be made with IP based restrictions either in the ssh configuration, firewall configuration or other considerations.

It should be explained further that this script actually connects to the destination server as the root user, using ssh key authentication. It then initiates a remote rsync command on the destination server back to the backup server as a user called “backupuser”. So that means that not only does the ssh key need to be installed for root on the destination servers, but a user called “backupuser” needs to be added on the backup server itself, with the ssh keys of all the destination servers installed for the remote rsync.

Hopefully I did not over complicate this, because it really is quite simple :

Backup Server –> root –> destination server to backup — > backupuser rsync –> Backup Server

Once you implement the script and do a few dry run tests then it might be ready to implement a scheduled task for each destination server. Here is an example of one cron entry for a server to be backed up :

Script to distribute SSH Keys across many servers

Hello once again!

You may remember an earlier post that detailed how to implement SSH Key based authentication.

We believe it is important, when administering many (sometimes hundreds or thousands) of servers, to implement a strategy that can allow systems administrators to seamlessly run scripts, system checks or critical maintenance across all the servers.

SSH Key authentication allows for this potential. It is a very powerful strategy and should be maintained and implemented with security and efficiency as a top priority.

Distributing keys for all authorized systems administrators is something that would allow for the maintenance of this authentication system much easier — when an admin leaves or is dismissed, you need to be able to remove his or her’s keys from the “pool” quickly.

The idea behind this script is to have a centralized, highly secure and restricted key repository server. Each server in your environment would run this script to “pull” the updated key list from the central server. The script would run as a cron job and can run as often as you like. Ideally every 5-10 minutes would allow for quick key updates / distribution.

Here is the perl script :

Note that it downloads the public keys via http with wget. This can be easily modified to utilize https, if necessary, or perhaps even another protocol to make the transfer. HTTP Was chosen because the public keys are harmless and http is the easiest method. HTTPS would be desirable, however.

We hope this script helps you along the way towards making your life easier! 😉

SSH Key based authentication

Administering a large number of servers is never easy. Sometimes it’s necessary to streamline many “menial” tasks across the board. This is especially true when performing simple audits across all your linux and unix based servers.

What makes this process seamless and efficient is to implement ssh key authentication to replace password based authentication.

When implemented, commands can be executed via scripts in order to execute and gather all the information that’s required in order to complete your task.

Sound easy? It is!

Public Key Setup

First, confirm that OpenSSH is the SSH software installed on the client system. Key generation may vary under different implementations of SSH. The ssh -V command should print a line beginning with OpenSSH, followed by other details.

Key Generation

A RSA key pair must be generated on the client system. The public portion of this key pair will reside on the servers being connected to, while the private portion needs to remain on a secure local area of the client system, by default in ~/.ssh/id_rsa. The key generation can be done with the ssh-keygen(1) utility.

Depending on your work environment and the security policies and procedures, you may want to enter a passphrase. This effectively defeats the purpose of this, however. Entering an empty passphrase will allow you to complete a seamless connection.

The file permissions should be locked down to prevent other users from being able to read the key pair data. OpenSSH may also refuse to support public key authentication if the file permissions are too open. These fixes should be done on all systems involved.

Key Distribution

The public portion of the RSA key pair must be copied to any servers that will be accessed by the client. The public key information to be copied should be located in the ~/.ssh/id_rsa.pub file on the client. Assuming that all of the servers use OpenSSH instead of a different SSH implementation, the public key data must be appended into the ~/.ssh/authorized_keys file on the servers.

first, upload public key from client to server:

next, setup the public key on server:

Be sure to append new public key data to the authorized_keys file, as multiple public keys may be in use. Each public key entry must be on a different line.

Many different things can prevent public key authentication from working, so be sure to confirm that public key connections to the server work properly. If the following test fails, consult the debugging notes.

Key distribution can be automated with module:authkey and CFEngine. This script maps public keys stored in a filesystem repository to specific accounts on various classes of systems, allowing a user key to be replicated to all systems the user has access to.

If exporting the public key to a different group or company, consider removing or changing the optional public key comment field to avoid exposing the default username and hostname.


Quick tips using FIND , SSH, TAR , PS and GREP Commands

Administering hundreds of systems can be tedious. Sometimes scripting repetitive tasks, or replicating tasks across many servers is necessary.

Over time, I’ve jotted down several quick useful notes regarding using various linux/unix commands. I’ve found them very useful when navigating and performing various tasks. I decided to share them with you, so hopefully you will find them a useful reference at the very least!

To find files within a time range and add up the total size of all those files :

To watch a command’s progress :

Transfer a file / folders, compress it midstrem over the network, uncompress the file on the recieving end:

Below will return any XYZ PID that is older than 10 hours.

Check web logs on www server for specific ip address access:

Those are just a few of the useful commands that can be applied to many different functions. I particularly like sending files across the network and compressing them mid stream 🙂

The above kind of administration is made even easier when you employ ssh key based authentication — your commands can be scripted to execute across many servers in one sitting (just be careful) 😉