Migrate from Linux to Xen with Rsync

I decided to write this little guide to provide the relatively simple steps needed to migrate your linux system to a Xen (HVM) virtual instance.

It is assumed that on your source and destination boxes, that you only have one root “/” partition. If you partitioned out your file system differently, you will have to accommodate that based on these instructions.

The following steps walk you through the process of migrating linux to Xen from start to finish :

1. Install the exact same version of linux on your destination server
This isn’t really 100% necessary, obviously. You could always boot into Finnix, partition your disk and install Grub. If you are uncomfortable doing that, install the distribution from start to finish. The file system will be overwritten anyways.

2. Boot into finnix on the destination system
If you have never used Finnix, it is a “self contained, bootable linux distribution”. I like it alot actually and have used it for similar purposes, rescue operations and the like.

3. Setup networking on both destination and source systems
If both systems are on the same network, you could assign local IP addresses to ensure the process of synchronisation is speedy and unobstructed.

Ensure you configure networking either way and that you set a root password and start ssh :

passwd
/etc/init.d/ssh start

4. Mount the partition that you want to copy to on the destination server
Remember, so far everything you are doing has been on the destination server. Mount the destination partition within finnix :

mount /dev/xvdb

5. On the source server, rsync all the files of the source partition to the destination partition
When logged into the source server, simply issue the following rsync command and direct it to the destination server’s partition you just mounted :

rsync -aHSKDvz -e ssh / root@12.34.56.78:/mnt/xvdb/

The rsync process will complete and the partition on the destination server should be ready to boot into. Remember to change the networking configuration if you dont want any IP conflicts to happen.

I hope this helps!

Creating a Xen template

One way to increase the efficiencies of Xen based systems is to utilize templates. VMware talks about this in their whitepaper for ESX2 best practices.

With Xen, you have to create your own. Here is a straight forward guide for how to do it.

1. Bootstrap a DomU named -tpl (e.g. centos4-tpl).

I recommend using a file-backed VBD, but partition or LVM volume will work fine as well. Here is an example /etc/xen/centos4-tpl

kernel = "/boot/vmlinuz-2.6.12.6-xenU"
memory = 256
name = "centos4-tpl" 
disk = [  'file:/opt/xen/domains/centos4-tpl/diskimage,sdb1,w','file:/opt/xen/domains/centos4-tpl/swapimage,sdb2,w'  ]
root = "/dev/sdb1 ro"
dhcp="dhcp

This is just a normal system (DomU) install – see Centos-4 on Xen for an example. Un-customize files

2.Inside the VM, edit the following files

/etc/hosts
remove any address lines other than localhost

/etc/sysconfig/network
use a generic hostname which will be unique to each deployment

NETWORKING=yes
HOSTNAME=centos4-tpl-changeme.example.com

/etc/sysconfig/network-scripts/ifcfg-eth0
should look like this:

DEVICE=eth0
ONBOOT=yes
BOOTPROTO=dhcp

also important – remove any line starting with HWADDR, e.g.:

HWADDR=00:10:5A:XX:YY:ZZ

Other configuration files to consider tweaking include /etc/dhclient.conf & /etc/hosts

3. Files to remove:

– SSH Host key files (auto-created at boot time)

rm -f /etc/ssh/*host*

4. Shutdown the template VM

xm shutdown centos4-tpl

You might normally link your VMs into /etc/xen/auto. I recommend against this as the template VM can be left shutdown until/unless you want to update it, saving valuable RAM and CPU cycles.

Clone the virtual disk Now we can deploy from the template by cloning the data into a clean diskimage (or partition or LVM volume). Create the diskimage using an appropriate size (must be larger than the template). Oh -the nice thing here is that there is flexibility. For instance, you can have a file-based diskimage and clone the data onto LVM volumes. As long as you can mount the (virtual) disks, you can clone templatized systems.

Here we use /mnt/disk to mount the new system disk, and /mnt/image to mount the template disk.

First, mount the template disk.

mount -o loop /opt/xen/domains/centos4-tpl/diskimage /mnt/image

Next, create and mount the new system (DomU) disk space & swap space.

mkdir -p /opt/xen/domains/cloned
cd /opt/xen/domains/cloned
dd if=/dev/zero of=diskimage bs=1024k count=2048
dd if=/dev/zero of=swapimage bs=1024k count=256
mkfs.ext3 diskimage
mkswap swapimage
mkdir -p /mnt/disk
mount -o loop /opt/xen/domains/cloned/diskimage /mnt/disk

Create the exclude file in /tmp/XenCloneExclude

proc/*
users/*
tmp/*
lost+found/
etc/mtab

Copy the data across

rsync -av -SHWD --exclude-from="/tmp/XenCloneExclude" /mnt/image/ /mnt/disk

Chroot into the newly copied template and fixup certain files

chroot /mnt/disk /bin/bash

Fix the hostname, etc in the files we “un-customized” in the template.

Exit, unmount both the template image and volume

umount /mnt/disk
umount /mnt/image

Setup your Xen config and be on your way!

cd /etc/xen
cp centos4-tpl cloned
(edit cloned to change name and paths to disk and swap)
xm create -c cloned

MySQL Query Log – diagnosing and debugging mysql

The general query log is a general record of what mysqld is doing. The server writes information to this log when clients connect or disconnect, and it logs each SQL statement received from clients. The general query log can be very useful when you suspect an error in a client and want to know exactly what the client sent to mysqld.

mysqld writes statements to the query log in the order that it receives them, which might differ from the order in which they are executed. This logging order contrasts to the binary log, for which statements are written after they are executed but before any locks are released. (Also, the query log contains all statements, whereas the binary log does not contain statements that only select data.)

To enable the general query log, start mysqld with the –log[=file_name] or -l [file_name] option.

If no file_name value is given for –log or -l, the default name is host_name.log in the data directory.

Server restarts and log flushing do not cause a new general query log file to be generated (although flushing closes and reopens it). On Unix, you can rename the file and create a new one by using the following commands:

shell> mv host_name.log host_name-old.log
shell> mysqladmin flush-logs
shell> cp host_name-old.log backup-directory
shell> rm host_name-old.log

Before 5.0.17, you cannot rename a log file on Windows while the server has it open. You must stop the server and rename the file, and then restart the server to create a new log file. As of 5.0.17, this applies only to the error log. However, a stop and restart can be avoided by using FLUSH LOGS, which causes the server to rename the error log with an -old suffix and open a new error log.

Remove mail headers in Postfix outgoing mail

This post is intended for people who want to set up Postfix to remove specific headers within emails that pass through their systems. The most common use for this is to set up a relaying server that will remove any reference of where source emails originated and relevant information about the sender’s computer. Another useful application for this type of header_checks is to remove details about additional functions of your mail server that you do not want made available to the world.

This guide focuses on postfix’s header_checks capabilities, and although there are other ways to do so, we’ve found that this is by far the simplest.

IMPORTANT NOTES

Use these instructions at your own risk. Never test things out in a production environment!

In order for this to work, your main.cf file will have to have a reference to the header_checks file as follows:

header_checks = regexp:/etc/postfix/maps/header_checks

It is recomended that you keep all of your postfix map files in one directory along with any checks files. In this case, these will be kept in /etc/postfix/maps.

HEADER_CHECKS DETAILS

In addition to any spam filters (see our header_checks file for more information), the below lines should be added to your header_checks file to preserve privacy and remove headers for the internal operations of your mail server:

    # Sample For Dropping Headers: 	
    #/^Header: IfContains/ 	IGNORE
    /^Received: from 127.0.0.1/ 	IGNORE
    /^User-Agent:/ 	IGNORE
    /^X-Mailer:/ 	IGNORE
    /^X-Originating-IP:/ 	IGNORE

Each line above will search for headers tha have the content between the /^ and the / and will remove each line within the email headers that matches. As an example, the line “/^Received: from 127.0.0.1 .*/ IGNORE” will erase any lines from the email headers that list previous handoffs from an internal mail process to another. This is most commonly used for antivirus or antispam functions on a mail server.

The following lines are related to Anomy Sanitizer and SpamAssassin – two very useful products. These three lines will remove references from the headers for the two software packages, making sure that the users of the system will not easily identify the software that is running on the back end.

    # Sample For Dropping Headers: 	
    #/^Header: IfContains/ 	IGNORE
    /^Received: from 127.0.0.1/ 	IGNORE
    /^X-Sanitizer:/ 	IGNORE
    /^X-Spam-Status:/ 	IGNORE
    /^X-Spam-Level:/ 	IGNORE

If one were to want to remove all headers relevant to personal information and previous hosts on which the email has passed, the following would be a possible configuration. Note that by removing all of this information, some mail servers will automatically identify emails passing through this system as spam. You will also be removing useful information for troubleshooting any problems that may arise with the mail server.

    # Sample For Dropping Headers: 	
    #/^Header: IfContains/ 	IGNORE
    /^Received:/ 	IGNORE
    /^User-Agent:/ 	IGNORE
    /^Message-ID:/ 	IGNORE
    /^X-Mailer:/ 	IGNORE
    /^X-MimeOLE:/ 	IGNORE
    /^X-MSMail-Priority:/ 	IGNORE
    /^X-Spam-Status:/ 	IGNORE
    /^X-Spam-Level:/ 	IGNORE
    /^X-Sanitizer:/ 	IGNORE
    /^X-Originating-IP:/ 	IGNORE

Hopefully this will help you clean your mail headers up! šŸ™‚

Automatically Deploy Debian Firewalls with bash scripting

Automation is as necessary as any other aspect of systems administration in any critical or production environment where growth and scalability are moving at a significant pace.

Growth in any organization is obviously a good thing. In the systems administrator’s perspective, however, growth can mean more time spent deploying systems and less time spent focusing on other duties.

Automating the server deployment process is the natural next step when your organization has grown to a point where time efficiency becomes more relevant and noticeable to your business owners.

This is the first in a series of posts here where we will explain and share shell scripts that automate the deployment process of several key debian linux based systems. These scripts automate the patching, configuration and implementation of said systems.

They will certainly have to be modified to fit your organization’s needs and standards obviously, but hopefully it will give you a starting point to base your automation / roll-out policies.

Making your life easier and more automated is always a good thing! šŸ˜‰

#!/bin/sh
# Debian FW deployer script
# Version 1.0

PROGNAME="$0"
VERSION="1.0"

# working directory for deployer process.
WORKDIR="/root"

# tasks left (this is updated every step to accommodate recovery during
# the deployer  process)
TASKS="./deploy-fw.tasks"

init_tasks() {
	# This function will write a new tasks file.
	# it's called from the main body of the script if a tasks file does not exist.
	cat > $TASKS< /etc/hostname
	echo `hostname` > /etc/mailname
	echo "done."
	return 0
}

usage() {
	echo "[+] Usage: $PROGNAME"
	echo
	return 0
}

###############################
### MAIN SCRIPT STARTS HERE ###
###############################

# installer_splash
installer_splash

# fix working dir.
cd $WORKDIR

# does our installer file exist? if not, initalize it.
if [ ! -f $TASKS ]
then
	echo "[+] No task file found, installation will start from beginning."
	init_tasks
	if (($? != 0))
	then
		echo "[!] ERROR: Cannot create tasks file. Installation will not continue."
		exit 1
	fi
else 
	echo "[+] Tasks file located - starting where you left off."
fi

# start popping off tasks from the task list and running them.
# pop first step off of the list
STEP=`head -n 1 $TASKS`
while [ ! -z $STEP ]
do
	# execute the function.
	echo -e "nn###################################"
	echo "[+] Running step: $STEP"
	echo -e "###################################nn"
	$STEP
	if (($? != 0))
	then
		# command failed.
		echo "[!] ERROR: Step $STEP failed!"
		echo "    Installation will now abort - you can pick it up after fixing the problem"
		echo
		exit 1
	fi
	# throw up a newline just so things don't look so crowded
	echo
	# remove function from function list.
	perl -pi -e "s/$STEPn?//" $TASKS || exit 1
	STEP=`head -n 1 $TASKS`
done

# clean_up_and_reboot
echo "[+] Installation finished - cleaning up."
clean_up_and_reboot

# script is done now - termination should happen with clean_up_and_reboot.
echo "[!] Should not be here!"
exit 1

Relay Exim mail to google mail in Debian Linux

Sometimes its necessary to relay your mail through a third party provider. If your server environment has a dedicated sendmail server (most do), then this scenario is applicable to you. It is ideal to centralize your outgoing mail to one server so that changes, policies and configuration is located in a single place.

In this scenario, outgoing mail is relayed to google’s domain mail in an Exim mail environment. These steps are fairly straightforward and will hopefully help you to utilize google’s free mail service to send your mail.

Note that google has queuing and mass mail restrictions so if you plan on sending alot of mail this way, you will just get blocked.

    Step 1

Run dpkg-reconfigure exim4-config

1. Choose mail sent by smarthost; received via SMTP or fetchmail

2. Type System Mail Name: e.g. company.com

3. Type IP Adresses to listen on for incoming SMTP connections: 127.0.0.1

4. Leave Other destinations for which mail is accepted blank

5. Leave Machines to relay mail for: blank

6. Type Machine handling outgoing mail for this host (smarthost): smtp.gmail.com::587

7. Choose NO, don’t hide local mail name in outgoing mail.

8. Chose NO, don’t keep number of DNS-queries minimal (Dial-on-Demand).

9. Choose mbox

10. Choose NO, split configuration into small files

11. Mail for postmaster. Leaving blank will not cause any problems though it is not recommended

    Step 2

1. Open the file /etc/exim4/exim4.conf.template
2. Find the line .ifdef DCconfig_smarthost DCconfig_satellite and add the following in that section

 send_via_gmail:
 driver = manualroute
 domains = ! +local_domains
 transport = gmail_smtp
 route_list = * smtp.gmail.com

If you have any other smarthost defined with ā€œdomains = ! +local_domainsā€ remove that smarthost.

3. Find the ā€œbegin authenticatorsā€. In that section add the following

 gmail_login:
 driver = plaintext
 public_name = LOGIN
 client_send = : yourname@gmail.com : YourGmailPassword

Make sure you have no other authenticators with the same public_name (LOGIN). Comment them out if needed (Thanks Jakub for reminding me)

4. Find the comment ā€œtransport/30_exim4-config_remote_smtp_smarthostā€. In that section add

 gmail_smtp:
 driver = smtp
 port = 587
 hosts_require_auth = $host_address
 hosts_require_tls = $host_address
    Step 3

1. Run update-exim4.conf

2. Do /etc/init.d/exim4 restart

That should be it. You can test by using the command line mail client.

Test :

 echo "test" | mail -s "subject" test@email-to-send-to.com

Integrate your custom IPTables script with Linux

How do I integrate my custom iptables script with Red Hat Enterprise Linux?

A custom iptables script is sometimes necessary to work around the limitations of the Red Hat Enterprise Linux firewall configuration tool. The procedure is as follows:

1. Make sure that the default iptables initialization script is not running:

service iptables stop

2. Execute the custom iptables script:

sh [custom iptables script]

3. Save the newly created iptables rules:

service iptables save

4. Restart the iptables service:

service iptables restart

5. Verify that the custom iptables ruleset have taken effect:

service iptables status

6. Enable automatic start up of the iptables service on boot up:

chkconfig iptables on

The custom iptables script should now be integrated into the operating system.

Troubleshooting Mail : Basic Tips

In a shared or managed hosting environment, many people would agree that one of the most commonly reported issue with our helpdesk is email.

It has become a best practice to perform basic email troubleshooting , depending on the error / bounce back reported. These steps could be performed before or after the issue has been fixed — either to confirm it is working or to reproduce a particular problem.

The purpose of this document is to describe how mail gets from one users email client to another users mailbox. Each action that takes place can be tested and verified by an some level of admin with a few exceptions. After going through this document you should understand what takes place when a user sends email or has email sent to them.

Test SMTP Authentication

SMTP users BASE64 encoding to transmit usernames and passwords. You will need to take the username of the client ( user@domain.com ) and the password and convert them to BASE64. Links provided below to do this:

* http://www.webpan.com/customers/Email/base64_conversion.htm
* http://makcoder.sourceforge.net/demo/base64.php
* http://www.opinionatedgeek.com/dotnet/tools/Base64Encode/

telnet mail.domainname.com 25
helo mailserver
auth login
BASE64 Encoding of USERNAME@domain.com
BASe64 Encoding of PASSWORD

Test Sucessful if: 235 Authentication successful

Testing if user is local or remote

Our mailserver will only accept email for a remote host if you authenticate first. Without authentication it will only accept email for local domains.

telnet mail.domainname.com 25
helo mailserver
mail from:  testingadmin@stardothosting.com
rcpt to: superadmin@gmail.com

Local Domain if: 250 ok
Remote Domain if: 553 sorry, that domain isn’t in my list of allowed rcpthosts

Looking up an MX record

Two or three lookups should be done off of various DNS servers. ns1.stardothosting.com, a public recursive NS server (168.144.1.130), and the NS server specified in the WHOIS of the domain. Ideally these will all match, if they don’t then connections to the incorrect mailserver may be made.

WINDOWS COMMAND PROMPT

nslookup
set type=mx
server  NS SERVER NAME OR IP
DOMAIN
server NS SERVER NAME OR IP
DOMAIN
server NS SERVER NAME OR IP
DOMAIN

Testing to see if remote server will accept a message

Similar to the test we use to see if our mail server will take the message as a local delivery you can do the same test on the remote server to see if it will take the message.

telnet mail.domainname.com 25
helo mailserver
mail from:  testingadmin@stardothosting.com
rcpt to: ADDRESS CLIENT IS TRYING TO EMAIL

Test Sucessful if: 250 ok

Send an entire message to a mailserver

telnet mail.domainname.com 25
helo mailserver
auth login
BASE64 Encoding of USERNAME
BASe64 Encoding of PASSWORD
mail from:      testing@stardothosting.com
rcpt to:        sales@stardothosting.com
data
To:     StarDot Sales sales@stardothosting.com
From:   testing@stardothosting.com
Subject: This is a test email
Dear SDH,
This is the body of your email.
.

Test Sucessful if: 250 ok

Check to see if a remote server is in an RBL

You will need to find out what the IP address of the machine that is actually sending the email out to the internet. There are many tools out there to check you rdomain / IP address againts many RBL lists :

* http://www.robtex.com/rbl/
* http://www.anti-abuse.org/multi-rbl-check/
* http://checker.msrbl.com/

RBL’s Checked Against:

* relays.ordb.org
* sbl-xbl.spamhaus.org
* bl.spamcop.net

SpamAssassin Troubleshooting

Via the webmail (Squirrel Mail) place the address that is trying to email the customer in the whitelist. When the email arrives check the email headers which will show any SpamAssassin checks that applied points to the message. Details on the SA tests and how to check email headers is available on the links below:

* Headers: http://www.spamcop.net/fom-serve/cache/19.html
* SA Tests: http://spamassassin.apache.org/tests_3_1_x.html

Quick tips using FIND , SSH, TAR , PS and GREP Commands

Administering hundreds of systems can be tedious. Sometimes scripting repetitive tasks, or replicating tasks across many servers is necessary.

Over time, I’ve jotted down several quick useful notes regarding using various linux/unix commands. I’ve found them very useful when navigating and performing various tasks. I decided to share them with you, so hopefully you will find them a useful reference at the very least!

To find files within a time range and add up the total size of all those files :

find /opt/uploads -mtime -365 -printf "%sn"|awk '{sum+=$0}END{print sum}'

To watch a command’s progress :

watch -n1 'du -h -c --max-depth=1'

Transfer a file / folders, compress it midstrem over the network, uncompress the file on the recieving end:

ssh -l root 00.00.000.000 '(cd /opt/uploads/ && tar -czf - . -C /opt/uploads)' | tar -xzf -

Below will return any XYZ PID that is older than 10 hours.

ps -ef | grep XYZ | awk '{ print $7 ":" $2 }' | awk 'BEGIN { FS =":" }; {if ($1 > 10) print $4}'

Check web logs on www server for specific ip address access:

grep "ip_address" [a-h]*/logs/*access*4		<-- check a-h websites
grep "ip_address" [A-Z,0-9]*/logs/*access*4	<-- check A-Z, 0-9 websites

Those are just a few of the useful commands that can be applied to many different functions. I particularly like sending files across the network and compressing them mid stream :)

The above kind of administration is made even easier when you employ ssh key based authentication -- your commands can be scripted to execute across many servers in one sitting (just be careful) ;)

Menu