Foundry Load Balancers HTTP sticky sessions

This post is intended to be a general guide for configuring “stickied” load balanced HTTP servers. Whether it’s F5 load balancers, foundry load balancers or open source based load balancers (keepalived/lvs), the concepts are the same and can be migrated across said platforms.

If you have a paid of foundry’s and are looking to configure stickied load balanced HTTP servers, hopefully this guide will provide some assistance.

    Logging into the load balancer

Telnet to the box and ‘enable’ to allow admin access. The first thing you want to do is show the current configuration to view the existing setup for other working boxes :

> telnet 192.x.x.x
Trying 192.x.x.x...
Connected to 10.x.x.x.
Escape character is '^]'.

User Access Verification

Please Enter Login Name: admin
Please Enter Password: 

User login successful.

Enable Password:
Error - Incorrect username or password.
Enable Password:
    Real servers : defining the multiple load balanced boxes

Show the existing configuration on the foundary :

sh config

Take a look at the configuration of two “real” servers, which are the two servers that are behind the load balancer that will have balanced sticky connections :

server real serverposapp01-tomcat01
 port default disable
 port 8001
server real serverposapp03-tomcat01
 port default disable
 port 8001

The above example is balancing TCP 8001 traffic, which is for TOMCAT. Here are entries for two servers doing simple HTTP traffic :

server real serverapp01-vhost01
 port default disable                                             
 port http
 port http keepalive
 port http url "HEAD /"
server real serverapp02-vhost01
 port default disable
 port http
 port http keepalive
 port http url "HEAD /"

This example is similar to the tomcat example, except you have several options. “port default disable” disables all other ports. “port http keepalive” and “port http url “HEAD /”” define the http checks that take place to ensure apache is running on that box. If not , it will fail over to the second box and stop sending traffic to it.

    SSL Connections

SSL incoming connections are handled by the load balancer initially, then passed off to the actual server as regular http / port 80 traffic. The internal box configuration would be similar to the above configuration examples :

server virtual portal
 port default disable
 port ssl sticky
 port ssl ssl-terminate portal
 bind ssl serverapp01-portal01 http

Notice how instead of "port http sticky" , its "port ssl sticky". First of all, the sticky option is only set on the "virtual" configuration directives. Secondly, the SSL traffic is bound to the real servers via http in the last line of this example. Its pretty self explanatory.
[edit] Regular HTTP Sticky Connections

If no SSL Is being used on the site at all, then all you need is to set an HTTP virtual configuration :

server virtual serverapp-vhost01
 port default disable
 port http sticky                                                 
 bind http serverapp02-vhost01 http
    Configuring the external IP to NAT to the internal virtual

Typically, you will have a firewall in front of the load balancer that actaully holds the external ip addresses. The traffic is filtered initially by the firewall, then NAT'd to the virtual ip (VIP) of the load balancer, which then handles balancing the traffic.

You will need to either establish a new external ip , or use an existing one (for instance, if you are moving from 1 web server to 2 web servers , and want to balance the traffic using the load balancer). You need to setup the external IP address, and NAT it to the internal VIP.

    Verifying the configuration works

Once everything is setup properly, and the external IP is being NAT'd to the load balancer, it is time to ensure the load balancer is seeing the connections. You could do this before doing the switchover on the firewall as well, just to ensure everything looks right before actually doing the switchover.

To see the active connections being load balanced, issue the following command (replacing the servername for whichever one you want to check) :

sh server real serverapp02-vhost01

That should display information similar to this :

Real Servers Info
State(St) - ACT:active, ENB:enabled, FAL:failed, TST:test, DIS:disabled,
            UNK:unknown, UNB:unbind, AWU:await-unbind, AWD:await-delete

Name: serverapp02-vhost01     State: Active       Cost: 0  IP:   1
Mac: 0012.7990.d06a          Weight: 0                  MaxConn: 2000000
SrcNAT: not-cfg, not-op      DstNAT: not-cfg, not-op    Serv-Rsts: 0
tcp conn rate:udp conn rate = 1:0, max tcp conn rate:max udp conn rate = 8:0
BP max local conn configured No: 0 0 0 0 0 0 
BP max conn percentage configured No: 0 0 0 0 0 0 
Use local conn : No

Port    St  Ms ServerConn TotConn    Rx-pkts   Tx-pkts   Rx-octet   Tx-octet   Reas
----    --  -- ------- -------    -------   -------   --------   --------   ----
default DIS 0  0       0          0         0         0          0          0  
http    ACT 0  104     13094      181671    150813    162364862  20325115   0  

Server  Total  104     13094      181671    150813    162364862  20325115   0   

The above is displaying the specific connection details for a single real server. To check the VIP / Virtual server :

sh server virtual tomcat

Which will display the following :

Virtual Servers Info

Name: tomcat                 State: Enabled             IP:   1
Pred: least-conn             ACL-Id: 0                  TotalConn: 149959

Port    State     Sticky  Concur  Proxy  DSR   ServerConn  TotConn  PeakConn  
----    -----     ------  ------  -----  ---   -------  -------  --------  

default disabled  NO      NO      NO     NO    0        0        0         
ssl     enabled   YES     NO      NO     NO    46       149959   443       

You can see that "ServerConn" is displaying 46 connections. Thats it!

Creating a Xen template

One way to increase the efficiencies of Xen based systems is to utilize templates. VMware talks about this in their whitepaper for ESX2 best practices.

With Xen, you have to create your own. Here is a straight forward guide for how to do it.

1. Bootstrap a DomU named -tpl (e.g. centos4-tpl).

I recommend using a file-backed VBD, but partition or LVM volume will work fine as well. Here is an example /etc/xen/centos4-tpl

kernel = "/boot/vmlinuz-"
memory = 256
name = "centos4-tpl" 
disk = [  'file:/opt/xen/domains/centos4-tpl/diskimage,sdb1,w','file:/opt/xen/domains/centos4-tpl/swapimage,sdb2,w'  ]
root = "/dev/sdb1 ro"

This is just a normal system (DomU) install – see Centos-4 on Xen for an example. Un-customize files

2.Inside the VM, edit the following files

remove any address lines other than localhost

use a generic hostname which will be unique to each deployment


should look like this:


also important – remove any line starting with HWADDR, e.g.:


Other configuration files to consider tweaking include /etc/dhclient.conf & /etc/hosts

3. Files to remove:

– SSH Host key files (auto-created at boot time)

rm -f /etc/ssh/*host*

4. Shutdown the template VM

xm shutdown centos4-tpl

You might normally link your VMs into /etc/xen/auto. I recommend against this as the template VM can be left shutdown until/unless you want to update it, saving valuable RAM and CPU cycles.

Clone the virtual disk Now we can deploy from the template by cloning the data into a clean diskimage (or partition or LVM volume). Create the diskimage using an appropriate size (must be larger than the template). Oh -the nice thing here is that there is flexibility. For instance, you can have a file-based diskimage and clone the data onto LVM volumes. As long as you can mount the (virtual) disks, you can clone templatized systems.

Here we use /mnt/disk to mount the new system disk, and /mnt/image to mount the template disk.

First, mount the template disk.

mount -o loop /opt/xen/domains/centos4-tpl/diskimage /mnt/image

Next, create and mount the new system (DomU) disk space & swap space.

mkdir -p /opt/xen/domains/cloned
cd /opt/xen/domains/cloned
dd if=/dev/zero of=diskimage bs=1024k count=2048
dd if=/dev/zero of=swapimage bs=1024k count=256
mkfs.ext3 diskimage
mkswap swapimage
mkdir -p /mnt/disk
mount -o loop /opt/xen/domains/cloned/diskimage /mnt/disk

Create the exclude file in /tmp/XenCloneExclude


Copy the data across

rsync -av -SHWD --exclude-from="/tmp/XenCloneExclude" /mnt/image/ /mnt/disk

Chroot into the newly copied template and fixup certain files

chroot /mnt/disk /bin/bash

Fix the hostname, etc in the files we “un-customized” in the template.

Exit, unmount both the template image and volume

umount /mnt/disk
umount /mnt/image

Setup your Xen config and be on your way!

cd /etc/xen
cp centos4-tpl cloned
(edit cloned to change name and paths to disk and swap)
xm create -c cloned