Use AWS CLI to automate the removal and addition of instances in your ELB

Hello!

Sometimes its necessary to automate the removal and addition of instances in your elastic load balancer. Perhaps for the purposes of auto scaling or deploying updates to your web application. Either way, there is many tools at the disposal of the systems administrator to automate this process. Below we will share some simple steps as well as some (very) simple scripts to make it that much easier to manipulate the instances that are receiving live traffic via the ELB.

Install AWS CLI

This is pretty straightforward (and obvious). Amazon themselves provide a great guide to installing the AWS Cli toolset on your linux distribution. Below I’ll provide the shorthand for setting up AWS Cli on CentOS/RedHat or Ubuntu/Debian systems.

CentOS/RedHat

Ubuntu/Debian

So simple, right? There are other ways to install the toolset, such as through Python or to download the source and install directly from the source. After installing it, you will want to configure it with the access credentials in order to authenticate against your AWS account :

Before you do that, it might be a good idea to create a new IAM user with restricted access.

Create IAM user in AWS Security Console to access only your ELB

Restricting access for your IAM user is a good best practice. It will ensure that the access you delegate will never go beyond what was originally intended and will also mitigate any damage a malicious user might do should they be able to gain access to the credentials.

What you would want to do is create a group first, with the following two policies attached : AmazonEC2ReadOnlyAccess (a pre-made policy that you can search for and attach automatically), and a custom policy detailed below.

The policy above will allow the users in the IAM group to only access the specified ELB (where “your-elb-name” is specified). If you are in a different availability zone, you would also want to change us-east-1 to whatever zone your in.

Once the policy is attached to the group, then you simply need to create the user, add them to the group you created and create the access credential key/secret to use with the aws configure command.

The purposes of the script for us was to create a script on each actual instance so that we could simply run the script locally and automatically take the instance out of the pool. This means, for us, we ran the aws configure command on each instance that the script was running on. If you are using a centralized server (i.e. Jenkins, Ansible, Puppet, etc) then your script may look different. Perhaps your script in this case would parse the instances that are currently active in the ELB, and then iterate through each, taking them out of the ELB, running the update (or whatever you need to do) and then putting them back before moving on to the next.

Bash script to automate adding and removing servers to an ELB

This bash script is dead simple. We are simply grabbing the instance id of the script its running on and then read the user input to determine if the request is to add or remove the instance in question from the ELB. The script can definitely be improved further to perhaps include an error checker to make sure if you are removing the instance that there is at least 1 other active instance in the ELB before doing that (to avoid outages).

Pretty straightforward! Again, better error checking and perhaps parsing the output of the aws commands may add better checks and balances to this kind of manipulation of your ELBs. For that level of checking and parsing it might be worth exploring using Python instead of Bash.

Varnish Caching with Joomla

Hello There!

One of the exciting new technologies to come out in the last few years is a tremendously efficient and dynamic caching system called Varnish (see : http://www.varnish-cache.org).

We have been employing the use of Varnish for high traffic websites for the purposes of user experience improvements as well as for redundancy and load balancing purposes.

Varnish can do it all – complex load balancing and polling based on many different weighting methodologies for fail over, as well as holding on to a “stale” cache in the event of a back end web server outage, or perhaps for geographic redundancy (holding a stale cache in a secondary data center).

One of the challenges we have faced in the many different implementations of varnish into web stacks, is dealing with dynamic and user session (i.e. “logged in”) content.

If the Internet was full of only static (see 1995) html files, varnish would work beautifully out of the box. Unfortunately the web is a complicated mess of session based authentication, POSTS, GETS and query strings among a few things.

One of our recent accomplishments was getting the Joomla 1.5 content management system to work with Varnish 2.1.

The biggest challenge for Joomla was that it creates a session cookie for all users. This means the session is created and established for any guest visiting the site, and if they decide to log in , that same session is used to establish a logged in session through authentication. This is an apparent effort to deter or avoid session hijacking.

The problem with this is that Varnish ends up caching all the logged in session content, as well as the anonymous front page content.

I spent a significant amount of time fine tuning my VCL (varnish configuration language) to play nice with Joomla. Unfortunately it became apparent that some minor modifications to the Joomla code was necessary in order for it to communicate properly with Varnish.

Step 1 : Move the login form off the front page

I realize this might be a hard decision. I cant offer an alternative. If you have an integrated login form on the front page of your site, and you wish to cache that page with varnish, you will likely have to chose one or the other. It would probably be ideal to replace that login form with a button to bring the user to a secondary page off the main page.

For the sake of argument, lets call our site “example.com” and the login page url within Joomla should look like the following :

http://www.example.com/index.php?option=com_user&view=login

Take note of login URI in this string.

The reason we need the login form on a secondary page is because we need an almost “sandboxed” section of the site where the anonymous session cookie can be established, and passed through the authentication process to a logged in session. We will tell varnish to essentially ignore this page.

Step 2 : Modify Joomla to send HTTP headers for user/guest sessions

This isn’t that hard. In the Joomla code, there is a section where it defines the HTTP headers it sends to the browser for cache variables such as expire times and whatnot. I’m going to assume you have turned off the built-in Joomla caching system.

What you need to do is tell Joomla to send a special HTTP header that will give either a True or False value if the user is logged in or not. This is useful information. It will allow varnish to not cache any logged in content such as “Welcome back, USERNAME” after the user is passed back to the front page from logging in.

In my joomla installation, I modified the following file :

The parent folder being the public_html / root folder for your Joomla installation. In this file, please find the line that determines if the Joomla caching system is disabled :

After this line, you will see about 5 HTTP header declarations (expires, last-modified, cache-control, cache-control again and pragma). Above those declarations , add the following 6 lines of code :

If you read the above code, its fairly straight forward. I do a check to see if the user is a guest (aka anonymous) or not. If they are logged in I send an HTTP header called “X-Logged-In”, and assign a “True” value to it. If the user is not logged in, it sets it to “False”.

Pretty easy, right?

This will allow varnish to avoid caching a logged in user’s page.

Step 3 : Configure Varnish

This is the part that took the most time during this entire process. Mind you patching the Joomla code and whatnot took some time as well, this process took a lot of experimentation and long hours examining session cookies and host headers.

What I will do is break down the generalized configuration directives into two groups : VCL_RECV and VCL_FETCH.

VCL_RECV

In here, I set a bunch of IF statement directives to tell varnish what it should look up in the cache and what it should pipe to the backend and what it should pass. This could probably be optimized and improved upon, but it works for me :

VCL_FETCH

The fetch section is a little bit easier. I only have about 5 directives. The first one is the most important one you want to look at. It “unsets” the cookie from any page on the site, EXCEPT the login page. This allows varnish to properly establish the logged in session. The subsequent rules determine what to deliver and what to pass based on URI or HTTP header checks :

Thats it! I just saved you many sleepless nights (I hope!). Hopefully your headers will look something like this after you implement varnish in front of Joomla :

UPDATE : 12/08/2011

I realize I made a mistake and have corrected this post. In vcl_fetch, i had the following :

Well I realize I should be unsetting the response cookie, not the set cookie. For some reason, the above (erroneous) directive works only right after you login. If you start clicking around the site, your logged in session disappears. I suspect this is because either joomla or varnish is mistakenly unsetting a logged in session.

This is the correct entry (I have fixed it in my original post as well) :

After making the above change, I can login and browse the site and my session stays intact. Mind you, the Joomla site I am testing with is definitely not a vanilla Joomla installation.

I’d love to hear from anyone who has accomplished the above scenario either way!

Foundry Load Balancers HTTP sticky sessions

This post is intended to be a general guide for configuring “stickied” load balanced HTTP servers. Whether it’s F5 load balancers, foundry load balancers or open source based load balancers (keepalived/lvs), the concepts are the same and can be migrated across said platforms.

If you have a paid of foundry’s and are looking to configure stickied load balanced HTTP servers, hopefully this guide will provide some assistance.

    Logging into the load balancer

Telnet to the box and ‘enable’ to allow admin access. The first thing you want to do is show the current configuration to view the existing setup for other working boxes :

    Real servers : defining the multiple load balanced boxes

Show the existing configuration on the foundary :

Take a look at the configuration of two “real” servers, which are the two servers that are behind the load balancer that will have balanced sticky connections :

The above example is balancing TCP 8001 traffic, which is for TOMCAT. Here are entries for two servers doing simple HTTP traffic :

This example is similar to the tomcat example, except you have several options. “port default disable” disables all other ports. “port http keepalive” and “port http url “HEAD /”” define the http checks that take place to ensure apache is running on that box. If not , it will fail over to the second box and stop sending traffic to it.

    SSL Connections

SSL incoming connections are handled by the load balancer initially, then passed off to the actual server as regular http / port 80 traffic. The internal box configuration would be similar to the above configuration examples :

    Configuring the external IP to NAT to the internal virtual

Typically, you will have a firewall in front of the load balancer that actaully holds the external ip addresses. The traffic is filtered initially by the firewall, then NAT’d to the virtual ip (VIP) of the load balancer, which then handles balancing the traffic.

You will need to either establish a new external ip , or use an existing one (for instance, if you are moving from 1 web server to 2 web servers , and want to balance the traffic using the load balancer). You need to setup the external IP address, and NAT it to the internal VIP.

    Verifying the configuration works

Once everything is setup properly, and the external IP is being NAT’d to the load balancer, it is time to ensure the load balancer is seeing the connections. You could do this before doing the switchover on the firewall as well, just to ensure everything looks right before actually doing the switchover.

To see the active connections being load balanced, issue the following command (replacing the servername for whichever one you want to check) :

That should display information similar to this :

The above is displaying the specific connection details for a single real server. To check the VIP / Virtual server :

Which will display the following :

You can see that “ServerConn” is displaying 46 connections. Thats it!