Load Balancing Archives - Managed Wordpress Hosting | Managed VPS Hosting | Stack Star

Use AWS CLI to automate the removal and addition of instances in your ELB

admin — Thu, 15 Dec 2016 17:59:42 +0000

Hello!

Sometimes its necessary to automate the removal and addition of instances in your elastic load balancer. Perhaps for the purposes of auto scaling or deploying updates to your web application. Either way, there is many tools at the disposal of the systems administrator to automate this process. Below we will share some simple steps as well as some (very) simple scripts to make it that much easier to manipulate the instances that are receiving live traffic via the ELB.

Install AWS CLI

This is pretty straightforward (and obvious). Amazon themselves provide a great guide to installing the AWS Cli toolset on your linux distribution. Below I’ll provide the shorthand for setting up AWS Cli on CentOS/RedHat or Ubuntu/Debian systems.

CentOS/RedHat

yum install aws-cli

Ubuntu/Debian

apt-get install awscli

So simple, right? There are other ways to install the toolset, such as through Python or to download the source and install directly from the source. After installing it, you will want to configure it with the access credentials in order to authenticate against your AWS account :

aws configure

Before you do that, it might be a good idea to create a new IAM user with restricted access.

Create IAM user in AWS Security Console to access only your ELB

Restricting access for your IAM user is a good best practice. It will ensure that the access you delegate will never go beyond what was originally intended and will also mitigate any damage a malicious user might do should they be able to gain access to the credentials.

What you would want to do is create a group first, with the following two policies attached : AmazonEC2ReadOnlyAccess (a pre-made policy that you can search for and attach automatically), and a custom policy detailed below.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "Stmt1479414049000",
            "Effect": "Allow",
            "Action": [
                "elasticloadbalancing:DeregisterInstancesFromLoadBalancer",
                "elasticloadbalancing:DescribeInstanceHealth",
                "elasticloadbalancing:RegisterInstancesWithLoadBalancer"
            ],
            "Resource": [
                "arn:aws:elasticloadbalancing:us-east-1:113224244507:loadbalancer/your-elb-name"
            ]
        }
    ]
}

The policy above will allow the users in the IAM group to only access the specified ELB (where “your-elb-name” is specified). If you are in a different availability zone, you would also want to change us-east-1 to whatever zone your in.

Once the policy is attached to the group, then you simply need to create the user, add them to the group you created and create the access credential key/secret to use with the aws configure command.

The purposes of the script for us was to create a script on each actual instance so that we could simply run the script locally and automatically take the instance out of the pool. This means, for us, we ran the aws configure command on each instance that the script was running on. If you are using a centralized server (i.e. Jenkins, Ansible, Puppet, etc) then your script may look different. Perhaps your script in this case would parse the instances that are currently active in the ELB, and then iterate through each, taking them out of the ELB, running the update (or whatever you need to do) and then putting them back before moving on to the next.

Bash script to automate adding and removing servers to an ELB

This bash script is dead simple. We are simply grabbing the instance id of the script its running on and then read the user input to determine if the request is to add or remove the instance in question from the ELB. The script can definitely be improved further to perhaps include an error checker to make sure if you are removing the instance that there is at least 1 other active instance in the ELB before doing that (to avoid outages).

#!/bin/sh

#check command input
if [ -z "$1" ];
then
        echo "STACKSTAR ELB MANIPULATION"
        echo "--------------------------"
        echo ""
        echo "Usage : ./elbstack add"
        echo "Usage : ./elbstack remove"
        echo ""
        exit 0
fi

current_instance=`/usr/bin/wget -q -O - http://instance-data/latest/meta-data/instance-id`

if [ "$1" -eq "add" ];
then
        echo "Removing $current_instance..."
        /usr/bin/aws elb register-instances-with-load-balancer --load-balancer-name your-elb-name --instances $current_instance
elif [ "$1" -eq "remove" ];
then
        echo "Removing $current_instance..."
        /usr/bin/aws elb deregister-instances-from-load-balancer --load-balancer-name your-elb-name --instances $current_instance
else
        echo "Invalid command argument given."
fi

Pretty straightforward! Again, better error checking and perhaps parsing the output of the aws commands may add better checks and balances to this kind of manipulation of your ELBs. For that level of checking and parsing it might be worth exploring using Python instead of Bash.

The post Use AWS CLI to automate the removal and addition of instances in your ELB appeared first on Managed Wordpress Hosting | Managed VPS Hosting | Stack Star.

Varnish Caching with Joomla

admin — Mon, 08 Aug 2011 21:05:50 +0000

Hello There!

One of the exciting new technologies to come out in the last few years is a tremendously efficient and dynamic caching system called Varnish (see : http://www.varnish-cache.org).

We have been employing the use of Varnish for high traffic websites for the purposes of user experience improvements as well as for redundancy and load balancing purposes.

Varnish can do it all – complex load balancing and polling based on many different weighting methodologies for fail over, as well as holding on to a “stale” cache in the event of a back end web server outage, or perhaps for geographic redundancy (holding a stale cache in a secondary data center).

One of the challenges we have faced in the many different implementations of varnish into web stacks, is dealing with dynamic and user session (i.e. “logged in”) content.

If the Internet was full of only static (see 1995) html files, varnish would work beautifully out of the box. Unfortunately the web is a complicated mess of session based authentication, POSTS, GETS and query strings among a few things.

One of our recent accomplishments was getting the Joomla 1.5 content management system to work with Varnish 2.1.

The biggest challenge for Joomla was that it creates a session cookie for all users. This means the session is created and established for any guest visiting the site, and if they decide to log in , that same session is used to establish a logged in session through authentication. This is an apparent effort to deter or avoid session hijacking.

The problem with this is that Varnish ends up caching all the logged in session content, as well as the anonymous front page content.

I spent a significant amount of time fine tuning my VCL (varnish configuration language) to play nice with Joomla. Unfortunately it became apparent that some minor modifications to the Joomla code was necessary in order for it to communicate properly with Varnish.

Step 1 : Move the login form off the front page

I realize this might be a hard decision. I cant offer an alternative. If you have an integrated login form on the front page of your site, and you wish to cache that page with varnish, you will likely have to chose one or the other. It would probably be ideal to replace that login form with a button to bring the user to a secondary page off the main page.

For the sake of argument, lets call our site “example.com” and the login page url within Joomla should look like the following :

http://www.example.com/index.php?option=com_user&view=login

Take note of login URI in this string.

The reason we need the login form on a secondary page is because we need an almost “sandboxed” section of the site where the anonymous session cookie can be established, and passed through the authentication process to a logged in session. We will tell varnish to essentially ignore this page.

Step 2 : Modify Joomla to send HTTP headers for user/guest sessions

This isn’t that hard. In the Joomla code, there is a section where it defines the HTTP headers it sends to the browser for cache variables such as expire times and whatnot. I’m going to assume you have turned off the built-in Joomla caching system.

What you need to do is tell Joomla to send a special HTTP header that will give either a True or False value if the user is logged in or not. This is useful information. It will allow varnish to not cache any logged in content such as “Welcome back, USERNAME” after the user is passed back to the front page from logging in.

In my joomla installation, I modified the following file :

libraries/joomla/environment/response.php

The parent folder being the public_html / root folder for your Joomla installation. In this file, please find the line that determines if the Joomla caching system is disabled :

if (JResponse::allowCache() === false)

After this line, you will see about 5 HTTP header declarations (expires, last-modified, cache-control, cache-control again and pragma). Above those declarations , add the following 6 lines of code :

$user =& JFactory::getUser();
if (!$user->guest) {
JResponse::setHeader( 'X-Logged-In', 'True', true);
} else {
JResponse::setHeader( 'X-Logged-In', 'False', true );
}

If you read the above code, its fairly straight forward. I do a check to see if the user is a guest (aka anonymous) or not. If they are logged in I send an HTTP header called “X-Logged-In”, and assign a “True” value to it. If the user is not logged in, it sets it to “False”.

Pretty easy, right?

This will allow varnish to avoid caching a logged in user’s page.

Step 3 : Configure Varnish

This is the part that took the most time during this entire process. Mind you patching the Joomla code and whatnot took some time as well, this process took a lot of experimentation and long hours examining session cookies and host headers.

What I will do is break down the generalized configuration directives into two groups : VCL_RECV and VCL_FETCH.

VCL_RECV

In here, I set a bunch of IF statement directives to tell varnish what it should look up in the cache and what it should pipe to the backend and what it should pass. This could probably be optimized and improved upon, but it works for me :

# If user sends an http POST, pipe to backend
if (req.request == "POST") {
set req.backend = iamloggedin;
return(pipe);
}

# http authenticated sessions are piped
if (req.http.Authenticate || req.http.Authorization) {
set req.backend = iamloggedin;
return(pipe);
}

# if the user is coming FROM the login page, pipe to backend
if (req.http.referer ~ "(?i)(com_user|login)") {
set req.backend = iamloggedin;
return(pipe);
}

VCL_FETCH

The fetch section is a little bit easier. I only have about 5 directives. The first one is the most important one you want to look at. It “unsets” the cookie from any page on the site, EXCEPT the login page. This allows varnish to properly establish the logged in session. The subsequent rules determine what to deliver and what to pass based on URI or HTTP header checks :

# discard backend setcookie unless it equals the following
if (!req.url ~ "(?i)(login|com_user|user|logout)") {
unset beresp.http.Set-Cookie;
}

if (req.http.referer ~ "(?i)(com_user|login|logout)") {
set req.backend = iamloggedin;
return(pass);
}

if (beresp.http.x-logged-in ~ "False"){
set req.backend = webfarm;
return(deliver);
}

if (beresp.http.x-logged-in ~ "True"){
set req.backend = iamloggedin;
return(pass);
}

if (req.http.Authenticate || req.http.Authorization) {
set req.backend = iamloggedin;
return(pass);
}

Thats it! I just saved you many sleepless nights (I hope!). Hopefully your headers will look something like this after you implement varnish in front of Joomla :

Set-Cookie example_auth_129bf15asdfasdf52f3afaafawef; path=/
P3P CP="NOI ADM DEV PSAi COM NAV OUR OTRo STP IND DEM"
X-Logged-In False
Expires Mon, 1 Jan 2001 00:00:00 GMT
Last-Modified Mon, 08 Aug 2011 20:49:37 GMT
Cache-Control post-check=0, pre-check=0
Pragma no-cache
Content-Type text/html; charset=utf-8
Content-Length 85898
Date Mon, 08 Aug 2011 21:01:52 GMT
X-Varnish 761778669 761751685
Age 735
Via 1.1 varnish
Connection keep-alive
X-Cache-Svr cache.example.com
X-Cache HIT
X-Cache-Hits 121

UPDATE : 12/08/2011

I realize I made a mistake and have corrected this post. In vcl_fetch, i had the following :

# discard backend setcookie unless it equals the following
if (!req.url ~ "(?i)(login|com_user|user|logout)") {
unset req.http.Set-Cookie;
}

Well I realize I should be unsetting the response cookie, not the set cookie. For some reason, the above (erroneous) directive works only right after you login. If you start clicking around the site, your logged in session disappears. I suspect this is because either joomla or varnish is mistakenly unsetting a logged in session.

This is the correct entry (I have fixed it in my original post as well) :

# discard backend setcookie unless it equals the following
if (!req.url ~ "(?i)(login|com_user|user|logout)") {
unset beresp.http.Set-Cookie;
}

After making the above change, I can login and browse the site and my session stays intact. Mind you, the Joomla site I am testing with is definitely not a vanilla Joomla installation.

I’d love to hear from anyone who has accomplished the above scenario either way!

The post Varnish Caching with Joomla appeared first on Managed Wordpress Hosting | Managed VPS Hosting | Stack Star.

Foundry Load Balancers HTTP sticky sessions

admin — Mon, 30 Aug 2010 19:15:06 +0000

This post is intended to be a general guide for configuring “stickied” load balanced HTTP servers. Whether it’s F5 load balancers, foundry load balancers or open source based load balancers (keepalived/lvs), the concepts are the same and can be migrated across said platforms.

If you have a paid of foundry’s and are looking to configure stickied load balanced HTTP servers, hopefully this guide will provide some assistance.

Logging into the load balancer

Telnet to the box and ‘enable’ to allow admin access. The first thing you want to do is show the current configuration to view the existing setup for other working boxes :

> telnet 192.x.x.x
Trying 192.x.x.x...
Connected to 10.x.x.x.
Escape character is '^]'.

User Access Verification

Please Enter Login Name: admin
Please Enter Password: 

User login successful.

SLB-telnet@XXXX>enable
Enable Password:
Error - Incorrect username or password.
SLB-telnet@XXXX>enable
Enable Password:
SLB-telnet@XXXX#

Real servers : defining the multiple load balanced boxes

Show the existing configuration on the foundary :

sh config

Take a look at the configuration of two “real” servers, which are the two servers that are behind the load balancer that will have balanced sticky connections :

server real serverposapp01-tomcat01 192.168.1.141
 port default disable
 port 8001
!                                                                 
!
server real serverposapp03-tomcat01 192.168.1.143
 port default disable
 port 8001

The above example is balancing TCP 8001 traffic, which is for TOMCAT. Here are entries for two servers doing simple HTTP traffic :

server real serverapp01-vhost01 192.168.1.195
 port default disable                                             
 port http
 port http keepalive
 port http url "HEAD /"
!
server real serverapp02-vhost01 192.168.1.196
 port default disable
 port http
 port http keepalive
 port http url "HEAD /"

This example is similar to the tomcat example, except you have several options. “port default disable” disables all other ports. “port http keepalive” and “port http url “HEAD /”” define the http checks that take place to ensure apache is running on that box. If not , it will fail over to the second box and stop sending traffic to it.

SSL Connections

SSL incoming connections are handled by the load balancer initially, then passed off to the actual server as regular http / port 80 traffic. The internal box configuration would be similar to the above configuration examples :

server virtual portal 192.168.1.104 port default disable port ssl sticky port ssl ssl-terminate portal bind ssl serverapp01-portal01 http Notice how instead of "port http sticky" , its "port ssl sticky". First of all, the sticky option is only set on the "virtual" configuration directives. Secondly, the SSL traffic is bound to the real servers via http in the last line of this example. Its pretty self explanatory. [edit] Regular HTTP Sticky Connections If no SSL Is being used on the site at all, then all you need is to set an HTTP virtual configuration :

server virtual serverapp-vhost01 192.168.1.106
 port default disable
 port http sticky                                                 
 bind http serverapp02-vhost01 http

Configuring the external IP to NAT to the internal virtual

Typically, you will have a firewall in front of the load balancer that actaully holds the external ip addresses. The traffic is filtered initially by the firewall, then NAT'd to the virtual ip (VIP) of the load balancer, which then handles balancing the traffic.

You will need to either establish a new external ip , or use an existing one (for instance, if you are moving from 1 web server to 2 web servers , and want to balance the traffic using the load balancer). You need to setup the external IP address, and NAT it to the internal VIP.

Verifying the configuration works

Once everything is setup properly, and the external IP is being NAT'd to the load balancer, it is time to ensure the load balancer is seeing the connections. You could do this before doing the switchover on the firewall as well, just to ensure everything looks right before actually doing the switchover.

To see the active connections being load balanced, issue the following command (replacing the servername for whichever one you want to check) :

sh server real serverapp02-vhost01

That should display information similar to this :

Real Servers Info
========================
State(St) - ACT:active, ENB:enabled, FAL:failed, TST:test, DIS:disabled,
            UNK:unknown, UNB:unbind, AWU:await-unbind, AWD:await-delete

Name: serverapp02-vhost01     State: Active       Cost: 0  IP:192.168.1.196:   1
Mac: 0012.7990.d06a          Weight: 0                  MaxConn: 2000000
SrcNAT: not-cfg, not-op      DstNAT: not-cfg, not-op    Serv-Rsts: 0
tcp conn rate:udp conn rate = 1:0, max tcp conn rate:max udp conn rate = 8:0
BP max local conn configured No: 0 0 0 0 0 0 
BP max conn percentage configured No: 0 0 0 0 0 0 
Use local conn : No

Port    St  Ms ServerConn TotConn    Rx-pkts   Tx-pkts   Rx-octet   Tx-octet   Reas
----    --  -- ------- -------    -------   -------   --------   --------   ----
default DIS 0  0       0          0         0         0          0          0  
http    ACT 0  104     13094      181671    150813    162364862  20325115   0  

Server  Total  104     13094      181671    150813    162364862  20325115   0

The above is displaying the specific connection details for a single real server. To check the VIP / Virtual server :

sh server virtual tomcat

Which will display the following :

Virtual Servers Info

Name: tomcat                 State: Enabled             IP:192.168.1.101:   1
Pred: least-conn             ACL-Id: 0                  TotalConn: 149959

Port    State     Sticky  Concur  Proxy  DSR   ServerConn  TotConn  PeakConn  
----    -----     ------  ------  -----  ---   -------  -------  --------  

default disabled  NO      NO      NO     NO    0        0        0         
ssl     enabled   YES     NO      NO     NO    46       149959   443

You can see that "ServerConn" is displaying 46 connections. Thats it!

The post Foundry Load Balancers HTTP sticky sessions appeared first on Managed Wordpress Hosting | Managed VPS Hosting | Stack Star.