Use Varnish and Nginx to follow, hide and cache 301 / 302 redirects

Hello!

Varnish is a robust, open source and stable caching solution that has been employed on many different high traffic environments with a significant amount of success.

One of the things that we have come across, specifically with environments such as Amazon Web Services is that websites tend to spread their web stack across multiple services. For example static media such as JS, CSS and image files may be hosted on Amazon S3 storage. This would require either implementing additional CNAMES for your domain (i.e. static.yourdomain.com) that point to the S3 URL, or have your CMS redirect requests for static media to the S3 backend.

Remember with S3, you have to generate the static files and copy them over to S3, so these URLs may need to be generated and maintained by the CMS often times with a redirect (301 or 302) that rewrites the URL to the S3 backend destination.

When Varnish is caching a website, and it comes across a request that is rewritten in a 301/302 redirect by the backend response (“beresp”), varnish typically will simply cache the 301/302 redirect as a response, saving that minuscule amount of processing power that needed to happen to process the request and send the rewrite. Some may argue that that is simply negligible!

Wouldn’t it be nice to actually cache the content after the redirect happens? There’s two ways one could go about doing this.

Cache 301 / 302 Redirects with Varnish 4

In simpler setups, Varnish can simply process the rewrite/redirect and change the url and issue a return(restart) to restart the request with the new URL. This means that Varnish is processing the rewrite and returning the new url so that it can be re-requested and ultimately cached. So in vcl_deliver you can add the following :

The above should work for you if, lets say, you are using Varnish in front of a simple apache/nginx server that is then processing the request. If Varnish is sending traffic to another proxy (i.e. nginx + proxy_pass), then this above directive may not work for you. The reason why one may want to proxy traffic from varnish to another proxy like nginx may be in a scenario where you want to do some fancy redirection of traffic + DNS resolution. Still confused?

Lets say varnish is at the edge of a network, caching a complicated website. Requests to varnish need to go to another load balancer (i.e. an Amazon ELB). ELB endpoints are never going to be a static IP address and Varnish (as of v4) cannot do DNS resolution of hostnames on a per request basis, so you would need to proxy the request to Nginx which would handle the reverse proxy over to ELB which would then load balance the backend fetch to the CMS.

If your scenario sounds more like the aforementioned one, then you could try following the 301/302 redirect with nginx instead of varnish.

Cache 301 / 302 Redirects with Nginx

Nginx and Varnish seem to go hand in hand. They’re great together! In this scenario you are using Varnish as your edge cache and sending all backend requests to an nginx proxy_pass directive. In order to tell Nginx to follow a redirect before sending any response to Varnish (and ultimately the end-user), you can tell varnish to simply save the redirect location and return the response after redirecting back to Varnish so it can simply cache the response!

You can see that the proxy_pass directive is configured normally. In the event of any 301, 302 or 307, process it with the @handle_redirects location directive. Then simply proxy pass the $saved_redirect_location as if it were the backend server! This means that even if the proxy_pass location is not even in your Varnish configuration as a valid hostname (i.e. random S3 url) Varnish will still cache it, thinking it is backend-server.com.

Hopefully this will help someone out there!

Force SSL for your site with Varnish and Nginx

Hello!

For those of you who depend on Varnish to offer robust caching and scaling potential to your web stack, hearing about Google’s prioritization (albeit arguably small, for now) of sites that force SSL may cause pause in how to implement.

Varnish currently doesn’t have the ability to handle SSL certificates and encrypt requests as such. It may never actually have this ability because its focus is to cache content and it does a very good job I might add.

So if Varnish can’t handle the SSL traffic directly, how would you go about implementing this with Nginx?

Well, nginx has the ability to proxy traffic. This is one of the many reasons why some admins choose to pair Varnish with Nginx. Nginx can do reverse proxying and header manipulation out of the box without custom modules or configuration. Combine that with the lightweight production tested scalability of Nginx over Apache and the reasons are simple. We’re not interested in that debate here, just a simple implementation.

Nginx Configuration

With Nginx, you will need to add an SSL listener to handle the ssl traffic. You then assign your certificate. The actual traffic is then proxied to the (already set up) non-https listener (varnish).

The one thing to note before going further is the second last line of the configuration. That is important because it allows you to avoid an infinite redirect loop of a request proxying to varnish, varnish redirecting non-ssl to ssl and back to nginx for a proxy. You’ll notice that pretty quickly because your site will ultimately go down 🙁

What nginx is doing is defining a custom HTTP header and assigning a value of “https” to it :

So the rest of the nginx configuration can remain the same (the configuration that varnish ultimately forwards requests in order to cache).

Varnish

What you’re going to need in your varnish configuration is a minor adjustment :

What the above snippet is doing is simply checking if the header “X-Forwarded-Proto” (that nginx just set) exists and if the value equals (case insensitive) to “https”. If that is not present or matches , it sets a redirect to force the SSL connection which is handled by the nginx ssl proxy configuration above. Its also important to note that we are not just doing a clean break redirect, we are still appending the originating request URI in order to make it a smooth transition and potentially not break any previously ranked links/urls.

The last thing to note is the error 750 handler that handles the redirect in varnish :

You can see that were using a 302 temporary redirect instead of a permanent 301 redirect. This is your decision though browsers tend to be stubborn in their own internal caching of 301 redirects so 302 is good for testing.

After restarting varnish and nginx you should be able to quickly confirm that no non-SSL traffic is allowed anymore. You can not only enjoy the (marginal) SEO “bump” but you are also contributing to the HTTPS Everywhere movement which is an important cause!

Auto updating Atomicorp Mod Security Rules

Hello!

If any of you use mod_security as a web application firewall, you might have enlisted the services of Atomicorp for regularly updating your mod_security ruleset with signatures to protect against constantly changing threats to web applications in general.

One of the initial challenges, in a managed hosting environment, was to implement a system that utilizes the Atomicorp mod_security rules and update them regularly on an automated schedule.

When you subscribe to their service, they provide access credentials in order to pull the rules. You then need to integrate the rule files into your mod_security implementation and gracefully restart apache or nginx to ensure all the updated rules are loaded.

We developed a very simple python script, intended to run as a cron scheduled task, in order to accomplish this. We thought we would share it here in case anyone else may find it useful at all to accomplish the same thing. This script could easily be modified to download rules from any similar service, alternatively. This script was written for nginx, but can be changed to be integrated with apache.

Find the code below. Enjoy!

Web based system to purge multiple Varnish cache servers

Hello!

We have been working with varnish for quite a while. And there is quite a lot of documentation out there already for the different methods for purging cache remotely via Curl, the varnish admin tool sets and other related methods.

We deal with varnish in the Amazon Cloud as well as on dedicated servers. In many cases varnish sits in a pool of servers in the web stack before the web services such as Nginx and Apache. Sometimes purging specific cache urls can be cumbersome when you’re dealing with multiple cache servers.

Depending on the CMS you are using, there is some modules / plugins that are available that offer the ability to purge Varnish caches straight from the CMS, such as the Drupal Purge module.

We have decided to put out a secure, web accessible method for purging Varnish cached objects across multiple varnish servers. As always, take the word “secure” with a grain of salt. The recommended way to publish a web accessible method on apache or nginx that gives the end-user the ability to request cache pages be purged would be to take these fundamentals into consideration :

– Make the web accessible page available only to specific source IPs or subnets
– Make the web accessible page password protected with strong passwords and non-standard usernames
– Make the web accessible page fully available via SSL encryption

On the varnish configuration side of things, with security still in mind, you would have to set up the following items in your config :

ACL

Set up an access control list in varnish that only allows specific source IPs to send the PURGE request. Here is an example of one :

vcl_recv / vcl_hit / vcl_miss / vcl_pass

This is self explanatory (I hope). Obviously you would be integrating the following logic into your existing varnish configuration.

The code itself is available on our GitHub Project page. Feel free to contribute and add any additional functionality.

It should be important to note that what differentiates our solution among the existing ones out there is that our script will manipulate the host headers of the Curl request in order to submit the same hostname / url request across the array of varnish servers. That way the identical request can be received by multiple varnish servers with no local host file editing or anything like that.

There is lots of room for input sanity checks, better input logic and other options to perhaps integrate with varnish more intuitively. Remember this is a starting point, but hopefully it is useful for you!

Web based system to push your GIT code

Hello!

Since posting recently about our Web based SVN push system , we have decided to take what we did there one step further and implement a very similar system for GIT, but with more options!

The web based GIT push system is, as mentioned, very similar to the web based SVN push system, with the exception that you can select branches before exporting the code.

I should stress before continuing that this system is not intended to be publicly visible on a website. Strict access controls need to be implemented in front of this implementation to protect the integrity and protect from malicious users. For example, only making this system available on a Development LAN, or putting it behind an IP restricted firewall, with IP restricted apache/nginx rules, web authentication and SSL will allow for a much more secure implementation of this system. My advice is to always assume everything is vulnerable at any time. Working backwards with that assumption has always been a good policy for me.

First of all the entire solution is available on GitHub for you to preview.

I’ll go through each file individually, briefly explaining what each file does.

index.php
This is a straightforward file. There is a small amount of php code embedded in this file with HTML to present the push page in a simple HTML table. An array is built for all the sites you want to push (in this example case its a Dev and Prod site). The array makes it very easy to add additional sites. Each array propagates a source, destination, site name and site url within.

The only field that is really used is the “pushname” variable in each site array. That variable gets passed to the shell script that actually takes care of the pushing mechanism.

The remaining php code in this file builds a list of sites based on the array, as well as pulling the current branch by running a function included in functions.inc.php that pulls all the branches associated with a repository and saves it to a text file for easy parsing. The other function pulls the last time the site was pushed or “exported”, giving an easy reference when dealing with multiple developers.

It should be noted that it is best to implement apache/nginx web based access on a per-user basis in order to access this page. This is because the index.php file parses the username of who is accessing the site for logging purposes. So every user that needs to access this needs an htpasswd user/password created for them for security and accountability purposes.

functions.inc.php
This file is where many of the functions lie (obviously). There is a crossite scripting function that is used to filter any submit input. I realize this is not very secure, but with the security considerations I mentioned in the beginning of this post, it should suffice. A good systems administrator would implement many hardware, software and intrusion layers to prevent malicious users from injecting content such as snort and mod_security. Nothing beats the security of a completely offline web accessible page on an internal LAN, obviously.

Next we have some functions that grab the branches, get the current branch that the site has been previously pushed on, some log file functions for storing the log file info and writing the log data and displaying it as well. All of these functions are intended to help keep the development process very organized and easy to maintain.

gitupdate_process.php
This file is where the index.php file POSTS the data of the site you want to push. This file receives the data as a $_POST (with the XSS cleaner function mentioned earlier sanitizing as best as it can) and then passes that variable to the push bash shell script in order to do the actual file synchronization.

It might be possible to do all the file synchronization in php, but I felt that separating the actual git pulling and rsync process into a separate shell script made the process less obfuscated and confusing. The shell script rarely needs to change unless a new site is added obviously.

log.php
This file is simply loaded as an iframe within index.php when someone clicks to view the export log. It parses the log.txt file and displays it. The export log format can be customized obviously, but usually would contain the site name, username who pushed, date and time as well as the branch pushed.

log.txt
This is self explanatory and contains the log information detailed in log.php

push.sh
This is the push bash shell script that gitupdate_process.php calls. Again this can be consolidated to be 100% PHP but I felt segmenting it was a good idea. You can see that the command line arguments are parsed from a $_POST in gitupdate_process.php and then passed to the shell script as an argument. This is very simple and shouldn’t be too hard to understand. The arguments would basically be the site name ($1) and the git branch name that was selected from the dropdown box before hitting the export button ($2).

That’s it! This package for GIT has made many developers’ life easier and caused less headaches when diagnosing problems or even rolling back to a stable branch. Keeping a stable and organized development environment is key here, with the security considerations I mentioned earlier being paramount above everything else.

I hope that this script was helpful and would welcome any suggestions to improve it further 🙂