Nginx Split Testing Basics

Split testing is a method of conduting controlled, randomized experiments with a goal of improving a website materic such as clicks, form completions or purchases. Incoming traffic to the website is distributed between the original and the different variations of a page without any of the visitors knowing that they are part of an experiment.

Using NGINX/OpenResty for A/B Testing

Nginx and its full pack Openresty variant provides out of the box method for controlling where the traffic is sent via its http_split_clients module. For a simple scenario we will create two different pages (index.new.html and index.old.html) to be randomly served via the same URL (http://example.com/)

The NGINX configuration could be as simple as this:

http {
    split_clients "app${remote_addr}${http_user_agent}${date_gmt}"   $variant {
        80%     .new;
        *       .old;
    }
server {
        # ...
        listen 80;
        location / {
           index index${variant}.html
        }
    }
}

Session Persistence

The above exmple will work but it doesn’t give the user a persistent experience. If you refresh the page, there is a chance you will switch between the variations, and this can be a horrid user experience.

A simple approach to solve the above problem is to set a cookie with an expiration time for the duration of the test and then check the cookie existence to ensure a uniform user experience.

We will use NGINX’s map directive to map the $cookie_split_test_version to the different variants (.new|.old) we created earlier.

The new configuration now looks like:

http {
    map $cookie_split_test_version $upstream {
        default $variant;
        v1 ".old";
        v2 ".new";
    }
    split_clients "app${remote_addr}${http_user_agent}${date_gmt}"   $variant {
        80%     .new;
        *       .old;
    }
server {
        # ...
        listen 80;
        location / {
            add_header Set-Cookie "split_test_version=$upstream;Path=/;Max-Age=518400;";
            index index$upstream.html
        }
    }
}

NGINX prefixes the cookies with $cookie_

In next article we will cover to advances topics such as upstreams and setting the cookie from an application server

comments powered by Disqus