Server-side AB testing in ASP.NET Core web applications

You've likely heard of Optimizely, or ABTasty, but what if you don't want to shell out money to do your own server-side testing? You know the benefits of server-side testing, it is invaluable. Whether you are a big shop, or small team, you can implement server-side testing today in your ASP.NET Core web application. Today, we will explore how to implement server-side testing in ASP.NET Core web applications with OSWTF.

A/B testing

Disclaimer: I am the author of OSWTF; it is based on work I have previously done working in an ecommerce team. The framework itself does not offer any analytics integration, as that is likely company/project-specific and can easily be added yourself.

*Please be aware before the package gets released as a major release, the framework is subject to breaking changes.

What is OSWTF

OSWTF stands for Open Source Web Testing Framework. The framework is a very simple integration into your web applications to enable server-side testing capabilities. The framework relies solely on filters and options to enable you to set up your own server-side tests.

Let's look at how you can use OSWTF in less than 5 minutes!

Getting your solution set up

1. Install the NuGet package in your project (Tip: install the package from the Package Manager Console -> "Install-Package OSWTF.Server -Version 0.0.5-alpha")
2. Modify your Startup.cs class
   

   using OSWTF.Server.Models;
   
   namespace MyWebProject
   {
       public class Startup
       {
           public void ConfigureServices(IServiceCollection services)
           {
               // Add this line
               services.Configure<TestingOptions>(Configuration.GetSection("OSWTF"));
           }
       }
   }
   

3. Add an entry into your appsettings.json file
   

   {
     "Logging": {
       "LogLevel": {
         "Default": "Warning"
       }
     },
     "AllowedHosts": "*",
     "OSWTF": {
       "Tests": [
         {
           "Name": "MyFirstTest",
           "Split": "50,50",
           "Begin": "2/17/2019",
           "End": "3/17/2019"
         }
       ]
     }
   }
   

4. Modify your controller action you'd like to AB test
   

   namespace MyWebProject.Controllers
   {
       public class HomeController : Controller
       {
           [TypeFilter(typeof(ServerSideTestFilter),
               Arguments = new object[] { new string[] { "MyFirstTest" } })]
           public IActionResult Index()
           {
               string variation = ViewBag.ServerSideTests["MyFirstTest"];
   
               // Branch based on variation value
   
               return View();
           }
       }
   }
   

That's it!

Configuring your test

You have a number of options to configure your test with, all of these options are configurable within the appsettings.json file. A sample appsettings.json file with all available options is shown below.

{
  // rest of appsettings.json.. 

  "OSWTF": {
    "Prefix": "oswtf_",
    "Tests": [
      {
        "Name": "MyFirstTest",
        "Split": "50,50",
        "Begin": "2/15/2019",
        "End": "3/15/2019",
        "Exclude": [],
        "TurnOff": false
        "ForceVariation": "1" 
      }
    ]
  }
}

• Prefix - sets the prefix of the cookies of the server-side tests. This defaults to "oswtf_".
• Name - the name of the test.
• Split - how you want to split the traffic to different variations of the test. Each number represents a percentage (totaling to 100%). You can pass any number of comma-separated numbers totaling 100.
• Begin - the begin date of the server-side test.
• End - the end date of the server-side test. This is optional - if not provided, the server-side test will run indefinitely.
• Exclude - an array of string values that match to any Name parameter/s. If provided, a given server-side test will be put in variation 0 (the control) if the user is already in any of the excluded server-side tests.
• TurnOff - a flag to turn off the test. If not provided (or equal to false) the test is assumed to be active.
• ForceVariation - used to force a particular variation for users who have not already been put into the server-side test. This value is zero-indexed (starts at 0) and can have a max value of one less than the number of elements in Split.

How does it work

If you want to go into the code, head on over to the Github page. The code you'll want to find is in the OSWTF.Server folder. You can also run a demo from the OSWTF.Demo folder.

For those that want a brief overview.

1. When a user navigates to a webpage that is decorated with the ServerSideTestFilter, the filter will be executed.
2. Depending on the test's settings, the user will be put into a variation. If the user is already in a variation, as noted by the presence of a cookie that holds the variation value, the cookie's value will be used instead.
3. The variation value will be saved in a cookie (this makes server-side testing with a web farm that much easier) and persisted to the ViewBag.
4. You, the programmer, use the value of the variation and code the server-side test as necessary.

FAQ

While using OSWTF, you should be aware of these facts.

• The variation value that is available to you in the controller is zero-based. That is, if a given test is splitting traffic 50-50, the variation value can either be 0 or 1.
• The values that you pass in the TypeFilter in your controllers must match the Name key in your appsettings.json file.
• You can have a single action process multiple server-side tests, simply by adding more elements to the string array parameter in the ServerSideTestFilter is all you need to do.

Suggestions / Bugs

Please submit a issue.