A/B Testing for Bluemix application.

Some of you may heard about A/B testing. I believe this became famous when Obama Digital team produced campaign site. Simply stated, different web pages will be displayed on the same URL. This is useful to test the hypothesis which page are more attractive. There is A site and B site for the same URL. Some users traffic goes to A site, and the rest goes to B site. The team carefully monitor the access to the site to see which site is more attractive.

I wondered how this can be done with Bluemix. Here is my experiment steps.

1. Create a single route

> cf create-route dev mybluemix.net -n amntabtesting

2. Create applications with no-route option.

>cf push --no-route -p A -b https://github.com/cloudfoundry-community/nginx-buildpack -m 64M amntatesting
>cf push --no-route -p B -b https://github.com/cloudfoundry-community/nginx-buildpack -m 64M amntbtesting

3. Bind the both applications to the same URL.

> cf map-route amntatesting mybluemix.net -n amntabtesting
> cf map-route amntbtesting mybluemix.net -n amntabtesting

4. Check the application URL

> cf apps
amntatesting     started           1/1         64M      1G     amntabtesting.mybluemix.net
amntbtesting     started           1/1         64M      1G     amntabtesting.mybluemix.net

As you can see the same URL is mapped to different applications. I tested  the Web URL, and hit “reload” button several times. As I expected, I see different sites for the same URL.

2014-07-24  web site a

2014-07-24 web site b

So the Bluemix is not only provide horizontal scalability, but also provide A/B testing capability. I believe this may be useful for canary testing too. In this test, some of new features are added and deployed to Bluemix to see if the new features are working correctly. If not, developer can simply unbind the route to new  application. When testing goes well, then developer can unbind old applications. I may explorer more on how to do canary testing on Bluemix.

This entry was posted in Bluemix. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s