Canary deployment for Bluemix application

In the last my blog, I experimented A/B testing for Bluemix application. This is useful to validate assumption. Suppose feature B was attractive. So I’d like to deploy only B.  However, how can we deploy the application with (near) zero downtime ?

Canary Deployment

There is a deployment technique of new features called “Canary Deployment”. This new emerging technology is frequently talked among web developers recently. Figure show how it is done.

2014-07-28 canary deployment

The developer prepare the machine environment called canary cluster, and deploy new applications to it. After the acceptance testing, the developer setup route to new application, and send the small traffic (very small amount) to the canary cluster. If somethings go wrong, developer delete the route to canary cluster. Otherwise, gradually increases the traffic up to 100%. When everything worked perfectly, developer delete the route to old cluster, and eventually delete the application running environment.

Web-scale IT

Gartner predicts that by 2017, enterprise web architecture will move to Web-scale IT architecture. This is common architecture in large Web IT company such as Google, Netflix now. The common keyword here is “horizontal scale” rather than vertical. The vertical means to increase the memory, increase the number of CPU cores. The horizontal means to increase the number of application or service instances.

How this can be done with Bluemix ?

Bluemix is based on Cloud Foundry, an open base PaaS infrastructure. From the beginning, it is designed for Web-Scale IT. It has feature for horizontal scalability.

 cf scale -i

The “cf scale” does support vertical scale using “-m (memory)” option too. The number of CPU also scales based on memory assignment.


I have created very slow PHP application V1. It has artificial sleep(5) command in the PHP application.

echo ‘<p>Hello World</p>’;

The application scale parameter is set to 5 instances and 64Mbyte.

>cf app amntphpv1
requested state: started
instances: 5/5
usage: 64M x 5 instances

Now, I run very primitive performance test using ab (ApacheBench) command.

$ ab -n 20 -c 20
Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      197  201   3.0    202     208
Processing:  5215 5890 1234.4   5258   10211
Waiting:     5215 5890 1234.4   5258   10211
Total:       5414 6092 1236.1   5457   10416

As I expected, the connection took more than 5 seconds. Then, I fixed the problem by removing “sleep(5)” command from PHP application. And deploy the application as V2 with 2 instances, and scale down the V1 application to 3.

>cf push amntphpv2 -b -m 64M -n amntphp -i 2
>cf scale -i 3 amntphpv1
>cf apps
amntphpv2       started           2/2         64M      1G
amntphpv1       started           3/3         64M      1G

Did the same test.

$ ab -n 20 -c 20

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      197  202   2.7    202     206
Processing:   213 1744 2361.6    270    5432
Waiting:      213 1744 2361.7    270    5432
Total:        412 1946 2361.5    475    5636

The response time became better. So I’d like to scale the V2 to 5, and then unmap the V1 application.

$ cf scale -i 5 amntphpv2
$ cf unmap-route amntphpv1 -n amntphp

The “cf unmap-route” command means that no traffic will goes to V1 application. And then, run the same test again.

$ ab -n 20 -c 20

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:      197  201   3.0    202     207
Processing:   210  261  56.0    221     348
Waiting:      210  261  56.0    221     348
Total:        408  462  56.3    423     550

As I expected, the performance was much better now.

In the actual situation, the deployment must be done carefully. For example, developer needs to test the new application using different route. Also, developer may need to take care of existing sessions. In the cloud application, it is recommended that application is stateless, and recover the session from existing cache.

This blog entry was to show how we can do canary deployment for Bluemix application, so I omitted the detail technology discussion.

This entry was posted in Bluemix. Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s