In the last my blog, I experimented A/B testing for Bluemix application. This is useful to validate assumption. Suppose feature B was attractive. So I’d like to deploy only B. However, how can we deploy the application with (near) zero downtime ?
There is a deployment technique of new features called “Canary Deployment”. This new emerging technology is frequently talked among web developers recently. Figure show how it is done.
The developer prepare the machine environment called canary cluster, and deploy new applications to it. After the acceptance testing, the developer setup route to new application, and send the small traffic (very small amount) to the canary cluster. If somethings go wrong, developer delete the route to canary cluster. Otherwise, gradually increases the traffic up to 100%. When everything worked perfectly, developer delete the route to old cluster, and eventually delete the application running environment.
Gartner predicts that by 2017, enterprise web architecture will move to Web-scale IT architecture. This is common architecture in large Web IT company such as Google, Netflix now. The common keyword here is “horizontal scale” rather than vertical. The vertical means to increase the memory, increase the number of CPU cores. The horizontal means to increase the number of application or service instances.
How this can be done with Bluemix ?
Bluemix is based on Cloud Foundry, an open base PaaS infrastructure. From the beginning, it is designed for Web-Scale IT. It has feature for horizontal scalability.
cf scale -i
The “cf scale” does support vertical scale using “-m (memory)” option too. The number of CPU also scales based on memory assignment.
I have created very slow PHP application V1. It has artificial sleep(5) command in the PHP application.
echo ‘<p>Hello World</p>’;
The application scale parameter is set to 5 instances and 64Mbyte.
>cf app amntphpv1 requested state: started instances: 5/5 usage: 64M x 5 instances urls: amntphp.mybluemix.net
Now, I run very primitive performance test using ab (ApacheBench) command.
$ ab -n 20 -c 20 http://amntphp.mybluemix.net/ <snip> Connection Times (ms) min mean[+/-sd] median max Connect: 197 201 3.0 202 208 Processing: 5215 5890 1234.4 5258 10211 Waiting: 5215 5890 1234.4 5258 10211 Total: 5414 6092 1236.1 5457 10416
As I expected, the connection took more than 5 seconds. Then, I fixed the problem by removing “sleep(5)” command from PHP application. And deploy the application as V2 with 2 instances, and scale down the V1 application to 3.
>cf push amntphpv2 -b https://github.com/dmikusa-pivotal/cf-php-build-pack -m 64M -n amntphp -i 2 >cf scale -i 3 amntphpv1 >cf apps amntphpv2 started 2/2 64M 1G amntphp.mybluemix.net amntphpv1 started 3/3 64M 1G amntphp.mybluemix.net
Did the same test.
$ ab -n 20 -c 20 http://amntphp.mybluemix.net/ Connection Times (ms) min mean[+/-sd] median max Connect: 197 202 2.7 202 206 Processing: 213 1744 2361.6 270 5432 Waiting: 213 1744 2361.7 270 5432 Total: 412 1946 2361.5 475 5636
The response time became better. So I’d like to scale the V2 to 5, and then unmap the V1 application.
$ cf scale -i 5 amntphpv2 $ cf unmap-route amntphpv1 mybluemix.net -n amntphp
The “cf unmap-route” command means that no traffic will goes to V1 application. And then, run the same test again.
$ ab -n 20 -c 20 http://amntphp.mybluemix.net/ Connection Times (ms) min mean[+/-sd] median max Connect: 197 201 3.0 202 207 Processing: 210 261 56.0 221 348 Waiting: 210 261 56.0 221 348 Total: 408 462 56.3 423 550
As I expected, the performance was much better now.
In the actual situation, the deployment must be done carefully. For example, developer needs to test the new application using different route. Also, developer may need to take care of existing sessions. In the cloud application, it is recommended that application is stateless, and recover the session from existing cache.
This blog entry was to show how we can do canary deployment for Bluemix application, so I omitted the detail technology discussion.