Access to Cloud Foundry back-end services via tunnel over HTTP (alternative to vmc tunnel)

I recently read about blog post by Takeshi Morikawa on Chisel ( https://github.com/morikat/chisel ). Chisel was originally developed by Jaime Pillora (https://github.com/jpillora). This software brings TCP tunnel over HTTP. Morikawa-san’s idea is to use this software to access back-end services on Cloud Foundry base PaaS, such as Cloudn or IBM Bluemix . Some of Cloud Foundry users may remember that there was “tunneling feature (vmc tunnel)” in Cloud Foundry V1. This allowed to connect to back-end services such as mysql, postgresql from client command line. Unfortunately, this feature was gone in Cloud Foundry V2.

Morikawa-san found out a way of similar functionality to us. “Connect to Service(PostgreSQL & MySQL) on Cloudn PaaS v2(CloudFoundry)” (written in Japanese)  .  This is excerpt of Morikawa-san’s blog on how to use Chisel for Cloud Foundry base PaaS. This example used “mysql” service available on Bluemix, but should be able to provide access to any services. I have to modify some of the sample code to be able to run Chisel, especially Go dependency did not work well on Bluemix, so I have worked around via old .godir method. Here is a procedure to deploy chisel application.

$ cf create-service mysql 100 amnt-mysql        # Create mysql service
$ git clone https://github.com/morikat/chisel   # Clone Chisel application source code
$ cd chisel
$ mv Godeps /tmp
$ echo "github.com/morikat/chisel" > .godir
$ cf push --no-start amnt-chisel                # no-start to bind service later
$ cf bind-service amnt-chisel amnt-mysql
$ cd start amnt-chisel

To make is easy to verify the chisel application, I have also deployed phpMyAdmin application so that I can create table from the GUI as a reference.

$ git clone https://github.com/dmikusa-pivotal/cf-ex-phpmyadmin
$ cd cf-ex-phpmyadmin
$ mv manifest.yml manifest.yml-original
$ cf push --no-start -m 256M -b https://github.com/cloudfoundry/php-buildpack.git amnt-phpmyadmin
$ cf push --no-start -m 256M amnt-phpmyadmin
$ cf bind-service amnt-phpmyadmin amnt-mysql
$ cf start amnt-phpmyadmin

I have created SAMPLE table shown in the attached screenshot. Please refer to the result of “SELECT * FROM SAMPLE” in the bottom of the screenshot. It has FRISTNAME, LASTNAME and EMAIL.

2015-04-07 11_40_22-myphpadmin

Connect to mysql service via “Chisel” application

A shell script called “caldecott.sh” is provided to access to the service as a sample. I needed to supply application GUID (because I’m running go version of Cloud Foundry CLI). I have modified caldecott.sh to supply my application GUID in “APP_GUID” environmental variable in the caldecott.sh. Now, I run the command.

Connect to mysql service via “Chisel” application. The application name is provided as an argument to “caldecott.sh” script.

$ ./bin/caldecott.sh amnt-chisel
2015/04/07 17:34:36 client: Connecting to ws://amnt-chisel.mybluemix.net:80

mysql >

Now, I see “mysql” prompt here. I queried the table which I created at phpMyAdmin application.

mysql> select * from SAMPLE;
+——-+———–+———-+—————-+
| ID    | FIRSTNAME | LASTNAME | EMAIL          |
+——-+———–+———-+—————-+
| 12345 | Takehiko  | Amano    | amano@hoge.com |
+——-+———–+———-+—————-+
1 row in set (0.44 sec)

I was able to query the mysql services on Bluemix from my local client now !

Advertisements
Posted in Bluemix | Leave a comment

SSH access to Bluemix container via cf-ssh

There are frequent questions related to how to access Bluemix container via ssh command. Bluemix is a PaaS. Hence it is not providing infrastructure, but a platform to run application with various run-times (Java, Ruby, Node, PHP, etc) without worry about infrastructure too much. But sometimes, developers want to login solely for debugging purpose. There several articles which mentions about how to access container. For example, “Easy SSH Sessions into Bluemix”  mentions about using “cf-ssh” to access Bluemix container. However, unfortunately it does not tell how to run your application for the debugging.

This blog entry is a kind second article mentioned above to actually run the application. The original idea comes from article by excellent engineer Kusama-san (@jacopen) (link to article written in Japanese). This blog entry is mostly relies on his article.

cf-ssh

First of all, what is “cf-ssh” ?  This is a utility to access container for Cloud Foundry base PaaS via ssh. As both articles explains, the idea comes from Dr. Nic (link to article) using tmate technology. This technology share’s console so that developer can easily take a look at container through console. His article explains about cf-ssh script. There is yet another, but much simpler implementation of “cf-ssh” in Cloud Foundry community (https://github.com/cloudfoundry-community/cf-ssh).

This utility can be downloaded from here . Cloud Foundry CLI (cf) and ssh command must be installed to use “cf-ssh” utility. I verified this utility on Linux platform (Ubuntu), so this method may not work on Windows platform.

The usage is very simple. Under the directory of application, run:

$ cf-ssh -f manifest.yml

That’s it !

Note that manifest.yml is required to use this utility. If it succeeds, it opens a connection to public terminal sharing on tmate.io, and then login to that terminal. The application sources as well as buildpack (run-time) is also deployed on the container. The application is deployed as <appname>-ssh with no route. Note that the session is one time only.

2015-01-15 15_09_02 terminal

Create a route

The application is deployed without route. But we can create a route for this application so that we can use web browser to access to the application.

$ cf create-route <space> mybluemix.net -n <appname>-ssh
Creating route <appname>.mybluemix.net ...
OK

$ cf map-route <appname>-ssh mybluemix.net -n <appname>-ssh
Adding route ...
OK

Run the application

A tmate-bootstrap is run as an application for “<appname>-ssh”. This also works as a reverse proxy. The expected application port is 8080. So we need to modify the listen port from VCAP_ENV_PORT to 8080. For example, in case of Node application, I need to modify like this:

//var port = (process.env.VCAP_APP_PORT || 3000);
var port = 8080;

Then, I can run the application (Node application in this case) from the terminal.

vcap@18ckfi1j8nh:~$ node app.js

As I imagined, I can view the application from web browser, and also take a look at console at the same time !

2015-01-15 15_14_12-run the application

2015-01-15 15_13_15-application

Most likely, this technique can be used for debugging purpose. For example, there is a situation that the problem only happens in Bluemix environment, not in development environment, and “cf logs” gives no clue for us.

It may be possible to access to the backend services from this terminal. For example, upload initial data to mysql, query existing database, so on. Unfortunately, I have not explored this option yet (but put the idea as my blog backlog 🙂 ).

I have to remind that for true terminal services, we can leverage Bluemix container service, as “cf-ssh” is not intended to provide terminal for the developers.

Posted in Bluemix | Leave a comment

Handy JavaScript editor as Bluemix application

Sometimes, I want simple text editor on the web for developing JavaScript. This is to share the codes among multiple desktop (I’m using many PCs), and work with other developers (e.g. pair programming). I normally use IBM DevOps Services to do it. However, sometimes, I want very simple just for experiment. This is very similar situation where GitHub provides Code Snippet page to host very ad-hoc code snippet.

So I’ve tried Node version of Orion editor to run on Bluemix. This is not perfect solution for me because when the application stops, my editor contents will be gone. Eventually, code storage option may be developed to store files on Swift based object storage or Amazon S3. But for now, I experimented to see if my idea works or not.

First of all, I need source of Node version of Orion Editor.

$ npm install orion

The command installed orion node module inside node_modules. I have to edit source a bit so that it can be run on IBM Bluemix.

$ cd node_modules/orion
$ vi server.js

And I have modified port variable.

var port = process.env.PORT || args.port || args.p || 8081;

Then, I pushed this application to Bluemix.

$ cf push --random-route -c "node server.js"

As I imaged, I now have my own editor on the web as Bluemix application !

2015-01-14 12_07_33-my orion editor

Note: I have removed Bluemix application URL so that my codes are not copied somewhere. That’s why I used “–random-route” option in “cf push”. 🙂

I may explorer this solution more so that files created to be permanent. However, the priority is very low, since we have IBM DevOps services. This editor is just handy to fulfill my purpose to share code snippet with other PCs and other developers in remote location.

Posted in Bluemix, JazzHub | Leave a comment

Host Bluemix application with the same URL on multiple regions

Bluemix now has multiple region (US and UK) to host applications. So user has choice of selecting regions depending on network connectivity. I was wondering that because Bluemix is based on open platform called Cloud Foundry, it should be able to support the same application on each Cloud Foundry instances via DNS round robin.

This is a preliminary experiment for hosting the same application on both region using the same URL. Bluemix can bring your own domain. For example, I can have “example.com” domain for my applications. I noticed that I can set up the same domain for each region in US and UK.

US

$cf domains
name             status
mybluemix.net    shared
ng.bluemix.net   shared
example.com      owned

UK

$cf domains
name                  status
eu-gb.mybluemix.net   shared
eu-gb.bluemix.net     shared
example.com           owned

I have setup DNS server locally, and setup it to do round robin for each host. For example, dnstest.example.com has IP address, and the same host has another IP address.

2014-12-08 15_33_19-dns server

And created an alias dnsapp for the host (dnstest). If I run “nslookup” command to search for dnsapp.example.com, it surely returns IP address via round robin.

 

$ nslookup dnsapp.example.com
 Name:    dnstest.example.com
 Addresses:  5.10.124.141  75.126.81.68
 Aliases:  dnsapp.example.com
$ nslookup dnsapp.example.com
 Name:    dnstest.example.com
 Addresses:  75.126.81.68  5.10.124.141
 Aliases:  dnsapp.example.com

Then, I pushed the same application with slight modification to show the region (i.e. US or UK).

# Push the application to US region
$cf login -a https://api.ng.bluemix.net
$cf push dnsapp -d example.com
# Push the application to UK region
$cf login -a https://api.eu-gb.bluemix.net
$cf push dnsapp -d example.com

The result was not that I expected one. I always got the same result. It seems either computer or browser seems to cache the IP address. So I need to bring two PCs. And open the http://dnsapp.example.com. Here is a result. I now see the same application with the same URL runs on both Bluemix regions.

2014-12-08 15_27_13-us region 2014-12-08 15_26_39-UK region

If this works, I think this can become yet another solution for redundancy. For example, we might get some disruption in the region due to security update (e.g. Shell Shock). But if we have the same application with the same URL in the different region, it is likely that the application will be able to serve to end-user without significant impact.

I also think this can be used on another Cloud Foundry instances, such as NTT’s Cloudn PaaS, Pivotal’s one, or your own. This means the same application with the same URL should be able to run on various Cloud Foundry instances. Our application scales beyond just one cloud vendor. This is the power of Cloud Foundry openness. By the way, this blog is still experimental state, but surely good to start about scalability beyond region.

Posted in Bluemix | 3 Comments

Leverage Bluemix application on IBM DevOps Services dashboard

I noticed about technical article on how to leverage Bluemix application in WebSphere portal (https://developer.ibm.com/digexp/docs/docs/integration/websphere-portal-bluemix-integration/). This inspired me to add widget to access Bluemix application inside IBM DevOps Services dashboard.

IBM DevOps Services offers core application development lifecycle utilities, such as software configuration management, defect tracking, task management, build utilities and deployment to Bluemix. One of the capability is dashboard which powered by Rational Team Concert ( https://jazz.net/products/rational-team-concert/features/report ). This is very useful for Agile team who want to track and measure software development lifecycle. For example, it offers:

  • Burn-down chart to determine if the development is healthy or not.
  • Major defects to resolve as technical debt.
  • and much more.

This dashboard can call other site URL through “External Content” widget.

2014-10-30 13_43_03 external content

For example, I can add Bluemix documentation in the dashboard.

2014-10-30 13_59_34-dashboard with Bluemix app

Like this, IBM DevOps Services dashboard can be extended to include Bluemix application just like portal. I think there are several usage for it.

  • Hold project information website which cannot be expressed just with README.md file.
  • Fun pages like team’s photo and/or software demonstration video pages.
  • etc

If we remember about Scrum process, demonstration must be done at the end of iteration. My company’s team seems to record iteration demo as video so that even if stakeholders can not attend the session, they can watch the demonstration. So if we build very simple application to hold videos, we can embed this Bluemix application in the dashboard so that stakeholder not only able to monitor team’s process but also watch demonstrations to understand what features were implemented without leaving dashboard.

Since the simple web site probably requires minimal memory (say 64M), there will be less impact on the cost of project. Leveraging Bluemix application for software development can be useful to on IBM DevOps services, just like WebSphere portal is quite useful to aggregate information in a single page.

Posted in Bluemix, JazzHub | Leave a comment

Debug Bluemix application on your local PC

When I discuss with Bluemix application developers, I frequently hear about how to debug the application on Bluemix. There are several technique for it. One of the best known technique is probably to use cf-debug-tools written by Pivotal engineer. This tool opens a websocket connection between web browser and container, and it accept command. To be able to use this tool on Bluemix, web socket URL needs to be changed from “ws” to “wss”. I used Chrome browser to run it.

2014-09-17 cf-debug-tools sample screen

Another promising way to debug Bluemix application is to use “Cloud Focker” from CloudCredo team ( https://github.com/CloudCredo/cloudfocker ). This project has just started, but I like this project from various view points.

  • I can run the application in my local environment with less memory resource compared to 1VM Cloud Foundry (Bosh-lite).
  • I can try new buildpack, and potentially debug the buildpack itself.
  • Prepare for Docker integration expected to happen in Cloud Foundry community space.

Running Cloud Focker is very simple.

  1. Install Vagrant and VirtualBox in advance.
  2. Clone the Cloud Focker Project.
    $ git clone https://github.com/CloudCredo/cloudfocker.git
  3. Deploy Cloud Focker image and run it.
    $ cd cloudfocker
    $ vagrant up –provider virtualbox
    $ vagrant ssh

Good thing of Cloud Focker is support for Cloud Foundry community buildpacks  ( some of buildpacks may not run with it). For example, I can add Node.js buildpack, and then deploy the application as follows.

 $ fock add-buildpack https://github.com/cloudfoundry/nodejs-buildpack
 $ cd <node-js-app>
 $ fock up

The last command kicks off Cloud Foundry container image, and then just like Cloud Foundry PaaS, it will stage and run the application on the container.

2014-09-17 15_20_19-Cloud Focker

For example, I can now debug my favorite “Sentiment Analysis App” application available on IBM DevOps Service in my local PC (note about URL is localhost:8080 ).

2014-09-04 16 twitter sentiment

Another good thing is I can enter into container image using “nsentor” command.

$ docker ps
$ PID=$(docker inspect --format {{.State.Pid}} <container id>)
$ sudo nsenter --target $PID --mount --uts --ipc --net --pid

2014-09-17 inside docker container

Again, Cloud Focker project has just started, so probably you need to have skill on Docker and Golang to trouble shoot if there is any issue, and then contribute. Hopefully, I’ll be able to contribute to this project as well.

Posted in Bluemix | 2 Comments

Take a look at Bluemix container image

Sometime, I want to check what are there in Bluemix application container. There is simple way to take a look at the inside Bluemix container image.

The output of “cf stacks” command returns “lucid64” which is build from Cloud Foundry stacks project ( https://github.com/cloudfoundry/stacks ). Here is how I build the lucid64. I’ve installed VirtualBox and Vagrant as well as Cygwin (with several necessary environment such as Ruby, g++, libcrypt-devel, etc) in advance.

$ git clone https://github.com/cloudfoundry/stacks.git
$ cd stacks
$ gem install bundler
$ ./init

It takes a while, since Vagrant try to download OS image. Then, I run build_stack command as follows:

$ ./build_stack

This command created the rootfs for container in /tmp/warden/rootfs. Now, I can go through most of file systems (except /proc) and commands available in Bluemix application container image.

2014-09-03 lucid64 stack

For example, I was able to find “perl” command.

$ export LC_ALL=POSIX
$ sudo chroot /tmp/warden/rootfs
# /usr/bin/perl -version
This is perl, v5.10.1 (*) built for x86_64-linux-gnu-thread-multi

Since the rootfs image is built using Ubuntu lucid image, there is no wonder that I can use “perl” because this is default package in the release ( http://releases.ubuntu.com/releases/lucid/ )

In addition to the default packages, stack build process add several additional packages described in “build” script of stacks ( https://raw.githubusercontent.com/cloudfoundry/stacks/master/lucid64/build ).

The list contains package like “imagemagick”. So this means, the Bluemix application can utilize the image conversion library in various languages. Of course, the application should be able to run Perl codes at server-side.

If you wonder what commands and libraries are available for you to build wonderful Bluemix application, it may be worthwhile to take a look at actual image.

Posted in Bluemix | 1 Comment