IBM Bluemix OpenWhisk REST API

IBM recently announced event driven application execution environment called OpenWhisk. This is what we can find from the page.

OpenWhisk provides a distributed compute service to execute application logic in response to events.

Event driven programing are used for a long time, especially in designing application (both client native and web application). For example, user clicked mouse, user input text, so on. Programing model normally use event handler to handle those event. Likewise, the needs to handle internet events are rapidly growing. Mostly, the rise of internet of things (IoT) are overwhelming around us.

These things ( including human ) produce events, such as location changes, temperature changes in case of sensor devices. In case of human, they produce text or pictures in SNS timelines. These also can be thought as events on the internet.

IBM Bluemix provides a service called IoT Foundation for IoT devices, and Message Hub for general purpose event store or broker. However these service do not provide so called “Event Handler” as is. OpenWhisk is a solution for it. The OpenWhisk also is designed to popular “server less” style favor developers, since it accept single purpose code, and immediately runs on the cloud. This is sometimes called “microservice” type application because it provides single purpose business.

OpenWhisk project itself has good tutorial on how to run code on IBM Bluemix OpenWhisk platform. If you are not familiar with this technology, I recommend to run tutorial found in Bluemix document for OpenWhisk.

This blog is intended to provide additional information, “how to run REST API” against IBM Bluemix OpenWhisk. The REST API document may come soon. But early adapter may want to try first like myself, since I was the one who loves microservice architecture.

Calling REST API via “curl”

The API endpoint at the time of writing is this:

https://openwhisk.ng.bluemix.net/api/v1

And to call API, the following header must be set.

Content-Type: application/json
Authorization: Basic <based64 encoded key>

The <base64 encoded key> is a base64 encoded Authorization Key. If you run tutorial, this is set via “wsk property set” command. base64 encodeing can be done like:

$ wsk property get --auth | awk '{printf("%s", $3)}' | base64 --wrap=0

I have put these values as TYPE and AUTH environmental variable. I also put my name space in NAMESPACE environmental variable.

$ export TYPE='Content-Type: application/json'
$ export AUTH='Authorization: Basic '`wsk property get --auth | awk '{printf("%s", $3)}' | base64 --wrap=0`
$ export NAMESPACE=<my-name-space>

REST API are documented in swagger API (Bluemix, OpenWhisk). For example, if I want a list of actions, call “<api-end-point>/namespaces/<namespace>/actions” will show us a list of them.

$ curl -s -X GET -H "$TYPE" -H "$AUTH" https://openwhisk.ng.bluemix.net/api/v1/namespaces/$NAMESPACE/actions | jq ".[] | .name"
"weather"
"hello"

I got actions “weather” and “hello” which I have created by executing tutorial. “jq” command is used to filter JSON object. In this case, it shows name of actions. To execute an action,  call <api-end-point>/namespaces/<namespace>/actions/<action-name> with POST method. Data must be provided in json format if action requires parameters.

$ curl -s -X POST --data '{"location": "kyoto"}' -H "$TYPE" -H "$AUTH" https://openwhisk.ng.bluemix.net/api/1/namespaces/$NAMESPACE/actions/weather?blocking=true | jq '.response.result.msg'
"It is 10 degrees in kyoto and Partly Cloudy"

I have supplied “?blocking=true” parameter to obtain the result immediately. This is equivalent to specify “-b” option in “wsk action invoke” command.

Calling REST API within Node.js application

Calling this REST API from application should be straight forward. For example, I can write to call API like this:

    var result;
    var apihost = WhiskApi.host;
    var apipath = WhiskApi.path + "/actions/" + action + "?blocking=true";
    var opts = {
        host: apihost,
        path: apipath,
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': apiKey
        }
    };

    // https request to Whisk 
    var req = https.request(opts, function(res) {
        var output = '';
        res.on('data', function(chunk) {
            output += chunk;
        });
        res.on('end', function() {
            result = JSON.parse(output);
            callback(result, null);
        });
    });

    // Send request body : action parameters.
    req.write(JSON.stringify(param));
    req.end();

First, set the options to pass to “https.request()” method, and then wait for response.

I wrote sample web application, it worked very well.🙂

whisk-demo-web-app

So I was able to add “weather” feature to my web application very easily. This is good part of OpenWhisk feature as microservice.

 

Posted in Bluemix | 4 Comments

Easy usage of Bluemix Object Storage service

IBM Bluemix evolution continues. Since last year (2015/Nov), Bluemix has been providing “Object Storage” service as beta. This service is built using OpenStack swift technology.  Most of people may not be aware of, but Bluemix is not only Cloud Foundry platform, but also runs several OpenStack services or similar ones. Object Storage is one of them. Good things for cloud developers is that Object Storage service has “free” amount. This is currently 5Gbyte at the time of writing (2016/Feb). This is mostly equivalent to file storage service such as Amazon, or Microsoft OneDrive.  I had trouble in accessing this service from my development environment, but I have a solution to it. I thought it may be worth while to share the solution for those who have similar problem.

The problem

I have been working on a project which uses OpenStack. And the primary OpenStack environment is setup to use target OpenStack. Unfortunately, my environment is not against latest OpenStack version. While Bluemix Object Storage service is based on V3 which is quite latest version. For example, I was not even able to run “swift list” command.

$ swift --version
swift 2.0.3
$ swift list
Unauthorised. Check username, password and tenant name/id

And fortunately, python provides a way to install various version of command via “virtualenv” mechanism. It is possible to run various version of python in a virtual environment. And surely it worked for Bluemix Object Storage service too.

In this articles, I’d like to show how I setup the client environment and show simple usage of Object Storage service.

Setup client and test Object Storage service

My environment is Ubuntu 14.04. The other OS users may need to consult Python and OpenStack document to set up virtualenv and swift client.

First of all, I need to install virtualenv package.

% sudo apt-get install python-virtualenv

And then prepare virtual environment and install swift command under it like:

$ virtualenv virt
$ cd virt
$ bin/pip install python-swiftclient
$ bin/pip install python-keystoneclient
$ bin/pip install --upgrade ndg-httpsclient

Since this is virtual environment, I need to install keystone client, and some additional module (ndg-httpsclient) to run swift command. I already had one container called “notebook”. I have created a new “test” container for this article.

2016-02-02 15_44_26-object storage UI

This article focus on command line interface called “swift”. I’ve setup environmental variables to access Object Storage like below.

export OS_USER_ID=<user-id>
export OS_PASSWORD=<password>
export OS_PROJECT_ID=<project-id>
export OS_AUTH_URL='https://identity.open.softlayer.com/v3'
export OS_REGION_NAME=dallas
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_VERSION=3

The values like <user-id>, <password> and <project-id> can be obtained from Bluemix Object Storage service UI.  Then, I can run several swift commands. For example, I can see the container list from “swift list” command just like shown in Bluemix Object Storage UI.

$ swift list
notebooks
test

For a simple test, I tried to upload a simple text file “hello.txt”.

$ echo  "Hello" > hello.txt
$ cat hello.txt
Hello
$ swift upload test hello.txt
hello.txt
$ swift list test
hello.txt

And then I removed the “hello.txt” file from my local machine, and then downloaded from Object Storage to see what happens.

$ rm hello.txt
$ cat hello.txt
cat: hello.txt: No such file or directory

$ swift download test hello.txt
hello.txt [auth 1.016s, headers 2.028s, total 2.028s, 0.000 MB/s]
$ cat hello.txt
Hello

I was able to recover the lost file.  It looks like it’s working fine🙂

Version control files

My intention to use Object Storage is not only save relevant files in a safe way, but also want to version control them. Of course I use software configuration management tool like Rational Team Concert or Git for open source related, but I do not want to commit each files every time when I just want to test a slight modification.

OpenStack Object Storage provides a way to keep previous files when the same file is uploaded. By default, upload overrides objects. To enable a container to save versions, put meta-data called “X-Versions-Location” and set the name of container. I need to create “versions” container in advance, and set the values to “X-Versions-Location”.

$ swift post versions
$ swift post --header "X-Versions-Location: versions" test

Let’s see how it works. I create a next version of hello.txt file. Then upload to “test” container.

$ echo Version1 >> hello.txt
$ cat hello.txt
Hello
Version1
$ swift upload test hello.txt
hello.txt

And then I modify the file slightly and upload the file again.

$ vi hello.txt
$ cat hello.txt
Hello
Version2
$ swift upload test hello.txt
hello.txt

Now let’s see how “versions” container looks like. I run “swift list” command against “versions” container, and obtain the file contents with “swift download” command.

$ swift list versions
009hello.txt/1454395632.91154
009hello.txt/1454399672.55827

$ swift download versions 009hello.txt/1454395632.91154 -o -
Hello

$ swift download versions 009hello.txt/1454399672.55827 -o -
Hello
Version1

The option “-o -” tell to output into “stdout” instead of normal file. Now I see the original and version1 document very easily.

Creating a temporary url to share a file with your team member

Sometimes it is good idea to share a file (especially large one) temporary with your team member. Bluemix Object Storage service also offer to create temporary URL to download a file from a browser just like the other object storage.

First of all, we need to obtain the AUTH_account and set a key to swift account.

$ swift stat | grep " Account:" | awk '{print $2}'
$ swift post -m Temp-URL-Key:<key>

The first command obtains AUTH_account, and the second one set the <key>. Note that <key> is arbitrary string (e.g. result of command “$ openssl rand -base64 6 | md5sum” ). After this setup, temporary url can be create via “swift tempurl” command. For example if we want to share hello.txt file on test container for 1 hour, run like:

$ swift tempurl GET 3600 /v1/<AUTH_account>/test/hello.txt <key>

Append this url to https://dal.objectstorage.open.softlayer.com. The overall url will look like this:

https://dal.objectstorage.open.softlayer.com/v1/<AUTH_account>/test/hello.txt?temp_url_sig=5922b0ed1ca5a43fe9f12167be71f65a867cee6a&temp_url_expires=1454469261

Now you can pass this url to your team mate. This is probably useful to share a large file in your company.

Summary

This article tried to share how to use “swift” command in a virtual environment so that it does not break your current development environment. I also wanted to share minimal usage of Object Storage. Hope this is useful in your daily development activities !

 

 

Posted in Bluemix | 2 Comments

Using Splunk for IBM Bluemix syslog drain

One of the repeating voice of IBM Bluemix users are how to manage application logs. A while ago, I wrote a blog entry  to retrieve logs from IBM Bluemix using 3rd party logging service. However, some users may have concerns about using public SaaS, since log may contain some relevant information such as PI (personal information). I had a discussion with Daniel Toczala (one of emerging technology leaders inside IBM), and he suggested about the trial of “Splunk“. Because it offers some trial period, or community version for small team (free with some limitation), I was able to evaluate it.

Fortunately, Cloud Foundry itself does support Splunk, so this blog entry is about how I tried Splunk for IBM Bluemix syslog drain.

Setup Splunk

Firstly, I have created a virtual machine using IBM Bluemix VM(β) service. I have chosen Ubuntu as image. Installation of Splunk was very simple. Download Splunk Light from the download site, and then install it with single line of command.

# dpkg -i  ~/splunklight-6.3.0-aa7d4b1ccb80-linux-2.6-amd64.deb

The software was installed on /opt/splunk. Looks I like I need to install and patch log format to be able to integrate with Bluemix. I have follow the guideline from Cloud Foundry documentation ( here ) .  Based on the guide, I added “rfc5424-syslog” package, since this format is used in loggreator syslog drain output.

# cd /opt/splunk/etc/apps
# tar xvfz ~/rfc5424-syslog_11.tgz

Then, I modified the file “/opt/splunk/etc/apps/rfc5424/default/transforms.conf”

[rfc5424_host]
DEST_KEY = MetaData:Host
REGEX = <\d+>\d{1}\s{1}\S+\s{1}(\S+)
FORMAT = host::$1

[rfc5424_header]
REGEX = <(\d+)>\d{1}\s{1}\S+\s{1}\S+\s{1}(\S+)\s{1}(\S+)\s{1}(\S+)
FORMAT = prival::$1 appname::$2 procid::$3 msgid::$4
MV_ADD = true

After this setup, I need to open some ports on ubuntu machine to accept incoming syslog drain (5140) and Splunk web UI (8000) because IBM Bluemix VM image has firewall setup by default.

# iptables -A INPUT -p tcp --dport 5140 -j ACCEPT
# iptables -A INPUT -p tcp --sport 5140 -j ACCEPT
# iptables -A INPUT -p tcp --dport 8000 -j ACCEPT
# iptables -A INPUT -p tcp --sport 8000 -j ACCEPT

And then, I run splunk.

# /opt/splunk/bin/splunk start --accept-license

Splunk setting

I need to setup Splunk to accept syslog drain from IBM Bluemix. I need to create “Data Input” for syslog. From left hand side of Web UI, selected “Data -> Data inputs” menu.

add data input

This opens a list of inputs that Splunk (by default). Syslog drain uses TCP protocol, so I selected TCP.

select TCP

This opened a new entry for TCP. I have put 5140 as a port number. This was setup as open port for the communication, keep the rest of fields as blank and clicked “Next”.

setup listen port

I need to select “rfc5424_syslog” which I have installed for this integration.

select rfc

I selected IP for method. Index needs to be created as well.

create-new-index-blemix

I created “bluemix” as index.

add index bluemix

 

In the review, I confirmed the setting is right.

review

And then enabled this input.

enable TCP

IBM Bluemix setting

Bluemix application needs to bind syslog drain service in user space. I have created like this:

% cf cups splunk -l syslog://dummyhost:5140

Note that dummyhost is not real name, but to hide the actual host name. And then bind the service and restage the application.

$ cf bind-service myapp splunk
$ cf restage myapp

I played with my application, then I put the query string in the Splunk web UI  like:

source="tcp:5140" index="bluemix" sourcetype="rfc5424_syslog"

I got a stream of logs in my Splunk web interface.

actual log stream - need to edit

Even though this is Splunk Light, I can retain logs 500MB per day. This is quite useful for my experiment and evaluation of this integration for who needs this kind of solution.

Posted in Bluemix | Leave a comment

Start learning IBM Knowledge Anyhow Workbench

IBM’s jStart team presents coming emerging technology (or even predict so to speak). IBM Bluemix was one of them. Now, they offer iPython Notebook as emerging technology too. So it’s good to start learning it as there is a great chance of business.

IBM offer two versions of iPython Notebook running environment on the Cloud. One is SaaS called IBM Knowledge Anyhow Workbench which runs on Softlayer.  And the other is PaaS – Apache Spark Starter which runs on IBM Bluemix. Since I’m beginner, I choose IBM Knowledge Anyhow Workbench to start learning.

It comes with several tutorials. For example, tutorial 2 will offer to analyze Winter Olympic games. If we run the tutorial, it analyze like “Which country won the medals ?”. By executing the tutorial, it gives me the following graph.

2015-08-25 13_03_47-IBM Knowledge Anyhow Workbench - 1

The result were Norway won most medals.

This tool offer to save the data on the cloud or read from external URL. So there is no need to have local environment at all. This workbench also have several tutorials like plot functions, calling IBM Cloud Services (e.g. IBM Cloudant service).

2015-08-25 13_32_35-IBM Knowledge Anyhow Workbench-2

Many developers are building software for innovations. But just like jStart team, I also believe innovation also comes from analyzing big data scientifically. IBM Knowledge Anyhow Workbench is a really good starting point to learn about the basics and start to create a new innovation.

Posted in Big Data | Leave a comment

Developing Node-RED application for Bluemix

Most of IBM Bluemix users probably knows about the existence of Node-RED starter in the boilerplate. Starter kit is supposed to be starting point of development of application using specific run-times. However, I rarely see the actual applications which uses Node-red run-times (based on Node.js SDK). If you download the Node-RED starter kit, it contains the places for developing software.

nodered starter package

Node-RED is basically information stream processing engine. For example, information came from various sources such as IoT devices ( Phones, Devices which are connected to internet), or came from social network (Twitter, etc). Then, these information are processed and then final output is made. I see that Bluemix users use Node-RED flow editors to create this kind of flow engine, and mostly end up with printing information in the debug output.

The other day, I raised the question,  “How I can obtain the output of Node-RED process in the web page ? “.  I got an answer from developer. He instructed me to this sample – Dynamic Word Cloud. The key was to utilize “websocket” to communicate with Web client and server (Node-RED).

If I run the sample “Dynamic Word Cloud”,

2015-05-25 14_24_14-word cloud flow

it gave me really cool word cloud.

2015-05-25 14_33_25-wordcloud

Then, I have modified the sample code to display twitter message.

2015-05-25 14_15_03-nodered-flow-editor

I have re-tweeted some Bluemix related message, then I immediately see on my web page.

realtime twitter

So I really think Node-RED boiler plate is actually starter kit for real-time information processing and display on the Web platform. If we take a look at http://flows.nodered.org/, I see there are bunch of cool applications. For example, there is an application Weather Prediction with Node-Red, forecast.io and a Raspberry Pi , Calculate how far away from the International Space Station you are , so on.

Using Node-RED with websocket interaction with web pages will create a really cool web site.

Posted in Bluemix | Leave a comment

Integrate Kiuwan static test services with IBM Bluemix DevOps Services

I’m located at Tokyo, but I sometimes collaborate beyond border to assist customers. I’m working with Smith Naik who has been managers of various IBM software at IBM labs, now works for various customers to provide best solutions for them.

We were wondering how we can provide “static analysis” on the cloud using IBM Bluemix DevOps Services, and Smith found the vendor Kiuwan who provides the solution for it ( https://github.com/kiuwan/ibm-devOps-services ) . Smith double checked with product manager of DevOps Services and it turned out that Kiuwan is on the list of IBM Cloud market place (i.e. our partner). So we have started to investigate.

This is preliminary blog entry how I verified the features of this integration. First of all, I choose this tutorial article because this is public, everybody can double check it. I have chosen to add “Build and Deploy” feature in “Add Git” step. After a while, build and deploy was ready to use in the project.

2015-05-19 14_56_40-initial status

I have modified “Build” stage to include Test job. The job will provide a text area to enter test script.

2015-05-19 14_57_32-add test

I have added the following texts ( I have just followed instruction found in GitHub readme.md ). I have changed the KIUWAN_APPLICATOIN_NAME, KIUWAN_USER and KIUWAN_PASSWORD parameters.

#!/bin/bash
# invoke tests here

#Set parameters for Kiuwan Local Analyzer
export KIUWAN_SOURCE_CODE_DIRECTORY="src"
export KIUWAN_APPLICATION_NAME="amntliberty"
export KIUWAN_ANALYSIS_LABEL="`date`"
export KIUWAN_USER="<my login at Kiuwan>"
export KIUWAN_PASSWORD="<my password at Kiuwan>"
export KIUWAN_TECHNOLOGIES_TO_ANALYZE="java,javascript"
export KIUWAN_INCLUDE_PATTERNS=""
export KIUWAN_EXCLUDE_PATTERNS="**/src/test/**,**/__MACOSX/**,**/*.min.js,**/*.Designer.vb,**/*Reference.vb,**/*Service.vb,**/*Silverlight.vb,**/*.Designer.cs,**/*Reference.cs,**/*Service.cs,**/*Silverlight.cs"

#Retrieve and execute the analysis with Kiuwan Local Analyzer
curl -L https://raw.githubusercontent.com/kiuwan/ibm-devOps-services/master/scripts/kiuwan.sh | bash

And then, save the configuration, and run it. I see the test log during the operation.

2015-05-19 15_23_24-test log

After the test end, The log gave a URL to view the test result. By opening the URL, I can see several test result of Kiuwan static analysis. Here is a result of our sample code.

2015-05-19 15_03_13-static test result

It indicated it has defects in Maintainability. What kind of error it was ? They were:

“Provide Javadoc comments for public methods.”

Yes, these are actually defects. :)   Looks like the sample code has less defects in the other area, security, portability, reliability, so on.  These are quite useful information.

 

Posted in Bluemix, JazzHub | 2 Comments

Download files from Bluemix using Cloud Foundry CLI plugin

I see several questions  in the Bluemix forum about the desire to download files of running applications. Cloud Foundry CLI provides “cf files” command to just to take a look at each file/folder, not entire files. There is easy way to do it. Use Cloud Foundry CLI plugin “cf-download”.

Cloud Foundry CLI v6.7 or higher now provides plugin functionality. See this dev guide . Some of Bluemix users may have already noticed new command “cf install-plugin”. This enables to add features to existing commands.

% cf install-plugin -help
NAME:
install-plugin - Install the plugin defined in command argument

USAGE:
cf install-plugin URL or LOCAL-PATH/TO/PLUGIN [-r REPO_NAME]

The command will download the plugin binary from repository if '-r' is provided

EXAMPLE:
cf install-plugin https://github.com/cf-experimental/plugin-foobar
cf install-plugin ~/Downloads/plugin-foobar
cf install-plugin plugin-echo -r My-Repo


OPTIONS:
-r   repo name where the plugin binary is located

Cloud Foundry Foundation provides community base plugins at http://plugins.cloudfoundry.org. There are plugin such as:

  • autopilot : zero downtime deploy plugin for cf applications. This is useful to switch to new application without worry about stopping and deleting old application.
  • Targets : Can be used to change the target (e.g. to Bluemix US , to Bluemix London, or to the other Cloud Foundry base PaaS provider such as Cloudn PaaS from NTT ).
  • etc, etc.

Most of plugins are quite useful. Anyway, go back to “cf-download”. To add the plugin is very easy. Add plugin repository, and then install the plugins.

% cf add-plugin-repo CF-Community http://plugins.cloudfoundry.org/
OK
http://plugins.cloudfoundry.org/list added as 'CF-Community'

% cf install-plugin cf-download -r CF-Community
Looking up 'cf-download' from repository 'CF-Community'
8637440 bytes downloaded...
Installing plugin <path>...
OK
Plugin cf-download v1.0.0 successfully installed.

% cf plugins
Listing Installed Plugins...
OK
Plugin Name   Version   Command Name   Command Help
cf-download   1.0.0     download       Download contents of a running app's file directory

Now, I login to Bluemix, and then tried to download files (Nodered boilerplate in my case).

% cf download # just to see syntax.

Error: Missing App Name
NAME:
   download - Download contents of a running app's file directory

USAGE:
   cf download APP_NAME [PATH] [--overwrite] [--verbose] [--omit ommited_paths] [-i instance_num]

% cf download mynoderedapp .

It took several minutes to download all files. After the command completes, I can see enter files. To check if the files are identical. I took a look at run.pid file.

$ cat run.pid

31

$ cf files mynoderedapp ./run.pid
Getting files for app ...
OK

31

Looks like they are identical.  It is so easy to download files from Bluemix application now.🙂

Posted in Bluemix | 2 Comments