Easy usage of Bluemix Object Storage service

IBM Bluemix evolution continues. Since last year (2015/Nov), Bluemix has been providing “Object Storage” service as beta. This service is built using OpenStack swift technology.  Most of people may not be aware of, but Bluemix is not only Cloud Foundry platform, but also runs several OpenStack services or similar ones. Object Storage is one of them. Good things for cloud developers is that Object Storage service has “free” amount. This is currently 5Gbyte at the time of writing (2016/Feb). This is mostly equivalent to file storage service such as Amazon, or Microsoft OneDrive.  I had trouble in accessing this service from my development environment, but I have a solution to it. I thought it may be worth while to share the solution for those who have similar problem.

The problem

I have been working on a project which uses OpenStack. And the primary OpenStack environment is setup to use target OpenStack. Unfortunately, my environment is not against latest OpenStack version. While Bluemix Object Storage service is based on V3 which is quite latest version. For example, I was not even able to run “swift list” command.

$ swift --version
swift 2.0.3
$ swift list
Unauthorised. Check username, password and tenant name/id

And fortunately, python provides a way to install various version of command via “virtualenv” mechanism. It is possible to run various version of python in a virtual environment. And surely it worked for Bluemix Object Storage service too.

In this articles, I’d like to show how I setup the client environment and show simple usage of Object Storage service.

Setup client and test Object Storage service

My environment is Ubuntu 14.04. The other OS users may need to consult Python and OpenStack document to set up virtualenv and swift client.

First of all, I need to install virtualenv package.

% sudo apt-get install python-virtualenv

And then prepare virtual environment and install swift command under it like:

$ virtualenv virt
$ cd virt
$ bin/pip install python-swiftclient
$ bin/pip install python-keystoneclient
$ bin/pip install --upgrade ndg-httpsclient

Since this is virtual environment, I need to install keystone client, and some additional module (ndg-httpsclient) to run swift command. I already had one container called “notebook”. I have created a new “test” container for this article.

2016-02-02 15_44_26-object storage UI

This article focus on command line interface called “swift”. I’ve setup environmental variables to access Object Storage like below.

export OS_USER_ID=<user-id>
export OS_PASSWORD=<password>
export OS_PROJECT_ID=<project-id>
export OS_AUTH_URL='https://identity.open.softlayer.com/v3'
export OS_REGION_NAME=dallas
export OS_IDENTITY_API_VERSION=3
export OS_AUTH_VERSION=3

The values like <user-id>, <password> and <project-id> can be obtained from Bluemix Object Storage service UI.  Then, I can run several swift commands. For example, I can see the container list from “swift list” command just like shown in Bluemix Object Storage UI.

$ swift list
notebooks
test

For a simple test, I tried to upload a simple text file “hello.txt”.

$ echo  "Hello" > hello.txt
$ cat hello.txt
Hello
$ swift upload test hello.txt
hello.txt
$ swift list test
hello.txt

And then I removed the “hello.txt” file from my local machine, and then downloaded from Object Storage to see what happens.

$ rm hello.txt
$ cat hello.txt
cat: hello.txt: No such file or directory

$ swift download test hello.txt
hello.txt [auth 1.016s, headers 2.028s, total 2.028s, 0.000 MB/s]
$ cat hello.txt
Hello

I was able to recover the lost file.  It looks like it’s working fine 🙂

Version control files

My intention to use Object Storage is not only save relevant files in a safe way, but also want to version control them. Of course I use software configuration management tool like Rational Team Concert or Git for open source related, but I do not want to commit each files every time when I just want to test a slight modification.

OpenStack Object Storage provides a way to keep previous files when the same file is uploaded. By default, upload overrides objects. To enable a container to save versions, put meta-data called “X-Versions-Location” and set the name of container. I need to create “versions” container in advance, and set the values to “X-Versions-Location”.

$ swift post versions
$ swift post --header "X-Versions-Location: versions" test

Let’s see how it works. I create a next version of hello.txt file. Then upload to “test” container.

$ echo Version1 >> hello.txt
$ cat hello.txt
Hello
Version1
$ swift upload test hello.txt
hello.txt

And then I modify the file slightly and upload the file again.

$ vi hello.txt
$ cat hello.txt
Hello
Version2
$ swift upload test hello.txt
hello.txt

Now let’s see how “versions” container looks like. I run “swift list” command against “versions” container, and obtain the file contents with “swift download” command.

$ swift list versions
009hello.txt/1454395632.91154
009hello.txt/1454399672.55827

$ swift download versions 009hello.txt/1454395632.91154 -o -
Hello

$ swift download versions 009hello.txt/1454399672.55827 -o -
Hello
Version1

The option “-o -” tell to output into “stdout” instead of normal file. Now I see the original and version1 document very easily.

Creating a temporary url to share a file with your team member

Sometimes it is good idea to share a file (especially large one) temporary with your team member. Bluemix Object Storage service also offer to create temporary URL to download a file from a browser just like the other object storage.

First of all, we need to obtain the AUTH_account and set a key to swift account.

$ swift stat | grep " Account:" | awk '{print $2}'
$ swift post -m Temp-URL-Key:<key>

The first command obtains AUTH_account, and the second one set the <key>. Note that <key> is arbitrary string (e.g. result of command “$ openssl rand -base64 6 | md5sum” ). After this setup, temporary url can be create via “swift tempurl” command. For example if we want to share hello.txt file on test container for 1 hour, run like:

$ swift tempurl GET 3600 /v1/<AUTH_account>/test/hello.txt <key>

Append this url to https://dal.objectstorage.open.softlayer.com. The overall url will look like this:

https://dal.objectstorage.open.softlayer.com/v1/<AUTH_account>/test/hello.txt?temp_url_sig=5922b0ed1ca5a43fe9f12167be71f65a867cee6a&temp_url_expires=1454469261

Now you can pass this url to your team mate. This is probably useful to share a large file in your company.

Summary

This article tried to share how to use “swift” command in a virtual environment so that it does not break your current development environment. I also wanted to share minimal usage of Object Storage. Hope this is useful in your daily development activities !

 

 

Advertisements
This entry was posted in Bluemix. Bookmark the permalink.

8 Responses to Easy usage of Bluemix Object Storage service

  1. Natasha Dsouza says:

    Excellent e. Thank you very much. But I had a query – the export commands do not seem to work. Any particular reason for the same?

    • ibmamnt says:

      The “export” command is a shell command to setup environmental variable. Normally, used in bash shell. How to set environmental varialbles will depends on the machine. For example, on Windows “set” command is used. Please refer to your OS manual how to set environmental variables.

  2. Thanks for this..One question Iam able to upload only one file at a time but I want to upload all the files at a time to container.How can we do it?
    I tried >swift upload –all

  3. It worked using swift upload containername .

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s