2012-04-15: I found it easier to work with signed URLs instead of a separate header so the blog post was updated to use these.

I like to find new ways to use Ubuntu One as easily as possible. I have to admit that the way described in headless Ubuntu One article is not quite reliable since the file is read completely into the memory prior to being transferred. For large files this can be a problem. There are ways to make this work for streaming, but while investigating this I found out that we can use curl for uploads and downloads, without any python wrapper.

Basically,

curl -H "Content-Type: text/plain"
    -T filename `oauth-sign -m PUT -f url http://.../~/Folder/filename`

But in order to make it work this way, you need just a tiny bit of preparation.

First of all, you must know that all Ubuntu One clients do not use your password for authentication. Instead, a mechanism called OAuth is used. This means that for every connected device a separate set of credentials is being assigned allowing you to invalidate the access whenever you need and keep the really secret password to yourself.

OAuth is not natively supported by curl but since this is a standard, there are multiple solutions you can use to generate the signed URL as in the example, there are no proprietary extensions or cookie storage.

If you have ruby, installing oauth gem will give you an oauth script. In case you are after the python one, you can grab my implementation from ubuntuone-scripts branch.

First you need to get the tokens, we are using pretty much the same ubuntuone-sso-login.py script I mentioned in the article about the REST client, however in order to make it simpler to pass the arguments to other application, i added the --format (-f) option, which does the following:

$ python ubuntuone-sso-login.py --format shell
Ubuntu SSO Login: sso-test-2@rtg.in.ua
Password:
export OAUTH_CONSUMER_KEY=RwLrAWeWF
export OAUTH_CONSUMER_SECRET=fGztsAJHqfWFirGBwXNhiEinrfqCvDVtEHJ
export OAUTH_TOKEN=HFvMpWiwuOcGOsSSRbkaHZHMARjHZrfYYRa
export OAUTH_TOKEN_SECRET=BjnVKEhQqCaTcHWBEywVGMhlqvtTBzFFeykYC

Copy the lines with export to some shell script file, e.g. ~/.ubuntuone-credentials, we will re-use the names further.

Now create the signed URL with ruby implementation: We are requesting account information, so the URL is https://one.ubuntu.com/api/account/.

$ SIGNED_URL=`oauth --consumer-key $OAUTH_CONSUMER_KEY
        --consumer-secret $OAUTH_CONSUMER_SECRET
        --method GET --uri https://one.ubuntu.com/api/account/
        --secret $OAUTH_TOKEN_SECRET
        --token $OAUTH_TOKEN -v -Q sign | grep 'OAuth Request URI:'
                                    | sed 's/OAuth Request URI: //'`

The weird grep/sed dance is needed to get the URL to the format we will use later, you will not need to do this in python oauth-sign version.

$ curl $SIGNED_URL
{"username": "https://login.ubuntu.com/+id/RwLWeWF", "openid":
"https://login.ubuntu.com/+id/RwLWeWF", "last_name": "Roman"
...

Great, it works. Now let’s work with the actual files.

According to the docs the file root is https://files.one.ubuntu.com/ and PUT is being used to upload the files.

Please note the following paragraph in the docs -

Note also that content_paths may not be rooted under the API root, and may change without warning, so they must be read from the API and not hardcoded.

The current way assumes that the content path is /content/~/path/to/volume/path/to/file.txt, this works for now, but may change in the future.

First let’s construct the URL we will be using

$ URL=https://files.one.ubuntu.com/content/~/Ubuntu%20One/hello.txt

Now we need to get the signed URL. Let’s use the python version. Since it reads the environment variables, there is no need to specify the token:

$ SIGNED_URL=`oauth-sign -f url -m PUT $URL`

Yes, that’s it.

Now create a hello.txt file locally and then upload it with curl, specifying -T makes curl use “PUT” request and transfer the file without buffering it in the memory:

$ echo "Hello, Ubuntu One" > hello.txt
$ curl -T hello.txt -H "Content-Type: text/plain" $SIGNED_URL

{"kind": "file", "public_url": null, "hash":
"sha1:b50a416c31dbf3620b72cd7841980541805db44e", ...

Great! Now let’s get the file:

$ curl `oauth-sign -f url $URL`
Hello, Ubuntu One

Please note that I am specifying Content-Type header too. Usually when you ask curl to upload with --data-binary, it sets the Content-Type to application/octet-stream. This is a mandatory header for Ubuntu One servers. In case you omit the header, you will receive an error.

I have some SQL databases of text messages, expenses and a test openfire installation. I upload the backup regularly to Ubuntu One with the following simple script. I put the OAUTH exports to ~/.ubuntuone-credentials file which gets sourced.

#!/bin/bash

URL=https://files.one.ubuntu.com/content/~/Ubuntu%20One

source ~/.ubuntuone/credentials

for DB in webapp openfire; do
    RESULT=fridge-$DB.sql.gz

    LOCALFILE=`mktemp /tmp/backup-XXXXXX`

    pg_dump -h localhost -U backup $DB | gzip > $LOCALFILE

    REMOTE=$URL/$RESULT

    SIGNED_URL=`~/bin/oauth-sign -f url -m PUT $REMOTE`

    HEADERS=`mktemp /tmp/headers-XXXXXX`
    OUTPUT=`mktemp /tmp/output-XXXXXX`

    STATUS=`curl --silent --fail --dump-header $HEADERS
        -T "$LOCALFILE"
        -H "Content-Type: application/octet-stream"
        -w '%{http_code}' $SIGNED_URL -o $OUTPUT`
    rv=$?
    if [ "$rv" != "0" -o ( "$STATUS" != "201" -a "$STATUS" != "200" ) ]; then
        echo "Backup of $DB failed!"
        echo "The server replied with:"
        echo
        cat $HEADERS
        echo
        cat $OUTPUT
    fi

    rm $OUTPUT
    rm $HEADERS
    rm $LOCALFILE

done

while not keypressed

Upload to Ubuntu One using curl