2012-04-15: I found it easier to work with signed URLs instead of a
separate header so the blog post was updated to use these.
I like to find new ways to use Ubuntu One as easily as possible. I have to
admit that the way described in headless Ubuntu One article is not quite
reliable since the file is read completely into the memory prior to being
transferred. For large files this can be a problem. There are ways to make this
work for streaming, but while investigating this I found out that we can use
curl for uploads and downloads, without any python wrapper.
Basically,
curl -H "Content-Type: text/plain"
-T filename `oauth-sign -m PUT -f url http://.../~/Folder/filename`
But in order to make it work this way, you need just a tiny bit of preparation.
First of all, you must know that all Ubuntu One clients do not use your
password for authentication. Instead, a mechanism called OAuth is used. This means that
for every connected device a separate set of credentials is being assigned
allowing you to invalidate the access whenever you need and keep the really
secret password to yourself.
OAuth is not natively supported by curl but since this is a standard, there are
multiple solutions you can use to generate the signed URL as in the example,
there are no proprietary extensions or cookie storage.
If you have ruby, installing oauth gem will give you an oauth script. In case
you are after the python one, you can grab my implementation from
ubuntuone-scripts branch.
First you need to get the tokens, we are using pretty much the same
ubuntuone-sso-login.py script I mentioned in the article about the REST client,
however in order to make it simpler to pass the arguments to other application,
i added the --format (-f) option, which does the following:
$ python ubuntuone-sso-login.py --format shell
Ubuntu SSO Login: sso-test-2@rtg.in.ua
Password:
export OAUTH_CONSUMER_KEY=RwLrAWeWF
export OAUTH_CONSUMER_SECRET=fGztsAJHqfWFirGBwXNhiEinrfqCvDVtEHJ
export OAUTH_TOKEN=HFvMpWiwuOcGOsSSRbkaHZHMARjHZrfYYRa
export OAUTH_TOKEN_SECRET=BjnVKEhQqCaTcHWBEywVGMhlqvtTBzFFeykYC
Copy the lines with export to some shell script file, e.g.
~/.ubuntuone-credentials, we will re-use the names further.
Now create the signed URL with ruby implementation: We are requesting account
information, so the URL is https://one.ubuntu.com/api/account/.
$ SIGNED_URL=`oauth --consumer-key $OAUTH_CONSUMER_KEY
--consumer-secret $OAUTH_CONSUMER_SECRET
--method GET --uri https://one.ubuntu.com/api/account/
--secret $OAUTH_TOKEN_SECRET
--token $OAUTH_TOKEN -v -Q sign | grep 'OAuth Request URI:'
| sed 's/OAuth Request URI: //'`
The weird grep/sed dance is needed to get the URL to the format we will use
later, you will not need to do this in python oauth-sign version.
$ curl $SIGNED_URL
{"username": "https://login.ubuntu.com/+id/RwLWeWF", "openid":
"https://login.ubuntu.com/+id/RwLWeWF", "last_name": "Roman"
...
Great, it works. Now let’s work with the actual files.
According to the docs the file root is https://files.one.ubuntu.com/ and PUT is
being used to upload the files.
Please note the following paragraph in the docs -
Note also that content_paths may not be rooted under the API root, and may
change without warning, so they must be read from the API and not hardcoded.
The current way assumes that the content path is
/content/~/path/to/volume/path/to/file.txt, this works for now, but may change
in the future.
First let’s construct the URL we will be using
$ URL=https://files.one.ubuntu.com/content/~/Ubuntu%20One/hello.txt
Now we need to get the signed URL. Let’s use the python version. Since it reads the environment variables, there is no need to specify the token:
$ SIGNED_URL=`oauth-sign -f url -m PUT $URL`
Yes, that’s it.
Now create a hello.txt file locally and then upload it with curl,
specifying -T makes curl use “PUT” request and transfer the file without
buffering it in the memory:
$ echo "Hello, Ubuntu One" > hello.txt
$ curl -T hello.txt -H "Content-Type: text/plain" $SIGNED_URL
{"kind": "file", "public_url": null, "hash":
"sha1:b50a416c31dbf3620b72cd7841980541805db44e", ...
Great! Now let’s get the file:
$ curl `oauth-sign -f url $URL`
Hello, Ubuntu One
Please note that I am specifying Content-Type header too. Usually when you ask
curl to upload with --data-binary, it sets the Content-Type to
application/octet-stream. This is a mandatory header for Ubuntu One servers. In
case you omit the header, you will receive an error.
I have some SQL databases of text messages, expenses and a test openfire
installation. I upload the backup regularly to Ubuntu One with the following
simple script. I put the OAUTH exports to ~/.ubuntuone-credentials file which
gets sourced.
#!/bin/bash
URL=https://files.one.ubuntu.com/content/~/Ubuntu%20One
source ~/.ubuntuone/credentials
for DB in webapp openfire; do
RESULT=fridge-$DB.sql.gz
LOCALFILE=`mktemp /tmp/backup-XXXXXX`
pg_dump -h localhost -U backup $DB | gzip > $LOCALFILE
REMOTE=$URL/$RESULT
SIGNED_URL=`~/bin/oauth-sign -f url -m PUT $REMOTE`
HEADERS=`mktemp /tmp/headers-XXXXXX`
OUTPUT=`mktemp /tmp/output-XXXXXX`
STATUS=`curl --silent --fail --dump-header $HEADERS
-T "$LOCALFILE"
-H "Content-Type: application/octet-stream"
-w '%{http_code}' $SIGNED_URL -o $OUTPUT`
rv=$?
if [ "$rv" != "0" -o ( "$STATUS" != "201" -a "$STATUS" != "200" ) ]; then
echo "Backup of $DB failed!"
echo "The server replied with:"
echo
cat $HEADERS
echo
cat $OUTPUT
fi
rm $OUTPUT
rm $HEADERS
rm $LOCALFILE
done
