Accessing a webservice using libsoup and liboauth

OAuth 1.0 (RFC 5849) is a common method used by web services for authorization. If you are working on a GLib based project there are two libraries that can help with this - libosoup and liboauth. The bad news for liboauth is it's not being developed anymore. The good news is OAuth 2.0 (RFC 6749) is replacing OAuth 1.0 so I guess that wont matter into the future.

There are a number of ways OAuth can be used, in this example I'll show how to obtain Ubuntu One credentials and use these to post a review for an Ubuntu application. This example will use JSON so refer to my previous tutorial on how to do this with JSON-GLib.

The first step is to get the credentials from Ubuntu One. After reading the API documentation this is fairly easy to acomplish.  For my application, I need to create a named token (I'll use "test-oauth"). To do this, I send my email address and password in JSON form to the server:

// Create JSON request
JsonBuilder *builder = json_builder_new ();

json_builder_begin_object (builder);
json_builder_set_member_name (builder, "email"); 
json_builder_add_string_value (builder, "test@example.com");
json_builder_set_member_name (builder, "password");
json_builder_add_string_value (builder, "secret");
json_builder_set_member_name (builder, "token_name");
json_builder_add_string_value (builder, "test-oauth"); 
json_builder_end_object (builder);

// Convert request into a string
JsonGenerator *generator = json_generator_new ();
json_generator_set_root (generator, json_builder_get_root (builder));
gsize length;
gchar *data = json_generator_to_data (generator, &length);

// Send request to the server
SoupSession *session = soup_session_new_with_options (SOUP_SESSION_USER_AGENT, "test-oauth", NULL);
SoupMessage *message = soup_message_new (SOUP_METHOD_POST, "https://login.ubuntu.com/api/v2/tokens/oauth");
soup_message_headers_append (message->request_headers, "Accept", "application/json");
soup_message_set_request (message, "application/json", SOUP_MEMORY_TAKE, data, length);
guint status_code = soup_session_send_message (session, message);
g_assert (status_code == SOUP_STATUS_CREATED);

If everything is correct, then the server will send back OAuth information for this token like this:

{
  "token_key": "vt81TzXi2DdNkS9WRa27FZOt4SMhrn4XlPI4YgyplS6TbxoCqnFZtVAWEDfn",
  "token_secret": "r232LlyogYQIN72AinyjGg32yKcEDP0w73Vwau0EpOzjC3XhQD9ID1X99OtaZcZk2yNUrG",
  "consumer_key": "K6Jl5et",
  "consumer_secret": "U23tFbCv9HCZJvvVZiBDBxZ2qcidOx"
}

The consumer_key is basically your user ID (named weirdly for historical reasons). token_key is what you use to mark requests with the "test-oauth" token. consumer_secret and token_secret are encryption keys you use to prove that consumer_key and token_key are yours. These must be kept secret (the hint is in the name)!

Now I have my OAuth token I can use it to review an Ubuntu package. I should also save this information so I can re-use it without returning to login.ubuntu.com.

I'll make a message to send to the reviews server:

// Create JSON request
JsonBuilder *builder = json_builder_new (); 
json_builder_begin_object (builder);
json_builder_set_member_name (builder, "package_name");
json_builder_add_string_value (builder, "no-such-package");
json_builder_set_member_name (builder, "rating");
json_builder_add_int_value (builder, 5);
// Add other required fields...
json_builder_end_object (builder);

// Convert request into a string
JsonGenerator *generator = json_generator_new ();
json_generator_set_root (generator, json_builder_get_root (builder));
gsize length;
gchar *data = json_generator_to_data (generator, &length);

// Prepare request for the server
SoupSession *session = soup_session_new_with_options (SOUP_SESSION_USER_AGENT, "test-oauth", NULL);
SoupMessage *message = soup_message_new (SOUP_METHOD_POST, "https://reviews.ubuntu.com/reviews/api/1.0/reviews/");
soup_message_headers_append (message->request_headers, "Accept", "application/json");
soup_message_set_request (message, "application/json", SOUP_MEMORY_TAKE, data, length);

If I send the message like this, the server will reject it because there's no valid HTTP Authorization header. To populate this I'll use liboauth.

The liboauth API is a bit confusing, so I'll run through the functions. Firstly the URL needs splitting into the parts that are needed for signing:

char **url_parameters = NULL;
int url_parameters_length = oauth_split_url_parameters ("https://reviews.ubuntu.com/reviews/api/1.0/reviews/", &url_parameters);

Next step is to generate the Oauth fields and the signature. These new fields are written back into url_parameters. Note the NULL argument - this is required if the HTTP request has content in the form application/x-www-form-urlencoded. In my case it's application/json so I don't have to bother.

oauth_sign_array2_process (&url_parameters_length, &url_parameters,
                           NULL, // No postargs
                           O_HMAC,
                           "POST",
                           oauth_consumer_key, oauth_consumer_secret,
                           oauth_token, oauth_token_secret);

Finally all the OAuth fields in url_parameters need to be combined into a valid OAuth field. The 6 is a bitfield (no defines in the liboauth API)  that specifies how to correctly choose and format the required fields.

char *p = oauth_serialize_url_sep (url_parameters_length, 1, url_parameters, ", ", 6);
gchar *authorization_text = g_strdup_printf ("OAuth realm=\"Ratings and Reviews\", %s", p);

And now authorization_text contains a valid Authorization HTTP header! I can now add this to the SoupMessage and send it to the server:

soup_message_headers_append (message->request_headers, "Authorization", authorization_text);
guint status_code = soup_session_send_message (session, message);
g_assert (status_code == SOUP_STATUS_OK);

And that's all it takes to send an OAuth authenticated request.

The full program:

// gcc -g -Wall example-oauth.c -o example-oauth `pkg-config --cflags --libs libsoup-2.4 oauth json-glib-1.0`

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <libsoup/soup.h>
#include <oauth.h>
#include <json-glib/json-glib.h>

static void
add_string_member (JsonBuilder *builder, const gchar *name, const gchar *value)
{
    json_builder_set_member_name (builder, name);
    json_builder_add_string_value (builder, value);
}

static void
add_int_member (JsonBuilder *builder, const gchar *name, gint64 value)
{
    json_builder_set_member_name (builder, name);
    json_builder_add_int_value (builder, value);
}

static void
set_request (SoupMessage *message, JsonBuilder *builder)
{
    JsonGenerator *generator = json_generator_new ();
    json_generator_set_root (generator, json_builder_get_root (builder));
    gsize length;
    gchar *data = json_generator_to_data (generator, &length);
    soup_message_set_request (message, "application/json", SOUP_MEMORY_TAKE, data, length);
    g_object_unref (generator);
}

static const char *
getfield (const char *prompt)
{
    static char value[1024];
    printf ("%s", prompt);
    fgets (value, 1024, stdin);
    g_strchomp (value);
    return value;
}

static void
sign_message (SoupMessage *message, const gchar *realm, OAuthMethod method,
              const gchar *oauth_consumer_key, const gchar *oauth_consumer_secret,
              const gchar *oauth_token, const gchar *oauth_token_secret)
{
    gchar *url, *oauth_authorization_parameters;
    gchar **url_parameters = NULL;
    int url_parameters_length;
    gchar *authorization_text;

    url = soup_uri_to_string (soup_message_get_uri (message), FALSE);

    url_parameters_length = oauth_split_url_parameters (url, &url_parameters);
    oauth_sign_array2_process (&url_parameters_length, &url_parameters,
                               NULL, // No postargs
                               method,
                               message->method,
                               oauth_consumer_key, oauth_consumer_secret,
                               oauth_token, oauth_token_secret);
    oauth_authorization_parameters = oauth_serialize_url_sep (url_parameters_length, 1, url_parameters, ", ", 6);
    authorization_text = g_strdup_printf ("OAuth realm=\"%s\", %s", realm, oauth_authorization_parameters);
    soup_message_headers_append (message->request_headers, "Authorization", authorization_text);

    g_free (url);
    oauth_free_array (&url_parameters_length, &url_parameters);
    free (oauth_authorization_parameters);
}

int main (int argc, char **argv)
{
    const gchar *token_name = "test-oauth";

    SoupSession *session = soup_session_new_with_options (SOUP_SESSION_USER_AGENT, "test-oauth", NULL);

    /* Load the OAuth key, token and secrets */
    GKeyFile *config = g_key_file_new ();
    gchar *path = g_build_filename (g_get_user_config_dir (), "oauth-test.conf", NULL);
    g_key_file_load_from_file (config, path, G_KEY_FILE_NONE, NULL);
    gchar *oauth_consumer_key = g_key_file_get_string (config, token_name, "consumer-key", NULL);
    gchar *oauth_consumer_secret = g_key_file_get_string (config, token_name, "consumer-secret", NULL);
    gchar *oauth_token = g_key_file_get_string (config, token_name, "token", NULL);
    gchar *oauth_token_secret = g_key_file_get_string (config, token_name, "token-secret", NULL);
    if (!oauth_consumer_key || !oauth_consumer_secret || !oauth_token || !oauth_token_secret)
    {
        /* Request a token from the Canonical Identity Provider */
        SoupMessage *message = soup_message_new (SOUP_METHOD_POST, "https://login.ubuntu.com/api/v2/tokens/oauth");
        soup_message_headers_append (message->request_headers, "Accept", "application/json");
        JsonBuilder *builder = json_builder_new ();
        json_builder_begin_object (builder);
        add_string_member (builder, "email", getfield ("Email address: "));
        add_string_member (builder, "password", getpass ("Password: "));
        add_string_member (builder, "otp", getfield ("Verification code: "));
        add_string_member (builder, "token_name", token_name);
        json_builder_end_object (builder);
        set_request (message, builder);
        g_object_unref (builder);
        guint status_code = soup_session_send_message (session, message);
        g_assert (status_code == SOUP_STATUS_CREATED);

        /* Parse response */
        JsonParser *parser = json_parser_new ();
        gboolean result = json_parser_load_from_data (parser, message->response_body->data, -1, NULL);
        g_assert (result);
        JsonNode *root = json_parser_get_root (parser);
        g_assert (JSON_NODE_HOLDS_OBJECT (root));
        JsonObject *object = json_node_get_object (root);
        oauth_consumer_key = g_strdup (json_object_get_string_member (object, "consumer_key"));
        oauth_consumer_secret = g_strdup (json_object_get_string_member (object, "consumer_secret"));     
        oauth_token = g_strdup (json_object_get_string_member (object, "token_key"));
        oauth_token_secret = g_strdup (json_object_get_string_member (object, "token_secret"));

        /* Write config so we don't do this the second time */
        g_key_file_set_string (config, token_name, "consumer-key", oauth_consumer_key);
        g_key_file_set_string (config, token_name, "consumer-secret", oauth_consumer_secret);
        g_key_file_set_string (config, token_name, "token", oauth_token);
        g_key_file_set_string (config, token_name, "token-secret", oauth_token_secret);
        g_key_file_save_to_file (config, path, NULL);

        g_object_unref (message);
        g_object_unref (parser);
    }
    g_free (path);

    /* Make the request using a HTTP POST signed with OAuth */
    SoupMessage *message = soup_message_new (SOUP_METHOD_POST, "https://reviews.ubuntu.com/reviews/api/1.0/reviews/");
    g_assert (message != NULL);
    soup_message_headers_append (message->request_headers, "Accept", "application/json");
    JsonBuilder *builder = json_builder_new ();
    json_builder_begin_object (builder);
    add_string_member (builder, "package_name", "no-such-package");
    add_string_member (builder, "summary", "★★★★★");
    add_string_member (builder, "review_text", "This is the best non existant application ever!");
    add_string_member (builder, "language", "en");
    add_string_member (builder, "origin", "ubuntu");
    add_string_member (builder, "distroseries", "xenial");
    add_string_member (builder, "version", "42");
    add_int_member (builder, "rating", 5);
    add_string_member (builder, "arch_tag", "amd64");
    json_builder_end_object (builder);
    set_request (message, builder);
    g_object_unref (builder);
    sign_message (message, "Ratings and Reviews", OA_HMAC, oauth_consumer_key, oauth_consumer_secret, oauth_token, oauth_token_secret);
    guint status_code = soup_session_send_message (session, message);
    g_printerr ("%d '%s'\n", status_code, message->response_body->data);
    g_assert (status_code == SOUP_STATUS_OK);

    /* Parse the data in JSON format */
    JsonParser *parser = json_parser_new ();
    gboolean result = json_parser_load_from_data (parser, message->response_body->data, -1, NULL);
    g_assert (result);
    JsonNode *root = json_parser_get_root (parser);
    g_assert (JSON_NODE_HOLDS_OBJECT (root));
    JsonObject *object = json_node_get_object (root);
    gint64 review_id = json_object_get_int_member (object, "id");
    g_print ("Review created with ID %" G_GUINT64_FORMAT "\n", review_id);

    /* Clean up */
    g_object_unref (session);
    g_object_unref (message);
    g_object_unref (parser);
    g_free (oauth_consumer_key);
    g_free (oauth_consumer_secret);
    g_free (oauth_token);
    g_free (oauth_token_secret);
 
    return 0;
}

Accessing a JSON webservice using libsoup and JSON-GLib

A lot of web services use the JSON format. If you are working a GLib based project and need to access a service like this there are two great libraries to help you - libsoup and JSON-Glib.

For my example, I'm going to grab some review data from Ubuntu (API) which looks something like this:

[
    {
        "ratings_total": 229,
        "ratings_average": "3.84",
        "app_name": "",
        "package_name": "simple-scan",
        "histogram": "[35, 13, 22, 42, 117]"
    },
    {
        "ratings_total": 546,
        "ratings_average": "4.66",
        "app_name": "",
        "package_name": "audacity",
        "histogram": "[17, 7, 17, 63, 442]"
    },
    ...
]

The data is a single array of objects that contain the statistics for each package. For this example I'll print out the number of ratings for each package by getting the package_name and ratings_total members from each object.

Firstly, I need to download the data. The data is retrieved using a HTTP GET request; in libsoup you can do this with:

    SoupSession *session = soup_session_new_with_options (SOUP_SESSION_USER_AGENT, "test-json", NULL);
    SoupMessage *message = soup_message_new (SOUP_METHOD_GET, "https://reviews.ubuntu.com/reviews/api/1.0/review-stats/any/any");
    soup_session_send_message (session, message);

Now I have the server text in message->response_body->data but it needs to be decoded. JSON-GLib can parse it with:

    JsonParser *parser = json_parser_new ();
    json_parser_load_from_data (parser, message->response_body->data, -1, NULL);
    JsonNode *root = json_parser_get_root (parser);

Now I have an in-memory tree of the JSON data which can be traversed. After checking the root node is an array as expected I'll iterate over each object:

    g_assert (JSON_NODE_HOLDS_ARRAY (root));
    array = json_node_get_array (root);
    for (i = 0; i < json_array_get_length (array); i++)
    {
        JsonNode *node = json_array_get_element (array, i);
        /* do stuff... */
    }

For each object, I extract the required data:

        g_assert (JSON_NODE_HOLDS_OBJECT (node));
        JsonObject *object = json_node_get_object (node);
        const gchar *package_name = json_object_get_string_member (object, "package_name");
        gint64 ratings_total = json_object_get_int_member (object, "ratings_total");
        if (package_name)
            g_print ("%s: %" G_GUINT64_FORMAT "\n", package_name,
 
Combined into a program, I can print out the number of reviews for each package:

simple-scan: 229
audacity: 546
...

The full program:

// gcc -g -Wall example-json.c -o example-json `pkg-config --cflags --libs libsoup-2.4 json-glib-1.0`

#include
#include

int main (int argc, char **argv)
{
    SoupSession *session;
    SoupMessage *message;
    guint status_code;
    JsonParser *parser;
    gboolean result;
    JsonNode *root;
    JsonArray *array;
    gint i;

    /* Get the data using a HTTP GET */
    session = soup_session_new_with_options (SOUP_SESSION_USER_AGENT, "test-json", NULL);
    message = soup_message_new (SOUP_METHOD_GET, "https://reviews.ubuntu.com/reviews/api/1.0/review-stats/any/any");
    g_assert (message != NULL);
    status_code = soup_session_send_message (session, message);
    g_assert (status_code == SOUP_STATUS_OK);

    /* Parse the data in JSON format */
    parser = json_parser_new ();
    result = json_parser_load_from_data (parser, message->response_body->data, -1, NULL);
    g_assert (result);

    /* The data should contain an array of JSON objects */
    root = json_parser_get_root (parser);
    g_assert (JSON_NODE_HOLDS_ARRAY (root));
    array = json_node_get_array (root);
    for (i = 0; i < json_array_get_length (array); i++)
    {
        JsonNode *node;
        JsonObject *object;
        const gchar *package_name;
        gint64 ratings_total;

        /* Get the nth object, skipping unexpected elements */
        node = json_array_get_element (array, i);
        if (!JSON_NODE_HOLDS_OBJECT (node))
            continue;

        /* Get the package name and number of ratings from the object - skip if has no name */
        object = json_node_get_object (node);
        package_name = json_object_get_string_member (object, "package_name");
        ratings_total = json_object_get_int_member (object, "ratings_total");
        if (package_name)
            g_print ("%s: %" G_GINT64_FORMAT "\n", package_name, ratings_total);
    }

    /* Clean up */
    g_object_unref (session); 
    g_object_unref (message);
    g_object_unref (parser);

    return 0;
}

And to show you can do the same thing with GIR bindings, here's the same in Vala:

// valac example-json.vala --pkg soup-2.4 --pkg json-glib-1.0

public int main (string[] args)
{
    /* Get the data using a HTTP GET */
    var session = new Soup.Session.with_options (Soup.SESSION_USER_AGENT, "test-json");
    var message = new Soup.Message ("GET", "https://reviews.ubuntu.com/reviews/api/1.0/review-stats/any/any");
    assert (message != null);
    var status_code = session.send_message (message);
    assert (status_code == Soup.Status.OK);

    /* Parse the data in JSON format */
    var parser = new Json.Parser ();
    try
    {
        parser.load_from_data ((string) message.response_body.data);
    }
    catch (Error e)
    {
    }

    /* The data should contain an array of JSON objects */
    var root = parser.get_root ();
    assert (root.get_node_type () == Json.NodeType.ARRAY);
    var array = root.get_array ();
    for (var i = 0; i
    {
        /* Get the nth object, skipping unexpected elements */
        var node = array.get_element (i);
        if (node.get_node_type () != Json.NodeType.OBJECT)
            continue;

        /* Get the package name and number of ratings from the object - skip if has no name */
        var object = node.get_object ();
        var package_name = object.get_string_member ("package_name");
        var ratings_total = object.get_int_member ("ratings_total");
        if (package_name != null)
            stdout.printf ("%s: %" + int64.FORMAT + "\n", package_name, ratings_total);
    }

    return 0;
}

and Python:

#!/usr/bin/python

from gi.repository import Soup
from gi.repository import Json

# Get the data using a HTTP GET
session = Soup.Session.new ()
session.set_property (Soup.SESSION_USER_AGENT, "test-json")
message = Soup.Message.new ("GET", "https://reviews.ubuntu.com/reviews/api/1.0/review-stats/any/any")
assert (message != None)
status_code = session.send_message (message)
assert (status_code == Soup.Status.OK)

# Parse the data in JSON format
parser = Json.Parser ()
parser.load_from_data (message.response_body.data, -1)

# The data should contain an array of JSON objects
root = parser.get_root ()
assert (root.get_node_type () == Json.NodeType.ARRAY)
array = root.get_array ()
for i in xrange (array.get_length ()):
    # Get the nth object, skipping unexpected elements
    node = array.get_element (i)
    if node.get_node_type () != Json.NodeType.OBJECT:
        continue

    # Get the package name and number of ratings from the object - skip if has no name
    object = node.get_object ()
    package_name = object.get_string_member ("package_name")
    ratings_total = object.get_int_member ("ratings_total")
    if package_name != None:
        print ("%s: %d" % (package_name, ratings_total))

I wrote some more apps for Ubuntu Phone

In a previous post I talked about the experiences of writing my first five applications for Ubuntu Phone. Since then, I've written three more.

As before, all these apps are GPL 3 licensed and available on Launchpad. What's new is now you can browse them online due to a great unofficial web appstore made by Brian Douglass. This solves one of my previous gripes about not being able to find new applications.

One thing I have changed is I've started to use the standard page header. This makes it easy to add more pages (e.g. help, high scores). I initially wasn't a fan of the header due to the amount of space it took up but for simple apps it works well and it makes them predictable to use. I also went back and updated Dotty to use this style.

Let me introduce...

Dice Roller (lp:dice-roller)

It's a simple utility to simulate dice rolling. Not much more to say. 431 lines of QML.

Morse Sender (lp:morse-sender)

I was playing with uTorch and watching a spy movie and thinking "hey, wouldn't it be cool to flash the camera and send morse code messages. I wonder if the phone responds fast enough to do that." Apparently it does. 584 lines of QML.

Yatzy (lp:yatzy)

So I thought, now I have this nice dice code from Dice Roller, it would be cool to make a full dice game with it. I've played a fair bit of Yahtzee in my time and searching Wikipedia I found there was a similar public domain game called Yatzy. 999 lines of QML.