Interesting things about the GIF image format

I recently took a deep dive into the GIF format. In the process I learnt a few things by reading the specification.

A GIF is made up of multiple images

 

I thought the GIF format would just contain a set of pixels. In fact, a GIF is made up of multiple images. So a simple example like:

 Could actually be made up of multiple images like this:

 

GIF has transparency, but that doesn't mean you have transparent GIFs

 

In the above example the sun and house images have the background in them. If the background was very detailed then this would be inefficient. So instead you can set a transparent colour index for each image. Pixels with this index don't replace the background pixels when the images are composited together.

That's the only transparency in the specification. The background colour is actually encoded in the file so technically a GIF picture has all pixels set to a colour. However at some point renderers decided they wanted transparency and ignored the background colour and set it to transparent instead. It's not in the spec, but it's what everyone does. This is the reason that GIF transparency looks bad - there's no alpha channel, just a hack abusing another feature.

You can have more than 256 colours

 

GIFs are well known for having a palette of only up to 256 colours. However, you can have a different palette for each image in the GIF. That means in the above example you could use a palette with lots of greens and blues for the background, lots of reds for the house and lots of yellows for the sun. The combined image could have up to 768 colours! With some clever encoding you can have a GIF file that uses up to 24 million colours.

Animation is just delaying the rendering 

GIFs are most commonly used for small animations. This wasn't in the original specification but at some point someone realised if you inserted a delay between each image you could make an animation! In the above example we could animate by adding more images of the sun that were rotated from the previous frame with a delay before them:

 

Why we can't have nice things

With all of the above GIF is both a simple but powerful format. You can make an animation that is made up of small updates efficiently encoded.

Sadly however someone decided that all images inside a GIF file should be treated as animation frames. And they should have a minimum delay time (including zero delays being rounded up to 20ms or so). So if you want you GIF to look as you intended you're stuck with one image per frame and only 256 colours per frame unless the common decoders are fixed. It seems the main reason they continue to be like this is there are badly encoded GIF files online and they don't want them to stop working.

GIF, you are a surprisingly beautiful format and it's a shame we don't see your full potential!

GIFs in GNOME

Here is the story of how I fell down a rabbit hole and ended up learning far more about the GIF image format than I ever expected...

We had a problem with users viewing a promoted snap using GNOME Software. When they opened the details page they'd have huge CPU and memory usage. Watching the GIF in Firefox didn't show a problem - it showed a fairly simple screencast demoing the app without any issues.

I had a look at the GIF file and determined:

• It was quite large for a GIF (13Mb).
• It had a lot of frames (625).
• It was quite high resolution (1790×1060 pixels).
• It appeared the GIF was generated from a compressed video stream, so most of the frame data was just compression artifacts. GIF is lossless so it was faithfully reproducing details you could barely notice.

GNOME Software uses GTK+, which uses gdk-pixbuf to render images. So I had a look a the GIF loading code. It turns out that all the frames are loaded into memory. That comes to 625×1790×1060×4 bytes. OK, that's about 4.4Gb... I think I see where the problem is. There's a nice comment in the gdk-pixbuf source that sums up the situation well:

 /* The below reflects the "use hell of a lot of RAM" philosophy of coding */

They weren't kidding. 🙂

While this particular example is hopefully not the normal case the GIF format has has somewhat come back from the dead in recent years to be a popular format. So it would be nice if gdk-pixbuf could handle these cases well. This was going to be a fairly major change to make.

The first step in refactoring is making sure you aren't going to break any existing behaviour when you make changes. To do this the code being refactored should have comprehensive tests around it to detect any breakages. There are a good number of GIF tests currently in gdk-pixbuf, but they are mostly around ensuring particular bugs don't regress rather than checking all cases.

I went looking for a GIF test suite that we could use, but what was out there was mostly just collections of GIFs people had made over the years. This would give some good real world examples but no certainty that all cases were covered or why you code was breaking if a test failed.

If you can't find what you want, you have to build it. So I wrote PyGIF - a library to generate and decode GIF files and made sure it had a full test suite. I was pleasantly surprised that GIF actually has a very well written specification, and so implementation was not too hard. Diversion done, it was time to get back to gdk-pixbuf.

Tests plugged in, and the existing code actually has a number of issues. I fixed them, but this took a lot of sanity to do so. It would have been easier to replace the code with new code that met the test suite, but I wanted the patches to be back-portable to stable releases (i.e. Ubuntu 16.04 and 18.04 LTS).

And with a better foundation, I could now make GIF frames load on demand. May your GIF viewing in GNOME continue to be awesome.

Counting Code in GNOME Settings

I've been spending a bit of time recently working on GNOME Settings. One part of this has been bringing some of the older panel code up to modern standards, one of which is making use of GtkBuilder templates.

I wondered if any of these changes would show in the stats, so I wrote a program to analyse each branch in the git repository and break down the code between C and GtkBuilder. The results were graphed in Google Sheets:

This is just the user accounts panel, which shows some of the reduction in C code and increase in GtkBuilder data:

Here's the breakdown of which panels make up the codebase:

I don't think this draws any major conclusions, but is still interesting to see. Of note:

• Some of the changes make in 3.28 did reduce the total amount of code! But it was quickly gobbled up by the new Thunderbolt panel.
• Network and Printers are the dominant panels - look at all that code!
• I ignored empty lines in the files in case differing coding styles would make some panels look bigger or smaller. It didn't seem to make a significant difference.
• You can see a reduction in C code looking at individual panels that have been updated, but overall it gets lost in the total amount of code.

I'll have another look in a few cycles when more changes have landed (I'm working on a new sound panel at the moment).

GUADEC 2018 Almería

I recently attended the recent GNOME Users and Developers European Conference (GUADEC) in Almería, Spain. This was my fifth GUADEC and as always I was able to attend thanks to my employer Canonical paying for me to be there. This year we had seven members of the Ubuntu desktop team present. Almería was a beautiful location for the conference and a good trade for the winter weather I left on the opposite side of the world in New Zealand.

This was the second GUADEC since the Ubuntu desktop switched back to shipping GNOME and it’s been great to be back. I was really impressed how positive and co-operative everyone was; the community seems to be in a really healthy shape. The icing on the cake is the anonymous million dollar donation the foundation has received which they announced will be used to hire some staff.

The first talk of the week was from my teammates Ken VanDine, Didier Roche and Marco Treviño who talked about how we’d done the transition from Unity to GNOME in Ubuntu desktop. I was successful in getting an open talk slot and did a short talk about the state of Snap integration into GNOME. I talked about the work I’d done making snapd-glib and the Snap plugin in GNOME Software. I also touched on some of the work James Henstridge has been working on making Snaps work with portals. It was quite fun to see James be a bit of a celebrity after a long period of not being at a GUADEC - he is the JH in JHBuild!

After the first three days of talks the remaining three days are set for Birds of a Feather sessions where we get together in groups around a particular topic and discuss and hack on that. I organised a session on settings which turned out to be surprisingly popular! It was great to see everyone that I work with online in-person and allowed us to better understand each other. In particular I caught up with Georges Stavracas who has been very patient in reviewing the many patches I have been working on in GNOME Control Center.

I hope to see everyone again next year!

Setting up Continuous Integration on gitlab.gnome.org

Simple Scan recently migrated to the new gitlab.gnome.org infrastructure. With modern infrastructure I now have the opportunity to enable Continuous Integration (CI), which is a fancy name for automatically building and testing your software when you make changes (and it can do more than that too).

I've used CI in many projects in the past, and it's a really handy tool. However, I've never had to set it up myself and when I've looked it's been non-trivial to do so. The great news is this is really easy to do in GitLab!

There's lots of good documentation on how to set it up, but to save you some time I'll show how I set it up for Simple Scan, which is a fairly typical GNOME application.

To configure CI you need to create a file called .gitlab-ci.yml in your git repository. I started with the following:

build-ubuntu:
  image: ubuntu:rolling
  before_script:
    - apt-get update
    - apt-get install -q -y --no-install-recommends meson valac gcc gettext itstool libgtk-3-dev libgusb-dev libcolord-dev libpackagekit-glib2-dev libwebp-dev libsane-dev
  script:
    - meson _build
    - ninja -C _build install

The first line is the name of the job - "build_ubuntu". This is going to define how we build Simple Scan on Ubuntu.

The "image" is the name of a Docker image to build with. You can see all the available images on Docker Hub. In my case I chose an official Ubuntu image and used the "rolling" link which uses the most recently released Ubuntu version.

The "before_script" defines how to set up the system before building. Here I just install the packages I need to build simple-scan.

Finally the "script" is what is run to build Simple Scan. This is just what you'd do from the command line.

And with that, every time a change is made to the git repository Simple Scan is built on Ubuntu and tells me if that succeeded or not! To make things more visible I added the following to the top of the README.md:

[](https://gitlab.gnome.org/GNOME/simple-scan/pipelines)

This gives the following image that shows the status of the build:

And because there's many more consumers of Simple Scan that just Ubuntu, I added the following to.gitlab-ci.yml:

build-fedora:
  image: fedora:latest
  before_script:
    - dnf install -y meson vala gettext itstool gtk3-devel libgusb-devel colord-devel PackageKit-glib-devel libwebp-devel sane-backends-devel
  script:
    - meson _build
    - ninja -C _build install

Now it builds on both Ubuntu and Fedora with every commit!

I hope this helps you getting started with CI and gitlab.gnome.org. Happy hacking.

Retiring my Ubuntu Phone after 1000 days

With some sadness I recently replaced my Ubuntu Phone with a Nexus 5. It lasted me just over 1000 days (almost three years) as my everyday phone, and I last wrote about it at the 500 mark.

Even though this is the end for me and Ubuntu Phone the hope of a true open source phone platform continues on:

• The Ubuntu Phone project lives on ubports.
• As I put my Ubuntu phone to rest the Purism Librem 5 project was funded with over $2 million!

I wish both these projects all the best.

My thoughts on my time with Ubuntu Phone:

• It worked!
• While the hardware (Meizu MX4) was reasonable hardware, it would have been nice to see it on something newer/faster and have gone through some more iterations on software performance.
• The apps I missed most were:
• An app for my bank and network provider (that I could use to quickly check balances).
• Communication apps (e.g. Facebook messenger, WhatsApp)
• Uber
• I used a reasonable amount of webapps, which mostly filled the gap where apps weren't available. I does appear that most companies put more effort into their mobile apps than mobile web.

Snappy Sprint - London June 2017

I recently attended a Snappy Sprint in London, UK. As well as the Canonical people attending (including me) with experience in the whole Snappy stack (Snapcraft, the Snap store, snapd, snapd-glib) we had great representation from the Elementary, Fedora, GNOME, MATE and KDE communities. My goal was to help improve the Snap experience for desktop apps both on Ubuntu and other distributions.

We spent a lot of time working on improving snap metadata for use with desktop apps. Improvements included:

• Exposing the title field from the store down to clients.
• A plan to get standard license information (using SPDX) attached to snaps.
• We made progress on a solution for projects that use AppStream to be able to easily build snaps and provide some AppStream data that doesn't fit the Snap metadata model to pass through to clients.
• Fixing of many small issues in GNOME Software so it is suitable to work in Fedora and other distributions.
• Plans for a tool that allows graphical configuration of snap interfaces.
• A plan to solve the limitation on desktop clients able to install / remove snaps without a store login.
• Discussions around metadata translations.

I helped the MATE Software Boutique and KDE Discover developers make use of snapd-glib using GIR bindings in Python and the Qt bindings to make their stores work. It was great to see snapd-glib working in these different use cases and got back some great feedback and a few patches.

Thanks to all the community for attending, I found it very productive to work in-person with them all. If you're interested in following Snappy development check out the Snapcraft Forum where you'll find discussions about what I've described above and much more.

Back to GNOME

With recent changes in Ubuntu I found myself suddenly swung back into the GNOME orbit. It’s been a long journey and my GNOME contributions had reduced over time but it’s good to be back! Iain has written an excellent post about the challenges we face trying to balance the best possible experience for Ubuntu users while also having a mutually beneficial relationship with our upstreams.

After missing the last five GUADECs, I will be at Manchester this year. I hope to catch up with as many people as possible including many old friends I haven’t seen in person for quite some time. If you have any questions about Ubuntu please find me or others from our team, we’re excited to collaborate.

I’ll be spending a lot of my time this development cycle working on GNOME Software particularly around snap support. We’re already delivered some good changes to upstream GNOME like reviews and paid application support. GNOME Software has changed from being a stop-gap solution in Ubuntu to being our permanent software management solution (and has been working really well for us).

It is with some sadness that I say goodbye to the Unity desktop. In particular some things I will miss:

• The performance and stability of Unity. After some early teething troubles Unity was rock solid and reliable.
• Fullscreen window management. Unity was super efficient at making use of screen space and reducing distraction. I hope we can get a similar solution into GNOME Shell .
• Convervenge. While we weren’t able to make a commercial success of it in Ubuntu I hope it will return in the future when the time is right.
• The development experience of Ubuntu phone and clicks. I hope we can get that experience (and better) soon as next generation packaging systems start to take over. Luckily the click packages I worked on for the phone are being taken over by others in the community (as is the whole Unity 8 project). So I wish these projects success in the future.
• Unfortunately we have decided it’s not possible to continue to use LightDM in the default Ubuntu install. This project has had wide support amongst many distributions and has a number of features that I will miss.

Introducing snapd-glib

World, meet snapd-glib. It's a new library that makes it easy for GLib based projects (e.g. software centres) to access the daemon that allows you to query, install and remove Snaps. If C is not for you, you can use all the functionality in Python (using GObject Introspection) and Vala. In the future it will support Qt/QML through a wrapper library.

snapd uses a REST API and snapd-glib very closely matches that. The behaviour is best understood by reading the documentation of that API. To give you a taste of how it works, here's an example that shows how to find and install the VLC snap.

Step 1: Connecting to snapd

The connection to snapd is controlled through the SnapdClient object. This object has all the methods required to communicate with snapd. Create and connect with:

    g_autoptr(SnapdClient) c = snapd_client_new ();
    if (!snapd_client_connect_sync (c, NULL, &error))
        // Something went wrong

Step 2: Find the VLC snap 

Asking snapd to perform a find causes it to contact the remote Snap store. This can take some time so consider using an asynchronous call for this. This is the synchronous version:

    g_autoptr(GPtrArray) snaps =
        snapd_client_find_sync (c,
                                SNAPD_FIND_FLAGS_NONE, "vlc",
                                NULL, NULL, &error);
    if (snaps == NULL)
        // Something went wrong
    for (int i = 0; i < snaps->len; i++) {
        SnapdSnap *snap = snaps->pdata[i];
        // Do something with this snap information
    }

Step 3: Authenticate 

Some methods require authorisation in the form of a Macaroon (the link is quite complex but in practise it's just a couple of strings). To get a Macaroon you need to provide credentials to snapd. In Ubuntu this is your Ubuntu account, but different snapd installations may use another authentication provider.

Convert credentials to authorization with:

    g_autoptr(SnapdAuthData) auth_data =
        snapd_login_sync (email, password, code,
                          NULL, &error);
    if (auth_data == NULL)
        return EXIT_FAILURE;
    snapd_client_set_auth_data (c, auth_data)

Once you have a Macaroon you can store it somewhere and re-use it next time you need it. Then the authorization can be created with:

    g_autoptr(SnapdAuthData) auth_data =
        snapd_auth_data_new (macaroon, discharges);
    snapd_client_set_auth_data (c, auth_data);

Step 4: Install VLC 

In step 2 we could determine the VLC snap has the name "vlc". Since this involves downloading ~100Mb and is going to take some time the asynchronous method is used. There is a callback that gives updates on the progress of the install and one that is called when the operation completes:

 
    snapd_client_install_async (c,
                                "vlc", NULL,
                                progress_cb, NULL,
                                NULL,
                                install_cb, NULL);

static void
progress_cb (SnapdClient *client,
             SnapdTask *main_task, GPtrArray *tasks,
             gpointer user_data)
{
    // Tell the user what's happening
}

static void
install_cb (GObject *object, GAsyncResult *result,
            gpointer user_data)
{
    g_autoptr(GError) error = NULL;

    if (snapd_client_install_finish (SNAPD_CLIENT (object),
                                     result, &error))
        // It installed!
    else
        // Something went wrong...
}

Conclusion 

With snapd-glib and the above code as a starting point you should be able to start integrating Snap support into your project. Have fun!

Developing for Ubuntu Phone without the SDK

I'm not a fan of IDEs. So when developing apps for Ubuntu Phone, I went looking for a way of doing everything from the command line. I'll describe the method I'm currently using to do this. It's pretty rough and there are probably better ways of doing this, but it does work!

For an example, I've make a small application that:

• Has some C++ code
• Has some QML code
• Uses gettext for translations

I need to do the following things:

• Extract translatable strings from source files and merge in translations
• Cross-compile for the target device (ARM CPU) from my laptop (AMD64 CPU)
• Create a click package
• Deploy the click package to my phone for testing

I'm using make for the build system. It's pretty basic, but it's fairly easy to understand what it does.

The project

My example project has the following files:

 

Makefile

hello.apparmor

hello.cpp

hello.desktop.in

hello.h

hello.png

main.qml

manifest.json

po/fr.po

po/de.po
po/hello.robert-ancell.pot
share/locale/de/LC_MESSAGES/

share/locale/fr/LC_MESSAGES/

If you've done some Ubuntu phone apps hopefully these should be familiar.

Translations 

To make my app translatable to other languages (French and German in this example) I've used gettext:

• The .desktop file needs translations added to it. This is done by using hello.dekstop.in and prefixing the fields that need translating with '_' (e.g. _Name). These are then combined with the translations to make hello.desktop.
• In the .qml files translatable strings are marked with i18n.tr ("text to translate").
• The compiled message catalogues (.mo files) need to be in the click package as share/locale/(language)/(appid).mo.

Gettext / intltool are somewhat scary to use, but here's the magic rules that work for me:

hello.desktop: hello.desktop.in po/*.po
    intltool-merge --desktop-style po $< $@

po/hello.robert-ancell.pot: main.qml hello.desktop.in
    xgettext --from-code=UTF-8 --language=JavaScript --keyword=tr --keyword=tr:1,2 --add-comments=TRANSLATORS main.qml -o po/hello.robert-ancell.pot
    intltool-extract --type=gettext/keys hello.desktop.in
    xgettext --keyword=N_ hello.desktop.in.h -j -o po/hello.robert-ancell.pot
    rm -f hello.desktop.in.h

share/locale/%/LC_MESSAGES/hello.robert-ancell.mo: po/%.po
    msgfmt -o $@ $<

Cross-compiling for the target device

To compile our package I need to make a chroot:

$ sudo click chroot -a armhf -f ubuntu-sdk-15.04 create

The following Makefile rule runs make inside this chroot, then packages the results into a click file:

click:
        click chroot -a armhf -f ubuntu-sdk-15.04 run ARCH_PREFIX=arm-linux-gnueabihf- make
        click build --ignore=Makefile --ignore=*.cpp --ignore=*.h --ignore=*.pot --ignore=*.po --ignore=*.in --ignore=po .

Note the ARCH_PREFIX variable in the above. I've used this to run the correct cross-compiler when inside the chroot. When compiling from my laptop this variable is not set so it uses the local compiler.

hello_moc.cpp: hello.h
    moc $< -o $@

hello: hello.cpp hello_moc.cpp
    $(ARCH_PREFIX)g++ -g -Wall -std=c++11 -fPIC $^ -o $@ `$(ARCH_PREFIX)pkg-config --cflags --libs Qt5Widgets Qt5Quick`

Running 'make click' in this project will spit out hello.robert-ancell_1_armhf.click.

Testing

I connect my phone with a USB cable and copy the click package over and install it locally:

$ adb push hello.robert-ancell_1_armhf.click /tmp/hello.robert-ancell_1_armhf.click

$ phablet-shell

$ pkcon install-local --allow-untrusted /tmp/hello.robert-ancell_1_armhf.click

Then on the phone I quit any instances of this app running, refresh the app scope (pull down at the top) and run my test version.

Summary

Just one more rule to add to the Makefile, then it all works:

all: hello \
     hello.desktop \
     po/hello.robert-ancell.pot \
     share/locale/de/LC_MESSAGES/hello.robert-ancell.mo \
     share/locale/fr/LC_MESSAGES/hello.robert-ancell.mo

The whole example is available in Launchpad:

$ bzr branch lp:~robert-ancell/+junk/hello-example

Happy hacking!

Five hundred days using Ubuntu Phone

Today is my five hundredth day of using the Meizu MX4 Ubuntu Edition exclusively as my mobile phone. This is a nice piece of hardware (good power, good camera and simple but elegant design).

Here's what I've learnt.

I have written a bunch of phone apps you can install and blogged it. Writing for the Ubuntu phone is by far the easiest platform I've developed for. Click packaging works really well and the speed at which you can release to the Ubuntu store and get the update on your phone is incredible. QML allows you to build beautiful apps quickly however can be a challenge when apps get more complicated. Qt / C++ is functional, but feels lacking compared to more modern languages. If I could get Swift and an improved QML working together I'd be very happy. I initially used the Ubuntu SDK for building and deplying the apps but have now switched to doing everything on the command line (I've never found an IDE that doesn't feel over-engineered).

There's more than enough apps in the store to keep me happy. In fact, I have installed far more apps than I ever did on my Android phone. I think that is because I really trust the Ubuntu store in a way I never did in Android (too much crap there).

I initially thought webapps wouldn't be useful but they're a good option when there's no native app. I use webapps for social networking and news sites and am pretty happy with that. They're definitely not as good as a native app but feel slightly more integrated than just visiting using your web browser.

Scopes. I can see there's something there in the concept but even though I've tried I've never found them useful. The only scope I have is the app scope (i.e. the traditional grid of applications). I'm hoping a few more iterations and they will find a place on my phone.

Love getting updates. Both system and app updates occur frequently and bring improvements. Unless you had a Nexus device you are more or less abandoned in the Android world - with Ubuntu the complete opposite.

The polish is not quite there compared to Android, but it's getting better quickly. There's little quirks / crashes that are annoying but nothing that stops me from using it all day. A couple more releases and the unforgiving mainstream will be able to thrash it too.

Here's to another five hundred days!

Accessing a webservice using libsoup and liboauth

OAuth 1.0 (RFC 5849) is a common method used by web services for authorization. If you are working on a GLib based project there are two libraries that can help with this - libosoup and liboauth. The bad news for liboauth is it's not being developed anymore. The good news is OAuth 2.0 (RFC 6749) is replacing OAuth 1.0 so I guess that wont matter into the future.

There are a number of ways OAuth can be used, in this example I'll show how to obtain Ubuntu One credentials and use these to post a review for an Ubuntu application. This example will use JSON so refer to my previous tutorial on how to do this with JSON-GLib.

The first step is to get the credentials from Ubuntu One. After reading the API documentation this is fairly easy to acomplish.  For my application, I need to create a named token (I'll use "test-oauth"). To do this, I send my email address and password in JSON form to the server:

// Create JSON request
JsonBuilder *builder = json_builder_new ();

json_builder_begin_object (builder);
json_builder_set_member_name (builder, "email"); 
json_builder_add_string_value (builder, "test@example.com");
json_builder_set_member_name (builder, "password");
json_builder_add_string_value (builder, "secret");
json_builder_set_member_name (builder, "token_name");
json_builder_add_string_value (builder, "test-oauth"); 
json_builder_end_object (builder);

// Convert request into a string
JsonGenerator *generator = json_generator_new ();
json_generator_set_root (generator, json_builder_get_root (builder));
gsize length;
gchar *data = json_generator_to_data (generator, &length);

// Send request to the server
SoupSession *session = soup_session_new_with_options (SOUP_SESSION_USER_AGENT, "test-oauth", NULL);
SoupMessage *message = soup_message_new (SOUP_METHOD_POST, "https://login.ubuntu.com/api/v2/tokens/oauth");
soup_message_headers_append (message->request_headers, "Accept", "application/json");
soup_message_set_request (message, "application/json", SOUP_MEMORY_TAKE, data, length);
guint status_code = soup_session_send_message (session, message);
g_assert (status_code == SOUP_STATUS_CREATED);

If everything is correct, then the server will send back OAuth information for this token like this:

{
  "token_key": "vt81TzXi2DdNkS9WRa27FZOt4SMhrn4XlPI4YgyplS6TbxoCqnFZtVAWEDfn",
  "token_secret": "r232LlyogYQIN72AinyjGg32yKcEDP0w73Vwau0EpOzjC3XhQD9ID1X99OtaZcZk2yNUrG",
  "consumer_key": "K6Jl5et",
  "consumer_secret": "U23tFbCv9HCZJvvVZiBDBxZ2qcidOx"
}

The consumer_key is basically your user ID (named weirdly for historical reasons). token_key is what you use to mark requests with the "test-oauth" token. consumer_secret and token_secret are encryption keys you use to prove that consumer_key and token_key are yours. These must be kept secret (the hint is in the name)!

Now I have my OAuth token I can use it to review an Ubuntu package. I should also save this information so I can re-use it without returning to login.ubuntu.com.

I'll make a message to send to the reviews server:

// Create JSON request
JsonBuilder *builder = json_builder_new (); 
json_builder_begin_object (builder);
json_builder_set_member_name (builder, "package_name");
json_builder_add_string_value (builder, "no-such-package");
json_builder_set_member_name (builder, "rating");
json_builder_add_int_value (builder, 5);
// Add other required fields...
json_builder_end_object (builder);

// Convert request into a string
JsonGenerator *generator = json_generator_new ();
json_generator_set_root (generator, json_builder_get_root (builder));
gsize length;
gchar *data = json_generator_to_data (generator, &length);

// Prepare request for the server
SoupSession *session = soup_session_new_with_options (SOUP_SESSION_USER_AGENT, "test-oauth", NULL);
SoupMessage *message = soup_message_new (SOUP_METHOD_POST, "https://reviews.ubuntu.com/reviews/api/1.0/reviews/");
soup_message_headers_append (message->request_headers, "Accept", "application/json");
soup_message_set_request (message, "application/json", SOUP_MEMORY_TAKE, data, length);

If I send the message like this, the server will reject it because there's no valid HTTP Authorization header. To populate this I'll use liboauth.

The liboauth API is a bit confusing, so I'll run through the functions. Firstly the URL needs splitting into the parts that are needed for signing:

char **url_parameters = NULL;
int url_parameters_length = oauth_split_url_parameters ("https://reviews.ubuntu.com/reviews/api/1.0/reviews/", &url_parameters);

Next step is to generate the Oauth fields and the signature. These new fields are written back into url_parameters. Note the NULL argument - this is required if the HTTP request has content in the form application/x-www-form-urlencoded. In my case it's application/json so I don't have to bother.

oauth_sign_array2_process (&url_parameters_length, &url_parameters,
                           NULL, // No postargs
                           O_HMAC,
                           "POST",
                           oauth_consumer_key, oauth_consumer_secret,
                           oauth_token, oauth_token_secret);

Finally all the OAuth fields in url_parameters need to be combined into a valid OAuth field. The 6 is a bitfield (no defines in the liboauth API)  that specifies how to correctly choose and format the required fields.

char *p = oauth_serialize_url_sep (url_parameters_length, 1, url_parameters, ", ", 6);
gchar *authorization_text = g_strdup_printf ("OAuth realm=\"Ratings and Reviews\", %s", p);

And now authorization_text contains a valid Authorization HTTP header! I can now add this to the SoupMessage and send it to the server:

soup_message_headers_append (message->request_headers, "Authorization", authorization_text);
guint status_code = soup_session_send_message (session, message);
g_assert (status_code == SOUP_STATUS_OK);

And that's all it takes to send an OAuth authenticated request.

The full program:

// gcc -g -Wall example-oauth.c -o example-oauth `pkg-config --cflags --libs libsoup-2.4 oauth json-glib-1.0`

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <libsoup/soup.h>
#include <oauth.h>
#include <json-glib/json-glib.h>

static void
add_string_member (JsonBuilder *builder, const gchar *name, const gchar *value)
{
    json_builder_set_member_name (builder, name);
    json_builder_add_string_value (builder, value);
}

static void
add_int_member (JsonBuilder *builder, const gchar *name, gint64 value)
{
    json_builder_set_member_name (builder, name);
    json_builder_add_int_value (builder, value);
}

static void
set_request (SoupMessage *message, JsonBuilder *builder)
{
    JsonGenerator *generator = json_generator_new ();
    json_generator_set_root (generator, json_builder_get_root (builder));
    gsize length;
    gchar *data = json_generator_to_data (generator, &length);
    soup_message_set_request (message, "application/json", SOUP_MEMORY_TAKE, data, length);
    g_object_unref (generator);
}

static const char *
getfield (const char *prompt)
{
    static char value[1024];
    printf ("%s", prompt);
    fgets (value, 1024, stdin);
    g_strchomp (value);
    return value;
}

static void
sign_message (SoupMessage *message, const gchar *realm, OAuthMethod method,
              const gchar *oauth_consumer_key, const gchar *oauth_consumer_secret,
              const gchar *oauth_token, const gchar *oauth_token_secret)
{
    gchar *url, *oauth_authorization_parameters;
    gchar **url_parameters = NULL;
    int url_parameters_length;
    gchar *authorization_text;

    url = soup_uri_to_string (soup_message_get_uri (message), FALSE);

    url_parameters_length = oauth_split_url_parameters (url, &url_parameters);
    oauth_sign_array2_process (&url_parameters_length, &url_parameters,
                               NULL, // No postargs
                               method,
                               message->method,
                               oauth_consumer_key, oauth_consumer_secret,
                               oauth_token, oauth_token_secret);
    oauth_authorization_parameters = oauth_serialize_url_sep (url_parameters_length, 1, url_parameters, ", ", 6);
    authorization_text = g_strdup_printf ("OAuth realm=\"%s\", %s", realm, oauth_authorization_parameters);
    soup_message_headers_append (message->request_headers, "Authorization", authorization_text);

    g_free (url);
    oauth_free_array (&url_parameters_length, &url_parameters);
    free (oauth_authorization_parameters);
}

int main (int argc, char **argv)
{
    const gchar *token_name = "test-oauth";

    SoupSession *session = soup_session_new_with_options (SOUP_SESSION_USER_AGENT, "test-oauth", NULL);

    /* Load the OAuth key, token and secrets */
    GKeyFile *config = g_key_file_new ();
    gchar *path = g_build_filename (g_get_user_config_dir (), "oauth-test.conf", NULL);
    g_key_file_load_from_file (config, path, G_KEY_FILE_NONE, NULL);
    gchar *oauth_consumer_key = g_key_file_get_string (config, token_name, "consumer-key", NULL);
    gchar *oauth_consumer_secret = g_key_file_get_string (config, token_name, "consumer-secret", NULL);
    gchar *oauth_token = g_key_file_get_string (config, token_name, "token", NULL);
    gchar *oauth_token_secret = g_key_file_get_string (config, token_name, "token-secret", NULL);
    if (!oauth_consumer_key || !oauth_consumer_secret || !oauth_token || !oauth_token_secret)
    {
        /* Request a token from the Canonical Identity Provider */
        SoupMessage *message = soup_message_new (SOUP_METHOD_POST, "https://login.ubuntu.com/api/v2/tokens/oauth");
        soup_message_headers_append (message->request_headers, "Accept", "application/json");
        JsonBuilder *builder = json_builder_new ();
        json_builder_begin_object (builder);
        add_string_member (builder, "email", getfield ("Email address: "));
        add_string_member (builder, "password", getpass ("Password: "));
        add_string_member (builder, "otp", getfield ("Verification code: "));
        add_string_member (builder, "token_name", token_name);
        json_builder_end_object (builder);
        set_request (message, builder);
        g_object_unref (builder);
        guint status_code = soup_session_send_message (session, message);
        g_assert (status_code == SOUP_STATUS_CREATED);

        /* Parse response */
        JsonParser *parser = json_parser_new ();
        gboolean result = json_parser_load_from_data (parser, message->response_body->data, -1, NULL);
        g_assert (result);
        JsonNode *root = json_parser_get_root (parser);
        g_assert (JSON_NODE_HOLDS_OBJECT (root));
        JsonObject *object = json_node_get_object (root);
        oauth_consumer_key = g_strdup (json_object_get_string_member (object, "consumer_key"));
        oauth_consumer_secret = g_strdup (json_object_get_string_member (object, "consumer_secret"));     
        oauth_token = g_strdup (json_object_get_string_member (object, "token_key"));
        oauth_token_secret = g_strdup (json_object_get_string_member (object, "token_secret"));

        /* Write config so we don't do this the second time */
        g_key_file_set_string (config, token_name, "consumer-key", oauth_consumer_key);
        g_key_file_set_string (config, token_name, "consumer-secret", oauth_consumer_secret);
        g_key_file_set_string (config, token_name, "token", oauth_token);
        g_key_file_set_string (config, token_name, "token-secret", oauth_token_secret);
        g_key_file_save_to_file (config, path, NULL);

        g_object_unref (message);
        g_object_unref (parser);
    }
    g_free (path);

    /* Make the request using a HTTP POST signed with OAuth */
    SoupMessage *message = soup_message_new (SOUP_METHOD_POST, "https://reviews.ubuntu.com/reviews/api/1.0/reviews/");
    g_assert (message != NULL);
    soup_message_headers_append (message->request_headers, "Accept", "application/json");
    JsonBuilder *builder = json_builder_new ();
    json_builder_begin_object (builder);
    add_string_member (builder, "package_name", "no-such-package");
    add_string_member (builder, "summary", "★★★★★");
    add_string_member (builder, "review_text", "This is the best non existant application ever!");
    add_string_member (builder, "language", "en");
    add_string_member (builder, "origin", "ubuntu");
    add_string_member (builder, "distroseries", "xenial");
    add_string_member (builder, "version", "42");
    add_int_member (builder, "rating", 5);
    add_string_member (builder, "arch_tag", "amd64");
    json_builder_end_object (builder);
    set_request (message, builder);
    g_object_unref (builder);
    sign_message (message, "Ratings and Reviews", OA_HMAC, oauth_consumer_key, oauth_consumer_secret, oauth_token, oauth_token_secret);
    guint status_code = soup_session_send_message (session, message);
    g_printerr ("%d '%s'\n", status_code, message->response_body->data);
    g_assert (status_code == SOUP_STATUS_OK);

    /* Parse the data in JSON format */
    JsonParser *parser = json_parser_new ();
    gboolean result = json_parser_load_from_data (parser, message->response_body->data, -1, NULL);
    g_assert (result);
    JsonNode *root = json_parser_get_root (parser);
    g_assert (JSON_NODE_HOLDS_OBJECT (root));
    JsonObject *object = json_node_get_object (root);
    gint64 review_id = json_object_get_int_member (object, "id");
    g_print ("Review created with ID %" G_GUINT64_FORMAT "\n", review_id);

    /* Clean up */
    g_object_unref (session);
    g_object_unref (message);
    g_object_unref (parser);
    g_free (oauth_consumer_key);
    g_free (oauth_consumer_secret);
    g_free (oauth_token);
    g_free (oauth_token_secret);
 
    return 0;
}

Accessing a JSON webservice using libsoup and JSON-GLib

A lot of web services use the JSON format. If you are working a GLib based project and need to access a service like this there are two great libraries to help you - libsoup and JSON-Glib.

For my example, I'm going to grab some review data from Ubuntu (API) which looks something like this:

[
    {
        "ratings_total": 229,
        "ratings_average": "3.84",
        "app_name": "",
        "package_name": "simple-scan",
        "histogram": "[35, 13, 22, 42, 117]"
    },
    {
        "ratings_total": 546,
        "ratings_average": "4.66",
        "app_name": "",
        "package_name": "audacity",
        "histogram": "[17, 7, 17, 63, 442]"
    },
    ...
]

The data is a single array of objects that contain the statistics for each package. For this example I'll print out the number of ratings for each package by getting the package_name and ratings_total members from each object.

Firstly, I need to download the data. The data is retrieved using a HTTP GET request; in libsoup you can do this with:

    SoupSession *session = soup_session_new_with_options (SOUP_SESSION_USER_AGENT, "test-json", NULL);
    SoupMessage *message = soup_message_new (SOUP_METHOD_GET, "https://reviews.ubuntu.com/reviews/api/1.0/review-stats/any/any");
    soup_session_send_message (session, message);

Now I have the server text in message->response_body->data but it needs to be decoded. JSON-GLib can parse it with:

    JsonParser *parser = json_parser_new ();
    json_parser_load_from_data (parser, message->response_body->data, -1, NULL);
    JsonNode *root = json_parser_get_root (parser);

Now I have an in-memory tree of the JSON data which can be traversed. After checking the root node is an array as expected I'll iterate over each object:

    g_assert (JSON_NODE_HOLDS_ARRAY (root));
    array = json_node_get_array (root);
    for (i = 0; i < json_array_get_length (array); i++)
    {
        JsonNode *node = json_array_get_element (array, i);
        /* do stuff... */
    }

For each object, I extract the required data:

        g_assert (JSON_NODE_HOLDS_OBJECT (node));
        JsonObject *object = json_node_get_object (node);
        const gchar *package_name = json_object_get_string_member (object, "package_name");
        gint64 ratings_total = json_object_get_int_member (object, "ratings_total");
        if (package_name)
            g_print ("%s: %" G_GUINT64_FORMAT "\n", package_name,
 
Combined into a program, I can print out the number of reviews for each package:

simple-scan: 229
audacity: 546
...

The full program:

// gcc -g -Wall example-json.c -o example-json `pkg-config --cflags --libs libsoup-2.4 json-glib-1.0`

#include
#include

int main (int argc, char **argv)
{
    SoupSession *session;
    SoupMessage *message;
    guint status_code;
    JsonParser *parser;
    gboolean result;
    JsonNode *root;
    JsonArray *array;
    gint i;

    /* Get the data using a HTTP GET */
    session = soup_session_new_with_options (SOUP_SESSION_USER_AGENT, "test-json", NULL);
    message = soup_message_new (SOUP_METHOD_GET, "https://reviews.ubuntu.com/reviews/api/1.0/review-stats/any/any");
    g_assert (message != NULL);
    status_code = soup_session_send_message (session, message);
    g_assert (status_code == SOUP_STATUS_OK);

    /* Parse the data in JSON format */
    parser = json_parser_new ();
    result = json_parser_load_from_data (parser, message->response_body->data, -1, NULL);
    g_assert (result);

    /* The data should contain an array of JSON objects */
    root = json_parser_get_root (parser);
    g_assert (JSON_NODE_HOLDS_ARRAY (root));
    array = json_node_get_array (root);
    for (i = 0; i < json_array_get_length (array); i++)
    {
        JsonNode *node;
        JsonObject *object;
        const gchar *package_name;
        gint64 ratings_total;

        /* Get the nth object, skipping unexpected elements */
        node = json_array_get_element (array, i);
        if (!JSON_NODE_HOLDS_OBJECT (node))
            continue;

        /* Get the package name and number of ratings from the object - skip if has no name */
        object = json_node_get_object (node);
        package_name = json_object_get_string_member (object, "package_name");
        ratings_total = json_object_get_int_member (object, "ratings_total");
        if (package_name)
            g_print ("%s: %" G_GINT64_FORMAT "\n", package_name, ratings_total);
    }

    /* Clean up */
    g_object_unref (session); 
    g_object_unref (message);
    g_object_unref (parser);

    return 0;
}

And to show you can do the same thing with GIR bindings, here's the same in Vala:

// valac example-json.vala --pkg soup-2.4 --pkg json-glib-1.0

public int main (string[] args)
{
    /* Get the data using a HTTP GET */
    var session = new Soup.Session.with_options (Soup.SESSION_USER_AGENT, "test-json");
    var message = new Soup.Message ("GET", "https://reviews.ubuntu.com/reviews/api/1.0/review-stats/any/any");
    assert (message != null);
    var status_code = session.send_message (message);
    assert (status_code == Soup.Status.OK);

    /* Parse the data in JSON format */
    var parser = new Json.Parser ();
    try
    {
        parser.load_from_data ((string) message.response_body.data);
    }
    catch (Error e)
    {
    }

    /* The data should contain an array of JSON objects */
    var root = parser.get_root ();
    assert (root.get_node_type () == Json.NodeType.ARRAY);
    var array = root.get_array ();
    for (var i = 0; i
    {
        /* Get the nth object, skipping unexpected elements */
        var node = array.get_element (i);
        if (node.get_node_type () != Json.NodeType.OBJECT)
            continue;

        /* Get the package name and number of ratings from the object - skip if has no name */
        var object = node.get_object ();
        var package_name = object.get_string_member ("package_name");
        var ratings_total = object.get_int_member ("ratings_total");
        if (package_name != null)
            stdout.printf ("%s: %" + int64.FORMAT + "\n", package_name, ratings_total);
    }

    return 0;
}

and Python:

#!/usr/bin/python

from gi.repository import Soup
from gi.repository import Json

# Get the data using a HTTP GET
session = Soup.Session.new ()
session.set_property (Soup.SESSION_USER_AGENT, "test-json")
message = Soup.Message.new ("GET", "https://reviews.ubuntu.com/reviews/api/1.0/review-stats/any/any")
assert (message != None)
status_code = session.send_message (message)
assert (status_code == Soup.Status.OK)

# Parse the data in JSON format
parser = Json.Parser ()
parser.load_from_data (message.response_body.data, -1)

# The data should contain an array of JSON objects
root = parser.get_root ()
assert (root.get_node_type () == Json.NodeType.ARRAY)
array = root.get_array ()
for i in xrange (array.get_length ()):
    # Get the nth object, skipping unexpected elements
    node = array.get_element (i)
    if node.get_node_type () != Json.NodeType.OBJECT:
        continue

    # Get the package name and number of ratings from the object - skip if has no name
    object = node.get_object ()
    package_name = object.get_string_member ("package_name")
    ratings_total = object.get_int_member ("ratings_total")
    if package_name != None:
        print ("%s: %d" % (package_name, ratings_total))