Brain Flush

May 3, 2009

Introducing Signpost: Easy OAuth for Java and Apache HttpComponents (Android, too)

Filed under: Software Development & Programming — Tags: , , , , , , , — Matthias @ 10:58 am

I would like to announce a project I started, which I hope may be useful to Java (particularly Google Android) application developers who need to communicate with web services that leverage the OAuth protocol for accessing protected resources.

What is Signpost?

Signpost is the easy and intuitive solution for signing HTTP messages in conformance with the OAuth Core 1.0 standard. Signpost has been designed to work in conjunction with the Apache HttpComponents library, a proven, well-established HTTP library for the Java programming language.

Signpost is still in beta stage, which means it may contain bugs. The project is currently hosted at Google Code and can be downloaded, distributed and used under the terms of the Apache License, version 2.

Why another OAuth library for Java?

It was my discontendness with Netflix’s OAuth implementation which ultimately drove me to develop my own solution. My biggest gripes with it were:

  • Its clumsy, overly complicated API
  • Its tendency to do more than what’s actually in the standard (why does it implement an own HTTP layer? OAuth is about message signing, not about message sending)
  • Its limitations resulting from the last point, particularly:
    • Its ignorance towards RESTful web services (a 201 is treated as an error)
    • Its inability to process other HTTP verbs than GET (you must subclass in order to POST or PUT a resource)
    • Its inability to send more complex messages (don’t even try to send multipart requests)

Signpost attempts to avoid these issues as described below (a brief remark: Despite those issues, I highly appreciate Netflix’s work on the original implementation. In fact, Signpost is to some degree based on code from the reference implementation).

Goals of Signpost

Signpost has been designed with several principal goals in mind:


Using Signpost is as simple as it could possibly get — all actions are executed with only a few lines of code. For example, here is how you would sign an HTTP message using Signpost (assuming you have already created the involved HttpClient and OAuthConsumer objects):

        // create an HTTP request to a protected resource
        HttpGet request = new HttpGet("");

        // sign the request (consumer is a Signpost OAuthConsumer)

        // send the request
        HttpResponse response = httpClient.execute(request);

Signpost exposes a minimalistic API designed for two purposes: Signing HTTP messages and requesting tokens from an OAuth service provider. Everything else is beyond the scope of the OAuth specification, and is thus left to the HTTP messaging layer, where it belongs.

For more exhaustive examples, please refer to GettingStarted.


Signpost tries to be as unobtrusive as possible. Unlike the OAuth reference implementation for Java, Signpost does not wrap the entire HTTP layer and hides its features from the client. Instead, you simply pass an HttpRequest object to it, and Signpost will sign the message using the credentials it was configured with. This means that all the power and flexibility of the Apache HttpComponents? is still at your fingertips!


Simplicity doesn’t come free. Thus, Signpost currently makes certain assumptions to reduce the complexity of both the implementation and the API.

Deviations from the OAuth standard

  • Additional service provider parameters for retrieving request tokens are currently unsupported (cf. section 6.1)
  • Message signing using public key encryption (as per section 9.3) is currently unsupported. Message signing using the PLAINTEXT and HMAC-SHA1 is supported, however.
  • The OAuth standard demands that OAuth request parameters may be put in the URI query string or in the message payload. Signpost will never do that; instead, all OAuth protocol parameters are written to the HTTP Authorization header field. Anything you put there will be overwritten by Signpost.
  • Signpost does not support writing OAuth protocol params to the WWW-Authenticate header field

I believe that even with those restrictions in place, Signpost will work for 99% of its users. Trading in some flexibility only relevant for edge cases was a design decision. If that doesn’t work for your setup, then Signpost is probably not the best choice.

Thread Safety

Signpost is not thread safe and probably will never be. Signpost objects are very lightweight, so you are adviced to create an OAuthConsumer and OAuthProvider for every thread in your application that must send signed HTTP requests.

Google Android

Signpost works flawlessly in conjunction with Android, Google’s software stack for mobile devices. In fact, Signpost has already signed thousands of HTTP requests at this very moment, as it is an integral part of Qype Radar, our geo-sensitive mobile application for Android that finds the best places near you.

Since Android already ships with a recent version of the Apache HttpComponents?, you merely need to add the Signpost JAR to your Android application, and you’re all set.

October 17, 2008

Talking to Web Servers via HTTP in Android 1.0

Filed under: Mobile Devices, Software Development & Programming — Tags: , , , — Matthias @ 9:08 pm

If you have worked with Google Android prior to the 1.0 release, you probably have noticed that Google has upgraded the Apache HttpClient module in Android 1.0 from 3.x to a recent alpha release of version 4, which is a complete rewrite and brings with it a host of API changes. Unfortunately, the alpha version included in Android 1.0 is both terribly documented and lacking an important feature: supporting multipart requests using the multipart/form-data MIME type. Sending GET requests has changed significantly, too. Where you were formerly be able to add request parameters to a GetMethod object, you now will have to build a query string on your own and initialize an HttpGet object with it.

Here is how HTTP GET works in Android 1.0:

HttpClient httpClient = new DefaultHttpClient();

StringBuilder uriBuilder = new StringBuilder(SERVICE_ENDPOINT);
uriBuilder.append("?param0=" + param0);
uriBuilder.append("&param1=" + param1);
uriBuilder.append("&paramN=" + paramN);

HttpGet request = new HttpGet(uriBuilder.toString());
HttpResponse response = httpClient.execute(request);

int status = response.getStatusLine().getStatusCode();

// we assume that the response body contains the error message
if (status != HttpStatus.SC_OK) {
    ByteArrayOutputStream ostream = new ByteArrayOutputStream();
    Log.e("HTTP CLIENT", ostream.toString()));
} else {
    InputStream content = response.getEntity().getContent();
    // <consume response>
    content.close(); // this will also close the connection

Sending a multipart request first involves some environmental setup, since the required libraries are not bundled with Android 1.0. First, go to the Apache HttpClient download page and download the distribution called “Binary with dependencies”. In that package you’ll find two libraries: apache-mime4j-0.4.jar and httpmime-4.0-beta1.jar. Copy these files to e.g. the lib/ folder of your Android project and add them to the build path. You can now use a MultipartEntity to send multipart POSTs as such:

HttpClient httpClient = new DefaultHttpClient();

HttpPost request = new HttpPost(SERVICE_ENDPOINT);
MultipartEntity entity = new MultipartEntity();
entity.addPart("param0", new StringBody("value0"));
entity.addPart("param1", new StringBody(Double.toString(1.0)));
entity.addPart("paramN", new FileBody(new File("/bar/baz")));

HttpResponse response = httpClient.execute(request);
int status = response.getStatusLine().getStatusCode();

if (status != HttpStatus.SC_OK) {
    // see above   
} else {
    // see above

And that’s that. I assume Google has included this early build of HttpClient 4 in Android 1.0 so that they will remain API-stable for the years to come, without being stuck with a legacy HTTP component.


March 18, 2008

Using cURL for Testing Web Applications

Filed under: Software Development & Programming — Tags: , , , , , , , , — Matthias @ 11:05 am

This may be old news to some, but although I had heard of using cURL for downloading files off the internet from a command line (as a wget alternative), I didn’t know that it was actually capable of doing much, much more. In fact, cURL is a very good allrounder at doing anything related to transmitting and receiving data using popular protocols such as HTTP(S), (S)FTP, TELNET, LDAP and more. In particular, you can use it to test your Web application with very little effort!

I am currently programming for a Java Servlet based Web application, where I need to assemble HTTP requests on the client side and dissect them on the server side. More precisely, I have a servlet which extracts data from an HTTP multipart request (using Apache Commons FileUpload) and hand it over to the logic for further processing. Of course, I have to write lots of code for checking whether requests are valid and so forth. So, in order to test this code, I want to send “malformed” requests, by which I mean requests that do not carry data expected by the application in its current state. Instead of setting up a complex test environment or even modifying your client code to send bad data, you can simply use cURL.

cURL uses a simple command line interface; I will show you the most important flags. Let’s assume we have a simple RESTful Web application which manages books. We could use cURL to retrieve the list of available books like this:

curl http://localhost:8080/mybookstore/books -v

This will issue an HTTP GET on the specified URL. The -v flag tells cURL to be verbose, which means it will print all sent and received HTTP headers, the payload of the server reply, plus some additional status information to the standard output. The complete output might look something like this:

* About to connect() to localhost port 8080 (#0)
* Trying… connected
* Connected to localhost ( port 8080 (#0)
> GET /mybookstore/books HTTP/1.1
> User-Agent: curl/7.18.0 (i586-pc-mingw32msvc) libcurl/7.18.0 zlib/1.2.3
> Host: localhost:8080
> Accept: */*
< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< Content-Type: application/xml;charset=utf-8
< …
< <books><book author=”David Flanagan”>Java In A Nutshell</book><book author=”Stephen Hawking”>A Brief History Of Time</book>…</books>

The > symbol indicates data going to the server, while < indicates data coming from the server. If that’s not verbose enough for you, try using the --trace-ascii <filename> argument instead. It will log the client/server conversation to the specified file.

So, what about sending data? It’s actually just as easy. Suppose we want to store a new book by supplying its title and author:

curl http://localhost:8080/mybookstore/books -F “title=Wikinomics” -F “author=D. Tapscott, A. D. Williams”

What this does is issueing an HTTP POST to the specified address, submitting the given data using the multipart/form-data MIME type. Now that was easy! If you don’t want to type all the input data on the command line, you can also let cURL read data from a file by prefixing the value with @ followed by the file name, which makes perfect sense when transmitting binary data or long text documents. As to the way how cURL transmits the data, you can also use the -d flag, which will result in the data being POST-ed as an application/x-www-form-urlencoded string, or use the same -d flag in combination with -G to issue a GET request instead and sending the data as a URL parameter (query string).

Another noteworthy aspect about cURL is that you can use it to manage cookies. Let’s assume our book server requires the user to login first, and that the server remembers us by setting a session cookie. Ponder the commands below, where we use cURL first to authenticate with our server, and then add a new book in the same session:

curl http://localhost:8080/mybookstore/login -d “username=john” -d “password=doe” -c cookies.txt
curl http://localhost:8080/mybookstore/books -b cookies.txt -F “title=…” …

In the first line, we tell cURL to use the file cookies.txt as its “cookie jar”, which means that it will store all cookies set by the server in this file. We can leverage the session data stored in that file (for a Servlet container this would be a JSESSIONID) in consecutive requests in order to remain authenticated, by setting the value of the -b flag to the cookie jar. You can also use key/value pairs as an argument to -b, but using an intermediary file is more convenient.

cURL is free software and can be downloaded for a broad range of platforms from the cURL website.

Blog at