Signpost, my client-side OAuth library for Java, now comes modularized, so that you can use it with other HTTP libraries than Apache HTTP Components.

By default, Signpost now only supports signing java.net.HttpURLConnection type requests, which makes the core module completely independent of any specific HTTP messaging layer. Instead, additional HTTP libraries are supported via add-on JARs; adapters have been created for Jetty HTTP 6 and Apache HTTP Components 4.

More info on the project website: http://code.google.com/p/oauth-signpost

*UPDATE* The source code is now on GitHub. Feel free to fork ‘n fix.

*UPDATE* I have put up the fixed dom4j JAR for download. The Jaxen JAR in that archive I didn’t touch, it’s the same you would download from their website.

I have been very disappointed with Android’s XML parsing support from day one, it’s simply too low level, inconvenient to use, and is lacking important features (I was especially disappointed with the decision to exclude the JAXP XPath support from Android, which has become an integral part of the JSE).

This is not only about cosmetics. Parsing XML documents of only medium complexity already turned out to be error prone and very tedious on Android (white space normalization problems, broken Unicode entity ref expansion, etc.) and we would’ve had to rewrite stuff which existing Java XML libraries already do in a graceful and stable manner.

Since I have always been a big fan of dom4j, I fixed an issue with the latest source tree that prevented dom4j’s QNAME caching to work with Android’s Java implementation (or more precisely, with Apache Harmony’s SAX implementation — the Android Java implementation is based on a pre-release version of Apache Harmony).

I haven’t committed that change back to dom4j yet, because development seems to have stalled on that project, but if anyone is interested, I can host the source code and a working JAR somewhere (please drop a short line in the comments section, otherwise I won’t bother sharing it).

dom4j also works very well in conjunction with Jaxen (a free XPath implementation)!

Some example code to wet your mouth:

SAXReader reader = new SAXReader(); // dom4j SAXReader
Document document = reader.read(xmlInputStream); // dom4j Document

// select all link nodes with href "http://example.com"
List<Element> linkNodes = document.selectNodes("//link[@href='http://example.com']");

// select an attribute value
String val = linkNodes.get(0).attributeValue("href");

// select element text and trim it
String value = document.elementTextTrim("childNode");

etc. pp.

Simple, powerful, straight forward — and performance is also decent (it’s pretty slow in debug mode, but reasonably fast otherwise).

Read more >> Options >>

I would like to announce a project I started, which I hope may be useful to Java (particularly Google Android) application developers who need to communicate with web services that leverage the OAuth protocol for accessing protected resources.

What is Signpost?

Signpost is the easy and intuitive solution for signing HTTP messages in conformance with the OAuth Core 1.0 standard. Signpost has been designed to work in conjunction with the Apache HttpComponents library, a proven, well-established HTTP library for the Java programming language.

Signpost is still in beta stage, which means it may contain bugs. The project is currently hosted at Google Code and can be downloaded, distributed and used under the terms of the Apache License, version 2.

Why another OAuth library for Java?

It was my discontendness with Netflix’s OAuth implementation which ultimately drove me to develop my own solution. My biggest gripes with it were:

• Its clumsy, overly complicated API
• Its tendency to do more than what’s actually in the standard (why does it implement an own HTTP layer? OAuth is about message signing, not about message sending)
• Its limitations resulting from the last point, particularly:
  • Its ignorance towards RESTful web services (a 201 is treated as an error)
  • Its inability to process other HTTP verbs than GET (you must subclass in order to POST or PUT a resource)
  • Its inability to send more complex messages (don’t even try to send multipart requests)

Signpost attempts to avoid these issues as described below (a brief remark: Despite those issues, I highly appreciate Netflix’s work on the original implementation. In fact, Signpost is to some degree based on code from the reference implementation).

Goals of Signpost

Signpost has been designed with several principal goals in mind:

Simplicity

Using Signpost is as simple as it could possibly get — all actions are executed with only a few lines of code. For example, here is how you would sign an HTTP message using Signpost (assuming you have already created the involved HttpClient and OAuthConsumer objects):

        // create an HTTP request to a protected resource
        HttpGet request = new HttpGet("http://example.com/protected");

        // sign the request (consumer is a Signpost OAuthConsumer)
        consumer.sign(request);

        // send the request
        HttpResponse response = httpClient.execute(request);

Signpost exposes a minimalistic API designed for two purposes: Signing HTTP messages and requesting tokens from an OAuth service provider. Everything else is beyond the scope of the OAuth specification, and is thus left to the HTTP messaging layer, where it belongs.

For more exhaustive examples, please refer to GettingStarted.

Unobtrusiveness

Signpost tries to be as unobtrusive as possible. Unlike the OAuth reference implementation for Java, Signpost does not wrap the entire HTTP layer and hides its features from the client. Instead, you simply pass an HttpRequest object to it, and Signpost will sign the message using the credentials it was configured with. This means that all the power and flexibility of the Apache HttpComponents? is still at your fingertips!

Limitations

Simplicity doesn’t come free. Thus, Signpost currently makes certain assumptions to reduce the complexity of both the implementation and the API.

Deviations from the OAuth standard

• Additional service provider parameters for retrieving request tokens are currently unsupported (cf. section 6.1)
• Message signing using public key encryption (as per section 9.3) is currently unsupported. Message signing using the PLAINTEXT and HMAC-SHA1 is supported, however.
• The OAuth standard demands that OAuth request parameters may be put in the URI query string or in the message payload. Signpost will never do that; instead, all OAuth protocol parameters are written to the HTTP Authorization header field. Anything you put there will be overwritten by Signpost.
• Signpost does not support writing OAuth protocol params to the WWW-Authenticate header field

I believe that even with those restrictions in place, Signpost will work for 99% of its users. Trading in some flexibility only relevant for edge cases was a design decision. If that doesn’t work for your setup, then Signpost is probably not the best choice.

Thread Safety

Signpost is not thread safe and probably will never be. Signpost objects are very lightweight, so you are adviced to create an OAuthConsumer and OAuthProvider for every thread in your application that must send signed HTTP requests.

Google Android

Signpost works flawlessly in conjunction with Android, Google’s software stack for mobile devices. In fact, Signpost has already signed thousands of HTTP requests at this very moment, as it is an integral part of Qype Radar, our geo-sensitive mobile application for Android that finds the best places near you.

Since Android already ships with a recent version of the Apache HttpComponents?, you merely need to add the Signpost JAR to your Android application, and you’re all set.

UPDATE 15/12/2009: While the problem described here still applies to the latest version of Android, I discourage anyone from using the code linked here. I’ve bundled a better solution with my Droid-Fu library, which is a little more intrusive, but leads to much leaner code for the application developer.

Today I want to talk a little bit about concurrency in Android applications, and the problems it poses on the developer. If you have used Android on your phone before, it’s likely that you have stumbled upon applications which load data off the internet, or perform other time consuming operations. The problem with time consuming operations is that, well, they consume time, and if they aren’t perfomed concurrently to Android’s user interface thread (the main thread), then the UI will lock up — certainly not a good user experience. So, it’s pretty obvious that on internet phones like those based on Android, highly concurrent applications are more the rule than the exception.

I guess I don’t have to mention that developing concurrent applications is everything but simple. Keeping threads that share data in sync is not a trivial task and prone to errors. What makes it even more difficult in Android is the fact that while your application is loading data, it may suddenly be interrupted by an incoming phone call or because the user decided to flip the screen into a different orientation. You may think that your thread will get paused while the activity that created it is brought to the background (or even gets destroyed). That’s not what happens though: any thread will continue running until it completes, even if your activity or service is not alive anymore. And that’s where the pain starts.

Problem Statement

Consider the following news reader application (just as an example). To keep things simple, the application only supports one operation: Showing the latest news articles on the screen using pagination. When started the first time, the application loads the first 10 articles off the internet in a separate thread and renders a nifty spinner visual to keep the user happy. When the user wants to read the next 10 articles, a button is pressed, the spinner appears again, and the next 10 articles are loaded, etc. pp. We have two separate threads running concurrently in this application: The main UI thread that renders all the article items and the spinner graphic, and the thread that downloads data off the web and posts that data back to the UI thread upon completion, so the UI can update itself accordingly.

Now, what happens if a phone call comes in while we see the spinner graphic? What happens is that Android will pause your application (probably even destroy it) in order to launch the call activity on top of it, but your download thread will continue running in the background. It’s difficult to tell what is going to happen when that thread tries to post its result to an activity that doesn’t even exist anymore, but most probably that application will crash (for example when the download result handler in that activity tries to display a toast or dialog, which will be attached to a window that has already been destroyed). If that thread doesn’t terminate and still holds a reference to the calling activity, it may even produce a memory leak. Long story short, you will definitely want to introduce some mechanism that keeps the communication between activities, services, and any threads they run, in sync.

Solution

Lucky for you, it just so happens that I have written a module that does exactly that. Based on my former work on this problem and an excellent article by Eric Burke (who already presented an almost-working solution), I have come up with a Task class that you can use to dispatch long running operations from your activities and services and which also handles all the inconveniences arising from situations like resuming from an interruption (phone calls and the likes).

And here is how you use it:

public class Concurrency extends Activity implements TaskListener<String> {

    private static final int TASK1 = 0;

    private static final int TASK2 = 1;

    private Task<String> task1, task2;

    private Callable<String> callable1 = new Callable<String>() {

        public String call() throws Exception {
            try {
                System.out.println("task1 starting");
                Thread.sleep(3000);
            } catch (InterruptedException e) {
                System.out.println("task1 finished");
            }
            return "task1 result";
        };
    };

    private Callable<String> callable2 = new Callable<String>() {

        public String call() throws Exception {
            try {
                System.out.println("task2 starting");
                Thread.sleep(6000);
            } catch (InterruptedException e) {
                System.out.println("task2 finished");
            }
            return "task2 result";
        };
    };

    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
    }

    @Override
    protected void onPause() {
        super.onPause();

        task1.unregisterCallback();
        task2.unregisterCallback();
    }

    @Override
    protected void onResume() {
        super.onResume();

        task1 = Task.getOrCreate(this, TASK1);
        task2 = Task.getOrCreate(this, TASK2);

        switch (task1.state()) {
        case NOT_STARTED:
            task1.run(this, callable1);
            break;
        case RUNNING:
            System.out.println("task1 still running");
            break;
        case COMPLETED:
            System.out.println("task1 completed in background, result: "
                    + task1.getResult());
            break;
        }

        switch (task2.state()) {
        case NOT_STARTED:
            task2.run(this, callable2);
            break;
        case RUNNING:
            System.out.println("task2 still running");
            break;
        case COMPLETED:
            System.out.println("task2 completed in background, result: "
                    + task2.getResult());
            break;
        }

    }

    @Override
    public void onTaskFinished(Task<String> task) {

        if (task.failed()) {
            System.err.println("task" + task.getTaskId() + " failed. Reason: "
                    + task.getError().getMessage());
        } else {
            System.out.println("task" + task.getTaskId() + " finish handler: "
                    + task.getResult());
        }
    }

    @Override
    public boolean onKeyDown(int keyCode, KeyEvent event) {

        if (keyCode == KeyEvent.KEYCODE_BACK) {
            Task.cancelAll(this);
        }

        return super.onKeyDown(keyCode, event);
    }
}

Let’s walk through that code bit by bit. First, our activity defines two task objects, task1 and task2, with IDs TASK1 and TASK2. What those tasks are supposed to do is defined using two Callable objects, callable1 and callable2. Everything that happens inside the call() methods of those objects will be executed in a separate thread. We also have to tell those tasks what will happen should they complete. We do this by implementing the TaskListener interface, which currently only defines a single method: onTaskFinished(Task). We can check in that handler whether the task succeeded or not by calling its failed() method. If any exception was thrown during the execution of call(), this method will return true and the exception can be retrieved by calling its getError() method. Otherwise, the return value of getResult() is guranteed to be whatever you return in the callable. The Task class is generic: You instantiate it using the return type of the callable you pass to it. This ensures type safety when working with the result object. The same holds for TaskListener.

A closer look to onResume() reveals that everytime our activity is resumed, those task objects are either already in memory or will be created for us by calling Task.getOrCreate(). If we already started that task during a previous life-cycle of our activity, we can poll its status to check whether it has already completed or if it’s still running. The former is the case if the thread had terminated while our activity was paused or even completely destroyed; in that case, we can simply pick up whatever result the task came up with in the meantime. We also call Task.unregisterCallback() in onPause() in order to avoid being called back by the task when the activity goes poof (not doing so may result in memory leaks, as described above).

Right now, the activity will pick up any results of a task with a certain ID when being restarted, even when explicitly restarted by the user. If that’s not what you want, you can make a call to Task.cancelAll() in the key handler for the ‘back’ key. That way you can ensure that all tasks (or their results) are discarded when explicitly exiting the activity. You can also cancel a single task using task.cancel(). A canceled task will never post any result or error data back to the caller.

How it works

Internally, task state is maintained in a static hash mapping callers to their list of tasks. This assures that tasks are kept in memory as long as the Task class itself (or until they terminate of course). The Task class does all the locking, state updates and callback invocations for us; we can even create a ProgressDialog and assign it to a Task using Task.setProgressDialog(). The dialog will then automatically be displayed when the task starts running, and will close when the task finishes. A task (or more precisely: a list of tasks) is bound to its caller (the calling activity or service) by the caller’s ComponentName. That means, you can think of tasks being associated to a calling class, rather than a calling object. The task class will take care of removing tasks that finished and which have been posted back to the caller, but it will preserve all completed (uncanceled) tasks until the caller claims its results, in case the caller wasn’t reachable while the task was finishing.

You can download the Task module for free here.

If you have worked with Google Android prior to the 1.0 release, you probably have noticed that Google has upgraded the Apache HttpClient module in Android 1.0 from 3.x to a recent alpha release of version 4, which is a complete rewrite and brings with it a host of API changes. Unfortunately, the alpha version included in Android 1.0 is both terribly documented and lacking an important feature: supporting multipart requests using the multipart/form-data MIME type. Sending GET requests has changed significantly, too. Where you were formerly be able to add request parameters to a GetMethod object, you now will have to build a query string on your own and initialize an HttpGet object with it.

Here is how HTTP GET works in Android 1.0:

HttpClient httpClient = new DefaultHttpClient();

StringBuilder uriBuilder = new StringBuilder(SERVICE_ENDPOINT);
uriBuilder.append("?param0=" + param0);
uriBuilder.append("&param1=" + param1);
uriBuilder.append("&paramN=" + paramN);

HttpGet request = new HttpGet(uriBuilder.toString());
HttpResponse response = httpClient.execute(request);

int status = response.getStatusLine().getStatusCode();

// we assume that the response body contains the error message
if (status != HttpStatus.SC_OK) {
    ByteArrayOutputStream ostream = new ByteArrayOutputStream();
    response.getEntity().writeTo(ostream);
    Log.e("HTTP CLIENT", ostream.toString()));
} else {
    InputStream content = response.getEntity().getContent();
    // <consume response>
    content.close(); // this will also close the connection
}

Sending a multipart request first involves some environmental setup, since the required libraries are not bundled with Android 1.0. First, go to the Apache HttpClient download page and download the distribution called “Binary with dependencies”. In that package you’ll find two libraries: apache-mime4j-0.4.jar and httpmime-4.0-beta1.jar. Copy these files to e.g. the lib/ folder of your Android project and add them to the build path. You can now use a MultipartEntity to send multipart POSTs as such:

HttpClient httpClient = new DefaultHttpClient();

HttpPost request = new HttpPost(SERVICE_ENDPOINT);
MultipartEntity entity = new MultipartEntity();
entity.addPart("param0", new StringBody("value0"));
entity.addPart("param1", new StringBody(Double.toString(1.0)));
entity.addPart("paramN", new FileBody(new File("/bar/baz")));
request.setEntity(entity);

HttpResponse response = httpClient.execute(request);
int status = response.getStatusLine().getStatusCode();

if (status != HttpStatus.SC_OK) {
    // see above  
} else {
    // see above
}

And that’s that. I assume Google has included this early build of HttpClient 4 in Android 1.0 so that they will remain API-stable for the years to come, without being stuck with a legacy HTTP component.

–

References:
http://groups.google.com/group/android-developers/browse_frm/thread/e4230ed22c196772
http://wiki.apache.org/HttpComponents/HttpCoreTutorial
http://hc.apache.org/httpcomponents-client/examples.html

This may be old news to some, but although I had heard of using cURL for downloading files off the internet from a command line (as a wget alternative), I didn’t know that it was actually capable of doing much, much more. In fact, cURL is a very good allrounder at doing anything related to transmitting and receiving data using popular protocols such as HTTP(S), (S)FTP, TELNET, LDAP and more. In particular, you can use it to test your Web application with very little effort!

I am currently programming for a Java Servlet based Web application, where I need to assemble HTTP requests on the client side and dissect them on the server side. More precisely, I have a servlet which extracts data from an HTTP multipart request (using Apache Commons FileUpload) and hand it over to the logic for further processing. Of course, I have to write lots of code for checking whether requests are valid and so forth. So, in order to test this code, I want to send “malformed” requests, by which I mean requests that do not carry data expected by the application in its current state. Instead of setting up a complex test environment or even modifying your client code to send bad data, you can simply use cURL.

cURL uses a simple command line interface; I will show you the most important flags. Let’s assume we have a simple RESTful Web application which manages books. We could use cURL to retrieve the list of available books like this:

curl http://localhost:8080/mybookstore/books -v

This will issue an HTTP GET on the specified URL. The -v flag tells cURL to be verbose, which means it will print all sent and received HTTP headers, the payload of the server reply, plus some additional status information to the standard output. The complete output might look something like this:

* About to connect() to localhost port 8080 (#0)
* Trying 127.0.0.1… connected
* Connected to localhost (127.0.0.1) port 8080 (#0)
> GET /mybookstore/books HTTP/1.1
> User-Agent: curl/7.18.0 (i586-pc-mingw32msvc) libcurl/7.18.0 zlib/1.2.3
> Host: localhost:8080
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: Apache-Coyote/1.1
< Content-Type: application/xml;charset=utf-8
< …
< <books><book author=”David Flanagan”>Java In A Nutshell</book><book author=”Stephen Hawking”>A Brief History Of Time</book>…</books>

The > symbol indicates data going to the server, while < indicates data coming from the server. If that’s not verbose enough for you, try using the --trace-ascii <filename> argument instead. It will log the client/server conversation to the specified file.

So, what about sending data? It’s actually just as easy. Suppose we want to store a new book by supplying its title and author:

curl http://localhost:8080/mybookstore/books -F “title=Wikinomics” -F “author=D. Tapscott, A. D. Williams”

What this does is issueing an HTTP POST to the specified address, submitting the given data using the multipart/form-data MIME type. Now that was easy! If you don’t want to type all the input data on the command line, you can also let cURL read data from a file by prefixing the value with @ followed by the file name, which makes perfect sense when transmitting binary data or long text documents. As to the way how cURL transmits the data, you can also use the -d flag, which will result in the data being POST-ed as an application/x-www-form-urlencoded string, or use the same -d flag in combination with -G to issue a GET request instead and sending the data as a URL parameter (query string).

Another noteworthy aspect about cURL is that you can use it to manage cookies. Let’s assume our book server requires the user to login first, and that the server remembers us by setting a session cookie. Ponder the commands below, where we use cURL first to authenticate with our server, and then add a new book in the same session:

curl http://localhost:8080/mybookstore/login -d “username=john” -d “password=doe” -c cookies.txt
curl http://localhost:8080/mybookstore/books -b cookies.txt -F “title=…” …

In the first line, we tell cURL to use the file cookies.txt as its “cookie jar”, which means that it will store all cookies set by the server in this file. We can leverage the session data stored in that file (for a Servlet container this would be a JSESSIONID) in consecutive requests in order to remain authenticated, by setting the value of the -b flag to the cookie jar. You can also use key/value pairs as an argument to -b, but using an intermediary file is more convenient.

cURL is free software and can be downloaded for a broad range of platforms from the cURL website.

Direct Web Remoting, or DWR for short, is a Web technology for exposing server-side Java interfaces via a dynamically generated JavaScript API. A specific Servlet will translate incoming JavaScript calls to their Java equivalents which makes integration of applications programmed in client-side JavaScript with Web applications written in server-side Java a no-brainer.

DWR calls are typically dispatched from a JavaScript/JScript program running in a Web browser such as Mozilla Firefox or Microsoft’s Internet Explorer, but since all Mozilla products are built upon the Gecko layout engine and XUL (which are programmed using JavaScript modules), you can leverage DWR to dispatch remote calls from e.g. a Mozilla Thunderbird extension just as easily. Integrating DWR into a XUL based extension is actually pretty straight forward, but there are couple of subtleties to be aware of.

I will hereafter assume that a Web application called ‘MyWebApp’ has been deployed on localhost port 8080, that it’s up and running, that it exposes an interface called ‘MyService’ via DWR. Refer to the DWR documentation if you are unsure how to do that. It is further assumed that the DWR servlet is reached at dwr/ relative to the base URL. All following code examples refer to DWR version 1.1, which unfortunately has already been superseded by version 2.x at the time of this writing. If required at all, changing the code to run with newer versions of DWR should be easy enough however.

In order for your XUL component to be able to access `MyService’, add the following lines to your XUL file (e.g. myOverlay.xul):

<script type="application/x-javascript"
    src="http://localhost:8080/MyWebApp/dwr/engine.js"/>

<script type="application/x-javascript"
    src="http://localhost:8080/MyWebApp/dwr/util.js"/>

<script type="application/x-javascript"
    src="http://localhost:8080/MyWebApp/dwr/interface/MyService.js"/>

Don’t go looking for MyService.js in your file system, it’s not there. DWR will take care of dynamically creating it once your client requests it from the server. You can find both engine.js and util.js in the dwr.jar file which should reside at WEB-INF/lib of your deployed Web application.

Now you’re already prepared to invoke your service. However, when run outside a Web browser, as is the case with e.g. a Thunderbird extension, there is a catch: DWR won’t correctly resolve the target URL for your server, because it operates on relative URLs by default. You can fix that problem by modifying the MyService._path variable, which was auto-generated by DWR. Ponder the example program below, which invokes MyService.service() from a JavaScript program running in a Thunderbird extension:

function callService() {

    MyService._path = "http://localhost:8080/MyWebApp/dwr";

    MyService.service({
        callback: function(returnData) {
            // perform logic
        },

        errorHandler: function(error) {
            // handle error
        }
    });
}

That’s it basically; of course you may want to add additional call variables or use another invocation style altogether, but you get the idea.