Brain Flush

January 10, 2010

Introducing Calculon – a Java DSL for Android Activity testing

Filed under: Mobile Devices, Software Development & Programming — Tags: , , — Matthias @ 1:05 pm

I have been working on a new test library for Google Android lately, a project which I’m very excited about, and which I feel has loads of potential: Calculon, an Android testing library which lets you write functional tests for your Activities using a very concise and natural syntax.

I am a big fan of test-driven development, and I think there’s no arguing about the massive benefits you get from backing your software up with an automated test suite. The support for writing tests in Android, however, is lacking at best. Even though there has been a test runner since the 1.0 days, writing unit or functional tests in Android is far from being a pleasant experience. For me, the two most important things about tests are that they must be easy to write, and more importantly, easy to read.

That being said, here is a classic example of why Android testing sucks. Our test story is:

Assert that a click on ‘button’ launches a new BarActivity.

Without Calculon, what you do is this:

public void whatInHeavensNameDoesThisEvenTest() {
    final Button b = (Button) getActivity().findViewById(R.id.button);
    ActivityMonitor monitor = getInstrumentation().addMonitor(
        BarActivity.class.getCanonicalName(), null, false);
    getInstrumentation().runOnMainSync(new Runnable() {
        public void run() {
            b.performClick();
        }
    });
    getInstrumentation().waitForIdleSync();
    assertTrue(getInstrumentation().checkMonitorHit(monitor, 1));
}

So, I first have to retrieve a reference to the button that’s supposed to launch a new BarActivity. Then I have to create what Android calls an ActivityMonitor, that’s simply an object which just sits there and waits until an Activity is being launched that matches the description it was configured with. Then I have to perform the actual button click, but sorry, not allowed on the instrumentation thread, so I have to runOnMainSync(). We’re not yet there though: we have to sit and wait, because launching a new activity could take a while, so we also have to waitForIdleSync(). The assertion that follows also couldn’t be less expressive.

When I first wrote this test, my single thought was: Where the heck is my story? It’s buried under a heap of ugly boilerplate code.

I don’t care that the test runner cannot perform a view event on the instrumentation thread. I don’t care that it needs to sit and wait for the Activity to launch. I don’t care about ActivityMonitor and its crummy API. I just want to look at the test, and understand what it’s doing.

Let’s reiterate our test story:

Assert that a click on ‘button’ launches a new BarActivity.

With Calculon, this test is written as:

assertThat(R.id.button).click().starts(BarActivity.class);

Oh the beauty. Our test story was a single sentence, so is the test. That’s how it should be, really. There’s nothing worse than seeing someone else’s tests failing on the build server and having to dissect them line by line to figure out what it’s actually testing.

So let’s move on and look at some nifty Calculon tests. And remember: “Calculon never does two takes!”

Testing with Calculon

Calculon is in a very early stage of development, and currently it only supports writing functional Activity tests, using Android’s ActivityInstrumentationTestCase2. Here is what a Calculon test looks like:

public class FooTest extends FunctionalTest<FooActivity> {
    public FooTest() {
        super("com.example", FooActivity.class);
    }

    public void testStuff() {
        ...
    }
}

Nothing special here, really. You just inherit from a Calculon FunctionalTest.

Assertions

At the heart of a Calculon test are assertions. To create a new assertion, you use the assertThat() method:

public void testStuff() {
    // testing against activities
    assertThat()...
    assertThat(getActivity())...
    assertThat(someOtherActivity)...

    // testing against views
    assertThat(R.id.some_button)...
    assertThat(someButton)...

    // of course all Junit assertions work as well
    assertEquals(...)
    assertNotNull(...)
    ...
}

For now, there are two basic kinds of assertions: ActivityAssertion and ViewAssertion. There is also a third kind, ActionAssertion, but it takes a somewhat special role and is discussed at the end.

Activity Assertions

An ActivityAssertion is an assertion on the state of an Activity and is used like this:

public void testStuff() {
    // testing for an orientation
    assertThat().inPortraitMode();

    // testing for views
    assertThat().viewExists(R.id.some_button);

    // testing for input actions
    assertThat().keyDown(KeyEvent.KEYCODE_BACK)...

    // testing for custom predicates
    assertThat().satisfies(new Predicate<Activity>() {
        public boolean check(Activity target) {
            return target.isTaskRoot();
        }
    });
}

View Assertions

A ViewAssertion is basically the same, just that it tests the state and behavior of a view:

public void testStuff() {
    // testing for view state
    assertThat(R.id.button).isVisible();
    assertThat(R.id.button).isGone();

    // testing for input actions
    assertThat(R.id.button).keyDown(KeyEvent.KEYCODE_BACK)...
    assertThat(R.id.button).click()...
    assertThat(R.id.button).longClick()...

    // testing for custom predicates
    assertThat(R.id.button).satisfies(new Predicate<View>() {
        public boolean check(View target) {
            return target.getVisibility() == View.VISIBLE;
        }
    });
}

Action Assertions

This is a special kind of assertion. You always use it when you want to assert that some action (e.g. a click or a key press) does something. You can use it on both activity and view assertions, and it also allows you to delegate an assertion to another object:

public void testStuff() {
    // testing for actions that launch a new activity
    assertThat(R.id.b1).click().starts(BarActivity.class);

    // testing for actions that finish an activity
    assertThat(R.id.b2).keyDown(KeyEvent.KEYCODE_Q).finishesActivity();

    // testing for actions that change something
    assertThat(R.id.b3).click().implies(R.id.b2).isGone();
    assertThat(R.id.b4).click().implies(getActivity()).inLandscapeMode();
    assertThat(R.id.b5).click().implies(new Predicate<Model>() {
        public boolean check(Model target) {
            return target.someAttribute() == 5;
        }
    });
}

There’s more to come!

This is only a very early version. I intend to expand this library significantly over time. Of course it’s open source, so contributions are always welcome. You can fork the project on GitHub, add your changes, and send me a pull request.

About these ads

23 Comments »

  1. This looks epic! Can’t wait to play around with it.

    Comment by Michael Leung — January 10, 2010 @ 9:51 pm

  2. [...] Introducing Calculon – a Java DSL for Android Activity testing « Brain Flush (tags: android testing fluent)   [...]

    Pingback by links for 2010-01-10 « Object neo = neo Object — January 11, 2010 @ 5:30 am

  3. This looks really cool, I will try it out soon.

    One question: Is it intended for running within or outside the emulator? I have realized -the hard way- that most tests cannot be run outside the emulator, because most Android classes are stubbed.

    Please, comment on your experience and viewpoint regarding plain vanilla unit tests run outside the emulator using the IDE and/or ant/mvn, versus tests within the emulator and all time it takes to load it in.

    Comment by Jens Riboe — January 11, 2010 @ 4:03 pm

    • It’s intended to run as an instrumentation on the emulator (or device). As far as I know, you cannot use the Java classloader to load Android classes, so writing “normal” unit tests outside the Android runtime that use Android classes is impossible. That’s a major drawback and I’ve seen Google developers saying they’re aware of it and “looking into it”. My experience is that it’s a little buggy both on my Mac and Linux boxes. Sometimes the test runner will simply fail to execute and just sits there without terminating. It’s also significantly slower, because the deployer first installs the APK which is under test on the device, then the test APK, and then runs the tests.

      However, I plan to migrate our Android project to Maven (via maven-android-plugin), and integrate the builds into our Hudson build server. That way, you don’t have to wait for a build to complete and performance isn’t much of an issue. You just commit, and the tests will be triggered.

      Comment by Matthias Käppler — January 11, 2010 @ 4:49 pm

  4. I am SO GLAD to see something beyond to basic JUnit stuff. I will definitely be trying this out. Kudos for kicking it off.

    Comment by Luke — January 13, 2010 @ 5:38 pm

  5. [...] very excited to see Calculon – a Java DSL for Android Activity testing – Awesome – just when I thought nobody cared about TDD on [...]

    Pingback by dropping in for a minute « Teh Tech Blogz0r by Luke Meyer — January 13, 2010 @ 8:53 pm

  6. This may seem like a stupid question, but how do you get started with using Calculon? I’m assuming a start with a standard Android test-project, but then were do I through Calculon? Do I drop it’s source files into the /lib directory of my test-project?

    Comment by Bob McCormick — February 19, 2010 @ 11:32 pm

    • Well, currently it’s just a bunch of source files which I pulled out of another project. I’ll turn this into a Maven project tho, so anyone can deploy it as a JAR. For now, I guess you’ll have to simply checkout the sources and either depend on them or drop them into your project.

      Note that this library is still a very early prototype. Its API will probably change.

      Comment by Matthias Käppler — February 20, 2010 @ 12:14 am

  7. Very nice – I’ve written a few tests for my app & they are so easy to understand. One question (and from the source it looks like you’ve tackled it) – how do you test orientation changes in a functional test? The following sequence passes for my activity:

    assertThat().inPortraitMode();
    assertThat(R.id.element_0).isGone();

    getActivity().setRequestedOrientation(ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE);

    assertThat().inLandscapeMode(); *
    assertThat(R.id.element_0).isGone(); **

    However, I know that ** should fail, since it’s a bug in the app. Also, setting breakpoints in the Activity onCreate I notice that calling setRequestedOrientation appears to be kicking off another thread to instantiate a new (presumably landscape) Activity. However, the test is not synch’d with this thread & runs to completion before that second thread completes (though the test still reports AUT in landscape at *)? I’ve also tried calling setRequestedOrientation in a runOnMainSync Runnable – same problem.

    Comment by Rory Douglas — February 21, 2010 @ 6:37 pm

    • Yeah, this whole orientation thing is a bit of a mess — and by that I mean not only in Calculon, but in Android itself. I’ve already spent several hours on this, but couldn’t find a way to programatically change the screen orientation (screen orientation is not even consistently modeled throughout the framework APIs!). What you’re doing here with setRequestedOrientation() probably has no effect other than setting a flag, because it’s just a request; the actual orientation is not necessarily changed immediately. Unless we find a way to actually flip the screen immediately, you can’t write a test like that.

      Comment by Matthias Käppler — February 21, 2010 @ 6:08 pm

  8. [...] DSL java tools that are coming out that make unit testing easier for  example Calculon, and the introduction blog post about  it.  But the android developer still needs to do unit testing on device or [...]

    Pingback by A Mobile Build System « MobileBytes — April 10, 2010 @ 10:41 pm

  9. u should check this Library..
    I was annoying with these heck test code…

    http://code.google.com/p/robotium/

    i found that’s quite useful. now i’m using it and also making my own methods..

    Comment by mooncheol — May 12, 2010 @ 10:17 am

  10. I’m not sure if you’re still working on this project, but it would be great if you would migrate some of your work to Robotium; namely the ability to verify that new activities were started from another activity.

    Comment by glenviewjeff — September 25, 2010 @ 3:36 pm

    • Yes we’re currently rethinking whether we will continue using calculon anyway or if we switch to robotium. We have a few fundamental hurdles with test automation on android and I’m curious whether or not robotium will fix that.

      Comment by Matthias Käppler — September 26, 2010 @ 12:42 pm

      • No robotium will not fix that as to get testing on multiple devices/emulators need to do test script at build level not library framework..

        I am in process of investigating Groovy/gradle to do that..

        Comment by Fred Grott — October 1, 2010 @ 1:20 pm

      • we’re using Maven to execute the builds. Some problems we had were related to the device keyguard, which prevents generic key events from working (e.g. key presses to bring up the menu). We had to disable it from within the app…

        In fact we’ve looked a bit closer at Robotium during the last Android UG here in Hamburg, but we were a bit surprised by some of the design decisions that have been made. One thing we found a bit questionable was the way Robotium accesses views (either by tree index or label). What’s the rationale behind this? Both are cumbersome and either difficult to read (tree index) or very verbose (labels). Why not use view IDs? That’s what they’re here for!

        Since you mentioned verification of activity launches. I believe that’s already part of Robotium (assertCurrentActivity())? So there’s probably little we could merge back from Calculon, since it’s very immature, but we’ll potentially switch to Robotium entirely and help improving it further (for instance, I found the ability to test ListViews a bit lacking, because Robotium assumes that a list element is always a single TextView, an assumption that rarely holds true in a half-way complex app :-)

        Comment by Matthias Käppler — October 4, 2010 @ 11:02 am

    • I agree that seems odd..strangeness in droid-sugar/roboelectric as far as choices as well..

      Maybe its me I am so use to either firing up android-Mock or roboGuice to grab what I need to test but its cumbersome as than i have to remember all the DI that I forgot if using DI..

      I am going to try an experiment with Calculon..just a polite fork to see if the path I want to travel on makes sense..let you know how it turns out..

      Comment by Fred Grott — October 11, 2010 @ 10:48 am

    • By the way, I’ve been giving Robotium a closer look lately, and while I think it’s awesome (no arguing about that!), I have decided against using it and will instead work on Calculon from here on. My reasons:

      – Robotium focuses on end-to-end testing, not on unit testing. It’s well tailored towards that, but ActivityInstrumentationTestCase2 is slow, and most things can be tested in isolation. Calculon focuses on unit testing instead, which is a lot faster and better suited for exercising TDD.
      – some design choices for Robotium I find questionable, if not flat out cumbersome. My biggest gripe with is that it cannot select a view for test by ID, but only by tree index or text. IDs are stable and easy to use, it’s simply beyond we why Robotium cannot do that.
      – last time I checked, ListView support was bad. It expected all list elements to be TextViews. Our app makes heavy use of list based layouts, so Robotium was essentially useless to test these Activities
      – lack of assertions. While the solo exposes a nice syntax, I find the most striking benefit of Calculon are the assertions (well, it’s completely modeled around them, so that doesn’t surprise). Robotium focuses on commands, while Calculon focuses on assertions. Maybe it’s a matter of taste, but I’d weigh the latter in higher than the former.
      – Calculon will not only focus on test case syntax, but also on general test structure, readability, and transparency, e.g. in form of annotations. For instance, I’ve just added the @ExpectsExtras annotation, which you can add to an activity unit test. It will assert that the test case starts the activity under test in the expected way. It’s a nice take on the document-by-testing methodology, and only a first step, more to come!

      Last but not least, I think Calculon will add to the variety of a very small landscape of Android testing libraries. It takes new approaches compared to Robotium, and I don’t see why I would throw all that away. That does of course not mean that it wouldn’t make sense to share code across the two libraries. I will soon start working on Calculon’s story testing (end2end), and I will likely end up borrowing things from Robotium, and I’ll gladly contribute things back in that process.

      Comment by Matthias Käppler — January 8, 2011 @ 5:47 pm

  11. [...] another test framework which name has to sound familiar: Calculon. There is a short introduction on the author's blog. I might look into that [...]

    Pingback by Android UI testing with robotium | united-coders.com — October 31, 2010 @ 12:46 pm

  12. [...] Robotium myself and it is really nice. Furthermore, the vanilla Android testing API just sucks, as better explained in this blog post over at Brain [...]

    Pingback by Random Thoughts form Øredev 2010 « Haskellville — November 14, 2010 @ 10:06 pm

  13. This looks really interesting. I absolutely love Mockito, and have been playing around with Robotium for a few days. It seems Robotium is focused on black box testing, and what is needed is a better solution for TDD.

    I just came across another project in the Android TDD space: Robolectric (http://pivotal.github.com/robolectric/). I need to do more research to compare and contract the two projects, but I’m excited about the possibilities.

    Comment by Shawn Lauzon — February 10, 2011 @ 12:07 am

  14. This is really a good test library for Android. Is it possible to test native Android applications such as Browser, Calendar, etc.?

    Comment by Leo — June 26, 2012 @ 7:46 am

    • No, this is meant for white box tests. But you may want to have a look at Robotium or Android NativeDriver, those can even test apps that you don’t developer yourself.

      Comment by Matthias Käppler — June 26, 2012 @ 5:53 pm


RSS feed for comments on this post. TrackBack URI

Leave a Reply

Please log in using one of these methods to post your comment:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

The Shocking Blue Green Theme. Blog at WordPress.com.

Follow

Get every new post delivered to your Inbox.

Join 37 other followers

%d bloggers like this: