Testing mobile apps in the cloud

There are several services available for testing mobile apps in the cloud on real devices:

Sauce Labs

Sauce Labs is the veteran and was co-founded by Jason Huggins, the original creator of Selenium.  Sauce Labs also works on development of Appium.


BrowserStack is an economical choice for testing in the cloud, but while the offer “Live” mobile devices (manual testing only), it does not appear that they offer devices for test automation.

Xamarin Test Cloud

Xamarin is a pretty good contender, but focuses their tooling around Microsoft .NET / Visual Studio developers.  It does not appear that Appium is supported (directly) Still, they have a big selection of devices for those willing to stick to Microsoft tooling. It is also nice that their metering is a soft limit.

Perfecto Mobile

Perfecto Mobile has been around for a while as a service for Enterprises.  It is proprietary, but also supports Appium.  It comes with an Enterprise level price though.  You have to call and talk to a salesperson for any plan beyond proof of concept.

Keynote Mobile Testing

Keynote Mobile Testing used to be called Device Anywhere and their experience dates back to the days before smart phones.  I don’t have recent experience with them, but they used to offer only pixel based automation.  It appears they support Appium now.  But automation is not supported outside their enterprise sales channel.

Amazon Device Farm

Amazon Device Farm is the most recent entry in the field.  But it is not only quite expensive, but requires a lot of do-it-yourself.  Perhaps it will be more practical once tooling for recipes becomes available.


Price Comparision of Sauce Labs, Xamarin Test Cloud, Perfecto Mobile, & Amazon Device Cloud

Service Price Device time Concurrent Devices
Sauce Labs $129/month* 1000 minutes 4
$259/month* 2000 minutes 4
399/month* 3000 minutes 6
$499/month* 4000 minutes 8
Xamarin Test Cloud $99/month** 1 hour/day 1
$379/month** 5 hours/day 3
$799/month** 10 hours/day 5
Perfecto Mobile $399/month* 20 hours 1
Amazon Device Farm $0.17/minute 1
Amazon Device Farm $250/month unlimited 1

* 25% annual discount available

** 15% annual discount available


Setting system properties with Gradle

In Java you can pass system properties from the command line like this:

java -D MyProperty=foo MyClass

And you can then get them in your code like so:

public class MyClass {
  public String getMyProperty() {
    return System.getProperty("MyProperty");

Pretty easy, no?

You can pass system arguments the same way with Gradle:

gradle -D MyProperty=foo run

But what if you want to manipulate or use those properties in Gradle first?

Instead of -D you can use -P to pass properties to the Gradle object, and then you can do whatever you want with it.

gradle -P MyProperty=foo MyClass

And then you can use your properties in Gradle and then pass them to the System properties thusly:

task setProperty << {

    if (project.hasProperty("ENV")) {
        println "project has property ENV"
        System.properties["ENV"] = "$ENV"
    } else {
        println "project does not have property ENV"
        System.properties["ENV"] = "dev"

    println "ENV: " + System.properties["ENV"]


Some references:






When should you use BDD

1. Product and dev are talking to each other
2. You want to write something down to communicate
3. You want to read something to communicate
4. You don’t use documents as a way to avoid communicating.
3. You can come up specific examples of requirements
4. You are willing to test those requirements frequently
5. You will pay attention to the results of those tests
6. You are willing to update those requirements when they change
7. You want to execute tests automatically
8. You have an environment and data that enables you to test changes immediately.
9. You have a build and deploy process that enables you to deploy when any change is made.
10. You don’t think BDD is a way to solve any of these problems.

Who wants to write unit tests?

Who wants to write unit tests?

Not me.

It’s like asking who wants to eat healthy. We all know (or suspect) that it’s good for you, but what are the real benefits?

Writing unit tests is extra work, and not very fun.  And if you start with an existing code base that doesn’t have them, it can be intimidating and frustrating to get started.  And it isn’t always easy.  You might have to spend significant time refactoring and writing mocks and other test doubles to be able to test your code.


If you’re skeptical of the value of unit tests, come share your reasons.

If you write unit tests but find they don’t have any real value (or not enough to justify the costs) then let’s talk about why you do it.

If you want to write unit tests, but can’t find the time, or feel like you need help getting started, let me know, and I’ll be your personal trainer.  Book some 1 on 1 time with me.

If you love writing unit tests and the productivity boost and code quality gains they provide, don’t bother coming, unless you want to help convince others.

There are real challenges to writing unit tests but don’t let procrastination or fear be one of them.


I’ll be hosting a webinar about unit testing today at 1pm at Hart.com and next Wednesday at 1pm Pacific for everyone online.

Here’s the hangout url https://hangouts.google.com/hangouts/_/one-shore.com/whowantstowriteunittests?hl=en&authuser=0

Ping me if you want an invite in your calendar.

Standardizing tools can lead to testing “synergies”

It’s everyone’s favorite square on buzzword bingo.

The one with the word “synergies” written on it.

But just because managers and consultants overuse a word, doesn’t mean it doesn’t have real value.

I’m talking about standardizing testing tools.

I was just talking with someone about a consulting opportunity at a large company. Their challenge is that they have a bunch of cobbled together testing systems for different teams. They need to build it all under one system. The list of skills needed is miles long:

Java, Javascript, Python, Selenium, Protractor, QUnit, JUnit, TestNG, iOS, Android, Maven, Gradle, Puppet, Chef, Docker, Vagrant, VMWare, Angular, Backbone, …

My response:

It looks like the first big challenge will be standardizing tools.

There could be a lot of easy wins by picking, for example:

* Maven or Gradle for builds
* Puppet or Chef for deployments
* JUnit or TestNG for test runner

That could be quite a challenge for teams who “like what they know” and don’t see the value of change.

But if there are projects with neglected or troublesome build/deploy/test cycles, then standardization could be the price of admission into a concierge service that builds their code, runs unit tests and static analysis, deploys it to a test environment, and runs system & integration tests. The biggest challenge I’ve seen is getting permission and resources for the test environments, and it can be like pulling teeth to find out all the dependencies & data that need to be assembled (or mocked). At the end of it, though, there’s a virtual cycle that can’t be stopped.

And when you have standard tools in place and a smooth build/deploy/test in place, then teams really can start leveraging shared tools.

Synergies achieved. Without layoffs. BTW, they’re looking for additional SDETs in Seattle. Let me know if you’re interested in tackling a big project.

How to reduce the number of mocks for testing

We all know that testing objects in isolation is a good thing. (You should also test objects interacting, but that’s a different subject.)

Testing in isolation allows us to concentrate on testing a behavior without introducing additional complexity. It also allows us to strictly regulate the external state and eliminate potentially unexpected behavior. To accomplish this, we often turn to mock objects.

Mocks can do two main things:

  1. Isolate the object under test from its dependencies
  2. Isolate the object under test from external communication

Let’s say we want to make a self-driving car. We’ll start off with an existing vehicle.

car = new Car(make, model, year)

The first thing it needs is a robot driver.

car.driver = new RobotDriver()

Since the robot driver doesn’t need to take up a seat, we’ll throw him in the trunk so we can have another passenger. And we might as well remove the steering wheel, pedals, and shifter to make room for him. (Assume our robot driver will not malfunction so there is no need for a manual override.)


Now we can get to our first test. We want to check how many passengers can fit into our car now.


For our purposes, the driver doesn’t count as a passenger.

We probably should have checked the capacity before ripping out all that stuff, and then we could write a more flexible test. We’ll also create a factory that can transform a regular car into a self-driving car.

car = new Car(make, model, year)
initialPassengerCapacity = car.passengerCapacity()
selfDrivingCar = RobotCarFactory.transform(car)

All well and good, but we’ve done a lot of hand waving. Of course, there is no such robot car factory, and no robot driver (yet). That shouldn’t stop us from testing. And that’s where mocks come in.

What does a driver do?  First we need to define an interface.

interface driver {

Now, as long as both a human driver and robot driver can do all of these things, they can satisfy the interface and be substituted for one another by our car.

But you might remember that we don’t actually have a robot driver. (Developers say it should be ready by next week, but they’ve been saying that for a long time.) So we create a mock:

robotDriver = mock(driver)

And now we can test it.



You can see pretty explicitly here that a driver is a dependency of the car (if we want it to go anywhere.) In this case, our mock robot driver could just be a person with a remote control.

Our goal here is to show that our car works without a human driver and that there is room for an extra passenger. (So that we can obtain additional funding from our investors who want an edge in the ride-sharing marketplace.)

Stay tuned for part 2.

Using GVM to manage Go (golang) versions

I’m testing microservices build with Go [https://golang.org/] lately. So I need to set up a working Go environment.

On a mac, it’s as easy as `brew install go`, but on CentOS `yum install go` will give you an outdated 1.4 version of go.

There are a few tricks to using Go, but the basics are to set two environment variables: GOROOT and GOPATH

GOROOT is strictly not necessary (unless you’re dealing with multiple versions of go, which is officially not recommended.)

GOPATH is kinda like setting your PATH or JAVA_HOME, but GOPATH is additionally tricky because it determines where your dependencies are a installed, so it’s really more a bit like setting M2_HOME (if you use maven.)

Go `go get` is aware of git & mercurial repositories, it can install packages from github for you, but it is only semi-aware — in that it doesn’t understand versions. So you either need to fork each dependency’s repo at the version you need (and merge when you need updates) or just freeze it. So each project needs it’s own GOPATH (and a branch with each depndency tree.)

Enter GVM. Kinda like RVM, NVM but for Go. Here are the steps I use to set it up:

# install go dependencies on CentOS 7.2
yum install curl-devel expat-devel gettext-devel openssl-devel zlib-devel gcc perl-ExtUtils

# throw in the dev kitchen sink while your at it
yum groupinstall "Development Tools"

#install GVM (https://github.com/moovweb/gvm)
bash < <(curl -s -S -L https://raw.githubusercontent.com/moovweb/gvm/master/binscripts/gvm-installer)
source ~/.gvm/scripts/gvm

# install go 1.4 (to be able to build later versions)
gvm install go1.4
gvm use 1.4

# now I can install go 1.6
gvm install go1.6
gvm use 1.6

# create a packageset for my project
gvm pkgset create mygoproject
gvm pkgset use mygoproject

# finally, I can get some work done and build my Go project
git clone git@github.com:fijiaaron/mygoproject.git
cd mygoproject
go install