Kaspresso: The autotest framework that you have been looking forward to. Part I

The world is changing dazzlingly fast. The most valuable and important resource now is Time. Success belongs to those who can manage time better.

What does time mean in product development? You have to release your features as soon as possible, i.e. improve your time-to-market metric. I am sure you have heard this term, but let me remind you what it is. Time-to-market is the time during which a team develops a feature from the idea and delivers it to end-users.

How can we improve time-to-market? There are a lot of different options, and one of those is to reduce the time spent on manual regress tests. So, how to reduce lead time to release? Yep, just automate testing.

It’s so simple to say “Automate testing”. But, how to start?
The first obstacle you meet is choosing a tool.

You see a lot of different frameworks and tools for autotests. Currently, the main battle is between Appium and Espresso.
There is a big number of talks and articles devoted to comparing such tools, which you can read in your free time. In short, Espresso is better because of the following features:

• It’s a native tool. Google recommends it.
• Espresso is more stable and faster than Appium. This claim is justified by the absence of intermediate layers in Espresso.
• Espresso is more convenient for developers. A developer can write and debug UI-tests in Android Studio in the same project.

Sadly, Espresso is not so perfect and doesn’t satisfy all our needs. How come?

Readability

You will be like this man while trying to read such code.

What do you see? An API describing a simple function in a verbose manner. But this function just contains only three actions: searching a button, checking the button’s visibility, and clicking on the button. Can you understand it from this code? Sure, but you need some time.

Flakiness

The next unpleasant feature of any UI-test framework, not only Espresso, is flakiness.
Your test may succeed ten times in a row. But at the eleventh attempt, the test flakes. The reason and the point where the test fails could be absolutely undefined. And at the twelve attempt, the test is green again. All of this looks like black magic.

Logging

Beautiful logs and error messages provided by Espresso. This is a common error message. What happened? Ugh.
Also, you can’t observe what your test is doing during the run because Espresso doesn’t output any useful information.

Architecture of tests

When a team has written a bunch of UI-tests you notice a wide diversity of approaches to write UI-tests. Each developer == new code style. It’s a disaster.
In the world of autotests, there are no good general rules and architecture recommendations to write tests (almost).

Interaction with Android OS

Espresso doesn’t feature any tool to interact with Android OS. Espresso is useless if you desire to push some files to a device or simulate a phone call. Also, ADB is not possible in Espresso-tests. But Appium-tests can use it.

Upgrade scenarios

The architecture of Espresso-tests doesn’t allow you to automate upgrade scenarios (when you upgrade your app). You know that any interrupting of the application’s process fails the test. But, during an application’s upgrading, we stop and restart the application’s process.
That’s why the behavior of Android OS brakes the automatization of upgrade scenarios through Espresso.
Again, Appium can automate these scenarios.

Screenshotting

It’s an additional part that doesn’t relate to Espresso’s disadvantages directly. But, anyway, there are a lot of interesting points, hints and tricks here to simplify the process of producing screenshots, which is not as easy as you think.

All of this prevents you from writing clean, stable, maintainable and understandable UI-tests! And all teams who decide to write autotests are forced to struggle with these problems. It sucks.
That’s why we (developers at KasperskyLab, Avito, HeadHunter) have decided to join together to prepare a united library to address this lack.

Meet the new unique library — Kaspresso!

We’ve written a series of articles on how Kaspresso resolves your beloved tasks.
The first one is about problems with Readability, Flakiness, Logging and Architecture of UI-tests.

Let’s go!

Readability

To begin with, the Kakao library is going to improve readability. Kakao is a beautiful DSL wrapper over Espresso.
Just compare two pieces of code representing one test.
The first code is written with Espresso:

The second code is written with Kakao:

Pretty impressive.
You may notice the use of such things as MainScreen. MainScreen is an implementation of the Page Object pattern:

Shortly, Page Object describes views located on the screen. A more detailed explanation will come a little later. So, Kakao encourages us to write tests based on PageObjects.

Flaky tests and logging

Here, the real adventure begins. I want to draw your attention to the fact that there are no good and clean solutions to resolve flaky and logging problems. But we have tried to address this lack. What have we gotten?

As you remember Espresso doesn’t provide any mechanism to handle errors with a restarting of failed actions nor does it add automatic logs. Have a look at the entire scheme:

If we really wish to manage Espresso, then we have to put interceptors between the User and Espresso. They will be responsible for handling errors, restarting failed actions and logging, like on the following picture:

How to set interceptors here? What does it look like? To answer this question, let’s dive into Espresso!

Look at the Espresso test again:

I really like this API. So readable and maintainable. Pretty cool. (sarcasm =))
But let’s transform the code into a block-scheme to simplify the perception of Espresso:

Now, let’s consider the block-scheme step-by-step:

1. The onView method with ViewMatcher passed as the argument helps to get a special class describing the View with which we are working.
   The name of this class is ViewInteraction.
2. ViewIntercation is the most basic and important class in Espresso.
   All actions and assertions over a view are available only through the ViewIntercation class.
3. Actually, all actions and assertions over a view are available through two methods of the ViewIntercation. These are perform and check methods. These methods do a lot of different things with MainLooper, Views, Async operations and others under the hood. In arguments, we set implementations of simple interfaces: ViewAction and ViewAssertion by which we set rules for concrete action or assertion that we want to execute.

Now, let’s recall the initial scheme of the User-Espresso interaction:

When including the above interfaces and classes it transforms into:

I think you will predict the next step. Right, just wrap perform and check methods to manage Espresso:

This new intermediate layer is an ideal candidate where we can put our Interceptors. What are these interceptors and what do they look like?

We have implemented two kinds of Interceptors: BehaviorInterceptor and WatcherInterceptor.

BehaviorInterceptor

We shall now focus on the example with the viewIntercation.perform method. All we are going to discuss is absolutely identical for viewIntercation.check.

BehaviorInterceptor is the current Wrapper of viewIntercation.perform.
The Interceptor is responsible for:

1. Calling viewIntercation.perform as much as you need.
2. Handling the result of each execution of viewIntercation.perform.

The code is like this:

The first implementation behaviorInterceptor is the FlakyBehaviorInterceptor to overcome the flakiness problem. See the draft of FlakyBehaviorInterceptor’s implementation:

Pay attention to the action parameter. In our case, it’s viewInteraction.perform(ViewAction). The calling of action is wrapped by try-catch and do-while constructions which we can handle and manage Espresso’s behavior.

In case of error, flakyBehaviorInterceptor catches the exception and repeats the calling after intervalMs period. The number of attempts to execute the action is restricted by the timeoutMs parameter.

But we have not stopped with only the FlakyBehaviorInterceptor.

For example, another common failure is the nonvisibility of the view on the screen. You just need to scroll the parent layout to make your view visible. Surprisingly enough, Espresso can’t do that and throws an exception. That’s why the AutoscrollBehaviorInterceptor was designed.

One more cause of flakiness is the random appearing of android system dialogs, especially in real devices. To tackle it we have written the SystemDialogBehaviorInterceptor.

Keep in mind that the principle of described interceptors works like a Russian matryoshka. Have a glance at the image below:

FlakyBehaviorInterceptor calls AutoscrollBehaviorInterceptor, AutoscrollBehaviorINterceptor calls SystemDialogBehaviorInterceptor, SystemDialogBehaviorInterceptor calls Espresso’s code. But the handling of a result is going in the opposite way.

WatcherInterceptor

The second kind of Interceptors is WatcherInterceptor.
WatcherInterceptor can’t impact the behavior of Espresso. But it can get a lot of useful information about a concrete action. Such information is located in ViewAction. Thanks to this data you can build, for example, richer, more understandable and readable logs. So, how to get such information?

We have introduced the ViewActionProxy class which looks like on the following image:

The implementation’s draft of ViewActionProxy is:

As you see we just call all watcherInterceptors and send them all the needed information about concrete viewAction and view. After this activity, we call the original viewAction.perform.

Our default WatcherInterceptor logs each action and assertion of each view. Have a glance at the example below:

Brilliant, isn’t? The concept is really interesting and flexible. Nobody prevents you from adding additional interceptors and modify the common Espresso behavior. But, where are all interceptors located? How is it organized?

First, the full wrapper over Espresso is Kakao that hides all interactions with ViewInteraction, onView, and other inner Espresso things. That’s why we put the support of interceptors into the Kakao library (version 2.1).

Second, we have written a library that provides a simple and convenient way to manage interceptors; it gives a rich set of default interceptors handling flaky tests, improving the logging and such like. And you know the name of this library — Kaspresso.

See an example of how it works with the simple test written with Kakao:

How do I enable all default interceptors i.e. behavior and watcher interceptors?
You just need to add the following thing:

TestCase is a special parent class making all the magic. Here, our test has the handling of flakiness and can output more detailed logs. It’s quite sexy =)

Sweety cookies

The earlier described interceptors allowed us to implement some very attractive functions. See below.

flakySafety
It’s a method that receives a lambda and invokes it in the same manner as FlakyBehaviorInterceptor does. If you disable this interceptor or if you want to set some special flaky safety params for any view, you can use this method.

continuously
This function is similar to what flakySafely does, but for negative scenarios, where you need all the time to check that something does not happen.

compose
This is a method to make a composed action from multiple actions or assertions, and this action succeeds if at least one of its components succeeds. It is available as an extension function for any BaseView(base class for all views in Kakao), and just as a regular method (in this case, it can take actions on different views as well).

Architecture of tests

Have you ever had any recommendations on how to write conventional UI-tests? I am not sure you have. But when a developers’ team starts to write autotests without any code style and rules, then, the outcome can be unpredictable.
So, what do we offer? We are going to divide all the recommendations into two large groups: Abstractions and Test structure.

Abstractions

This part is built using a question-answer format.

How many abstractions can you have in your tests?

Only one! It’s a page object (PO), the term explained well by Martin Fowler in this article. As we discussed Page Object describes views located on the screen. It makes your code cleaner.
In Kakao a Screen class is the implementation of the PO. Each screen visible by the user, even a simple dialog, should be a separate PO.
Of course, there are cases when you need a new abstraction and it's ok. But our advice is to think well before you introduce a new abstraction. The fewer number of abstractions makes the possibility of involving testers and auto-testers into the process of autotest writing simpler.

Is it ok that your PO contains helper methods?

If these methods help to understand what the test is doing, then it’s ok.
For example, compare two parts of code:

and

I am sure that the method navigateToTasksScreen() is more "explicit" than the simple click on some shieldView.

Can Page Object contain inner state or logic?

No! Page Object doesn’t have an inner state or logic. It’s only a description of the UI of a concrete View. Remember about the Single responsibility principle.

Assert help methods inside of Page Object. Is it appropriate?

Experienced auto-testers who engaged in the automatization of tests on the Desktop application may remember about potential huge sizes of Page Objects in the case when a developer puts a lot of helper (asserts) methods inside the PO, as suggested in the article of Martin Fowler.

But Mobile and Desktop applications are different things. And, sure, the size of a screen on a Mobile and the size of a screen on a Desktop are different, too. That’s why we don’t see obstacles to add assert methods into PO.
To clarify the advantage of such an approach just compare three pieces of code:

The third code was inspired by Martin Fowler’s advice in the mentioned article.
To summarize, we are in favor of the first variant.

Test structure

Check this simple example:

We have two screens and three actions here. Can you predict how this test correlates with the test-case on which the test is based? How many steps are there in the test-case? Doubt.
What can a developer do here? The first way is to put comments:

Okay. But, we also wish to see what step is running in the logs of the test.
Let’s perform the code:

We have added logs in the test. Not bad.
Another desirable feature is to catch potential exceptions in each step. In the case of an exception, we want to output additional info in the logs and make a screenshot.
The current code transforms into:

Ugh. You see how your simple test converts into a real mess. The maintenance of such code is too hard.

That’s why we have created a special kind of “Kotlin” DSL to carry out all our wishes leaving the code without significant changes.
This DSL looks like this:

The method step is doing all what we have discussed:

• auto logging
• exceptions’ catching
• screenshot after an error
• etc.

Now, your test consists of separate and logical steps with understandable names and the huge logic under the hood:

Just evaluate logs step function produces:

Great!

Additional features of our DSL

During the automatization process, the DSL was evolving and improving. What else did we put into the DSL?

Scenario

“Scenario” is a group of steps that are repeated in almost all tests. An example is below:

Inner step

Sometimes, it would be useful to highlight each assertion in one step or to do something similar. In this case, you can use “inner step”:

Environment preparation

This part is a continuation of the previous part too. But we are going to consider a slightly different problem.

Very often you have to prepare your device or the environment before a test run. For example, you need to turn off the network before the test. What can we do?

We have added the zero-step to turn off the network.
The key topic here is to restore the device environment to its original state. We strongly recommend restoring the device’s environment after each test to avoid potential flakiness for future tests.

To be honest, these two steps are not part of the test-case. They only concern the environment. Such cases pushed us to extend our DSL to the following form:

Much better! The DSL provides before/after methods to handle the environment state before/after the test and provides a run method where we put all the steps of the test.

What else at DSL?

Please, read the documentation where you can find a detailed explanation of other features and possibilities.

Conclusion

After years of sweating on testing, you can safely reduce lead time without fear your manager will strangle you if you overlooked a bug.

Be free to highlight interesting topics for you.
Naturally, you need to have a look at the Kaspresso library.
And be happy while coding autotests in Android =)

What’s next?

In further articles, we are going to consider:

• Interaction with Android OS (ADB restoring)
• Automatization of upgrade scenarios
• Screenshotting
• Maybe something else =)