I want to run any number of Android UI tests on each PR. Cost. Part II
Series of articles:
- I want to run any number of Android UI tests on each PR. Your actions? Part I
- I want to run any number of Android UI tests on each PR. Cost. Part II
- I want to run any number of Android UI tests on each PR. Existing solutions (BrowserStack, Firebase Test Lab). Part III
- I want to run any number of Android UI tests on each PR. Existing solutions (SauceLabs, AWS Device Farm, LambdaTest, Perfecto Mobile). Part IV
- I want to run any number of Android UI tests on each PR. Existing solutions. Part V
Hello everyone!
In my previous article, we revisited the importance of running UI tests on Pull Requests and discussed the necessary requirements for selecting the right solution. In the following articles, we will explore the various solutions available in the market and evaluate them based on the aforementioned requirements.
[DISCLAIMER]
At the outset, I must say that I hold the position of Co-Founder at MarathonLabs, the company whose solution (Marathon Cloud) is being discussed in this series. While I have attempted to remain impartial, I acknowledge that some of my views may be biased. Therefore, I would appreciate any feedback to ensure that this research remains objective.
Also, I must clarify that this study does not aim to provide an exhaustive evaluation of the products in question. Rather, it is based on my brief interactions with each one and reflects my personal opinions. Assessing factors like stability and scalability can be difficult when working with trial versions of solutions, as there may be insufficient data or time to arrive at definitive conclusions. Therefore, some of my conclusions may be subjective or inaccurate. Nonetheless, I trust that these articles will serve as a useful guide in navigating the complex landscape of UI Testing Infrastructure.
So, let me present comparing solutions:
- Marathon Cloud
- Firebase Test Lab
- BrowserStack
- emulator.wtf
- SauceLabs
- AWS Device Farm
- Perfecto Mobile
- LambdaTest
In this article, I recommend focusing on the cost of solutions. It's crucial to highlight the significant differences in the cost policies of each option.
Cost
Let's get started. It's best to take into account the pricing for certain teams that plan to implement UI Test Infrastructure for running UI Tests on PRs.
Teams, assumptions, and Purpose
So, let’s imagine that we have two teams:
- The first team consists of 5 developers and has 50 UI tests that they are going to run on each PR.
- The second team consists of 30 developers and 300 UI Tests.
Also, let’s make the next assumptions:
- Each UI test takes an average of 1 minute.
- Each developer produces 3 PRs per day.
- Each month contains approximately 20 working days.
- The infrastructure needs to be able to handle the maximum load when all developers decide to open pull requests simultaneously, which is a typical scenario at the end of the workday.
The main goal of the infrastructure is to run all PRs, including UI tests, in 15–20 minutes without any queues or delays.
Two price policies
There are typically two types of solutions available: paying a monthly fee for a device (parallel or job) or paying for the amount of time spent using it.
Pay for a device per month
When renting, the user can opt to rent the entire device for a month. If additional parallels are required, the user can rent more devices. Notable solutions that offer this type of rental service include BrowserStack, Perfecto Mobile, LambdaTest, and AWS Device Farm.
Pay for spent hours/minutes
When it comes to test running time, you only pay for what you use. There are various options available such as Marathon Cloud and emulator.wtf that offer unlimited parallels, which can be quite appealing. However, solutions like SauceLabs and Firebase Test Lab do have limitations on the number of parallels (25 or 50). AWS Device Farm has a limit of 5 devices that can be used simultaneously, but it is possible to request an increase in this number.
Policies comparing
We can calculate the costs for the two teams mentioned using two different options, each from a different pricing group. It's important to note that prices within each group are similar, so we only need to compare one option from each group. The two options we'll consider are BrowserStack and Marathon Cloud.
First team
General calculations:
- Each PR takes = 50 UI tests * 1 minute (average duration of a test) = 50 minutes.
- The number of parallels for one PR to fit in 15–20 minutes = 50 (minutes for one PR) / 20 (general time for one PR) = 2.5 => 3.
- The number of parallels to overcome the maximum load = 5 (devs) * 3 (parallels per PR) = 15.
- The number of minutes spent on tests running per day = 5 (devs) * 3 (PRs) * 50 (minutes per each PR) = 750 minutes.
BrowserStack cost:
- Official price = 199$ per parallel.
- Price for the team = 15 (number of parallels) * 199$ = 2 985$ per month.
Marathon Cloud cost:
- Official price = 1.99$ per hour.
- Price for the team = 750 minutes (the load each day) / 60 (minutes in an hour) * 20(number of working days) * 1.99$ = 497.5$ per month.
Second team
General calculations:
- Each PR takes = 300 UI tests * 1 minute (average duration of a test) = 300 minutes.
- The number of parallels for one PR to fit in 15–20 minutes = 300 (minutes for one PR) / 20 (general time for one PR) = 15.
- The number of parallels to overcome the maximum load = 30 (devs) * 15 (parallels per PR) = 450.
- The number of minutes spent on tests running per a day = 30 (devs) * 3 (PRs) * 300 (minutes per each PR) = 27 000 minutes.
BrowserStack cost:
- Official price = 199$ per parallel.
- Price for the team = 450 (number of parallels) * 199$ = 89 550$ per month.
Marathon Cloud cost:
- Official price = 1.99$ per hour.
- Price for the team = 27 000 minutes (the load each day) / 60 (minutes in an hour) * 20(number of working days) * 1.99$ = 17 910$ per month.
Conclusion
I want to point out that the prices calculated may not be the final cost since companies often give big discounts for teams. Nevertheless, it's important to comprehend the distinctions between the two cost policies. While paying for a device every month may work well for occasional runs or overnight builds, it may not be the optimal choice for running PRs.
In the upcoming articles, I will discuss each solution while taking into account the requirements mentioned in the first article. Additionally, I will determine the cost policy of each solution based on the information presented in this article.
