How to Build in House Test Lab
Content
Share this eBook!
TABLE OF CONTENT
WHY YOU NEED AN IN-HOUSE TEST LAB
4
HOW TO CHOOSE DEVICES AND HOW MANY OF THEM
9
REQUIRED INFRASTRUCTURE 14
ENTERPRISE GRADE RELIABILITY 20
OPERATING YOUR IN-HOUSE TEST LAB 24
Share this eBook!
Introduction
Building a large-scale in-house test lab from scratch can be a big challenge. It takes time
and efforts to figure out the right equipment and tools that are needed for a successful
deployment. In addition, it can be trickier if something is not properly configured.
At Testdroid, we understand how daunting and time-consuming it can be. And we have
experienced it ourselves when we were building Testdroid Cloud. To help customers
with different needs, we have been building various in-house test labs ranging from 20
devices to several hundred devices with Testdroid Enterprise.
Here, we’ve gathered and put together a guide that discusses about why you need to
build an in-house test lab, what and how many devices you need, what infrastructure
is required, how to create enterprise grade reliability, and how to operate your test lab
without failure. In fact, the tips we cover in this ebook are mainly influenced by our own
setup at Testdroid Cloud.
With the help of this ebook, you will be able to create the best offline test lab with the
right setup that maximizes your return on investment, instead of spending tons of time
correcting the wrong configuration back and forth.
Share this eBook!
Chapter 1
The Reasons
for an In-House
Test Lab
Share this eBook!
Top Reasons for In-House Test Labs
The very first thing you should tackle is probably to make sure whether
or not you need to set up one in-house test lab. Before simply jumping
into its construction, you must know the reasons why in-house test labs,
rather than cloud based test automation services, are the best options to
fulfill your needs. The most typical reasons you should take into account
are:
PREMARKET DEVICES
Surprisingly many of top tier developers work
with pre-market devices. Typically you are working on a significant enough app that will ship with
a device and you have to make sure that your
app works even when the actual device software changes from release to another. Typically
you are working inside or together with a device
manufacturer, carrier or one of the TOP100 app
companies that do get access to pre-market devices from top manufacturers.
CORPORATE SECURITY POLICIES
Like many other companies, you probably
have strict security policies regarding uploading development versions of your unreleased applications outside of the corporate firewall. Quite often your company is
also very professional with your testing
and setting up your own in-house device
lab makes a lot of sense to get the scalability and
time to market benefits of test automation on
real devices while maintaining required security
levels.
Share this eBook!
AVAILABILITY AND USAGE LEVEL
Sometimes your QA teams need to run
very long soak tests that take all night or
there are tens of development teams in
your company relying on the availability of
the test results every day so the shared infrastructure of public cloud based services
does not fit well to your usage profile. If
you are running a lot of tests (potentially
from every code commit), your tests take
a long time to execute or you are relying
on having the test devices available when
you need them an in-house device lab is
the way to go.
SPECIAL SETUPS
“
Your company might also
by far the
is
b
la
e
s
u
o
-h
in
An
require specific hardware
cure setup
e
s
d
n
a
le
ib
or environment setup to be
x
e
fl
most
s for wide
te
a
able to fully test your softd
o
m
m
o
c
c
a
and it
ware. For instance some speardware seth
t
n
re
fe
if
d
f
o
cial Bluetooth or WIFI enabled
spectrum
rent usa ge
fe
if
d
ly
d
il
w
s
a
ll
accessory needs to be physicalups as we
uous testn
ti
n
o
c
ly in the same location with the
/7
4
2
m
o
rofiles fr
test devices or the devices need p
access to
te
o
m
re
g
in
id
v
ing to pro
to be in some specific location or
prototypes.
in some specific carrier network.
specific device
Many of the environment related
characteristics can be simulated but
not all and therefore this sort of special cases need to be run in specific inhouse labs.
“
As you can see the reasons for setting up in-house
device labs can be as varied as the companies that are setting those up. An in-house lab is by far the most flexible and
secure setup and it accommodates for wide spectrum of
different hardware setups as well as wildly different usage
profiles from 24/7 continuous testing to providing remote
access to specific device prototypes.
Share this eBook!
The Core Benefits of Continuous
Integration & Testing
One essential tool for mobile application developers to use in order
to automate their testing effort is continuous integration software –
for example Jenkins, which is currently the leading open source
continuous integration (CI) server monitoring executions
of repeated jobs. With continuous integration and
testing, you can reap the following benefits.
VERSION CONTROL AND BUILD REVISION
This rather old practice in software industry is the
same with mobile application development. The
quicker you can ensure your additions and regressions to builds work properly, the more quickly you
can move forward. On the other hand, if you face
problems or find any issues, all attention to fixing
those bugs can be drawn there right away. Fixing a
bug today costs much less than fixing the very same
bug tomorrow.
BUILD AUTOMATION, INSTANT EXECUTION
AND TESTING
Build Automation refers to the ability to either automatically trigger or build a clean version of your application.
You can take this further and make your systems built from
source code just committed to version control. This typically
shortens the build time significantly, but also makes the application ready for execution and testing instantly. This is one of the
greatest benefits that continuous integration can bring to you.
Share this eBook!
ALL TYPES OF TESTING ENABLED
By automating mobile application testing you can ensure that your application behaves correctly on every mobile OS version & build (30+ with
Android, 10+ with iOS), which will reduce costs and increase efficiency
of the overall testing process. Not only the basic unit and GUI testing,
but testing in continuous integration environment provides a way to
adopt any test frameworks, methods, complementary technologies and so on, for your continuous testing and
development environment.
“
“
inuous
t
n
o
c
g
ualin
i
q
t
e
p
h
o
t
d
g
a
By
mprovin
i
y
l
n
o
your
t
o
g
n
n
i
e
k
r
a
a
t also m
ion you
u
t
b
a
r
,
s
g
s
e
t
e
c
n
i
pro
er, more
t
d
s
n
a
f
a
h
e
c
d
r co
ffort mu
e
g
n
ty of you
i
t
s
asure.
and te
e
t
n
m
e
o
t
m
p
r
o
e
devel
nd easi
a
t
n
e
t
s
i
cons
FREQUENT COMMITS, CODE CONSOLIDATION
-> FAST BUILDS
Time is money. And fast builds give you that time. Fast builds allow you
to receive test results (= bug report) faster.
CONTINUOUS INTEGRATION/TESTING IS THE RIGHT WAY
TO GO FORWARD
By adopting continuous integration you are not only improving the
quality of your code and process, but also making your development
and testing effort much faster, more consistent and easier to measure.
These are significant benefits to be gained out from simple process
change and adopting something that already exists in processes.
Share this eBook!
Chapter 2
Devices for
In-House
Test Lab
Share this eBook!
Decide the Size of Your Device Pool
Today the number of different device variants, OS versions, resolutions and form factors is really large and to
make matters worse a lot of these different devices have
shipped tens of millions units making each of them significant in their own right. This problem is not limited
to only Android anymore but also on iOS. Probably you have already over 10 unique hardware and resolution combinations and 3-4
active OS versions, making the iOS device
matrix fairly sizeable as well.
So how many devices you need
to provide wide enough market
coverage? For Android you take a
great data published by OpenSignal
and from here you can see that for instance if you want to cover 50% of the
global market you will need about 60
devices, about 20% global market coverage can be reached with 12 devices
and if you want to go over 50% you will
need significantly more devices.
If you focus mainly on US market you can
find great and fresh information published by
app development company Double Encore. According to this data with well-selected 25 Android devices you can cover two-thirds of the Active users and for
90% US market coverage you will need 128 devices.
For estimating the number of needed different iOS devices, you can take a look at the data published by Apple.
In May 2014 there are 8 unique iPhone models, 7 unique
Share this eBook!
iPad models and 5 generations of iPod Touches, totaling at 20
unique devices.
Given Apple’s user base you can potentially safely leave out iPhone 2G, 3G and 3GS as well as the first three generations of iPod
Touch, since the number of these devices in active use is most
likely fairly small and especially these were introduced 5-7 years
ago and have not even been in production for several years. This
leaves 15 unique and still significantly represented iOS devices in
the market.
On iOS you also need to remember that while most of the iOS users do always update to the latest version, there are always some
users who don’t. For instance now about nine months after the
introduction of iOS 7, there are still 12% of users who have not
updated to the latest version. Although this is nothing close to
the situation in Android land, it is advisable that you keep some
iOS devices with the second latest iOS version as well.
Share this eBook!
Select the Right Devices for
Maximum Coverage
Now that we know how many devices you need for both Android and iOS, let’s take a bit
more detailed look at which devices are the correct ones. Assuming your target market
is North America we can again use Double Encore’s data as a good yardstick. So the list
of 25 most used Android devices looks like this:
1.
2.
3.
4.
5.
Samsung Galaxy S3 (17%)
Samsung Galaxy S4 (12.3%)
Motorola Droid Razr (3.7%)
Samsung Galaxy Note 3 (3.7%)
Samsung Galaxy Note 2 (3.3%)
6.
Asus Nexus 7 (3.2%)
7.
HTC One (1.9%)
8.
Samsung Galaxy S2 (1.8%)
9.
Samsung Galaxy Tab2 10.1 (1.6%)
10.
14.
Samsung Galaxy Note 10.1 (1.2%)
15.
LG G2 (1.1%)
16.
Samsung Galaxy S (<1%)
17.
Motorola Droid X (<1%)
18.
Motorola Droid Bionic (<1%)
19.
Samsung Galaxy S5 (<1%)
20.
Samsung Galaxy Nexus (<1%)
21.
LG Nexus 5 (<1%)
Droid Razr HD (1.4%)
22.
Motorola Moto X (<1%)
11.
Droid Razr M (1.4%)
23.
Samsung Galaxy Tab 3 10.1 (<1%)
12.
Samsung Galaxy S2 Epic (1.3%)
24.
LG Nexus 4 (<1%)
13.
Samsung Galaxy Tab 2 7.0 (1.3%)
25.
HTC Thunderbolt
Share this eBook!
Respectively if you want to cover the most used iOS devices, the list looks like this:
1.
2.
3.
4.
5.
iPhone 5S
iPhone 5C
iPhone 5
iPhone 4S
iPhone 4
8.
iPad Air
9.
iPad 4th generation
10.
iPad 3th generation
11.
iPad 2nd generation
12.
iPad 1st generation
6.
iPad Mini
13.
iPod Touch 5th generation
7.
iPad Mini 2nd generation
14.
iPod Touch 4th generation
The Frequency of Updating Your
Device List
Once you have your lab up and running, you need to think how often you should add
new devices to keep your market coverage. For iOS devices it is simple, as you only need
to add new devices when Apple announces new models (twice a year). But on Android
it is a bit more complex than that. A good rule of thumb is to check every quarter what
are the latest blockbusters (especially any new Galaxy devices) and add those to your
lab. You cannot really substitute the older ones because, shown in the TOP25 Android
device list, even 2-3 year old number one devices (think Samsung Galaxy S2 and Samsung Galaxy S) have such a huge installed base that they will hang on the list of most
used devices for years.
For Testdroid Cloud, we always expand our device pool with the latest Android and
iOS device models when they are available on the market
Share this eBook!
Chapter 3
Required
Infrastructure
Share this eBook!
Device Control Servers
After selecting your target devices, you need to take a look at other parts of a reliable
infrastructure. Regardless of the technology you use to build your device lab, you
will need some sort of device control node servers that will take care of managing
devices and possibly executing tests. Additionally these servers can collect, process
and store test results and do other test execution related tasks. At Testdroid we use
either generic PC-hardware running Ubuntu Linux for controlling Android devices or
Mac Minis for controlling iOS devices. Because we use the OS vendor provided methods to control the devices (Android Debug Bridge (ADB) on Android and Instruments
on iOS) the control nodes/servers need to have decent number of free USB ports as
each device will be connected to the device servers with USB cable.
Share this eBook!
In order to have stable test execution infrastructure you need
to take following things into account when selecting the
hardware for device control servers:
HARD DISKS
You need to have enough disk space for all test artifacts (logs,
screenshots, videos, performance data, memory dumps, network dumps etc.) and make sure that there is a mechanism that
either deletes the data created by previous test runs or transfers it to somewhere where there is enough storage space.
Running out of disk space is one of the most common failure
points in large-scale test automation. You also may want to
consider investing in SSD disks to make sure that writing the
test results to storage does not become a bottleneck
under heavy load.
RAM
You need enough RAM to run several virtual
machines in one physical device control server
as that is the best way to ensure that each test
run starts from identical starting point. Also running virtual machines is required to run several iOS
Instruments sessions on one Mac.
ENERGY EFFICIENCY
If you are building a large device lab, you will
potentially running tens of device control servers as well. So keep in mind that the more power your hardware draws, the more heat those
will generate and more cooling you will need.
Share this eBook!
USB Hubs
As all devices will be connected to the device control servers over USB, you need to pay
attention to the quality and capabilities of your USB hardware. This is one of the most
error-prone parts of the system. While there are no fixed hard limits on how many
devices you can plug into one USB, our experiments show that you should not connect over 16 devices per USB hub. If you connect more than 16, the devices may start
disconnecting randomly especially when a lot of data is being transferred either at the
beginning or end of each test.
The other important reason why good quality USB hubs are essential is that many devices (especially ones with a large display) have
problems with charging if the charging current is not high enough. The USB specification
mandates that the output current is limited to
500mA when the USB port is on data mode but
the best USB hubs on the market allow you to
programmatically switch between data and
charging modes allowing you to implement automatic charging cycles for devices that cannot
be charged in USB data mode.
QUICKTIPS!
To make sure that there are as
few USB related problems as
possible we prefer professional
USB hub hardware by company
called Cambrionix. Those are sold
under several names and at least
ones sold under Datamation brand
are great. Just be careful when
selecting those as some models are
just for charging and
some models are
just for iOS device.
Share this eBook!
WIFI Infrastructure
WIFI infrastructure is another very important area that is often overlooked when creating large-scale test lab. You can get to about 10 maybe even 15 devices with any WIFI infrastructure without any problems. But as the number of devices in your WIFI network
adds up, so do the problems. Quite often all devices are able to connect to the WIFI
without problems but the problems appear when all of those start transferring data
at the same time. Most WIFI access points are not designed for this kind of throughput
and you will start observing different kinds of timeouts on server responses etc. Network throughput on typical WIFI router is around 100-150 Mbit/s. But when you have
several dozens of wireless devices trying to download at tens of Mbits/s, you can easily
see where the bottleneck is.
QUICKTIPS!
There are several brands on the market
selling enterprise grade WIFI access
points. At Testdroid, we have tried
at least Aerohive, Cirrus and
Cisco Meraki. Our current
favorite is Cisco Meraki MR24 which has
three separate radios each capable
of working on either
2.4Ghz or 5Ghz frequency band and each
of them can transfer up to
300 Mbit/s giving whopping
900 Mbit/s from your devices to
the back end servers. Another nice thing
about Cisco Meraki WIFI infrastructure is
their cloud based management UI that has
features like throttling the speed of individual WIFI connection which can be useful for instance when simulating different
network conditions.
Share this eBook!
Integration to Your Build Process
The next important consideration is how you integrate your in-house device lab to your
build process. Most likely you have already a separate Continuous Integration server in
place and you have to make sure that without any manual interaction your test infrastructure can be controlled by your CI server. This means that everything needs to be
controllable over an API so that your CI jobs can seamlessly both trigger test runs and
use the test results from your in-house test lab to trigger further actions such as deploying your app to production.
Continuous
integration
Source Management
USER INTERFACE
API
application
server
API
UI
Test
execution
DEVICES
Test
management
reporting
admin
monitoring
users &
roles
DATABASE
Access to Your Organizations
Centralized Access Control System
Last but not least, if you are building your in-house device lab in an enterprise setting
you need to be able to connect to your company’s centralized user right management
system in order to validate, grant and revoke user rights for each user in your test automation system. This is just to ensure that any unauthorized users are not able to access
the data and binaries that they are not supposed to and to synchronize these access
rights when people leave the organization, change roles or simply do not anymore need
access to such data. Most common protocols for doing this are LDAP (Lightweight Directory Access Protocol) and OAuth. And your infrastructure should validate each user
login through such centralized system.
Share this eBook!
Chapter 4
Enterprise Grade
Reliability
Share this eBook!
Always Run Tests on Clean Devices
In addition to hardware infrastructure, you need a reliable software configuration for avoiding any test failures. One of the most common
causes for flaky test results is that the environment where you
run your tests i.e. your test device is not exactly in the same
state during each test session. This means that there may
be different processes running, different applications
or background services installed, different amount
of free memory or different amount of storage
space available. In order to avoid test flakiness,
you should make sure that there is a proper
cleaning cycle between every test session that
uninstalls all unwanted apps, sets the device
storage to the same state as it was before
the test and reboots the devices so that
there are no processes running from the
previous test session.
Create Intelligent
Retry Mechanisms
Some of the test execution related failures are
not actually related to the application under test
but are caused by some test infrastructure related
reason. The good news is that this type of failures are
easy to identify and tests that have failed because of
such reasons can be automatically retried as many times
as needed to have a test run that is free from infrastructure
related failures. The most common reasons for test infrastructure
related failures are: The connection between the device control server
and device fails, the connection between the device and your back end server fails,
either the device or device control server runs out of storage space, device runs out
of power in the middle of test session, and etc. All of these can be identified during
the test session and the tests can be automatically retried so that the only remaining
failures are real failures related to the application under test or the test scripts.
Share this eBook!
Check and automatically reconnect
the USB and Wireless data connections
Losing either the USB or wireless data connection at some point is unfortunately very
common on both Android and iOS devices. To make matters worse quite often, the
device reports that it has live connection over USB, WIFI or Wireless data but no data
is moving. You need to be able to automatically verify that both
USB connection and WIFI connection are actually transferring data and if that’s not the case, you need to
automatically disconnect & reconnect the connections to get them back up again.
Automate All Configuration Changes
When you have hundreds of devices and tens of device control servers in your test lab,
you cannot (and should not) really make any changes to settings or configurations manually because the risk of not doing the changes identically on all devices or servers is
simply too high. We automate everything we can by using Opsworks Chef and we have
implemented our on-device services that take care of changing the device settings etc.
Even if we have been very systematic with this, every now and then there are problems
because some small setting was done by hand and it was not replicated identically on
all devices or servers. This is one area where you cannot afford to cut corners if your
goal is 99.99% reliability of your in-house test lab.
Share this eBook!
Use Professional Monitoring
The last but not by any means least important way to ensure the robustness of your
in-house test lab is to set up a professional monitoring for all aspects of your test lab
infrastructure. The most important areas to monitor are:
DISK SPACES
– Automated tests create a lot of data in the form of logs, screenshots, videos, memory
dumps, and network dumps. When you are running tens or even hundreds of devices
sometimes 24/7, you may run out of disk space just out of the blue.
NETWORK CONNECTIONS, LATENCIES, PACKET LOSS RATES
– These have an impact on everything in your test lab and once you start troubleshooting the strangest and most randomly appearing problems, most often the root cause is
networking related.
POWER
– Disruptions on power supply (even very short ones) can cause very strange problems
when the system has lots of moving parts and all of them are dependent on steady
power supply. Use UPS backed power whenever you can and also make sure that your
servers close down and come up in a controlled way when longer power disruption
happens.
You can use any generic monitoring software for this. For Testdroid Cloud we are currently using AWS monitoring, Loggly and New Relic to handle our monitoring.
Share this eBook!
Chapter 5
Operating
In-House
Test Lab
Share this eBook!
g
o
L
l
a
n
o
i
t
a
r
e
p
O
t up
Once you’ve se
cftware infrastru
so
d
n
a
re
a
w
rd
ha
lab is the
g your device
n
ti
ra
e
p
o
,
re
tu
ke care of. The
ta
to
d
e
e
n
u
o
opnext thing y
s is to save all
n
o
ti
ra
e
p
o
f
o
docfirst aspect
teams should
n
o
ti
ra
e
p
O
s.
to
eration log
s that happen
ce
ie
p
d
n
a
s
it
b
runs.
ument all
vices and test
e
d
t,
n
e
m
n
o
ir
cs,
the env
ework with do
m
o
h
t
a
re
g
g
in
By do
and
le to maintain
b
a
e
b
l
il
w
s
m
tea
hen
nvironment w
e
e
th
e
d
ra
g
p
u
.
needed easily
Keep You
Devices C r
oo
l
Everyt
power
hin
n
o
waday g is about
its con
s. T
ant th sumption is he power a
ing to
contrib the most im nd
of the
po
u
tempe system. Eve te the tempe rtns
rat
ra
malfun ure can cau maller chan ture
ge
se
ct
estima ion (This sho that lots of s in
device
te
u
s
up the d!) and go o ld never be
u
ffl
device
n
i
d
n
e
e. Whe
poo
enoug
n you rh spac l, make sure
set
e for t
y
o
u
l
e
ho
a
“breat se devices t ve
h”.
o
e
h
t
r
o
t
i
n
o
M
s
e
c
i
v
e
D
f
o
g
Chargin
ke
need to ma
y
ll
a
r
tu
a
n
You
ugh power
o
n
e
t
e
g
s
evice
4/7
vices are 2
sure your d
e
d
e
s
o
th
evic. Once
gly, those d
to perform
in
is
r
p
r
u
s
, maybe
if they get
d
e
r
o
it
n
o
plugged in
tly m
are compo
be constan
w
d
r
a
h
s
a
y
es need to
atter
sily
wer. The b
ing and ea
o
g
r
p
a
h
h
g
c
u
ll
o
fu
n
e
ow
g
’t always all
t in chargin
p
e
k
y
tl
n
ta
nent doesn
ns
es!
hen it is co
those devic
r
o
it
n
o
gives up w
m
e sure you
mode. Mak
Share this eBook!
Use Staging
Environment
ftware deLike with any so
ent can
ging environm
a
st
t,
n
e
m
p
lo
ve
smaller chang
n
e
v
e
fy
ri
e
v
t
u
help you a lo
t. In staging yo
n
e
m
n
o
ir
v
n
e
l
d
es in critica
as expected an
rk
o
w
s
e
g
n
a
ch
concheck that
rations in that
u
g
fi
n
co
w
e
n
ll
new
validate a
d idea to roll in
o
o
g
a
o
ls
a
is
gh
text. It
devices) throu
n
e
v
(e
re
a
w
hard
es are ‘exotic’.
staging if devic
USB Tips
Standardize Your
Test Lab Hardwar
e
It is a
to use the lways a good idea
setting up same hardware w
hen
o
house lab r configuring a lar
ger in.
certain de This means that if
vice setup
you use
for serve
your ope
rs, it mak
rational t
es
eam’s life
very sam
easier. Th
e hardwa
e
re is cons
across th
tantly use
e environ
d
ment. Th
naturally
o
se device
differ, bu
s
t then ag
use the s
ain you c
ame way
an
trol and m s to connect, conaintain th
ose.
number of
m
u
im
x
a
m
h
Find the
ected to eac
n
n
o
c
e
b
n
a
tc
experience,
r
devices tha
u
o
m
o
fr
ne
erally, and
nected to o
n
o
c
server. Gen
s
e
ic
v
e
nectumber of d
metime con
o
S
.
0
the typical n
-2
5
1
rver
bustness is
the same se
in
ts
lo
s
server for ro
B
S
U
in different
.
ing devices
tter stability
e
b
h
c
u
m
e
may giv
to
ely pays off
it
n
fi
e
d
it
o
vS
ork
USB ports w
t
a
h
w
t
u
o
find
ne
o, there is o
ls
A
t.
s
e
b
e
th
nt
ery importa
minor but v
uge setups:
h
f
o
e
s
a
c
trick in
ed USB cad
o
-c
r
lo
o
c
e
Us
t
u know righ
o
y
d
n
a
s
le
b
t devices
away in wha
t with.
they connec
Share this eBook!
Conclusion
Building a large-scale in-house test lab is not a one- or two-day work. It needs all appropriate devices, infrastructure and software as basic ‘weapon’ for your on-premise testing environment. More than that, you also need to make sure that your test lab is well
developed and reliable enough to support your team running tests 24/7 with no flaws.
At Testdroid, we ourselves have been hosting an over 300-device cloud-based test lab –
Testdroid Cloud for the past years. All of the tips and tricks mentioned above are exactly
what we have experienced when hosting Testdroid Cloud. When you decide and start to
build your own in-house test lab, it is of great significance to take into account all of the
aspects (hardware, software and operations) mentioned above.
The purpose of this ebook is to give you a complete guide for creating offline test lab
and help you inspect potentially weak parts in your current in-house test lab settings
if you already have one in place. In addition, with plenty of experiences on hosting a
large-scale test lab, we are also able to deliver to you such a testing solution –Testdroid
Enterprise. Building your in-house test lab with Testdroid Enterprise will ease you at any
point and you can just sit back and relax.
Share this eBook!
Build Your In-House
Test-Lab with
interested?
Leave us your MESSAGE and we will get
back to you soon.
Learn More About Testdroid Enterprise HERE
Share this eBook!
TABLE OF CONTENT
WHY YOU NEED AN IN-HOUSE TEST LAB
4
HOW TO CHOOSE DEVICES AND HOW MANY OF THEM
9
REQUIRED INFRASTRUCTURE 14
ENTERPRISE GRADE RELIABILITY 20
OPERATING YOUR IN-HOUSE TEST LAB 24
Share this eBook!
Introduction
Building a large-scale in-house test lab from scratch can be a big challenge. It takes time
and efforts to figure out the right equipment and tools that are needed for a successful
deployment. In addition, it can be trickier if something is not properly configured.
At Testdroid, we understand how daunting and time-consuming it can be. And we have
experienced it ourselves when we were building Testdroid Cloud. To help customers
with different needs, we have been building various in-house test labs ranging from 20
devices to several hundred devices with Testdroid Enterprise.
Here, we’ve gathered and put together a guide that discusses about why you need to
build an in-house test lab, what and how many devices you need, what infrastructure
is required, how to create enterprise grade reliability, and how to operate your test lab
without failure. In fact, the tips we cover in this ebook are mainly influenced by our own
setup at Testdroid Cloud.
With the help of this ebook, you will be able to create the best offline test lab with the
right setup that maximizes your return on investment, instead of spending tons of time
correcting the wrong configuration back and forth.
Share this eBook!
Chapter 1
The Reasons
for an In-House
Test Lab
Share this eBook!
Top Reasons for In-House Test Labs
The very first thing you should tackle is probably to make sure whether
or not you need to set up one in-house test lab. Before simply jumping
into its construction, you must know the reasons why in-house test labs,
rather than cloud based test automation services, are the best options to
fulfill your needs. The most typical reasons you should take into account
are:
PREMARKET DEVICES
Surprisingly many of top tier developers work
with pre-market devices. Typically you are working on a significant enough app that will ship with
a device and you have to make sure that your
app works even when the actual device software changes from release to another. Typically
you are working inside or together with a device
manufacturer, carrier or one of the TOP100 app
companies that do get access to pre-market devices from top manufacturers.
CORPORATE SECURITY POLICIES
Like many other companies, you probably
have strict security policies regarding uploading development versions of your unreleased applications outside of the corporate firewall. Quite often your company is
also very professional with your testing
and setting up your own in-house device
lab makes a lot of sense to get the scalability and
time to market benefits of test automation on
real devices while maintaining required security
levels.
Share this eBook!
AVAILABILITY AND USAGE LEVEL
Sometimes your QA teams need to run
very long soak tests that take all night or
there are tens of development teams in
your company relying on the availability of
the test results every day so the shared infrastructure of public cloud based services
does not fit well to your usage profile. If
you are running a lot of tests (potentially
from every code commit), your tests take
a long time to execute or you are relying
on having the test devices available when
you need them an in-house device lab is
the way to go.
SPECIAL SETUPS
“
Your company might also
by far the
is
b
la
e
s
u
o
-h
in
An
require specific hardware
cure setup
e
s
d
n
a
le
ib
or environment setup to be
x
e
fl
most
s for wide
te
a
able to fully test your softd
o
m
m
o
c
c
a
and it
ware. For instance some speardware seth
t
n
re
fe
if
d
f
o
cial Bluetooth or WIFI enabled
spectrum
rent usa ge
fe
if
d
ly
d
il
w
s
a
ll
accessory needs to be physicalups as we
uous testn
ti
n
o
c
ly in the same location with the
/7
4
2
m
o
rofiles fr
test devices or the devices need p
access to
te
o
m
re
g
in
id
v
ing to pro
to be in some specific location or
prototypes.
in some specific carrier network.
specific device
Many of the environment related
characteristics can be simulated but
not all and therefore this sort of special cases need to be run in specific inhouse labs.
“
As you can see the reasons for setting up in-house
device labs can be as varied as the companies that are setting those up. An in-house lab is by far the most flexible and
secure setup and it accommodates for wide spectrum of
different hardware setups as well as wildly different usage
profiles from 24/7 continuous testing to providing remote
access to specific device prototypes.
Share this eBook!
The Core Benefits of Continuous
Integration & Testing
One essential tool for mobile application developers to use in order
to automate their testing effort is continuous integration software –
for example Jenkins, which is currently the leading open source
continuous integration (CI) server monitoring executions
of repeated jobs. With continuous integration and
testing, you can reap the following benefits.
VERSION CONTROL AND BUILD REVISION
This rather old practice in software industry is the
same with mobile application development. The
quicker you can ensure your additions and regressions to builds work properly, the more quickly you
can move forward. On the other hand, if you face
problems or find any issues, all attention to fixing
those bugs can be drawn there right away. Fixing a
bug today costs much less than fixing the very same
bug tomorrow.
BUILD AUTOMATION, INSTANT EXECUTION
AND TESTING
Build Automation refers to the ability to either automatically trigger or build a clean version of your application.
You can take this further and make your systems built from
source code just committed to version control. This typically
shortens the build time significantly, but also makes the application ready for execution and testing instantly. This is one of the
greatest benefits that continuous integration can bring to you.
Share this eBook!
ALL TYPES OF TESTING ENABLED
By automating mobile application testing you can ensure that your application behaves correctly on every mobile OS version & build (30+ with
Android, 10+ with iOS), which will reduce costs and increase efficiency
of the overall testing process. Not only the basic unit and GUI testing,
but testing in continuous integration environment provides a way to
adopt any test frameworks, methods, complementary technologies and so on, for your continuous testing and
development environment.
“
“
inuous
t
n
o
c
g
ualin
i
q
t
e
p
h
o
t
d
g
a
By
mprovin
i
y
l
n
o
your
t
o
g
n
n
i
e
k
r
a
a
t also m
ion you
u
t
b
a
r
,
s
g
s
e
t
e
c
n
i
pro
er, more
t
d
s
n
a
f
a
h
e
c
d
r co
ffort mu
e
g
n
ty of you
i
t
s
asure.
and te
e
t
n
m
e
o
t
m
p
r
o
e
devel
nd easi
a
t
n
e
t
s
i
cons
FREQUENT COMMITS, CODE CONSOLIDATION
-> FAST BUILDS
Time is money. And fast builds give you that time. Fast builds allow you
to receive test results (= bug report) faster.
CONTINUOUS INTEGRATION/TESTING IS THE RIGHT WAY
TO GO FORWARD
By adopting continuous integration you are not only improving the
quality of your code and process, but also making your development
and testing effort much faster, more consistent and easier to measure.
These are significant benefits to be gained out from simple process
change and adopting something that already exists in processes.
Share this eBook!
Chapter 2
Devices for
In-House
Test Lab
Share this eBook!
Decide the Size of Your Device Pool
Today the number of different device variants, OS versions, resolutions and form factors is really large and to
make matters worse a lot of these different devices have
shipped tens of millions units making each of them significant in their own right. This problem is not limited
to only Android anymore but also on iOS. Probably you have already over 10 unique hardware and resolution combinations and 3-4
active OS versions, making the iOS device
matrix fairly sizeable as well.
So how many devices you need
to provide wide enough market
coverage? For Android you take a
great data published by OpenSignal
and from here you can see that for instance if you want to cover 50% of the
global market you will need about 60
devices, about 20% global market coverage can be reached with 12 devices
and if you want to go over 50% you will
need significantly more devices.
If you focus mainly on US market you can
find great and fresh information published by
app development company Double Encore. According to this data with well-selected 25 Android devices you can cover two-thirds of the Active users and for
90% US market coverage you will need 128 devices.
For estimating the number of needed different iOS devices, you can take a look at the data published by Apple.
In May 2014 there are 8 unique iPhone models, 7 unique
Share this eBook!
iPad models and 5 generations of iPod Touches, totaling at 20
unique devices.
Given Apple’s user base you can potentially safely leave out iPhone 2G, 3G and 3GS as well as the first three generations of iPod
Touch, since the number of these devices in active use is most
likely fairly small and especially these were introduced 5-7 years
ago and have not even been in production for several years. This
leaves 15 unique and still significantly represented iOS devices in
the market.
On iOS you also need to remember that while most of the iOS users do always update to the latest version, there are always some
users who don’t. For instance now about nine months after the
introduction of iOS 7, there are still 12% of users who have not
updated to the latest version. Although this is nothing close to
the situation in Android land, it is advisable that you keep some
iOS devices with the second latest iOS version as well.
Share this eBook!
Select the Right Devices for
Maximum Coverage
Now that we know how many devices you need for both Android and iOS, let’s take a bit
more detailed look at which devices are the correct ones. Assuming your target market
is North America we can again use Double Encore’s data as a good yardstick. So the list
of 25 most used Android devices looks like this:
1.
2.
3.
4.
5.
Samsung Galaxy S3 (17%)
Samsung Galaxy S4 (12.3%)
Motorola Droid Razr (3.7%)
Samsung Galaxy Note 3 (3.7%)
Samsung Galaxy Note 2 (3.3%)
6.
Asus Nexus 7 (3.2%)
7.
HTC One (1.9%)
8.
Samsung Galaxy S2 (1.8%)
9.
Samsung Galaxy Tab2 10.1 (1.6%)
10.
14.
Samsung Galaxy Note 10.1 (1.2%)
15.
LG G2 (1.1%)
16.
Samsung Galaxy S (<1%)
17.
Motorola Droid X (<1%)
18.
Motorola Droid Bionic (<1%)
19.
Samsung Galaxy S5 (<1%)
20.
Samsung Galaxy Nexus (<1%)
21.
LG Nexus 5 (<1%)
Droid Razr HD (1.4%)
22.
Motorola Moto X (<1%)
11.
Droid Razr M (1.4%)
23.
Samsung Galaxy Tab 3 10.1 (<1%)
12.
Samsung Galaxy S2 Epic (1.3%)
24.
LG Nexus 4 (<1%)
13.
Samsung Galaxy Tab 2 7.0 (1.3%)
25.
HTC Thunderbolt
Share this eBook!
Respectively if you want to cover the most used iOS devices, the list looks like this:
1.
2.
3.
4.
5.
iPhone 5S
iPhone 5C
iPhone 5
iPhone 4S
iPhone 4
8.
iPad Air
9.
iPad 4th generation
10.
iPad 3th generation
11.
iPad 2nd generation
12.
iPad 1st generation
6.
iPad Mini
13.
iPod Touch 5th generation
7.
iPad Mini 2nd generation
14.
iPod Touch 4th generation
The Frequency of Updating Your
Device List
Once you have your lab up and running, you need to think how often you should add
new devices to keep your market coverage. For iOS devices it is simple, as you only need
to add new devices when Apple announces new models (twice a year). But on Android
it is a bit more complex than that. A good rule of thumb is to check every quarter what
are the latest blockbusters (especially any new Galaxy devices) and add those to your
lab. You cannot really substitute the older ones because, shown in the TOP25 Android
device list, even 2-3 year old number one devices (think Samsung Galaxy S2 and Samsung Galaxy S) have such a huge installed base that they will hang on the list of most
used devices for years.
For Testdroid Cloud, we always expand our device pool with the latest Android and
iOS device models when they are available on the market
Share this eBook!
Chapter 3
Required
Infrastructure
Share this eBook!
Device Control Servers
After selecting your target devices, you need to take a look at other parts of a reliable
infrastructure. Regardless of the technology you use to build your device lab, you
will need some sort of device control node servers that will take care of managing
devices and possibly executing tests. Additionally these servers can collect, process
and store test results and do other test execution related tasks. At Testdroid we use
either generic PC-hardware running Ubuntu Linux for controlling Android devices or
Mac Minis for controlling iOS devices. Because we use the OS vendor provided methods to control the devices (Android Debug Bridge (ADB) on Android and Instruments
on iOS) the control nodes/servers need to have decent number of free USB ports as
each device will be connected to the device servers with USB cable.
Share this eBook!
In order to have stable test execution infrastructure you need
to take following things into account when selecting the
hardware for device control servers:
HARD DISKS
You need to have enough disk space for all test artifacts (logs,
screenshots, videos, performance data, memory dumps, network dumps etc.) and make sure that there is a mechanism that
either deletes the data created by previous test runs or transfers it to somewhere where there is enough storage space.
Running out of disk space is one of the most common failure
points in large-scale test automation. You also may want to
consider investing in SSD disks to make sure that writing the
test results to storage does not become a bottleneck
under heavy load.
RAM
You need enough RAM to run several virtual
machines in one physical device control server
as that is the best way to ensure that each test
run starts from identical starting point. Also running virtual machines is required to run several iOS
Instruments sessions on one Mac.
ENERGY EFFICIENCY
If you are building a large device lab, you will
potentially running tens of device control servers as well. So keep in mind that the more power your hardware draws, the more heat those
will generate and more cooling you will need.
Share this eBook!
USB Hubs
As all devices will be connected to the device control servers over USB, you need to pay
attention to the quality and capabilities of your USB hardware. This is one of the most
error-prone parts of the system. While there are no fixed hard limits on how many
devices you can plug into one USB, our experiments show that you should not connect over 16 devices per USB hub. If you connect more than 16, the devices may start
disconnecting randomly especially when a lot of data is being transferred either at the
beginning or end of each test.
The other important reason why good quality USB hubs are essential is that many devices (especially ones with a large display) have
problems with charging if the charging current is not high enough. The USB specification
mandates that the output current is limited to
500mA when the USB port is on data mode but
the best USB hubs on the market allow you to
programmatically switch between data and
charging modes allowing you to implement automatic charging cycles for devices that cannot
be charged in USB data mode.
QUICKTIPS!
To make sure that there are as
few USB related problems as
possible we prefer professional
USB hub hardware by company
called Cambrionix. Those are sold
under several names and at least
ones sold under Datamation brand
are great. Just be careful when
selecting those as some models are
just for charging and
some models are
just for iOS device.
Share this eBook!
WIFI Infrastructure
WIFI infrastructure is another very important area that is often overlooked when creating large-scale test lab. You can get to about 10 maybe even 15 devices with any WIFI infrastructure without any problems. But as the number of devices in your WIFI network
adds up, so do the problems. Quite often all devices are able to connect to the WIFI
without problems but the problems appear when all of those start transferring data
at the same time. Most WIFI access points are not designed for this kind of throughput
and you will start observing different kinds of timeouts on server responses etc. Network throughput on typical WIFI router is around 100-150 Mbit/s. But when you have
several dozens of wireless devices trying to download at tens of Mbits/s, you can easily
see where the bottleneck is.
QUICKTIPS!
There are several brands on the market
selling enterprise grade WIFI access
points. At Testdroid, we have tried
at least Aerohive, Cirrus and
Cisco Meraki. Our current
favorite is Cisco Meraki MR24 which has
three separate radios each capable
of working on either
2.4Ghz or 5Ghz frequency band and each
of them can transfer up to
300 Mbit/s giving whopping
900 Mbit/s from your devices to
the back end servers. Another nice thing
about Cisco Meraki WIFI infrastructure is
their cloud based management UI that has
features like throttling the speed of individual WIFI connection which can be useful for instance when simulating different
network conditions.
Share this eBook!
Integration to Your Build Process
The next important consideration is how you integrate your in-house device lab to your
build process. Most likely you have already a separate Continuous Integration server in
place and you have to make sure that without any manual interaction your test infrastructure can be controlled by your CI server. This means that everything needs to be
controllable over an API so that your CI jobs can seamlessly both trigger test runs and
use the test results from your in-house test lab to trigger further actions such as deploying your app to production.
Continuous
integration
Source Management
USER INTERFACE
API
application
server
API
UI
Test
execution
DEVICES
Test
management
reporting
admin
monitoring
users &
roles
DATABASE
Access to Your Organizations
Centralized Access Control System
Last but not least, if you are building your in-house device lab in an enterprise setting
you need to be able to connect to your company’s centralized user right management
system in order to validate, grant and revoke user rights for each user in your test automation system. This is just to ensure that any unauthorized users are not able to access
the data and binaries that they are not supposed to and to synchronize these access
rights when people leave the organization, change roles or simply do not anymore need
access to such data. Most common protocols for doing this are LDAP (Lightweight Directory Access Protocol) and OAuth. And your infrastructure should validate each user
login through such centralized system.
Share this eBook!
Chapter 4
Enterprise Grade
Reliability
Share this eBook!
Always Run Tests on Clean Devices
In addition to hardware infrastructure, you need a reliable software configuration for avoiding any test failures. One of the most common
causes for flaky test results is that the environment where you
run your tests i.e. your test device is not exactly in the same
state during each test session. This means that there may
be different processes running, different applications
or background services installed, different amount
of free memory or different amount of storage
space available. In order to avoid test flakiness,
you should make sure that there is a proper
cleaning cycle between every test session that
uninstalls all unwanted apps, sets the device
storage to the same state as it was before
the test and reboots the devices so that
there are no processes running from the
previous test session.
Create Intelligent
Retry Mechanisms
Some of the test execution related failures are
not actually related to the application under test
but are caused by some test infrastructure related
reason. The good news is that this type of failures are
easy to identify and tests that have failed because of
such reasons can be automatically retried as many times
as needed to have a test run that is free from infrastructure
related failures. The most common reasons for test infrastructure
related failures are: The connection between the device control server
and device fails, the connection between the device and your back end server fails,
either the device or device control server runs out of storage space, device runs out
of power in the middle of test session, and etc. All of these can be identified during
the test session and the tests can be automatically retried so that the only remaining
failures are real failures related to the application under test or the test scripts.
Share this eBook!
Check and automatically reconnect
the USB and Wireless data connections
Losing either the USB or wireless data connection at some point is unfortunately very
common on both Android and iOS devices. To make matters worse quite often, the
device reports that it has live connection over USB, WIFI or Wireless data but no data
is moving. You need to be able to automatically verify that both
USB connection and WIFI connection are actually transferring data and if that’s not the case, you need to
automatically disconnect & reconnect the connections to get them back up again.
Automate All Configuration Changes
When you have hundreds of devices and tens of device control servers in your test lab,
you cannot (and should not) really make any changes to settings or configurations manually because the risk of not doing the changes identically on all devices or servers is
simply too high. We automate everything we can by using Opsworks Chef and we have
implemented our on-device services that take care of changing the device settings etc.
Even if we have been very systematic with this, every now and then there are problems
because some small setting was done by hand and it was not replicated identically on
all devices or servers. This is one area where you cannot afford to cut corners if your
goal is 99.99% reliability of your in-house test lab.
Share this eBook!
Use Professional Monitoring
The last but not by any means least important way to ensure the robustness of your
in-house test lab is to set up a professional monitoring for all aspects of your test lab
infrastructure. The most important areas to monitor are:
DISK SPACES
– Automated tests create a lot of data in the form of logs, screenshots, videos, memory
dumps, and network dumps. When you are running tens or even hundreds of devices
sometimes 24/7, you may run out of disk space just out of the blue.
NETWORK CONNECTIONS, LATENCIES, PACKET LOSS RATES
– These have an impact on everything in your test lab and once you start troubleshooting the strangest and most randomly appearing problems, most often the root cause is
networking related.
POWER
– Disruptions on power supply (even very short ones) can cause very strange problems
when the system has lots of moving parts and all of them are dependent on steady
power supply. Use UPS backed power whenever you can and also make sure that your
servers close down and come up in a controlled way when longer power disruption
happens.
You can use any generic monitoring software for this. For Testdroid Cloud we are currently using AWS monitoring, Loggly and New Relic to handle our monitoring.
Share this eBook!
Chapter 5
Operating
In-House
Test Lab
Share this eBook!
g
o
L
l
a
n
o
i
t
a
r
e
p
O
t up
Once you’ve se
cftware infrastru
so
d
n
a
re
a
w
rd
ha
lab is the
g your device
n
ti
ra
e
p
o
,
re
tu
ke care of. The
ta
to
d
e
e
n
u
o
opnext thing y
s is to save all
n
o
ti
ra
e
p
o
f
o
docfirst aspect
teams should
n
o
ti
ra
e
p
O
s.
to
eration log
s that happen
ce
ie
p
d
n
a
s
it
b
runs.
ument all
vices and test
e
d
t,
n
e
m
n
o
ir
cs,
the env
ework with do
m
o
h
t
a
re
g
g
in
By do
and
le to maintain
b
a
e
b
l
il
w
s
m
tea
hen
nvironment w
e
e
th
e
d
ra
g
p
u
.
needed easily
Keep You
Devices C r
oo
l
Everyt
power
hin
n
o
waday g is about
its con
s. T
ant th sumption is he power a
ing to
contrib the most im nd
of the
po
u
tempe system. Eve te the tempe rtns
rat
ra
malfun ure can cau maller chan ture
ge
se
ct
estima ion (This sho that lots of s in
device
te
u
s
up the d!) and go o ld never be
u
ffl
device
n
i
d
n
e
e. Whe
poo
enoug
n you rh spac l, make sure
set
e for t
y
o
u
l
e
ho
a
“breat se devices t ve
h”.
o
e
h
t
r
o
t
i
n
o
M
s
e
c
i
v
e
D
f
o
g
Chargin
ke
need to ma
y
ll
a
r
tu
a
n
You
ugh power
o
n
e
t
e
g
s
evice
4/7
vices are 2
sure your d
e
d
e
s
o
th
evic. Once
gly, those d
to perform
in
is
r
p
r
u
s
, maybe
if they get
d
e
r
o
it
n
o
plugged in
tly m
are compo
be constan
w
d
r
a
h
s
a
y
es need to
atter
sily
wer. The b
ing and ea
o
g
r
p
a
h
h
g
c
u
ll
o
fu
n
e
ow
g
’t always all
t in chargin
p
e
k
y
tl
n
ta
nent doesn
ns
es!
hen it is co
those devic
r
o
it
n
o
gives up w
m
e sure you
mode. Mak
Share this eBook!
Use Staging
Environment
ftware deLike with any so
ent can
ging environm
a
st
t,
n
e
m
p
lo
ve
smaller chang
n
e
v
e
fy
ri
e
v
t
u
help you a lo
t. In staging yo
n
e
m
n
o
ir
v
n
e
l
d
es in critica
as expected an
rk
o
w
s
e
g
n
a
ch
concheck that
rations in that
u
g
fi
n
co
w
e
n
ll
new
validate a
d idea to roll in
o
o
g
a
o
ls
a
is
gh
text. It
devices) throu
n
e
v
(e
re
a
w
hard
es are ‘exotic’.
staging if devic
USB Tips
Standardize Your
Test Lab Hardwar
e
It is a
to use the lways a good idea
setting up same hardware w
hen
o
house lab r configuring a lar
ger in.
certain de This means that if
vice setup
you use
for serve
your ope
rs, it mak
rational t
es
eam’s life
very sam
easier. Th
e hardwa
e
re is cons
across th
tantly use
e environ
d
ment. Th
naturally
o
se device
differ, bu
s
t then ag
use the s
ain you c
ame way
an
trol and m s to connect, conaintain th
ose.
number of
m
u
im
x
a
m
h
Find the
ected to eac
n
n
o
c
e
b
n
a
tc
experience,
r
devices tha
u
o
m
o
fr
ne
erally, and
nected to o
n
o
c
server. Gen
s
e
ic
v
e
nectumber of d
metime con
o
S
.
0
the typical n
-2
5
1
rver
bustness is
the same se
in
ts
lo
s
server for ro
B
S
U
in different
.
ing devices
tter stability
e
b
h
c
u
m
e
may giv
to
ely pays off
it
n
fi
e
d
it
o
vS
ork
USB ports w
t
a
h
w
t
u
o
find
ne
o, there is o
ls
A
t.
s
e
b
e
th
nt
ery importa
minor but v
uge setups:
h
f
o
e
s
a
c
trick in
ed USB cad
o
-c
r
lo
o
c
e
Us
t
u know righ
o
y
d
n
a
s
le
b
t devices
away in wha
t with.
they connec
Share this eBook!
Conclusion
Building a large-scale in-house test lab is not a one- or two-day work. It needs all appropriate devices, infrastructure and software as basic ‘weapon’ for your on-premise testing environment. More than that, you also need to make sure that your test lab is well
developed and reliable enough to support your team running tests 24/7 with no flaws.
At Testdroid, we ourselves have been hosting an over 300-device cloud-based test lab –
Testdroid Cloud for the past years. All of the tips and tricks mentioned above are exactly
what we have experienced when hosting Testdroid Cloud. When you decide and start to
build your own in-house test lab, it is of great significance to take into account all of the
aspects (hardware, software and operations) mentioned above.
The purpose of this ebook is to give you a complete guide for creating offline test lab
and help you inspect potentially weak parts in your current in-house test lab settings
if you already have one in place. In addition, with plenty of experiences on hosting a
large-scale test lab, we are also able to deliver to you such a testing solution –Testdroid
Enterprise. Building your in-house test lab with Testdroid Enterprise will ease you at any
point and you can just sit back and relax.
Share this eBook!
Build Your In-House
Test-Lab with
interested?
Leave us your MESSAGE and we will get
back to you soon.
Learn More About Testdroid Enterprise HERE
Share this eBook!
