Press "Enter" to skip to content

GTAC 2015 Mobile Game Test Automation Using Real Devices

GTAC 2015: Mobile Game Test Automation Using Real Devices ampgtampgtYvette Nameth: Next up, we have Jouko from Bitbar to talk about using image recognition for mobile app and game testing. ampgtampgtJouko Kaasila: Hello. Okay. So my name is Jouko Kaasila. Im the cofounder and CEO of Bitbar, the company behind a pretty awesome online service called Testdroid Cloud, which is a — its a device cloud of Android and iOS devices from every single market in the world, including Japan, China, Korea, and India.

We also host private device clouds for people who dont want to share their devices. And we license our technology to companies who build large-scale on-premise device labs. So in the next ten minutes, Im going to tell you how to get rid of 60 manual testers with — and replace them with one smart one.

And this is actually something that has happened with one of our customer organizations, when they moved from 100 manual testing using real devices to 100 automated testing using real devices. And at the end, Im going to have a gift for all of you in the spirit of the holiday season. So lets start. So if you look at the — the charts of the most grossing apps on Android or the Google Play today, on the top 50 — list of the most-grossing apps, there is only one app that is not a game. And if you look at the top 100 most grossing apps, there are only three apps that are not games.

So its pretty obvious who is making the money on the app stores. And usually, games are free to download, and they monetize with in-app purchases. So it takes a little bit of time, or you have to engage the user long enough to make really any money on that. Because currently, the average customer acquisition cost or the cost per download on Android and iOS is around $5. So it means that you have to keep the customer for long — as many releases as possible in order to recover that $5 investment and make some profit on top of it.

So its very high motivation for testing in general and making sure that your game works on every single device out there. So given the commercial incentive, why hasnt this all been automated a long time ago? You know, these guys do have a lot of money. Thats not the problem.

The problem is that mobile games are not really easy to automate. They are very difficult to automate. So the main reason is that the games use direct screen access in the form of OpenGL or active X. And they effectively bypass all the operating system controls or services you have. So that means that any of your — any of the mobile — native mobile test automation frameworks that we know cant really access any of the internal data of the game.

So you only have to resort to X and Y clicks and, you know, reading the screen buffer. You basically have to test everything from the outside of the game. And, of course, in terms of automation, thats pretty tricky. The second thing is that the games are very performance-driven. So, first of all, they consume a lot of resources.

So the game binaries are two to three gigabytes today. They use a lot of network traffic, they use a lot of memory, they use a lot of CPU, they use a lot of GPU, they utilize the sensors. So its pretty clear that, you know, these guys, they dont test anything on emulators and simulators. It doesnt just make any sense.

And theres another interesting incentive, is that one of our customers told us that, you know, if they can add one more very popular Chinese Android device on the list of their supported devices, it can bring up to 5 million revenue over the lifetime of the game. So you can invest quite a lot on optimizing your game on that model. So all the difficulties have actually led to this. So the guys have resources, and its difficult to automate games. So all the gaming companies, the large gaming companies, have very large manual device farms on their favorite low-cost locations.

And thats — of course, it doesnt scale very well. And that leads to another problem, that the QA process causes delays on the actual release of the game, which increases the time to market. And thats the real cost of the manual testing, that, you know, things get delayed, and you dont stick to the schedule. And, of course, even with games, there is — theres still a lot of room for manual testing. But that space is more for the qualitative, you know, testing of the fluidity of the game play and that sort of aspects, not to go through every single menu on every single language on every single device.

Thats the job for the automation. So the most typical assignment we get is that, you know, guys, we only need to automate the basic game — basic functionality of the game, so — on as many devices as possible and it has to run as fast as possible, because they want to automate it on every single build. And, of course, with these guys, the challenge is never the access to the devices. You know, for the manual testing, they have hundreds of devices.

Thats not a problem. The problem is that how to automate these, like, monotonous routines, and especially in a way that they produce actionable — actionable results in — such as, like, performance data logs, screenshots, and articles at scale. You know, with manual testing, you cant — you can get those, like, one by one, but you cant get it at scale. And another curve ball is that of course these teams, they dont have, like, programming skills or background on scripting, which severely limits the tools and the frameworks that can be used. And also, these guys — these teams are usually quite separated from the actual development team.

So any kind of, like, white box testing or instrumenting the game itself is not usually feasible for these guys. These guys get the APK thrown over the fence, and its like pure, 100 black box testing scenario. So with these two requirements in mind, so, need for, like, very simple scripting without any computer science skills, and the pure 100 black box approach, we started looking at the open source space that, you know, what sort of building blocks could we find to, like, gobble up a solution that actually solves this sort of problem. So as a foundation of the solution, we selected Appium.

And the fact that Appium is cross-platform test automation framework works really well here, because the game — usually the game is exactly the same on Android and iOS. So I — an ideal case, you can use the same script to run your automation on both Android and iOS. Also, Appium provides a pretty nice abstracted API that we can use for the next level of our automation to run. So in this scenario, we cannot use any of the, like, advanced features of Appium, like — like the object — inspecting the objects and those sort of things. We can only use, like, X and Y clicks and drags.

For the next level, we had to kind of figure out three things. How — first, how can we — how can we know where in the game flow we are? How can we get that info? Then, when we get that info, where should we click next? What is the next X and Y coordinate to click?

And then after we have clicked something, did we achieve what we wanted to do? Did the game go to the next stage as — as we expected? So the — to drive the execution 100 from the outside of the game, we selected OpenCV image recognition library.

So, basically, we feed the library with screenshots and reference images. And it does, like, a pixel comparison, pixels not matching. And when it finds the matching location, it will feed the X and Y coordinates to the Appium script that then executes that on the real devices.

Its pretty — The OpenCV library is very good for this, because its customizable, its resolution-agnostic, which is really good on real mobile testing — like, real device testing context, and it can even recognize images that are stretched or, like, somehow at an angle, so you can even test, like, 3D games as well. So the outcome is that theres only two simple tasks that the test automation engineers need to do. They need to cut the reference images and then change only, like, one line or — like, one line of code to define what kind of click needs to be done when the reference is matching or when you get the coordinates. So then parallelizing this was quite interesting. So in the — in the device cloud, like, remote device cloud context, the Appium client sits on the remote machine, which is typically a developer workstation or a mobile — like, a indiscernible integration server on the other side of the Internet where the devices are.

And this — Appium creates WebDriver session from the Appium client to the Appium server, which is in the cloud, and then, you know, using that server, all the data — all the data goes over the WebDriver session, including the logs and the screenshots and all the control commands. And the way you scale this is that you just spin a lot of WebDriver sessions on the same — same remote machine, and, you know, you just keep running it. But that effectively renders the — the remote machine, the bottleneck. And especially in our scenario, where we run the OpenCV image recognition stack on top of the Appium client and we are transferring very large screenshots all the time on all the devices, we really hit performance wall on, like, five, six, seven devices per remote machine.

It didnt scale. We had to look for something else. So our solution was that we moved all the processing, the whole test execution, on the server side. So we created virtual machine images that included the whole stack, including the OpenCV, the Appium client and the Appium server, and this virtual machine automatically connects to one individual Android or iOS device at a time.

Then we can run the — a lot of these — On one physical server, we can run as many of these as we want. And the server has enough processing power to do all the pixel comparison, all that. We dont have to transfer the screenshots over the Internet. And, actually — its actually made this very robust and very fast. So in a way, its like Appium on steroids, because, you know, it runs, like, three times faster than your normal Appium.

And its really a very scalable — and the end user doesnt have to worry of any of the — how to spin this up and how to scale it. From the end user point of view, its like running one session. They upload just script, and it runs — you know, their online system scales all that execution for them. So we prepared a demo on how all this works in practice. So can we start the article? ampgtampgtJouko Kaasila: Okay.

So thats a demo. And as promised, the gift. All the sample code, all the instructions, everything you can get access to those at testdroid.comgtac15.

Questions? ampgtampgtYvette Nameth: Sadly, we actually dont have time for questions. ampgtampgtJouko Kaasila: Good. ampgtampgtYvette Nameth: Good? Yeah, well, you are standing between these people and food. ampgtampgtJouko Kaasila: Yes. You can reach me today and tomorrow.

Im going to be here the whole day. So any questions on automating game testing or scaling Appium on the server side, just reach me. GTAC 2015: Mobile Game Test Automation Using Real Devices ampgtampgtYvette Nameth: Next up, we have Jouko from Bitbar to talk about using image recognition for mobile app and game testing. ampgtampgtJouko Kaasila: Hello.

Okay. So my name is Jouko Kaasila. Im the cofounder and CEO of Bitbar, the company behind a pretty awesome online service called Testdroid Cloud, which is a — its a device cloud of Android and iOS devices from every single market in the world, including Japan, China, Korea, and India.

We also host private device clouds for people who dont want to share their devices. And we license our technology to companies who build large-scale on-premise device labs. So in the next ten minutes, Im going to tell you how to get rid of 60 manual testers with — and replace them with one smart one.

And this is actually something that has happened with one of our customer organizations, when they moved from 100 manual testing using real devices to 100 automated testing using real devices. And at the end, Im going to have a gift for all of you in the spirit of the holiday season. So lets start. So if you look at the — the charts of the most grossing apps on Android or the Google Play today, on the top 50 — list of the most-grossing apps, there is only one app that is not a game.

And if you look at the top 100 most grossing apps, there are only three apps that are not games. So its pretty obvious who is making the money on the app stores. And usually, games are free to download, and they monetize with in-app purchases.

So it takes a little bit of time, or you have to engage the user long enough to make really any money on that. Because currently, the average customer acquisition cost or the cost per download on Android and iOS is around $5. So it means that you have to keep the customer for long — as many releases as possible in order to recover that $5 investment and make some profit on top of it.

So its very high motivation for testing in general and making sure that your game works on every single device out there. So given the commercial incentive, why hasnt this all been automated a long time ago? You know, these guys do have a lot of money.

Thats not the problem. The problem is that mobile games are not really easy to automate. They are very difficult to automate. So the main reason is that the games use direct screen access in the form of OpenGL or active X. And they effectively bypass all the operating system controls or services you have. So that means that any of your — any of the mobile — native mobile test automation frameworks that we know cant really access any of the internal data of the game.

So you only have to resort to X and Y clicks and, you know, reading the screen buffer. You basically have to test everything from the outside of the game. And, of course, in terms of automation, thats pretty tricky.

The second thing is that the games are very performance-driven. So, first of all, they consume a lot of resources. So the game binaries are two to three gigabytes today. They use a lot of network traffic, they use a lot of memory, they use a lot of CPU, they use a lot of GPU, they utilize the sensors. So its pretty clear that, you know, these guys, they dont test anything on emulators and simulators.

It doesnt just make any sense. And theres another interesting incentive, is that one of our customers told us that, you know, if they can add one more very popular Chinese Android device on the list of their supported devices, it can bring up to 5 million revenue over the lifetime of the game. So you can invest quite a lot on optimizing your game on that model. So all the difficulties have actually led to this. So the guys have resources, and its difficult to automate games.

So all the gaming companies, the large gaming companies, have very large manual device farms on their favorite low-cost locations. And thats — of course, it doesnt scale very well. And that leads to another problem, that the QA process causes delays on the actual release of the game, which increases the time to market. And thats the real cost of the manual testing, that, you know, things get delayed, and you dont stick to the schedule.

And, of course, even with games, there is — theres still a lot of room for manual testing. But that space is more for the qualitative, you know, testing of the fluidity of the game play and that sort of aspects, not to go through every single menu on every single language on every single device. Thats the job for the automation. So the most typical assignment we get is that, you know, guys, we only need to automate the basic game — basic functionality of the game, so — on as many devices as possible and it has to run as fast as possible, because they want to automate it on every single build.

And, of course, with these guys, the challenge is never the access to the devices. You know, for the manual testing, they have hundreds of devices. Thats not a problem. The problem is that how to automate these, like, monotonous routines, and especially in a way that they produce actionable — actionable results in — such as, like, performance data logs, screenshots, and articles at scale.

You know, with manual testing, you cant — you can get those, like, one by one, but you cant get it at scale. And another curve ball is that of course these teams, they dont have, like, programming skills or background on scripting, which severely limits the tools and the frameworks that can be used. And also, these guys — these teams are usually quite separated from the actual development team. So any kind of, like, white box testing or instrumenting the game itself is not usually feasible for these guys. These guys get the APK thrown over the fence, and its like pure, 100 black box testing scenario.

So with these two requirements in mind, so, need for, like, very simple scripting without any computer science skills, and the pure 100 black box approach, we started looking at the open source space that, you know, what sort of building blocks could we find to, like, gobble up a solution that actually solves this sort of problem. So as a foundation of the solution, we selected Appium. And the fact that Appium is cross-platform test automation framework works really well here, because the game — usually the game is exactly the same on Android and iOS. So I — an ideal case, you can use the same script to run your automation on both Android and iOS.

Also, Appium provides a pretty nice abstracted API that we can use for the next level of our automation to run. So in this scenario, we cannot use any of the, like, advanced features of Appium, like — like the object — inspecting the objects and those sort of things. We can only use, like, X and Y clicks and drags. For the next level, we had to kind of figure out three things.

How — first, how can we — how can we know where in the game flow we are? How can we get that info? Then, when we get that info, where should we click next? What is the next X and Y coordinate to click? And then after we have clicked something, did we achieve what we wanted to do?

Did the game go to the next stage as — as we expected? So the — to drive the execution 100 from the outside of the game, we selected OpenCV image recognition library. So, basically, we feed the library with screenshots and reference images.

And it does, like, a pixel comparison, pixels not matching. And when it finds the matching location, it will feed the X and Y coordinates to the Appium script that then executes that on the real devices. Its pretty — The OpenCV library is very good for this, because its customizable, its resolution-agnostic, which is really good on real mobile testing — like, real device testing context, and it can even recognize images that are stretched or, like, somehow at an angle, so you can even test, like, 3D games as well. So the outcome is that theres only two simple tasks that the test automation engineers need to do. They need to cut the reference images and then change only, like, one line or — like, one line of code to define what kind of click needs to be done when the reference is matching or when you get the coordinates.

So then parallelizing this was quite interesting. So in the — in the device cloud, like, remote device cloud context, the Appium client sits on the remote machine, which is typically a developer workstation or a mobile — like, a indiscernible integration server on the other side of the Internet where the devices are. And this — Appium creates WebDriver session from the Appium client to the Appium server, which is in the cloud, and then, you know, using that server, all the data — all the data goes over the WebDriver session, including the logs and the screenshots and all the control commands. And the way you scale this is that you just spin a lot of WebDriver sessions on the same — same remote machine, and, you know, you just keep running it.

But that effectively renders the — the remote machine, the bottleneck. And especially in our scenario, where we run the OpenCV image recognition stack on top of the Appium client and we are transferring very large screenshots all the time on all the devices, we really hit performance wall on, like, five, six, seven devices per remote machine. It didnt scale. We had to look for something else.

So our solution was that we moved all the processing, the whole test execution, on the server side. So we created virtual machine images that included the whole stack, including the OpenCV, the Appium client and the Appium server, and this virtual machine automatically connects to one individual Android or iOS device at a time. Then we can run the — a lot of these — On one physical server, we can run as many of these as we want. And the server has enough processing power to do all the pixel comparison, all that.

We dont have to transfer the screenshots over the Internet. And, actually — its actually made this very robust and very fast. So in a way, its like Appium on steroids, because, you know, it runs, like, three times faster than your normal Appium.

And its really a very scalable — and the end user doesnt have to worry of any of the — how to spin this up and how to scale it. From the end user point of view, its like running one session. They upload just script, and it runs — you know, their online system scales all that execution for them. So we prepared a demo on how all this works in practice.

So can we start the article? ampgtampgtJouko Kaasila: Okay. So thats a demo. And as promised, the gift. All the sample code, all the instructions, everything you can get access to those at testdroid.comgtac15.

Questions? ampgtampgtYvette Nameth: Sadly, we actually dont have time for questions. ampgtampgtJouko Kaasila: Good. ampgtampgtYvette Nameth: Good? Yeah, well, you are standing between these people and food. ampgtampgtJouko Kaasila: Yes. You can reach me today and tomorrow.

Im going to be here the whole day. So any questions on automating game testing or scaling Appium on the server side, just reach me.

Be First to Comment

    Leave a Reply

    Your email address will not be published. Required fields are marked *