by Eduardo Bravo Ortiz
General
Understand the platform. Testing on Android is not the same as testing on iOS. The testing tools and frameworks available for each platform are significantly different. (e.g., Android uses Java while iOS uses Objective-C, UI layouts are built differently on each platform, UI testing frameworks also work very differently in both platforms.)
Stabilize your test suite and test environments. Flaky tests are worse than having no tests, because a flaky test pollutes your build health and decreases the credibility of your suite.
Break down testing into manageable pieces. There are too many complex pieces when testing on mobile (e.g., emulator/device state, actions triggered by the OS).
Provide a hermetic test environment for your tests. Mobile UI tests are flaky by nature; don’t add more flakiness to them by having external dependencies.
Unit tests are the backbone of your mobile test strategy. Try to separate the app code logic from the UI as much as possible. This separation will make unit tests more granular and faster.
Android Testing
Unit Tests
Robolectric is a perfect tool for this; it stubs the implementation of the Android platform while running tests.
Hermetic UI Tests
hermetic environment , a white box testing framework like Espresso can simulate user actions on the UI and is tightly coupled to the app code. Espresso will also synchronize your tests actions with events on the UI thread, reducing flakiness. More information on Espresso is coming in a future Google Testing Blog article.
Diagram: Non-Hermetic Flow vs. Hermetic Flow
Monkey Tests
ANRs by stressing your Android application. They exercise pseudo-random events like clicks or gestures on the app under test. Monkey test results are reproducible to a certain extent; timing and latency are not completely under your control and can cause a test failure. Re-running the same monkey test against the same configuration will often reproduce these failures, though. If you run them daily against different SDKs, they are very effective at catching bugs earlier in the development cycle of a new release.
iOS Testing
Unit Tests
OCUnit , which comes bundled with Xcode, or GTMSenTestcase are both good choices.
Hermetic UI Tests
KIF has proven to be a powerful solution for writing Objective-C UI tests. It runs in-process which allows tests to be more tightly coupled with the app under test, making the tests inherently more stable. KIF allows iOS developers to write tests using the same language as their application.
Monkey Tests
Backend Testing
Guice for dependency injection, which allows us to easily swap out dependencies with fake implementations during testing and seed data fixtures.
Diagram: Normal flow vs Replay Tests flow
Conclusion
Unit tests: These should be your first priority in either Android or iOS. They run fast and are less flaky than any other type of tests.
Backend tests: Make sure your backend doesn’t break your mobile clients. Breakages are very likely to happen when the release cycle of mobile clients and backends are different.
UI tests: These are slower by nature and flaky. They also take more time to write and maintain. Make sure you provide coverage for at least the critical paths of your app.
Monkey tests: This is the final step to complete your mobile automation strategy.
Looking forward to getting my hands on Espresso. Would be interested to get your thoughts on appium.
ReplyDeleteMe too, looking anxious to see how Espresso works!
ReplyDeleteEspresso has been released http://googletesting.blogspot.com/2013/10/espresso-for-android-is-here.html
DeleteThanks for sharing this great article.
ReplyDeleteWhat do you do for testing the performance or responsiveness of an application? That becomes hard because it is dependent on the actual hardware and the internet connection.
ReplyDeleteNice post. I'd be really keen to hear how google approaches cross device/browser testing. Its always a big pain point for many organisations, but I don't hear too much about successfully approaches, or whether this is actually a valuable activity given the variance that exists.
ReplyDeleteWhat about actual user functionality? How does the Google+ team test that?
ReplyDeleteGuys, do you plan to share monkey for iOS to the world :) ?
ReplyDeleteRight now there are only two alternatives:
AntEater https://www.redant.com/anteater/
UI-AutoMonkey https://github.com/jonathanpenn/ui-auto-monkey
Hi Alexander,
DeleteWe are looking into opensource it in the near future, stay tuned.
Nice Article.
ReplyDeleteWill this tool mitigate a challenge of asserting mobile app. UI with respect to different device screen sizes.
Should I use Robolectric for anything beyond unit test? If I use that, than do I still need Espresso? And do you know how "real" is Robolectric compared with real emulator? I am afraid that I have to fix problems that are Robolecric specific.
ReplyDeleteHi，Greate Article.
ReplyDeleteI was wandering if there is any advice about how to test sdk which has no user interface.
For example, I have developed a tool which can get infomation about location，and other apps will use this as a ***.jar lib. Any advice?
Thumbs up..never try this but will follow the instruction it is very interesting...Thanks!
ReplyDeleteHi!
ReplyDeleteIs this article up-to-date? How do you test "in-the-wild"?
We are also following the same, not exactly but similar to this structure.
ReplyDeleteHi all,
ReplyDeleteI have question currently backend is not yet ready, And our plan to start frontend testing with stubs simulating the backend behavior.
The question is should we do phase before regression for checking after removing the stubs and testing with real testdata the backends are working fine OR regression will be enough. And what about real testdata wasn't available so could we do regression with the same stubs of backends?
How do you cover below testing for Mobile Apps :
ReplyDelete1. Battery Drain
2. Memory Testing
3. Performance Testing(App Side)