Google Testing Blog: Test Flakiness - One of the main challenges of automated testing (Part II)

Test Flakiness - One of the main challenges of automated testing (Part II)

Wednesday, March 24, 2021

This is part two of a series on test flakiness. The first article discussed the four components under which tests are run and the possible reasons for test flakiness. This article will discuss the triage tips and remedies for flakiness for each of these possible reasons.
Components
To review, the four components where flakiness can occur include:

The tests themselves
The test-running framework
The application or system under test (SUT) and the services and libraries that the SUT and testing framework depend upon
The OS and hardware and network that the SUT and testing framework depend upon

This was captured and summarized in the following diagram.

The reasons, triage tips, and remedies for flakiness are discussed below, by component.

The tests themselves

The tests themselves can introduce flakiness. This can include test data, test workflows, initial setup of test prerequisites, and initial state of other dependencies.

Reason for Flakiness	Tips for Triaging	Type of Remedy
Improper initialization or cleanup.	Look for compiler warnings about uninitialized variables. Inspect initialization and cleanup code. Check that the environment is set up and torn down correctly. Verify that test data is correct.	Explicitly initialize all variables with proper values before their use. Properly set up and tear down the testing environment. Consider an initial test that verifies the state of the environment.
Invalid assumptions about the state of test data.	Rerun test(s) independently.	Make tests independent of any state from other tests and previous runs.
Invalid assumptions about the state of the system, such as the system time.	Explicitly check for system dependency assumptions.	Remove or isolate the SUT dependencies on aspects of the environment that you do not control.
Dependencies on execution time, expecting asynchronous events to occur in a specific order, waiting without timeouts, or race conditions between the tests and the application.	Log the times when accesses to the application are made. As part of debugging, introduce delays in the application to check for differences in test results.	Add synchronization elements to the tests so that they wait for specific application states. Disable unnecessary caching to have a predictable timeline for the application responses. Note: Do NOT add arbitrary delays as these can become flaky again over time and slow down the test unnecessarily.
Dependencies on the order in which the tests are run. (Similar to the second case above.)	Rerun test(s) independently.	Make tests independent of each other and of any state from previous runs.

Table 1 - Reasons, triage tips, and remedies for flakiness in the tests themselves

The test-running framework

An unreliable test-running framework can introduce flakiness.

Reason for Flakiness	Tips for Triaging	Type of Remedy
Failure to allocate enough resources for the SUT, thus preventing it from running.	Check logs to see if SUT came up.	Allocate sufficient resources.
Improper scheduling of the tests so they “collide” and cause each other to fail.	Explicitly run tests independently in different order.	Make tests runnable independently of each other.
Insufficient system resources to satisfy the test requirements. (Similar to the first case but here resources are consumed while running the workflow.)	Check system logs to see if SUT ran out of resources.	Fix memory leaks or similar resource “bleeding.” Allocate sufficient resources to run tests.

Table 2 - Reasons, triage tips, and remedies for flakiness in the test running framework

The application or SUT and the services and libraries that the SUT and testing framework depend upon

Of course, the application itself (or the SUT) could be the source of flakiness.

An application can also have numerous dependencies on other services, and each of those services can have their own dependencies. In this chain, each of the services can introduce flakiness.

Reason for Flakiness	Tips for Triaging	Type of Remedy
Race conditions.	Log accesses of shared resources.	Add synchronization elements to the tests so that they wait for specific application states. Note: Do NOT add arbitrary delays as these can become flaky again over time.
Uninitialized variables.	Look for compiler warnings about uninitialized variables.	Explicitly initialize all variables with proper values before their use.
Being slow to respond or being unresponsive to the stimuli from the tests.	Log the times when requests and responses are made.	Check and remove any causes for delays.
Memory leaks.	Look at memory consumption during test runs. Use tools such as Valgrind to detect.	Fix programming error causing memory leak. This Wikipedia article has an excellent discussion on these types of errors.
Oversubscription of resources.	Check system logs to see if SUT ran out of resources.	Allocate sufficient resources to run tests.
Changes to the application (or dependent services) out of sync with the corresponding tests.	Examine revision history.	Institute a policy requiring code changes to be accompanied by tests.

Table 3 - Reasons, triage tips, and remedies for flakiness in the application or SUT

The OS and hardware that the SUT and testing framework depend upon

Finally, the underlying hardware and operating system can be sources of test flakiness.

Reason for Flakiness	Tips for Triaging	Type of Remedy
Networking failures or instability.	Check for hardware errors in system logs.	Fix hardware errors or run tests on different hardware.
Disk errors.	Check for hardware errors in system logs.	Fix hardware errors or run tests on different hardware.
Resources being consumed by other tasks/services not related to the tests being run.	Examine system process activity.	Reduce activity of other processes on test system(s).

Table 4 - Reasons, triage tips, and remedies for flakiness in the OS and hardware of the SUT

Conclusion

As can be seen from the wide variety of failures, having low flakiness in automated testing can be quite a challenge. This article has outlined both the components under which tests are run and the types of flakiness that can occur, and thus can serve as a cheat sheet when triaging and fixing flaky tests.

References

Where do our flaky tests come from?
Flaky Tests at Google and How We Mitigate Them
My Selenium Tests Aren't Stable!
TotT: Avoiding Flakey Tests
Test Flakiness - One of the main challenges of automated testing

3 comments :

Greg PaskalMarch 25, 2021 at 5:50:00 AM PDT
There are two main reasons for flaky automated tests.

1) Poor Locator Strategy. Find a methodology that is testable before you have to depend upon it in your automated testing. I just posted a video on this topic a week ago, which shares why our team exclusively utilizes xPath for some of the most reliable locators you can build. Realize xPath has got a bad rap and is often demonstrated online in some awful ways. This video will show you the beginning of how to build reliable locators. https://realworldtestautomation.com/basic-xpath-example/

2) Poor Synchronization Strategy. You should never need any type of sleep or wait in your test automation. These are almost always present in flaky automation, and it's no wonder why. Wait() is as good as Hope() when determining how long is long enough to pause for something to happen. Instead, build your automation with a more intelligent synchronization strategy based on these four fundamental methods. Exist, Not Exist, Wait to Exist, Wait to Not Exist. So the foundation to using these four synchronizations is to determine the state of a locator. You should be able to ask any locator does it Exist or Not Exist at any moment? You should also be able to ask your automation to check for a locator to exist for a determined period of time (Wait to Exist). Likewise, you should be able to ask your automation to check if a locator no longer exists for a determined period of time (Wait to Not Exist).

If you adopt these two fundamentals to your automation strategy, the majority of your flaky test will go away without any additional actions necessary.

Pursue the Craft of Testing in excellence.
ReplyDelete
Replies
Guillermo ChussirJune 6, 2021 at 11:26:00 AM PDT
Excellent guide!
Most of the flakiness I have come across come from the the tests themselves.
Totally agree with the Remedies recommended there.
ReplyDelete
Replies

Add comment

The comments you read and contribute here belong only to the person who posted them. We reserve the right to remove off-topic comments.

Testing Blog

Test Flakiness - One of the main challenges of automated testing (Part II)

3 comments :

Labels

Archive

Feed