Flaky, or non-deterministic, tests are a serious and prevalent problem in modern software development. If your application interacts with browsers, external devices or services, or has asynchronous behavior, it’s likely you have suffered from flaky tests. Martin Fowler has this to say about flaky tests:
Non-deterministic tests have two problems, firstly they are useless, secondly they are a virulent infection that can completely ruin your entire test suite. As a result they need to be dealt with as soon as you can, before your entire deployment pipeline is compromised. — Martin Fowler on Eradicating Non-Determinism in Tests
Flaky tests compromise deployment pipelines by slowing them down and decreasing confidence in the correctness of changes. When changes regularly fail due to unrelated flaky tests, time is wasted, features are delayed, and developers are demotivated to make changes.
Read More →