Failure Analytics

Eliminate Flaky Tests and Other Avoidable Failures

Unreliable builds cause downtime, waste compute resources and are a massive distraction. They also negatively affect the quality of the code that is shipped. Builds become unreliable when problems are too expensive to find, too hard to reproduce for root cause analysis, and when fixes can not be correctly prioritized because their relative impact is unknown. With Gradle Enterprise you can leverage analytics to proactively find unreliable builds and tests, learn how many people and environments are affected by the problem, share information about them, and understand the root cause efficiently.

The Gradle Enterprise Failure Analytics solution consists of three components: Flaky Test Management, Test Failure Analytics and Build & CI Failure Analytics.

Primary Benefits

End-users and customers have a great experience with better quality and faster updates

Developers have confidence in the test suites which encourages a culture of accountability and good behavior (like building earlier and more often and fixing broken tests)

Management improves metric and KPI outcomes used to drive business success, such as productivity and efficiency, quality of service, and speed of delivery

Key Solution Components

Flaky Test Management

A flaky test is a non-deterministic test caused by code that produces both a “passing” and “failing” test result. Gradle Enterprise detects flakiness by re-running failed tests as part of the same build execution. When they succeed after re-trying, the test is classified as flaky, but it still must be remedied.

Flakiness is not necessarily rooted in the test, it can also indicate flaky behavior in your product. Without a remedy you shift the consequences of unreliable code to users and customers and flakiness accumulates, which increases build and test time and wastes valuable compute resources. A less obvious consequence is the degree to which flaky tests poison your culture by reducing confidence in your test suite. This may result in engineers building less often and paying less attention to writing tests.

To address these pains, Gradle Enterprise provides Flaky Test Management to help you proactively detect, flaky tests and how often they break your build. This allows you to prioritize which if any to fix first.

Detect flaky tests reliably for all Maven and Gradle builds including local ones by using a retry-mechanism and Build Scans.

Prioritize flaky test remediation based on frequency of occurrence and impact.

Provide you with flaky test history to measure how their negative impact is decreasing or increasing over time.

Fix flaky tests efficiently using data reported by the Test Failure Dashboard and associated Build Scans. This includes test history across all local and CI builds, common traits among flaky runs, differences when compared to stable runs, and test methods used for specific runs.

Test Dashboard with Flaky Test Enabled

Flaky test report in Gradle Enterprise. Chart with new flaky test detected.

Test Failure Analytics

Test failures are the most common cause of build failures and many test failures are unnecessary. Further, not  all unnecessary test failures are flaky tests (e.g. unstable environments also cause a large number of test failures).  Test Failure Analytics addresses the systemic issues that result when these unnecessary test failures are left unaddressed and allowed to fester. The result is compounding test suite failures and the resulting lack of motivation by developers to address the problems because it’s gotten to the point where it’s difficult to determine where even to start. If only there was an easy way to triage failed tests and separate the healthy tests that detected a real bug in production code, from flaky tests, and completely broken tests that fail frequently and require immediate attention.

Determine priorities for remediation by generating a list of test cases grouped by classes and packages that fail most frequently.

Get a timeline view of when particular tests started to fail (“before” fix picture) which can be used to understand your starting baseline level of test failure volume and can be helpful in root cause determination.

Streamline root cause analysis with a listing of build environment metadata for failed test executions.

View performance characteristics of failed test executions compared to passing ones.

Verify the establishment of a new reduced test-failure-frequency baseline and monitor on an on-going basis with an extended timeline view (“after” fix picture).

Analyze test failures by Gradle task or Maven goal with a dashboard of Build & Test Failure Trends for local and CI builds.

Test Failure Dashboard

Search failed and flaky tests. Find failed local builds on git master branch.

Build & CI Failure Analytics

Build & CI Failure Analytics are used to detect and prioritize avoidable build failures for remediation based on impact and determine the root cause of the incident faster.

Classify non-verification (a.k.a avoidable) build failures from verification failures (e.g. JUnit, Checkstyle.)

Fix non-verification failures efficiently using data reported by the Failure Dashboard and associated Build Scans. This includes failure history across all local and CI builds, common traits among failed runs, differences when compared to successful runs

Prioritize remediation for failures based on the frequency of occurrence and impact.

Search build failures using any part of an error message (as you would on StackOverflow)

Provide you with non-verification failure history to measure how the negative impact of failures is decreasing or increasing over time

Build & CI Failures Dashboard

Search failed and flaky tests. Find failed local builds on git master branch.

Get started today

Request a 30-day free trial of Gradle Enterprise.
Free 30-day trial