Writing tests is hard. Fixing them is harder. Join the webinar.
Sustain performance gains over time with
Performance Continuity
The Develocity Performance Continuity solution provides a comprehensive set of complementary analytic and diagnostic tools to maintain performance continuity of build and test times for Gradle, Maven, and Bazel build environments. It provides observability of regressions and profiling for performance issues and pathologies that can be used to trigger proactive performance improvement interventions often before developers even notice and begin to complain. Performance Continuity can be viewed as a sub-practice of the broader practice of Developer Productivity Engineering.
Build Performance Continuity Solution Overview
With very few exceptions, organizations have no relevant and actionable insight into the performance and reliability of the toolchain their developers are using. The tech industry that has enabled other industries to become so efficient by providing data visibility and analytics has so far not applied productivity practices to itself. Develocity Performance Continuity performance profiling data and analytic techniques to optimize and sustain build performance levels over time, given code architecture, technology stack, and other constraints. It is now easy to (1) continuously monitor build and test performance metrics and trends and (2) observe and effectively determine the root cause of performance pathologies, bottlenecks, and regressions.
Performance Continuity works hand in hand with the other Develocity Performance Acceleration technologies—which include Build Cache, Test Distribution, and Predictive Test Selection—to deliver an integrated performance acceleration strategy. This means these technologies can be used to raise the theoretical maximum achievable performance and performance continuity techniques can then be used to sustain and minimize the variance of actual gains over time.
A critical mistake is to view these performance acceleration technologies as “fire and forget” solutions. Unmanaged, performance optimization opportunities will be squandered, and performance will regress overtime as the codebase grows in size and complexity, leading to potentially negative savings. Performance Continuity practices and the enabling technology provided by Develocity aim to prevent that.
Key Build Performance Continuity Solution Benefits
Consistent Build & Test Performance Levels
As inputs continuously change and the number of developers, lines of code, repositories, and dependencies grow, build performance can slowly regress and result in an unnoticed and suboptimal “new normal” for feedback cycle times. Performance Continuity tools are used to ensure that product delivery times are not systematically impacted by codebase and toolchain changes and cycle times continuously and incrementally progress towards the maximum achievable.
Improved DevProd and Developer Experience
Used in combination with performance acceleration technologies like Build Cache, Test Distribution and Predictive Test Selection, Performance Continuity tools help minimize build and test time. This makes developers more productive and happier since they spend less idle time waiting for feedback.
Fact-based Decision Making
Developer productivity stakeholders are so used to relying on tribal knowledge and outdated assumptions about their build environment, they often fail to see or believe what’s at the root cause of their performance issues. The collection of “factual” data may positively change the cultural environment in a way that will increase the success rate of your continuous improvement initiatives.
Observability for Proactive Management
Performance continuity provides analytic and trend data so that deviations from baselines and actual historical performance norms can be easily observed and dealt with proactively. This avoids being incident driven. This data is also used to prove the value of DPE best practices to management and increase the incentive of productivity engineers to take up the challenges since they now have reliable tools to measure their impact.
Build Performance Continuity Features
Develocity Performance Continuity provides comprehensive build and test performance profiling that is easily accessed and organized in a well-structured UI that facilitates root cause analysis for collaborative performance troubleshooting and continuous proactive improvement.
This results in increasing (1) the motivation of developers to report problems since they are more likely to get addressed than reports with limited context and data, (2) the rate at which problems are addressed successfully since it is easier to detect the root cause, and (3) the ease with which performance problems can be prioritized and determined to be worth fixing since it is easier to determine the overall impact on the organization.
Individually and collectively, detailed build performance data described below can help to qualify decisions about where to invest in further build optimization and where specifically to focus your experiments.
Performance Trends Dashboard | Build Timeline Profile | Build Performance Details Profile |
This is an interactive and multidimensional visualization of build performance metrics over day, week, and month time intervals and across builds that match your search criteria and filters. This allows easier discovery and analysis of builds with specific performance characteristics or anomalies related to build time, task parallelization, incremental build and build cache effectiveness. | The Build Scan build timeline provides an intuitive way to visualize the order in which build actions are executed, the critical path and concurrency of execution, execution duration, and details like time spent “snapshotting” inputs, compiling code, downloading dependencies, and running annotation processors. This makes it easier to assess where best to invest in optimization. | Build Scan provides a simple UI and record of the performance details for every build including total build time, initialization and config time versus execution time, total garbage collection time, and peak heap memory usage. |
Configuration Time Profile | Dependency Resolution Profile | Build Action Execution Profile |
Build Scan configuration time profiling provides total configuration time and breaks configuration execution time between the time spent compiling build scripts versus time spent applying plug-ins. This helps prioritize investments in minimizing configuration time and determine which bottlenecks to focus on like lifecycle callbacks. | Build Scan provides details on the time spent resolving dependencies for individual builds and Performance and Trend Dashboard visualizations show how much build time is spent obtaining dependencies. This is very useful for identifying the build time cost of ephemeral CI builds, poor network performance, or frequent dependency changes. | This provides an aggregated view of build execution details within Build Scan, like work avoidance statistics with supporting data (e.g., cache hit/miss metrics and incremental build savings). A ledger of Gradle Tasks, Maven Goals, or Bazel Targets is combined with these aggregate statistics so that the execution performance of each step of the build is understood. |
Build Cache & Test Distribution Profile | Predictive Test Selection Profile | Network Activity Profile |
Build Scan tracks cache hit/miss rates for each build. This data can be used to determine whether individual builds are taking full advantage of available remote and local caching resources, and view how much overhead is associated with caching. Test Distribution profiling information includes details on time saved resulting from parallel execution of tests. | The Predictive Test Selection Dashboard provides data accuracy and explainability metrics and details for the test sets it optimizes. Users can view the percentage of test failures that were correctly predicted and the overall work avoidance savings. Also, information on why Predictive Test Selection chose to run or skip a test can be viewed for each evaluated test. | Easily accessible network activity profile information includes names of all files downloaded, the size of all files downloaded and the download times. Knowing how much time is spent during a build waiting on network resources helps guide decisions about an organization’s artifact repository infrastructure strategy. |
Switches & Infrastructure Profile | Test Performance (Slow Test Analysis) Analysis | Settings and Suggestions |
Available at-a-glance on/off switch settings and execution environment information about the JDK used to execute the build, the machine’s host name, operating system, number of cores, memory allocation and more, is relevant to understanding build and test performance outcomes. | Test performance can be viewed on an individual-test and test-class level. Slowest test views identify the longest running tests within a single build and across builds to determine which tests are worth making faster. Filters can identify the slowest tests within a subset of a single task (e.g., within a particular test task or goal) while the Tests Dashboard makes identifying the slowest tests across builds easy. | Settings and suggestions provide further actionable insights like why a task was not cacheable, if parallel execution is enabled, or whether the build was run via the daemon which makes builds much faster. |
Performance Continuity Engineering Framework
Here is a simple framework which demonstrates how the analytic data, insights, and performance profiling information available in Develocity can be used to achieve high levels of performance continuity. Without the Performance Continuity data provided by Develocity, it is almost impossible to find performance related issues that occur irregularly (like so-called flaky performance issues) because they are extremely difficult to reproduce. But with the data of an instrumented toolchain at your fingertips, this is now practical to pursue.
Performance Continuity In Action
Tests Dashboard showing the build search criteria and (mean execution time) filter used to identify and prioritize slowest tests for remediation.
Use the Trends Dasbhoard to observe when your Maven or Gradle build times are regressing and the Performance Dashboard to understand why.
Combine Test Distribution with other Performance Acceleration Technologies for a Force-Multiplier Effect
While Performance Continuity does not depend on other Performance Acceleration technologies available in Develocity to be useful in observing and tuning your build performance achieved through existing processes and tools, combining Performance Continuity with Build Cache, Predictive Test Selection and Test Distribution can provide a force multiplier effect on feedback cycle times as illustrated below.
- Build Cache. Avoid unnecessarily running components of builds and tests whose inputs have not changed.
- Predictive Test Selection. Run only tests that are likely to provide useful feedback using machine learning.
- Test Distribution. Run the necessary and relevant remaining tests in parallel to minimize build time.
- Performance Continuity. Sustain Test Distribution and other performance improvements over time with data analytic and performance profiling capabilities.