Next Webcast: DevProdEng Lowdown: How Uber does Developer Productivity Engineering – July 14th (10am – 11am PDT). Learn more and register here.
Reduce Test Times Up to 90% with
Test Distribution is a modern approach to test parallelism that dramatically accelerates test execution by taking existing test suites and distributing them across remote agents to execute them faster—locally and on CI. The Gradle Enterprise Test Distribution product extension supports Gradle and Maven build tools.
Key Test Distribution Solution Benefits
Test time is one of the key drivers of build times. There are many factors that contribute to test time, including the growing number of integration and functional tests to cover a wide range of inputs, running tests sequentially, and dependencies on expensive external services. The Gradle Enterprise Test Distribution product extension for the Gradle and Maven build tools expands test parallelism by fanning out test execution to remote agents that run on premises inside your own network.
Tests are partitioned and distributed based on historical execution times. Test output files, such as code coverage reports, are transferred back from the agents and merged automatically. The number of remote agents can be scaled on demand via auto-scaling (e.g., when running in a Kubernetes cluster). The scheduling of tests during the build is adaptive and utilizes new agents as they become available. Finally, Test Distribution works for local and CI builds so developers no longer need to wait for their CI server to run tests.
Faster test feedback cycles
By distributing text execution to remote agents, test times and overall build times can be reduced up to 90%.
Improved DevProd and Developer Experience
Minimizing test time makes developers more productive and happier since they spend less idle time waiting for feedback.
Fast test feedback cycles mean developers build more often rather than pushing quality checks to later stages of the development cycle.
Key Test Distribution Features
|Uses single-machine parallelism when no agents are available or in combination with remote agents (ensures test distribution will never be slower)||Requires a simple change to the build configuration to enable||Partitions tests sets optimally based on text execution history.|
|Works for local and CI builds||Show the results of all tests via test reports and in the Build Scan™||Makes Information about used agents available in the Build Scan™|
|Transfers and merges outputs back to the build automatically (e.g., JaCoCo)||Allows running a subset of tests on specific agents based on requirements specified in the build configuration (e.g., certain tests should only run on agents with a specific OS)||Runs on-premise inside your own network infrastructure|
|Supports auto-scaling remote agents (e.g., via a Kubernetes cluster)||Resilient against temporary network failures||Utilizes remote agents as they become available to perform adaptive scheduling|
Key Advantages of Test Distribution Relative to Alternative Approaches
A good way to understand the advantages of Test Distribution is to understand the limitations of alternative approaches to test parallelization:
Not being able to parallelize your tests is a serious constraint which will become more and more costly as your test base grows, ultimately resulting in test times that take an order of magnitude longer to run compared to modern test distribution solutions.
- The size of a test set tends to grow along with its associated project, with very large or complex applications spending the majority of their time in the test phase of the build.
- Without a plan for parallelization of tests, test feedback cycle times will increase continuously as more tests are added to the set.
- New tests are necessary when new features are added, and though Test Impact Analysis can help to reduce the test set executed during each build, there are many scenarios where running tests is unavoidable.
Single-machine parallelism often results in a significant acceleration, but is limited by the number of parallel forks, which is dictated by a machine’s CPU and memory resources. This has several implications:
- Tests that consume a lot of local developer machine resources become unavailable for other tasks.
- Organizations invest in expensive, powerful machines to run CI agents and improve test execution time, resulting in massively underutilized machine capacity. As soon as a CI build starts, CI machine resources are fully dedicated to CI builds and are not available for other jobs.
- Resource sharing conflicts often result since single-machine parallelism may dictate running tests in multiple local JVMs to avoid interfering with each other. They share the same file system and often the same “process space” so servers started from tests need to use different ports.
CI fanout achieves parallel execution by creating multiple CI jobs per build, where each CI job runs on a subset of the tests on a different agent. While valuable when compared to not leveraging any form of test distribution, CI fanout has many disadvantages:
- There is no aggregate test reporting since test reports are separated per agent.
- There is no efficient load balancing across agents as test partitioning must be done manually.
- Significant CI administrative overhead may result from running tens or hundreds of CI jobs for the same logical build step.
- Debugging is inefficient since the same builds are run many times, each generating its own build log.
- Feedback cycles and resource consumption may be negatively impacted by the necessity to run full builds per agent.
- CI fanout only works on CI. As a result, a pull-request has to be created just to get fast feedback on tests, negatively impacting developer productivity and the developer experience.
Modern Test Distribution (TD)
Modern fine-grained and transparent TD solutions offer many advantages:
- TD is available for all local and CI builds regardless of where invoked (e.g., local command line, IDE).
- There is no change to the developer workflow since tests are run automatically in a distributed and transparent manner, instead of run locally.
- The CI user experience is better compared to CI fan out since there is only one build invocation and CI job. If properly implemented, test report aggregation happens as if all tests are run on one agent. There is only one build log.
- Resource consumption is dramatically reduced since the prerequisite build steps for running tests run only once.
- Reliability is increased since efficient fine-grained distribution allows for sequential execution of tests per agent.
- TD history and dynamic scheduling services can be used to balance the test execution load properly and elastically between test agents and optimize test execution time and compute resource consumption.
- To maximize available resources, TD supports local test execution fallback if no remote test agents are available or an optional hybrid local/remote execution mode.
Test Distribution Test-Time Savings Results
Here are actual test time savings experienced by Gradle Enterprise Test Distribution customers and how that translates directly into hard annual savings in developer years and cost.
Combine Test Distribution with other Performance Acceleration Technologies for a Force-Multiplier Effect
Combining Test Distribution with other performance acceleration technologies can provide a force-multiplier effect on feedback cycle times as illustrated below.
- Build Cache. Avoid unnecessarily running components of builds and tests when inputs have not changed.
- Predictive Test Selection. Run only tests that are likely to provide useful feedback using machine learning.
- Test Distribution. Run the necessary and relevant remaining tests in parallel to minimize build time.
- Performance Continuity. Sustain Test Distribution and other performance improvements over time with data analytic and performance profiling capabilities.