Discover the only event dedicated to Developer Productivity Engineering and Developer Experience—September 24-25 in San Francisco.
Develocity 2023.4
Develocity 2023.4 provides an additional detection mechanism for flaky tests, insights into artifact transform executions for Gradle builds, new API endpoints to access aggregate information such as the slowest and flakiest tests across a set of builds.
Read on for more details of these and other new features.
Use version 1.20.1 of the Develocity Maven extension, version 3.16.2 of the Develocity Gradle plugin, version 0.10.1 of the Develocity sbt plugin, version 2.1 of the Develocity Test Distribution agent, and version 18.1 of the Develocity Build Cache node for optimal usage of Develocity.
Use the latest versions of the Gradle Jenkins plugin, Build Scan plugin for TeamCity, Develocity Bamboo plugin, Gradle Build GitHub Action, and Develocity Gitlab Templates for optimal integration of your CI provider with Develocity.
Highlights
Detection of flaky tests across builds
In addition to tests retried in the same build with different outcomes, Develocity now identifies flaky tests across multiple builds. A test class or method is considered cross-build flaky when it had different outcomes for the same task/goal inputs across different builds. Develocity displays an icon to the right of the original test outcome to indicate a cross-build flaky test.
Whether a test is flaky across builds is independent of whether a build failed or passed. See the Develocity Flaky Test Detection Guide for more information about flaky test detection.
Increased predictive accuracy of Predictive Test Selection
Improvements to the machine learning model used by Predictive Test Selection increase the likelihood of predicting test failures, with little to no loss in time savings. When using the standard selection profile, testing on various enterprise and open-source projects showed an average of 49% fewer missed test failures than the previous version of Develocity. This was done by increasing the size and variety of the training dataset as well as updating the model’s input features to include more relevant information.
If you are using Predictive Test Selection, please be sure to read the upgrade notes below before upgrading.
Support for Cucumber tests in Test Distribution and Predictive Test Selection
Develocity users can now benefit from Predictive Test Selection and Test Distribution when running Cucumber tests. See the Develocity Predictive Test Selection User Manual and Develocity Test Distribution User Manual for more information.
Insights into Gradle’s artifact transform executions
Build Scan® for Gradle now provides build cache effectiveness and other finer-grained performance insights for artifact transform executions, building on the observability of planned artifact transforms that were added to the Timeline view in Develocity 2023.1.
Artifact transforms are a concept from Gradle’s dependency management. When Gradle resolves artifacts for a configuration and finds that they are not in the required format, Gradle can convert them to the requested format by executing artifact transforms. Users do not directly request these transforms; if a task requires artifacts as input, Gradle executes the appropriate transforms before the task runs or as part of task execution. Artifact transforms can be cacheable and benefit from the same avoidance optimizations as tasks.
Insights into artifact transform executions can be found on a new tab of the Performance view. A breakdown of the artifact transform executions allows you to spot possible performance problems in the build at first glance.
Users can drill down into the different categories from the breakdown to investigate certain categories or individual executions. Additional search criteria are available to restrict the data to what is necessary for the current investigation.
To find the most expensive artifact transform executions, users can sort the list by longest duration or group by transform action type.
The Requests tab on an individual artifact transform execution allows navigating to the tasks that requested the artifact transform execution.
Similarly, for an individual task, the Transform requests tab allows navigating to the artifact transform executions requested by the task.
Finally, the Develocity API provides the new effectiveWorkUnitExecutionTime and serialWorkUnitExecutionTime fields that include task and artifact transform executions.
Retrieve aggregate test information via the Develocity API
New API endpoints to retrieve test information for Gradle, Maven, and Bazel builds have been added in this release. They provide aggregated information about tests for a set of builds, such as which tests were the flakiest or slowest.
For example, these endpoints can automate monitoring for test classes or test cases that have recently been flaky or generating a periodic report on the slowest tests. The Develocity API sample application has been updated to demonstrate the usage of the new API capability.
See the reference documentation for the /api/tests/containers and /api/tests/cases endpoints in the Develocity API Documentation for more information.
Retrieve build network request information via the Develocity API
New API endpoints for Gradle and Maven provide information about dependency downloads and expose the impact of network requests on build time.
See the reference documentation for the /api/builds/{id}/gradle-network-activity and /api/builds/{id}/maven-dependency-resolution endpoints in the Develocity API Documentation for more information.
Retrieve detailed build information via the Develocity API more easily
Develocity 2023.3 added the ability to filter builds returned by the /api/builds API endpoint using a concise and powerful query language. This made it easier to work with a subset of builds, but still required one or more further API calls per matched build to access detailed information about each build. API consumers can now retrieve detailed information as part of the build list, simplifying API clients.
The /api/builds/{id} and /api/builds endpoints now support a multi-valued “models” parameter that specifies which additional information to include. The available information is the set of API endpoints available for a specific build, such as /api/builds/{id}/gradle-attributes or /api/builds/{id}/maven-build-cache-performance. To include the information these endpoints provide, you can use the endpoint names as the “models” parameter, e.g., /api/builds?models=gradle-attributes&models=maven-build-cache-performance.
See the reference documentation for the /api/builds/{id} and /api/builds endpoints in the Develocity API Documentation for more information.
Filter builds by build options and environment
Users can now filter builds by build options and the build environment. This allows for powerful queries to identify trends, compare different sets of builds, and identify issues. For example:
- Viewing performance trends of builds with specific build options enabled, such as the build cache
- Viewing test failures restricted to a subset of build hosts, based on hostname or resources like CPU and JVM memory
- Verifying that no builds are running without the expected set of build options
This filtering is available via the advanced search query in the Build Scan dashboards and when querying the /api/builds API endpoints.
See Advanced search syntax in the Develocity API User Manual for more information.
Filter builds by build cache errors
Users can now also filter builds to find those that had build cache errors. This is useful in identifying configuration issues in different environments and verifying a healthy build environment.
This filtering is available via the advanced search query in the Build Scan dashboards and when querying the /api/builds API endpoints. Finding errors via the API allows custom automation to collect metrics on or alert when errors occur, helping keep your infrastructure operating efficiently.See Advanced search syntax in the Develocity API User Manual for more information.
Upgrade notes
After upgrading your Develocity server to the most recent version, it is recommended to upgrade your builds to use the latest version of the Develocity Gradle plugin, Develocity Maven extension, Develocity sbt plugin, Develocity Test Distribution agent, and Develocity Build Cache node. Using the latest versions of all components ensures that you benefit from security updates, bug fixes, and feature enhancements. Please check the version compatibility matrix for further reference.
It is also recommended to update your configuration of the Gradle Jenkins plugin, Build Scan plugin for TeamCity, Develocity Bamboo plugin, Gradle Build GitHub Action, and Develocity Gitlab Templates to reference the latest version of the Develocity Gradle plugin.
Pre-upgrade database configuration required for superuser-less user-managed databases
Develocity installations can use an embedded database provided as part of the installation or a user-managed database for superior performance and operability. When using a user-managed database, Develocity can be configured with superuser access to the database so that it can initialize and configure the database appropriately. Alternatively, this initialization and configuration can be executed manually, avoiding the need to provide Develocity with superuser access to the database.
For installations using the superuser-less approach, new configuration requirements in Develocity 2023.4 necessitate manually executing an updated configuration script as part of the upgrade.
If your installation uses the superuser-less approach, download the new configuration scripts from the Develocity Upgrade Guide to obtain the new configuration scripts before upgrading. If you are unsure whether your installation uses this approach and requires this extra upgrade step, please contact Develocity support.
Temporarily degraded performance due to data reindexing
Upon upgrading, a data reindexing process will be initiated in the background, with Develocity being usable for its duration. CPU usage will be increased, and performance may be slightly degraded. The reindexing process may take several hours or potentially several days for large installations storing months of build history. During this time, some builds may be omitted from cross-build data visualizations. Recent builds are prioritized, making their data available sooner.
Predictive Test Selection is temporarily unavailable after upgrading
The improved machine learning model in this release requires re-analyzing build history, which is initiated automatically after upgrading. Attempting to use Predictive Test Selection while this occurs will result in all tests being selected for execution. The duration of this process depends on the amount of build history stored by your installation and the availability of CPU resources. It is estimated to take ~24 hours for an installation with a build history of 6 months of 60,000 builds per week and eight processor cores. Due to this, installations using Predictive Test Selection are recommended to be upgraded at the end of the week to allow this process to be completed over the weekend. A banner will be displayed on the Predictive Test Selection dashboard while this re-analyzing occurs.
Predictive Test Selection with Develocity 2023.4 requires Develocity Gradle plugin 3.16+ or Develocity Maven extension v1.20+. Attempting to use Predictive Test Selection with older versions will result in all tests being selected for execution. The Predictive Test Selection Simulator is unaffected by this requirement.
Review roles to ensure the correct set of users have the "Use Predictive Test Selection" permission
This release adds a dedicated ‘Use Predictive Test Selection’ permission. Previously, permission to use Predictive Test Selection was granted by the ‘View build scans and build data’ permission.
After upgrading, Develocity will automatically assign the ‘Use Predictive Test Selection’ permission to all existing roles with the ‘View build scans and build data’ permission. If your organization uses Predictive Test Selection it is recommended that you review all of the roles to ensure the ‘Use Predictive Test Selection’ permission is assigned to only the appropriate roles.
Review roles to ensure the correct set of users have the "Generate Support Bundle" permission
This release adds a dedicated ‘Generate Support Bundle’ permission. That permission grants generating support bundles from the web interface. Previously, administrative users needed to provide the system user’s password to generate a support bundle.
After upgrading, Develocity will automatically assign the ‘Generate Support Bundle’ permission to all existing roles with the ‘Administer Develocity’ permission. It is recommended that you review all of the roles to ensure the ‘Generate Support Bundle’ permission is assigned to only the appropriate roles.
Some access keys may be invalidated after upgrading to 2023.4.3
Starting with Develocity 2023.4.3, when authenticating with an access key, the issuer of an access key is validated. The issuer is based on Develocity’s public hostname. If the issuer on the access key does not match Develocity’s current public hostname, then the access key will be rejected and invalidated.
If a user’s access key was created before a Develocity hostname change (even if the hostname was updated on an earlier version of Develocity), then the access key will be invalidated the first time the user tries to use the access key after Develocity has been upgraded to 2023.4.3 or higher. In the My Settings / Access Keys page, the access key will be marked as “expired”. Clicking Reactivate will refresh the access key so that the issuer matches the current Develocity hostname, and the access key will once again be accepted for authentication.
If the Develocity public hostname has never been changed, then no access keys will be invalidated.
Admin CLI tools merged into new develocityctl with new Java 17 requirement
The admin-cli and scans-copy-cli CLI tools have been merged into the new “develocityctl” tool. Furthermore, the minimum JRE version required when using the JAR version of the tools has been increased from Java 11 to Java 17. Please refer to the Develocity Admin CLI User Manual for more information.
Users of the provided Docker image do not need to take any action as it provides an appropriate Java version.
The release of the develocityctl tool that will accompany the next major Develocity release will require a Java 21 runtime.
Migration to Helm-based object storage configuration is now simplified
For customers migrating the object storage configuration to the Helm-based configuration introduced in 2023.3, it is now simpler to remove the previous configuration. If both are set, a simple interface is now visible in the application administration interface to remove the old configuration once the new one is in place and working. For customers using advanced parameters to configure the object storage, verify that they are migrated to Helm. See the migration guide for more information.
Develocity Helm Chart upgrade notes
- Internal proxy logs are written to syslog instead of a file (includes a new container using 50m CPU & 32Mi memory)
- Init containers resources have been adjusted to match containers’ resources and improve performance.
- Develocity ‘monitoring’ component memory was increased (requests: 256 Mi to 512 Mi, limits 512 Mi to 768 Mi). Configuration was adjusted to ensure a better retention period, data compaction, and improved performance.
Changes
Feb 23, 2024
- [FIX] When generating a support bundle from the Admin UI, clicking other UI elements may cancel the bundle generation process
- [FIX] The Develocity Build Cache eviction process may fail to finish and cause 'out of disk' errors
Feb 13, 2024
- [FIX] Develocity may disable the build cache when storing cache data in object storage
- [FIX] Builds using Test Distribution are slower and may fail when Develocity is under load
Feb 08, 2024
- [FIX] Build Scan Timeline view does not render for planned transform nodes without execution requests
- [FIX] Tests dashboard may not render if multiple work units have the same fingerprint
- [FIX] Users cannot log in if their identity provider sends large SAML responses
- [FIX] The Build Cache uses unnecessary space for object storage
Jan 30, 2024
- [FIX] Build Scan Timeline view may not render for very large builds
- [FIX] Build Scan Timeline view never returns artifact transforms when searching by cacheability
- [FIX] Build Scan Tests view fails to handle not selected but executed test classes
- [FIX] Build Scan Tests view shows inconsistent task/goal outcome on the Overview page
- [FIX] Using project-level access control, advanced search queries fail for builds with build cache errors
- [FIX] Cleanup of Build Scan data fails if the payload does not exist in the GCP bucket
- [FIX] Build Cache node has socket read timeouts in some circumstances
- [Fix] Kubernetes may repeatedly restart pods due to liveness probe failures
Dec 22, 2023
- [FIX] Predictive test selection data may not be fully rebuilt after the upgrade
- [FIX] Data processing has errors when using project-level access control in some circumstances
- [FIX] Querying custom values with a comma results in an error
- [FIX] Incorrect "Miss (local)" build cache outcome is shown for non-cacheable transform executions
- [FIX] Build Scan performance page shows an error if the build ran with Gradle 8.6 with configure on demand and included-build logic
Dec 14, 2023
- [NEW] Support bundle generation progress is conveyed to users via Admin UI
- [FIX] Build scan data not visible for a period after upgrade
- [FIX] Metrics database extraction is more reliable during Support Bundle Generation
- [FIX] Develocity rejects license files with minor formatting changes
- [FIX] Access key section highlights correct menu item
- [FIX] PTS requests for older plugins do not fail with internal error when contacting 2023.4
- [FIX] Tests overview page in Build Scans crashes when executor released events are missing
Dec 05, 2023
- [NEW] Detection of flaky tests across builds
- [NEW] Increased predictive accuracy of Predictive Test Selection
- [NEW] Support for Cucumber tests in Test Distribution and Predictive Test Selection
- [NEW] Insights into Gradle’s artifact transform executions
- [NEW] Retrieve aggregate test information via the Develocity API
- [NEW] Retrieve build network request information for via the Develocity API
- [NEW] Retrieve detailed build information via the Develocity API more easily
- [NEW] Filter builds by build options and environment
- [NEW] Filter builds by build cache errors
- [NEW] Improved build cache horizontal scalability
- [NEW] Build Scan test method view shows Test Distribution agent, pool, and assignment reason
- [NEW] Permission to use Predictive Test Selection is granted by dedicated permission
- [NEW] Gradle Build Scan Performance section overview is simplified
- [NEW] Support bundle generation requires dedicated permission instead of system user password
- [NEW] Project ID is shown in Build Scan and build comparison sections
- [NEW] The task cache key is now included in the gradle-build-cache-performance API endpoint of the Develocity API
- [NEW] The goal cache key is now included in the maven-build-cache-performance API endpoint of the Develocity API
- [NEW] Serial execution factor is shown in Gradle and Maven Build Scans
- [FIX] Requested tasks filter for Gradle configuration-cache hit builds incorrectly matches tasks
- [FIX] The Timeline view incorrectly counts projects when filtering by Transform kind for Gradle Build Scans
- [FIX] Use of Predictive Test Selection may cause excessive database disk space usage
- [FIX] Gradle input file comparison fails when input directories outside of project directory share names with inputs inside project directory
- [FIX] Incorrect class outcome is displayed for flaky tests where no whole retry round was successful
- [FIX] Dedicated background processing component cannot be used when Network Policy is enabled
- [FIX] Test outcomes for builds where start time and upload time are on different days may be incorrect
- [FIX] Dedicated background processing component cannot be used in conjuction with project-level access control