Develocity 2024.1

Develocity 2024.1 marks the general availability release of Develocity for sbt, the build tool of choice for Scala projects, which includes the following new capabilities:

  • Timeline view of sbt build scans
  • Filtering tasks by name in the Console Log
  • Test results for all executed tests, including flaky tests
  • Test failure analysis
  • Build failure analysis

The release also provides new Build Scan® insights into configuration caching for Gradle builds and several Build Scan improvements for Bazel builds. In addition, this release enables short-lived access tokens for improved security, project-level access control for Test Distribution agent pools, and the ability to debug tests remotely executed on Test Distribution agents.

Another exciting new development is the addition of the Develocity Reporting and Visualization set of tools, which facilitates custom analysis of Develocity build data by allowing build data to be queried with an SQL-like language and visualized using the Grafana visualization platform.

Read on for more details of these and other new features.

Use version 1.21.4 of the Develocity Maven extension, version 3.17.4 of the Develocity Gradle plugin, version 1.0.1 of the Develocity sbt plugin, version 3.0.1 of the Develocity Test Distribution agent, and version 19.0 of the Develocity Build Cache node for optimal usage of Develocity.

Use the latest versions of the Gradle Jenkins plugin, Build Scan plugin for TeamCity, Develocity Gitlab templates, GitHub Gradle build action and Develocity Bamboo plugin for optimal integration of your CI provider with Develocity.

NOTICE: The 2024.1 release provides a new Develocity build cache service that replaces the existing service. Please be sure to read the accompanying upgrade notes to ensure optimal continuity of service after upgrading.

Highlights

Develocity for sbt is promoted to General Availability

The 2024.1 release of Develocity marks the general availability of Develocity for sbt, the most popular build tool for Scala projects. This release is packed with several features that substantially enhance the sbt Build Scan experience. Sbt build scans now provide visibility on all tasks executed during a build execution, report the result of each executed test, surface flaky test executions, and enable investigating tests and build failures across time.

You can publish an sbt Build Scan directly to the free scans.gradle.com service. If you are interested in connecting your sbt builds to your Develocity installation, please contact your customer success representative.

Task execution insights for sbt builds

The Timeline page of an sbt Build Scan provides a swimlane-style visualization of a build’s tasks, including detailed information about each executed task.

A lot occurs when you invoke common tasks such as compile, test, and package. The timeline helps tame this complexity, enabling you to identify and remediate the bottlenecks in a build.

See this pull request on the Apache Pekko project or this one on the Spotify Scio project for two real-world examples demonstrating how the timeline helps surface and address build bottlenecks.

Console log filtering for sbt builds

The Console log can now be filtered by a given task’s execution name, which is particularly useful for the intertwined logging of builds executed in parallel.

Tests results for sbt builds

The Tests section of an sbt Build Scan reports the results of all executed tests, including flaky tests.

The Slowest tests tab on the Tests page helps quickly identify the longest running tests across a build, which can be particularly useful for larger builds with several modules that contain tests.

Test failure analysis for sbt builds

This release of Develocity enables identifying and investigating test failure patterns, including flaky tests, on sbt builds. The charts at the top of the Tests page separate builds with failed tests from those hampered by flakiness. By default, a flaky test execution doesn’t result in a test failure.

Each test suite that you run can be explored, for instance, to understand which of the tests were a source of flakiness.

Read the Failure Analytics product page to learn more.

Build failure analysis for sbt builds

It’s now possible to identify and investigate failure patterns across sbt builds. The top failures report shows the most frequent failures, grouped by cause and charted over time.

Develocity separates source code verification failures, such as compiling, linking, testing, and linting from non-verification failures. This surfaces problems such as build configuration or infrastructure failures that are likely unexpected and disruptive.

Selecting a failure initiates a search for all occurrences of that failure, along with breakdowns by day, user, build hostname, and common tags.

You can search for failures that you’re interested in by their message. For example, you can search for https://your.artifactoryhost.com/ to quickly see how often builds fail due to an offline repository.

Read the Failure Analytics product page to learn more.

Test retry and flaky test detection for sbt builds

You can configure the Develocity sbt plugin to retry failed tests up to a user-specified maximum. If a test succeeds after a retry, the test is reported as FLAKY in the published Build Scan. This simplifies the task of identifying flaky tests while also enhancing the resilience of sbt builds.

To enable this functionality and allow up to three retries, include the following in your develocityConfiguration:

ThisBuild / develocityConfiguration :=
DevelocityConfiguration(
testRetryConfiguration = TestRetryConfiguration()
.withMaxRetries(3)
// rest of your Develocity configuration
)

By default, failed tests passing on retry prevent the test task from failing. This means that a build never fails because of a flaky test unless the maximum number of retry attempts is reached. This behavior can be changed, and it’s possible to fail a build if flaky tests are detected.

Develocity Common Custom User Data plugin for sbt

The sbt Develocity Common Custom User Data plugin enhances published build scans by adding a set of tags, links, and custom values that have proven useful for many projects built with Develocity. See the Develocity sbt plugin User Manual and the sbt Develocity Common Custom User Data plugin repository for more information.

New insights into configuration caching for Gradle builds

You can now delve deeper into configuration caching within a Gradle Build Scan. The Configuration tab of the Performance page is updated to show the overall result: HIT, MISS, or FAILED.

Furthermore, the Configuration tab also provides details about the configuration cache entry size on disk, which helps understand why it might be slow to store or load. If it’s a HIT, you can conveniently navigate to the origin build responsible for creating the configuration cache entry.

Improvements to the Bazel Build Scan® experience in Develocity

Develocity 2024.1 provides several additions and improvements for Bazel build scans, including the following:

  • Highlights Bazel’s exit codes for failing builds and tests to quickly differentiate a parsing error, build failure, test failure, or OOM error
  • Captures and displays stderr and stdout from failing actions, such as compilation errors
  • Gracefully handles aborted builds, displays test failures in the Failure page, and properly reports when tests time out

In addition, Develocity now displays Bazel’s reported Critical Path on the Timeline page to let users quickly see the longest chain of dependent actions with regard to wall execution time of Bazel build and test commands. Also, users can now directly download the Bazel profile.json file from the Timeline page to load into third-party profile viewers.

Debug tests that are executed remotely on Test Distribution agents

Tests executed remotely on Test Distribution agents can now be debugged using the standard debugging tools in your IDE. This allows developers to diagnose issues with tests that only occur when executed remotely. Moreover, this allows for a consistent experience when remote execution is preferred.

Restrict access to Test Distribution agent pools by project

When you enable project-level access control, it’s now possible to restrict the usage of Test Distribution agent pools to specific project groups. This is useful when Test Distribution agents provide access to secured resources, such as databases or other services, that should not be accessible to all projects. To prevent misconfiguration, such agent pools require pool-specific registration keys for authentication.

See Project groups in the Develocity Administration Manual for additional information about configuring project groups.

Performance information for Gradle artifact transform executions are now available from the Develocity API

A new endpoint provides information about individual artifact transform executions for Gradle builds. The endpoint allows users to monitor the performance of artifact transform executions, particularly those related to the build cache.

See Get the artifact transform execution list of a Gradle build in the API reference documentation for more information.

Gradle deprecation warnings are available from the Develocity API

This release adds a new endpoint to retrieve deprecation information for Gradle builds. The endpoint identifies the kinds of deprecations a build currently has and how often each kind appears in the build. Use the endpoint to monitor whether a build is ready for upgrade to the next major Gradle version.

See Get the Gradle Build Tool deprecations of a Gradle build in the Develocity API reference documentation for more information.

Create arbitrary reports and visualizations of Develocity build data

Previously, arbitrary and custom analysis of Develocity build data required creating or configuring systems to pull data from the Develocity API into an in-house system. With this release of Develocity, we are making this much easier to achieve by removing the need to create custom applications and providing specific support for a standalone end-to-end analysis stack using the Trino Distributed SQL query engine and the Grafana data visualization platform, or AWS Athena and Grafana for users on the AWS platform.

The standalone end-to-end stack, known as the Develocity Reporting Kit, can be installed separately from the Develocity server. It is available to all Develocity users at no additional cost.

The stack includes a set of useful predefined Grafana visualizations that visualize several aspects of Develocity build data, including network usage and dependency downloading to the slowest Gradle tasks and Maven goals across all builds and projects. These visualizations are based on Trino/Athena queries and also serve as useful starting points for developing custom queries and visualizations.

Please contact Develocity support to get started if you are interested in installing the Develocity Reporting Kit using the AWS-native equivalent stack, a native stack for a different cloud platform, or using different query or visualization tools.

Increase security with short-lived access tokens

Before 2024.1, access keys were the only available mechanism for builds to authenticate with Develocity, which are long-lived access credentials. Develocity now supports authenticating with short-lived access tokens. An access token is similar to an access key, except it is only valid for a short period of time and can be limited in scope.

See the appropriate manual for your build tool for instructions on how to generate and use access tokens:

Improvements to horizontal scalability of the build cache

Develocity provides a build cache service for Gradle, Maven, and Bazel builds as part of a server installation. This service now scales horizontally with the rest of the Develocity server, improving availability and supporting larger workloads. The remote build cache node, installed separately, can provide additional redundancy and geographically distributed presence.

Previously, each build cache node, built-in or remote, needed to nominate another node as a replication source when enabling replication between nodes. Now, this is simplified so that any replication-enabled node replicates to and from all other replication-enabled nodes, which always includes the Develocity server’s built-in build cache node. This eases replication configuration, particularly when creating a set of load-balanced remote nodes.

For more information about Build Cache replication, see the Replication section of the Develocity Administration Manual.

Develocity-based naming for build tool plugins and extensions

The Gradle plugin, Maven extension, and sbt plugin are now based on the “Develocity” product name. This affects plugin IDs, artifact coordinates, configuration DSLs, system properties, and general APIs.

Backwards compatibility with the “Gradle Enterprise” product name has been maintained, with the use of all symbols based on this name having been deprecated.

Please refer to the (Legacy) Gradle Enterprise Gradle Plugin User Manual and (Legacy) Gradle Enterprise Maven Extension User Manual for migration instructions.

Functional enhancements to CI integrations

The 2024.1 release renames all CI integrations for the new Develocity product name.

For more information, see the release notes for the CI integrations.

Upgrade notes

If upgrading from a version of Develocity prior to 2023.4, be sure to also consult the release notes for all interim versions.

After upgrading your Develocity server to the most recent version, it is recommended to upgrade your builds to use the latest version of the Develocity Gradle plugin, Develocity Maven extension, Develocity sbt plugin, Develocity Test Distribution agent, and Develocity Build Cache node. Using the latest versions of all components ensures that you benefit from security updates, bug fixes, and feature enhancements. Please check the version compatibility matrix for further reference.

It is also recommended to update your configuration of the Gradle Jenkins plugin, Build Scan plugin for TeamCity, and Develocity Bamboo plugin to reference the latest version of the Develocity Gradle plugin.

Build cache storage mechanism and configuration is changed

The 2024.1 release provides a new Develocity build cache service that replaces the existing service. Before upgrading, all Develocity administrators should read the build cache upgrade notes to guarantee migration of all existing build cache artifacts and to maintain any custom memory and resource settings:

Access control with username and password to a build cache node is deprecated and will be removed in 2024.3.

The Build Cache username and password access control is deprecated and will be removed in Develocity 2024.3.

Usages should be migrated to Develocity account-based access control. See Access control in the Develocity Administration Manual for more information on getting started with account-based access control.

If you currently use Build Cache username and password access control to allow shared access to the build cache, for example to allow CI build agents to read and write cache data, Develocity account-based access control supports this use-case. To setup a shared user account:

 

  1. Create a Develocity local user account and assign the account a role with the required Build Cache permissions. Develocity provides a ci-user role that facilitates typical CI agent access.
  2. Generate an access key for builds to authenticate with Develocity. To create a new access key, sign in to Develocity with the new account and access “My settings” via the user menu at the top right of the page. From there, use the “Access keys” section to generate an access key.
  3. Configure builds to authenticate with the generated access key. See Authenticated build access in the Develocity Administration Manual for how to get started with access key authentication for builds.

Review replication configuration for the build cache node

Develocity can now configure build cache nodes to replicate content both to and from all other replica-enabled nodes. This is the default replication mode for any new build cache nodes created with Develocity 2024.1.

Upon upgrade to 2024.1, existing build cache node replication associations operate in the pre-2024.1 (“Legacy”) replication mode: only build cache entries stored to the replication source are copied to the replication follower.

Administrators should enable the new replication mode for all build cache nodes, unless the node is in a “Legacy (follower)” role and builds are configured to write to the node. Please contact customer support if this is the case for advice.

To enable the new replication mode, select Enabled in the Replication section of the Nodes page for a build cache node.

Replication associations that had the built-in node as either a source or a follower have this replication mode enabled automatically upon upgrade.

See the Replication section in the Develocity Administration Manual for more information on Build Cache replication.

Build Cache configuration settings are moved

The Develocity-provided build cache is now configured on the Build Cache page of the Develocity Administration UI. Configuration is simplified to just the target storage size and maximum artifact size settings. The built-in node settings page is removed from the Build Cache page.

Existing maximum artifact size and cache access control settings for the built-in node are retained and enforced by Develocity. Administrators can access these settings on the Legacy Configuration page of the Build Cache dashboard.

 

Deprecation of API keys and unpooled agents

Support for Test Distribution API keys and unpooled Test Distribution agents is deprecated and will be removed in a future release. Instead of API keys, pool-specific registration keys should be used as they improve security by disallowing joining any agent pool with any API key. Moreover, connecting Test Distribution agents without assigning them to an agent pool is deprecated.

Please migrate your existing agent pools and agents to use registration keys and create agent pools for unpooled agents. Afterwards, revoke any remaining API keys.

Some project and project group display names may be automatically updated

Duplicate project and project group display names are automatically renamed when upgrading to Develocity 2024.1. Except for the first duplicate, renamed display items have (N) appended to their names, where (N) is an integer counter incremented for each duplicate display name. For example, if three projects have the display name Project A, their display names are updated to Project A, Project A (2), and Project A (3)

Deprecated SSL termination with the gradle-proxy component has been removed

SSL termination with the gradle-proxy component was deprecated in Develocity version 2022.4.5 and is now removed. Terminate SSL at Ingress or externally instead.

Supported Kubernetes and Helm versions have changed

The 2024.1 release drops support for older versions of Kubernetes and Helm. Develocity displays a Helm warning during installation if administrators are using the minimum supported or lower versions of Kubernetes and Helm. The Develocity Admin page also displays this warning.

The minimum supported versions for 2024.1 are:

  • Kubernetes 1.26.0
  • Helm 3.11.0

Please check the version compatibility matrix for further reference.

Develocity Helm chart upgrade notes

The 2024.1 release provides a new Develocity build cache service that replaces the existing service when enabled. The release also marks the general availability of object-based storage for the build cache, which improves the horizontal scalability of the Develocity-provided build cache. Both of these features require additional resources. Administrators must update their values.yaml configuration file before upgrading to 2024.1 to ensure that all cache entries are migrated to the new build cache service and that non-default values for resources are maintained after upgrade.

  • For standalone installations, read and perform the tasks in Appendix C of the Standalone Helm Chart Configuration Guide before upgrading.
  • For Kubenetes cluster installations, read and perform the tasks in Appendix D of the Kubernetes Helm Chart Configuration Guide before upgrading.

Removed deprecated SSL termination feature from gradle-proxy component.

Changes

May 06, 2024
  • [FIX] Users are not prompted to login again when their session expires
  • [FIX] Performance issues with cross-build flaky background processors
Apr 23, 2024
  • [NEW] Allow multiple users to use the same email
  • [NEW] Gradle 'Isolated Projects' feature is visible in Build Scan® switches section
  • [FIX] `gradle-metrics` doesn't fill its persistent volume
  • [FIX] Performance dashboard may display incorrect aggregate up-to-date avoidance savings for datasets containing at least one Maven build
  • [FIX] sbt test failure messages may contain ASCII control characters
  • [FIX] Develocity can't connect to its embedded-object-storage after re-installation
  • [FIX] Warning about multiple versions of the Develocity plugin applied is incorrectly shown for Gradle 5.x
  • [FIX] Develocity fails to load the performance tab for builds with artifact transforms that execute while storing to the configuration cache
  • [FIX] Test case executions in sbt build scans are not ordered
  • [FIX] Unauthorized users can publish a Build Scan® if anonymous users have the 'Publish build scans' permission
Apr 02, 2024
  • [NEW] Develocity reliably detects when Gradle disables the remote build cache due to failures
  • [NEW] Develocity displays avoidance savings and a link to the origin Build Scan for all up-to-date Gradle artifact transform executions
  • [NEW] Remove the previously deprecated Test Distribution agent pool status endpoint in favor of the corresponding Develocity API
  • [NEW] PathType property of the Kubernetes Ingress resource can now be set to ImplementationSpecific if required
  • [NEW] Gradle deprecation warnings are available from the Develocity API
  • [NEW] The names of the Gradle plugin, Maven extension, and sbt plugin are now prefixed with ‘Develocity’ rather than ‘Gradle Enterprise’
  • [NEW] The advanced query syntax now allows searching by verification and non-verification failures
  • [NEW] Develocity now supports Bazel when deployed on Google Kubernetes Engine
  • [NEW] Develocity reduces the target storage size of the build cache when there is insufficient storage
  • [NEW] Guidance for administrators when configuring the target storage size for the build cache
  • [NEW] Add allModels query parameter to /api/build/{id} and api/builds endpoints for requesting all build models
  • [NEW] The Timeline view for Gradle and Maven builds now enables grouping work executions by project
  • [NEW] The Jobs tab on the Test Distribution page now includes the number of currently connected and compatible agents
  • [FIX] Gradle artifact transform executions are now available with the Develocity API
  • [FIX] Export API may return incorrect event data for very small events
  • [FIX] Uniqueness of project and project group is not enforced
  • [FIX] Bazel build cache may crash when content is uploaded faster than it is persisted
  • [FIX] The /api/build and /api/builds API endpoints do not set a Content-Type HTTP response header
  • [FIX] Vertical scrolling does not work for the Usage and Simulations views in the Predictive Test Selection dashboard
  • [FIX] Generating a support bundle fails when users select then disable DateTime range
  • [FIX] The Tests view may not display failed Spock tests