Gradle Enterprise 2023.3

Gradle Enterprise 2023.3 brings project-level access control for enforcing organizational information-sharing boundaries, provides dependency downloading build time impact analysis for Gradle and Apache Maven™, adds new insights for sbt users, and is now compatible with Google Cloud Storage and Azure Blob Storage.

Read on for more details of these and other new features.

Use version 1.19.3 of the Gradle Enterprise Maven extension, version 3.15.1 of the Gradle Enterprise Gradle plugin, version 0.10 of the Gradle Enterprise sbt plugin, version 2.0.3 of the Gradle Enterprise Test Distribution agent, and version 17.1 of the Gradle Enterprise Build Cache node for optimal usage of Gradle Enterprise.

Use the latest versions of the Gradle Jenkins plugin, Build Scan plugin for TeamCity, and Gradle Enterprise Bamboo plugin for optimal integration of your CI provider with Gradle Enterprise.

Highlights

Project-level access control

This feature allows organizations to share a single Gradle Enterprise installation while ensuring that data and information about individual projects are restricted to an appropriate subset of users and teams, respecting organization information-sharing boundaries. This includes Build Scan® data, Build Cache data, test acceleration features (including Test Distribution and Predictive Test Selection), and analytics and insights derived from build data for Gradle, Apache Maven™, sbt, and Bazel.

Project-level access control requires registering a “project ID” for each project with Gradle Enterprise, assigning it to one or more project groups, and then assigning users to project groups (either directly or via SAML or LDAP role mappings). Users may only perform an action (e.g. viewing a Build Scan) for data related to a project if they are associated with the project via a project group. 

 

To leverage project-level access control builds must be configured with a project ID as part of their Gradle Enterprise configuration. This project ID is not a secret value and can be safely checked into version control. For environments where Gradle Enterprise functionality is automatically applied to many projects (such as via the Jenkins Gradle plugin, which also works for Maven builds) without any checked-in build configuration, the project ID can be specified via the build environment.

See the appropriate manual for your build tool for instructions on how to specify the project ID:

Projects and project groups can be created and managed manually within the Gradle Enterprise user interface, or the Gradle Enterprise API can be used to automate this step, potentially integrating with an organization’s project registry or project provisioning workflows. 

 

The related API operations are currently “beta” and may change due to user feedback over the subsequent major releases.

Project-level access control is disabled by default. See the Project-level access control in the Gradle Enterprise Administration Manual for Helm Installations for details on enabling and migrating to project-level access control.

Project-level access control is available with Gradle 5.0+ and Gradle Enterprise Gradle plugin 3.15+, with Maven 3.3.1+ and Gradle Enterprise Maven extension 1.19+, with sbt 1.6.0+ and Gradle Enterprise sbt plugin 0.10+, and, with Bazel 4.0.0+.

Observe build time cost of dependency downloading for Gradle and Apache Maven

An estimate of the actual build time contributed by downloading dependencies is now available in a Build Scan for Gradle and Maven. Previously, only a “serial time” number was observable which is the sum of time taken for all dependency downloading network operations. As dependency downloading frequently occurs in parallel with other requests and other types of build work, this number is typically significantly higher than the actual effect on elapsed, or “wall clock”, build time. The new wall clock time estimate provides a more insightful representation of the build time cost of downloading dependencies.

For many organizations, dependency downloading significantly contributes to CI build times due to short-lived or ephemeral build environments, where a lack of local dependency caches causes each build to download excessive amounts of data from slow and often unreliable artifact repositories. Organizations can use the new wall clock time estimates to assess this build time cost and quantify any future improvements or optimizations to CI or artifact repository infrastructure.

For Gradle builds, the per-build time can be found in the “Performance → Network activity” section.

For Maven builds, the per-build time can be found in the “Performance → Dependency resolution” section. 

 

The wall clock dependency downloading time can be visualized across builds via the “Performance dashboard”.

It can also be visualized in aggregate via the “Trends dashboard”.

Dependency download wall-clock time contribution insights is available with Gradle 6.2+ and Gradle Enterprise Gradle plugin 3.12+, and with all supported Maven / Gradle Enterprise Maven extension versions.

Richer time savings statistics for Predictive Test Selection usage analysis

The savings chart on the Predictive Test Selection (PTS) usage dashboard has been improved to provide two new insights.

The “Serial test time saved” number is now qualified with an overall percentage indicating how this number relates to the total serial test time where PTS was enabled.

The “Savings Potential” number indicates the amount of serial test time that PTS would have saved if it were enabled for all tests, providing context on how much more time could be saved by enabling PTS for more tests and more builds.

Additional advanced search fields

Gradle Enterprise 2023.1 introduced the ability to search for builds matching criteria via a new query language. This release introduces several new fields and a convention for build-tool-specific fields, that implicitly apply the corresponding build tool as a criterion. Thus, the advanced search query gradle.requestedTasks:verify selects only Gradle builds, whereas the query maven.requestedGoals:verify selects only Maven builds.

It is now possible to use an advanced search query to filter builds according to the applied version of the Gradle Enterprise plugin or extension, depending on the build tool, by using the fields gradle.pluginVersion, maven.extensionVersion, or sbt.pluginVersion.

There are new build-tool-specific fields: gradle.rootProjectName, maven.topLevelProjectName, and sbt.rootProjectName. The previous rootProjectName field, a tool-agnostic version of the same concept, has been renamed to project. 

There are new build-tool-specific fields gradle.requestedTasks, maven.requestedGoals, bazel.requestedTargets, and sbt.requestedTasks that complement the existing requested.

See the advanced query language reference for further details.

List and observe builds via API matching criteria

Previously, the /api/builds API endpoint allowed listing all known builds and observing new builds as they happen. Now, it can also list and observe only builds matching specific criteria, expressed as an advanced search. In many cases, this avoids making extra API calls to determine whether a given build is a build of interest before continuing with processing, making programs using the API simpler and faster.

Automatically enable Gradle Enterprise for Gradle and Maven builds when using GitHub Actions or GitLab CI

Features recently introduced in the Gradle GitHub Action and Gradle Enterprise GitLab templates make it easier to enable Gradle Enterprise for GitHub Actions and GitLab CI builds, without requiring any modification to existing build scripts.

This functionality is beneficial for organizations with an existing set of diverse projects. It makes the insights of a Build Scan available without requiring project owners to make changes to their build logic, limiting these changes to GitLab CI or GitHub Actions configuration files. By connecting more projects to Gradle Enterprise, teams can better monitor build performance and failure rates across all projects.

Note: Gradle GitHub Action is only able to instrument Gradle builds.

Similar integrations already exist for Jenkins, TeamCity and Bamboo.

Using OpenRewrite to integrate Gradle Enterprise at scale

Connecting hundreds or thousands of Gradle and Maven projects to Gradle Enterprise for the first time can be time-consuming. The canonical way requires modifying the existing build configuration settings, which is impractical at scale. That’s why we collaborated with OpenRewrite from Moderne to create pull requests to update all repositories automatically.

For further information, check out the OpenRewrite recipes for the Gradle Enterprise Gradle plugin and Gradle Enterprise Maven extension.

Contact your customer success representative if you’d like to learn more.

Simplified setup of the Gradle Enterprise Maven extension

A new init goal has been introduced to quickly set up the Gradle Enterprise Maven extension on any given project. Running the following command will create or update .mvn/extensions.xml to add the Gradle Enterprise Maven extension and configure it for use with the specified server.

mvn com.gradle:gradle-enterprise-maven-extension:init -Dgradle.enterprise.url=https://gradle-enterprise.mycompany.com

Please see the Gradle Enterprise Maven Extension User Manual for more information.

Store data in Google Cloud Storage and Azure Blob Storage

In addition to the previously supported Amazon S3, application data can now be stored in Azure Blob Storage, Google Cloud Storage and other S3-compatible stores. This makes using object storage available to installations in Google Cloud Platform and Microsoft Azure.

Opt-in beta for new Build Cache service

An evolved implementation of the build cache service is under active development. It is available as an opt-in beta for this release and replaces the existing service when enabled. It supports horizontal scaling of read-write instances for cluster deployments and leverages object storage for simpler persistence management. 

Opting into the new beta Build Cache service is suitable for users getting started with Gradle Enterprise who want to simplify their deployment configurations, or for existing users particularly interested in increased redundancy and availability. 

Please contact Gradle Enterprise support for information and guidance on enabling this feature.

Upgrade notes

If upgrading from a version of Gradle Enterprise prior to Gradle Enterprise 2023.2, be sure to also consult the release notes for all interim versions.

After upgrading your Gradle Enterprise server to the most recent version, it is recommended to upgrade your builds to use the latest version of the Gradle Enterprise Gradle plugin, Gradle Enterprise Maven extension, Gradle Enterprise sbt plugin, Gradle Enterprise Test Distribution agent, and Gradle Enterprise Build Cache node. Using the latest versions of all components ensures that you benefit from security updates, bug fixes, and feature enhancements. Please check the version compatibility matrix for further reference.

It is also recommended to update your configuration of the Gradle Jenkins plugin, Build Scan plugin for TeamCity, and Gradle Enterprise Bamboo plugin to reference the latest version of the Gradle Enterprise Gradle plugin.

Temporarily degraded performance due to data reindexing

Upon upgrading, a data reindexing process will be initiated in the background, with Gradle Enterprise being usable for its duration. CPU usage will be increased, and performance may be slightly degraded. The reindexing process may take several hours and potentially several days for large installations storing months of build history. During this time, some builds may be omitted from cross-build data visualizations. Recent builds are prioritized, making their data available sooner.

Object storage configuration now in Helm chart and not user interface

Prior releases of Gradle Enterprise required configuration of S3 storage via the Administration UI or unattended configuration. From 2023.3 onwards, the configuration of an external object store must be done via the Helm chart.

If an existing installation with S3 storage configured is upgraded to 2023.3 without migrating to the Helm configuration, Gradle Enterprise will continue to function, but a warning will be displayed. This backwards-compatible mode will be removed in a future release.

Advanced search field name changes

The previous search field rootProjectName has been renamed to project.

Gradle Enterprise Helm Chart upgrade notes

  • All StatefulSet usage was replaced with Deployment: enterprise-app, enterprise-app-background processor, keycloak & proxy are now Deployments. The transition will be transparent if you are using Helm to upgrade your installation. If you use a custom upgrade strategy, you should make sure that old StatefulSets are removed. If you use internal scripts or customizations referencing StatefulSet, please update it to Deployment instead.
  • Label usage was unified & standardised. You should no longer use in your custom scripts app/component labels. For example, app=gradle-enterprise & component=proxy should be replaced with app.kubernetes.io/part-of=gradle-enterprise & app.kubernetes.io/component=proxy
  • Applications no longer log into a filesystem, and the log volume usage has been removed. Applications are logging to the monitoring component. If you use a custom tool to capture logs generated by any component, please use  container standard output instead of log files now. To avoid unnecessary costs, you can additionally  delete all Persistent Volumes previously attached to a Persistence Volume Claims with annotation `app.kubernetes.io/for: logs` 
  • The logging retention period was reduced from 30 days to 7 days.
  • Storage.logs.capacity, storage.logs.directory, storage.logs.class helm values are no longer used.
  • An internal object storage component was introduced
  • `gradle-env-vars-config` has been split. If you used a post-rendering tool to modify part of it, it’s possible the value you wanted to change isn’t anymore located in this configMap. Please adjust your tool to modify the correct ConfigMap.

You can find resource requirements for cluster installations here and standalone installations here. If you want to compare all helm changes between your currently installed version and the version you are upgrading to, you can use the helm template command for each version and then compare the two templates.

Changes

Nov 20, 2023
  • [NEW] Airgap bundle script may use Skopeo for uploading images to an internal registry instead of Docker
  • [NEW] Dedicated Ingress is added to improve compatibility with load balancer for Standalone installation
  • [FIX] Tags are cut off in the PTS dashboard
  • [FIX] Proxy component does not start in OpenShift clusters
  • [FIX] Gradle Enterprise 2023.3 does not migrate advanced parameters for LegacyS3Config
Oct 31, 2023
  • [FIX] Upgrading from a version before 2023.3 is slow due to under-resourced init containers
  • [FIX] Support bundle generation is very slow due to slow log capture
  • [FIX] Proxy component access logs are not included in support bundle
  • [FIX] Startup is excessively slow due to the "monitoring" container being slow to start
  • [FIX] gradle-monitoring-data volume may runs out of disk space due to excessive log retention
  • [FIX] Embedded database startup is slow when using network-attached storage due to excessive file permission checking
  • [FIX] Enabling opt-in beta for new build cache service makes service unavailable for Bazel builds
  • [FIX] No feedback is given when using deprecated object storage configuration via application
Oct 18, 2023
  • [FIX] Build time in the Timeline section of Bazel Build Scan may be incorrect
  • [FIX] Targets section of Bazel Build Scan fails to load if the build fails before the analysis phase
  • [FIX] Using Bazel v6.2.1 and later causes excessive temporary database storage usage
  • [FIX] Project entries are created when submitting a request with an unknown project identifier
  • [FIX] Previously enqueued Predictive Test Selection data processing occurs after disabling the feature
  • [FIX] Built-in build cache opt-in beta may serve incorrect entries when the same entry is stored by different projects
  • [FIX] AWS S3 instance profile credentials are not respected when using a specific endpoint address
Sep 22, 2023
  • [FIX] Tests dashboard shows error after upgrading to 2023.3
  • [FIX] Test distribution agents providing more capabilities than any matching pool will not be assigned work
  • [FIX] SAML-based login fails if IdP does not specify encryption algorithm
  • [FIX] Upgrade from 2022.4 release to 2023.3 release may fail
Sep 13, 2023
  • [NEW] Project admins can enable and configure project level access control
  • [NEW] Dependency download wall clock time contribution insights in Build Scan
  • [NEW] Navigation from timeline to artifact transform log in Build Scan
  • [NEW] Performance and Trends dashboards include sbt builds
  • [NEW] Richer time savings statistics for Predictive Test Selection usage analysis
  • [NEW] Advanced query syntax has new build-tool-specific search fields
  • [NEW] Centralized background work scheduling, such as ingesting new build scans
  • [NEW] The /builds endpoint of the Enterprise API can be filtered using advanced query syntax
  • [NEW] Data can be stored in Google Cloud Storage and Azure Blob Storage
  • [NEW] Opt-in beta for new Build Cache service
  • [NEW] Multiple roles can be mapped to the same value in the IdP
  • [NEW] Application logs to the dedicated monitoring component instead of the filesystem
  • [NEW] Embedded database is upgraded to PostgreSQL version 14.9
  • [NEW] End of support for restoring backups from installations on versions before Gradle Enterprise 2021.1
  • [FIX] Lenient artifact transform failures are polluting the Failures dashboard
  • [FIX] Unscheduled planned transform steps are considered as scheduled ones
  • [FIX] Build Scan pages show errors for builds with Isolated Projects
  • [FIX] Build Cache tab does not display proper statistics when remote cache errors occur
  • [FIX] Error for test classes with a mix of "not selected" and other outcomes on Tests dashboard
  • [FIX] Error for inconsistent start/end timestamps of test events
  • [FIX] Embedded database restore fails when the database pod was not shut down cleanly
  • [FIX] Keycloak Admin connections are disallowed from public IPs
  • [FIX] Monitoring component is not compatible with Kubernetes cluster using public IPs
  • [FIX] The buildCacheNode.buildCacheBuiltin.value secret leaks
  • [FIX] Update to Java 17.0.8-7 for security updates and bug fixes