Accelerate
Troubleshoot
Observe
When not tackling flaky tests, Collibra’s 300+ global developers take time to meet and collaborate
About Collibra
Collibra is a software company specializing in unified governance of data and AI, and the first Belgian technology company ever to achieve “unicorn” status (a billion-dollar valuation) in 2018. Collibra’s Platform brings flexible data governance, continuous quality, and built-in privacy to over 700 customers around the world.
We spoke with Niels Doucet, Staff Software Engineer at Collibra, about how they use Develocity to address key pain points in their SDLC—specifically, managing and eradicating flaky tests, accelerating build and test cycle times, and reducing the MTTR for build and test failures.
The primary developer pain points
“Most developers want a build to be a one-time concern at the start of a new project. From then on, it should just work. But “bit rot” is a real problem here. Making sure you have centralized insights into the build health of all of your projects alleviates some of that pressure.”
– Niels Doucet, Staff Software Engineer, Collibra
Collibra employs about 300 developers who work on 200 active projects. The majority of their projects are in Java and use Gradle Build Tool; other projects use various other languages. Jenkins CI is their primary continuous integration server, along with GitHub Actions for some projects.
Collectively, Collibra’s development team generates approximately 1,500 local builds and 4,500 CI builds per day—each of which creates an easily accessible Build Scan® in Develocity as a “statement of record” for the work done.
Collibra’s primary pain points are likely familiar to many developers:
- Flaky tests – Tests that sometimes pass and sometimes fail without any changes to their input are a frequent and frustrating problem that erodes trust in the system.
- Slow feedback cycles for engineers – Slow feedback cycles lead to lower productivity, increased cognitive fatigue, and less frequent builds and tests (meaning that bugs and issues are found later in the SDLC).
- Lack of build insights and awareness – Without observability into how builds work, and what maintenance is needed for optimal performance, it’s difficult to meaningfully improve any KPIs.
Develocity enters the picture
When Collibra started using Develocity with their Gradle builds, they realized it would require some customizations to get what they needed. By creating a custom, in-house Gradle distribution, they were able to integrate all Gradle projects seamlessly into their Develocity environment. This allowed them to easily configure the remote Build Cache out of the box and create a new Build Scan by default (using tags) for each new build cycle.
Getting insights into average build times has helped Collibra make targeted improvements. With Build Scan, they have an easy way to track outcomes. This has led to considerable improvements in the feedback cycle time for some of their biggest projects—they went from about 1 hour to ~25 minutes for a full build (a 60% reduction).
Develocity Build Cache—and the ability to determine task avoidance opportunities through build optimization—made this possible. Develocity has enabled up to 78% avoidance savings in their longest running builds (see the image above).
Collibra uses Develocity Failure Analytics and Performance Insights to achieve the level of observability needed to inform decisions on areas for improvement, especially regarding flaky tests.
Flaky tests are a challenge for every software organization—in many cases, flaky tests are caused by badly isolated test contexts.
Collibra discovered that, for example, the order in which tests are run could influence their outcome; this is especially true for integration tests that rely on data generated in a test environment. Developers might see test data from other tests, or their own data might be “cleaned up” by another test’s tear-down logic being overzealous. Other common causes of test flakiness include reliance on unstable external systems or timing issues.
Every team regularly looks at the “Tests” dashboard, filters for their projects and sorts by test flakiness. The top items of that list become tickets to either solve the root cause of the flakiness or simply remove the tests if they no longer prove valuable.
Collibra’s go-to Develocity features
“Develocity has led to a 60% reduction in build times on CI, and has been instrumental in helping engineers rapidly address and solve flaky tests. Preliminary results using Predictive Test Selection have already shown a 300% ROI in test cycle time savings.”
– Niels Doucet, Staff Software Engineer, Collibra
Build Scan
Build Scan is a critical piece of Collibra’s toolchain. Every build uploads a scan by default, which is the starting point for debugging build issues in CI. It provides the fastest and easiest way to communicate with engineers about problems they experience with their builds—simply send a Build Scan URL.
Build Cache
Build Cache is also enabled by default, both on local developer machines and on CI (using Remote Build Cache). This has had a tremendous impact on build times—which are about 60% faster than before. This also onboards new members of the Collibra engineering team faster when they build for the first time, giving a positive impression to fresh talent.
Failure Analytics and Performance Insights
The Failures and Performance dashboards are used primarily to address one of Collibra’s biggest pain points: flaky tests. Develocity’s Flaky Test Detection capabilities guide many of the engineering teams’ efforts to improve their test suites, leading to more reliable and trusted build outcomes. Prior to Develocity, flaky test detection was a highly manual effort of trial and error, which significantly added to time to diagnose.
Next steps with Develocity
So far, Develocity has accelerated feedback cycle times, increased build and test observability for faster identification of opportunities, and provided a better team-wide understanding of how Gradle Build Tool works.
What will the Collibra team do next? They’ve got their eye on these additional Develocity features:
Predictive Test Selection
Recent trials of Predictive Test Selection have shown an 11% decrease in test cycle times, which already gives a 300% ROI in the first year. That said, the Collibra team has already discovered methods for boosting this number to about 40%. They are actively working on rolling out PTS to all teams.
Reporting & Visualization
Collibra recently reviewed Develocity Reporting & Visualization and believes it will dramatically improve observability into overall build health, as well as surface trends across all of their builds and projects. With greater insight into the health of their toolchain, Collibra can leverage an aggregate view on what data flows into their Develocity environment to prioritize optimization efforts.
Words of advice for DPE beginners
The team at Collibra believes it’s important to set expectations that Developer Productivity Engineering (DPE) is a long-term investment, not a one-time project. DPE improvements are generally not “big bang” events. They are smaller, steady improvements that add up over time.
For example, your plugins require regular maintenance and if you don’t pay close attention, before you know it you have old plugins that don’t fit current best practices, leading to sub-optimal builds.
“We believe that the best way to achieve DPE success is to dedicate a team to take on the challenge. For resource-challenged organizations (and who isn’t?), that’s a tall order. So another method is to develop engineering “champions” that can influence team initiatives to tackle developer productivity challenges.”
– Niels Doucet, Staff Software Engineer, Collibra