Every year at DPE Summit, Gradle hosts hundreds of Developer Productivity Engineering and Developer Experience experts for two days of presentations, discussions, and networking. One of the featured speakers from DPE Summit 2024 was Aida Issayeva, a staff software engineer and tech lead for the Point-of-Sale Productivity team at Toast. In her presentation, Aida shared simple yet effective strategies to improve Android build times, drawing from her extensive experience at Toast.

About Toast

Toast is a revolutionary restaurant point of sale (POS) and management system. Founded in 2012, Toast rapidly grew from around 1,000 restaurant customers in early 2015 to over 100,000 restaurants across the U.S. in late 2024, with the robust POS app operating on approximately 500,000 devices nationwide.

As Toast has grown and expanded, so has its Android-based POS app and the team that supports it. Initially numbering less than 10 contributors and about 2,000 builds annually, the Android POS team has scaled to 140 contributors and 26,000 builds annually.

Today, the project encompasses over 2 million lines of code and more than 300 Gradle modules. With this expansion, the number of pull requests has surged to an average of 150 PR builds a day. This massive growth quickly escalated many of the challenges related to build times. So, what were those challenges?

The build time challenge

As Toast’s Android user base and app features expanded, build times inevitably followed suit. By early 2023, the average build time had climbed from 42 minutes to a staggering 75 minutes—a hefty 44% increase in just a year. This rise in build times posed significant issues for both developers and end-users.

“Build times matter because when they increase, developers wait longer. This leads to distractions, hampering productivity and extending release cycles. Unhappy developers result in unhappy customers, ultimately affecting business outcomes.”

– Aida Issayeva

Specifically, Toast noticed these negative outcomes due to long build times:

• Long CI queue times: More builds in the queue means developers spend excessive time waiting, requiring more frequent context-switching and greater cognitive fatigue.
• Developer distractions and decreased efficiency: Prolonged build times break developers’ flow, diminishing the overall developer experience and slowing down innovation and responsiveness.
• Extended release cycles: Slower release cycles delay feature deployment, impacting business targets and ultimately, customer satisfaction.
• Business bottom line: Reduced customer satisfaction directly harms the business’ bottom line.

Tackling the challenge

In late 2023, Aida’s team embarked on a journey to dramatically reduce build times. First, they began using Develocity, the leading toolchain observability platform, to help identify build bottlenecks and then begin to address them. Through a number of strategic enhancements, they achieved a remarkable 61% reduction in build times—going from an average of 68 minutes to 27 minutes by the first quarter of 2024.

Low-Hanging Fruit

The phrase “you can’t fix what you can’t see” rings especially true when debugging builds. Develocity’s end-to-end observability enabled Aida to identify productivity issues and prioritize quick wins.

Hardware upgrades

The first action Aida’s team took was to upgrade their hardware. This delivered the most immediate improvement, albeit at a high price.

• Developer machines: Modern hardware significantly influences build performance. Upgrading from older MacBook Pro models with Intel-based chips to the latest models with Apple silicon chips gave a noticeable performance boost.
• Availability: Upgrading CI server agents to faster machines with more memory offered similar build performance improvements.

Gradle properties optimization

Another primary area of focus was optimizing Gradle properties. Enabling the following Gradle Build Tool configurations led to faster build cycles:

• Parallel Execution: Enable parallel execution of tasks in your gradle.properties file to reduce the overall time it takes to execute tasks.

org.gradle.parallel=true

• Build Cache: Enable Gradle’s Build Cache to reuse task outputs from your previous builds to avoid re-running unnecessary build steps.

org.gradle.caching=true

Take this even further by using Develocity’s remote Build Cache to re-use cached outputs generated by CI from the changes your team members contributed.

• Managing Workers: Adjust the maximum workers to match your system’s CPU capabilities. This ensures optimal memory usage and prevents out-of-memory issues on devices with a limited number of cores.

org.gradle.workers.max=<Available_Cores>/2

Take this even further by using Develocity Build Scan® to see resource usage data, such as CPU usage for build and build child processes, as well as memory, disk, and network usage. In addition, see which processes are using the most CPU and memory during the build.

Memory configuration

After that, the team looked into memory allocation for large projects, which turned out to be crucial.

• Heap Size Configuration: Proper heap size minimizes garbage collection time and maximizes throughput. It requires careful calibration based on trials.

org.gradle.jvmargs=-Xmx<Max_Heap_Size>

• Set Initial Heap Size: Defining an initial heap size helps in memory management. This customization ensures efficient memory usage, particularly in production-grade projects.

org.gradle.jvmargs=-Xms<Initial_Heap_Size>

When using these memory settings together, keep the values on a single line. For example, setting the max heap size to 1GB and the initial heap size to 512MB, the setting would look like:

org.gradle.jvmargs=-Xmx1g -Xms512m

Dependency management

Another strategy Toast took was to review dependencies, keeping them lean and updated.

• Removing unused dependencies: Eliminate unnecessary dependencies to speed up resolution times.
• Updating outdated dependencies: Outdated dependencies can slow down development with compatibility issues. Use plugins like Dependabot or Renovate to manage updates seamlessly.

CI script enhancement

Toast’s CI environment was also ripe with opportunities for optimization.

• Parallelizing CI stages: Running CI stages in parallel instead of sequentially resulted in immediate time savings.
• Streamlining Gradle tasks: Avoided re-compiling tasks in different stages for efficiency.
• Reducing plugin timeouts: Minimized wait times caused by high timeout settings for tasks such as code coverage analysis with SonarQube.

CI Container Optimization

Recognizing outdated configurations (and upgrading them) led to further improvements.

• Upgrading CI infrastructure: Transitioning to newer Jenkins instances allowed Toast access to advanced settings and higher agent counts.
• Managing agents: Introducing agent pooling helped reuse agents effectively, reducing startup times.

“By focusing on both software and hardware improvements, we drastically optimized our build times, enhancing both developer satisfaction and productivity.”

– Aida Issayeva

Next up: Modularization

Achieving dramatic build time improvements spurred Aida to look beyond low-hanging fruit. Her team realized the need for deeper structural changes like modularization. With over 2 million lines of code distributed across 300 modules, enhancing module development could unlock greater efficiencies. Having all their build data automatically collected and analyzed by Develocity has given Aida’s team the visibility to know where to start.

Wrapping it all up

Revamping the Android build environment at Toast was no small feat but well worth the effort, resulting in a streamlined development process and happier developers. From making hardware enhancements to enabling Build Cache to reviewing CI settings, every effort contributed to the goal of reducing build times.

Next steps

If, like Toast, you’re ready to start understanding your builds in a whole new way, we have a great free tool at your disposal. You can get granular analytic information about your Android project with Build Scan, powered by Develocity. It’s free forever for a single project, with no strings attached.

Learn more