Exploring the Developer Productivity Solution Landscape

Introduction

There are two primary—and complementary—approaches to improving developer productivity. Both aim to use resources more efficiently and help ship code faster, while improving the developer experience. 

1. Developer Productivity Management (DPM) focuses on the people, and answers questions like, “How can we get more output out of individual developers and teams by defining and tracking the right metrics?”  Such metrics typically help to quantify output, evaluate performance and competencies, build and manage teams, and optimize collaboration and time management.
2. Developer Productivity Engineering (DPE) focuses on the process and technology, and answers questions like, “How can we make feedback cycles faster using acceleration technologies to speed up builds and tests?”  and “How can we make troubleshooting easier, faster, and more efficient using data analytics?”

In analogous terms, suppose you’re the owner of an auto racing team. To win, you need both the best drivers and the fastest cars.  In the software engineering world, the drivers are your developers, while the cars are your highly performant processes and technology toolchain.

This article surveys the relative advantages and disadvantages of DPM and DPE, while ultimately recommending a DPE-first approach. In other words, no matter how good your drivers are, it’s hard to win races without the fastest cars.

Advantages of Developer Productivity Management

DPM can provide management insights through cohort analysis

DPM aims to give engineering management in-depth insight into the performance of their individual developers and teams who are frequently geographically spread throughout the world. These insights may be used to increase leadership’s understanding of the organization, give decision makers more confidence, and provide opportunities for continuous improvement.  DPM does this through cohort analysis, which filters productivity metrics and trends by team locations (on-site or local), domain (front-end or back-end), seniority, geography, team sizes, and technical environment. 

In contrast, while DPE also provides activity and outcome metrics by team member and technical environment, these metrics are mainly used to identify, prioritize, and fix toolchain specific problems like builds and tests.

DPM may be useful in building and managing teams and time

DPM can help you build high-performing teams through cohort analysis data about ideal team sizes, makeup, and structure. Further, DPM metrics can be used to improve project forecasts and estimates, prioritize work, evaluate project health and risk, and assign developers to teams based on demonstrated skills and needed competencies. 

Managing time is a primary focus of many DPM tool vendors. Using data to simply strike the right balance between coding time and meeting time, for example, can make a tremendous difference in team output. Deeper DPM data that focuses on the days of the week and times of the day that optimize meeting effectiveness can further improve productivity.

DPM may be useful in optimizing individual performance

DPM metrics can be used both to measure individual output, and as inputs for evaluating individual performance. They can also help with identifying and closing skill gaps and allowing individuals to showcase their own domain expertise and competencies. Perhaps most importantly, DPM can be used to monitor changes in individual activity levels (e.g., velocity) to detect if a developer is stuck and requires some kind of intervention.

Limitations of Developer Productivity Management

Human productivity metrics are flawed

DPM assumes there is a reliable and meaningful way to measure developer productivity. After all, you can’t measurably improve productivity if you can’t define a baseline. In “The Pragmatic Engineer”, Gergely Orosz argues that you can’t measure the productivity of knowledge workers like doctors, lawyers, chemists, and software engineers. He concludes that for software engineering in particular, individual metrics can be gamed, don’t tell the whole story or,  in some cases, are completely arbitrary. 

Many metrics violate “Do no harm”

The fact is there is no real consensus on what is a valuable human productivity metric that makes sense across the industry. Moreover, many human productivity metrics can lead to behavior changes—such as the number of commits per day—that can be counterproductive and antithetical to the business objectives they are designed to support. 

In “The Myth of Developer Productivity” (Nortal Cloud Team, 2018), the authors review several common developer productivity management metrics, such as hours worked, source lines of code (SLOC) produced, bugs closed, function points, and defect rates. They conclude that most are more harmful than useful in managing developer productivity, in part because they lack context. Consider the developer whose commits and code review metrics are down, but only because s/he spent more time mentoring others and helping them increase their own productivity.  

Gut instinct is still king

According to developer productivity thought leader Bill Harding (“Measuring developer productivity in 2020 for data-driven decision makers”), due to the challenges of measuring developer output and using that data to improve team productivity, managers often hone their own intuition to serve as their ‘north star’. Thus, a key goal of DPM is to find and retain technical managers whose past intuitions led to success. But how do you define success, and how is that measured?

Big brother can be a tough sell

Another potential DPM challenge is gaining buy-in from the developer teams that are being measured. Defendants of DPM dismiss the inevitable developer pushback as largely a matter of messaging, and claim that it’s a natural evolution of a maturing software development management process. They point to all of the great companies that initially resisted external measures of performance—like Consumer Reports, Glassdoor, and Yelp—and later embraced them, failing to acknowledge that they had no other option. 

DPM is a choice.  To be effective, it should be implemented with great care,  considering your company culture, objectives, and ability to execute (i.e,. the ability to translate insight into action). Developers understand that their activities may be monitored (e.g., whether they’re active on Slack or not) and understand the need for measurement. The sensitivity stems from the purpose of the measurement and the underlying problems management is trying to solve. When management is focused on understanding the root cause of delivery speed variations, for example, DPM may be welcome, but expect the developer antibodies to surface when metrics are used as input for comparative performance evaluation.

DPM solutions may collide with other tools

The benefits that DPM targets often overlap with existing and established technologies, including IDEs, collaboration and workflow tools, project management systems, and learning management platforms. As a result, it’s up to you to decide where one system/process starts and another ends, and deal with potentially multiple integration points to rationalize workflows and facilitate reporting.

In contrast, DPE is a mostly accretive solution in terms of the kind of data it generates and the ways in which it solves problems and augments workflows. Integrations with systems of record for management reporting (e.g., Tableau) or CI integration (like Jenkins and TeamCity) are simple and straightforward.

Advantages of Developer Productivity Engineering

DPE provides measures of outcomes

Perhaps the most clear distinction between DPM and DPE is that DPM provides mostly measures of activity and DPE focuses more on measuring outcomes. Compare DPM metrics like hours worked, SLOC, and bugs closed to DPE metrics such as build and test times for local and CI builds, failure rates, and build comparison data that can be used to pinpoint the root cause of regressions.

DPE metrics cannot be gamed, are not arbitrary, and can be more easily marshalled to make logical connections to higher-level performance and efficiency metrics. These metrics can, in turn, be aligned unequivocally with C-level business objectives, like time-to-market, quality of services, and cost savings.

DPE delivers measurable ROI

If you Google “developer productivity measurement benefits”, you will be hard-pressed to find a straightforward explanation from industry commentators or DPM tool vendors on the benefits of tools used to measure individual developer and team productivity. It may be that a direct answer to the query is hard to find not because it doesn’t exist, but because it’s obvious and assumed. That is, more productive developers means higher quality code that ships faster and at a lower cost. It would not be unreasonable to take this on faith, but to know if the return on your investment in DPM is a lot or a little is impossible to say. 

With DPE, measuring ROI for several key benefits is straightforward. For example, to calculate the hard savings from achieving faster build and test feedback cycles using DPE acceleration technologies like build caching and test distribution, the formula is simple: Multiply the average time saved waiting for builds to complete by the number of builds per year to get total time savings in engineering years, and then multiply that by the cost of a fully-loaded engineering year. For many moderate-size development teams, this quickly translates into double-digit savings, measured in engineering years, or millions of dollars if put into budgetary terms. 

DPE improves the developer experience

While DPM tool vendors may claim that many features support a better developer experience (e.g., team assignments that play to their skill and competency strengths), the limitations of DPM described above suggest DPM may do more to negatively impact the developer experience. As a result, developers are not the driver for DPM adoption—in fact, they are more likely to be the primary skeptics, while software engineering management is the primary user/practitioner. 

Compare this scenario to DPE, in which the senior and lead developers are not only the drivers of DPE initiatives, but are also the power users of the tools. The benefits they enjoy are not soft, indirect, or unquantifiable. They often directly experience the benefits of faster build times and more efficient troubleshooting a dozen times or more a day. As a result, it is not uncommon for companies with established and dedicated  DPE teams to use their DPE practice as a tool for recruiting and retaining top engineering talent.

DPE has more robust success stories

It is difficult to find evidence of DPM successes beyond testimonial quotes. In contrast, you can find hour-long webcasts presented by DPE leaders within some of the world’s leading business and technology brands. They cover business drivers and objectives, solutions deployed, verifiable benefits achieved, and next steps in their DPE journeys.

Limitations of Developer Productivity Engineering

DPE tools have yet to leverage AI/ML

While both DPM and DPE tools provide analytic data that can be analyzed to generate insights and develop recommendations for action, that process is primarily manual today. However, some DPM tools are already starting to incorporate AI/ML to better automate insight generation and other analyses.

DPE is relatively new and less well understood

Compared to DPM, DPE is a relatively new and emerging discipline that is in the process of socializing new concepts and solutions—including local and remote build caching, test distribution, build scans, and build and test failure analytics. Some concepts get confused, which may create some initial friction in their adoption, such as the functional difference between build caches and binary repositories. However, this is a normal part of the growing pains associated with establishing a new software solution category.

DPE solutions do not yet support all developer environments

DPE solutions, like Gradle Inc’s Develocity, currently focus on the JVM and Android ecosystems and may provide more limited support for other environments. However, it is expected that DPE solutions will expand their footprint over time to support fully all popular developer environments. While some DPM tool metrics are toolchain/technology specific, some apply more broadly across ecosystems.

Conclusion

DPM and DPE are complementary, not competing approaches to improving developer productivity. However, they can compete for the lion’s share of both your mind and your wallet. If you’re living in a Java ecosystem, start with investing in the fastest car: Developer Productivity Engineering. 

***

To see Developer Productivity Engineering technology in action, check out the Develocity Demo Video.