Engineering Process Is Overvalued · [Thinking inside a large box];

Engineering Process Is Overvalued

Many engineers and managers focus too much on process and overlook much more important aspects of a successful engineering team.

Here’s a quick reminder about what really matters and what engineering managers should spend their time and energy on.


I’ve spent this year interviewing a lot of people. One question that inevitably comes up: “What’s your engineering process, are you an agile team?”. The terms agile, Scrum and sprint are thrown around rather loosely these days and I argue that we barely have a shared understanding of what they mean. How did we get from the original agile manifesto to where we are today?

Process is a focal point when it comes to building engineering teams. In my opinion though it is one of the least important aspects. People gravitate towards it because it is tangible and easy.

What is the stuff that really matters?

Engineering teams are formed to build and improve software products that customers desire. For most organizations it is important that these teams deliver value at a somewhat predictable cadence. It is also important that these teams maintain and improve the technical foundation of their products, such that future development can happen at a reasonable speed.

What is required to achieve these goals?

This is not a complete list, but it highlights the aspects that have been the most important in my experience. They fall roughly into two categories, capabilities of team members and the way they work together.

  • Capabilities:
    • Engineers need to have an understanding of the technology that is used to build the product and need to have an interest in the type of problems the team needs to solve
    • Engineers need to be able to evaluate the technical effort of multiple prioritized initiatives such that they can optimize the cost/benefit ratio and define milestones
    • Engineers need to understand the system they are working in well enough to identify areas that require maintenance and reduction of technical debt in order to not reduce future velocity
    • Engineers on the team need to be able to collaborate and communicate well with each other
  • How work is done:
    • The team needs to understands the customer’s desires and therefore be able prioritize initiatives (typically with the help of a product manager)
    • Engineers need to be engaged in their work to deliver maximum performance. This typically requires understanding the impact of their work as well as having some degree of creative freedom
    • Engineers need lots of time to focus
    • Engineers need to be trusted with the details of their work and need to be challenged and held accountable for outcomes
    • The team needs to feel momentum. Frequent value delivery, combined with analyzing the impact of the team’s work via metrics is very important for motivation. Momentum can be best achieved by delivering projects in an iterative fashion
    • The team needs to have limited dependencies on other parts of the organization to get its work done

What can a engineering manager do to help meet these requirements?

  • Hiring, hiring, hiring. Many of the bullet points above require strong individual capabilities. Composition matters as well. A strong team has a good mix of engineers of different seniority, who enjoy working together, such that the team can find engagement for engineering work of various complexity
  • Contextualizing the work the team is doing with respect to the overall product and organization
  • Defining small, packages of work with measurable outcomes. Keeping packages small is the best way to arrive at realistic milestones
  • Creating a work environment with limited distractions
  • Enabling engineers to spend some time on work they believe is important by letting them fix technical debt and enabling and leveraging their creativity for product work
  • Enabling engineers to participate in planning their work and setting milestones, while holding them accountable for outcomes
  • Identifying and removing dependencies that the team has on the rest of the organization

Can process help with this?

It might. But it can only address a small subset of the requirements for a great team and in my experience rarely succeeds at even that.

I frequently see teams implement a formal Scrum process, but combining it with multi-month long projects and a waterfall approach. This leads to a lot of overhead for sprint planning, estimation and retros but doesn’t provide the momentum of an iterative development approach and typically does not lead to a predictable schedule, since even the best engineers are bad at estimating huge projects. This process also rarely leaves room for creativity, as teams are churning through predefined stories and tasks from sprint to sprint.

My main issue though is not the implementation of a specific process. It is that many teams put too much emphasis on process in the first place, while neglecting more important areas of building a great team.

Conclusion

I wanted to keep this post short. Specific prescriptions about how to run an engineering team can fill entire books. The goal of this post is to remind engineers, engineering managers and organizations about what really matters and to encourage them to debug their engineering performance by looking at these important requirements for a successful team.

Where process can help addressing these requirements it should be implemented. But the focus should lie on hiring a great team, building momentum, enabling collaboration and engaging and motivating each team member - process won’t get that important job done.


P.S.: Here’s a somewhat related and great post from the HubSpot blog: Why our engineering leaders focus on product over process.