Top Engineering Metrics
The metrics I use to understand the capabilities of a new engineering team or project.
When first joining a company or a new project, as a technology leader, you need to understand the state of play quickly to devise a strategy and roll it out in order to have an impact. One of the key ways I like to do this is by collecting a number of key metrics, as well as more qualified information from 1:1s with key people.
Consideration
There are a whole array of different metrics that people say are good or bad, so the question is, how do you know which ones to track? I decided to put together a starting list to help me onboard into organisations and here are some of the considerations I took into account:
- I wanted to keep the set as small as possible, whilst giving a rounded holistic view of the team. Focussing too much on one area could cause problems in others.
- This is a generic set for an initial look into how things are going on a team. It is important not to always stick with these, but to adapt the scorecard as you learn more.
- I have included owners of the metrics to distribute the responsibility of implementing the method of collecting them and to give goals for their improvement to the relevant people.
- I put together some starting targets, but these are not perfect or necessarily attainable. They do however help surface potential problems or areas to celebrate.
- Metrics are indicators and not the truth. I always use them to start conversations, not to come to conclusions straight away. Taking time to understand the history is really important. There may be some you want to add/remove for political reasons.
- Product metrics are more important but often harder to measure. These metrics mean nothing unless you are building something the customer wants or that provides value.
➕Health
Team Health
Owner: Engineering Manager
Benefits
- Ensuring the team is happy and motivated is key to getting the most out of the team, keeping the retention rate high, and making your organisation is a great place to work.
Measuring
At the start of each retrospective, go round the team and ask each team member to give a score between 1 and 5, 1 being their worst sprint ever and 5 being their best sprint ever. As you go around, ask each team member to also give a short explanation of why they have picked this number. Sometimes it can be an amazing sprint for one person, as they were able to work on new tech, whilst it is terrible for another as all they did was fix bugs.
The whole team needs to hear and understand each other's experiences. For example, in the case above, maybe the team will ensure that the person who picked up all the bugs gets a more exciting task next time.
The score from each team member is then taken as a team average.
Starting Targets
Red: < 3
Green: > 4
Backlog Health
Owner: Product Owner
Benefits
- By having a well-defined backlog of items, the team will be more efficient in getting the work done and have less of a dependency on stakeholders being available to provide the details.
- People can easily pick up the next most important thing in the backlog.
- You can start to plan out beyond a sprint as per the agile onion.
Measuring
The number of sprints of estimated work that meets the definition of ready for the team.
total story points / team velocity = backlog health
Starting Targets
Red: < 2
Green: > 3
Infrastructure Costs
Owner: Engineering Manager / Tech Lead
Benefits
- You gain an understanding of the scale the company is operating at.
- Helps identify possible cost-saving measures (such as turning off dev environments over weekends).
- Allows you and your team to understand the impact of architectural decisions and to keep it at the forefront of their mind when designing solutions.
Measuring
Most cloud providers come with some kind of cost analysis tool, such as GCP outputting the data straight into BigQuery. You can then point Looker Studio to that and easily build out dashboards to understand your costs.
A key part of FinOps is to tag resources accordingly to understand what clients, components, or features are costs.
Starting Targets
These targets have to be determined by speaking to the business and technical teams about what are appropriate levels of costs.
Generally speaking, it is good to have a dividing number. For example the cost per X transactions/requests/events.
🚀 Delivery
Reliability
Owner: Delivery Lead
Benefits
- POs can confidently report on progress to stakeholders.
- Dependencies between teams can more easily be worked out.
- Creates urgency in the team to delivery what they commit to.
- Teams can start to plan beyond a sprint and for a plan/release.
Measuring
The number of points set to ‘done’ is divided by the number of points committed to at the start of the sprint.
completed points / committed points * 100
Starting Targets
Red: < 80%
Green: > 100%
Deployment Frequency
Owner: Engineering Manager
Benefits
- One of the DORA metrics that identifies high-performing teams.
- Deploying frequently reduces the risk and scope of introducing bugs.
- Creates quick feedback loops for changes and reduces delivery times.
- Identifies the team is thinking incrementally, rather than in a waterfall, all-at-the-end approach.
Measuring
Deployments to prod / Number of team members = Deployment Frequency
Starting Targets
Red: < value that equals once a sprint
Green: > 1
Change/Failure Rate
Owner: QA
Benefits
- Another one of the DORA metrics for identifying high-performing teams.
- Identifies how good the team is at doing repeatable releases.
- Highlights if the QA practices in the team are sufficient.
Measuring
failed deployments / total deployments * 100
Starting Targets
Red: > 25%
Green: < 15%
⚙️ Engineering
Productivity
Owner: Engineering Manager
Benefits
- Flags potential problems with underperforming engineers who may need additional support.
- Can be a good indicator if engineers have to spend too much time in unnecessary meetings or performing business analysis-type activities.
- Helps identify solid contributors to your team, who may be introverted or not be able to ‘talk the talk’ so well.
- A low score may also mean the technical solution is too complex or there are too many dependencies.
Measuring
This is one of the harder metrics to track and the most controversial. My favorite way of doing this is through the productivity metric with Pluralsight Flow as it looks at the complexity of commits, not just the volume. They also provide benchmarks for a primary and occasional contributor.
Starting Targets
The targets will depend upon the tool being used. For flow I use the following:
Red: < Occasional
Green: > Primary
Coding Days
Owner: Engineering Manager
Benefits
- Pushing at least once a day removes ‘bus factor’ and ensures you don’t lose access to the work done when people are off sick.
- Promotes a practice of frequent commits and saving points within the code base.
- It also identifies if it is commonplace for people to be working over weekends or extra days which might lead to burnout and much bigger problems.
Measuring
The number of days in a week that changes have been pushed up to the repo.
Starting Targets
Red: < 2
Green: > 4
Unreviewed PRs
Owner: Engineering Manager
Benefits
- All code should be reviewed to ensure quality.
- Reduces the chances of malicious code entering production.
- PRs help in knowledge sharing, up-skilling, and technical excellence.
- Often needed for certification standards and looked at as part of due diligence on selling a company.
Measuring
A tool such as GitHub pull requests is key for measuring this. This is one of the easiest metrics to improve upon quickly.
number of PRs not reviewed / number of PRs * 100
Starting Targets
Red: > 10%
Green: 0%
👍 Quality
Code Coverage
Owner: QA/1st Line Support
Benefits
- Unit tests generally improve the quality of the software, especially non-front-end code.
- Technically excellent teams, with well-written code tend to have high unit test coverage.
Measuring
This metric is easily measured with test tooling provided by different languages. It can be nicely tracked over time using SonarQube.
Starting Targets
Green: > 80%
Red: < 40%
Incident Count
Owner: QA
Benefits
- A clear indicator of the QA strategy within the organisation and the fragility of the code base.
- Reacting to incidents makes it harder to plan for work and puts stress on the team, as well as damages the company and the team's reputation.
- This figure will determine how important it is to have a robust out-of-hours support process.
Measuring
The number of P1 or P2 incidents.
Starting Targets
Red: > 2
Green: 0
Security Vulnerabilities
Owner: Engineering Manager / Tech Lead
Benefits
- Understanding where you stand with security is always important and should never be neglected.
- If you have big holes, then you need to get them fixed ASAP to not cause you bigger problems down the line.
- Organisations will want to know your approach to security and you have measures in place before they trust you with their data.
- You don’t want the hassle (and stress!) of dealing with data breaches.
Measuring
The number of high or critical vulnerabilities detected. There are a number of tools that provide this information on the market, including SonarQube and Snyk to name a couple of the most popular.
Starting Targets
Targets should be based on the product/organisation's appetite for risk. For example, security for a military or health application will always be more important than your personal blog.
