Metrics give insights into what's happening at all stages of the DevOps pipeline, from design to development to deployment. Metrics are objective measures. They strengthen the feedback loops that are essential to DevOps. Collecting metrics and displaying them via dashboards or scorecards should be automated. It's important to map these metrics to business needs.
What factors make for a good DevOps metric?
A good DevOps metric must ideally be all of these:
- Obtainable: A metric that can't be measured is useless.
- Reviewable: It must be relevant to the business and stand up to scrutiny.
- Incorruptible: It should be free from influence of teams and team members.
- Actionable: It should suggest improvements to workflows, policies, incentives, tools, etc.
- Traceable: It should be possible to trace the metric to root causes.
- What's the process of working with DevOps metrics?
A typical process involved identifying the metrics, putting in place methods to measure them, measuring and displaying them on dashboards, evaluating the metrics in terms status and trends, acting on the metrics to effect change, and continually assessing if the metrics are aligned to business goals.
Since DevOps is cross-functional (process, people, tools) and cross-teams (dev, ops, testing), metrics should not narrowly focus on only some parts of the value chain. Metrics should capture a holistic view of the entire value chain.
What are some important DevOps metrics?
There are dozens of metrics spread across all phases of a DevOps pipeline. Some have attempted to group them into categories:
- Velocity: lead time, change complexity, deployment frequency, MTTR
- Quality: deployment success rate, application error rate, escaped defects, number of support tickets, automated test pass percentage
- Performance: availability, scalability, latency, resource utilization
- Satisfaction: usability, defect age, subscription renewals, feature usage, business impact, application usage and traffic
Another grouping can be host-based metrics, application metrics, network metrics, server pool metrics and external dependency metrics.
There are also metrics for application build cycles, metrics for application performance, metrics for delivery performance, metrics organized by infrastructure, system and team health, and metrics for building or running apps.
At a minimum, aim for more deployments per week, shorter lead time from code commit to deployment, lower failure rate in production, and shorter time to repair failures. Have metrics to measure these.
Likewise, a study from 2019 identified lead time, deployment frequency, mean time to restore (MTTR) and change fail percentage as key metrics.
What are the important metrics in the world of microservices and serverless architectures?
For microservices, metrics to include are number of requests per second, number of failed requests per second, and distribution of request service times.
For serverless, the concern shifts from monitoring infrastructure to the application itself. Metrics include performance such as function runtime; scaling such as concurrency limits or memory limits; tracing event-triggered call flows across services or functions; and errors such as code bug, wrong invocation or function timeout.
For both microservices and serverless, it's important to instrument the code. Observability is an important aspect, which means that communication across services and functions needs to be accessible. A single request must be correlated to the sequence of service calls that followed it. Istio is a tool that requires strong observability.
From traditional engineering, DevOps has adopted the following metrics:
- Mean Time To Detect (MTTD): This is the average time to discover a problem. It's an indication of how effective is your incident management tools and processes.
- Mean Time To Failure (MTTF): This is an indication of how long on average the system or a component can run before failing. This can suggest preventive maintenance. This metric relates to improving system uptime.
- Mean Time Between Failures (MTBF): This is the average time between failures. It's a measure of reliability and availability.
- Mean Time To Repair (MTTR): This is the average time to repair/resolve/recover after failure is detected. This metric relates to reducing system downtime. Code complexity is one aspect that affects MTTR.
The goal is to reduce MTTD and MTTR while increasing MTTF and MTBF. DevOps is about incremental changes. If many changes are introduced at once, it will take longer to detect and fix issues.
Are there DevOps metrics that one should avoid?
Teams transitioning to DevOps might end up adopting the wrong metrics. In fact, traditional metrics such as MTBF could be seen as irrelevant for DevOps where some failures are expected due to the speed of delivery. Look beyond such costs. Instead, improve total economic impact. Others to avoid are metrics that focus on business velocity at the expense of quality or culture; metrics that are optimized for one team and causing negative impact on others.
Avoid conflict metrics that promote individuals rather than teams or pit one team versus another. These include ranking individuals or teams based on failure metrics (broken builds, etc.), rewarding top performers who don't collaborate or having different standards for different teams.
Avoid vanity metrics that promote quantity or speed over quality: number of lines of code, number of deployments per week, number of bugs fixed, number of tests added.
Don't collect a specific metric just because it's easy. Don't use a metric that encourages negative behaviours.
For those new to DevOps metrics, start with metrics that are simpler to collect and manage. Get the momentum going. For better focus, don't apply too many metrics. Choose metrics aimed at broader organizational goals or process health issues. Measure fast to enable real-time feedback loops.
Because automated system-based metric collection is hard to do, you may want to start with surveys. In fact, both these are complementary. Surveys are good for metrics on culture or things outside the system.
Use metrics that suit your business model. Adopt value stream mapping in which each metric is mapped to business values. For example, measuring website responsiveness becomes more useful if you can map it to business outcomes such as customer churn or abandoned shopping carts.
Metrics can also be role-based (business vs engineering): give teams the choice to customize their own dashboards. In fact, dashboards are essential for tracking all metrics in one place. Compare trends, not teams. Look for outliers. Measure lead time to production, not just completion.
Evolve your metrics as new technologies and tools enter your DevOps pipeline.