Observability and monitoring

I've come to highly value observability and (good) monitoring, having seen the huge difference it makes to the pace and quality of delivery, as well as its importance in production support.

Having learnt from experience, I would put it very near the top of a playbook for fixing a struggling project, even ahead of improvements to things like CI/CD and automated testing, although these have great value too. On a greenfield project, I'd certainly give it a greal deal of thought from the outset.

Being able to see and understand the actual behaviour of our software and its users, in every environment, in real time, gives us insights, feedback loops and guardrails that support us at every stage of the delivery lifecycle, giving us the confidence to move quickly and the ability to respond well to the unexpected.

Defining telemetry, observability and monitoring

Telemetry generally consists of logs, traces and metrics. These are the foundations that allow us to monitor and observe a software system.

Observability and monitoring are related and overlapping, but do have different meanings.

Observability is a property of our product, like security, accessibility, testability, and so on. We increase the product's observability by good use of logs and traces, which can be analysed by dedicated tools and can be used to produce custom metrics. We can make use of our system's observability for ad hoc debugging and investigation. 

Monitoring is more about practices that give us oversight of a system's general health and performance. It will include dashboards that we design to show key metrics, like request statuses, latency or CPU usage. It will also include alerts that notify us if specific metrics are outside a specified range, such as a high rate of request failures or a big degradation in performance.

Well-designed monitoring might contribute to overall observability, but doesn't necessarily provide good observability itself.

You could have a system that is quite good on observability, because its rich logs and traces allow easy debugging, but bad on monitoring, because there's no clear overview of system health and no automated alerts to get our attention when something is obviously wrong.

Vice versa, you could also have a system that is pretty good on monitoring, with dashboards and alerts covering the high-level health of all the infra, but is poor on obvservability, lacking a detailed, joined-up account of each transaction flowing through the system.

How it helps us

• Learning and understanding the system architecture.

• Understanding user behaviour and experience.

• Informing design choices.

• Quick feedback in development and testing.

• Identifying bugs and incidents.

• Revealing inefficiencies and poor performance.

• A vital accessory to performance testing.

• Confidence in new features and fixes.

• Visualising the back-end for non-technical colleagues and stakeholders.