Observability has come a long way over the last decade. We have better tooling, better standards, and more telemetry than ever before. But having more telemetry doesn’t necessarily mean you have better observability.
You can collect millions of logs, instrument every service and build dozens of dashboards, yet still struggle to answer a fairly basic question when something goes wrong: what actually happened?
After years of building and operating production systems, I’ve seen the same observability mistakes come up repeatedly. Most come down to how observability is implemented across an engineering organisation and how much useful context is preserved when something goes wrong.
Here are some of the most common.
1. Not tracing requests across services
One of the most common problems I’ve seen is good instrumentation within individual services, but no way to trace a request end-to-end.
Say a request flows through:
API Gateway -> User Service -> Order Service -> Payment Service -> Database
You know the request failed, and you might even know that the Order Service returned an error. Without distributed tracing, though, investigating what happened often means looking at each service independently.
You search the API logs, find a timestamp, jump to the next service, search around the same time, and gradually try to reconstruct the request yourself. This becomes increasingly difficult as systems grow and requests pass through more services.
A request should have an identity that follows it through the system. Distributed tracing gives you a record of which services handled it, which operations were performed, how long they took, and where errors occurred.
This is especially useful in microservice architectures, where a single user action can involve several services. You shouldn’t have to reconstruct that path manually from timestamps and individual log entries.
2. Logs without request context
Finding the right log entry is often only the start of an investigation.
A typical debugging session might involve searching through thousands of entries until you eventually find something useful:
Failed to process payment for customer 12345
The message tells you that something went wrong while processing a payment, but there is still a lot you don’t know. You need to find the request that produced the log, what happened before and after it, which services were involved, and whether there were other warnings or errors during the same transaction.
Without that context, the next step is usually another search using the timestamp, customer ID, request ID, or whatever information happens to be available.
Logs are far more useful when they can be correlated with the request or transaction in which they occurred. A log entry is one event within a larger execution, and you should be able to use it as a starting point for investigating that execution.
Ideally, finding an interesting log entry should also give you access to the surrounding logs, spans, services and other activity from the same request.
That context often tells you far more than the individual log message.
3. Inconsistent logging between teams and services
Logging tends to evolve independently across teams.
One service might log:
customerId=12345
Another:
customer_id=12345
And another:
Processing request for customer 12345
The frontend might call the same value userId.
Each log can make sense in isolation, but inconsistencies become a problem when you need to search and correlate information across services.
Structured logging helps here, although switching everything to JSON doesn’t solve the problem by itself. Teams also need some agreement on how important fields are named and what they mean.
For example:
{
"message": "Processing order",
"customer.id": "12345",
"order.id": "67890"
}If customer.id and order.id have the same meaning everywhere, you can reliably search for them regardless of which team owns the service that produced the telemetry.
There is no need to standardise every log message across an organisation. Developers should still log whatever information is useful for their service. The important part is agreeing on the context that needs to survive across service and team boundaries: request identifiers, customer identifiers, order identifiers, environment, service name, and any other fields regularly used during investigations.
4. Treating every exception as an incident
Exceptions are useful signals, but an exception occurring doesn’t always mean that an operation failed.
Consider an application that tries to parse a header. Occasionally the header isn’t in the expected format and the parsing code throws an exception. The application knows how to handle this case, falls back to another value and successfully continues processing the request.
If that exception is still reported as an error, your observability data suggests that something failed even though the request completed successfully.
Over time, engineers become accustomed to these errors and start ignoring them. They can also cause problems during real incidents. An engineer sees an exception around the time of the failure, assumes it is related, and spends time investigating something that had no effect on the user.
If an exception is expected and the application can safely recover from it, handle it explicitly. Log the relevant information if it could still be useful for debugging, then continue processing.
try operation
catch expected exception
log useful context
continue processingThis doesn’t mean suppressing genuine failures. The aim is to make the distinction between an exception in the code and an operation that actually failed clear in your telemetry.
When an engineer sees an error, it should be worth investigating.
5. Too many false alarms
Noisy exceptions are one source of false alarms, but alert fatigue is a wider problem.
Alerts work when engineers trust them. If an alert fires regularly and nothing needs to be done, people quickly learn to ignore it. Notifications accumulate, Slack channels become noisy, and important alerts become harder to distinguish from everything else.
An alert should correspond to a condition that someone is expected to act on.
Before creating one, it is worth asking a few practical questions:
- Does this indicate a real problem?
- Does somebody need to do something when it fires?
- How quickly do they need to respond?
- Are customers or an important business process affected?
- What happens if nobody responds?
If there isn’t a clear answer to what someone should do when an alert fires, it probably doesn’t need to page anyone.
You can still record the condition, expose it through metrics, include it on a dashboard or investigate it later. That information may be useful without requiring someone’s immediate attention.
6. Tagging infrastructure instead of business context
Tags, attributes, labels and dimensions are a major part of most observability setups. The problem is that they often describe infrastructure while leaving out the information engineers actually need during an investigation.
Infrastructure attributes such as these are useful:
region=eu-west-1
availability_zone=eu-west-1a
instance_type=m7g.large
host=i-1234567890
If you’re investigating an infrastructure issue, you want to know where the affected service was running.
But many production investigations start somewhere else. A customer reports a failed order. Support gives you a customer ID. An alert fires and you need to know which team owns the affected service and whether it is customer-facing.
For those situations, you need business and operational context alongside the infrastructure data:
customer.id=12345
order.id=67890
team=payments
tier=customer-facing
domain=checkout
environment=prod
When these attributes are consistently available across your telemetry, they give engineers useful ways to navigate and filter it.
If someone reports that order 67890 failed, you should be able to search for that identifier and find the requests, logs, services and errors associated with it. You shouldn’t have to hope that an engineer happened to include the order ID somewhere in an unstructured log message.
The same principle applies to incident prioritisation. An error in environment=dev usually doesn’t deserve the same response as an error in environment=prod. Similarly, an issue affecting a customer-facing checkout service is likely to have a different priority from an issue affecting a non-critical internal tool.
Infrastructure metadata remains important for understanding where something happened. Business and operational metadata tells you what was affected and helps you decide what to do about it. A useful observability setup needs both.
The common theme: context
A lot of observability problems eventually come back to missing context.
If a trace ends at a service boundary, you lose part of the request path. If a log can’t be connected to its request, you have to reconstruct the surrounding activity yourself. If every service uses a different name for the same customer identifier, correlation becomes unreliable. Noisy exceptions and alerts make it harder to distinguish real failures from events that don’t require action.
Collecting more telemetry doesn’t fix these problems by itself. The telemetry needs enough structure and context to help an engineer understand what happened, what was affected, and where to investigate next.
That is a much better measure of an observability setup than the number of logs, metrics or dashboards it produces.
Observability is becoming context for AI
These practices also matter as AI becomes part of the debugging workflow.
An AI agent given a single exception or log entry has many of the same limitations as an engineer looking at that information in isolation. It may know what error occurred without knowing which request produced it, what happened earlier in the request, which services were involved, or whether the error affected the user at all.
Distributed traces, correlated logs, consistent attributes and business context give an AI agent more information to work with when investigating a problem.
As more engineering teams use AI to investigate production issues and generate fixes, the quality of their observability data will increasingly affect the quality of those investigations. Giving an AI agent access to telemetry is useful. Giving it well-structured telemetry with the context around each event is much more useful.