This quote from Charity Majors is probably the best summary of the current state of observability in the tech industry - a total, mass confusion. Everyone is confused. What is a trace? What is a span? Is log line a span? Do I need traces if I have logs? Why I need traces if I have great metrics? The list of questions like these goes on.
Charity - together with other great folks from observability system called Honeycomb - have been doing a great job shedding light on these questions. Yet, per my own experience it's still extremely hard to explain what does Charity meant by "logs are thrash", let alone the fact that logs and traces are essentially the same things.
Why is everyone so confused? With the risk of being spicy a little, I'm going to blame Open Telemetry. Yes, it's powering the modern observability stack and yet I blame it for the mass confusion. Not because it's a bad solution - it's great! But the presentation and the whole approach of explaining what Open Telemetry is and what it does makes the observability look tricky and complex.
First, Open Telemetry from the very beginning makes a clear distinctions between traces, metrics and logs: Open Telemetry is a collection of APIs, SDKs, and tools. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software's performance and behavior. Then it goes deeper in explaining each of these 3. This is screenshot from the part of open telemetry website introducing traces. Based on my experience talking to people who work with Open Telemetry, this presentation has indeed become one of the main pictures being associated with observability. For some, this IS the observability. And it also sets traces apart from anything else. This is clear not a log, is it? This also doesn't look like a metric, right? It's something special, probably a bit sacred, and requiring learning investment.
Per my experience, once people learned about traces, they only think about them in the context of this picture and the whole set of terms like spans, root spans, nested spans and the rest. Open Telemetry site has a glossary page with more than 60 terms! This all is insanely complex! But what's more important - does this focus on "logs, metrics and traces" represent the true power of observability? It does cover some scenarios that's true, but when it comes to the distributed systems at scale what's more important is an ability to "dig" into data - "slice and dice" it, build and analyse various views, correlate, search for anomalies... And systems that offer all of this do exist.
When I was working at Meta, I wasn't aware that I was privileged to be using the best observability system ever. This system is called Scuba and it's the top 1 thing by a large margin that people miss when they leave Meta. The basic idea of Scuba is extremely simple and doesn't require a glossary page for people to grasp. It operates with Wide Events. Wide Event is just a collection of fields with names and values, pretty much like a json document. If you need to record some information - whether it's the current state of the system, or an event caused by an API call, background job or whatever - you can just write some Wide Event to Scuba. For instance, if a system serves ads, the natural desire would be to record Ad Impressions - the facts that a certain ad has been seen by a user. The corresponding Wide Event might be looking like this: {"Timestamp": "1707951423","Ad Id": "542508c92f6f47c2916691d6e8551279","User Country": "US","Placement": "mobile_feed","Campaing Type": "direct_ads","User OS": "Android","OSVersion": "14","App Version": "798de3c28b074df9a24a479ce98302b6",...}
Such events called wide, because it's encouraged to dump to them all the information one can think of. Anything that might be relevant in the context of a certain data - just put it there, it might be useful later. This approach is laying the groundwork for dealing with unknown unknowns - something that's very hard to do with traditional logs, metrics and traces.
Source: Hacker News · Summarized by HeadlinesBriefing