feat(otlp): introduce "kind" to spans #4540
Conversation
OTLP spans have a property called `kind` that helps specify the relationship between that span and its parent or child. Sentry users should be able to see the span kind and search/filter on it when it is sent. During ingestion, write a string representation of the `kind` to the `span.kind` attribute. (A string representation is chosen over the default integer one because it is both self-describing and also makes search straightforward (no need for translations between string and integer in the frontend or EAP. We are relying on Clickhouse's compression to make the additional data usage reasonable.) [0] https://github.com/open-telemetry/opentelemetry-proto/blob/d7770822d70c7bd47a6891fc9faacc66fc4af3d3/opentelemetry/proto/trace/v1/trace.proto#L152-L178 Co-authored-by: Ash Anand <[email protected]>
| #[metastructure(field = "span.kind")] | ||
| pub span_kind: Annotated<String>, |
There was a problem hiding this comment.
Please add a small doc string, ideally what values it can have and when it should be filled.
There was a problem hiding this comment.
See opentelemetry-proto:
// Distinguishes between spans generated in a particular context. For example,
// two spans with the same name may be distinguished using `CLIENT` (caller)
// and `SERVER` (callee) to identify queueing latency associated with the span.
relay-spans/src/span.rs
Outdated
| @@ -201,6 +202,11 @@ pub fn otel_to_sentry_span(otel_span: OtelSpan) -> EventSpan { | |||
| description = Some(format!("{http_method} {http_route}")); | |||
| } | |||
|
|
|||
| data.insert( | |||
| "span.kind".to_owned(), | |||
There was a problem hiding this comment.
In opentelemetry-proto this is a top-level field on the span rather than an attribute. Could we align with their schema and create a field on our Span, too? Note we did the same with is_remote (mapping it out from flags).
There was a problem hiding this comment.
@jan-auer Can do. Just to confirm, I'm assuming that would mean adding kind as a field to both Span and SpanKafkaMessage for transfer to Snuba? (And then EAP can choose to store it however it makes sense, column or attribute?)
Also, in that case the choice of how to save the value is being made by EAP instead - in that case should we transfer it as the original integer value rather than our string representation, maybe? (Leaving it to Snuba to stringify, in storage or via RPC?)
There was a problem hiding this comment.
that would mean adding kind as a field to both Span and SpanKafkaMessage for transfer to Snuba?
That's right.
And then EAP can choose to store it however it makes sense, column or attribute?
We can consider aligning with how we want kind to show up in the product & how otel SDKs track it. In my previous comment, I assumed we would then also show it as span.kind rather than span.data.kind. If we want to show it as an attribute, then we should probably keep your current approach. Otherwise, we have a strange drift between the SDK protocol, pipeline, and the product-side data model.
in that case should we transfer it as the original integer value rather than our string representation, maybe?
Definitely possible, though I'd advise against it in this instance. In both cases - string or integer - would need to share an enum between Relay, Sentry (pipeline) and Snuba. However, if we convert to string representation right away, we can now use the same enums in the processing pipeline and in all workers/endpoints that pull data out of EAP. Essentially, the pre- and post-EAP data model would be identical, which I think would be desirable.
|
@jan-auer I've updated the PR to use a new |
mjq
changed the title
OTLP spans have a property called
kindthat helps specify the relationship between that span and its parent or child. Sentry users should be able to see the span kind and search/filter on it when it is sent (via OTLP or theOtelSpanenvelope item type). During ingestion, write a string representation of thekindto thespan.kindattribute.(A string representation is chosen over the default integer one because it is both self-describing and also makes search straightforward (no need for translations between string and integer in the frontend or EAP. We are relying on Clickhouse's compression to make the additional data usage reasonable.)