processor_sampling: new trace sampling processor

[edsiper]

This PR introduces a new trace sampling processor designed with a pluggable architecture, allowing easy extension to support multiple sampling strategies and backends.

Samplers

• probabilistic (head sampling)
• tail (tail sampling)
  • conditions:
    • status_code
    • latency
    • string_attribute
    • numeric_attribute
    • boolean_attribute
    • span_count
    • trace_state

Head Sampling

For head sampling needs, it provides a probabilistic sampler.

sampling settings	description	default
sampling_percentage	sets the probability of sampling traces (0-100%). For example, 40 samples 40% of traces randomly.	10

Configuration example

pipeline:
  inputs:
    - name: opentelemetry
      port: 4318

      processors:
        traces:
          - name: sampling
            type: probabilistic
            sampling_settings:
              sampling_percentage: 40

  outputs:
    - name: stdout
      match: '*'

Tail Sampling

sampling settings	description	default
decision_wait	Specifies how long to buffer spans before making a sampling decision, allowing full trace evaluation. For example, 5s waits 5 seconds before deciding whether to sample a trace	30s

This sampler supports conditionals to sample traces if any of their spans meet a specific condition.

Condition: status_code

Samples traces based on span status codes (OK, ERROR, UNSET)

property	description	default
status_codes	defines an array of span status codes (OK, ERROR, UNSET) to filter traces. Traces are sampled if any span matches a listed status code. For example, status_codes: [ERROR, UNSET] captures traces with errors or unset statuses

example:

pipeline:
  inputs:
    - name: opentelemetry
      port: 4318
      processors:
        traces:
          - name: sampling
            type: tail
            sampling_settings:
              decision_wait: 5s
            conditions:
              - type: status_code
                status_codes: [ERROR]
  outputs:
    - name: stdout
      match: '*'

Condition: latency

Samples traces based on span duration. Uses threshold_ms_low to capture short traces and threshold_ms_high for long traces

condition settings	description	default
threshold_ms_low	specifies the lower latency threshold. Traces with a duration <= this value will be sampled.	0
threshold_ms_high	specifies the upper latency threshold. Traces with a duration >= this value will be sampled.	0

example:

pipeline:
  inputs:
    - name: opentelemetry
      port: 4318
      processors:
        traces:
          - name: sampling
            type: tail
            sampling_settings:
              decision_wait: 5s
            conditions:
                - type: latency
                  threshold_ms_high: 200
                  threshold_ms_high: 3000
  outputs:
    - name: stdout
      match: '*'

This tail-based sampling configuration waits 5 seconds before making a decision. It samples traces based on latency, capturing short traces of 200ms or less and long traces of 3000ms or more. Traces between 200ms and 3000ms are not sampled unless another condition applies.

Condition: string_attribute

The string_attribute conditional allows traces to be sampled based on specific span or resourceattributes. Users can define key-value filters (e.g., http.method=POST) to selectively capture relevant traces.

condition settings	description	default
key	specifies the span or resource attribute to match (e.g., "service.name").
values	defines an array of accepted values for the attribute. A trace is sampled if any span contains a matching key-value pair: ["payment-processing"]
match_type	defines how attributes are compared: strict ensures exact value matching, while exists checks if the attribute is present regardless of its value (note that string type is enforced)	strict

example:

pipeline:
  inputs:
    - name: opentelemetry
      port: 4318
      processors:
        traces:
          - name: sampling
            type: tail
            sampling_settings:
              decision_wait: 5s
            conditions:
              - type: string_attribute
                match_type: strict
                key: "http.method"
                values: ["GET"]
              - type: string_attribute
                match_type: exists
                key: "service.name"
  outputs:
    - name: stdout
      match: '*'

Condition: numeric_attribute

Allows traces to be sampled based on numeric attribute values. Users can define min and max thresholds.

condition settings	description
min_value	The minimum inclusive value for the numeric attribute; traces with values >= min_value are considered.
max_value	The maximum inclusive value for the numeric attribute; traces with values <= max_value are considered.
match_type	Defines how attribute values are evaluated: strict matches exact values (default), while exists checks if the attribute is present, regardless of its value.

example:

pipeline:
  inputs:
    - name: opentelemetry
      port: 4318

      processors:
        traces:
          - name: sampling
            type: tail
            sampling_settings:
              decision_wait: 2s
            conditions:
              - type: numeric_attribute
                key: "http.status_code"
                min_value: 400
                max_value: 504
  outputs:
    - name: stdout
      match: '*'

Condition: boolean_attribute

The boolean_attribute sampling policy filters traces based on a boolean attribute’s value (e.g., true or false). This allows selecting traces based on flags like error indicators or debug modes.

condition settings	description
value	Specifies whether the trace is sampled based on the attribute being explicitly true or false.
match_type	Defines how attribute values are evaluated: strict matches exact values (default), while exists checks if the attribute is present, regardless of its value.

example:

pipeline:
  inputs:
    - name: opentelemetry
      port: 4318

      processors:
        traces:
          - name: sampling
            type: tail
            sampling_settings:
              decision_wait: 2s
            conditions:
              - type: numeric_attribute
                key: "http.status_code"
                min_value: 400
                max_value: 504
  outputs:
    - name: stdout
      match: '*'

Condition: span_count

Allows to sample traces that contain a specific number of spans defined by a configurable range.

condition settings	description
min_spans	The minimum number of spans a trace must have to be sampled.
max_spans	The maximum number of spans a trace can have to be sampled.

example:

pipeline:
  inputs:
    - name: opentelemetry
      port: 4318

      processors:
        traces:
          - name: sampling
            type: tail
            sampling_settings:
              decision_wait: 2s
            conditions:
              - type: span_count
                min_spans: 3
                max_spans: 5

Condition: trace_state

Uses a conditional for the trace_state field allowing traces to be sampled based on metadata stored in the W3C trace_state field.

condition settings	description
values	Defines a list of key-value pairs to match against the trace_state. A trace is sampled if any of the specified values exist in the trace_state. Matching follows OR logic, meaning at least one value must be present for sampling to occur.

example:

pipeline:
  inputs:
    - name: opentelemetry
      port: 4318

      processors:
        traces:
          - name: sampling
            type: tail
            sampling_settings:
              decision_wait: 2s
            conditions:
              - type: trace_state
                values: [debug=false, priority=high]
  outputs:
    - name: stdout
      match: '*'

Test

Fluent Bit v4.0.0
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _             ___  _____
|  ___| |                | |   | ___ (_) |           /   ||  _  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __/ /| || |/' |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| ||  /| |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /\___  |\ |_/ /
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/     |_(_)___/

[2025/02/28 16:46:00] [ info] [fluent bit] version=4.0.0, commit=0e885e2d60, pid=778903 [2025/02/28 16:46:00] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/02/28 16:46:00] [ info] [simd    ] disabled
[2025/02/28 16:46:00] [ info] [cmetrics] version=0.9.9
[2025/02/28 16:46:00] [ info] [ctraces ] version=0.6.0
[2025/02/28 16:46:00] [ info] [input:opentelemetry:opentelemetry.0] initializing
[2025/02/28 16:46:00] [ info] [input:opentelemetry:opentelemetry.0] storage_strategy='memory' (memory only)
[2025/02/28 16:46:00] [ info] [input:opentelemetry:opentelemetry.0] listening on 0.0.0.0:4318
[2025/02/28 16:46:00] [ info] [processor:sampling:sampling.0] initializing probabilistic sampling processor
[2025/02/28 16:46:00] [ info] [sp] stream processor started
[2025/02/28 16:46:00] [ info] [output:stdout:stdout.0] worker #0 started

🔍 Debug sampling 'probabilistic' (0x779068027940): before
   ┌─────────────────────────────────────────────────────────────────┐
   │ trace_id=5b8efff798038103d269b633813fc60c                       │
   ├─────────────────────────────────────────────────────────────────┤
   │ spans:                                                          │
   │   ├── id=eee19b7ec3c1b174 name=I'm a server span                │
   │   ├── id=eee19b7ec3c1b175 name=Child span of server span        │
   │   ├── id=eee19b7ec3c1b176 name=Database query                   │
   └─────────────────────────────────────────────────────────────────┘

   ┌─────────────────────────────────────────────────────────────────┐
   │ trace_id=6a9dfff798038103d269b633813fc60d                       │
   ├─────────────────────────────────────────────────────────────────┤
   │ spans:                                                          │
   │   ├── id=fff19b7ec3c1b174 name=A span in another trace          │
   └─────────────────────────────────────────────────────────────────┘

   ┌─────────────────────────────────────────────────────────────────┐
   │ trace_id=7c8efff798038103d269b633813fc60e                       │
   ├─────────────────────────────────────────────────────────────────┤
   │ spans:                                                          │
   │   ├── id=0000000000000000 name=Slow request                     │
   └─────────────────────────────────────────────────────────────────┘

   ┌─────────────────────────────────────────────────────────────────┐
   │ trace_id=8d9efff798038103d269b633813fc60f                       │
   ├─────────────────────────────────────────────────────────────────┤
   │ spans:                                                          │
   │   ├── id=0000000000000000 name=High traffic span                │
   │   ├── id=0000000000000000 name=Load testing event               │
   └─────────────────────────────────────────────────────────────────┘

   ┌─────────────────────────────────────────────────────────────────┐
   │ trace_id=9a1bfff798038103d269b633813fc610                       │
   ├─────────────────────────────────────────────────────────────────┤
   │ spans:                                                          │
   │   ├── id=0000000000000000 name=Faulty transaction               │
   │   ├── id=0000000000000000 name=Database rollback                │
   └─────────────────────────────────────────────────────────────────┘

🔍 Debug sampling 'probabilistic' (0x779068027940): after
   ┌─────────────────────────────────────────────────────────────────┐
   │ trace_id=6a9dfff798038103d269b633813fc60d                       │
   ├─────────────────────────────────────────────────────────────────┤
   │ spans:                                                          │
   │   ├── id=fff19b7ec3c1b174 name=A span in another trace          │
   └─────────────────────────────────────────────────────────────────┘

   ┌─────────────────────────────────────────────────────────────────┐
   │ trace_id=7c8efff798038103d269b633813fc60e                       │
   ├─────────────────────────────────────────────────────────────────┤
   │ spans:                                                          │
   │   ├── id=0000000000000000 name=Slow request                     │
   └─────────────────────────────────────────────────────────────────┘

|-------------------- RESOURCE SPAN --------------------|
  resource:
     - attributes: - service.name: 'other.service'
     - dropped_attributes_count: 0 - schema_url: "" [scope_span] instrumentation scope:
        - name                    : other.library
        - version                 : 2.0.0
        - dropped_attributes_count: 0
        - attributes: undefined
    schema_url: ""
    [spans]
         [span #0 'A span in another trace']
             - trace_id                : 6a9dfff798038103d269b633813fc60d
             - span_id                 : fff19b7ec3c1b174
             - parent_span_id          : undefined
             - kind                    : 2 (server)
             - start_time              : 1544712660000000000
             - end_time                : 1544712662000000000
             - dropped_attributes_count: 0
             - dropped_events_count    : 0
             - dropped_links_count     : 0
             - trace_state             : (null)
             - status:
                 - code    : 0
             - attributes: none
             - events: none
             - [links]
|-------------------- RESOURCE SPAN --------------------|
  resource:
     - attributes:
            - service.name: 'latency.test.service'
     - dropped_attributes_count: 0
     - schema_url: ""
  [scope_span]
    instrumentation scope:
        - name                    : latency.test.library
        - version                 : 3.0.0
        - dropped_attributes_count: 0
        - attributes: undefined
    schema_url: ""
    [spans]
         [span #0 'Slow request']
             - trace_id                : 7c8efff798038103d269b633813fc60e
             - span_id                 : 0000000000000000
             - parent_span_id          : undefined
             - kind                    : 2 (server)
             - start_time              : 1544712660000000000
             - end_time                : 1544712675000000000
             - dropped_attributes_count: 0
             - dropped_events_count    : 0
             - dropped_links_count     : 0
             - trace_state             : (null)
             - status:
                 - code    : 0
             - attributes: none
             - events: none
             - [links]

Manual test

Using this JSON file: trace_sampling_extended.json , try with curl:

curl -X POST -H "Content-Type: application/json" -d @trace_sampling_extended.json -i localhost:4318/v1/traces

---

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.