Fluent-bit crashes when handling envoy empty spans #9391

carlos4ndre · 2024-09-16T15:06:30Z

Bug Report

Describe the bug

I’m having an issue using fluent-bit (3.1.7) when processing spans sent by Envoy.

The flow is like this: Envoy -> fluent-bit -> OTel collector

From what I can see in the logs, Envoy is sending some periodic data (every 10s) but with empty spans, probably some sort of health check and causes fluent-bit to crash.

Not sure if this is a well known issue. Envoy is configured to send traces over gRPC.

To Reproduce

Create config files for fluent-bit, envoy and OTel collector services

fluent-bit.conf

[SERVICE]
  Flush        1
  Log_Level    debug
  Daemon       off

[INPUT]
  Name       opentelemetry
  Listen     0.0.0.0
  Port       4318
  Tag        otel

#[INPUT]
#  Name  event_type
#  Type  traces
#  Tag   otel

[OUTPUT]
  Name  stdout
  Match otel

[OUTPUT]
  Name       opentelemetry
  Match      otel
  Host       otel-collector
  Port       4318

otel-collector.yaml

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [logging]

envoy.yaml

admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    traffic_direction: OUTBOUND
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          tracing:
            spawn_upstream_span: true
            verbose: false
            provider:
              name: envoy.tracers.opentelemetry
              typed_config:
                "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
                grpc_service:
                  envoy_grpc:
                    cluster_name: fluentbit_agent
                  timeout: 2s
                service_name: front-envoy
            client_sampling:
              value: 100
            random_sampling:
              value: 100
            overall_sampling:
              value: 100
          codec_type: AUTO
          stat_prefix: ingress_http
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: proxy_routes
            virtual_hosts:
            - name: proxy
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/echo"
                direct_response:
                  status: 200
                  body:
                    inline_string: "OK"
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: "/dev/stdout"
              log_format:
                json_format:
                  traceparent: "%REQ(TRACEPARENT)%"
                  tracestate: "%REQ(TRACESTATE)%"
  clusters:
  - name: fluentbit_agent
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    load_assignment:
      cluster_name: fluentbit_agent
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: fluent-bit
                port_value: 4318

Create docker-compose.yaml file

services:
  envoy:
    image: envoyproxy/envoy:distroless-v1.31.1
    volumes:
    - ./envoy.yaml:/etc/envoy/envoy.yaml
    ports:
    - "10000:10000"
    - "9901:9901"

  fluent-bit:
    image: fluent/fluent-bit:3.1.7
    container_name: fluent-bit
    volumes:
    - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    ports:
    - "4318:4318"

  otel-collector:
    image: otel/opentelemetry-collector:0.109.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
    - ./otel-collector.yaml:/etc/otel-collector-config.yaml

Run the following command to bring everything up:

~/demo $ tree
.
├── docker-compose.yaml
├── envoy.yaml
├── fluent-bit.conf
└── otel-collector.yaml

0 directories, 4 files


$ docker compose up --build -d
[+] Running 3/3
 ✔ Container demo-otel-collector-1  Started                                                                                                            0.2s
 ✔ Container demo-envoy-1           Started                                                                                                            0.2s
 ✔ Container fluent-bit             Started

Fluent-bit crashes when receiving payload sent by Envoy

$ docker ps -a
CONTAINER ID   IMAGE                                  COMMAND                  CREATED              STATUS                            PORTS                                                                   NAMES
c61911c888d6   otel/opentelemetry-collector:0.109.0   "/otelcol --config=/…"   About a minute ago   Up About a minute                 4317/tcp, 0.0.0.0:8888->8888/tcp, 55678/tcp, 0.0.0.0:55679->55679/tcp   demo-otel-collector-1
5154ff9bc29b   envoyproxy/envoy:distroless-v1.31.1    "/usr/local/bin/envo…"   About a minute ago   Up About a minute                 0.0.0.0:9901->9901/tcp, 0.0.0.0:10000->10000/tcp                        demo-envoy-1
b82479cb74c9   fluent/fluent-bit:3.1.7                "/fluent-bit/bin/flu…"   About a minute ago   Exited (133) About a minute ago                                                                           fluent-bit

$ docker logs b82479cb74c9
Fluent Bit v3.1.7
* Copyright (C) 2015-2024 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  __
|  ___| |                | |   | ___ (_) |         |____ |/  |
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`| |
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \ | |
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /_| |_
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)___/

[2024/09/16 18:37:12] [ info] Configuration:
[2024/09/16 18:37:12] [ info]  flush time     | 1.000000 seconds
[2024/09/16 18:37:12] [ info]  grace          | 5 seconds
[2024/09/16 18:37:12] [ info]  daemon         | 0
[2024/09/16 18:37:12] [ info] ___________
[2024/09/16 18:37:12] [ info]  inputs:
[2024/09/16 18:37:12] [ info]      opentelemetry
[2024/09/16 18:37:12] [ info] ___________
[2024/09/16 18:37:12] [ info]  filters:
[2024/09/16 18:37:12] [ info] ___________
[2024/09/16 18:37:12] [ info]  outputs:
[2024/09/16 18:37:12] [ info]      stdout.0
[2024/09/16 18:37:12] [ info]      opentelemetry.1
[2024/09/16 18:37:12] [ info] ___________
[2024/09/16 18:37:12] [ info]  collectors:
[2024/09/16 18:37:12] [ info] [fluent bit] version=3.1.7, commit=c6e902a43a, pid=1
[2024/09/16 18:37:12] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2024/09/16 18:37:12] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2024/09/16 18:37:12] [ info] [cmetrics] version=0.9.5
[2024/09/16 18:37:12] [ info] [ctraces ] version=0.5.5
[2024/09/16 18:37:12] [ info] [input:opentelemetry:opentelemetry.0] initializing
[2024/09/16 18:37:12] [ info] [input:opentelemetry:opentelemetry.0] storage_strategy='memory' (memory only)
[2024/09/16 18:37:12] [debug] [opentelemetry:opentelemetry.0] created event channels: read=21 write=22
[2024/09/16 18:37:12] [debug] [downstream] listening on 0.0.0.0:4318
[2024/09/16 18:37:12] [ info] [input:opentelemetry:opentelemetry.0] listening on 0.0.0.0:4318
[2024/09/16 18:37:12] [debug] [stdout:stdout.0] created event channels: read=24 write=25
[2024/09/16 18:37:12] [debug] [opentelemetry:opentelemetry.1] created event channels: read=31 write=32
[2024/09/16 18:37:12] [debug] [router] match rule opentelemetry.0:stdout.0
[2024/09/16 18:37:12] [debug] [router] match rule opentelemetry.0:opentelemetry.1
[2024/09/16 18:37:12] [ info] [sp] stream processor started
[2024/09/16 18:37:12] [ info] [output:stdout:stdout.0] worker #0 started
[2024/09/16 18:37:17] [debug] [task] created task=0xffffa68396e0 id=0 OK
[2024/09/16 18:37:17] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2024/09/16 18:37:17] [engine] caught signal (SIGSEGV)
...
|-------------------- RESOURCE SPAN --------------------|
  resource:
     - attributes:
            - service.name: 'front-envoy'
     - dropped_attributes_count: 0
  schema_url:
  [scope_span]
    schema_url:
    [spans]
[2024/09/16 18:37:17] [debug] [output:opentelemetry:opentelemetry.1] ctraces msgpack size: 1562
[2024/09/16 18:37:17] [debug] [output:stdout:stdout.0] ctr decode msgpack returned : 6
[2024/09/16 18:37:17] [debug] [out flush] cb_destroy coro_id=0
#0  0xaaaac47bcc03      in  process_traces() at plugins/out_opentelemetry/opentelemetry.c:389
#1  0xaaaac47bcc03      in  cb_opentelemetry_flush() at plugins/out_opentelemetry/opentelemetry.c:485
#2  0xaaaac4b765a7      in  co_switch() at lib/monkey/deps/flb_libco/aarch64.c:133
#3  0xffffffffffffffff  in  ???() at ???:0

Now use the event_type input to show that opentelemetry output works fine with normal span payloads:

#[INPUT]
#  Name       opentelemetry
#  Listen     0.0.0.0
#  Port       4318
#  Tag        otel

[INPUT]
  Name  event_type
  Type  traces
  Tag   otel
...

Send a request to generate a span

$ curl http://localhost:10000/echo
OK

Check that the span is correctly forward from fluent-bit to otel-collector by looking into their logs

$ docker ps
$ docker logs <container_id>
...
|-------------------- RESOURCE SPAN --------------------|
  resource:
     - attributes:
            - service.name: 'Fluent Bit Test Service'
     - dropped_attributes_count: 5
  schema_url: https://ctraces/resource_span_schema_url
  [scope_span]
    instrumentation scope:
        - name                    : ctrace
        - version                 : a.b.c
        - dropped_attributes_count: 3
        - attributes: undefined
    schema_url: https://ctraces/scope_span_schema_url
    [spans]
         [span 'main']
             - trace_id                : 526274528d3beab4f98a82f459bdc77c
             - span_id                 : 1442ef1b690cd8b0
             - parent_span_id          : undefined
             - kind                    : 1 (internal)
             - start_time              : 1726512123955380387
             - end_time                : 1726512123955380387
             - dropped_attributes_count: 0
             - dropped_events_count    : 0
             - status:
                 - code        : 0
             - attributes:
                 - agent: 'Fluent Bit'
                 - year: 2022
                 - open_source: true
                 - temperature: 25.5
                 - my_array: [
                     'first',
                     2,
                     false,
                     [
                         3.1000000000000001,
                         5.2000000000000002,
                         6.2999999999999998
                     ]
                 ]
                 - my-list:
                     - language: 'c'

             - events:
                 - name: connect to remote server
                     - timestamp               : 1726512123955392720
                     - dropped_attributes_count: 0
                     - attributes:
                         - syscall 1: 'open()'
                         - syscall 2: 'connect()'
                         - syscall 3: 'write()'
             - [links]
         [span 'do-work']
             - trace_id                : 526274528d3beab4f98a82f459bdc77c
             - span_id                 : a58aca2253f4b054
             - parent_span_id          : 1442ef1b690cd8b0
             - kind                    : 3 (client)
             - start_time              : 1726512123955397137
             - end_time                : 1726512123955397137
             - dropped_attributes_count: 0
             - dropped_events_count    : 0
             - status:
                 - code        : 0
             - attributes: none
             - events: none
             - [links]
                 - link:
                     - trace_id             : 41354bad670f86e7a9fe6077a7ae3a4c
                     - span_id              : 820d8bab3a51c548
                     - trace_state          : aaabbbccc
                     - dropped_events_count : 2
                     - attributes           : none
...

Expected behaviour

Fluent-bit does not crash when receiving empty spans from Envoy, or allow a way to filter them.

Your Environment

Version used:
- fluent-bit: 3.1.7
- envoy: v1.31.1
- otel-collector: 0.109.0
Environment name and version: Docker (27.0.3)
Operating System and version: MacOS (14.6.1)
Filters and plugins: No additional

The text was updated successfully, but these errors were encountered:

edsiper · 2024-09-26T20:48:45Z

fixed in #9430

agup006 · 2025-02-17T14:28:10Z

Regression: in 3.2.6 with Envoy 1.30.16

agup006 · 2025-02-17T14:39:23Z

Repro for below

$ cat docker-compose.yaml
services:
  envoy:
    image: envoyproxy/envoy:distroless-v1.30.9
    volumes:
    - ./envoy.yaml:/etc/envoy/envoy.yaml
    ports:
    - "10000:10000"
    - "9901:9901"

  fluent-bit:
    image: fluent/fluent-bit:3.2.6
    container_name: fluent-bit
    volumes:
    - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf
    ports:
    - "4318:4318"

  otel-collector:
    image: otel/opentelemetry-collector:0.110.0-arm64
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
    - ./otel-collector.yaml:/etc/otel-collector-config.yaml

$ cat otel-collector.yaml
receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [logging]

$ cat fluent-bit.conf
[SERVICE]
  Flush        1
  Log_Level    debug
  Daemon       off

[INPUT]
  Name       opentelemetry
  Listen     0.0.0.0
  Port       4318
  Tag        otel

#[INPUT]
#  Name  event_type
#  Type  traces
#  Tag   otel

[OUTPUT]
  Name  stdout
  Match otel

[OUTPUT]
  Name       opentelemetry
  Match      otel
  Host       otel-collector
  Port       4318

$ cat envoy.yaml
admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    traffic_direction: OUTBOUND
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          tracing:
            spawn_upstream_span: true
            verbose: false
            provider:
              name: envoy.tracers.opentelemetry
              typed_config:
                "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
                grpc_service:
                  envoy_grpc:
                    cluster_name: fluentbit_agent
                  timeout: 2s
                service_name: front-envoy
            client_sampling:
              value: 100
            random_sampling:
              value: 100
            overall_sampling:
              value: 100
          codec_type: AUTO
          stat_prefix: ingress_http
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: proxy_routes
            virtual_hosts:
            - name: proxy
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/echo"
                direct_response:
                  status: 200
                  body:
                    inline_string: "OK"
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: "/dev/stdout"
              log_format:
                json_format:
                  traceparent: "%REQ(TRACEPARENT)%"
                  tracestate: "%REQ(TRACESTATE)%"
  clusters:
  - name: fluentbit_agent
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    load_assignment:
      cluster_name: fluentbit_agent
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: fluent-bit
                port_value: 4318

$ docker compose up --build -d
[+] Running 4/4
 ✔ Network fluent-bit-crash_default             Created                                                                                                0.0s
 ✔ Container fluent-bit-crash-otel-collector-1  Started                                                                                                0.2s
 ✔ Container fluent-bit                         Started                                                                                                0.2s
 ✔ Container fluent-bit-crash-envoy-1           Started                                                                                                0.2s

$ curl http://localhost:10000/echo
OK
$ curl http://localhost:10000/echo
OK
$ curl http://localhost:10000/echo
OK
$ curl http://localhost:10000/echo
OK

$ docker logs 643d7ceefcc5 -f
Fluent Bit v3.2.6
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io/

______ _                  _    ______ _ _           _____  _____
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2025/02/17 11:30:37] [ info] Configuration:
[2025/02/17 11:30:37] [ info]  flush time     | 1.000000 seconds
[2025/02/17 11:30:37] [ info]  grace          | 5 seconds
[2025/02/17 11:30:37] [ info]  daemon         | 0
[2025/02/17 11:30:37] [ info] ___________
[2025/02/17 11:30:37] [ info]  inputs:
[2025/02/17 11:30:37] [ info]      opentelemetry
[2025/02/17 11:30:37] [ info] ___________
[2025/02/17 11:30:37] [ info]  filters:
[2025/02/17 11:30:37] [ info] ___________
[2025/02/17 11:30:37] [ info]  outputs:
[2025/02/17 11:30:37] [ info]      stdout.0
[2025/02/17 11:30:37] [ info]      opentelemetry.1
[2025/02/17 11:30:37] [ info] ___________
[2025/02/17 11:30:37] [ info]  collectors:
[2025/02/17 11:30:37] [ info] [fluent bit] version=3.2.6, commit=66256ae467, pid=1
[2025/02/17 11:30:37] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2025/02/17 11:30:37] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/02/17 11:30:37] [ info] [simd    ] disabled
[2025/02/17 11:30:37] [ info] [cmetrics] version=0.9.9
[2025/02/17 11:30:37] [ info] [ctraces ] version=0.5.7
[2025/02/17 11:30:37] [ info] [input:opentelemetry:opentelemetry.0] initializing
[2025/02/17 11:30:37] [ info] [input:opentelemetry:opentelemetry.0] storage_strategy='memory' (memory only)
[2025/02/17 11:30:37] [ info] [output:stdout:stdout.0] worker #0 started
[2025/02/17 11:30:37] [debug] [opentelemetry:opentelemetry.0] created event channels: read=25 write=26
[2025/02/17 11:30:37] [debug] [downstream] listening on 0.0.0.0:4318
[2025/02/17 11:30:37] [ info] [input:opentelemetry:opentelemetry.0] listening on 0.0.0.0:4318
[2025/02/17 11:30:37] [debug] [stdout:stdout.0] created event channels: read=28 write=29
[2025/02/17 11:30:37] [debug] [opentelemetry:opentelemetry.1] created event channels: read=37 write=38
[2025/02/17 11:30:37] [debug] [router] match rule opentelemetry.0:stdout.0
[2025/02/17 11:30:37] [debug] [router] match rule opentelemetry.0:opentelemetry.1
[2025/02/17 11:30:37] [ info] [sp] stream processor started
[2025/02/17 11:31:03] [debug] [task] created task=0xffff946396e0 id=0 without routes, dropping.
[2025/02/17 11:31:03] [debug] [task] destroy task=0xffff946396e0 (task_id=0)
[2025/02/17 11:31:08] [debug] [task] created task=0xffff946395a0 id=0 without routes, dropping.
[2025/02/17 11:31:08] [debug] [task] destroy task=0xffff946395a0 (task_id=0)
[2025/02/17 11:31:13] [debug] [task] created task=0xffff94639640 id=0 without routes, dropping.
[2025/02/17 11:31:13] [debug] [task] destroy task=0xffff94639640 (task_id=0)

lecaros · 2025-03-12T20:04:41Z

hi @carlos4ndre
This doesn't reproduce with 3.2.8 (current latest version). If you're still able to reproduce, could you please share your conf?
Thanks

carlos4ndre · 2025-03-13T08:23:17Z

@lecaros Thank you for your help.

I've tested with the latest version of fluent-bit 3.2.8 but unfortunately still not receiving any traces on the otel collector, all seems to point out that traces are being dropped by fluent-bit.

Fluent-bit logs:

Fluent Bit v3.2.8
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

______ _                  _    ______ _ _           _____  _____
|  ___| |                | |   | ___ (_) |         |____ |/ __  \
| |_  | |_   _  ___ _ __ | |_  | |_/ /_| |_  __   __   / /`' / /'
|  _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / /   \ \  / /
| |   | | |_| |  __/ | | | |_  | |_/ / | |_   \ V /.___/ /./ /___
\_|   |_|\__,_|\___|_| |_|\__| \____/|_|\__|   \_/ \____(_)_____/


[2025/03/13 09:05:47] [ info] Configuration:
[2025/03/13 09:05:47] [ info]  flush time     | 1.000000 seconds
[2025/03/13 09:05:47] [ info]  grace          | 5 seconds
[2025/03/13 09:05:47] [ info]  daemon         | 0
[2025/03/13 09:05:47] [ info] ___________
[2025/03/13 09:05:47] [ info]  inputs:
[2025/03/13 09:05:47] [ info]      opentelemetry
[2025/03/13 09:05:47] [ info] ___________
[2025/03/13 09:05:47] [ info]  filters:
[2025/03/13 09:05:47] [ info] ___________
[2025/03/13 09:05:47] [ info]  outputs:
[2025/03/13 09:05:47] [ info]      stdout.0
[2025/03/13 09:05:47] [ info]      opentelemetry.1
[2025/03/13 09:05:47] [ info] ___________
[2025/03/13 09:05:47] [ info]  collectors:
[2025/03/13 09:05:47] [ info] [fluent bit] version=3.2.8, commit=d13e8e4ab2, pid=1
[2025/03/13 09:05:47] [debug] [engine] coroutine stack size: 196608 bytes (192.0K)
[2025/03/13 09:05:47] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/03/13 09:05:47] [ info] [simd    ] disabled
[2025/03/13 09:05:47] [ info] [cmetrics] version=0.9.9
[2025/03/13 09:05:47] [ info] [ctraces ] version=0.6.0
[2025/03/13 09:05:47] [ info] [input:opentelemetry:opentelemetry.0] initializing
[2025/03/13 09:05:47] [ info] [input:opentelemetry:opentelemetry.0] storage_strategy='memory' (memory only)
[2025/03/13 09:05:47] [debug] [opentelemetry:opentelemetry.0] created event channels: read=25 write=26
[2025/03/13 09:05:47] [debug] [downstream] listening on 0.0.0.0:4318
[2025/03/13 09:05:47] [ info] [input:opentelemetry:opentelemetry.0] listening on 0.0.0.0:4318
[2025/03/13 09:05:47] [debug] [stdout:stdout.0] created event channels: read=28 write=29
[2025/03/13 09:05:47] [debug] [opentelemetry:opentelemetry.1] created event channels: read=37 write=38
[2025/03/13 09:05:47] [ info] [output:stdout:stdout.0] worker #0 started
[2025/03/13 09:05:47] [debug] [router] match rule opentelemetry.0:stdout.0
[2025/03/13 09:05:47] [debug] [router] match rule opentelemetry.0:opentelemetry.1
[2025/03/13 09:05:47] [ info] [sp] stream processor started
[2025/03/13 09:07:27] [debug] [task] created task=0xffff81c396e0 id=0 without routes, dropping.
[2025/03/13 09:07:27] [debug] [task] destroy task=0xffff81c396e0 (task_id=0)
[2025/03/13 09:07:32] [debug] [task] created task=0xffff81c395a0 id=0 without routes, dropping.
[2025/03/13 09:07:32] [debug] [task] destroy task=0xffff81c395a0 (task_id=0)
[2025/03/13 09:07:35] [debug] [task] created task=0xffff81c39640 id=0 without routes, dropping.
[2025/03/13 09:07:35] [debug] [task] destroy task=0xffff81c39640 (task_id=0)
[2025/03/13 09:07:37] [debug] [task] created task=0xffff81c395a0 id=0 without routes, dropping.
[2025/03/13 09:07:37] [debug] [task] destroy task=0xffff81c395a0 (task_id=0)
[2025/03/13 09:07:38] [debug] [task] created task=0xffff81c396e0 id=0 without routes, dropping.
[2025/03/13 09:07:38] [debug] [task] destroy task=0xffff81c396e0 (task_id=0)

OTel collector logs:

2025-03-13T09:05:47.171Z	info	[email protected]/service.go:193	Setting up own telemetry...
2025-03-13T09:05:47.171Z	info	builders/builders.go:26	Development component. May change in the future.	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}
2025-03-13T09:05:47.172Z	info	[email protected]/service.go:258	Starting otelcol...	{"Version": "0.121.0", "NumCPU": 10}
2025-03-13T09:05:47.172Z	info	extensions/extensions.go:40	Starting extensions...
2025-03-13T09:05:47.172Z	info	[email protected]/otlp.go:173	Starting HTTP server	{"otelcol.component.id": "otlp", "otelcol.component.kind": "Receiver", "endpoint": "0.0.0.0:4318"}
2025-03-13T09:05:47.172Z	info	[email protected]/service.go:281	Everything is ready. Begin running and processing data.

docker-compose.yaml

services:
  envoy:
    image: envoyproxy/envoy:distroless-v1.30.9
    volumes:
    - ./envoy.yaml:/etc/envoy/envoy.yaml
    ports:
    - "10000:10000"
    - "9901:9901"

  fluent-bit:
    image: fluent/fluent-bit:3.2.8 # 3.1.10 (works)
    container_name: fluent-bit
    volumes:
    - ./fluent-bit.conf:/fluent-bit/etc/fluent-bit.conf

  otel-collector:
    image: otel/opentelemetry-collector:0.121.0-arm64
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
    - ./otel-collector.yaml:/etc/otel-collector-config.yaml

fluent-bit.conf

[SERVICE]
  Flush        1
  Log_Level    debug
  Daemon       off

[INPUT]
  Name       opentelemetry
  Listen     0.0.0.0
  Port       4318
  Tag        otel

#[INPUT]
#  Name  event_type
#  Type  traces
#  Tag   otel

[OUTPUT]
  Name  stdout
  Match otel

[OUTPUT]
  Name       opentelemetry
  Match      otel
  Host       otel-collector
  Port       4318

otel-collector.yaml

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

exporters:
  debug:
    verbosity: detailed

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [debug]

envoy.yaml

admin:
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

static_resources:
  listeners:
  - address:
      socket_address:
        address: 0.0.0.0
        port_value: 10000
    traffic_direction: OUTBOUND
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          tracing:
            spawn_upstream_span: true
            verbose: false
            provider:
              name: envoy.tracers.opentelemetry
              typed_config:
                "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
                grpc_service:
                  envoy_grpc:
                    cluster_name: fluentbit_agent
                  timeout: 2s
                service_name: front-envoy
            client_sampling:
              value: 100
            random_sampling:
              value: 100
            overall_sampling:
              value: 100
          codec_type: AUTO
          stat_prefix: ingress_http
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
          route_config:
            name: proxy_routes
            virtual_hosts:
            - name: proxy
              domains:
              - "*"
              routes:
              - match:
                  prefix: "/echo"
                direct_response:
                  status: 200
                  body:
                    inline_string: "OK"
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: "/dev/stdout"
              log_format:
                json_format:
                  traceparent: "%REQ(TRACEPARENT)%"
                  tracestate: "%REQ(TRACESTATE)%"
  clusters:
  - name: fluentbit_agent
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    typed_extension_protocol_options:
      envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
        "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
        explicit_http_config:
          http2_protocol_options: {}
    load_assignment:
      cluster_name: fluentbit_agent
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: fluent-bit
                port_value: 4318

Run docker compose up --build -d and trigger traces with curl http://localhost:10000/echo, random sampling is set at 100%.

Other notes, also tried with the latest version of envoy (v1.33.0) but achieved the same results.

lecaros · 2025-03-14T21:33:38Z

Hi @carlos4ndre,
from your config I assume you expect the data to be matched by the tag (tag otel). For this to work, you must set tag_from_uri to false. Then, the value from tag will be used for the ingested data.
I think the docs weren't clear so I've submitted a PR to improve that.

carlos4ndre · 2025-03-15T12:40:54Z

@lecaros Thanks! that was the issue, setting tag_from_uri to false did the trick:

[SERVICE]
  Flush     1
  Log_Level debug
  Daemon    off

[INPUT]
  Name         opentelemetry
  Listen       0.0.0.0
  Port         4318
  Tag_from_uri false
  Tag          otel

[OUTPUT]
  Name  stdout
  Match otel

[OUTPUT]
  Name  opentelemetry
  Match otel
  Host  otel-collector
  Port  4318

Traces are properly processed and forwarded by fluent-bit:

[2025/03/15 12:32:36] [debug] [task] created task=0xffff814396e0 id=0 OK
[2025/03/15 12:32:36] [debug] [output:stdout:stdout.0] task_id=0 assigned to thread #0
[2025/03/15 12:32:36] [debug] [output:opentelemetry:opentelemetry.1] ctraces msgpack size: 708
[2025/03/15 12:32:36] [debug] [output:opentelemetry:opentelemetry.1] final payload size: 499
[2025/03/15 12:32:36] [debug] [output:stdout:stdout.0] ctr decode msgpack returned : 6
[2025/03/15 12:32:36] [debug] [out flush] cb_destroy coro_id=0
|-------------------- RESOURCE SPAN --------------------|
  resource:
     - attributes:
            - service.name: 'front-envoy'
     - dropped_attributes_count: 0
  schema_url:
  [scope_span]
    schema_url:
    [spans]
         [span 'egress localhost:10000']
             - trace_id                : ef25e55fa0a291be8de61012611faee3
             - span_id                 : cdbe42ce62f4da23
             - parent_span_id          : undefined
             - kind                    : 2 (server)
             - start_time              : 1742041953778219000
             - end_time                : 1742041953778505000
             - dropped_attributes_count: 0
             - dropped_events_count    : 0
             - status:
                 - code        : 0
             - attributes:
                 - node_id: ''
                 - zone: ''
                 - guid:x-request-id: 'c7c28ab5-cc5c-999b-9651-017115e1cf28'
                 - http.url: 'http://localhost:10000/echo'
                 - http.method: 'GET'
                 - downstream_cluster: '-'
                 - user_agent: 'curl/7.84.0'
                 - http.protocol: 'HTTP/1.1'
                 - peer.address: '172.18.0.1'
                 - request_size: '0'
                 - response_size: '2'
                 - component: 'proxy'
                 - http.status_code: '200'
                 - response_flags: '-'
             - events: none
             - [links]
[2025/03/15 12:32:36] [debug] [upstream] KA connection #51 to otel-collector:4318 is connected
[2025/03/15 12:32:36] [ info] [output:opentelemetry:opentelemetry.1] otel-collector:4318, HTTP status=200

Traces are successfully delivered to the otel collector:

2025-03-15T12:32:36.047Z	info	ResourceSpans #0
Resource SchemaURL:
Resource attributes:
     -> service.name: Str(front-envoy)
ScopeSpans #0
ScopeSpans SchemaURL:
InstrumentationScope
Span #0
    Trace ID       : ef25e55fa0a291be8de61012611faee3
    Parent ID      :
    ID             : cdbe42ce62f4da23
    Name           : egress localhost:10000
    Kind           : Server
    Start time     : 2025-03-15 12:32:33.778219 +0000 UTC
    End time       : 2025-03-15 12:32:33.778505 +0000 UTC
    Status code    : Unset
    Status message :
Attributes:
     -> node_id: Str()
     -> zone: Str()
     -> guid:x-request-id: Str(c7c28ab5-cc5c-999b-9651-017115e1cf28)
     -> http.url: Str(http://localhost:10000/echo)
     -> http.method: Str(GET)
     -> downstream_cluster: Str(-)
     -> user_agent: Str(curl/7.84.0)
     -> http.protocol: Str(HTTP/1.1)
     -> peer.address: Str(172.18.0.1)
     -> request_size: Str(0)
     -> response_size: Str(2)
     -> component: Str(proxy)
     -> http.status_code: Str(200)
     -> response_flags: Str(-)
	{"otelcol.component.id": "debug", "otelcol.component.kind": "Exporter", "otelcol.signal": "traces"}

However it is important to note that I've had to downgrade fluent-bit to 3.2.6 for this to work, this issue was already reported in #10071.

carlos4ndre · 2025-03-15T12:42:31Z

Closing this issue since the bug initially reported has been fixed.

carlos4ndre added the status: waiting-for-triage label Sep 16, 2024

edsiper assigned cosmo0920 Sep 18, 2024

cosmo0920 mentioned this issue Sep 24, 2024

encode_opentelemetry: Handle minimal otel payloads fluent/ctraces#60

Merged

edsiper closed this as completed Sep 27, 2024

edsiper added this to the Fluent Bit v3.2.0 milestone Sep 27, 2024

agup006 reopened this Feb 17, 2025

carlos4ndre closed this as completed Mar 15, 2025

patrick-stephens mentioned this issue Mar 17, 2025

inputs: opentelemetry: add clarification to tag and tag_from_uri fluent/fluent-bit-docs#1587

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fluent-bit crashes when handling envoy empty spans #9391

Fluent-bit crashes when handling envoy empty spans #9391

carlos4ndre commented Sep 16, 2024 •

edited

Loading

edsiper commented Sep 26, 2024

agup006 commented Feb 17, 2025

agup006 commented Feb 17, 2025 •

edited

Loading

lecaros commented Mar 12, 2025

carlos4ndre commented Mar 13, 2025 •

edited

Loading

lecaros commented Mar 14, 2025

carlos4ndre commented Mar 15, 2025 •

edited

Loading

carlos4ndre commented Mar 15, 2025

Fluent-bit crashes when handling envoy empty spans #9391

Fluent-bit crashes when handling envoy empty spans #9391

Comments

carlos4ndre commented Sep 16, 2024 • edited Loading

Bug Report

edsiper commented Sep 26, 2024

agup006 commented Feb 17, 2025

agup006 commented Feb 17, 2025 • edited Loading

lecaros commented Mar 12, 2025

carlos4ndre commented Mar 13, 2025 • edited Loading

lecaros commented Mar 14, 2025

carlos4ndre commented Mar 15, 2025 • edited Loading

carlos4ndre commented Mar 15, 2025

carlos4ndre commented Sep 16, 2024 •

edited

Loading

agup006 commented Feb 17, 2025 •

edited

Loading

carlos4ndre commented Mar 13, 2025 •

edited

Loading

carlos4ndre commented Mar 15, 2025 •

edited

Loading