bug: record_accessor isnt working with opensearch

[deepend-dev]

Bug Report

Describe the bug

** Opensearch plugin is not working with record accessor as per [document (https://docs.fluentbit.io/manual/pipeline/outputs/opensearch) with public ecr image

It seems there is difference between fluent bit images on public ecr and dockerhub.

Issue:

Record accessor is not reading expressions and sending it as a string.
For Ex.
Index $kubernetes['namespace_name']

It will create an index in opensearch litreally called "$kubernetes['namespace_name']"

Source

Above issue comes with all fluent bit ECR images aws-observability/aws-for-fluent-bit. I have tried withlatest, 2.31.3, 2.31.2, 2.30.0, 2.21.0, 2.22.0
but it works with docker hub image fluent/fluent-bit

To Reproduce

• Set below settings and send logs to opensearch:

[SERVICE]
        Parsers_File /fluent-bit/parsers/parsers.conf
    [INPUT]
        Name              tail
        Tag               kube.*
        Path              /var/log/containers/*.log
        DB                /var/log/flb_kube.db
        Parser            docker
        Docker_Mode       On
        Mem_Buf_Limit     5MB
        Skip_Long_Lines   On
        Refresh_Interval  10
    [FILTER]
        Name                kubernetes
        Match               kube.*
        Kube_URL            https://kubernetes.default.svc.cluster.local:443
        Merge_Log           On
        Merge_Log_Key       data
        Keep_Log            On
        K8S-Logging.Parser  On
        K8S-Logging.Exclude On
        Buffer_Size         32k
    [OUTPUT]
        Name                opensearch
        Match               *
        AWS_Region          ap-southeast-1
        AWS_Auth            On
        Host                127.0.0.1
        Port                443
        tls                true
        Buffer_Size         5m
        Index               $kubernetes['namespace_name']
        Type                _doc
        Logstash_Format     Off
        Logstash_Prefix     logstash
        Logstash_DateFormat %Y.%m.%d
        Time_Key            @timestamp
        Time_Key_Format     %Y-%m-%dT%H:%M:%S
        Time_Key_Nanos      Off
        Include_Tag_Key     Off
        Tag_Key             _flb-key
        Generate_ID         Off
        Write_Operation     create
        Replace_Dots        Off
        Trace_Output        Off
        Trace_Error         Off
        Current_Time_Index  Off
        Suppress_Type_Name  On

• Example log message if applicable:

{"@timestamp":"2023-02-25T15:32:19.108Z","log":"2023-02-25T15:32:09.273701033Z stdout F {\"@timestamp\":\"2023-02-25T15:32:08.701Z\",\"log\":\"2023-02-25T15:31:59.237567594Z stdout F \\\\\\\\\\\\\\\\\\\\\\\\\\\\\\\",\\\"kubernetes\\\":{\\\"pod_name\\\":\\\"aws-for-fluent-bit-hm5vx\\\",\\\"namespace_name\\\":\\\"aws-for-fluent-bit\\\",\\\"pod_id\\\":\\\"3da2b3d7-7d5f-4ba3-bf4d-8271b106b219\\\",\\\"labels\\\":{\\\"app.kubernetes.io/instance\\\":\\\"aws-for-fluent-bit\\\",\\\"app.kubernetes.io/name\\\":\\\"aws-for-fluent-bit\\\",\\\"controller-revision-hash\\\":\\\"b8588dd9f\\\",\\\"pod-template-generation\\\":\\\"7\\\"},\\\"annotations\\\":{\\\"checksum/config\\\":\\\"4d9bbde653197b2024ed70fd46d50fe42a895b7cfdaa66da7862282d231e3a91\\\",\\\"kubectl.kubernetes.io/restartedAt\\\":\\\"2023-02-25T15:24:02Z\\\",\\\"kubernetes.io/psp\\\":\\\"eks.privileged\\\"},\\\"host\\\":\\\"ip-172-31-11-123.ap-southeast-1.compute.internal\\\",\\\"container_name\\\":\\\"aws-for-fluent-bit\\\",\\\"docker_id\\\":\\\"68f58046c64c598285d7418cddc348aa1b2a1dce9c696ebf6b64ebdcc1239883\\\",\\\"container_image\\\":\\\"public.ecr.aws/aws-observability/aws-for-fluent-bit:latest\\\"}}\",\"kubernetes\":{\"pod_name\":\"aws-for-fluent-bit-hm5vx\",\"namespace_name\":\"aws-for-fluent-bit\",\"pod_id\":\"3da2b3d7-7d5f-4ba3-bf4d-8271b106b219\",\"labels\":{\"app.kubernetes.io/instance\":\"aws-for-fluent-bit\",\"app.kubernetes.io/name\":\"aws-for-fluent-bit\",\"controller-revision-hash\":\"b8588dd9f\",\"pod-template-generation\":\"7\"},\"annotations\":{\"checksum/config\":\"4d9bbde653197b2024ed70fd46d50fe42a895b7cfdaa66da7862282d231e3a91\",\"kubectl.kubernetes.io/restartedAt\":\"2023-02-25T15:24:02Z\",\"kubernetes.io/psp\":\"eks.privileged\"},\"host\":\"ip-172-31-11-123.ap-southeast-1.compute.internal\",\"container_name\":\"aws-for-fluent-bit\",\"docker_id\":\"68f58046c64c598285d7418cddc348aa1b2a1dce9c696ebf6b64ebdcc1239883\",\"container_image\":\"public.ecr.aws/aws-observability/aws-for-fluent-bit:latest\"}}","kubernetes":{"pod_name":"aws-for-fluent-bit-hm5vx","namespace_name":"aws-for-fluent-bit","pod_id":"3da2b3d7-7d5f-4ba3-bf4d-8271b106b219","labels":{"app.kubernetes.io/instance":"aws-for-fluent-bit","app.kubernetes.io/name":"aws-for-fluent-bit","controller-revision-hash":"b8588dd9f","pod-template-generation":"7"},"annotations":{"checksum/config":"4d9bbde653197b2024ed70fd46d50fe42a895b7cfdaa66da7862282d231e3a91","kubectl.kubernetes.io/restartedAt":"2023-02-25T15:24:02Z","kubernetes.io/psp":"eks.privileged"},"host":"ip-172-31-11-123.ap-southeast-1.compute.internal","container_name":"aws-for-fluent-bit","docker_id":"68f58046c64c598285d7418cddc348aa1b2a1dce9c696ebf6b64ebdcc1239883","container_image":"public.ecr.aws/aws-observability/aws-for-fluent-bit:latest"}}
76
{"create":{"_index":"$kubernetes['namespace_name']"}}

• Steps to reproduce the problem:

Expected behavior

For above example
Read $kubernetes['namespace_name'] and create index named aws-for-fluent-bit

Screenshots

Your Environment

• Version used: fluent bit 2.31.2 with opensearch 2.3
• Configuration:
• Environment name and version (e.g. Kubernetes? What version?): EKS
• Server type and version:
• Operating System and version:
• Filters and plugins:

Additional context

[nokute78]

Cc @PettitWesley
I think v2.31.3 doesn't support record accessor feature.

https://github.com/aws/aws-for-fluent-bit/releases/tag/v2.31.3
This version contains Fluent-bit v1.9.10.

Record accessor feature was supported from v2.0.5.
https://docs.fluentbit.io/manual/pipeline/outputs/opensearch

[deepend-dev]

@nokute78 thanks for pointing it out. Somehow I missed it.

I was looking for this mapping but couldn't find it.

Great to know we capture them in release notes.

I just assumed that fluentbit latest versions map with aws-fluent-bit latest versions.

Good thing is that we now know the possible solution.