feat: outdated aws-for-fluent-bit chart updated to use new high performance cloudwatch_logs plugin

[deepend-dev]

This PR is raised in extension of #903 which got closed because of some conflicting changes.

Issue

Resolves below issue/s:
#903
#901

Also can be added as resolver of:
#304
#719
#436
#671

Current fluent bit helm chart is outdated.

Aligning with recommendations from current repo to use New Higher Performance Core Fluent Bit Plugin.(https://github.com/fluent/fluent-bit/tree/master/plugins/out_cloudwatch_logs) is directly integrated into fluent bit.

It can achieve higher throughput and will consume less CPU and memory. The [new cloudwatch plugin].
It also provides features to send metrics to cloudwatch with cloudwatch metric namespaces and dimesions.
Has better credential management with sts_endpoint.

Description of changes

• Added cloudwatch_logs plugin parameters
• Updated fluent-bit image to latest stable 2.28.4
• Added support for all new cloudwatch_logs features.
• Updated readme to reflect the change.

Checklist

[ x ] Added/modified documentation as required (such as the README.md for modified charts)
[ x ] Incremented the chart version in Chart.yaml for the modified chart(s)
[ x ] Manually tested. Describe what testing was done in the testing section below
[ x ] Make sure the title of the PR is a good description that can go into the release notes

Testing

added as part of argocd addon repository deployed with eks terraform blueprint. Here's the addon repo

configfile is correctly reflecting as per provided config

Helm chart deployed successfully
Chart version: 0.1.24

All pods up:

Testing template feature:
Added below config to generate a new loggroup per namespace and new stream per app

    enable: true
    logGroupName: /eks/newchart/workload-logs/fallback
    logStreamPrefix: fallback-
    logGroupTemplate: /eks/newchart/workload-logs/$kubernetes['namespace_name']
    logStreamTemplate: $kubernetes['pod_name'].$kubernetes['container_name']

Namespace isolated logs being generated:

Application stream:

Additional features added

Support for S3

Support for opensearch

Sample data

Known issue

doesn't impact our implementation or this chart as its an issue with fluent bit itself

I noticed an issue that opensearch plugin doesnt work with record accessor, although it is mentioned in fluent-bit docs that it should.

I tried multiple combinations but it just doesnt seem to work. Raised an issue on fluentbit repo - fluent/fluent-bit#6914 . Have you seen this behaviour before?

Here's a setting example

[OUTPUT]
        Name                opensearch
        Match               *
        AWS_Region          ap-southeast-1
        AWS_Auth            On
        Host                search-fluent-bit-logging-g6lh44s35e7ytj5vh2yrsds7qa.ap-southeast-1.es.amazonaws.com
        Port                443
        tls                true
        Buffer_Size         5m
        Index               $kubernetes['namespace_name']
        Type                _doc
        Logstash_Format     Off
        Logstash_Prefix     logstash
        Logstash_DateFormat %Y.%m.%d
        Time_Key            @timestamp
        Time_Key_Format     %Y-%m-%dT%H:%M:%S
        Time_Key_Nanos      Off
        Include_Tag_Key     Off
        Tag_Key             _flb-key
        Generate_ID         Off
        Write_Operation     create
        Replace_Dots        Off
        Trace_Output        On
        Trace_Error         Off
        Current_Time_Index  Off
        Suppress_Type_Name  On

Here's fluent bit log showing that index api call is not correct. Its literally sending in $kubernetes['namespace_name'] as string without resolving :

FYI I have raised below issues:
fluent bit : fluent/fluent-bit#6914
aws-for-fluent-bit: aws/aws-for-fluent-bit#557

Solution:
As per the notes on the issue, it is happening because latest public ecr image 2.31.3 is created with fluentbit 1.9.10 while the record accessor feature is introduced with fluent-bit 2.0.5. Current docker hub image for fluentbit is at 2.0.9 hence it works with that image.
Having said that we now know that the issue will resolve with later ecr image releases which will have base fluent bit image > 2.0.5

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

[dnaprawa-capgemini]

Looking forward to merge this PR.

[deepend-dev]

Test: Trying to reopen original PR