libbeat: increase total_fields.limit to 12500

[AndersonQ]

Proposed commit message

libbeat: increase index template total_fields.limit to 12500

It increased the index.mapping.total_fields.limit from 10000 to 12500 in order to avoid ingestion failures caused by too many field in the index. Since 8.15.0 the limit started to be hit.
The field count being exceeded is on the index, counting all mapped fields and the dynamic fields. That's why a small event might trigger the error, the event contains new fields to be mapped which would exceed the total field limit if mapped.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Disruptive User Impact

Whereas the index.mapping.total_fields.limit might impact Elasticsearch's performance, see here, we do not anticipate any significant impact.

How to test this PR locally

check for errors

Follow the tutorial for system metrics integration: https://YOUR_CLUSTER/app/home#/tutorial/systemMetrics

• configure the elasticsearch output
• the run the following

./metricbeat modules enable system
./metricbeat setup
./metricbeat -e | grep -e 'has been exceeded while adding new fields'

• you should not see any log

check the new index settings

GET /metricbeat-9.0.0/_settings

you should get an answer like:

{
  ".ds-metricbeat-9.0.0-2024.11.14-000001": {
    "settings": {
      "index": {
        "mapping": {
          "total_fields": {
            "limit": "12500"
          }
        },

without the fix, it'd be 10000.

Related issues

• Relates Known issue 8.15.x field limit reached on Metricbeat #41614

Logs

error log before the fix:

# failed to parse: Limit of total fields [10000] has been exceeded while adding new fields [1]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Limit of total fields [10000] has been exceeded while adding new fields [1]\"

{"log.level":"warn","@timestamp":"2024-08-13T14:04:46.708Z","log.logger":"elasticsearch","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.(*Client).applyItemStatus","file.name":"elasticsearch/client.go","file.line":489},"message":"Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Meta:null, Fields:null, Private:interface {}(nil), TimeSeries:false}, Flags:0x0, Cache:publisher.EventCache{m:mapstr.M(nil)}, EncodedEvent:(*elasticsearch.encodedEvent)(0xc003933700)} (status=400): {\"type\":\"document_parsing_exception\",\"reason\":\"[1:2756] failed to parse: Limit of total fields [10000] has been exceeded while adding new fields [1]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"Limit of total fields [10000] has been exceeded while adding new fields [1]\"}}, dropping event!","service.name":"metricbeat","log.type":"event","ecs.version":"1.6.0"}

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[mergify]

backport-8.x has been added to help with the transition to the new branch 8.x.
If you don't need it please use backport-skip label and remove the backport-8.x label.

[pierrehilbert]

Looks like you need also to change

beats/libbeat/template/load_test.go

Line 173 in 367d94e

"limit": 10000,

[cmacknz]

There is a planned 8.15.5 with feature freeze on Nov 19

[cmacknz]

Added 8.15 backport label