add ecs support #228

kaisecheng · 2021-04-27T08:53:52Z

This PR adds ECS support and maps the following

Legacy	ECS
cloudfront_fields	[@metadata][s3][cloudfront][fields]
cloudfront_version	[@metadata][s3][cloudfront][version]

Closed: #196

karenzone

Doc review: Made some suggestions in line for your consideration. Please give me a chance to re-review so that we don't break the doc build.

Do you have an example that would help clarify this info for users?

karenzone · 2021-04-27T18:16:11Z

docs/index.asciidoc

+==== Event Metadata and the Elastic Common Schema (ECS)
+This plugin adds cloudfront metadata to event.
+By default, the value is stored in the root level.
+When ECS is enabled, it is stored in the `@metadata`.


Suggested change

When ECS is enabled, it is stored in the `@metadata`.

When ECS is enabled, the value is stored in the `@metadata`.

karenzone · 2021-04-27T18:17:30Z

docs/index.asciidoc

+|=======================================================================
+| disabled | v1 | Availability | Description
+
+| cloudfront_fields | [@metadata][s3][cloudfront][fields] | available when the file is a cloudfront metadata | columns name of metadata


I don't understand this wording. Maybe "When the file contains cloudfront metadata?"
If I don't have the wording right, please let me know. We can work together to find the right wording.

my word metadata is not accurate. It should be a log file specific to CloudFront, which records the access details.
cloudfront_fields and cloudfront_version are only available for CloudFront log.
So, may be change it to "available when the file is a CloudFront log"

karenzone · 2021-04-27T18:17:57Z

docs/index.asciidoc

+| disabled | v1 | Availability | Description
+
+| cloudfront_fields | [@metadata][s3][cloudfront][fields] | available when the file is a cloudfront metadata | columns name of metadata
+| cloudfront_version | [@metadata][s3][cloudfront][version] | available when the file is a cloudfront metadata | version of metadata


Same comment as previous

karenzone · 2021-04-27T18:26:29Z

docs/index.asciidoc

+* Default value depends on which version of Logstash is running:
+** When Logstash provides a `pipeline.ecs_compatibility` setting, its value is used as the default
+** Otherwise, the default value is `disabled`.
+


Suggested change

Controls this plugin's compatibility with the

{ecs-ref}[Elastic Common Schema (ECS)].

See <<plugins-{type}s-{plugin}-ecs_metadata,ECS mapping>> for detailed information.

karenzone · 2021-04-27T18:31:53Z

docs/index.asciidoc

@@ -31,6 +31,20 @@ Files ending in `.gz` are handled as gzip'ed files.

 Files that are archived to AWS Glacier will be skipped.

+==== Event Metadata and the Elastic Common Schema (ECS)


Suggested change

==== Event Metadata and the Elastic Common Schema (ECS)

[id="plugins-{type}s-{plugin}-ecs_compatibility"]

==== Event Metadata and the Elastic Common Schema (ECS)

yaauie

Changes in code and specs look good.

I've left a few suggestions in docs, and we should probably get 👀 from @karenzone before we merge.

docs/index.asciidoc

yaauie · 2021-04-27T18:27:37Z

docs/index.asciidoc

+Here’s how ECS compatibility mode affects output.
+[cols="<l,<l,e,<e"]
+|=======================================================================
+| disabled | v1 | Availability | Description


yaauie · 2021-04-27T18:27:54Z

docs/index.asciidoc

+|=======================================================================
+| disabled | v1 | Availability | Description
+
+| cloudfront_fields | [@metadata][s3][cloudfront][fields] | available when the file is a cloudfront metadata | columns name of metadata


docs/index.asciidoc

yaauie · 2021-04-27T18:29:10Z

spec/inputs/s3_spec.rb

+        ecs_compatibility_matrix(:disabled, :v1) do |ecs_select|
+          before(:each) do
+            allow_any_instance_of(described_class).to receive(:ecs_compatibility).and_return(ecs_compatibility)
+            # subject.register


Suggested change

# subject.register

docs/index.asciidoc

kaisecheng · 2021-04-28T11:56:29Z

Doc review: Made some suggestions in line for your consideration. Please give me a chance to re-review so that we don't break the doc build.

Do you have an example that would help clarify this info for users?

the cloudfront related fields store in invisible [@metadata] in ECS v1. The data exists but just not visible. I am afraid the example with missing fields causes more confusion. Do you think I should show the example? @karenzone

karenzone · 2021-04-28T12:20:18Z

Regarding:

the cloudfront related fields store in invisible [@metadata] in ECS v1. The data exists but just not visible. I am afraid the example with missing fields causes more confusion. Do you think I should show the example? @karenzone

@kaisecheng Using examples to illustrate concepts is one of many areas where you excel. If you believe that an example would add more confusion, then I agree that we shouldn't add one.

karenzone

@kaisecheng I suggested two more fixes inline. One was the result of my error in suggesting a heading anchor that was already in use.

docs/index.asciidoc

Co-authored-by: Karen Metts <[email protected]>

karenzone

LGTM 🎉

fix #228

add ecs support

Unverified

This user has not yet uploaded their public signing key.

GPG key ID: 836031A2C9F15C08

Learn about vigilant mode

2c3606e

roaksoax added the status:work-in-progress label Apr 27, 2021

ecs doc

Unverified

This user has not yet uploaded their public signing key.

GPG key ID: 836031A2C9F15C08

Learn about vigilant mode

af85d98

kaisecheng assigned elasticsearch-bot Apr 27, 2021

kaisecheng marked this pull request as ready for review April 27, 2021 14:05

roaksoax added status:needs-review and removed status:work-in-progress labels Apr 27, 2021

yaauie assigned yaauie and unassigned elasticsearch-bot Apr 27, 2021

karenzone reviewed Apr 27, 2021

View reviewed changes

yaauie reviewed Apr 27, 2021

View reviewed changes

update doc

bfefd91

kaisecheng requested a review from karenzone April 29, 2021 12:45

karenzone reviewed Apr 30, 2021

View reviewed changes

docs/index.asciidoc Outdated Show resolved Hide resolved

docs/index.asciidoc Outdated Show resolved Hide resolved

kaisecheng and others added 2 commits May 3, 2021 10:56

Update docs/index.asciidoc

fab8f22

Co-authored-by: Karen Metts <[email protected]>

Update docs/index.asciidoc

4ee1378

Co-authored-by: Karen Metts <[email protected]>

kaisecheng requested a review from karenzone May 3, 2021 09:02

karenzone approved these changes May 3, 2021

View reviewed changes

kaisecheng merged commit 65f156c into logstash-plugins:master May 3, 2021

kaisecheng mentioned this pull request Jun 24, 2021

bump version to 3.7.0 #230

Merged

kaisecheng added a commit that referenced this pull request Jun 24, 2021

bump version to 3.7.0 (#230)

0ce8625

fix #228

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

add ecs support #228

add ecs support #228

kaisecheng commented Apr 27, 2021

karenzone left a comment

karenzone Apr 27, 2021

karenzone Apr 27, 2021

karenzone Apr 27, 2021

kaisecheng Apr 28, 2021

karenzone Apr 27, 2021

karenzone Apr 27, 2021

karenzone Apr 27, 2021

karenzone Apr 27, 2021

yaauie left a comment

yaauie Apr 27, 2021

yaauie Apr 27, 2021

yaauie Apr 27, 2021

kaisecheng commented Apr 28, 2021 •

edited

Loading

karenzone commented Apr 28, 2021 •

edited

Loading

karenzone left a comment •

edited

Loading

karenzone left a comment

	When ECS is enabled, it is stored in the `@metadata`.
	When ECS is enabled, the value is stored in the `@metadata`.

	\| cloudfront_fields \| [@metadata][s3][cloudfront][fields] \| available when the file is a cloudfront metadata \| columns name of metadata
	\| cloudfront_fields \| [@metadata][s3][cloudfront][fields] \| available when the file contains cloudfront metadata \| columns name of metadata

	\| cloudfront_version \| [@metadata][s3][cloudfront][version] \| available when the file is a cloudfront metadata \| version of metadata
	\| cloudfront_version \| [@metadata][s3][cloudfront][version] \| available when the file contains cloudfront metadata \| version of metadata

+Controls this plugin's compatibility with the
+{ecs-ref}[Elastic Common Schema (ECS)].
+See <<plugins-{type}s-{plugin}-ecs_metadata,ECS mapping>> for detailed information.

		@@ -31,6 +31,20 @@ Files ending in `.gz` are handled as gzip'ed files.

		Files that are archived to AWS Glacier will be skipped.

		==== Event Metadata and the Elastic Common Schema (ECS)

	==== Event Metadata and the Elastic Common Schema (ECS)
	[id="plugins-{type}s-{plugin}-ecs_compatibility"]
	==== Event Metadata and the Elastic Common Schema (ECS)

	\| disabled \| v1 \| Availability \| Description
	\| ECS disabled \| ECS v1 \| Availability \| Description

add ecs support #228

add ecs support #228

Conversation

kaisecheng commented Apr 27, 2021

karenzone left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yaauie left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kaisecheng commented Apr 28, 2021 • edited Loading

karenzone commented Apr 28, 2021 • edited Loading

karenzone left a comment • edited Loading

Choose a reason for hiding this comment

karenzone left a comment

Choose a reason for hiding this comment

kaisecheng commented Apr 28, 2021 •

edited

Loading

karenzone commented Apr 28, 2021 •

edited

Loading

karenzone left a comment •

edited

Loading