You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
secret_id | Yes | String | The AWS secret name or ARN. |
149
150
region | No | String | The AWS region of the secret. Defaults to `us-east-1`.
150
151
sts_role_arn | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to the AWS Secrets Manager. Defaults to `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html).
151
-
refresh_interval | No | Duration | The refreshment interval for AWS secrets extension plugin to poll new secret values. Defaults to `PT1H`. See [Automatically refreshing secrets](#automatically-refreshing-secrets) for details.
152
+
refresh_interval | No | Duration | The refreshment interval for the AWS Secrets extension plugin to poll new secret values. Defaults to `PT1H`. For more information, see [Automatically refreshing secrets](#automatically-refreshing-secrets).
153
+
disable_refresh | No | Boolean | Disables regular polling on the latest secret values inside the AWS secrets extension plugin. Defaults to `false`. When set to `true`, `refresh_interval` will not be used.
The `geoip_service` extension configures all [`geoip`]({{site.url}}{{site.baseurl}}/data-prepper/pipelines/configuration/processors/geoip) processors in Data Prepper.
12
+
13
+
## Usage
14
+
15
+
You can configure the GeoIP service that Data Prepper uses for the `geoip` processor.
16
+
By default, the GeoIP service comes with the [`maxmind`](#maxmind) option configured.
17
+
18
+
The following example shows how to configure the `geoip_service` in the `data-prepper-config.yaml` file:
19
+
20
+
```
21
+
extensions:
22
+
geoip_service:
23
+
maxmind:
24
+
database_refresh_interval: PT1H
25
+
cache_count: 16_384
26
+
```
27
+
28
+
## maxmind
29
+
30
+
The GeoIP service supports the MaxMind [GeoIP and GeoLite](https://dev.maxmind.com/geoip) databases.
31
+
By default, Data Prepper will use all three of the following [MaxMind GeoLite2](https://dev.maxmind.com/geoip/geolite2-free-geolocation-data) databases:
32
+
33
+
* City
34
+
* Country
35
+
* ASN
36
+
37
+
The service also downloads databases automatically to keep Data Prepper up to date with changes from MaxMind.
38
+
39
+
You can use the following options to configure the `maxmind` extension.
40
+
41
+
Option | Required | Type | Description
42
+
:--- | :--- | :--- | :---
43
+
`databases` | No | [database](#database) | The database configuration.
44
+
`database_refresh_interval` | No | Duration | How frequently to check for updates from MaxMind. This can be any duration in the range of 15 minutes to 30 days. Default is `PT7D`.
45
+
`cache_count` | No | Integer | The maximum cache count by number of items in the cache, with a range of 100--100,000. Default is `4096`.
46
+
`database_destination` | No | String | The name of the directory in which to store downloaded databases. Default is `{data-prepper.dir}/data/geoip`.
47
+
`aws` | No | [aws](#aws) | Configures the AWS credentials for downloading the database from Amazon Simple Storage Service (Amazon S3).
48
+
`insecure` | No | Boolean | When `true`, this options allows you to download database files over HTTP. Default is `false`.
49
+
50
+
## database
51
+
52
+
Option | Required | Type | Description
53
+
:--- | :--- | :--- | :---
54
+
`city` | No | String | The URL of the city in which the database resides. Can be an HTTP URL for a manifest file, an MMDB file, or an S3 URL.
55
+
`country` | No | String | The URL of the country in which the database resides. Can be an HTTP URL for a manifest file, an MMDB file, or an S3 URL.
56
+
`asn` | No | String | The URL of the Autonomous System Number (ASN) of where the database resides. Can be an HTTP URL for a manifest file, an MMDB file, or an S3 URL.
57
+
`enterprise` | No | String | The URL of the enterprise in which the database resides. Can be an HTTP URL for a manifest file, an MMDB file, or an S3 URL.
58
+
59
+
60
+
## aws
61
+
62
+
Option | Required | Type | Description
63
+
:--- | :--- | :--- | :---
64
+
`region` | No | String | The AWS Region to use for the credentials. Default is the [standard SDK behavior for determining the Region](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/region-selection.html).
65
+
`sts_role_arn` | No | String | The AWS Security Token Service (AWS STS) role to assume for requests to Amazon S3. Default is `null`, which will use the [standard SDK behavior for credentials](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials.html).
66
+
`aws_sts_header_overrides` | No | Map | A map of header overrides that the AWS Identity and Access Management (IAM) role assumes when downloading from Amazon S3.
67
+
`sts_external_id` | No | String | An STS external ID used when Data Prepper assumes the STS role. For more information, see the `ExternalID` documentation in the [STS AssumeRole](https://docs.aws.amazon.com/STS/latest/APIReference/API_AssumeRole.html) API reference.
Copy file name to clipboardExpand all lines: _data-prepper/pipelines/configuration/processors/date.md
+43-11
Original file line number
Diff line number
Diff line change
@@ -9,24 +9,32 @@ nav_order: 50
9
9
# date
10
10
11
11
12
-
The `date` processor adds a default timestamp to an event, parses timestamp fields, and converts timestamp information to the International Organization for Standardization (ISO) 8601 format. This timestamp information can be used as an event timestamp.
12
+
The `date` processor adds a default timestamp to an event, parses timestamp fields, and converts timestamp information to the International Organization for Standardization (ISO) 8601 format. This timestamp information can be used as an event timestamp.
13
13
14
14
## Configuration
15
15
16
16
The following table describes the options you can use to configure the `date` processor.
17
17
18
+
<!-- vale off -->
18
19
Option | Required | Type | Description
19
20
:--- | :--- | :--- | :---
20
-
match | Conditionally | List | List of `key` and `patterns` where patterns is a list. The list of match can have exactly one `key` and `patterns`. There is no default value. This option cannot be defined at the same time as `from_time_received`. Include multiple date processors in your pipeline if both options should be used.
21
-
from_time_received | Conditionally | Boolean | A boolean that is used for adding default timestamp to event data from event metadata which is the time when source receives the event. Default value is `false`. This option cannot be defined at the same time as `match`. Include multiple date processors in your pipeline if both options should be used.
22
-
destination | No | String | Field to store the timestamp parsed by date processor. It can be used with both `match` and `from_time_received`. Default value is `@timestamp`.
23
-
source_timezone | No | String | Time zone used to parse dates. It is used in case the zone or offset cannot be extracted from the value. If the zone or offset are part of the value, then timezone is ignored. Find all the available timezones [the list of database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List) in the **TZ database name** column.
24
-
destination_timezone | No | String | Timezone used for storing timestamp in `destination` field. The available timezone values are the same as `source_timestamp`.
25
-
locale | No | String | Locale is used for parsing dates. It's commonly used for parsing month names(`MMM`). It can have language, country and variant fields using IETF BCP 47 or String representation of [Locale](https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html) object. For example `en-US` for IETF BCP 47 and `en_US` for string representation of Locale. Full list of locale fields which includes language, country and variant can be found [the language subtag registry](https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry). Default value is `Locale.ROOT`.
21
+
`match` | Conditionally | [Match](#Match) | The date match configuration. This option cannot be defined at the same time as `from_time_received`. There is no default value.
22
+
`from_time_received` | Conditionally | Boolean | When `true`, the timestamp from the event metadata, which is the time at which the source receives the event, is added to the event data. This option cannot be defined at the same time as `match`. Default is `false`.
23
+
`date_when` | No | String | Specifies under what condition the `date` processor should perform matching. Default is no condition.
24
+
`to_origination_metadata` | No | Boolean | When `true`, the matched time is also added to the event's metadata as an instance of `Instant`. Default is `false`.
25
+
`destination` | No | String | The field used to store the timestamp parsed by the date processor. Can be used with both `match` and `from_time_received`. Default is `@timestamp`.
26
+
`output_format` | No | String | Determines the format of the timestamp added to an event. Default is `yyyy-MM-dd'T'HH:mm:ss.SSSXXX`.
27
+
`source_timezone` | No | String | The time zone used to parse dates, including when the zone or offset cannot be extracted from the value. If the zone or offset are part of the value, then the time zone is ignored. A list of all the available time zones is contained in the **TZ database name** column of [the list of database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List).
28
+
`destination_timezone` | No | String | The time zone used for storing the timestamp in the `destination` field. A list of all the available time zones is contained in the **TZ database name** column of [the list of database time zones](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List).
29
+
`locale` | No | String | The location used for parsing dates. Commonly used for parsing month names (`MMM`). The value can contain language, country, or variant fields in IETF BCP 47, such as `en-US`, or a string representation of the [locale](https://docs.oracle.com/javase/8/docs/api/java/util/Locale.html) object, such as `en_US`. A full list of locale fields, including language, country, and variant, can be found in [the language subtag registry](https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry). Default is `Locale.ROOT`.
30
+
<!-- vale on -->
26
31
27
-
<!---## Configuration
32
+
### Match
28
33
29
-
Content will be added to this section.--->
34
+
Option | Required | Type | Description
35
+
:--- | :--- | :--- | :---
36
+
`key` | Yes | String | Represents the event key against which to match patterns. Required if `match` is configured.
37
+
`patterns` | Yes | List | A list of possible patterns that the timestamp value of the key can have. The patterns are based on a sequence of letters and symbols. The `patterns` support all the patterns listed in the Java [DatetimeFormatter](https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html) reference. The timestamp value also supports `epoch_second`, `epoch_milli`, and `epoch_nano` values, which represent the timestamp as the number of seconds, milliseconds, and nanoseconds since the epoch. Epoch values always use the UTC time zone.
30
38
31
39
## Metrics
32
40
@@ -40,5 +48,29 @@ The following table describes common [Abstract processor](https://github.com/ope
40
48
41
49
The `date` processor includes the following custom metrics.
42
50
43
-
*`dateProcessingMatchSuccessCounter`: Returns the number of records that match with at least one pattern specified by the `match configuration` option.
44
-
*`dateProcessingMatchFailureCounter`: Returns the number of records that did not match any of the patterns specified by the `patterns match` configuration option.
51
+
*`dateProcessingMatchSuccessCounter`: Returns the number of records that match at least one pattern specified by the `match configuration` option.
52
+
*`dateProcessingMatchFailureCounter`: Returns the number of records that did not match any of the patterns specified by the `patterns match` configuration option.
53
+
54
+
## Example: Add the default timestamp to an event
55
+
The following `date` processor configuration can be used to add a default timestamp in the `@timestamp` filed applied to all events:
56
+
57
+
```yaml
58
+
- date:
59
+
from_time_received: true
60
+
destination: "@timestamp"
61
+
```
62
+
63
+
## Example: Parse a timestamp to convert its format and time zone
64
+
The following `date` processor configuration can be used to parse the value of the timestamp applied to `dd/MMM/yyyy:HH:mm:ss` and write it in `yyyy-MM-dd'T'HH:mm:ss.SSSXXX` format:
The `decompress` processor decompresses any Base64-encoded compressed fields inside of an event.
12
+
13
+
## Configuration
14
+
15
+
Option | Required | Type | Description
16
+
:--- | :--- | :--- | :---
17
+
`keys` | Yes | List<String> | The fields in the event that will be decompressed.
18
+
`type` | Yes | Enum | The type of decompression to use for the `keys` in the event. Only `gzip` is supported.
19
+
`decompress_when` | No | String| A [Data Prepper conditional expression](https://opensearch.org/docs/latest/data-prepper/pipelines/expression-syntax/) that determines when the `decompress` processor will run on certain events.
20
+
`tags_on_failure` | No | List<String> | A list of strings with which to tag events when the processor fails to decompress the `keys` inside an event. Defaults to `_decompression_failure`.
21
+
22
+
## Usage
23
+
24
+
The following example shows the `decompress` processor used in `pipelines.yaml`:
25
+
26
+
```yaml
27
+
processor:
28
+
- decompress:
29
+
decompress_when: '/some_key == null'
30
+
keys: [ "base_64_gzip_key" ]
31
+
type: gzip
32
+
```
33
+
34
+
## Metrics
35
+
36
+
The following table describes common [abstract processor](https://github.com/opensearch-project/data-prepper/blob/main/data-prepper-api/src/main/java/org/opensearch/dataprepper/model/processor/AbstractProcessor.java) metrics.
37
+
38
+
| Metric name | Type | Description |
39
+
| ------------- | ---- | -----------|
40
+
| `recordsIn` | Counter | The ingress of records to a pipeline component. |
41
+
| `recordsOut` | Counter | The egress of records from a pipeline component. |
42
+
| `timeElapsed` | Timer | The time elapsed during execution of a pipeline component. |
43
+
44
+
### Counter
45
+
46
+
The `decompress` processor accounts for the following metrics:
47
+
48
+
* `processingErrors`: The number of processing errors that have occurred in the `decompress` processor.
0 commit comments