Skip to content

Latest commit

 

History

History
3003 lines (1832 loc) · 109 KB

data-prepper.change-log-2.4.0.md

File metadata and controls

3003 lines (1832 loc) · 109 KB

  • Fix for kafka source issue #3264 (aws glue excetion handling) (#3265) (#3266)

    opensearch-trigger-bot[bot] - Sun, 27 Aug 2023 15:37:31 -0500

    EAD -> refs/heads/2.4, tag: refs/tags/2.4.0, refs/remotes/upstream/2.4 (cherry picked from commit d908c40ad946625ef9b183466a9c55ce8033dbb8) Co-authored-by: Hardeep Singh <[email protected]>

  • Fix for kafka source issue #3247 (offset commit stops on deserialization error) (#3260) (#3262)

    opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 17:18:11 -0700

    Signed-off-by: Hardeep Singh <[email protected]> (cherry picked from commit 8114ab40e75c0471bf47b9e78e35f3d4fd11e6a2) Co-authored-by: Hardeep Singh <[email protected]>

  • Disallow the combination of a user-defined schema and include/exclude keys (#3254) (#3261)

    opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 16:10:12 -0700

    Disallow the combination of a user-defined schema and include/exclude keys in the Parquet/Avro sink codecs. Resolves #3253. Signed-off-by: David Venable <[email protected]> (cherry picked from commit 01981a0d07169d359bbf63a1bd03b2a107335239) Co-authored-by: David Venable <[email protected]>

  • Fixes build broken by RELEASING.md spotless check. (#3258) (#3259)

    opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 15:41:53 -0700

    Signed-off-by: David Venable <[email protected]> (cherry picked from commit 1bcf9f6f5a437802722212cbaebff13253c0f3d4) Co-authored-by: David Venable <[email protected]>

  • Adds a RELEASING.md file to the root of the project (#3251) (#3257)

    opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 14:19:11 -0700

    Adds a RELEASING.md file to the root of the project. This has updated instructions for the new release workflow. Resolves #3108. Signed-off-by: David Venable <[email protected]> (cherry picked from commit 1eec63f169a2cba3d32a5cab2cd5349c49a570c6) Co-authored-by: David Venable <[email protected]>

  • Fixes a bug with the S3 parquet codec which was not calculating size correctly. Require the parquet codec only with in_memory which is how it is buffering data. Some debugging help. (#3249) (#3256)

    opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 14:18:07 -0700

    Signed-off-by: David Venable <[email protected]> (cherry picked from commit 7914e93b345cec3969e828ca082d246c7c067b37) Co-authored-by: David Venable <[email protected]>

  • Improve the S3 sink integration tests combinations. The tests are now more consistent and avoid some redundant tests, thus also running faster. Sets up to have fewer combinations while testing all codecs. (#3199) (#3248)

    opensearch-trigger-bot[bot] - Fri, 25 Aug 2023 10:17:34 -0700

    Signed-off-by: David Venable <[email protected]> (cherry picked from commit f9f37299ce1f39e090752c8864255203df1ed798) Co-authored-by: David Venable <[email protected]>

  • Behavioral change to Avro codecs and schema handling (#3238) (#3245)

    opensearch-trigger-bot[bot] - Thu, 24 Aug 2023 14:47:33 -0700

    Change the behavior of Avro-based codecs. When a schema is defined, rely on the schema rather than the incoming event. If the schema is auto-generated, then the incoming event data must continue to match. Fix Avro arrays which were only supporting arrays of strings previously. Signed-off-by: David Venable <[email protected]> (cherry picked from commit f17e8338225aefab9d0241d1b540fdd2face2d81) Co-authored-by: David Venable <[email protected]>

  • Batch the errors writing to the S3 sink to reduce the number of errors reported. (#3242) (#3244)

    opensearch-trigger-bot[bot] - Thu, 24 Aug 2023 14:47:05 -0700

    Signed-off-by: David Venable <[email protected]> (cherry picked from commit 2cca4b187c706fd3ab079b422c96c53bfb8b9b40) Co-authored-by: David Venable <[email protected]>

  • Catch when no object exists and mark as completed in s3 scan (#3241) (#3243)

    opensearch-trigger-bot[bot] - Thu, 24 Aug 2023 14:19:40 -0700

    Signed-off-by: Taylor Gray <[email protected]> (cherry picked from commit c2fc9495c09ba936994f58ff8dbbd94330a03582) Co-authored-by: Taylor Gray <[email protected]>

  • Fix for kafka source not committing offsets issue #3231 (#3232) (#3239)

    opensearch-trigger-bot[bot] - Thu, 24 Aug 2023 13:30:58 -0700

    Signed-off-by: Hardeep Singh <[email protected]> (cherry picked from commit 89e8f3945607499a292d59c1e9fc1ccbbf5b2218) Co-authored-by: Hardeep Singh <[email protected]>

  • Removes @cmanning09 from the CODEOWNERS file to allow the release build to proceed. (#3225) (#3227)

    opensearch-trigger-bot[bot] - Wed, 23 Aug 2023 15:48:59 -0700

    Signed-off-by: David Venable <[email protected]> (cherry picked from commit a4df0bd80e2d47dd8b5392bccb39587307bc544b) Co-authored-by: David Venable <[email protected]>

  • Improvements in the release.yml GitHub Action: Better conditional to fail the promote if the build fails, increased the timeout, added the issues write permissions, string literal correction. (#3224) (#3226)

    opensearch-trigger-bot[bot] - Wed, 23 Aug 2023 14:27:09 -0700

    Signed-off-by: David Venable <[email protected]> (cherry picked from commit 2cd9e693365d41a79d0d9850a963a2823dfef0e3) Co-authored-by: David Venable <[email protected]>

  • Add 2.4 release notes (#3220) (#3222)

    opensearch-trigger-bot[bot] - Wed, 23 Aug 2023 09:16:00 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]> (cherry picked from commit 990497a91c5df9d45cc71b1ed96f33ec0cd7f6e4) Co-authored-by: Asif Sohail Mohammed <[email protected]>

  • Kafka source fixes: commit offsets, consumer group mutations, consumer shutdown (#3207) (#3218)

    opensearch-trigger-bot[bot] - Tue, 22 Aug 2023 15:27:14 -0700

    Removed acknowledgments_timeout config from kafka source Signed-off-by: Hardeep Singh <[email protected]> (cherry picked from commit b5443634a4704cec4c33ec386747e12270aed073) Co-authored-by: Hardeep Singh <[email protected]>

  • Updates documentation for the Avro codec and S3 sink. Resolves #3162. (#3211) (#3219)

    opensearch-trigger-bot[bot] - Tue, 22 Aug 2023 15:18:17 -0700

    Signed-off-by: David Venable <[email protected]> (cherry picked from commit 3d67468ab1ec7008378410fd14bb2e8d6742f9a2) Co-authored-by: David Venable <[email protected]>

  • Catch exceptions when writing to the output codec and drop the event. (#3210) (#3216)

    opensearch-trigger-bot[bot] - Tue, 22 Aug 2023 15:17:57 -0700

    Catch exceptions when writing to the output codec and drop the event. Correctly release failed events in the S3 sink. Signed-off-by: David Venable <[email protected]> (cherry picked from commit c1cbb22fc5a80003d881a66bd028772909b33bba) Co-authored-by: David Venable <[email protected]>

  • Set release version to 2.4 (#3214)

    Asif Sohail Mohammed - Tue, 22 Aug 2023 15:16:28 -0700

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Removes some experimental projects from the 2.4 release and branch. (#3217)

    David Venable - Tue, 22 Aug 2023 15:15:21 -0700

    Signed-off-by: David Venable <[email protected]>

  • Generated THIRD-PARTY file for fecb842 (#3212) (#3213)

    opensearch-trigger-bot[bot] - Tue, 22 Aug 2023 14:20:56 -0700

    Signed-off-by: GitHub <[email protected]> Co-authored-by: dlvenable <[email protected]> (cherry picked from commit 733e7bc8381d1fe02c30b6c75c4cc2dd8984c21c) Co-authored-by: opensearch-trigger-bot[bot] <98922864+opensearch-trigger-bot[bot]@users.noreply.github.com>

  • Normalize the include/exclude keys in the JacksonEvent implementation in order to fix a problem where the ndjson codec was not correctly including/excluding keys. (#3209)

    David Venable - Tue, 22 Aug 2023 13:47:29 -0700

    Signed-off-by: David Venable <[email protected]>

  • Fix bug where enum for s3 select was using NotBlank annotation (#3208)

    Taylor Gray - Tue, 22 Aug 2023 14:04:46 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Adds an auto_schema flag to require a user to be explicit in using auto-schema generation. (#3206)

    David Venable - Tue, 22 Aug 2023 10:25:03 -0700

    Signed-off-by: David Venable <[email protected]>

  • Add jenkins release workflow and refactor GHA workflow (#3204)

    Sayali Gaikawad - Tue, 22 Aug 2023 09:31:00 -0700

    Signed-off-by: Sayali Gaikawad <[email protected]>

  • Removes code which isn't used for the Avro and Parquet codecs. This will keep untested and errant code paths out of the project. Resolves #3201. (#3205)

    David Venable - Tue, 22 Aug 2023 07:40:32 -0700

    Signed-off-by: David Venable <[email protected]>

  • Updates to the S3 sink to speed up the unit test time. There are a few major changes - use the Duration class instead of a nebulous long to have millisecond options and clarity; inject the retry sleep time so that the tests can sleep for shorter time; using mocking where possible to avoid unnecessary sleeps. (#3203)

    David Venable - Tue, 22 Aug 2023 07:05:30 -0700

    Signed-off-by: David Venable <[email protected]>

  • Fix: IllegalArgument Exception in String converter (#3172)

    Asif Sohail Mohammed - Mon, 21 Aug 2023 12:06:13 -0700

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Corrects the output codecs for Avro/Parquet to use the include/exclude keys. Also adds a shouldNotIncludeKey method to OutputCodecContext. (#3197)

    David Venable - Mon, 21 Aug 2023 08:56:19 -0700

    Signed-off-by: David Venable <[email protected]>

  • Correctly add compression extensions to the generated S3 sink keys. If compression is internal, does not utilize. Resolves #3158. (#3196)

    David Venable - Mon, 21 Aug 2023 08:47:04 -0700

    Signed-off-by: David Venable <[email protected]>

  • Fix uncaught exception causing pipeline shutdown (#3189)

    Hai Yan - Mon, 21 Aug 2023 10:15:19 -0500

    • Catch ClassCastException in JacksonOtelLog.toJsonString()
    • Add overwrite option to parse-json processor

    Signed-off-by: Hai Yan <[email protected]>

  • Updates the Data Prepper tar.gz artifact to include JDK 17.0.8_7 which is the current latest version available. (#3136)

    David Venable - Fri, 18 Aug 2023 18:17:15 -0500

    Signed-off-by: David Venable <[email protected]>

  • Updated scan start_time ,end_time and range combinations (#3188)

    Asif Sohail Mohammed - Fri, 18 Aug 2023 16:56:54 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Corrects the JSON output codec to write Events as provided rather than convert to string. Also fixes the include/exclude keys. Adds a boolean check in OutputCodecContext so that this can be used by other codecs. (#3195)

    David Venable - Fri, 18 Aug 2023 16:44:40 -0500

    Signed-off-by: David Venable <[email protected]>

  • Suppport null values in Avro and Parquet output codecs when the schema is null. Auto-generate schemas that are nullable so that null values can be included in these schemas. Resolves part of #3158. (#3194)

    David Venable - Fri, 18 Aug 2023 16:15:56 -0500

    Signed-off-by: David Venable <[email protected]>

  • Added validations in include and exclude keys (#3181)

    Asif Sohail Mohammed - Fri, 18 Aug 2023 13:46:27 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • FIX: set default value for enable_compression (#3190)

    Qi Chen - Fri, 18 Aug 2023 10:47:14 -0500

    Signed-off-by: George Chen <[email protected]>

  • Automatically promote the releases in the release workflow once the release issue has been approved by two maintainers. Resolves #2122. (#3149)

    David Venable - Fri, 18 Aug 2023 08:30:16 -0500

    Signed-off-by: David Venable <[email protected]>

  • Continue calling S3SinkService::output even if records is empty to flush stale batches (#3187)

    Chase - Fri, 18 Aug 2023 08:29:49 -0500

    Continue calling S3SinkService::output even if records is empty to flush stale batches (#3187)

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • merged with latest (#3182)

    kkondaka - Thu, 17 Aug 2023 16:23:06 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Correct Parquet support for the S3 sink and a new multipart buffer type (#3186)

    David Venable - Thu, 17 Aug 2023 16:35:25 -0500

    Correct the ParquetOutputCodec and moved into the S3 sink project for now. It has a few corrections including support for compression and avoiding multiple S3 copies. This PR also adds a new buffer type to the S3 sink - Multipart uploads. This PR also includes a number of refactorings to the project and the integration tests. Signed-off-by: David Venable <[email protected]>

  • Update Kafka source/sink to use ByteCount (#3183)

    kkondaka - Thu, 17 Aug 2023 14:24:30 -0700

    • Retry without seeking incase of AWSSchemaRegistryException Signed-off-by: Krishna Kondaka <[email protected]>

    • Updated to link usage and config to DataPrepper documentation Signed-off-by: Krishna Kondaka <[email protected]>

    • Update Kafka source/sink to use ByteCount Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add Exemplars to metrics generated in aggregate processor (#3165)

    kkondaka - Thu, 17 Aug 2023 09:43:58 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add limit to cardinality key, and metric for cardinality overflow (#3173)

    Jonah Calvo - Thu, 17 Aug 2023 08:40:39 -0700

    • Add limit to cardinality key, and metric for cardinality overflow Signed-off-by: Jonah Calvo <[email protected]>

    • Change cardinality overflow warning from once to every five minutes Signed-off-by: Jonah Calvo <[email protected]>

    Signed-off-by: Jonah Calvo <[email protected]>

  • Adds a new integration test to the S3 sink which can test different scenarios. This currently is testing against ndjson since this codec generally works. (#3179)

    David Venable - Wed, 16 Aug 2023 14:50:36 -0500

    efs/heads/3158-avro-schema Signed-off-by: David Venable <[email protected]>

  • Only log that the scan is complete one time for s3 scan (#3168)

    Taylor Gray - Wed, 16 Aug 2023 13:33:10 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Fix NPE on s3 source stopping without sqs, stop s3 scan worker thread on stopping of the s3 source (#3178)

    Taylor Gray - Wed, 16 Aug 2023 13:09:41 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Support other scan time configs= combinations (#3151)

    Asif Sohail Mohammed - Wed, 16 Aug 2023 12:53:02 -0500

    • Added support for additional time comibinations in s3 scan Signed-off-by: Asif Sohail Mohammed <[email protected]>

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Deprecate document_id_field and add support for document_id with formatting (#3153)

    Taylor Gray - Wed, 16 Aug 2023 12:27:29 -0500

    Deprecate document_id_field and add support for document_id with formatting Signed-off-by: Taylor Gray <[email protected]>

  • Minor fixes to Kafka Source (#3174)

    kkondaka - Wed, 16 Aug 2023 10:23:15 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Reword circuit breaker configuration log message (#3175)

    Taylor Gray - Wed, 16 Aug 2023 12:18:00 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Fix timestamp format (#3171)

    Chase - Wed, 16 Aug 2023 11:52:53 -0500

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Fix S3 sink writing to closed stream exception (#3170)

    Chase - Wed, 16 Aug 2023 11:22:07 -0500

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Snappy as a compression option in the S3 sink: adds new option and engine, adds missing unit test for the CompressionOption class, make other compression engine classes package private. (#3155)

    David Venable - Wed, 16 Aug 2023 10:39:35 -0500

    Signed-off-by: David Venable <[email protected]>

  • Add catching and logging of exceptions for s3 scan worker (#3159)

    Taylor Gray - Tue, 15 Aug 2023 15:36:04 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Support s3:// prefix (#3156)

    Asif Sohail Mohammed - Tue, 15 Aug 2023 10:39:01 -0500

    Support s3:// prefix Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Fixes two flaky unit tests that have failed recently (#3150)

    David Venable - Mon, 14 Aug 2023 16:02:18 -0500

    Fixes two unit tests that have failed and are probably flaky. The ParseTreeEvaluatorListenerTest appears to be using negative values sometimes, which seems to be unsupported. The OTelLogsSourceTest test failed as well, but it appears this code may not always be executed because it was mixing Hamcrest and Mockito. Signed-off-by: David Venable <[email protected]>

  • Fix flaky integration test by wrapping a list in a new list to avoid a ConcurrentModificationException. Resolves #3139. (#3152)

    David Venable - Mon, 14 Aug 2023 16:02:01 -0500

    Signed-off-by: David Venable <[email protected]>

  • Fix flakey test caused by RCF variance. Update metric for RCF Instances (#3145)

    Jonah Calvo - Mon, 14 Aug 2023 13:10:21 -0500

    • Fix flakey test caused by RCF variance Signed-off-by: Jonah Calvo <[email protected]>

    • Change metric name and type. Update test readability Signed-off-by: Jonah Calvo <[email protected]>

    • Fix unit test to account for metric change Signed-off-by: Jonah Calvo <[email protected]>

    Signed-off-by: Jonah Calvo <[email protected]>

  • Fix Null pointer exception when schema registry not specified (#3147)

    kkondaka - Fri, 11 Aug 2023 18:35:17 -0700

    • Fix Null pointer exception when schema registry not specified Signed-off-by: Krishna Kondaka <[email protected]>

    • Fix failing test cases Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Glue registry fixes. Fixed a bug in getMSKBootstrapServers (#3142)

    kkondaka - Fri, 11 Aug 2023 10:28:08 -0700

    • Glue registry fixes. Fixed a bug in getMSKBootstrapServers Signed-off-by: Krishna Kondaka <[email protected]>

    • Changed default auto commit reset to earliest Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add default value to cardinality keys (#3144)

    Jonah Calvo - Fri, 11 Aug 2023 11:14:43 -0500

    Signed-off-by: Jonah Calvo <[email protected]>

  • Adds S3 sink compression. Resolves #3130. (#3138)

    David Venable - Fri, 11 Aug 2023 10:58:22 -0500

    Signed-off-by: David Venable <[email protected]>

  • Add support for scheduled scan to s3 scan (#3140)

    Taylor Gray - Fri, 11 Aug 2023 10:54:42 -0500

    Add support for scheduled scan to s3 scan Signed-off-by: Taylor Gray <[email protected]>

  • The rss-source is defined twice in the settings.gradle and this removes the extra one. (#3134)

    David Venable - Thu, 10 Aug 2023 20:02:37 -0500

    Signed-off-by: David Venable <[email protected]>

  • Adding cardinality key support for AD processor (#3073)

    Jonah Calvo - Thu, 10 Aug 2023 13:33:20 -0700

    • Adding cardinality key support for AD processor Signed-off-by: Jonah Calvo <[email protected]>

    • Refactor hash function to common package. Add metrics for RCF instances. Implement optional verbose mode for RCF Signed-off-by: Jonah Calvo <[email protected]>

    Signed-off-by: Jonah Calvo <[email protected]>

  • S3 single scan improvements (#3124)

    Asif Sohail Mohammed - Thu, 10 Aug 2023 15:30:28 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Correct the behavior of the JSON output codec to write a JSON object first. Adds a configurable keyName for the array. (#3132)

    David Venable - Thu, 10 Aug 2023 15:13:27 -0500

    Signed-off-by: David Venable <[email protected]>

  • Add metrics to Kafka Source (#3118)

    kkondaka - Thu, 10 Aug 2023 11:38:43 -0700

    • Add metrics to Kafka Source Signed-off-by: Krishna Kondaka <[email protected]>

    • Removed debug print statement Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed failing test case Signed-off-by: Krishna Kondaka <[email protected]>

    • Added total committed metric and fixed tests Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed number of committed records stat. Also fixed bug when acknowledgements enabled Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments. Fixed acknowledgements related bug Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed to use counters for records/bytes consumed metrics Signed-off-by: Krishna Kondaka <[email protected]>

    • Removed unused code Signed-off-by: Krishna Kondaka <[email protected]>

    • Added a metric for keeping track of number of consumers without any partitions assigned Signed-off-by: Krishna Kondaka <[email protected]>

    • Added unit test for KafkaTopicMetrics Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Update the developer_guide.md with updated instructions for user documentation. Updated the PR template to include documentation, and updated the Issues Resolved section to encourage use of "Resolves #". (#2772)

    David Venable - Thu, 10 Aug 2023 11:51:16 -0500

    Signed-off-by: David Venable <[email protected]>

  • Improved Avro error reporting related to schemas (#3110)

    David Venable - Thu, 10 Aug 2023 11:22:33 -0500

    Check the Avro schema when starting Data Prepper and throw a clearer exception when an Avro field is missing from the schema. Some code clean-up. Signed-off-by: David Venable <[email protected]>

  • Updates to the Avro codec README.md to include an example of using an Avro schema for VPC Flow Logs. Updates the YAML to make the string easier to handle. (#3111)

    David Venable - Thu, 10 Aug 2023 10:21:05 -0500

    Signed-off-by: David Venable <[email protected]>

  • MAINT: add integ test coverage for ODFE 0.10.0 (#3131)

    Qi Chen - Wed, 9 Aug 2023 19:29:13 -0500

    • MAINT: add integ test coverage for ODFE 0.10.0 Signed-off-by: George Chen <[email protected]>

  • Adds Apache commons-lang3 to the Gradle version catalog and updates it to version 3.13.0. (#3120)

    David Venable - Wed, 9 Aug 2023 17:23:16 -0500

    Signed-off-by: David Venable <[email protected]>

  • Fixed issue with CloudWatchLogMetrics not updating counters when no event handles are present (#3114)

    Marcos Gonzalez Mayedo - Wed, 9 Aug 2023 16:20:10 -0500

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]> Co-authored-by: Marcos Gonzalez Mayedo <[email protected]>

  • Add ttl to all dynamo source coordination store items on creation, not just when they are COMPLETED (#3121)

    Taylor Gray - Wed, 9 Aug 2023 13:26:55 -0500

    efs/heads/csv-validation-up-front Add ttl to all dynamo source coordination store items on creation, not just when they are COMPLETED Signed-off-by: Taylor Gray <[email protected]>

  • Add include_keys and exclude_keys to S3 sink (#3122)

    Aiden Dai - Wed, 9 Aug 2023 13:15:39 -0500

    Signed-off-by: Aiden Dai <[email protected]>

  • Default keys (#3075)

    Katherine Shen - Tue, 8 Aug 2023 12:19:36 -0500

    Signed-off-by: Kat Shen <[email protected]> Co-authored-by: Kat Shen <[email protected]>

  • S3 scan enhancements (#3049)

    Asif Sohail Mohammed - Mon, 7 Aug 2023 16:50:20 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • -Support for kafka-sink (#2999)

    rajeshLovesToCode - Mon, 7 Aug 2023 09:07:18 -0700

  • Centralize exception handling and fix behavior for RequestTimeoutException (#3063)

    Chase - Fri, 4 Aug 2023 16:05:54 -0500

    • Centralize exception handling and fix behavior for RequestTimeoutException Signed-off-by: Chase Engelbrecht <[email protected]>

    • Fix existing tests Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add unit tests for exception handlers Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add copyright headers Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add better default messages Signed-off-by: Chase Engelbrecht <[email protected]>

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Fix code to set max poll interval and fetch min bytes config (#3115)

    kkondaka - Fri, 4 Aug 2023 12:50:56 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Sns Sink Plugin with junit test cases (#2995)

    Uday Chintala - Fri, 4 Aug 2023 12:58:24 -0500

    Sns Sink Plugin with junit test cases

    Signed-off-by: Uday Kumar Chintala <[email protected]> Signed-off-by: Uday Chintala <[email protected]>

  • Prometheus Sink Boiler plate code for issue #1744. (#3078)

    mallikagogoi7 - Fri, 4 Aug 2023 10:10:49 -0500

    • Prometheus Sink boiler plate code for issue #1744. Signed-off-by: mallikagogoi7 <[email protected]>

    • Prometheus Sink Fix for issue #1744. Signed-off-by: mallikagogoi7 <[email protected]>

    • Prometheus Sink review comment rsolved for issue #1744. Signed-off-by: mallikagogoi7 <[email protected]>

  • ENH: support custom index template for ES6 in opensearch sink (#3061)

    Qi Chen - Thu, 3 Aug 2023 11:59:06 -0500

    Signed-off-by: George Chen <[email protected]>

  • GitHub-Issue#2778: Added README for CloudWatch Logs Sink (#3101)

    Marcos Gonzalez Mayedo - Wed, 2 Aug 2023 17:19:06 -0500

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    Co-authored-by: Marcos Gonzalez Mayedo <[email protected]> Co-authored-by: Mark Kuhn <[email protected]>

  • Bump com.opencsv:opencsv from 5.7.1 to 5.8 (#3097)

    dependabot[bot] - Wed, 2 Aug 2023 17:13:11 -0500

    Bumps com.opencsv:opencsv from 5.7.1 to 5.8.

    updated-dependencies:

    • dependency-name: com.opencsv:opencsv dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • GitHub-Issue#2778: Refactored Config to include Duration and ByteCount types (#3099)

    Marcos Gonzalez Mayedo - Wed, 2 Aug 2023 16:43:33 -0500

    • Converted data types in the configuration to Data-Prepper types Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added Duration to backOffTime Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Attempting to fix unused imports Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]> Co-authored-by: Marcos Gonzalez Mayedo <[email protected]>

  • Updated GitHub Actions to use "Data Prepper" in the job titles for consistency and aligning with the project name. (#3104)

    David Venable - Wed, 2 Aug 2023 13:59:04 -0500

    Signed-off-by: David Venable <[email protected]>

  • Fix consumer synchronization. Fix consumer to use user-specified groupId (#3100)

    kkondaka - Wed, 2 Aug 2023 10:43:37 -0700

    • Fix consumer synchronization. Fix consumer to use user-specified groupId Signed-off-by: Krishna Kondaka <[email protected]>

    • Fix check style error Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed to retry if consume records encounters an exception Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Bump org.xerial.snappy:snappy-java in /data-prepper-plugins/common (#3095)

    dependabot[bot] - Wed, 2 Aug 2023 09:00:13 -0500

    Bumps org.xerial.snappy:snappy-java from 1.1.10.1 to 1.1.10.3.

    Commits

    updated-dependencies:

    • dependency-name: org.xerial.snappy:snappy-java dependency-type: direct:production update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • GitHub-Issue#2778: Added CloudWatchLogsSink (#3084)

    Marcos Gonzalez Mayedo - Tue, 1 Aug 2023 15:58:09 -0500

    GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4)

    Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]> Co-authored-by: Taylor Gray <[email protected]> Co-authored-by: Marcos <[email protected]>

  • Fix Negative acknowledgement handling and other minor issues (#3082)

    kkondaka - Tue, 1 Aug 2023 10:10:22 -0700

    • Fix Negative acknowledgement handling and other minor issues Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed check style errors Signed-off-by: Krishna Kondaka <[email protected]>

    • Cleanup of unused files and config Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • HttpSink Plugin Functionality for #874. (#3036)

    mallikagogoi7 - Tue, 1 Aug 2023 08:57:06 -0700

  • Added Translate Processor README.md file (#3033)

    Vishal Boinapalli - Mon, 31 Jul 2023 16:19:08 -0700

    Signed-off-by: Vishal Boinapalli <[email protected]>

  • GitHub-Issue#2778: Added CouldWatchLogsService, Tests and RetransmissionException (#3023)

    Marcos Gonzalez Mayedo - Mon, 31 Jul 2023 13:03:33 -0500

    • Elasticsearch client implementation with pit and no context search (#2910) Create Elasticsearch client, implement search and pit apis for ElasticsearchAccessor Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4) Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added fixes from comments to code (including pathing and nomenclature syntax)

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Refactoring config (#5) Added default params for back_off and log_send_interval alongside test cases for ThresholdConfig. Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Fixed deleted AwsConfig file Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Removed the s3 dependency from build.gradle, replaced the AwsAuth.. with AwsConfig. Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added modifiable back_off_timer, added threshold test for back_off_timer and params to AwsConfig Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added fixes to gradle file, added tests to AwsConfig, and used Reflective mapping to tests CwlSink Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added default value test to ThresholdConfig and renamed getter for maxRequestSize Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Removed unnecessary imports Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added cloudwatch-logs to settings.gradle Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added a quick fix to the back_off_time range Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added Buffer classes, ClientFactory similar to S3, and ThresholdCheck Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Removed unnecessary default method from ClientFactory Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added comments in Buffer Interface, change some default values to suit the plugin use case more Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Removed unused imports Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Changed the unused imports, made parameters final in the ThresholdCheck Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Made changes to the tests and the method signatures in ThresholdCheck, made fixes to gradle file to include catalog Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Removed unused methods/comments Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added CloudWatchLogsService, CloudWatchLogsServiceTest and RetransmissionLimitException Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Fixed retransmission logging fixed value Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Fixed unused imports Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Fixed making ThresholdCheck public Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added fixes to ThresholdCheck and CloudWatchLogsService to decouple methods Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Fixed syntax start import in CloudWatchLogsServiceTest Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Extracted LogPusher and SinkStopWatch classes for code cleanup. Addded fixes to variables and retry logic for InterruptExceptions Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Changed method uses in CloudWatchLogsService and removed logging the batch size in LogPusher Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added Multithreaded CloudWatchLogsDispatcher for handling various async calls to perform PLE's and added tests Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added fixesto test and defaulted the parameters in the config to CloudWatchLogs limits, customer can change this in config file Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added exponential backofftime Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Fixed unused imports Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Fixed up deepcopy of arraylist for service workers in CloudWatchLogsService, and fixed Log calling methods Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added CloudWatchLogsDispatcher builder pattern, fixed tests for Service and Dispatcher and modified backOffTimeBase Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Removed unused imports Signed-off-by:Marcos Gonzalez Mayedo <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added resetBuffer method, removed unnecessary RetransmissionException, and added logString pass in parameter for staging log events. Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Started making changes to the tests to implement the new class structure (performance enhancement) Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Refactored the CloudWatchLogsDispatcher into two classes with the addition of Uploader, introduced simple multithread tests for CloudWatchLogsService Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Fixed issues with locking in try block and added final multithreaded tests to the CloudWatchLogsService class Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added CloudWatchLogsMetricsTest, changed upper back off time bound and scale, and refactoring changes for better code syntax (renaming, refactoring methods for conciseness, etc...) Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added changes to javadoc Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Update data-prepper-plugins/cloudwatch-logs/src/main/java/org/opensearch/dataprepper/plugins/sink/client/CloudWatchLogsDispatcher.java

    Co-authored-by: Mark Kuhn <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Fixed comment on CloudWatchLogsDispatcher Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]> Co-authored-by: Taylor Gray <[email protected]> Co-authored-by: Marcos <[email protected]>

    Co-authored-by: Mark Kuhn <[email protected]>

  • Config changes and support for adding different modes to put kafka key in the event (#3076)

    kkondaka - Sun, 30 Jul 2023 22:54:47 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add include_keys and exclude_keys to sink (#2989)

    Aiden Dai - Fri, 28 Jul 2023 16:43:20 -0700

    Signed-off-by: Aiden Dai <[email protected]>

  • Removed IterateOn otpion (#3050)

    Vishal Boinapalli - Fri, 28 Jul 2023 09:33:08 -0700

    Signed-off-by: Vishal Boinapalli <[email protected]>

  • Changed non-exact mathching logic (#3046)

    Vishal Boinapalli - Fri, 28 Jul 2023 09:31:02 -0700

    Signed-off-by: Vishal Boinapalli <[email protected]>

  • Translate Processor : Added support for S3 file mappings (#3054)

    Vishal Boinapalli - Fri, 28 Jul 2023 09:30:43 -0700

    • Added support for S3 file Signed-off-by: Vishal Boinapalli <[email protected]>

    • Changed logic for retrieving mappings from S3 file Signed-off-by: Vishal Boinapalli <[email protected]>

    Signed-off-by: Vishal Boinapalli <[email protected]>

  • Moves the S3 sink and HTTP sink into their own packages. This fixes an issue where there are class conflicts at runtime. (#3067)

    David Venable - Thu, 27 Jul 2023 19:37:18 -0500

  • Exclude keys (#3055)

    Katherine Shen - Thu, 27 Jul 2023 11:35:48 -0500

    Signed-off-by: Kat Shen <[email protected]> Co-authored-by: Kat Shen <[email protected]>

  • Parquet Sink Codec (#2928)

    umayr-codes - Thu, 27 Jul 2023 08:35:31 -0700

    Co-authored-by: umairofficial <[email protected]>

  • Adds new configurations to the S3 source to better define bucket ownership. Resolves #2012. (#3012)

    David Venable - Wed, 26 Jul 2023 14:41:29 -0500

    Signed-off-by: David Venable <[email protected]>

  • -Support for Sink Codecs (#2986)

    umayr-codes - Wed, 26 Jul 2023 11:52:00 -0700

    Co-authored-by: umairofficial <[email protected]>

  • Add support for Glue registry (#3056)

    kkondaka - Tue, 25 Jul 2023 15:32:30 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Improves some of the timeouts for the peer forwarder tests to reduce testing time. (#3020)

    David Venable - Tue, 25 Jul 2023 12:49:46 -0500

    Signed-off-by: David Venable <[email protected]>

  • Improves some of the test timing in Data Prepper core tests which are showing somewhat high test times. (#3021)

    David Venable - Tue, 25 Jul 2023 12:49:32 -0500

    Signed-off-by: David Venable <[email protected]>

  • Work to reduce test time by reducing some repeated tests, using Awaitility, and reducing delays (#3019)

    David Venable - Tue, 25 Jul 2023 12:49:18 -0500

    Work to reduce test time by reducing some repeated tests, using Awaitility, and reducing some wait times. Signed-off-by: David Venable <[email protected]>

  • FIX: missing request index (#3058)

    Qi Chen - Tue, 25 Jul 2023 12:03:25 -0500

    Signed-off-by: George Chen <[email protected]>

  • Fix bug where exception is thrown when csv source key does not exist or is null (#3053)

    Taylor Gray - Tue, 25 Jul 2023 12:01:25 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • ENH: support es 6 in sink (#3045)

    Qi Chen - Mon, 24 Jul 2023 15:52:17 -0500

  • Kafka source Confluent schema registry connectivity and OAuth implementation (#3037)

    Ajeesh Gopalakrishnakurup - Mon, 24 Jul 2023 11:27:48 -0700

    • Schema registry connectivity with the oauth configurations Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    • Junit fixes Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    • Defect fixes Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    • Updated the review comments for the PR3037 Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

  • Remove brackets feature option (#3035)

    Katherine Shen - Mon, 24 Jul 2023 13:17:14 -0500

    *add remove brackets feature option Signed-off-by: Kat Shen <[email protected]>

    Signed-off-by: Kat Shen <[email protected]> Co-authored-by: Kat Shen <[email protected]>

  • Add Support for Auth/NoAuth with/without Encryption in Kafka with integration tests (#3042)

    kkondaka - Thu, 20 Jul 2023 09:20:15 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Bump word-wrap from 1.2.3 to 1.2.4 in /release/staging-resources-cdk (#3044)

    dependabot[bot] - Thu, 20 Jul 2023 10:26:17 -0500

    Bumps word-wrap from 1.2.3 to 1.2.4.

    updated-dependencies:

    • dependency-name: word-wrap dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Translate Processor - Added file_path functionality for local files (#3034)

    Vishal Boinapalli - Wed, 19 Jul 2023 14:20:24 -0700

    • Changed target_type option name to type Signed-off-by: Vishal Boinapalli <[email protected]>

    • Added file_path functionality for local file Signed-off-by: Vishal Boinapalli <[email protected]>

    • Fixed file parsing issue and changed error msgs Signed-off-by: Vishal Boinapalli <[email protected]>

    • Added IOException to log, made testcase change for mappings validation Signed-off-by: Vishal Boinapalli <[email protected]>

    Signed-off-by: Vishal Boinapalli <[email protected]>

  • Consolidate logic related to extracting data from a BulkOperation (#3041)

    Chase - Wed, 19 Jul 2023 13:15:41 -0500

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Kafka Source - Cleanup and Enhancements for MSK (#3029)

    kkondaka - Tue, 18 Jul 2023 15:54:21 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Fix race condition in data prepper sources using e2e acknowledgements (#3039)

    kkondaka - Tue, 18 Jul 2023 15:53:41 -0700

    • Fix race condition in data prepper sources using e2e acknowledgements Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed checkStyle error Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add support for using expressions with formatString in JacksonEvent, use for index in OpenSearch sink (#3032)

    Taylor Gray - Tue, 18 Jul 2023 12:10:24 -0500

    Add support for using expressions with formatString in JacksonEvent, use for index in OpenSearch sink Signed-off-by: Taylor Gray <[email protected]>

    Signed-off-by: Taylor Gray <[email protected]>

  • -Support for Sink Codecs (#3030)

    Omkar - Mon, 17 Jul 2023 13:28:58 -0700

  • Duplicate values (#3026)

    Katherine Shen - Mon, 17 Jul 2023 09:35:57 -0700

    Signed-off-by: Kat Shen <[email protected]> Signed-off-by: Katherine Shen <[email protected]> Co-authored-by: Kat Shen <[email protected]>

  • Connection code of HttpSink Plugin for #874. (#2987)

    mallikagogoi7 - Mon, 17 Jul 2023 09:49:32 -0500

    Connection code of HttpSink Plugin. Signed-off-by: mallikagogoi7 <[email protected]>

  • GitHub-issue#253 : Implemented GeoIP processor integration test (#2927)

    venkataraopasyavula - Fri, 14 Jul 2023 14:54:33 -0700

    • GitHub-issue#253 : Implemented GeoIP processor integration test

    Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor integration test

    Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor integration test

    Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor integration test

    Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor integration test

    Signed-off-by: venkataraopasyavula <[email protected]>

  • Add support for Data Prepper expressions in the document_id_field of the OpenSearch sink, add opensearch prefix to opensearch source metadata keys (#3025)

    Taylor Gray - Fri, 14 Jul 2023 10:51:25 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Translate Plugin: Simplified Config. (#3022)

    Vishal Boinapalli - Thu, 13 Jul 2023 22:18:06 -0700

    • Translate Plugin: Simplified Config. Added functionality for multiple sources and multiple targets Signed-off-by: Vishal Boinapalli <[email protected]>

    • Moved helper methods out of config file Signed-off-by: Vishal Boinapalli <[email protected]>

    Signed-off-by: Vishal Boinapalli <[email protected]>

  • Adds the Data Prepper 2.3.2 change log. (#3024)

    David Venable - Thu, 13 Jul 2023 12:02:45 -0500

    Signed-off-by: David Venable <[email protected]>

  • Updated Kafka security configuration (#2994)

    kkondaka - Thu, 13 Jul 2023 08:20:50 -0700

    • Add Kafka Security Configurations Signed-off-by: Krishna Kondaka <[email protected]>

    • Modified kafka security config. Added new fields to AwsConfig Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Modified AwsConfig to have msk option that can take multiple options Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • GitHub-Issue#2778: Added CloudWatchLogs Buffer, ThresholdCheck, and ClientFactory utilities. (#2982)

    Marcos Gonzalez Mayedo - Wed, 12 Jul 2023 16:35:47 -0500

    Added CloudWatchLogs Buffer, ThresholdCheck, and ClientFactory utilities.

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    Co-authored-by: Marcos <[email protected]>

  • Whitespace (#3004)

    Katherine Shen - Wed, 12 Jul 2023 13:28:59 -0500

    implement transform_key feature Signed-off-by: Kat Shen <[email protected]>

    Signed-off-by: Kat Shen <[email protected]> Co-authored-by: Kat Shen <[email protected]>

  • Release notes for Data Prepper 2.3.2 (#3016)

    David Venable - Wed, 12 Jul 2023 13:13:11 -0500

    Signed-off-by: David Venable <[email protected]>

  • GitHub-issue#253 : Implemented GeoIP processor functionality (#2925)

    venkataraopasyavula - Wed, 12 Jul 2023 08:58:22 -0700

    • GitHub-issue#253 : Implemented GeoIP processor functionality Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor functionality Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor functionality Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor functionality Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor functionality Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Fixed the test-case-failed issue. Signed-off-by: Deepak Sahu <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Implemented GeoIP processor functionality. Addressed Code review comments Signed-off-by: venkataraopasyavula <[email protected]>

    Signed-off-by: Deepak Sahu <[email protected]> Co-authored-by: Deepak Sahu <[email protected]>

  • Fix bucket ownership validation. Resolves #3005 (#3009)

    David Venable - Tue, 11 Jul 2023 21:30:00 -0500

    Signed-off-by: David Venable <[email protected]>

  • Introduce option to measure bulk sizes with or without compression (#2985)

    Chase - Tue, 11 Jul 2023 18:17:33 -0500

    • Initial bulk estimation improvements Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add setting to enable/disable estimation with compression Signed-off-by: Chase Engelbrecht <[email protected]>

    • Make max local compressions configurable Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add documentation for new settings Signed-off-by: Chase Engelbrecht <[email protected]>

    • Remove debug comment Signed-off-by: Chase Engelbrecht <[email protected]>

    • Remove another debug log Signed-off-by: Chase Engelbrecht <[email protected]>

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Fix race condition in SqsWorker when acknowledgements are enabled (#3001)

    kkondaka - Tue, 11 Jul 2023 15:09:26 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Retry s3 reads on socket exceptions. (#2992)

    Adi Suresh - Tue, 11 Jul 2023 12:25:20 -0500

    • Retry s3 reads on socket exceptions. S3 will reset the conenction on their end frequently. To not lose data, data prepper should retry all socket exceptions by attempting to re-open the stream. Signed-off-by: Adi Suresh <[email protected]>

    • Bubble up parquet exceptions. Signed-off-by: Adi Suresh <[email protected]>

    Signed-off-by: Adi Suresh <[email protected]>

  • Fix SqsWorker error messages (#2991)

    kkondaka - Fri, 7 Jul 2023 10:25:47 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Translate Plugin - Target Type implementation (#2979)

    Vishal Boinapalli - Thu, 6 Jul 2023 16:29:02 -0700

    • Translate Plugin - Target Type implementation Signed-off-by: Vishal Boinapalli <[email protected]>

    • addressed review comments Signed-off-by: Vishal Boinapalli <[email protected]>

    Signed-off-by: Vishal Boinapalli <[email protected]>

  • Implement transform_key feature (#2977)

    Katherine Shen - Thu, 6 Jul 2023 11:37:55 -0500

    implement transform_key feature Signed-off-by: Kat Shen <[email protected]>

  • Fix S3 errors around end of file behavior. (#2983)

    Adi Suresh - Wed, 5 Jul 2023 16:20:12 -0500

    Signed-off-by: Adi Suresh <[email protected]>

  • Fix Stdout and File sink (#2978)

    kkondaka - Wed, 5 Jul 2023 10:15:15 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Boiler plate code of HttpSink for #874. (#2916)

    mallikagogoi7 - Wed, 5 Jul 2023 09:27:57 -0700

    • Boiler plate code of HttpSink for #874. Signed-off-by: mallikagogoi7 <[email protected]>

    • Added copyright on classes of HttpSink for #874. Signed-off-by: mallikagogoi7 <[email protected]>

    • Moved Accumulator package to common for #874. Signed-off-by: mallikagogoi7 <[email protected]>

    • Test cases added for accumulator related classes for #874. Signed-off-by: mallikagogoi7 <[email protected]>

    • Added HttpSink related methos in accumulator for #874. Signed-off-by: mallikagogoi7 <[email protected]>

    • Removed plugin specific methods from common for #874. Signed-off-by: mallikagogoi7 <[email protected]>

  • GitHub-Issue#2778: Added CloudWatchLogs Sink Config Files (#2922)

    Marcos Gonzalez Mayedo - Wed, 5 Jul 2023 09:12:03 -0700

    • Elasticsearch client implementation with pit and no context search (#2910) Create Elasticsearch client, implement search and pit apis for ElasticsearchAccessor Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • GitHub-Issue#2778: Refactoring config files for CloudWatchLogs Sink (#4) Added Config Files for CloudWatchLogs Sink. Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added fixes from comments to code (including pathing and nomenclature syntax)

    Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Refactoring config (#5) Added default params for back_off and log_send_interval alongside test cases for ThresholdConfig. Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Fixed deleted AwsConfig file Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Removed the s3 dependency from build.gradle, replaced the AwsAuth.. with AwsConfig. Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added modifiable back_off_timer, added threshold test for back_off_timer and params to AwsConfig Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added fixes to gradle file, added tests to AwsConfig, and used Reflective mapping to tests CwlSink Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added default value test to ThresholdConfig and renamed getter for maxRequestSize Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Removed unnecessary imports Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added cloudwatch-logs to settings.gradle Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    • Added a quick fix to the back_off_time range Signed-off-by: Marcos Gonzalez Mayedo <[email protected]>

    Signed-off-by: Taylor Gray <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]> Signed-off-by: Marcos Gonzalez Mayedo <[email protected]> Co-authored-by: Taylor Gray <[email protected]> Co-authored-by: Marcos <[email protected]>

  • Bump org.apache.commons:commons-compress in /data-prepper-plugins/common (#2960)

    dependabot[bot] - Wed, 5 Jul 2023 10:59:41 -0500

    Bumps org.apache.commons:commons-compress from 1.21 to 1.23.0.

    updated-dependencies:

    • dependency-name: org.apache.commons:commons-compress dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump commons-io:commons-io from 2.11.0 to 2.13.0 in /data-prepper-api (#2900)

    dependabot[bot] - Wed, 5 Jul 2023 10:57:43 -0500

    Bumps commons-io:commons-io from 2.11.0 to 2.13.0.

    updated-dependencies:

    • dependency-name: commons-io:commons-io dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • Bump com.fasterxml.jackson.datatype:jackson-datatype-jsr310 (#2796)

    dependabot[bot] - Wed, 5 Jul 2023 10:55:11 -0500

    Bumps com.fasterxml.jackson.datatype:jackson-datatype-jsr310 from 2.14.2 to 2.15.2.

    updated-dependencies:

    • dependency-name: com.fasterxml.jackson.datatype:jackson-datatype-jsr310 dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

  • -Support for Sink Codecs (#2881)

    Omkar - Wed, 5 Jul 2023 10:53:44 -0500

    -Support for Sink Codecs Signed-off-by: omkarmmore95 <[email protected]>

  • Added Kafka config to support acknowledgments and MSK arn (#2976)

    kkondaka - Tue, 4 Jul 2023 21:25:09 -0700

    • Added Kafka config to support acknowledgments and MSK arn Signed-off-by: Krishna Kondaka <[email protected]>

    • Modified to use data-prepper-core in testImplementation Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed review comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Addressed failing test Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Define multiple keys for type conversion (#2934)

    wanghd89 - Tue, 4 Jul 2023 21:42:07 -0500

    • feat: add include_key options to KeyValueProcessor Signed-off-by: Haidong <[email protected]>

    Signed-off-by: Haidong <[email protected]> Co-authored-by: Haidong <[email protected]>

  • Translate Plugin - Added functionality for iterate_on, default, exact options (#2953)

    Vishal Boinapalli - Mon, 3 Jul 2023 14:32:32 -0500

    Added IterateOn functionality, default, exact and testcases for translate processor Signed-off-by: Vishal Boinapalli <[email protected]>

  • Kafka Source code refactoring (#2951)

    kkondaka - Fri, 30 Jun 2023 13:01:29 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Add disable_authentication flag to the opensearch source (#2942)

    Taylor Gray - Thu, 29 Jun 2023 11:37:34 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Fix CVE-2023-35165, CVE-2023-34455, CVE-2023-34453, CVE-2023-34454, C… (#2948)

    Asif Sohail Mohammed - Thu, 29 Jun 2023 09:59:03 -0500

    • Fix CVE-2023-35165, CVE-2023-34455, CVE-2023-34453, CVE-2023-34454, CVE-2023-2976 Signed-off-by: Asif Sohail Mohammed <[email protected]>

    • Updated snappy version in build.gradle files Signed-off-by: Asif Sohail Mohammed <[email protected]>

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Update simple_pipelines.md (#2947)

    Katherine Shen - Thu, 29 Jun 2023 09:29:49 -0500

    Removed a space between the second ${PWD} and the rest of the line Signed-off-by: Katherine Shen <[email protected]>

  • Adding Translate Processor functionality and config files (#2913)

    Vishal Boinapalli - Wed, 28 Jun 2023 13:45:02 -0700

    • Adding MapValues Processor functionality and config file Signed-off-by: Vishal Boinapalli <[email protected]>

    • Changed logic for splitting of Keys, Added config file for Regex option Signed-off-by: Vishal Boinapalli <[email protected]>

    • Added testcases, made changes addressing previous review comments, Changed the processor name from map_values to translate Signed-off-by: Vishal Boinapalli <[email protected]>

    • Added regexConfigTests, made code structure changes, added check for patterns under regex Signed-off-by: Vishal Boinapalli <[email protected]>

    • Removed * imports Signed-off-by: Vishal Boinapalli <[email protected]>

    Signed-off-by: Vishal Boinapalli <[email protected]>

  • Sqs Source Integration Tests (#2874)

    Uday Chintala - Wed, 28 Jun 2023 10:02:25 -0500

    Sqs Source Integration Tests Signed-off-by: Uday Kumar Chintala <[email protected]>

  • Remove validation that made keys starting or ending with . - or _ invalid, catch all exceptions in the parse json processor (#2945)

    Taylor Gray - Tue, 27 Jun 2023 16:25:19 -0500

    Remove validation that made keys starting or ending with . - or _ invalid, catch all exceptions in the parse json processor Signed-off-by: Taylor Gray <[email protected]>

  • Kafka source integration test (#2891)

    Ajeesh Gopalakrishnakurup - Tue, 27 Jun 2023 10:11:52 -0700

    • Integration testcases Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    • Fix for the Integration testcases Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    • Fix for the white source issue Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    • Fixes for the merge conflicts Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

  • Add basic opensearch source documentation for config (#2940)

    Taylor Gray - Tue, 27 Jun 2023 12:02:36 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Add support for writing tags along with events to Sink (#2850)

    kkondaka - Tue, 27 Jun 2023 08:42:27 -0700

    • Updated to pass SinkContext to Sink constructors as suggested in the previous comments Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed check style errors and renamed RoutedPluginSetting to SinkContextPluginSetting Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed s3-sink integration test Signed-off-by: Krishna Kondaka <[email protected]>

    • Added javadoc for SinkContext Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Temporarily reduce coverage minimum (#2937)

    kkondaka - Mon, 26 Jun 2023 16:50:03 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Implement scroll apis for ElasticSearch Accessor (#2930)

    Taylor Gray - Mon, 26 Jun 2023 14:58:36 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Sqs Source initial changes (#2786)

    Uday Chintala - Mon, 26 Jun 2023 13:12:28 -0500

    Signed-off-by: Uday Kumar Chintala <[email protected]> Signed-off-by: Uday Chintala <[email protected]>

  • Fix DLQ writer writing empty list (#2931)

    kkondaka - Mon, 26 Jun 2023 09:48:31 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • S3 Scan time range improvements (#2883)

    Hai Yan - Mon, 26 Jun 2023 11:11:40 -0500

    • When no time range set, default to scan all objects; allow setting time range for specific bucket Signed-off-by: Hai Yan <[email protected]>

    Signed-off-by: Hai Yan <[email protected]>

  • Implement searching with scroll contexts for OpenSearch (#2923)

    Taylor Gray - Fri, 23 Jun 2023 15:26:52 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Fix addTags API in EventMetadata (#2926)

    kkondaka - Fri, 23 Jun 2023 10:07:09 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • GitHub-issue#253 : Implemented GeoIP Plugin Configuration Code JUnit test cases (#2909)

    venkataraopasyavula - Thu, 22 Jun 2023 09:11:48 -0700

    Signed-off-by: venkataraopasyavula <[email protected]>

  • Elasticsearch client implementation with pit and no context search (#2910)

    Taylor Gray - Wed, 21 Jun 2023 13:59:00 -0500

    Create Elasticsearch client, implement search and pit apis for ElasticsearchAccessor Signed-off-by: Taylor Gray <[email protected]>

  • Added Readme.md (#2714)

    Ajeesh Gopalakrishnakurup - Wed, 21 Jun 2023 11:49:55 -0700

    Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

  • OpenSearch Sink Optimizations (#2908)

    Chase - Wed, 21 Jun 2023 10:12:26 -0500

    • Fix size estimation for compression. Maintain requests across iterations for better packing. Limit bulk response size Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add unit tests, slight refactors Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add null handling Signed-off-by: Chase Engelbrecht <[email protected]>

    • Increase gradle heap Signed-off-by: Chase Engelbrecht <[email protected]>

    • Set flush timeout in IT Signed-off-by: Chase Engelbrecht <[email protected]>

    • Set flush timeout to 0 in ITs Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add documentation for flush_timeout and fix OpenSearchSinkITs Signed-off-by: Chase Engelbrecht <[email protected]>

    • Add default to documentation Signed-off-by: Chase Engelbrecht <[email protected]>

    • Set flush_timeout to 5s in e2e tests to fall within timeouts Signed-off-by: Chase Engelbrecht <[email protected]>

    Signed-off-by: Chase Engelbrecht <[email protected]>

  • Updated the release date (#2911)

    Asif Sohail Mohammed - Tue, 20 Jun 2023 13:45:17 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Kafka source retry (#2708)

    Ajeesh Gopalakrishnakurup - Tue, 20 Jun 2023 09:50:26 -0700

    • Code rebase for the PR2708 Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    • Fixes the code rebase issue for the PR2708 Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    • Fixes the code rebase issue and code refactoring Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    • Fixes the build issue Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    • Refactored the consumer code for plaintext,json and avro Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    • Updated the review comments for the PR2708 Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

    Signed-off-by: Ajeesh Gopalakrishnakurup <[email protected]>

  • Configuration PR for HttpSink for #874. (#2890)

    mallikagogoi7 - Mon, 19 Jun 2023 11:40:49 -0700

    Signed-off-by: mallikagogoi7 <[email protected]>

  • Added 2.3.1 release notes (#2871)

    Asif Sohail Mohammed - Mon, 19 Jun 2023 12:44:40 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Added 2.3.1 change log (#2872)

    Asif Sohail Mohammed - Mon, 19 Jun 2023 12:44:25 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • GitHub-issue#253 : Implemented GeoIP Plugin boiler plate Code Structure (#2840)

    venkataraopasyavula - Mon, 19 Jun 2023 10:01:15 -0700

    Signed-off-by: venkataraopasyavula <[email protected]>

  • Change the title to be consistent with configuration (#2899)

    kkondaka - Mon, 19 Jun 2023 09:13:06 -0700

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Do not suppress logs when there are exception in s3 source. (#2896)

    Adi Suresh - Sat, 17 Jun 2023 10:04:23 -0500

    Signed-off-by: Adi Suresh <[email protected]>

  • Change log for index name format failure in opensearch sink (#2894)

    Taylor Gray - Fri, 16 Jun 2023 16:09:38 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Updated s3 sink metrics (#2888)

    Asif Sohail Mohammed - Fri, 16 Jun 2023 08:48:44 -0500

  • Implement NoSearchContextWorker to search with search_after and not use pit or scroll, allow override with search_context_type parameter (#2873)

    Taylor Gray - Thu, 15 Jun 2023 16:14:33 -0500

    • Implement NoSearchContextWorker to search with search_after and not use pit or scroll, allow override with search_context_type parameter Signed-off-by: Taylor Gray <[email protected]>

  • Fix silent dropping of data when index format has null keys, write to dlq if configured (#2885)

    Taylor Gray - Thu, 15 Jun 2023 15:12:30 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • GitHub-issue#253 : Implemented GeoIP Plugin Configuration Code (#2811)

    venkataraopasyavula - Thu, 15 Jun 2023 12:31:04 -0700

    • GitHub-issue#253 : Implemented GeoIP Plugin Configuration Code

    Signed-off-by: venkataraopasyavula <[email protected]>

    • GitHub-issue#253 : Incorporated code review comments of GeoIP Plugin Configuration Code Signed-off-by: venkataraopasyavula <[email protected]>

  • Update README.md for S3 sink (#2878)

    Travis Benedict - Thu, 15 Jun 2023 10:39:53 -0500

    Signed-off-by: Travis Benedict <[email protected]>

  • Add exception when gzip input stream not have magic header. (#2879)

    Adi Suresh - Thu, 15 Jun 2023 09:28:17 -0500

    Signed-off-by: Adi Suresh <[email protected]>

  • FIX: concurrentModification (#2876)

    Qi Chen - Wed, 14 Jun 2023 21:24:07 -0500

    Signed-off-by: George Chen <[email protected]>

  • S3 EventBridge and security lake support (#2861)

    Asif Sohail Mohammed - Tue, 13 Jun 2023 17:53:18 -0500

    • EventBridge initial working draft Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Add STS external ID to all STS configurations. (#2862)

    Adi Suresh - Tue, 13 Jun 2023 16:08:30 -0500

    STS external ID is required by some AWS services when making an STS AssumeRole call. Signed-off-by: Adi Suresh <[email protected]>

  • Save searchAfter to state so pagination can continue where it left off when using PIT on opensearch source (#2856)

    Taylor Gray - Mon, 12 Jun 2023 15:56:57 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Consolidate BufferAccumulator to buffer-api module (#2857)

    Taylor Gray - Mon, 12 Jun 2023 13:29:56 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • Add end-to-end acknowledgement support to Stdout and File Sinks (#2860)

    kkondaka - Mon, 12 Jun 2023 11:18:18 -0700

    • Add end-to-end acknowledgement support to Stdout and File Sinks Signed-off-by: Krishna Kondaka <[email protected]>

    • Fixed code to check for object is an instance of event Signed-off-by: Krishna Kondaka <[email protected]>

    Signed-off-by: Krishna Kondaka <[email protected]> Co-authored-by: Krishna Kondaka <[email protected]>

  • Sqs Source configuration code changes for #2679 (#2801)

    Uday Chintala - Mon, 12 Jun 2023 12:04:53 -0500

    Sqs Source configuration code changes for #2679 Signed-off-by: Uday Kumar Chintala <[email protected]>

  • Os source buffer backoff retry (#2849)

    Taylor Gray - Mon, 12 Jun 2023 09:40:10 -0500

    Use buffer accumulator in opensearch source to backoff and retry Signed-off-by: Taylor Gray <[email protected]>

  • -Support for Sink Codecs (#2845)

    Omkar - Fri, 9 Jun 2023 13:40:05 -0500

    Support for Sink Codecs Signed-off-by: omkarmmore95 <[email protected]>

  • Implement basic search with point in time and search after (#2847)

    Taylor Gray - Fri, 9 Jun 2023 10:06:50 -0500

    Implement basic search with point in time and search after Signed-off-by: Taylor Gray <[email protected]>

  • Fix CVE in maven-artifact by excluding that dependency (#2848)

    Taylor Gray - Thu, 8 Jun 2023 15:52:39 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • -Support for Sink Codecs (#2842)

    Omkar - Thu, 8 Jun 2023 14:47:08 -0500

    Signed-off-by: omkarmmore95 <[email protected]>

  • Create and delete point in time for processing an index (#2839)

    Taylor Gray - Thu, 8 Jun 2023 11:32:22 -0500

    Signed-off-by: Taylor Gray <[email protected]>

  • GitHub-issue#2822: Define null characters in convert processor (#2844)

    saydar31 - Thu, 8 Jun 2023 11:28:37 -0500

    GitHub-issue#2822: Define null characters in convert processor Signed-off-by: Aidar Shaidullin <[email protected]> Signed-off-by: saydar31 <[email protected]>

    Signed-off-by: saydar31 <[email protected]> Co-authored-by: saydar31 <[email protected]>

  • Implement opensearch index partition creation supplier and PitWorker without processing indices (#2821)

    Taylor Gray - Tue, 6 Jun 2023 17:59:59 -0500

    Implement opensearch index partition creation supplier and PitWorker without processing indices Signed-off-by: Taylor Gray <[email protected]>

  • Added change log for 2.3 (#2836)

    Asif Sohail Mohammed - Tue, 6 Jun 2023 12:32:55 -0500

    Signed-off-by: Asif Sohail Mohammed <[email protected]>

  • Adds release notes for Data Prepper 2.3.0. (#2833)

    David Venable - Tue, 6 Jun 2023 10:59:57 -0500

    Signed-off-by: David Venable <[email protected]>

  • Updates main to Data Prepper 2.4. (#2832)

    David Venable - Tue, 6 Jun 2023 10:44:47 -0500

    Signed-off-by: David Venable <[email protected]>

  • Fix bug where s3 stream was closing too early. (#2830)

    Adi Suresh - Mon, 5 Jun 2023 20:21:40 -0500

    Signed-off-by: Adi Suresh <[email protected]>

  • Generated THIRD-PARTY file for 3a70e73 (#2828)

    opensearch-trigger-bot[bot] - Mon, 5 Jun 2023 17:06:11 -0500

    Signed-off-by: GitHub <[email protected]> Co-authored-by: dlvenable <[email protected]>