Skip to content

Optimistic Concurrency Control Support #271

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
HDDTHR opened this issue Mar 6, 2025 · 1 comment
Open

Optimistic Concurrency Control Support #271

HDDTHR opened this issue Mar 6, 2025 · 1 comment
Labels
enhancement Increases software capabilities beyond original client specifications

Comments

@HDDTHR
Copy link

HDDTHR commented Mar 6, 2025

Is your feature request related to a problem? Please describe.
Although the Bulk API supports the if_seq_no and if_primary_term options for each document in the request below, the OpenSearch output plugin doesn't provide an option to set those values for the request.

POST _bulk
{"index":{"_index":"some-index","_id":"1", "if_seq_no": 100 "if_primary_term": 20 }}
{"amount": 500 }

Describe the solution you'd like
Adding two new options to support Optimistic Concurrency Control for the output plugin. It could look like this.

        opensearch {
            id => "os-output"
            hosts => "..."
            user => "..."
            password => "..."
            index => "some-index"
            document_id => "..."
            sequence_number => "[@metadata][_seq_no]"
            primary_term => "[@metadata][_seq_no]"
        }

Describe alternatives you've considered
I had to figure out a workaround, where I fetch the _version field alongside the document from OpenSearch, manually increment it by 1, and configure my output plugin as shown below.

        opensearch {
            id => "os-output"
            hosts => "..."
            user => "..."
            password => "..."
            index => "some-index"
            document_id => "..."
            version => "%{[@metadata][_version]}"
            version_type => "external"
        }

While this does work, I would rather not have to manage my version externally (and introduce unnecessary points of failure), as the only logic I need is to update the document in OpenSearch if there hasn't been any updates while the pipeline is running.

@HDDTHR HDDTHR added the enhancement Increases software capabilities beyond original client specifications label Mar 6, 2025
@krisfreedain
Copy link
Member

Catch All Triage - 1 2

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Increases software capabilities beyond original client specifications
Projects
None yet
Development

No branches or pull requests

2 participants