Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: more precise and reliable way to force new leaders in tests #10066

Merged
2 commits merged into from
Aug 16, 2022

Conversation

lenaschoenburg
Copy link
Member

Description

This PR changes two things:

  1. after promoting a new leader, we assert that the new leader is as expected
  2. retries the promote request to allow for temporary failures, for example unavailable receivers

Related issues

closes #9921

Definition of Done

Not all items need to be done depending on the issue and the pull request.

Code changes:

  • The changes are backwards compatibility with previous versions
  • If it fixes a bug then PRs are created to backport the fix to the last two minor versions. You can trigger a backport by assigning labels (e.g. backport stable/1.3) to the PR, in case that fails you need to create backports manually.

Testing:

  • There are unit/integration tests that verify all acceptance criterias of the issue
  • New tests are written to ensure backwards compatibility with further versions
  • The behavior is tested manually
  • The change has been verified by a QA run
  • The impact of the changes is verified by a benchmark

Documentation:

  • The documentation is updated (e.g. BPMN reference, configuration, examples, get-started guides, etc.)
  • New content is added to the release announcement
  • If the PR changes how BPMN processes are validated (e.g. support new BPMN element) then the Camunda modeling team should be informed to adjust the BPMN linting.

Please refer to our review guidelines.

This commit changes two things:
1. after promoting a new leader, we assert that the new leader is as
expected
2. retries the promote request to allow for temporary failures, for
example unavailable receivers
@github-actions
Copy link
Contributor

github-actions bot commented Aug 15, 2022

Test Results

   599 files   -    244     599 suites   - 244   21m 40s ⏱️ - 1h 29m 32s
4 319 tests  - 2 297  4 310 ✔️  - 2 295  9 💤  - 2  0 ±0 
4 494 runs   - 2 306  4 485 ✔️  - 2 304  9 💤  - 2  0 ±0 

Results for commit afdc843. ± Comparison against base commit 1f09e85.

♻️ This comment has been updated with latest results.

@lenaschoenburg lenaschoenburg marked this pull request as ready for review August 15, 2022 12:32
Copy link
Member

@ChrisKujawa ChrisKujawa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @oleschoenburg I have some comment see below, please check them before merging :)

.getServer();

Awaitility.await("Promote request is successful")
.pollInterval(Duration.ofMillis(500))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

❓ Any specifc reason for that poll interval?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not at all, 500ms just felt like a reasonable interval before retrying 🤷

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm ok I would just remove it, otherwise it looks like it is important 🤷

@lenaschoenburg
Copy link
Member Author

Thanks @Zelldon!

bors r+

ghost pushed a commit that referenced this pull request Aug 16, 2022
10066: fix: more precise and reliable way to force new leaders in tests r=oleschoenburg a=oleschoenburg


## Description

This PR changes two things:
1. after promoting a new leader, we assert that the new leader is as expected
2. retries the promote request to allow for temporary failures, for example unavailable receivers


## Related issues

<!-- Which issues are closed by this PR or are related -->

closes #9921 



Co-authored-by: Ole Schönburg <[email protected]>
@ghost
Copy link

ghost commented Aug 16, 2022

Build failed:

@lenaschoenburg
Copy link
Member Author

bors retry

@ghost
Copy link

ghost commented Aug 16, 2022

Build succeeded:

@ghost ghost merged commit 2df4fac into main Aug 16, 2022
@ghost ghost deleted the 9921-flaky-force-leader-change branch August 16, 2022 07:40
@remcowesterhoud
Copy link
Contributor

remcowesterhoud commented Jul 12, 2023

Flaky on stable/8.0 still. Let me try backporting

/backport

@backport-action
Copy link
Collaborator

Successfully created backport PR for stable/8.0:

ghost pushed a commit that referenced this pull request Jul 12, 2023
13453: [Backport stable/8.0] Limit error message size in job and incident records r=remcowesterhoud a=nicpuppa

backport #12066 
relates to #11460 

13455: [Backport stable/8.0] fix: more precise and reliable way to force new leaders in tests r=remcowesterhoud a=backport-action

# Description
Backport of #10066 to `stable/8.0`.

relates to #9921

Co-authored-by: camunda\berkay.can <[email protected]>
Co-authored-by: Ole Schönburg <[email protected]>
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Flaky InstallRequestHandlingTest.shouldBeAbleToBecomeLeader
5 participants