Increase instance size used for RHEL tests [DI-435] #917
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The RHEL smoke test regularly fails due to: > error: timed out waiting for the condition on pods/test-13631969388-1-17-hazelcast-enterprise-mancenter-0 After extensive investigation, I _believe_ the root cause is an inadequate instance size. Specifically, my hypothesis is: - MC [regularly takes ~30 seconds to start, even if ultimately successful](https://github.com/hazelcast/hazelcast-docker/actions/runs/13631962459) on our current instances - locally this is <5 seconds - In the Helm chart, [we allow 30 seconds before beginning liveness probes](https://github.com/hazelcast/charts/blob/7cf90413100187335332ebccefd58781873fd696/stable/hazelcast-enterprise/values.yaml#L499) - If a liveness probe fails (i.e. it's still starting), the instance is restarted - This leads to regular MC test instance restarts - normally eventually _an_ invocation starts up quick enough, but sometimes not. Due to this transient behaviour it's difficult to be certain, but tested: - with the existing runner, [failed after 3 re-runs](https://github.com/hazelcast/hazelcast-docker/actions/runs/13783658071/job/38574527022) - with a faster runner, [did not fail _with this error_ after 8 re-runs](https://github.com/hazelcast/hazelcast-docker/actions/runs/13792764938) Fixes: [DI-435](https://hazelcast.atlassian.net/browse/DI-435) Post-merge: - [ ] backport - [ ] retag
nishaatr
approved these changes
Mar 12, 2025
ldziedziul
approved these changes
Mar 14, 2025
JackPGreen
deleted the
Increase-instance-size-used-for-RHEL-tests-DI-435]
branch
This was referenced Mar 14, 2025
JackPGreen
added a commit
that referenced
this pull request
Mar 14, 2025
JackPGreen
added a commit
that referenced
this pull request
Mar 14, 2025
JackPGreen
added a commit
that referenced
this pull request
Mar 14, 2025
JackPGreen
added a commit
that referenced
this pull request
Mar 14, 2025
JackPGreen
added a commit
that referenced
this pull request
Mar 14, 2025
JackPGreen
added a commit
that referenced
this pull request
Mar 14, 2025
JackPGreen
added a commit
that referenced
this pull request
Mar 14, 2025
This reverts commit dc05127.
JackPGreen
added a commit
that referenced
this pull request
Mar 14, 2025
Reverts #917 Despite testing being unable to reproduce the issue, as soon as merged [it failed in the same way as before](https://github.com/hazelcast/hazelcast-docker/actions/runs/13858020706) showing this fix does nothing.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The RHEL smoke test regularly fails due to:
After extensive investigation, I believe the root cause is an inadequate instance size.
Specifically, my hypothesis is:
Due to this transient behaviour it's difficult to be certain, but tested:
Fixes: DI-435
Post-merge: