Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Truncate errorMessage to prevent exceeding maxMessageSize #11460

Closed
lenaschoenburg opened this issue Jan 20, 2023 · 3 comments · Fixed by #12066
Closed

Truncate errorMessage to prevent exceeding maxMessageSize #11460

lenaschoenburg opened this issue Jan 20, 2023 · 3 comments · Fixed by #12066
Assignees
@remcowesterhoud @berkaycanbc
Labels
area/reliability Marks an issue as related to improving the reliability of our software (i.e. it behaves as expected) area/ux Marks an issue as related to improving the user experience component/engine kind/bug Categorizes an issue or PR as a bug onboarding version:8.1.15 Marks an issue as being completely or in parts released in 8.1.15 version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0

Comments

@lenaschoenburg
Copy link
Member

Similar to https://github.com/camunda/connector-sdk/issues/345, Zeebe should truncate error messages it produces to a sensible value. This is to prevent extremely large generated error messages from exceeding the maxMessageSize which, for example, can lead to incidents not being created.

For users, a truncated error message is more helpful than getting no response at all.
@lenaschoenburg lenaschoenburg added kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc. area/ux Marks an issue as related to improving the user experience area/reliability Marks an issue as related to improving the reliability of our software (i.e. it behaves as expected) component/engine labels Jan 20, 2023
@remcowesterhoud
Copy link
Contributor

@oleschoenburg Could you explain a situation in which this would occur? Didn't we already fix this with #10402?

@lenaschoenburg
Copy link
Member Author

@remcowesterhoud Maybe? I'm not sure tbh. @saig0 recommended to create an issue for this, maybe he can provide more context.

@saig0
Copy link
Member

saig0 commented Feb 1, 2023

@remcowesterhoud it's about improving the UX in Zeebe. We want to limit the errorMessage in the job and the incident record.

If a client (i.e. a job worker) provides a too-large error message, we still want to fail the job and create an incident. Since it makes not much sense to store a huge error message and it could exceed the batch size, we cant to shrink the error message to a reasonable size.

We raised the issue after seeing that a connector tried to fail a job with a huge error message. By accident, the error message contained a big JSON payload. As a result, the connector was not able to fail the job. See the details here.

This situation could happen to any job worker. Or, even within the engine processing, for example, if an expression failed and we try to create an incident with a huge error message.

@remcowesterhoud remcowesterhoud added kind/bug Categorizes an issue or PR as a bug onboarding and removed kind/toil Categorizes an issue or PR as general maintenance, i.e. cleanup, refactoring, etc. labels Feb 8, 2023
@berkaycanbc berkaycanbc mentioned this issue Mar 20, 2023
Merged
14 tasks
ghost pushed a commit that referenced this issue Mar 23, 2023
@berkaycanbc
merge: #12066
8471221 
12066: Limit error message size in job and incident records r=remcowesterhoud a=berkaycanbc

## Description

Currently if a client (e.g. job worker) provides a too large error message (e.g. stack trace of the error), we try to store the whole provided message. That can cause the overall broker message size (set to 4mb at the moment) to be exceeded. Therefore, we fail to create records at all.

This PR limits the job and incident error message size to 500 characters in order to be able to create job and incident records without hitting the max message size limit.

## Related issues

<!-- Which issues are closed by this PR or are related -->

closes #11460 



Co-authored-by: camunda\berkay.can <[email protected]>
@ghost ghost closed this as completed in 0dae0bd Mar 23, 2023
@npepinpe npepinpe added the version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0 label Apr 5, 2023
@remcowesterhoud remcowesterhoud mentioned this issue Jun 9, 2023
Closed
@nicpuppa nicpuppa mentioned this issue Jul 12, 2023
Merged
ghost pushed a commit that referenced this issue Jul 12, 2023
@berkaycanbc
merge: #13453
2adbcc7 
13453: [Backport stable/8.0] Limit error message size in job and incident records r=remcowesterhoud a=nicpuppa

backport #12066 
relates to #11460 

Co-authored-by: camunda\berkay.can <[email protected]>
@nicpuppa nicpuppa mentioned this issue Jul 12, 2023
Merged
ghost pushed a commit that referenced this issue Jul 12, 2023
@berkaycanbc
merge: #13453
8cb2f84 
13453: [Backport stable/8.0] Limit error message size in job and incident records r=remcowesterhoud a=nicpuppa

backport #12066 
relates to #11460 

Co-authored-by: camunda\berkay.can <[email protected]>
ghost pushed a commit that referenced this issue Jul 12, 2023
@berkaycanbc
merge: #13454
06a6686 
13454: [Backport stable/8.1] Limit error message size in job and incident records r=nicpuppa a=nicpuppa

backport #12066 
relates to #11460 

Co-authored-by: camunda\berkay.can <[email protected]>
ghost pushed a commit that referenced this issue Jul 12, 2023
@berkaycanbc @lenaschoenburg
merge: #13453 #13455
cf4d53e 
13453: [Backport stable/8.0] Limit error message size in job and incident records r=remcowesterhoud a=nicpuppa

backport #12066 
relates to #11460 

13455: [Backport stable/8.0] fix: more precise and reliable way to force new leaders in tests r=remcowesterhoud a=backport-action

# Description
Backport of #10066 to `stable/8.0`.

relates to #9921

Co-authored-by: camunda\berkay.can <[email protected]>
Co-authored-by: Ole Schönburg <[email protected]>
ghost pushed a commit that referenced this issue Jul 12, 2023
@berkaycanbc
merge: #13454
345fe89 
13454: [Backport stable/8.1] Limit error message size in job and incident records r=nicpuppa a=nicpuppa

backport #12066 
relates to #11460 

Co-authored-by: camunda\berkay.can <[email protected]>
ghost pushed a commit that referenced this issue Jul 12, 2023
@berkaycanbc
merge: #13454
718d5a1 
13454: [Backport stable/8.1] Limit error message size in job and incident records r=nicpuppa a=nicpuppa

backport #12066 
relates to #11460 

Co-authored-by: camunda\berkay.can <[email protected]>
@abbasadel abbasadel added the version:8.1.15 Marks an issue as being completely or in parts released in 8.1.15 label Aug 2, 2023
This issue was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@remcowesterhoud remcowesterhoud

@berkaycanbc berkaycanbc

Labels
area/reliability Marks an issue as related to improving the reliability of our software (i.e. it behaves as expected) area/ux Marks an issue as related to improving the user experience component/engine kind/bug Categorizes an issue or PR as a bug onboarding version:8.1.15 Marks an issue as being completely or in parts released in 8.1.15 version:8.2.0 Marks an issue as being completely or in parts released in 8.2.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Limit error message size in job and incident records camunda/camunda
7 participants
@npepinpe @megglos @abbasadel @lenaschoenburg @saig0 @remcowesterhoud @berkaycanbc