Clarify scope of EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS test suite

[alisevych]

Description

Clarify scope of generated suite with EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS.

Expected behavior

Is it correct that only explicitly thrown exceptions without try-catch should be located in this suite?

Environment

Fuzzing is on.

Context

Example:

Test with String.repeat(-1) throws [java.lang.IllegalArgumentException: count is negative: -1]

Expected result

The test should be put into FUZZING: IMPLICITLY_THROWN_UNCHECKED_EXCEPTIONS region.

Actual result

The test is currently put into FUZZING: EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS region.

Originally posted by @alisevych in #1102 (comment)

[amandelpie]

I am happy to discuss about clusters for fuzzing tests bazed on the utExecution.result.

At this moment all unsuccessful fuzzing execution are grouped to the one cluster with the name "EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS", probably the common name is incorrect

Also, I could create a separate clusters for all types of utExecution.result

• UtConcreteExecutionFailure
• UtExplicitlyThrownException
• UtImplicitlyThrownException
• UtOverflowFailure
• UtSandboxFailure
• UtTimeoutException
  But @Markoutte looks like not all results are possible for Fuzzing execution, am I right?

[Markoutte]

If it collects exceptions of any kind, I'd prefer name like "EXCEPTIONS_THROWN" without any details about checked/unchecked.

[korifey]

If it collects exceptions of any kind, I'd prefer name like "EXCEPTIONS_THROWN" without any details about checked/unchecked.

We should at least differentiate Error suite from regression suite

[alisevych]

As a suggestion:
EXPECTED_EXCEPTIONS
UNEXPECTED_EXCEPTIONS

[amandelpie]

@alisevych what do you mean under EXPECTED_EXCEPTIONS and UNEXPECTED_EXCEPTIONS from the following list?
Is it the same as the proposed by @korifey the splitting of exceptions and errors (failures)?

Am I right?

1. UtConcreteExecutionFailure (UNEXPECTED_EXCEPTIONS)
2. UtExplicitlyThrownException (EXPECTED_EXCEPTIONS)
3. UtImplicitlyThrownException (EXPECTED_EXCEPTIONS)
4. UtOverflowFailure (UNEXPECTED_EXCEPTIONS)
5. UtSandboxFailure (UNEXPECTED_EXCEPTIONS)
6. UtTimeoutException (EXPECTED_EXCEPTIONS)

[amandelpie]

@korifey is it correct to say that all result types could be grouped in Errors and Exceptions?

Exceptions:
UtExplicitlyThrownException
UtImplicitlyThrownException
UtTimeoutException

Failures:
UtOverflowFailure
UtSandboxFailure
UtConcreteExecutionFailure

[amandelpie]

@korifey @Markoutte @alisevych

Also as a varinat we could unify it with the clusters produced by the Symbolic Engine.
It has the following logic based on type of exceptions (checked/unchecked)

    is UtExecutionSuccess -> ExecutionGroup.SUCCESSFUL_EXECUTIONS
    is UtImplicitlyThrownException -> if (this.exception.isCheckedException) ExecutionGroup.CHECKED_EXCEPTIONS else ExecutionGroup.ERROR_SUITE
    is UtExplicitlyThrownException -> if (this.exception.isCheckedException) ExecutionGroup.CHECKED_EXCEPTIONS else ExecutionGroup.EXPLICITLY_THROWN_UNCHECKED_EXCEPTIONS
    is UtOverflowFailure -> ExecutionGroup.OVERFLOWS
    is UtTimeoutException -> ExecutionGroup.TIMEOUTS
    is UtConcreteExecutionFailure -> ExecutionGroup.CRASH_SUITE
    is UtSandboxFailure -> ExecutionGroup.SECURITY

[alisevych]

@amandelpie @Markoutte @korifey
Please correct me if needed:

As for Overflow exceptions - they should be treated as errors or ignored - based on the setting:

Also , what is meant by "ignored": just skipped?

From my point of view implicitly thrown (by other classes/packages) and explicitly thrown (by user code) - should be in different suites.
Implicitly - possible errors. Should be added to error suite (?)
Explicitly - expected ones. Should be treated as regression suite - add to successful executions (?)

Sandbox failures should be shown separately - with distinct explanation how to add permissions and regenerate the tests if needed.

Timeout exceptions should be added to error suite.

Concrete execution failures - are UnitTestBot issues. Should be shown in error suite or separately.

[alisevych]

@rudolf101 @zishkaz
Currently generated test cases for Java are divided into the following regions:

<Source name> : <Suite name> for method <Method signature>

You can find some details in PR - #1250.

Also there is Errors report region to collect internal errors from concrete execution (only count, full details should be in logs).

Let's consider and discuss an appropriate solution for JavaScript test cases division into suites and their naming.