Manual testing of Release 2022.12.1 Part 2

[alisevych]

Initial set-up

Check that the IntelliJ Idea UTBot plugin can be successfully installed

• Choose appropriate build
  01/03 New build taken
• Download plugin
• Check downloaded zip-file size < 100 MB
  It is 101 MB
• Open IntelliJ IDE 2022.2
• Remove previously installed UTBot plugin
• Clone or reuse UTBot project (https://github.com/UnitTestBot/UTBotJava.git)
• Open the project in the IDE
• Install the downloaded plugin

JDK <-> Framework Pair:

All possible combinations of JDK and Framework are selected.
Project type is iterated independently.
Parametrization is OFF.

Scenario:

1. Create new project with specified project type, JDK.
2. Remove default test dependency from Maven or Gradle configuration and refresh project structure.
3. Copy paste ... and ... from UTBot project into source code folder.
4. Invoke test generation for ... file with specific Framework - all test frameworks suggested to be installed.
5. Generate tests - tests are generated without error with proper test Framework.
6. Run tests - tests run without error.
7. Invoke test generation again for ..., all except installed framework suggested to be installed.
8. Generate and run tests with the same framework - tests are generated and executed without errors.

Project	JDK	Framework	Sample
Intellij	1.8	JUnit 4	/reflection/NewInstanceExample
Intellij	11	JUnit 4	/substitution/StaticSubstitutionExamples
Intellij	17	JUnit 4	/exceptions/JvmCrashExamples

Settings to check in pairs
add sample name

Retest with the following bug fixes: #1877, #1878 using (this build for ideType=IU](https://github.com/UnitTestBot/UTBotJava/actions/runs/4373176419))

Mocks/Mode pairs
tested with Project, JDK, Framework

Mocks/Mode	Fuzzing + Symbolic	Symbolic 100%	Fuzzing 100%
outside package + static	mock/provider	mock/service	CommonMockExamples
outside class + static	mock/aliasing, MockStaticMethodExample	MockStaticMethodExample	MockStaticMethodExample
outside package + no static	MockStaticMethodExample, MockWithSideEffectExample	algorithms/errors
outside class + no static	MockStaticMethodExample	MockStaticMethodExample	MockStaticMethodExample
no mocks	#1936

Mocks/Project pairs
tested with Project, JDK, Framework

Mocks/Project	Gradle	Maven	IntelliJ
outside package + static	mock/CommonMockExamples	mock/provider
outside class + no static	mock/MockStaticMethodExample	mock/aliasing
no mocks

[alisevych]

Reproducing bug on mock/provider/ProviderImpl:

• HashCode of a mocked object is used in path constraints #286

[alisevych]

New bugs found on mock/provider/ProviderImpl with Mock outside package settings found:

• MockConstruction is mocking only the first constructor #1875
• Use deepEquals to assert Objects if one (or both) of them is a mock #1877
• Remove switch and mockClassCounter in mockConstruction in simple mockConstruction cases #1878
• Incorrect warning about the Destination package #1915

[alisevych]

Reproducing bug:

• com.intellij.util.IncorrectOperationException is thrown down to IDEA #1775

[alisevych]

Verified with Yuri - this is by design.

The first call of constructor is the special case for algorithms/errors/Taxi class.

    ///region Test suites for executable org.utbot.examples.algorithms.errors.Taxi.runTest

    ///region SYMBOLIC EXECUTION: SUCCESSFUL EXECUTIONS for method runTest(int, int[])

    /**
     * @utbot.classUnderTest {@link Taxi}
     * @utbot.methodUnderTest {@link Taxi#runTest(int, int[])}
     * @utbot.iterates iterate the loop {@code for(int i = 0; i < length; i++)} once
     * @utbot.returnsFrom {@code return findAnswer(length, groups);}
     */
    @Test
    @DisplayName("runTest: -> return findAnswer(length, groups)")
    public void testRunTest_IterateForLoop() {
        org.mockito.MockedConstruction mockedConstruction = null;
        try {
            int[] intArray = {};
            java.util.concurrent.atomic.AtomicInteger mockClassCounter = new java.util.concurrent.atomic.AtomicInteger();
            mockedConstruction = mockConstruction(org.utbot.examples.algorithms.Sort.class, (org.utbot.examples.algorithms.Sort SortMock, org.mockito.MockedConstruction.Context context) -> {
                switch (mockClassCounter.get()) {
                    case 0:
                        (when(SortMock.mergeSort(any()))).thenReturn(intArray);
                        break;
                }
                mockClassCounter.getAndIncrement();
            });
            /* WARNING!!! Automatically used "Mockito static mocking" framework for mocking statics
            because execution encountered flaky methods
            To change this behaviour edit [Settings -> UtBot -> Force static mocking] */
            Taxi taxi = new Taxi();
            int[] groups = {1};

            int actual = taxi.runTest(1, groups);

            assertEquals(1, actual);
        } finally {
            mockedConstruction.close();
        }
    }

    /**
     * @utbot.classUnderTest {@link Taxi}
     * @utbot.methodUnderTest {@link Taxi#runTest(int, int[])}
     * @utbot.iterates iterate the loop {@code for(int i = 0; i < length; i++)} once
     * @utbot.returnsFrom {@code return findAnswer(length, groups);}
     */
    @Test
    @DisplayName("runTest: -> return findAnswer(length, groups)")
    public void testRunTest_IterateForLoop_1() {
        org.mockito.MockedConstruction mockedConstruction = null;
        try {
            int[] intArray = {1};
            java.util.concurrent.atomic.AtomicInteger mockClassCounter = new java.util.concurrent.atomic.AtomicInteger();
            mockedConstruction = mockConstruction(org.utbot.examples.algorithms.Sort.class, (org.utbot.examples.algorithms.Sort SortMock, org.mockito.MockedConstruction.Context context) -> {
                switch (mockClassCounter.get()) {
                    case 0:
                        (when(SortMock.mergeSort(any()))).thenReturn(intArray);
                        break;
                }
                mockClassCounter.getAndIncrement();
            });
            /* WARNING!!! Automatically used "Mockito static mocking" framework for mocking statics
            because execution encountered flaky methods
            To change this behaviour edit [Settings -> UtBot -> Force static mocking] */
            Taxi taxi = new Taxi();

            int actual = taxi.runTest(1, intArray);

            assertEquals(1, actual);
        } finally {
            mockedConstruction.close();
        }
    }
    ///endregion

    ///endregion

[alisevych]

Retest

with the latest build from unit-test-bot/rc3102023 branch

Open IntelliJ IDE 2022.2

JDK <-> Framework Pair:

All possible combinations of JDK and Framework are selected.
Project type is iterated independently.
Parametrization is OFF.

Scenario:

1. Create new project with specified project type, JDK.
2. Remove default test dependency from Maven or Gradle configuration and refresh project structure.
3. Copy paste ... and ... from UTBot project into source code folder.
4. Invoke test generation for ... file with specific Framework - all test frameworks suggested to be installed.
5. Generate tests - tests are generated without error with proper test Framework.
6. Run tests - tests run without error.
7. Invoke test generation again for ..., all except installed framework suggested to be installed.
8. Generate and run tests with the same framework - tests are generated and executed without errors.

Project	JDK	Framework	Sample
Intellij	1.8	JUnit 4	mock/CommonMockExamples
Intellij	11	JUnit 4	collections/Maps
Gradle	17	JUnit 4	arrays/ArrayOfArrays

Settings to check in pairs
add sample name

Retest with the following bug fixes: #1877, #1878 using (this build for ideType=IU](https://github.com/UnitTestBot/UTBotJava/actions/runs/4373176419))

Mocks/Mode pairs
tested with Project, JDK, Framework

Mocks/Mode	Fuzzing + Symbolic	Symbolic 100%	Fuzzing 100%
outside package + static	arrays/ArrayOfArrays
outside class + static	collections/Maps
outside package + no static	mock/CommonMockExamples
outside class + no static	MockFinalClassExample
no mocks	MockFinalClassExample

Mocks/Project pairs
tested with Project, JDK, Framework

Mocks/Project	Gradle	Maven	IntelliJ
outside package + static	arrays/ArrayOfArrays	mock/MockReturnExample	mock/MockFinalClassExample
outside class + no static			collections/Maps
no mocks		mock/ MockFinalClassExample

[alisevych]

Reproducing:

• IndexNotReadyException thrown in IDEA with installed UnitTestBot plugin #1236
• Server.EngineProcessModel.generateTestReport is not bound to a protocol #1183

[alisevych]

New bugs:

• import JUnit5 @Timeout is missing #1982
• Several identical cases in switch in MockedConstruction for Random #1983
• Missing SARIF-reports and classes on Problems view #1986
• Identical cases with and without mocking are not minimized #1988
• IllegalArgumentExceptions in Instrumented process for MockReturnObjectExample #1990