Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeout tests are sometimes generated by Fuzzing for simple functions #2262

Open
alisevych opened this issue Jun 6, 2023 · 1 comment
Open
Assignees
Labels
comp-fuzzing Issue is related to the fuzzing comp-instrumented-process Issue is related to Instrumented process ctg-bug Issue is a bug lang-java Issue is related to Java support spec-false-positive False-positive error test is generated

Comments

@alisevych
Copy link
Member

alisevych commented Jun 6, 2023

Description

Timeout tests are sometimes generated by Fuzzing for simplest functions.
For example for a+b

To Reproduce

  1. Install UnitTestBot plugin built from main in IntelliJ IDEA
  2. Open/create a project (Maven with JDK 17 was used)
  3. Generate tests for the following class with default settings:
public class Addition {
    public int addition(int a, int b){
        return a + b;
    }
}

Expected behavior

One test is expected to be generated.

Actual behavior

Once there was also a timeout test generated by Fuzzing.

Screenshots, logs

import org.junit.jupiter.api.DisplayName;
import org.junit.jupiter.api.Test;

import java.time.Duration;
import java.util.concurrent.TimeUnit;

import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertTimeoutPreemptively;

public final class AdditionTest {
    ///region Test suites for executable org.example.Addition.addition

    ///region SYMBOLIC EXECUTION: SUCCESSFUL EXECUTIONS for method addition(int, int)

    /**
     * @utbot.classUnderTest {@link Addition}
     * @utbot.methodUnderTest {@link Addition#addition(int, int)}
     * @utbot.returnsFrom {@code return a + b;}
     */
    @Test
    @DisplayName("addition: -> return a + b")
    public void testAddition_ReturnAPlusB() {
        Addition addition = new Addition();

        int actual = addition.addition(-255, -255);

        assertEquals(-510, actual);
    }
    ///endregion

    ///region FUZZER: TIMEOUTS for method addition(int, int)

    /**
     * @utbot.classUnderTest {@link Addition}
     * @utbot.methodUnderTest {@link Addition#addition(int, int)}
     */
    @Test
    @DisplayName("addition: a = -1 (mutated from -4194305), b = -2147483646 (mutated from min)")
    @org.junit.jupiter.api.Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
    public void testAddition() {
        Addition addition = new Addition();
        
        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
        assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> addition.addition(-1, -2147483646));
    }
    ///endregion

    ///endregion
}

There seems to be nothing special in logs with this exact Fuzzing test generation:
utbot-engine-current_part.log
idea_part.log

Environment

IntelliJ IDEA 2023.1 (Community Edition)
Project - Maven
JDK - corretto-17

Additional context

Sometimes reproducing for simple functions of different times.
But has not reproduced on same function in a row.
Probably there is some performance issue with concrete execution.

@alisevych alisevych added ctg-bug Issue is a bug comp-fuzzing Issue is related to the fuzzing comp-instrumented-process Issue is related to Instrumented process lang-java Issue is related to Java support labels Jun 6, 2023
@alisevych alisevych added the spec-false-positive False-positive error test is generated label Jun 7, 2023
@alisevych
Copy link
Member Author

alisevych commented Sep 25, 2023

Reproducing quite regularly on different code.
Now on spring-petclinic simple Entity class Person:

	///region FUZZER: TIMEOUTS for method getFirstName()

	/**
	 * @utbot.classUnderTest {@link Person}
	 * @utbot.methodUnderTest {@link Person#getFirstName()}
	 */
	@Test
	@DisplayName("getFirstName: ")
	@Timeout(value = 1000L, unit = TimeUnit.MILLISECONDS)
	public void testGetFirstName() {
		Person person = new Person();
		person.setFirstName("ZX");
		person.setLastName("10");

        /* This execution may take longer than the 1000 ms timeout
         and therefore fail due to exceeding the timeout. */
		assertTimeoutPreemptively(Duration.ofMillis(1000L), () -> person.getFirstName());
	}
	///endregion

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
comp-fuzzing Issue is related to the fuzzing comp-instrumented-process Issue is related to Instrumented process ctg-bug Issue is a bug lang-java Issue is related to Java support spec-false-positive False-positive error test is generated
Projects
Status: Todo
Development

No branches or pull requests

2 participants