Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve sandbox-relative description in generated tests #782

Open
Markoutte opened this issue Aug 24, 2022 · 3 comments
Open

Improve sandbox-relative description in generated tests #782

Markoutte opened this issue Aug 24, 2022 · 3 comments
Assignees
Labels
ctg-enhancement New feature, improvement or change request

Comments

@Markoutte
Copy link
Collaborator

Description

After #625 was implemented methods that try to run some unsafe code now generates into tests like this:

@Disabled("Disabled due to sandbox")
public void testPropertyWithBlankString() {
        SecurityCheck securityCheck = new SecurityCheck();
        
        /* This test fails because method [com.company.security.SecurityCheck.property] produces [java.security.AccessControlException: access denied ("java.util.PropertyPermission" "   " "read")]
            java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
            java.security.AccessController.checkPermission(AccessController.java:886)
            java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
            java.lang.SecurityManager.checkPropertyAccess(SecurityManager.java:1294)
            java.lang.System.getProperty(System.java:719)
            com.company.security.SecurityCheck.property(SecurityCheck.java:32) */
    }

We have several problems with this view:

  1. User have no information how to change settings of security manager
  2. TestNG has no @Disabled annotation
  3. Is it good idea to print stacktrace?

Expected behavior

Generated test should be improved. We can add little doc or even hyperlink to add permissions into correct file.

@Markoutte Markoutte self-assigned this Aug 24, 2022
@korifey korifey moved this to Todo in UTBot Java Aug 24, 2022
@alisevych
Copy link
Member

alisevych commented Aug 24, 2022

@Markoutte @EgorkaKulikov
Could you please also consider the following:

  1. Disabling tests is made in a bit different way in different test frameworks:
    JUnit4 has only @Ignore("<comment>")
    JUnit5 has @Disabled("<comment>")
    TestNG has possibility to disable tests inside @Test( enabled=false ), there can be also description parameter inside, that Summaries is using. So need to render it once and correctly.

  2. How these tests will be rendered when parameterization is on?

@alisevych
Copy link
Member

@Markoutte @EgorkaKulikov
Tested cases from previous comment - tests are rendered correctly.

@alisevych alisevych added the ctg-enhancement New feature, improvement or change request label Sep 26, 2022
@alisevych
Copy link
Member

alisevych commented Sep 26, 2022

"This test fails.." is not a correct note. The test would pass if enabled.
"The test was not generated..." would be more clear. Or "The test generation stopped on unpermitted operation..."
Also it would be great to suggest the user to allow UTBot to generate and run the tests for these unsafe operations - @Markoutte 's idea.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ctg-enhancement New feature, improvement or change request
Projects
Status: Todo
Development

No branches or pull requests

2 participants