Add testing module to allow better structuring of test suites

[awvwgk]

For prior discussions on this topic see #162

This is an attempt to improve the test suite in stdlib. It uses a vendored version of test-drive, the testing framework I wrote for fpm, named stdlib_test for now. It currently has the status of a testing dependency, i.e. the module is available in stdlib's test suite but not part of stdlib's public API, downstream projects cannot use it.

Why vendoring a dependency?

• test-drive can be automatically fetched and included in CMake, meson and fpm projects
• embedding in the manual Makefile turned out to be difficult, I gave up on this after several attempts
• can be devendored and replaced with a proper dependency later (if make is dropped)

I'm not a fan of vendoring dependencies, but I found no better solution given the constraints of stdlib's build systems.

Why not vegetables?

• vegetables can be automatically fetched and included in CMake and fpm projects
• similar issue as before, we can't use it in make
• vendoring is not an option because of its dependencies

I ported the tests for stdlib_ascii module by replacing the check function of stdlib_error and added the test suite for the testing frame itself as well. The former uses the finalizer to check for failed tests while the latter is propagating the error back. Both will correctly report on failure, the latter approach can additionally recover and run the remaining tests as well.

I'm opening this for feedback. Let me know what you think and whether we want to adopt this testing framework.

If we agree on this approach I will move all existing tests in the new format. In a follow-up patch I would try to bundle all tests into a single test executable and allow selection of the individual suites by the framework. This simplifies the logic for setting up the tests and it be easier to adopt fypp in the test suite without over-engineering the CMake macro / make include file to generate the tests.

Output from stdlib_ascii tester

# Testing: ascii
  Starting is_alphanum_short ... (1/38)
       ... is_alphanum_short [PASSED]
  Starting is_alphanum_long ... (2/38)
       ... is_alphanum_long [PASSED]
  Starting is_alpha_short ... (3/38)
       ... is_alpha_short [PASSED]
  Starting is_alpha_long ... (4/38)
       ... is_alpha_long [PASSED]
  Starting is_lower_short ... (5/38)
       ... is_lower_short [PASSED]
  Starting is_lower_long ... (6/38)
       ... is_lower_long [PASSED]
  Starting is_upper_short ... (7/38)
       ... is_upper_short [PASSED]
  Starting is_upper_long ... (8/38)
       ... is_upper_long [PASSED]
  Starting is_digit_short ... (9/38)
       ... is_digit_short [PASSED]
  Starting is_digit_long ... (10/38)
       ... is_digit_long [PASSED]
  Starting is_octal_digit_short ... (11/38)
       ... is_octal_digit_short [PASSED]
  Starting is_octal_digit_long ... (12/38)
       ... is_octal_digit_long [PASSED]
  Starting is_hex_digit_short ... (13/38)
       ... is_hex_digit_short [PASSED]
  Starting is_hex_digit_long ... (14/38)
       ... is_hex_digit_long [PASSED]
  Starting is_white_short ... (15/38)
       ... is_white_short [PASSED]
  Starting is_white_long ... (16/38)
       ... is_white_long [PASSED]
  Starting is_blank_short ... (17/38)
       ... is_blank_short [PASSED]
  Starting is_blank_long ... (18/38)
       ... is_blank_long [PASSED]
  Starting is_control_short ... (19/38)
       ... is_control_short [PASSED]
  Starting is_control_long ... (20/38)
       ... is_control_long [PASSED]
  Starting is_punctuation_short ... (21/38)
       ... is_punctuation_short [PASSED]
  Starting is_punctuation_long ... (22/38)
       ... is_punctuation_long [PASSED]
  Starting is_graphical_short ... (23/38)
       ... is_graphical_short [PASSED]
  Starting is_graphical_long ... (24/38)
       ... is_graphical_long [PASSED]
  Starting is_printable_short ... (25/38)
       ... is_printable_short [PASSED]
  Starting is_printable_long ... (26/38)
       ... is_printable_long [PASSED]
  Starting is_ascii_short ... (27/38)
       ... is_ascii_short [PASSED]
  Starting is_ascii_long ... (28/38)
       ... is_ascii_long [PASSED]
  Starting to_lower_short ... (29/38)
       ... to_lower_short [PASSED]
  Starting to_lower_long ... (30/38)
       ... to_lower_long [PASSED]
  Starting to_upper_short ... (31/38)
       ... to_upper_short [PASSED]
  Starting to_upper_long ... (32/38)
       ... to_upper_long [PASSED]
  Starting to_upper_string ... (33/38)
       ... to_upper_string [PASSED]
  Starting to_lower_string ... (34/38)
       ... to_lower_string [PASSED]
  Starting to_title_string ... (35/38)
       ... to_title_string [PASSED]
  Starting to_sentence_string ... (36/38)
       ... to_sentence_string [PASSED]
  Starting reverse_string ... (37/38)
       ... reverse_string [PASSED]
  Starting to_string ... (38/38)
       ... to_string [PASSED]

Output from stdlib_test tester

# Testing: check
  Starting success ... (1/44)
       ... success [PASSED]
  Starting failure ... (2/44)
       ... failure [EXPECTED FAIL]
  Message: Non-zero exit code encountered
  Starting skipped ... (3/44)
       ... skipped [SKIPPED]
  Message: This test is always skipped
  Starting failure-message ... (4/44)
       ... failure-message [EXPECTED FAIL]
  Message: Custom message describing the error
  Starting failure-with-more ... (5/44)
       ... failure-with-more [EXPECTED FAIL]
  Message: Non-zero exit code encountered
           with an additional descriptive message here
  Starting expression ... (6/44)
       ... expression [PASSED]
  Starting expression-fail ... (7/44)
       ... expression-fail [EXPECTED FAIL]
  Message: Condition not fullfilled
  Starting expression-message ... (8/44)
       ... expression-message [EXPECTED FAIL]
  Message: index("info!", "!") == 0
  Starting expression-with-more ... (9/44)
       ... expression-with-more [EXPECTED FAIL]
  Message: Condition not fullfilled
           index("info!", "?")
  Starting real-single-abs ... (10/44)
       ... real-single-abs [PASSED]
  Starting real-single-rel ... (11/44)
       ... real-single-rel [PASSED]
  Starting real-single-nan ... (12/44)
       ... real-single-nan [EXPECTED FAIL]
  Message: Exceptional value 'not a number' found
  Starting real-single-abs-fail ... (13/44)
       ... real-single-abs-fail [EXPECTED FAIL]
  Message: Floating point value missmatch
           expected 2.00000000 but got 1.00000000 (difference: 1.00000000)
  Starting real-single-rel-fail ... (14/44)
       ... real-single-rel-fail [EXPECTED FAIL]
  Message: Floating point value missmatch
           expected 1.50000000 but got 1.00000000 (difference: 33%)
  Starting real-double-abs ... (15/44)
       ... real-double-abs [PASSED]
  Starting real-double-rel ... (16/44)
       ... real-double-rel [PASSED]
  Starting real-double-nan ... (17/44)
       ... real-double-nan [EXPECTED FAIL]
  Message: Exceptional value 'not a number' found
  Starting real-double-abs-fail ... (18/44)
       ... real-double-abs-fail [EXPECTED FAIL]
  Message: Floating point value missmatch
           expected 2.0000000000000000 but got 1.0000000000000000 (difference: 1.0000000000000000)
  Starting real-double-rel-fail ... (19/44)
       ... real-double-rel-fail [EXPECTED FAIL]
  Message: Floating point value missmatch
           expected 1.5000000000000000 but got 1.0000000000000000 (difference: 33%)
  Starting integer-char ... (20/44)
       ... integer-char [PASSED]
  Starting integer-char-fail ... (21/44)
       ... integer-char-fail [EXPECTED FAIL]
  Message: Integer value missmatch
           expected -4 but got 3
  Starting integer-char-message ... (22/44)
       ... integer-char-message [EXPECTED FAIL]
  Message: Actual value is not seven
  Starting integer-char-with-more ... (23/44)
       ... integer-char-with-more [EXPECTED FAIL]
  Message: Integer value missmatch
           expected 3 but got 0
           with an additional descriptive message here
  Starting integer-short ... (24/44)
       ... integer-short [PASSED]
  Starting integer-short-fail ... (25/44)
       ... integer-short-fail [EXPECTED FAIL]
  Message: Integer value missmatch
           expected -4 but got 3
  Starting integer-short-message ... (26/44)
       ... integer-short-message [EXPECTED FAIL]
  Message: Actual value is not seven
  Starting integer-short-with-more ... (27/44)
       ... integer-short-with-more [EXPECTED FAIL]
  Message: Integer value missmatch
           expected 3 but got 0
           with an additional descriptive message here
  Starting integer-default ... (28/44)
       ... integer-default [PASSED]
  Starting integer-default-fail ... (29/44)
       ... integer-default-fail [EXPECTED FAIL]
  Message: Integer value missmatch
           expected -4 but got 3
  Starting integer-default-message ... (30/44)
       ... integer-default-message [EXPECTED FAIL]
  Message: Actual value is not seven
  Starting integer-default-with-more ... (31/44)
       ... integer-default-with-more [EXPECTED FAIL]
  Message: Integer value missmatch
           expected 3 but got 0
           with an additional descriptive message here
  Starting integer-long ... (32/44)
       ... integer-long [PASSED]
  Starting integer-long-fail ... (33/44)
       ... integer-long-fail [EXPECTED FAIL]
  Message: Integer value missmatch
           expected -4 but got 3
  Starting integer-long-message ... (34/44)
       ... integer-long-message [EXPECTED FAIL]
  Message: Actual value is not seven
  Starting integer-long-with-more ... (35/44)
       ... integer-long-with-more [EXPECTED FAIL]
  Message: Integer value missmatch
           expected 3 but got 0
           with an additional descriptive message here
  Starting logical-default-true ... (36/44)
       ... logical-default-true [PASSED]
  Starting logical-default-false ... (37/44)
       ... logical-default-false [PASSED]
  Starting logical-default-fail ... (38/44)
       ... logical-default-fail [EXPECTED FAIL]
  Message: Logical value missmatch
           expected F but got T
  Starting logical-default-message ... (39/44)
       ... logical-default-message [EXPECTED FAIL]
  Message: Logical value is not true
  Starting logical-default-with-more ... (40/44)
       ... logical-default-with-more [EXPECTED FAIL]
  Message: Logical value missmatch
           expected F but got T
           with an additional descriptive message
  Starting character ... (41/44)
       ... character [PASSED]
  Starting character-fail ... (42/44)
       ... character-fail [EXPECTED FAIL]
  Message: Character value missmatch
           expected 'negative' but got 'positive'
  Starting character-message ... (43/44)
       ... character-message [EXPECTED FAIL]
  Message: Character string should be negative
  Starting character-with-more ... (44/44)
       ... character-with-more [EXPECTED FAIL]
  Message: Character value missmatch
           expected 'negative' but got 'positive'
           with an additional descriptive message
# Testing: select
  Starting always-pass ... (1/6)
       ... always-pass [PASSED]
  Starting always-fail ... (2/6)
       ... always-fail [EXPECTED FAIL]
  Message: Always failing test
  Starting run-good-suite ... (3/6)
       ... run-good-suite [PASSED]
  Starting run-bad-suite ... (4/6)
       ... run-bad-suite [PASSED]
  Starting run-selected ... (5/6)
       ... run-selected [PASSED]
  Starting select-missing ... (6/6)
       ... select-missing [PASSED]

[milancurcic]

Thank you for starting this, I think it has been much needed for stdlib.

[awvwgk]

We already have a sizeable test suite in stdlib, porting all tests will take a while. I'll keep this list here up-to-date with the state of this patch.

• src/tests/ascii/test_ascii.f90 (1017 956 lines)
• src/tests/bitsets/test_stdlib_bitset_64.f90 (743 613 lines)
• src/tests/bitsets/test_stdlib_bitset_large.f90 (1468 1127 lines)
• src/tests/io/test_loadtxt.f90 (63 lines)
• src/tests/io/test_loadtxt_qp.f90 (30 lines)
• src/tests/io/test_open.f90 (99 lines)
• src/tests/io/test_parse_mode.f90 (187 lines)
• src/tests/io/test_savetxt.f90 (119 lines)
• src/tests/io/test_savetxt_qp.f90 (63 lines)
• src/tests/linalg/test_linalg.f90 (581 714 lines)
• src/tests/logger/test_stdlib_logger.f90 (845 lines)
• src/tests/math/test_linspace.f90 (422 lines)
• src/tests/math/test_logspace.f90 (272 lines)
• src/tests/math/test_math_arange.f90 (86 lines)
• src/tests/math/test_stdlib_math.f90 (223 lines)
• src/tests/optval/test_optval.f90 (376 473 lines)
• src/tests/quadrature/test_gauss.f90 (533 lines)
• src/tests/quadrature/test_simps.f90 (505 lines)
• src/tests/quadrature/test_trapz.f90 (236 lines)
• src/tests/sorting/test_sorting.f90 (939 lines)
• src/tests/stats/test_corr.f90 (387 lines)
• src/tests/stats/test_cov.f90 (614 lines)
• src/tests/stats/test_distribution_PRNG.f90 (108 lines)
• src/tests/stats/test_mean.fypp (55 393 lines)
• src/tests/stats/test_mean_f03.fypp (302 lines)
• src/tests/stats/test_median.fypp (234 lines)
• src/tests/stats/test_moment.f90 (712 lines)
• src/tests/stats/test_rawmoment.f90 (624 lines)
• src/tests/stats/test_var.f90 (463 lines)
• src/tests/stats/test_varn.f90 (359 lines)
• src/tests/string/test_string_assignment.f90 (72 117 lines)
• src/tests/string/test_string_derivedtype_io.f90 (80 122 lines)
• src/tests/string/test_string_functions.f90 (518 689 lines)
• src/tests/string/test_string_intrinsic.f90 (489 692 lines)
• src/tests/string/test_string_match.f90 (72 118 lines)
• src/tests/string/test_string_operator.f90 (123 186 lines)
• src/tests/string/test_string_strip_chomp.f90 (110 216 lines)
• src/tests/string/test_string_to_string.f90 (176 lines)
• src/tests/system/test_sleep.f90 (33 lines)
• src/tests/test/test_check.f90 (886 lines)
• src/tests/test/test_select.f90 (171 lines)
• total (16153 lines)

[jvdp1]

Thanks @awvwgk . This is highly needed, indeed!
If everybody agree on this approach, I would be happy to help you to move all tests.

[milancurcic]

@awvwgk as an exercise and if you agree, I'd like to tackle all the tests for the io module.

[awvwgk]

@milancurcic @jvdp1 thanks for the offer, feel free to send a PR against my fork, but please ping me there or I might overlook it.

[jvdp1]

I will give it a start.

@awvwgk : is it better to keep all stats functions in different tests? Or should I merge them in a single program?

[awvwgk]

is it better to keep all stats functions in different tests? Or should I merge them in a single program?

The testing framework should encourage writing smaller tests, such that one unit test is only testing one thing.

For now I suggest the following strategy:

1. transform internal procedures to module procedures, add an allocatable error_type to their interface
2. create a collect_* subroutine to register all unit tests, export only the collect_* subroutine as public
3. create a minimal driver running only the single testsuite
   (see test-drive Readme or copy the driver from an already converted suite)
4. replace all check calls from stdlib_error with the ones from stdlib_test, relevant signature change:
   • first argument is the error_type
   • remove warn argument
   • msg= keyword argument becomes positional argument after testing condition
   • keep the condition our use a more specific check procedure
     (see test-drive Readme for all available ones)
5. run the testsuite, everything should compile and pass now
6. add if (allocated(error)) return between the checks or split the unit tests to smaller checks

Don't spend too much time on step 3, we will refactor all driver programs at a later point to become more flexible.

[jvdp1]

@awvwgk , an additional question: it seems that it is not possible to check if 2 (float) arrays are the same. should I use the boolean check? Or should I extend check_float to higher dimensions?

[awvwgk]

an additional question: it seems that it is not possible to check if 2 (float) arrays are the same. should I use the boolean check? Or should I extend check_float to higher dimensions?

Using a boolean check for now seems best. I prefer to only change one thing at a time to keep patches small. This one is already a huge patch and we still have to go through all the tests, therefore I would defer implementing new features in the testing framework at the moment.

I was planning to transfer the test-drive project to @fortran-lang when it is used in stdlib and fpm. Than we can develop its capabilities further. Checks on arrays are a bit more difficult because there are different possible ways to define a threshold, therefore I would like to discuss this first before implementing something ad-hoc.

[awvwgk]

The following tests can't be ported, because they are not thread-safe:

• src/tests/logger/test_stdlib_logger.f90 depends on a global state since it is using the global logger
• src/tests/system/test_sleep.f90 requires a separate process with timeout limit

[milancurcic]

For the logger we can use the local logger instance, no? I don't know what to do about the other one, but it wouldn't be the end of the world if we omitted a test or two.

[awvwgk]

For the logger we can use the local logger instance, no? I don't know what to do about the other one, but it wouldn't be the end of the world if we omitted a test or two.

It might be possible to rewrite the tests with a local logger instance. The main difficulty is to figure out whether a test depends on the final state generated by the previous test at the moment. This did cost me quite a while in the bitsets test, which saved a few lines of initialization every test by exploiting, that a previous test already initialized a global variable.

[milancurcic]

I opened a PR with io tests here: awvwgk#87 (just in case there were no notifications, I'm new to this workflow).

The tests there won't pass with ifort until #506 is merged. I think there are two ways forward:

1. Temporarily disable test_loadtxt and test_savetxt and merge Port stdlib_io tests to test-drive awvwgk/fortran-stdlib#87. Merge this into master when ready. Then merge Portable savetxt and loadtxt #506 with tests enabled (pending the approval of the fix) into master.
2. Merge Portable savetxt and loadtxt #506 into Port stdlib_io tests to test-drive awvwgk/fortran-stdlib#87, then merge that here. Once done, merge into master.

[awvwgk]

I think it is best to target #506 against this branch, since this PR is anyway quite large already and touching a lot of things. We can discuss the change in #506 first to give it a proper review, but since it is fixing an actual bug, I think we can move it forward a little bit faster.

[milancurcic]

I opened awvwgk#90 which addresses the failing ifort tests.

[milancurcic]

That's fine, you ported so many already.

[awvwgk]

I think there is some value in getting this PR merged at the current stage, since it will make the testing framework available, port the largest share of the test and allow us to port the rest of the tests without risking merge conflicts by making smaller PRs.

From my side this PR is complete and ready for review. As described above, I have plans for following up once it is merged.

I can update the PR before or after the review, that is up to you. But I don't want to do it twice, because it takes at least an hour of work to check all the commits when rebasing.

[certik]

I think the main change in this PR, to move to a unit testing framework, is fine.

The main changes in a typical test seems minor and also quite minimal.

So unless anybody has any objections, I think why don't we merge this. That is, let's bring this PR to merge cleanly and all tests pass. The rest of the tests do not have to be ported, they can just continue running the old way. I think both ways can coexist. And then we just port the tests as we have time once this PR is merged.

[awvwgk]

I'm all for moving forward here. However, the list of things to keep in mind and fix has been growing week by week now. I think we also have a hard blocker, which requires some work first: #534, since many tests are now moved to using fypp.

I can for sure dedicate an evening/night to update this branch. I'm currently just waiting for a clear signal here on what to do next.

[milancurcic]

I was hoping to get all initial tests ported since we're so close, but this "bar" is arbitrary. I don't see a good reason not to merge this as is, pending resolved conflicts. @awvwgk let me know if you'd like me to help with resolving outstanding conflicts.

[jvdp1]

I was hoping to get all initial tests ported since we're so close, but this "bar" is arbitrary. I don't see a good reason not to merge this as is, pending resolved conflicts. @awvwgk let me know if you'd like me to help with resolving outstanding conflicts.

Is the list in this comment up-to-date? If yes, I will try to find some times to move some of the remaining tests. Otherwise, we can always open an issue and assign it to me for later.

[milancurcic]

@awvwgk I brought this branch up to date with master and resolved conflicts.

There's an outstanding issue with fpm deployment that I need some advice on. The issue currently is that stdlib_test.fypp which provides the test-drive to stdlib, is located in src/tests so it's not picked up by the fpm-deployment.sh:

stdlib/ci/fpm-deployment.sh

Lines 34 to 39 in d14fca8

	# Preprocess stdlib sources
	find src -maxdepth 1 -iname "*.fypp" \
	| cut -f1 -d. | xargs -P "$njob" -I{} "$fypp" "{}.fypp" "$destdir/{}.f90" $fyflags
	# Collect stdlib source files
	find src -maxdepth 1 -iname "*.f90" -exec cp {} "$destdir/src/" \;

so when the CI runs fpm test we get:

<ERROR>*cmd_run*:targets error:Unable to find source for module dependency: "stdlib_test" used by "test/test_check.f90"

I see a few solutions:

1. Set -maxdepth 2 so that the script picks up src/tests/stdlib_test.{fypp,f90}; I think this is the simplest "hack" around this problem.
2. Move stdlib_test.fypp from src/tests to src. This goes against your original rationale to make this internal to stdlib and not part of the public API.
3. Do not include stdlib_test.fypp in the fpm deployment, but add a dependency on fortran-lang/test-drive in fpm.toml, assuming the API is the same between test-drive and stdlib_test.fypp. This would of course, require renaming all imports in current tests from stdlib_test to testdrive.

I think 3rd is the best approach but a little bit more work. What do you think?

And if so, should we sort that out in this PR, or do a quick fix here (option 1) and then have a separate PR to switch to test-drive as an fpm dependency?

[awvwgk]

Let's go with option 3, I opened fortran-lang/test-drive#6 to make test-drive aware of quadruple precision and started working on a solution in awvwgk#103.

For using test-drive directly we have to:

• allow make to dynamically fetch the test-drive source
• allow cmake to dynamically fetch or find a system package to make test-drive available
• update the fpm generator to depend on test-drive

[awvwgk]

We might have to increase the threshold for the sleep test on MacOS a bit further:

      Start 37: sleep
37/46 Test #37: sleep ............................***Failed    0.27 sec
# Testing: sleep
  Starting sleep ... (1/1)
       ... sleep [FAILED]
  Message: Floating point value missmatch
           expected 100.00000000000000 but got 230.29599999999999 (difference: 130%)
1 test(s) failed!

[jvdp1]

OK for me as it is currently.
I have no time now to move the remaining tests on stats. An issue might be assigned to me when merging this PR.

[awvwgk]

Many thanks to everybody who contributed to this project. To move forward, I'll go ahead and merge this PR now.