Proposal to use Testing Framework

[everythingfunctional]

Rather than do ad-hoc testing of the standard library, we should adopt and use a dedicated testing framework. We'll use this issue to discuss possible candidates, their pros and cons, and what requirement we might have for choosing one.

[everythingfunctional]

I'm going to firstly suggest my library, vegetables.

Cons:

• It comes with some additional dependencies

Pros:

• As I'm also working on fpm, and the long term goal is for us to start using it, I will likely port it to fpm, alleviating the above con

I'm open to suggestions about others though.

[certik]

I think we need a testing framework. We should integrate it with fpm, so that things just work. Dependencies will then not be a problem.

In fact, I think Cargo allows you to use any testing framework you want. One simply specifies it in [dev-dependencies]. So we should design fpm in the same way. That way people can use any testing framework they like friction free.

So we can use vegetables. The issue I have is with the API. See:

#121 (comment)

I don't like assertEquals. I think Fortran's conventions are to use something like assert_equal or assert_eq, see STYLE_GUIDE.md#variable-and-procedure-naming.

@everythingfunctional would you be willing to consider using such names, assuming most people would agree?

[milancurcic]

Of all the testing frameworks I looked at, I like vegetables the best, minus the camelCase but that's a nit pick :).

What I'm still confused about is where do you write actual tests. Brad wrote here that testing should be done from the outside. Concretely, what does that mean for stdlib?

1. Tests are written in a dedicated project stdlib-tests which has vegetables as a dependency?
2. Tests are written in fpm which has vegetables as a dependency?
3. Tests are written in vegetables?

Of the three, only the 1st makes some sense to me. Of course, the most sensible solution to me is to write tests in stdlib, and then the framework is a dependency there (if pure Fortran or C, not a biggie). But I got the impression that that's not what you're trying to do.

[certik]

I propose to do what Rust / Cargo does (as well as Haskell / Stack). Have a tests directory, and each file there gets compiled to a binary. That file internally can import many other modules and use a test framework to execute tests.

Regarding dependencies, fpm build will only build stdlib without tests, so no dependency on vegetables. fpm test builds both stdlib and the tests, and this command will pull in vegetables as a [dev-dependencies] item.

[everythingfunctional]

Like @certik said. The tests and library are in the same project, just kept in separate folders. The convention fpm is leaning towards is library code goes in src and test code goes in tests.

[everythingfunctional]

@certik , I could be convinced to switch styles. Do you know if there is any kind of consensus from the community on that? Is camel case really frowned on in Fortran?

[milancurcic]

The tests and library are in the same project, just kept in separate folders. The convention fpm is leaning towards is library code goes in src and test code goes in tests.

Okay, I was confused with what you wrote before. So, to confirm, stdlib (and likewise, any other project) would have:

stdlib/
  src/
  tests/

and tests are not built within stdlib itself, only by fpm test. But each project carries its test sources. Correct?

[certik]

@milancurcic correct. This would be the standard layout as expected by fpm that works by default. Non-standard layout can still be used (by explicitly telling fpm in fpm.toml), but generally discouraged.

@everythingfunctional

Do you know if there is any kind of consensus from the community on that? Is camel case really frowned on in Fortran?

Let's discuss at j3-fortran/fortran_proposals#168, so that we keep discussions focused and at one place.

[aradi]

I just wanted to let you know, that I once also wrote a simple unit testing framework, which is based on the Fypp preprocessor. FyTest is in spirit similar to the Google-test framwork, leveraging the power of Fypp. It is a header-only library (it is a single Fypp-include file), so it does not add any external dependencies. It offers unit tests, test suites, fixtures and parameterized tests. Unfortunately, it is not that well documented yet, and has not been used by many projects yet, so it far less tested as Fypp. For a first impression, see those examples.

Please note, that I am the author of FyTest, but I am not pushing in any sense for its use. FyTest was just a toy project of mine, and Vegetables is for sure much-much more mature. But FyTest represents a slightly different testing approach, and I wanted to offer the opportunity to discuss about the approach first.

[awvwgk]

Having contributed some patches to stdlib now, I found writing the unit testing terrible cumbersome every time. Even writing the specs is fun compared to creating tests... Failing a test is annoying because I just get a meaningless error stop about a failed check instead of some hint which test of the plenty internal procedures just broke.

I think it is urgent to find some solution for the unit testing. Vegetables would be nice if upstream could provide some CMake integration (I will even volunteer to write the CMake for this!). There is also a cleaned up version of fytest @aradi recently added for @dftbplus, which could also be an option, check this file for an example. Or I can put together a small framework like the one I wrote for fpm (the latest iteration is integrated here) and we just add it as special testing only object library here.

[milancurcic]

Failing a test is annoying because I just get a meaningless error stop about a failed check instead of some hint which test of the plenty internal procedures just broke.

Have you tried giving it a message and letting check not fail but warn only? (spec and examples)

call check(a == 5, 'a == 5 failed.', warn=.true.)

I found that it works well and covers most of my needs.

But I agree that some minimal framework would be helpful. I like your small framework best because it's relatively simple and gives us everything we need.

[everythingfunctional]

I've used Vegetables in some CMake projects, as a git submodule and just written the CMake file in the parent project. I could be convinced to add support for CMake to Vegetables if there is a desire. Can you provide some more details about how you would expect to consume it? I.e. as a git submodule, a CMake superbuild, something else?

[awvwgk]

Have you tried giving it a message and letting check not fail but warn only?

Fair enough, didn't consider this usage yet. Still this won't cover all needs for testing, because I only get a printout and no programmatic access to the error.

Can you provide some more details about how you would expect to consume it?

FetchContent is usually the way to go:
https://github.com/fortran-lang/stdlib-cmake-example/blob/6a67d6e/subprojects/CMakeLists.txt#L7-L20

But this still means I have to handle the dependencies in vegetables myself downstream (even if it is just iso_varying_string) and maintaining the build system externally from the source is a bit fragile IMO.

[everythingfunctional]

FetchContent is usually the way to go:
https://github.com/fortran-lang/stdlib-cmake-example/blob/6a67d6e/subprojects/CMakeLists.txt#L7-L20

Ok, that makes sense and should be easy enough to support.

But this still means I have to handle the dependencies in vegetables myself downstream (even if it is just iso_varying_string) and maintaining the build system externally from the source is a bit fragile IMO.

Agreed. It's certainly not the ideal situation.

I'll try and spend some time in the next day or two seeing if I can figure out how to make it work.

[everythingfunctional]

@awvwgk , I think I've got it set up so it will work based on the example you gave. Can you take a look here, here and here and let me know what you think? If it will work for you I'll get stuff merged in with the appropriate git tags and version numbers.

[awvwgk]

Thanks for taking on the effort, I'm running into issues with Intel and coarray support when building vegetables and are also seeing some install instructions bleeding through into dependent projects.

[awvwgk]

@everythingfunctional I think I figured out most of the issues, I put together some patches on the merge requests.

It's simply astonishing to see that even the most basic features have to be implemented every time from scratch in CMake, I'm really looking forward to see how we get this insanity encoded into the fpm CMake backend.

[everythingfunctional]

@awvwgk , it should all be usable now. Let me know if you run into more issues.

[awvwgk]

I opened #494 to integrate a version of test-drive into stdlib. I added some notes about using an external testing framework there as well, in short the manual make build system is really narrowing down the options we have available.