Unit testing framework

[awvwgk]

I seems like we will stay with my (temporary) testing framework for now. I'm actually surprised that this idea worked out.

Nevertheless, we should discuss if there might be better alternatives or in case we want to stay with the current model, how it should behave as we implement more test suites. If we want to allow preprocessor or external (non-Fortran) dependencies we have a much wider range of testing frameworks available.

Regarding the current behaviour of the testing framework:

• one executable for all test suites to reduce [[test]] entries in fpm.toml (related to Automatically discover executables & tests #164)
• failing tests within a suite will not cause the testing to halt
  • better overview in a CI run about errors
  • potential for parallelisation over a test suite
• a failing test suite will halt the testing framework
  • not sure about this one (any opinions or suggestions?)
  • will be changed by Allow selective testing of single suites and tests #177
• array constructor to register unit tests (and also test suites)
  • name and procedure pointer required (could be redundant)
  • could become fragile for very large test suites (compiler limits for array constructors)

[milancurcic]

I think your framework does exactly what we need and stays out of the way. Good to have this thread open for discussion, but unless concrete problems arise I don't think we need to look further for now.

[everythingfunctional]

I was going to suggest vegetables at some point, as it solves those problems. I also like it because it encourages the test descriptions to read more like a requirements specification. Your framework is actually fairly similar in terms of definition/registration of the tests, so I wasn't worried about switching, and if we do need to the transition won't seem very awkward. I also have a tool with it that automates finding all the test suites and constructing the main program.

a failing test suite will halt the testing framework

I wouldn't recommend this. It makes the output of the framework dependent on the order that it runs the tests. If the tests are independent (and they should be), the order that they're run shouldn't have any bearing on the outputs.

could become fragile for very large test suites (compiler limits for array constructors)

I've used vegetables for a project that had hundreds of tests. I don't know if fpm will have more than that, but it seems to me more likely to hit the continuation line limit before any array constructor limit (assuming there is one).

[LKedward]

I'm happy with your framework since it's lightweight and easy to use as a developer.
I will echo the general sentiment here that I prefer testing to continue in the presence of test failures - this is more useful when debugging IMO.

I usually split test suites between multiple executables (hence #164) for two reasons:

1. I can run tests in parallel trivially;
2. Uncaught fatal errors don't stop other suites from running (granted, we shouldn't have any uncaught failures here).

With that said, I don't mind the current single-executable approach since our tests don't have any significant runtime.

[awvwgk]

Agreed, a failing test suite should not stop the testing. I'll address this in #177.

[arjenmarkus]

In my Ftnunit unit test package I use a small batch file/shell script to repeatedly run the test executable in case of run-time failures. It relies on a small file being updated and read before each test to see if that test needs to be run or has already been run. The problem is that if a test leads to a crash of the program, you simply cannot continue and this mechanism works around that. Op wo 16 sep. 2020 om 11:01 schreef Sebastian Ehlert < [email protected]>:

…

Agreed, a failing test suite should not stop the testing. I'll address this in #177 <#177>. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#176 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAN6YR6FY65K6RHDZTADFPTSGB5FLANCNFSM4RNZK6LQ> .

[awvwgk]

In case of a test crashing we would have to run it with gdb or valgrind to debug anyway. This is not that straight-forward with fpm right now, but would probably use a similar mechanism as an external script to launch tests.

We could have something like the build-script in the library section to support more elaborated testing frameworks not only for this project but for all fpm packages:

[[test]]
name = "tester"
build-script = "collect-tests"
test-script = "run-tests"

fpm test would than call <test-script> <name> instead of just the binary.

[certik]

I am ok with any testing framework for fpm itself. The current one is fine with me.

As long as fpm test works with any testing framework. I assume we all agree on that.

[LKedward]

Related to fortran-lang/stdlib#162 and fortran-lang/stdlib#345, I am wondering whether there is motivation to extract the lightweight unit testing framework as a separate fpm package?

I liked it enough to incorporate a version of it in my fhash package, and @awvwgk is similarly maintaining a version in mctc-lib. With a cmake configuration it could also be adopted by stdlib. What do people think?

[awvwgk]

The mctc library already offers a CMake integration for the testing framework here to allow parallelisation over the testsuites and the individual unit tests. I think neither fpm nor stdlib want to depend on a computational chemistry IO library for testing, so spinning of a new project is the way to go.

I didn't made it its own project yet because it's an awful lot of work to maintain those and most of my use cases are covered with the mctc library now. If you are interested in co-maintaining such a project under the @fortran-lang namespace, I'll setup a project for this purpose.

[LKedward]

Awesome. Yep I'm happy to help co-maintain this. Having it under the fortran-lang namespace seems like a good idea as it is used by fpm and it would hopefully encourage people to write tests for their new fpm packages!

[awvwgk]

I separated the testing framework from the mctc library to the repository here https://github.com/awvwgk/test-drive. It is currently very rudimentary but should already be working more or less.

[certik]

Awesome, thanks! Fpm is under MIT license, is there a reason to change the tests to to Apache? It's probably a relatively minor issue, but I don't have that much experience with Apache. Also I feel the less licenses types we use under fortran-lang, the better.

[awvwgk]

I prefer Apache over other permissive licenses because it actually contains clauses that cover contributions to the project and offers some basic protection for the copyright holder, which is the basic minimum I'm expecting from a open source license I'm putting on a project I'm personally responsible for.

[certik]

All right then. I think Apache is fine. You should also ask anybody who contributed to this code if they are ok with the license change.

[awvwgk]

No worries there, the relevant part in fpm is still in its original form: https://github.com/fortran-lang/fpm/blame/master/fpm/test/fpm_test/testsuite.f90, but it was not used to spin off the test-drive project in the first place anyway.