Skip to content

Code coverage #167

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tlepoint opened this issue Aug 14, 2017 · 11 comments · May be fixed by #2072
Open

Code coverage #167

tlepoint opened this issue Aug 14, 2017 · 11 comments · May be fixed by #2072
Assignees
Labels
enhancement New feature or request

Comments

@tlepoint
Copy link
Contributor

Add code coverage using coveralls

@tlepoint tlepoint added the enhancement New feature or request label Aug 14, 2017
@tlepoint tlepoint self-assigned this Aug 14, 2017
@dstebila dstebila added the future-work Will not be fixed in current release cycle label Feb 28, 2020
@ajbozarth ajbozarth moved this to Todo in liboqs planning Jul 23, 2024
@aidenfoxivey
Copy link
Contributor

In all honesty, I don't have a tonne of experience with setting up code coverage in non-Golang environments.

I think that https://coveralls.io looks very nice and has a good interface. That said, I'm always a little bit wary to adopt SaaS when there could be a more "long lived" solution. (for example, choosing astyle over clang-format despite astyle existing on Sourceforge)

I was recommended gcov as well. I'm probably going to read the manual for gcov and then give it a go. If anyone has firm opinions and/or recommendations, I would be very interested to hear them.

Seems like a lot of C test coverage tools are proprietary.

@SWilson4 SWilson4 linked a pull request Feb 7, 2025 that will close this issue
7 tasks
@SWilson4 SWilson4 removed the future-work Will not be fixed in current release cycle label Feb 7, 2025
@SWilson4 SWilson4 self-assigned this Feb 7, 2025
@SWilson4 SWilson4 moved this from Todo to In Progress in liboqs planning Feb 7, 2025
@aidenfoxivey
Copy link
Contributor

aidenfoxivey commented Apr 17, 2025

Well, this is a bit later...

Image

(included image is of lcov being run to generate code coverage)

Turns out it wasn't actually that hard. I was just being kind of daft.

@aidenfoxivey
Copy link
Contributor

I'm not 100% sure this actually makes 100% sense, so please do check me if something this diagnostic is showing seems inaccurate.

It seems like coveralls has a nice interface, but it's also just using gcov underneath.

So much for a diverse ecosystem of C code coverage tools! :)

@dstebila
Copy link
Member

Thanks for revisiting this. Is this output for a single execution of test_kem for a BIKE algorithm? Is it able to aggregate coverage across multiple runs?

@aidenfoxivey
Copy link
Contributor

Image

Yeah it appears to be able to do a decent job of aggregating across multiple tests being run. I just need to figure out how to run all the tests haha.

@aidenfoxivey
Copy link
Contributor

Image

also works for AES.

@aidenfoxivey
Copy link
Contributor

Seems to use pytest if I'm not wrong.

@aidenfoxivey
Copy link
Contributor

Image

@aidenfoxivey
Copy link
Contributor

Alright, looks like this is the approximate flow:

# build with --coverage flag
cmake -DCMAKE_BUILD_TYPE=Debug -DCMAKE_C_FLAGS="--coverage" -DCMAKE_CXX_FLAGS="--coverage" ..

# run files
<trying to sort this bit out>

# use lcov to automate running gcov for us
lcov --capture --directory . --output-file coverage.info

# generate an HTML report
genhtml coverage.info --output-directory coverage_html

@SWilson4
Copy link
Member

Hi @aidenfoxivey! I took a stab at this a couple of months ago but haven't had time to properly see it through. Maybe my branch would be helpful to you?

@aidenfoxivey
Copy link
Contributor

That's awesome. Thanks @SWilson4. I'll give that a look.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: In Progress
Development

Successfully merging a pull request may close this issue.

4 participants