File becomes uncovered when using both includeUncoveredFiles and processUncoveredFiles

[Slamdunk]

Q	A
php-code-coverage version	9.1.5
PHP version	7.4.9
Driver	PCOV
Installation Method	Composer
Usage Method	PHPUnit
PHPUnit version (if used)	9.3.7

Hi, I've recently updated from 9.1.4 to 9.1.5 and a file, previously reported as 100% covered, disappeared from CC and is now reported as 0%.

After a git bisect session and few debugging, I've narrowed it down to 7e91099 and both includeUncoveredFiles="true" and processUncoveredFiles="true" set in the <coverage> tag of my phpunit.xml.

If any of the two is changed to false, the CC gets back to 100% again.

The config in question is a test one from ParaTest, so I don't know whether it should or should not be correct to use both these options, but still seems to me an unexpected or undocumented behavior for a patch release.

[sebastianbergmann]

CC @dvdoug

[dvdoug]

includeUncoveredFiles defaults to true, setting processUncoveredFiles to true too should be perfectly fine - php-code-coverages own test suite does that...

A file that was there now has coverage removed? 👀

[Slamdunk]

Not removed, I expressed bad myself: just reported as completely uncovered.

[dvdoug]

So before/after the file is in the report, just after the coverage is 100->0 ? That would be...unexpected from that PR

[Slamdunk]

Exactly. I'd like to prepare a test case so you can debug it better, but I'm not able to do it soon.

Still you can replicate the issue with the following steps:

1. Download ParaTest https://github.com/paratestphp/paratest/releases/tag/5.0.4 and composer install
2. Run vendor/bin/phpunit --coverage-text test/Unit/Runners/PHPUnit/BaseRunnerTest.php --filter testGeneratesCoverageTypes
3. See how ParaTest\Runners\PHPUnit\BaseRunner is reported as Methods: 0.00% ( 0/ 9) Lines: 0.00% ( 0/ 62)
4. Edit test/fixtures/phpunit-fully-configured.xml and remove processUncoveredFiles="true"
5. Rerun the test and now the same class is reported as Methods: 66.67% ( 6/ 9) Lines: 95.16% ( 59/ 62)

[dvdoug]

OK I'll take a look and see if I can figure it out from that but obviously a reduced testcase would be helpful 😄. I've tried some other codebases and can't find any change to coverage

[sebastianbergmann]

Thank you for looking into this, @dvdoug.

[dvdoug]

This appears to be specific to ParaTest not PHPUnit generally, and I think maybe just ParaTest's own test suite rather than users of ParaTest. It self-testing causing the issue here.

It's something like this:

1. PHPUnit starts coverage, starts executing the test
2. Paratest's test suite tries to prove it can generate coverage
3. As part of generating the coverage, it generates a report
4. Generating a report with processUncoveredFiles starts coverage again with Id UNCOVERED_FILES, does the includes and then stops coverage. That means the coverage from that particular test is stored under UNCOVERED_FILES rather than the testcase name
5. The testcase ends, PHPUnit stops coverage but because the data was already collected under UNCOVERED_FILES there is nothing left.

In the previous releases, processUncoveredFiles happened before test suite execution and ParaTest reaching into coverage functions from within it's own test suite appeared to work.

[dvdoug]

If I've understood that correctly, I think ParaTest should run that particular test with a mock driver rather than the real in-process one

[Slamdunk]

Thank you very much @dvdoug for your analysis and the effort you put into all this.

It seems you are right, this issue shouldn't be considered a proper bug anymore.

As a final note though it would be useful to pinpoint somewhere that nested CC analysis are not idempotent to non-nested ones anymore.