Code coverage appears broken on HHVM

[tomzx]

Hi,

I use to have phpunit running with code coverage under 4.7.7 and not report any error. Although I cannot confirm if it did indeed produce meaningful output (since it was only executed on travis and sending the report to scrutinizer-ci), my guess is that it did.

However, it appears that changes that have been done in 2.2.x might have broken what used to not generate errors.

My guess would be that the change 245f47c#diff-b2fbc27b0e544bbffcdfba89b3fb1dd4L99 changed the condition just enough to break.

It changed from PHP_SAPI === 'phpdbg' (such that it would use phpdbg) to !$this->isHHVM() && PHP_SAPI === 'phpdbg';. Thus, it prevents the use of the phpdbg driver and defaults to xdebug, which isn't supported by hhvm.

Here's an example callstack of the issue.

PHPUnit 4.7.7 by Sebastian Bergmann and contributors.


Fatal error: Uncaught exception 'PHP_CodeCoverage_Exception' with message 'This driver requires Xdebug' in /home/travis/build/tomzx/ditto/vendor/phpunit/php-code-coverage/src/CodeCoverage/Driver/Xdebug.php:25
Stack trace:
#0 /home/travis/build/tomzx/ditto/vendor/phpunit/php-code-coverage/src/CodeCoverage.php(914): PHP_CodeCoverage_Driver_Xdebug->__construct()
#1 /home/travis/build/tomzx/ditto/vendor/phpunit/php-code-coverage/src/CodeCoverage.php(99): PHP_CodeCoverage->selectDriver()
#2 /home/travis/build/tomzx/ditto/vendor/phpunit/phpunit/src/TextUI/TestRunner.php(360): PHP_CodeCoverage->__construct()
#3 /home/travis/build/tomzx/ditto/vendor/phpunit/phpunit/src/TextUI/Command.php(147): PHPUnit_TextUI_TestRunner->doRun()
#4 /home/travis/build/tomzx/ditto/vendor/phpunit/phpunit/src/TextUI/Command.php(99): PHPUnit_TextUI_Command->run()
#5 /home/travis/build/tomzx/ditto/vendor/phpunit/phpunit/phpunit(36): PHPUnit_TextUI_Command::main()
#6 {main}

[sebastianbergmann]

I do not use HHVM myself and don't have the time to follow it's development. I was told that PHP_CodeCoverage_Driver_HHVM (https://github.com/sebastianbergmann/php-code-coverage/blob/2.1/src/CodeCoverage/Driver/HHVM.php) stopped working because HHVM removed the fb_*_code_coverage() functions. I was also told that HHVM now uses Xdebug's API. Looking at https://github.com/facebook/hhvm/blob/master/hphp/runtime/base/code-coverage.cpp I get the impression that code coverage is implemented completely different.

If the HHVM community wants code coverage support in PHPUnit / PHP_CodeCoverage then it is up to the HHVM community to provide patches.

[Koc]

I also cann't run code coverage on hhvm: https://travis-ci.org/Koc/Sphinxy/jobs/73683002

[sebastianbergmann]

Adding "it also does not work for me" comments does not help. I have nothing to add to what I wrote in #365 (comment), sorry.

[Koc]

Let's wrap call of xdebug_start_code_coverage with defined('HHVM_VERSION') and do not pass any of constants to it for HHVM.

[sebastianbergmann]

No clue what you're talking about. Since sebastianbergmann/environment@3e39374 and 0edb5c6 PHP_CodeCoverage won't try to use PHP_CodeCoverage_Driver_Xdebug on HHVM.

[sebastianbergmann]

@Koc Please discuss pull requests such as #366 here first. You may start by commenting on what I wrote in #365 (comment).

I doubt that your changes in 5ebf085 have any effect due to sebastianbergmann/environment@3e39374 and 0edb5c6.

[tomzx]

I ran a configuration which used to not generate errors (phpunit @ 4.7.3 + php-code-coverage @ 2.1.5 + HHVM @ 3.5.0).

At that point in time, phpunit would not generate any code coverage, it would simply run the tests and be done.

$ php vendor/bin/phpunit
PHPUnit 4.7.3 by Sebastian Bergmann and contributors.
Warning:    The Xdebug extension is not loaded
        No code coverage will be generated.

...........

Time: 1.7 seconds, Memory: 8.44Mb

OK (11 tests, 24 assertions)

The command "php vendor/bin/phpunit" exited with 0.

After looking a bit on the Internet, it appears that you have to use
if [[ $TRAVIS_PHP_VERSION =~ ^hhvm ]]; then echo 'xdebug.enable = On' >> /etc/hhvm/php.ini; fi in order to enable xdebug for hhvm when running @ travis-ci. At that point, the behavior is the same as before: no code coverage is generated and it doesn't throw an exception.

phpunit @ 4.7.7 + php-code-coverage @ 2.2.1 + HHVM @ 3.5.0

$ php vendor/bin/phpunit
PHPUnit 4.7.7 by Sebastian Bergmann and contributors.

...........

Time: 2.26 seconds, Memory: 8.44Mb

OK (11 tests, 24 assertions)

The command "php vendor/bin/phpunit" exited with 0.

[sebastianbergmann]

Which versions of php-code-coverage and environment did you run your tests with?

[tomzx]

phpunit @ 4.7.7 + php-code-coverage @ 2.2.1
This is all running @ travis-ci on HHVM 3.5.0.

[sebastianbergmann]

Do I understand you correctly that code coverage works on HHVM by HHVM exposing Xdebug's API when xdebug.enable = On is set in HHVM's configuration?

[tomzx]

No. It does not work. As you can see there is no message such as

Generating code coverage report in Clover XML format ... done

Generating code coverage report in HTML format ... done

being outputted.

All that xdebug.enable = On does is enable xdebug but it does not appear to generate any code coverage.

[tomzx]

Looking at the code, it appears that \SebastianBergmann\Environment\Runtime::canCollectCodeCoverage returns false, thus preventing any code coverage from being collected.

At that point, if I force it to return true, I get a similar error to what @Koc reported.

Fatal error: XDEBUG_CC_UNUSED and XDEBUG_CC_DEAD_CODE constants are not currently supported. 

Then, forcing \PHP_CodeCoverage_Driver_Xdebug::start call to xdebug_start_code_coverage to have no argument (like it is suggested in @Koc PR), everything works fine:

hhvm vendor/phpunit/phpunit/phpunit
PHPUnit 4.7.7 by Sebastian Bergmann and contributors.

................................

Time: 11.69 seconds, Memory: 4.78Mb

OK (32 tests, 34 assertions)

Generating code coverage report in Clover XML format ... done

Generating code coverage report in HTML format ... done

My suggestion (after testing it out) is to merge @Koc #366 and my PR sebastianbergmann/environment#9.

[sebastianbergmann]

sebastianbergmann/environment#9 has been merged and a new release of sebastian/environment has been rolled. Thanks for the patch!

Hopefully tonight (~ 12 hours from now) or tomorrow (~ 24 hours from now) I will have the time to review and merge #366.

Thanks for your effort and patience.

[sebastianbergmann]

Turns out I had a bit more time to spend on this right now :-)

Please test the current state and provide feedback, especially when things are broken. Thanks!

[tomzx]

Looks good to me 👍.

hhvm @ 3.5.0
phpunit/phpunit @ 4.7.7
phpunit/php-code-coverage @ 2.2.1
sebastian/environment @ 1.3.2

For code coverage to work on travis, one needs to make sure to enable it (as it is not enabled by default it seems):
if [[ $TRAVIS_PHP_VERSION =~ ^hhvm ]]; then echo 'xdebug.enable = On' >> /etc/hhvm/php.ini; fi