llvm-cov support

[gsauthof]

LLVM clang also understands the gcc options for enabling coverage collection (e.g. --coverage and -fprofile-arcs -ftest-coverage). But - in that compatibility mode - llvm targets a certain gcc/gcov version. For example one that isn't supported anymore by current gcov/lcov (yielding inaccuracies and lcov errors).

Thus, to increase the portability of the cmake codecov module, it would be useful to also add support for llvm-cov.

[alehaa]

If your compiler is clang >= 3.5, FindGcov should use llvm-cov by default. The llvm-cov-wrapper will be used for compatibility with Lcov.

Can you give me some sample output to debug the situation?

[gsauthof]

I see, the llvm-cov-wrapper that comes with CMake-codecov calls llvm-cov gcov.
This indeed does work when I compile with clang.

With this issue-ticket I am trying to suggest to add support for LLVM's native coverage mode.

Currently, when I compile with CMake-codecov and clang it is compiled like this:

/usr/bin/clang++     -O0 -g -fprofile-arcs -ftest-coverage ...

In contrast, to use the native mode one would compile like this:

 /usr/bin/clang++     -O0 -g -fprofile-instr-generate -fcoverage-mapping ...

With that mode one also would have to execute different steps to generate the report, e.g.:

llvm-profdata merge default.profraw -o default.profi
llvm-cov show -instr-profile default.profi foo

So this issue is definitely intended as a feature request.

[alehaa]

At the moment this CMake module is intended to generate coverage reports for tools which understand gcov data only, like lcov and online tools like codecov.io. I think if the native LLVM coverage format will be enabled, this will break the gcov data which might not be intended, right?

A separate option for compiling for native LLVM coverage format might be added to enable building the LLVM reports. It will be default off, so the normal user will have no problems using this module for gcov data, but interested users might generate LLVM coverage reports. Is this the behavior you're requesting?

[gsauthof]

Yes, a separate option would be fine.

[rickyzhang82]

Hi Gentlemen,

Thanks for sharing your cmake file. It saves me a lot of leg work.

It works perfectly in Linux GCC. But in Mac OS X clang, it can't generate HTML report. I can see *.gcda and *.gcno are generated . In Mac OS 10.12.2 XCode 8.2, I found that

ls -al /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
...
lrwxr-xr-x   1 root  wheel     8B Oct 29 09:31 gcov@ -> llvm-cov
-rwxr-xr-x   1 root  wheel   590K Dec  6 18:53 llvm-cov*

gcov is a symlink to llvm-cov. There is no more geninfo and genhtml. But it seems that new llvm-cov can generate HTML report as well in the same command.

I wonder if the CMake-codecov is intended to support Mac OS X? For your debug, here is my CMake output:

-- Found gcov evaluation for Clang: /Users/Ricky/repo/xlparser/cmake/llvm-cov-wrapper
-- Could NOT find lcov (missing:  LCOV_BIN GENINFO_BIN GENHTML_BIN)

[rickyzhang82]

I dig it deeper. The script llvm-cov-wrapper does try to address the llvm-cov. But it doesn't solve lcov problem. I installed lcov from GNU in Mac OS X to generate report from 'llvm-cov gcov'. GNU lcov doesn't seem to be compatible with llvm-cov.

I tried gcovr, a Python implementation report coverage generator. The detailed result doesn't match.

So IMHO I need to use llvm-cov native mode and generate report by 'llvm-cov show --format=html'

/usr/bin/clang++     -O0 -g -fprofile-instr-generate -fcoverage-mapping ...

[alehaa]

Ok. As far as I understand we need a new mode for the codecov scripts to handle a) native llvm-cov and b) exporting the llvm-cov reports as HTML instead of using lcov. Right?

[rickyzhang82]

I'm working on it now.

[rickyzhang82]

No, it turns out that Mac OS X llvm-cov is not the same as Clang llvm-cov. It doesn't support -format argument. It still can't show html. Any suggestions?

[alehaa]

I'll try to generalize the modules, so we can plug different analyzing-tools. Then we can investigate how to generate the reports for llvm-cov in XCode. ;)

[rickyzhang82]

Sorry, I don't follow your plan. Can you be more specific on "the modules" and "different analyzing-tools"?