A new code coverage bot

Project Infrastructure LLVM Dev List Archives
1

Hi,

I'd like to announce a new code coverage bot:

    http://lab.llvm.org:8080/coverage/coverage-reports/clang/index.html

The purpose of the bot is to make high-quality coverage reports available to
llvm developers, and to provide additional testing for clang's code coverage
implementation.

The coverage data clang generates allows the reporting tool to render execution
counts for code regions nested within lines or macros. The tool is also capable
of displaying coverage information for each instantiation of a function.

The bot runs "check-{llvm,clang}" with an instrumented version of the latest
top-of-tree compiler. It generates new reports roughly every four hours. These
reports can be used to find dead code, find fishy code, and to gauge the
effectiveness of our smoke tests.

If you have any feedback about the bot or the coverage reports, please let me
know!

vedant

p.s: There is already a coverage bot which runs "check-all" and uses gcov. If
you'd prefer those reports, you can find them here --

    http://llvm.org/reports/coverage/

2

Would it be possible to add lld to the mix? :slight_smile:

Thanks,
Rafael

3

Nice!

Would be good to get this linked up to llvm.org so I can find it easily next time I am interested in coverage info.

  • Matthias
4

I'm testing this out and will let you know when it works.

vedant

5

Hi Vedant,

This is awesome! Thanks for this great work :slight_smile:

6

I'm testing this out and will let you know when it works.

vedant

Thank you very much!

7

Very nice. Any chance you could git-clone polly into tools/polly, to get
Polly coverage information as well?

Best,
Tobias

8

I've configured the bot to test lld and polly. The first batch of reports for
the new tools are not ready yet, but anyone can monitor the build:

    http://lab.llvm.org:8080/green/view/Experimental/job/clang-stage2-coverage-R/

Matthias, I will ask around about adding a link to this bot on llvm.org once it
graduates from the Experimental pane on greendragon.

thanks,
vedant

9

Thank you! Polly is now available, but individual sites are not
accessible:

http://lab.llvm.org:8080/coverage/coverage-reports/clang/coverage/Users/buildslave/jenkins/sharedspace/clang-stage2-coverage-R@2/llvm/tools/polly/lib/Transform/FlattenSchedule.cpp.html

I get: Forbidden

You don't have permission to access
/coverage/coverage-reports/clang/coverage/Users/buildslave/jenkins/sharedspace/clang-stage2-coverage-R@2/llvm/tools/polly/lib/Transform/FlattenSchedule.cpp.html
on this server.

Best,
Tobias

10

The bot hiccupped earlier but looks stable now. The average turnaround seems to
be 3.5 hours.

clang:
  http://lab.llvm.org:8080/coverage/coverage-reports/clang/index.html

lld:
  http://lab.llvm.org:8080/coverage/coverage-reports/lld/index.html

polly:
  http://lab.llvm.org:8080/coverage/coverage-reports/polly/index.html

I've configured the bot to test lld and polly. The first batch of reports
for
the new tools are not ready yet, but anyone can monitor the build:

   http://lab.llvm.org:8080/green/view/Experimental/job/clang-stage2-coverage-R/

Matthias, I will ask around about adding a link to this bot on llvm.org
once it
graduates from the Experimental pane on greendragon.

Thank you! Polly is now available, but individual sites are not
accessible:

http://lab.llvm.org:8080/coverage/coverage-reports/clang/coverage/Users/buildslave/jenkins/sharedspace/clang-stage2-coverage-R@2/llvm/tools/polly/lib/Transform/FlattenSchedule.cpp.html

I get: Forbidden

You don't have permission to access
/coverage/coverage-reports/clang/coverage/Users/buildslave/jenkins/sharedspace/clang-stage2-coverage-R@2/llvm/tools/polly/lib/Transform/FlattenSchedule.cpp.html
on this server.

This should be fixed now.

vedant

11

I noticed that certain files which exist in ToT don't appear to be getting mentioned despite having test coverage within the test suite.

As a particular example, consider lib/Transforms/Scalar/RewriteStatepointsForGC.cpp. There may also be others, but that's the one I happened to notice.

It's particular weird in that the gcc runs do include this particular file.

Philip

12

Perfect. I added a link from the Polly website.

Best,
Tobias

13

The bot hiccupped earlier but looks stable now. The average turnaround
seems to
be 3.5 hours.

clang:
  http://lab.llvm.org:8080/coverage/coverage-reports/clang/index.html

lld:
  http://lab.llvm.org:8080/coverage/coverage-reports/lld/index.html

This seems like a bug in the coverage generation:

http://lab.llvm.org:8080/coverage/coverage-reports/lld/coverage/Users/buildslave/jenkins/sharedspace/clang-stage2-coverage-R@2/llvm/tools/lld/ELF/Symbols.cpp.html#L31

It shows `switch (Body.kind()) {` with zero counts even though code it
dominates has counts.

-- Sean Silva

14

Also, I think that at some point we could teach coverage about
llvm_unreachable() (or the other way around, maybe annotating), so
that we don't get zero counts as in here, e.g. void TargetInfo::relaxTlsIeToLe()
http://lab.llvm.org:8080/coverage/coverage-reports/lld/coverage/Users/buildslave/jenkins/sharedspace/clang-stage2-coverage-R@2/llvm/tools/lld/ELF/Target.cpp.html

15

The bot hiccupped earlier but looks stable now. The average turnaround
seems to
be 3.5 hours.

clang:
http://lab.llvm.org:8080/coverage/coverage-reports/clang/index.html

lld:
http://lab.llvm.org:8080/coverage/coverage-reports/lld/index.html

This seems like a bug in the coverage generation:

http://lab.llvm.org:8080/coverage/coverage-reports/lld/coverage/Users/buildslave/jenkins/sharedspace/clang-stage2-coverage-R@2/llvm/tools/lld/ELF/Symbols.cpp.html#L31

It shows `switch (Body.kind()) {` with zero counts even though code it
dominates has counts.

Yeah, this is a bug :|. I already have a report about this.

Also, I think that at some point we could teach coverage about
llvm_unreachable() (or the other way around, maybe annotating), so
that we don't get zero counts as in here, e.g. void TargetInfo::relaxTlsIeToLe()
http://lab.llvm.org:8080/coverage/coverage-reports/lld/coverage/Users/buildslave/jenkins/sharedspace/clang-stage2-coverage-R@2/llvm/tools/lld/ELF/Target.cpp.html

I have a bug open about this too. I like the idea of introducing a function
attribute like 'nocoverage'. I'll float the idea on cfe-dev, along with some
alternatives we've considered (new pragma, function blacklists).

vedant

16

This is happening because RewriteStatepointsForGC.cpp.o isn't linked into
clang, causing its coverage mapping data to be discarded. Maggie (CC'd) is
working on a patch that should allow llvm-cov to produce a combined report
given multiple binaries.

For now, I've configured the bots to produce separate reports for opt and llc
(adding to the reports we have for clang, lld, and polly). The next time it
updates, you should be able to see coverage for this file in the report for
opt.

I'll send out a link once it's ready.

vedant

17

It's ready. There are links to the each of the reports in the bot description:

    http://lab.llvm.org:8080/green/view/Experimental/job/clang-stage2-coverage-R/

Sorry about the maze of links. I'll put together a more convenient landing page
soon.

vedant

18

The bot hiccupped earlier but looks stable now. The average turnaround
seems to
be 3.5 hours.

clang:
  http://lab.llvm.org:8080/coverage/coverage-reports/clang/index.html

lld:
  http://lab.llvm.org:8080/coverage/coverage-reports/lld/index.html

This seems like a bug in the coverage generation:

http://lab.llvm.org:8080/coverage/coverage-reports/lld/
coverage/Users/buildslave/jenkins/sharedspace/clang-
stage2-coverage-R@2/llvm/tools/lld/ELF/Symbols.cpp.html#L31

It shows `switch (Body.kind()) {` with zero counts even though code it
dominates has counts.

I committed a fix for this issue yesterday. The coverage reports should be
looking better now when it comes to switches.

Alex

19

The bot hiccupped earlier but looks stable now. The average turnaround
seems to
be 3.5 hours.

clang:
  http://lab.llvm.org:8080/coverage/coverage-reports/clang/index.html

lld:
  http://lab.llvm.org:8080/coverage/coverage-reports/lld/index.html

This seems like a bug in the coverage generation:

http://lab.llvm.org:8080/coverage/coverage-reports/lld/cover
age/Users/buildslave/jenkins/sharedspace/clang-stage2-
coverage-R@2/llvm/tools/lld/ELF/Symbols.cpp.html#L31

It shows `switch (Body.kind()) {` with zero counts even though code it
dominates has counts.

I committed a fix for this issue yesterday. The coverage reports should be
looking better now when it comes to switches.

Thanks!

-- Sean Silva