function level coverage

[mtozjh]

I need to get info about what functions are tested. Currently there is only line coverage data.
if this tool can be enhanced to report that data, it will be great

[przecze]

I would also very like to see such feature added. Currently we have to use some home-brewed script that parses normal coverage report and tries to extract info about function, this has some issues so "native" support from pycoverage would be great.

[IterableTrucks]

By now there's only one plugin called pytest-func-cov for function coverage. But it does not support file reporting.

[nedbat]

I need some clarification about what people mean by "function-level coverage". The pytest-func-cov plugin considers a function tested if it is called directly by a test, but not if it is called indirectly. Is that what is meant? Or are indirect calls sufficient?

But it does not support file reporting.

The pytest-func-cov README shows a report with file names, so I'm not sure what is missing?

[getim]

We are looking to complete our current coverage report with this data as well. It looks like pytest-func-cov's definition of function coverage (a function must be called directly from a test) is different from what other sources say. Atlassian, Wikipedia and lcov's data format docs all define a function as covered if it's hit during a test at any point.

For our particular use case, having the FNF (functions found) and FNH (functions hit) numbers in the lcov report generated by coveragepy would complete the HTML reports we currently generate with genhtml from the lcov report.

[jab]

Being able to see what functions were covered by tests would be useful for me too, and being called indirectly should absolutely still count.

Currently we have to use some home-brewed script that parses normal coverage report and tries to extract info about function

Anything you can share here, by chance?

The pytest-func-cov plugin considers a function tested if it is called directly by a test, but not if it is called indirectly.

This makes pytest-func-cov unusable for my purposes. Also, it needs to be reimplemented due to a design flaw, and isn't actively maintained. Plus it'd be better to have this feature directly in coverage.py itself or a coverage plugin, as opposed to something external (with its own separate data format / possibly no structured data format).

[nedbat]

I have some thoughts about how to implement this, and have started some of the foundation work. I'm curious if people have ideas about how to present the data? Do you have examples of other coverage tools (possibly in other languages) that do a good job showing the results?

[jab]

That's awesome, thanks @nedbat!

A useful way of presenting this data for me would be something like:

❯ coverage report --functions --sort-by=function-coverage  # I made up these options

Function                   Stmts   Miss Branch BrPart  Cover
------------------------------------------------------------
mypkg:func1                  245      7     98      3  97.1%
mypkg:MyCls.method1           51      5      2      0  90.6%
...

...as well as an associated JSON representation. Even just a list of (function, cov_percent) pairs would be enough to answer questions like, "What are the best (and worst) covered functions?"

That said, just having this data collected in the first place would be useful to have landed before any presentation support is landed. The harder part for users is getting this data to be collected. But once it's collected, it's easier for us to handle pulling it out of the coverage database ourselves in whatever format we need. In case it makes things easier for you to ship the collection support first, with presentation support left for a followup release.

[nedbat]

This is an interesting point, because the function-level data doesn't need to be stored in the database at all, just as the missing lines aren't in the database. The database captures the measured data. The reporting phase can determine what is missing, and could also calculate statistics per function.

A middle ground could be something like a JSON report that includes function data.

[jab]

Outputting JSON about function-level coverage would be great!

Just to improvise something:

❯ coverage json --rollup=functions
{
  "files": {
    "mypkg/__init__.py": {
       "functions": {
         "mypkg:func1": {"percent_covered": 1.0, ...},
         "mypkg:MyCls.method1": {"percent_covered": 1.0, ...}
       }
    }
  }
}

And if passing --show-contexts could also show which test functions caused which functions to be covered (when using dynamic_context = test_function), that would be especially awesome. But the function-level rollup would be super useful even without that.

[nedbat]

If anyone wants to try some really hacked together functionality as a work in progress: install this temporary branch of coverage.py:

python3 -m pip install git+https://github.com/nedbat/coveragepy@nedbat/public-analysis#egg=coverage==0.0

After running coverage as usual, run regions.py: https://github.com/nedbat/coveragepy/blob/nedbat/public-analysis/lab/regions.py

It will list functions and classes, with their total percentage.

Keep in mind, this is temporary. I'm interested to hear if this is useful data.

[jab]

Definitely useful, thank you!

Some prior art just for completeness:

lcov's genhtml includes function coverage by default:

And gcovr does as well:

(Here's a zip of the source files and the html output I used to produce those screenshots, and here's a post about those two tools, in case it's helpful.)

[nedbat]

Thanks. I'm wondering how those tools cope with real projects with thousands of functions. How do they present the information in a useful way?

[jab]

Looks like both allow you to view the functions in a single file at a time (and hopefully there aren't thousands of functions in a single file). In genhtml's case, that looks like the only way to view function coverage, while gcovr also includes a view including all functions across all files (among several other views). Both tools generate trees of views that let you drill down into subdirectories (and provide associated rollups of coverage data per subdirectory). And I guess you can always use lcov's --extract or --remove options if you need to slice the data further before generating html from it via either tool

Also looks like only gcovr shows the percent of the lines in a given function that were covered ("block coverage"). In contrast, genhtml only shows "hit count". I think it's definitely worth including all the line-based coverage counts per function (hits, misses, total percent covered, ...).

[nedbat]

I've implemented more real function and class reports. Please try them out: https://nedbatchelder.com/blog/202404/try_it_functionclass_coverage_report.html