I use the C++ Core Guidelines[1] during code review. clang-tidy provides many checks to automatically enforce the C++ Core Guidelines. However, it’s unclear to me and other code reviewers how much of the C++ Core Guidelines is covered by clang-tidy and what I have to manually review.
I propose adding a section to the clang-tidy documentation with tables tracking what C++ Core Guidelines rules correspond to each clang-tidy linter check. There will be 4 states for every rule, green for full enforcement, red for no enforcement, yellow for partial enforcement, and purple for “cannot enforce” (e.g. subjective unenforceable rules like “express intent”). The table will also link the documentation for each check.
This will also make it easier to track progress towards the goal of getting 100% coverage on the enforceable parts of the C++ Core Guidelines. Since this is a change to the goals of clang-tidy, I want to get consensus this is a good idea before making the PR.
I think one important thing that is missing is if a guideline is partially implemented or not implemented yet even if it is fully enforceable.
Linking towards a living document can also be problematic since a link can potentially be broken after a certain time for an older version of clang-tidy and it puts a lot of effort on maintenance. C++ Core Guidelines does no have different released versions of the document.
To be far, I think you are free to do the mapping yourself.
Yellow will mean partially implemented in clang-tidy.
Linking towards a living document can also be problematic since a link can potentially be broken after a certain time for an older version of clang-tidy and it puts a lot of effort on maintenance. C++ Core Guidelines does no have different released versions of the document.
The C++ Core Guidelines does have “releases”, but they haven’t cut one in over 8 years. We could create a PR to increment to 0.8. clang-tidy is already facing this problem since the documentation for eachcppcoreguidelines-* check links to the applicable C++ Core Guideline. e.g.:
FWIW, I do not believe this is an existing goal. Historically people have had a hard time implementing these rules for a number of reasons: enforceable in theory but not in practice, ambiguous, unclear, incomplete, too strict, poor feedback loops with the authors, etc.
The fact that it’s a living document is also a problem, they can change rules at any time and suddenly a rule that was fully enforced becomes partially enforced, for example. I do not believe it makes sense to target release 0.8 from 8 years ago.
It would also seem strange to have a section dedicated to coverage of the C++ Core Guidelines when there are many other guidelines as well - should those get a coverage section too? Otherwise this might suggest clang-tidy endorses or somehow gives preference to the C++ Core Guidelines.
What could be done perhaps is to name the checks with the corresponding rule number, for easier mapping/reference, but it’s a rather breaking change.
what I have to manually review.
As a side note, I personally think this is the wrong way to look at it. Guidelines are to be enforced by static analysis tools, it’s impossible for humans to manually do it, for all rules, for all commits.
A better approach is to enable as many compiler warnings and clang-tidy checks (and treat them as errors in CI) as are reasonable in your project - that will give you the best quality code you can enforce automatically. Combine it also with sanitizers. Then let humans review what humans are good at: architecture, design, structure, algorithms, etc.
Thank you for putting work in “C++ Core Guidelines” (later CCG) direction, here is my thoughts on original RFC:
CCG has far more rules than there are clang-tidy checks, so I’m afraid your list with all CCG rules will be mostly “red” and “purple” with a little bit of “green” and “yellow”. It can be hard to navigate this way and look for relevant information. Do you plan to remove/suppress some CCG rules to keep this list relatively small?
Together with this color, some description of what part of this CCG rule is not covered would be helpful.
Apart from original RFC I’d suggest implementing this mapping on CCG document side. It has already tools support which can be extended to be a list with clang-tidy checks. CERT guidelines do this, plus we can mention compiler warning/clang static analyzer warning that enforce CCG rules (like CERT do).
I think the user will expect to find some links to enforcement tools in the actual C++ Core Guidelines paper, rather than deep-diving at clang-tidy docs.
I like the idea of providing a summary table that shows what support clang-tidyhas for various coding standards such as CERT and C++ Core Guidelines.
However, I would be opposed to the “red for no enforcement” lines as I think they will just clutter up the table and there’s a whole bunch of guidelines for which clang-tidyisn’t the right tool and won’t ever provide any sort of detection or automated fix.
I do see value in a table for CERT guildelines and C++ Core Guidelines that summarizes the support provided by clang-tidy.
I’m a little against naming rules only by CCG rule number because it significantly reduces readability. We can just add rule number at the end of check names, like cppcoreguidelines-avoid-goto-ES-76 make check name cppcoreguidelines-ES-76 as an alias for human-readable misc-avoid-goto.
It’s very frustrating right now that, e.g. check cert-err58-cpp can be actually a human-readable bugprone-static-object-exception, but clang-tidy user can’t see that without reading all CERT guidelines.
It doesn’t, but I don’t see a why it should in the first place.
If a user wants to look what clang-tidy can offer as CPPCore/CERT guidelines conformance tool, he would look at check-list.
If a user wants to choose a tool for guidelines conformance, he would visit the corresponding page in guidelines, like CERT and CCG.
I agree, I meant adding the rule number to the existing name, like cppcoreguidelines-ES-76-avoid-goto.
I don’t think this issue has to do with aliases, it’s simply the way the check name was chosen. It could still be an alias and be called cert-static-object-exception.
Red means “no enforcement, but it should be”. Purple would be “will not enforce”.
CCG has far more rules than there are clang-tidy checks, so I’m afraid your list with all CCG rules will be mostly “red” and “purple” with a little bit of “green” and “yellow”. It can be hard to navigate this way and look for relevant information. Do you plan to remove/suppress some CCG rules to keep this list relatively small?
The goal is, when doing code review, I will look at the clang-tidy list for what I need to manually check for. I was expecting most of the list to be green, honestly, but it’s hard to know without creating the list.
When I have been working with MISRA C:2023 this is actually something that I have been looking into. Initially to see which of the rules and directives that I had already analyzed and implemented. So I created an extra json file to track all this information. This is especially needed for all the rules and directives that requires manually reviewing (or is impossible to trigger due to how clang analyzes source code) since those will not have any other file to put this meta data into.
while my ”extra” json file might not be the best solution, I think there is still a need to cover how much of any guidelines a tool covers even if it is just 10%. While it is not necessary for a normal project, it is actually needed if the tool is used in a regulatory industry. I understand that non of this exist yet but I think this could actually be a helpful way of increasing how much a tool actually covers. I think the use case initially is more in increasing coverage rather than using it to see how much that is not covered.
Not all of the checks have cppcoreguidelines aliases even when implementing them. e.g misc-const-correctness implements ES.25 but doesn’t have an alias.
It’s also not always a 1-to-1 mapping. A guideline can be implemented in multiple checks or one check can cover multiple guidelines.
To give an example, cppcoreguidelines-macro-usage implements ES.31 and ES.32
That is unfortunate but understandable, read in multiple threads that it was quite work intensive to introduce an alias. Also it runs multiple times so it is not great from a performance point of view either.
With MISRA guidelines I tried to have a 1-1 mapping but it is more due to how you are suppose to handle deviations where a rule is not followed for a certain code.
That is probably better from a performance point of view even if it can make the check more complex.
With that in mind, I think it is better to have a list of all the C++ core guidelines and link to 1 or more checks with how much those checks cover rather than the other way around. Also, it makes it more natural if there is no check.