[RFC] Add a new text diagnostics format that supports nested diagnostics

Clang Frontend
1

This RFC consists of two main parts

  1. the addition of nesting information to the diagnostics (engine) and consumers
  2. the introduction of a new diagnostics output format to take advantage of that information.

For the current state of the implementation, see [Clang] [Diagnostics] Add a new text diagnostics format that supports nested diagnostics by Sirraide · Pull Request #151234 · llvm/llvm-project · GitHub.

Both parts are described in greater detail below, but to summarise, I think it would be beneficial to be able to arrange diagnostics hierarchically beyond the simple ‘notes are attached to warnings/errors’ model that we have at the moment. However, altering the way in which we currently render diagnostics to the terminal would probably be too much of a breaking change for a number of people, so I propose adding a new format instead.

Thanks also to @AaronBallman for feedback on this RFC.

1. Nesting

In the current implementation, Diagnostic, StoredDiagnostic, and DiagnosticStorage now have a NestingLevel member, which is an (optional) unsigned value. The diagnostics engine keeps track of a ‘global nesting level’, which can be incremented via an RAII object; when a diagnostic is emitted (i.e. passed to the DiagnosticConsumer), the nesting level of the diagnostic is set to the current global nesting level (or that level +1 if it is a note). This means that, by default, the nesting level is 1 for note and 0 for any other type of diagnostic.

StoredDiagnostic and DiagnosticStorage store an UnsignedOrNone, which is unset if the nesting level hasn’t been computed yet; if set, the nesting level will not be recomputed when the diagnostic is emitted. This is to allow diagnostic consumers to store diagnostics and emit them later on without altering their nesting level.

How to Make Use of Nesting

On the consumer side, in and of itself, the nesting level does nothing; it is up to diagnostic consumers to make use of it.

On the producer side, the intended (and indeed only) way to nest diagnostics is to use the aforementioned RAII object (DiagnosticsEngine::NestingLevelRAII); I’ve introduced nesting to a few of the DiagnoseTypeTraitDetails diagnostics since they were previously pointed out to me as being good candidates for this. An example of this is the following:

 static void DiagnoseNonTriviallyCopyableReason(Sema &SemaRef,
                                                SourceLocation Loc, QualType T) {
   SemaRef.Diag(Loc, diag::note_unsatisfied_trait)
       << T << diag::TraitName::TriviallyCopyable;

+  DiagnosticsEngine::NestingLevelRAII IncrementNestingLevel{SemaRef.Diags};
   if (T->isReferenceType())
     SemaRef.Diag(Loc, diag::note_unsatisfied_trait_reason)
         << diag::TraitNotSatisfiedReason::Ref;

   const CXXRecordDecl *D = T->getAsCXXRecordDecl();
   if (!D || D->isInvalidDecl())
     return;

   if (D->hasDefinition())
     DiagnoseNonTriviallyCopyableReason(SemaRef, Loc, D);

+  IncrementNestingLevel.Increment();
   SemaRef.Diag(D->getLocation(), diag::note_defined_here) << D;
 }

(The Increment() function allows ‘reusing’ an RAII object rather than having to create a new one every time you want to increment the nesting level. It’s possible to make do w/o it by just adding another scope—which is what I did initially—but I ended up including it because after migrating a mere 4 or so functions, I was already using it in 2 places, so chances are there will be a quite a few more cases where it will come in handy…)

Basically, if you know you’re about to emit diagnostics which you’d like to be more deeply nested, you simply increment the global nesting level, and then any code after that doesn’t have to know or care about what nesting level it’s at. This is also (currently) the only way to alter the nesting level of a diagnostic. The intention of this is to make it possible to introduce nesting without having to fundamentally alter the way (or order) in which diagnostics are emitted—though of course, there will most likely be places where we will want to perform some amount of rearranging or grouping to take full advantage of nesting.

I briefly experimented with making it possible to write something along the lines of Diag(...) << diag::AddNestingLevel(1), which would add an increment to the nesting level of a single diagnostic, but I found the RAII object to be both simpler to implement and easier to reason about. One thing we definitely don’t want to start doing is setting the nesting level to an absolute value (e.g. Diag(...) << diag::SetNestingLevel(5) or whatever), since that would break in the presence of recursive nesting.

2. The New ‘Fancy’ Diagnostics Format

Pass -fdiagnostics-format=fancy to enable it. If someone can think of a better name for this, please let me know (note: I didn’t want to call this the ‘nested’ diagnostic format or anything like that because it does more than just add nesting).

While we could simply update the existing text diagnostic printer to e.g. indent diagnostics by some amount for each nesting level, I don’t think this is a good idea: the current format is fairly well-established, and I wouldn’t be surprised if doing even something as simple as adding leading indentation would mess with systems that expect <file>:<line>:<col>: error: at the start of a line.

Moreover, I would argue that our current default format has a few shortcomings, though this might be fairly subjective (a while back I did look into a few papers about writing ergonomic diagnostics in the course of this, but I don’t consider this to be an exact science…); e.g. consider this diagnostic:

In file included from test.cc:17:
/usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/variant:103:21: error: static assertion failed due to requirement '2UL < sizeof...(_Types)'
  103 |       static_assert(_Np < sizeof...(_Types));
      |                     ^~~~~~~~~~~~~~~~~~~~~~~

I’d argue that the first thing you’d probably want to see when looking at a compiler error is the actual error message, but in our current format, you first have to make it past the include stack and the file path; in this case, the path is so long that the error message flows off the screen (I’ve been working on improving that part too, but it’s turned out to be more complicated than expected; see #148745 for that).

Example

So, what I suggest we do instead is add a completely new format which can both make use of nesting and hopefully be a bit more ergonomic. In the present version of this format, the error above would end up being formatted as

error: static assertion failed due to requirement '2UL < sizeof...(_Types)'
|
|     - included from test.cc:17:10: 
|     - at /usr/lib/gcc/x86_64-redhat-linux/15/../../../../include/c++/15/variant:103:21: 
|
| 103 |       static_assert(_Np < sizeof...(_Types));
|     |                     ^~~~~~~~~~~~~~~~~~~~~~~

which I would argue is a bit clearer structurally speaking. Of course, this is just what I’ve managed to arrive at; if anyone has any suggestions as to how to improve on this (or if you find it completely unreadable or don’t like it at all), please let me know.

Connecting Lines

The line in the first column might seem a bit odd in isolation, but it’s there to join to any nested diagnostics following it, e.g.

error: static assertion failed due to requirement '__is_trivially_copyable(Y)'
|
|     - at test.cc:14:15: 
|
|  14 | static_assert(__is_trivially_copyable(Y));
|     |               ^~~~~~~~~~~~~~~~~~~~~~~~~~
|
|-- note: 'Y' is not trivially copyable
|
|         - at test.cc:14:15: 
|
|
|------ note: because it has a user provided copy constructor
|
|             - at test.cc:14:15: 
|
|           6 | struct Y { Y(const Y&); }; 
|             |            ~~~~~~~~~~~
|           7 | 
|           8 | void g() {
|           9 |     FOO
|          10 | }
|          11 | 
|          12 | static_assert(__is_assignable(int&, void));
|          13 | static_assert(__is_empty(X&));
|          14 | static_assert(__is_trivially_copyable(Y));
|             |               ^
|
|---------- note: 'Y' defined here
|
|                 - at test.cc:6:8: 
|
|               6 | struct Y { Y(const Y&); }; 
|                 |        ^

I will say, I personally prefer this without the lines connecting to the notes (I think just the nesting+spacing is enough), but @AaronBallman liked them so I’ve included them here.

Alternative Formatting Ideas and Buffering

Certain more sophisticated formatting operations would require us to buffer diagnostics. Consider for instance a group of nested diagnostics that we might want to print in the following hypothetical format:

A
|
|-- B
|   |
|   |-- C
|
|-- D
    |
    |-- E

Printing this is impossible without buffering: when printing C, we need to also print the | characters in the first column that make up the line connecting to D; conversely, when printing E, we don’t print any |s in the first column, because there is nothing else to connect to. The issue, clearly, is that we have no way of knowing whether there will be some future diagnostic that we need to draw a connecting line to… until we either see that diagnostic or another top-level diagnostic.

As a compromise, the current formatter resorts to printing a tree structure more similar to the following (the indentation is the same; it’s just the connecting lines that are different):

A
|
|-- B
|   
|------ C
|
|-- D
|
|------ E

I personally don’t think this looks too good, but I don’t think we can do much better if we want to both draw connecting lines and avoid buffering. At the same time, my plan at least for this format is for us to progressively improve it, which might eventually entail buffering diagnostics. I’ve been experimenting with this and I think it’s not too hard to do, but I also feel like something like that is best left to a follow-up pr.

And just for comparison, if we were to get rid of the lines entirely, the error above would look like this:

error: static assertion failed due to requirement '__is_trivially_copyable(Y)'

      - at test.cc:14:15:

   14 | static_assert(__is_trivially_copyable(Y));
      |               ^~~~~~~~~~~~~~~~~~~~~~~~~~

    note: 'Y' is not trivially copyable

          - at test.cc:14:15:


        note: because it has a user provided copy constructor

              - at test.cc:14:15:

            6 | struct Y { Y(const Y&); };
              |            ~~~~~~~~~~~
            7 |
            8 | void g() {
            9 |     FOO
           10 | }
           11 |
           12 | static_assert(__is_assignable(int&, void));
           13 | static_assert(__is_empty(X&));
           14 | static_assert(__is_trivially_copyable(Y));
              |               ^

            note: 'Y' defined here

                  - at test.cc:6:8:

                6 | struct Y { Y(const Y&); };
                  |        ^

Potential Applications

In addition to experimenting with buffering diagnostics, I can think of several places in Clang where we would be able to make use of nesting: diagnostics around overload resolution, template instantiations, and recursive calls in the constant evaluator are things that I think would become more comprehensible if we actually made the underlying hierarchical structure more obvious instead of just emitting a flood of errors or notes all next to each other. Of course, that will probably require some more sophisticated refactoring than just incrementing the nesting level in a few places.

If anyone can think of other diagnostics that could make use of nesting, please let me know so I can add them to my list of things to look into. :eyes:

6 Likes
2

If you are interested in grabbing ideas from elsewhere, Flang-new’s Message class has the concept of attachments, which are a list of Messages (that themselves can have attachments). This is how the Fortran compiler can emit a message that points to one place in the source and is formatted with other messages that point to declarations, include “because:” notes, and so forth. See flang/include/flang/Parser/message.h. The compiler buffers up all messages until the end of semantics and then sorts them into source order.

2 Likes
3

I had a similar thought a long time ago motivated by a template error followed by a huge number of notes (similarly, an error on overloading function followed by lots of notes about candidates) – those notes are useful sometimes but not always. It would be really helpful if we can have a way to easily tell errors from notes, or even better, have an easier way to quickly skip to the next error.

At that time I was thinking about a “collapsable” diagnostics, where we can somehow fold those notes under its error message, and expand them only when needed. Of course it would be nearly impossible to actually “fold” them in a normal terminal without some fancy support from the terminal emulator. But it might be a perfect fit for environment with GUI like IDE, where notes are placed in a dropdown list folded into the parent error message and expanded when users clicked it – and I think your proposal here can make that happen.

Because IIRC, Clang support different DiagnosticEngine backends (not sure if this is the right term), so if you could create a hierarchical structure for diagnostic messages, the DiagnosticEngine backend for an IDE could materialize it into the dropdown list I described earlier.

1 Like
4

Yeah, that’s an alternative implementation that I also considered; it would require quite a bit more refactoring to how we emit diagnostics, but it would also give us more of a hierarchical structure to work with.

Oh interesting. I recall Aaron mentioning that it might be nice to have diagnostics sorted by file, so if we end up buffering diagnostics we could experiment w/ buffering all of them rather than one top-level diagnostics + any nested diagnostics at a time.

I think the main complication at the moment is that the way in which we emit diagnostics in a few places is kind of ‘backwards’. As in, if there is an error in a template instantiation, we issue a diagnostic that points to the error in the template definition and then add a note that points to the user code that triggered the instantiation. I think it should be the other way around, i.e. the user code that ultimately ‘caused’ the error (by existing, if you will) should be at the top level, and the ‘fallout’ (that is, however many errors the failed template instantiation causes) should be nested within. Afaik the main reason we don’t do it this way is simply because we currently don’t support e.g. nesting errors within errors.

This presents an issue for the attachment-based approach you mentioned (I don’t believe Fortran has templates, so flang doesn’t have this issue?): we’d have to somehow collect errors emitted during template instantiation to then attach them to some other error so we can emit them as a group. I can think of two ways of dealing with that.

  1. With the nesting level approach, one thing that I’ve been experimenting with is that if we buffer diagnostics, I think it’d be possible to emit them ‘out of order’ and have e.g. the diagnostics consumer rearrange them. That is, we could increment the global nesting level, emit a bunch of diagnostics w/ e.g. nesting level 1 and 2, and then emit a diagnostic with nesting level 0; this would require having some way of ‘flushing’ diagnostics, i.e. telling the engine/consumer that any future nested diagnostics should not be grouped under the previous top-level diagnostic, but rather under the next one.

  2. The alternative I think would be to collect diagnostics somewhere during template instantiation, emit the top-level diagnostic, and then emit all those stored diagnostics.

I think both approaches are more or less equivalent because they’d require some form of buffering and sorting either way…

5

Yeah, overloading is definitely another example of how nesting makes it easier to tell what is and isn’t part of a single error.

I generally call them ‘diagnostic consumers’ (because the base class is DiagnosticConsumer); and yes, the plan currently is that nesting information will be provided by the diagnostics engine, and it’s up to the consumer to make use of it, e.g. by indenting nested diagnostics when printing them to the terminal, or as you mentioned, I’d imagine IDEs could make use of it too; updating the SARIF diagnostic consumer to actually include nesting information would probably help there (the SARIF standard supports nesting, but since we currently don’t, we just emit one big list of diagnostics, which isn’t exaclty great…).

1 Like
6

Adding new capabilities to the reporting sounds exciting!

I have a couple use cases in mind to consider. We could make lifetime related warnings way more ergonomic in Clang. Usually, with these warnings there are multiple points of interests:

  • Lifetime starts
  • Lifetime ends
  • Other points of interests, e.g., where we apply the annotations (like when function returns something with the same lifetime as one of its arguments).
  • Point of use after the lifetime ends

However, one could argue that reasoning about individual points might not be the most ergonomic, often where lifetime ends is just a closing curly. It would be nice to be able to draw a vertical line next to the source code representing the region where the object in question is alive. This would greatly improve the understandability of these diagnostics. Some of your examples look like that they could be extended to support this use case.

The other interesting use case is the Clang Static Analyzer that often produces path sensitive reports. It has its own diagnostics engine that can print (inter-procedural) execution paths in the terminal or generate reports as static HTMLs, SARIF, and in some other formats. It would be nice to attempt to remove the duplication and share as much infrastructure within the compiler as possible.
Some of the considerations for the CSA:

  • Visualizing a path/trace across multiple function calls. Maybe nesting could help us visualize how deep the trace is in the call stack.
  • Visualizing events that does not have a corresponding statement or expression. For example, a value can become dead (i.e., it is never read after that program point) at any time. And we often report diagnostics like memory being leaked when that happened. But there is no “statement” or “expression” that leaks the memory.
  • The analyzer ofter tries to add additional explanation to these traces: values of variables, whether we take a branch because of an assumption or taking a branch is inevitable due to the analysis state. Sometimes it even tried to add explanations that are not related to the execution path that we explored. E.g., a value is uninitialized, and we called a function that only initializes memory on a path that we did not take.

For a separate topic, could you show some examples how do you envision fixit hints appearing in the new format?

cc. @gribozavr, @usx95, @ymand for awareness for lifetimes.
cc. @NoQ, @steakhal, @DonatNagyE for the clang static analyzer.

1 Like
7

Afaik what’s happening there is if you include multiple source ranges on different lines in a single diagnostic, it just prints that entire region of code; it would probably be possible to add support for specifying two or more source ranges that are ‘linked’, which would indicate that a line should be drawn connecting them (and maybe we could let you specify some additional text to be printed next to the line too). That seems like it’d be orthogonal to what this RFC introduces though and could probably also be applied to our current default format.

That does sound interesting. Since I’m not too familiar w/ the static analyser, could you give me an example of a diagnostic for which we print execution paths and/or point me to where that diagnostics renderer is implemented so I can have a look at it?

I haven’t really changed anything about fixits, so they’re still printed in the same way as in the default format:

error: use of undeclared identifier 'bar1'; did you mean 'bar'?
|
|     - at fixit.cc:4:2: 
|
|   4 |         bar1();
|     |         ^~~~
|     |         bar
|
|-- note: 'bar' declared here
|
|         - at fixit.cc:1:6: 
|
|       1 | void bar() {}
|         |      ^
8 
$ cat t2.f90
type parameterized(intKind)
  integer, kind :: intKind
  integer(kind=intKind) component
end type
type(parameterized(7)) thereIsNoKind7
end
$ flang-new t2.f90
error: Semantic errors in t2.f90
./t2.f90:3:25: error: KIND parameter value (7) of intrinsic type INTEGER did not resolve to a supported value
    integer(kind=intKind) component
                          ^^^^^^^^^
./t2.f90:5:1: in the context: instantiation of parameterized derived type 'parameterized(intkind=7_4)'
  type(parameterized(7)) thereIsNoKind7
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Because all messages are buffered in flang-new, it is common to both (1) thread a reference to a message buffer through a particular analysis or check to collect (or defer) its messages, or (2) swap out the current active buffer in some context with a new one, collect messages, and put the old one back into place, and then transform the collected messages into an attachment to a new one.

$ cat t3.f90
module mod
  integer :: moduleVar = 0
 contains
  pure subroutine setToOne(n)
    integer, intent(out) :: n
    n = 1
  end
  pure subroutine badSideEffect
    call setToOne(moduleVar)
  end
end
$ flang-new t3.f90
error: Semantic errors in t3.f90
./t3.f90:9:19: error: Actual argument associated with INTENT(OUT) dummy argument 'n=' is not definable
      call setToOne(moduleVar)
                    ^^^^^^^^^
./t3.f90:9:19: because: 'modulevar' may not be defined in pure subprogram 'badsideeffect' because it is host-associated
      call setToOne(moduleVar)
                    ^^^^^^^^^
./t3.f90:2:14: Declaration of 'modulevar'
    integer :: moduleVar = 0
               ^^^^^^^^^
9

@Sirraide Thanks for proposal. This looks like a step in the right direction.

Have you given any thought how your proposal interacts with fix-its?

Having worked with fix-its in the past I ran into a whole bunch of problems with them and I wonder if your proposal is a potential avenue for slowly working to fix them. Here are a few issues I’ve observed

  • The existing rendering of fix-its on the command line is really not great. IIRC the fix-it is rendering as a caret diagnostic in green around the snippet of text that is shown for the diagnostic. There are two problems with this. 1) if the fix-it is for code nowhere near to the diagnostic source location then nothing is rendered at all. 2) if the diagnostic doesn’t mention there’s a fix then it’s super unobvious that Clang is even suggesting a fix. If we are reworking how diagnostics are rendered on the command line then this might be an opportunity to fix the discoverability of fix its.
  • Clang currently has an implicit assumptions about how fix-its attached to diagnostics should be treated (see “Clang” CFE Internals Manual — Clang 22.0.0git documentation ). Currently if you attach a fix it to an error or warning it has to be a “high confidence” fix and every other fix should be attached to a note that’s attached to the warning/error. When there are multiple choices for fix-its I’ve attached multiple notes (each with a different fixit) to the same warning/error diagnostic to represent a disjunction of fix-it choices (I’m not sure if that’s an actually convention that others follow). I have observed that consumers of these diagnostics don’t necessarily understand these implicit assumptions. It might be better for properties of fix-its to be explicit rather than being in implicit from the position in the tree of diagnostics.
10

Not much; my plan was to come back to fixits later and focus on the other aspects for now since there are already enough design decision to be made there. The new format is also not supposed to be stable—at least not right away—and my idea is that we should be able to make (potentially large) adjustments to it as we figure out how to best present all of the information that our diagnostics encode.

Yeah, it’s not great; I haven’t really thought to much about it, but one option would be to more or less copy what rustc does and just print the source line in question twice, i.e. both as written and with the fixit applied (Compiler Explorer):

error: Rust has no postfix increment operator
  --> <source>:11:8
   |
11 |     num++;
   |        ^^ not a valid postfix operator
   |
help: use `+= 1` instead
   |
11 -     num++;
11 +     num += 1;
   |

(Side note: you can’t tell here, but the ++ gets highlighted in red and the += 1 in green.)

Yeah, I agree that that would make sense.

11

Here is one example text diagnostic: Compiler Explorer

1 Like
12

Oh dear, yeah, er, I see what you mean; that really is basically just a soup of notes… It definitely would be nice if we could combine all of that somehow.

13

Excited to see work on making diagnostics more helpful!

I think we could give a multi point diagnostic to give complete lifetime regions (e.g., Diag(LifetimeStart->getExprLoc(), diag::note_lifetime_validity_region) << LifetimeEnd->getEndLoc();). This would give a verbose code snippet with two markers. This verbosity may be better nested under a single lifetime violation error. A better solution could have been to annotate both these start/end markers but I guess that is an orthogonal problem.

Another use case I see is C++20 concepts (similar to recursive constant evaluation). Concepts can easily be nested and chained using logical operators like ||. When a type fails to satisfy a complex concept, the current error messages can become very verbose and difficult to follow. The current diagnostics explain the failure by chaining a series of notes with “because ..”, “and ..” This flat structure doesn’t clearly reflect the hierarchical nature of the concept definition. For example, if a concept is defined as A || B, the output will list the failure of A and the failure of B and both of them can be individually arbitrarily nested. eg: https://godbolt.org/z/YcE13j8qc.

14

Yeah, those concept errors could definitely benefit from this. Earlier I also had another idea: I wonder if we could combine a few of these diagnostics, in the sense that we render them together; that is, if multiple consecutive (or in some other way related) diagnostics point to the same location, but possibly with different carets, we could print the code snippet only once and put the messages next to their carets—I haven’t tried implementing this yet though, so no idea how feasible that’d be in the general case, and it would probably also be something we’d want to control on a per-diagnostic basis because I think some diagnostics would benefit from this a lot more than others (e.g. those concept notes).

We have a number of cases where information is spread out across a number of different notes that could probably be condensed quite a bit, so having some means of doing that would be useful I think.

(Side note: diagnostics consumers that don’t know how to (or don’t care to) handle this sort of formatting could of course simply fall back to emitting individual notes.)