slog: Support structured logging

[ellemouton]

This PR does a few things:

• turns this package into a go module
• Expands the existing Logger interface with new structured logging messages (DebugS, InfoS etc)
• let's the existing default implementation implement these new methods (in an unstructured way)
• add a new log/slog implementation of the Logger that uses proper structured logging.

Details

• This new implementation is flexible enough to allow callers to pass in their own slog.Handlers (which underneath could be a json formatter, a logfmt formatter or any anything else like a charm Handler

Usage

The idea is that the msg passed to an S method should be a constant. The attr... params are then used for key-value pairs.

log.InfoS(ctx, "User performed a request",
			"user_id", id,
			"request_uri: ", uri)

Each new method also takes a context.Context. The WithCtx helper can be used to attach any key-value pairs to a
context and then when the logger is called, the pairs will be extracted from the context and added as attributes in the log.

For example:

ctx := WithCtx(context.Background(), "user_id", id)

// This log will include both the `user_id` and `request_uri` attributes.
log.InfoS(ctx, "User performed a request", "request_uri", uri)

[guggero]

Very nice! LGTM, just a couple of nits and/or suggestions.
Really excited to get this in so we can have structured logs in lnd.

[ellemouton]

Thanks @guggero 📁

[ellemouton]

Note to reviewers: Im going to look into writing a Default Handler that can be passed to the NewSLog() constructor that will result in the output looking identical to the existing sublogger output. If this is done, then we can actually just completely remove the subLog impl in this PR

[ellemouton]

Note: I have updated the PR to include a default Handler that can be used with the new NewSlogger method :)

[ellemouton]

I have also added a test now

[guggero]

Great stuff! Super cool that we can get all of those features with zero additional dependencies 💯

[guggero]

Do we need to process the attrs first? Or is the assumption that msg would never contain formatting directives? Then perhaps we should call into l.Trace() instead.
Example of what I mean:

bar := "bar"
subLog.TraceS(ctx, "incorrectly used directive: %v", "foo", bar)

What I think would be printed:

incorrectly used directive: foo%!(EXTRA string=bar)

What the user would expect to be printed:

incorrectly used directive: foo=bar

[ellemouton]

cool yeah agreed

[ellemouton]

ah - the thing is thought that for this implementation, under the hood there is not actually structured logging going on. So the output we get will not look like "key=value" etc. So I used the *f methods just so that we are passing the msg and attr to some underlying impl. the non f methods only take ...attr so the message would not be used.

So this was just to somewhat complete this implementation. In reality if callers want structured logging, they should use the new slog implementation. In the LL fork PR, I completely remove this impl anyways. So I think instead of trying to get this to work as if it is using slog underneath, let's just add a warning to the impl saying it works like the *f methods.

Taking a step back: i think we arent gonna get this into this repo anyways and so we will carry on with the forked version which removes this impl anyways

[ellemouton]

EDIT given latest updates to the convo: perhaps I should just look into a best effort way of converting the given attributes to k=v pairs and writing those to the writer

[guggero]

nit: missing full stop.

[yyforyongyu]

Pending Oli's comments, otherwise LGTM🙏

[ellemouton]

Ok yall: thanks so much for review here! I think we are gonna go with the forking path here as I dont think the maintainer(s) of the repo is/are active.

So let's move discussion & final review there: lightninglabs#1

EDIT: nevermind... if we fork this then we'd also need to fork every other btcsuite PR that we use that imports the logger....
Ideally we need merge here.

[Roasbeef]

cc @davecgh

[davecgh]

@Roasbeef What's up? I guess maybe perms? I really no longer have anything to do with maintaining btc things.

A couple of things I noticed on a quick review though is:

1. Making this a module that requires Go 1.21 will force every single consumer, even those that do not want to use the new logging semantics, to be hoisted forward and themselves require Go 1.21. We no longer use this code, so it wouldn't affect us, but if we did, this change would 100% be a showstopper as a result of that.
2. Allowing and encouraging ANSI codes (styled output) is just asking for log injection attacks. That is something I personally would never allow in anything even remotely related to cryptocurrency code.

[davecgh]

Aside from the two aforementioned things, and some nits like missing copyright headers, the changes otherwise look fine.

As I said though, I no longer maintain anything btc, so I'll defer to @Roasbeef to make merge decisions.

[ellemouton]

Making this a module that requires Go 1.21

The reason that go1.21 was chosen is cause log/slog is only available in that version onwards.

Allowing and encouraging ANSI codes (styled output) is just asking for log injection attacks

it is off by default - but if we are worried about this, i can update the PR to just allow the user to pass in various Style(string)string call-backs instead of specifying anything specific here. Then the risk is totally left up to the caller.

so I'll defer to @Roasbeef to make merge decisions.

Ok cool

[davecgh]

The reason that go1.21 was chosen is cause log/slog is only available in that version onwards.

Aye, I understand the reasoning. The code was changed over to define Level as a slog.Level (and some technically incompatible changes to the constant values, but since everything really should be using the constants, that doesn't matter so much) and it also now embeds a new instance of a log/slog.

Regardless of the rationale though, the effect is that anything that uses it will necessarily be hoisted forward thereby forcing everything that uses those things to also be hoisted forwards all the way down the dependency stack.

For example, take github.com/btcsuite/btcd/wire. It currently only requires Go 1.17 and it's imported by nearly 3000 modules, a huge number of which themselves only require Go 1.17. This would force every single one of those, and anything that uses them to now require Go 1.21. For many projects, that would prevent them from being able to update to the latest versions of the code.

Even something relatively simple like github.com/btcsuite/btcd/btcutil/psbt currently only requires Go 1.17 and is imported by ~100 modules, many of which themselves also only require Go 1.17.

Without looking super closely, I imagine it would be possible to make this compatible with Go 1.17 by implementing the relevant slog interface and not defining anything explicitly in terms of the slog types. That way, it'll still work fine with older Go versions and still be able to interop with the log/slog package in Go 1.21+.

[ellemouton]

the effect is that anything that uses it will necessarily be hoisted forward thereby forcing everything that uses those things to also be hoisted forwards all the way down the dependency stack.

But this is only the case if they want to update to the latest version right? And they would only want to update to the latest version if they want the new features which in this case is slog stuff which would anyways require them to run go 1.21.

The only other reason they'd want to update their btclog dep is if there is some bug found in the old code & they want to update to include just that fix - but for that, we can just maintain a branch that only includes the old implementation & backfills the fix. That way they can point to that branch.

It just seems like we should be able to move logging forward and take advantage of new features that go provides us in terms of logging so that those users that want to take advantage of newer features can do so

I imagine it would be possible to make this compatible with Go 1.17 by implementing the relevant slog interface and not defining anything explicitly in terms of the slog types. That way, it'll still work fine with older Go versions and still be able to interop with the log/slog package in Go 1.21+.

Meaning just update the interface but have the implementation be defined elsewhere (ie not in this repo)? This would work except for the tricky situation with the slog.Levels since I think slog requires the use of its Level numbers... so that makes things tricky I think... I'll double check this to make sure though

[Roasbeef]

I think we might be able to get away with using the old version of the package: https://pkg.go.dev/golang.org/x/exp/slog. It doesn't look to use as many recent language features.