Issues with given prioritization change

[tgodzik]

With 3.7.0 the new given prioritization comes into effect, which might cause libraries like chimney or cats-effect to not work properly even if compiled with an earlier version.

So in essence:

1. Library gets compiled and tests with 3.3.5 LTS . It heavily uses the old priority of givens
2. Someone uses that library in 3.7.0 and the code doesn't work since the compiler behaviour changed.

Do we have a way to fix it for users? We could potentially use the source flag in the user code to revert to older priority, but that makes them unable to use any new features and stuck with 3.6.0.

We could also add a separate flag for that case, but that make everyone need to use that flag until cats reimplement the given resolution, which I am not even sure is doable.

Another, bad idea, is to cross compile pre and after 3.7.0, but that is not something we wanted to ever do.

Do we have any sensible solution to that problem? Cats-effect is one of the wider used libraries, so it would be great to be able to fix the issue even before the new priority comes into effect.

Since I don't know much about details it would be great to hear from someone who knows more details

CC
@armanbilge @lbialy @WojciechMazur

I think Wojciech and Łukasz were doing some tests, but not sure if they managed to find a conclusion.

I have limited knowledge here, so it's possible it's not an issue.

[tgodzik]

@odersky I wanted to discuss it during the previous core meeting since it was raised by a couple of open source contributors as a possible issue.

[djspiewak]

I'll let Arman speak to it since he probably remembers more of the context than I do, but he and I discussed this at length back when the SIP was going through its paces. Our conclusion at the time was that the new prioritization would have no effect on Cats Effect and probably wouldn't affect Cats either, but obviously this stuff tends to be subtle and surprising. There's basically no way to be sure outside of trying to compile a ton of downstream code.

The one place where I could see surprising user-facing things popping up is we tend to do some stuff which looks like this:

  def apply[F[_], E](implicit F: GenConcurrent[F, E]): F.type = F
  def apply[F[_]](implicit F: GenConcurrent[F, ?], d: DummyImplicit): F.type = F

These are the summoners for the Concurrent typeclass. Note that this is a bit different from how it's conventionally done in Cats, where the return type would just be GenConcurrent. We do the path-dependent thing here because we want to ensure that the specific type information is carried through from the inference. This in turn means that users can do Concurrent[F] and the type of that expression might be Async[F] (as an example). The new given priorities definitely don't break this mechanism, but I wonder if we've thought through all the source-level implications at the call site. As I said, it's really hard to tell.

The risk here is real though. Manipulating given prioritization in general requires tucking things into varying levels of the inheritance hierarchy, and so if we have a situation where compiling against 3.7 results in libraries outright breaking, it's possible (though not guaranteed) that we would have no way to fix this in a way which is binary compatible with previous library releases, to say nothing of source compatibility with previous Scala releases.

[bishabosha]

you can detect the tasty version of the symbol and change semantic (sounds evil)

[tgodzik]

The change in question didn't actually go through SIP process I think. I think it was deemed to be minor change, but in reality it might not be.

#19300

[djspiewak]

Ah my bad. Like I said, hazy memory. I was thinking of the PR.

[sjrd]

you can detect the tasty version of the symbol and change semantic (sounds evil)

No you can't. Candidate implicits can come from two sources, with both old and new semantics, but you have to pick one.

[lbialy]

We did in fact execute a spike into this problem. The motivation for this research was that:

a) there were some rather uncomfortable warnings present in the OCB log when the original blast radius analysis was done for this change. Initially the assumption was that these were all fine and the number of ambiguous clashes was very low in OCB so we assumed it's going to be fine.

b) @MateuszKubuszok gave us a hint that this change does break Chimney as it depends on a significance of typeclass hierarchy - namely for Transformer typeclass there's a super type Transformer.AutoDerived[A, B] that has instances generated by a macro and a subtype Transformer[A, B] that is left for user to provide for specific cases. Macro itself looks for Transformer instances only (only instances given by the user) and if there aren't any, attempts to derive instances recursively (by providing AutoDerived[A, B] for these types). This is done for reasons of macro efficiency, better error messages etc. Example of how this fails to compile under new rules can be seen here:

//> using scala 3.7.0-RC1

trait TypeClass[A] extends TypeClass.AutoDerived[A]
object TypeClass {
  trait AutoDerived[A]
  object AutoDerived {
    given [A]: AutoDerived[A] = new TypeClass[A] {}
  }
}

class Foo
object Foo {
  given TypeClass[Foo] = new TypeClass[Foo] {}
}

def use[A: TypeClass.AutoDerived]: Unit = println(summon[TypeClass.AutoDerived[A]])

@main def main = use[Foo]
//-- [E172] Type Error: ----------------------------------------------------------
//16 |@main def main = use[Foo]
//   |                         ^
//   |Ambiguous given instances: both given instance given_AutoDerived_A in object AutoDerived and given instance //given_TypeClass_Foo in object Foo match type TypeClass.AutoDerived[Foo] of a context parameter of method use
//1 error found

There is no way for this to be resolved under new rules and Chimney will have to deal with this by having two parallel major releases, on for scala 2.13 & 3.x where x < 7 and for 3.7+. To make Chimney possible for Scala 3.7+ a new macro combinator was introduced by @jchyb, summonIgnoring, that takes a list of symbols to ignore in implicit search which allows to encode the same logic of finding user-provided instances while ignoring the macro itself. This means however that the hierarchy has to be abolished altogether and that chimney for 3.7+ will only have Transformer[A, B] typeclass, hence new major release.

This realisation made us wonder if the same will happen to cats / cats-effect and typelevel ecosystem and given the potential blast radius of this possibility, we spent some time investigating. The result of our spike is that we haven't been able to find conclusive evidence of breakage of significance similar to the one in Chimney. We have explored three approaches:

1. first approach was to force all cats typeclasses to be resolved using new rules. To do that we had to declare them using givens. I forked cats and cats-effect with the goal of using these forks in OCB to find compile and runtime issues at scale. I replaced all implicit uses beside implicit conversions (things like implicit def f[F[_]: Constraint](a: A)(implicit F: Constraint[F]): FOps[F, A]) with givens, then did the same with cats-effect. This was a rather non-trivial piece of work given that there are some implicit patterns that are impossible to translate to givens (for example in some settings givens are final and overriding them is impossible) and that literal replacement of implicits with givens led to at least some cases of deadlocks on lazy vals (@WojciechMazur can elaborate on this). The farthest I was able to get with this was that main scope did compile and publish locally for both libraries (test scope did not however!). Execution on OCB with these forks did not yield any new and significant information about breakage from what I recall but it was also plagued with issues arising from problems orthogonal to the given priority change - breakage introduced by forks being subtly incompatible version of Cats/CE in comparison to versions used in OCB targets, difficulty in comprehensive replacement in imports for import cats.implicits.given etc (many places use specialized imports and it's quite difficult to correctly mark these imports as import given at scale), aforementioned lazy val deadlocks locking tests for things that did manage to compile etc.

Cats fork
Cats Effect fork

Disclaimer: these forks were in no way an attempt to port cats/CE to S3 givens in a sane and comprehensive way.

2. another approach was to leverage what compiler can do for us - @jchyb implemented a change to the dotty parser that made it treat all implicits as givens. This worked to some degree but also run into many limitations and orthogonal problems. No new breakage information was found using this approach.

3. in depth investigation of cases found in the original report by @WojciechMazur - prior to Scalar I tried to investigate the nature of warnings of code indicated as impacted in the gist and also of errors in cats/cats-effect test scopes. This was quite straightforward - I explored what is the structure of the typeclass hierarchy in cats for several positions highlighted as changing the resolved type in 3.7+ and tried to prove that it is in fact selecting a different concrete instance. In all cases I looked into there were several types in the hierarchy and just a single concrete instance that implemented them. Moreover, in most of these cases the warning was caused by user depending on the more general typeclass, old resolution rules providing instance matching lowest subtype of the typeclass and new rules providing instance matching the demanded general super type like in this case from gvolpe/trading:

      {
        "change": {
          "target": "cats.kernel.Eq[A]",
          "first": "cats.kernel.Order[A]",
          "second": "cats.kernel.Eq[A]",
          "currentChoice": "FirstAlternative",
          "newChoice": "SecondAlternative"
        },
        "count": 1
      }

The hierarchy is Eq[A] <- PartialOrder[A] <- Order[A], user demands Eq[A], the instance that exists implements Order (so transitively it also implements Eq), selection rules change from Order to Eq but in fact it is exactly the same concrete instance that is being resolved. This logic applied to all examples I have manually checked but that wasn't a huge number of cases. It did however manage to convince me that it's quite probable that we won't run into a catsastrophe as I feared with the release and subsequent migration of Cats/CE to new Scala 3 LTS version. There still remains the problem of issues with test scope in forks of Cats and CE where I did run into ambiguity errors. Short exploration of these issues made me think that these problems are related to how test instances are provided and that's why we didn't run into these issues when compiling code at scale in OCB. It would probably still be good if someone who is more intimately familiar with how cats/CE typeclasses are organized and implemented by concrete instances looked into these test scope compile failures in our forks.

[djspiewak]

@lbialy That's a wonderful bit of investigation thank you. I'll try to look at your forks at some point this week.

Your explanation did make me realize that we might have some problems related to this which wouldn't show up in functional testing. In particular, there are a bunch of places both in Cats and Cats Effect where a more specific instance of a typeclass overrides the definition of some concrete method with a more efficient implementation, basically leveraging the stricter guarantees it has available to it by being lower in the hierarchy. We actually even surface this information (internally) at runtime and use it to manually specialize certain things. In particular, look at Queue, as well as the overrides of GenConcurrent#deferred.

Now, for concrete types like IO there's really only one possible instance anyway (IO.asyncForIO), so it's basically fine, but anyone touching monad transformers (or Resource, which is really widely used) is going to get a whole slew of plausible derived instances, and specificity starts to really matter. Getting the most general derived instance would result in silent deoptimization of call sites. The exact order of magnitude of impact here is hard to predict, and it's very use-case specific, but in some benchmarks the more optimized Async-based Queue is 5-10x faster than the more generic Concurrent-based Queue (this is deceptive though because usually thread contention dominates). That sounds pretty bad, particularly since major libraries like Http4s do indeed heavily leverage monad transformers and Resource, but Queue itself isn't actually used heavily because Fs2 provides Channel (which has only a single implementation). Channel is based on Deferred though, which does do this kind of runtime specialization stuff.

So in other words… I do think there's probably some danger here. I really would be surprised if anything functionally breaks at compile- or runtime, but I could definitely see cases where performance is silently regressed by significant amounts.

[MateuszKubuszok]

It might or it might not be relevant but...

When I was checking whether there's a way to work around the issue in Chimney without a new major version, I found out that while:

foo.transformInto[Bar] // summons Transformer.AutoDerived[Foo, Bar]

triggers an issue on 3.6, and and ambiguity error on 3.7

foo.into[Bar].transform

does not. The later triggers a macro that does look for implicits but it if does not find one, it generates an inlined transformation. I checked that it did find the implicit, so it used the old semantics. One inline def with summonInline later and I was certain:

• it was said that for implicit-only the old semantics is used
• if using or given pops out anywhere, the new semantics is used
• but if implicit search is triggered by the macro, it would also use the old semantics

so there's a chance that some issues would come from the type class derivation, in libraries that use macros, once that "bug" is fixed.

Since the type class derivation with Mirrors usually relies on summonInline (which is a macro), that might have some impact.

[lbialy]

oh, I think the fact that summonInline uses implicit semantics inside quite definitely needs it's own issue 😱

[lbialy]

@djspiewak In this case the best way to verify this (given my limited knowledge) would be to create a catalog of all C/CE typeclasses along with their parents and verify which instances get summoned for which scala version (and I'm making a generous assumption here that you can't get different instances depending on the import you use - cats.syntax.stdSomething vs cats.implicits.* for example).

[djspiewak]

@lbialy Here's some quick resources:

• Very old, but the Cats part mostly hasn't changed (Cats Effect and Cats MTL have changed dramatically): https://github.com/tpolecat/cats-infographic
• Up to date, but only Cats Effect: https://github.com/typelevel/cats-effect?tab=readme-ov-file#abstraction

Might be easier to generate something more bespoke by traversing the inheritance hierarchies.

[joroKr21]

There is no way for this to be resolved under new rules and Chimney will have to deal with this by having two parallel major releases, on for scala 2.13 & 3.x where x < 7 and for 3.7+. To make Chimney possible for Scala 3.7+ a new macro combinator was introduced by @jchyb, summonIgnoring, that takes a list of symbols to ignore in implicit search which allows to encode the same logic of finding user-provided instances while ignoring the macro itself. This means however that the hierarchy has to be abolished altogether and that chimney for 3.7+ will only have Transformer[A, B] typeclass, hence new major release.

Why do you need a special primitive for that? Have you tried using summonFrom? I've been using summonFrom with opaque types to control implicit search for some time now and it seems to work ok. We even added it to Shapeless 3 recently: https://github.com/typelevel/shapeless-3/blob/main/modules/deriving/src/main/scala/shapeless3/deriving/deriving.scala#L91-L111

  opaque type OrElse[A, B] = A | B
  inline given orElse[A, B]: OrElse[A, B] = summonFrom:
    case instance: A => OrElse(instance)
    case instance: B => OrElse(instance)

first approach was to force all cats typeclasses to be resolved using new rules. To do that we had to declare them using givens. I forked cats and cats-effect with the goal of using these forks in OCB to find compile and runtime issues at scale. I replaced all implicit uses beside implicit conversions (things like implicit def f[F[_]: Constraint](a: A)(implicit F: Constraint[F]): FOps[F, A]) with givens, then did the same with cats-effect.

How did shepeless3, kittens and cats-tagless fare in those OCB runs? These projects use givens for derived type classes because they have a totally different implementation for Scala 3.

Since the type class derivation with Mirrors usually relies on summonInline (which is a macro), that might have some impact.

But then this would mean it was not really tested as intended.

The farthest I was able to get with this was that main scope did compile and publish locally for both libraries (test scope did not however!).

This is concerning, I would assume a green OCB would be a prerequisite for this kind of big change.

The point about optimization in subclasses is a really good one. Specifically for generic types and monad transformers, to provide an instance you need to require an instances for your type parameters. E.g. Eq for Option would depend on Eq for the element, Order for Option would depend on Order for the element, Functor for OptionT would depend on Functor for the type parameter and so on. So in those cases it cannot be a single instance which implements everything. And if you couple that with the recommendation to require the least powerful type class needed to implement a piece of functionality, then most probably optimizations will be lost.

[lbialy]

OCB was mostly green for this change! It wasn't for my forks where I refactored cats/cats-effect to define implicits as given/using. The problem is a bit deeper. Cats, cats-mtl and cats-effect define their implicits as, well, implicits and not givens. Then, to make things worse, a LOT of existing code uses implicit arg lists instead of using. The true impact of this change will only surface once foundational libs like these 3 will move to givens and more users start to use using to demand evidences. This will be probably a quite long process so I hope we can avoid sudden "everything is broken and we need to release a separate artifact family for those who want to use 3.7+" phase.

Regarding the special primitive - I believe that's due to a) existing code that can't be broken that b) was written during 2.11, then refactored during 2.12 and 2.13 and then updated to work with Scala 3. No summonFrom on 2.x so no clever Scala 3 hacks possible without breaking changes for Chimney. @MateuszKubuszok can probably provide a lot more context on this. Not that 3.7+ won't mean breaking changes for Chimney - it will, unfortunately, but with summonIgnoring there's at least a path forward. Mateusz, would that clever trick with summonFrom work for your use case?

[joroKr21]

The true impact of this change will only surface once foundational libs like these 3 will move to givens and more users start to use using to demand evidences.

Wouldn't it be quite confusing that implicit and given have different resolution rules? At the very least because they can be mixed in the same chain. I'm not even sure what are the new rules now, when do they apply?

[MateuszKubuszok]

Why do you need a special primitive for that? Have you tried using summonFrom?

I think I may need to add some context, why Chimney needed such scheme.

1. at some point we noticed, that with macros we can add much better error messages
2. we also noticed that with 2 type classes: Transformer[From, To] (From => To) and PartialTransformer[From, To] (From => partial.Result[To]) only a fraction of operations inside Partial actually needed the error handling - so the naive approach when we lift everything from total to partial computations and then use something-like an applicative to glue them together - generated a lot of unnecessary boxing
3. we also measured that when a single macro expansion avoids calling another macro for its subexpression, but instead just performs recursion inside a single expansion, it's much less taxing on typer, resulting in faster compilations (often faster even than using semi-auto everywhere)

In other words, having the macro avoid implicit search that resolves to itself, enables better error messages (even for nested fields), better runtime performance and better compilation times. It does not affect correctness, since it's only about whether there is several intermediate type class instances instead of one, having their code inlined, but it improves only UX which is probably why nobody cared about doing such fancy tricks in their libraries.

Anyway, this pattern was a way to achieve just that: recursive derivation that is not broken by enabling automatic derivation: having extension methods summon either user-provided instances or falling back to derived ones, and having macros always treating user-provided instances as "overrides" of the default behavior, and using plain-old recursion otherwise.

trait TypeClass[A] extends TypeClass.AutoDerived[A]
object TypeClass {
  trait AutoDerived[A]
  object AutoDerived {
    // Macro looks only for TypeClass, if there is none, it generates the transformation
    // while handling the transformation recursively
    inline given [A]: AutoDerived[A] = ${ ... }
  }
}

def extension[From](value: A)
  // If user provided TypeClass[A], it would be preferred
  // it not, it will be derived as TypeClass.AutoDerived[A]
  def useTypeClass(using t: TypeClass.AutoDerived[A]): A = ...

This was designed in such a way to have the requirement fulfilled AND keep the same API on: 2.12, 2.13, 3. If somebody does foo.transformInto[Bar](using explicitlyProvided) they can, on all Scala versions.

I did researched summonFrom as a workaround, but it still:

• is a breaking change - to use it I have to implement inline def that takes no parameters, so whoever used foo.transformInto[Bar](using explicitlyProvided) now would have to rewrite the code
• it requires splitting hierarchies of Transformer and Transformer.AutoDerived otherwise

  summonFrom {
    case t: Transformer[From, To] => ...
    case t: Transformer.AutoDerived[From, To] => ... // <--- ambiguous implicit again!
  }
• the split hierarchies would result in code like:

  given myCustomImplicit[A, B](using inner: Transformer.AutoDerived[A, B]): Transformer[Something[A], Something[B]] = ...
  no longer working when given Transformer[A, B] would be provided manually by the user (since it could no longer we simply upcasted to Transformer.AutoDerived[A, B])

so we would have to add some other hacky workarounds for that as well...

Then, there is the risk that yet another SIP would pull the rug from under us again, and then we would need another rewrite, and another, and another...

Luckily for us, summonIgnoring was added, so the whole scheme becomes unnecessary, I just ban some method symbols in the macro, and it happily recurses. I could use it to reimplement derivation in some other libraries (to provide better error messages, compile times and performance).

But still, it only works for 3.7.0+, while the previous version worked on 2.12, 2.13 and everything before 3.7.0, so at least for us the prophecy about needing to support 2 versions: pre- and post-3.7.0 has fulfilled.

[odersky]

I have asked @EugeneFlesselle to study this and comment.

[EugeneFlesselle]

I have asked @EugeneFlesselle to study this and comment.

This comment is more of a summary of the thread than proposing solutions.

The original question was how to address situations for libraries (with a particular focus on the Chimney and Cats libraries) where:

1. Library gets compiled and tests with 3.3.5 LTS . It heavily uses the old priority of givens
2. Someone uses that library in 3.7.0 and the code doesn't work since the compiler behaviour changed.

The two main cases discussed where selecting the most general instance was not the desirable outcome are:

1. For specialization Issues with given prioritization change #22913 (comment)
2. For prioritization of user-provided instances over library-derived instances
   • using a design pattern where the latter are given for some super trait of the type class (created specifically for de-prioritization) Issues with given prioritization change #22913 (comment)

Another concern having been brought up is that

a LOT of existing code uses implicit arg lists instead of using. The true impact of this change will only surface once foundational libs like these 3 will move to givens and more users start to use using

The high-level conclusions from the two libraries were that:

Chimney will have to deal with this by having two parallel major releases, on for scala 2.13 & 3.x where x < 7 and for 3.7+. [each tailored for the corresponding prioritization scheme to be adequate a use-sites]

[cats / cats-effect] we haven't been able to find conclusive evidence of breakage of significance similar to the one in Chimney. We have explored three approaches: #22913 (comment)
[Cats] I really would be surprised if anything functionally breaks at compile- or runtime, but I could definitely see cases [not picking the specialized instance] where performance is silently regressed by significant amounts.

The decision is now whether there is anything to do from the point of view of the language or if there are recommendations we should make for library authors/users.

[odersky]

Thanks for the summary! One conclusion for me is that we might want to move forward with #22580, so that we can give early warnings to library developers about possible breakages when moving from implicits to givens.

[EugeneFlesselle]

The later triggers a macro that does look for implicits but it if does not find one, it generates an inlined transformation. I checked that it did find the implicit, so it used the old semantics. One inline def with summonInline later and I was certain:

• it was said that for implicit-only the old semantics is used
• if using or given pops out anywhere, the new semantics is used
• but if implicit search is triggered by the macro, it would also use the old semantics

so there's a chance that some issues would come from the type class derivation, in libraries that use macros, once that "bug" is fixed.

This is very surprising. AFAIK, there is no logic applying a different resolution scheme from the inliner and/or macros. I verified with @hamzaremmal that we do select the most general instance for compiletime.summonInline as well, including when it is used in a macro.

I'm not familiar with the definitions used in the two code snippets from #22913 (comment), @MateuszKubuszok could there be another reason they yielded different results?

[lbialy]

@MateuszKubuszok you had a minimizer available, didn't you?

[MateuszKubuszok]

@lbialy, @EugeneFlesselle I have this Scastie snippet - it shows how:

• an extension method with using breaks with the new semantics
• macros/summonInline keep working (which is why Ducktape has no such issue even though it copied Chimney's approach)

AFAIR The warning appeared in 3.5 together with -source 3.6 flag, in 3.6 it always appears, and in 3.7 turns into error - so it's either unexpected side-effect of new semantics, or unintended change in behavior got shipped together with it.