Touching a file in src/test/resources triggers test:fastOptJS

[japgolly]

If I touch a file in src/test/resources, it causes a rebuild of my test JS and taking about 5 seconds.

webapp-client> last test:fastOptJS
[info] Fast optimizing .../target/scala-2.11/webapp-client-test-fastopt.js
[debug] Linker: Read info: 25703 us
[debug] Linker: Compute reachability: 475989 us
[debug] Linker: Assemble LinkedClasses: 859315 us
[debug] Linker: cache stats: reused: 15325 -- invalidated: 0 -- trees read: 0
[debug] Linker: 1367511 us
[debug] Inc. optimizer: Batch mode: false
[debug] Inc. optimizer: Incremental part: 18493 us
[debug] Inc. optimizer: Optimizing 0 methods.
[debug] Inc. optimizer: Optimizer part: 50 us
[debug] Inc. optimizer: 32347 us
[debug] Refiner: Compute reachability: 364890 us
[debug] Refiner: Assemble LinkedClasses: 19782 us
[debug] Refiner: 384803 us
[debug] Emitter: Class tree cache stats: reused: 4193 -- invalidated: 0
[debug] Emitter: Method tree cache stats: resued: 20894 -- invalidated: 0
[debug] Emitter (write output): 278975 us

I think this is separate from the incremental linker issue in that regardless of how fast linking can be, it shouldn't be triggered if the underlying compilation units haven't changed.

[gzm0]

It definitely is separate from the incremental linker issue. But it is actually pretty hard to solve in our current classpath model.
@sjrd what do you think about that one? It is definitely fixable, the question is more how backwards compatible can we make the tools API...

[gzm0]

I'll try and have a look at this one.

[sjrd]

No need to change or classpath utilities, nor should we.

We can supply have the Linker short-circuit right after updateFiles if no file has been read again from disk, nor disappeared. The rest of the pipeline can be short-circuited as well from there. At this point we should still probably touch the final .js file, though, or this much will be repeated over and over again until some sjsir finally changes.

[gzm0]

This is harder than it sounds, notably because of the builder abstraction: Once we instantiate a builder for a file, that file is gone, even before we start the compilation pipeline.

An option is to add the following method to JSTreeBuilder:

/** Tells the builder, that the resulting tree is unchanged.
  *
  *  If the builder replies true, the building terminates.
  *  If the builder replies false, the caller must rebuild the tree.
  */
def unchanged(): Boolean = false

This will effectively allow the client of the pipeline decide whether it wants the pipeline to continue (e.g. file was deleted).

However, this also requires to change the JSFileBuilder (and friends) to not take streams but files (or probably virtual files), since otherwise they are unable to deal with the unchanged method.

Further, I considered implementing the change propagation by adding a version field to the linking unit. It seems better than having each method in the Linker return an optional value (and pushing force parameters down the callstack). Again, this allows the client to decide, whether it wants to continue the pipeline anyways a posteriori.

The downside is, that we now hold on to the last linking unit produced by the Linker, which can be memory heavy. Although I have not benchmarked this (I don't have a running version of this yet).

[gzm0]

WIP branch here: https://github.com/gzm0/scala-js/tree/no-link-unchanged

[gzm0]

@sjrd I was hoping on a comment from your side, whether you think the above is a reasonable approach. It seems to be a more fundamental change than the ticket initially suggests.

[sjrd]

I think you are over-engineering it. It's not necessary to pipe stuff through the pipeline. It's fine to rerun the whole pipeline if one IR file changed in the beginning, even if that leads to producing the same LinkedUnit somewhere in the middle. That event should be rare enough that it doesn't matter.

What we want to avoid is rerunning the pipeline if the set of IR files hasn't changed a bit (but other stuff on the classpath, such as resources, have been touched).

Therefore, I think the version in LinkedUnit is overkill. Generalizing JSTreeBuilder also seems too broad.

Instead of changing Linker.link to return (linkingUnit, hasChanged), I would separate the operation in two methods:

  def prepareLink(irInput: Traversable[VirtualScalaJSIRFile], logger: Logger): Boolean
  def link(logger: Logger,
       reachOptimizerSymbols: Boolean, bypassLinkingErrors: Boolean,
       noWarnMissing: Seq[NoWarnMissing],
       @deprecatedName('checkInfos) checkIR: Boolean): LinkingUnit

The first one does updateFiles and returns whether something has changed or not. With that ScalaJSOptimizer can decide what to do; it already works with a VirtualFileWriter, so it should know how to test whether it's deleted or not, etc. If ScalaJSOptimizer wants the linker to proceed, it calls link with the rest of the parameters.

A cleaner separation of concerns would be to simply extract the VirtualScalaJSIRFile cache into a separate class, separated from Linker. Linker.link would take a cache as a parameter. ScalaJSOptimizer would maintain its own instance of the cache, and use that to first update it with the new classpath, then optionally calling Linker.link.

This way, we avoid complicating the entire pipeline.

(We can also keep the old link method, which calls the other two; so we keep full compatibility here.)