Add more diagnostics for compiler performance analysis #5760
Conversation
- `-Yprofile` will output the difference between snapshots of GC and CPU
snapshot data, sourced from platform MBeans
- This output can be sent to a file with `-Yprofile-destination <filename>`
By default, output is to console
- Use `-Yprofile-external-tool` will generate a call to the static method
`before` and `after` around each of the the given phases. This can be
used to communicate with an external profiler, such as YourKit, to
generate a profile for a subset of the compiler.
- `-Yprofile-run-gc` runs the GC after each phase to help more accurately
attribute retained heap to a given phase.
Co-Authored by: Jason Zaugg <[email protected]>
This is handy when collecting samples in YourKit. The actual result of this class is just an approximation, we rely on JMH in scala/compiler-benchmark for more rigourous statistics. ``` ./build/quick/bin/scala -J-Dscala.benchmark.iterations=2000 scala.tools.nsc.MainBench sandbox/test.scala ```
|
Rebase of #5758 |
There was a problem hiding this comment.
I started reviewing this today and got sucked into getting per-phase profiling with JFR. This is my WIP branch: lrytz@9d16bb3
- Run the compiler with
-Yprofile-enabled -J-XX:+UnlockCommercialFeatures - In JMC, for the compiler process, open the "MBean Server" console and go to the "Triggers" tab
- Import the config https://gist.github.com/lrytz/aa0e72e00a3f1fadee92d62f9708dfbf, adjust hard coded paths
- enable the triggers, recording starts and stops automatically during
typer
I haven't figured out how to do that from the command line, unfortunately..
| withPostSetHook( _ => YprofileEnabled.value = true ) | ||
| val YprofileExternalTool = PhasesSetting("-Yprofile-external-tool", "Enable profiling for a phase using an external tool hook. Generally only useful for a single phase", "typer"). | ||
| withPostSetHook( _ => YprofileEnabled.value = true ) | ||
| val YprofileRunGcBetweenPhases = PhasesSetting("-Yprofile-run-gc", "Run a GC between phases - this allows heap size to be accurate at the expense of more time. Specify a list of phases, or *", "_"). |
There was a problem hiding this comment.
both _ and * are not valid, should use all instead
| s2 - s1 | ||
| } | ||
| private def doGC(): Unit = { | ||
| System.gc() |
There was a problem hiding this comment.
Javadoc says: "When control returns from the method call, the Java Virtual Machine has made a best effort to reclaim space from all discarded objects", I guess that's still true for concurrent GC? Anyway, we just have to be aware that System.gc is probably not the most reliable tool.
There was a problem hiding this comment.
the GC before/after was intended to provide some indication of the ratio of allocation vs retained sizes. Generally the information that this tool provide is indicative, and not 100 % reproducible, but with sufficient iteration and post processing of the data can provide a high confidence that a particular PR affected a certain metric
There was a problem hiding this comment.
for the record - it also has inaccuracies related to finalization and object graphs that require multiple GC/finalization cycles to reclaim, and also with soft references
|
tentatively retargeted for 2.12.3, change it back if that's wrong |
|
Hi @retronym we have updated the base profiler to support capturing of stats from background threads (needed for #5815), like thread CPU time and allocations This also includes better output control and formatting in https://github.com/rorygraves/scalac_perf/tree/2.12.x_profile2 is it best to take these additonal commits to a new PR or adjust the base of this one? |
|
I merged the original PR instead. I'll salvage any useful changes in this PR in a new one. |
-Yprofilewill output the difference between snapshots of GC and CPUsnapshot data, sourced from platform MBeans
-Yprofile-destination <filename>By default, output is to console
-Yprofile-external-toolwill generate a call to the static methodbeforeandafteraround each of the the given phases. This can beused to communicate with an external profiler, such as YourKit, to generate a
profile for a subset of the compiler.
-Yprofile-run-gcruns the GC after each phase to help more accuratelyattribute retained heap to a given phase.