Build time regression on Linux

[MichalStrehovsky]

As seen on https://aka.ms/aspnet/nativeaot/benchmarks.

I think this is from object writer change. I did a quick test with dotnet new webapiaot with/without the LLVM objwriter and I can see a difference on Linux (or... when targeting Linux - I was on Windows). FWIW, Windows seems to be better than before.

TodosApi app:

BasicMinimalApi app

I made a quick profile and at minimum we're spending a lot of time zeroing memory:

Cc @filipnavara

Tagging subscribers to this area: @agocke, @MichalStrehovsky, @jkotas
See info in area-owners.md if you want to be subscribed.

Issue Details

---

As seen on https://aka.ms/aspnet/nativeaot/benchmarks.

I think this is from object writer change. I did a quick test with dotnet new webapiaot with/without the LLVM objwriter and I can see a difference on Linux (or... when targeting Linux - I was on Windows). FWIW, Windows seems to be better than before.

TodosApi app:

BasicMinimalApi app

I made a quick profile and at minimum we're spending a lot of time zeroing memory:

Cc @filipnavara

Author:	MichalStrehovsky
Assignees:	-
Labels:	area-NativeAOT-coreclr
Milestone:	9.0.0

[filipnavara]

That’s somewhat surprising because I tested these scenarios and there was not a regression on my machine. I was testing with the DOTNET_USE_LLVM_OBJWRITER switch set to 0/1 to avoid influence from other factors, like the LINQ change that happened to be in the same time frame and caused size regression.

If you have the scenario setup locally I would really appreciate if you can compare the time of the same build just with the DOTNET_USE_LLVM_OBJWRITER environment variable set to 0/1.

There are definitely some ways to improve the performance of the code in the DWARF debug info writer but I would like to understand the magnitude of the regression.

[MichalStrehovsky]

For dotnet new webapiaot, I was seeing these total times (measured with ptime):

Managed objwriter:
8.635 s
9.148 s
9.206 s
9.194 s

Native objwriter:
8.697 s
8.499 s
8.429 s

Then switched to managed again:
8.949 s
9.015 s

It is a bit all over the place but I think there's a theme and 6% of total CPU spent in ZeroMemory looks like a thing we can improve.

Note I measured ILC-compiled-with-ILC.

[filipnavara]

Thanks for the data. Notably, I run most of the x64 tests on a Ryzen 7950X machine which happens to be quite fast at zeroing memory (even in ilc-published). I agree that there's a room for improvement. Lot of the places in the profile don't actually need to zero memory, it looks like a side effect of ArrayBufferWriter.Clear in a place where we could use ArrayBufferWriter.ResetWrittenCount.

[filipnavara]

Another round of spot the mistake with too much memory clearing:

        public readonly void EmitSymbolReference(
            RelocType relocType,
            string symbolName,
            long addend = 0)
        {
            IBufferWriter<byte> bufferWriter = _sectionData.BufferWriter;
            int size = relocType == RelocType.IMAGE_REL_BASED_DIR64 ? sizeof(ulong) : sizeof(uint);
            Span<byte> buffer = bufferWriter.GetSpan(size);
            buffer.Clear();
            _objectWriter.EmitRelocation(
                SectionIndex,
                Position,
                buffer,
                relocType,
                symbolName,
                addend);
            bufferWriter.Advance(size);
        }

I'll submit a fix later today.

[filipnavara]

FWIW: This is the profile of building webapiaot template with #96956:

  100%   EmitObject  •  3,200 ms  •  ILCompiler.ObjectWriter.ObjectWriter.EmitObject(String, IReadOnlyCollection, IObjectDumper, Logger)
  ► 26.5%   GetData  •  847 ms  •  ILCompiler.DependencyAnalysis.DehydratableObjectNode.GetData(NodeFactory, Boolean)
  ► 16.0%   EmitObjectFile  •  514 ms  •  ILCompiler.ObjectWriter.ElfObjectWriter.EmitObjectFile(String)
  ► 8.83%   GetMangledName  •  282 ms  •  ILCompiler.ObjectWriter.ObjectWriter.GetMangledName(ISymbolNode)
  ► 8.47%   GetData  •  271 ms  •  ILCompiler.DependencyAnalysis.MetadataNode.GetData(NodeFactory, Boolean)
  ► 3.66%   GetData  •  117 ms  •  ILCompiler.DependencyAnalysis.DehydratedDataNode.GetData(NodeFactory, Boolean)
  ► 3.30%   EmitDebugSections  •  105 ms  •  ILCompiler.ObjectWriter.UnixObjectWriter.EmitDebugSections(IDictionary)
  ► 3.12%   EmitSymbolTable  •  100 ms  •  ILCompiler.ObjectWriter.ElfObjectWriter.EmitSymbolTable(IDictionary, SortedSet)
  ► 2.95%   GetTypeIndex  •  95 ms  •  ILCompiler.UserDefinedTypeDescriptor.GetTypeIndex(TypeDesc, Boolean)
  ► 2.38%   EmitRelocations  •  76 ms  •  ILCompiler.ObjectWriter.ElfObjectWriter.EmitRelocations(Int32, List)
  ► 2.34%   EmitDebugFunctionInfo  •  75 ms  •  ILCompiler.ObjectWriter.UnixObjectWriter.EmitDebugFunctionInfo(UInt32, String, SymbolDefinition, INodeWithDebugInfo, Boolean)
  ► 2.27%   GetData  •  73 ms  •  ILCompiler.DependencyAnalysis.ReflectionInvokeMapNode.GetData(NodeFactory, Boolean)
  ► 1.86%   GetUndefinedSymbols  •  59 ms  •  ILCompiler.ObjectWriter.ObjectWriter.GetUndefinedSymbols()
  ► 1.83%   EmitOrResolveRelocation  •  59 ms  •  ILCompiler.ObjectWriter.ObjectWriter.EmitOrResolveRelocation(Int32, Int64, Span, RelocType, String, Int64)
    1.48%   IsInstanceOfClass  •  48 ms  •  System.Runtime.CompilerServices.CastHelpers.IsInstanceOfClass(Void*, Object)
  ► 1.45%   ShouldSkipEmittingObjectNode  •  46 ms  •  ILCompiler.DependencyAnalysis.EETypeOptionalFieldsNode.ShouldSkipEmittingObjectNode(NodeFactory)
  ► 1.11%   Any  •  36 ms  •  System.Linq.Enumerable.Any(IEnumerable)
  ► 1.11%   GetData  •  35 ms  •  ILCompiler.DependencyAnalysis.InterfaceDispatchMapNode.GetData(NodeFactory, Boolean)
  ► 0.75%   EmitUnwindInfo  •  24 ms  •  ILCompiler.ObjectWriter.UnixObjectWriter.EmitUnwindInfo(SectionWriter, INodeWithCodeInfo, String)
  ► 0.73%   GetData  •  23 ms  •  ILCompiler.DependencyAnalysis.FieldRvaDataNode.GetData(NodeFactory, Boolean)
  ► 0.72%   GetMethodFunctionIdTypeIndex  •  23 ms  •  ILCompiler.UserDefinedTypeDescriptor.GetMethodFunctionIdTypeIndex(MethodDesc)
  ► 0.72%   ShouldSkipEmittingObjectNode  •  23 ms  •  ILCompiler.DependencyAnalysis.EETypeNode+WritableDataNode.ShouldSkipEmittingObjectNode(NodeFactory)
  ► 0.56%   GetData  •  18 ms  •  ILCompiler.DependencyAnalysis.GenericTypesHashtableNode.GetData(NodeFactory, Boolean)
  ► 0.55%   GetData  •  18 ms  •  ILCompiler.DependencyAnalysis.ExactMethodInstantiationsNode.GetData(NodeFactory, Boolean)
  ► 0.53%   GetData  •  17 ms  •  ILCompiler.DependencyAnalysis.StackTraceMethodMappingNode.GetData(NodeFactory, Boolean)
  ► 0.39%   get_Offset  •  13 ms  •  ILCompiler.DependencyAnalysis.EETypeNode.get_Offset()
  ► 0.38%   GetData  •  12 ms  •  ILCompiler.DependencyAnalysis.ResourceIndexNode.GetData(NodeFactory, Boolean)
    0.38%   TryGetValue  •  12 ms  •  System.Collections.Generic.Dictionary`2.TryGetValue(TKey, out TValue)
  ► 0.38%   GetOrCreateSection  •  12 ms  •  ILCompiler.ObjectWriter.ObjectWriter.GetOrCreateSection(ObjectNodeSection, String, String)
  ► 0.37%   GetData  •  12 ms  •  ILCompiler.DependencyAnalysis.StaticsInfoHashtableNode.GetData(NodeFactory, Boolean)
  ► 0.36%   GetData  •  12 ms  •  ILCompiler.DependencyAnalysis.ModuleInitializerListNode.GetData(NodeFactory, Boolean)
  ► 0.36%   GetData  •  12 ms  •  ILCompiler.DependencyAnalysis.GenericMethodsHashtableNode.GetData(NodeFactory, Boolean)
  ► 0.36%   EmitSymbolDefinition  •  11 ms  •  ILCompiler.ObjectWriter.SectionWriter.EmitSymbolDefinition(String, Int64, Int32, Boolean)
  ► 0.34%   GetData  •  11 ms  •  ILCompiler.DependencyAnalysis.ReflectionVirtualInvokeMapNode.GetData(NodeFactory, Boolean)
  ► 0.34%   ShouldSkipEmittingObjectNode  •  11 ms  •  ILCompiler.DependencyAnalysis.EETypeNode.ShouldSkipEmittingObjectNode(NodeFactory)
  ► 0.19%   GetData  •  6.0 ms  •  ILCompiler.DependencyAnalysis.EETypeNode+WritableDataNode.GetData(NodeFactory, Boolean)
  ► 0.19%   GetData  •  6.0 ms  •  ILCompiler.DependencyAnalysis.ArrayOfEmbeddedDataNode`1.GetData(NodeFactory, Boolean)
  ► 0.19%   GetData  •  6.0 ms  •  ILCompiler.DependencyAnalysis.SealedVTableNode.GetData(NodeFactory, Boolean)
  ► 0.19%   GetData  •  6.0 ms  •  ILCompiler.DependencyAnalysis.GenericTypesTemplateMap.GetData(NodeFactory, Boolean)
  ► 0.19%   GetData  •  6.0 ms  •  ILCompiler.DependencyAnalysis.ArrayMapNode.GetData(NodeFactory, Boolean)
  ► 0.19%   GetData  •  6.0 ms  •  ILCompiler.DependencyAnalysis.TypeMetadataMapNode.GetData(NodeFactory, Boolean)
    0.19%   HasFlag  •  6.0 ms  •  System.Enum.HasFlag(Enum)
  ► 0.19%   GetData  •  6.0 ms  •  ILCompiler.DependencyAnalysis.ExternalReferencesTableNode.GetData(NodeFactory, Boolean)
  ► 0.19%   GetData  •  6.0 ms  •  ILCompiler.DependencyAnalysis.GenericVirtualMethodTableNode.GetData(NodeFactory, Boolean)
    0.19%   GetSection  •  6.0 ms  •  ILCompiler.DependencyAnalysis.TypeMetadataMapNode.GetSection(NodeFactory)
  ► 0.17%   GetData  •  5.5 ms  •  ILCompiler.DependencyAnalysis.ReflectionFieldMapNode.GetData(NodeFactory, Boolean)
  ► 0.17%   GetData  •  5.5 ms  •  ILCompiler.DependencyAnalysis.ResourceDataNode.GetData(NodeFactory, Boolean)
  ► 0.17%   ShouldSkipEmittingObjectNode  •  5.5 ms  •  ILCompiler.DependencyAnalysis.TypeGenericDictionaryNode.ShouldSkipEmittingObjectNode(NodeFactory)
    0.17%   MoveNext  •  5.5 ms  •  System.Collections.Immutable.ImmutableArray`1+EnumeratorObject.MoveNext()

The time in spent in EmitDebugSections was about 22% before the change. Overall this dropped about a second of the runtime on my machine.

[MichalStrehovsky]

Thank you! I've submitted a job on the benchmark machine and we're now in the ~40 second range. I'll leave this unresolved until we have more conclusive evidence the build time is at least same as it was (we'll probably know tomorrow).

One thing to note is that it would be better to profile the native AOT-compiled compiler - I've seen some samples in interface dispatch that might be due to IBufferWriter not getting devirtualized without dynamic PGO (I did not dig into it since you jumped on it and I didn't want to step on your toes).

[filipnavara]

One thing to note is that it would be better to profile the native AOT-compiled compiler

I decided to fix the low-hanging fruit first, and that was visible in the managed profiler as well.

I've seen some samples in interface dispatch that might be due to IBufferWriter not getting devirtualized without dynamic PGO

That would be a bit unfortunate. We only use a single implementation of the interface at the moment so I would have expected the devirtualization to kick in (*).

(*) …unless there’s a System.Text.Json reference and it brings along its implementation.