| Age | Commit message (Collapse) | Author |
|---|
|
* switch to using CSTRMarshaller instead of AnsiBSTR
* cleanup
* Add CleanupManaged back
Updated the cleanup logic to the one implemented in ILOptimizedAllocMarshaler::EmitClearNative
* update MAX_LOCAL_BUFFER_LENGTH
* CR feedback
|
|
|
|
* compile composite with avx2 on x64
* fix xml coding style
|
|
* Revive #48505
Revive https://github.com/dotnet/runtime/pull/48505 and try to find out why the change caused official builds to break.
* Use OutputRid when creating .nupkgs, instead of PackageRID
Co-authored-by: Adeel Mujahid <3840695+am11@users.noreply.github.com>
|
|
Task.Delay(1).Wait() is variant of sync-over-async anti-patern. It does not work well with threadpool used to run parallel AOT compilation. It can lead to too many threadpool threads getting created and progress slowing down to crawl, at least temporarily.
|
|
I have expanded the PerfMap format produced by Crossgen2 and
R2RDump to produce metadata in form of pseudo-symbol records with
high addresses. In this version I have implemented four metadata
entries - output GUID, target OS, target architecture and perfmap
format version number. I have verified for System.Private.CoreLib
and for the composite framework that Crossgen2 and R2RDump
produce identical metadata.
To facilitate a smooth transition to the new perfmap format, in
accordance with Juan's suggestion I have introduced a new command-line
option to explicitly specify the perfmap format revision. As of today,
0 corresponds to the legacy Crossgen1-style output where the
perfmap file name includes the {MVID} section, perfmap format #1
corresponds to current Crossgen2 with its new naming scheme.
As of today there are no differences in the file content.
Thanks
Tomas
|
|
|
|
exceeded the int range (2 billion). (#54832)
Fix is to just use 64 bit ints instead.
|
|
|
|
|
|
|
|
This is required for R2R relocations to be processed by the OS loader on Windows 7.
|
|
The kernel requires the function table to be 4-byte aligned.
|
|
- Update ILVerify to build for NetCoreAppToolCurrent
|
|
* Improve dotnet-pgo output and fix null type handles
* When comparing .mibc files, find a unique name to print for both files
by repeatedly appending another parent directory on it.
* When comparing .mibc files, print a comparison of type handle
histogram data
* Fix a potential division by zero
* Print some info when producing .mibc that are based on samples/LBR
data
* Fix .mibc read type handles with value 0 to actually return IsNull
* Use different devirt threshold for interfaces and classes
* Add ClassProfileFlags to PgoFormat.cs
|
|
|
|
Address deficiencies in current devirtualization infrastructure
- Remove the responsibility of creating a CORINFO_RESOLVED_TOKEN structure from the JIT and make it a responsibility of the VM side of the jit interface.
- This enables the component (crossgen2) which has deeper understanding of the requirements here to correctly handle scenarios that would otherwise require expressing crossgen2 specific details across the jit interface.
- Add a new set of fixups (`READYTORUN_FIXUP_Check_VirtualFunctionOverride` and `READYTORUN_FIXUP_Verify_VirtualFunctionOverride`) these are used to validate that the behavior of the runtime and crossgen2 compiler is equivalent for a virtual resolution event
- `READYTORUN_FIXUP_Check_VirtualFunctionOverride` will ensure that the virtual resolution decision is the same at crossgen2 time and runtime, and if the decision differs, any generated code affected by the decision will not be used.
- `READYTORUN_FIXUP_Verify_VirtualFunctionOverride` will perform the same checks as `READYTORUN_FIXUP_Check_VirtualFunctionOverride`, but if it fails the check, the process will be terminated with a fail-fast. It is intended for use under the `--verify-type-and-field-layout` stress mode.
- Currently only the `READYTORUN_FIXUP_Verify_VirtualFunctionOverride` is actually generated, and it is only generated when using the `--verify-type-and-field-layout` switch to crossgen2. Future work will identify if there are scenarios where we need to generate the `READYTORUN_FIXUP_Check_VirtualFunctionOverride` flag. One area of possible concern is around covariant returns, another is around handling of type equivalence.
- In order to express the fixup signature for the VirtualFunctionOverride fixups, a new flag has been added to `ReadyToRunMethodSigFlags`. `READYTORUN_METHOD_SIG_UpdateContext` will allow the method signature to internally specify the assembly which is associated with the method token, instead of relying on the ambient context.
- R2RDump and the ReadyToRun format documentation have been updated with the details of the new fixups/flags.
- Update the rules for handling unboxing stubs
- See #51918 for details. This adds a new test, as well as proper handling for unboxing stubs to match the JIT behavior
- Also revert #52605, which avoided the problem by simply disabling devirtualization in the presence of structs
- Adjust the rules for when it is legal to devirtualize and maintain version resiliency
- The VersionsWithCode and VersionsWithType rules are unnecessarily restrictive.
- Instead Validate that the metadata is safely checkable, and rely on the canInline logic to ensure that no IL that can't be handled is inlined.
- This also involved adding a check that the chain of types from the implementation type to the declaration method table type is within the version bubble.
- And changing the `VersionsWithType` check on the implementation type, to a new `VersionsWithTypeReference` check which can be used to validate that the type can be referred to, in combination with using `VersionsWithType` on the type definition.
- By adjusting the way that the declMethod is referred to, it becomes possible to use the declMethod without checking the full method is `VersionsWithCode`, and it just needs a relationship to version matching code.
- In `resolveVirtualMethod` generate the `CORINFO_RESOLVED_TOKEN` structures for the jit
- In particular we are now able to resolve to methods where the decl method is the resolution result but is not within the version bubble itself. This can happen if we can prove that the decl method is the only method which can possibly implement a virtual.
- Add support for devirtualization reasons to crossgen2
- Port all devirtualization abort conditions to crossgen2 from runtime that were not already present
- Fix devirtualization from a canonical virtual method when the actual implementation is more exact
- Fix variant interface override scenario where there is an interface that requires implementation of the variant interface as well as the variant interface itself.
|
|
* excluding crossgen from build.
* Fix cross-component build break
* Fix tests to not use crossgen
* more test fixes.
* switching R2R jobs to use cg2 by default
* switch helix runs to use crossgen2
|
|
|
|
|
|
Manish investigated this issue and he found out that the problem
is caused by the fact that in the method
Newtonsoft.Json.Utilities.ReflectionUtils.GetFields
Crossgen2 produces a NewObject fixup for the incorrect type
System.Collections.Generic.List`1<System.Type>
instead of the right type
System.Collections.Generic.List`1<System.Reflection.MemberInfo>
This was caused by a bug in the token harvesting logic; at some
point resolveToken was asked to resolve the token 0x0A02BA,
string [System.Runtime/*23000001*/]System.Type/*01000019*/::get_Name() /* 0A0002BA */
As part of the token harvesting logic we looked at the
MethodReference.Parent (01000019) and stored it in the harvesting
table along with the OwningType of the method. The problem is
that the translation of the MemberReference to MethodDesc
resolves the method on a base class,
string [System.Reflection/*23000009*/]System.Reflection.MemberInfo/*01000076*/::get_Name() /* 0A0002B5 */
As a result we were storing the incorrect [token - type] pair
[01000019 - System.Reflection.MemberInfo] into the type token
translation table and that confused the signature encoder because
01000019 is System.Type.
The trivial solution is to separately translate the memberref
parent entity handle to the owning type entity instead than
extracting it from the MethodDesc. I have verified using R2RDump
that I now see the correct NewObject fixup getting generated
in the method.
Thanks
Tomas
|
|
* Fixing a regression compiling VT arrays
* Update src/coreclr/tools/Common/TypeSystem/Interop/IL/MarshalHelpers.cs
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
|
|
These numbers changed in #53424. One of the reasons why I'm not a huge fan of too much commenting...
|
|
The regression test
<code>src\tests\JIT\Regressions\JitBlue\Runtime_46239</code>
exercises various interesting corner cases of type layout that
weren't handled properly in Crossgen2 on x86 and ARM[32]. This
change fixes the remaining deficiencies and it also adds
provisions for better runtime logging upon type layout mismatches.
With this change, the only remaining pipelines using Crossgen1 are
"r2r.yml", "r2r-extra.yml" and "release-tests.yml". I haven't yet
identified the pipeline running the "release-tests.yml" script;
for the "r2r*.yml", these now remain the only pipelines exercising
Crossgen1. I don't think it makes sense to switch them over to
CG2 as we already have their CG2 counterparts; my expectation is
that, once CG1 is finally decommissioned, they will be just deleted.
Thanks
Tomas
|
|
* Integrate changes from NativeAOT branch
Includes faster virtual method enumerator that should help crossgen2 too
* Fix tests
|
|
|
|
* Add support for AvxVnni instructions under Experimental.
* Add support for AvxVnni instructions
* Add preveiw feature attribute
* Handle operands in lsra
* Undo changes for Experimental
* Update JITEEVersionIdentifier and fix remaining issues
* Resolve Mono CI failure
* Disable tests
* Disable Vector128 tests
* Modify disable tests
Co-authored-by: Tanner Gooding <tagoo@outlook.com>
|
|
- Change up the definition of an MDArray which doesn't need a special descriptor in the embedded signature data to match the type of mdarray thay everyone always uses. This fixes the virtual function handling behavior for all reasonable cases. (All cases where the MDArray is used for a base type in the expected fashion.)
- Add a devirtualization test for this specific scenario
Fixes #52444
|
|
In circumstances where the JIT doesn't provide exact enough details about the impl type and interface method to identify exactly which method should be called
- In particular, when the impl class implements multiple copies of the target interface, and they are canonically compatible with the interface method that is to be called
- Simply disable devirtualization for these scenarios
Fixes #51982
|
|
Several changes to help better diagnose PGO and devirtualization issues:
* Report the source of the PGO data to the jit
* Report the reason for a devirtualization failure to the jit
* Add checking mode that compares result of devirtualization to class profile
* Add reporting mode to assess overall rates of devirtualization failure
when the jit has exact type information.
Also fix a loophole where in some case we'd still devirtualize if not
optimizing.
Note crossgen2 does not yet set devirtualization failure reasons.
|
|
- More code sharing for System.Exception
- Some support for default interface methods
- Move callling convention helpers around to avoid duplication
- Unify Environment.StackTrace
|
|
* Integrate misc changes from nativeaot
* Fix RemoteStack serialization
* Delete confusing String.Empty comment
|
|
* Override BaseType in TypeRefTypeSystemType
BaseType is being used to determine whether the type is a value type or
class type. This meant that `dotnet-pgo merge` would always produce
metadata saying that all types were class types.
In practice, this meant that we had the following behavior:
```
.\dotnet-pgo.exe merge --input IBCTrace28800.mibc --output identity.mibc
.\dotnet-pgo.exe merge --input IBCTrace28800.mibc --input identity.mibc --output foo.mibc
Opening D:\dev\dotnet\dotnet-optimization\output\temp\IBCTrace28800.mibc
Opening D:\dev\dotnet\runtime\artifacts\bin\coreclr\windows.x64.Debug\dotnet-pgo\foo.mibc
Unhandled exception. System.Exception: Same type `[S.P.CoreLib]System.ReadOnlySpan`1` used as both ValueType and non-ValueType
at Microsoft.Diagnostics.Tools.Pgo.TypeRefTypeSystem.TypeRefTypeSystemType.SetIsValueType(Boolean isValueType)
at Microsoft.Diagnostics.Tools.Pgo.TypeRefTypeSystem.TypeRefTypeSystemContext.TypeRefSignatureParserProvider.GetTypeFromReference(MetadataReader reader, TypeReferenceHandle handle, Byte rawTypeKind)
```
The same problem would occur when using compare-mibc to compare a
merged mibc file with some of its constituents. These scenarios work
now.
* Re-abstract BaseType in MetadataType
|
|
|
|
|
|
* Moved R2RDump statistics from console to --out file
* Automated the fixup statistics report into a self-maintaining component.
* Optimized the minimum values by setting them at the beginning, and removing the Math.Max() calls.
|
|
* Support checking for overridden methods during P/Invoke inline pass
* Objective-C runtime msgSend* functions are never inlinable.
* Update CrossGen2.
* Add comment on details of how we are testing the
SetMessageSendPendingException API.
* Disable the crossgen2smoke on OSX ARM64
|
|
Co-authored-by: Jan Kotas <jkotas@microsoft.com>
|
|
* Fix logic error in PGO schema comparison
* Revert "Fix logic error in PGO schema comparison"
This reverts commit a3a977c4be10e545774d371642c87693f2989414.
* Fix merging of edge counts
Co-authored-by: Jakob Botsch Nielsen <jakob.botsch.nielsen@gmail.com>
|
|
This allows dotnet-pgo to generate .mibc files using the sample data
stored in the trace that it is processing. It implements support for
both last branch record (LBR) data and normal IP samples. The latter can
be produced using PerfView as normal while the former currently requires
using xperf with LBR mode enabled. For posterity, to enable both logging
required .NET events and LBR, the following commands can be used (on
Windows):
```
xperf.exe -start "NT Kernel Logger" -on LOADER+PROC_THREAD+PMC_PROFILE -MinBuffers 4096 -MaxBuffers 4096 -BufferSize 4096 -pmcprofile BranchInstructionRetired -LastBranch PmcInterrupt -setProfInt BranchInstructionRetired 65537 -start clr -on e13c0d23-ccbc-4e12-931b-d9cc2eee27e4:0x40000A0018:0x5 -MinBuffers 4096 -MaxBuffers 4096 -BufferSize 4096
scenario.exe
xperf.exe -stop "NT Kernel Logger" -stop clr -d xperftrace.etl
```
SPGO does not currently do well with optimized code as the mapping
IP<->IL mappings the JIT produces there are not sufficiently accurate.
To collect data in tier-0 one can enable two environment variables
before running the scenario:
```
$env:COMPlus_TC_QuickJitForLoops=1
$env:COMPlus_TC_CallCounting=0
```
When samples are used the associated counts will not typically look
valid, i.e. they won't satisfy flow conservation. To remedy this,
dotnet-pgo performs a smoothing step after assigning samples to the
flow-graph of each method. The smoothing is based on [1] and the code
comes from Midori.
The commit adds some new commands to dotnet-pgo. The --spgo flag can be
specified to create-mibc to use samples to create the .mibc file. Also,
even if --spgo is specified, instrumented data will still be preferred
if available in the trace. If spgo is not specified, the behavior should
be the same as before.
--spgo-with-block-counts and --spgo-with-edge-counts control whether
dotnet-pgo outputs the smoothed block or edge counts (or both). By
default block counts are output. The JIT can use both forms of counts
but will be most happy if only one kind is present for each method.
--spgo-min-samples controls how many samples must be in each method
before smoothing is applied and the result included in the .mibc. SPGO
is quite sensitive to low sample counts and the produced results are not
good when the number of samples is low. By default, this value is 50.
The commit also adds a new compare-mibc command that allows to compare
two .mibc files. Usage is dotnet-pgo compare-mibc --input file1.mibc
--input file2.mibc. For example, comparing a .mibc produced via
instrumentation and one produced via SPGO (in tier-0) for some JIT
benchmarks produces the following:
```
Comparing instrumented.mibc to spgo.mibc
Statistics for instrumented.mibc
# Methods: 3490
# Methods with any profile data: 865
# Methods with 32-bit block counts: 0
# Methods with 64-bit block counts: 865
# Methods with 32-bit edge counts: 0
# Methods with 64-bit edge counts: 0
# Methods with type handle histograms: 184
# Methods with GetLikelyClass data: 0
# Profiled methods in instrumented.mibc not in spgo.mibc: 652
Statistics for spgo.mibc
# Methods: 1107
# Methods with any profile data: 286
# Methods with 32-bit block counts: 286
# Methods with 64-bit block counts: 0
# Methods with 32-bit edge counts: 0
# Methods with 64-bit edge counts: 0
# Methods with type handle histograms: 0
# Methods with GetLikelyClass data: 0
# Profiled methods in spgo.mibc not in instrumented.mibc: 73
Comparison
# Methods with profile data in both .mibc files: 213
Of these, 213 have matching flow-graphs and the remaining 0 do not
When comparing the flow-graphs of the matching methods, their overlaps break down as follows:
100% █ (1.9%)
>95% █████████████████████████████████▌ (61.0%)
>90% ████████ (14.6%)
>85% ████▏ (7.5%)
>80% ████▋ (8.5%)
>75% █▊ (3.3%)
>70% █ (1.9%)
>65% ▎ (0.5%)
>60% ▎ (0.5%)
>55% ▏ (0.0%)
>50% ▏ (0.0%)
>45% ▏ (0.0%)
>40% ▎ (0.5%)
>35% ▏ (0.0%)
>30% ▏ (0.0%)
>25% ▏ (0.0%)
>20% ▏ (0.0%)
>15% ▏ (0.0%)
>10% ▏ (0.0%)
> 5% ▏ (0.0%)
> 0% ▏ (0.0%)
(using block counts)
```
I also made the dump command print some statistics about the .mibc that
was dumped. Hopefully some of this tooling can help track down #51908.
[1] Levin R., Newman I., Haber G. (2008) Complementing Missing and
Inaccurate Profiling Using a Minimum Cost Circulation Algorithm. In:
Stenström P., Dubois M., Katevenis M., Gupta R., Ungerer T. (eds) High
Performance Embedded Architectures and Compilers. HiPEAC 2008. Lecture
Notes in Computer Science, vol 4917. Springer, Berlin, Heidelberg.
https://doi.org/10.1007/978-3-540-77560-7_20
|
|
|
|
references into version bubble (#51555)
|
|
This change implements initial CoreCLR runtime support for static virtual interface methods and adds over 1700 test cases covering various aspects of this new runtime feature.
1) In the JIT, we are relaxing the assumption that ".constrained" must be always followed by a "callvirt" by allowing two more IL instructions to be constrained, "call" and "ldftn".
2) In the JIT interface, we're adding bits of code to CEEInfo::getCallInfo to properly handle the new static virtual methods. We're extending constrained method resolution to cater for the static virtual method case in a new method "ResolveStaticVirtualMethod".
3) In our work on the implementation we found a pre-existing JIT interface issue - the interplay between getCallInfo and embedGenericHandle doesn't work well in case of constrained calls as the existing API surface doesn't provide any means for communicating compile-time resolution of the constraint from getCallInfo to embedGenericHandle. In this change we're working around this by deferring to runtime resolution of the constraint in the presence of shared generics.
4) In the method table builder, we're newly tracking a flag saying whether we're implementing an interface declaring static virtual methods (fHasVirtualStaticMethods) that is used to speed up the runtime checks by skipping irrelevant interfaces.
5) For Crossgen / Crossgen2, this change only adds minimalistic support in the form of bailing out in getCallInfo whenever we hit a call to a static virtual method, cancelling AOT compilation of the caller and deferring it to runtime JIT.
|
|
image for runtime repo in parallel path (#52579)
* Move all crossgen2 tasks/targets used in Runtime repo to be local to the repo
* Move ResolveReadyToRunCompilers task into the Crossgen2Tasks dll
* Create composite bundle and package
- Note that it doesn't build properly, but hopefully that's a crossgen2 task bug
* Composite image production
* Refactor how composite image creation works
- Move it to allow mixed composite and non-composite r2r in the same build, controlled by the PublishReadyToRunCompositeExclusions list
- Enhance the accuracy of how msbuild tracks which files are input vs reference for composite images
- Implement the R2R exclusion list for composite images
- Remove the always on usage of --inputbubble when compiling a composite image
* - Add a scheme to perform public signing with a PublicKeyToken to a composite of the composite image if required
- Required adding a new parameter to crossgen2
|
|
- Make regular (non-memory-mapped file based) stresslog work for clrgc.dll
- Fix interaction issue between GC and thread filters in StressLogAnalyzer
- Interpret small numbers for COMPlus_StressLogSize and COMPlus_TotalStressLogSize as GB rather than bytes
- Increment SOS_BREAKING_CHANGE_VERSION
|
|
* Modify JIT to support double mapped memory
This change modifies JIT so that it can generate code into
double mapped memory. The code is written into RW mapped memory,
but the relative offsets are computed using the related RX locations.
The change doesn't modify the runtime to provide double mapped memory
yet, the JIT2EE interface allocMem returns the same addresses as RW and
RX. The runtime changes will be part of follow-up PRs. However, it was
already tested with the double mapping enabled locally.
It also changes signature of allocMem to pass everything in a single structure
|
|
|
|
* Initial implementation of crossgen2 pipeline mode
- Add --out-near-input option, which adds .ni. suffix to input filepath
and stores resulting ni.dll near original dll. In this mode --out option can be skipped.
- Add --single-file-compilation mode, which allows to compile all input files separately.
* [crossgen2] Optimize pipeline mode
Load input and reference modules only once by crossgen2 start.
Signed-off-by: Timur Mustafin <t.mustafin@partner.samsung.com>
* Feedback: make typeSystemContext with opened files
Construct new typeSystemContext with already opened PEReaders and PDBReaders.
Signed-off-by: Timur Mustafin <t.mustafin@partner.samsung.com>
Co-authored-by: Gleb Balykov <g.balykov@samsung.com>
|
|
Co-authored-by: Youssef Victor <youssefvictor00@gmail.com>
Co-authored-by: Marek Safar <marek.safar@gmail.com>
|
|
Add COMPlus_JitCollect64BitCounts which makes the JIT instrument using
64-bit counts instead of 32-bit counts. No support for consuming these
counts is added, only support for producing them.
I also changed the printing of relocs to include their values when
diffable disassembly is off.
|
