diff options
author | David Wrighton <davidwr@microsoft.com> | 2022-11-08 03:19:53 +0300 |
---|---|---|
committer | GitHub <noreply@github.com> | 2022-11-08 03:19:53 +0300 |
commit | 939f050e32e5fdee41dbe75b870c5666c9e5316c (patch) | |
tree | e954025c7136311028edc11e7d76474fb4fd0c97 /docs | |
parent | cd3bd4cdacc61e09d4c24cdc41d1938c9d40b822 (diff) |
Improve coreclr test infra (#77929)
These fixes were built for PR #74886; however, as that PR is so utterly large and unreviewable, I've pulled out the test infra changes for separate review
Changes
- Increase the number of trampolines in the llvm aot compilation process to 20,000 from 10,000 (This avoids running out of them in some of the hardware intrinsics tests
- Add a concept of striping tests when running under GC stress
- To use this new feature, specify <NumberOfStripesToUseInStress>N</NumberOfStripesToUseInStress> within the merged test assembly. If this value is set, then the tests within that merged test assembly will be run across N different work items instead of 1 when running under any form of GC stress based scenario. At this moment the largest supported value of N is 99
- Emit the testresults.xml file as a file which is exported from the tests. This is useful for debugging testresult.xml parsing failures
- Fix the testresults summary generator to never emit an empty CDATA string. If one is present the parser may fail the parse.
- In the XUnitWrapperGenerator fix the implementation of the Outerloop and ActiveIssue when used with a conditional member.
- Add PlatformDetection.IsMonoLLVMAOT, PlatformDetection.IsMonoLLVMFULLAOT, and PlatformDetection.IsMonoInterpreter boolean properties to the PlatformDetection type for use with the ActiveIssue attribute
- Add some documentation about project files for coreclr tests, as well as some documentation on the command line parameters for merged test runner assemblies
Diffstat (limited to 'docs')
-rw-r--r-- | docs/workflow/testing/coreclr/test-configuration.md | 26 |
1 files changed, 26 insertions, 0 deletions
diff --git a/docs/workflow/testing/coreclr/test-configuration.md b/docs/workflow/testing/coreclr/test-configuration.md index 4f5d11cf6ab..1655f3f639a 100644 --- a/docs/workflow/testing/coreclr/test-configuration.md +++ b/docs/workflow/testing/coreclr/test-configuration.md @@ -49,8 +49,20 @@ Therefore the managed portion of each test **must not contain**: * e.g. `<DisableProjectBuild>true</DisableProjectBuild>` * Exclude test from GCStress runs by adding the following to the csproj: * `<GCStressIncompatible>true</GCStressIncompatible>` +* Exclude test from HeapVerify testing runs runs by adding the following to the csproj: + * `<HeapVerifyIncompatible>true</HeapVerifyIncompatible>` * Exclude test from JIT stress runs runs by adding the following to the csproj: * `<JitOptimizationSensitive>true</JitOptimizationSensitive>` +* Exclude test from NativeAOT runs runs by adding the following to the csproj: + * `<NativeAotIncompatible>true</NativeAotIncompatible>` +* Exclude the test from ilasm round trip testing by adding the following to the csproj + * `<IlasmRoundTripIncompatible>true</IlasmRoundTripIncompatible>` +* Exclude the test for unloadability (collectible assemblies) testing + * `<UnloadabilityIncompatible>true</UnloadabilityIncompatible>` +* If the test is specific for testing crossgen2, and should be compiled as such in all test modes + * `<AlwaysUseCrossGen2>true</AlwaysUseCrossGen2>` +* When `CrossGenTest` is set to false, this test is not run with standard R2R compilation even if running an R2R test pass. + * `<CrossGenTest>false</CrossGenTest>` * Add NuGet references by updating the following [test project](https://github.com/dotnet/runtime/blob/main/src/tests/Common/test_dependencies/test_dependencies.csproj). * Any System.Private.CoreLib types and methods used by tests must be available for building on all platforms. This means there must be enough implementation for the C# compiler to find the referenced types and methods. Unsupported target platforms @@ -94,3 +106,17 @@ should simply `throw new PlatformNotSupportedException()` in its dummy method im * CMake reference: `<CMakeProjectReference Include="../NativeDll/CMakeLists.txt" />` 1. Build the test. 1. Follow the steps to re-run a failed test to validate the new test. + +### Creating a merged test runner project +1. Use an existing test such as `<repo_root>\src\tests\JIT\Methodical\Methodical_d1.csproj` as a template. +1. If your new merged test runner has MANY tests in it, and takes too long to run under GC Stress, set `<NumberOfStripesToUseInStress>` to a number such as 10 to make it possible for the test to complete in a reasonable timeframe. + +#### Command line arguments for merged test runner projects +Unless tests are manually run on the command line to repro a problem, these parameters are handled internally by the test infrastructure, but for running tests locally, there are a set of standard parameters that these merged test runners support. + +`[testFilterString] [-stripe <whichStripe> <totalStripes>]` + +`testFilterString` is any string other that `-stripe`. The only filters supported today are the simple form supported in 'dotnet test --filter' (substrings of the test's fully qualified name). + +Either the -stripe <whichStripe> <totalStripes> parameter can be used or the TEST_HARNESS_STRIPE_TO_EXECUTE environment variable may be used to control striping. The TEST_HARNESS_STRIPE_TO_EXECUTE environment variable must be set to a string of the form `.<whichStripe>.<totalStripes>` if it is used. `<whichStripe>` is a 0 based index into the count of stripes, `<totalStripes>` is the total number of stripes. + |