Move MSBuild projects out of the native build scripts (#31701)

* Move coreclr managed tools that don't need to be published and System.Private.CoreLib into their own subsets in Subsets.props and drive their builds via Arcade instead of the coreclr scripts.

* Remove building System.Private.CoreLib from build scripts. Eventing sources TBD.

* Build packages via an Arcade subset instead of within build.cmd/build.sh

* Remove packages and test build from build.cmd. Clean up options on build.cmd and build.sh

* Generate eventing sources for System.Private.CoreLib within MSBuild instead of in the build.cmd/sh scripts. Clean up the build.sh usage documentation (lots of copy-paste errors).

* Remove check-definitions.py script and supporting goo.

* Fix eventing generation.

* Wire up  TargetArchitecture and OSGroup to coreclr's BuildArch and BuildOS to enable building System.Private.CoreLib for the correct output given the script parameters.

* Stop passing down now non-existent skiptests parameter.

* Fix coreclr .nuget rid parsing to account for linux-musl correctly.

* Update build scripts to use Arcade to build packages.

* Convert from Properties to AdditionalProperties.

* Build crossgen as part of Arcade (self-contained publish TBD).

* Pull down .NET runtime CLI for helix runs with crossgen2.

* Output crossgen2 to the BinDir/crossgen when built from the project file. Enable the test shell scripts to either resolve the dotnet CLI from the path or from relative to Core_Root if the __DotNet variable is undefined.

* Generate distro rid during windows build.

* Various fixes to get files binplaced correctly and builds to work on Linux.

* Fix search for dotnet in repo on bash.

* Remove __BuildTest variable and fix up the script.

* Fix steps in build-job to build S.P.CL before the runtime and build managed tools after the runtime.

* Fix casing typo.

* Refactor detecting system OS and Architecture.

* Fix wasm rid calculation.

* Run crossgen2 on corerun so we run it on the runtime we're going to ship it with.

* Assemble a live self-contained publish in the crossgen2 package.

* Fix init-os-and-arch resolution.

* Ensure that the runtime files we package in crossgen2 are the crossgened ones.

* Remove extraneous /bl

* Build R2RDump to a subfolder instead of the product dir.

* Seperate out the crosssgening corelib step into a separate script.

* Fix permissions on crossgen-corelib.sh

* Rename build.cmd/sh to build-runtime.cmd/sh

* Fix parameters in crossgen2determinism test.

* Run crossgen on System.Private.CoreLib in CI.

* Restore original build.cmd/sh scripts with a deprecation warning and update instructions.

* Initialize developer prompt in crossgen-corelib.cmd script so we can find DIA on CI machines.

* Make sure vs tools are enabled so we can resolve DiaSymReader.

* Call build-native instead of build.

* Fix build-native -> build-runtime.

* Fix permissions.

* Remove dead MSBuild code.

* Factor out python locating targets to make them available to the whole repo.

* PR Feedback/simplification.

* Create a self-contained deps.json for crossgen2 and add it to the package.

* Pass runtimeconfiguration as configuration for crossgen2 deps file generation.

* Try specifying rids directly in crossgen2 project to get the restore correct for the live-build pipeline.

* Run crossgen2 via corerun in R2R.SuperILC.

* Add binlogs to build-runtime.cmd

* Combine non build-runtime steps into a single step so we get one binlog that doesn't get overwritten.

* Add reference to issue describing new workflow.

* Quote python path

* Set arch default.

* PR Feedback and clean up crossgen-corelib's argument list.

* Fix subset name matching to work when one subset name is a substring of another.

* Pass the official build id as an msbuild arg at all times.
Additional cleanup for crossgen-corelib step.

* PR Feedback.

* Fix default subset list construction.

* Add deprecation warnings at beginning and end of build.cmd and build.sh.

* Setup dirs in crossgen-corelib.sh

* Move enforcepgo argument to before the official build arg so that it gets consumed correctly.

* Add cross-os DAC builds as subsets.
Clean up build-runtime parameters and passing the official build ID from runtime.proj.

* Update script usage docs.

* Skip the cross-os dac on x86.

* Build windows->linux cross dac on PR and CI.

* Don't build the cross-OS linux dac on Windows x86.

1 files changed, 267 insertions, 0 deletions

diff --git a/src/coreclr/crossgen-corelib.cmd b/src/coreclr/crossgen-corelib.cmd
new file mode 100644
index 00000000000..7cd05c3e8e0
--- /dev/null
+++ b/src/coreclr/crossgen-corelib.cmd
@@ -0,0 +1,267 @@
+@if not defined _echo @echo off
+setlocal EnableDelayedExpansion EnableExtensions
+
+:: Define a prefix for most output progress messages that come from this script. That makes
+:: it easier to see where these are coming from. Note that there is a trailing space here.
+set "__MsgPrefix=CROSSGEN-CORELIB: "
+
+echo %__MsgPrefix%Starting Build at %TIME%
+
+set __ThisScriptFull="%~f0"
+set __ThisScriptDir="%~dp0"
+
+:: Note that the msbuild project files (specifically, dir.proj) will use the following variables, if set:
+::      __BuildArch         -- default: x64
+::      __BuildType         -- default: Debug
+::      __BuildOS           -- default: Windows_NT
+::      __ProjectDir        -- default: directory of the dir.props file
+::      __RepoRootDir       -- default: directory two levels above the dir.props file
+::      __RootBinDir        -- default: %__RepoRootDir%\artifacts\
+::      __BinDir            -- default: %__RootBinDir%\%__BuildOS%.%__BuildArch.%__BuildType%\
+::      __IntermediatesDir
+::      __PackagesBinDir    -- default: %__BinDir%\.nuget
+::
+:: Thus, these variables are not simply internal to this script!
+
+:: Set the default arguments for build
+set __BuildArch=x64
+set __BuildType=Debug
+set __BuildOS=Windows_NT
+
+set "__ProjectDir=%~dp0"
+:: remove trailing slash
+if %__ProjectDir:~-1%==\ set "__ProjectDir=%__ProjectDir:~0,-1%"
+set "__RepoRootDir=%__ProjectDir%\..\.."
+
+set "__RootBinDir=%__RepoRootDir%\artifacts"
+set "__LogsDir=%__RootBinDir%\log"
+
+set __BuildArchX64=0
+set __BuildArchX86=0
+set __BuildArchArm=0
+set __BuildArchArm64=0
+
+set __BuildTypeDebug=0
+set __BuildTypeChecked=0
+set __BuildTypeRelease=0
+
+set __PgoInstrument=0
+set __IbcTuning=
+
+REM __PassThroughArgs is a set of things that will be passed through to nested calls to build.cmd
+REM when using "all".
+set __PassThroughArgs=
+
+set "__args= %*"
+set processedArgs=
+set __CrossgenAltJit=
+set __CrossArch=
+
+:Arg_Loop
+if "%1" == "" goto ArgsDone
+
+if /i "%1" == "/?"     goto Usage
+if /i "%1" == "-?"     goto Usage
+if /i "%1" == "/h"     goto Usage
+if /i "%1" == "-h"     goto Usage
+if /i "%1" == "/help"  goto Usage
+if /i "%1" == "-help"  goto Usage
+if /i "%1" == "--help" goto Usage
+
+if /i "%1" == "-x64"                 (set __BuildArchX64=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%1" == "-x86"                 (set __BuildArchX86=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%1" == "-arm"                 (set __BuildArchArm=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%1" == "-arm64"               (set __BuildArchArm64=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+
+if /i "%1" == "-debug"               (set __BuildTypeDebug=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%1" == "-checked"             (set __BuildTypeChecked=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%1" == "-release"             (set __BuildTypeRelease=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+
+if /i "%1" == "-ci"                  (set __ErrMsgPrefix=##vso[task.logissue type=error]&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+
+REM All arguments after this point will be passed through directly to build.cmd on nested invocations
+REM using the "all" argument, and must be added to the __PassThroughArgs variable.
+if [!__PassThroughArgs!]==[] (
+    set __PassThroughArgs=%1
+) else (
+    set __PassThroughArgs=%__PassThroughArgs% %1
+)
+if /i "%1" == "-ibcinstrument"       (set __IbcTuning=/Tuning&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%1" == "-pgoinstrument"       (set __PgoInstrument=1&set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+if /i "%1" == "-crossgenaltjit"      (set __CrossgenAltJit=%2&set processedArgs=!processedArgs! %1 %2&shift&shift&goto Arg_Loop)
+if /i "%1" == "--"                  (set processedArgs=!processedArgs! %1&shift&goto Arg_Loop)
+
+:ArgsDone
+
+set /A __TotalSpecifiedBuildArch=__BuildArchX64 + __BuildArchX86 + __BuildArchArm + __BuildArchArm64
+if %__TotalSpecifiedBuildArch% GTR 1 (
+    echo Error: more than one build architecture specified.
+    goto Usage
+)
+
+if %__BuildArchX64%==1      set __BuildArch=x64
+if %__BuildArchX86%==1      set __BuildArch=x86
+if %__BuildArchArm%==1 (
+    set __BuildArch=arm
+    set __CrossArch=x86
+)
+if %__BuildArchArm64%==1 (
+    set __BuildArch=arm64
+    set __CrossArch=x64
+)
+
+set /A __TotalSpecifiedBuildType=__BuildTypeDebug + __BuildTypeChecked + __BuildTypeRelease
+if %__TotalSpecifiedBuildType% GTR 1 (
+    echo Error: more than one build type specified.
+    goto Usage
+)
+
+if %__BuildTypeDebug%==1    set __BuildType=Debug
+if %__BuildTypeChecked%==1  set __BuildType=Checked
+if %__BuildTypeRelease%==1  set __BuildType=Release
+
+set "__BinDir=%__RootBinDir%\bin\coreclr\%__BuildOS%.%__BuildArch%.%__BuildType%"
+set "__IntermediatesDir=%__RootBinDir%\obj\coreclr\%__BuildOS%.%__BuildArch%.%__BuildType%"
+set "__CrossComponentBinDir=%__BinDir%"
+
+
+if NOT "%__CrossArch%" == "" set __CrossComponentBinDir=%__CrossComponentBinDir%\%__CrossArch%
+set "__CrossGenCoreLibLog=%__LogsDir%\CrossgenCoreLib_%__BuildOS%__%__BuildArch%__%__BuildType%.log"
+set "__CrossgenExe=%__CrossComponentBinDir%\crossgen.exe"
+
+if not exist "%__BinDir%"              md "%__BinDir%"
+if not exist "%__IntermediatesDir%"    md "%__IntermediatesDir%"
+if not exist "%__LogsDir%"             md "%__LogsDir%"
+
+
+call "%__ThisScriptDir%"\setup_vs_tools.cmd
+if NOT '%ERRORLEVEL%' == '0' goto ExitWithError
+
+if defined VS160COMNTOOLS (
+    set "__VSToolsRoot=%VS160COMNTOOLS%"
+    set "__VCToolsRoot=%VS160COMNTOOLS%\..\..\VC\Auxiliary\Build"
+    set __VSVersion=vs2019
+) else if defined VS150COMNTOOLS (
+    set "__VSToolsRoot=%VS150COMNTOOLS%"
+    set "__VCToolsRoot=%VS150COMNTOOLS%\..\..\VC\Auxiliary\Build"
+    set __VSVersion=vs2017
+)
+
+REM Need VC native tools environment for the host arch to find Microsoft.DiaSymReader.Native in the Visual Studio install.
+set __VCBuildArch=x86_amd64
+if /i "%__BuildArch%" == "x86" ( set __VCBuildArch=x86 )
+if /i "%__BuildArch%" == "arm" (
+    set __VCBuildArch=x86_arm
+)
+if /i "%__BuildArch%" == "arm64" (
+    set __VCBuildArch=x86_arm64
+)
+
+echo %__MsgPrefix%Using environment: "%__VCToolsRoot%\vcvarsall.bat" !__VCBuildArch!
+call                                 "%__VCToolsRoot%\vcvarsall.bat" !__VCBuildArch!
+@if defined _echo @echo on
+
+if not defined VSINSTALLDIR (
+    echo %__ErrMsgPrefix%%__MsgPrefix%Error: VSINSTALLDIR variable not defined.
+    goto ExitWithError
+)
+if not exist "!VSINSTALLDIR!DIA SDK" goto NoDIA
+
+echo %__MsgPrefix%Generating native image of System.Private.CoreLib for %__BuildOS%.%__BuildArch%.%__BuildType%. Logging to "%__CrossGenCoreLibLog%".
+if exist "%__CrossGenCoreLibLog%" del "%__CrossGenCoreLibLog%"
+
+REM Need VS native tools environment for the **target** arch when running instrumented binaries
+if %__PgoInstrument% EQU 1 (
+    set __VCExecArch=%__BuildArch%
+    if /i [%__BuildArch%] == [x64] set __VCExecArch=amd64
+    echo %__MsgPrefix%Using environment: "%__VCToolsRoot%\vcvarsall.bat" !__VCExecArch!
+    call                                 "%__VCToolsRoot%\vcvarsall.bat" !__VCExecArch!
+    @if defined _echo @echo on
+    if NOT !errorlevel! == 0 (
+        echo %__ErrMsgPrefix%%__MsgPrefix%Error: Failed to load native tools environment for !__VCExecArch!
+        goto ExitWithError
+    )
+
+    REM HACK: Workaround for [dotnet/coreclr#13970](https://github.com/dotnet/coreclr/issues/13970)
+    set __PgoRtPath=
+    for /f "tokens=*" %%f in ('where pgort*.dll') do (
+        if not defined __PgoRtPath set "__PgoRtPath=%%~f"
+    )
+    echo %__MsgPrefix%Copying "!__PgoRtPath!" into "%__BinDir%"
+    copy /y "!__PgoRtPath!" "%__BinDir%" || (
+        echo %__ErrMsgPrefix%%__MsgPrefix%Error: copy failed
+        goto ExitWithError
+    )
+    REM End HACK
+)
+
+if defined __CrossgenAltJit (
+    REM Set altjit flags for the crossgen run.
+    echo %__MsgPrefix%Setting altjit environment variables for %__CrossgenAltJit%.
+    echo %__MsgPrefix%Setting altjit environment variables for %__CrossgenAltJit%. >> "%__CrossGenCoreLibLog%"
+    set COMPlus_AltJit=*
+    set COMPlus_AltJitNgen=*
+    set COMPlus_AltJitName=%__CrossgenAltJit%
+    set COMPlus_AltJitAssertOnNYI=1
+    set COMPlus_NoGuiOnAssert=1
+    set COMPlus_ContinueOnAssert=0
+)
+
+REM Need diasymreader.dll on your path for /CreatePdb
+set PATH=%PATH%;%WinDir%\Microsoft.Net\Framework64\V4.0.30319;%WinDir%\Microsoft.Net\Framework\V4.0.30319
+
+    for /f "tokens=*" %%f in ('where Microsoft.DiaSymReader.Native.amd64.dll') do (
+        echo "%%~f"
+    )
+
+set NEXTCMD="%__CrossgenExe%" /nologo %__IbcTuning% /Platform_Assemblies_Paths "%__BinDir%\IL" /out "%__BinDir%\System.Private.CoreLib.dll" "%__BinDir%\IL\System.Private.CoreLib.dll"
+echo %__MsgPrefix%!NEXTCMD!
+echo %__MsgPrefix%!NEXTCMD! >> "%__CrossGenCoreLibLog%"
+!NEXTCMD! >> "%__CrossGenCoreLibLog%" 2>&1
+if NOT !errorlevel! == 0 (
+    echo %__ErrMsgPrefix%%__MsgPrefix%Error: CrossGen System.Private.CoreLib build failed. Refer to %__CrossGenCoreLibLog%
+    REM Put it in the same log, helpful for CI
+    type %__CrossGenCoreLibLog%
+    goto ExitWithError
+)
+
+set NEXTCMD="%__CrossgenExe%" /nologo /Platform_Assemblies_Paths "%__BinDir%" /CreatePdb "%__BinDir%\PDB" "%__BinDir%\System.Private.CoreLib.dll"
+echo %__MsgPrefix%!NEXTCMD!
+echo %__MsgPrefix%!NEXTCMD! >> "%__CrossGenCoreLibLog%"
+!NEXTCMD! >> "%__CrossGenCoreLibLog%" 2>&1
+if NOT !errorlevel! == 0 (
+    echo %__ErrMsgPrefix%%__MsgPrefix%Error: CrossGen /CreatePdb System.Private.CoreLib build failed. Refer to %__CrossGenCoreLibLog%
+    REM Put it in the same log, helpful for CI
+    type %__CrossGenCoreLibLog%
+    goto ExitWithError
+)
+
+REM =========================================================================================
+REM ===
+REM === All builds complete!
+REM ===
+REM =========================================================================================
+
+echo %__MsgPrefix%Crossgenning of System.Private.CoreLib succeeded.  Finished at %TIME%
+echo %__MsgPrefix%Product binaries are available at !__BinDir!
+exit /b 0
+
+
+REM =========================================================================================
+REM === These two routines are intended for the exit code to propagate to the parent process
+REM === Like MSBuild or Powershell. If we directly exit /b 1 from within a if statement in
+REM === any of the routines, the exit code is not propagated.
+REM =========================================================================================
+:ExitWithError
+exit /b 1
+
+:ExitWithCode
+exit /b !__exitCode!
+
+:NoDIA
+echo Error: DIA SDK is missing at "%VSINSTALLDIR%DIA SDK". ^
+Did you install all the requirements for building on Windows, including the "Desktop Development with C++" workload? ^
+Please see https://github.com/dotnet/runtime/blob/master/docs/workflow/requirements/windows-requirements.md ^
+Another possibility is that you have a parallel installation of Visual Studio and the DIA SDK is there. In this case it ^
+may help to copy its "DIA SDK" folder into "%VSINSTALLDIR%" manually, then try again.
+exit /b 1