Move MSBuild projects out of the native build scripts (#31701)

* Move coreclr managed tools that don't need to be published and System.Private.CoreLib into their own subsets in Subsets.props and drive their builds via Arcade instead of the coreclr scripts.

* Remove building System.Private.CoreLib from build scripts. Eventing sources TBD.

* Build packages via an Arcade subset instead of within build.cmd/build.sh

* Remove packages and test build from build.cmd. Clean up options on build.cmd and build.sh

* Generate eventing sources for System.Private.CoreLib within MSBuild instead of in the build.cmd/sh scripts. Clean up the build.sh usage documentation (lots of copy-paste errors).

* Remove check-definitions.py script and supporting goo.

* Fix eventing generation.

* Wire up  TargetArchitecture and OSGroup to coreclr's BuildArch and BuildOS to enable building System.Private.CoreLib for the correct output given the script parameters.

* Stop passing down now non-existent skiptests parameter.

* Fix coreclr .nuget rid parsing to account for linux-musl correctly.

* Update build scripts to use Arcade to build packages.

* Convert from Properties to AdditionalProperties.

* Build crossgen as part of Arcade (self-contained publish TBD).

* Pull down .NET runtime CLI for helix runs with crossgen2.

* Output crossgen2 to the BinDir/crossgen when built from the project file. Enable the test shell scripts to either resolve the dotnet CLI from the path or from relative to Core_Root if the __DotNet variable is undefined.

* Generate distro rid during windows build.

* Various fixes to get files binplaced correctly and builds to work on Linux.

* Fix search for dotnet in repo on bash.

* Remove __BuildTest variable and fix up the script.

* Fix steps in build-job to build S.P.CL before the runtime and build managed tools after the runtime.

* Fix casing typo.

* Refactor detecting system OS and Architecture.

* Fix wasm rid calculation.

* Run crossgen2 on corerun so we run it on the runtime we're going to ship it with.

* Assemble a live self-contained publish in the crossgen2 package.

* Fix init-os-and-arch resolution.

* Ensure that the runtime files we package in crossgen2 are the crossgened ones.

* Remove extraneous /bl

* Build R2RDump to a subfolder instead of the product dir.

* Seperate out the crosssgening corelib step into a separate script.

* Fix permissions on crossgen-corelib.sh

* Rename build.cmd/sh to build-runtime.cmd/sh

* Fix parameters in crossgen2determinism test.

* Run crossgen on System.Private.CoreLib in CI.

* Restore original build.cmd/sh scripts with a deprecation warning and update instructions.

* Initialize developer prompt in crossgen-corelib.cmd script so we can find DIA on CI machines.

* Make sure vs tools are enabled so we can resolve DiaSymReader.

* Call build-native instead of build.

* Fix build-native -> build-runtime.

* Fix permissions.

* Remove dead MSBuild code.

* Factor out python locating targets to make them available to the whole repo.

* PR Feedback/simplification.

* Create a self-contained deps.json for crossgen2 and add it to the package.

* Pass runtimeconfiguration as configuration for crossgen2 deps file generation.

* Try specifying rids directly in crossgen2 project to get the restore correct for the live-build pipeline.

* Run crossgen2 via corerun in R2R.SuperILC.

* Add binlogs to build-runtime.cmd

* Combine non build-runtime steps into a single step so we get one binlog that doesn't get overwritten.

* Add reference to issue describing new workflow.

* Quote python path

* Set arch default.

* PR Feedback and clean up crossgen-corelib's argument list.

* Fix subset name matching to work when one subset name is a substring of another.

* Pass the official build id as an msbuild arg at all times.
Additional cleanup for crossgen-corelib step.

* PR Feedback.

* Fix default subset list construction.

* Add deprecation warnings at beginning and end of build.cmd and build.sh.

* Setup dirs in crossgen-corelib.sh

* Move enforcepgo argument to before the official build arg so that it gets consumed correctly.

* Add cross-os DAC builds as subsets.
Clean up build-runtime parameters and passing the official build ID from runtime.proj.

* Update script usage docs.

* Skip the cross-os dac on x86.

* Build windows->linux cross dac on PR and CI.

* Don't build the cross-OS linux dac on Windows x86.

1 files changed, 256 insertions, 0 deletions

diff --git a/src/coreclr/build-runtime.sh b/src/coreclr/build-runtime.sh
new file mode 100755
index 00000000000..149b1944fa9
--- /dev/null
+++ b/src/coreclr/build-runtime.sh
@@ -0,0 +1,256 @@
+#!/usr/bin/env bash
+
+# resolve python-version to use
+if [[ -z "$PYTHON" ]]; then
+    if ! PYTHON=$(command -v python3 || command -v python2 || command -v python || command -v py)
+    then
+       echo "Unable to locate build-dependency python!" 1>&2
+       exit 1
+    fi
+fi
+# validate python-dependency
+# useful in case of explicitly set option.
+if ! command -v "$PYTHON" > /dev/null
+then
+   echo "Unable to locate build-dependency python ($PYTHON)!" 1>&2
+   exit 1
+fi
+
+export PYTHON
+
+usage_list+=("-nopgooptimize: do not use profile guided optimizations.")
+usage_list+=("-pgoinstrument: generate instrumented code for profile guided optimization enabled binaries.")
+usage_list+=("-skipcrossarchnative: Skip building cross-architecture native binaries.")
+usage_list+=("-skiprestoreoptdata: skip restoring optimization data.")
+usage_list+=("-staticanalyzer: skip native image generation.")
+
+setup_dirs_local()
+{
+    setup_dirs
+
+    mkdir -p "$__LogsDir"
+    mkdir -p "$__MsbuildDebugLogsDir"
+
+    if [[ "$__CrossBuild" == 1 ]]; then
+        mkdir -p "$__CrossComponentBinDir"
+    fi
+}
+
+restore_optdata()
+{
+    local OptDataProjectFilePath="$__ProjectRoot/src/.nuget/optdata/optdata.csproj"
+    if [[ "$__SkipRestoreOptData" == 0 && "$__IsMSBuildOnNETCoreSupported" == 1 ]]; then
+        echo "Restoring the OptimizationData package"
+        "$__RepoRootDir/eng/common/msbuild.sh" /clp:nosummary $__ArcadeScriptArgs \
+                                               $OptDataProjectFilePath /t:Restore /m \
+                                               $__CommonMSBuildArgs $__UnprocessedBuildArgs \
+                                               /nodereuse:false
+        local exit_code="$?"
+        if [[ "$exit_code" != 0 ]]; then
+            echo "${__ErrMsgPrefix}Failed to restore the optimization data package."
+            exit "$exit_code"
+        fi
+    fi
+
+    if [[ "$__PgoOptimize" == 1 && "$__IsMSBuildOnNETCoreSupported" == 1 ]]; then
+        # Parse the optdata package versions out of msbuild so that we can pass them on to CMake
+
+        local PgoDataPackagePathOutputFile="${__IntermediatesDir}/optdatapath.txt"
+
+        # Writes into ${PgoDataPackagePathOutputFile}
+        "$__RepoRootDir/eng/common/msbuild.sh" /clp:nosummary $__ArcadeScriptArgs $OptDataProjectFilePath /t:DumpPgoDataPackagePath ${__CommonMSBuildArgs} /p:PgoDataPackagePathOutputFile=${PgoDataPackagePathOutputFile} > /dev/null 2>&1
+        local exit_code="$?"
+        if [[ "$exit_code" != 0 || ! -f "${PgoDataPackagePathOutputFile}" ]]; then
+            echo "${__ErrMsgPrefix}Failed to get PGO data package path."
+            exit "$exit_code"
+        fi
+
+        __PgoOptDataPath=$(<"${PgoDataPackagePathOutputFile}")
+    fi
+}
+
+build_cross_architecture_components()
+{
+    local intermediatesForBuild="$__IntermediatesDir/Host$__CrossArch/crossgen"
+    local crossArchBinDir="$__BinDir/$__CrossArch"
+
+    mkdir -p "$intermediatesForBuild"
+    mkdir -p "$crossArchBinDir"
+
+    __SkipCrossArchBuild=1
+    # check supported cross-architecture components host(__HostArch)/target(__BuildArch) pair
+    if [[ ("$__BuildArch" == "arm" || "$__BuildArch" == "armel") && ("$__CrossArch" == "x86" || "$__CrossArch" == "x64") ]]; then
+        __SkipCrossArchBuild=0
+    elif [[ "$__BuildArch" == "arm64" && "$__CrossArch" == "x64" ]]; then
+        __SkipCrossArchBuild=0
+    else
+        # not supported
+        return
+    fi
+
+    __CMakeBinDir="$crossArchBinDir"
+    CROSSCOMPILE=0
+    export __CMakeBinDir CROSSCOMPILE
+
+    __CMakeArgs="-DCLR_CMAKE_TARGET_ARCH=$__BuildArch -DCLR_CROSS_COMPONENTS_BUILD=1 $__CMakeArgs"
+    build_native "$__CrossArch" "$__ProjectRoot" "$__ProjectRoot" "$intermediatesForBuild" "cross-architecture components"
+
+    CROSSCOMPILE=1
+    export CROSSCOMPILE
+}
+
+handle_arguments_local() {
+    case "$1" in
+
+        ignorewarnings|-ignorewarnings)
+            __IgnoreWarnings=1
+            __CMakeArgs="-DCLR_CMAKE_WARNINGS_ARE_ERRORS=OFF $__CMakeArgs"
+            ;;
+
+        nopgooptimize|-nopgooptimize)
+            __PgoOptimize=0
+            __SkipRestoreOptData=1
+            ;;
+
+        pgoinstrument|-pgoinstrument)
+            __PgoInstrument=1
+            ;;
+
+        skipcrossarchnative|-skipcrossarchnative)
+            __SkipCrossArchNative=1
+            ;;
+
+        staticanalyzer|-staticanalyzer)
+            __StaticAnalyzer=1
+            ;;
+
+        *)
+            __UnprocessedBuildArgs="$__UnprocessedBuildArgs $1"
+            ;;
+    esac
+}
+
+echo "Commencing CoreCLR Repo build"
+
+# Argument types supported by this script:
+#
+# Build architecture - valid values are: x64, ARM.
+# Build Type         - valid values are: Debug, Checked, Release
+#
+# Set the default arguments for build
+
+# Obtain the location of the bash script to figure out where the root of the repo is.
+__ProjectRoot="$(cd "$(dirname "$0")"; pwd -P)"
+__RepoRootDir="$(cd "$__ProjectRoot"/../..; pwd -P)"
+
+__BuildArch=
+__BuildType=Debug
+__CodeCoverage=0
+__IgnoreWarnings=0
+
+# Set the various build properties here so that CMake and MSBuild can pick them up
+__Compiler=clang
+__CompilerMajorVersion=
+__CompilerMinorVersion=
+__CommonMSBuildArgs=
+__ConfigureOnly=0
+__CrossBuild=0
+__DistroRid=""
+__PgoInstrument=0
+__PgoOptDataPath=""
+__PgoOptimize=1
+__PortableBuild=1
+__ProjectDir="$__ProjectRoot"
+__RootBinDir="$__RepoRootDir/artifacts"
+__SignTypeArg=""
+__SkipConfigure=0
+__SkipNative=0
+__SkipCrossArchNative=0
+__SkipGenerateVersion=0
+__SkipManaged=0
+__SkipRestore=""
+__SkipRestoreOptData=0
+__SourceDir="$__ProjectDir/src"
+__StaticAnalyzer=0
+__UnprocessedBuildArgs=
+__UseNinja=0
+__VerboseBuild=0
+__ValidateCrossArg=1
+__CMakeArgs=""
+
+source "$__ProjectRoot"/_build-commons.sh
+
+if [[ "${__BuildArch}" != "${__HostArch}" ]]; then
+    __CrossBuild=1
+fi
+
+# Set dependent variables
+__LogsDir="$__RootBinDir/log"
+__MsbuildDebugLogsDir="$__LogsDir/MsbuildDebugLogs"
+
+# Set the remaining variables based upon the determined build configuration
+__BinDir="$__RootBinDir/bin/coreclr/$__BuildOS.$__BuildArch.$__BuildType"
+__IntermediatesDir="$__RootBinDir/obj/coreclr/$__BuildOS.$__BuildArch.$__BuildType"
+__ArtifactsIntermediatesDir="$__RepoRootDir/artifacts/obj/coreclr"
+export __IntermediatesDir __ArtifactsIntermediatesDir
+__CrossComponentBinDir="$__BinDir"
+
+__CrossArch="$__HostArch"
+if [[ "$__CrossBuild" == 1 ]]; then
+    __CrossComponentBinDir="$__CrossComponentBinDir/$__CrossArch"
+fi
+
+# CI_SPECIFIC - On CI machines, $HOME may not be set. In such a case, create a subfolder and set the variable to set.
+# This is needed by CLI to function.
+if [[ -z "$HOME" ]]; then
+    if [[ ! -d "$__ProjectDir/temp_home" ]]; then
+        mkdir temp_home
+    fi
+    HOME="$__ProjectDir"/temp_home
+    export HOME
+    echo "HOME not defined; setting it to $HOME"
+fi
+
+# Specify path to be set for CMAKE_INSTALL_PREFIX.
+# This is where all built CoreClr libraries will copied to.
+__CMakeBinDir="$__BinDir"
+export __CMakeBinDir
+
+# Make the directories necessary for build if they don't exist
+setup_dirs_local
+
+# Set up the directory for MSBuild debug logs.
+MSBUILDDEBUGPATH="${__MsbuildDebugLogsDir}"
+export MSBUILDDEBUGPATH
+
+# Check prereqs.
+check_prereqs
+
+# Restore the package containing profile counts for profile-guided optimizations
+restore_optdata
+
+# Build the coreclr (native) components.
+__CMakeArgs="-DCLR_CMAKE_PGO_INSTRUMENT=$__PgoInstrument -DCLR_CMAKE_OPTDATA_PATH=$__PgoOptDataPath -DCLR_CMAKE_PGO_OPTIMIZE=$__PgoOptimize -DCLR_REPO_ROOT_DIR=\"$__RepoRootDir\" $__CMakeArgs"
+
+if [[ "$__SkipConfigure" == 0 && "$__CodeCoverage" == 1 ]]; then
+    __CMakeArgs="-DCLR_CMAKE_ENABLE_CODE_COVERAGE=1 $__CMakeArgs"
+fi
+
+if [[ "$__SkipNative" == 1 ]]; then
+    echo "Skipping CoreCLR component build."
+else
+    build_native "$__BuildArch" "$__ProjectRoot" "$__ProjectRoot" "$__IntermediatesDir" "CoreCLR component"
+
+    # Build cross-architecture components
+    if [[ "$__SkipCrossArchNative" != 1 ]]; then
+        if [[ "$__CrossBuild" == 1 ]]; then
+            build_cross_architecture_components
+        fi
+    fi
+fi
+
+# Build complete
+
+echo "Repo successfully built."
+echo "Product binaries are available at $__BinDir"
+exit 0