Improve generation and use of ref/ projects (#17311)

* Remove useless src/PackageArchive files
- not used outside 2.x branches

* Improve use of ref/ assemblies
- compile against ref/ assemblies but do not change package metadata
  - update the metadata of implementation projects to include the ref/ assembly path
  - update `@(ReferenceAssembly)` metadata for Extensions packages, not `@(PackageReference)`
  - can be disabled using `$(CompileUsingReferenceAssemblies)` e.g. when generating ref/ projects
- include ref/ projects in source build by default
  - remove `$(ExcludeFromSourceBuild)` overrides from ref/ project files
- use latest package references and use project references even when _not_ building the targeting packs
- restore previous `@(Reference)` -> `@(PackageReference)` logic
  - add build-only Microsoft.Internal.Extensions.Refs package reference in most cases
- remove IndirectReferences.props and `@(_ExtensionInternalRefAssemblies)`; no longer needed

* Improve ref/ project generation
- use ../src/**/AssemblyInfo.cs files instead of including attributes in *.Manual.cs files
  - for same reason, copy `@(InternalsVisibleto)` items from src/ to ref/ projects
- use eng/targets/CSharp.ReferenceAssembly.props instead of ref/Directory.Build.props files
- use TFM-specific *.Manual.cs files in ref/ project files instead of ref/Directory.Build.props files

optimizations and usability improvements:
- add `$(BuildMainlyReferenceProviders)` property to focus on reference providers when generating ref/ projects
- disable `$(UseReferenceAssemblyInImplementation)` to avoid using ref/ projects while generating them

nits:
- clean up whitespace and remove blank lines in ref/ project files

* Perform smaller cleanup
- remove `$(IsTargetingPackPatching)`; use only `$(IsTargetingPackBuilding)`
- remove `$(DisableServicingFeatures)`; enable the servicing features we need
  - suppress baseline references even in servicing builds
- restore `$(AdditionalGenApiCmdOptions)`; useful when updating *.Manual.cs files

nits:
- simplify conditions using `$(HasReferenceAssembly)`
- correct spelling in comments
- shorten long lines

* Use a response file for GenAPI commands
- work around dotnet/arcade#4021 and help with additional ref assemblies
- mimic 111462e0c24f and integrate w/ other changes here

* Undo some manual ref/ project changes
- now done automatically or centrally
- remove manual `[TypeForwardedTo]` and `[InternalsVisibleTo]` attributes
  - fully qualify a type now that `using` is gone
- remove dupe `@(Compile)` items for *.Manual.cs files; included in the ref/ project files
- remove redundant `$(AllowUnsafeBlocks)` and `$(NoWarn)` settings

nits:
- rename a *.Manual.cs file that's not TFM-specific
- remove `private` members

* Correct use of `@(ProjectReference)` items for reference providers
- use `@(Reference)` instead

* Remove recently-added `@(Compile)` and `@(Reference)` items
- were added due to missing `[InternalsVisibleTo]` attributes in ref/ assemblies or as early workarounds
  - plus, now transitive references **Just Work:tm:**
- expose `ClosedGenericMatcher` in the usual (*.Manual.cs) way
- also undo Microsoft.Extensions.ApiDescription.Server workaround

* Remove `private` members from ref/ *.Manual.cs files
- not useful and bloat the ref/ assemblies

* Cleanup warnings
- avoid "CSC warning CS2008: No source files specified." building site extensions
- correct warnings (as errors) about `RenderToStringResult` being obsolete
  - add Microsoft.AspNetCore.SpaServices.Tests to Middleware solution

* Remove `@(RuntimeHostConfigurationOption)` workarounds
- deps files are unaffected by new ref/ assembly handling and test projects aren't special-cased
- also execute a test previously skipped due to deps file problems

* Regenerate ref/ projects
- pick up the latest generation changes

(unclear why Mvc.RazorPages/ref/Microsoft.AspNetCore.Mvc.RazorPages.netcoreapp3.0.cs changed but works)

* Fill in missing `internal` types 1 of n
- rename Microsoft.AspNetCore.Components.netstandard2.0.Manual.cs; need `RenderTreeFrame` type everywhere
- add types needed in unit and perf tests to *.Manual.cs files

* Clean up recent commits
- remove recently-added `private` members
- restore `_dummyPrimitive` fields in Microsoft.AspNetCore.Server.HttpSys.Manual.cs

* Add *.Manual.cs files for more projects

* !fixup! fields in *.Manual.cs `struct`s
- GenAPI sometimes generates `_dummy` and `_dummyPrimitive` fields _instead of_ visible members
- what GenAPI generates sometimes have the right length but actual fields don't hurt
  - that is, using the real fields corrects both the visible API and `struct`s' sizes

nits:
- consolidate `namespace`s in Microsoft.AspNetCore.Mvc.Core.Manual.cs

* Remove special case for generating ref/ projects on non-Windows
- referenced issue was closed with no action but workaround still not required
- no tabs in generated content

* Only create ref/ projects for assemblies in the shared framework
- restrict when `$(HasReferenceAssembly)` is `true` by default
- add warnings when `$(IsAspNetCoreApp)` or `$(HasReferenceAssembly)` have unexpected values

* Remove "extra" ref/ projects
- associated implementation projects no longer have `$(HasReferenceAssembly)` set to `true`

* Add a few GenAPI exclusions
- see dotnet/arcade#4488
- generation for these members leads to NREs

* Add more `internal` types and members
- Identity/Core
- Identity/Extensions.Core
- Mvc/Mvc.ViewFeatures

* Add direct dependencies to work around CS1705 errors
- add direct references to some test and sample projects to make intent clear i.e. address CS1705 root cause
  - these projects must use implementation assemblies for those direct references
  - requirement also applies to anything depending on them e.g. functional tests
  - for simplicity, use `$(CompileUsingReferenceAssemblies)` instead of targeted `@(Reference)` metadata
- leads to ~40 projects that do not themselves add ref/ metadata
  - this is _not_ transitive i.e. it applies only to projects that override `$(CompileUsingReferenceAssemblies)`

* nits: Remove a few more `private` members in *.Manual.cs files

* !fixup! correct namespaces of a few types in *.Manual.cs files

* Try another way to fix Microsoft.AspNetCore.Blazor.Build.Tests

* Try another way to fix missing targets in Web.JS.npmproj

1 files changed, 9 insertions, 12 deletions

diff --git a/Directory.Build.targets b/Directory.Build.targets
index 6066ff7fc6..70f9528919 100644
--- a/Directory.Build.targets
+++ b/Directory.Build.targets
@@ -9,9 +9,10 @@
     -->
     <IsPackable Condition="'$(IsAspNetCoreApp)' == 'true' AND '$(IsShippingPackage)' != 'true'">false</IsPackable>
 
-    <!-- Only build assemblies in Microsoft.AspNetCore.App in source build -->
+    <!-- Only build Microsoft.AspNetCore.App and ref/ assemblies in source build. -->
     <!-- Analyzer package are needed in source build for WebSDK -->
-    <ExcludeFromSourceBuild Condition="'$(ExcludeFromSourceBuild)' == '' and '$(DotNetBuildFromSource)' == 'true' and '$(IsAspNetCoreApp)' != 'true' and '$(IsAnalyzersProject)' != 'true'">true</ExcludeFromSourceBuild>
+    <ExcludeFromSourceBuild
+        Condition="'$(ExcludeFromSourceBuild)' == '' and '$(DotNetBuildFromSource)' == 'true' and '$(IsAspNetCoreApp)' != 'true' and '$(IsReferenceAssemblyProject)' != 'true' and '$(IsAnalyzersProject)' != 'true'">true</ExcludeFromSourceBuild>
   </PropertyGroup>
 
   <PropertyGroup Label="Resx settings">
@@ -58,15 +59,11 @@
 
   <Import Project="eng\Baseline.Designer.props" />
 
-  <PropertyGroup
-      Condition=" '$(IsPackable)' != 'false' AND '$(AspNetCorePatchVersion)' != '0' AND '$(DisableServicingFeatures)' != 'true' ">
-    <IsPackageInThisPatch Condition="'$(IsPackageInThisPatch)' == ''">true</IsPackageInThisPatch>
-  </PropertyGroup>
-
   <PropertyGroup Condition=" '$(IsPackable)' != 'false' AND '$(IsServicingBuild)' == 'true' ">
+    <IsPackageInThisPatch Condition="'$(IsPackageInThisPatch)' == ''">true</IsPackageInThisPatch>
     <!-- Used to distinguish between packages building -->
     <IsPackableInNonServicingBuild>true</IsPackableInNonServicingBuild>
-    <!-- Suppress creation of .nupkg for servicing builds. -->
+    <!-- Suppress creation of .nupkg for servicing builds of non-shipping projects. -->
     <IsPackable Condition=" '$(IsPackageInThisPatch)' != 'true' ">false</IsPackable>
   </PropertyGroup>
 
@@ -88,7 +85,7 @@
     <Version Condition="'$(NoBuild)' == 'true' AND '$(DesignTimeBuild)' == 'true'">$(BaselinePackageVersion)</Version>
     <PackageVersion Condition="'$(NoBuild)' == 'true' AND '$(DesignTimeBuild)' == 'true'">$(BaselinePackageVersion)</PackageVersion>
 
-    <!-- For servicing builds, we want to resolve basline versions of project packages that aren't building, always -->
+    <!-- For servicing builds, we want to resolve baseline versions of project packages that aren't building, always -->
     <PackageVersionForPackageVersionInfo>$(BaselinePackageVersion)</PackageVersionForPackageVersionInfo>
   </PropertyGroup>
 
@@ -99,7 +96,8 @@
     <!-- This determines whether a project is available as a <Reference> to other projects in this repo. -->
     <IsProjectReferenceProvider Condition=" '$(IsProjectReferenceProvider)' == '' AND '$(IsImplementationProject)' == 'true' AND '$(PackAsTool)' != 'true' ">true</IsProjectReferenceProvider>
 
-    <HasReferenceAssembly Condition="'$(HasReferenceAssembly)' == '' AND '$(IsProjectReferenceProvider)' == 'true'">true</HasReferenceAssembly>
+    <HasReferenceAssembly
+        Condition=" '$(HasReferenceAssembly)' == '' AND '$(IsProjectReferenceProvider)' == 'true' AND '$(IsAspNetCoreApp)' == 'true' ">true</HasReferenceAssembly>
     <HasReferenceAssembly Condition="'$(HasReferenceAssembly)' == ''">false</HasReferenceAssembly>
 
     <IsPackable Condition="'$(IsPackable)' == '' AND ('$(IsImplementationProject)' == 'true' OR '$(IsAnalyzersProject)' == 'true') ">true</IsPackable>
@@ -161,13 +159,12 @@
   </Target>
 
   <Import Project="eng\Workarounds.targets" />
-  <Import Project="eng\IndirectReferences.props" Condition="'$(IsTestProject)' == 'true' OR '$(IsTestAssetProject)' == 'true' OR '$(IsBenchmarkProject)' == 'true' OR '$(IsSampleProject)' == 'true'" />
   <Import Project="eng\targets\ResolveIisReferences.targets" Condition=" '$(MSBuildProjectExtension)' != '.vcxproj' " />
   <Import Project="eng\targets\Cpp.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.vcxproj'" />
   <Import Project="eng\targets\CSharp.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.csproj'" />
   <Import Project="eng\targets\FSharp.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.fsproj'" />
   <Import Project="eng\targets\Wix.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.wixproj'" />
   <Import Project="eng\targets\Npm.Common.targets"  Condition="'$(MSBuildProjectExtension)' == '.npmproj'" />
-  <Import Project="eng\targets\ReferenceAssembly.targets" Condition=" '$(HasReferenceAssembly)' == 'true' " />
+  <Import Project="eng\targets\ReferenceAssembly.targets" Condition=" $(HasReferenceAssembly) " />
 
 </Project>