Add support for reflection invoking ref return methods

This is a port of dotnet/coreclr#17732. Fixes bug 603305.

Per popular demand from people who would like to serialize ref returning properties or use them in data binding, adding support for this in MethodInfo.Invoke. The choice was to make these dereference the result and return it like that. For the corner case of returning a ref to a null, a `NullReferenceException` will be thrown (this behavior has a very unfortunate side effect of slowing down the general Invoke path because we can't throw this exception from a place where we would end up wrapping this in a TargetInvocationException).

I also made tweaks to byref parameters path to fix byref pointer parameters, but this is horribly broken on the CLR so I didn't bother to test it here. Should work.

Also includes a slight size on disk optimization that merges common tails of the static/instance paths of the invoker thunk.

Testing of the calling convention converter portion was done by disabling invoke thunks generation in NUTC and running UnitTests. I also ran it with GCStresss (RH_GCStressThrottleMode=1, RH_GCStress_Mode=1). Let me know if there's more testing needed. This has a huge test matrix for being such a corner case thing.

[tfs-changeset: 1709919]

2 files changed, 6 insertions, 4 deletions

diff --git a/src/System.Private.Reflection.Core/src/Resources/Strings.resx b/src/System.Private.Reflection.Core/src/Resources/Strings.resx
index 6488cc434..820a41e26 100644
--- a/src/System.Private.Reflection.Core/src/Resources/Strings.resx
+++ b/src/System.Private.Reflection.Core/src/Resources/Strings.resx
@@ -156,9 +156,6 @@
   <data name="Arg_HTCapacityOverflow" xml:space="preserve">
     <value>Hashtable's capacity overflowed and went negative. Check load factor, capacity and the current size of the table.</value>
   </data>
-  <data name="NotSupported_ByRefReturn" xml:space="preserve">
-    <value>ByRef return value not supported in reflection invocation.</value>
-  </data>
   <data name="Arg_DlgtTargMeth" xml:space="preserve">
     <value>Cannot bind to the target method because its signature is not compatible with that of the delegate type.</value>
   </data>
diff --git a/src/System.Private.Reflection.Core/src/System/Reflection/Runtime/MethodInfos/RuntimeMethodInfo.cs b/src/System.Private.Reflection.Core/src/System/Reflection/Runtime/MethodInfos/RuntimeMethodInfo.cs
index 33d73b16a..4af12bfc9 100644
--- a/src/System.Private.Reflection.Core/src/System/Reflection/Runtime/MethodInfos/RuntimeMethodInfo.cs
+++ b/src/System.Private.Reflection.Core/src/System/Reflection/Runtime/MethodInfos/RuntimeMethodInfo.cs
@@ -354,7 +354,12 @@ namespace System.Reflection.Runtime.MethodInfos
                 if (methodInvoker == null)
                 {
                     if (ReturnType.IsByRef)
-                        throw new NotSupportedException(SR.NotSupported_ByRefReturn);
+                    {
+                        // The invoker is going to dereference and box (for structs) the result of the invocation
+                        // on behalf of the caller. Can't box byref-like types and can't box void.
+                        if (ReturnType.GetElementType().IsByRefLike || ReturnType.GetElementType() == CommonRuntimeTypes.Void)
+                            throw new NotSupportedException();
+                    }
                     methodInvoker = _lazyMethodInvoker = this.UncachedMethodInvoker;
                 }
                 return methodInvoker;