[mono] merge wasm-threading-eventpipe into main (#68232)

Merge initial work on multi-threaded WebAssembly.

The normal wasm build is single-threaded.  There should be no functional changes to its behavior.

To enable a multi-threaded build pass `/p:WasmEnableThreads=true`.  See `src/mono/wasm/threads.md` for details.

The big changes are:
1. The normal ref assemblies related to threading retain `[UnsupportedOSPlatform("browser")]` attributes for various threading-related functions
2. In System.Private.CoreLib, the `[UnsupportedOSPlatform]` attributes are removed, and functions that used to always throw PNSE nwo do a runtime check using `System.Threading.Thread.IsThreadStartSupported` to check if threading is enabled in the current build. 
3. A new nuget `Microsoft.NET.WebAssembly.Threading` is created.  It contains experimental ref assemblies without the `[UnsupportedOSPlatform]` attributes.  The intention is that code opting into experimenting with multithreading will include this nuget by setting some property that will be used by the wasm MSBuild SDK to pick a multi-threaded runtime build and configure things appropriately.  (The SDK updates don't exist yet).
4. In the multi-threaded runtime we don't use Emscripten's "main thread" option (ie: the browser thread is the main one); we also continue to run certain runtime-internal jobs (finalizers, GC pump) on the main thread even in the multi-threaded runtime.

Remaining work is tracked in the related issue https://github.com/dotnet/runtime/issues/68162

---

* Initial changes for emscripten 2.0.34

* Use emcc-link.rsp in build targets

* Use updated docker images

* Fix compiler warnings

* Put `--profiling-funcs` to `_EmccLinkFlags`

* Fix build

    src/mono/mono/mini/mini-runtime.c:3407:25: error: ‘invoke’ undeclared (first use in this function); did you mean ‘revoke’?
       3407 |                         invoke = mono_marshal_get_runtime_invoke_dynamic ();

* Add shell to the environment

   Environment setting https://github.com/emscripten-core/emscripten/blob/2.0.34/src/settings.js#L616-L641

   From emscripten 2.0.25 release notes

    - Support for the 'shell' environment is now disabled by default.  Running under
      `d8`, `js`, or `jsc` is not something that most emscripten users ever want to
      do, so including the support code is, more often than not, unnecessary.  Users
      who want shell support can enable it by including 'shell' in `-s ENVIRONMENT`
      (#14535).

   Example of the the size increase for bench sample:

    -a---          12/10/2021  3:35 PM         382113 dotnet.js
    -a---          12/13/2021 10:37 AM         383589 dotnet.js

* Add emcc-link.rsp to PlatformManifestFileEntry

* Feedback

   https://github.com/emscripten-core/emscripten/blob/2fda25eea756c78c8cb024aa5b6c2b188bf7990f/src/settings.js#L1173-L1176

   -s EXPORT_ES6 is link option

* Bump emscripten version

* Bump llvm package version and use its libclang

* Use newer docker images with emscripten 3.1.1

* Remove unused variable

* Add runtime support for threads in the wasm build

   To enable, pass `/p:WasmEnableThreads` when building the runtime

   ./build.sh -Subset mono+libs -os Browser -arch wasm /p:WasmEnableThreads=true

* Prevent runtime from starting twice when loaded in a web worker

* Automatically populate the emscripten mainScriptUrlOrBlob property so that worker initialization can find dotnet.js

* Add compatibility shim so that emscripten's generated worker.js can properly get a Module instance, since we broke the API

* Checkpoint

* Bring back threadpool and add some tracing in diagnostics

* Add comments and fix a typo

* Introduce 'MonoObjectRef' ts type. Migrate mono_string_intern to not have a retval

* Checkpoint (strings are broken for some reason)

* Fix string interning

* Migrate ObjectToString and GetDateValue

* Checkpoint gc safe/unsafe region work

* More ref conversion

* Checkpoint (broken?)

* Fix missing method

* Fix incorrect signatures

* Fix lint

* Add new 'R' signature char for 'ref object'

* Remove AddEventListener and RemoveEventListener Checkpoint

* eslint fixes

* Update call_method signature to avoid passing raw object pointers

* Ref-ify one websocket API and fix types and incorrect rooting of two others

* Deprecation metadata

* More ref

* Remove outdated comments

* Convert typed_array_new to ref

* Add volatile modifiers, satisfy eslint

* Update src/mono/wasm/runtime/corebindings.c

* Missing conflict

* Fix build, set coop gc, and always copy dotnet.worker.js when it's around for apps

* Disable sample profiler, add some functions that were missing from katelyn's PR.

* Add safepoint around ep_rt_wait_event_set.  Tweak sample to explicitly exit wasm in order to properly flush event to the nettrace file (w/ rundown events).

* [gc] Start the GC Finalizer thread on threaded WASM

* [mono] add GC Unsafe in mono_assembly_load; remove in EventPipe

   Remove GC Unsafe hack in ep_rt_wait_event_set

* post-merge cleanup: delete duplicated definitions

* [sample] Env vars should be stringy

* updated dotnet.d.ts

* Add mono_threads_wasm_async_run_in_main_thread; do background work on main

   Don't start a finalizer thread

   Queue all background work to run on the main thread

* [mono] Fix non-threaded wasm build

* Add a System.Threading.Thread.WebAssembly.Threading ref assembly

* Update the browser sample to use System.Threading.Thread.WebAssembly.Threading

* Rationalize System.Threading.Thread build

   In CoreLib, never add the [UnsupportedOSPlatform("browser")] attribute. Rely on runtime checks (`ThrowIfNoThreadStart()`). In System.Threading.Thread ref assembly, always add the unsupported platform
attribute.  Add mismatches to ApiCompat baseline. In System.Threading.Thread.WebAssembly.Threading don't add the attributes, and also set DisablePackageBaselineValidation to prevent Restore from looking for a System.Threading.Thread.WebAssembly.Threading nuget (fixes in-tree ProjectReferences for testing)

* only turn on analyzers for the browser sample

   not all wasm samples

* Make a single Microsoft.NET.WebAssembly.Threading nupkg

   that holds all the special ref assemblies

* works: sample has ProjectReference to the Microsoft.NET.WebAssembly.Threading.proj

* Add System.Threading.ThreadPool.WebAssembly.Threading ref assembly

* ThreadPool: throw PNSE if no thread start

* Update wasm threads.md

* apicompat: correct warnings for System.Threading.ThreadPool

* Add dotnet.worker.js to the runtime pack; update PlatformManifestFileEntry

* [wasm] startup: detect Blazor dotnet.[version].[hash].js location

   Blazor injects a `<link rel="modulepreload" />` element into the header when it boots; detect it and extract the URL of the script.

   This is needed by Emscripten's dotnet.worker.js to run WorkerGlobalScope.importScripts

* one more fix to Microsoft.NET.WebAssembly.Threading

   Seems to be necessary in order for the resulting .nupkg not to reference non-existent nugets for the ProjectReferences

* rename sample to browser-mt-eventpipe

* bring back unmodified browser sample

   The multithreading sample is browser-mt-eventpipe

* update browser-mt-eventpipe sample to use ref assembly refs

   Referencing the rollup Microsoft.NET.WebAssembly.Threading.proj doesn't work (it ends up bundling the ref assemblies into the publish dir and breaking the app)

* Use correct ifdef in AppContext.AnyOS.cs

* [build] support WasmEnableThreads and WasmEnablePerfTracing

   These toplevel options either turn on multi-threading in general, or turn on multithreading only for eventpipe internals.

   For libraries these define `FeatureWasmThreads` and `FeatureWasmPerfTracing` properties and the `FEATURE_WASM_THREADS` and `FEATURE_WASM_PERFTRACING` compiler constants.

   For the native code, they control `DISABLE_THREADS` and `DISABLE_WASM_USER_THREADS` cmake and preprocessor settings.

* Only add the portable threadpool on wasm if threading is enabled

* rename browser-mt-eventpipe csproj and main assembly

   Give it a unique name distinct from Wasm.Browser.CJS.Sample.csproj

* fix /p:WasmEnableThreads=false build after merge

* now fix the WasmEnableThreads=true build

* we need two ThreadPoolBoundHandle implementation stubs

   one for PortableThreadPool when threads are enabled, one for single-threaded wasm

* Add a System.Diagnostics.Tracing ref assembly gated by FeatureWasmPerfTracing

* [eventpipe] use the correct cmake option name

   see src/mono/mono.proj

* revert debug printf and commented out code

* turn off additional logging

* hack: set FeatureWasmPerfTracing in the browser-mt-eventpipe sample

   It would be better to drive this (the inclusion of the tracing runtime component) from a user-visible flag. not the FeatureWasmPerfTracing implementation detail

* remove unused variable, remove unneeded configure checks, revert line damage; add better comment in export.ts

* Exclude Microsoft.NET.WebAssembly.Threading from testPackages.proj

* review feedback, Apply suggestions from code review

* Use a testPackages settings file to skip package runtime item verification

* remove unneeded Directory.Build.props for ref package

   since ti doesn't compile its own assembly, none of these properties are needed

* use one ProjectReference item to share metadata for the ref assemblies

* remove ProjectReference comment and NoTargetsDoNotReferenceOutputAssemblies prop

* Remove unneeded target

* packaging simplification

   - move `_ExperimentalUpdateFileVersion` target to packaging.targets, conditioned on a new `IsExperimentalRefAssembly` attribute.  (The target increments the file version of the ref assembly to make it easier to distinguish from the real non-experimental ref assembly)
   - Remove unneeded src subdirectories in ref assembly libraries
   - Move properties that are only used in the ref assembly projects directory into the projects and delete Directory.Build.props in the experimental ref assembly subdirectories.

* move and rename UpdateExperimentalRefAssemblyFileVersion target

   packages.targets is only included for IsPackable=true projects, and these ref assemblies are not packable.

* Assorted code review nits

* Don't build/pack the multi-threaded sample on single-threaded runtime

* remove gratuitous debug printfs

* Apply suggestions from code review

* merge followup: nullable is enabled by default now

* make eslint happy

* include wasm-config.h in wasm runtime host

* include wasm-config.h into the runtime pack

   fixes aot compilation

* Add wasm-config.h to manifest

* put wasm-config.h into include/wasm from the outset

* put back noExitRuntime replacement for CJS

Co-authored-by: Radek Doulik <radekdoulik@gmail.com>
Co-authored-by: Radek Doulik <radekdoulik@google.com>
Co-authored-by: Zoltan Varga <vargaz@gmail.com>
Co-authored-by: Steve Pfister <steve.pfister@microsoft.com>
Co-authored-by: Katelyn Gadd <kg@luminance.org>
Co-authored-by: Viktor Hofer <viktor.hofer@microsoft.com>

9 files changed, 82 insertions, 3 deletions

diff --git a/src/mono/wasm/runtime/CMakeLists.txt b/src/mono/wasm/runtime/CMakeLists.txt
index dfd9e0d8c01..dac8d63e719 100644
--- a/src/mono/wasm/runtime/CMakeLists.txt
+++ b/src/mono/wasm/runtime/CMakeLists.txt
@@ -2,10 +2,13 @@ cmake_minimum_required(VERSION 3.14.5)
 
 project(mono-wasm-runtime C)
 
+option(DISABLE_THREADS "defined if the build does NOT support multithreading" ON)
+option(DISABLE_WASM_USER_THREADS "defined if the build does not allow user threads to be created in a multithreaded build" OFF)
+
 set(CMAKE_EXECUTABLE_SUFFIX ".js")
 add_executable(dotnet corebindings.c driver.c pinvoke.c)
 
-target_include_directories(dotnet PUBLIC ${MONO_INCLUDES} ${MONO_OBJ_INCLUDES})
+target_include_directories(dotnet PUBLIC ${MONO_INCLUDES} ${MONO_OBJ_INCLUDES} ${CMAKE_CURRENT_BINARY_DIR}/include/wasm)
 target_compile_options(dotnet PUBLIC @${NATIVE_BIN_DIR}/src/emcc-default.rsp @${NATIVE_BIN_DIR}/src/emcc-compile.rsp -DCORE_BINDINGS -DGEN_PINVOKE=1)
 
 set_target_properties(dotnet PROPERTIES COMPILE_FLAGS ${CONFIGURATION_EMCC_FLAGS})
@@ -33,3 +36,5 @@ set_target_properties(dotnet PROPERTIES
 if(CMAKE_BUILD_TYPE STREQUAL "Release")
     add_custom_command(TARGET dotnet POST_BUILD COMMAND ${EMSDK_PATH}/upstream/bin/wasm-opt --enable-exception-handling --strip-dwarf ${NATIVE_BIN_DIR}/dotnet.wasm -o ${NATIVE_BIN_DIR}/dotnet.wasm)
 endif()
+
+configure_file(wasm-config.h.in include/wasm/wasm-config.h)
diff --git a/src/mono/wasm/runtime/cjs/dotnet.cjs.extpost.js b/src/mono/wasm/runtime/cjs/dotnet.cjs.extpost.js
index 9326071c42e..daa0782d477 100644
--- a/src/mono/wasm/runtime/cjs/dotnet.cjs.extpost.js
+++ b/src/mono/wasm/runtime/cjs/dotnet.cjs.extpost.js
@@ -1,6 +1,7 @@
 var require = require || undefined;
 // if loaded into global namespace and configured with global Module, we will self start in compatibility mode
-let ENVIRONMENT_IS_GLOBAL = typeof globalThis.Module === "object" && globalThis.__dotnet_runtime === __dotnet_runtime;
+const __isWorker = typeof globalThis.importScripts === "function";
+let ENVIRONMENT_IS_GLOBAL = !__isWorker && (typeof globalThis.Module === "object" && globalThis.__dotnet_runtime === __dotnet_runtime);
 if (ENVIRONMENT_IS_GLOBAL) {
     createDotnetRuntime(() => { return globalThis.Module; }).then((exports) => exports);
 }
diff --git a/src/mono/wasm/runtime/corebindings.c b/src/mono/wasm/runtime/corebindings.c
index 8cc2ff15a1e..8d6d29cd947 100644
--- a/src/mono/wasm/runtime/corebindings.c
+++ b/src/mono/wasm/runtime/corebindings.c
@@ -13,6 +13,7 @@
 #include <mono/metadata/object.h>
 #include <mono/jit/jit.h>
 
+#include "wasm-config.h"
 #include "gc-common.h"
 
 //JS funcs
diff --git a/src/mono/wasm/runtime/dotnet.d.ts b/src/mono/wasm/runtime/dotnet.d.ts
index 3e5083662d8..0a4a8d48e8e 100644
--- a/src/mono/wasm/runtime/dotnet.d.ts
+++ b/src/mono/wasm/runtime/dotnet.d.ts
@@ -382,6 +382,8 @@ declare const BINDING: {
      * @deprecated Renamed to conv_string_root
      */
     conv_string_rooted: typeof conv_string_root;
+    mono_obj_array_new_ref: (size: number, result: MonoObjectRef) => void;
+    mono_obj_array_set_ref: (array: MonoObjectRef, idx: number, obj: MonoObjectRef) => void;
     js_string_to_mono_string_root: typeof js_string_to_mono_string_root;
     js_typed_array_to_array_root: typeof js_typed_array_to_array_root;
     js_to_mono_obj_root: typeof js_to_mono_obj_root;
diff --git a/src/mono/wasm/runtime/driver.c b/src/mono/wasm/runtime/driver.c
index 2007d25bc22..3a5977ee2c9 100644
--- a/src/mono/wasm/runtime/driver.c
+++ b/src/mono/wasm/runtime/driver.c
@@ -28,6 +28,7 @@
 #include <mono/jit/jit.h>
 #include <mono/jit/mono-private-unstable.h>
 
+#include "wasm-config.h"
 #include "pinvoke.h"
 #include "gc-common.h"
 
@@ -60,6 +61,8 @@ void mono_free (void*);
 int32_t mini_parse_debug_option (const char *option);
 char *mono_method_get_full_name (MonoMethod *method);
 
+static void mono_wasm_init_finalizer_thread (void);
+
 #define MARSHAL_TYPE_NULL 0
 #define MARSHAL_TYPE_INT 1
 #define MARSHAL_TYPE_FP64 2
@@ -464,6 +467,12 @@ mono_wasm_load_runtime (const char *unused, int debug_level)
 	mono_wasm_link_icu_shim ();
 #endif
 
+	// We should enable this as part of the wasm build later
+#ifndef DISABLE_THREADS
+	monoeg_g_setenv ("MONO_THREADS_SUSPEND", "coop", 0);
+	monoeg_g_setenv ("MONO_SLEEP_ABORT_LIMIT", "250", 0);
+#endif
+
 #ifdef DEBUG
 	// monoeg_g_setenv ("MONO_LOG_LEVEL", "debug", 0);
 	// monoeg_g_setenv ("MONO_LOG_MASK", "gc", 0);
@@ -575,6 +584,10 @@ mono_wasm_load_runtime (const char *unused, int debug_level)
 	mono_initialize_internals();
 
 	mono_thread_set_main (mono_thread_current ());
+
+	// TODO: we can probably delay starting the finalizer thread even longer - maybe from JS
+	// once we're done with loading and are about to begin running some managed code.
+	mono_wasm_init_finalizer_thread ();
 }
 
 EMSCRIPTEN_KEEPALIVE MonoAssembly*
@@ -1203,6 +1216,7 @@ mono_wasm_exec_regression (int verbose_level, char *image)
 EMSCRIPTEN_KEEPALIVE int
 mono_wasm_exit (int exit_code)
 {
+	mono_jit_cleanup (root_domain);
 	exit (exit_code);
 }
 
@@ -1323,3 +1337,13 @@ mono_wasm_load_profiler_aot (const char *desc)
 }
 
 #endif
+
+static void
+mono_wasm_init_finalizer_thread (void)
+{
+	// At this time we don't use a dedicated thread for finalization even if threading is enabled.
+	// Finalizers periodically run on the main thread
+#if 0
+	mono_gc_init_finalizer_thread ();
+#endif
+}
diff --git a/src/mono/wasm/runtime/exports.ts b/src/mono/wasm/runtime/exports.ts
index 37578887dde..45124fc219a 100644
--- a/src/mono/wasm/runtime/exports.ts
+++ b/src/mono/wasm/runtime/exports.ts
@@ -170,6 +170,7 @@ let exportedAPI: DotnetPublicAPI;
 
 // this is executed early during load of emscripten runtime
 // it exports methods to global objects MONO, BINDING and Module in backward compatible way
+// At runtime this will be referred to as 'createDotnetRuntime'
 // eslint-disable-next-line @typescript-eslint/explicit-module-boundary-types
 function initializeImportsAndExports(
     imports: { isESM: boolean, isGlobal: boolean, isNode: boolean, isShell: boolean, isWeb: boolean, locateFile: Function, quit_: Function, ExitStatus: ExitStatusError, requirePromise: Promise<Function> },
@@ -293,6 +294,14 @@ function initializeImportsAndExports(
 
     configure_emscripten_startup(module, exportedAPI);
 
+    // HACK: Emscripten expects the return value of this function to always be the Module object,
+    // but we changed ours to return a set of exported namespaces. In order for the emscripten
+    // generated worker code to keep working, we detect that we're running in a worker (via the
+    // presence of globalThis.importScripts) and emulate the old behavior. Note that this will
+    // impact anyone trying to load us in a web worker directly, not just emscripten!
+    if (typeof ((<any>globalThis)["importScripts"]) === "function")
+        return <any>exportedAPI.Module;
+
     return exportedAPI;
 }
 
diff --git a/src/mono/wasm/runtime/pinvoke.c b/src/mono/wasm/runtime/pinvoke.c
index 3804005cb5a..eb422204826 100644
--- a/src/mono/wasm/runtime/pinvoke.c
+++ b/src/mono/wasm/runtime/pinvoke.c
@@ -1,3 +1,4 @@
+#include "wasm-config.h"
 #include "pinvoke.h"
 
 #include <stdint.h>
diff --git a/src/mono/wasm/runtime/startup.ts b/src/mono/wasm/runtime/startup.ts
index 49427b3539c..a8f3b7ce85b 100644
--- a/src/mono/wasm/runtime/startup.ts
+++ b/src/mono/wasm/runtime/startup.ts
@@ -26,6 +26,29 @@ export const mono_wasm_runtime_is_initialized = new Promise((resolve, reject) =>
 let ctx: DownloadAssetsContext | null = null;
 
 export function configure_emscripten_startup(module: DotnetModule, exportedAPI: DotnetPublicAPI): void {
+    // HACK: Emscripten expects us to provide it a fully qualified path where it can find
+    //  our main script so that it can be loaded from inside of workers, because workers
+    //  have their paths relative to the root instead of relative to our location
+    // In the browser we can create a hyperlink and set its href to a relative URL,
+    //  and the browser will convert it into an absolute one for us
+    if (
+        (typeof (globalThis.document) === "object") &&
+        (typeof (globalThis.document.createElement) === "function")
+    ) {
+        // blazor injects a module preload link element for dotnet.[version].[sha].js
+        const blazorDotNetJS = Array.from (document.head.getElementsByTagName("link")).filter(elt => elt.rel !== undefined && elt.rel == "modulepreload" && elt.href !== undefined && elt.href.indexOf("dotnet") != -1 && elt.href.indexOf (".js") != -1);
+        if (blazorDotNetJS.length == 1) {
+            const hr = blazorDotNetJS[0].href;
+            console.log("determined url of main script to be " + hr);
+            (<any>module)["mainScriptUrlOrBlob"] = hr;
+        } else {
+            const temp = globalThis.document.createElement("a");
+            temp.href = "dotnet.js";
+            console.log("determined url of main script to be " + temp.href);
+            (<any>module)["mainScriptUrlOrBlob"] = temp.href;
+        }
+    }
+
     // these could be overriden on DotnetModuleConfig
     if (!module.preInit) {
         module.preInit = [];
@@ -715,4 +738,4 @@ export type DownloadAssetsContext = {
     resolved_promises: (MonoInitFetchResult | undefined)[] | null;
     loaded_files: { url?: string, file: string }[],
     loaded_assets: { [id: string]: [VoidPtr, number] },
-}
\ No newline at end of file
+}
diff --git a/src/mono/wasm/runtime/wasm-config.h.in b/src/mono/wasm/runtime/wasm-config.h.in
new file mode 100644
index 00000000000..c5650ed200b
--- /dev/null
+++ b/src/mono/wasm/runtime/wasm-config.h.in
@@ -0,0 +1,13 @@
+// Licensed to the .NET Foundation under one or more agreements.
+// The .NET Foundation licenses this file to you under the MIT license.
+//
+#ifndef __MONO_WASM_CONFIG_H__
+#define __MONO_WASM_CONFIG_H__
+
+/* Support for threads is disabled */
+#cmakedefine DISABLE_THREADS
+
+/* Support for starting user threads is disabled */
+#cmakedefine DISABLE_WASM_USER_THREADS
+
+#endif/*__MONO_WASM_CONFIG_H__*/