Assets: add Asset Catalog system

Catalogs work like directories on disk (without hard-/symlinks), in that
an asset is only contained in one catalog.

See T90066 for design considerations.

#### Known Limitations

Only a single catalog definition file (CDF), is supported, at
`${ASSET_LIBRARY_ROOT}/blender_assets.cats.txt`. In the future this is
to be expanded to support arbitrary CDFs (like one per blend file, one
per subdirectory, etc.).

The current implementation is based on the asset browser, which in
practice means that the asset browser owns the `AssetCatalogService`
instance for the selected asset library. In the future these instances
will be accessible via a less UI-bound asset system.

The UI is still very rudimentary, only showing the catalog ID for the
currently selected asset. Most notably, the loaded catalogs are not
shown yet. The UI is being implemented and will be merged soon.

#### Catalog Identifiers

Catalogs are internally identified by UUID. In older designs this was a
human-readable name, which has the problem that it has to be kept in
sync with its semantics (so when renaming a catalog from X to Y, the
UUID can be kept the same).

Since UUIDs don't communicate any human-readable information, the
mapping from catalog UUID to its path (stored in the Catalog Definition
File, CDF) is critical for understanding which asset is stored in which
human-readable catalog. To make this less critical, and to allow manual
data reconstruction after a CDF is lost/corrupted, each catalog also has
a "simple name" that's stored along with the UUID. This is also stored
on each asset, next to the catalog UUID.

#### Writing to Disk

Before saving asset catalogs to disk, the to-be-overwritten file gets
inspected. Any new catalogs that are found thre are loaded to memory
before writing the catalogs back to disk:

- Changed catalog path: in-memory data wins
- Catalogs deleted on disk: they are recreated based on in-memory data
- Catalogs deleted in memory: deleted on disk as well
- New catalogs on disk: are loaded and thus survive the overwriting

#### Tree Design

This implements the initial tree structure to load catalogs into. See
T90608, and the basic design in T90066.

Reviewed By: Severin

Maniphest Tasks: T91552

Differential Revision: https://developer.blender.org/D12589

2 files changed, 28 insertions, 0 deletions

diff --git a/source/blender/editors/space_file/CMakeLists.txt b/source/blender/editors/space_file/CMakeLists.txt
index 993a52b9084..b60f9df82f6 100644
--- a/source/blender/editors/space_file/CMakeLists.txt
+++ b/source/blender/editors/space_file/CMakeLists.txt
@@ -49,6 +49,7 @@ set(SRC
 )
 
 set(LIB
+  bf_blenkernel
 )
 
 if(WITH_HEADLESS)
diff --git a/source/blender/editors/space_file/filelist.c b/source/blender/editors/space_file/filelist.c
index 511b5b255e9..4f881184990 100644
--- a/source/blender/editors/space_file/filelist.c
+++ b/source/blender/editors/space_file/filelist.c
@@ -56,6 +56,7 @@
 #endif
 
 #include "BKE_asset.h"
+#include "BKE_asset_library.h"
 #include "BKE_context.h"
 #include "BKE_global.h"
 #include "BKE_icons.h"
@@ -386,6 +387,7 @@ typedef struct FileList {
   eFileSelectType type;
   /* The library this list was created for. Stored here so we know when to re-read. */
   AssetLibraryReference *asset_library_ref;
+  struct AssetLibrary *asset_library;
 
   short flags;
 
@@ -1758,6 +1760,13 @@ void filelist_clear_ex(struct FileList *filelist, const bool do_cache, const boo
   if (do_selection && filelist->selection_state) {
     BLI_ghash_clear(filelist->selection_state, NULL, NULL);
   }
+
+  if (filelist->asset_library != NULL) {
+    /* There is no way to refresh the catalogs stored by the AssetLibrary struct, so instead of
+     * "clearing" it, the entire struct is freed. It will be reallocated when needed. */
+    BKE_asset_library_free(filelist->asset_library);
+    filelist->asset_library = NULL;
+  }
 }
 
 void filelist_clear(struct FileList *filelist)
@@ -3136,6 +3145,9 @@ typedef struct FileListReadJob {
    * The job system calls #filelist_readjob_update which moves any read file from #tmp_filelist
    * into #filelist in a thread-safe way.
    *
+   * #tmp_filelist also keeps an `AssetLibrary *` so that it can be loaded in the same thread, and
+   * moved to #filelist once all categories are loaded.
+   *
    * NOTE: #tmp_filelist is freed in #filelist_readjob_free, so any copied pointers need to be set
    * to NULL to avoid double-freeing them. */
   struct FileList *tmp_filelist;
@@ -3266,6 +3278,13 @@ static void filelist_readjob_do(const bool do_lib,
     BLI_stack_discard(todo_dirs);
   }
   BLI_stack_free(todo_dirs);
+
+  /* Check whether assets catalogs need to be loaded. */
+  if (job_params->filelist->asset_library_ref != NULL) {
+    /* Load asset catalogs, into the temp filelist for thread-safety.
+     * #filelist_readjob_endjob() will move it into the real filelist. */
+    job_params->tmp_filelist->asset_library = BKE_asset_library_load(filelist->filelist.root);
+  }
 }
 
 static void filelist_readjob_dir(FileListReadJob *job_params,
@@ -3355,6 +3374,8 @@ static void filelist_readjob_startjob(void *flrjv, short *stop, short *do_update
   BLI_mutex_lock(&flrj->lock);
 
   BLI_assert((flrj->tmp_filelist == NULL) && flrj->filelist);
+  BLI_assert_msg(flrj->filelist->asset_library == NULL,
+                 "Asset library should not yet be assigned at start of read job");
 
   flrj->tmp_filelist = MEM_dupallocN(flrj->filelist);
 
@@ -3415,6 +3436,12 @@ static void filelist_readjob_endjob(void *flrjv)
   /* In case there would be some dangling update... */
   filelist_readjob_update(flrjv);
 
+  /* Move ownership of the asset library from the temporary list to the true filelist. */
+  BLI_assert_msg(flrj->filelist->asset_library == NULL,
+                 "asset library should not already have been allocated");
+  flrj->filelist->asset_library = flrj->tmp_filelist->asset_library;
+  flrj->tmp_filelist->asset_library = NULL; /* MUST be NULL to avoid double-free. */
+
   flrj->filelist->flags &= ~FL_IS_PENDING;
   flrj->filelist->flags |= FL_IS_READY;
 }