Fix T88756: crash when baking with autosmooth

When baking some data, we create a new Mesh with edits and modifiers applied.
However, in some cases (e.g. when there is no modifier), the returned Mesh is
actually referencing the original one and its data layers. When autosmooth is
enabled we also split the Mesh. However, since the new Mesh is referencing the
original one, although `BKE_mesh_split_faces` is creating new vertices and edges,
the reallocation of the custom data layers is preempted because of the
reference, so adding the new vertices and edges overwrites valid data

To fix this we duplicate referenced layers before splitting the faces.

Reviewed By: brecht

Differential Revision: https://developer.blender.org/D11703

3 files changed, 19 insertions, 0 deletions

diff --git a/source/blender/blenkernel/BKE_customdata.h b/source/blender/blenkernel/BKE_customdata.h
index ed319948160..6d6639b1e70 100644
--- a/source/blender/blenkernel/BKE_customdata.h
+++ b/source/blender/blenkernel/BKE_customdata.h
@@ -233,6 +233,9 @@ void *CustomData_duplicate_referenced_layer_named(struct CustomData *data,
                                                   const int totelem);
 bool CustomData_is_referenced_layer(struct CustomData *data, int type);
 
+/* Duplicate all the layers with flag NOFREE, and remove the flag from duplicated layers. */
+void CustomData_duplicate_referenced_layers(CustomData *data, int totelem);
+
 /* set the CD_FLAG_NOCOPY flag in custom data layers where the mask is
  * zero for the layer type, so only layer types specified by the mask
  * will be copied
diff --git a/source/blender/blenkernel/intern/customdata.c b/source/blender/blenkernel/intern/customdata.c
index b1bb8b9715e..ea05ab99d88 100644
--- a/source/blender/blenkernel/intern/customdata.c
+++ b/source/blender/blenkernel/intern/customdata.c
@@ -2813,6 +2813,14 @@ void *CustomData_duplicate_referenced_layer_named(CustomData *data,
   return customData_duplicate_referenced_layer_index(data, layer_index, totelem);
 }
 
+void CustomData_duplicate_referenced_layers(CustomData *data, int totelem)
+{
+  for (int i = 0; i < data->totlayer; i++) {
+    CustomDataLayer *layer = &data->layers[i];
+    layer->data = CustomData_duplicate_referenced_layer(data, layer->type, totelem);
+  }
+}
+
 bool CustomData_is_referenced_layer(struct CustomData *data, int type)
 {
   /* get the layer index of the first layer of type */
diff --git a/source/blender/blenkernel/intern/mesh.c b/source/blender/blenkernel/intern/mesh.c
index 08d3236f3a9..40b95f9535b 100644
--- a/source/blender/blenkernel/intern/mesh.c
+++ b/source/blender/blenkernel/intern/mesh.c
@@ -2087,6 +2087,14 @@ void BKE_mesh_split_faces(Mesh *mesh, bool free_loop_normals)
   SplitFaceNewVert *new_verts = NULL;
   SplitFaceNewEdge *new_edges = NULL;
 
+  /* Ensure we own the layers, we need to do this before split_faces_prepare_new_verts as it will
+   * directly assign new indices to existing edges and loops. */
+  CustomData_duplicate_referenced_layers(&mesh->vdata, mesh->totvert);
+  CustomData_duplicate_referenced_layers(&mesh->edata, mesh->totedge);
+  CustomData_duplicate_referenced_layers(&mesh->ldata, mesh->totloop);
+  /* Update pointers in case we duplicated referenced layers. */
+  BKE_mesh_update_customdata_pointers(mesh, false);
+
   /* Detect loop normal spaces (a.k.a. smooth fans) that will need a new vert. */
   const int num_new_verts = split_faces_prepare_new_verts(
       mesh, &lnors_spacearr, &new_verts, memarena);