Mesh Edit: preserve Custom Normal vectors in topology operators.

Custom Loop Normals are normally encoded relative to the default
normals, similar to normal maps, allowing them to naturally follow
mesh deformations. Changes to mesh topology however often result
in nonsensical effects that are not desired.

The Remove Doubles operation especially (now known as Merge By
Distance) is intended as a purely topological operation, and
definitely should not change the vector of the custom normals.

This patch implements that behavior by converting the relative
encoding into an absolute vector layer for the duration of the
operation. It also modifies other Merge types in this way for
consistency, the Rip operator as their inverse counterpart;
and also Delete, Dissolve, Connect Path and Knife operators
as other examples more related to topology than shape.

On the technical side, this ports mesh_normals_loop_custom_set
to BMesh, and then uses a temporary Custom Data layer to store
the normals as vectors for the duration of the above mentioned
operations. When the normals are converted back to custom data,
the caller can choose whether to mark edges as sharp to preserve
distinct normals, or just average them instead. All but Remove
Doubles choose to average for now.

Differential Revision: https://developer.blender.org/D4994

1 files changed, 4 insertions, 0 deletions

diff --git a/source/blender/editors/mesh/editmesh_rip.c b/source/blender/editors/mesh/editmesh_rip.c
index 5d9923c6a7d..e805507351c 100644
--- a/source/blender/editors/mesh/editmesh_rip.c
+++ b/source/blender/editors/mesh/editmesh_rip.c
@@ -1053,6 +1053,8 @@ static int edbm_rip_invoke(bContext *C, wmOperator *op, const wmEvent *event)
      * useful selection for grabbing.
      */
 
+    BM_custom_loop_normals_to_vector_layer(bm);
+
     /* BM_ELEM_SELECT --> BM_ELEM_TAG */
     BM_ITER_MESH (e, &iter, bm, BM_EDGES_OF_MESH) {
       BM_elem_flag_set(e, BM_ELEM_TAG, BM_elem_flag_test(e, BM_ELEM_SELECT));
@@ -1070,6 +1072,8 @@ static int edbm_rip_invoke(bContext *C, wmOperator *op, const wmEvent *event)
       continue;
     }
 
+    BM_custom_loop_normals_from_vector_layer(bm, false);
+
     BLI_assert(singlesel ? (bm->totvertsel > 0) : (bm->totedgesel > 0));
 
     if (bm->totvertsel == 0) {