Fix T96810: Invalid sculpt normals after some operations

Mask and color brushes were using the existing PBVH vertex "update tag"
to mark their modifications. This was mostly unnecessary, and causes
unnecessary calculation of normals. It also caused errors though,
because they didn't tag the corresponding PBVH node for normal
recalculation, causing problems on the borders of nodes, since one
node might accumulate into another's vertex normals, but the other
node wouldn't also accumulate and normalize the normals.

The solution is to only use the update tag for tagging deformed
vertices that need recalculated normals. Everything else is handled at
the PBVH node level (which was already the case, but it wasn't clear).

The update tag was also used for undo to tag the nodes corresponding to
changed vertices. This was wrong though, because normals and visibility
would also be recalculated for just color or mask undo steps. Instead,
just use local arrays to map from vertices to nodes.

Differential Revision: https://developer.blender.org/D15581

1 files changed, 1 insertions, 7 deletions

diff --git a/source/blender/editors/sculpt_paint/sculpt_ops.c b/source/blender/editors/sculpt_paint/sculpt_ops.c
index f16763be735..8f96f5cddea 100644
--- a/source/blender/editors/sculpt_paint/sculpt_ops.c
+++ b/source/blender/editors/sculpt_paint/sculpt_ops.c
@@ -882,9 +882,6 @@ static void do_mask_by_color_contiguous_update_nodes_cb(
       continue;
     }
     update_node = true;
-    if (vd.mvert) {
-      BKE_pbvh_vert_mark_update(ss->pbvh, vd.index);
-    }
   }
   BKE_pbvh_vertex_iter_end;
   if (update_node) {
@@ -1001,13 +998,10 @@ static void do_mask_by_color_task_cb(void *__restrict userdata,
       continue;
     }
     update_node = true;
-    if (vd.mvert) {
-      BKE_pbvh_vert_mark_update(ss->pbvh, vd.index);
-    }
   }
   BKE_pbvh_vertex_iter_end;
   if (update_node) {
-    BKE_pbvh_node_mark_redraw(data->nodes[n]);
+    BKE_pbvh_node_mark_update_mask(data->nodes[n]);
   }
 }