Fix T96810: Invalid sculpt normals after some operations

Mask and color brushes were using the existing PBVH vertex "update tag"
to mark their modifications. This was mostly unnecessary, and causes
unnecessary calculation of normals. It also caused errors though,
because they didn't tag the corresponding PBVH node for normal
recalculation, causing problems on the borders of nodes, since one
node might accumulate into another's vertex normals, but the other
node wouldn't also accumulate and normalize the normals.

The solution is to only use the update tag for tagging deformed
vertices that need recalculated normals. Everything else is handled at
the PBVH node level (which was already the case, but it wasn't clear).

The update tag was also used for undo to tag the nodes corresponding to
changed vertices. This was wrong though, because normals and visibility
would also be recalculated for just color or mask undo steps. Instead,
just use local arrays to map from vertices to nodes.

Differential Revision: https://developer.blender.org/D15581

1 files changed, 3 insertions, 3 deletions

diff --git a/source/blender/blenkernel/intern/pbvh.c b/source/blender/blenkernel/intern/pbvh.c
index 6cebcdfea4e..8c1f19f0909 100644
--- a/source/blender/blenkernel/intern/pbvh.c
+++ b/source/blender/blenkernel/intern/pbvh.c
@@ -1964,7 +1964,7 @@ bool BKE_pbvh_node_fully_unmasked_get(PBVHNode *node)
   return (node->flag & PBVH_Leaf) && (node->flag & PBVH_FullyUnmasked);
 }
 
-void BKE_pbvh_vert_mark_update(PBVH *pbvh, int index)
+void BKE_pbvh_vert_tag_update_normal(PBVH *pbvh, int index)
 {
   BLI_assert(pbvh->type == PBVH_FACES);
   pbvh->vert_bitmap[index] = true;
@@ -2123,7 +2123,7 @@ void BKE_pbvh_node_get_bm_orco_data(PBVHNode *node,
   *r_orco_coords = node->bm_orco;
 }
 
-bool BKE_pbvh_node_vert_update_check_any(PBVH *pbvh, PBVHNode *node)
+bool BKE_pbvh_node_has_vert_with_normal_update_tag(PBVH *pbvh, PBVHNode *node)
 {
   BLI_assert(pbvh->type == PBVH_FACES);
   const int *verts = node->vert_indices;
@@ -2991,7 +2991,7 @@ void BKE_pbvh_vert_coords_apply(PBVH *pbvh, const float (*vertCos)[3], const int
       /* no need for float comparison here (memory is exactly equal or not) */
       if (memcmp(mvert->co, vertCos[a], sizeof(float[3])) != 0) {
         copy_v3_v3(mvert->co, vertCos[a]);
-        BKE_pbvh_vert_mark_update(pbvh, a);
+        BKE_pbvh_vert_tag_update_normal(pbvh, a);
       }
     }