Fix T96810: Invalid sculpt normals after some operations

Mask and color brushes were using the existing PBVH vertex "update tag"
to mark their modifications. This was mostly unnecessary, and causes
unnecessary calculation of normals. It also caused errors though,
because they didn't tag the corresponding PBVH node for normal
recalculation, causing problems on the borders of nodes, since one
node might accumulate into another's vertex normals, but the other
node wouldn't also accumulate and normalize the normals.

The solution is to only use the update tag for tagging deformed
vertices that need recalculated normals. Everything else is handled at
the PBVH node level (which was already the case, but it wasn't clear).

The update tag was also used for undo to tag the nodes corresponding to
changed vertices. This was wrong though, because normals and visibility
would also be recalculated for just color or mask undo steps. Instead,
just use local arrays to map from vertices to nodes.

Differential Revision: https://developer.blender.org/D15581

1 files changed, 2 insertions, 2 deletions

diff --git a/source/blender/blenkernel/BKE_pbvh.h b/source/blender/blenkernel/BKE_pbvh.h
index f517ff3a949..5641ff0ba34 100644
--- a/source/blender/blenkernel/BKE_pbvh.h
+++ b/source/blender/blenkernel/BKE_pbvh.h
@@ -308,7 +308,7 @@ void BKE_pbvh_node_fully_unmasked_set(PBVHNode *node, int fully_masked);
 bool BKE_pbvh_node_fully_unmasked_get(PBVHNode *node);
 
 void BKE_pbvh_mark_rebuild_pixels(PBVH *pbvh);
-void BKE_pbvh_vert_mark_update(PBVH *pbvh, int index);
+void BKE_pbvh_vert_tag_update_normal(PBVH *pbvh, int index);
 
 void BKE_pbvh_node_get_grids(PBVH *pbvh,
                              PBVHNode *node,
@@ -526,7 +526,7 @@ void BKE_pbvh_node_get_bm_orco_data(PBVHNode *node,
  * however this is important to avoid having to recalculate bound-box & sync the buffers to the
  * GPU (which is far more expensive!) See: T47232.
  */
-bool BKE_pbvh_node_vert_update_check_any(PBVH *pbvh, PBVHNode *node);
+bool BKE_pbvh_node_has_vert_with_normal_update_tag(PBVH *pbvh, PBVHNode *node);
 
 // void BKE_pbvh_node_BB_reset(PBVHNode *node);
 // void BKE_pbvh_node_BB_expand(PBVHNode *node, float co[3]);