Fix sculpt neighbor iterator not taking visibility into account

Sculpting tools are designed to ignore hidden geometry and behave like
hidden geometry does not exist.
When getting the neighbors of a vertex, now this takes into account
hidden geometry to avoid returing neighbors which connected edge is not
visible. This should make corner cases of a lot of tools work properly,
especially when working in low poly meshes when is common to have a
single face loop hidden.

Reviewed By: JacquesLucke

Differential Revision: https://developer.blender.org/D11007

1 files changed, 5 insertions, 1 deletions

diff --git a/source/blender/blenkernel/BKE_paint.h b/source/blender/blenkernel/BKE_paint.h
index 228b52123f3..73413b61456 100644
--- a/source/blender/blenkernel/BKE_paint.h
+++ b/source/blender/blenkernel/BKE_paint.h
@@ -486,7 +486,11 @@ typedef struct SculptSession {
   /* Total number of polys of the base mesh. */
   int totfaces;
   /* Face sets store its visibility in the sign of the integer, using the absolute value as the
-   * Face Set ID. Positive IDs are visible, negative IDs are hidden. */
+   * Face Set ID. Positive IDs are visible, negative IDs are hidden.
+   * The 0 ID is not used by the tools or the visibility system, it is just used when creating new
+   * geometry (the trim tool, for example) to detect which geometry was just added, so it can be
+   * assigned a valid Face Set after creation. Tools are not intended to run with Face Sets IDs set
+   * to 0. */
   int *face_sets;
 
   /* BMesh for dynamic topology sculpting */