Fix boundary edges detection ignoring Face Set visibility

If one of the faces connected to a vertex is hidden in the face sets, we
can assume that the vertex is part of a boundary edge, so it should be
cosidered like that in all automasking and edge detection functions.

Reviewed By: jbakker

Differential Revision: https://developer.blender.org/D7126

1 files changed, 4 insertions, 0 deletions

diff --git a/source/blender/editors/sculpt_paint/sculpt.c b/source/blender/editors/sculpt_paint/sculpt.c
index 07c452d291c..68d44472dc5 100644
--- a/source/blender/editors/sculpt_paint/sculpt.c
+++ b/source/blender/editors/sculpt_paint/sculpt.c
@@ -625,6 +625,10 @@ static bool sculpt_vertex_is_boundary(SculptSession *ss, const int index)
         return false;
       }
 
+      if (!SCULPT_vertex_all_face_sets_visible_get(ss, index)) {
+        return false;
+      }
+
       for (int i = 0; i < vert_map->count; i++) {
         const MPoly *p = &ss->mpoly[vert_map->indices[i]];
         unsigned f_adj_v[2];