Fix Sculpt Relax operation when deforming mesh boundaries

Previously, mesh boundaries were relaxed as any other vertex, which was
causing artifacts and unwanted deformation. In order to prevent this,
the mesh filter was using the automasking system to lock the boundary
vertices, which was hacked into the tool. For the brush, the only
solution was to enable boundary automasking to lock those vertices
in plance.

Now the relax vertex function slides the boundary vertices along the
mesh boundary edges, relaxing all the topology correctly while
preserving the shape of the mesh. The automasking hack in the relax
mesh filter was also removed as now vertices slide correctly along
the boundary.

Reviewed By: sergey

Differential Revision: https://developer.blender.org/D8350

1 files changed, 1 insertions, 0 deletions

diff --git a/source/blender/editors/sculpt_paint/sculpt_face_set.c b/source/blender/editors/sculpt_paint/sculpt_face_set.c
index 031b4f8731d..1940b007cb0 100644
--- a/source/blender/editors/sculpt_paint/sculpt_face_set.c
+++ b/source/blender/editors/sculpt_paint/sculpt_face_set.c
@@ -205,6 +205,7 @@ void SCULPT_do_draw_face_sets_brush(Sculpt *sd, Object *ob, PBVHNode **nodes, in
   TaskParallelSettings settings;
   BKE_pbvh_parallel_range_settings(&settings, true, totnode);
   if (ss->cache->alt_smooth) {
+    SCULPT_boundary_info_ensure(ob);
     for (int i = 0; i < 4; i++) {
       BLI_task_parallel_range(0, totnode, &data, do_relax_face_sets_brush_task_cb_ex, &settings);
     }