Sculpt: Edge Automasking

This automasking option protects the open boundary edges of the mesh from the brush deformation. This is needed to sculpt cloths and it works nicely with the cloth brush.
It has a Propagation Steps property that controls the falloff of the mask from the edge.

Limitations:
- The automask is recalculated at the beginning of each stroke, creating a little bit of lag in high poly meshes, but it is not necessary. This can be fixed in the future by caching the edge distances, increasing a little bit the complexity of the code.
- The boundary vertex detection in meshes is not ideal and it fails with triangulated geometry, but it is the same as in the smooth brush. After fixing this, we should refactor the smooth brush to use the API and let the automasking option manually control the affected vertices.
- It does not work in Multires (it needs to be implemented in the API). The smooth brush in Multires is also not making boundary vertices.
- The falloff has a visible line artifact on grid patterns. We can smooth the final automasking factors several iterations, but it will make the initialization much slower. This can also be added in the future if we decided to cache the distances.

Reviewed By: jbakker

Differential Revision: https://developer.blender.org/D6705

1 files changed, 4 insertions, 2 deletions

diff --git a/source/blender/makesdna/DNA_brush_types.h b/source/blender/makesdna/DNA_brush_types.h
index 3f703558e54..2168a63940f 100644
--- a/source/blender/makesdna/DNA_brush_types.h
+++ b/source/blender/makesdna/DNA_brush_types.h
@@ -321,6 +321,7 @@ typedef enum eGP_Sculpt_Mode_Flag {
 typedef enum eAutomasking_flag {
   BRUSH_AUTOMASKING_TOPOLOGY = (1 << 0),
   BRUSH_AUTOMASKING_FACE_SETS = (1 << 1),
+  BRUSH_AUTOMASKING_BOUNDARY_EDGES = (1 << 2),
 } eAutomasking_flag;
 
 typedef struct Brush {
@@ -426,7 +427,7 @@ typedef struct Brush {
   char gpencil_sculpt_tool;
   /** Active grease pencil weight tool. */
   char gpencil_weight_tool;
-  char _pad1_[6];
+  char _pad1[6];
 
   float autosmooth_factor;
 
@@ -446,7 +447,9 @@ typedef struct Brush {
   int curve_preset;
   float hardness;
 
+  /* automasking */
   int automasking_flags;
+  int automasking_boundary_edges_propagation_steps;
 
   /* Factor that controls the shape of the brush tip by rounding the corners of a square. */
   /* 0.0 value produces a square, 1.0 produces a circle. */
@@ -497,7 +500,6 @@ typedef struct Brush {
   float mask_stencil_pos[2];
   float mask_stencil_dimension[2];
 
-  char _pad6[4];
   struct BrushGpencilSettings *gpencil_settings;
 
 } Brush;