Sculpt: Edge Automasking

This automasking option protects the open boundary edges of the mesh from the brush deformation. This is needed to sculpt cloths and it works nicely with the cloth brush.
It has a Propagation Steps property that controls the falloff of the mask from the edge.

Limitations:
- The automask is recalculated at the beginning of each stroke, creating a little bit of lag in high poly meshes, but it is not necessary. This can be fixed in the future by caching the edge distances, increasing a little bit the complexity of the code.
- The boundary vertex detection in meshes is not ideal and it fails with triangulated geometry, but it is the same as in the smooth brush. After fixing this, we should refactor the smooth brush to use the API and let the automasking option manually control the affected vertices.
- It does not work in Multires (it needs to be implemented in the API). The smooth brush in Multires is also not making boundary vertices.
- The falloff has a visible line artifact on grid patterns. We can smooth the final automasking factors several iterations, but it will make the initialization much slower. This can also be added in the future if we decided to cache the distances.

Reviewed By: jbakker

Differential Revision: https://developer.blender.org/D6705

1 files changed, 8 insertions, 0 deletions

diff --git a/source/blender/blenloader/intern/versioning_280.c b/source/blender/blenloader/intern/versioning_280.c
index 68f0abe9b3e..2d1c57b1495 100644
--- a/source/blender/blenloader/intern/versioning_280.c
+++ b/source/blender/blenloader/intern/versioning_280.c
@@ -4811,5 +4811,13 @@ void blo_do_versions_280(FileData *fd, Library *UNUSED(lib), Main *bmain)
         }
       }
     }
+
+    /* Boundary Edges Automasking. */
+    if (!DNA_struct_elem_find(
+            fd->filesdna, "Brush", "int", "automasking_boundary_edges_propagation_steps")) {
+      for (Brush *br = bmain->brushes.first; br; br = br->id.next) {
+        br->automasking_boundary_edges_propagation_steps = 1;
+      }
+    }
   }
 }