Fix mesh shrinking when using the relax mesh filter.

If the relax mesh filter was used on a non manifold mesh with open
boundaries, all the vertices were relaxed and the mesh was shrinking.
This was an unintended behavior that was making the filter unusable with
these meshes.

The mesh filter is now initializing an automasking buffer using the same
boundary automasking function from the brush code. Now edges are
preserved and the relax filter works as it should.

Reviewed By: jbakker

Differential Revision: https://developer.blender.org/D7097

1 files changed, 3 insertions, 0 deletions

diff --git a/source/blender/editors/sculpt_paint/sculpt_intern.h b/source/blender/editors/sculpt_paint/sculpt_intern.h
index 7f90f86edd4..4e36ae7aa5e 100644
--- a/source/blender/editors/sculpt_paint/sculpt_intern.h
+++ b/source/blender/editors/sculpt_paint/sculpt_intern.h
@@ -663,6 +663,9 @@ typedef struct FilterCache {
   float *prev_mask;
   float mask_expand_initial_co[3];
 
+  /* Used to prevent undesired results on certain mesh filters. */
+  float *automask;
+
   int new_face_set;
   int *prev_face_set;