Fix T78747: Fix mesh boundary detection and automasking

This issue was produced by a hack in the sculpt mode code from 2.80
 when the sculpt API for connectivity info was not available.
The smooth brush was the only brush that needed connectivity info,
so there were 3 different smooth functions with the connectivity
queries implemented for dyntopo, meshes and grids. The mesh version
of smoothing was checking the number of connected faces to a vertex
to mask the mesh boundaries, which was not covering all cases and
was hardcoded in the smooth function itself.

This patch removes all those legacy functions and unifies all
smooth functions into a single one using the new API and the
automasking system. In order to achieve this, there were needed
some extra changes:

- The smooth brush now does not automasks the boundaries by default,
so its default preset needs to be updated to enable automasking

- The mesh boundary info is extracted once and cached in a
bitmap, similar to the disconnected elements IDs. This makes
boundary detection work as expected in all cases, solving a lot
of known issues with the smooth brush. In multires, this info is
extracted and cached only at the base mesh level, so it is much
more memory efficient than the previous automasking system.

- In order to keep the brushes responsive as they were before,
the automasking system can now skip creating the cache when it
is not needed for the requested options. This means that for
high poly meshes and simple automasking options the brushes
won't lag on start.

Reviewed By: sergey

Maniphest Tasks: T78747

Differential Revision: https://developer.blender.org/D8260

1 files changed, 4 insertions, 0 deletions

diff --git a/source/blender/blenkernel/BKE_paint.h b/source/blender/blenkernel/BKE_paint.h
index e08d3fe26fb..e3a6fb4ba2e 100644
--- a/source/blender/blenkernel/BKE_paint.h
+++ b/source/blender/blenkernel/BKE_paint.h
@@ -24,6 +24,7 @@
  * \ingroup bke
  */
 
+#include "BLI_bitmap.h"
 #include "BLI_utildefines.h"
 #include "DNA_object_enums.h"
 
@@ -291,6 +292,9 @@ typedef struct SculptPersistentBase {
 typedef struct SculptVertexInfo {
   /* Idexed by vertex, stores and ID of its topologycally connected component. */
   int *connected_component;
+
+  /* Indexed by base mesh vertex index, stores if that vertex is a boundary. */
+  BLI_bitmap *boundary;
 } SculptVertexInfo;
 
 typedef struct SculptFakeNeighbors {