Fix Boundary brush not working when the whole mesh is inside the brush radius

When creating the boundary edit data, the loop can stop because a new
vertex was found further from the boundary than the brush radius or
because all vertices of the mesh were already processed. In this second
case, the max_propagation_step was not set, so the code that laters
calculates the falloff was not working, preventing the mesh from
deforming.

Reviewed By: sergey

Differential Revision: https://developer.blender.org/D9215

1 files changed, 3 insertions, 9 deletions

diff --git a/source/blender/editors/sculpt_paint/sculpt_boundary.c b/source/blender/editors/sculpt_paint/sculpt_boundary.c
index 84f2721c31a..5dcaf7d9468 100644
--- a/source/blender/editors/sculpt_paint/sculpt_boundary.c
+++ b/source/blender/editors/sculpt_paint/sculpt_boundary.c
@@ -346,9 +346,9 @@ static void sculpt_boundary_edit_data_init(SculptSession *ss,
   float accum_distance = 0.0f;
 
   while (true) {
-    /* This steps is further away from the boundary than the brush radius, so stop adding more
-     * steps. */
-    if (accum_distance > radius) {
+    /* Stop adding steps to edit info. This happens when a steps is further away from the boundary
+     * than the brush radius or when the entire mesh was already processed. */
+    if (accum_distance > radius || BLI_gsqueue_is_empty(current_iteration)) {
       boundary->max_propagation_steps = num_propagation_steps;
       break;
     }
@@ -416,12 +416,6 @@ static void sculpt_boundary_edit_data_init(SculptSession *ss,
       BLI_gsqueue_push(current_iteration, &next_v);
     }
 
-    /* Stop if no vertices were added in this iteration. At this point, all the mesh should have
-     * been initialized with the edit data. */
-    if (BLI_gsqueue_is_empty(current_iteration)) {
-      break;
-    }
-
     num_propagation_steps++;
   }