Fix T89164: Sculpt "Smooth" brush crash with zero pressure

Caused by {rB3e5431fdf439}

Issue is that sculpting could start with using `SCULPT_smooth` and
(because of the Pressure sensitivity dropping to zero) code would switch
to `SCULPT_enhance_details_brush` at strength zero.
Issue with this though is that this can be in the middle or end of a
stroke and the necessary `ss->cache->detail_directions` are only
initialized for the first brush step (see
`SCULPT_stroke_is_first_brush_step` in `SCULPT_enhance_details_brush`).
With these missing, it could only go downhill from there.

Suggest to prevent the "mode-flip" from `SCULPT_smooth` to
`SCULPT_enhance_details_brush` (happening solely because of pressure
strength) by changing the condition.
Now do `SCULPT_enhance_details_brush` only if strength is **below** zero
and `SCULPT_smooth` else (flipping from enhance_details to regular smooth
is fine).

If inverting the brush in the middle of the stroke will be supported at
some point, the codepath of `smooth` would have to inform the cache that
invert changed and detail_directions would have to be initialized then
(even if not at the start of the stroke).

Maniphest Tasks: T89164

Differential Revision: https://developer.blender.org/D12676

1 files changed, 6 insertions, 1 deletions

diff --git a/source/blender/editors/sculpt_paint/sculpt_smooth.c b/source/blender/editors/sculpt_paint/sculpt_smooth.c
index 38165b7622f..1bfe8e1cbf1 100644
--- a/source/blender/editors/sculpt_paint/sculpt_smooth.c
+++ b/source/blender/editors/sculpt_paint/sculpt_smooth.c
@@ -395,7 +395,12 @@ void SCULPT_smooth(Sculpt *sd,
 void SCULPT_do_smooth_brush(Sculpt *sd, Object *ob, PBVHNode **nodes, int totnode)
 {
   SculptSession *ss = ob->sculpt;
-  if (ss->cache->bstrength <= 0.0f) {
+
+  /* NOTE: The enhance brush needs to initialize its state on the first brush step. The stroke
+   * strength can become 0 during the stroke, but it can not change sign (the sign is determined
+   * in the beginning of the stroke. So here it is important to not switch to enhance brush in the
+   * middle of the stroke. */
+  if (ss->cache->bstrength < 0.0f) {
     /* Invert mode, intensify details. */
     SCULPT_enhance_details_brush(sd, ob, nodes, totnode);
   }