Fix Edit Face Set operator not being cancelled when the cursor is not over the mesh

If the cursor is not over the mesh, the operator was still using the
last Face Set ID updated by the drawing cursor code when the cursor was
over the mesh.

This now cancels the operator instead of modifying a Face Set that will
look random to the user.

Reviewed By: sergey

Differential Revision: https://developer.blender.org/D9545

1 files changed, 4 insertions, 1 deletions

diff --git a/source/blender/editors/sculpt_paint/sculpt_face_set.c b/source/blender/editors/sculpt_paint/sculpt_face_set.c
index f67c8f701f7..7c8a4c5a857 100644
--- a/source/blender/editors/sculpt_paint/sculpt_face_set.c
+++ b/source/blender/editors/sculpt_paint/sculpt_face_set.c
@@ -1306,7 +1306,10 @@ static int sculpt_face_set_edit_invoke(bContext *C, wmOperator *op, const wmEven
    * tool without brush cursor. */
   SculptCursorGeometryInfo sgi;
   const float mouse[2] = {event->mval[0], event->mval[1]};
-  SCULPT_cursor_geometry_info_update(C, &sgi, mouse, false);
+  if (!SCULPT_cursor_geometry_info_update(C, &sgi, mouse, false)) {
+      /* The cursor is not over the mesh. Cancel to avoid editing the last updated Face Set ID. */
+      return OPERATOR_CANCELLED;
+  }
   const int active_face_set = SCULPT_active_face_set_get(ss);
 
   switch (mode) {