Fix sculpting/painting with viewport clipping and radial symmetry

This was reported for sculpting, the same is true for weightpaint or
vertexpaint though.

When viewport clipping and radial symmetry are enabled, the
'sculpt_brush_test_clipping()' function was not considering radial
symmetry at all, so if the coordinate was outside the clipping planes,
no action would take place. Now the coordinte is brought back to where
the stroke actually happens and that is checked against clipping.

Since other mirroring options while painting/sculpting (as well as
editmode operations with mirroring) usually still take place even if the
mirrored coord is outside the clipping planes, this should also be the
case for radial symmetry.

This grows the 'SculptBrushTest' struct a bit, but should be acceptable?

Fixes T81466

Maniphest Tasks: T81466

Differential Revision: https://developer.blender.org/D9120

2 files changed, 10 insertions, 0 deletions

diff --git a/source/blender/editors/sculpt_paint/sculpt.c b/source/blender/editors/sculpt_paint/sculpt.c
index 1a2848f1b38..806a9e4b0fb 100644
--- a/source/blender/editors/sculpt_paint/sculpt.c
+++ b/source/blender/editors/sculpt_paint/sculpt.c
@@ -1591,10 +1591,14 @@ void SCULPT_brush_test_init(SculptSession *ss, SculptBrushTest *test)
   if (ss->cache) {
     copy_v3_v3(test->location, ss->cache->location);
     test->mirror_symmetry_pass = ss->cache->mirror_symmetry_pass;
+    test->radial_symmetry_pass = ss->cache->radial_symmetry_pass;
+    copy_m4_m4(test->symm_rot_mat_inv, ss->cache->symm_rot_mat_inv);
   }
   else {
     copy_v3_v3(test->location, ss->cursor_location);
     test->mirror_symmetry_pass = 0;
+    test->radial_symmetry_pass = 0;
+    unit_m4(test->symm_rot_mat_inv);
   }
 
   /* Just for initialize. */
@@ -1622,6 +1626,9 @@ BLI_INLINE bool sculpt_brush_test_clipping(const SculptBrushTest *test, const fl
   }
   float symm_co[3];
   flip_v3_v3(symm_co, co, test->mirror_symmetry_pass);
+  if (test->radial_symmetry_pass) {
+    mul_m4_v3(test->symm_rot_mat_inv, symm_co);
+  }
   return ED_view3d_clipping_test(rv3d, symm_co, true);
 }
 
diff --git a/source/blender/editors/sculpt_paint/sculpt_intern.h b/source/blender/editors/sculpt_paint/sculpt_intern.h
index e6710e27115..84a47ab86ba 100644
--- a/source/blender/editors/sculpt_paint/sculpt_intern.h
+++ b/source/blender/editors/sculpt_paint/sculpt_intern.h
@@ -735,6 +735,9 @@ typedef struct SculptBrushTest {
   float dist;
   int mirror_symmetry_pass;
 
+  int radial_symmetry_pass;
+  float symm_rot_mat_inv[4][4];
+
   /* For circle (not sphere) projection. */
   float plane_view[4];