Fix T80970: Copy Rotation constraint incorrect under shear

Orthogonalize the constraint target's matrix before decomposing it into
Euler angles. This removes sheer, and is actually a requirement for
correct decomposition.

It's conceivable that someone has used the incorrect behaviour in a rig.
As the shear caused unpredictable flipping of the constrained object,
this is unlikely.

Reviewed By: angavrilov, sybren

Differential Revision: https://developer.blender.org/D8915

1 files changed, 8 insertions, 2 deletions

diff --git a/source/blender/blenkernel/intern/constraint.c b/source/blender/blenkernel/intern/constraint.c
index 974d536d83e..bc918df84ed 100644
--- a/source/blender/blenkernel/intern/constraint.c
+++ b/source/blender/blenkernel/intern/constraint.c
@@ -1833,8 +1833,14 @@ static void rotlike_evaluate(bConstraint *con, bConstraintOb *cob, ListBase *tar
     /* To allow compatible rotations, must get both rotations in the order of the owner... */
     mat4_to_eulO(obeul, rot_order, cob->matrix);
     /* We must get compatible eulers from the beginning because
-     * some of them can be modified below (see bug T21875). */
-    mat4_to_compatible_eulO(eul, obeul, rot_order, ct->matrix);
+     * some of them can be modified below (see bug T21875).
+     * Additionally, since this constraint is based on euler rotation math, it doesn't work well with shear.
+     * The Y axis is chosen as the main axis when we orthoganalize the matrix because constraints are
+     * used most commonly on bones. */
+    float mat[4][4];
+    copy_m4_m4(mat, ct->matrix);
+    orthogonalize_m4_stable(mat, 1, true);
+    mat4_to_compatible_eulO(eul, obeul, rot_order, mat);
 
     /* Prepare the copied euler rotation. */
     bool legacy_offset = false;